MTS 525-0
Special Topics Research Seminar
Section 20: Generalizing about Message Effects
Spring 2020
SYLLABUS: TOPIC 3 (part 2)
TOPIC 3: The “replication crisis” and affiliated ideas (part 2)
Overall topic 3 outline:
3.1 Crises of replication (especially in psychology)
3.2 Underlying elements
3.3 Toward improved research practices
3.4 Being a vigilant research consumer
part 2 outline:
3.3 Toward improved
research practices
3.3.1 Changes in research culture
3.3.2 Editorial practices: Reporting requirements,
registered reports, etc.
3.3.3 Researcher practices
3.3.4 Alpha reduction/justification
3.4 Being a vigilant
research consumer (and producer)
3.4.1 Statcheck
3.4.2 GRIM (Granularity
Related Inconsistent Means)
3.4.3 PubPeer
3.4.4 General principles
3.3 Toward improved research practices
3.3.1 Changes in research culture
Nosek, B. A., Spies, J. R., & Motyl, M. (2012). Scientific utopia II. Restructuring
incentives and practices to promote truth over publishability.
Perspectives on Psychological Science, 7,
615 – 631. https://doi.org/10.1177/1745691612459058
Nelson, L. D., Simmons, J. P.,
& Simonsohn, U. (2018). Psychology’s renaissance.
Annual Review of Psychology, 69,
511-534. doi:10.1146/annurev-psych-122216-011836
Dienlin, T., Johannes, N., Bowman,
N. D., Masur, P. K., Engesser,
S., Kümpel, A. S., Lukito,
J., Bier, L. M., Zhang, R., Johnson, B. K., Huskey,
R., Schneider, F. M., Breuer, J., Parry, D. A., Vermeulen,
I., Fisher, J. T., Banks, J., Weber, R., Ellis, D. A., Smits, T., Ivory, J. D.,
Trepte, S., McEwan, B., Rinke, E. M., Neubaum, G., Winter, S., Carpenter, C. J., Krämer, N., Utz, S., Unkel, J., Wang, X., Davidson, B. I., Kim, N., Won, A. S., Domahidi, E., Lewis, N. A., de Vreese,
C. (in press). An agenda for open
science in communication. Journal of Communication. https://doi.org/10.1093/joc/jqz052
For further reading:
Koole,
S. L., & Lakens, D. (2012). Rewarding
replications: A sure and simple way to improve psychological science. Perspectives on Psychological Science, 7, 608-614. doi:10.1177/1745691612462586
Levine,
T. R. (2013). A defense of publishing nonsignificant (ns) results. Communication Research Reports, 30,
270-274. doi:10.1080/08824096.2013.806261
McEwan,
B., Carpenter, C. J., & Westerman, D. (2018). On
replication in communication science. Communication
Studies, 69, 235-241. doi:10.1080/10510974.2018.1464938
Lindsay, D.
S. (in press). Seven steps toward transparency and replicability in
psychological science. Canadian
Psychology. Preprint available at PsyArXiv. https://doi.org/10.31234/osf.io/32uz6
3.3.2 Editorial practices: Reporting requirements,
registered reports, etc.
Fidler,
F., Thomason, N., Cumming, G., Finch, S., & Leeman,
J. (2004). Editors can lead researchers to confidence intervals, but can’t make
them think: Statistical reform lessons from medicine. Psychological Science, 15, 119-126.
van 't Veer, A. E., & Giner-Sorolla,
R. (2016). Pre-registration in social psychology: A discussion and suggested
template. Journal of Experimental Social
Psychology, 67, 2-12. doi:10.1016/j.jesp.2016.03.004
Scheel, A. M., Schijen,
M., & Lakens, D. (2020, February 5). An excess of positive results: Comparing the
standard Psychology literature with Registered Reports. PsyArXiv
manuscript. https://doi.org/10.31234/osf.io/p6e9c
For further reading:
Fritz,
A., Scherndl, T., & Kühberger,
A. (2013). A comprehensive review of reporting practices in psychological
journals: Are effect sizes really enough? Theory
and Psychology, 23, 98-122. https://doi.org/10.1177/0959354312436870
Simons, D.
J., Holcombe, A. O., & Spellman, B. A. (2014). An introduction to
registered replication reports at Perspectives
on Psychological Science. Perspectives
on Psychological Science, 9(5), 552-555. doi:10.1177/1745691614543974
Bruce,
R., Chauvin, A., Trinquart, L., Ravaud,
P., & Boutron, I. (2016). Impact of interventions
to improve the quality of peer review of biomedical journals: A systematic
review and meta-analysis. BMC Medicine,
14, 85. doi:10.1186/s12916-016-0631-5
Button, K. S., Bal, L., Clark, A.,
& Shipley, T. (2016). Preventing the ends from justifying the means: Withholding
results to address publication bias in peer-review. BMC Psychology, 4, article no. 59.
https://doi.org/10.1186/s40359-016-0167-7
Findley, M. G., Jensen, N. M., Malesky, E. J., & Pepinsky,
T. B. (2016). Can results-free review reduce publication bias? The results and
implications of a pilot study. Comparative
Political Studies, 49(13), 1667–1703. https://doi.org/10.1177/0010414016655539
Claesen, A., Gomes, S. L. B. T., Tuerlinckx,
F., & Vanpaemel, W. (2019, May 9). Preregistration: Comparing dream to reality.
PsyArXiv manuscript. https://doi.org/10.31234/osf.io/d8wex
Lakens, D. (2019, November 18). The value of preregistration for psychological science: A conceptual
analysis. PsyArXiv manuscript. https://doi.org/10.31234/osf.io/jbh4w
Chambers,
C. (2019). The registered reports revolution: Lessons in cultural reform. Significance, 16, 23-27.
doi:10.1111/j.1740-9713.2019.01299.x
3.3.3 Researcher practices
Wicherts, J. M., Veldkamp, C. L. S., Augusteijn, H.
E. M., Bakker, M., van Aert, R. C. M., & van Assen, M. A. L. M. (2016). Degrees of freedom in planning, running, analyzing,
and reporting psychological studies: A checklist to avoid p-hacking. Frontiers in Psychology, 7, article no.
1832. doi:10.3389/fpsyg.2016.01832
Klein, O., Hardwicke, T. E., Aust, F., Breuer, J., Danielsson,
H., Hofelich Mohr, A., Ijzerman,
H., Nilsonne, G., Vanpaemel,
W., & Frank, M. C. (2018). A practical guide for transparency in
psychological science. Collabra: Psychology, 4(1), 20. http://doi.org/10.1525/collabra.158
For further reading:
Westfall, J., Judd, C. M.,
& Kenny, D. A. (2015). Replicating studies in which samples of participants
respond to samples of stimuli. Perspectives
on Psychological Science, 10, 390-399. doi:10.1177/1745691614564879
LeBel, E. P., & John, L. K. (2017). Toward transparent
reporting of psychological science. In S. O. Lilienfeld
& I. D. Waldman (Eds.), Psychological
science under scrutiny: Recent challenges and proposed solutions (pp. 73-84). John Wiley & Sons.
3.3.4 Alpha reduction/justification
For further reading:
Benjamin,
D. J., Berger, J. O., Johannesson, M., Nosek, B. A., Wagenmakers, E.-J.,
Berk, R., Bollen, K. A., Brembs, B., Brown, L., Camerer,
C., Cesarini, D., Chambers, C. D., Clyde, M., Cook,
T. D., De Boeck, P., Dienes, Z., Dreber,
A., Easwaran, K., Efferson,
C., Fehr, E., Fidler, F., Field, A. P., Forster, M.,
George, E. I., Gonzalez, R., Goodman, S., Green, E., Green, D. P., Greenwald,
A., Hadfield, J. D., Hedges, L. V., Held, L., Ho, T.-H., Hoijtink,
H., Jones, J. H., Hruschka, D. J., Imai, K., Imbens, G., Ioannidis, J. P. A., Jeon, M., Kirchler, M., Laibson, D., List,
J., Little, R., Lupia, A., Machery, E., Maxwell, S.
E., McCarthy, M., Moore, D., Morgan, S. L., Munafó,
M., Nakagawa, S., Nyhan, B., Parker, T. H., Pericchi, L., Perugini, M., Rouder, J., Rousseau, J., Savalei,
V., Schönbrodt, F. D., Sellke,
T., Sinclair, B., Tingley, D., Van Zandt, T., Vazire, S., Watts, D. J., Winship,
C., Wolpert, R. L., Xie,
Y., Young, C., Zinman, J., & Johnson, V. E.
(2018). Redefine statistical significance. Nature Human Behaviour, 2.
6-10. doi:10.1038/s41562-017-0189-z
Crane,
H. (2017). Why “redefining statistical significance” will not improve
reproducibility and could make the replication crisis worse. arXiv manuscript. arXiv:1711.07801v1
Holbert, R.
L., Hardy, B. W., Park, E., Robinson, N. W., Jung, H., Zeng,
C., … Sweeney, K. (2018). Addressing a statistical power-alpha
level blind spot in political- and health-related media research: Discontinuous
criterion power analyses. Annals of the
International Communication Association, 42, 75-92. doi:10.1080/23808985.2018.1459198
Trafimow, D., Amrhein, V., Areshenkoff, C.N., Barrera-Causil,
C., Beh, E.J., Bilgiç, Y.,
Bono, R., Bradley, M.T., Briggs, W.M., Cepeda-Freyre,
H.A., Chaigneau, S.E., Ciocca,
D.R., Carlos Correa, J., Cousineau, D., de Boer,
M.R., Dhar, S.S., Dolgov,
I., Gómez-Benito, J., Grendar, M., Grice, J.,
Guerrero-Gimenez, M.E., Gutiérrez, A., Huedo-Medina, T.B., Jaffe, K., Janyan,
A., Karimnezhad, A., Korner-Nievergelt,
F., Kosugi, K., Lachmair,
M., Ledesma, R., Limongi, R., Liuzza,
M.T., Lombardo, R., Marks, M., Meinlschmidt, G., Nalborczyk, L., Nguyen, H.T., Ospina,
R., Perezgonzalez, J.D., Pfister,
R., Rahona, J.J., Rodríguez-Medina, D.A., Romão, X., Ruiz-Fernández, S.,
Suarez, I., Tegethoff, M., Tejo,
M., van de Schoot, R., Vankov,
I., Velasco-Forero, S., Wang, T., Yamada, Y., Zoppino, F.C., & Marmolejo-Ramos,
F. (2018). Manipulating the alpha level cannot cure significance testing. Frontiers in Psychology, 9, 699. Available at https://doi.org/10.3389/fpsyg.2018.00699
Lakens, D., Adolfi, F. G.,
Albers, C. J., Anvari, F., Apps, M. A. J., Argamon, S. E., … Zwaan, R. A.
(2018). Justify your alpha. Nature Human
Behaviour, 2, 168-171. doi:10.1038/s41562-018-0311-x
Miller,
J., & Ulrich, R. (2019). The quest for an optimal alpha. PLoS ONE, 14, e0208631. doi:10.1371/journal.pone.0208631
3.4 Being a vigilant research consumer (and
producer)
3.4.1 Statcheck
Nuijten, M. B., Hartgerink, C. H. J., van Assen,
M. A. L. M., Epskamp, S., & Wicherts,
J. M. (2016). The prevalence of statistical reporting errors in psychology
(1985-2013). Behavior Research Methods, 48, 1205-1226. doi:10.3758/s13428-015-0664-2
https://mbnuijten.com/statcheck/
3.4.2 GRIM (Granularity
Related Inconsistent Means)
Brown, N. J. L., & Heathers, J. A. J. (2017). The GRIM
test: A simple technique detects numerous anomalies in the reporting of results
in psychology. Social Psychological and
Personality Science, 8, 363-369. doi:10.1177/1948550616673876
http://www.prepubmed.org/general_grim/
http://www.prepubmed.org/grim_test/
3.4.3 PubPeer
https://pubpeer.com/static/extensions
3.4.4 General principles
Sainani, K. L. (2020), How to be a
statistical detective. PM&R, 12(2), 211-215. doi:10.1002/pmrj.12305
For further reading:
Brown, A.
W., Kaiser, K. A., & Allison, D. B. (2018). Issues with data and analyses. PNAS, 115(11), 2563-2570. doi:10.1073/pnas.1708279115