MTS 525-0
 Special Topics Research Seminar

Section 20: Generalizing about Message Effects
 Spring 2020

SYLLABUS: TOPIC 3 (part 1)

TOPIC 3: The “replication crisis” and affiliated ideas (part 1)


Overall topic 3 outline:

3.1  Crises of replication (especially in psychology)

3.2  Underlying elements

3.3  Toward improved research practices

3.4  Being a vigilant research consumer



Part 1 outline:

3.1  Crises of replication (especially in psychology)

            3.1.1  Precipitating events

            3.1.2  Fallout

3.2  Underlying elements

            3.2.1  Low statistical power

            3.2.2  Weaknesses of peer review

            3.2.3  Questionable research practices

            3.2.4  Lack of replication5

            3.2.4  Errors (numerical and otherwise)






3.1  Crises of replication (especially in psychology)


3.1.1  Precipitating events


Ioannidis, J. P. A. (2005). Why most published research findings are false. PLoS Medicine, 2, 696-701. doi: 10.1371/journal.pmed.0020124


For further reading:

            Ioannidis, J. P. (2005). Contradicted and initially stronger effects in highly cited clinical research. JAMA, 294, 218-228. doi: 10.1001/jama.294.2.218

            Ioannidis, J. P. A. (2008). Why most discovered true associations are inflated. Epidemiology, 19, 640-648. doi: 10.1097/EDE.0b013e31818131e7


            Bem, D. J. (2011). Feeling the future: Experimental evidence for anomalous retroactive influences on cognition and affect. Journal of Personality and Social Psychology, 100, 407-425.

            Wagenmakers, E. J., Wetzels, R., Borsboom, D., & van der Maas, H. L. (2011). Why psychologists must change the way they analyze their data: The case of psi: Comment on Bem 2011. Journal of Personality and Social Psychology, 100, 426–432.

            Galak, J., LeBoeuf, R. A., Nelson, L. D., Simmons, J. P. (2012). Correcting the past: Failures to replicate ψ. Journal of Personality and Social Psychology, 103, 933–948.


            Bargh, J. A., Chen, M., & Burrows, L. (1996). Automaticity of social behavior: Direct effects of trait construct and stereotype priming on action. Journal of Personality and Social Psychology, 71, 230-244.

            Doyen, S., Klein, O., Pichon, C. L., & Cleeremans, A. (2012). Behavioral priming: It’s all in the mind, but whose mind? PLoS One, 7, e29081. 

            Srivastava, S. (2012). Some reflections on the Bargh-Doyen elderly walking priming brouhaha. Blog post.




3.1.2  Fallout


Pashler, H., & Harris, C. R. (2012). Is the replicability crisis overblown? Three arguments examined. Perspectives on Psychological Science, 7, 531-536. doi:10.1177/1745691612463401


Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349, no. 6251. doi:10.1126/science.aac4716



For further reading:

            Chang, A. C., & Li, P. (2015). Is economics research replicable? Sixty published papers from thirteen journals say ”usually not.” Finance and Economics Discussion Series 2015-083. Washington: Board of Governors of the Federal Reserve System. doi:10.17016/FEDS.2015.083

            Camerer, C. F., Dreber, A., Forsell, E., Ho, T.-H., Huber, J., Johannesson, M., Kirchler, M., Almenberg, J., Altmejd, A., Chan, T., Heikensten, E., Holzmeister, F., Imai, T., Isaksson, S., Nave, G., Pfeiffer, T., Razen, M., & Wu, H.. (2016). Evaluating replicability of laboratory experiments in economics. Science, 351, 1433-1436. doi:10.1126/science.aaf0918

            Lilienfeld, S. O., & Waldman, I. D. (Eds.). (2017). Psychological science under scrutiny: Recent challenges and proposed solutions. Malden, MA: Wiley Blackwell.





3.2  Underlying elements


3.2.1  Low statistical power


Cohen, J. (1962). The statistical power of abnormal-social psychological research. Journal of Abnormal and Social Psychology, 65, 145-153.  doi:10.1037/h0045186


Sedlmeier, P., & Gigerenzer, G. (1989). Do studies of statistical power have an effect on the power of studies? Psychological Bulletin, 105(2), 309–316. doi:10.1037/0033-2909.105.2.309


Christley, R. M. (2010). Power and error: Increased risk of false positive results in underpowered studies. The Open Epidemiology Journal, 3, 16-19. doi:10.2174/1874297101003010016


For further reading: 

            Chase, L. J., & Tucker, R. K. (1975). A power-analytic examination of contemporary communication research. Communication Monographs, 42, 29-41. doi:10.1080/03637757509375874 

            Garrison, J. P., & Andersen, P. A. (1979). A reassessment of statistical power analysis in Human Communication Research. Human Communication Research, 5, 343–350. doi:10.1111/j.1468-2958.1979.tb00647.x 

            Zaller, J. (2002). The statistical power of election studies to detect media exposure effects in political campaigns. Electoral Studies, 21, 297-329. doi:10.1016/S0261-3794(01)00023-3

            Maxwell, S. E. (2004). The persistence of underpowered studies in psychological research: Causes, consequences, and remedies. Psychological Methods, 9, 147-163.

            Button, K. S., Ioannidis, J. P. A., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S. J., & Munafo, M. R. (2013). Power failure: Why small sample size undermines the reliability of neuroscience. Nature Reviews Neuroscience, 14, 365-376. doi:10.1038/nrn3475

            Szucs, D., & Ioannidis, J. P. A. (2017). Empirical assessment of published effect sizes and power in the recent cognitive neuroscience and psychology literature. PLoS Biology, 15, e2000797. doi:10.1371/journal.pbio.2000797 

            Ioannidis, J. P. A., Stanley, T. D., & Doucouliagos, H. (2017). The power of bias in economics research. The Economic Journal, 127, F236-F265.  doi:10.1111/ecoj.12461 

            Nord, C. L., Valton, V., Wood, J., & Roiser, J. P. (2017). Power-up: A reanalysis of 'power failure' in neuroscience using mixture modeling. Journal of Neuroscience, 37, 8051-8061. doi:10.1523/JNEUROSCI.3592-16.2017 

            Elson, M., & Przybylski, A. K. (2017). The science of technology and human behavior: Standards, old and new. Journal of Media Psychology, 29, 1–7. doi:10.1027/1864-1105/a000212 

            Brodeur, A., Cook, N., & Heyes, A. (2018). Methods matter: P-hacking and causal inference in economics. IZA Institue of Labor Economics Discussion Paper 11796.


For further reading: misunderstanding of statistical power

            Hoenig, J. M., & Heisey, D. M. (2001). The abuse of power: The pervasive fallacy of power calculations for data analysis. The American Statistician, 55, 19-24.

            O’Keefe, D. J. (2007). Post hoc power, observed power, a priori power, retrospective power, prospective power, achieved power: Sorting out appropriate uses of statistical power analyses. Communication Methods and Measures, 1, 291-299. doi:10.1080/19312450701641375

            Wang, L. L. (2010). Retrospective statistical power: Fallacies and recommendations. Newborn and Infant Nursing Reviews, 10, 55-59. doi:10.1053/j.nainr.2009.12.012 



3.2.2  Weaknesses of peer review


For further reading:

            Sterling, T . D. (1959). Publication decision and the possible effects on inferences drawn from tests of significance-or vice versa. Journal of the American Statistical Association, 54(285), 30-34. doi:10.1080/01621459.1959.10501497 

            Mahoney, M. J. (1977). Publication prejudices: An experimental study of confirmatory bias in the peer review system. Cognitive Therapy and Research, 1, 161–175.

            Peters, D. P., & Ceci, S. J. (1982). Peer-review practices of psychological journals: The fate of published articles, submitted again. Behavioral and Brain Sciences, 5(2), 187-195. doi:10.1017/S0140525X00011183 

            Ernst, E., & Resch, K. L. (1994). Reviewer bias – a blinded experimental study. Journal of Laboratory and Clinical Medicine, 124, 178-182.

            Sterling, T. D., Rosenbaum, W. L., & Weinkam, J. J. (1995). Publication decisions revisited: The effect of the outcome of statistical tests on the decision to publish and vice versa. The American Statistician, 49, 108-112. 







3.2.3  Questionable research practices: Selective publication, p-hacking, etc.


Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22, 1359-1366. doi:10.1177/0956797611417632


For further reading: 

            Rosenthal, R. (1979). The “file drawer problem” and tolerance for null results. Psychological Bulletin, 86, 638‑641. doi:10.1037/0033-2909.86.3.638

            Kerr, N. L. (1998). HARKing: Hypothesizing after the results are known. Personality and Social Psychology Review, 2, 196-217.  doi:10.1207/s15327957pspr0203_4

            Chan, A. W., Hrobjartsson, A., Haahr, M. T., Gøtzsche, P. C., & Altman, D. G. (2004). Empirical evidence for selective reporting of outcomes in randomized trials: Comparison of protocols to published articles.  JAMA, 291, 2457-2465. doi:10.1001/jama.291.20.2457

            Chan, A. W., Krleza-Jeric´, K., Schmid, I., & Altman, D. G. (2004). Outcome reporting bias in randomized trials funded by the Canadian Institutes of Health Research. CMAJ, 171, 735-740. doi: 10.1503/cmaj.1041723

            Gerber, A., & Malhotra, N. (2008). Do statistical reporting standards affect what is published? Publication bias in two leading political science journals. Quarterly Journal of Political Science, 3, 313-326. 10.1561/100.00008024

            Gerber, A. S., & Malhotra, N. (2008). Publication bias in empirical sociological research: Do arbitrary significance levels distort published reports? Sociological Methods & Research, 37, 3-30. 10.1177/0049124108318973

            Turner, E. H., Matthews, A. M., Linardatos, E., Tell, R. A., & Rosenthal, R. (2008). Selective publication of antidepressant trials and its influence on apparent efficacy. New England Journal of Medicine, 358, 252-260.

            Dwan, K., Altmanm D. G., Arnaiz, J. A., Bloom, J., Chan, A.-W., Cronin, E. Decullier, E., Easterbrook, P. J., Von Elm, E., Gamble, C., Ghersi, D. Ioannidis, J. P. A., Simes, J., & Williamson, P. R. (2008). Systematic review of the empirical evidence of study publication bias and outcome reporting bias.  PLoS One, 3(8), e3081.

            Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2009). Publication bias. Chapter 30 (pp. 277-292) in: Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2009). Introduction to meta-analysis. Chichester, West Sussex, UK: Wiley.

            Sutton, A. J. (2009). Publication bias. In H. Cooper, L. V. Hedges, & J. C. Valentine (Eds.), The handbook of research synthesis and meta-analysis (2nd ed., pp. 435-452). New York: Russell Sage Foundation.

            Bertamini, M., & Munafò, M. (2012). Bite-size science and its undesired side effects. Perspectives on Psychological Science, 7, 67-71.

            John, L. K., Loewenstein, G., & Prelec. D. (2012). Measuring the prevalence of questionable research practices with incentives for truth telling. Psychological Science, 23, 524-532. 10.1177/0956797611430953

            Franco, A., Malhotra, N., & Simonovits, G. (2014). Publication bias in the social sciences: Unlocking the file drawer. Science, 345(6203), 1502-1505. doi:10.1126/science.1255484

            Vermeulen, I., Beukeboom, C. J., Batenburg, A., Avramiea, A., Stoyanov, D., van de Velde, B., & Oegema, D. (2015). Blinded by the light: How a focus on statistical “significance” may cause p-value misreporting and an excess of p-values just below .05 in communication science. Communication Methods and Measures, 9, 253-279. doi:10.1080/19312458.2015.1096333 

            Seaman, C. S., & Weber, R. (2015). Undisclosed flexibility in computing and reporting structural equation models in communication science. Communication Methods and Measures, 9, 208-232. doi:10.1080/19312458.2015.1096329 

            Vermeulen, I. E., & Hartmann, T. (2015). Questionable research and publication practices in communication science. Communication Methods & Measures, 9, 189–192. doi:10.1080/19312458.2015.1096331

            Matthes, J., Marquart, F., Naderer, B., Arendt, F., Schmuck, D., & Adam, K. (2015). Questionable research practices in experimental communication research: A systematic analysis from 1980 to 2013. Communication Methods and Measures, 9, 193-207. doi:10.1080/19312458.2015.1096334 

            Head, M. L., Holman, L., Lanfear, R., Kahn, A. T., & Jennions, M. D. (2015). The extent and consequences of p-hacking in science. PLoS Biology, 13(3), e1002106. 

            Franco, A., Malhotra, N., & Simonovits, G. (2015). Underreporting in political science survey experiments: Comparing questionnaires to published results. Political Analysis, 23, 306-312. doi:10.1093/pan/mpv006 

            Franco, A., Malhotra, N., & Simonovits, G. (2016). Underreporting in psychology experiments: Evidence from a study registry.  Social Psychological and Personality Science, 7(1), 8-12.  doi:10.1177/1948550615598377 

            Renkewitz, F., & Keiner, M. (2019). How to detect publication bias in psychological research. Zeitschrift für Psychologie, 227(4), 261-279.  doi:10.1027/2151-2604/a000386

            Cairo, A. H., Green, J. D., Forsyth, D. R., Behler, A. M. C., & Raldiris, T. L. (in press). Gray (literature) matters: Evidence of selective hypothesis reporting in social psychological research. Personality and Social Psychology Bulletin.  doi:10.1177/0146167220903896 


For further reading about effect size and sample size relationships:

            Gerber, A. S., Green, D. P., & Nickerson, D. (2001). Testing for publication bias in political science. Political Analysis, 9(4), 385-392.

            Levine, T., Asada, K. J., & Carpenter, C. (2009). Sample sizes and effect sizes are negatively correlated in meta-analyses: Evidence and implications of a publication bias against non-significant findings. Communication Monographs, 76, 286-302. doi:10.1080/03637750903074685

            Kühberger, A., Fritz, A., & Scherndl, T. (2014). Publication bias in psychology: A diagnosis based on the correlation between effect size and sample size. PLoS One, 9, e105825. doi:10.1371/journal.pone.0105825


3.2.4  Lack of replications


Easley, R. W., Madden, C. S., & Dunn, M. G. (2000). Conducting marketing science: The role of replications in the research process. Journal of Business Research, 48, 83-92. doi:10.1016/S0148-2963(98)00079-4 


For further reading:

            Smith, N. C., Jr. (1970). Replication studies: A neglected aspect of psychological research. American Psychologist, 25, 970-975. doi:10.1037/h0029774

            Reid, L. N., Soley, L. C., & Wimmer, R. D. (1981). Replication in advertising research: 1977, 1978, 1979. Journal of Advertising, 10(1), 3–13. doi:10.1080/00913367.1981.10672750 

            Neuliep, J. W. (Ed.). (1990). Handbook of replication studies in the social and behavioral sciences. Newbury Park, CA: Sage.

            Hunter, J. E. (2001). The desperate need for replications. Journal of Consumer Research, 28, 149-158. doi:10.1086/321953

            Evanschitzky, H., Baumgarth, C., Hubbard, R., & Armstrong, J. S. (2007). Replication research’s disturbing trend. Journal of Business Research, 60, 411-415. doi:10.1016/j.jbusres.2006.12.003





3.2.5  Errors (numerical and otherwise) 


García-Berthou, E., & Alcaraz, C. (2004). Incongruence between test statistics and P values in medical papers. BMC Medical Research Methodology, 4, article no. 13.


For further reading:

            Dewald, W. G., Thursby, J. G., & Anderson, R. G. (1986). Replication in empirical economics: The Journal of Money, Credit and Banking project. American Economic Review, 76, 587-603.  

            Edouard, L., & Senthilselvan, A. (1997). Observer error and birthweight: Digit preference in recording. Public Health, 111(2), 77-79.  doi:10.1016/s0033-3506(97)90004-4 

            Bakker, M., & Wicherts, J. M. (2011). The (mis)reporting of statistical results in psychology journals. Behavior Research Methods, 43, 666–678. doi:10.3758/s13428-011-0089-5

            Elson, M., & Przybylski, A. K. (2017). The science of technology and human behavior: Standards, old and new. Journal of Media Psychology, 29, 1–7. doi:10.1027/1864-1105/a000212 

            Brown, A. W., Kaiser, K. A., & Allison, D. B. (2018). Issues with data and analyses. PNAS, 115(11), 2563-2570.  doi:10.1073/pnas.1708279115 





Home page for this course
Daniel J. O'Keefe home page
Daniel J. O'Keefe email