MTS 525-0
 Special Topics Research Seminar

Section 20: Generalizing about Message Effects
 Spring 2020



TOPIC 6:  Message design guidance: Variations on, and alternatives to, RCT-based generalization



6.1  Randomized controlled trials (RCTs): Variations and alternatives

            6.1.1  Multiphase optimization strategy (MOST), sequential multiple assignment randomized trials (SMART), etc.

            6.1.2  Pragmatic trials

            6.1.3  Effectiveness-implementation designs

6.2  Assessment of the messages of interest

            6.2.1  Summary discussions of message pretesting methods

            6.2.2  Message properties

                Armstrong’s Persuasive Principles Index (PPI)

                Other index-based assessments of message properties

            6.2.3  Message perceptions

                Perceived message effectiveness

                Discrete choice experiments

                Message liking

            6.2.4  Message effects

                Familiar experimental arrangements and outcomes

                Neural responses

                A/B testing

                ARS persuasion scores




6.1  Randomized controlled trials (RCTs): Variations and alternatives


6.1.1  Multiphase optimization strategy (MOST), sequential multiple assignment randomized trials (SMART), etc.


Collins, L. M., Murphy, S. A., & Strecher, V. (2007). The multiphase optimization strategy (MOST) and the sequential multiple assignment randomized trial (SMART): New methods for more potent eHealth interventions. American Journal of Preventive Medicine, 32(Suppl), S112-118. doi:10.1016/j.amepre.2007.01.022 


For further reading:

            Plackett, R. L., & Burman, J. P. (1946). The design of optimum multifactorial experiments. Biometrika, 33, 305-325.

            Bell, G. H., Ledolter, J., & Swersey, A. J. (2006). Experimental design on the front lines of marketing: Testing new ideas to increase direct mail sales. International Journal of Research in Marketing, 23, 309-319.

            Nair, V., Strecher, V., Fagerlin, A., Ubel, P., Resnicow, K., Murphy, S., Little, R., Charkraborty, B., & Zhang, A. J. (2008). Screening experiments and the use of fractional factorial designs in behavioral intervention research. American Journal of Public Health, 98, 1354-1358.

            Collins, L. M., Dziak, J. J., & Li, R. Z. (2009). Design of experiments with multiple independent variables: a resource management perspective on complete and reduced factorial designs. Psychological Methods, 14(3), 202-224. doi: 10.1037/a0015826  

            Chakraborty, B., Collins, L. M., Strecher, V. J., & Murphy, S. A. (2009). Developing multicomponent interventions using fractional factorial designs. Statistics in Medicine, 28, 2687-2708.

            Brown, C. H., Ten Have, T. R., Jo, B., Dagne, G., Wyman, P. A., Muthen, B., & Gibbons, R. D. (2009). Adaptive designs for randomized trials in public health. Annual Review of Public Health, 30, 1-25.

            Almirall, D., Compton, S. N., Gunlicks-Stoessel, M., Duan, N., & Murphy, S. A. (2012). Designing a pilot sequential multiple assignment randomized trial for developing an adaptive treatment strategy. Statistics in Medicine, 31, 1887-1902. doi:10.1002/sim.4512 

            Collins, L. M., Trail, J. B., Kugler, K. C., Baker, T. B., Piper, M. E., & Mermelstein, R. J. (2014). Evaluating individual intervention components: Making decisions based on the results of a factorial screening experiment. Translational Behavioral Medicine, 4, 238-251. doi:10.1007/s13142-013-0239-7 

            Collins, L. M. (2018). Optimization of behavioral, biobehavioral, and biomedical interventions: The multiphase optimization strategy (MOST). Springer.

            Collins, L. M., & Kugler, K. (Eds.). (2018). Optimization of behavioral, biobehavioral, and biomedical interventions: Advanced topics. Springer.

            Miller, L. C., Shaikh, S. J., Jeong, D. C., Wang, L., Gillig, T. K., Godoy, G. G., Appleby, P. R., Corsbie-Massay, C. L., Marsella, S., Christensen, J. L., & Read, S. J..(2019). Causal inference in generalizable environments: Systematic representative design. Psychological Inquiry, 30(4), 173-202.  doi:10.1080/1047840X.2019.1693866 

            Michela, A., van Rooij, M. M. J. W., Klumpers, F., van Peer, J. M., Roelofs, K., & Granic, I. (2019). Reducing the noise of reality. Psychological Inquiry, 30(4), 203-210. doi:10.1080/1047840X.2019.1693872 



6.1.2  Pragmatic trials


For further reading:

            Schwartz, D., & Lellouch, J. (1967). Explanatory and pragmatic attitudes in therapeutic trials. Journal of Chronic Diseases, 20, 637-648.  doi:10.1016/0021-9681(67)90041-0 

            Patsopoulos, N. A. (2011). A pragmatic view on pragmatic trials. Dialogues in Clinical Neuroscience, 13(2), 217–224.

            Ford, I., & Norrie, J. (2016). Pragmatic trials. New England Journal of Medicine, 375, 454-463. doi:10.1056/NEJMra1510059 

            Troxel, A. B., Asch, D. A., & Volpp, K. G. (2016). Statistical issues in pragmatic trials of behavioral economic interventions. Clinical Trials, 13, 478-483. doi:10.1177/1740774516654862 

            Zuidgeest, M. G. P., Goetz, I., Groenwold, R. H. H., Irving, E., van Thiel, G. J. M. W., Grobbee, D. E., & GetReal Work Package 3. (2017). Series: Pragmatic trials and real world evidence: Paper 1. Introduction. Journal of Clinical Epidemiology, 88, 7-13. doi:10.1016/j.jclinepi.2016.12.023 



6.1.3  Effectiveness-implementation designs


For further reading:

            Curran, G., Bauer, M., Mittman, B., Pyne, J., & Stetler, C. (2012). Effectiveness-implementation hybrid designs: Combining elements of clinical effectiveness and implementation research to enhance public health impact. Medical Care, 50, 217-226. doi:10.1097/MLR.0b013e3182408812 

            Wolfenden, L., Williams, C. M., Wiggers, J., Nathan, N., &Yoong, S. L. (2016). Improving the translation of health promotion interventions using effectiveness–implementation hybrid designs in program evaluations. Health Promotion Journal of Australia, 27(3), 204–207.

            Luszczynska, A. (2020) It’s time for effectiveness-implementation hybrid research on behaviour change. Health Psychology Review, 14(1), 188-192. doi:10.1080/17437199.2019.1707105 


6.2  Assessment of the messages of interest


6.2.1  Summary discussions of message pretesting methods


For further reading:

            Bertrand, J. T. (1978). Communications pretesting.  Chicago, IL: University of Chicago Community and Family Study Center.

            Stewart, D. W., Furse, D. H., & Kozak, R. P. (1983). A guide to commercial copytesting services. Current Issues and Research in Advertising, 6, 1–43.  doi:10.1080/01633392.1983.10505330

            Pechmann, C., & Andrews, J. C. (2011). Copy test methods to pretest advertisements. In J. N. Sheth & N. K. Malhotra (Eds.), Wiley international encyclopedia of marketing. Chichester, West Sussex, UK: Wiley.  doi:10.1002/9781444316568.wiem04007

            Willoughby, J. F., & Furberg, R. (2015). Underdeveloped or underreported? Coverage of pretesting practices and recommendations for design of text message–based health behavior change interventions. Journal of Health Communication, 20, 472-478. doi:10.1080/10810730.2014.977468



6.2.2  Message properties  Armstrong’s Persuasive Principles Index (PPI)


Armstrong, J. S., Du, R., Green, K. C., & Graefe, A. (2016). Predictive validity of evidence-based persuasion principles: An application of the index method. European Journal of Marketing, 50, 276-292. doi:10.1108/EJM-10-2015-0728


For further reading:

            Armstrong, J. S., & Patnaik, S. (2009). Using quasi-experimental data to develop empirical generalizations for persuasive advertising. Journal of Advertising Research, 49, 170-175.  doi:10.2501/S0021849909090230 

            Armstrong, J. S. (2010). Persuasive advertising: Evidence-based principles. New York: Palgrave Macmillan.

            Armstrong, J. S. (2011). Evidence-based advertising: An application to persuasion. International Journal of Advertising, 30, 743-767. doi:10.2501/IJA-30-5-743-767

            O’Keefe, D. J. (2016). Evidence-based advertising using persuasion principles. European Journal of Marketing, 50, 294-300. doi:10.1108/EJM-11-2015-0801 

            Sharp, B., & Hartnett, N. (2016). Generalisability of advertising persuasion principles. European Journal of Marketing, 50, 301-305. doi:10.1108/EJM-12-2015-0842

            Woodside, A. G. (2016). Predicting advertising execution effectiveness: Scale development and validation. European Journal of Marketing, 50, 306-311. doi:10.1108/EJM-11-2015-0809 

            Wright, M. J. (2016). Predicting what? The strengths and limitations of a test of persuasive advertising principles. European Journal of Marketing, 50, 312-316. doi:10.1108/EJM-12-2015-0833

            Green, K. C., Armstrong, J. S., Du, R., & Graefe, A. (2016). Persuasion Principles Index: Ready for pretesting advertisements. European Journal of Marketing, 50, 317-326. doi:10.1108/EJM-12-2015-0838 

            Pereira, J. J. S. (2018).  A ciência da publicidade : Conhecimento intuitivo e uso de princípios de mudança comportamental por especialistas para influenciar consumidores  [The science of advertising: Experts’ intuitions and usage of behavioral change principles to influence consumers] (doctoral dissertation, University of Brasilia).  Other index-based assessments of message properties


For further reading:

            Paul, C. L., Redman, S., & Sanson-Fisher, R. W. (1997). The development of a checklist of content and design characteristics for printed health education materials. Health Promotion Journal of Australia, 7(3), 153-159.

            Cole, G. E., Keller, P. A., Reynolds, J., Schaur, M., & Krause, D. (2016). CDC MessageWorks: Designing and validating a social marketing tool to craft and defend effective messages. Social Marketing Quarterly, 22(1), 3-18. doi:10.1177/1524500415614817

            Wildeboer, G., Kelders, S. M., & van Gemert-Pijnen, J. E. (2016). The relationship between persuasive technology principles, adherence and effect of web-based interventions for mental health: A meta-analysis. International Journal of Medical Informatics, 96, 71-85. doi:10.1016/j.ijmedinf.2016.04.005 

            Baumel, A., Faber, K., Mathur, N., Kane, J. M., & Muench, F. (2017). Enlight: A comprehensive quality and therapeutic potential evaluation tool for mobile and web-based eHealth interventions. Journal of Medical Internet Research, 19, e82. doi:10.2196/jmir.7270   

            Lim, K., Kilpatrick, C., Storr, J., & Seale, H. (2018). Exploring the use of entertainment-education YouTube videos focused on infection prevention and control. American Journal of Infection Control, 46, 1218-1223. doi:10.1016/j.ajic.2018.05.002

            Huhmann, B. A., & Albinsson, P. A. (2019). Assessing the usefulness of taxonomies of visual rhetorical figures. Journal of Current Issues & Research in Advertising, 40, 171-195. doi:10.1080/10641734.2018.1503106





6.2.3  Message perceptions  Perceived message effectiveness


Dillard, J. P., Weber, K. M., & Vail, R. G. (2007). The relationship between the perceived and actual effectiveness of persuasive messages: A meta-analysis with implications for formative campaign research. Journal of Communication, 57, 613-631. doi:10.1111/j.1460-2466.2007.00360.x


O’Keefe, D. J. (2018). Message pretesting using assessments of expected or perceived persuasiveness: Evidence about diagnosticity of relative actual persuasiveness. Journal of Communication, 68, 120-142. doi:10.1093/joc/jqx009 


O’Keefe, D. J. (2020). Message pretesting using perceived persuasiveness measures: Reconsidering the correlational evidence. Communication Methods and Measures, 14(1), 25-37. doi:10.1080/19312458.2019.1620711


For further reading:

            Dillard, J. P., Shen, L., & Vail, R. G. (2007). Does perceived message effectiveness cause persuasion or vice versa? 17 consistent answers. Human Communication Research, 33, 467-488. doi:10.1111/j.1468-2958.2007.00308.x 

            Yzer, M., LoRusso, S., & Nagler, R. H. (2015). On the conceptual ambiguity surrounding perceived message effectiveness. Health Communication, 30, 125-134. doi:10.1080/10410236.2014.974131

            Meschtscherjakov, A., Gärtner, M., Mirnig, A., Rödel, C., & Tscheligi, M. (2016). The Persuasive Potential Questionnaire (PPQ): Challenges, drawbacks, and lessons learned. In A. Meschtscherjakov, B. De Ruyter, V. Fuchsberger, M. Murer, & M. Tscheligi (Eds.), Persuasive technology: 11th international conference, PERSUASIVE 2016 (pp. 162-175). Cham, Switzerland: Springer. [LNCS (Lecture Notes in Computer Science) vol. 9638]

            Noar, S. M., Barker, J., & Yzer, M. (2018). Measurement and design heterogeneity in perceived message effectiveness studies: A call for research. Journal of Communication, 68, 990-993. doi:10.1093/joc/jqy047

            Cappella, J. N. (2018). Perceived message effectiveness meets the requirements of a reliable, valid, and efficient measure of persuasiveness. Journal of Communication, 68, 994-997.  doi:10.1093/joc/jqy044 

            Davis, K. C., & Duke, J. C. (2018). Evidence of the real-world effectiveness of public health media campaigns reinforces the value of perceived message effectiveness in campaign planning. Journal of Communication, 68, 998-1000.  doi:10.1093/joc/jqy045

            O’Keefe, D. J. (2018). Whistling past the graveyard: Response to commentaries. Journal of Communication, 68, 1001-1005. doi:10.1093/joc/jqy046

            Noar, S. M., Bell, T., Kelley, D., Barker, J., & Yzer, M. C. (2018). Perceived message effectiveness measures in tobacco education campaigns: A systematic review. Communication Methods and Measures, 12, 295-313. doi:10.1080/19312458.2018.1483017 

            Baig, S. A., Noar, S. M., Gottfredson, N. C., Boynton, M. H., Ribisl, K. M., & Brewer, N. T. (2019). UNC perceived message effectiveness: Validation of a brief scale. Annals of Behavioral Medicine, 53(8), 732-742.  doi:10.1093/abm/kay080 

            Thomas, R. J., Masthoff, J., & Oren, N. (2019). Can I influence you? Development of a scale to measure perceived persuasiveness and two studies showing the use of the scale. Frontiers in Artificial Intelligence, 2.  doi:10.3389/frai.2019.00024 

            Kim, M., & Cappella, J. N. (2019). Reliable, valid and efficient evaluation of media messages: Evaluation protocol for effectiveness and empirical evidence. Journal of Communication Management, 23, 179-197.  doi:10.1108/JCOM-12-2018-0132 

            Noar, S. M., Barker, J., Bell, T., & Yzer, M. C. (2020). Does perceived message effectiveness predict the actual effectiveness of tobacco education messages? A systematic review and meta-analysis. Health Communication, 35(2), 148-157. doi:10.1080/10410236.2018.1547675  Discrete choice experiments


Thrasher, J. F. Anshari, D., Lambert-Jessup, V., Islam, F., Mead, E., Popova, L., Salloum, R., Moodie, C., Louviere, J., & Lindblom, E. N. (2018). Assessing smoking cessation messages with a discrete choice experiment. Tobacco Regulatory Science, 4, 73-87. doi:10.18001/TRS.4.2.7


For further reading:

            Promberger, M., Dolan, P., & Marteau, T. M. (2012). “Pay them if it works”: Discrete choice experiments on the acceptability of financial incentives to change health related behaviour. Social Science and Medicine, 75(12), 2509-2514. doi:10.1016/j.socscimed.2012.09.033 

            Clark, M., D., Determann, D., Petrou, S., Moro, D., & de Bekker-Grob, E. W. (2014). Discrete choice experiments in health economics: A review of the literature. PharmacoEconomics, 32(9), 883-902. doi:10.1007/s40273-014-0170-x 

            Janssen, E. M., Marshall, D. A., Hauber, A. B., & Bridges, J. F. P. (2017). Improving the quality of discrete-choice experiments in health: How can we assess validity and reliability? Expert Review of Pharmacoeconomics & Outcomes Research, 17(6), 531-542, doi:10.1080/14737167.2017.1389648 

            Crutchfield, T. M., & Kistler, C. E. (2017). Getting patients in the door: Medical appointment reminder preferences. Patient Preference and Adherence, 11, 141-150. doi:10.2147/ppa.s117396 

            de Bekker-Grob, E. W., Veldwijk, J., Jonker, M., Donkers, B., Huisman, J., Buis, S., Swait, J., Lancsar, E., Witteman, C. L. M., Bonsel, G., & Bindels, P. (2018). The impact of vaccination and patient characteristics on influenza vaccination uptake of elderly people: A discrete choice experiment. Vaccine, 36(11), 1467-1476. doi: 10.1016/j.vaccine.2018.01.054

            Terris-Prestholt, F., Neke, N., Grund, J. M., Plotkin, M., Kuringe, E., Osaki, H., Ong, J. J., Tucker, J. D., Mshana, G., Mahler, H., Weiss, H. A., Wambura, H., & The VMMC study team. (2019). Using discrete choice experiments to inform the design of complex interventions. Trials, 20(1), 157. doi:10.1186/s13063-019-3186-x 

            McGrady, M. E., Pai, A. L. H., & Prosser, L. A. (in press). Using discrete choice experiments to develop and deliver patient-centered psychological interventions: A systematic review. Health Psychology Review.  doi:10.1080/17437199.2020.1715813  Message liking


For further reading: 

            Haley, R. I., & Baldinger, A. L. (1991). The ARF Copy Research Validity Project. Journal of Advertising Research, 31(2), 11–32.

            Rossiter, J. R., & Eagleson, G. (1994). Conclusions from the ARF’s Copy Research Validity Project. Journal of Advertising Research, 34(3), 19-32.

            Haley, R. I. (1994). A rejoinder to “Conclusions from the ARF’s Copy Research Validity Project.” Journal of Advertising Research, 34(3), 33. 

            Hollis, N. S. (1995). Like it or not, liking is not enough. Journal of Advertising Research, 35(5), 7-16. 

            Bergkvist, L., & Rossiter, J. R. (2008). The role of ad likability in predicting an ad’s campaign performance. Journal of Advertising, 37(2), 85-97.




6.2.4  Message effects  Familiar experimental arrangements and outcomes


Judah, G., Aunger, R., Schmidt, W. P., Michie, S., Granger, S., & Curtis, V. (2009). Experimental pretesting of hand-washing interventions in a natural setting. American Journal of Public Health, 99, S405-S411. doi:10.2105/AJPH.2009.164160   


For further reading:

            Faulkner, M. & Kennedy, R. (2008). A new tool for pre-testing direct mail. International Journal of Market Research, 50, 469-490.  [

            Whittingham, J., Ruiter, R. A. C., Zimbile, F., & Kok, G. (2008).  Experimental pretesting of public health campaigns: A case study. Journal of Health Communication, 13, 216-230. doi:10.1080/10810730701854045 

            Yang, B., Owusu, D., & Popova, L. (2019). Testing messages about comparative risk of electronic cigarettes and combusted cigarettes. Tobacco Control, 28, 440–448. doi:10.1136/tobaccocontrol-2018-054404  Neural responses


Falk, E. B., O’Donnell, M. B., Tompson, S., Gonzalez, R., Dal Cin, S., Strecher, V., . . . An, L. (2016). Functional brain imaging predicts public health campaign success. Social Cognitive and Affective Neuroscience, 11, 204-214. doi:10.1093/scan/nsv108 


For further reading:

            Kennedy, R., Northover, H., Leighton, J., Bird, G., & Lion, S. (2010, June). Pre-test advertising: Proposing a new validity project. Paper presented at the 39th European Marketing Academy (EMAC) conference, Copenhagen, Denmark.

            Falk, E. B., Berkman, E. T., Mann, T., Harrison, B., & Lieberman, M. D. (2010). Predicting persuasion-induced behavior change from the brain. Journal of Neuroscience, 30, 8421-8424. doi: 10.1523/JNEUROSCI.0063-10.201

            Falk, E. B., Berkman, E. T., & Lieberman, M. D. (2012). From neural responses to population behavior: Neural focus group predicts population-level media effects. Psychological Science, 23, 439-445.  doi:10.1177/0956797611434964 

            Cascio, C. N., Dal Cin, S., & Falk, E. B. (2013). Health communications: Predicting behavior change from the brain. In P. A. Hall (Ed.), Social neuroscience and public health (pp. 57-71). New York: Springer.

            Venkatraman, V., Dimoka, A., Pavlou, P. A., Vo, K., Hampton, W., Bollinger, B., Hershfield, H. E., Ishihara, M., & Winer, R. S. (2015). Predicting advertising success beyond traditional measures: New insights from neurophysiological methods and market response modeling. Journal of Marketing Research, 52, 436-452. doi:10.1509/jmr.13.0593  

            Weber, R., Huskey, R., Mangus, J. M., Westcott-Baker, A., & Turner, B. O. (2015). Neural predictors of message effectiveness during counterarguing in antidrug campaigns. Communication Monographs, 82, 4-30. doi:10.1080/03637751.2014.971414

            Falk, E. B., Cascio, C. N., & Coronel, J. C. (2015). Neural prediction of communication-relevant outcomes. Communication Methods and Measures, 9, 30-54. doi:10.1080/19312458.2014.999750    

            Pegors, T. K., Tompson, S., O'Donnell, M. B., & Falk, E. B. (2017). Predicting behavior change from persuasive messages using neural representational similarity and social network analyses. NeuroImage, 157, 118-128.  doi:10.1016/j.neuroimage.2017.05.063 

            Bellman, S., Nenycz-Thiel, M., Kennedy, R., Larguinat, L., McColl, B., & Varan, D. (2017). What makes a television commercial sell? Using biometrics to identify successful ads. Journal of Advertising Research, 57, 53-66. doi:

            Kranzler, E. C., Schmälzle, R., Pei, R., Hornik, R. C., & Falk, E. B. (2019). Message-elicited brain response moderates the relationship between opportunities for exposure to anti-smoking messages and message recall. Journal of Communication, 69(6), 589–611.  doi:10.1093/joc/jqz035

            Doré, B. P., Tompson, S. H., O’Donnell, M. B., An, L., Strecher, V., & Falk, E. B. (2019). Neural mechanisms of emotion regulation moderate the predictive value of affective and value-related brain responses to persuasive messages. Journal of Neuroscience, 39, 1293-1300. doi:10.1523/JNEUROSCI.1651-18.2018  A/B testing


Kohavi, R., & Longbotham, R. (2017). Online controlled experiments and A/B testing. In C. Sammut & G.I. Webb (Eds.), Encyclopedia of machine learning and data mining. doi:10.1007/978-1-4899-7502-7 891-1 


For further reading:

            Kohavi, R., Henne, R. M., & Sommerfield, D. (2007). Practical guide to controlled experiments on the web: Listen to your customers not to the hippo. KDD '07: Proceedings of the 13th ACM SIGKDD international conference on knowledge discovery and data mining (pp. 959-967).  

            Kohavi, R., Deng, A., Frasca, B., Longbotham, R., Walker, T., & Xu, Y. (2012). Trustworthy online controlled experiments: Five puzzling outcomes explained. KDD '12: Proceedings of the 18th ACM SIGKDD international conference on Knowledge discovery and data mining (pp. 786–794). 

            Berman, R., Pekelis, L., Scott, A., & Van den Bulte, C. (2018). p-hacking and false discovery in A/B testing.  SSRN working paper. (December 11, 2018). Available at SSRN: or  ARS persuasion scores


For further reading: 

            Kuse, A. R. (1997). The measurement of advertising effectiveness: Empirical learning and application. In W. D. Wells (Ed.), Measuring advertising effectiveness (pp. 301-322). Mahwah, NJ: Erlbaum.

            Blair, M. H., & Rabuck, M. J. (1998). Advertising wearin and wearout: Ten years later—More empirical evidence and successful practice. Journal of Advertising Research, 38(5), 7–18.    

            Pechmann, C., & Andrews, J. C. (2011). Copy test methods to pretest advertisements. In J. N. Sheth & N. K. Malhotra (Eds.), Wiley international encyclopedia of marketing. Chichester, West Sussex, UK: Wiley.  doi:10.1002/9781444316568.wiem04007  (see pp. 5-7)