MTS 525-0
Special Topics Research Seminar
Section 20: Generalizing about Message Effects
Spring 2020
SYLLABUS: TOPIC 6
TOPIC 6: Message design guidance: Variations on, and alternatives to, RCT-based generalization
6.1 Randomized controlled
trials (RCTs): Variations and alternatives
6.1.1 Multiphase optimization strategy (MOST),
sequential multiple assignment randomized trials (SMART), etc.
6.1.2 Pragmatic trials
6.1.3 Effectiveness-implementation designs
6.2 Assessment of the
messages of interest
6.2.1 Summary discussions of message pretesting
methods
6.2.2 Message properties
6.2.2.1 Armstrong’s Persuasive Principles Index (PPI)
6.2.2.1 Other index-based assessments of message
properties
6.2.3 Message perceptions
6.2.3.1 Perceived message effectiveness
6.2.3.2 Discrete choice experiments
6.2.3.3 Message liking
6.2.4 Message effects
6.2.4.1 Familiar experimental arrangements and
outcomes
6.2.4.2 Neural responses
6.2.4.3 A/B testing
6.2.4.4 ARS persuasion scores
6.1 Randomized controlled trials (RCTs):
Variations and alternatives
6.1.1 Multiphase optimization strategy (MOST),
sequential multiple assignment randomized trials (SMART), etc.
Collins, L. M., Murphy, S. A., &
Strecher, V. (2007). The multiphase optimization
strategy (MOST) and the sequential multiple assignment randomized trial
(SMART): New methods for more potent eHealth interventions. American Journal of Preventive Medicine, 32(Suppl), S112-118. doi:10.1016/j.amepre.2007.01.022
For further reading:
Plackett, R. L., & Burman, J. P. (1946). The design of
optimum multifactorial experiments. Biometrika, 33,
305-325.
Bell, G.
H., Ledolter, J., & Swersey,
A. J. (2006). Experimental design on the front lines of marketing: Testing new
ideas to increase direct mail sales. International
Journal of Research in Marketing, 23, 309-319.
Nair, V., Strecher, V., Fagerlin, A., Ubel, P., Resnicow, K., Murphy,
S., Little, R., Charkraborty, B., & Zhang, A. J.
(2008). Screening experiments and the use of fractional factorial designs in
behavioral intervention research. American
Journal of Public Health, 98, 1354-1358.
Collins, L.
M., Dziak, J. J., & Li, R. Z. (2009). Design of
experiments with multiple independent variables: a resource management
perspective on complete and reduced factorial designs. Psychological Methods, 14(3), 202-224. doi: 10.1037/a0015826
Chakraborty,
B., Collins, L. M., Strecher, V. J., & Murphy, S.
A. (2009). Developing multicomponent interventions using fractional factorial
designs. Statistics in Medicine, 28,
2687-2708.
Brown, C.
H., Ten Have, T. R., Jo, B., Dagne, G., Wyman, P. A.,
Muthen, B., & Gibbons, R. D. (2009). Adaptive
designs for randomized trials in public health. Annual Review of Public Health, 30, 1-25.
Almirall, D., Compton, S. N., Gunlicks-Stoessel,
M., Duan, N., & Murphy, S. A. (2012). Designing a
pilot sequential multiple assignment randomized trial for developing an
adaptive treatment strategy. Statistics in Medicine, 31,
1887-1902. doi:10.1002/sim.4512
Collins, L.
M., Trail, J. B., Kugler, K. C., Baker, T. B., Piper,
M. E., & Mermelstein, R. J. (2014). Evaluating
individual intervention components: Making decisions based on the results of a
factorial screening experiment. Translational
Behavioral Medicine, 4, 238-251. doi:10.1007/s13142-013-0239-7
Collins, L.
M. (2018). Optimization of behavioral, biobehavioral, and biomedical interventions: The multiphase
optimization strategy (MOST). Springer.
Collins, L.
M., & Kugler, K. (Eds.). (2018). Optimization of behavioral, biobehavioral, and biomedical interventions: Advanced
topics. Springer.
(2019).
Causal inference in generalizable environments: Systematic
representative design. Psychological Inquiry,
30(4), 173-202. doi:10.1080/1047840X.2019.1693866
Michela, A., van Rooij, M. M. J.
W., Klumpers, F., van Peer, J. M., Roelofs, K., & Granic, I.
(2019). Reducing the noise of reality. Psychological
Inquiry, 30(4), 203-210. doi:10.1080/1047840X.2019.1693872
6.1.2 Pragmatic trials
For further reading:
Schwartz,
D., & Lellouch, J. (1967). Explanatory and
pragmatic attitudes in therapeutic trials. Journal
of Chronic Diseases, 20, 637-648.
doi:10.1016/0021-9681(67)90041-0
Patsopoulos, N. A. (2011). A pragmatic view on pragmatic
trials. Dialogues in Clinical Neuroscience,
13(2), 217–224.
Ford,
I., & Norrie, J. (2016). Pragmatic trials. New England Journal of Medicine, 375, 454-463. doi:10.1056/NEJMra1510059
Troxel, A. B., Asch, D. A., & Volpp,
K. G. (2016). Statistical issues in pragmatic trials of behavioral economic
interventions. Clinical Trials, 13,
478-483. doi:10.1177/1740774516654862
Zuidgeest, M. G. P., Goetz, I., Groenwold,
R. H. H., Irving, E., van Thiel, G. J. M. W., Grobbee,
D. E., & GetReal Work Package 3. (2017). Series:
Pragmatic trials and real world evidence: Paper 1. Introduction. Journal of Clinical Epidemiology, 88,
7-13. doi:10.1016/j.jclinepi.2016.12.023
6.1.3 Effectiveness-implementation designs
For further reading:
Curran, G.,
Bauer, M., Mittman, B., Pyne,
J., & Stetler, C. (2012).
Effectiveness-implementation hybrid designs: Combining elements of clinical
effectiveness and implementation research to enhance public health impact. Medical Care, 50, 217-226.
doi:10.1097/MLR.0b013e3182408812
Wolfenden, L., Williams, C. M., Wiggers,
J., Nathan, N., &Yoong, S. L. (2016). Improving
the translation of health promotion interventions using effectiveness–implementation
hybrid designs in program evaluations. Health
Promotion Journal of Australia, 27(3), 204–207.
Luszczynska, A. (2020) It’s time for
effectiveness-implementation hybrid research on behaviour change. Health Psychology Review, 14(1), 188-192.
doi:10.1080/17437199.2019.1707105
6.2 Assessment of the messages of interest
6.2.1 Summary discussions of message pretesting methods
For further reading:
Bertrand,
J. T. (1978). Communications pretesting. Chicago, IL: University of Chicago Community
and Family Study Center.
Stewart, D.
W., Furse, D. H., & Kozak,
R. P. (1983). A guide to commercial copytesting services. Current Issues and
Research in Advertising, 6, 1–43. doi:10.1080/01633392.1983.10505330
Pechmann, C., & Andrews, J. C. (2011). Copy test
methods to pretest advertisements. In J. N. Sheth
& N. K. Malhotra (Eds.), Wiley international encyclopedia of marketing.
Chichester, West Sussex, UK: Wiley. doi:10.1002/9781444316568.wiem04007
Willoughby,
J. F., & Furberg, R. (2015). Underdeveloped or
underreported? Coverage of pretesting practices and recommendations for design of
text message–based health behavior change interventions. Journal of Health Communication, 20, 472-478.
doi:10.1080/10810730.2014.977468
6.2.2 Message properties
6.2.2.1 Armstrong’s Persuasive Principles Index (PPI)
Armstrong, J. S., Du, R., Green,
K. C., & Graefe, A. (2016). Predictive validity
of evidence-based persuasion principles: An application of the index
method. European Journal of
Marketing, 50, 276-292. doi:10.1108/EJM-10-2015-0728
For further reading:
Armstrong, J. S., & Patnaik, S. (2009). Using
quasi-experimental data to develop empirical generalizations for persuasive
advertising. Journal of Advertising
Research, 49, 170-175. doi:10.2501/S0021849909090230
Armstrong, J. S. (2010). Persuasive advertising: Evidence-based
principles. New York: Palgrave Macmillan.
Armstrong, J. S. (2011). Evidence-based advertising: An
application to persuasion. International
Journal of Advertising, 30, 743-767. doi:10.2501/IJA-30-5-743-767
O’Keefe,
D. J. (2016). Evidence-based advertising using persuasion principles. European Journal of Marketing, 50, 294-300.
doi:10.1108/EJM-11-2015-0801
Sharp,
B., & Hartnett, N. (2016). Generalisability of
advertising persuasion principles. European
Journal of Marketing, 50, 301-305. doi:10.1108/EJM-12-2015-0842
Woodside,
A. G. (2016). Predicting advertising execution effectiveness: Scale development
and validation. European Journal of
Marketing, 50, 306-311. doi:10.1108/EJM-11-2015-0809
Wright,
M. J. (2016). Predicting what? The strengths and limitations of a test of
persuasive advertising principles. European
Journal of Marketing, 50, 312-316. doi:10.1108/EJM-12-2015-0833
Green,
K. C., Armstrong, J. S., Du, R., & Graefe, A.
(2016). Persuasion Principles Index: Ready for pretesting advertisements. European
Journal of Marketing, 50, 317-326.
doi:10.1108/EJM-12-2015-0838
Pereira,
J. J. S. (2018). A ciência da publicidade
: Conhecimento intuitivo e uso de princípios de mudança comportamental por especialistas para influenciar consumidores [The
science of advertising: Experts’ intuitions and usage of behavioral change
principles to influence consumers] (doctoral dissertation, University of
Brasilia). http://repositorio.unb.br/handle/10482/32938
6.2.2.1 Other index-based assessments of message
properties
For further reading:
Paul, C.
L., Redman, S., & Sanson-Fisher, R. W. (1997).
The development of a checklist of content and design characteristics for
printed health education materials. Health
Promotion Journal of Australia, 7(3), 153-159.
Cole, G.
E., Keller, P. A., Reynolds, J., Schaur, M., &
Krause, D. (2016). CDC MessageWorks: Designing and
validating a social marketing tool to craft and defend effective messages. Social Marketing Quarterly, 22(1), 3-18.
doi:10.1177/1524500415614817
Wildeboer, G., Kelders, S. M.,
& van Gemert-Pijnen, J. E. (2016). The
relationship between persuasive technology principles, adherence and effect of
web-based interventions for mental health: A meta-analysis. International Journal of Medical Informatics,
96, 71-85.
doi:10.1016/j.ijmedinf.2016.04.005
Baumel, A., Faber, K., Mathur,
N., Kane, J. M., & Muench, F. (2017). Enlight: A comprehensive quality and therapeutic potential
evaluation tool for mobile and web-based eHealth interventions. Journal of Medical Internet Research, 19,
e82. doi:10.2196/jmir.7270
Lim, K.,
Kilpatrick, C., Storr, J., & Seale, H. (2018). Exploring
the use of entertainment-education YouTube videos focused on infection
prevention and control. American Journal
of Infection Control, 46, 1218-1223. doi:10.1016/j.ajic.2018.05.002
Huhmann, B. A., & Albinsson,
P. A. (2019). Assessing the usefulness of taxonomies of visual rhetorical
figures. Journal of Current Issues &
Research in Advertising, 40, 171-195. doi:10.1080/10641734.2018.1503106
6.2.3 Message perceptions
6.2.3.1 Perceived message effectiveness
Dillard, J. P., Weber, K. M., & Vail, R. G. (2007). The
relationship between the perceived and actual effectiveness of persuasive
messages: A meta-analysis with implications for formative campaign research. Journal
of Communication, 57, 613-631.
doi:10.1111/j.1460-2466.2007.00360.x
O’Keefe, D. J. (2018). Message pretesting using assessments
of expected or perceived persuasiveness: Evidence about diagnosticity of
relative actual persuasiveness. Journal
of Communication, 68, 120-142. doi:10.1093/joc/jqx009
O’Keefe, D. J. (2020). Message pretesting using perceived
persuasiveness measures: Reconsidering the correlational evidence. Communication Methods and Measures, 14(1),
25-37.
doi:10.1080/19312458.2019.1620711
For further reading:
Dillard, J.
P., Shen, L., & Vail, R. G. (2007). Does perceived message effectiveness
cause persuasion or vice versa? 17 consistent answers. Human Communication Research, 33, 467-488.
doi:10.1111/j.1468-2958.2007.00308.x
Yzer, M., LoRusso, S., & Nagler, R. H.
(2015). On the conceptual ambiguity surrounding perceived message
effectiveness. Health Communication, 30,
125-134. doi:10.1080/10410236.2014.974131
Meschtscherjakov, A., Gärtner,
M., Mirnig, A., Rödel, C.,
& Tscheligi, M. (2016). The Persuasive Potential
Questionnaire (PPQ): Challenges, drawbacks, and lessons learned. In A. Meschtscherjakov, B. De Ruyter,
V. Fuchsberger, M. Murer, & M. Tscheligi (Eds.), Persuasive
technology: 11th international conference, PERSUASIVE 2016 (pp. 162-175).
Cham, Switzerland: Springer. [LNCS (Lecture Notes in Computer Science) vol.
9638]
Noar, S.
M., Barker, J., & Yzer, M. (2018). Measurement and design heterogeneity in
perceived message effectiveness studies: A call for research. Journal of Communication, 68, 990-993. doi:10.1093/joc/jqy047
Cappella,
J. N. (2018). Perceived message effectiveness meets the requirements of a
reliable, valid, and efficient measure of persuasiveness. Journal of Communication, 68, 994-997. doi:10.1093/joc/jqy044
Davis, K.
C., & Duke, J. C. (2018). Evidence of the real-world effectiveness of
public health media campaigns reinforces the value of perceived message
effectiveness in campaign planning. Journal
of Communication, 68, 998-1000. doi:10.1093/joc/jqy045
O’Keefe, D.
J. (2018). Whistling past the graveyard: Response to commentaries. Journal of Communication, 68, 1001-1005. doi:10.1093/joc/jqy046
Noar, S.
M., Bell, T., Kelley, D., Barker, J., & Yzer, M. C. (2018). Perceived
message effectiveness measures in tobacco education campaigns: A systematic
review. Communication Methods and
Measures, 12, 295-313. doi:10.1080/19312458.2018.1483017
Baig, S. A., Noar, S. M., Gottfredson,
N. C., Boynton, M. H., Ribisl, K. M., & Brewer,
N. T. (2019). UNC perceived message effectiveness: Validation of a brief scale.
Annals of Behavioral Medicine, 53(8),
732-742. doi:10.1093/abm/kay080
Thomas, R.
J., Masthoff, J., & Oren, N. (2019). Can I
influence you? Development of a scale to measure perceived persuasiveness and
two studies showing the use of the scale. Frontiers
in Artificial Intelligence, 2.
doi:10.3389/frai.2019.00024
Kim, M.,
& Cappella, J. N. (2019). Reliable, valid and efficient evaluation of media
messages: Evaluation protocol for effectiveness and empirical evidence. Journal of Communication Management, 23,
179-197.
doi:10.1108/JCOM-12-2018-0132
Noar, S.
M., Barker, J., Bell, T., & Yzer, M. C. (2020). Does perceived message
effectiveness predict the actual effectiveness of tobacco education messages? A
systematic review and meta-analysis. Health
Communication, 35(2), 148-157. doi:10.1080/10410236.2018.1547675
6.2.3.2 Discrete choice experiments
Thrasher, J. F. Anshari, D., Lambert-Jessup, V., Islam, F., Mead, E.,
Popova, L., Salloum, R., Moodie,
C., Louviere, J., & Lindblom, E. N. (2018).
Assessing smoking cessation messages with a discrete choice experiment. Tobacco Regulatory Science, 4, 73-87.
doi:10.18001/TRS.4.2.7
For further reading:
M., P.,
& T. M. (2012). “Pay them if it
works”: Discrete choice experiments on the acceptability of financial
incentives to change health related behaviour. Social Science and
Medicine, 75(12), 2509-2514. doi:10.1016/j.socscimed.2012.09.033
M.,
D., D., S., D.,
& E. W. (2014).
Discrete choice experiments in health economics: A
review of the literature. PharmacoEconomics,
32(9), 883-902. doi:10.1007/s40273-014-0170-x
E. M.,
D.
A., A. B.,
& J. F. P. (2017). Improving the quality of discrete-choice experiments in
health: How can we assess validity and reliability? Expert Review of Pharmacoeconomics & Outcomes Research, 17(6), 531-542, doi:10.1080/14737167.2017.1389648
T. M.,
& C. E. (2017). Getting patients in
the door: Medical appointment reminder preferences. Patient
Preference and Adherence, 11, 141-150. doi:10.2147/ppa.s117396
E. W., J., M., B., J., S., Swait, J., Lancsar, E., Witteman, C. L. M., Bonsel, G.,
& BP. (2018). The impact of
vaccination and patient characteristics on influenza vaccination uptake of
elderly people: A discrete choice experiment. Vaccine, 36(11), 1467-1476. doi:
10.1016/j.vaccine.2018.01.054
F., N., J. M., M., E., Osaki,
H., Ong, J. J., Tucker, J. D., Mshana, G.,
Mahler, H., Weiss, H. A., Wambura, H., & The VMMC
study team. (2019). Using discrete choice
experiments to inform the design of complex interventions. Trials,
20(1), 157. doi:10.1186/s13063-019-3186-x
McGrady, M. E., Pai, A. L. H., &
Prosser, L. A. (in press). Using discrete choice experiments to develop and
deliver patient-centered psychological interventions: A systematic review. Health Psychology Review. doi:10.1080/17437199.2020.1715813
6.2.3.3 Message liking
For further reading:
Haley, R. I., & Baldinger,
A. L. (1991). The ARF Copy Research Validity Project. Journal of Advertising
Research, 31(2), 11–32.
Rossiter, J. R., & Eagleson,
G. (1994). Conclusions from the ARF’s Copy Research Validity Project. Journal
of Advertising Research, 34(3), 19-32.
Haley, R.
I. (1994). A rejoinder to “Conclusions from the ARF’s Copy Research Validity
Project.” Journal of Advertising
Research, 34(3), 33.
Hollis, N.
S. (1995). Like it or not, liking is not enough. Journal of Advertising Research, 35(5), 7-16.
Bergkvist, L., & Rossiter, J.
R. (2008). The role of ad likability in predicting an ad’s campaign
performance. Journal of Advertising, 37(2),
85-97.
6.2.4 Message effects
6.2.4.1 Familiar experimental arrangements and
outcomes
Judah, G., Aunger, R., Schmidt, W.
P., Michie, S., Granger, S., & Curtis, V. (2009). Experimental pretesting
of hand-washing interventions in a natural setting. American Journal of
Public Health, 99, S405-S411. doi:10.2105/AJPH.2009.164160
For further reading:
Faulkner, M. & Kennedy, R. (2008). A new
tool for pre-testing direct mail. International Journal of Market Research,
50, 469-490. [
Whittingham, J., Ruiter, R. A. C., Zimbile,
F., & Kok, G. (2008). Experimental
pretesting of public health campaigns: A case study. Journal of Health Communication, 13, 216-230.
doi:10.1080/10810730701854045
Yang,
B., Owusu, D., & Popova, L. (2019). Testing
messages about comparative risk of electronic cigarettes and combusted
cigarettes. Tobacco Control, 28, 440–448.
doi:10.1136/tobaccocontrol-2018-054404
6.2.4.2 Neural responses
Falk, E. B.,
O’Donnell, M. B., Tompson, S., Gonzalez, R., Dal Cin, S., Strecher, V., . . . An,
L. (2016). Functional brain imaging predicts public health campaign
success. Social Cognitive and Affective
Neuroscience, 11, 204-214. doi:10.1093/scan/nsv108
For further reading:
Kennedy,
R., Northover, H., Leighton, J., Bird, G., &
Lion, S. (2010, June). Pre-test
advertising: Proposing a new validity project. Paper presented at the 39th
European Marketing Academy (EMAC) conference, Copenhagen, Denmark.
Falk, E.
B., Berkman, E. T., Mann, T., Harrison, B., & Lieberman, M. D. (2010).
Predicting persuasion-induced behavior change from the brain. Journal of Neuroscience, 30, 8421-8424.
doi: 10.1523/JNEUROSCI.0063-10.201
Falk, E.
B., Berkman, E. T., & Lieberman, M. D. (2012). From neural responses to
population behavior: Neural focus group predicts population-level media
effects. Psychological Science, 23,
439-445. doi:10.1177/0956797611434964
Cascio, C. N., Dal Cin, S., &
Falk, E. B. (2013). Health communications: Predicting behavior change from the
brain. In P. A. Hall (Ed.), Social
neuroscience and public health (pp. 57-71). New York: Springer.
Venkatraman, V., Dimoka, A., Pavlou, P. A., Vo, K., Hampton, W., Bollinger, B., Hershfield, H. E., Ishihara, M., & Winer, R. S. (2015).
Predicting advertising success beyond traditional measures: New insights from
neurophysiological methods and market response modeling. Journal of Marketing Research, 52, 436-452. doi:10.1509/jmr.13.0593
Weber,
R., Huskey, R., Mangus, J. M.,
Westcott-Baker, A., & Turner, B. O. (2015). Neural predictors of message
effectiveness during counterarguing in antidrug campaigns. Communication Monographs, 82, 4-30. doi:10.1080/03637751.2014.971414
Falk,
E. B., Cascio, C. N., & Coronel, J. C. (2015).
Neural prediction of communication-relevant outcomes. Communication Methods and Measures, 9, 30-54. doi:10.1080/19312458.2014.999750
Pegors, T. K., Tompson, S.,
O'Donnell, M. B., & Falk, E. B. (2017). Predicting behavior change from
persuasive messages using neural representational similarity and social network
analyses. NeuroImage, 157, 118-128. doi:10.1016/j.neuroimage.2017.05.063
Bellman,
S., Nenycz-Thiel, M., Kennedy, R., Larguinat, L., McColl, B., & Varan,
D. (2017). What makes a television commercial sell? Using biometrics to
identify successful ads. Journal of
Advertising Research, 57, 53-66. doi:
Kranzler, E. C., Schmälzle, R.,
Pei, R., Hornik, R. C., & Falk, E. B. (2019). Message-elicited brain
response moderates the relationship between opportunities for exposure to
anti-smoking messages and message recall. Journal
of Communication, 69(6), 589–611. doi:10.1093/joc/jqz035
Doré, B. P., Tompson, S. H.,
O’Donnell, M. B., An, L., Strecher, V., & Falk,
E. B. (2019). Neural mechanisms of emotion regulation moderate the predictive
value of affective and value-related brain responses to persuasive messages. Journal of Neuroscience, 39, 1293-1300.
doi:10.1523/JNEUROSCI.1651-18.2018
6.2.4.3 A/B testing
Kohavi, R., & Longbotham, R.
(2017). Online controlled experiments and A/B testing. In C. Sammut & G.I. Webb (Eds.), Encyclopedia of machine learning and data mining.
doi:10.1007/978-1-4899-7502-7 891-1
For further reading:
Kohavi, R., Henne, R. M., & Sommerfield, D. (2007). Practical guide to controlled
experiments on the web: Listen to your customers not to the hippo. KDD '07: Proceedings of the 13th ACM SIGKDD
international conference on knowledge discovery and data mining (pp.
959-967). https://doi.org/10.1145/1281192.1281295
Kohavi,
R., Deng, A., Frasca, B., Longbotham,
R., Walker, T., & Xu, Y. (2012). Trustworthy online controlled experiments:
Five puzzling outcomes explained. KDD
'12: Proceedings of the 18th ACM SIGKDD international conference on Knowledge
discovery and data mining (pp. 786–794). https://doi.org/10.1145/2339530.2339653
Berman, R.,
Pekelis, L., Scott, A., & Van den Bulte, C. (2018). p-hacking
and false discovery in A/B testing.
SSRN working paper. (December 11, 2018). Available at SSRN:
https://ssrn.com/abstract=3204791 or http://dx.doi.org/10.2139/ssrn.3204791
6.2.4.4 ARS persuasion scores
For further reading:
Kuse, A. R. (1997). The measurement of advertising
effectiveness: Empirical learning and application. In W. D. Wells (Ed.), Measuring advertising effectiveness (pp.
301-322). Mahwah, NJ: Erlbaum.
Blair, M. H.,
& Rabuck, M. J. (1998). Advertising wearin and wearout: Ten years
later—More empirical evidence and successful practice. Journal of
Advertising Research, 38(5), 7–18.
Pechmann, C., & Andrews, J. C. (2011). Copy test
methods to pretest advertisements. In J. N. Sheth
& N. K. Malhotra (Eds.), Wiley international encyclopedia of marketing.
Chichester, West Sussex, UK: Wiley. doi:10.1002/9781444316568.wiem04007 (see pp. 5-7)