Effect of Physician-Delivered COVID-19 Public Health Messages and Messages Acknowledging Racial Inequity on Black and White Adults’ Knowledge, Beliefs, and Practices Related to COVID-19
Abstract
IMPORTANCE Social distancing is critical to the control of COVID-19, which has disproportionately affected the Black community. Physician-delivered messages may increase adherence to these behaviors.
OBJECTIVES To determine whether messages delivered by physicians improve COVID-19 knowledge and preventive behaviors and to assess the differential effectiveness of messages tailored to the Black community.
DESIGN, SETTING, AND PARTICIPANTS This randomized clinical trial of self-identified White and Black adults with less than a college education was conducted from August 7 to September 6, 2020. Of 44 743 volunteers screened, 30 174 were eligible, 5534 did not consent or failed attention checks, and 4163 left the survey before randomization. The final sample had 20 460 individuals (participation rate, 68%). Participants were randomly assigned to receive video messages on COVID-19 or other health topics.
INTERVENTIONS Participants saw video messages delivered either by a Black or a White study physician. In the control groups, participants saw 3 placebo videos with generic health topics. In the treatment group, they saw 3 videos on COVID-19, recorded by several physicians of varied age, gender, and race. Video 1 discussed common symptoms. Video 2 highlighted case numbers; in one group, the unequal burden of the disease by race was discussed. Video 3 described US Centers for Disease Control and Prevention social distancing guidelines. Participants in both the control and intervention groups were also randomly assigned to see 1 of 2 American Medical Association statements, one on structural racism and the other on drug price transparency.
MAIN OUTCOMES AND MEASURES Knowledge, beliefs, and practices related to COVID-19, demand for information, willingness to pay for masks, and self-reported behavior.
RESULTS Overall, 18223 participants(9168Black;9055White) completed the survey (9980 [55.9%] women, mean [SD] age, 40.2 [17.8] years). Overall, 6303 Black participants (34.6%) and 7842 White participants (43.0%) were assigned to the intervention group, and 1576 Black participants (8.6%) and 1968 White participants (10.8%) were assigned to the control group. Compared with the control group, the intervention group had smaller gaps in COVID-19 knowledge (incidence rate ratio [IRR], 0.89 [95% CI, 0.87-0.91]) and greater demand for COVID-19 information (IRR, 1.05 [95% CI, 1.01-1.11]), willingness to pay for a mask (difference, $0.50 [95% CI, $0.15-$0.85]). Self-reported safety behavior improved, although the difference was not statistically significant (IRR, 0.96 [95% CI, 0.92-1.01]; P = .08). Effects did not differ by race (F = 0.0112; P > .99) or in different intervention groups (F = 0.324; P > .99).
CONCLUSIONS AND RELEVANCE In this study, a physician messaging campaign was effective in increasing COVID-19 knowledge, information-seeking, and self-reported protective behaviors among diverse groups. Studies implemented at scale are needed to confirm clinical importance.
TRIAL REGISTRATION ClinicalTrials.govIdentifier: NCT04502056 JAMA Network Open. 2021;4(7):e2117115. doi:10.1001/jamanetworkopen.2021.17115
Introduction
Physical distancing and mask-wearing remain essential to the control of COVID-19, yet vigilance has decreased over time.1 To address fatigue, health care professionals have used social media to spread public health messages.2 There is evidence that these messages improve knowledge, but there are less data on whether they change behavior.3
Black US residents have been disproportionately affected by the pandemic.4 This reflects the cumulative impact of systemic racism, acknowledged as a public health threat by the American Medical Association (AMA) in a June 2020 statement.5 This raises the question on whether the effectiveness of public health messages regarding COVID-19 would be enhanced if tailored to the Black community. The focus of this study was to identify whether messages delivered by physicians increase COVID-19 knowledge and improve preventive behaviors for White and Black individuals and to assess whether various ways of increasing the relevance of messages to the Black community (ie, physician race, AMA acknowledgments of racial injustices, or information about the disproportionate burden of COVID-19 on the Black community) affects their impact on both White and Black participants.
Methods
Trial Design and Oversight
The trial flowchart (Figure 1; eFigure 1 in Supplement 1) describes the factorial design and the allocation of participants to each intervention arm. The design was approved by the ethical review boards of Massachusetts Institute of Technology (MIT) and Stanford, with Massachusetts General Hospital, Yale, and Harvard ceding authority to MIT. All participants provided written informed consent. The trial and the outcomes were registered on ClinicalTrials.gov (NCT04502056). Planned analyses were published on the American Economic Association trial registry (AEARCTR-0006177). The pre-analysis plan and institutional review board–approved protocol are available in
Supplement 2. This study followed the Consolidated Standards of Reporting Trials (CONSORT) and American Association for Public Research (AAPOR) reporting guidelines.
Participants
Individuals were recruited online throughout the United States by the survey company Lucid from August 7, 2020, to September 6, 2020. Lucid recruits survey participants by advertising surveys to third-party suppliers, including double opt-in panels, publishing networks, social media, and other types of online communities. Participants aged 18 years or older, self-identifying as White or Black, and without a college degree were eligible. We focused on these 2 groups because we were interested in tailoring messages toward the Black community as well as the reaction of the White community to the tailoring of those messages. Latinx individuals were included in our previous study,3 along with specific tailoring toward this community. Recruitment used quotas to match the 2018 population estimates by age, sex, and race issued by the US Census Bureau.6
Interventions
Intervention vs Control Messages
After reading the informed consent form (approved by the institutional review board) (eAppendix 1 in Supplement 1) and agreeing to participate, all individuals answered sociodemographic questions, saw 3 videos, and then completed the outcome survey questions. In the control groups, participants saw 3 placebo videos with generic health topics, including fitness guidelines, recommended sugar intake, and the importance of adequate sleep. In the treatment group, they saw 3 videos on COVID-19, recorded by several physicians of varied age, gender, and race. Participants in each group (placebo and intervention) saw video messages delivered either by a Black or a White study physician (including L.O.-N., M.A., F.C.S., and E.W.).
Video 1 defined COVID-19 and discussed common symptoms associated with COVID-19 as well as asymptomatic transmission. Video 2 reminded the viewer that COVID-19 was actively circulating in the United States. Video 3 described the Centers for Disease Control and Prevention social distancing guidelines (complete scripts appear in eAppendix 1 in Supplement 1).
Unequal Burden of COVID-19
Video 2 in the COVID-19 intervention had 2 randomized variants. Script 1 emphasized the number of new cases in the week of July 6, 2020. Script 2 added that, controlling for age, Black individuals were 3 times as likely to become infected as White individuals and 4 times as likely to die from it. These 2 variants of video 2 were cross-randomized with the intervention.
AMA Antiracism or Placebo Statement
At the beginning of the study, all participants saw a video of an actor delivering the script of a public statement by the AMA. The AMA antiracism statement, issued on June 7, 2020, “recognizes that racism in its systemic, structural, institutional, and interpersonal forms is an urgent threat to public health, the advancement of health equity, and a barrier to excellence in the delivery of medical care.”5 The AMA placebo intervention was an AMA statement on drug pricing.7 The race and gender of the person reading the statement were randomized to each recipient.
Outcomes
Most outcomes were measured online immediately following the intervention or the placebo. The prespecified primary outcomes were knowledge, beliefs, and practices related to COVID-19, measured immediately after the intervention; intended behavior, measured immediately after the intervention; and knowledge and behavior, measured a few days after the intervention.
eAppendix 2 in Supplement 1 describes the outcome measurement in detail. Primary outcomes presented in the main text include 5 outcomes. First, knowledge gap, which measures knowledge and beliefs. Participants were asked to identify ways to prevent COVID-19 spread and identify 4 common symptoms. The knowledge gap outcome is an integer that can have values from 0 (no error) to 10 (10 errors). Second, information seeking was measured by offering participants the option of requesting additional information on COVID-19–related resources by clicking on up to 5 links that included more content. We measured information-seeking behavior as the number of links in which participants expressed interest, a count variable between 0 (lowest information-seeking behavior) and 5 (greatest information-seeking behavior). Third, self-reported safety behavior was measured a few days after the initial intervention among a subsample that was eligible for follow-up and could be tracked. Participants were asked about how often they engaged in 4 behaviors of interest: (1) whether they wore a mask indoors; (2) whether they wore a mask outdoors; (3) whether they washed their hands; and (4) whether they followed social distancing guidelines. The safety gap index had values of 0 (if a participant reported that they always practiced the 4 behaviors of interest) to 4 (participant reported that they practiced none of the behaviors). Fourth, at the end of the survey, each participant was asked the price they would be willing to pay for high-quality masks. Each participant was entered into a draw to receive either a coupon for masks or a gift card to an online retailer. When a participant was selected by the draw, a price was then randomly drawn for the coupon. If their reported willingness to pay was greater than the price, they would receive the masks; otherwise, they would receive the gift card. Therefore, it was in participants’ best interest to report their true willingness to pay for the masks. This type of procedure has been shown to lead to truthful reporting.8
We collected data on 3 secondary outcomes specified in our pre-analysis plan (Supplement 2). First, we asked participants to report their judgment of how well federal and state policies balanced opening the economy and limiting the health impacts of COVID-19. Second, we measured how participants prioritized COVID-19 protection vs other issues by asking the participants how they would want to allocate a donation of $1000 (which the research team would fund) between 2 charities: Give a Mask or the Alzheimer Association. Third, we asked whether they would prioritize a COVID-19 relief donation to a Black-focused charity (the BET COVID-19 relief fund) or a general charity (the Give Directly Project 100+ relief program).
Randomization
Randomization into intervention vs control and between interventions was stratified according to sex, age (45 years), race, and self-identified partisan affiliation (Republican vs other affiliation). The flowchart (Figure 1) summarizes the randomization; a more detailed version is available as eFigure 1 in Supplement 1.
Participants were first randomized into the AMA antiracism or placebo statements, with equal probability. They were then randomly assigned to intervention or control. One of 5 participants was assigned to control; the remainder were assigned to an intervention.
Within each group, participants were randomized into Black or White physician groups with equal probability. Intervention participants were further randomized, with equal probability, into 1 the 2 arms for video 2: they either received the information about the unequal burden of disease or did not. Randomization was performed using the Qualtrics platform, using a randomizer block within each stratum with the option to evenly present elements.
Statistical Analysis
We determined that a sample of 20 000 individuals (10 000 Black and 10 000 White) would provide 85% power to detect effect sizes of 0.05 SDs for intervention relative to control and for effects of specific variations in message content. These are small effect sizes that would justify scale up of this inexpensive intervention.
The analysis was performed by original assigned group, and it included all participants who completed the survey. Multivariable regression models include the stratifying variables
(age × sex × race × Republican identity dummies).
Effect of Any Video Message Intervention Relative to Control
To analyze the effect of seeing any video message on the knowledge gap, information-seeking behavior, and safety behavior outcomes, we fit the following negative binomial regression model for the count outcome:
log(μ ) = β + β intervention + βT stratum , i01i2i
where μi is the estimated mean outcome value (knowledge gap count, count of demanded links, or safety behavior count), interventioni is an indicator that equals 1 if the individual received the intervention videos and 0 if they received the placebo videos, and stratumi is a vector of indicator variables. Similar models are estimated for binary regressions (using the logistic regression equation) and continuous variable (using ordinary least squares) (eAppendix 2 in Supplement 1). Because there were multiple outcomes, we provide P values and q values adjusted for false discovery rates.
Effect of Variation in the Message Framing
To analyze the impact of different arms, we fit a negative binomial regression model to the count data:
log(μi) = β0 + β1Black physiciani + β2AMA antiracismi + β3interventioni + (β4Black
physiciani × interventioni) + (β5AMA antracismi × interventioni) + (β6mortality
difference × intervention ) + βT stratum , ii7i
where Black physiciani is an indicator that equals 1 if the physician was a Black individual and 0 otherwise; AMA antiracismi is an indicator for the AMA statement featuring the antiracism message (rather than the drug pricing message); mortality differencei is an indicator for video 2 mentioning the excess mortality from COVID-19. Similar specifications are fitted with logistic regression for binary variables and ordinary least squares for continuous variables.
To address possible bias stemming from nonrandom attrition for the follow-up survey, we weighted the follow-up data using Hainmueller entropy weights,9 which ensures that the observed baseline characteristics of the follow-up sample matches the original sample as closely as possible (eAppendix 2 in Supplement 1). eAppendix 3 in Supplement 1 describes further robustness checks and subgroups analysis.
Analyses were performed using R version 4.0.3 (R Project for Statistical Computing). Statistical significance was set at P < .05, and all tests were 2-tailed.
Results
Trial Sample
The trial sample was enrolled from August 7 through September 6, 2020. Of 44 743 screened, 30 174 were eligible for participation, 5534 individuals did not consent or failed 2 simple attention checks, 4163 left the survey before randomization, and 17 were excluded from the analysis due to unknown race or multiple survey completion. The final sample at randomization had 20 460 individuals, for a participation rate of 68%. After attrition, 18 223 individuals were included in the study. Summary statistics (Table 1) were computed on the sample that was randomized and that
completed all main baseline variables. Our sample included 9880 (55.9%) women. The mean (SD) age was 40.2 (17.8) years, 4206 (23.8%) reported household incomes greater than $60 000, 4228 Black participants (53.7%) and 2749 White participants (28.0%) identified as members of the Democratic party, 12 106 (68.4%) reported always wearing a mask indoors (outside the home), and 5517 (31.2%) reported always wearing a mask when outdoors. Black participants were twice as likely as White participants to report always wearing a mask outdoors (3513 [44.6%] vs 2004 [20.4%]) and were also more likely to report practicing hand hygiene (5084 [64.5%] vs 5695 [58.1%]) and physical distancing (4587 [58.2%] vs 4874 [49.7%]). Baseline covariates were balanced between intervention and control at all stages (eAppendix 5 and eTable 1A in Supplement 1).
Attrition
Due to the online nature of the survey, participants could exit at any point after watching the video messages without finishing the survey. To include as many participants as possible in the analysis, we included everyone who answered knowledge questions for the knowledge outcome. Overall, 18 762 participants included in the initial randomization were included at that stage (9445 Black; 9317 White). For other outcomes, except follow-up outcomes, we included 18 223 participants (9168 Black; 9055 White) who completed the survey. Attrition was similar in all groups (eTable 1B in Supplement 1).
The adherence to safety behavior was collected a few days later among a smaller follow-up sample that experienced more attrition. Overall, 12 591 individuals were included in the follow-up sample to track; only people who had given permission to Lucid to be recontacted were included in this sample. Of those, we successfully contacted 6217. Attrition at this stage was 51.8% in the treatment group and 51.0% in the control group. A systematic analysis of attrition at all 3 stages (eTable 1B in Supplement 1) revealed no systematic difference between the characteristic of attritors in the treatment and comparison groups. At the realized sample size, the power was 0.055 SDs for the knowledge sample (18 762 respondents), 0.056 SDs for all other outcomes in the first survey, and 0.095 SDs for the follow up sample (6217 respondents).
Effects of Any Video Message, Intervention vs Control
Receiving any COVID-19 video improved knowledge of COVID-19 and adherence to preventive practices. Table 2 shows main outcomes overall and by racial group.
Main Sample
The knowledge gap incidence rate was 0.241 (95% CI, 0.235-0.246) in the control group and 0.214 (95% CI, 0.211-0.217) in intervention group. The intervention had a significant effect on reducing knowledge gaps relative to the control group (estimated incidence rate ratio [IRR], 0.89 [95% CI, 0.87-0.91]; P < .001; q < .001). In the control group, 315 participants (8.4%) had no gap in knowledge (Figure 2). The proportion increased to 13.0% (1915 participants) in the intervention group; 1352 participants (35.9%) in the control group had a 1-point knowledge gap compared with 6636 (44.2%) in the intervention group. The incidence rate of information seeking behavior was 0.320 in the control group and 0.338 in the intervention group (estimated IRR, 1.05 [95% CI, 1.01-1.11]; P = .03;
q = .04). The willingness to pay for a mask increased from $14.07 in the control group to $14.58 in the intervention group (difference, $0.50 [95% CI, $0.15-$0.85]; P = .005; q = .013).
The intervention was impactful for both Black and White participants. It was more impactful for White participants vs Black participants on knowledge (IRR, 0.80 [95% CI, 0.76-0.83] vs 0.94 [95% CI 0.91-0.97]; P for difference < .001), but equally impactful for all the other measures (eTable 2C in Supplement 1). The F statistics for the hypothesis that the coefficients across all outcomes are jointly different for both races was 0.0112 (P > .99).
Follow-up Survey
At the follow-up survey (Table 2), which was realized on a smaller sample with larger attrition, the safety gap index incidence rate was 0.47 (95% CI, 0.45-0.48) in the control group and 0.45 (95% CI, 0.44-0.46) in the treatment group (IRR, 0.96 [95% CI 0.92-1.01]; P = .08, q = .08). Overall, 244 participants (20.1%) and 218 participants (18.0%) in the control group and 1040 participants (21.6%) and 837 participants (17.4%) in the intervention group reported respecting all and none, respectively, of 4 safety practices (eFigure 2 in Supplement 1).
Supplemental Outcomes
eTable 3 in Supplement 1 reports effects on secondary outcomes. One noteworthy result is that we found significant effects of the intervention on donations both to a COVID-19 charity (vs Alzheimer) and to a COVID-19 relief charity specific to Black US residents (vs COVID-19 economic relief for everyone).
Effects of the Framing of the Videos
None of the efforts to make the messages more relevant to the Black community had a differential effect on knowledge or individual behavior (Table 3). An F statistic that all effects for Black physician, AMA antiracism statement, and disproportionate burden statement are jointly different from 0 was 0.324 (P > .99).
The only outcome that was affected by some of these variants was the desired donation to a Black-specific COVID-19 charity (eTable 4 in Supplement 1). eTable 5 in Supplement 1 shows that the combination of the AMA antiracism statement, a Black physician, and a video acknowledging racial disparities significantly increased how White and Black participants allocated donations to a charity focused on Black communities (ordinary least squares coefficient, $30.60 [95% CI, $10.93-$50.27]; P = .002).
Heterogeneity by Sex, Education, Income, and Political Affiliation
Across all conditions, there were no statistically significant differences by sex or political affiliation (eTable 2A and eTable 2E in Supplement 1). The effect of intervention relative to control on knowledge gaps was more pronounced for participants with a high school education or more (eTable 2B in Supplement 1). However, there was no significant difference for the demand for links and willingness to pay for a mask. The intervention was more impactful among participants with lower incomes (ie, <$60 000).
Discussion
Exposure to public health video messages about COVID-19 recorded by a diverse set of physicians decreased knowledge gaps on COVID-19 symptoms, preventive behaviors, and transmission among Black and White participants with modest incomes, relative to a control condition that saw placebo videos. The effect on knowledge was substantial and clear. This replicates the results of our prior study conducted in May 20203 and extends it to White participants. New to this study, we also found a modest but statistically significant increase in the demand for more information, the willingness to pay for high quality masks, and self-reported behavior at follow-up.
Despite the heightened awareness of racial justice issues during the period of this intervention and increased polarization in the political discourse in the run-up to the presidential election, effects are remarkably similar across racial and political lines. These results suggest that physicians still have the ability to inform and persuade members of society from a broad range of backgrounds.
Our results also indicate that tailoring the message to specific communities did not affect its impact on behavior. Both White and Black physicians were able to effectively convey the importance of masking and social distancing to Black and White participants (unlike the previous study, in which concordance was important to change behavior3). The AMA antiracism statement did not affect participants’ attentiveness to the message delivered, for Black or White respondents. Acknowledgment of structural racism remains important, but it may not be sufficient to increase the level of trust from the Black community.
Importantly, the intervention made both Black and White participants more willing to focus resources both toward COVID-19 in general and toward the Black community in particular. Highlighting health conditions that disproportionately affect the Black community is one step toward increasing public consciousness of structural racism.
Limitations
There are several limitations of the study. First, it was conducted online, and the participants may not be representative of the population with less than a college degree, given that they have access to the internet and are used to participating in online studies. Second, although information-seeking behavior and willingness to pay for masks were objectively measured, participants’ preventive health behaviors were not directly observed. Outcomes were self-reported. Third, outcomes might be subject to social desirability bias. Fourth, there may be bias due to attrition, particularly for the self- reported safety behavior, given that only a small fraction of the sample could be followed up a few days after the initial intervention. While the observable variables remain balanced, the unobservable may not be. Furthermore, while we found consistent effects on knowledge, information seeking, the willingness to pay for masks, and self-reported behavior, the final clinical significance of these findings is uncertain because effects on all were quantitatively small.
Conclusions
These results suggest physician messaging campaigns may be effective and trust in Black and White physicians is equally high. There is no evidence of preexisting bias that would have led the intervention to have a negative effect. Because it is inexpensive, it could be a promising way to encourage behavior at scale. However, future studies implemented at a large scale are needed to confirm whether these kinds of interventions can change behavior in a way that will affect clinical outcomes. In ongoing work, we will study scale up messaging by doctors using social media.
ARTICLE INFORMATION
Accepted for Publication: May 9, 2021.
Published: July 14, 2021. doi:10.1001/jamanetworkopen.2021.17115
Open Access: This is an open access article distributed under the terms of the CC-BY License. © 2021 Torres C et al. JAMA Network Open.
Corresponding Author: Esther Duflo, PhD, Department of Economics, Massachusetts Institute of Technology, 77 Massachusetts Ave, Building E52-544, Cambridge, MA 02139 (eduflo@mit.edu).
Author Affiliations: Harvard Medical School, Boston, Massachusetts (Torres, Ogbu-Nwobodo, Stanford, Warner); Department of Pediatrics–General Pediatrics, Massachusetts General Hospital for Children, Boston (Torres); Department of Psychiatry, Massachusetts General Hospital, Boston (Ogbu-Nwobodo, Warner); Department of Psychiatry, McLean Hospital, Belmont, Massachusetts (Ogbu-Nwobodo); Harvard Kennedy School of Government, Cambridge, Massachusetts (Alsan); Department of Medicine–Neuroendocrine Unit, Department of Pediatrics–Endocrinology, Massachusetts General Hospital, Boston (Stanford); Department of Economics, Massachusetts Institute of Technology, Cambridge (Banerjee, Karnani, Olken, Vautrey, Duflo); Department of Economics, Harvard University, Cambridge, Massachusetts (Breza); Department of Economics, Stanford University, Stanford, California (Chandrasekhar); Department of Economics, Ludwig Maximilian University of Munich, Munich, Germany (Eichmeyer); Paris School of Economics, Paris, France (Loisel); Yale School of Management, New Haven, Connecticut (Goldsmith-Pinkham); Clinical Translational Epidemiology Unit, Massachusetts General Hospital, Boston (Warner).
Author Contributions: Dr Duflo had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.
Concept and design: Torres, Ogbu-Nwobodo, Alsan, Banerjee, Breza, Chandrasekhar, Eichmeyer, Goldsmith- Pinkham, Olken, Vautrey, Duflo.
Acquisition, analysis, or interpretation of data: Ogbu-Nwobodo, Stanford, Banerjee, Breza, Chandrasekhar, Karnani, Loisel, Goldsmith-Pinkham, Olken, Vautrey, Warner, Duflo.
Drafting of the manuscript: Ogbu-Nwobodo, Alsan, Stanford, Chandrasekhar, Loisel, Olken, Vautrey, Warner, Duflo. Critical revision of the manuscript for important intellectual content: Torres, Ogbu-Nwobodo, Stanford, Banerjee,
Breza, Eichmeyer, Karnani, Goldsmith-Pinkham, Olken, Warner.
Statistical analysis: Breza, Chandrasekhar, Karnani, Loisel, Goldsmith-Pinkham, Olken, Vautrey, Duflo.
Obtained funding: Breza, Goldsmith-Pinkham, Duflo.
Administrative, technical, or material support: Torres, Alsan, Stanford, Breza, Karnani, Goldsmith- Pinkham, Warner.
Supervision: Torres, Ogbu-Nwobodo, Breza, Vautrey, Duflo.
Conflict of Interest Disclosures: Dr Olken reported receiving ad credits from Facebook outside the submitted
work. No other disclosures were reported.
Funding/Support: This work was supported by grant 2029880 from the National Science Foundation to Drs Alsan and Duflo, the Physician/Scientist Development Award from the Executive Committee on Research at Massachusetts General Hospital to Dr Stanford, grant P30 DK040561from the National Institutes of Health to Dr Stanford, grant L30 DK118710 from the National Institutes of Health to Dr Stanford, and the Clinician-Teacher Development Award from the Massachusetts General Physicians Organization at Massachusetts General Hospital to Dr Torres.
Role of the Funder/Sponsor: The funders had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.
Group Members: The members of the COVID-19 Working Group appear in Supplement 3.
Disclaimer: The findings and conclusions expressed are solely those of the authors and do not represent the views
of their funders.
Data Sharing Statement: See Supplement 4.
Additional Contributions: We thank the Center for Diversity and Inclusion, especially Sandra Pena Ordonez, BS, and Elena Olson, JD for their assistance in this project. We thank Minjeong Joyce Kim, BS (Stanford University), and Sirena Yu (Massachusetts Institute of Technology), for their research assistance. These individuals were not compensated for their time.
REFERENCES
1. Zhao J, Lee M, Ghader S, Younes H, Darzi A, Xiong C, Zhang L. Quarantine fatigue: first-ever decrease in social distancing measures after the COVID-19 pandemic outbreak before reopening United States. Preprint updated June 11, 2020. Accessed June 8, 2021. https://arxiv.org/abs/2006.03716
2. McDonnellNietodelRioG.DoctorspleadwithAmericanstotakethevirussurgeseriously.TheNewYorkTimes. Accessed June 8, 2021. https://www.nytimes.com/live/2020/11/15/world/covid-19-coronavirus#doctors-plead- with-americans-to-take-the-virus-surge-seriously
3. AlsanMS,StanfordFC,BanerjeeA,etal.Comparisonofknowledgeandinformation-seekingbehaviorafter general COVID-19 public health messages and messages tailored for Black and Latinx communities: a randomized controlled trial. Ann Intern Med. 2021;174(4):484-492.
4. USCentersforDiseaseControlandPrevention.RiskforCOVID-19infection,hospitalization,anddeathbyrace/ ethnicity. Updated May 26, 2021. Accessed January 21, 2021. https://www.cdc.gov/coronavirus/2019-ncov/covid- data/investigations-discovery/hospitalization-death-by-race-ethnicity.html
5. AmericanMedicalAssociation.AMABoardofTrusteespledgesactionagainstracism,policebrutality.June7, 2020. Accessed June 8, 2021. https://www.ama-assn.org/press-center/ama-statements/ama-board-trustees- pledges-action-against-racism-police-brutality
6. US Census Bureau. 2018 Population estimates by age, sex, race and Hispanic origin. June 19, 2019. Accessed June 19, 2020. https://www.census.gov/newsroom/press-kits/2019/detailed-estimates.html
7. AmericanMedicalAssociation.AMAapplaudscongressionalactiononprescriptiondrugtransparency.April9, 2019. Accessed June 8, 2021. https://www.ama-assn.org/press-center/ama-statements/ama-applauds- congressional-action-prescription-drug-transparency
8. WertenbrochK,SkieraB.Measuringconsumers’willingnesstopayatthepointofpurchase.JMarketingRes. 2002;39(2):228-241. doi:10.1509/jmkr.39.2.228.19086
9. HainmuellerJ.Entropybalancingforcausaleffects:amultivariatereweightingmethodtoproducebalanced samples in observational studies. Political Analysis. 2012;20(1):25-46. doi:10.1093/pan/mpr025
SUPPLEMENT 1.
eAppendix 1. Survey Design and Videos
eAppendix 2. Supplementary Methods
eAppendix 3. Robustness Checks and Subgroup Analysis
eFigure 1. Full Study Flowchart
eFigure 2. Distribution of the Safety Gap Score in the Control and Intervention Groups eTable 1. Balance and Attrition
eTable 2. Outcomes by Subgroup
eTable 3. Effects of Any Message Intervention: Effects on Additional Outcomes eTable 4. Effects of Tailoring Messages on Additional Outcomes
eTable 5. Effects of All Black Treatments on Outcomes
SUPPLEMENT 2.
Pre-analysis Plan and Trial Protocol
SUPPLEMENT 3.
Nonauthor Contributors
SUPPLEMENT 4.
Data Sharing Statement