Shchetkina, Anya and Ron Berman, "When Is Heterogeneity Actionable for Targeting?" Accepted at ACM EC 2024. [Abstract] [Paper]
We apply five popular personalization approaches to two large-scale field experiments with many interventions (aka megastudies) aimed at increasing vaccination rates (the Walmart study of Milkman et al. (2022) and the Penn-Geisinger study of Milkman et al. (2021) and Patel et al. (2023)). We find limited value of targeting in the Walmart experiment and a four times higher value of targeting in the Penn-Geisinger experiment. We seek to explain the difference in the gains from personalization between the two studies and show that the presence of heterogeneity alone is not sufficient to predict whether a targeting exercise will be successful. Instead, a specific form of heterogeneity, which we call ``actionable'' heterogeneity, determines the value of targeting. We demonstrate how the amount of actionable heterogeneity depends on three forces: (1) within- and (2) cross-treatment heterogeneity, as well as (3) cross-treatment correlation. For studies with many interventions, such as the ones we analyze, determining the magnitude of actionable heterogeneity can be challenging. To aid this task, we develop a model that estimates the value of personalized policies compared to the best untargeted intervention using three simple summary statistics of the data. We find that the value of actionable heterogeneity of the Penn-Geisinger study is higher than that of the Walmart study, which can explain the difference in the observed values of targeting. Our model also illustrates conditions when adding more treatments to an experiment may hurt the value of targeting even in infinite samples.
De La Rosa, Wendy, Ron Berman, Christophe Van den Bulte, Nidhi Agrawal, Adam L. Alter, Christopher J. Bechler, Jonathan E. Bogard, J. Anthony Cookson, Kylie Davis, Ayelet Fishbach, Craig R. Fox, Ayelet Gneezy, Hal E. Hershfield, Tatiana Homonoff, Aziza C. Jones, Lena Kim, John G. Lynch, Tamutswa Mahari, Erick M. Mas, Eesha Sharma, Anya Shchetkina, Jackie Silverman, Abigail B. Sussman, Patricia Torres, Stephanie M. Tully, Broderick L. Turner, Jr., Esther Uduehi, Oleg Urminsky, Adrian F. Ward, Vince Dorie, Gwen Rino, Maximilian Hell, and Eric Giannella, “Increasing Interest in Claiming a Tax Credit: Evidence from Two Large-Scale A/B/n Field Experiments Among Lower Income People”. R&R at Marketing Science.
Dew, Ryan, Nicolas Padilla, and Anya Shchetkina "Your MMM Is Broken: Identification of Nonlinearities and Dynamics in Marketing Mix Models" [Abstract] [Paper]
Recent years have seen a resurgence in interest in marketing mix models (MMMs), which are aggregate-level models of marketing effectiveness. Often these models incorporate nonlinear effects, and either implicitly or explicitly assume that marketing effectiveness varies over time. In this paper, we show that nonlinear and time-varying effects are often not identifiable from standard marketing mix data: while certain data patterns may be suggestive of nonlinear effects, such patterns may also emerge under simpler models that incorporate dynamics in marketing effectiveness. This lack of identification is problematic because nonlinearities and dynamics suggest fundamentally different optimal marketing allocations. We examine this identification issue through theory and simulations, wherein we explore the exact conditions under which conflation between the two types of models is likely to occur. In doing so, we introduce a flexible Bayesian nonparametric model that allows us to both flexibly simulate and estimate different data-generating processes. We show that conflating the two types of effects is especially likely in the presence of autocorrelated marketing variables, which are common in practice, especially given the widespread use of stock variables to capture long-run effects of advertising. We illustrate these ideas through numerous empirical applications to real-world marketing mix data, showing the prevalence of the conflation issue in practice. Finally, we show how marketers can avoid this conflation, by designing experiments that strategically manipulate spending in ways that pin down model form.