mHealth Monitor

Author Name
Published Date
Wed, December 21, 2016
Blog Topics
Behavior Change
Behavioral Health
Monitoring and Evaluation
Smoke Free
Smoking Cessation

“You can’t always get what you want, but if you try sometimes, you just might find, you get what you need…” – Mick Jagger, Rolling Stones

Now, I don’t think Mick was thinking about the evaluation of health interventions when he wrote these lyrics, but they ring true for anyone who has ever tried to measure behavioral outcomes in a real world setting; for example, measuring outcomes associated with using an easily accessible, population-level intervention in which the users are free to enter and leave the program as they choose.  Understanding the relationship between health interventions and a behavioral outcome can be incredibly difficult, especially if you are doing it outside of a study environment (e.g., not within a randomized controlled trial). Using smoking cessation as an example, unless you are physiologically measuring the presence (or biochemical evidence) of a recent smoking event, there is no way to tell for sure if someone has actually quit smoking. In a communication environment that is shifting more and more towards digital delivery, and a funding environment that places prominence on cost-effective and direct lines of care, it makes it even more challenging to know if your intervention efforts are making any difference. How can you really know if your intervention is working if you can’t directly observe what your intervention is intended to impact?

Power to the Proxy

That’s where the proxy indicator comes in. A proxy indicator can be defined as an “indirect measure or sign that approximates or represents a phenomenon in the absence of a direct measure or sign.” (http://www.businessdictionary.com/definition/proxy-indicator.html#ixzz4HcLv4axH). When you can’t get what you want (final outcome), you find a measure or measures that will get you what you need (proxy indicator) to inform intervention development. Think of proxy indicators as smoke and your outcome as a fire; the proxy can provide the data you need to evaluate the efficacy of your intervention without the expense and methodological obstacles that can come with difficult-to-measure outcomes.

Choosing Your Proxy: A Case Study 

Knowing that you need a proxy measure is all well and good, but how do you actually go about choosing them for your measurement plan? Let’s turn again to smoking cessation for a case study. The Smokefree.gov Initiative (Smokefree) is a wide-ranging digital smoking cessation program from the National Cancer Institute. Smokefree uses websites, social media, mobile apps, and text messaging programs to help Americans quit smoking. Quitting smoking is therefore the primary outcome measure of interest. As previously mentioned, measuring a successful quit at the individual level within a population-level intervention can be a huge challenge. For example, Smokefree is a federally funded digital program that reaches millions of persons each year which presents large cost hurdles for the collection of biologically-confirmed or directly observed outcome measures.  Within this context, the Gold Standard is self-reported data on smoking status which can be used to gauge quit status.  However, self-report can be unreliable due to response bias (including a likely sense of pressure to report being smoke-free, even if the user hasn’t quit) and high nonresponse rates to assessment as users in population-level programs are “Free Range Humans” with minimal incentive to respond to repeated assessment questions.  In addition, multiple data management challenges arise including privacy concerns, structuring and analyzing “big data” (dealing with massive datasets with multiple variables and values), and a number of other factors that require intense resources and effort to implement and analyze. 

So, what can you easily measure to get you as close as possible to a reliable report of “I quit smoking?” An ideal option is to choose a measure that has been shown either in your own work or in the literature to have a strong relationship to your outcome of interest. For example, a study by Richardson et al. (2013) found an association between how many times an individual visited a web-based intervention and smoking cessation rates. So, in the simplest of terms, the more times that an individual visited the website, the more likely they were to quit smoking. While the site in the study was not a Smokefree website, the features and content aligned well with Smokefree. Based on these findings and similar findings from internal web metric analyses, Smokefree regularly tracks engagement (e.g., visits, visitors, time on site, pages per visit, bounce rate, specific event tracking, form submission) and return engagement with their various websites. In particular, the proportion of return visitors to the website and visitors who visit or use other Smokefree resources (social media, text messaging, and mobile apps) are used as an approximation for a quit attempt. While these data are not at the individual level, and exact rates of cessation are not possible, it does provide an indirect way for Smokefree to gauge the efficacy of their efforts to have people attempt to quit smoking. Such data can also be used in combination, as multiple indicators of engagement and usage can paint a clearer picture of a quit attempt. Proportion of return visitors can also be combined with a visitor’s average time spent on the website to paint a clearer picture of the depth of the quit attempt.  These engagement measures, tracked over time and compared via more detailed analyses, get us closer to “I quit” and can guide intervention improvements.*

Two last things to note: first, racking proxy measures like engagement will provide an indicator of a final outcome, such as quit smoking attempts or success, but it may not provide a hyper-specific number associated with the final outcome. It is not a 1:1 equation or detailed estimate, it is more of a guiding post to inform your intervention work. If possible, finding ways to do confirmatory studies with your own intervention that shows the association between your proxy measure and a specific outcome is ideal, so you know with certainty that your proxy is valid. It may even provide the jumping off point for an equation or algorithm that allows you to leverage the proxy to estimate outcomes such as quit smoking attempts or successful quits. Second, theory can guide the selection of proxy outcomes as well as data observed in previous studies. In this way, any number of behavioral health conceptual frameworks or studies that align well with your own outcomes could help to narrow down the measures that should be examined.

 *Note: Smokefree utilizes a variety of different usage, engagement, and secondary outcome measures to inform the development and improvement of intervention offerings. This is just a single example of a proxy indicator, one of many that is used by the program. 

Wrapping Things Up

So, if you are managing or working on a health intervention that may not have the time or resources to conduct full-fledged outcome studies, consider looking into using proxy indicators. Each health behavior and intervention is unique, so be sure to do your homework to find studies or conduct preliminary analyses that align with your goals and that you feel confident will get you close to your primary outcome. Even if you can’t directly measure a health outcome, understanding the steps to or around that behavior can still provide valuable data on the success of your intervention efforts.

NIH… Turning Discovery Into Health®