Smokefree.gov mHealth Evaluation Experts Meeting Re-cap

Topics: mHealth

On December 3, 2015, the National Cancer Institute’s Smokefree.gov team convened a group of 18 behavioral scientists for a day-long meeting to discuss scientific methods for evaluating technology-mediated behavior change interventions, with an emphasis on mobile smoking cessation interventions.

Part of the day was spent answering a series of questions about the current state of evaluation in mHealth research, as well as predictions about and pathways to a future state. This blog post recounts the ideas generated during the December 3 meeting, and details how the group answered three questions about evaluation methods in mHealth research.

Question 1: Has the field, to date, done an adequate job of linking process and clinical outcomes together, and what is the value of doing so?

The group had been primed to think about “process outcomes” as events or behaviors between having access or exposure to an mHealth application, and “clinical outcomes” as outcomes specifically associated with the behavior change of interest (e.g., smoking cessation, lapse, or relapse). It was proposed to the group that a key process outcome is use of the mHealth application – a proposed definition for engagement. Furthermore, engagement is the process outcome that mediates the relationship between access or exposure to an mHealth application and clinical outcomes.

The group countered that this was too simplistic – they argued that while use of an mHealth app is critical to achieving the desired impact of the app, other interim outcomes are relevant, too, such as mood or incremental change towards a goal. The group also suggested that having consensus on a definition of “engagement” would be useful to the field, and while it may be appropriate to define engagement in terms of use of an mHealth application, what constitutes meaningful engagement will differ between users.  For example, one user may achieve smoking cessation after using an app for two days, while another may not achieve cessation until Day 10. Both users have achieved the same clinical outcome – initial cessation – at very different levels of engagement. While it may be that it is reasonable to suspect that more use is associated with better progress towards and achievement of clinical outcomes, the group maintained that the “dose” of engagement required to achieve a clinical outcome may differ between users. Further, the group characterized engagement as a dynamic construct, not a dichotomous one, pointing out that users may initiate, maintain, cease, and re-start use of an mHealth application, and that stopping use (disengaging) and re-starting (re-engaging) is an especially important sequence to understand in a user’s journey with a mobile application. For smoking cessation specifically, given the importance of the first 48 to 72 hours of initial abstinence, the group felt process and clinical measures during this time are especially important.

Finally, the group acknowledged that the industry segment associated with mHealth is more focused on process outcomes as compared to academic mHealth research, which heavily favors an emphasis on clinical outcomes.  This disconnect, they suggested, is not only problematic from a conceptual standpoint, but is also part of what makes successful industry-academic partnerships problematic.

Question 2: What are other examples of process and clinical outcomes?

Even though the group was primed to think about use of an mHealth application as a key process outcome and cessation-associated outcomes (cessation, lapse, relapse) as clinical outcomes, the group was asked to consider other examples of these outcomes as well.  For other process outcomes, some listed were “stickiness” of an mHealth application, or a measure of how consistently people use it. Passively collected data were pointed to as good process measures, including location data, as well as use of other interventions during the time an individual is using the mHealth app of focus. Regarding other clinical outcomes, the group identified proximal, or intermittent outcomes that are known or found to be associated with the primary behavioral outcomes. These included things like psychological factors associated with behavior change (e.g., self-efficacy); self-regulatory capacity; and networking metrics like social support and social connections. The group noted that beyond categorizing variables as process or clinical outcomes, many could be further categorized as contextual (associated with external or internal environmental factors) or theory-related (associated with mediators or moderators identified within theories of health behavior).

Question 3: What limitations of the current state should be considered?

Before answering this question, the group was asked to think of limitations beyond the ones typically identified in mHealth research and evaluation. These typically identified limitations include the differences in the speed at which research is usually conducted and the speed at which technology iterates; how mHealth applications can iterate during the course of a research study, leaving the research team to consider whether participants should continue to use an “old” version of an application or migrate to a new-and-improved release; the larger emphasis on clinical and distal outcomes over proximal or intermittent ones; challenges to maintaining methodological rigor when conducting research outside of controlled laboratory settings; and balancing the need to often collect data from research participants using ecological momentary assessment (EMA), which may not be scalable when thinking about implementing an mHealth application at a population level.

The group identified several additional limitations and challenges that the field of mHealth research and evaluation currently faces. For example, there are limitations to the conclusions one can draw about the use or effect of an mHealth application when the application is being used in the context of other mobile applications or clinical interventions that may target the same behavior. Another limitation related to the relatively short length of time that mHealth research has been active, is standardizing agreement on what constitutes sufficient proof of effect. Additionally, the group felt that there is often an expectation that an mHealth application will have the same effect as a non-mobile version of a behavior change intervention, but that this expectation may not be realistic or appropriate.

However, many in the group felt that these limitations can be addressed methodologically. Specifically, the group pointed to the data collection capabilities that are possible using a mobile application, and felt that intensive data collection would provide insights into the use and effects of an mHealth application, even when participants are using the application in a non-controlled environment. Further, increased data volume affords opportunities for insight into the mechanisms by which mHealth applications can influence behavior. The group felt strongly that the field of mHealth research will not be able to meaningfully advance unless studies carefully capture the data that will enable an understanding of what worked, what didn’t, and for whom and under what circumstances. Capturing these kinds of data will be instrumental to improving the design and implementation of mHealth application over time.


First Name: 
Last Name: