Smokefree.gov mHealth Evaluation Experts Meeting Re-cap
On December 3, 2015, the National Cancer Institute’s Smokefree.gov team convened a group of 18 behavioral scientists for a day-long meeting to discuss scientific methods for evaluating technology-mediated behavior change interventions, with an emphasis on mobile smoking cessation interventions.
Part of the day was spent answering a series of questions about the current state of evaluation in mHealth research, as well as predictions about and pathways to a future state. This blog post recounts the ideas generated during the December 3 meeting, and details how the group answered three questions about evaluation methods in mHealth research.
Question 1: Has the field, to date, done an adequate job of linking process and clinical outcomes together, and what is the value of doing so?
The group had been primed to think about “process outcomes” as events or behaviors between having access or exposure to an mHealth application, and “clinical outcomes” as outcomes specifically associated with the behavior change of interest (e.g., smoking cessation, lapse, or relapse). It was proposed to the group that a key process outcome is use of the mHealth application – a proposed definition for engagement. Furthermore, engagement is the process outcome that mediates the relationship between access or exposure to an mHealth application and clinical outcomes.
Finally, the group acknowledged that the industry segment associated with mHealth is more focused on process outcomes as compared to academic mHealth research, which heavily favors an emphasis on clinical outcomes. This disconnect, they suggested, is not only problematic from a conceptual standpoint, but is also part of what makes successful industry-academic partnerships problematic.
Question 2: What are other examples of process and clinical outcomes?
Even though the group was primed to think about use of an mHealth application as a key process outcome and cessation-associated outcomes (cessation, lapse, relapse) as clinical outcomes, the group was asked to consider other examples of these outcomes as well. For other process outcomes, some listed were “stickiness” of an mHealth application, or a measure of how consistently people use it. Passively collected data were pointed to as good process measures, including location data, as well as use of other interventions during the time an individual is using the mHealth app of focus. Regarding other clinical outcomes, the group identified proximal, or intermittent outcomes that are known or found to be associated with the primary behavioral outcomes. These included things like psychological factors associated with behavior change (e.g., self-efficacy); self-regulatory capacity; and networking metrics like social support and social connections. The group noted that beyond categorizing variables as process or clinical outcomes, many could be further categorized as contextual (associated with external or internal environmental factors) or theory-related (associated with mediators or moderators identified within theories of health behavior).
Question 3: What limitations of the current state should be considered?
Before answering this question, the group was asked to think of limitations beyond the ones typically identified in mHealth research and evaluation. These typically identified limitations include the differences in the speed at which research is usually conducted and the speed at which technology iterates; how mHealth applications can iterate during the course of a research study, leaving the research team to consider whether participants should continue to use an “old” version of an application or migrate to a new-and-improved release; the larger emphasis on clinical and distal outcomes over proximal or intermittent ones; challenges to maintaining methodological rigor when conducting research outside of controlled laboratory settings; and balancing the need to often collect data from research participants using ecological momentary assessment (EMA), which may not be scalable when thinking about implementing an mHealth application at a population level.
The group identified several additional limitations and challenges that the field of mHealth research and evaluation currently faces. For example, there are limitations to the conclusions one can draw about the use or effect of an mHealth application when the application is being used in the context of other mobile applications or clinical interventions that may target the same behavior. Another limitation related to the relatively short length of time that mHealth research has been active, is standardizing agreement on what constitutes sufficient proof of effect. Additionally, the group felt that there is often an expectation that an mHealth application will have the same effect as a non-mobile version of a behavior change intervention, but that this expectation may not be realistic or appropriate.
However, many in the group felt that these limitations can be addressed methodologically. Specifically, the group pointed to the data collection capabilities that are possible using a mobile application, and felt that intensive data collection would provide insights into the use and effects of an mHealth application, even when participants are using the application in a non-controlled environment. Further, increased data volume affords opportunities for insight into the mechanisms by which mHealth applications can influence behavior. The group felt strongly that the field of mHealth research will not be able to meaningfully advance unless studies carefully capture the data that will enable an understanding of what worked, what didn’t, and for whom and under what circumstances. Capturing these kinds of data will be instrumental to improving the design and implementation of mHealth application over time.