Well-valuation: Ripple Effects Mapping as a Method for Wellness Program Evaluation

Well_valuation

Wellness professionals are increasingly required to demonstrate program impacts. Evaluation of these impacts can be elusive because they are contingent upon dynamic social processes of change that may not be fully understood, controllable, or predictable. In these situations, qualitative evaluation methods like Ripple Effects Mapping are well-suited for constructing holistic information about wellness program outcomes. This post describes how to implement the five phases of the REM evaluation process.

Ripple Effects Mapping

Ripple effect mapping (REM) is a participatory method of qualitative data collection that seeks to identify intended and unintended effects (positive and negative) of a program using Appreciative Inquiry (AI). It is useful in situations where the results of programs occur over time within complex settings and can be used to explore outcomes at both the individual and organizational levels. In REM, participants are asked to consider the social, emotional, physical, intellectual, occupational, and spiritual dimensions of their life as they share their experiences in, and after, the program.

Synergies with Wellness

Wellness is defined by the National Wellness Institute as an active process through which people become aware of, and make choices toward, a more successful existence. It is a process in which the social, spiritual, occupational, intellectual, emotional, and physical dimensions of individuals and groups function in harmony. Wellness promotion is a humanistic practice that leverages methods grounded in salutogenensis, self-determination, authentic relationship, inter-professional collaboration, and an inter-disciplinary knowledge base to cultivate wellness. Like wellness, Ripple Effects Mapping is a holistic, context-responsive, and person-centered evaluation method that can help understand how a program impacts the wellness journey. The Appreciative Inquiry style questioning used in REM is similar in nature to wellness coaching in that it is non-directive, strengths-based, and growth-oriented. Instead of focusing on solving problems, AI seeks to generate new ways of thinking, identify opportunities, and catalyze new patterns of behavior by cultivating “more good” instead of “less bad”. Therefore, the use of REM in evaluation is particularly congruent with the aims of wellness promotion.

Implementing REM and Building a Grounded Theory of Change

There are two phases to the REM process: interactive interviewing and group mind mapping. In a third phase, the evaluation team uses a grounded theory style analysis to construct a theory of change to explain the origins the intended and unintended outcomes that emerge. This approach to REM works best when conducted with a sequence of stakeholder groups consisting of up to 10 individuals each. The sampling strategy may entail seeking out extreme or deviant cases, typical cases, maximum variation, critical cases, politically sensitive cases, or convenience sampling at each phase. Once the first group of participants has been selected, the REM sessions can begin.

Phase I: Peer-to-Peer Interviews

In phase I of REM, session participants begin by interviewing each other using a semi-structured, pre-determined appreciative inquiry protocol. Participants share stories in pairs about their peak experience in the program, what they value about the program, and what they wish for the program in the future. Opening the REM session with peer-to-peer sharing allows participants to become comfortable in the environment and recall their past experiences. Participants are instructed to think holistically about their experiences and describe these to their partner. This phase may last approximately 10-15 minutes.

Phase II: Group Mind Mapping

In the second phase of the REM process, the group comes together to create a mind map. A mind map is a diagram that concentrically organizes information to show relationships among elements of a system over time. This phase should be audio-recorded so the evaluation team can return to the transcripts for rich details. The facilitator begins the initial round of questions from an interview guide and each participant responds by relating their experience orally. After communicating orally, the facilitator invites the participant to write a brief statement about this experience within a set of concentric circles.These concentric circles can be interpreted as changes over time, with the inner ripple representing changes occurring at a time more proximal to program participation and the outer ripple representing changes occurring at a time more distal to program participation.

Questions in the initial round may include but are not limited to:

  1. What is a highlight, achievement, or success you experienced as a direct result of your involvement with the program?
  2. What unexpected things (positive or negative) have happened as a result of your involvement with the program? And,
  3. What connections with others—new and/or deepened—have you made as a result of the program? The facilitator may probe participants to expand upon their answers as needed.

Transitional

Once the facilitator has completed the first ripple of questioning and probing for the transactional (ripple 1) changes, the facilitator draws another circle concentrically around participants’ initial responses and moves to protocol questions that elicit experiences for the transitional (ripple 2) and transformational (ripple 3) ripples. The protocol continues with questions such as:

  1. How did you benefit from [experience participant wrote in previous circle]?
  2. What happened because of [experience participant wrote in previous circle]?
  3. As a result of [experience participant wrote in previous circle], how has your attitude or behavior changed? and,
  4. How are these changes benefiting others or changing what others do?

As before, participants will orally communicate their experience and write a brief note about it in the new circle. This time, they will connect the new experience to its generating experience with a line. This sequence can be repeated multiple times to gather as many transitional (ripple 2) and transformational (ripple 3) outcomes as participants are willing to share and lasts approximately 1-1.5 hours.

3 level

 

At the end of each mind-mapping session, participants are asked to provide their interpretation of the experiences shared by the group and to identify the most valuable outcome they experienced as a result of their participation in the program. These responses can be elicited using questions like:

  1. What relationships do you see between different aspects of the map?
  2. What do you think is the most significant change on the map? Why?
  3. How valuable are these changes? Why? and
  4. What from this map that will be helpful in improving the program for the future?

This final round of questioning contributes to the ability of an evaluator to fulfill their role in making informed judgement about the merit, worth, and significance of the program. It also aids evaluators in constructing a parsimonious theory of change that, as Weiss (1995) recommends, concentrates on key aspects of the program identified as most valuable or significant by program stakeholders.

Phase III: Interviewing Other Stakeholders

Ripple Effect Mapping application typically brings all stakeholder groups together to create maps that represent the range of possible experiences and interpretations. In some circumstances, different stakeholder groups may participate in the REM sessions alongside one another (blended sessions). However, in other circumstances, blended REM sessions may not be appropriate due to power dynamics or concerns for confidentiality. Therefore, different stakeholder groups may participate in separate REM sessions divided along power structures (split sessions). A third option is to conduct individual interviews with key stakeholders who are unable to attend group sessions (individual sessions). In this third situation, the evaluator can imitate the REM sessions using the previously constructed mind maps.

Using digital mind mapping tools, the evaluator can convert the large paper-based mind maps into condensed, de-identified digital mind maps. Then, in each interview, after the evaluator asks questions similar to those asked in the group REM session s/he shows the digital mind map from the group REM sessions to the interviewee. After the interviewee has a change to review the mind map, the evaluator continues the interview with questions like:

  1. What surprised you about the map?
  2. What themes or patterns do you see in this map?
  3. What was confirmed by the map that you already knew? and
  4. What was missing in the map that you thought you would see?

This line of questions allows for divergent assumptions to emerge about how a program works and compare them with what is developing in the data.

Phase IV: Analysis & Codifying a Theory of Change

Labeling the analysis and theory construction as phase IV is slightly misleading. Since grounded theory leads the evaluator to make use of an iterative, comparative process, each stage of data collection should be followed by a stage of analysis and theory construction generating new levels of complexity and sophistication in the emerging theory. After each REM session or interview, the evaluator transcribes the audio files and creates initial codes. Completing this prior to engaging in subsequent REM sessions helps the evaluator to start forming ideas and hypotheses about the emerging themes to adapt the interview protocol and probing in theoretically important directions. The next step in analysis is coding the transcripts. If you are unfamiliar with qualitative coding, Saldana is a great resource.

The theoretical codes generated from the data should help link program activities, transactional changes, transitional changes, and transformational changes to each other and construct a coherent story about how the program works and its impacts. In thinking about these inter-relationships within the program system, Williams & Hummelbrunner (2010) provide the evaluator several questions to ask themselves:

  1. How do the categories affect each other? Does a category have a reinforcing or dampening effect on the categories to which it is linked?
  2. What controls the way in which resources flow through the situation? How does this affect behavior?
  3. What are the key processes for achieving the outcomes of the program?
  4. How do we make sense of the patterns that emerge from those processes, with what consequences, and for whom?
  5. What are the consequences for achieving effects if those processes do not take place as foreseen?

The theory of change that emerges from this rich, systematic process is grounded in the lived experiences of stakeholders interacting with the program in a given time, rather than a priori beliefs or assumptions of the underlying program mechanisms that link activities to outcomes. The theory of change must adequately connect and describe expected causal connections without becoming so complex that the utility of the model is weakened.  Given the complex system dynamics, this theory must also be open to continuous change as new information emerges.

Phase V: Member Checking

Stakeholder involvement in evaluation is considered good practice and member checking is recognized as a method for enhancing both participation and rigor in qualitative inquiry. This final phase of the REM process aims to improve the plausibility, credibility, and relevance of the evaluation by including stakeholders in the interpretation and confirmation of the emerging theory of change. During member checking, stakeholders are invited to comment on the findings in the form of individual interviews, focus groups, or written communications to “test” the plausibility, credibility, and relevance of the emerging theory. It also helps ensure that evaluation findings are presented in a way that is accessible to all stakeholders. Member checking can increase acceptance of the evaluation findings and implementation of recommendations but requires careful ethical considerations in the evaluation protocol.

Final Thoughts

Ripple effect mapping is a participatory data collection method that helps an evaluator explain program outcomes and the process through which they emerge. Such insights into a program may remain buried with goal-oriented, quantitative evaluation methods. Although this paper makes a distinction between intended and unintended outcomes, this approach to REM within a goal-free evaluation framework can avoid making a distinction between intended outcomes (goals) and side-effects because inn task of evaluating merit, worth and significance, “what counts is not the specific intentions, but the results” (p. 97).

Mixed-method evaluations can help identify and address unintended consequences that may occur during program implementation. It can also improve the credibility, relevance, and ultimate use of the evaluation findings among stakeholders with different information needs. Ripple-effect mapping may be integrated into a mixed-method design that includes additional qualitative or quantitative components. For example, an evaluator may use a survey to examine intended outcomes among a large sample and recruit REM participants. Alternatively, the evaluator might follow the REM process with a survey to assess the theory of change among a larger sample, allowing for additional revision and refinement.   

 

Thanks for reading! For a sample REM facilitation guide click Ripple Effects Mapping Sample Facilitation Guide.

 

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s