obÌåÓý

Overview and Importance

Step 5 involves generating evaluation conclusions, which are answers to the evaluation questions supported by the data collected in Step 4. This step involves the following:

• Reviewing the evidence expectations identified previously
• Planning data analysis
• Conducting data analysis
• Interpreting the analytic findings
• Forming recommendations

Data analysis and interpretation in this step transform the raw data into meaningful and actionable recommendations. As you begin Step 5, refer to the expectations of interest holders established in Step 4. Specifically, engage interest holders in:

• Discussing the analysis plan (may include quantitative and qualitative analysis techniques)
• Interpreting analyses
• Drawing evaluative conclusions
• Testing the feasibility of potential recommendations to ensure conclusions are responsive to the context and underlying data
• Clearly planning for how and when you will involve interest holders in each of these activities

Refer to the full-length obÌåÓý Program Evaluation Action Guide for additional information, examples and worksheets to apply the concepts discussed in this step.

Review Expectations with Interest Holders

The expectation setting with interest holders that happened in Step 4 can be the basis for developing the data analysis plan. Review what the interest holders stated regarding:

• What evidence will be used to answer the evaluation questions
• What expectations they have about the type, quality, and quantity of data needed
• What indicators and measures are most important
• What changes, trends, or patterns suggest the program is on track or doing well

Planning Data Analysis

A data analysis plan includes how and by whom the data would be organized, analyzed, and synthesized. A plan that is feasible, appropriate, and aligns to the evaluation questions and data collection methods identified in earlier steps is more likely to produce findings that will be used by the interest holders. Evaluators will want to collaboratively engage interest holders to obtain agreement on the techniques for analysis and interpreting findings before data collection begins to promote transparency.

Develop the data analysis plan early because it will outline the series of choices you will make. The plan will also include the program's context, external factors, and implementation changes. Prepare your data analysis plans before data collection begins to ensure that data collected can be analyzed and meets the needs of interest holders, including program leadership. Involve interest holders in reviewing these plans and the expectations for success established in Step 4.

Conducting Data Analysis

Common steps for quantitative and qualitative analysis are described below.

Common key steps for quantitative data analysis:

• Clean and pre-process raw data for accuracy and consistency. Check for missing data, standardize data formats, and make any other corrections needed.
• Import data into a statistical software program such as SAS, STATA, SPSS, R, or Excel.
• Analyze data based on the indicators defined in Step 4.
• Calculate descriptive statistics to describe, summarize, and compare key characteristics about the data (totals, frequency counts, or percentages). For some indicators, data will need to be stratified or grouped based on variables of interest first.
• Explore distributions and variations within the data. Analyze trends, patterns, and relationships within the data.
• Draw conclusions from the findings.
• Evaluation questions looking to determine the association between program activities and the intervention or desired outcomes (changes in knowledge, attitudes, behavior, health status, or systems changes) will require more advanced statistical analyses such as means comparisons or regression analysis.

Common key steps for qualitative data analysis:

• Transcribe recordings from interviews or focus groups and/or enter narrative comments from surveys or field notes into a word processing or qualitative analysis software such as nVivo, Excel, MAXQDA, or ATLAS.ti.
• Review and annotate the data to understand and familiarize yourself with the content. This could involve reading interview transcripts or listening to recordings.
• Organize the data for instance, by date, by data collection type, or by question asked.
• Code the text to identify and label key themes (trends or ideas that appear throughout) that correspond to your evaluation questions.
• Group text by key themes.
• Review the themes and codes for refinement and to identify any sub-themes that emerge.
• Draw conclusions from the findings.
• More advanced qualitative analysis techniques include the use of multiple coders, calculation of interrater reliability, and within or between-case analysis.

Interpreting Analytic Findings

The interpretation process is where meaning is made from the analytic results. When reviewing results, consider how the findings compare to the expectations for success outlined with interest holders in Step 4. Findings also can be reviewed alongside the existing evidence base or applicable scientific theories or models.

A shared interpretation process allows evaluators and interest holders to collaborate and discuss the strengths, limitations, and interpretations of the findings within context. Seek perspectives from various interest holders, especially those directly impacted by the program and familiar with the program context. The evaluator may have to balance interpreting findings with including interest holder perspectives as this will influence on the interpretation of findings. When different but equally well-supported conclusions exist, each could be presented with a summary of its strengths and weaknesses. The culmination of this step is generating evaluative conclusions.

There are several ways to conduct a collaborative process of reviewing analytic results, translating findings, and validating the interpretation, including:

• Using data dashboards to share and generate discussion on interim findings
• Establishing collaborative interpretation activities such as data walks
• Hosting roundtable discussions
• Facilitating data storytelling (for instance, by using narratives and visualizations to convey insights from data)
• Re-purposing project meetings to discuss findings throughout the evaluation

It is important to think through the best approaches for your evaluation's unique interest holders, and the format may vary between different groups of interest holders. At the end of the process, answers to the following questions may be generated:

• How do the findings compare to the expectations for success established among interest holders prior to data collection?
• Do any changes need to be made because of the evaluation findings?

Forming Recommendations

After interpreting the analytic results of the evaluation and forming evaluative conclusions, the next step is to outline recommendations for action. Ensure recommendations are clear and concise, evidence-based, and grounded in the program context. Facilitating recommendation discussions with interest holders allows you to gauge the feasibility of putting them into action. Prioritize and outline which recommendations are more critical and feasible to address first, who will address them, on what timeline, and provide multiple options for action when possible. Establish a process to document and share recommendations with interest holders, as well as a process to follow up on progress.

Applying the Cross-Cutting Actions and Evaluation Standards to Step 5

As with all the evaluation framework steps, it is important to integrate the evaluation standards and cross-cutting actions when generating and supporting conclusions in Step 5. See Table 9 in the obÌåÓý Program Evaluation Framework, 2024 to determine if you have effectively applied the evaluation standards and cross-cutting actions.