Use Evaluation Synthesis for Evidence-Based Decision-Making

Evaluation synthesis can be a powerful tool for synthesizing the results of multiple evaluations and gaining a comprehensive understanding of the effectiveness of programs, projects or interventions. Recognizing its importance United Nations and development agencies are increasingly making its use for evidence-based decision making. .

Evaluation synthesis can also be particularly useful in addressing complex or cross-cutting issues. By bringing together the results, best practices, and recommendations of evaluations across different areas, it is possible to identify the broader social and economic context in which the intervention is taking place. One of the key benefits of evaluation synthesis is that it allows for a more comprehensive and nuanced understanding of the effectiveness of programmes and projects. Synthesis of the results of evaluations also allows to identify common themes, patterns, and trends that may not be apparent from individual evaluations. This can help to identify areas where improvements can be made and to inform future program or policy development.

‍

Types of Evaluation Synthesis

‍

There are four types of evaluation-based syntheses: Rapid Evidence Assessments (REAs), Synthesis Review, Meta-Synthesis, and Meta-Analysis. Each of these methods has its strengths and weaknesses and selection of synthesis type depends on the best use scenario depends on needs and available resources.

Rapid Evidence Assessments (REAs)	REAs are a fast and systematic review of the evidence that provides a quick overview of a particular subject area or in response to a specific research question. They can provide quick lessons learned and resourcefulness in time- and resource-scarce contexts Prone to confirmation bias and can lead to false sense of evidence-based decision-making.	Prone to confirmation bias and can lead to false sense of evidence-based decision-making.
Synthesis Review	A synthesis review is a high-level overview of primary research on a focused question that identifies, selects, reviews, and appraises all reliable research evidence relevant to that question. They can provide recommendations, assessing available evaluations guided by a specific question and in cases where there is uncertainty on a particular intervention theme or cross-cutting issue.	Requires subjective judgment calls.
Meta-Synthesis	A meta-synthesis qualitatively aggregates available data to not just review previous research, but also help testable working hypotheses on why an intervention may work (or not). Hence, it is often a first step in developing hypotheses for future testing with more reliable data. Meta-synthesis is the most common type of synthesis used in evaluation.	Provides only a qualitative summary of included studies and may not justify investments.
Meta-Analysis	A meta-analysis quantitatively synthesizes studies that address a common research question, providing reliable evidence and uncovering causal mechanisms. It requires statistically trained researchers and is more expensive and demanding than the other methods.	More expensive and demanding than the other methods.

‍

Evaluation Synthesis Tools

Evaluation synthesis often involves analyzing large amounts of data from various sources, which can be a time-consuming and challenging task. These are some of the software that can be sued to help qualitative data (coding) and quantitative analysis.

NVivo: NVivo is commonly used for qualitative analysis, including organizing and coding large amounts of textual data. NVivo allows for easy tagging and categorization of data, making it useful for meta-synthesis and other types of evaluation synthesis.

Atlas.ti:.Atlas.ti can also be used to code evaluation reports for identifying patterns and themes in data.

R: A free, open-source programming language, R is widely used for quantitative data analysis. It has a vast array of statistical analysis tools and packages that can be used for meta-analysis and other types of evaluation synthesis.

Stata: Similar to R, Stata is a statistical software package used for data analysis, visualization, and management. It has a wide range of built-in tools and commands for meta-analysis and other types of evaluation synthesis.

Excel: Excel can be a useful tool for organizing and analyzing both qualitative and quantitative data. It can be used for creating coding sheets, keeping track of data, and creating visualizations.

Practical use of Evaluation Synthesis

A good evaluation synthesis is based on multiple data sources and mixed methods of analysis, validation, and triangulation of evidence. The robustness of evaluation synthesis mainly depends on the quality of the evaluations it is synthesizing. Evaluation Quality Assessment (EQA) of Evaluations can also help ensure that evaluations meet criteria to increase the usefulness and credibility of the synthesized results. A post-hoc assessment system can facilitate the screening of good evaluations, and a meta-evaluation can be used to assess the quality of individual evaluations.

In addition to ensuring the quality of evaluations, involving stakeholders throughout the process can further increase the credibility and usefulness of evaluation synthesis. By consulting with stakeholders from the outset and throughout the process, priorities can be confirmed, expectations can be managed, and the analysis can be validated and enriched with their input. This can also help with disseminating and sharing the results.

Human rights & intersectionality

Intersectionality in evaluation synthesis recognizes that individuals are not defined by a single identity or characteristic, but rather by the complex and interconnected nature of their identities. By acknowledging the intersection of various identities such as race, gender, age, and socio-economic status, evaluation synthesis can better capture the nuanced experiences of different groups of people. By incorporating an intersectional lens in evaluation synthesis, it is possible to identify disparities and inequalities that may not be evident from a single identity perspective. Intersectional approach also helps decision-makers better understand the needs and experiences of different groups and develop more targeted and inclusive interventions in future.

Conclusion

Choosing the appropriate type of evaluation synthesis requires careful consideration of requirements, available resources, and time frame. Rapid assessment is suitable for time-sensitive decisions, while meta-synthesis is useful for developing testable hypotheses. Meta-analysis is ideal for understanding the impact of a body of work, while systematic review can provide recommendations based on findings. Properly conducted evaluation synthesis can provide valuable insights for evidence-based decision making by bringing together the results of multiple evaluations to provide a comprehensive understanding of the effectiveness of a program, projects, or intervention. To ensure the credibility and usefulness of evaluation synthesis, evaluators should use a rigorous and transparent methodology, triangulate and validate data, involve stakeholders throughout the process, and conduct proper evaluability assessments. With the availability of innovative tools such as NVivo and Atlas.ti and online available database of evaluation reports, evaluators can more efficiently and accurately conduct evaluation synthesis.

Want to know more or learn how evaluation synthesis might help your organization? Feel free to get in touch by emailing us at: care@deftedge.com

‍