JANUARY 4, 2005

Review of the quality of evaluations across Canadian departments and agencies

(Reproduction of the executive summary of a study conducted by the Centre of Excellence for Evaluation and available on its Web site)

Introduction

Evaluation supports the Government of Canada's aim to becoming a learning organization. It does this by helping senior executives, program managers and policy makers discover whether or not their initiatives work and are meeting objectives, whether or not there is a continued need for their initiatives, and how their initiatives can be better designed and delivered to meet objectives in a cost-effective manner. The quality of evaluation reports is fundamental if the evaluation function is to deliver upon these information needs.

Purpose

In 2001, TBS created the Centre of Excellence for Evaluation (CEE) and established a new Evaluation Policy to strengthen the evaluation function and the quality of reporting. A key objective of this report is to address whether the quality of reports is acceptable and whether there has been an improvement in quality. An important aspect of this work is to promote quality evaluation reports. This review represents one piece of CEE's overall strategy to monitor and strengthen the quality of reporting. Other activities include: best practice research; an annual survey of the health of departmental and small agency evaluation units; individual meetings; ongoing review of evaluations, RMAFS, departmental evaluation plans; and, an annual report documenting evaluation findings and how they contribute to strengthening accountability and the government's Expenditure Review exercise.

Methodology

A number of sources were used to develop the criteria used for this review including the "Guide for the Review of Evaluation Reports", prepared by the Centre of Excellence for Evaluation, TBS, January 2004 and excerpts from the OAG 1993 Report on Program Evaluation ("Criticisms re Evaluation Reports"). A reference group of department and agency evaluation units was also consulted.

Findings

The findings of this review indicate that most federal evaluation reports are acceptable in quality, though almost one-quarter of the evaluations (23 per cent) were rated as inadequate overall. No clear and consistent variations in quality were observed for federal organizations of different sizes and for departments versus agencies. A comparison of reports completed pre- versus post-April 2002 indicates, however, that quality has improved on a number of criteria in the more recent evaluations. For example, this includes: addressing cost-effectiveness issues; methodological rigour; identifying alternatives; presentation of evidence-based findings; and, formal recommendations. This increase in quality over time suggests that TBS's efforts to improve the quality of evaluation may be having a positive impact (i.e., allowing one year, until April 2002, for the Policy to be fully understood by departments/agencies and for the new Centre for Excellence in Evaluation to begin operating). Still, there is a pressing need for further improvement as indicated by the findings noted below.

Key strengths of the evaluations examined in this review include:

• a comprehensive description of the program/initiative under review including its resources, beneficiaries and stakeholders;
• a clear statement of the evaluation objectives;
• the use of multiple lines of evidence in the methodology;
• a strong presentation of findings, in particular, on relevance and delivery/implementation issues;
• formal recommendations or suggested improvements that flow logically from the findings and conclusions; and
• reports that are well-written and well-organized.

On the other hand, some weaknesses of evaluations/reports are:

• only six in ten evaluation reports indicated the timing and significance of the evaluation;
• most of the reports only listed (two-thirds) and very few discussed (about one-quarter) the evaluation issues;
• superficial coverage of cost-effectiveness issues;
• many reports lacked a full description of the key methodological details. While just over half of reports described the methodology, four in ten only listed a few details and only one-quarter referenced a technical document;
• there is little incorporation of data from a performance measurement system;
• only a minority of the evaluation designs included features to optimize the rigour of the research such as a comparison group (13 per cent), baseline measures (14 per cent) or a comparison to norms, literature or some other benchmark (22 per cent). Only 26 percent used interviews with independent key informants with no stake in the program;
• only about four in ten evaluation reports included a statement of the limitations or constraints of the evaluation;
• only about one-third of evaluations presented findings on whether the program duplicates or works at cross purposes with other programs/initiatives;
• only one-quarter of the evaluations discussed unintended outcomes (25 per cent) or addressed incremental impacts (26 per cent);
• only 26 per cent of evaluations provided findings on alternative, potentially more cost-effective approaches, though coverage of this issue has increased in more recent reports (31 per cent post-April 2002 versus 16 per cent pre-April 2002);
• almost one-quarter of evaluations (24 per cent) were rated as inadequate in their provision of objective, evidence-based conclusions related to relevance, success and/or cost-effectiveness;
• among the reports with recommendations, only 26 per cent identified alternative scenarios ; and,
• less than half of the evaluation reports included a management response (48 per cent) or action plan (33 per cent).
• 25% of reports with recommendations included a recommendation related to overall funding, and in all of these cases, the recommendation was to increase funding.
• No reports presented evidence indicating that a program was not relevant or not needed.

Conclusions and Recommendations

On balance, most evaluations that were assessed in this review are of reasonable quality. The majority received an overall rating of adequate (45 per cent) or more than adequate (32 per cent). Still, a considerable proportion of the evaluations (23 per cent) were rated as inadequate and this finding warrants attention. To this end, the report recommends that the TBS Centre of Excellence for Evaluation:

1. Encourage evaluation divisions in federal departments and agencies to strengthen their evaluation reports by addressing the major weaknesses identified in this review;
2. Refine Treasury Board guidelines/criteria for the expected features of (1) evaluation methodologies and (2) evaluation reports and disseminate them;
3. Continue to implement a rigorous approach to monitoring the quality of evaluations, and use this as a basis for the development of individual report cards on the quality and overall health of the evaluation function by department and small agency; and,
4. Identify measures, including an incentive structure and standards, to ensure that departments and agencies submit completed evaluations and reviews in a responsible, reasonable manner. Departments and agencies adherence to such standards should be made a public record.

 Return to the home page
 List all news