For every complex problem there is an answer that is clear, simple, and wrong.
(H. L. Mencken)

Document T060

Performance Measurement and Evaluation Framework for the Canada Research Chairs Program 2000—2004

Report presented to the Chairs/NCE Evaluation Steering Committee, December 2001

The Canada Research Chairs Program (hereafter, the program) is a key piece in Canada's strategy to become a world leader in the knowledge-based economy. The program's contribution to the strategy is to enable Canadian universities to create research opportunities that will attract and retain the outstanding researchers. It is aimed at developing 2000 university chairs between years 2000 and 2005. The 2000 Budget provided $900 million over five years toward that objective — in addition to $250 million earmarked within the Canadian Foundation for Innovation (CFI) to support the program. The program is managed by the Social Science and Humanities Research Council on behalf of a management committee where the Natural Sciences and Engineering Research Council, the Canadian Institutes of Health Research, the Canada Foundation for Innovation as well as Industry Canada are also represented.

Program logic

The program is based on the pre-allocation of research chairs to universities and disciplines based on funds received from granting councils in the past. Universities present nominations which are assessed through a peer-review process. The review takes into consideration both the established excellence of the researchers (for Tier I chairs, valid for seven years and renewable without limit) or the potential for excellence (for Tier II chairs, valid for five years and renewable once) and the contribution that the nominee could make to the realization of the university strategic research plan.

The program aims at producing two critical short term impacts: retaining the best Canadian researchers as well as attracting excellent researchers from other countries (or Canadian expatriates). The program also pursues a peripheral objective of ensuring the effective use of research resources through strategic planning by the institutions as well as the inter-institutional and inter-sectoral collaboration.

The establishment of a more productive research infrastructure environment is expected to contribute to offsetting "brain-drain" pressures. A series of impacts on the university research environment should flow from the reduction of pressures on key personnel: easing the construction of dynamic research teams; reinforcing the training of graduate students through inclusion in a world-class research environment; producing more and better graduate students and improving retention of highly qualified personnel in Canada.

In the long run, the goal of the program is to lead to a strengthened and more highly competitive research environment in Canada's universities as compared to our competitors in the world, and enhanced Canadian visibility in the global knowledge-based economy. Ultimately, the program should contribute to the overall government objectives of a strong economy and of a better quality of life.

Evaluation issues

This evaluation framework paves the way for ongoing performance monitoring, the mid-term review and the evaluation planned in the fifth year of program implementation. It is based on a review of program documentation, interviews with 30 key informants and a literature scan.

Evaluation issues have been identified within three groups:

process issues deal with: program take-up, the allocation formula, the peer review process, selection criteria, gender equity, the disposition of reserve chairs through an open competition and the efficiency of the CFI and CRC interface;
short-term impact issues include: integration with other council programs and other chair programs, the appropriateness of the number of chairs, the retention and attraction effects, the excellence of the chair holders, the balance between Tier I and Tier II chairs, the displacement of personnel, the effects beyond the hiring of a researchers, university commitment to the chairs and the generation of additional research funds;
longer-term impact issues comprise: risks taken by universities, the stability of chair holders, contribution to the training of highly qualified personnel, impacts on the capacity of the university research system's capacity to produce and apply new knowledge, contribution to inter-institutional and inter-sectoral collaboration, distinctive effects on smaller universities, progress in the realisation of the strategic research plans and possible unintended effects, in particular, the possible impact on the equilibrium of the Canadian university system.

Evaluation methods

Priorities have been established within the set of issues and timing of the study of the issues has been laid out. Indicators and data sources have been associated with each issue. The following data sources have been identified as requirements for a full evaluation of the program:

statistical reports using program data;
annual chair and university reports;
interviews with university representatives, key stakeholders and management committee members;
special requests for information from universities;
statistical reports using data from the three granting councils;
selection committee reporting forms;
survey of faculty members — chair holders and other faculty members;
survey of selection committee members;
case studies;
a series of special studies: a gender equity study, an international peer review of chair holders, a comparison of selection criteria worldwide, a study on the open competition concept and an international comparison of the ratio of chairs to the number of faculty members.

Ongoing performance measurement was given particular attention. It includes program activity data drawn from statistical reports produced by the program as well as information on immediate impacts from special requests to universities and annual chair and university reports.

Evaluation options were developed. They correspond to a minimal but credible package which deals with highest priority issues; a more complete package which provides more qualitative information relative to the incrementality of the program and more complete data collection on the issue of excellence; and a comprehensive package including all indicators and data sources.

52 pages, 647k [PDF format]

Document T079

Evaluation of the Networks of Centres of Excellence Program

Prepared for the Interagency Evaluation Steering Committee, October 2007

This evaluation study was the Networks of Centres of Excellence (NCE) Program's third in the ten years. It was required as part of the renewal of the program's Terms and Conditions and it was conducted for the Interagency Evaluation Steering Committee on behalf of the NCE Steering Committee.

98 pages, 506K [PDF format]

Document T080

Evaluation of the Tri-Agency Indirect Costs Program

Prepared for the Interagency Evaluation Steering Committee, July 2009

The present evaluation study of the Indirect Costs Program (ICP) was conducted in preparation for the renewal of its terms and conditions. It was conducted for the presidents of the Canadian Institutes of Health Research (CIHR), the Natural Sciences and Engineering Research Council of Canada (NSERC) and the Social Sciences and Humanities Research Council of Canada (SSHRC).

118 pages, 474K [PDF format]

Document T081

Canada Graduate Scholarships (CGS) Program and Related Programs Review

Prepared for the Interagency Evaluation Steering Committee, November 2008

This evaluation study of the Canada Graduate Scholarships (CGS) program was conducted in preparation for the renewal of its Terms and Conditions. It was conducted for the Presidents of the Canadian Institutes of Health Research (CIHR), the Natural Sciences and Engineering Research Council of Canada (NSERC) and the Social Sciences and Humanities Research Council of Canada (SSHRC). It was managed by the Interagency Evaluation Steering Committee which is comprised of program and evaluation representatives from all three Agencies as well as Industry Canada.

167 pages, 714K [PDF format]