JANUARY 11, 2013

Evaluation Use as Influence: Article Review

An article recently published by Sarah Appleton-Dyer, Janet Clinton, Peter Carswell and Rob McNeill in the American Journal of Evaluation (vol. 3, no. 4, pp. 532-546) piqued our interest. In a time when evaluation budgets for government programs are shrinking, the question of evaluation use becomes increasingly salient. The authors, in Understanding Evaluation Influence Within Public Sector Partnerships: A Conceptual Model, ground their endeavour to explore this theme in the literature.

As Appleton-Dyer, Clinton, Carswell and McNeill point out, "evaluation use (...) is a complex phenomenon that is still not well understood." (p. 532) They choose to leave behind the four typically-inventoried types of evaluation use that they briefly describe (instrumental, conceptual, symbolic and process use ) and focus on evaluation use as influence, because "this term encompasses traditional conceptions of use, as well as various changes at the individual, interpersonal and collective levels." (p. 533) The authors also see value in highlighting "the numerous factors that can affect the influence of an evaluation, including context." (p. 533) The article draws on the literature of other disciplines to map out elements that may contribute to evaluation use. These elements are broken down into various categories, which, based on the literature, may interact with one another:

• Evaluation attributes;
• Partnership characteristics;
• Partnership functioning;
• Partnership evaluation behaviour;
• Mechanism of evaluation influence;
• Individual characteristics;
• Outcomes of evaluation influence; and
• Other contextual factors.

Comment

• While we appreciate the authors' efforts to systematize the profession's thinking about evaluation use as influence in public sector partnership, we feel like their research constitutes more of a scan of the available literature than an effort to think outside the box. However, we can hope that a heightened focus on evaluation use as influence will lead to more innovative enquiries on the topic.
• Of the elements that may contribute to evaluation use that are summarized in the article, it appears that only a few are within the evaluator's control. For instance, the evaluator has a limited role to play in the determination of the partnership characteristics, and, in the case of external evaluators, the evaluation attributes are already determined before the evaluation team becomes involved.
• Moreover, we think that contextual factors should be given more importance in the analysis of the elements contributing to evaluation use. Matters of organizational culture, political climate, etc. were only touched upon, when they may indeed play a central role in the evaluator's ability to steer an evaluation's influence for the best.
• Lastly, notwithstanding the authors' intentions, we remain unsure that the model of evaluation influence portrayed in this article really addresses the question of complexity as it applies to the topics that tend to generate partnerships in public health. As the authors put it, "in public health specifically, partnerships are often working to tackle issues such as obesity, teenage pregnancy, and social exclusion." (p. 533) These types of issues have been characterized as wicked problems by other authors[1], and it is said that these problems' complexity and shifting nature needs to be embraced to allow for creative, collaborative problem solving. However, whether the model for evaluation use as influence proposed in the article can indeed be applied to complex – wicked – public health issues remains to be explored.

  [1] The term was originally coined in the field of urban planning in 1973 by Rittel and Webber. See the followign brief overviews: New Tools For Resolving Wicked Problems and An Introduction to Wicked Problems

   Return to the home page
   List all news