FEBRUARY 7, 2012

Evaluation of Web 2.0 initiatives: a lot of ground to cover

The Treasury Board Secretariat (TBS) recently made public its new Guideline for External Use of Web 2.0, which is intended to cover "Internet-based tools and services that allow for participatory multi-way information sharing, dialogue, and user-generated content". The Guideline suggests some uses of Web 2.0, identifies risks and emphasises the role of evaluation in this new era of communication.

The Guideline document identifies several potential uses for Web 2.0 tools, such as recruitment, risk and emergency communication, services to the public, stakeholder outreach and education, collaboration, and consultation.

On the subject of uses of Web 2.0 by government, Barkat, Jaeggli and Dorsaz recently published a report documenting several examples of fruitful uses of social media in government settings. The report, Citizen 2.0: 17 examples of social media and government innovation, presents ways in which citizen have engaged governments: "By providing inexpensive and widely available tools to make it easier to organize and voice challenge, new technologies have contributed to empower citizens while improving governments' responsiveness and accountability. A new citizen is emerging." (p. 3)

The TBS Guideline raises numerous risks and challenges. They include:

• difficulties reconciling existing obligations with technological constraints (e.g., ensuring bilingualism in a 140-character message);
• potential misuse of content shared, for instance when subject to a third party's terms of use;
• misuse of Web 2.0 platforms;
• misunderstanding of the boundary between individuals' opinions & online activities and official Government of Canada positions;
• challenges in protecting user privacy, be they personnel or members of the general public; and
• legally binding terms of use, which may present legal risks when not adequately understood or respected.

According to TBS's Guideline, departments should ensure that their use of Web 2.0 feeds into their mandate by building it into regular departmental planning exercises. In addition, TBS suggests that departments evaluate Web 2.0 initiatives regularly to assess:

• the extent to which the tools or services are meeting their expected outcomes;
• that the tools or services are reaching their intended target audiences;
• the level of compliance with policies, procedures and legal obligations; and
• the extent to which evaluation findings are leveraged to improve the initiative.

However, the evaluation community may not be moving as fast as the public, as very little research was identified about evaluating Web 2.0 or other deliberative initiatives. As Abelson et al, explain, "the paucity of rigorous evaluations is [...] of concern for those looking to draw generalizable lessons to inform the design of more effective participation processes in the future." (p. 249) [1]. Similarly, a Canadian Health Services Research Foundation (CHSRF) paper lead by the same author underscores the gap in knowledge around public consultations: "Despite broad mandates for involving the public, we know little about what influences the choice of consultation method, the results obtained using different approaches, or the extent to which they have been evaluated." (p. 2).

In the Social Science and Medicine article, the authors suggest that Weber's framework for evaluating public deliberation models be utilized. Weber's framework revolves around criteria of fairness and competence; they also place value in Beierle's view of the following elements as "key components of any evaluation of a deliberative process: (1) representation; (2) the structure of the process or procedures; (3) the information used in the process; and (4) the outcomes and decisions arising from the process." (p. 244)

While citizens and public instances are becoming increasingly amenable to using Web 2.0 tools in public policy contexts, significant research ground remains to be covered in the field of evaluation to follow the trend.

[1] Abelson, Julia, Forest, Pierre-Gerlier, Eyles, John, Smith, Patricia, Martin, Elisabeth and Gauvin, François-Pierre (2003). "Deliberations about deliberative methods: issues in the design and evaluation of public participation processes", in Social Science and Medicine 57, pp. 239-251.

 Return to the home page
 List all news