You are entitled to your opinion. But you are not entitled to your own facts.
(Daniel Patrick Moynihan)

QUALITY CRITERIA FOR SURVEY RESEARCH

(The most complete and up-to-date version of the document describing the survey research assessment framework is available on this site).

Traditionally, the assessment of survey research has been more of an art than a science. Every researcher uses their own criteria and put more or less emphasis on various aspects of the research. This occurs in the literature when one author comments on the weight of the evidence brought to bear on a competing or supporting point. It also takes place when clients assess the value to give survey evidence they commissioned or that is used in support of a business decision. It happens during one's self-examination of a research report to ensure that all elements of the required proof are offered.

However, the depth of examination of research evidence is nowhere more profound than when survey research is used in the judicial system. In this instance, two or more parties have every reason to either accept or reject the evidence; all parties tend to emphasize only the areas of the proposed evidence which support their position. Meanwhile, the courts and the quasi-judicial bodies need to be able to rely on a proven approach to assessing survey research so as to ensure that no important component of the research is left unexamined.

The experience of Circum's president, Benoît Gauthier, with the judicial system as a survey expert has motivated the development of a framework to assess survey evidence. It has also been used as a self-assessment tool, as a checklist to confirm the quality and completeness of the research. As a matter of course, Circum Network survey-related work abides by the principles and criteria defined in this framework for quality survey research.

The assessment framework is structured around the six building blocks of survey research, on the one hand, and on four fundamental principles of assessment on the other hand. They are listed below, along with a brief definition.

Fundamental Principles

Components of Survey Research

RIGOUR
Does the study systematically apply recognized best practices of survey research? Rigour is, of course, relative to the situation at hand. Certain research situations call for some methodological choices while others require other approaches. By essence, rigour is the most technical of the principles used here; yet, the specific rules of assessment are likely to evolve through time as we develop more knowledge on best survey practices.

NEUTRALITY
Does the study observe the phenomena of interest without biassing the observations? The object of research is to depict reality faithfully; therefore, sound research must take steps to ensure that its depiction is neutral in its representation, that reality has every chance of surging in a representative way from the observations made.

BALANCE
Does the study use sufficient resources to demonstrate its thesis adequately, but no more than are necessary for this purpose? Balance demands both sufficiency and parsimony. Balance is the position one adopts to achieve stated objectives with a minimum use of resources, efforts, burden, etc.. According to the Oxford dictionary, balance refers to, among other things, "harmony of design and proportion" and the verb "to balance" is defined as "to bring into or to keep in equilibrium".

TRANSPARENCY
Does the study provide all the relevant background needed for an informed assessment? Reproduction, a basic building block of the formation of scientific knowledge, requires shared knowledge or intersubjectivity. The corresponding fundamental requirement is the full disclosure of all elements of methods which may have a bearing on the quality of the observations and on the strength of the conclusions. Transparency is probably the most fundamental precept: without it, the other criteria have no meaning in terms of public communication of research results.

QUESTIONNAIRE
The tangible form of issues raised by the research, it must perform the translation of concepts into indicators and allow for the back-translation of observations into conceptual interpretations. The questionnaire also shapes the design of the proof available to the researcher.

SAMPLING
The selection of a subset of the study target population, sampling raises issues of representativity and precision.

DATA COLLECTION
Data collection comprises the procedures used to collected the required information. In survey research, data collection usually takes on five forms: telephone surveying, in-person interviews, paper-and-pencil questionnaires, electronic data collection and mixed mode approaches.

DATA MANAGEMENT
Data management subsumes procedures which transform data to make it ready for analysis. It includes data adjustments, data coding, data weighting, computations. This episode is often (and wrongly) relegated to junior staff.

DATA ANALYSIS
The analysis of the data collection, using statistical and graphical tools, this phase constructs the proof required by the issue at hand.

REPORT
The report presents the research journey and the results obtained. It usually takes the form of a written report. Different issues are raised by scientific and management reports.

As is the case with Generally Accepted Principles of Sound Management (from which the structure of this survey research quality assessment framework is borrowed), the survey quality criteria are produced by the combination of the fundamental principles with the components of survey research. This combination produces a 6 x 4 matrix. Therefore, our survey quality framework offers criteria in areas such as the rigour of the sampling, the neutrality of data collection, the balance of data management and the transparency of reporting. As of May 2001, the 24-cell assessment matrix contained 58 quality criteria ranging from the absence of bias in the questionnaire to the avoidance of excessive weighting. The most recent complete (to date) description of the assessment framework is available on this site.

Even armed with such a systematic quality assurance tool, research analysts must still exercise a healthy dose of judgment in the assessment of survey research. However, the use of a full disclosed and exhaustive analysis tool will help discipline the assessment of survey research as well as improve the confidence that the survey researcher himself/herself has that their research is really up to standards.