Programming will begin after specifications that adhere to SRO guidelines for instrument or application programming and design are final. SRO CAI instruments for interviewer-administered surveys are programmed in Blaise, an interviewing software system developed by Statistics Netherlands. SRO has developed guidelines for specifying, programming, and documenting Blaise Surveys using the Michigan Questionnaire Documentation System (MQDS).  

Web surveys have been programmed in Illume (developed by DatStat, Inc. in Seattle) and Blaise IS (Internet Survey, development by Statistics Netherlands), among other software tools. SRO uses the University-provided Web survey tool Qualtrics for internal surveys, including training assessments. For the most part, SRO Web survey guidelines apply to any Web survey software.

Testing verifies that the instrument matches the specifications and is error free.  The main objectives are to identify and correct defects and to make sure that the instruments and systems meets stakeholder requirements. Figure 7.2 provides a flowchart of the typical testing process, which is iterative.

The project Technical Leader assigns a member of the team the Testing Coordinator role, whose responsibilities are to:

• Develop the plan and schedule for testing;
• Provide comprehensive testing documentation;
• Provide ongoing support to testers;
• Ensure that the scheduled completion dates are met;
• Identify and arbitrate conflicting tester feedback;
• Summarize changes to be made for programmers; and
• Ensure that all changes are made and tested.

The plan is a guide used to coordinate testing and breaks the process into manageable pieces using key milestones, such as releases or deliverables. It sets programming and testing goals for each cycle of testing, and outlines testing staff resources and their assigned tasks. In addition, it should always include integrated instrument, systems, and protocol testing on production platforms, and performing dataset quality assurance checks.  

The testing plan may include testing scenarios for complex instruments. Such scenarios generally have a table or list of “respondent types” that generally reflect target subpopulations with specific characteristics or responses (e.g., women over 44 or woman under 44 who have never been pregnant). These facilitate testing instrument skip logic or sample management and selection protocols.

The Technical Leader may serve as the Testing Coordinator, or may assign coordination to another technical team member

The CAI Testing Tool (CTT) provides a user interface to assist those testing Blaise applications. It allows the user to record bugs or changes needed in the application at the question level, and collects and manages comments across all testers. This reduces the programming time needed for making corrections and changes. CTT provides a seamless point of entry in the instrument, as well as reports to help manage all the notes collected during testing.

Programming and testing may involve use of evaluation methods such as prototyping, usability tests, and reviews by substantive or methodological experts to assess the “usability” of the instrument. These methods may be used iteratively during CAI instrument development, with early usability tests of prototypes of new question or section designs and later expert reviews or usability tests of a completed instrument. Usability tests usually take place in the SRO Instrument Development Laboratory. They may supplement pretests of CAI instruments (see next section), through the use of software for recording interviewer-respondent interaction and screen images on interviewer laptops.

Conventional pretests.  SRO strongly recommends conducting a conventional pretest for all survey instruments and procedures, in order to evaluate them in the field prior to data collection. A pretest is usually completed on a small sample. It may include as few as 20 respondents or as many as several hundred, depending on the size of the expected production sample and number of targeted subgroups. Pretests can use a convenience sample or some type of random sample. See Section 7.8 for additional information on pretests.

Behavior coding.  Taped pretest interviews may also be behavior coded (Oksenberg et al, 1991).   Behavior coding is a method of recording the frequency of interviewer and respondent behaviors that may indicate problems with questions or potential interviewer effects.  Interviewer behaviors include whether or not questions were asked as worded (major and minor wording changes) and whether or not interviewers used appropriate probing and reinforcement. Respondent behaviors that may indicate question problems are asking for clarification or asking the interviewer to repeat the question or response options.  

Behavior coding data has been shown to reliably identify problems, but not necessarily the source of the problems. Thus, they are sometimes used in conjunction with cognitive interviews, where interviewers can probe respondents directly about what they are thinking as they respond to questions.

Expert reviews.  Expert reviews (“desktop pretests”; Biemer and Lyberg, 2004) may supplement the conventional pretest. In such reviews, one or more experts (substantive and/or methodological) review the instrument to identify questions that may be misunderstood, have formatting or layout problems, confusing instructions, etc.  They may be unstructured or structured. RTI’s Questionnaire Appraisal System (QAS-99; see references) provides a model for a structured review of an instrument.

The goal of translating questionnaires is to achieve different language versions of the English questionnaire that are conceptually (i.e., semantically and culturally) equivalent in each of the target countries or cultures. That is, the instrument should be equally natural and acceptable and it should perform in the same way in practice. The focus is on cultural rather than linguistic or literal equivalence.

Best practice for translation is to use a team approach, in which a group of people work together. The group includes translators, reviewers, and at least one adjudicator. The team should include those knowledgeable about the substantive research area, the principles of questionnaire design, and the cultural norms of the targeted population. 

Translators produce a draft translation, review it with reviewers, and seek the advice of an adjudicator where there are differences in proposed translation. As with a source language instrument, any translated instruments should be pretested, using one or more evaluation methods (e.g., conventional pretest, cognitive interviews, focus groups, etc.). Finally, the translation and pretesting protocol should be documented. This approach is known as the TRAPD Translation Model (Translation, Review, Adjudication, Pretesting, and Documentation).

The purpose of a pretest is to ensure that:

• The questions are clear and easily understood by the target population;
• The answers to the questions follow the intent of the question as written;
• The flow of the questions is conversationally logical;
• Interviewer or respondent instructions are clear; and
• The interviewer or respondent can record an answer in the expected format.

Pretest procedures should mirror the actual production procedures as closely as possible, including sample administration. If interviewer-administered, interviewers should minimally attend a briefing outlining the goals of the pretest. When study procedures are complex or innovative, consideration should be given to conducting a full pretest interviewer training prior to pretest interviewing.

Interviewer debriefings should be conducted after pretesting to obtain feedback on any potential problems with the instrument or the study procedures. Debriefing sessions can be in person or via conference call.  

For CAI studies, data management quality checks should be performed, and a dataset and data dictionary from the pretest interviews should be delivered to the client in the same format as specified for production data delivery. Production should not start until the client signs off on the data dictionary and data delivered.

Audit trail reports.  Since one of the key goals of pretesting is to obtain an accurate estimate on the length of the interview, data on section and question timings should also be delivered.  Item timings and items at which certain functions are used frequently (consistency checks, backing up, suspensions, etc.), may also be used to identify problem questions or sections.   Blaise collects item times, keystrokes, and functions used in an audit trail keystroke file (.ADK) for each interview. The Audit Trail Reporting System (ATReport) provides reports (case-level, item- or section-level, and interviewer-level) on ADK data, and a means of exporting them for further analysis. Project team members may also access audit trail by attaching directly to the project’s Audit Trail Database, and generate custom reports.

Before the start of pretest or production interviewing, the questionnaire must be formally approved by the PI or a designated member of the project staff and the appropriate IRB review committee(s).  See Chapter 3, “Project Management,” for more detail on IRB submissions.

Before production starts, the PI must also sign off on the dataset and data dictionary.

Biemer, P.P., and L.E. Lyberg.  2003.  “Overview of Survey Error and Evaluation Methods,” Chapter 8 in Introduction to Survey Quality.  New York:  John Wiley & Sons, Inc.

Converse, J.M., and S. Presser.  1986.  Survey Questions: Handcrafting the Standardized Questionnaire.  Thousand Oaks, CA:  Sage Publications, Inc.

Couper, Mick P., Michael W. Traugott, and Mark J. Lamias. 2001. Web Survey Design and Administration. Public Opinion Quarterly 65(2): 230-253.
 
Dillman, Don. A. 2007 [2000].  Mail and Internet Surveys: The Tailored Design Method.  New York: John Wiley and Sons, Inc.

Groves R.M., F.J. Fowler Jr., M.P. Couper, J.M. Lepkowski, E. Singer, and R. Tourangeau.  2009.   “Methods of Data Collection” (Chapter 5), “Questions and Answers in Surveys” (Chapter 7), and “Evaluating Survey Questions” (Chapter 8), in Survey Methodology. New York: John Wiley & Sons, Inc.
 
Forsyth, B.,  and J. Lessler.  2004.  “Cognitive Laboratory Methods: A Taxonomy” (Chapter 20), in P. P. Biemer, R.M. Groves, L.E. Lyberg, N.A. Mathiowetz, S. Sudman (Eds.), Measurement Errors in Surveys.  New York:  John Wiley & Sons, Inc.

Oksenberg, L., C. Cannell, and G. Kalton.  1991.  “New Strategies for Pretesting Survey Questions.”  Journal of Official Statistics 7(3):349-365.

Presser, S., M.P. Couper, J.T. Lessler, E. Martin, J. Martin, J.M. Rothgeb, and E. Singer (Eds), Methods for Testing and Evaluating Survey Questionnaires.  New York:
  
Presser, S., and J. Blair.  1994.  “Survey Pretesting:  Do Different Methods Produce Different Results?”  Sociological Methodology 24:73-104.

Stewart, D.W., P.M. Shamdasani, and D.W. Rook.  2007 [1990].  Focus Groups: Theory and Practice.  Thousand Oaks, CA:  Sage Publications, Inc.
 
Sudman, S., and N. Bradburn.  1982.  Asking Questions:  A Practical Guide to Questionnaire Design.  San Francisco:  Jossey-Bass.

Willis, G.B.  2005.  Cognitive Interviewing:  A Tool for Improving Questionnaire Design.  Thousand Oaks, CA:  Sage Publications, Inc.

Willis, G.B., and J.T. Lessler.  1999.  Questionnaire Appraisal System:  QAS-99.  Rockville, MD :  Research Triangle Institute.  http://appliedresearch.cancer.gov/areas/cognitive/qas99.pdf

http://www.websm.org/ — A Website focused on research on Web survey methodology and, more widely, the use of technology for data collection.