11. Post Data Collection Processing
11.1 Introduction
This chapter covers the project activities and deliverables necessary for post data collection processing for SRO projects.
SRO Standard Project Procedures
Following are mandatory and conditional post data collection deliverables for SRO data collection:
Deliverables
- Data operations project checklist;
- Cleaned dataset(s);
- Dataset documentation;
- Data release approval form(s); and
- Technical sampling, weighting, and outcome rate report.
Deliverables
- Survey weights and the components of the survey weights;
- Complex sample design (stratum, cluster) variables;
- Editing checklist;
- Coding progress report;
- Coding verification report;
- Data entry progress report;
- Data entry verification report;
- Confidential materials receipt; and
- Memo of understanding.
Note that these latter deliverables are neither mandatory nor optional. Rather, they are conditional, based on the study design. For example, a coding verification report is not required if SRO does not code open-ended questions for the project.
Figure 11.1 provides a flow diagram of post data collection processing activities and related project management processes.
SRO best practices for post data collection processing are described in sections 11.2 through 11.12. The chapter closes with references.
- 11.2 Data extraction;
- 11.3 Editing and cleaning data;
- 11.4 Coding open ended questions and other/specify text;
- 11.5 Data entry;
- 11.6 Scanning;
- 11.7 Weighting and estimation;
- 11.8 Outcome rate calculations;
- 11.9 Technical report;
- 11.10 Confidentiality assurance;
- 11.11 Dataset documentation; and
- 11.12 Dataset delivery.
11.2 Data Extraction
Data processing begins with the extraction of data from the data collection software [e.g., computer assisted interviewing (CAI), data entry, or Web] and the sample management system used for the project. “Data Extraction” may also occur in the form of receipt of a raw data set from a third party vendor, such as a scanning company or test scoring company.
Activities for this task could include, but are not limited to:
Specify data sets to be produced
The Project Leader will specify for the Data Manager the number and type of data sets to be produced. This information will be included in the project plan and work scope definition, which are reviewed and approved by the Senior Project Advisor (SPA) and Principal Investigator (PI)–see chapter 6 of the Data Operation’s Project checklist for an example of a data delivery table.
Specify variables to extract
The Project Leader works with the PI and Data Manager to define the variables to extract from each system, and to specify the destination of the output files. Depending on the type of project and source of the data, the dataset may need more extensive up-front definition, including: dataset structure (e.g., vertical – so that data from multiple waves of collection can be appended, with variable names the same across waves, or horizontal – so that each wave of collection is separate, with variable names unique to each wave); values and their labels; dataset order; or determining whether variables from other sources should be merged. For consistency and comparability, care should be taken to ensure that variables shared across files and waves have the same specifications.
Check for identifying information
Special care should be taken to ensure that variables containing identifying information are kept in separate and secure files, and are not included in any data set with survey answers.
11.3 Editing & Cleaning Data
Data cleaning will be performed on all data sets before they are delivered to the PI. The amount and type of data cleaning will be specified in the contract or description of work.
Activities for this task could include, but are not limited to:
Specify and program consistency checks
The best way to produce a clean dataset is to identify and program consistency checks into the CAI, Data Entry (DE), or Web application, to catch and correct errors at the data collection stage. The Project Leader will include in the scope statement a description of the amount and type of consistency checking that will be included in the CAI or DE application.
Specify data cleaning requirements
The Project Leader will include in the scope statement the amount and type of post-interview data cleaning that will be done—see chapter 6 of the Data Processing and Quality Control Project Checklist for data cleaning options).
Data Processing and Quality Control Project checklist
Clean data
The Data Manager will clean all datasets as defined in the scope statement, contract, and/or description of work.
Specify editing work scope for paper/pencil questionnaires
The Project Leader will include in the scope statement a description of the amount and type of editing and consistency checking required for the project.
Edit paper questionnaires
Paper questionnaires must be reviewed and edited before being sent to data entry. This includes assigning numeric codes for open or other/specify text responses, adding missing data codes, and making sure there is no ambiguity about what should be entered (e.g., resolving illegible handwriting or bad skips).
11.4 Coding Open-Ended Questions & Other (Specify) Text
Most surveys have some text responses, in the form of open ended questions or other/specify items. Responses to these questions are reviewed by a coder and a numeric code is assigned for use in analysis. The amount and type of coding to be done on a project should be included in the work scope description and approved by the PI and SPA.
Activities for this task could include, but are not limited to:
Build a code frame
A code frame is a list of categories into which the answers will be placed. If no code frame exists, Survey Services Lab (SSL) staff can create a frame from pretest interviews, or from the first set of production interviews. All code frames are approved by the Project Leader, SPA, and the PI before they are implemented.
Specify and program CAI coding application
The coding application is specified by the Project Leader and programmed by the SSL staff. SSL coding applications often display the value of related variables to provide context within which the verbatim answer will be coded.
Write coding instructions and train coders
The Project Leader and the SSL coding supervisor write instructions for coders (often including objectives for complex questions) and provide a training class that coders must attend before working on a project.
Code opens and “other—specifies”
SSL staff members assign numeric codes to the text answers in the questionnaire, and enter the data into the Blaise application. Where necessary, the Project Leader or other members of the research team are consulted for clarification or changes to the code frame.
Check-code open ends and “other—specifies”
Cases are independently coded by different coders and decisions are compared for at least 10% of the cases (with a higher percentage of check-coding at the beginning of the study and for new coders). The purpose of check-coding is to catch differences between coders, to ensure that each coder understands and applies the code categories in the same way. For each discrepancy found between the original coder and the check-coder, a coding supervisor reviews the answer and chooses the most appropriate code to include in the final data set.
Provide feedback to interviewers
Where possible, it is best to begin coding early in the data collection process so that feedback can be provided to interviewers if they are not probing properly, or if there appears to be a misunderstanding about the meaning of a question. This allows for both individual and group re-training on specific questions as needed.
Provide feedback to the PI
At times the coding staff will identify a question that is not “working”– i.e., it seems to be misunderstood by a large number of respondents. The coding staff will contact the PI and Project Leader, and alert them to the problem, providing suggestions for changes to the instrument.
Report progress
The SSL data entry system generates a variety of reports to assist the coding supervisor and Project Leader in evaluating quality and efficiency of the coding staff. These include verification reports (showing the level of agreement between coders and data entry staff), and the average amount of time it takes to code a case.
11.5 Data Entry
High-speed data entry for self-administered questionnaires or other paper/pencil questionnaires is done by the Survey Services Lab staff. Activities for this task could include, but are not limited to:
Specify and program DE application
SRO data entry is done using Blaise software. The data entry application is specified by the Project Leader and programmed by the SSL staff.
Write instructions and train editing and data entry staff
The Project Leader and the SSL coding supervisor write instructions for the editing and data entry staff and provide training for the staff.
Edit questionnaires
Questionnaires are edited to prepare them for the data entry step. Missing data codes are assigned, marginal comments are assessed and codes adjusted as appropriate, and all steps on the editing checklist are completed. Where necessary, the Project Leader or other members of the research team are consulted for clarification or changes to the code frame.
Enter data
SSL staff members enter the numeric codes into the Blaise application. Cases generally are independently entered by two different staff members. SRO normally double-enters 100% of the cases, but a smaller percentage may be specified in the work scope if appropriate. Discrepancies are flagged and the coding supervisor works with the data entry team to resolve discrepancies and gives feedback to entry staff on their error rates.
Example Data Entry and Data Verification Progress Report
Example Data Entry and Data Verification Coder Report
11.6 Scanning
Scan forms are an alternative to data entry of paper questionnaires. In the scanning process, a scan form questionnaire or data collection instrument is scanned, using special software and equipment, by a third-party vendor.
Because much of the information needed for scanning is encoded into the scan form itself, many of the activities related to preparing for this post-production task are accomplished during questionnaire or form development. Activities include, but are not limited to:
Establish the scanning timeline
The Project Leader includes in the management plan a schedule for designing, printing, and shipping and receiving of scan forms.
Design the questionnaire or form
The Project Leader works with the PI to establish the questionnaire content and order, then works with the scanning company to format the instrument.
Specify the conventions for scanning or coding
The Project Leader specifies the variable names, values, value labels, codes for missing data and multiple marks, and which open-ended variables should be hand entered.
Specifying quantities for printing
The Project Leader specifies the amount of each form to print.
Coordinate shipping and receiving of forms for scanning
The Project Leader coordinates the shipping of forms to off-site locations, and receipt control of completed forms.
11.7 Weighting & Estimation
Appropriate estimation of population characteristics must take disproportionate representation into account. This is accomplished by assigning a weight to each respondent, where the weights account for the sample design and reflect the appropriate proportional representation of the various types of individuals in the population. The computation of sampling weights typically involves the following steps as appropriate based upon the sample design and population characteristics:
- Calculate the probability of selection weight
- Caluclate a non-response adjustment
- Calculate a poststratification adjustment
- Trim weights to help control variance
When specified in the work scope, SRO will provide sample weights for the survey data. If weights are not calculated by SRO, the final data set(s) will contain variables needed for the PI to calculate the weights.
Create complex sample design codes
SRO routinely creates complex sample design codes (stratum and cluster variables) to allow analysts to account for the complex sample design in their variance estimates.
Compute design variance estimates
The design effect is a measure of the loss and/or increase in precision incurred from clustered and stratified sampling designs, which are commonly used in SRO studies.
Complete quality control checks
Quality control checks include:
- Running descriptive statistics on all final weights and their components
- Ensuring that the sum of the probability of selection weights equals to number of elements on the sampling frame
- Ensuring that the sum of the final weights are close to the estimated number of elements in the population
- Having a second statistician review the primary statistician’s weighting calculations
11.8 Outcome Rate Calculations
Response rates are often used in the interpretation of survey quality Therefore, reporting response rates and other outcome rates based on an established survey research standard is an important part of dissemination and publication. Additionally, outcome rates often serve as indicators of a survey organization’s general performance. Activities for this task include, but are not limited to:
Calculate response rates
SRO calculates American Association for Public Opinion Research (AAPOR) response rates two and four, using SRO’s standard disposition codes, which follow AAPOR guidelines.
Calculate other outcome rates
SRO typically calculates others AAPOR outcome rates, such as contact, cooperation or refusal rates.
AAPOR Standard Definitions: Final Case Dispositions and Outcome Rates
Quality control
To check for potential errors, SRO typically has a second statistician review the primary statistician’s final outcome rate calculations.
11.9 Technical Report
At the conclusion of every data collection report, the Statistics and Methods Unit will provide a technical report to the Project Leader for inclusion in the final project report. This report includes descriptions of the sample design and sample selection procedure, (if necessary) the weighting procedure and how each weighting component was calculated, and final outcome rates and how they were calculated.
11.10 Confidentiality Assurance
SRO adheres to all University, ISR and SRC policies regarding respondent confidentiality, as well as any contractual requirements, such as approval of questionnaires and procedures by the Office of Management and Budget (OMB).
Requirements for this task could include, but are not limited to:
Pledges of confidentiality
All SRO employees sign a Pledge to Safeguard Respondent Privacy at the time of employment, and renew that pledge annually.
Policy on Safeguarding Respondent Privacy
Knowledge of policies
SRO employees should be aware of and adhere to all policies at the University, Institute, and Center levels as they relate to confidentiality and data security.
Confidentiality policies for University of Michigan PI’s
PIs with an appointment at the University of Michigan are required to sign a receipt for any files or hard copy materials with identifying information, and a confidentiality memorandum ensuring that the material will be stored in a secure location.
Confidentiality policies for external PIs
Providing PIs outside the University of Michigan with files or hard copy materials containing identifying information must be approved by the Center Senior Staff Advisory Committee (SSAC) and have explicit IRB approval. Evaluating whether or not to release identifying information must be handled on a case-by-case basis.
11.11 Dataset Documentation
SRO will prepare documentation for all datasets delivered to PIs. Documentation will vary by project. The amount and type of documentation will be specified in the contract or description of work. Basic documentation will usually be in the form of a data codebook or description of tables (e.g., for data from sample management systems). More extensive documentation may include materials in addition to the codebook, and will typically be bound and published as a final product.
Activities for this task could include, but are not limited to:
Create codebook
The Data Manager will create a codebook or data dictionary in the specified format for all variables in the final dataset, as specified in the contract or statement of work.
Prepare post-processing documentation
All SRO procedures and datasets developed for the computation of weights, variance estimation, and imputation will be documented and given to the PI as a study deliverable.
Create data frequencies
The Data Manager will run data frequencies containing all variables in the final dataset, as specified in the contract or statement of work, in the specified format.
Prepare data book documentation
A data book write-up (summary of procedures and data) may be required for some projects. A summary of this type typically includes: chapters that describe the overall project and purpose; the sample and results; methodology used; a description of the data and data handling, cleaning and processing; and appendices containing support documents.
11.12 Dataset Delivery
Data delivery procedures will be consistent with ISR’s Policy on Safeguarding Respondent Privacy. All datasets will be delivered in a standard format readable by third-party software. The basic format options available are ASCII (with data dictionary), SAS, or SPSS. The specific format for this deliverable will be specified in the contract or statement of work.
Raw data identifying respondents, including geographic information will not be part of SRO datasets unless explicitly noted in the contract or statement of work. Appropriate protection for Human Subjects must be approved by the University of Michigan before SRO can supply these data to non-ISR staff. (See Section 11.10 for details on confidentiality procedures and forms.)
Activities for this task could include, but are not limited to:
Deliver data
Data will be delivered to PIs as specified in the contract or statement of work.