6. Systems Development
6.1 Introduction
This phase covers the tasks, activities, and deliverables necessary for setup and development of technical systems for an SRO project, in accordance with the Survey Research Operations Standards (2004).
Technical systems development activities result in the following mandatory and conditional deliverables:
Deliverables
- Specifications for technical systems;
- Specifications for production reports;
- Specifications for quality checks and monitoring reports;
- A testing plan;
- Technical systems setup and development quality management; and
- Principal Investigator (PI) approval of the data collection systems, data structures, and quality checks and reports.
SRC provides a variety of data collection tools. Generally, every data collection project needs to specify basic and project-specific sample management system requirements, and what sample management, data collection, reporting, and quality assurance tools are needed (e.g., SurveyTrak, WebTrak, paradata dashboards). In addition, a clear testing plan should cover testing of both new technical systems development and project-specific utilization of existing systems (e.g., Playback and ADTReport), as well as any computer assisted instruments (see Chapter 7, “Instrument Development”). Finally, PI (or designate) approval is required for all technical specifications, including data structures.
In addition to mandatory deliverables, each project will have a series of additional deliverables, such as the following:
Deliverables
- SurveyTrak specifications;
- CATI Sample Management System (SMS) specifications;
- WebTrak specifications;
- ADT (Blaise audit trail) reports specifications;
- Weblog reports specifications;
- Specifications for project-specific new data collection systems development;
- Testing scenarios;
- Training scenarios;
- Prototypes; and
- Usability test reports.
Note that these deliverables are neither mandatory nor optional. Rather, they are conditional, based on the study design. Generally a project will require either SurveyTrak (CAPI) or SMS (CATI) specifications, as well as WebTrak specifications for reporting purposes. If a project has receipt control or tracking requirements, it may need WebLog specifications as well. Interview quality assurance may require interview verification (done through DRI/CARI/OLIVE) and/or review of Blaise interviews while listening to (portions of) recorded interviews.
The development process is an iterative series of steps, and specifications will evolve over the development lifecycle. Technical specifications should always reflect the project scope, and should be checked against scope throughout the development lifecycle. Changes in scope require a scope change memo and approval of the SRO Administrative Group (see Chapter 3, “Project Management”).
Technical systems development is coordinated by the Technical Lead, who ensures that all systems are fully specified and tested. Figure 6.1 provides a flow diagram of project management processes. SRO best practices for technical systems development are described in sections 6.2 through 6.7, and cover the following developmental tasks:
6.2 Specify requirements;
6.3 Set up systems;
6.4 Design components;
6.5 Program components;
6.6 Test components and integrated systems; and
6.7 Approve systems and data structures.
Figure 6.1. Technical Systems Development Processes
The project Technical Lead is responsible for the overall design of technical systems required to meet the project needs, as well as an overview of integrated systems and components and how they relate to one another. This responsibility should be made explicit in the project plan.
6.2 Specify Requirements
It is necessary to specify completely project sample management, reporting, and quality assurance requirements before development begins. Time to schedule writing such requirements must be built into the overall project management plan. With current systems, specifications will focus on either CAPI or CATI sample management, and SurveyTrak or SMS, respectively. Requirements for any Web components are specified separately, but linkages to current systems and information to be transferred to and from those systems are specified with those system requirements.
Sample management specifications (SurveyTrak or SMS) identify instruments that will collect survey and contact observation data, instrument preload, windows and tabs the interviewer will use, wizard driven call record actions, respondent incentives and letters, tracking requirements and respondent profile for panel surveys, requirements for generating (or spawning) new sample lines, and so on.
For both CATI and CAPI projects, reporting specifications include WebTrak and reporting requirements, such as levels of management, group and sample types, searching needs, main pages (e.g., sample, addresses, and verification), detail pages (e.g., profiles and case notes), quality control (e.g., verification and evaluation), result codes, report columns and rows, and key project statistics to store in SurveyTrak.
Any receipt control (WebLog) and quality assurance requirements are also specified, as well as any new or auxiliary system requirements.
Finally, specifications may also include training scripts for new components not covered in prior SRO trainings, and testing scenarios (see Section 7.5). SRO provides two tools for developing training scripts and testing scenarios, ScriptWriter and CTT (CAI Testing Tools; see also “SRO Instrument Development”).
6.3 Set Up Systems
Once initial requirements specifications are complete, Programmers and/or Data Managers set up the sample management systems appropriate for the project (generally SurveyTrak or CATI SMS). While generic or core system setup can start without specifications (e.g., Project Name, Path for Data Access, number of projects), further setup should not proceed without clear project specifications.
6.4 Design Components
Design is closely linked with requirements specifications, and begins when programmers have enough information about requirements to begin designing components in consultation with the Technical Lead and the project team. This stage may require storyboarding and paper prototyping of new windows, tabs, and system components. As the design becomes more formal, the specifications and related elaborated design documents become the project’s systems documentation. The final project specifications should reflect the production systems as implemented, and should be archived with other formal project documentation (see Chapter 12, “Archiving and Documentation”). This includes documentation of data structures, such as the SurveyTrak datamodel.
6.5 Program Components
Programming can begin once specifications and design are complete for the system or one of its components. This stage may involve development of prototypes of components or screen designs.
Programming involves conducting basic testing of systems against specifications before releasing them for further testing.
6.6 Test Components & Integrated Systems
Testing is the process of verifying that the systems work according to specifications. The objectives of testing are to ensure that systems meet stakeholder requirements, and to minimize the risk of product failure during production. Thus, all components and systems must be tested thoroughly, whether newly written or modified from an existing application.
Generally, all members of the project team are involved in some way in testing, from programmers to production staff, and their efforts are coordinated. Testing coordination is the responsibility of a designated member of the project team (study director, project manager, project lead, or technical lead, as appropriate). This responsibility should be made explicit in the project plan.
Testing is not viewed only as the final step in development. Project teams should test components and systems iteratively throughout the development lifecycle. As a result, testing can lead to re-specification, re-design, and/or re-programming, and thus additional testing. Every new version, module addition, or enhancement needs to undergo comprehensive testing, which the management and testing plans must accommodate.
Note that usability evaluation is form of testing, and may be included in a testing plan, particularly for prototypes and development of new systems. Usability tests can be formal (videotaping, event coding, and formal report) or informal (observations and summary of key findings as input to further programming). They are most useful at early stages of development.
In the absence of usability testing, expert review is an option for evaluation of new components or systems (the experts may be programming, management, or project staff, and/or survey methodologists).
All testing plans should include both unit (component) and integrated system testing. Integrated system testing should precede trainings prior to pretest(s) and production. In addition, testing metric data and reports should be retained and archived (see Chapter 12, “Archiving and Documentation”).
Evaluating the effectiveness of programming and testing should be part of the quality management plan which should include testing tools (e.g., CAI Testing Tool, Bug Log, SurveyTrak) for tracking and test groups (e.g., Interviewer Help Desk for systems integration testing).
6.7 Approve Systems & Data Structures
Before the start of pretest or production interviewing, the technical systems and data structures must be formally approved by the PI or designated member of the project staff. This includes review and approval of a test dataset. This signoff step (including freeze date) should appear explicitly in the project management plan and project schedule.