Establishing an IDS
There are several components necessary to establish a successful IDS. While all states and counties have administrative data systems that manage programs and services in a siloed environment, integrating these systems can provide policy makers with a more comprehensive understanding of how policies and programs affect the individuals they are intended to serve. When establishing an integrated data system, the following components must be fully addressed in order to ensure the IDS is a useful and efficient tool.
The critical methodology for creating integrated data systems is the process of record linkage, which refers to the joining or merging of data on the basis of common data fields. The data fields are usually personal identifiers (e.g., name, birth date, social security number), or an encrypted version of those identifiers. Other examples of data fields within IDSs are system-generated client tracking numbers, encrypted “unique IDs,” or addresses.
Accurate data are essential to maintaining the integrity of research. In order for an integrated data system to produce the most dependable results, the governing board overseeing the IDS must establish appropriate methods for assessing the reliability and validity of data elements to maximize the utility of the information they contain. Below lists several established procedures researchers can use to evaluate the data reliability:
- Variable-level auditing – looks for out-of-range codes or codes that may have changed over time
- Reliability measures – variables that are scored with reliability measures such that external requestors are aware of the reliability of a given variable
- Establishing common audit routines – helps measure the completeness of a given variable (degree of missing data), the accuracy (the proportion of valid codes), and coverage (gaps in time periods reported, or providers reporting, etc.).
- Reliability and validity testing are important data-auditing tasks for evaluating the scientific capacity of data to be included in the IDS. It ensures that data collected on a variable actually represent the phenomenon in question. Because reliability and validity testing can be time-consuming, it is important for IDS leadership to partner with data-sharing agencies to periodically seek funding to accomplish these important audits. When two data sources are available for a given measure (e.g., diagnosis associated with a hospitalization), the redundant data sources can be compared to assess the degree of agreement between them. This can include comparing redundant measures of the same variable from different databases, and can also include sampling charts or other documentation to compare it to the electronic records. Although time consuming, charts can be reviewed for their comparability with administrative records on a periodic basis. Administrators may want to include data accuracy and completeness as a performance measure for service contracts with providers and other agencies as a way of improving data quality.
Every integrated data system requires a formal governance process. Governing boards are typically composed of representatives from source agencies of various databases. These agencies have legal responsibilities for use of their data, and, depending on the data source, often must approve individual requests for data use. The governing board may also include data quality and management experts or legal advisors. Separately, some jurisdictions may have a Research Advisory Board that reviews the scientific merit of proposed projects and gives general advice on IDS research operations. Lastly, a jurisdiction may choose to have an advisory board of community stakeholders, which may include foundations, The United Way, citizens groups, and service provider organizations, who can provide input as to policy and research priorities from the community perspective, and who can consider the translational issues associated with study results. Special projects may have their own ad-hoc advisory boards to commission research, review results and offer recommendations for policy and program change.
Data request proposals are typically reviewed by the governing board on their merit and feasibility, as well as fit with agencies’ priorities. A common rubric includes the following questions:
- Does the proposal look to answer a question that’s in-line with agency and/or local priorities?
- Are appropriate protections in place for data handling, sharing, and storing?
- Does the proposed team have the research expertise to execute the project?
- Does the integrated data system have the reliable data needed to execute the project?
- Does the integrated data system have the resources needed to facilitate the timely execution of the project, and if not, what would be the additional resources required?
Integrated data systems exist in order to because providers, researchers, and executive leaders come together with the goal of answering complex questions about how social services can more effectively and efficiently meet the needs of the public. Because of this, integrating data is a process of evaluation, discovery, and improvement by stakeholders with various perspectives. It’s important to note that the main focus of an integrated data system is NOT technology. Rather, the essence of an IDS lies in partnerships across different stakeholder groups that seek to set priorities, lead inquiry, and translate results into actionable intelligence that improves policy and practice. An integrated data SYSTEM is primarily composed of RELATIONSHIPS among a group of individuals committed to improving services for the public.
Integrated data systems require administrative processes to be successful when sharing data. Below are IDS research administration activities:
- Logging in research requests
- Tracking the progress of a research request through the approval process
- Documenting data use agreements (DUAs) and required data security certifications
- Training on the handling of sensitive data
- Reporting back results to data sharing agencies for their review and approving the release of results
- Tracking data delivery through FTP or other secure means
- Tracking data destruction or return of data files as stipulated in a given DUA
Integrated data systems require attention to scientific issues (such as data quality and data integration methods) in order to produce reliable results. Therefore, research questions must be able to be tested and answered using appropriate methods with integrated data. Scientific expertise frequently aids in the interpretation of results and plays a part in the translation of findings into policy and practice change. There are also established procedures for data integration and record linkage based on probabilities for records identified with incomplete or inaccurate information. Sites with integrated data systems should have robust scientific and analytic capabilities to support their work and produce actionable intelligence of high scientific rigor. An IDS site may want to have a scientific advisory board or research advisory board in place to provide counsel to the IDS’ governing body and operations staff on effective use of the data for research.