Framework for Evaluating Public Health Surveillance Systems
for Early Detection of Outbreaks
Recommendations from the CDC Working Group
Prepared by
James W. Buehler, M.D.,1
Richard S. Hopkins, M.D.,2
J. Marc Overhage, M.D.,3
Daniel M. Sosin, M.D.,2
Van Tong, M.P.H.2
1 Department of Epidemiology, Rollins School of Public Health, Emory University
2 Division of Public Health Surveillance and Informatics, Epidemiology Program Office, CDC
3 Indiana University School of Medicine
The material in this report originated in the Epidemiology Program Office, Stephen B. Thacker, M.D., Director, and the Division of Public Health Surveillance and Informatics, Daniel M. Sosin, M.D., Director.
Summary
The threat of terrorism and high-profile disease outbreaks has drawn attention to public health surveillance systems for
early detection of outbreaks. State and local health departments are enhancing existing surveillance systems and developing
new systems to better detect outbreaks through public health surveillance. However, information is limited about the usefulness
of surveillance systems for outbreak detection or the best ways to support this function. This report supplements previous guidelines for evaluating public health surveillance systems. Use of this framework is intended to improve decision-making regarding the implementation of surveillance for outbreak detection. Use of a standardized evaluation methodology, including description
of system design and operation, also will enhance the exchange of information regarding methods to improve early detection
of outbreaks. The framework directs particular attention to the measurement of timeliness and validity for outbreak detection. The evaluation framework is designed to support assessment and description of all surveillance approaches to early
detection, whether through traditional disease reporting, specialized analytic routines for aberration detection, or surveillance using early indicators of disease outbreaks, such as syndromic surveillance.
Introduction
Public health surveillance is the ongoing, systematic collection, analysis, interpretation, and dissemination of data about
a health-related event for use in public health action to reduce morbidity and mortality and to improve health
(1). Surveillance serves at least eight public health functions. These
include supporting case detection and public health
interventions, estimating the impact of a disease or injury, portraying the natural history of a health condition, determining the distribution and spread of illness, generating hypotheses and stimulating research, evaluating prevention and control
measures, and facilitating planning (2). Another important public health function of surveillance is outbreak detection (i.e., identifying an increase in frequency of disease above the background occurrence of the disease).
Outbreaks typically have been recognized either based on accumulated case reports of reportable diseases or by clinicians
and laboratorians who alert public health officials about clusters of diseases. Because of the threat of terrorism and the increasing availability of electronic health data, enhancements are being made to existing surveillance systems, and new
surveillance systems have been developed and implemented in public health jurisdictions with the goal of early and complete detection of
outbreaks (3). The usefulness of surveillance systems for early detection and response to outbreaks has not been
established, and substantial costs can be incurred in developing or enhancing and managing these surveillance systems and investigating false alarms (4). The measurement of the performance of public health surveillance systems for
outbreak detection is needed to establish the relative value of different approaches and to provide information needed to improve
their efficacy for detection of outbreaks at the earliest
stages.
This report supplements existing CDC guidelines for evaluating public health surveillance systems
(1). Specifically, the report provides a framework to evaluate timeliness for outbreak detection and the balance among sensitivity, predictive value positive (PVP), and predictive value negative (PVN) for detecting outbreaks. This framework also encourages
detailed description of system design and operations and of their
experience with outbreak detection.
The framework is best applied to systems that have data to demonstrate the attributes of the system under
consideration. Nonetheless, this framework also can be
applied to systems that are in early stages of development or in the planning phase
by using citations from the published literature to support conclusions. Ideally, the evaluation should compare the
performance of the surveillance system under scrutiny to alternative surveillance systems and produce an assessment of the relative usefulness for early detection of outbreaks.
Background
Early detection of outbreaks can be achieved in three ways: 1) by timely and complete receipt, review, and investigation of disease case reports, including the prompt recognition and reporting to or consultation with health departments by
physicians, health-care facilities, and laboratories consistent with disease reporting laws or regulations; 2) by improving the ability to recognize patterns indicative of a possible outbreak early in its course, such as through analytic tools that improve
the predictive value of data at an early stage of an outbreak or by lowering the threshold for investigating possible outbreaks;
and 3) through receipt of new types of data that can signify an outbreak earlier in its course. These new types of data might include health-care product purchases, absences from work or school, presenting symptoms to a health-care provider, or laboratory test orders (5).
Disease Case Reports
The foundation of communicable disease surveillance in the United States is the state and local application of
the reportable disease surveillance system known as the National Notifiable Disease Surveillance System (NNDSS),
which includes the listing of diseases and laboratory findings of public health interest, the publication of case definitions for
their surveillance, and a system for passing case reports from local to state to CDC. This process
occurs best where two-way communication occurs between public health agencies and the clinical community: clinicians and laboratories report cases and clusters of reportable and unusual diseases, and health departments consult on case diagnosis and management,
alerts, surveillance summaries, and clinical and public health recommendations and policies. Faster, more specific and affordable diagnostic methods and decision-support tools for diseases with substantial outbreak potential could improve the timely recognition of reportable diseases. On-going health-care provider and laboratory outreach, education, and 24-hour access to public health professionals are needed to enhance
reporting of urgent health threats. Electronic laboratory reporting (i.e.,
the automated transfer of designated data from a laboratory database to a public health data repository using a
defined message structure) also will improve the timeliness and completeness of reporting notifiable conditions
(6--8) and can serve as a model for electronic reporting of a wider range of clinical information. A comprehensive surveillance effort supports timely investigation (i.e., tracking of cases once an outbreak has been recognized) and data needs for managing the public
health response to an outbreak or terrorist event.
Pattern Recognition
Statistical tools for pattern recognition and aberration
detection can be applied to screen data for patterns warranting
further public health investigation and to enhance recognition of subtle or obscure outbreak patterns
(9). Automated analysis and visualization tools can lessen the need for
frequent and intensive manual analysis of surveillance data.
New Data Types
Many new surveillance systems, loosely termed syndromic surveillance systems, use data that are not diagnostic of a
disease but that might indicate the early stages of an outbreak. The scope of this framework is broader than these novel systems, yet the wide-ranging definitions and expectations of syndromic surveillance require clarification. Syndromic surveillance for early outbreak detection is an investigational approach
where health department staff, assisted by automated data acquisition
and generation of statistical signals, monitor disease indicators continually (real-time) or at least daily (near real-time) to detect outbreaks of diseases earlier and more completely than might otherwise be possible with traditional public health
methods (e.g., by reportable disease surveillance and telephone consultation). The distinguishing characteristic of
syndromic surveillance is the use of indicator data types. For example, a laboratory is a data source that can support traditional
disease case reporting by submitting reports of confirmatory laboratory results for notifiable conditions; however, test requests are a type of laboratory data that might be used as an outbreak indicator by tracking excess volume of test
requests for diseases that typically cause outbreaks. New data types have been used by public health to enhance
surveillance, reflecting events that might precede a clinical diagnosis (e.g., patient's chief complaints in emergency departments, clinical impressions on ambulance log sheets, prescriptions filled, retail drug and product purchases, school or work
absenteeism, and constellations of medical signs and symptoms in persons seen in various clinical settings).
Outbreak detection is the overriding purpose of syndromic surveillance for terrorism preparedness. Enhanced case-finding and monitoring the course and population characteristics of a recognized outbreak also are potential benefits
of syndromic surveillance (4). A manual syndromic surveillance system was used to detect additional anthrax cases in the fall
of 2001 when the outbreak was recognized
(10). Complicating the understanding of syndromic surveillance is that
syndromes have been used for case detection and management of diseases when the condition is infrequent and the syndrome is
relatively specific for the condition of interest. Acute flaccid paralysis is a syndromic marker for poliomyelitis and is used to detect single cases of suspected polio in a timely way to initiate investigation and control measures. In this case, the syndrome is
relatively uncommon and serious and serves as a proxy for polio
(11). Syndromes also have been used effectively for surveillance
in resource-poor settings for sexually transmitted disease detection and control where laboratory confirmation is not possible or practical (12). However, syndromic surveillance for terrorism is not intended for early detection of single cases or limited outbreaks because the early clinical manifestations of diseases that might be caused by
terrorism are common and nonspecific (13).
Framework
This framework is intended to support the evaluation of all public health surveillance systems for the timely detection
of outbreaks. The framework is organized into four categories: system description, outbreak detection, experience,
and conclusions and recommendations. A comprehensive evaluation will address all four categories.
A. System Description
1. Purpose. The purpose(s) of the system should be explicitly and clearly described and should include the intended uses
of the system. The evaluation methods might be prioritized differently for different purposes. For example, if terrorism
is expected to be rare, reassurance might be the primary purpose of the terrorism surveillance system. However, for
reassurance to be credible, negative results must be accurate and the system should have a demonstrated ability to detect outbreaks of the kind and size being dismissed.
The description of purpose should include the indications for implementing the system; whether the system is designed
for short-term, high-risk situations or long-term, continuous use; the context in which the system operates (whether it stands alone or augments data from other surveillance systems); what type of outbreaks the system is intended to detect; and what secondary functional value is desired. Designers of the system should specify the desired sensitivity and specificity of the system and whether it is intended to capture small or large events.
2. Stakeholders. The stakeholders of the system should be listed. Stakeholders include those who provide data for
the system and those who use the information generated by the system (e.g., public health practitioners; health-care
providers; other health-related data providers; public safety officials; government officials at local, state, and federal levels; community
residents; nongovernmental organizations; and commercial systems developers). The stakeholders might vary among
different systems and might change as conditions change. Listing stakeholders helps define who the system is intended to serve
and provides context for the evaluation results.
3. Operation. All aspects of the operation of the syndromic surveillance system should be described in detail to
allow stakeholders to validate the description of the system and for other interested parties to understand the complexity
and resources needed to operate such a system. Detailed system description also will facilitate evaluation by highlighting
variations in system operation that are relevant to variations in system performance (Figure 1). Such a conceptual model can facilitate the description of the system. The description of the surveillance process should address
1)systemwide characteristics (data flow [Figure 2]), including data and transmission standards to facilitate
interoperability and data sharing between
information systems, security, privacy, and confidentiality; 2) data sources (used broadly in this framework to include the data-producing facility [i.e., the entity sharing data with the public health surveillance system], the data type [e.g., chief complaint, discharge diagnosis, laboratory test order], and the data format [e.g., electronic or paper, text descriptions of events or illnesses, or structured data reworded or stored in standardized format]); 3) data processing before analysis (the data collation, filtering, transformation, and routing functions required for public health to use the data, including the classification and assigning of syndromes); 4) statistical analysis (tools for automated screening of data for potential outbreaks); and
5) epidemiologic analysis, interpretation, and investigation (the rules, procedures, and tools that support decision-making in response to a system signal, including adequate staffing with trained epidemiologists who can review, explore, and interpret the data in
a timely manner).
B. Outbreak Detection
The ability of a system to reliably detect an outbreak at the earliest possible stage depends on the timely capture and processing of the data produced by transactions of health behaviors (e.g., over-the-counter pharmaceutical sales,
emergency department visits, and nurse call-line volume) or health-care activities (e.g., laboratory test volume and triage categorization of chief complaint) that might indicate an outbreak; the validity of the data for measuring the conditions of interest at
the earliest stage of illness and the quality of those data; and the detection methods applied to these processed surveillance data
to distinguish expected events from those indicative of an outbreak.
1. Timeliness. The timeliness of surveillance approaches for outbreak
detection is measured by the lapse of time from exposure to the disease agent to the initiation of a public health intervention. A timeline with interim milestones is proposed to improve the specificity of timeliness measures (Figure 3).
Although measuring all of the time points that define
the intervals might be impractical or inexact in an applied outbreak setting, measuring intervals in a consistent way can be used
to compare alternative outbreak-detection approaches and specific surveillance systems.
Onset of exposure:
By anchoring the timeline on
exposure, the timeliness advantage of different data sources can
be assessed and compared. Exposure can most easily be estimated in a point-source outbreak. Time of exposure is
often inferred from knowledge of the agent (e.g., incubation
period) and the epidemiology of the outbreak.
Onset of symptoms: The
interval to symptom onset in each case is defined by the incubation period for
the agent. Time of symptom onset might be estimated using case interviews or
existing knowledge of the agent and the time of exposure. The incubation period
might vary according to host factors and the route and dose of the exposure.
Onset of behavior: Following
symptom onset, several health behaviors can occur (e.g., purchasing
over-the-counter medication from a store, calling in sick to work, or visiting
an urgent-care center). When an affected person interacts with the health-care
system, a variety of health-care provider behaviors might be performed (e.g.,
order of a laboratory test and admission to a hospital). The selection of data
sources for a system has a strong influence on timeliness. Some of those
experiencing symptoms will initiate a health behavior or stimulate a health-care
provider behavior that is a necessary step to being captured in the surveillance
system.
Capture of data: The timing of
the capture of a behavior by the data-providing facility varies by data type and
can be influenced by system design. A retail purchase might be entered in an
electronic database at the moment the transaction is completed, or a record
might not be generated in a clinical setting until hours after health care was
sought.
Completion of data processing:
Time is required for the facility providing the data to process the data and
produce the files needed for public health. Records might be transmitted to a
central repository only periodically (e.g., weekly). Data form can influence
processing time (e.g., transcription from paper to electronic form and coding
text-based data), and data manipulations needed to de-identify data and prepare
necessary files can affect processing time.
Capture of data in public health surveillance system:
The time required to transfer data from the data
providing facility to the public health entity varies according to the frequency
established for routine data transmission and by the data transmission method
(e.g., Internet, mail, or courier).
Application of pattern recognition tools/algorithms:
Before analytic tools can be applied to the data in the surveillance system,
certain processing steps are necessary (e.g., categorization into syndrome
categories, application of case definition, and data transformations).
Generation of automated alert:
The detection algorithm's alerting interval is a product of how often the
algorithm is run and a report generated and the capacity of the algorithm to
filter noise and detect an aberration as early as possible in the course of the
outbreak.
Initiation of public health investigation:
The initiation of a public health investigation
occurs when a decision is made to acquire additional data. Analysis and judgment
are applied by public health-care providers to the processed surveillance data
and other available information to decide whether new data collection is
warranted to confirm the existence of an outbreak. The challenge of interpreting
data from multiple surveillance systems could diminish potential advantages in
timeliness. The focus on outbreak detection allows for investigations of
potential outbreaks to proceed before a specific clinical diagnosis is obtained.
Initiation of public health intervention:
When an outbreak of public health significance is
confirmed, interventions can be implemented to control the severity of disease
and prevent further spread. Interventions might be of a general nature directed
to the recognition of an outbreak (e.g., apply respiratory infection precautions
and obtain clinical specimens for diagnosis) or can be specific to the diagnosis
(e.g., antibiotic prophylaxis or vaccination).
2.Validity. Measuring the validity of a system for outbreak detection requires an operational definition of an
outbreak. Although a statistical deviation from a baseline rate can be useful for triggering further investigation, it is not sufficient
for defining an outbreak. In practice, the confirmation of an outbreak is a judgment that
depends on past experience with the condition, the severity of the condition, the communicability of the condition, confidence in the diagnosis of the condition, public health concern about outbreaks at the time, having options for effective prevention or control, and the
resources required and available to respond. Operationally, an outbreak is defined by the affected public health jurisdiction when
the occurrence of a condition has changed sufficiently to warrant public health attention.
The validity of a surveillance system for outbreak detection varies according to the outbreak
scenario and surveillance system factors. These factors can confound the comparison of systems and must be carefully described in the evaluation. For example, the minimum size of an outbreak that can be
detected by a system cannot be objectively compared among systems
unless they are identical or differences are accounted
for in several ways.
Case definitions: Establish
the specificity and sensitivity for the condition of interest on the basis of
the data source, data type, and response criteria.
Baseline estimation: Determine
the stability of the background occurrence of cases. Estimations are affected by
factors such as population size and geographic distribution. The performance of
detection algorithms will vary by the quality and duration and inherent
variability of baseline data.
Reporting delays: Result in
incomplete data, introducing bias that will diminish the performance of
detection algorithms.
Data characteristics: Includes
underlying patterns in the data (e.g., seasonal variation) and systematic errors
inherent in the data (e.g., product sales that influence purchasing behaviors
unrelated to illness).
Outbreak characteristics:
Results from agent, host, and environmental factors that affect the epidemiology
of the outbreak. For example, a large aerosol exposure with an agent causing
serious disease in a highly susceptible population will have different detection
potential than an outbreak of similar size spread person-to-person over a longer
time and dispersed distribution.
Statistical analysis: Defines
how data are screened for outbreak detection. Detection algorithms have
different performance characteristics under different outbreak conditions.
Epidemiologic analysis, interpretation, and investigation:
The procedures, resources, and tools for
analysis, interpretation, and response that can substantially affect the ability
to detect and respond to outbreaks.
Validation Approaches
Different approaches to outbreak detection need to be evaluated under the same conditions to isolate the unique features of the system (e.g., data type) from the outbreak characteristics and the health department capacity. The data needed to evaluate and compare the performance of surveillance systems for early outbreak detection can be obtained from naturally occurring outbreaks or through simulation.
Controlled comparisons of surveillance systems for detection of deliberately induced outbreaks will be difficult because of the infrequency of such outbreaks and the diversity of systems and outbreak settings. However,
understanding the value of different surveillance approaches to early detection will
increase as descriptions of their experience with detecting and
missing naturally occurring outbreaks accumulate. Accumulation of experience descriptions is made more difficult by not having standard methods for measuring outbreak detection successes and failures across systems and by the diversity of
surveillance system and outbreak factors that influence performance. Standardized classification of system and outbreak factors will enable comparison of experiences across systems. Pending the development of classification standards, descriptive evaluation
should include as much detail as possible. Proxy outbreak scenarios reflect the types of naturally occurring outbreaks that should
not be missed to instill confidence in the ability of these systems to detect outbreaks caused by terrorism.
Examples of proxy events or outbreaks include seasonal events (e.g., increases in influenza, norovirus gastroenteritis, and other infectious respiratory agents) and community outbreaks (e.g., foodborne, waterborne, hepatitis A, child-care--associated shigellosis,
legionellosis, and coccidioidomycosis and histoplasmosis in areas where the
diseases are endemic).
The measurement of outbreaks detected, false alarms, and outbreaks missed or detected late should be designed as a
routine part any system workflow and conducted with minimal effort or complexity. Routine reporting should be automated where possible. Relevant information needs include: the number of statistical aberrations detected at a set threshold in a defined period of time (e.g., frequency per month at a given p-value); the action taken as a result of the signals (e.g.,
review for data errors, in-depth follow-up analysis of the specific conditions within the syndrome category, manual epidemiologic analysis to characterize a signal, examining data from other systems, and increasing the frequency of
reporting from affected sites); resources directed to the follow-up of the alert; public health response that resulted (e.g., an alert to clinicians, timely dissemination of information to other health entities, a vaccination campaign, or no further response);
documentation of how every recognized outbreak in the
jurisdiction was detected; an assessment of the value of the follow-up effort (e.g., the
effort was an appropriate application of public health resources); a detailed description of the agent, host, and
environmental conditions of the outbreak; and the number of outbreaks detected only late in their course or in retrospect.
To evaluate the relative value of different methods for outbreak detection, a direct comparison approach is needed.
For example, if a health department detects a substantial number of its outbreaks through telephone consultations, then a phone call tracking system might produce the data needed to compare telephone consults with other approaches for
early detection of outbreaks.
As an alternative to naturally occurring outbreaks, simulations can allow for the control and modification of agent, host, and environmental factors to study system performance across a range of common scenarios. However, simulations are limited
in their ability to mimic the diversity and unpredictability of real-life events. Whenever possible, simulated outbreaks should
be superimposed on historical trend data. To evaluate detection algorithms comparatively, a shared challenge problem and
data set would be helpful. Simulation is limited by the availability of well-documented outbreak scenarios (e.g., organism or agent characteristics, transmission characteristics, and population characteristics). Simulations should incorporate data for each of the factors described previously. Multiple simulation runs should be used to test algorithm performance in different
outbreak scenarios, allowing for generation of operating characteristic curves that reflect
performance in a range of conditions.
Focused studies to validate the performance of limited
aspects of systems (e.g., data sources, case definitions,
statistical methods, and timeliness of reporting) can provide indirect evidence of system performance. Component studies also can test assumptions about outbreak scenarios and support better data simulation. Syndrome case definitions for certain specific
data sources need to be validated. Component validation studies should emphasize outbreak detection over case detection. These studies contain explicit hypotheses and research questions and should be shared in a manner to advance the development of outbreak detection systems without unnecessary
duplication.
Statistical Assessment of Validity
Surveillance systems must balance the risk for an outbreak, the value of early intervention, and the finite resources
for investigation. Perceived high risk and high value of timely detection support high sensitivity and low thresholds
for investigation. A low threshold can prompt resource-intensive investigations and occupy vital staff, and a high threshold might delay detection and intervention. The perceived threat of an outbreak, the community value attached to early detection, and the investigation resources available might vary over time. As a result, specifying a fixed relation between optimal sensitivity and predictive value for purposes of evaluation might be difficult.
The sensitivity and PVP and PVN are closely linked and considered together in this framework. Sensitivity is the percentage of outbreaks occurring in the jurisdiction detected by the system. PVP reflects the probability of a system signal being an outbreak. PVN reflects the probability that no outbreak is occurring when the system does not yield a signal. The calculation of sensitivity and predictive value is described in detail in the updated guidelines for evaluating public health
surveillance systems (1). Measurement of sensitivity requires an alternative data source of high quality (e.g., "gold" standard) to confirm outbreaks in the population that were missed by the surveillance system. Sensitivity for outbreak detection could be
assessed through capture-recapture techniques with two independent data sources
(14). The high costs associated with responding
to false alarms and with delayed response to outbreaks demand efforts to quantify and limit the impact of both. As long as
the likelihood of terrorism is extremely low, PVP will remain near zero and a certain level of nonterrorism signals will be
a necessary part of conducting surveillance for the detection of terrorism. Better performance can be achieved in one
attribute (e.g., sensitivity) without a performance decrement in another (e.g., PVP) by changing the system (e.g., adding a data type or applying a better detection algorithm). Improving sensitivity by lowering the cut-off for signaling an outbreak will reduce
PVP. Sensitivity and PVP for these surveillance systems will ultimately be calibrated in each system to balance the secondary benefits (e.g., detection of naturally occurring outbreaks, disease case finding and management, reassurance of no outbreak during periods of heightened risk, and a stronger reporting and consultation relation between public health and clinical medicine) with the locally acceptable level of false alarms.
Data Quality
The validity of syndromic surveillance system data is
dependent on data quality. Error-prone systems and data prone
to inaccurate measurement can negatively affect detection of unusual trends. Although data quality might be a less
critical problem for screening common, nonspecific indicators for statistical aberrations, quality should be evaluated and improved to the extent possible. Measuring data quality is dependent on a standard (e.g., medical record review or fabricated test data
with values known to the evaluator). The updated guidelines for evaluating public health surveillance systems (1) describe data quality in additional detail.
Representativeness: When case ascertainment within a population is incomplete (e.g., in a sentinel system or a statistically based sample), representativeness reflects whether a system accurately describes the distribution of cases by time, place, and person. Geographic representativeness is particularly important for detecting outbreaks of
infectious diseases.
Completeness of data: The
frequency of unknown or blank responses to data items in the system can be used
to measure the level of completeness. For systems that update data from previous
transmissions, time should be factored into measurement by indicating the
percentage of records that are complete (i.e., all variables are captured for a
record) on initial report and within an appropriate interval (e.g., 48 hours) of
submission. Sites with substantial reporting delays can be flagged for
reliability concerns and targeted for improvement. Incomplete data can require
follow-up before analysis, with associated decreases in timeliness and increase
in cost. When multiple data providers contribute to a common data store for
statistical analysis, the percentage of reporting sources that submit their data
on a routine interval (e.g., every 24 hours) conveys the completeness of the
aggregate database for routine analysis. Evaluation of completeness should
include a description of the problems experienced with manual data management
(e.g., coding errors or loss of data) and the problems with automated data
management (e.g., programming errors or inappropriate filtering of data).
C. System Experience
The performance attributes described in this section
convey the experience that has accrued in using the system.
1. System usefulness. A surveillance system is useful for outbreak detection
depending on its contribution to the early detection of outbreaks of public health significance that leads to an effective intervention. An assessment of usefulness goes beyond detection to address the impact or value added by its application. Measurement of usefulness is inexact. As
with validity, measurement will benefit from common terminology and standard data elements. In the interim, detailed efforts to describe and illustrate the consequences of early
detection efforts will improve understanding of their
usefulness.
Evaluation should begin with a review of the objectives of the system and should consider the priorities. To the extent possible, usefulness should be described by the disease
prevention and control actions taken as a result of the analysis
and interpretation of the data from the system.
The impact of the surveillance system should be contrasted with other mechanisms available for outbreak detection.
An assessment of usefulness should list the outbreaks detected and the role that different methods played in the identification of each one. Examples of how the system has been used to detect or track health problems other than outbreaks in
the community should be included. The public health response to the outbreaks and health problems detected should
be described as well as how data from new or modified surveillance systems support inferences about disease patterns that would not be possible without them.
Surveillance systems for early outbreak detection are sometimes justified for the reassurance they provide when
aberrant patterns are not apparent during a heightened risk period or when the incidence of cases declines during an outbreak. When community reassurance is claimed as a benefit of the surveillance system, reassurance should be defined and the measurement quantified (e.g., number of phone calls from the public on a health department hotline, successful press conferences, satisfaction of public health decision-makers, or resources to institutionalize the new surveillance system). A description should include who is reassured and of what they are reassured, and reassurance should be evaluated for validity by estimating the PVN.
2. Flexibility. The flexibility of a surveillance system refers to the system's ability to change as needs change. The adaptation to changing detection needs or operating conditions should occur with minimal additional time, personnel, or other resources. Flexibility generally improves the more data processing is handled centrally rather than distributed to individual data-providing facilities because fewer system and operator behavior changes are needed. Flexibility should
address the ability of the system to apply evolving data standards and code sets as reflected in Public Health Information Network (PHIN) standards (http://www.cdc.gov/phin). Flexibility includes the adaptability of the system to shift from outbreak detection to
outbreak management. The flexibility of the system to meet changing detection needs can include the ability to add unique data to refine signal detection, to capture exposure and other data relevant to managing an outbreak, to add data providers to increase population coverage and detect or track low frequency events, to modify case definitions (the aggregation of codes into syndrome groupings), to improve the detection algorithm to filter random variations in trends more efficiently, and to adjust the detection threshold. Flexibility also can be reflected by the ability of the system to detect and monitor naturally occurring outbreaks in the absence of terrorism. System flexibility is needed to balance the risk for an outbreak, the value of early intervention, and the resources for investigation as understanding of these factors changes.
3. System acceptability. As with the routine evaluation of public health surveillance systems
(1), the acceptability of a surveillance system for early outbreak detection is reflected by the willingness of participants and stakeholders to contribute to the data collection and analysis. This concept includes the authority and willingness to share electronic health data and
should include an assessment of the legal basis for the collection of prediagnosis data and the implications of privacy laws (e.g.,
Health Insurance Portability and Accountability Act Privacy Rule) (15). All states have broad disease-reporting laws that
require reporting of diseases of public health importance, and many of these laws appear compatible with the authority to
receive syndromic surveillance data (16). The authority to require reporting of indicator data for persons who lack evidence of a reportable condition and in the absence of an emergency is less clear and needs to be verified by jurisdictions.
Acceptability can vary over time as the threat level, perceived value of early detection, support for the methods of surveillance, and resources fluctuate.
Acceptability of a system can be inferred from the extent of its adoption. Acceptability is reflected by the participation
rate of potential reporting sources, by the completeness of data reporting, and by the timeliness of person-dependent steps in the system (e.g., manual data entry from emergency department logs as distinguished from electronic data from
the normal clinical workflow).
4. Portability. The portability of a surveillance system
addresses how well the system could be duplicated in another
setting. Adherence to the PHIN standards can enhance portability by reducing variability in the application of information technology between sites. Reliance on person-dependent steps, including judgment and action criteria (e.g., for analysis
and interpretation) should be fully documented to improve system portability. Portability also is influenced by the simplicity of the system. Examples should be provided of the deployment
of similar systems in other settings, and the experience of
those efforts should be described. In the absence of examples, features of the system that might support or detract from portability should be described.
5. System stability. The stability of a surveillance system refers to its resilience to system changes (e.g., change in coding from International Classifications of Disease, Ninth
Revision [ICD-9] to ICD-10). Stability can be demonstrated by
the duration and consistent operation of the system. System stability is distinguished from the reliability of data elements within the system. The consistent representation of the condition under surveillance (reliability) is an aspect of data quality. Stability can be measured by the frequency of system outages or downtime for servicing during periods of need, including downtime of data providers, the frequency of personnel deficiencies from staff turnover, and budget constraints. Ongoing support
by system designers and evolving software updates might improve system stability. Stability also can be reflected in the extent of control over costs and system changes that the sponsoring agency maintains.
6. System costs. Cost is a vital factor in assessing the relative value of surveillance for terrorism preparedness. Cost-effectiveness analyses and data modeling are needed under a range of scenarios to estimate the value of innovations
in surveillance for outbreak detection and terrorism preparedness
(17). Improved methods of measuring cost and impact
are needed. Costs borne by data providers should be noted; however, the cost perspective should be that of the community (societal perspective) to account for costs of prevention and treatment born by the community.
Direct costs include the fees paid for software and data, the personnel salary and support expenses (e.g., training, equipment support, and travel), and other resources needed to
operate the system and produce information for public health
decisions (e.g. office supplies, Internet and telephone lines, and other communication equipment). Fixed costs for running the system should be differentiated from the variable costs of responding to system alarms. Variable costs include the cost of follow-up activities (e.g., for diagnosis, case-management, or community interventions). The cost of responding to false alarms represents a variable but inherent inefficiency of an early detection system that should be accounted for in the evaluation.
Similarly, variable costs include the financial and public health costs of missing outbreaks entirely or recognizing them late. Costs vary because the sensitivity and timeliness of the detection methods can be modified according to changes in tolerance for
missing outbreaks and for responding to false alarms. Similarly, the threshold and methods for investigating system alarms can vary with the perceived risk and need to respond. Costs from public health response to false alarms with traditional surveillance systems need to be measured in a comparable way when assessing the relative value of new surveillance methods. Cost savings should be estimated by assessing the impact of prevention and control efforts (e.g., health-care costs and productivity losses averted) Questions to answer include the following:
How many investigations were initiated as a result of these data?
What response was made and
what cost was incurred through follow-up of flagged events?
What were the indications for
responding?
How much staff time was
required for follow-up?
Was anxiety raised
unnecessarily by false alarms?
Was benefit obtained (e.g.,
through improved communication and confidence in the responsibility and
performance of public health) when false alarms were investigated?
Who was affected?
What costs did partners incur
in follow-up of signals (e.g., medical record staff work and clinical staff
efforts)? Follow-up costs for false alarms should be distinguished from costs
related to investigations that uncover real outbreaks that warrant a public
health response.
Did the health department fail
to respond to a true event because of complacency or the response burden
resulting from false alarms?
Did late recognition of an
outbreak result in unnecessary morbidity?
Have lessons learned from
earlier events reduced costs as the system has continued to operate?
D. Conclusions and Recommendations for Use and Improvement of Systems
for Early Outbreak Detection
The evaluation should be summarized to convey the strengths and weaknesses of the system under scrutiny.
Summarizing and reporting evaluation findings should facilitate the comparison of systems for those making decisions about new or existing surveillance methods. These conclusions should be validated among stakeholders of the system and modified
accordingly. Recommendations should address adoption, continuation, or modification of the surveillance system so that it can better achieve its intended purposes. Recommendations should be disseminated widely and actively interpreted for all appropriate audiences.
An Institute of Medicine study concluded that although innovative surveillance methods might be increasingly helpful
in the detection and monitoring of outbreaks, a balance is needed between strengthening proven approaches (e.g., diagnosis
of infectious illness and strengthening the liaison
between clinical-care providers and health departments) and the
exploration and evaluation of new approaches
(17). Guidance for the evaluation of surveillance systems for outbreak detection is
on-going. Many advances are needed in understanding of systems and outbreak characteristics to improve performance metrics. For example, research is needed to understand the personal health and clinical health care
behaviors that might serve as early indicators of priority diseases; analytic methods are needed to improve pattern recognition and to integrate multiple streams of data; a shared vocabulary is needed for describing outbreak conditions, managing text-based information, and supporting
case definitions; and evaluation research is needed, including
cost-effectiveness of different surveillance models for early
detection, both in real-life comparisons and in simulated data environments, to characterize the size and nature of epidemics that can
be detected through innovative surveillance
approaches. Pending more robust measures of system performance, the goal of
this framework is to improve public health surveillance systems for early outbreak
detection by providing practical guidance for
evaluation.
Acknowledgements
This report includes contributions by Daniel S. Budnitz, M.D., National Center for Injury Prevention and Control; Richard
L. Ehrenberg, M.D., National Institute for Occupational Safety and Health;
Timothy Doyle, Robert R. German, Dr.P.H., Timothy
A. Green, Ph.D., Samuel L. Groseclose, D.V.M., Division of Public Health Surveillance and Informatics,
Denise Koo, M.D., Division of Applied Public Health Training, Carol A. Pertowski, M.D., Stephen B. Thacker M.D., Epidemiology Program Office; José G. Rigau-Pérez, M.D., National Center for Infectious Diseases, CDC, Atlanta, Georgia. Melvin A. Kohn, M.D., State Epidemiologist, Oregon Department of Human Services, Portland. Steven C. Macdonald, Ph.D., Office of Epidemiology, Washington State Department
of Health, Olympia. Nkuchia M. M'ikanatha, Dr.P.H., Pennsylvania Department of Health and Department, Harrisburg. Kelly
Henning, M.D., New York City Department of Health and Mental Hygiene.
Dan E. Peterson, M.D., Cereplex, Inc., Gaithersburg,
Maryland. Michael L. Popovich, Scientific Technologies Corporation, Tucson, Arizona. Scott F. Wetterhall, M.D., DeKalb County Board of
Health, Decatur, Georgia. Christopher W. Woods, M.D., Division of Infectious Diseases, Duke University Medical Center, Durham,
North Carolina.
Teutsch SM, Churchill RE. Principles and practice of public health surveillance.
2nd ed. Oxford, New York: Oxford University Press, 2000.
Lober WB, Karras BT, Wagner MM, et al. Roundtable on bioterrorism detection: information system-based surveillance. J Am Med Inform
Assoc 2002;9:105--15.
Reingold A. If syndromic surveillance is the answer, what is the question? Biosecurity and Bioterrorism: Biodefense Strategy, Practice, and
Science 2003;1:1--5.
Wagner MM, Tsui FC, Espino JU, et al. The emerging science of very early detection of disease outbreaks. J Pub Health Mgmt Pract 2001;6:51--9.
Effler P, Ching-Lee M, Bogard A, Ieong MC, Nekomoto T, Jernigan D. Statewide system of electronic notifiable disease reporting from
clinical laboratories: comparing automated reporting with conventional methods. JAMA 1999;282:1845--50.
Panackal AA, M'ikanatha NM, Tsui FC, et al. Automatic electronic laboratory-based reporting of notifiable infectious diseases at a large
health system. Emerg Infect Dis. 2002;8:685--91.
Hoffman MA, Wilkinson TH, Bush A, et al. Multijurisdictional
approach to biosurveillance, Kansas City. Emerg Infect Dis. 2003:9;1281--6.
Hutwagner L, Thompson W, Seeman GM, Treadwell, T. The bioterrorism preparedness and response early aberration reporting system (EARS).
J Urban Health 2003;80:89--96.
Tan CG, Sandhu HS, Crawford DC, et al. Surveillance for anthrax cases associated with contaminated letters, New Jersey, Delaware,
and Pennsylvania, 2001. Emerg Infect Dis. 2002;8:1073--6.
Robertson SE, Suleiman AJM, Mehta FR, Al-Dhahry SHS, El-Bualy MS. Poliomyelitis in Oman: acute flaccid paralysis surveillance leading to
early detection and rapid response to a type 3 outbreak. Bull WHO 1994;72:907--14.
Grosskurth H, Mosha F, Todd J, et al. Impact of improved treatment of sexually transmitted diseases on HIV infection in rural
Tanzania: randomized controlled trial. Lancet 1995;346:530--6.
Zeng X, Wagner M. Modeling the effects of epidemics on routinely collected data. Proc AMIA Annu Symp 2001:781--5.
Bishop YNM, Fienberg SE, Holland PW. Estimating the size of a closed population. In: Discrete multivariate analysis: theory and
practice. Cambridge, MA: MIT Press, 1975.
Broome CV, Horton HH, Tress D, Lucido SJ, Koo D. Statutory basis for public health reporting beyond specific diseases. J Urban
Health 2003:80:14--22.
Smolinski MS, Hamburg MA, Lederberg J. Microbial threats to health: emergence, detection, and response. Washington, DC: National
Academies Press, 2003.
CDC Evaluation Working Group on Public Health Surveillance Systems
For Early Detection of Outbreaks
Chair: Dan Sosin, M.D., Division of Public Health Surveillance and Informatics, Epidemiology Program Office, CDC.
Members: Claire Broome, M.D., Office of the Director; Richard Hopkins, M.D., Henry Rolka, M.S., Division of Public Health Surveillance
and Informatics; Van Tong, M.P.H., Epidemiology Program Office, CDC Atlanta, Georgia. James W. Buehler, M.D., Dept of Epidemiology, Rollins School of Public Health, Emory University, Atlanta, Georgia. Louise Gresham, Ph.D., San Diego Health and Human Services, Public Health Services, San Deigo, California. Ken Kleinman, Sc.D., Harvard Pilgrim Health Care, Cambridge, Massachusetts. Farzad Mostashari, M.D., New York
City Department of Health and Mental Hygiene, New York. J. Marc Overhage, M.D., Indiana University School of Medicine, Indianapolis. Julie
Pavlin, M.D., Division of Preventive Medicine, Walter Reed Army Institute of Research, Silver Springs, Maryland. Robert Rolfs, M.D., Utah Department of Health, Salt Lake City. David Siegrist, M.S., Potomac Institute for Policy Studies, Arlington, Virginia.
Use of trade names and commercial sources is for identification only and does not imply endorsement by the U.S. Department of
Health and Human Services.References to non-CDC sites on the Internet are
provided as a service to MMWR readers and do not constitute or imply
endorsement of these organizations or their programs by CDC or the U.S.
Department of Health and Human Services. CDC is not responsible for the content
of pages found at these sites. URL addresses listed in MMWR were current as of
the date of publication.
Disclaimer
All MMWR HTML versions of articles are electronic conversions from ASCII text
into HTML. This conversion may have resulted in character translation or format errors in the HTML version.
Users should not rely on this HTML document, but are referred to the electronic PDF version and/or
the original MMWR paper copy for the official text, figures, and tables.
An original paper copy of this issue can be obtained from the Superintendent of Documents,
U.S. Government Printing Office (GPO), Washington, DC 20402-9371; telephone: (202) 512-1800.
Contact GPO for current prices.
**Questions or messages regarding errors in formatting should be addressed to
mmwrq@cdc.gov.
