Biosurveillance: The Difference Between Detection, Signal Recognition, and Warning

For the last 15 or so years, we have seen an explosion of different electronic disease surveillance systems emerge.  Everything from laboratory diagnostics and reporting systems to syndromic surveillance to using the internet for so-called "event-based surveillance".  And there has been a virtual orgy of academic frenzy around the "bright lights and shiny object" effect of the technology.  This is representative of billions of taxpayer dollars over these years dispensed from a variety of federal mechanisms.

Yet, with all of this expenditure (much of which is ongoing), we have still so very little proof of operational value.  The "research and development ice cream cone" continues to lick itself without any apparent metric held by the funders for the operational "so what".  Arthur Reingold's challenge points from 2003 still appear to be unanswered ten years later.  Below, his points have been adjusted to fit a more broad consideration of the topic:

1. Demonstrating that [x, y, z type of] surveillance reduces morbidity and / or mortality following [an infectious disease crisis / disaster].
2. Types of [infectious disease crises / disasters] that [x, y, z type of] surveillance is likely to detect.
3. Response to apparent increases in illnesses signaled by [x, y, z type of] surveillance.
4. Plausibility that [x, y, z type of] surveillance can yield more timely identification of the population at risk in [an infectious disease crisis / disaster].
5. Likelihood that [x, y, z type of] surveillance will produce useful information about naturally occurring infectious diseases.
6. Circumstances under which [x, y, z type of] surveillance is likely to strengthen local and state health departments.

Certainly Reingold was not the only person during the heyday of syndromic surveillance to raise questions about the validity of the approach, but the gist of the above challenges were echoed several times in the literature.  But the academics lept onward undeterred, and the marketing associated with their programs grew to astounding levels of claimed accomplishments.

One such claim was that made by John Brownstein of HealthMap on Aug 3rd at an IARPA OSI Program Proposer's Day, who stated (paraphrased) that HealthMap was "the first surveillance system in the world to detect and publicly report the 2009 H1N1 influenza pandemic".  This impressive claim invokes scrutiny into the defintion of detection and how it applies to evaluating any disease surveillance system... and of course, the taxpayer funds associated with that program.

Merriam Webster defines the word detect as

1. to discover the true character of;
2. to discover or determine the existence, presence, or fact of
   

Webster's definition of recognize goes deeper than detect, however:

1. to acknowledge formally;
2. to acknowledge or take notice of in some definite way;
3. to perceive to be something or someone previously known

However it is the word's Latin origin, cognoscere that brings this point into focus.  Cognoscere means to know.  Of course the word know implies that a human being's mental processes have been engaged.  Which implies 1) there is a human being looking at the data and thinking about it. 

The definition of warning is provided by Webster as:

1. to give notice to beforehand especially of danger or evil
2. to give adminishing advice to
3. to call to one's attention

So this implies a human being, having thought about the data, has recognized and prioritized a concern... and communicated that specific concern to others.

The analytic team that warned the senior health intelligence officer in CDC's Director's Emergency Operations Center (DEOC) noted at the time that HealthMap's information was 1) not posted on their website as fast as the original Mexican source posting 2) was not highlighted in any unusual way to call singular attention to the information among dozens (hundreds?) of other colored pins on their map, and 3) where was no transfer of information to ProMED or CDC from HealthMap evident to suggest that human beings had communicated about the information during early to mid April 2009.  Therefore, the following may be concluded:

1. The first point of detection was actually the Mexican journalist who wrote the first article on the La Gloria situation.  HealthMap simply functioned as a Google News parser- a function anyone could have performed using Google Alerts.  If you happened to click that one pin on the map, you would have 'seen' the report, albeit then you would have to understand whether what was being reported was unusual...
   
2. Human beings were required to recognize unusual indicator patterns in the initial reports (n=1 to 3), not through recognition of simply volume of reporting as captured by HealthMap's Heat Index. 
3. Human to human communication clearly had to occur to push the warning through, as information provided via pull websites and push email notifications were inadequate due to daily information overload at WHO and CDC.

The point of the above is not to singularly criticize HealthMap but to make a point: collection of data is insufficient, by itself, to generate operational impact.  This was a very hard lesson learned by the syndromic surveillance community as well.  And if you are fighting an embedded, decades-old professional culture imbued with a "not invented here" bias (i.e. public health), it is unlikely the data will be exploited properly.

So let's give Brownstein a break for a minute and focus on one of his collaborators: the Canadian Global Public Health Network (GPHIN).  Brownstein's claim at the IARPA meeting was a duplication of GPHIN's claim to the world of having been the first public health system to detect SARS (see attached manuscript, Figure 1). However there was unverified rumor at the time that the web-harvested Chinese media article published on Nov 27, 2002 went unnoticed by GPHIN's staff for some time due to lack of machine translation for Chinese (unverified) and lack of a Chinese linguist.  Even if these rumors were false, and GPHIN's system immediately punched a warning out to its customers, there is no evidence anyone was paying attention.  In the multiple manuscripts and powerpoints given by GPHIN staff, not one contains copies of emails or phone records to indicate human beings had 1) recognized the problem, 2) prioritized the problem (out of the dozens of other reports for the day), and 3) communicated directly with the Canadian PHAC, the US CDC, or WHO specifically about this prioritized problem.  This is concerning when considering how much information was actually missed at the time (see attached report by Polyak et al).

The analogy is football: it's like being thrilled you threw a "Hail Mary" pass, but unawares that 1) no one caught the ball and 2) ran it to the end zone... 3) and thus you lost the game in the process.  Sound familiar?  It should.  These are very similar issues to those in the original 9/11 Commission Report.

Currently the federal government continues to chase its tail, spending more funds on biosurveillance research: the self-licking ice cream cone.  Billions of dollars and 10-15 years later, we still are not holding these systems' collective feet to the fire asking "show me don't tell me".  We continue to reach for the bright lights and shiney objects represented by the latest fad technology ("social media" being the current one) without thinking about how to promote the human process of understanding the data, accumulating knowledge, and using that knowledge to impact the potential outcome of the threat.

These are lessons currently highlighted in the recent IOM report on the DHS National Biosurveillance Integration Center (NBIC), however the problem is far bigger than just NBIC, DHS, or the people funded by DHS.  This is a biosurveillance community-wide problem that affects the credibility of the entire field. 

And if we do not fix it, the funding and public support may eventually disappear.

Download GPHIN Manuscript

Download Emergence of SARS in 2002