Here we conduct a simple comparison between the output of the Ascel Bio Black Canyon Infectious Disease Forecasting Station, Google's Flu Trends, HealthMap, and SickWeather. We are going to compare the output of the enhanced forecast provided by the Black Canyon Station that includes integrated national and local health intelligence data.
Black Canyon Infectious Disease Forecasting Station
Currently, we are dealing with a probable large-scale seasonal outbreak of human metapneumovirus (hMPV) here in Delta County, Colorado. Large enough we saw two more families affected with several more RSV- and influenza-negative cases of pneumonia. Disruptive enough to keep a large portion of our patients out of school for days as well as moms and dads. Disruptive enough to infect 2/4 of our clinical staff, prompting time off to recuperate and infect a third who worked through her illness.
This event represents a validated forecast specific for both the clinical syndrome and pathogen issued 30 days ago for the geographic coordinate Lat/Long: 38° 44' 43.4394", -108° 2' 49.1994" (Delta County, Colorado).
Pneumonia, as forecasted, has reached a peaking score on the Infectious Disease Medical Impact Scale (IDMIS), which indicates we are reaching an expected level of strain in terms of incidence and clinical severity.
RSV bronchiolitis is creeping upward as well, in keeping with forecasted expectations. Influenza activity has been zero in our pediatric population, meaning to-date we have not had a single positive test for influenza. Acute URI, otitis media, and bronchitis have all been observed, consistent with our forecasts. We are aware that Group A Streptococcus, rhinovirus, hMPV, adenovirus, and coronavirus are circulating at heightened levels as well- all of which were also forecasted. ... And this is but a snapshot of the nearly 100 diseases the station is currently forecasting and tracking live.
All disease activity considered, we are approaching our peak clinical load in terms of combined disease incidence and clinical severity, with evidence of mild infrastructure strain due to healthcare worker illness and absenteeism.
Flu Trends
This is the most current Flu Trends view of the state of Colorado regarding influenza:
The team at Google provided the following statement to the journal Nature:
Because the relative frequency of certain queries is highly correlated with the percentage of physician visits in which a patient presents with influenza-like symptoms, we can accurately estimate the current level of weekly influenza activity in each region of the United States, with a reporting lag of about one day.
According to Google, we are now at moderate levels of influenza for the state of Colorado. This is grossly incorrect. We have not seen a single positive influenza sample in any of our patients in our county, which is located west of the Front Range. To quote the latest influenza surveillance report from the State Department of Health, which utilized data up to Jan 28th, which is 5 days before the Flu Trends graph:
For the week ending January 28th, influenza activity remained low but increased somewhat based on sentinel hospital lab data and influenza-like illness (ILI) data. Influenza activity remained along the Front Range, as well as the northeast and southwest parts of the state.
According to the above Flu Trends graph for the state of Colorado, we were moving away from low levels of influenza activity in mid September.
Had we used Flu Trends, we would have been told there was moderate activity when activity was clinically non-existent. Further, there is no indication of whether influenza was at worse levels than prior years. In essence, a useless source of information for the purposes of operational biosurveillance or clinical medicine.
Below is a screenshot, taken on Friday, Feb 3, 2012 at 2000.
What is remarkable is the appearance of minimal infectious disease activity, not only for Delta County (which is located southeast of Grand Junction), but for the entire state of Colorado. This is most certainly not the case.
Clicking the only two pins shown for the state of Colorado reveals the following:
This is the limitation of solely relying on media reports and other internet-based reporting of infectious disease events.
SickWeather recently tweeted:
"#doctors #nurses be prepared for #patients - see what's going around this #cold & #flu season using http://t.co/9z59kOFA"
Had we been using SickWeather's map for our clinic, it too would have failed us. Below are screenshots of all of the symptoms we are seeing. There is no indication, based on the SickWeather mapping of the Western Slope, of the volume and clinical severity of the illnesses we are now seeing as physicians.
[The above figures were updated per SickWeather's comment- to be fair to them!] The only symptom displayed by SickWeather is pneumonia, with no indication of the true pervasiveness and severity in our kids that we are seeing. What would be interesting is to see how long that signal had been there and how closely it matched our clinical records. Either way, we have been issuing pre-event forecasts for pneumonia in our area for weeks prior to its appearance.
SickWeather claims to forecast infectious disease activity,
however this is incorrect and misleading. There is no pre-event (meaning an expression of what will happen before the event has actually occurred). It is highly doubtful that a patient would tweet a symptom before our knowledge of what was already happening in our community. Particularly regarding events of significance such as the outbreak of viral pneumonia we are seeing and associated medical infrastructure strain at the outpatient level.
Summary
The combined consideration of situational awareness provided by Flu Trends, HealthMap, and SickWeather failed to provide an accurate picture of current infectious disease activity in Delta County, or even the state of Colorado. None of these capabilities provided true forecasting of infectious disease activity.
While flashy, web-based technologies capture the attention and fancy of the media, academic journals, TED talks, blogs, and the like, there remains the practical reality of the need for experienced professionals to choose the appropriate data, assess the significance of the information, and practically answer the "so what" question. The current operational state of the art actually does not predominantly use the data provided by technologies such as Flu Trends, HealthMap, or SickWeather.
To place the world of operational biosurveillance in context, we often use the Microscope Analogy:
We consider open source / web-based event or horizon scanning, as offered by Flu Trends, HealthMap, or SickWeather, to be analogous to the low power lens of the microscope. The low power lens helps one become oriented to a specimen slide (i.e. a given location in the world), especially if one has not seen that slide before. Syndromic surveillance data represents a medium power lens. Clicking further as one zooms in for more comprehensive detail, one is using clinical laboratory diagnostic data.
This is precisely how the Haiti Epidemic Advisory System functioned. Initially open source information was used for orientation prior to deliberate social network engineering. Eventually, open source information was discarded in favor of direct ground observations and clinical data. One could not execute emergency medical response that included helo drops based solely on open source data.
It is important to view the media hype and the marketing around these technologies with a skeptical eye... because operational reality is quite a bit different: better accuracy, far faster, more complex, operates in the pre-event domain, and has been used clinically on a daily basis.
Recent Comments