Below are the results of our assessment of forecast validation for Artibonite Province, Haiti. We are quite pleased with the results from an operational perspective.
This is the area:
Basically, the forecasts vs. the actual outcomes for:
1) Anse-Rouge was outstanding with three peaks in both, and with both the timing and the amplitudes of the 2nd and 3rd peaks right on.
2) Govaives was solid with three peaks in both but only okay as the timing and amplitude of the 1st and 3rd peaks are off while that of the 2nd peak was right on.
3) Marmelade was solid with three peaks in both and okay as the timing and amplitude of the 1st and 3rd peaks are right on while that of the very broad 2nd peak was off.
4) TerrNeuve was off in the timing of the peaks, but the amplitudes of the peaks were solid and the number of peaks, with six predicted and six occurring, was outstanding.
5) Ennery was very good as both the timing and amplitudes of the actual and predicted 5th peaks were right on, however while the forecast was for four peaks and there were five actual, the predicted 3rd peak fell between the 3rd and 4th actual peaks, the agreement was not too bad overall. Additionally, considering the relatively high and low numbers of actual and predicted cases, the comparison is very good.
6) Grosmorne was okay considering the low number of cases.
7) St. Michel shows a mixed outcome as the only one peak was predicted and the timing and amplitude of that peak vs. the actual realization was outstanding, however the forecast missed the 2nd and 3rd peaks entirely.
So why did the model do well in some cases and not so well in others?
We believe that there are several fundamental reasons.
On the positive side, there is good structure and momentum in the individual and collective time series. As such the memory in the systems is capable of being carried forward in a probabilistic sense. So, one can conclude that there are individual and distinct signals which can be and have been separated from the noise in each of the time series.
Albeit, one must recognize that the forecasts are totally probabilistic and not deterministic. Moreover. and this point cannot be overstated, the time series are very short (~2 years) in all cases, so there are very few data points. Also, there have not been any correlative analyses done with other likely relevant parameters and variables, other then the cross correlations that we conducted between the stations.
The challenge created by the short time series can be briefly summarized with a simple example. To wit, if Dave left his office to walk across campus and the first two students he saw were wearing NC State tee shirts, then what would he guess, probabilistically speaking, that the next student would be wearing; an NC State tee shirt or not; 50/50?
Again, probabilistically speaking, if Bob found that using 117 years of hourly temperature data that the coldest two weeks of the year occur during the 2nd and 3rd weeks of January, then what about the forecast for 2013? What are the odds that it would necessarily be those two weeks or could it end up being the 4th week of January and the 1st week of February, et cetera? As such, probabilistic forecasts are not deterministic.
In terms of the timing and amplitude, while we have several
observations, they cover only a bit over 1 year so we have had only one look at
a pattern that is not necessarily going to repeat the next year. The 117
year example gets at the idea of probabilistic forecasting but our case is much
worse, having seen not 117 instances of a yearly pattern but only a bit over
In our forecasts we did not account for any of the external variables that the different regional facilities are located in, such as: relationships to weather and climate factors, with the former and latter including such conditions as warm and humid conditions, altitude, arid versus no-arid conditions, thus low or high low precipitation, transportation availability, water contamination, the ability of the stricken to get to the hospitals, the exposure of the populace to disease carrying hosts, such as outsiders carrying the disease into the locale(s), polluted water, and so on.
So the solution to the improvement of forecasts would be to have longer time series at individual and multiple locations which document incidence and also to be able to conduct correlative analyses with other factors that may be conditional and or causal in nature. We actually showed the power of such studies and analyses with our cross-correlations between station to station cholera data and also our cross-correlation study of the other diseases and medical conditions. These powerful relationships, or lack therein, were in the report as well.
All of that said, the results are very encouraging, especially given the paucity of data, suggesting that highly reliable relationships can be created given appropriate data sets, and sufficient time and effort.