Motivations for weighting

The weighting scheme is incorporated because scores may be context-dependent. In other words, users may have varying concerns about specific components being evaluated given the timing, historical data patterns, and specific goals of their plausibility assessment. Below we have included several examples highlighting use-cases for applying the weighting scheme to plane_score():

• If users are retrospectively analyzing forecast signals after the ground-truth, observed data for that horizon has been reported, they may be aware that a large jump in reported cases actually occurred. In this scenario, they might be less interested in the difference component, which evaluates the forecast and raises a flag if there is a point-to-point difference greater than any difference found in the observed seed. However, the users might still be concerned about any unreasonable jumps in forecasted cases. Rather than eliminating the difference component altogether, the users can reduce its weight within plane_score().
• During an expected large uptick in cases, users might increase the weight of the trend component to more heavily penalize unexpected dips in cases. This ensures that significant trends are given the appropriate emphasis in the analysis.
• At the beginning of a season, when zeros may be very common in some locations, users might decide to reduce the relative weight of the zero and repeat components. This adjustment may help account for the seasonality and expected patterns in the data.
• When working with relatively short time series seed data (e.g., only several months), users may encounter many shape flags due to the limited number of shapes found in the seed data. In such cases, users can reduce the relative weight of the shape component.
• If evaluating forecasts operationally, users may be more interested in ensuring that prediction intervals are appropriately calibrated (e.g., not too narrow). In this case, the cover and taper components might merit higher weighting.

Reliability of seed data

The plausibility analysis in rplanes assumes that users have access to observed data to establish baseline characteristics of the time series. Presumably, this observed signal is trustworthy and is a faithful representation of what one should expect from the signal to be evaluated. However, in practice, data issues such as lagged reporting, backfill, and other systematic biases in ascertainment may lead to unexpected behaviors in rplanes (i.e., too many or too few flags raised). If there are known issues within the seed data, users should carefully consider all individual components used in the plausibility scoring and determine how they might impact the results of the plausibility analysis. For example, consider observed data that may lack consistent reporting in certain locations, particularly early on in the time series. In this case, the seed data might contain many consecutive zeros across a long time span followed by more reliable reporting at these locations, and therefore rplanes would rarely (if ever) raise flags for the repeat and zero components. For this scenario, truncating the observed data to begin when reporting becomes more reliable before creating the seed may be appropriate.

Length of seed data

Besides the reliability, the length (i.e., number of observations) of the observed signal used to create the seed can influence the rplanes plausibility scoring. In general, more data provides higher resolution for the characteristics that could manifest in the evaluated signal. However, users should be aware of computational costs and a potential reduction in sensitivity in some components as the amount of the available seed data increases. Here we provide several considerations and examples of balancing the length of observed data used to create a seed.

In scenarios when the seed has been created with a relatively small number of observations, users may notice that some components have higher sensitivity resulting in more flags raised. Some of the individual components have a required seed to signal length ratio that must be met for the function to run (e.g., shape requires that the seed is at least four times the length of the forecast being evaluated). However, even with a built-in minimum length, there are cases where the seed data may be too short to produce reasonable results. For example, consider evaluating a forecast of four weeks ahead. In this case, the seed must contain at least 16 weekly observations for the given location. However, 16 weeks is roughly four months of data, which (depending on seasonality and timing of the observations) may not adequately capture all of the shapes that could plausibly manifest four weeks into the future. Below is a table detailing some potential complications caused by a seed object that is too short. For all of these possible issues, we recommend that users should manually examine flagged locations as feasible and consider reducing their relative weights within the plane_score() function.

While too few observations in the seed may lead to limitations described above, too many seed values can also trigger unexpected behavior. As the amount of seed data increases, the plausibility components will have lower sensitivity, which my result in fewer flags. The rplanes package includes options to mitigate this. When using the repeat component, increasing the “prepend” length and decreasing the repeat tolerance should increase the sensitivity. In some circumstances, the decreased sensitivity cannot be mitigated by adjusting component parameters. When evaluating the shape component, a longer seed time-series will likely contain more unique shapes, resulting in fewer potentially novel shapes in forecasts and therefore fewer flags being raised. Having observed a similar epidemiological signal before is not (alone) enough justification to infer that this shape is not unusual and should not be flagged. For example, it may be appropriate to flag forecasts of COVID-19 activity in 2024 that exhibit trajectories similar to the most extreme surges in 2020-2021. However, in this situation if a user’s reference seed data included pandemic activity levels, the flag would not be raised.

Unlike situations when the seed data is too short, changing the relative weights of components not flagged will not have as much of an impact on overall scores. Down-weighting the components when using a short seed object causes potentially erroneous flags to have less of an effect on the overall score, however increasing the weights will not cause more flags to be raised (i.e., changing the weights does not change the sensitivity of components). Further, manual inspection of “missed flags” is much more challenging and time-consuming. We suggest that if users suspect that the sensitivity of components in plane_score() is negatively impacted by having too many seed values that they consider truncating the data prior to creating the seed.