1. Problem Context

The forecasting target considered here is log realized volatility. Volatility is an instructive test case because it is persistent, clustered, and highly nonstationary. The process frequently alternates between relatively calm periods and abrupt transitions in which the conditional scale changes quickly. A model that appears adequate in one interval can become misaligned soon afterward, not because the forecast equation is intrinsically poor, but because the underlying drift level has shifted.

This creates a specific statistical difficulty. Standard time-series models represent persistence well, but they are less effective when the underlying level of the process moves over time in a way that is neither constant nor fully captured by periodic refitting. In practice, the forecaster then faces repeated forecast lag: after a transition, the model reacts slowly, produces directional bias, and accumulates error until the parameter estimates catch up.

In volatility forecasting, that lag matters. A drift misspecification can affect both point forecasts and derived event probabilities. If the model underestimates the current volatility level during a transition, the point forecast is biased downward and threshold-based probability assessments become miscalibrated. For this reason, drift modeling is not a secondary detail. It is a central part of the forecasting problem.

ARIMA-style models typically deal with drift in one of two ways. They either include a fixed intercept or allow adaptation only through rolling or repeated refitting. Both approaches are reasonable under mild nonstationarity, but both effectively assume that drift evolves slowly relative to the retraining schedule. When the process moves more quickly, the assumption becomes restrictive.

The central question of the analysis is therefore the following: can the drift term be updated online in a disciplined way, using forecast error itself as feedback, rather than relying only on periodic re-estimation?

Model evolution

Baseline ARIMA
      |
      v
Adaptive Drift Adjustment
      |
      v
Regime-Aware Drift
      |
      v
Composite Drift Model

2. Baseline Modeling Framework

The starting point is a conventional one-step-ahead forecasting framework for log realized volatility. The baseline model family includes naive persistence, EWMA smoothing, static AR(1), static ARIMA, rolling AR(1), and rolling ARIMA. These specifications provide a standard comparison set: some are deliberately simple and some allow limited parameter adaptation through rolling estimation.

The adaptive formulations preserve the same basic predictive structure,

\[\hat{y}_t = \mu_t + \phi(y_{t-1} - \mu_t),\]

where \(\phi\) governs short-run persistence and \(\mu_t\) is the drift or local equilibrium level. In the static case, \(\mu_t\) is fixed. In the adaptive case, the equation is unchanged except for the rule used to update \(\mu_t\) over time. That design choice is important because it isolates the effect of drift adaptation from broader changes in model class.

Evaluation is strictly walk-forward. Each forecast is constructed using information available up to the forecast origin. No future observations are used in fitting or updating. This is a stronger design than a pooled in-sample comparison because it exposes regime transitions, forecast lag, and accumulation of error in real time.

The primary forecast metrics are root mean squared error, mean absolute error, and correlation between the forecast and the realized target. For the probability experiments, the main diagnostics are expected calibration error, Brier score, and log loss. Together, these metrics cover both level accuracy and probabilistic reliability.

baseline_top = point_phase_1 |>
  dplyr::arrange(rmse) |>
  dplyr::slice_head(n = 8)

knitr::kable(
  baseline_top,
  digits = 4,
  caption = "Leading models from the initial walk-forward comparison"
)

3. Limitations of Static Drift

The initial comparison indicates that fixed or slowly updated drift creates at least three problems.

First, forecasts tend to lag when the latent volatility level changes rapidly. The model inherits persistence from the recent past but adapts too slowly to the new regime. In practice this appears as underreaction near volatility upshifts and overreaction after conditions normalize.

Second, static drift encourages systematic forecast bias during transition periods. The model may still have reasonable average accuracy over a long interval, but that aggregate score can conceal localized episodes in which the sign of the forecast error is persistently one-sided.

Third, the effect on probabilities is often more severe than the effect on point error. A forecast with tolerable RMSE can still produce event probabilities that are poorly calibrated if the conditional distribution has shifted. This matters whenever the forecast is used for threshold-based decisions rather than only for ranking or smoothing.

The first phase of results shows that adaptive variants are competitive with or better than the stronger rolling baselines, but the margins are not large enough to justify complexity without a clear mechanism. That is precisely why a more explicit motivation is needed.

knitr::kable(
  phase_1_rank_summary |>
    dplyr::slice_head(n = 8),
  digits = 4,
  caption = "Cross-run rank summary from the initial controller comparison"
)

4. Control-Theoretic Motivation

Control theory becomes relevant once forecast drift is viewed as an online adjustment problem rather than a purely static estimation problem.

The analogy is straightforward. In a feedback control system, one defines a target, observes the deviation between the target and the realized state, and updates a control input to reduce the deviation. In the forecasting setting, the realized value is observed after the forecast is issued, and the forecast error can be treated as the relevant deviation signal. The drift term then plays the role of a control input: it shifts the forecast center in response to persistent error.

This perspective is attractive for nonstationary forecasting because it focuses directly on the quantity that reveals model mismatch. If the process is drifting upward, forecast errors will tend to be positive; if it is drifting downward, forecast errors will tend to be negative. An adaptive drift rule can therefore respond to the empirical direction and persistence of error without requiring a full model re-estimation at every step.

The objective is not to convert the forecasting problem into a physical control problem literally. The objective is to borrow a disciplined feedback logic: use observed error to adjust the local operating point of the forecast equation in a stable and interpretable way.

5. PID Control Background

Proportional-integral-derivative control is one of the most common feedback mechanisms in engineering.

The proportional component reacts to the current error. If the system is above target, the proportional term pushes in the opposite direction immediately. If it is below target, it pushes back upward. Proportional correction is therefore fast, but by itself it can leave a residual steady-state error.

The integral component reacts to accumulated error over a recent window. If small errors persist in the same direction for many periods, the integral term grows and forces a stronger correction. In engineering systems, this is the component that removes sustained bias.

The derivative component reacts to the change in error. It anticipates turning points by looking at the slope of the error signal rather than only its current level. In practice, derivative correction can reduce overshoot or improve responsiveness around transitions, though it is also more sensitive to noise.

Together, the three terms offer a simple decomposition of feedback:

• proportional for immediate response
• integral for persistent bias
• derivative for acceleration or turning-point behavior

These properties make PID control a natural candidate for drift adjustment in a forecasting system whose main failure mode is delayed response to shifts in the local level.

6. PID-Based Drift Adjustment

The adaptive drift formulation keeps the AR(1)-style forecast equation but allows the drift parameter to evolve over time. Let \(e_t = y_t - \hat{y}_t\) denote the forecast error and let \(z_t\) denote a normalized version of that error. The drift update is then constructed from three components:

• a proportional term based on the current normalized error
• an integral term based on the recent average of normalized errors
• a derivative term based on the change in normalized error

Conceptually, the update has the form

\[ \Delta \mu_t = K_p z_t + K_i \bar{z}_t + K_d (z_t - z_{t-1}), \]

with clipping or other stabilization to keep the adjustment within a feasible range. The new drift becomes

\[ \mu_{t+1} = \mu_t + \Delta \mu_t. \]

This rule has an intuitive interpretation in the forecasting setting. If the current forecast is persistently too low, recent errors are positive and the proportional and integral terms push the drift upward. If the error is changing rapidly, the derivative term sharpens or dampens the update depending on whether the misspecification is accelerating or reversing.

The appeal of the construction is that the model remains interpretable. The persistence parameter still controls short-run autoregressive dynamics, while the drift update controls how quickly the local level is allowed to move in response to forecast error.

7. Experimental Development

knitr::kable(
  phase_2_pid_table,
  digits = 6,
  caption = "Phase 2 probability metrics for the PID-based forecast family"
)

knitr::kable(
  phase_3_forecast_top,
  digits = 6,
  caption = "Phase 3 forecast metrics recomputed from retained prediction paths"
)

knitr::kable(
  phase_3_prob_pid,
  digits = 6,
  caption = "Phase 3 probability metrics for the PID-based forecast family"
)

knitr::kable(
  phase_4_forecast_top,
  digits = 6,
  caption = "Phase 4 forecast metrics recomputed from the integrated result panel"
)

knitr::kable(
  phase_4_prob_pid,
  digits = 6,
  caption = "Phase 4 probability metrics for the PID-based forecast family"
)

8. Model Comparison

knitr::kable(
  comparison_table,
  digits = 6,
  caption = "Concise comparison of major model variants"
)

The table highlights a consistent pattern. The main gain in point forecasting comes from adaptive drift, especially when the model is allowed to adjust during regime transitions. The main gain in probability forecasting comes from adaptive calibration, especially when the calibration map is updated smoothly rather than controlled too aggressively.

9. Interpretation

Several substantive conclusions follow from the experiments.

First, control-theoretic ideas appear useful because they address the right failure mode. The key empirical problem is not lack of autoregressive structure. It is delayed response to a changing local level. Feedback-based drift updates target exactly that issue.

Second, the best-performing adaptive methods are not the most elaborate ones. The initial controller comparison suggests that moving from static drift to adaptive drift matters more than moving from a simple adaptive controller to a highly parameterized one. This is consistent with the idea that the dominant benefit comes from correcting the direction and persistence of drift, not from modeling every local nuance.

Third, the experiments separate point forecasting from calibration in an instructive way. The strongest probability improvements came from adaptive calibration rules, while the strongest point-forecast improvements came from adaptive or regime-aware drift adjustments. These are related problems, but not identical ones.

Fourth, some intermediate gains weaken when the evaluation design is tightened. The regime-adjusted variant is strongest in the third phase, but the aligned integrated comparison retains a simpler adaptive drift forecast as the best point model. This suggests that some apparent gains are phase-specific and that model retention should depend on stable comparative evidence rather than on a single favorable run.

Remaining limitations should also be noted. The analysis is centered on one asset and one forecasting target. The adaptive rules are intentionally simple. They improve responsiveness, but they do not explicitly model deeper structural drivers of regime change. In addition, the strongest point model and the strongest probability model are not identical, which means practical deployment would require clarity about the forecasting objective.

10. Conclusions

The analysis supports a clear conclusion: control-theoretic drift adjustment is a useful and interpretable way to handle nonstationary forecasting problems when the main failure mode is lag in the local level.

Three findings are especially important.

1. Static or periodically refit drift specifications react too slowly during transitions in the volatility environment.
2. Online drift adjustment improves point forecasting, but the strongest gains come from disciplined simplicity rather than from maximal controller complexity.
3. Adaptive calibration delivers a large improvement in probability quality and complements, rather than replaces, adaptive point forecasting.

The broader lesson is methodological. Treating forecast error as a feedback signal provides a practical way to update drift without reconstructing the entire model at every step. In this setting, that idea was strong enough to survive several rounds of refinement and to produce both better point forecasts and materially better calibrated event probabilities.

Several research directions remain open. A natural next step is to extend the adaptive drift framework to richer state-space or multivariate volatility settings. Another is to formalize regime detection rather than using it only indirectly through forecast and calibration signals. A third is to examine whether the same feedback logic improves downstream decision rules, not only forecast accuracy and calibration diagnostics.

Minimal supporting visual

forecast_plot_data = forecast_plot_data |>
  dplyr::mutate(
    hover_text = paste0(
      "Date: ", date,
      "<br>Series: ", model,
      "<br>Value: ", sprintf("%.4f", dplyr::if_else(model == "actual", actual, forecast))
    )
  )

actual_trace = phase_3_predictions |>
  dplyr::select(date, actual) |>
  dplyr::distinct() |>
  dplyr::mutate(date = as.Date(date)) |>
  dplyr::filter(date >= max(date) - 260)

plotly::plot_ly() |>
  plotly::add_lines(
    data = actual_trace,
    x = ~date,
    y = ~actual,
    name = "Actual",
    line = list(color = "#222222", width = 2)
  ) |>
  plotly::add_lines(
    data = forecast_plot_data |> dplyr::filter(model == "ar1_rolling"),
    x = ~date,
    y = ~forecast,
    name = "AR(1) rolling",
    line = list(color = "#1f77b4", width = 1.2)
  ) |>
  plotly::add_lines(
    data = forecast_plot_data |> dplyr::filter(model == "pid_adaptive"),
    x = ~date,
    y = ~forecast,
    name = "PID adaptive",
    line = list(color = "#ff7f0e", width = 1.2)
  ) |>
  plotly::add_lines(
    data = forecast_plot_data |> dplyr::filter(model == "pid_adaptive_regime_adj"),
    x = ~date,
    y = ~forecast,
    name = "PID adaptive regime-adjusted",
    line = list(color = "#2ca02c", width = 1.4)
  ) |>
  plotly::layout(
    title = list(text = "Representative forecast comparison over a recent evaluation window"),
    xaxis = list(title = ""),
    yaxis = list(title = "Log realized volatility"),
    legend = list(orientation = "h", x = 0, y = -0.2)
  )