Pooled residuals under a fast regime

A conformal threshold pools residuals across a hidden state that switches fast. How far its conditional coverage is from nominal depends on what the residual remembers: the regime at the end of the horizon, its average along the way, or the predictor's own slow state.

The model

A hidden state $y_t \in \{-1, +1\}$ flips at rate $1/\varepsilon$ in each direction. A forecast made at time $0$ for horizon $\Delta$ has a residual, or conformity score, $S$ whose law depends on the state; write $c = \Delta/\varepsilon$ for the number of regime lifetimes in the horizon. Three scores, each with independent Gaussian noise $\sigma Z$:

The first depends on the regime at the end of the horizon, as an error in a level does. The second aggregates the path, as an error in a cumulative quantity does. The third is the state of a predictor that filters the regime with its own memory $1/\kappa$, and the information it conditions on is $X_0$, not $y_0$.

Parameters for the checks: $\mu = \sigma = 1$, coverage $1 - \alpha = 0.9$, and $\kappa = 1$, $1/\varepsilon = 5$ for the slow state.

The quantity

The pooled threshold $q$ is the $(1-\alpha)$ quantile of the stationary law of $S$, the population version of a split-conformal threshold calibrated on residuals from every state. Its marginal coverage is $1 - \alpha$ by construction. The quantity is the coverage conditional on the forecaster's information at time $0$: $P(S \le q \mid y_0)$ for the first two scores and $P(S \le q \mid X_0)$ for the third.

Averaging

When the state switches infinitely fast the residual law is the same in every state, the pooled threshold is the conditional one, and pooling costs nothing. The corrector is the conditional coverage error at finite $\varepsilon$, and the three scores give it three different shapes.

Terminal score

The regime at the end of the horizon is the starting one with probability $\tfrac12(1 + e^{-2c})$. The pooled threshold does not depend on $c$, and the conditional coverage error is, exactly,

It is exponential in the horizon: two regime lifetimes bring it under $0.002$ here.

Path-average score

The average of the state over the horizon remembers the start algebraically,

because the starting regime holds for about $\varepsilon/2$ of a horizon of length $\Delta$, however long the horizon. The exact conditional law follows from the transform of the occupation time, $\mathbb E[e^{iu\mu A_\Delta} \mid y_0] = \big[\exp\big(\Delta Q + iu\mu\operatorname{diag}(1, -1)\big)\mathbf 1\big]_{y_0}$ with $Q$ the generator, inverted against the Gaussian noise. For small $\mu/\sigma$ the coverage error is $-(\mu/\sigma)\,\varphi(z_{1-\alpha})\,\mathbb E[A_\Delta \mid y_0]$, with $\varphi$ the normal density and $z_{1-\alpha}$ its quantile.

The two scores start together: for $c$ below one the forecaster's regime is still in force and both errors are of order one. Beyond that the terminal error dies like $e^{-2c}$ and the path error like $1/(2c)$, so a horizon that makes a terminal score safe leaves a path score biased, by a factor of 20 at $c = 2$ and 601 at $c = 4$. The first-order formula is within 10% at $\mu/\sigma = 0.3$ and overstates the error at $\mu/\sigma = 1$.

A predictor with a slow state

Let the predictor carry $X_t$, an exponentially weighted average of the regime with memory $1/\kappa$. In the stationary state $\tfrac12(1 + X)$ has the Beta law with both parameters $1/(\kappa\varepsilon)$, and $P(y_t = +1 \mid X_t = x) = \tfrac12(1 + x)$, both read off the stationary Fokker–Planck equation of the pair. Conditioning on the predictor's state and integrating the regime out,

The regime's own memory $e^{-2\lambda t}$ is there, but the leading term decays at the predictor's rate $\kappa$: the residual forgets the information it was conditioned on only as fast as the predictor does, however fast the regime switches. Fast switching shrinks the amplitude instead, since a typical $X$ is of size $\sqrt{\kappa\varepsilon/2}$.

The coverage error is the first-order formula with $x\,R(t)$ in place of the regime memory, while the endpoint memory has long vanished.

Calibration dependence is a separate effect

The threshold is estimated from residuals of forecasts made every $h$. For the terminal score their correlation comes only from the regimes at their endpoints, $r = e^{-2h/\varepsilon}$, whatever the horizon. The indicator $1\{S \le q\}$ has conditional means $F_\pm = \Phi((q \mp \mu)/\sigma)$ given the endpoint regime, so the covariance of two indicators $j$ steps apart is $r^j(F_+ - F_-)^2/4$ and the variance of the empirical coverage at $q$ is inflated by

So the horizon sets the bias of the pooled threshold given the current information, and the spacing sets how many effective calibration points there are. The two are controlled separately, and neither is fixed by the other. Block-permutation and mixing-coefficient treatments of the second effect are in Chernozhukov, Wüthrich and Zhu (2018) and Oliveira, Orenstein, Ramos and Romano (2024); for a finite chain the covariance sum above is the exact version of their bounds.

Thresholds

A regime-aware threshold, the Mondrian construction of Vovk, Gammerman and Shafer (2005) with the regime as the category, uses the quantile $q_\pm$ of the residual law given $y_0$ and has exact conditional coverage. The pooled threshold lies between $q_+$ and $q_-$: too narrow when the regime is high, too wide when it is low.

Weighting the calibration residuals by the posterior of the current regime interpolates between the two; for the slow-state predictor that posterior is $\tfrac12(1 + X_0)$. The pooled threshold is the choice that needs no regime at all, and the tables say what it costs.

Scope

The chain is symmetric with two states and the noise Gaussian; the thresholds are population quantiles, so the finite-sample randomness of a calibrated threshold enters only through the variance in the calibration section. Nothing here is a distribution-free conditional guarantee: the errors are computed in the model, and that is the point.

Nettasinghe, Chatterjee, Tipireddy and Halappanavar (2023) obtain exact marginal validity under a hidden Markov chain by randomizing over blocks; the pooled threshold studied here is the ordinary one, and its conditional error is what the corrector measures. The conformal prediction site gives the view from the other side: the guarantee holds without the corrector, and the sharpness given up is priced.

Certificate: verify_pooled_residuals.py.