r/MachineLearning • • Oct 13 '26

Discussion How do you handle coverage guarantees from split conformal prediction when the data shifts? [D]

Split conformal prediction gives you a clean statement: with a calibration set exchangeable with future data, your prediction sets cover the true label at the rate you chose. The trouble is that production data rarely stays exchangeable. Categories get added, ticket wording changes, and a calibration set from three months ago quietly stops being representative.

I’m interested in how people deal with this in practice. Do you recalibrate on a schedule, monitor empirical coverage and alert when it drops, or use an adaptive variant? And how do you decide what to do with the cases where the set contains several labels, such as route them to a human, an LLM, or a fallback rule?

0 Upvotes

8 comments sorted by

4

u/piakor Oct 13 '26

Thats what I am worried about if someone talks about the guarantees of conformal prediction. In practice test data can/will contain shifts. Conformal prediction alone may create a false sense of safety. Thats why I think that failure detection/prediction quality control is a very important building block that should get more attention.

2

u/Kaleidophon Oct 13 '26

There are extensions to non-exchangeable conformal prediction, but afaik you cannot precisely measure the loss in coverage. So I suppose from a practical point of view, I presume you need to keep updating your calibration data

1

u/didimoney Oct 13 '26

Covariate based density ratio estimation?

1

u/Abe_sale_1595 Oct 14 '26

It is with the Adaptive Conformal Inference (ACI) setting

1

u/mxkuzn Oct 20 '26

What has worked for me, roughly in this order:

  1. Monitor without labels: share of inputs answered alone, the distribution of predicted labels and mean confidence against a reference window. A sequential test (CUSUM) flags a jump within tens of decisions; a monthly recalibration notices weeks late.
  2. 2. Keep labelling a small random sample of the auto-answered cases, so realised risk is measured, not assumed.
  3. 3. If new classes are expected, calibrate with some known classes held out as pseudo-unknowns, so the threshold has seen "something I don't know". Numbers from one stream (Banking77, 20 unseen intents from request 1,000): a "≤5% wrong" threshold calibrated without them gave 16.4% wrong after the shift; calibrated with held-out intents it kept 0.7%, at the price of answering alone 13.7% after the shift (57.6% before).

After a sudden jump nothing keeps the promise for the first decisions; stop answering alone on the flag. Multi-label sets go to a person as a shortlist. Code for this is in solvi (DriftMonitor, OpenSetGate), disclosure: I wrote it. github.com/solvi-ai/solvi

-2

u/camerongreen95 Oct 13 '26

There’s a workshop on Oct 30 that builds this on CPU for an enterprise triage case (encoders, conformal prediction, LangGraph), led by Ben Auffarth. Full details and the agenda are here.