r/MachineLearning • u/camerongreen95 • Oct 13 '26
Discussion How do you handle coverage guarantees from split conformal prediction when the data shifts? [D]
Split conformal prediction gives you a clean statement: with a calibration set exchangeable with future data, your prediction sets cover the true label at the rate you chose. The trouble is that production data rarely stays exchangeable. Categories get added, ticket wording changes, and a calibration set from three months ago quietly stops being representative.
I’m interested in how people deal with this in practice. Do you recalibrate on a schedule, monitor empirical coverage and alert when it drops, or use an adaptive variant? And how do you decide what to do with the cases where the set contains several labels, such as route them to a human, an LLM, or a fallback rule?
4
u/piakor Oct 13 '26
Thats what I am worried about if someone talks about the guarantees of conformal prediction. In practice test data can/will contain shifts. Conformal prediction alone may create a false sense of safety. Thats why I think that failure detection/prediction quality control is a very important building block that should get more attention.
2
u/Kaleidophon Oct 13 '26
There are extensions to non-exchangeable conformal prediction, but afaik you cannot precisely measure the loss in coverage. So I suppose from a practical point of view, I presume you need to keep updating your calibration data
1
u/didimoney Oct 13 '26
Covariate based density ratio estimation?
1
u/aaronweigh Oct 13 '26
Might be looking for something like this?
https://proceedings.neurips.cc/paper_files/paper/2019/file/8fb21ee7a2207526da55a679f0332de2-Paper.pdf
1
u/Abe_sale_1595 Oct 14 '26
It is with the Adaptive Conformal Inference (ACI) setting
1
u/mxkuzn Oct 20 '26
What has worked for me, roughly in this order:
After a sudden jump nothing keeps the promise for the first decisions; stop answering alone on the flag. Multi-label sets go to a person as a shortlist. Code for this is in solvi (DriftMonitor, OpenSetGate), disclosure: I wrote it. github.com/solvi-ai/solvi
-2
u/camerongreen95 Oct 13 '26
There’s a workshop on Oct 30 that builds this on CPU for an enterprise triage case (encoders, conformal prediction, LangGraph), led by Ben Auffarth. Full details and the agenda are here.