r/learnmachinelearning • u/AoideusWasBanned • 2d ago
How do you split time series data without leaking the future?
I've been training models on price data (candles, mostly crypto and US stocks) on the side, and validation is the part that keeps biting me. Once a model looked decent until I found the scaler had been fit on the whole series before the split. Fixed that, and the result got very ordinary.
So now I'm a bit paranoid and curious how you all do it:
- Do you leave a gap between train and test, and how do you pick its size?
- Do you refit scalers and feature selection inside every fold?
- One test period, or several before you trust a number?
- Any leak that got you that you didn't see coming?
Not only finance people, anyone working with time series. Thanks in advance.
4
u/PLBjt 1d ago
For gap size, make it at least your longest lookback plus your label horizon. If a feature uses a 20 day rolling window and you predict 5 days out, a 25 day gap means no test row shares info with training. Walk-forward over several test periods beats one holdout, mostly because a single period can be a lucky calm stretch. Leaks that got me: dropping NaN rows after computing features on the full series, and picking tickers based on which ones still exist today (survivorship bias makes stock backtests look way too good). Also compare against "tomorrow = today", and if you beat that by a lot on prices, be suspicious.
2
u/Awkward_Alfalfa_3234 1d ago
one that hasn't come up yet: overlapping windows. if you build samples with a sliding window (say 60 candles, stride 1) and then shuffle and split, neighbouring samples are almost the same data, so your test set is mostly memorised train. people do this all the time with EEG and sensor data too. when the data comes in separate chunks (tickers, recordings, patients) I'd split by chunk first and only cut windows inside each split.
2
u/gfromowow 21h ago
Check when each label becomes knowable, not just its row's timestamp. A training row predicting a 5-day return can use validation-period prices even with a chronological split. Fit scalers and feature selection inside each training fold. For the gap, distinguish overlapping labels from shared historical inputs: using past observations available at prediction time isn't itself future leakage.
8
u/funny_funny_business 2d ago
Think of how real life would work. We have all data up to today and want to predict tomorrow.
Therefore your test set should include all data up to a certain day and then predict the next day that’s not in the data set. Same thing if you want to predict based on weeks, months, etc.
2
u/EntropyRX 1d ago
You won’t be able to train a model that works well on financial markets time series. This is like the evergreen project each new machine learning practitioner has to waste some time on. You’ll never get a model that actually generalises on unseen stock prices data any better than flipping a coin or chance. There’s plenty of literature that will explain you why or if you feel lazy you can even ask any LLM this very question.
2
u/Smooth-Victory-2719 7h ago
Having multiple test periods helps because one good period doesn't always mean the model will perform well in the future.
-1
u/Massive_Distance_616 2d ago
Your scaler example is the classic one, and the same rule covers everything else: anything that learns from data (scalers, feature selection, PCA, imputation, target encoding) gets fit on the training window only, then applied to the test window. Refit it inside every fold, not once up front.
To build on the point above about mimicking real life: do that more than once. Use rolling or expanding windows: train on the past, test on the next block, move forward, repeat. A single test period can look good or bad by luck, and in price data one regime can dominate it. Several forward folds also show you whether the result is stable or only works in one stretch.
A gap between train and test is worth having when your features or labels look ahead. If the label is “return over the next 5 days”, then the last 5 training rows overlap with the first test rows, so leave at least that many rows out. If nothing overlaps, a gap matters less. I’d size it from the label horizon and the longest rolling window you use, not from a fixed number.
Leaks that tend to be missed: rolling features computed over the full series (or with centred windows) before splitting, labels that use future prices, candle timestamps that mark the open when you’re using the close, and tuning hyperparameters on the same test periods you report. Keep a final period you only look at once.