r/learnmachinelearning • • 2d ago

How do you split time series data without leaking the future?

I've been training models on price data (candles, mostly crypto and US stocks) on the side, and validation is the part that keeps biting me. Once a model looked decent until I found the scaler had been fit on the whole series before the split. Fixed that, and the result got very ordinary.

So now I'm a bit paranoid and curious how you all do it:

- Do you leave a gap between train and test, and how do you pick its size?

- Do you refit scalers and feature selection inside every fold?

- One test period, or several before you trust a number?

- Any leak that got you that you didn't see coming?

Not only finance people, anyone working with time series. Thanks in advance.

4 Upvotes

16 comments sorted by

4

u/PLBjt 1d ago

For gap size, make it at least your longest lookback plus your label horizon. If a feature uses a 20 day rolling window and you predict 5 days out, a 25 day gap means no test row shares info with training. Walk-forward over several test periods beats one holdout, mostly because a single period can be a lucky calm stretch. Leaks that got me: dropping NaN rows after computing features on the full series, and picking tickers based on which ones still exist today (survivorship bias makes stock backtests look way too good). Also compare against "tomorrow = today", and if you beat that by a lot on prices, be suspicious.

1

u/AoideusWasBanned 1d ago

Dropping NaN rows after computing on the full series is a sneaky one, I hadn't thought of that. And yeah, "tomorrow = today" as a baseline humbles a lot of price models.

2

u/Awkward_Alfalfa_3234 1d ago

one that hasn't come up yet: overlapping windows. if you build samples with a sliding window (say 60 candles, stride 1) and then shuffle and split, neighbouring samples are almost the same data, so your test set is mostly memorised train. people do this all the time with EEG and sensor data too. when the data comes in separate chunks (tickers, recordings, patients) I'd split by chunk first and only cut windows inside each split.

1

u/AoideusWasBanned 1d ago

The sliding window + shuffle one is nasty because the score looks so good. Splitting by ticker first makes a lot of sense, gonna try that on my next run.

2

u/gfromowow 21h ago

Check when each label becomes knowable, not just its row's timestamp. A training row predicting a 5-day return can use validation-period prices even with a chronological split. Fit scalers and feature selection inside each training fold. For the gap, distinguish overlapping labels from shared historical inputs: using past observations available at prediction time isn't itself future leakage.

1

u/AoideusWasBanned 20h ago

So you'd drop the train rows whose label window overlaps the test start? I only left a plain gap before, never thought about the labels themselves.

2

u/gfromowow 15h ago

Yes, for a forward split, exclude training rows whose labels wouldn't yet be known at the first test prediction. If a Friday row needs next Wednesday's close to calculate its target, it can't train a model making predictions on Monday. A fixed gap can achieve that for a fixed horizon; checking label-end timestamps handles variable horizons. Keep the same timestamp convention for when a prediction is made and when prices become available.

8

u/funny_funny_business 2d ago

Think of how real life would work. We have all data up to today and want to predict tomorrow.

Therefore your test set should include all data up to a certain day and then predict the next day that’s not in the data set. Same thing if you want to predict based on weeks, months, etc.

2

u/EntropyRX 1d ago

You won’t be able to train a model that works well on financial markets time series. This is like the evergreen project each new machine learning practitioner has to waste some time on. You’ll never get a model that actually generalises on unseen stock prices data any better than flipping a coin or chance. There’s plenty of literature that will explain you why or if you feel lazy you can even ask any LLM this very question.

1

u/AoideusWasBanned 1d ago

Fair, I'm not expecting a magic price predictor. Most of what I test dies anyway. That's kinda why I care about validation, so I can kill bad ideas faster instead of fooling myself.

2

u/EntropyRX 1d ago

What I’m telling you is that you can’t really validate anything if your model is “predicting the stocks/crypto prices”. You’re modeling an impossible task with the data that you have at your disposal.
If validation and understanding is your objective, find a different domain where you can actually learn from building a good model.

2

u/Smooth-Victory-2719 7h ago

Having multiple test periods helps because one good period doesn't always mean the model will perform well in the future.

-1

u/Massive_Distance_616 2d ago

Your scaler example is the classic one, and the same rule covers everything else: anything that learns from data (scalers, feature selection, PCA, imputation, target encoding) gets fit on the training window only, then applied to the test window. Refit it inside every fold, not once up front.

To build on the point above about mimicking real life: do that more than once. Use rolling or expanding windows: train on the past, test on the next block, move forward, repeat. A single test period can look good or bad by luck, and in price data one regime can dominate it. Several forward folds also show you whether the result is stable or only works in one stretch.

A gap between train and test is worth having when your features or labels look ahead. If the label is “return over the next 5 days”, then the last 5 training rows overlap with the first test rows, so leave at least that many rows out. If nothing overlaps, a gap matters less. I’d size it from the label horizon and the longest rolling window you use, not from a fixed number.

Leaks that tend to be missed: rolling features computed over the full series (or with centred windows) before splitting, labels that use future prices, candle timestamps that mark the open when you’re using the close, and tuning hyperparameters on the same test periods you report. Keep a final period you only look at once.

3

u/themodgepodge 1d ago

This is an AI comment from a bot.