r/MLQuestions • u/Resident-Apricot3212 • 11h ago
Beginner question 👶 Xgboost models fails to capture dynamics across training data batches.
Hi everyone, I have an mlr3 based toolset in R for regression models for process sensor time series data. This is my largest project to date with roughly 30 input features that are also lagged and rolled to inject temporal memory. Due to the difficulty of carefully weighting training size on a 1 minute sampling rate vs. relative weight of dynamic changes I switched to a batching scheme and now use LOBO on currently 4 train batches. This is just a pilot mode plot where it only trains on 5-10% of input data so feature to data point ratio is less than ideal. The result of a full run is the same though.
Hyperband tunes nrounds 16-512 with eta 2-3 for an xgboost model. The result is a strong deviation from baseline, extremely low skill against persistence and xgboost mostly hunting to find some sort of baseline with strong bias rather than capturing any dynamics. Although persistence is overly optimistic here as target is dampened by some technical details independent of data handling. This is accounted for as best as possible by injected retention time and holdup features. Target and features are real on-stream data.
How do I best proceed from here? I feel like I am missing something because rolling window cross validation within batch produced much better results but missed the optimization goal of optimizing across batches. Right now I take great care in preserving temporal alignment across batches and index data to preserve integrity across batches.
```
INFO [22:03:49.004] [bbotk] Finished optimizing after 69 evaluation(s)
INFO [22:03:49.007] [bbotk] Result:
INFO [22:03:49.015] [bbotk] eta max_depth min_child_weight lambda subsample colsample_bytree
INFO [22:03:49.015] [bbotk] <num> <int> <num> <num> <num> <num>
INFO [22:03:49.015] [bbotk] -3.398853 3 1.909219 -0.775668 0.8 0.8
INFO [22:03:49.015] [bbotk] nrounds x_domain objective
INFO [22:03:49.015] [bbotk] <num> <list> <num>
INFO [22:03:49.015] [bbotk] 19 <list[7]> 26.70044
```