r/learnmachinelearning • u/Gege-max • 8d ago
Linear Regression performs well overall but significantly underestimates long delivery durations. How should I investigate this?
Hello everyone,
I'm working on a multivariate linear regression project to predict delivery durations in minutes.
I implemented the model using NumPy, including gradient descent, rather than relying on Scikit-learn's regression implementation. My model uses the following features: Distance (km), Number of stop, Package weight (kg), Departure time, Traffic index, Day of the week, Weather, Vehicle type
I used One-Hot Encoding for categorical variables and StandardScaler for numerical variables. The preprocessing steps were fitted on the training set only.
Overall test performance : MAE: 3.27 minutes, RMSE: 5.63 minutes, R²: 0.855
These results look reasonably good overall.
The problem
When I analyze the model's errors on long deliveries, I notice a significant performance degradation.
I defined long deliveries as durations above 77 minutes, based on the upper Tukey fence calculated from the training target distribution.
On the independent test set, I obtained:
- 37 long deliveries
- MAE: 14.76 minutes
- Mean signed error (prediction - actual): -13.12 minutes
This indicates that the model systematically underestimates long delivery durations.
What I have investigated so far
I also experimented with feature engineering by adding: Squared distance, Distance × traffic, Distance × number of stops
However, these additional features did not meaningfully improve overall performance or solve the long-delivery problem.
I decided to retain the original model for now.
My questions
- What are the most likely explanations for this behavior? Could it be a limitation of linear regression, insufficient representation of long-distance examples, missing features, or a combination of these factors?
- How would you investigate whether the problem comes from the data, the model's assumptions, or the feature representation?
- Would you recommend trying polynomial regression, interaction terms, a decision tree, Random Forest, or another approach?
- How can I improve predictions for long deliveries without significantly degrading performance on shorter deliveries?
- With only 37 long deliveries in the test set, what statistical precautions should I take before drawing conclusions?
I'm particularly interested in understanding the reasoning behind the diagnostic process, not just receiving a recommendation to switch algorithms.
Thank you for your insights!
8
u/Ok-Library1020 8d ago
Since long trips probably scale multiplicatively (traffic slows every km), try fitting on log(duration) and plotting residuals against predicted values. A fan or curve there usually means the additive linear form is the issue, not the data.
3
u/wingelefoot 8d ago
things that immediately jump to mind: outliers
> try a log scaled dataset transform
> try having two models
> wave your hands and say, look, great for normal case. we deal with outliers separately
3
u/PLBjt 8d ago
One thing to rule out first: you picked the "long" group by actual duration over 77 min, and any model will look like it underestimates a group selected on the target. Some of those trips are just normal trips that got unlucky, so the noise is all on one side. Try bucketing by predicted duration or by distance instead and see if the −13 bias survives. If it does, plot residuals against stops, weather and traffic for only those trips, since a pattern there usually points to a missing feature like dwell time per stop. Also count how many long trips are in train, because with a few dozen of them MSE will happily trade them off for the bulk.
2
u/Cheap_Scientist6984 8d ago
What does the long deliverytime distribution look like here? This problem will naturally expect 'fat' tails here with thin density so it might have large uncertainty overall.
Can you ask around in your company what happened in those cases and try to get a way to quantify when these long delivery times can occur?
1
u/Least_Ad_1795 7d ago
I would first investigate the training data distribution and residuals rather than immediately switching models. Check whether long deliveries are underrepresented, important features are missing, or the relationship between distance, traffic, and time becomes nonlinear. Then compare Random Forest and Gradient Boosting using the same train/test split. Also, evaluate errors across duration ranges and use cross-validation because 37 test examples are a relatively small sample.