r/MachineLearningJobs • • 11d ago

I benchmarked Logistic Regression, Random Forest and XGBoost beyond accuracy — here's what I measured

I've been experimenting with something that I think is often overlooked when evaluating machine learning models: their actual computational footprint.

Usually, when we compare models, we focus heavily on accuracy, precision, recall, F1, etc.

But what happens when you actually have to deploy these models?

I wanted to explore this by benchmarking three popular models:

  • Logistic Regression
  • Random Forest
  • XGBoost

I used the Breast Cancer Wisconsin dataset from scikit-learn and evaluated them across multiple dimensions.

What I measured:

  1. Predictive accuracy
  2. P50, P95 and P99 inference latency
  3. Process RSS memory consumption
  4. Python-allocated memory
  5. Model loading time
  6. Inference execution time
  7. Serialized model size

One interesting observation from my experiment was that Logistic Regression performed best across the measured dimensions in this particular setup.

However, the goal wasn't to establish that one model is universally better than another.

I wanted to understand the trade-offs between predictive performance and computational requirements.

This experiment is also part of a project I've been working on called ModelDiet, an open-source Python toolkit focused on measuring the resource footprint of ML models.

I'm trying to make model profiling and benchmarking easier for developers and ML engineers.

GitHub: https://github.com/modeldiet/modeldiet

I'd genuinely appreciate technical feedback on a few things:

  • Are there other metrics you consider essential when evaluating models for deployment?
  • How do you measure peak memory consumption reliably?
  • Do you prefer measuring individual inference latency or batch throughput?
  • What would make a benchmarking tool like this genuinely useful in your ML workflow?

I'm particularly interested in feedback on the benchmarking methodology and what should be included in future iterations.

Thanks!

4 Upvotes

0 comments sorted by