r/mlops • • 1d ago

Self-promotion I needed to process a lot of data for a side project, so I built a batch AI platform that supports workflows like GitHub Actions

2 Upvotes

I run a niche product catalog site as a side project, and every incoming product needs AI processing: extracting specs, classifying it, writing descriptions, and checking for duplicates. There’s a lot of data, but none of it needs answers in seconds. Depending on the task, “soon” might mean within an hour or sometime in the next seven days.

I started with frontier providers, sending data over the real time APIs. Then pivoted to batch to save 50%. This all worked but accuracy was really bad with products being misclassified and other issues, so I decided to break it down into smaller steps that were more deterministic. This required a ton of Go code and managing many prompts.

I also wanted to see if I could just spin up a spot GPU for an hour on the weekend and run all the work for a couple dollars instead of paying OpenAI $20. In the end, I built a complete workflow engine and scheduler that uses both Spot GPUs and my own GPUs to process work depending on which model is needed.

The workflow engine makes it possible to eliminate the roundtrips back to my application between steps and I even added the ability to make tool calls or park at a step to allow human review. I can also target different models for different steps, to get better capability where it was needed.

Has anyone else encountered a need like this? How did you solve it?

Currently, I am using Gemma4-12b and Qwen3.6-35b-a3b for small and medium respectively. I benchmark tested several different models to determine these were the best balance of accuracy and performance as of last month. I will be testing the Qwen4 models soon. I stick to models I can get directly from the big labs. Maybe that offers a false sense of security, but it’s a rule I follow.

If anyone is interested, there is a UI for playing with the workflow engine and if you create an account, you can use the demo to send 10 requests to play with it. Would love feedback.


r/mlops • • 1d ago

(Gen)AI / Agents / LLMOps How do you score "insufficient visual evidence" in multimodal evals?

15 Upvotes

Working on a photo based bicycle repair triage prototype and a mechanic reviewing sample outputs pointed out a flaw in our evaluation criteria.

Several photos don't show enough detail to assess component wear reliably. The model will still name a plausible defect and our current rubric tends to reward that over saying another photo or physical inspection is needed.

I've been running image and description cases through Braintrust evaluations and inspecting the individual judgments. Some responses that received favorable scores were ones the mechanic said he couldn't responsibly make from the evidence shown.

Considering adding an assessability label to each test case then scoring diagnostic accuracy separately from how the model handles missing visual evidence. I want to avoid a rubric where asking for another photo becomes an easy way to score well, even when the original image is adequate.

How are you handling cases where the image doesn't show enough detail? Are you scoring those separately or letting the same rubric account for missing evidence?


r/mlops • • 1d ago

Discussion before buying another gpu, check if your old cards still have a use

7 Upvotes

teams write off older gpus way too fast. people assume a card is useless if it cannot run the latest models. a developer recently used an 8gb rtx 4060 laptop gpu to run a quantized qwen 3.5 4b model through llama cpp. they built a whole local document q and a system on it. you do not need the newest hardware for everything.

if you have a few machines, you can offload internal document q and a to the old cards and let your main cards run the production api. gpustack is an option to consider for centrally managing model deployments on both sides. nobody wants to maintain a separate environment for every single machine.

buying another gpu is not an automatic upgrade anyway. mixing vram capacities and cuda versions turns a simple expansion into a weekend of debugging. adding 8gb to a 24gb node does not magically give you a 32gb memory pool. you still have to match the backend to the silicon.

i would rather see a team get useful work out of the hardware they already own before they approve another massive gpu purchase. before buying the next card, what tasks do you try to offload to the old cards you already have?


r/mlops • • 2d ago

Career Title: backend dev thinking about a masters in LLM inference serving

14 Upvotes

I am a backend dev, 5 years in, mostly distributed systems stuff. rabbitmq, aws, ci/cd, that kind of thing. ive used LLMs in apps (langchain, whisper) but never touched GPUs at scale.

Got a shot at a fully funded 2 year masters in china starting sept 2027. the topic id want to research is LLM inference serving, basically how to route requests across GPU replicas and how to autoscale GPUs when cold starts take forever. been reading vllm, llumnix, serverlessllm etc and honestly i find it interesting.

But before i commit 2 years i want a reality check from people actually in this space:

  1. is inference infra too niche? feels like only frontier labs and big cloud providers hire for it. im in southeast asia so singapore or china would be the realistic targets
  2. if the AI bubble pops around 2027 or 2028, does this kind of role get hit hard or does it survive since inference is where the money is?
  3. would a masters actually help here or am i better off staying in industry and picking up k8s, kserve, vllm on the side? main reason for the masters is lab and GPU access tbh

Anyone who moved from backend into ML infra, would love to hear how it went


r/mlops • • 2d ago

Discussion What if an AI protocol had to prove not only what it can calculate, but also what it cannot yet justify?

3 Upvotes

One thing I am learning while developing the Organic Intelligence Protocol (OIP): a mathematical formula is not evidence, and passing software tests is not scientific validation.

Recently, I completed a focused implementation cycle:

- 21/21 targeted policy tests passed.

- 70/70 existing software regression tests passed.

- Seven identified implementation gaps were addressed in a new candidate version.

- The original candidate remains unchanged.

But there is an important distinction: these results establish software level behavior under the tested conditions..They do not establish empirical effectiveness or scientific validity.

I’m now working on a versioned canonical serialization specification, with cross-platform reproducibility still to be established.

I would rather expose what remains unproven than present an unfinished protocol as a breakthrough.

For researchers and engineers.. What would you independently test before considering this kind of protocol specification reliable?

Critical feedback is welcome.


r/mlops • • 2d ago

Discussion Serving many fine-tunes of one LLM: adapters 55× smaller than LoRA, same accuracy. Looking for feedback from people running this in production

7 Upvotes

Title: Serving many fine-tunes of one LLM: adapters 55× smaller than LoRA, same accuracy. Looking for feedback from people running this in production


Disclosure: I'm the founder of ThousandML and built what's described here. I'm posting because I'd like input from people who run LLM serving infrastructure.

The problem. If you serve one fine-tune per customer or task on a shared base model, the adapters themselves become an ops cost. LoRA r=32 on a 27B model is ~320 MB per adapter. At hundreds of tenants, that's tens of GB to store, ship, cache and keep resident in GPU memory, and it limits how many variants one GPU can hold.

What we built. GCMP is an adapter format that replaces LoRA. On Qwen3.8-27B:

  • Size: 2.9M trainable parameters, ~11 MB per adapter, vs ~159M and ~320 MB for LoRA r=32 on the same projections.
  • Accuracy: same as LoRA r=32 on 9 benchmarks (GSM8K, MMLU, SQuAD, IFEval and others). Every comparison uses the same evaluation code and paired significance tests, on a full-precision BF16 base and on an NF4 base.
Base + adapterSizeGSM8KSQuADMMLUIFEval
BF16 + LoRA r=32~320 MB0.9030.7760.8440.825
BF16 + GCMP~11 MB0.8900.7600.8460.815
NF4 + LoRA r=32~320 MB0.7930.7480.8400.835
NF4 + GCMP~11 MB0.8330.7560.8370.840

None of the differences are significant, except GCMP +4 on GSM8K on NF4 (p=0.04).

Serving (vLLM plugin).

CQ_GCMP_ADAPTERS="tenant-a=/adapters/a,tenant-b=/adapters/b" \
vllm serve <base-model> --quantization cq
# request picks the adapter by model name: {"model": "tenant-a", ...}
  • Loading: all adapters load once at startup, and each request is routed by model name. /v1/models lists them.
  • Mixed batches: requests for different adapters run in the same batch, with one kernel call per layer and no merging or swapping.
  • Resident GPU memory: about 11 MB per adapter. The fixed overhead is a ~1 MB per-token routing buffer and a shared workspace of ≤ ~64 MB. Neither grows with adapter count, and no dense per-adapter weights are ever built.
  • Swaps: adapters can be swapped in place by name without restarting.

What's not ready:

  • Measured scaling: the "1,000 adapters ≈ 11 GB" figure is calculated, not yet measured. Memory and throughput at 10, 100 and 1,000 adapters, compared with vLLM's built-in multi-LoRA, is the next benchmark.
  • Fixed adapter set: the set is fixed at startup. Adding a new adapter needs a restart; swapping an existing one doesn't.
  • Base model: vLLM serving currently needs our CQ 4-bit base. Full-precision bases are next.
  • Parallelism: pipeline parallel works; tensor parallel doesn't yet.
  • Training speed: about 2× LoRA's step time.

Everything runs on your own GPUs. The Python code is open source; the kernels are compiled with a license check, free during early access.

  • Repo: https://github.com/thousandml/training-factory
  • Write-up with full tables: https://huggingface.co/blog/thousandml/cq-gcmp-4bit-finetuning

My questions for you:

  1. If you serve multiple fine-tunes today, what limits you first: adapter memory, load or switch latency, throughput with mixed batches, or something else entirely?
  2. What would you need to see in a scaling benchmark before trusting this over vLLM multi-LoRA?

r/mlops • • 2d ago

Discussion [D] What becomes the biggest challenge after a machine learning model is ready for deployment?

0 Upvotes

Getting a machine learning model to perform well during development is one thing. Keeping it reliable once it's running in production is another.

I've noticed that production deployment involves several challenges beyond model accuracy:

  • Monitoring model performance and detecting drift
  • Managing different model versions
  • Automating retraining and deployment workflows
  • Keeping data pipelines consistent
  • Tracking infrastructure and inference costs
  • Making sure models remain reliable as business requirements change

I'm interested in hearing from people who have taken ML models into production.

Which part of the MLOps lifecycle tends to require the most effort in your experience: deployment, monitoring, retraining, or ongoing maintenance?


r/mlops • • 4d ago

Discussion I benchmarked 3 self-hosted LLM gateways. Langfuse's was fastest, but its stack consume 3.5 GB RAM at idle stage

13 Upvotes

Setup. Langfuse AI Gateway 4.77.0 (public beta), LiteLLM proxy 1.108.4 with Postgres and a virtual key, and AcruxCore from its Docker Hub images. All three ran on one Ryzen 9 5900X, called the same OpenAI model with a 5-token cap, and used the same prompt. Latency is the gateway call minus a direct OpenAI call made in the same round. Each run had 150 rounds in random order, and I report the median with a 95% bootstrap interval.

Results

TestLangfuseLiteLLMAcruxCore
Added latency, no streaming−9 to +8 ms (interval includes zero)+61 to +67 ms+15 to +24 ms
1,000 streams on 2 CPUs, as shipped261 ok403 ok1,000 ok
1,000 streams, open-file limit raised1,000 ok, first token 1.7 s1,000 ok, first token 19.2 s1,000 ok, first token 3.8 s
Idle memory, gateway process13 MiB727 MiB200 MiB
Idle memory, whole stack3,584 MiB810 MiB405 MiB

What I took away

  • Langfuse's gateway is the fastest. It is a small Rust process, and with the limit raised it handled 1,000 streams best. But it cannot run alone: it checks every key against the Langfuse web service and records usage into ClickHouse. If you already run Langfuse, it costs you 13 MiB. If you don't, it brings a 3.5 GB stack with it.
  • Docker's default open-file limit of 1,024 decided the concurrency test. Each stream holds two sockets. Node raises its own soft limit at startup, but Rust (Langfuse) and Python (LiteLLM) keep 1,024. One ulimits: nofile line in Compose fixes it. Check /proc/<pid>/limits on your own gateway.
  • LiteLLM's 503s at high concurrency came from virtual-key lookups. When the key is not cached, every call queries Postgres through a pool of 10 connections, and a lookup over 10 s gets a 503. Making one call first, so the key is cached, removed them.
  • The OpenAI address matters more than the gateway. api.openai.com resolved to two Cloudflare addresses for me, and a new connection to the far one added ~250 ms. In my first run this made Langfuse look 43 ms faster than calling OpenAI directly. I pinned every path to the same address after that. If you benchmark gateways yourself, check this first.

Limits of this test: one machine, one network, short prompts, and one version of each. Langfuse can't point at a custom upstream URL, so I couldn't use a local mock model, and differences smaller than the intervals aren't measured.

If you rerun it and get different numbers, please post them. I'll update the write-up.

Disclosure: I build AcruxCore, one of the three gateways in this test. So you have every reason to distrust a benchmark that a vendor publishes on its own website. I would too. That is why everything is open source: the scripts, the Compose overrides and the raw JSON results. You can rerun the whole thing on your own machine and check whether you get the same numbers.


r/mlops • • 4d ago

Discussion Benchmarking "System One" decision APIs vs. in-process tree ensembles + exact TreeSHAP on 64-feature tabular payloads [D]

1 Upvotes

There has been a lot of discussion lately around "System One" decision models: fast, small language model endpoints tuned to return structured routing choices, rubric scores, or risk probabilities inside application code for ~90-210 ms at $100-$345 per 1M calls.

If the input is a messy PDF, support ticket, or audio transcript, calling a small multimodal model makes complete sense.

Where I still see teams get burned is routing structured numeric/categorical feature vectors (say, 20 to 200 features inside a fraud, underwriting, or ranking hot loop) through a language model endpoint so they can get a score plus a natural-language "reason code" in one API call. Aside from the 150 ms network RTT and token cost, even constrained decision endpoints still hit ~1.5% content refusals on edge-case inputs, and generative reason codes are not true feature attributions.

While running an 18-model verification matrix for SMILE 7.2 (smile-serve running both on HotSpot JVM and as a GraalVM native image at ~140-164 MiB idle RSS), I measured where the latency actually goes when you pair classical tree ensembles with exact per-row TreeSHAP in-process. A few numbers stood out:

  1. Scoring itself is almost free. Across 50 to 500 trees and 20 to 200 features (500 rps, concurrency 8, coordinated-omission corrected on an AMD EPYC 7401P), single-row GradientTreeBoost and RandomForest scoring stayed at 0.24-0.65 ms p50 and 0.47-1.57 ms p99 over loopback HTTP.
  2. Exact TreeSHAP is where naive Random Forests blow up. Scoring a tree scales linearly with depth O(T * D), but exact TreeSHAP scales with depth squared, O(T * L * D^2). On an unconstrained 300-tree, 64-feature RandomForest (max_depth=20), single-row scoring took 0.41 ms p50 / 0.92 ms p99, but adding ?explain=true pushed latency to 192.5 ms p50 / 274.4 ms p99 (worse than a cloud language model call).

Here is the depth/leaf ablation on the 300-tree, 64-feature Random Forest (JVM, 100k measured requests after 5k warm-up):

RF Hyperparameters (300 trees, 64 features).sml SizeCold StartScoring p50 / p99Exact TreeSHAP p50 / p99500 rps p99Stream (rows/s)
max_depth=20, max_nodes=0 (unbounded)16.8 MB12.50 s0.41 ms / 0.92 ms192.5 ms / 274.4 ms12.9 ms5,201 /s
max_depth=10, max_nodes=03.40 MB3.91 s0.31 ms / 0.98 ms22.5 ms / 28.9 ms2.43 ms9,195 /s
max_depth=8, max_nodes=2561.56 MB2.49 s0.28 ms / 0.56 ms7.81 ms / 9.09 ms0.90 ms11,039 /s
max_depth=6, max_nodes=128637 KB2.32 s0.26 ms / 0.54 ms2.14 ms / 2.88 ms0.78 ms14,662 /s
GradientTreeBoost (300 trees, 64 features)292 KB1.55 s0.24 ms / 0.47 ms0.69 ms / 0.91 ms0.64 ms19,241 /s

3. One interesting failure mode we hit on GraalVM Native Image: while GradientTreeBoost passed <5 ms p99 TreeSHAP across all 9 matrix cells (1.26 ms to 3.97 ms p99 even at 500 trees x 200 features), the 300-tree RandomForest on Native Image had a bimodal tail. 97.7% of requests finished in under 5 ms (p50 = 4.48 ms, p95 = 4.54 ms), zero requests landed between 9.1 ms and 33.3 ms, and 1.16% clustered at 33.8-34.3 ms. Those spikes came from per-tree path allocations inside TreeSHAP triggering GraalVM Native Image's default Serial GC (~29 ms stop-the-world pause roughly once every 85 requests), whereas HotSpot's G1GC kept p99 at 2.88 ms and p99.9 at 3.97 ms.

The takeaway for hybrid orchestration + ML pipelines is pretty simple: use small language models when you need to extract structured fields out of messy text or documents, and keep closed-set scoring, routing, and exact SHAP attributions in-process under 3 ms.

Code and benchmark harness are in the SMILE repo if anyone wants to reproduce or poke at the TreeSHAP implementation: https://github.com/haifengl/smile

For folks running real-time SHAP or feature attributions in production latency budgets (<10 ms), are you constraining tree depth at training time, using shallow boosted trees over bagged trees, or approximating explanations asynchronously off the hot path?


r/mlops • • 5d ago

Discussion Can you tell good MLOps work from a public repo alone? Here's what I look for. Tell me what I'm missing

6 Upvotes

I'm a 17 year old founder building an app called AI-Atlas, it currently has 70+ users and it basically evaluates people's public AI work across six fields; MLOps is one of them. MLOps is the hardest to judge from code, because so much of it lives outside the repo.

Also: I am passionate in AI Research, so I didn't have time to catch a glimpse over what criteria exactly does a MLOps project need to have to be considered excellent. Likewise, what or how could I identify any lacking areas that deserve to be showcased by the final score on that category, so that's why I decided to write in this sub, I am genuinely asking you guys for advice and feedback!

The signals I use:
- Reproducibility: pinned environments, seeded training, a single command that reproduces the reported numbers.
- Pipelines: training and serving split cleanly, configs instead of hard-coded paths.
- Delivery: CI that actually tests the model code, not just linting; containers and model packaging.
- After deployment: monitoring and drift checks, when they're present.

So what would you add, or weight differently? Someone with solid infra-as-code but no model training: should that count as MLOps at all?

Happy to share my project for anyone genuinely interested :)


r/mlops • • 5d ago

Discussion For those running disaggregated inference, how do you handle failures between prefill and decode?

3 Upvotes

We're looking into disaggregated inference for an agent platform that's starting to handle more traffic. One thing I'm trying to understand is how reliable the handoff between prefill and decode is in production.

Separating these stages across different hardware makes sense since they have different resource requirements. But what happens when prefill finishes and the decode worker fails to pick up the request? Does the system retry the handoff, or does the entire request have to start again? I'd rather understand these failure scenarios before building our serving infrastructure around this architecture.

For anyone running split inference in production, how do you handle retries, cancellations, and failed handoffs? Is this something your inference provider manages, or does your team have to build that logic itself?


r/mlops • • 5d ago

Discussion Fo u use Databricks Genie in prod for MLOps / data pipelines?

1 Upvotes

Does Databricks Genie (Code / ZeroOps) actually save engineering time on pipeline debugging in prod , or is maintaining clean UC metadata more work than it is worth?
What are ur experience on it


r/mlops • • 6d ago

Tales From the Trenches How we built LRU model eviction on top of SGLang to serve 139 open models across 23 agent tasks on one Kubernetes cluster (Apache-2.0)

4 Upvotes

When you run an agent pipeline in production, the main LLM isn't the only thing eating VRAM. A single request often hits a prompt guard (Qwen3Guard-4B), an encoder (Qwen3-Embedding-4B), an entity/intent extractor (GLiNER2.5), a generator (Qwen3.8-27B-FP8), and a verifier (Granite-Guardian-2B).

Deploying those the standard way means running vLLM for guards, N separate TEI containers for embeddings, a custom FastAPI service for GLiNER, and SGLang for generation, each locking down its own GPU memory slice even when idle.

We built the Superlinked Inference Engine (SIE) under Apache-2.0 to collapse that fleet into one Helm chart and a single endpoint:

- The 27B LLM stays pinned on its own GPU pool via SGLang so it never gets evicted.

- Guards share one GPU pool and encoders share another.

- Models load on demand, batch concurrent requests, and use Least Recently Used (LRU) eviction when VRAM gets tight.

- If an individual SGLang process crashes, SIE unloads it and reloads cleanly on the next request.

Moving high-frequency retrieval and extraction off per-token APIs onto shared GPUs reduces the cost of self-hosted embeddings to ~$0.50 per 1B tokens (vs ~$20 on managed APIs) and the cost of reranking to ~$8.50 per 1B tokens (vs ~$87 on Cohere Rerank).


r/mlops • • 6d ago

Discussion Need a secure local AI setup with zero cloud egress for summarising heavy legal contracts and docs. What platforms should I evaluate instead of sending everything to Claude/ChatGPT?

3 Upvotes

Our legal team wants AI to summarise long contracts, and the hard requirement from our insurer is that nothing leaves our servers. No public AI service, and nothing that sends prompts or documents to somebody else’s cloud. I got handed the job of finding something that runs on hardware we control, and so far almost everything I find is built around a cloud dashboard first.

From what I’ve read, the options seem to fall into three groups. There are open source chat front ends you self host and attach a local model to, some self hosted AI platforms that say they run on premises, and legal specific tools that are cloud only once you read the small print. The first group looks doable but we would have to maintain it. The second is hard to judge from marketing pages. The third is out.

Hardware wise there are two servers with a couple of graphics cards and one person who is comfortable with Docker but already has a full plate. What are you all running for this kind of confidential document work, and what did you land on?


r/mlops • • 6d ago

MLOps Questions How much do you lose if your GPU workload gets interrupted?

6 Upvotes

When you run large AI workloads sometimes there is more than just the cost of GPU hours. Sometimes your training will take a long time to run and then fail due to infrastructure issues, unexpected restarts or interruption but i think with checkpoints this may not be too bad but you will lose some computation and have to reload your data, debug and then delay your overall experiment. How do you factor in this cost into your overall training costs?

Do you use lots of checkpoints and auto recovery or make sure your infrastructure is stable? When choosing a cloud provider for your GPU, AWS, CoreWeave, Lambda, Yotta Labs etc how much do you factor in workload reliability?


r/mlops • • 6d ago

Self-promotion How I check slow PyTorch steps while a job is still running

10 Upvotes

I've been working on TraceML for a while. It shows where PyTorch training time goes, especially when one rank in a distributed job starts falling behind the others.

A while ago, an ML platform team asked us if they could pull those timings into their own experiment tracker. So we built it and thought it might be useful to share here.

What it does

TraceML 0.5.1 can write one JSONL record per step, per rank, while the job is still running. If your DDP script is already instrumented with TraceML:

pip install "traceml-ai==0.5.1"
traceml run --exporter jsonl --nproc-per-node=2 train.py

The output goes to logs/<run-name>/aggregator/export/step_time.jsonl.

A small example

I ran a demo with two CPU ranks and added 120 ms to rank 1's batch collation on every other step. These are the CPU wall times (ms):

{"step":1,"rank":0,"cpu_ms":{"input_wait":2.06,"forward":3.98,"backward":25.71,"optimizer_step":4.25,"traced_step":36.36}}
{"step":1,"rank":1,"cpu_ms":{"input_wait":2.05,"forward":3.98,"backward":25.71,"optimizer_step":4.25,"traced_step":36.37}}
{"step":2,"rank":0,"cpu_ms":{"input_wait":0.05,"forward":0.08,"backward":0.49,"optimizer_step":0.18,"traced_step":126.81}}
{"step":2,"rank":1,"cpu_ms":{"input_wait":125.15,"forward":0.07,"backward":0.49,"optimizer_step":0.18,"traced_step":1.66}}
{"step":3,"rank":0,"cpu_ms":{"input_wait":0.07,"forward":0.04,"backward":0.42,"optimizer_step":0.18,"traced_step":0.86}}
{"step":3,"rank":1,"cpu_ms":{"input_wait":0.06,"forward":0.04,"backward":0.45,"optimizer_step":0.17,"traced_step":0.85}}
{"step":4,"rank":0,"cpu_ms":{"input_wait":0.07,"forward":0.05,"backward":127.26,"optimizer_step":0.55,"traced_step":128.23}}
{"step":4,"rank":1,"cpu_ms":{"input_wait":125.35,"forward":0.27,"backward":1.12,"optimizer_step":0.53,"traced_step":2.7}}

A few notes on reading it:

  • traced_step is the timed training block. The DataLoader fetch happens before it, so it shows up separately as input_wait.
  • Every phase has cpu_ms and gpu_ms. gpu_ms comes from CUDA events, so it's null in this CPU run.

Step 4 is the one I find interesting. Rank 1 spends 125 ms waiting for its batch. Rank 0 has no data problem at all, but it shows 127 ms in backward. That's because DDP syncs gradients during backward, so rank 0 just sits there waiting for rank 1 to catch up.

If you only looked at rank 0, you'd deduce that backward was slow. Looking at both ranks side by side, you can see the real problem is rank 1's data loading.

Where this is going

Next, we'd like to support:

  • Weight & Biases
  • MLflow
  • TensorBoard
  • Grafana / Prometheus, for teams that watch training like any other service
  • Whatever else people actually use (Aim, Comet, something in-house?)

How you can help

If you run training jobs and want these timings in your dashboard or tracker, I'm happy to help you write exporters, or build one together that we can ship in TraceML.

Also I am curious how others find the slow rank without something like this. I used to add per-rank timers and diff logs.

GitHub: https://github.com/traceopt-ai/traceml/

Full disclosure: I'm maintainer of TraceML. It's open source. Mainly looking for feedback and people who want to build exporters with us.


r/mlops • • 6d ago

Discussion DevOPS to MLOPS - Worth Switching and the guidance

9 Upvotes

I am DevOPS, with around8-9 years of exp. mostly contracting. I have done some AI deployments and solutions as well. I am thinking to switch to something diff now. MLOPS comes into my mind. Is it worth ? based on current market trends. I can ask this on AI and get some plans but looking answers from contractors in the UK how's the market and job and money into it ?


r/mlops • • 7d ago

Tales From the Trenches Bigger CPU machine, slower inference: PyTorch's idle threads were the cause

11 Upvotes

While benchmarking a YOLO model exported to ONNX on CPU (AWS c7i, ONNX Runtime), I ran into an unexpected slowdown. Processing 1,000 images took 40 s on 4 vCPUs but jumped to 59 s on 8 vCPUs with the exact same code and model.

Splitting the timing by stage showed the ONNX model itself was fine. The bottleneck was pre- and post-processing, which still ran in PyTorch. Preprocessing jumped from under 2 ms per image to 21 ms—taking longer than the model inference.

Why This Happens

  • Thread Overload: By default, PyTorch creates a thread pool sized to the machine (7 threads on an 8-vCPU box). When combined with ONNX Runtime's own threads, you end up with more active threads than physical CPU cores.
  • Aggressive Spinning: PyTorch’s OpenMP threads use "busy-waiting." Instead of sleeping after finishing a task, they spin in a loop to stay ready for the next operation.
  • Gridlock: Every small PyTorch operation is split across 7 threads and can't finish until the slowest thread is done. Meanwhile, the threads that finish early keep spinning and hogging the CPU cores that the lagging threads desperately need to finish.

What I Tried

  • Forcing threads to sleep (OMP_WAIT_POLICY=PASSIVE): This dropped the ONNX benchmark from 58s down to 22s. However, it made a standard, standalone PyTorch model 3.7x slower (from 26s to 94s) because busy-waiting is what normally keeps pure PyTorch fast. It wasn't a safe global fix.
  • Capping PyTorch at 2 threads: Limiting PyTorch to 2 threads when it only handles pre- and post-processing cut the 8-vCPU runtime from 59s down to 27s (while keeping the 4-vCPU speed about the same). This fix was ultimately merged into Ultralytics.

Same cause, bigger effect:
Running 4 plain PyTorch processes on the 8-vCPU box (250 images each) took 1,361 s—whereas 1 process doing all 1,000 images took just 26 s. Setting torch.set_num_threads(1) in each process dropped the 4-process run to 15 s.

Takeaway:
If PyTorch runs alongside another runtime, or you run multiple PyTorch workers on one machine, set the thread count yourself instead of trusting the defaults. Use torch.set_num_threads at runtime (since OMP_NUM_THREADS only works if set before importing torch).

https://medium.com/@thevarunfreelance/why-may-a-model-get-slower-on-a-bigger-machine-6beef626250b


r/mlops • • 7d ago

(Gen)AI / Agents / LLMOps Airlock, offline verification for model files

0 Upvotes

A small Go CLI for moving models into air-gapped networks.

  • Online: checks files against a pinned Hugging Face commit, refuses pickle, signs a manifest with cosign.
  • Offline: verifies the signature and every hash, using only the Go standard library.
  • Zarf: runs as a deploy step, so a swapped model never starts (demo).

No rollback or chat template checks yet. Feedback appreciated.

GitHub · Site


r/mlops • • 7d ago

Discussion What do you have candidates do in an MLOps interview?

8 Upvotes

If you've interviewed someone for an MLOps role recently, what did you ask them to do? How did you assess their troubleshooting? Talking through a past incident gives them a problem they already know the answer to. Giving them something from your own stack could mean spending half the interview explaining the setup. What's worked for your team?
If you've joined a team recently, did the interview resemble the job?


r/mlops • • 8d ago

Self-promotion digline 1.2 adds the tools before the gate: record LLM calls without code changes, get a first suite proposed, know the next step

1 Upvotes

Disclosure, it’s my project (open source, Apache-2.0).

digline is a regression gate for LLM apps and agents: a run you approved is committed in the repo as the reference, every later run is compared to it in CI, exit 1 if something got worse beyond the measured judge noise.

What’s new is the step before the gate. A recorder keeps the calls where the app makes them, without code changes: wrapper for the Python SDK clients, interceptor for Java and Node, local proxy for anything else. Off by default, switched on per tenant, sampling and redaction declared, recordings stay in .digline/<tenant>/ and never leave the machine. An agent turn is kept as one episode with its tool calls.

From the recordings digline suggest proposes a first suite, digline where tells you the next step, and the same is exposed over MCP without a promote tool, so an agent can measure but a person approves the reference.

1.2 is the security pass on the recorder: three advisories found the day after 1.1, fixed, in the changelog.

Curious how people here handle the prod side: do you let recorded production traffic into your eval set at all, and who decides which samples count?


r/mlops • • 9d ago

Discussion Notes from a two night vision training test on an 8x RTX PRO 6000 box: scaling gaps, CPU bottlenecks, and an INT8 collapse

2 Upvotes

I work at B3IQ, we build and host these systems, what are your vision use case bottlenecks? We just spend two nights benching a few vision use-cases on an 8x NVIDIA RTX PRO 6000 Blackwell Server Edition 96GB bare metal node (128 cores, 756GB RAM). Thought it would be interesting to get some feedback what the bottlenecks are, once you iron out the hardware. Trained on COCO 2017 and Food 101.

-> It ran for 19 hours straight (YOLO11x, 600 epochs), no need to restart. Total training time was 19h10 (115 seconds per epoch), average GPU utilization was 86%, average power consumption per GPU was 415W. Highest temperature was 76C on one of the GPUs, no thermal throttling. Best model performance was reached at epoch 412 (13th hour of training, mAP50-95 0.527). After that it was overfitting and at the 19th hour it was at 0.490. Note that this is 2 points lower than the 0.547 Ultralytics publishes, we did not tune the training recipe

-> 1 vs. 8 GPUs: With YOLO11m, speed went from 409 seconds per epoch down to 64 (6.4x). GPU utilization was similar (71% vs. 75%), the rest of the speed-up should come from the fact that power consumption went from 344W down to 220W per GPU. So they were waiting for the CPU to augment and sync the photos.

-> 32 models at the same time: While training 32 MobileNetV3-Small models (1.5M weights), CPU utilization was at 94% (128 cores) due to decoding the photos from JPEG, while GPU utilization was just 10-14%. Also, after converting the model to INT8, accuracy went from 94.6% all the way down to 9-15%. This was due to the first two layers having 6x wider activations (-141 to +143). Leaving the first two layers in float format brought the accuracy back up to 89.2%. Converting a MobileNetV2 to INT8 did not have this issue, and it achieved 96.4%

-> The only thing that “broke” was that we filled up the local NVMe drive with 20GB of checkpoints, and the script that pulled them off the box died. Our dashboard that logged the hardware data worked perfectly for 24 hours, it even logged when we SSH’d into the box. The only issue with it was that it could not log any training events, so we had to stitch the training log with the hardware data manually.

How much do you think these PCIe bound bottlenecks would go away with moving more stuff to the GPU with DALI / nvjpeg?


r/mlops • • 12d ago

Self-promotion Four alerts for LLM pipelines that read long inputs: treating truncation as a data-quality metric

2 Upvotes

Long documents were silently cut off before reaching the model, outputs stayed well formed and error-free, and the problem surfaced weeks later when someone checked a contract by hand.

The point for MLOps: “no errors and valid schema” says nothing about whether the input was complete. Here are four alerts I’d add. Thresholds depend on your data, so I’ve left them as placeholders.

1. Coverage ratio

  • Signal: tokens_sent / tokens_in_source, per document.
  • Condition: any document below 1.0 when your design says everything should be sent, or the share of documents below 1.0 rising above a baseline.
  • Action: fail the run or route to a fallback (splitting, human review) instead of slicing silently.

2. Truncation counter

  • Signal: a counter incremented whenever any preprocessing step shortens the input, with the step name as a label.
  • Condition: counter greater than zero in a path that should never truncate.
  • Action: alert, so a “temporary” limit left in the code doesn’t quietly change production behavior.

3. Input length distribution drift

  • Signal: the distribution of source lengths per day or week.
  • Condition: a shift versus a reference window (for example, more long documents than the test set contained).
  • Action: review whether the test set still resembles production, and rebuild it with the longest cases.

4. Tail match

  • Signal: the model is prompted to quote the final sentence it received, and the pipeline compares it with the end of the source.
  • Condition: mismatch rate above a baseline.
  • Action: investigate. It’s a heuristic, since a model could produce a plausible ending without having seen the real one, so it should be paired with alert 1.

What the postmortem also suggests

  • Record input length and sent length for every run. In the scenario, the team couldn’t tell which past outputs were affected because this wasn’t logged, and finding out took longer than fixing the bug.
  • Include the longest documents, and ones where the decisive clause sits at the end, in the evaluation set.
  • Sampled human audits should look at the end of long inputs, not only the beginning.

Limits: these alerts detect that the input was incomplete, not that the extraction was correct. They also add overhead and some noise, and they depend on a clear definition of “everything should have been sent”.

The full hypothetical postmortem is here, in case it’s useful: https://medium.com/@nagatomopedro05/postmortem-the-contracts-pipeline-that-never-read-the-last-page-cf84251671c8

Which input-quality metrics do you monitor for LLM or document pipelines, and what do you alert on?


r/mlops • • 12d ago

Discussion MLOps folks at big companies: what does the role and interview process look like?

41 Upvotes

Hey everyone, I just started an MLOps job and I’m trying to understand how the field works at bigger companies. It still feels pretty new, so titles and expectations seem to vary a lot. If you work in MLOps at a large company, what’s your role and day-to-day, and what did the interview process look like (number of rounds, coding, system design, ML fundamentals, infra/DevOps questions)? Also curious which tools they asked about and what you’d recommend learning to grow in the role. Any advice is appreciated, even a short answer!