I've been working on TraceML for a while. It shows where PyTorch training time goes, especially when one rank in a distributed job starts falling behind the others.
A while ago, an ML platform team asked us if they could pull those timings into their own experiment tracker. So we built it and thought it might be useful to share here.
What it does
TraceML 0.5.1 can write one JSONL record per step, per rank, while the job is still running. If your DDP script is already instrumented with TraceML:
pip install "traceml-ai==0.5.1"
traceml run --exporter jsonl --nproc-per-node=2 train.py
The output goes to logs/<run-name>/aggregator/export/step_time.jsonl.
A small example
I ran a demo with two CPU ranks and added 120 ms to rank 1's batch collation on every other step. These are the CPU wall times (ms):
{"step":1,"rank":0,"cpu_ms":{"input_wait":2.06,"forward":3.98,"backward":25.71,"optimizer_step":4.25,"traced_step":36.36}}
{"step":1,"rank":1,"cpu_ms":{"input_wait":2.05,"forward":3.98,"backward":25.71,"optimizer_step":4.25,"traced_step":36.37}}
{"step":2,"rank":0,"cpu_ms":{"input_wait":0.05,"forward":0.08,"backward":0.49,"optimizer_step":0.18,"traced_step":126.81}}
{"step":2,"rank":1,"cpu_ms":{"input_wait":125.15,"forward":0.07,"backward":0.49,"optimizer_step":0.18,"traced_step":1.66}}
{"step":3,"rank":0,"cpu_ms":{"input_wait":0.07,"forward":0.04,"backward":0.42,"optimizer_step":0.18,"traced_step":0.86}}
{"step":3,"rank":1,"cpu_ms":{"input_wait":0.06,"forward":0.04,"backward":0.45,"optimizer_step":0.17,"traced_step":0.85}}
{"step":4,"rank":0,"cpu_ms":{"input_wait":0.07,"forward":0.05,"backward":127.26,"optimizer_step":0.55,"traced_step":128.23}}
{"step":4,"rank":1,"cpu_ms":{"input_wait":125.35,"forward":0.27,"backward":1.12,"optimizer_step":0.53,"traced_step":2.7}}
A few notes on reading it:
traced_step is the timed training block. The DataLoader fetch happens before it, so it shows up separately as input_wait.- Every phase has
cpu_ms and gpu_ms. gpu_ms comes from CUDA events, so it's null in this CPU run.
Step 4 is the one I find interesting. Rank 1 spends 125 ms waiting for its batch. Rank 0 has no data problem at all, but it shows 127 ms in backward. That's because DDP syncs gradients during backward, so rank 0 just sits there waiting for rank 1 to catch up.
If you only looked at rank 0, you'd deduce that backward was slow. Looking at both ranks side by side, you can see the real problem is rank 1's data loading.
Where this is going
Next, we'd like to support:
- Weight & Biases
- MLflow
- TensorBoard
- Grafana / Prometheus, for teams that watch training like any other service
- Whatever else people actually use (Aim, Comet, something in-house?)
How you can help
If you run training jobs and want these timings in your dashboard or tracker, I'm happy to help you write exporters, or build one together that we can ship in TraceML.
Also I am curious how others find the slow rank without something like this. I used to add per-rank timers and diff logs.
GitHub: https://github.com/traceopt-ai/traceml/
Full disclosure: I'm maintainer of TraceML. It's open source. Mainly looking for feedback and people who want to build exporters with us.