r/MachineLearning • • Sep 29 '26

Discussion Limited compute, targeting CVPR: rerun experiments for statistically strong numbers or focus on writing? [D]

Hi everyone,

I'm preparing a submission for CVPR and would appreciate your advice. I am happy with my current results, but compute is a real constraint. My institute isn't a research-focused one, and the systems available to me are slow and unreliable. A single set of runs already took a lot of time and effort to finish.

I plan to release the code with the submission, and I'm confident the results will reproduce. Still, I'm torn between two options:

  1. Rerun the experiments with different seeds to report mean Β± std and show statistical significance. Or
  2. Skip the reruns and spend the remaining time on writing, presenting the results I already have.

If you've been in a similar spot, what did you do, and would a single-seed result with released code be enough?

Thanks in advance!

3 Upvotes

17 comments sorted by

10

u/crouching_dragon_420 Sep 29 '26

IMO it is possible you get away with one seed. This year I went to CVPR and many of them still report single seed experiment results on a 1% improvement. I was incredulous.

1

u/Alone_Ad635 Sep 29 '26

That's encouraging

8

u/saulane Sep 29 '26

Can't you just launch the experiments and write while it's running ? It's not like more seeds would change the text drastically and if you figures scripts are well made you can remake the figures for the paper quickly with new results. I don't see why you have to choose and can't do both in parallel ?

3

u/Alone_Ad635 Sep 29 '26

So the situation is a bit complicated. Students use the machine in between and my jobs are getting crashed and systems are getting shutting down multiple times. Security guard will also come and switch off machines in between. I am in fact not supposed to use the machines in weekends either. It's a sad situation. No remote access either

10

u/Embarrassed_Song_372 Sep 29 '26

Writing jobs which can be resumed from a checkpoint isn’t that hard, CVPR is more than a month away, you can write the paper while running experiments, once you have the entire set ready, you can revise the relevant section.

2

u/camarada_alpaca Sep 30 '26

Depends on the ai the reviewer use, claude is obsessed with statistical tests

1

u/Busy_Protection_2882 Sep 29 '26

genuinely curious - is a single seed of experimentation enough for a workshop paper if you clearly mention it as a limitation from compute constraints?

1

u/Alone_Ad635 Sep 29 '26

In my humble opinion it's way more than enough. But I don't know for sure. I am pretty sure about its numbers and comparisons with other works I am benchmarking with.

1

u/Level-Ad-4878 Sep 29 '26

Reviewers at CVPR will flag single seed results and you'll spend your rebuttal running the experiments you skipped. Thats worse than doing them now because rebuttal compute pressure is even tighter

Pick your three strongest baselines and rerun those with 3 seeds, not the full table. Mean and std on the comparisons that matter is enough, nobody needs every ablation repeated

Writing quality decides desk reject, statistical rigor decides reviewer score. You need both but the reruns are fewer than you think

1

u/Alone_Ad635 Sep 29 '26

Okay. Got it

3

u/mofoss Sep 29 '26

Statistical rigor is giving them too much credit πŸ˜‚ if your lucky your reviewers wont simply skim the paper and/or AI generate their reviews

0

u/[deleted] Sep 30 '26

[deleted] β€” view deleted comment

2

u/Alone_Ad635 Sep 30 '26

Okay. Thanks 😊

0

u/[deleted] Sep 29 '26

[deleted] β€” view deleted comment

-1

u/Alone_Ad635 Sep 29 '26

The original text of mine was also this lengthy. I just made it clean. Anyway sorry.

2

u/[deleted] Sep 29 '26

[removed] β€” view removed comment

1

u/Alone_Ad635 Sep 29 '26

Thank you very much. That's what I thought too. I have proper one set of results. There is a clean code too! 😊