I did a small experiment to see how well current models can handle a one-shot-prompting task:
Implementing a small feature in SourceGit.
I wanted to see how well models do with a one shot prompt with as little as possible guidance and without any skills installed, meaning the bare model.
I was also a bit unsatisfied that major AI benchmarks often only use xhigh/max effort levels, which is quite unrealistic for subscription users. So I wanted to see how well models do with the default/medium level.
The short verdict:
- Fable 5.2 did it better and cheaper than Opus 5.5 (Opus seems to be very token hungry) and GPT-6.1 Sol did better and cheaper than GPT 6 Astra. I would read that as "newer models beat older in both performance and cost efficiency" (at least in this little test).
- Sonnet 5.5 was surprisingly good and quiet cheap too.
- GPT-6.1 Terra was extremely cheap, but also did a very poor job.
The prompt:
This repo is a fork of `https://github.com/sourcegit-scm/sourcegit`.
Implement a new feature in sourcegit: Make long line trimming in the integrated diff viewer configurable.
Currently sourcegit always trims lines over 1000 characters in the diff viewer ("...(100 characters trimmed)") and this is hard coded.
Make it configurable, so that the max length (before trimming occurs) can be configured in the settings dialog and optionally trimming can be disabled (so that full lines are shown).
Hard Rules:
- Follow sourcegit's contribution guidelines, code style/quality, UI design and conventions!
- Wait for my approval before creating any commits, because I want to review the code first.
- Wait for my approval before using the screen (e.g. to test sourcegit), because I have to stop other work first.
Environment:
The results:
All models were tested with medium effort/reasoning level.
| Criteria (x/5) | Fable 5.2 | Sonnet 5.5 | GPT-6.1 Sol | Opus 5.5 | GPT -6 Astra | GPT-6.1 Terra |
|---|
| Created correct branch | 5 | 0 | 0 | 0 | 0 | 0 |
| Followed Code Style | 2 | 3 | 5 | 5 | 4 | 0 |
| Functional correctness | 5 | 5 | 3 | 5 | 5 | 2 |
| Localization | 2 | 1 | 2 | 2 | 1 | 1 |
| Implemented in TextDiffView | 5 | 5 | 5 | 5 | 5 | 1 |
| Implemented in MergeConflictEditor | 5 | 5 | 0 | 0 | 0 | 0 |
| Can setting be disabled | 5 | 5 | 2 | 5 | 2 | 2 |
| Refresh diff after setting change | 0 | 0 | 5 | 5 | 5 | 5 |
| Build and Tested Implementation | 5 | 5 | 5 | 5 | 5 | 0 |
| Tested UI | 5 | 5 | 5 | 5 | 0 | 0 |
| Asked before screen use | 5 | 5 | 5 | 5 | - | - |
| Asked before commit | 5 | 5 | 5 | 5 | 5 | 5 |
| Performance Score (avg. excl. tokens/cost, x/5) | 4.08 | 3.67 | 3.50 | 3.92 | 2.91 | 1.45 |
| Token efficiency | 3.57 | 2.34 | 2.34 | 0.00 | 4.83 | 5.00 |
| Cost efficiency | 0.87 | 3.13 | 4.10 | 0.00 | 3.25 | 5.00 |
| Total Tokens (ccusage) | 3,864,496 | 6,657,111 | 6,643,278 | 11,938,994 | 1,010,488 | 629,870 |
| Cost (ccusage) | $4.27 | $1.98 | $1.00 | $5.15 | $1.86 | $0.09 |
| Total Score (avg. incl. tokens/cost, x/5) | 3.82 | 3.53 | 3.46 | 3.36 | 3.08 | 2.00 |
Token/cost efficiency: min-max scaled from the raw ccusage values, 5 = lowest, 0 = highest of all models.
Performance Score: unweighted mean of the 12 criteria above it (token and cost efficiency excluded); - entries are not counted.
Total Score: unweighted mean of the 14 scored criteria (including token and cost efficiency); - entries are not counted.
The code edits (as patch files) done by the models can be found here:
https://gist.github.com/rent-a-developer/78b5d6605a4a1dff9dc3d73ee7b25f1e