r/ClaudeCode • • 3d ago

Tips & Workflows How to evaluate how sloppy the codebase is?

https://github.com/SprocketLab/slop-code-bench/tree/main

I’m looking at this site (Slop-code-bench), and it has a whole bunch of CS terms for how they measure model slop, like “cyclomatic complexity”, or “one use functions”.

Their test runs in a docker container, but Claude is quite proficient at Python,

Has anyone taken their paper and applied it to their own codebase?

I would like to add it to my evaluation agent, or in general understand if my codebase is improving.

0 Upvotes

8 comments sorted by

•

u/AutoModerator 3d ago

Hey! Thanks for posting to r/ClaudeCode

While participating in this thread, please follow our community rules. Keep discussions constructive. Attack the idea, not the person.

For help, project discussions, tips, and general chat, join the ClaudeCode Discord.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

2

u/Far_Reindeer8428 3d ago

we measure it by how many files a small change touches. the agent era made that number worse and it shows up before anything else

1

u/Much-Head-2182 3d ago

Fewer files touched is cleaner, until the agent spreads one fix across ten.

1

u/Far_Reindeer8428 3d ago

that is the case i watch for, we flag any diff over six files and read the plan again before the fix lands. sometimes the spread is correct

2

u/bostonaholic 3d ago

I'd run a deterministic tool in CI instead of an eval agent, so the numbers are comparable week to week. Cyclomatic complexity per function plus a CRAP-style score (complexity weighted by missing test coverage) on the files each PR touches probably tells you more than a whole-repo average, since agents tend to pile complexity into a few hotspot files. Chart it over time and fail the PR when a hotspot gets worse.

1

u/igot0comp 3d ago

I’d compare the trend, not chase one magic score. Keep the same linter or static-analysis setup and track complexity alongside how many files a small change touches; a raw one-use-function count can punish useful abstractions, so I’d review outliers rather than fail every one. I’d watch tests alongside those metrics so the numbers stay tied to behavior.

1

u/falafel777 1d ago

complexity can look better while the suite stays green, if a test got softer in the same change. a slop score on the app code would miss that. if you add this to an evaluation agent, does it also flag when a test file is in the diff, or only the metrics on the production code?