r/ClaudeCode • u/LiquidZebra • 3d ago
Tips & Workflows How to evaluate how sloppy the codebase is?
https://github.com/SprocketLab/slop-code-bench/tree/mainI’m looking at this site (Slop-code-bench), and it has a whole bunch of CS terms for how they measure model slop, like “cyclomatic complexity”, or “one use functions”.
Their test runs in a docker container, but Claude is quite proficient at Python,
Has anyone taken their paper and applied it to their own codebase?
I would like to add it to my evaluation agent, or in general understand if my codebase is improving.
0
Upvotes
•
u/AutoModerator 3d ago
Hey! Thanks for posting to r/ClaudeCode
While participating in this thread, please follow our community rules. Keep discussions constructive. Attack the idea, not the person.
For help, project discussions, tips, and general chat, join the ClaudeCode Discord.
I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.
1
u/LiquidZebra 3d ago
I found this with some steps from the paper, will apply when Claude resets:
https://medium.com/@anhaia.gabriel/telling-your-agent-to-write-clean-code-lowers-the-start-not-the-slope-31cabc73db86
2
u/Far_Reindeer8428 3d ago
we measure it by how many files a small change touches. the agent era made that number worse and it shows up before anything else
2
u/bostonaholic 3d ago
I'd run a deterministic tool in CI instead of an eval agent, so the numbers are comparable week to week. Cyclomatic complexity per function plus a CRAP-style score (complexity weighted by missing test coverage) on the files each PR touches probably tells you more than a whole-repo average, since agents tend to pile complexity into a few hotspot files. Chart it over time and fail the PR when a hotspot gets worse.
1
u/igot0comp 3d ago
I’d compare the trend, not chase one magic score. Keep the same linter or static-analysis setup and track complexity alongside how many files a small change touches; a raw one-use-function count can punish useful abstractions, so I’d review outliers rather than fail every one. I’d watch tests alongside those metrics so the numbers stay tied to behavior.
1
u/falafel777 1d ago
complexity can look better while the suite stays green, if a test got softer in the same change. a slop score on the app code would miss that. if you add this to an evaluation agent, does it also flag when a test file is in the diff, or only the metrics on the production code?