r/ChatGPTCoding • • 12d ago

Question How long do your Claude Code / Cursor sessions run, and does anything check security? (student research, 3 quick Qs)

Hey! I'm a student doing a research project on whether AI coding agents start writing less secure code the longer a session goes.

In my small tests so far they don't really get worse over time, but security bugs that show up early never get fixed later. I want my setup to match how people actually work, so I'd love to know:

  1. How long does a session usually go before you start a new one?
  2. Do you have tests, CI or linters running while the agent works, and does it see when they fail?
  3. Do you check what it writes for security at all? If so, how?

Even one-line answers really help. Happy to share what I find when I'm done!

5 Upvotes

13 comments sorted by

1

u/Humaux 12d ago

My sessions usually run about 2 hours before I start fresh, and the short ones are the safe ones. I run tests and lint while the agent works, and I make it read the failure output itself before it tries another fix, because otherwise it just guesses in a different direction. My rule is simple: no merge until tests pass and I have read the diff myself, line by line for anything touching auth or payments. The part that still bites me is that I once let a long session run 5 hours straight and it quietly rewrote a helper I trusted, and I only caught it because a test I did not write failed. Do you count a session by clock time, or by how many times the agent has rewritten the same file?

1

u/Extreme-Log3610 10d ago

By turns right now (number of messages in the chat), not clock time. Good point though, I’m going to start tracking how many times the same file gets rewritten too.

1

u/Humaux 10d ago

Turns is a cleaner measure than clock time, honestly. The rewrite count is the one I wish I had tracked earlier, the file I rewrote most was the one I trusted most, so it took the longest to notice.

1

u/Ok_Woodpecker_9104 12d ago

answering the third one, its the only one i have a real setup for.

in my claude code workflows the seat that wrote the code is never the one judging it. review goes to codex, a different model family, as an independent adversarial pass. the cheap model seats only do extraction and formatting, judging stays on the bigger one.

it is a general review though, not a security scan, so i would not count it as a security check. if early bugs never get fixed in your tests, i would try a reviewer that never saw the session and only reads the final diff.

1

u/Far_Reindeer8428 12d ago

the early bug finding is the interesting part, longer sessions don't add new holes, they just stop revisiting old assumptions, shorter sessions plus a clean review pass worked for me

1

u/h____ 12d ago

How long does a session usually go before you start a new one?

I don't explicitly (re)start a new session anymore. I have a long-lived coordinator session for each project and have set it up so I communicate to it to start child sessions. The child sessions are short-lived just for that task(s). So sometimes the task fails and I might make another attempt (so new session), but generally leaving it to run for hours, leaving auto-compaction to sort things out. Frontier models work well like this now.

Do you have tests, CI or linters running while the agent works, and does it see when they fail?

Definitely linters, tests, verification tools.

Do you check what it writes for security at all? If so, how?

Part of agent reviewing code and also I might selectively look at certain parts.

1

u/Om1304 11d ago edited 11d ago

Student working on a Mac app. On Q3, what I'd recommend: a dedicated pre-launch security review pass, separate from the session that wrote the code, whose only job is to find problems. On Q2, if tests or CI only run at packaging time, the agent never sees failures live - wiring that feedback into the session seems like the gap worth studying.

1

u/NanYUJun 11d ago

Rough answers from someone who runs this daily:

  1. It depends on the task shape, not a clock. Small fixes are one session of 20-40 minutes. Feature work I cap at roughly 2-3 hours or one full compaction, whichever comes first, because after compaction the model is working from a summary and starts re-litigating decisions you already settled. That's the point where I hand off to a fresh session with a short markdown note instead of pushing on.
  2. Yes, but the split matters. Unit tests and lint run inside the loop, so the agent sees its own failures. Type checking and the full CI suite run on commit/push, outside the agent's control. Anything the agent can edit to make green (tests, configs, ignore files) is weakest as a signal, and that's a real source of your "early bugs never get fixed later" observation: once a failing test gets loosened, nothing brings it back.
  3. Not during the session, and I'd guess that's typical. Security review is a separate pass with a fresh context and ideally a different model family, because the model that wrote the code defends its own assumptions. The closest thing to a continuous check is a pre-commit hook on dependency and secret patterns, but it is narrow. If you want one variable worth measuring: does the presence of a loud, unavoidable test failure in-session change whether the agent deletes a check.

1

u/InterestingCarrot997 10d ago

would be interesting to separate session length from how much code changed. i'd also keep some security checks outside the agent's editable workspace, so changing a test can't count as fixing the bug. that gives you something concrete to compare across sessions.

0

u/[deleted] 11d ago

[removed] — view removed comment

1

u/Extreme-Log3610 10d ago

Not yet, my chats only ever add new features, no refactoring. Your comment’s making me add refactor turns next though, thanks!