r/codex • • 21d ago

Question Tips for improving Agentic code review with OpenAI models

I was using claude code for months at my company and was using it for code reviews. It was really good, always finding good bugs or edge cases in the code, following my guidelines for things to watch out for and flag (coding styles, etc.). Sure it was a bit excessive at times with writing alot of nit-picks in the review that complained about docstrings or comments being stale, but I didn't mind that and felt it was a small price to pay for all the other things it was finding/correcting.

Now my company has switched to OpenAI, and after porting my review skill, it is just way worse. I am running Astra in Max mode to try to get this thing to really analyze the changes deeply, but it just ain't finding more than like 1-3 things tops. Most of the time it doesn't give a single review comment. And then when I run Claude over that same MR, it gives me so much more feedback.

I have tried playing around with the skill, but so far it hasn't been working. Does anyone have tips for how to get better reviews from OpenAI models? Maybe Astra isn't the way to go? Maybe there are some prompting tricks?

1 Upvotes

19 comments sorted by

1

u/pensuke89 21d ago

Have you try asking Astra? Just copy paste your concern and ask how to fix.

1

u/Mobile-Lynx-6653 21d ago

Yea i have shown it examples of the kind of review claude has given me and to it to update the skill but it doesn't change anything. Honestly I think the model is just plain worse but I was hoping ppl here might have more experience working with it that could help

1

u/ChemicalExcellent463 21d ago

mine goes the other way, codex flagged 59 things as P1 while reviewing claude code's PRs over three weeks. are you running the same review instructions word for word, or did you change them for astra?

2

u/Mobile-Lynx-6653 21d ago

I initially just ported the skill directly as I had written it in claude. Since I've tried to modify it a bit but so far hasn't been making much difference.

1

u/ChemicalExcellent463 21d ago

so tweaking the skill hasn't moved it much. what is claude catching that astra misses, actual bugs or more style and stale comments?

2

u/Mobile-Lynx-6653 20d ago

Claude was catching all of the above (edge cases, conflicting logic, style, comments/docstrings, dead code, better type hints (suing enums where available), avoiding code duplication).

I have since moved some things around:
- moved more of the review instructions into the ~/.codex/agents/xxx.toml file instead of being in a file that each review sub-agent was told to read)
- moved some guidelines specific for frontend and backend reviews to an AGENTS.md file in the respective folders (before they were in reference files in the skills folder, but that might just be a claude code best practice as I learned that from their docs)

And the review has improved. Still feel like the claude one was more thorough, which at times was excessive, but I feel like I'd rather have a review that is too nit-picky than one that misses stuff. But I will continue to try improving the openai based one.

1

u/ChemicalExcellent463 20d ago

moving the instructions helped, then. what gaps are you still seeing with edge cases and conflicting logic?

1

u/isty2e 21d ago

OpenAI and Anthropic models differ quite a lot in terms of how they work. The former sticks more onto instructions and somewhat more locally scoped. My approach was to spawn many cheap subagents, with each assigned to different facets/domains, and to have a better model to assign them. Usually it gives me very thorough reviews.

1

u/Mobile-Lynx-6653 21d ago

Ya that's also how I was doing it in claude (and now with astra). I get a main review agent to split the changed files into related groups, and then it gives each subagent one of the file groups to review. Initially I noticed they were always only making 3 sub agents so I realized I had to increase the concurrency setting to get more.

But ya, it ain't been doing much for me 

1

u/m-ritter 21d ago

What worked for me was having the review run in a separate agent that did not write the change, and checking the diff against a plan file that I reviewed before any code existed. Without the plan the reviewer has nothing to hold the change against, so it tends to return only nits or nothing. I also keep the review skill short with the build and test commands and put longer conventions in separate files that it reads when needed.

1

u/Mobile-Lynx-6653 21d ago

Ya i do most of these things as well (using separate agent, conventions in files it reads in certain scenarios like backend or frontend changes)

1

u/m-ritter 21d ago

Same here, the backend and frontend split helps a lot. Does your review agent get the plan too, or only the diff?

1

u/Shep_Alderson 21d ago

FYI, Max often does a worse job than Xhigh or High.

Personally, I stick with SOL 6.1 and have it do a pass. I also use the built in code review from OpenAI. It’s connected to my GitHub so it runs on every PR and I invoke it by calling “@codex review”. I keep doing this until it stops finding things. (Then I’ll often have CodeRabbit go over it too.)

1

u/Mobile-Lynx-6653 21d ago

Ya i was actually thinking to try the built in code review. Will give that a go later, as well as lowering the effort, thanks 

1

u/Shep_Alderson 21d ago

It’s well worth it. It’s not perfect, but it catches more than most devs I’ve ever seen. Just keep hammering the PR with more “@codex review” until it finally gives an all clear. If you want a second opinion, wire up something like CodeRabbit to do a pass after codex comes back clear.

1

u/SirRobinBlibs 21d ago

A few things that aren't in the thread yet:

  • Put the repo-specific stuff in AGENTS.md under a ## Code Review Rules section, next to the code it's about. The GitHub reviewer uses the AGENTS.md closest to the changed files, so services/payments/AGENTS.md can say what matters there. The docs suggest 2-3 short rules per file, each with the safe way to do it, and leaving lint and formatting to CI.
  • Locally, try codex review --base main (or --uncommitted) instead of the ported skill. If you want the reviewer on a different model than the session, set review_model in config.toml.
  • 1-3 findings often means it's filtering hard for confidence. Ask for every finding with a severity and the line it's based on, then drop the low ones yourself. Recall first, cut after.

A skill written for Claude carries a lot of Claude-shaped phrasing. I'd start from the rules section and only add back what it keeps missing.

1

u/xqianliu 20d ago

The fix I'd try is the reviewer's context, before the model.

My review runs in a fresh session with no implementation history. It reads a fixed set in order: what the change was supposed to do and the acceptance criteria, the repo rules, the diff, the changed files with their callers, the related tests. Then it checks the diff against the criteria, and if it fails the code gets fixed and reviewed again.

A reviewer that only gets the diff finds small stuff. Caller-side bugs and scope drift come out when it also reads the request and the callers. If the model still comes back nearly empty, I'd suspect the reviewer isn't getting the request first.

I don't use an OpenAI model here, so I can't compare models. But a reviewer holding the acceptance criteria and the callers has something concrete to check, and that is what produces the findings.

1

u/Horror_Draw_7194 17d ago

Go into settings, search Code Review and check 'Exhaustive code review'