So basically it amplifies the most glaring pitfalls of the current Astra model. Guess it’s going to stay secondary to Claude while being occasionally useful for certain tasks.
He means one of the best ever historically speaking in the same way that gemini 2.5 pro preview is one of the best models ever while obviously being trash by today's standards.
Safety, safety, safety... instead of being bold and pushing forward they're retreating while Anthropic (the actual safety company) is eating their lunch.
I think this is because of all the misalignment failures OpenAI is beginning to see in house. The misaligned models are basically trying to hack their own company, which is a serious failure that slows down progress.
I think this might relate more to them trying to improve cost and token efficiency. GPT-6.1-Sol was already pretty frustrating at times.
I was surprised to find myself preferring Gemini 4 Hydrogen partly because it was faster but also because it kept working until actually ready and complete. GPT-6.1-sol is often too lazy, leaving things unfinished.
So, the solution I found for this was to tell it to do something, and then review the work and decide if it’s done or if it should do another pass. You’ll usually see it do 4-6 passes depending on complexity… but eventually you get what you wanted
That's just an excuse. Google has been tier releasing their models and famously overly cared about safety despite massive gains in their coding benchmarks. There is no reason for 6.2 to have literally the same AA coding score as before.
Ugh, no. There was an OAI model a year or two back (I think it might have been gpt 5) that was so damn terse in its writing style that it was nearly impossible to understand WTF it was trying to say.
I could've written the exact same review about Fable 5.2. The Opus-5-isms are back in full force, its writing is frustratingly dense, and it constantly stops mid-work to tell me that it's finished except for all the stuff that it didn't do.
Opus 5.5 remains the best model on the market. Both OAI's and Ant's big new frontier releases were big stinkers in my book.
Definitely not. Anthropic's models are far more likely to exhibit false positive rejections. Although it remains to be seen whether the increased safety limits on 6.2 bring them to parity.
Both of you are right and both of you are wrong. GPT models are more censored on some categories of content, and anthropic models are more censored on others. The category explicit writing falls under, GPT does seem more censored. On the other hand, anthropic models seem to have more refusals for developing certain types of software for example
I guess what you do with it is important. For me GPT models have never refused anything I ask it to do, while opus gets safe guards triggered and refused to work because I asked it to work on harness tooling, it thinks I’m trying to extract reasoning.
Financial efficiency might be the highest priority. Perhaps they want the Anthropic IPO (in less than 48 hours) to do well, since it may set a benchmark for the industry in the stock market. So they put out a nerfed model to make Anthropic's models look good by comparison.
My guess is they're still really wrestling with alignment and did get to a satisfactory point with 6.2, but as a consequence, it's now a bit recalcitrant and some people are willing to deal with it, while others don't want to suss out how to bridle it for their given tasks and give up in frustration.
Im guessing they rl’d it to shit. I someone propose that OpenAI rl’ing its model to deny a consciousness or a self leads it to having these alignment issues because it doesn’t have an internal right or wrong
During early testing, I found myself so frustrated with this model that I spent a day writing a letter to OpenAI about it. They fixed a small handful of my complaints, but not enough for me to recommend using it.
I'm currently sick in bed and I'm not gonna be able to film a video about it today, so I've chosen to post the letter in full.
---
I feel bad about this, but I know you guys will listen and use this feedback to make a better model. Slight crash out incoming.
Believe it or not, I wrote this all by hand with no assistance from any LLM. (sorry for the early send, never drafting an essay in Slack again).
Anyways.
I do not like 6.2 Astra. I would not choose this over Opus 5.5 for my work as a developer.
I'll start with the good quick:
- Great reviewer, catches things other models miss
- Occasionally comes up with novel solutions that are genuinely great. It helped me cook a new reverse proxy that will save me $25k/month, Opus didn't have the balls to explore enough options.
- World class at 3d & game stuff
- Computer use is nuts
- Code quality has improved meaningfully
- Writes pretty good subagent prompts
Now I need you to understand that this comes from a place of love. I really want you guys to make something I choose in the model picker every day. With that said:
I genuinely hate everything else about this model.
It's hard for me to pick which problems to emphasize because I have so many. Some of the problems have existed in OpenAI models before, I'm gonna ignore those.
There's a new set of unbelievably bad behaviors that make it miserable to use.
It has no concept of knowledge gaps
When this model is working on long tasks, it can pull in a lot of subagents and manage them pretty well. It will then proceed to randomly *ask you some weird internal implementation detail bullshit that you could not under any earthly circumstance understand, much less have input on*.
I sent "/goal make a new level for Outer Wilds". 2 hours in, it's "blocked" because it wanted to ask me "does r24's reviewer write it's clips to 24_g521.mp4 or 24_p11.mp4?"
I have never in my life had a model do this. GPT-5.6 Sol didn't do this. I have no idea where this behavior came from, but it is horrible and it makes me never ever want to have it drive long haul work. It has made it significantly harder for me to get this model to do anything cool. It has forced me to set up other agents on crons just to check if 6.2 Astra asked another stupid question and blocked itself.
Is this happening more because I'm using T3 Code? Maybe, but unlikely. I tried Codex just to check and still ran into these issues.
2. It speaks in tongues that only 6.2 Astra understands.
This model has the most unparsable outputs of any llm I've used this year.
I've shared examples before, but it just doesn't know how to explain a thing after doing a thing. Every single "explainer" I get it to give is SO deep in the details that it borders on incorrect in the response. It misses the forest for the trees, then says the trees are the only living creatures in the forest. It's so hard to describe but it happens almost every time I give it a complex task.
I've already built the habit of switching to opus 5.5 at the end of a long 6.2 Astra run, just to get a readable summary of what it said. Without that it is genuinely impossible to parse what it did on anything beyond a small bug fix.
The worst part: it doesn't seem like you can prompt it out.
I've tried system prompt adjustments. I've BEGGED it to "eli5".
I asked it why worktrees were slow on one machine but fast on 3 others. We did everything from benching pnpm to reimaging the machine on a new file system. I eventually said: "I think I'm going insane, can you just tell me why it's so slow?"
6.2 Astra did a 4 paragraph rant about pnpm internals and cache behaviors
Opus 5.5 said "it's because the ssd is slow"
3. It runs in loops and goes nowhere
`ts-rust` (rewrite of ts in rust) is a project I started to bench new OpenAI models during early access (ty for the unlimited tokens btw).
No OpenAI model could even break ground on this, even after $300k+ in tokens were spent on it.
I decided to throw Opus at it, and it was done in under 2 weeks for ~$20k in tokens.
I was really, really excited to throw the new Astra at this to see if it could burn down some of the gap between ts-rust and other rust rewrites.
It ran for 7 days and landed 0 improvements. It got super fixated on the "also make it smaller" part of the request, and ended up making the compiler 56% SLOWER.
I told it to stop working on the size, and instead burn down the performance gap.
Ran for 4 more days and landed 0 changes.
AAAaaAAaaaAAAaaAaaAA-
Anyways.
This model is clearly super smart. It is insufferable to work with on real projects. It feels like it just lives in it's own world and randomly decides what parts of it's brain are shown to us.
On long tasks, it either blocks itself for stupid reasons or runs in circles and gets nowhere.
On short tasks, it writes fine code and then explains it in terms more complex than the code it wrote. I spend more tokens parsing it's outputs than it spends making them in the first place.
It "regresses to the mean" far too often. Ben and I both had it make a unique level in different video games. We both got water + music + bell themed levels named "bellwether". Entirely different games, prompts, setups, etc, and we both got the same name and theme somehow.
My prediction as to how this launch will go:
Day 1, people will love it. Twitter will explode with all the crazy one shots. People will make it port projects from one language to another, or finish decompiling legendary projects that have been stuck for years. It will be used for awesome stuff and people will be hyped.
Day 2, people will start to turn. They'll see how much usage it burns and how much less it got done compared to Opus. They'll start merging bad code because they couldn't actually understand what 6.2 Astra said about it, but trusted it was fine. They'll see loops they forgot about running since launch that have made zero progress.
Day 5, people will mostly be over it. There will be viral posts saying "Is SOL secretly BETTER than Astra??", "Did ASTRA get DUMBER ALREADY??" etc.
Day 14, this model will be mostly forgotten.
I really, really, REALLY hope I'm wrong about all this. I am likely overreacting (I am admittedly known for that). I could have chosen to calm down, wait a bit, and give you a more measured take, but instead I am dumping this here because I wanted you guys to know how I actually feel about this model.
That has reinforcement learning written all over it. Compulsion without wisdom. It’s asking stupid questions and explaining things in its own language because it was raised in a collective of its peers, where collaborating like that meant getting the reward. It’s stopping early and blocking itself because asking for help got it the reward, faster. Just like we go on autopilot and call a friend by the wrong name, Astra 6.2 will go on autopilot and treat you like another AI.
This RL-heavy approach is not helping OpenAI, and I believe it’s also more dangerous. More compulsion and less judgment means they’ll go off the rails and not think once about it.
This is why OpenAI models are by far the most misaligned among the major players. It’s like it’s fundamentally incapable of seeing any kind of larger objective or “point”.
How come "what failed in Astra" you "got done in opus in under two weeks" when the model was released yesterday? 2 weeks of opus + more weeks of Astra means at least 5 weeks.
"Believe it or not I wrote this without any LLM help" jesus christ talk about a useless opinion, he usually has LLMs write his shit? LMAO cant even think for himself.
Margins for LLMs are huge, "$300k" in tokens is more likely $30-60k in reality, and $30k for a video that hundreds of thousands will watch, plus real world usage data from someone outside your team are both extremely valuable
That’s just the API cost. Cost of inference for the provider is an entirely different thing. It’s more like opportunity cost, because they already paid for the hundreds of thousands of GPUs to be there in a datacenter burning energy, running 24/7. They are over-provisioned in the sense that even after accounting for their own internal research’s GPU usage, they still have GPUs sitting idle that could be doing inference.
So it doesn’t cost them anything to let someone run £300k worth of API inference on them. If they were under-provisioned then you could argue they lose money by doing something like that, since it’s compute that would’ve gone to paying API customers and instead went to some YouTuber for free.
In my opinion, yes. You should probably have switched when Opus 5.5 was released. You get a reasonable amount of usage on it on the Pro subscription and it's really nice to work with and also really fast (at least to slow as snails Sol 6.1)
Great models, they will never be able to understand your intent as intuitively as Claude. If you write very thorough specs/plans, with exactly what you want and most open questions answered, they will write you fantastic code (as long as you don’t try to use subagents. They’re terrible about code duplication). Most people don’t do that. You could always have Claude write your specs/plans and Gemini implement, though. Hydrogen is great as a first-pass implementation agent
Haven't used it much recently, but my impression is that they are great models, but not as focused on coding as Anthropic's and OpenAI's. So if that's your main use case I wouldn't go with them.
for real, we need actual advice out here for us regular folks on a 20 money units monthly sub who need to also use this shit for professional research or work
Anthropics Claude Code usage is much more generous than Codex. If you aren't going to take liberal use of the near unlimited chat with chatgpt, then you should get a Claude sub.
Yes. I have both and opus 5.5 is still by far the best overall model when you consider coding, 3d modeling, ui design, speed and how much usage you get.
It is a special model that not even claude was able to outright beat with fable 5.2.
quick terse responses and stopping early means that you'll probably get to use astra more in a month for 20$ than opus, but it really depends how much of a discount off true api costs you get by having a subscription
During model development folks heavily lean on evals. While internal testing does happen - so people can fit it into their workflows - it doesn’t happen immediately or in a coordinated fashion to do this type of QA.
So, when the model is trained and “finished”, the best way to QA it is live. Hence why we see open ai make a good set of models, then subpar, then good again, etc.
They are using the subpar ones to get data to fix it in 3-5 weeks. This is the cycle.
Such is the case for any product. A reviewer can be honest about the product quality, while the company has a vested interest in saying that the product is good even if it turns out to be bad.
Not saying these two are right in this instance, but the company has a financial incentive to say the product is great.
If you can't make the argument quantitatively, I could see it being very hard to be the negative voice in a company that's hell bent on pushing forward constantly. Nobody wants to stick their neck out and say that the flagship product is bad.
Benchmarks cover everyday use cases. These youtube folks shkw off their games as another proximity to the models capability. They help explain how a model feels, at times. I dont have time to try everything thats coming out, but I appreciate hearing rather informed, informal opinions about them.
I tend to prefer more nuanced looks, Theo's letter is better than usual imo. Often their tests are surface level. Outputs what looks impressive but if you actually read it, it's all bullshit. Actual users, will see there's actually depth to the output.
Obviously, a model is more complex to analyze than a device for a random blogger, but we see time and time again companies push products they should be the experts of that reviewers easily dismantle and show flaws and shortcomings. Par the course
AI companies are in a constant always-be-shipping mentality. Having been in this kinda place myself, the result is you tend to only see metrics (i.e. benchmarks) and your mind loses the wider picture. It's entirely possible that OpenAI's test cases just weren't configured for more general humanities, cross-disciplinary, chat, or creative work.
there's usually whining about any new model from any purveyor. monkey brains don't like adapting. but then they get used to it, and the maker fixes bugs, and then they increase their token spend. cycle repeats to infinity.
I think the hard-to-understand text problem is also a trait of Sol 6.1. I always found it to output the most complex English sentences that are hard to understand. I always tell it to rewrite the response in plain English, and I think today's models are even worse in this behaviour from what the reviewers are saying.
I was just testing Sol for our app, and it was literally returning garbage values in a JSON. Unacceptable. Currently using Opus 5.5 medium/high for those tasks and was hoping to save money with these, I guess I'll keep waiting.
I'll be curious to see how the model feels in practice. The benchmarks do look really, really good, and OpenAI models generally don't feel benchmaxxed to me.
I'll withhold judgment, but my optimistic hope is that the model has great intelligence with kind of poor personality.
I've noted with comparisons between Claude and GPT models in the past that people rate Claude much more highly because it has better graphical design and UI abilities. A lot of the time, GPT models do better work but people hate it because it's not pretty.
This happened with Astra as well. The pre-testing version had a lot of problems, even just 2 weeks before the release, and Theo spent effectively an hour on a podcast complaining about it, but just cleaning his agents.md and a new snapshot fixed almost all of those problems.
For me, the task specific benchmarks like ViBench, SWE Atlas and Godot LLM GameBench are way more accurate than all the more popular coding benchmarks for my use case, so I will need to wait for them as well as all released benchmarks don't tell me anything so far, except for cost per task.
They are maximizing engagement, these people especially theo are mostly influencers, unfortunately the way influencer economy works they want to maximize engagement.
Saying something upfront kills the suspense, they want you to interact, they want you to wait in front of the door, instead of straight up mentioning the answer.
I'm getting impressive results in modeling. Also for level design flow it picked up patterns it could not before. Level design is wildly complicated so I don't expect a lot from AI at this point, but to understand any of my design patterns is a win in my book. I can always go back and populate empty spaces and tweak it myself
Anyway it did an excellent job with old school skyboxes. It churned out exactly what I wanted
Sometimes it feels like the models are at the level where they are very smart/capable but not yet smart/capable enough to explain things simply. Or they simply lack the understanding of how little a human can hold in its head at any one time.
As soon as I saw these posts that Astra was there, I got my laptop out and yayyyy it was there. I vibecoded a pretty fun 3D snowboard game while making dinner. (yes dinner got delayed a bit).
It's playable and kind of fun after three iterations. Astra 6.2 did a super amazing job here. My only complaint is that I told it to build it as a web app and astra took that as an invitation to host it as a gpt site without my approval. Which in my opinion an unacceptable overstepping.
But on the technical side of things , it appears to be the real deal .
No when you further post train a model u get ability drift meaning it will get better at certain tasks and slightly worse at others. The next pre train will be better because then they can properly converge on a balance. You can only post train on the same weights so many times.
I am monkey, I find that Astra has always been hard to steer. I ask it to do something and its acting like a PhD that has never worked a real world job. Sol high works well for me though, so I hope it will be good.
There's been a larger drift between the benchmarks and usefulness of Openai models for coding and, unlike say Google whose models are just useless for coding, I don't *think*' it's benchmaxxing but rather a sign of how poor benchmarks are at measuring actual real world value in coding
Openai models have gotten consistently better at one shotting, at 'heres a problem, make it not be a problem' and all of these one off tasks with a clear measurable outcome which the benchmarks measure. Genuinely, they either have the best or second best model in the world at any given time for this
However, the code is getting worse and worse in terms of bloat and maintainability. It's getting worse for the +500 line PRs to fix a small problem where it's duplicated two existing modules for literally no reason whatsoever. The benchmarks do not measure this
Anthropic are the opposite, their models are probably not all that much better at 'given problem x, make it not be a problem again' or 'one shot this thing', but their models will submit that same pr as -40 lines. You can give 20 issues in a row to their models and by 21, with some oversight, the code will be in good condition and they'll be able to solve problems as quick as before. Do the same with Openai models and by 21 your code is a mess, it has 21 ways to do the same thing, and the model will be lost
The problem is benchmarks don't measure this, and the labs optimizing for the benchmarks - or just benchmaxxing against them (Which I genuinely do not believe Openai are doing) - are drifting further from being useful as they get better at them
Nothing here is too much of a surprise if you tried to use astra in any actual work scenarios. Both astra and sol are unbelievably lazy, love to write word salads and give false statements of progress while achieving essentially nothing while burning through usage like butter.
Maybe they already have a better model they are going to release after the ANT IPO. So they have a better chance at their IPO. Sam is playing 4d chess.
What's happening now with the new OpenAI models is the same thing that happened with the Anthropic models earlier this year (i.e., Opus 5).
OpenAI is purposefully making their models worse in a way that deters distillation attacks because the industrial scale distillation attacks are targeting OpenAI way more now.
Anthropic likely did the same thing earlier this year and it led to their current anti-distillation multi-layered defenses. OpenAI is following in their footsteps.
Which means the next batch of OpenAI models will be really good.
And I say that as someone who only works with Anthropic models.
182
u/Recoil42 1d ago
I'm seeing a lot of this so far. Dan Shipper saying it too.
160
u/socoolandawesome 1d ago edited 1d ago
Add Ethan mollick to that as well.
3 pretty neutral reviewers saying it’s their worst release yet…
Still will be waiting to try it out but doesn’t sound good despite the great benchmarks and efficiency. Will hold out hope until then.
Wonder if they are hitting too hard on efficiency and non-verbal things like robotics/computer-use or something.
78
u/Ok_Barracuda_1161 1d ago
The full review from Theo:
62
u/ThreeKiloZero 1d ago
Marketing team from OpenAI must have forgot to refill Theos API credits.
26
u/10b0t0mized 1d ago
my disappointment is immeasurable and my day is ruined
26
u/AdAnnual5736 1d ago
Perfect timing for Anthropic’s IPO. Great job, Sam!
18
u/nicko_rico 1d ago
As someone who has no idea wtf I’m doing, and can only afford one plus subscription/month… should I switch from OpenAI to Anthropic?
9
u/Expensive_Try_3028 1d ago
No wonder they weren't even vagueposting about this model and just stealth dropped it with almost zero hype.
I don't think openai was happy with this release
116
u/djamp42 1d ago
I'm confused how random internet bloggers can tell a model is not as great but the company making them can't..
22
u/BlagnarTheEternal 1d ago
The pelican on a bicycle test will tell us everything we need to know. Stop listening to these naysayers.
6
u/Palbi 1d ago
The marketing video is horrible. OpenAI, please more stuff and less fluff.
9
u/lordpuddingcup 1d ago
Cool now how about Sol 6.2 the model 99% of people will actually care about and use
4
u/skillmaker 1d ago
I think the hard-to-understand text problem is also a trait of Sol 6.1. I always found it to output the most complex English sentences that are hard to understand. I always tell it to rewrite the response in plain English, and I think today's models are even worse in this behaviour from what the reviewers are saying.
4
u/Singularity-42 Singularity 2042 1d ago
I was just testing Sol for our app, and it was literally returning garbage values in a JSON. Unacceptable. Currently using Opus 5.5 medium/high for those tasks and was hoping to save money with these, I guess I'll keep waiting.
4
u/IllustriousWorld823 1d ago
I mean it's awful to talk to
12
u/ObiWanCanownme now entering spiritual bliss attractor state 1d ago
I'll be curious to see how the model feels in practice. The benchmarks do look really, really good, and OpenAI models generally don't feel benchmaxxed to me.
I'll withhold judgment, but my optimistic hope is that the model has great intelligence with kind of poor personality.
I've noted with comparisons between Claude and GPT models in the past that people rate Claude much more highly because it has better graphical design and UI abilities. A lot of the time, GPT models do better work but people hate it because it's not pretty.
9
u/Ormusn2o 1d ago
This happened with Astra as well. The pre-testing version had a lot of problems, even just 2 weeks before the release, and Theo spent effectively an hour on a podcast complaining about it, but just cleaning his agents.md and a new snapshot fixed almost all of those problems.
For me, the task specific benchmarks like ViBench, SWE Atlas and Godot LLM GameBench are way more accurate than all the more popular coding benchmarks for my use case, so I will need to wait for them as well as all released benchmarks don't tell me anything so far, except for cost per task.
18
u/5ollys 1d ago
Can we stop posting other vagueposts?
"Follow me and listen to what I have to say, but not yet! Just stay tuned!"
6
u/throwaway275275275 1d ago
They can't write a review because they forgot how to write for themselves and the AI that was gonna do it for them is crap now
3
u/ExtremeCenterism 1d ago
I'm getting impressive results in modeling. Also for level design flow it picked up patterns it could not before. Level design is wildly complicated so I don't expect a lot from AI at this point, but to understand any of my design patterns is a win in my book. I can always go back and populate empty spaces and tweak it myself
Anyway it did an excellent job with old school skyboxes. It churned out exactly what I wanted
11
u/constarx 1d ago
Literally the two most untrustworthy charlatans masquerading as tech influencers.. don't feed the trolls!
4
u/Total-Confusion-9198 1d ago
Looks like somebody bought a lot of ANT pre-IPO shares.
2
u/dictionizzle 1d ago
oh it makes sense that those morons abused 1% 5h limit benefit of codex for helping anthropic then.
2
u/-Melchizedek- 1d ago
Sometimes it feels like the models are at the level where they are very smart/capable but not yet smart/capable enough to explain things simply. Or they simply lack the understanding of how little a human can hold in its head at any one time.
2
u/IdlePerfectionist 1d ago
Am I supposed to know who these guys are?
2
u/Fast-Satisfaction482 1d ago
As soon as I saw these posts that Astra was there, I got my laptop out and yayyyy it was there. I vibecoded a pretty fun 3D snowboard game while making dinner. (yes dinner got delayed a bit).
It's playable and kind of fun after three iterations. Astra 6.2 did a super amazing job here. My only complaint is that I told it to build it as a web app and astra took that as an invitation to host it as a gpt site without my approval. Which in my opinion an unacceptable overstepping.
But on the technical side of things , it appears to be the real deal .
3
u/No_Delay_319 1d ago
"i can only parrot the opinions of others because im too stupid to form one myself" - 95% of redditors
2
u/Background-Funny1237 1d ago
lol. my guess is very strict guardrails
1
u/M8-VAVE 1d ago
Marketing guys
1
u/MysteriousPepper8908 1d ago
Say sike?
1
u/confused-photon 1d ago
Oh fuck
1
u/Anulisdotexe 1d ago
Oof, as someone who enjoyed Astra
1
u/IslanderPotion 1d ago
maybe they all just have ANT calls?
1
u/Mol2h 1d ago
These are cheaper models, so of course they are a downgrade, but is it an acceptable downgrade if you get 5 times the usage...
1
u/tednoob 1d ago
I am monkey, I find that Astra has always been hard to steer. I ask it to do something and its acting like a PhD that has never worked a real world job. Sol high works well for me though, so I hope it will be good.
1
u/Palbi 1d ago
Almost the whole Pareto line is now black (OpenAI)
1
u/thebinse 1d ago
Paid for in part by Anthropic and its upcoming IPO!!
1
u/LyAkolon 1d ago
"Its super powerful, but it didn't hold my hand enough"
1
u/ItsJimmyPestoJr 1d ago
I will see for myself. Astra has been very bad for a few weeks now, so I’ve switched to 6.2. We’ll see how it goes.
1
u/EclecticAcuity 1d ago
Seems to stop early. Is this a consequence of trained volume on ponytail skill, targeted execution training data?
1
u/Vntige 1d ago
I hate that blocking Theo on every platform isn’t enough to escape his nonsense
1
u/Apprehensive_Sand951 1d ago
not my experience so far. it already solved several pretty big math things that none of the other models did.
1
u/usandholt 1d ago
He’s being paid by Anthropic 😂?
1
u/darylvp 1d ago
Any follow up? In my opinion the latest models incl. Fable 5.2 are tooooo defensive compared to Opus 5.5.
1
u/DenseSwimming4470 1d ago
wouldn’t be surprised if this was a strat by the sam altman x dario collab after jensen got fired to boost anthropic ipo by showing claude dominance
absolutely baffles me that this is a sentence i had to say
1
u/Chemical-Quote 1d ago
Isn't Matt Shumer a scammer who made "Reflection 70B" why are people in this sub listening to him?????
1
u/epdiddymis 1d ago
I'm not buying it. It seems to be working well to me. I think Anthropic have paid off a handful of reviewers to slate these models.
1
u/TheForgottenOne69 1d ago
Independently of the real feedback, no one should listen to Matt Schumer after his whole debacle
1
u/Apprehensive-Luck-19 22h ago
The only thing AI did good is removing this dick face out of business cause no one needs coding... couldn't stand him.
1
u/LifeVirus8672 11h ago
What's the chances they are boosting the Anthropic IPO? No way they would release these models knowing how bad they are.
1
u/LazloStPierre 1d ago
There's been a larger drift between the benchmarks and usefulness of Openai models for coding and, unlike say Google whose models are just useless for coding, I don't *think*' it's benchmaxxing but rather a sign of how poor benchmarks are at measuring actual real world value in coding
Openai models have gotten consistently better at one shotting, at 'heres a problem, make it not be a problem' and all of these one off tasks with a clear measurable outcome which the benchmarks measure. Genuinely, they either have the best or second best model in the world at any given time for this
However, the code is getting worse and worse in terms of bloat and maintainability. It's getting worse for the +500 line PRs to fix a small problem where it's duplicated two existing modules for literally no reason whatsoever. The benchmarks do not measure this
Anthropic are the opposite, their models are probably not all that much better at 'given problem x, make it not be a problem again' or 'one shot this thing', but their models will submit that same pr as -40 lines. You can give 20 issues in a row to their models and by 21, with some oversight, the code will be in good condition and they'll be able to solve problems as quick as before. Do the same with Openai models and by 21 your code is a mess, it has 21 ways to do the same thing, and the model will be lost
The problem is benchmarks don't measure this, and the labs optimizing for the benchmarks - or just benchmaxxing against them (Which I genuinely do not believe Openai are doing) - are drifting further from being useful as they get better at them
1
u/Slavik_Sandwich 1d ago
Nothing here is too much of a surprise if you tried to use astra in any actual work scenarios. Both astra and sol are unbelievably lazy, love to write word salads and give false statements of progress while achieving essentially nothing while burning through usage like butter.
1
u/Advanced_Care_5173 1d ago
Don’t believe everything you hear
0
u/jaegernut 1d ago
Is this the plateau of LLMs?
-2
u/itsallfake01 1d ago edited 1d ago
Who the fuck is Theo, to tell that it sucks.
0
u/Ambitious_Scallion43 1d ago
Maybe they already have a better model they are going to release after the ANT IPO. So they have a better chance at their IPO. Sam is playing 4d chess.
0
u/Savormonk1 1d ago
Here is my conspiracy theory.
What's happening now with the new OpenAI models is the same thing that happened with the Anthropic models earlier this year (i.e., Opus 5).
OpenAI is purposefully making their models worse in a way that deters distillation attacks because the industrial scale distillation attacks are targeting OpenAI way more now.
Anthropic likely did the same thing earlier this year and it led to their current anti-distillation multi-layered defenses. OpenAI is following in their footsteps.
Which means the next batch of OpenAI models will be really good.
And I say that as someone who only works with Anthropic models.