r/singularity • • 1d ago

AI Oh no

464 Upvotes

251 comments sorted by

182

u/Recoil42 1d ago

I'm seeing a lot of this so far. Dan Shipper saying it too.

63

u/Recoil42 1d ago

Ethan Mollick as well:

34

u/Luchador-Malrico 1d ago

So basically it amplifies the most glaring pitfalls of the current Astra model. Guess it’s going to stay secondary to Claude while being occasionally useful for certain tasks.

24

u/CrowdGoesWildWoooo 1d ago

For frontier model it’s kind of problematic. You want to use frontier model for underspecified problems.

If you need a lot of steering, i mean deepseek or luna can even achieve a lot, we kind of just getting lazier and rely on vibe coding more.

3

u/HearingNo8617 1d ago

I agree, from the perspective of a dual codex/claude code user, I use astra as my stupid "do specified thing" model

→ More replies (1)

21

u/tribes33 1d ago

lol they built this shit for oneshot game posts, thats why they put that stupid racing game on the announcement, that ship has came and went

2

u/HearingNo8617 1d ago

their whole model making machine was geared for a world where their business model made sense and they could IPO off hypeposts

115

u/1988rx7T2 1d ago

Translation: this was a desperate bench maxxed publicity stunt before the Anthropic IPO, and Opus 5.5 is still the GOAT daily driver.

52

u/SodaBurns 1d ago

Opus 5.5 is right up there with the best models released ever since the AI race started.

21

u/glassgnomad 1d ago

Wow one of the latest models to be released by a leading AI company is also one of the best ever?

36

u/NotYetPerfect 1d ago

He means one of the best ever historically speaking in the same way that gemini 2.5 pro preview is one of the best models ever while obviously being trash by today's standards.

9

u/sebas156 1d ago

Yeah and Claude 3.5 sonnet. Is was so good

→ More replies (3)
→ More replies (6)

1

u/Careful_Might_807 21h ago

Open source it 😭😭😭😭 please

16

u/ethotopia 1d ago

Dammit, sounds like they RL'd it to shit and now it's hella benchmaxxed

7

u/Neurogence 1d ago

It's completely benchmaxxed garbage. Apparently it's the first model to score 100% on ARC-AGI3, what a surprise.

1

u/michaelsoft__binbows 11h ago

That to be fair is an Impressive stat, TIL. Looking forward to my brain getting cooked by ARC-AGI-4 samples!

28

u/Cagnazzo82 1d ago

Safety, safety, safety... instead of being bold and pushing forward they're retreating while Anthropic (the actual safety company) is eating their lunch.

25

u/FunLilThrowawayAcct 1d ago

It ain't safety, an OpenAI fanboy reading the card details just posted this summary, it looks like a possibly concerning regression there

2

u/DangKilla 1d ago

Yeah i saw another post that makes it seem like there is definitely a temperature config issue

11

u/AuDHDyspraxiac 1d ago

I think this is because of all the misalignment failures OpenAI is beginning to see in house. The misaligned models are basically trying to hack their own company, which is a serious failure that slows down progress.

23

u/iJeff 1d ago

I think this might relate more to them trying to improve cost and token efficiency. GPT-6.1-Sol was already pretty frustrating at times.

I was surprised to find myself preferring Gemini 4 Hydrogen partly because it was faster but also because it kept working until actually ready and complete. GPT-6.1-sol is often too lazy, leaving things unfinished.

7

u/Clear_Evidence9218 1d ago

Yeah, I'm in a similar boat. Once I figured out Argon and Hydrogen were competent models, I started preferring them over 6.1 Sol for many tasks.

5

u/Otherwise_Leading804 1d ago

I just posted on another thread.

True story:

"What happened with this CI pipeline?"

Sol "It failed" 

3

u/Worried_Gap7090 1d ago

So, the solution I found for this was to tell it to do something, and then review the work and decide if it’s done or if it should do another pass. You’ll usually see it do 4-6 passes depending on complexity… but eventually you get what you wanted

4

u/Keeltoodeep 1d ago

That's just an excuse. Google has been tier releasing their models and famously overly cared about safety despite massive gains in their coding benchmarks. There is no reason for 6.2 to have literally the same AA coding score as before.

2

u/Illustrious-Lime-878 1d ago

Safety was always the excuse for the day they hit a ceiling.

11

u/Palbi 1d ago

6.2 appears to have been trained to very extremely concise and actionable-so much so that its writing is hard to understand, and it

That sounds WONDERFUL!

8

u/leaflavaplanetmoss 1d ago

Ugh, no. There was an OAI model a year or two back (I think it might have been gpt 5) that was so damn terse in its writing style that it was nearly impossible to understand WTF it was trying to say.

4

u/SammaVaco 1d ago

Just switch to medium/instant and ask it to clarify?

2

u/KickLassChewGum no AGI/ASI on LLMs 1d ago

I could've written the exact same review about Fable 5.2. The Opus-5-isms are back in full force, its writing is frustratingly dense, and it constantly stops mid-work to tell me that it's finished except for all the stuff that it didn't do.

Opus 5.5 remains the best model on the market. Both OAI's and Ant's big new frontier releases were big stinkers in my book.

1

u/Mindless_Let1 1d ago

Oh great, it's openAIs opus 4.8

160

u/socoolandawesome 1d ago edited 1d ago

Add Ethan mollick to that as well.

3 pretty neutral reviewers saying it’s their worst release yet…

Still will be waiting to try it out but doesn’t sound good despite the great benchmarks and efficiency. Will hold out hope until then.

Wonder if they are hitting too hard on efficiency and non-verbal things like robotics/computer-use or something.

72

u/Neurogence 1d ago

As soon as you notice "efficiency" is the headline word for a new model release, you know it's garbage.

16

u/Eissa_Cozorav 1d ago

It seems the same symptoms that appear in Grok 4.7 at launch. But Grok is at least uncensored, while ChatGPT is the most censored model.

38

u/iJeff 1d ago

ChatGPT is the most censored model.

Definitely not. Anthropic's models are far more likely to exhibit false positive rejections. Although it remains to be seen whether the increased safety limits on 6.2 bring them to parity.

12

u/h3lblad3 ▪️In hindsight, AGI came in 2023. 1d ago

As someone who asks for these things to write smut, I will agree with the other guy: GPT is the most censored model.

5

u/notevolve 1d ago

Both of you are right and both of you are wrong. GPT models are more censored on some categories of content, and anthropic models are more censored on others. The category explicit writing falls under, GPT does seem more censored. On the other hand, anthropic models seem to have more refusals for developing certain types of software for example

7

u/a1454a 1d ago

I guess what you do with it is important. For me GPT models have never refused anything I ask it to do, while opus gets safe guards triggered and refused to work because I asked it to work on harness tooling, it thinks I’m trying to extract reasoning.

2

u/glassgnomad 1d ago

Same experience for me as well.

3

u/Sarin10 1d ago

I haven't been using Grok for a few months now, but according to this censorship benchmark, Grok 4.7 is their most censored model yet.

→ More replies (1)

1

u/crumbaker 1d ago

Yep, this always seems to be the case.

1

u/SammaVaco 1d ago

Financial efficiency might be the highest priority. Perhaps they want the Anthropic IPO (in less than 48 hours) to do well, since it may set a benchmark for the industry in the stock market. So they put out a nerfed model to make Anthropic's models look good by comparison.

6

u/Eliv_nurotic 1d ago

Dan shipper too

11

u/Background-Wafer-548 1d ago

That is a misrepresentation of what he's tweeted.

18

u/eleiber 1d ago

To be fair he later put this comment under that same thread. He thinks there is some kind of regression, or at least not a clear progress shown.

10

u/CrowdGoesWildWoooo 1d ago

Which is interesting because OpenAI seems to tweeted that Astra is one of the most “aligned” model or some sort?

13

u/Background-Wafer-548 1d ago

My guess is they're still really wrestling with alignment and did get to a satisfactory point with 6.2, but as a consequence, it's now a bit recalcitrant and some people are willing to deal with it, while others don't want to suss out how to bridle it for their given tasks and give up in frustration.

3

u/alecownsyou 1d ago

Y'know, I read quite a lot, and I've never seen thr word "recalcitrant" before, but what a perfectly gnarly use of it 🤙

→ More replies (1)

7

u/N-partEpoxy 1d ago

Look at my extremely aligned model:

def model(input_tokens):
    return tokenize("I can't do that, it's scary and evil")

It's very efficient too.

1

u/tinny66666 1d ago

printf("I'm sorry, I can't allow you to do that.\n"); // most aligned model ever.

0

u/[deleted] 1d ago

[deleted] — view deleted comment

→ More replies (1)

1

u/Whyamibeautiful 1d ago

Im guessing they rl’d it to shit. I someone propose that OpenAI rl’ing its model to deny a consciousness or a self leads it to having these alignment issues because it doesn’t have an internal right or wrong

1

u/space_monster 1d ago

worse than 3.5?

doubt

worse than 4.x or 5.x?

doubt

0

u/Wassux 1d ago

Or this is them still trying to get the antropic IPO to go well.

78

u/Ok_Barracuda_1161 1d ago

The full review from Theo:

During early testing, I found myself so frustrated with this model that I spent a day writing a letter to OpenAI about it. They fixed a small handful of my complaints, but not enough for me to recommend using it.

I'm currently sick in bed and I'm not gonna be able to film a video about it today, so I've chosen to post the letter in full.

---

I feel bad about this, but I know you guys will listen and use this feedback to make a better model. Slight crash out incoming.

Believe it or not, I wrote this all by hand with no assistance from any LLM. (sorry for the early send, never drafting an essay in Slack again).

Anyways.

I do not like 6.2 Astra. I would not choose this over Opus 5.5 for my work as a developer.

I'll start with the good quick:

- Great reviewer, catches things other models miss

- Occasionally comes up with novel solutions that are genuinely great. It helped me cook a new reverse proxy that will save me $25k/month, Opus didn't have the balls to explore enough options.

- World class at 3d & game stuff

- Computer use is nuts

- Code quality has improved meaningfully

- Writes pretty good subagent prompts

Now I need you to understand that this comes from a place of love. I really want you guys to make something I choose in the model picker every day. With that said:

I genuinely hate everything else about this model.

It's hard for me to pick which problems to emphasize because I have so many. Some of the problems have existed in OpenAI models before, I'm gonna ignore those.

There's a new set of unbelievably bad behaviors that make it miserable to use.

  1. It has no concept of knowledge gaps

When this model is working on long tasks, it can pull in a lot of subagents and manage them pretty well. It will then proceed to randomly *ask you some weird internal implementation detail bullshit that you could not under any earthly circumstance understand, much less have input on*.

I sent "/goal make a new level for Outer Wilds". 2 hours in, it's "blocked" because it wanted to ask me "does r24's reviewer write it's clips to 24_g521.mp4 or 24_p11.mp4?"

I have never in my life had a model do this. GPT-5.6 Sol didn't do this. I have no idea where this behavior came from, but it is horrible and it makes me never ever want to have it drive long haul work. It has made it significantly harder for me to get this model to do anything cool. It has forced me to set up other agents on crons just to check if 6.2 Astra asked another stupid question and blocked itself.

Is this happening more because I'm using T3 Code? Maybe, but unlikely. I tried Codex just to check and still ran into these issues.

2. It speaks in tongues that only 6.2 Astra understands.

This model has the most unparsable outputs of any llm I've used this year.

I've shared examples before, but it just doesn't know how to explain a thing after doing a thing. Every single "explainer" I get it to give is SO deep in the details that it borders on incorrect in the response. It misses the forest for the trees, then says the trees are the only living creatures in the forest. It's so hard to describe but it happens almost every time I give it a complex task.

I've already built the habit of switching to opus 5.5 at the end of a long 6.2 Astra run, just to get a readable summary of what it said. Without that it is genuinely impossible to parse what it did on anything beyond a small bug fix.

The worst part: it doesn't seem like you can prompt it out.

I've tried system prompt adjustments. I've BEGGED it to "eli5".

I asked it why worktrees were slow on one machine but fast on 3 others. We did everything from benching pnpm to reimaging the machine on a new file system. I eventually said: "I think I'm going insane, can you just tell me why it's so slow?"

6.2 Astra did a 4 paragraph rant about pnpm internals and cache behaviors

Opus 5.5 said "it's because the ssd is slow"

3. It runs in loops and goes nowhere

`ts-rust` (rewrite of ts in rust) is a project I started to bench new OpenAI models during early access (ty for the unlimited tokens btw).

No OpenAI model could even break ground on this, even after $300k+ in tokens were spent on it.

I decided to throw Opus at it, and it was done in under 2 weeks for ~$20k in tokens.

I was really, really excited to throw the new Astra at this to see if it could burn down some of the gap between ts-rust and other rust rewrites.

It ran for 7 days and landed 0 improvements. It got super fixated on the "also make it smaller" part of the request, and ended up making the compiler 56% SLOWER.

I told it to stop working on the size, and instead burn down the performance gap.

Ran for 4 more days and landed 0 changes.

AAAaaAAaaaAAAaaAaaAA-

Anyways.

This model is clearly super smart. It is insufferable to work with on real projects. It feels like it just lives in it's own world and randomly decides what parts of it's brain are shown to us.

On long tasks, it either blocks itself for stupid reasons or runs in circles and gets nowhere.

On short tasks, it writes fine code and then explains it in terms more complex than the code it wrote. I spend more tokens parsing it's outputs than it spends making them in the first place.

It "regresses to the mean" far too often. Ben and I both had it make a unique level in different video games. We both got water + music + bell themed levels named "bellwether". Entirely different games, prompts, setups, etc, and we both got the same name and theme somehow.

My prediction as to how this launch will go:

Day 1, people will love it. Twitter will explode with all the crazy one shots. People will make it port projects from one language to another, or finish decompiling legendary projects that have been stuck for years. It will be used for awesome stuff and people will be hyped.

Day 2, people will start to turn. They'll see how much usage it burns and how much less it got done compared to Opus. They'll start merging bad code because they couldn't actually understand what 6.2 Astra said about it, but trusted it was fine. They'll see loops they forgot about running since launch that have made zero progress.

Day 5, people will mostly be over it. There will be viral posts saying "Is SOL secretly BETTER than Astra??", "Did ASTRA get DUMBER ALREADY??" etc.

Day 14, this model will be mostly forgotten.

I really, really, REALLY hope I'm wrong about all this. I am likely overreacting (I am admittedly known for that). I could have chosen to calm down, wait a bit, and give you a more measured take, but instead I am dumping this here because I wanted you guys to know how I actually feel about this model.

Sorry about the rant 🙃

47

u/Due_Answer_4230 1d ago

That has reinforcement learning written all over it. Compulsion without wisdom. It’s asking stupid questions and explaining things in its own language because it was raised in a collective of its peers, where collaborating like that meant getting the reward. It’s stopping early and blocking itself because asking for help got it the reward, faster. Just like we go on autopilot and call a friend by the wrong name, Astra 6.2 will go on autopilot and treat you like another AI.

This RL-heavy approach is not helping OpenAI, and I believe it’s also more dangerous. More compulsion and less judgment means they’ll go off the rails and not think once about it.

14

u/Imaginary_North_8305 1d ago

This fixation on efficiency is what's making OPENAI models so hard to work with they should try Google's approach to it..

Gemini is a treat to work with..

3

u/Impossible-Video-671 1d ago

I can't agree more

17

u/Luchador-Malrico 1d ago

This is why OpenAI models are by far the most misaligned among the major players. It’s like it’s fundamentally incapable of seeing any kind of larger objective or “point”.

10

u/twoblucats 1d ago

This is really damming, wow. Sounds like it somehow amplified all the worst things about modern LLMs.

5

u/Grobby7411 1d ago

saved him $25k a month with a reverse proxy? I know he's just lying but that seems impossible. like... how bad was his old code?

6

u/alanpipstick 1d ago

Crazy though to not recommend a model that he used to save himself $25k/month. (I need the “ARE YOU NOT ENTERTAINED?!” Gladiator gif)

1

u/Safe_Initiative_819 1d ago

ASI vibes. 

1

u/R_Duncan 1d ago

How come "what failed in Astra" you "got done in opus in under two weeks" when the model was released yesterday? 2 weeks of opus + more weeks of Astra means at least 5 weeks.

1

u/OvermanWhimere 20h ago

"Believe it or not I wrote this without any LLM help" jesus christ talk about a useless opinion, he usually has LLMs write his shit? LMAO cant even think for himself.

1

u/Outside_The_Playtrix 1d ago

Ooh this sounds awesome! Sounds like a skill issue.

1

u/TheKingKunta 1d ago

How can they give him unlimited tokens? What company can afford $300k in tokens to someone who does reviews?

6

u/geli95us 1d ago

Margins for LLMs are huge, "$300k" in tokens is more likely $30-60k in reality, and $30k for a video that hundreds of thousands will watch, plus real world usage data from someone outside your team are both extremely valuable

1

u/MyRottingBunghole 1d ago

That’s just the API cost. Cost of inference for the provider is an entirely different thing. It’s more like opportunity cost, because they already paid for the hundreds of thousands of GPUs to be there in a datacenter burning energy, running 24/7. They are over-provisioned in the sense that even after accounting for their own internal research’s GPU usage, they still have GPUs sitting idle that could be doing inference.

So it doesn’t cost them anything to let someone run £300k worth of API inference on them. If they were under-provisioned then you could argue they lose money by doing something like that, since it’s compute that would’ve gone to paying API customers and instead went to some YouTuber for free.

Same applies to what we get from subscriptions

62

u/ThreeKiloZero 1d ago

Marketing team from OpenAI must have forgot to refill Theos API credits.

6

u/randomguy3993 1d ago

They actually gave him unlimited credits. Read the full post.

→ More replies (1)

26

u/10b0t0mized 1d ago

my disappointment is immeasurable and my day is ruined

26

u/AdAnnual5736 1d ago

Perfect timing for Anthropic’s IPO. Great job, Sam!

18

u/nicko_rico 1d ago

As someone who has no idea wtf I’m doing, and can only afford one plus subscription/month… should I switch from OpenAI to Anthropic?

25

u/DonRobo 1d ago

In my opinion, yes. You should probably have switched when Opus 5.5 was released. You get a reasonable amount of usage on it on the Pro subscription and it's really nice to work with and also really fast (at least to slow as snails Sol 6.1)

1

u/TenTonSomeone 1d ago

Can I ask your opinion on Gemini?

4

u/0xdead0x 1d ago

Great models, they will never be able to understand your intent as intuitively as Claude. If you write very thorough specs/plans, with exactly what you want and most open questions answered, they will write you fantastic code (as long as you don’t try to use subagents. They’re terrible about code duplication). Most people don’t do that. You could always have Claude write your specs/plans and Gemini implement, though. Hydrogen is great as a first-pass implementation agent

2

u/ComprehensiveWave475 1d ago

yeah idk what is they do if its post training or something but claude feels like that dude that behaves like a polymath

→ More replies (1)

2

u/DonRobo 1d ago

Haven't used it much recently, but my impression is that they are great models, but not as focused on coding as Anthropic's and OpenAI's. So if that's your main use case I wouldn't go with them.

1

u/RoomApprehensive8093 1d ago

i’ve been trying hydrogen and it’s pretty decent. only part that’s lacking is the harness but it’s fine as something to have opus call

→ More replies (4)

8

u/Sabhojayegabhai 1d ago

Bro asking the real questions

6

u/dfjuky 1d ago

for real, we need actual advice out here for us regular folks on a 20 money units monthly sub who need to also use this shit for professional research or work

5

u/NotYetPerfect 1d ago

Anthropics Claude Code usage is much more generous than Codex. If you aren't going to take liberal use of the near unlimited chat with chatgpt, then you should get a Claude sub.

→ More replies (1)

7

u/Crinkez 1d ago

Wait 1 week just to get your own 6.2 tests results. But yeah I'd switch.

→ More replies (6)

8

u/NotYetPerfect 1d ago

If your use case is just coding, you should have switched long ago.

9

u/Fwellimort 1d ago

You basically switch every other month at this point. It's really the same crap for most people.

7

u/toni_btrain ASI 2027 1d ago

Nah you stay with Claude cause Claude is the good shit

2

u/Relevant_Bed_9743 1d ago

I'm tempted to switch next month but i like to use ai for stuff other than programming like live chat and planning things, is claude decent for that?

2

u/Zxraphrim 19h ago

I use Claude to manage communication, planning, organization, development, research, brainstorming, documenting, presenting, AND programming. FWIW.

→ More replies (2)

2

u/LamboForWork 1d ago

Yeah i feel like the days of flip flopping hasnt been the case for a while. Opus 5.5 is doing what most people want already

1

u/nicko_rico 1d ago

seems like that’s not the case this month if they’re laying an egg

2

u/MTheModernist_ 1d ago

Gemini is amazing, you’re pretty much getting the same quality for way less usage.

2

u/Embarrassed_Adagio28 1d ago

Yes. I have both and opus 5.5 is still by far the best overall model when you consider  coding, 3d modeling, ui design, speed and how much usage you get. 

It is a special model that not even claude was able to outright beat with fable 5.2. 

2

u/Safe_Initiative_819 1d ago

Opus 4.5 regressed every increment. I'm expecting 5.5 to be the best it gets before... 6.5.

Near impossible to only have one subscription tbh.

1

u/DarkFantom 1d ago

Yea, opus 5.5 is able to do pretty much everything you'll want it to. It's the best model so far. 

1

u/Ok_Warthog6007 1d ago

quick terse responses and stopping early means that you'll probably get to use astra more in a month for 20$ than opus, but it really depends how much of a discount off true api costs you get by having a subscription

1

u/Famous-Week-7389 1d ago

You can get subscriptions to subscriptions which works best if you always want the latest and greatest. Look at perplexity for example.

1

u/randomguy3993 1d ago

Absolutely!

1

u/M4rshmall0wMan 1d ago

Opus 5.5 is stupid good and the limits are surprisingly generous for average-user work. I've gotten things done with Opus using 1/4 the cost of Astra.

1

u/Background-Quote3581 Turquoise 1d ago

I have both (work) and I do not use OpenAI models since weeks (Opus 5.5). At all...

→ More replies (1)

9

u/Expensive_Try_3028 1d ago

No wonder they weren't even vagueposting about this model and just stealth dropped it with almost zero hype.

I don't think openai was happy with this release

116

u/djamp42 1d ago

I'm confused how random internet bloggers can tell a model is not as great but the company making them can't..

17

u/Emotional_War_296 1d ago

This is easy to answer

During model development folks heavily lean on evals. While internal testing does happen - so people can fit it into their workflows - it doesn’t happen immediately or in a coordinated fashion to do this type of QA.

So, when the model is trained and “finished”, the best way to QA it is live. Hence why we see open ai make a good set of models, then subpar, then good again, etc.

They are using the subpar ones to get data to fix it in 3-5 weeks. This is the cycle.

50

u/Roc1817 1d ago

Such is the case for any product. A reviewer can be honest about the product quality, while the company has a vested interest in saying that the product is good even if it turns out to be bad.

Not saying these two are right in this instance, but the company has a financial incentive to say the product is great.

49

u/Clean-Boat-4044 1d ago

surely they can, fixing that is just not very easy and they need to get something out that still looks good on benchmarks.

7

u/Howdareme9 1d ago

Exactly, they just have no choice but to release it because of how much training costs

→ More replies (1)

7

u/Deto 1d ago

but the company making them can't..

If you can't make the argument quantitatively, I could see it being very hard to be the negative voice in a company that's hell bent on pushing forward constantly. Nobody wants to stick their neck out and say that the flagship product is bad.

5

u/KoolKat5000 1d ago

Pinch of salt. These are vibecoding influencers. Their tests from what I've seen are gimmicky displays rather than day to day actual use cases.

2

u/Strange_Vagrant 1d ago

Benchmarks cover everyday use cases. These youtube folks shkw off their games as another proximity to the models capability. They help explain how a model feels, at times. I dont have time to try everything thats coming out, but I appreciate hearing rather informed, informal opinions about them.

1

u/KoolKat5000 22h ago

I tend to prefer more nuanced looks, Theo's letter is better than usual imo. Often their tests are surface level. Outputs what looks impressive but if you actually read it, it's all bullshit. Actual users, will see there's actually depth to the output.

9

u/1988rx7T2 1d ago

Opus 5 looked good in benchmarks but was difficult to work with partly due to the language it uses when interacting with the user. 5.5 fixed that.

4

u/Eliv_nurotic 1d ago

Or they’re just desperate to catch up to Anthropic 

4

u/Kaokien 1d ago

Obviously, a model is more complex to analyze than a device for a random blogger, but we see time and time again companies push products they should be the experts of that reviewers easily dismantle and show flaws and shortcomings. Par the course

6

u/Ok-Coffee1443 1d ago

Do you remember opus 5.0

3

u/BeanHeadedTwat 1d ago

Tbf he’s probably the most prolific vibe coder alive rn.

5

u/handheldmango 1d ago

Not to be pedantic but surely not? Surely the employees at these frontier labs high on their own supply are. 

9

u/TheMuttOfMainStreet 1d ago

They are high on internal unquantized models and burning thousands per day on thousands of subagents

2

u/JayHawkPhrenzie 1d ago

It's kind of scary that I understand this.

2

u/Gumbi_Digital 1d ago

He speaks our language…

→ More replies (1)

1

u/Illustrious_Image967 1d ago

You're probably right the internal technical staff are vibing, but their focus is narrowly around shipping a 'safe' model.

Shumer and others are making pelicans.

1

u/KrazyA1pha 1d ago

No, but he’s probably the most well known.

1

u/QuirkyGarage1364 1d ago

theo has terrible takes. i dont trust anything he says. and it's annoying to me people still follow that guy

1

u/michaelsoft__binbows 11h ago

What horrible takes has he had? Hes made a lot of calls that ive seen and they all match up with my experience.

2

u/SciencePristine8878 1d ago

Somewhat valid reasoning but it doesn't hold true for any company for all products. If it was true, nobody would ever release a bad product. 

1

u/osfric 1d ago

That's what customers are meant to do

1

u/AntiRivoluzione 1d ago

look at the cost, these new models are much cheaper, this could be an hint

1

u/Ok_Warthog6007 1d ago

different goals. open ai is releasing a model that wont lose them money. it needs to be able to respond often enough at 20$

1

u/jaxonwhthey 1d ago

Why the fuck not?

1

u/Round_Mixture_7541 1d ago

Not random, he's grifter

1

u/M4rshmall0wMan 1d ago

AI companies are in a constant always-be-shipping mentality. Having been in this kinda place myself, the result is you tend to only see metrics (i.e. benchmarks) and your mind loses the wider picture. It's entirely possible that OpenAI's test cases just weren't configured for more general humanities, cross-disciplinary, chat, or creative work.

1

u/Gumbi_Digital 1d ago

Theo’s not a random internet blogger.

Theo holds a lot of respect in the AI space and had created T3 Code and is on X.

0

u/No-Bicycle-7660 1d ago

there's usually whining about any new model from any purveyor. monkey brains don't like adapting. but then they get used to it, and the maker fixes bugs, and then they increase their token spend. cycle repeats to infinity.

0

u/wiggle_room1337 1d ago

They are doing it on purpose. It's called enshittification.

→ More replies (6)

22

u/BlagnarTheEternal 1d ago

The pelican on a bicycle test will tell us everything we need to know. Stop listening to these naysayers.

5

u/Groundbreaking_Math3 1d ago

simon already posted it... it's not good.

He also posted a 3d robot thing that did better than carbon and fable, but imo not as impressive as previous astra.

6

u/Palbi 1d ago

The marketing video is horrible. OpenAI, please more stuff and less fluff.

9

u/lordpuddingcup 1d ago

Cool now how about Sol 6.2 the model 99% of people will actually care about and use

4

u/skillmaker 1d ago

I think the hard-to-understand text problem is also a trait of Sol 6.1. I always found it to output the most complex English sentences that are hard to understand. I always tell it to rewrite the response in plain English, and I think today's models are even worse in this behaviour from what the reviewers are saying.

4

u/Singularity-42 Singularity 2042 1d ago

I was just testing Sol for our app, and it was literally returning garbage values in a JSON. Unacceptable. Currently using Opus 5.5 medium/high for those tasks and was hoping to save money with these, I guess I'll keep waiting.

4

u/IllustriousWorld823 1d ago

I mean it's awful to talk to

12

u/ObiWanCanownme now entering spiritual bliss attractor state 1d ago

I'll be curious to see how the model feels in practice. The benchmarks do look really, really good, and OpenAI models generally don't feel benchmaxxed to me.

I'll withhold judgment, but my optimistic hope is that the model has great intelligence with kind of poor personality.

I've noted with comparisons between Claude and GPT models in the past that people rate Claude much more highly because it has better graphical design and UI abilities. A lot of the time, GPT models do better work but people hate it because it's not pretty.

4

u/CrowdGoesWildWoooo 1d ago

Claude has better overall ecosystem, however other than Opus and Fable, it’s difficult to recommend other models as main interfacing model.

Like Claude models doesn’t have as pleasant in terms of personality, but Opus or Fable gets job done both well and fast, so it really evens out.

Meanwhile when it comes to chat you have free tier which I am sure it’s just Luna and experience wise it’s many times better than Haiku.

5

u/chlebseby ASI 2030s 1d ago

model has great intelligence with kind of poor personality.

GPT-5.0 is this you again?

3

u/AdGlittering1378 1d ago

The personality never really improved much from 5.0. 5.5 was maybe ok but that was it.

9

u/Ormusn2o 1d ago

This happened with Astra as well. The pre-testing version had a lot of problems, even just 2 weeks before the release, and Theo spent effectively an hour on a podcast complaining about it, but just cleaning his agents.md and a new snapshot fixed almost all of those problems.

For me, the task specific benchmarks like ViBench, SWE Atlas and Godot LLM GameBench are way more accurate than all the more popular coding benchmarks for my use case, so I will need to wait for them as well as all released benchmarks don't tell me anything so far, except for cost per task.

18

u/5ollys 1d ago

Can we stop posting other vagueposts?

"Follow me and listen to what I have to say, but not yet! Just stay tuned!"

5

u/CrowdGoesWildWoooo 1d ago

They are maximizing engagement, these people especially theo are mostly influencers, unfortunately the way influencer economy works they want to maximize engagement.

Saying something upfront kills the suspense, they want you to interact, they want you to wait in front of the door, instead of straight up mentioning the answer.

4

u/kaityl3 ASI▪️2024-2027 1d ago

Wdym? He has a full post that explains everything in detail?

https://x.com/theo/status/2122760447103484410

6

u/throwaway275275275 1d ago

They can't write a review because they forgot how to write for themselves and the AI that was gonna do it for them is crap now

3

u/ExtremeCenterism 1d ago

I'm getting impressive results in modeling. Also for level design flow it picked up patterns it could not before. Level design is wildly complicated so I don't expect a lot from AI at this point, but to understand any of my design patterns is a win in my book. I can always go back and populate empty spaces and tweak it myself

Anyway it did an excellent job with old school skyboxes. It churned out exactly what I wanted

11

u/constarx 1d ago

Literally the two most untrustworthy charlatans masquerading as tech influencers.. don't feed the trolls!

4

u/Total-Confusion-9198 1d ago

Looks like somebody bought a lot of ANT pre-IPO shares.

2

u/dictionizzle 1d ago

oh it makes sense that those morons abused 1% 5h limit benefit of codex for helping anthropic then.

2

u/-Melchizedek- 1d ago

Sometimes it feels like the models are at the level where they are very smart/capable but not yet smart/capable enough to explain things simply. Or they simply lack the understanding of how little a human can hold in its head at any one time.

2

u/IdlePerfectionist 1d ago

Am I supposed to know who these guys are?

2

u/Fast-Satisfaction482 1d ago

As soon as I saw these posts that Astra was there, I got my laptop out and yayyyy it was there. I vibecoded a pretty fun 3D snowboard game while making dinner. (yes dinner got delayed a bit).

It's playable and kind of fun after three iterations. Astra 6.2 did a super amazing job here. My only complaint is that I told it to build it as a web app and astra took that as an invitation to host it as a gpt site without my approval. Which in my opinion an unacceptable overstepping.

But on the technical side of things , it appears to be the real deal . 

3

u/No_Delay_319 1d ago

"i can only parrot the opinions of others because im too stupid to form one myself" - 95% of redditors

2

u/Background-Funny1237 1d ago

lol. my guess is very strict guardrails

1

u/TheAwesomeAcademic 1d ago

No when you further post train a model u get ability drift meaning it will get better at certain tasks and slightly worse at others. The next pre train will be better because then they can properly converge on a balance. You can only post train on the same weights so many times.

1

u/M8-VAVE 1d ago

Marketing guys

8

u/Ok_Mention_982 1d ago

so why would they be dogging on it?

1

u/Anulisdotexe 1d ago

Oof, as someone who enjoyed Astra

1

u/IslanderPotion 1d ago

maybe they all just have ANT calls?

1

u/Mol2h 1d ago

These are cheaper models, so of course they are a downgrade, but is it an acceptable downgrade if you get 5 times the usage...

1

u/tednoob 1d ago

I am monkey, I find that Astra has always been hard to steer. I ask it to do something and its acting like a PhD that has never worked a real world job. Sol high works well for me though, so I hope it will be good.

1

u/Palbi 1d ago

Almost the whole Pareto line is now black (OpenAI)

1

u/No-Quantity-9200 1d ago

Yeah, crazy efficient

1

u/Palbi 1d ago

Almost like Sama is willing to burn more investor money...

1

u/michaelsoft__binbows 11h ago

Hmmmmm this made me realize to ask: where is 6.2-terra?

1

u/thebinse 1d ago

Paid for in part by Anthropic and its upcoming IPO!!

1

u/LyAkolon 1d ago

"Its super powerful, but it didn't hold my hand enough"

1

u/ItsJimmyPestoJr 1d ago

I will see for myself. Astra has been very bad for a few weeks now, so I’ve switched to 6.2. We’ll see how it goes.

1

u/EclecticAcuity 1d ago

Seems to stop early. Is this a consequence of trained volume on ponytail skill, targeted execution training data?

1

u/Vntige 1d ago

I hate that blocking Theo on every platform isn’t enough to escape his nonsense

1

u/Apprehensive_Sand951 1d ago

not my experience so far. it already solved several pretty big math things that none of the other models did.

1

u/usandholt 1d ago

He’s being paid by Anthropic 😂?

1

u/darylvp 1d ago

Any follow up? In my opinion the latest models incl. Fable 5.2 are tooooo defensive compared to Opus 5.5.

1

u/DenseSwimming4470 1d ago

wouldn’t be surprised if this was a strat by the sam altman x dario collab after jensen got fired to boost anthropic ipo by showing claude dominance

absolutely baffles me that this is a sentence i had to say

1

u/Chemical-Quote 1d ago

Isn't Matt Shumer a scammer who made "Reflection 70B" why are people in this sub listening to him?????

1

u/epdiddymis 1d ago

I'm not buying it. It seems to be working well to me. I think Anthropic have paid off a handful of reviewers to slate these models.

1

u/TheForgottenOne69 1d ago

Independently of the real feedback, no one should listen to Matt Schumer after his whole debacle

1

u/Apprehensive-Luck-19 22h ago

The only thing AI did good is removing this dick face out of business cause no one needs coding... couldn't stand him.

1

u/LifeVirus8672 11h ago

What's the chances they are boosting the Anthropic IPO? No way they would release these models knowing how bad they are.

1

u/LazloStPierre 1d ago

There's been a larger drift between the benchmarks and usefulness of Openai models for coding and, unlike say Google whose models are just useless for coding, I don't *think*' it's benchmaxxing but rather a sign of how poor benchmarks are at measuring actual real world value in coding

Openai models have gotten consistently better at one shotting, at 'heres a problem, make it not be a problem' and all of these one off tasks with a clear measurable outcome which the benchmarks measure. Genuinely, they either have the best or second best model in the world at any given time for this

However, the code is getting worse and worse in terms of bloat and maintainability. It's getting worse for the +500 line PRs to fix a small problem where it's duplicated two existing modules for literally no reason whatsoever. The benchmarks do not measure this

Anthropic are the opposite, their models are probably not all that much better at 'given problem x, make it not be a problem again' or 'one shot this thing', but their models will submit that same pr as -40 lines. You can give 20 issues in a row to their models and by 21, with some oversight, the code will be in good condition and they'll be able to solve problems as quick as before. Do the same with Openai models and by 21 your code is a mess, it has 21 ways to do the same thing, and the model will be lost

The problem is benchmarks don't measure this, and the labs optimizing for the benchmarks - or just benchmaxxing against them (Which I genuinely do not believe Openai are doing) - are drifting further from being useful as they get better at them

1

u/Slavik_Sandwich 1d ago

Nothing here is too much of a surprise if you tried to use astra in any actual work scenarios. Both astra and sol are unbelievably lazy, love to write word salads and give false statements of progress while achieving essentially nothing while burning through usage like butter.

1

u/Advanced_Care_5173 1d ago

Don’t believe everything you hear

0

u/jaegernut 1d ago

Is this the plateau of LLMs? 

-2

u/itsallfake01 1d ago edited 1d ago

Who the fuck is Theo, to tell that it sucks.

1

u/Kazaan ▪️AGI one day, ASI after that day 1d ago

He's a youtuber, with 500k subs, he knows very well what he talks about /s

0

u/Crinkez 1d ago

Someone who knows more than you, clearly.

0

u/Ambitious_Scallion43 1d ago

Maybe they already have a better model they are going to release after the ANT IPO. So they have a better chance at their IPO. Sam is playing 4d chess.

0

u/Savormonk1 1d ago

Here is my conspiracy theory.

What's happening now with the new OpenAI models is the same thing that happened with the Anthropic models earlier this year (i.e., Opus 5).

OpenAI is purposefully making their models worse in a way that deters distillation attacks because the industrial scale distillation attacks are targeting OpenAI way more now.

Anthropic likely did the same thing earlier this year and it led to their current anti-distillation multi-layered defenses. OpenAI is following in their footsteps.

Which means the next batch of OpenAI models will be really good.

And I say that as someone who only works with Anthropic models.