GPT-5.2 and the Difference Between Benchmarks and Real-World Usage

Open AI have one GPT 5.2 thinking as scoring 100%. And by the way guys, benchmarks are the bane of all of our existence. It's all comes down to real world usage and what you get on with. But it's hard to ignore. The one that I find hard to ignore is the SWE overtaking Opus on the SWE bench. Now I've been using ChachiBT 5.2 to today and my god is it slow through the API or where through the API I set it on a pretty hard task and cuz here's the thing we talked about this last week uh when I and I talked about it in my video about running local models right a lot of these benchmarks a lot of these oneshot things they throw in a single page app and but once you put it into a code base where it's got to read loads of context that's when they start to to falter Saying all that, I gave the GBT 5.2 a task where it was just like, "Go do this one little task. Don't worry about the rest of the codebase." Honestly, I fixed about four or five bugs using Opus in the time that it was running and running and running and then it still didn't fix it. Still didn't fix the bug that it was trying to to figure out. So, you know, it is what it is. But the real the real question which we've been talking about recently is like does it does it actually even matter the everyday user a f I think they're not really see people are still stuck on GPT40 they want 40 back they don't care they're not caring about tone was better the tone yeah exactly right um what makes a what makes a code what makes a model good is not these benchmarks and not these numbers it's the real world usage and people still were more than happy a 40. Um, so whilst open AAI are barking up this tree of like AGI and the best model, people are still they're like, don't we have a is it like a saying or something along like the line uh the lines when when the moment you start to have a test or like a benchmark, it's obsolete. Well, yeah. I mean that you know there's there's an argument to say that um you know they're optimizing specifically for these benchmarks but the thing is I see it all over YouTube you know currently it's like people us you I the viewers all of us we look for that one magic pill that just tells us whether something's good or bad when we we've sort of lost the ability to even consider that actually things need a little bit of work a little bit of time a little bit of adjustment but we're we're just we're addicted to that oneshot prompt. We're addicted to those sort of bombastic headlines to say this is, you know, the best or this is the worst or whatever. We we're losing that ability for nuance, but we've lost it completely. Do you know what I mean? Um, but yeah, there is an argument to say that the they are they are optimizing for these benchmarks cuz I still showed you the benchmarks even though I have this opinion that they don't even matter. So, have you done any of the benchmarks yourself? The RKGI ones? No. No. No. Like there are games you can you can play them and there there are like no instructions so you have to do something and I'm not going to spoil it. I've done like a few of them. Um and they can be tricky sometimes but what's your score? 20% less than JPT. Now I don't know. I don't know. I I never got a score. You just get or I maybe I haven't done enough of it but like you you go to the game and there are like levels. So level one is simple and then you figure it out and then you you advance the levels. Um but you get a feeling for it like may maybe that is the human part like okay so I need to rotate the block or I need to to to interact with that color but not with that color. So it's that type of things and I I I guess for AI it takes many runs that that's that's what I what my understanding is. It takes many runs to get all that necessary information. Okay, this is how I interact with this game. Um when while for us humans it's probably just order of magnitude easier because we are very efficient. Right. Right. We're like our brains are like incredibly efficient and the amount of data that we take uh visually and through sound and everything else is just like insanely uh huge and the computation that we do at the same time. So what what is uh interesting also about these um models getting better at these um AGI at at these like uh what what are they called again? the uh the RKGI tests uh is the cost because it used to take like thousands and thousands of dollars to just do one test and now it's just uh much much less I believe uh it was like Theo mentioned like $11 or something like this something that was in thousands. Well, funny you say that cuz I wanted to just raise the price real quick. Uh it's $175 in $14 out. So, it's it's gone up in price. Uh, you mentioned Theo. Um, he like he he released it and I was in the comment section with people and people are like tired. People are like, "Oh, like whatever." Like, you know, and and on the one hand, I get it and it I said it surprises me because we're at the cutting edge of technology. We're supposed to be excited about new technology and look at how stagnant Apple was. And I know, you know, comparing apples and oranges potentially, but when Apple just weren't releasing any meaningful updates to any of their computers and then people complained about that cuz there were no updates. Like I I for sure think we should celebrate any and all improvements to technology. However, in the name of crossing over a thousand subscribers on our on our YouTube, right, what do you want to hear about these new models? like do you you know balancing that we should be celebrating progress and we do want things to get better. Um rather than just kind of like dismissing it cuz it's just you're not the new model. I would like to make these sections more interesting and more kind of like you know maybe it's more anecdotes. Maybe you just don't want to hear about them at all cuz cuz whatever. One guy was like wake me when when GBT7 is out sort of thing. And on the one hand, I get it, but on the other hand, it's like any progress on any any progress is it should be celebrated. I don't know. Let us know what you want to hear about these new models. GBT 5.2, I'm here for it, but not really going to affect my dayto-day. Uh, especially in coding. Uh, I don't know. I don't know what what what they're going for at this point because the normies are happy with 40. So, yeah. Let us know if what you're doing with chat GBT, whether you use it for anything kind of cool, but they're just they're just a meh company for me. This is part of a larger conversation on my show, Command AI, which we stream live every single week. We discuss the news and all things related to AI in the world of design and web. Catch us next week and join in the banter.