Grok 4 in 6 Minutes

There is this saying that the moment you have a test or benchmark, that moment that test is obsolete. Unless you've been living under a rock, you'll have known that Grock 4 has basically just blasted out onto the scene as the most intelligent model and they have their own statistics to back that claim up. And this is the first time we've seen equal amounts spent on reinforced learning as pre-training. Now reinforced learning is sort of like with verifiable rewards is basically saying we'll give you a reward if you get this right which is quite funny but um Elon Musk during this live stream has said that this is PhD level thinking if not better under exam conditions now we've spoken in the past you know whether writing essays and things like that are a valid way to to justify someone's intelligence we don't know these are the scores it got so it got 41% on math it got 9% off his and these these with humanity's last exam which sounds so scary for some reason to me other models struggle to get double digits on these things but I mean look at math it's like 41 10% on artificial intelligence chemistry other I don't know whether what other entails but physics you know so this is something that scored really really well on now this was a three-step reveal that old Elon Musk with no tool calls it it got to 26 6% I think this was already ahead of the other models but then when it gave it access to tool calls which apparently has been trained on tool calls so it knows how to handle them it then achieved a 41% in the uh scaling hle tools then they revealed test time compute this is when they gave it basically more time and this to me is what was tree quest they employed the tree here where they had multiple agents running together, working together over longer periods, it was able to achieve 50. I can't prove that. I don't know. But this is my hunch. This is just putting two and two together. Oh, it did some predictions on stats on on who won the who would win the World Series, things like that. Simulations on black holes. Here's humanity's last exam, which is a very very very hard exam. scored 100% on the ame 25 which is insane. It's just beating pretty much every other model at everything except for speed and price will Yes. Well, I've used it today already and it's it's it's quite slow. It's something to go by and there's even word to say that they potentially optimize specifically for benchmarks. Let's talk about the price. 30 bucks a month for Super Grock and then Supergro heavy which is Gro 4 heavy model 300 bucks a month and then the yearly price is 300 bucks for the year or 3,000 bucks for super Grock heavy. So it's a very pricey model I think totally worth it if you're if you're doing academic research scientific things like that. It's available in cursor so you can actually play with it in code. They create their own benchmark for like operating a vending machine. What is the point of this? I think it's to do with how it sort of manipulates the the how much stock it has and how much it's basically able to get at the end of it and how much it's able to sell because you've got units sold here. Human was only able to sell 344 units. Claude Opus, 140 uh 1,412 and then Croc 4 got this. So, it's their own little benchmark that they did which is just quite funny, quite interesting. Uh oh, for gaming as well. So, they made this someone made this G vibe game this 3D game which is really really cool. And then this is the release cycle. So we've just hit this now. Grock 4 release. They're going to release a coding model in August. Then a multimodal agent and then a video generation model by October. So we've got a bit of a timeline to sort of wait. Only has a 256 context window. 256,000 context window. And this is not the be all and end all. And if anything, it teaches you that we need to be more efficient with the models. Can't just chuck everything at it. That these million because Gemini is a million, you know, over a million. In one other benchmark they are also really powerful the snitch bench if it will snitch on you this they got a perfect score 100%. So if if you do anything um that you know it it's illegal or it thinks that it's illegal it will try to contact the authorities and snitch you to them. I did see on a video somewhere that they tried to get it to design some UI and it was not you know still not there from the UI perspective unfortunately. Let's talk about this. So there is ARC AGI another benchmark to measure these LLMs to see how smart an actual you know AI is compared to a real human. And these tests in in the first you know look they are very simple. they are very simple for human to solve them but apparently very difficult for AI models. So what's really interesting is that uh other models are getting I think between 1 to 8% while humans get 90 plus% and Grock is getting what I think 15 or 16% which is doubling any other uh model. It's going to be also interesting to see what if AI can hit 100%. And what happens if we still find out AIs are not the real AGI, they are not truly that super smart being. So will we have other type of tests? Because there is this saying that the moment you have a test or benchmark that moment that test is obsolete. I would be interested to know how does that work? How can they be sure that the data for this is not in the learning material for the AI? This is part of a larger conversation on my show, Command AI, which we stream live every single week. We discuss the news and all things related to AI in the world of design and web. Catch us next week and join in the banter.