Did Cloudflare Just Undermine Their Entire Product?

Cloudflare. Ah, that that one was the memes command AI. So, Cloudflare um have released a new crawl endpoint that lets you crawl entire websites in a single API call which you can return HTML, you can return markdown, you can actually return structured JSON on the website. So, you know, I'm guessing that would be like heading uh headers and paragraphs and things like this. Let me show you the nerdy stuff first before we get to the memes. Well, you'd start inside of Cloudflare and you'd create an API key. Uh, obviously, I can't show you that cuz I don't want to reveal my API key, but you you create an API key with browser rendering permissions, right? So just log into Cloudflare, create an API key, and then you can literally jump into your terminal and run this. And what this does, curl request is just a request. Um, under the hood, a browser is basically doing a curl request. Um, and then you are sending a post request to this URL and then you put in your account ID there and your API. Um, and then your yeah your API code token in a sec. And then that's the URL. Oh, okay. And then you pass it authorization header with bearer. And there's your API token application JSON. And then the data you send is the URL that you want to crawl. So I put my company website here. I put my API key here. And I put my account ID here. Hit enter. It takes a little bit. And it then returns a um a an ID. And then you just say you get you put that job ID that you get back there. You put your account ID there and you put your API key token there. And that would just that just dumps you a ton of HTML in your terminal. It's all in your terminal. But then you start to pass it some particular bits of data um such as JSON options. You can even prompt it. Look, it's an AI endpoint. So you can prompt it. get me product name, price, description, and availability. And it will just do that. It will just return that into the a into the uh interesting API. So, this is a cleaner way of doing it. It might be a great way to do um website migrations. I'll tell you that. Or extracting the data from one website to input in another one. I I wanted to ask like this is also going to be great if you want to create some like health checking tools that that would grab the content of a website and then you could run some tests. You could do a lot. It's it's a handy bloody tool, but I think as you're going to get into it's the memes that are quite funny. So yeah, like get give it to me in HTML HTML and markdown. Um you can even authenticate uh all of this. So it's really useful. Yeah. But I think it's the it's the manner in which it was done. I think is the funny thing, right? Yeah. I I don't know if it's that being the issue or just people Well, I've got this. If I I'll show you this tweet that I found. Cloudflare spent years fighting scrapers and selling anti-brot protection. Now, they've launched their crawl API that can scrape an entire website with one request, sending a single AP. So I could I could crawl your website and I could do whatever I want. I could prompt crawl your website with this. It's not like it protect like it's not like it protects you or anything like that or or that you can only crawl websites that you have hosted on cloudare any website any website. So very interesting and it's a very different approach from a company that its whole entire job is to protect websites. Yeah. And prevent crawling. So it's just quite ironic there really. Yeah. It's this uh it's this meme, you know it, you either die a hero. Yeah. Or live long enough to see yourself become the Yeah. Exactly. Uh Yeah. So, but the memes aside, but I find it really useful for potentially like doing client research. Uh, like you you get a new lead or you want to reach out to somebody. Uh, you want to build an outreach tool. Maybe somebody maybe wants to build an outreach tool for themsel and maybe I can dump a bunch of websites that I found a bunch of leads with their websites and I can run some automated uh tests and yeah even the results that I get I could use them to write them the outreach materials like to to kind of like okay I understand what this tool does um I I see what the issues are. Yeah, that that could be very handy for that and as you mentioned to migrate content to Yeah, this is part of a larger conversation on my show Command AI which we stream live every single week. We discuss the news and all things related to AI in the world of design and web. Catch us next week and join in the banter. Command AI.