Most Creative Use of Depth Maps to Date

Unless you've been living under a rock, you probably have seen this video. It has almost 10 million views and look Command AI I saw it and then immediately I was like Oh, holy [ __ ] is this real time? And then I read it, it is real time. This is happening real time. But That light is not there. Quite frankly. And through some sort of process or whatever it's able to determine not only the shadows Well, first but the intensity of the light in the various areas in which it's being cast upon including blocking it completely from his hand. So, to do all this in real time is is computationally very very cool. Yeah, not to mention this is a 2D image. It's a video but it's at the end of the day it's an image. And we are like casting a 3D uh light at a 3D feature actually into a two at a 2D image. So, we have to determine depth. We have to determine okay, what is the closest to the camera? What is the further furthest away from it? Kind of like a Z index. And then based on that we can spread the light into in a fake 3D space. What's really crazy, this is done locally and in real time. So, depth aware light injection in type GPU, we have covered it in a really like past video. I got a 448 um by 448 monocolor depth model. I'll show you what that means. I will just show you in a second. Depth model down to 8 milliseconds on my M4 Pro across 250 uh dispatches. I'm not sure what this means actually. To be to be I think this is like the intensity or like the layers. I will show it and maybe we can determine what that mean. Uh which is fast enough to use uh in real time since No, actually that's not the layers. Anyway, uh to use in real time since the interface is written directly in type GPU, I can just feed the depth buffer straight into uh the lighting pass. It never has to leave the GPU or go through any extra synchronization uh or interrupt step. So, this is really cool and I saw another video just just like that, which is really cool like right? Uh and the way it works, I can show you what I have, which is not real time and not for the same purpose, but similarly you see I have like this seemingly 3D um model. It's not exactly 3D. It's actually it was one image. I did cut it in multiple pieces, but it's essentially a depth map. Well, multiple depth maps like the drink is also different depth map. And where we have the light, the the white, it's the closest to the camera and dark is the further away from it. And based at that information, we can do all sorts of stuff. We can uh distribute light intensity. We can also create animations like what I have here to create this fake effect where we are manipulating the pixels are that are closest to the camera more compared to the pixels that are more further away. So, here you see the drink looks like it's moving because the pixels are closer to the camera and I'm moving them more compared to the other pixels. And it looks also like a bit 3D. So, fake 3D. This is a a model, a machine learning model that you can run locally made by Apple. You feed it an image and it gives you another image that is a depth map essentially and you basically put these on top of one another and frankly, I didn't do any of this. I just I was aware of this and I just told my Claude, "Hey Fable, would you do this for me?" So, basically Fable did this. And what this guy has done is much more impressive, not to compare. I'm just showing what I was like actually working on and mine is, you know, not nearly as impressive, especially given that this is run um in real time and entirely on the GPU level and yeah, it's really cool. People see impressive stuff and yeah, it they click. It's really cool. Cuz I I don't think it's immediately obvious what how the depth map translates to this light image and that's the creative side, right? They've they've taken something as a view. I'm I'm guessing that the majority of this parallax effect is actually more so GSAP moving the individual PNG files. Um Well, I can turn that off. Yes. Yeah. Yes. But then you're also using a you're having a layer of depth maps, right? Yeah. the depth map isn't responsible for the parallax effect. Is that Is that right? Um the parallax is done in two ways, both with the depth map and with having the layers in separate layers. You could never achieve the depth the the extent of that 3D though, just with the depth map. You could probably cheat a little bit. This is a good example. So, I can definitely push it further, but you already see some artifacts here. It's hard to see it, but now you see it. If I get the further away, you begin to see some jagged corners because you cannot like cut the image and you know, under it you need to have a you need to have something under it. In the example that I showed here, where I'm cutting things, I'm actually regenerating the image. So, the sky is not, you know, empty where the model exists. The sky has been regenerated after that. So, all of these pieces are regenerated. In this example, this is one simple image, one Claude prompt saying, "Hey, use this Apple Depth Pro to to to generate a depth map and then animate based on a mouse hover interaction to create a 3D effect in addition to a light uh source uh reflecting on the image on when the mouse moves over the image. So, essentially something like that. And that's the result you see. It is already like giving a really nice vibe, but it's nowhere near this depth, you know? Right. So, this is what Claude would have done under the hood, which is go to this GitHub repo and clone it. Yes. It creates a little environment just to kind of make sure everything's encapsulated, and then it runs that to download the models. So, I wonder, get pre-trained models will be a Here it is. I wonder what the model is. Make directory checkpoints get ml site. So, let's just copy that. Interesting. What's that? Oh, no, it's just downloading now. So, this is actually the model. Okay, so this is a PT How big is the model? Dot uh 1.8 gig. Sorry, I can't zoom in anymore. not too bad. 1.8 gig. So, that's the model. Not too Yeah. So, once all that Once you've got the model, once you've you know, downloaded the download everything, you just run depth pro run. And I is input, so that means input, and then you just point it to an image. Yeah. drag an image into your terminal, and then then output it will just export an an image that's black and or gray and white. Um yeah, using that model. Or, you can even build applications with it. Like, you can even work it into your application. So, you could, in theory, well, no, you can, build like a nice little UI, like a web app that you just drag an image in, and it'll just pop it out. So, this this is what this is for. But, of course, 100%. 2026, you just ask your agent to do all of this. But, it's just nice to know, and I think it's helpful to understand what's going on uh from a code perspective. I think it just you know, I know we're asking agents everything, but it's also just nice to know that you're you're you're along for the ride. It's not It's not this [ __ ] big wolf that you're taking out for a walk, and it's just like like dragging you down the street, and you don't know where you're going or what's going on. It's just nice to know what's happening. So, pretty simple stuff there. Do you Do you know how long it took for you to fast. It's quite fast. 1.8 gig is not much. Yeah, I think it's done in matters of seconds, if like not minutes. It's seconds. And I did talk to Claude about it. I was like, "What configurations do I have? Like, can I Can I add a prompt and be like, 'Make the face to you you know, more bold, or like, you know, add more light, essentially make it more white, so it's it looks closer.'" And I Claude was Claude was like, "Nope, you just feed it an image, and it returns you an image. There is nothing you configure. It has a specific a range of pixels that it can support, so the image should not be too big or it will downscale it automatically. And that's it. Kudos to Apple, you know? They open source it. Yeah. They are quite sneaky. Again, this is probably something they've used it's 1.8 gig, right? They probably use it on your on your probably already got it on your phone, you know, it's able to if you've got a Well, they do that. They do that all the time. Yeah. Did you know that? Yeah, yeah, yeah. So By the way, shout out I took this image in London, you know, when where we met. Exactly, there we go. Yeah, exactly. So, that image has a depth and that is not leaving my phone. This is happening on on my phone. Apple looks at my images and thinks, "Okay, this one can have depth." So, I'm not sure if you can see it the top that little button. Oh, yeah, that too. Yeah. So, it's that probably it. That's probably it there. So, now I can just twist that and you can kind of see it doing it, I think. I'm not sure. see it. Yeah, it's less pronounced there, but yeah. Yeah. So, that's probably it. you have an iPhone if you have an iPhone take a picture of your friend showing you the middle finger when the hand is extended and then do this thing and you will see it looks very uh 3D. But I think it can only work on iPhone 15 upwards. This is part of a larger conversation on my show Command AI, which we stream live every single week. We discuss the news and all things related to AI in the world of design and web. Catch us next week and join in the banter. Command AI