When a foreground object is moved, how are the newly visible contents of the background filled?
The demos show either totally internal modifications (bouncing blanket changing shape / statue cheeks changing) or isolated with white background images that have been clipped out. Based on the description of how they generate the splats, I think you’d auto select the item out of the background, do this with it, then paste it back.
The splatting process uses a pretty interesting idea, which is to imagine two cameras, one the current “view” of the image, the other one 180 degrees opposite looking back, but at a “flat” mirror image of the front. This is going to constrain the splats away from having weird rando shapes. You will emphatically not get the ability to rotate something a long a vertical axis here, (e.g. “let me just see a little more of that statue’s other side”). You will instead get a nice method to deform / rearrange.
It probably isn't.
The most logical use of this is to replace mesh-transform tools in Photoshop or Adobe Illustrator. In this case, you probably work with a transparent map anyway.
Now THIS is the kind of shit I signed up for when AI started to become able to understand images properly: no shitty prompt-based generators that puke the most generalised version of every motif while draining the whole illustration industry from life.
It's just good-ass tooling for making cool-ass art. Hell yes! Finally, there is some useful AI tooling that empowers artistic creativity rather than drains it.
Pardon the French; I just think this is too awesome for normal words.
There's a ton of room for using today's ML techniques to greatly simplify photo editing. The problem is, these are not billion dollar ideas. You're not gonna raise a lot of money at crazy valuations by proposing to build a tool for relighting scenes or removing unwanted to objects from a photo. Especially since there is a good chance that Google, Apple, or Adobe are going to just borrow your idea if it pans out.
On the other hand, you can raise a lot of money if you promise to render an entire industry or an entire class of human labor obsolete.
The end result is that far fewer people are working on ML-based dust or noise removal than on tools that are generating made-up images or videos from scratch.
Yep, there's a similar refrain amongst 3D artists who are begging for AI tools which can effectively speed up the tedious parts of their current process like retopo and UV unwrapping, but all AI researchers keep giving them are tools which take a text prompt or image and try to automate their entire process from start to finish, with very little control and invariably low quality results.
There have been some really nice AI tools to generate bump and diffusion maps from photos. So you could photograph a wall and get a detailed meshing texture with good light scatter and depth.
That's the kind of awesome tech that got me into AI in the first place. But then prompt generators took over everything.
Denoising is another good practical application of AI in 3D, you can save a lot of time without giving up control by rendering an almost noise-free image and then using a neural network to clean it up, rather than brute forcing a clean image. Intel did some good work there with their open source OIDN library, but then genAI took over and now all the research focus is on trying to completely replace precise 3D rendering workflows with stochastic diffusion slot machines, rather than continuing to develop smarter denoisers.
Your fault for poor prompting. If you don't provide distinctive prompts you can expect generalised answers
Let's say you want to rotate a cat's head in an existing picture by 5 degrees, as in the most basic example suggested here. No prompt will reliably do that.
A mesh-transform tool and some brush touchups could. Or this tool could. Diffusion models are too uncontrollable, even in the most basic examples, to be meaningfully useful for artists.
This is honestly genius. If I understand it correctly, instead of manipulating pixels, you turn any 2D image to a 3D model and then manipulate that model.
Yes! This really feels next-gen. After all, you're not actually interested in editing the 2D image, that's just an array of pixels, you want to edit what it represents. And this approach allows exactly that. Will be very interesting to see where this leads!
Or analogous of how you convert audio waveform data into frequencies with the fast-fourier transform, modify it in the frequency spectrum and convert it back into waveform again.
Their examples does however only look a bit like distorted pixel data. The hands of the children seem to warp with the cloth, something they could have easily prevented.
The cloth also looks very static despite it being animated, mainly because the shading of it never changes. If they had more information about the scene from multiple cameras (or perhaps inferred from the color data), the Gaussian splat would be more accurate and could even incorporate the altered angle/surface-normal after modification to cleverly simulate the changed specular highlights as it animates.
The type of 3D model, Gaussian splatting, is also pretty neat and has been getting a lot of attention lately.
There's been some good previous discussion on it here, like this one:
Gaussian splatting is pretty cool https://news.ycombinator.com/item?id=37415478
Gaussian splatting is clearly going to change a lot of things in 3D assets, surprise to see it doing the same for 2D here.
https://carowl.co.uk/car/full-history-check/ for full history checks on any car in the UK.
Or, you can just learn how to do 3d modeling and animation.
The kind of edits and animations this can do are currently not possible with 3D modelling and animation, with or without this tech.
This kind of warping of 2D images is currently used extensively, but a lot more manually (See Live2d, the Photoshop mesh transform tool, or Adobe Illustrator). So, this does not replace 3D modelling; it replaces some 2D editing techniques where 3D modelling isn't even on the table as an applicable tool.
This kind of 2d image warping is useful in advertisements, game art, photo touchups, concept art, photo bashing and digital illustration.
So for making an edit you're proposing to extract every (possibly partial) objects/subjects out of a picture, create 3d models out of them and then animate? And if I don't know how, first learn how to do it?
A sleep-deprived artist somewhere (not me): No, thank you, I need to get this for an ad tomorrow morning.
What's the line there though? To "learn" 3d modeling, should I learn to program and write my own modeler/cad system? To learn to program, should I start shoveling sand into a kiln and make my own hardware?
Sometimes new techniques can augment or replace old ones.
Imagine telling the Toy Story team “or you can just draw each frame like normal”
Just draw the rest of the owl.
Is that all?
"Or you can learn to read and write in Latin and be a practicing member of the clergy."
Everyone should have the means of being able to express the ideas in their heads visually in front of others. It shouldn't require arcane and difficult to use tools. Nor should it cost millions of dollars and require a studio apparatus to tell stories to an audience.
Pretty soon creating worlds, filming movies, and sculpting images and three dimensional forms will be accessible to everyone.
What do you mean by “should”? Everyone does and always has had the right to express themselves visually with the best tools available. But why shouldn’t it require difficult to use tools or cost a lot? That depends entirely on what ideas you’re trying to express and what tools exist, does it not?
Blockbuster movies will never get significantly cheaper, no matter what tools we invent. They will spend as much or more money on using the new tools to make more grandiose movies that individual people have no hope of doing themselves. There’s already a hundred year history of this happening, despite the tools having always been accessible to everyone.
I think this Gaussian splatting tool is great, and I’m in favor of forward progress in imaging tools, and I work in this field. But where is the line between having accessible tools, and putting time and effort into your craft to make something that’s high quality, or telling a story with images that are personal and new? And if you’re really talking about AI (even though this Gaussian splat tool isn’t really AI), where is the line between being able to express your ideas effortlessly and stealing the means to do that from artists who had to put it effort to produce the content that trained the AI?
It is already available to everyone. You can make a movie on your phone, create a song, edit images.
A.I. art is for the lazy.
I'll take "People who feel threatened by AI" for $500, Alex.
Or you could just learn to draw the images on a physical medium.
"You can make a movie on your phone, create a song, edit images."
These are 2010-era tools. We're modernizing.
You wouldn't ask musicians today to stick to only pre-synthesizer, pre-DAW era tools to make their music. You wouldn't ask illustrators to ditch their tablets force them to mix their own cerulean unless that's what they wanted to do.
The tools that are coming are going to be fantastic and they're going to enable artists to do wild new things. That isn't subtractive, it's additive. There's more space to explore.
New tools don't even prevent folks from making music and media the old fashioned way: there are still people shooting on film to this day. And there's a vibrant community that embraces anachronistic tools to make new art.
I'm looking forward to a world where teenagers cut their teeth on making their own Marvel movies instead of just watching them. We'll have so many young Scorseses before they even graduate.
Current generative AI isn't additive. It's generative. That's about half of the problem. DAWs don't revert your changes back to means, but genAI always do, being a statistical model. The roughly other half is that the output is inexplicably bad, not always noticeable to everyone but often obvious to artists and connoisseurs, so connoisseurs can't promote themselves into artists by use of AI.
The almost violent anti-AI sentiment seen among art cohort is sometimes hard to understand to subgroups of tech literates without enough training epochs in human generative image data(especially the kind prevalent on the Internet), and I would understand that without grasp of rather subjective quality issues it could indeed look like an artificially incited luddite conspiracy.
Once someone makes an AI that would be additive and outputs entertainment worthy, then the "luddites" will change, they must. Until then, it's just a technically remarkable white noise generator.