Google has been doing more R&D and internal deployment of AI and less trying to sell it as a product. IMHO that difference in focus makes a huge difference. I used to think their early work on self-driving cars was primarily to support Street View in thier maps.

Maybe they were focusing on a real world use that basically requires AI, but not LLMs.

Tesla claimed that all their "real world" recording would give them a moat on FSD.

Waymo is showing that a) you need to be able to incorporate stuff that isn't "real" when training, and b) you get a lot more information from alternate sensors to visible spectrum only.

> Suddenly all this focus on world models by Deep mind starts to make sense

Google's been thinking about world models since at least 2018: https://arxiv.org/abs/1803.10122

Tesla built something like this for FSD training, they presented many years ago. I never understood why they did productize it. It would have made a brilliant Maps alternative, which country automatically update from Tesla cars on the road. Could live update with speed cameras and road conditions. Like many things they've fallen behind

I always understood this to be why Tesla started working on humanoid robots

  >I've never really thought of Waymo as a robot in the same way as e.g. a Boston Dynamics humanoid, but of course it is a robot of sorts.

https://www.youtube.com/watch?v=ODSJsviD_SU&t=3594s

Internal firewalls and poor management means that the vast majority of integration opportunities are missed.

Grok/xAI is a joke at this point. A true money pit without any hopes for a serious revenue stream.

They should be bought by a rocket company. Then they would stand a chance.

Which is why it's embarrassing how much worse Gemini is at searching the web for grounding information, and how incredibly bad gemini cli is.

> I've never really thought of Waymo as a robot in the same way as e.g. a Boston Dynamics humanoid, but of course it is a robot of sorts.

I view Tesla also more as a robot company than anything else.

Also known as a monopoly. This should terrify us all.

The flywheel is starting to spin......

What an upsetting comment. I'm glad you came around but what did you think was going to be effective before you came around to world models?

"Waymo as a robot in the same way"

Erm, a dishwasher, washing machine, automated vacuum can be considered robots. Im confused as to this obsession of the term - there are many robots that already exist. Robotics have been involved in the production of cars for decades.

......

It's a 3500lb robot that can kill you.

Boston Robotics is working on a smaller robot that can kill you.

Anduril is working on even smaller robots that can kill you.

The future sucks.

>or grok's porn

I know it’s gross, but I would not discount this. Remember why Blu-ray won over HDDVD? I know it won for many other technical reasons, but I think there are a few historical examples of sexual content being a big competitive advantage.

The vertical integration argument should apply to Grok. They have Tesla driving data (probably much more data than Waymo), Twitter data, plus Tesla/SpaceX manufacturing data. When/if Optimus starts on the production line, they'll have that data too. You could argue they haven't figured out how to take advantage of it, but the potential is definitely there.

But somehow google fails to execute. Gemini is useless for programming and I don’t think even bother to use it as chat app. Claude code + gpt 5.2 xhigh for coding and gpt as chat app are really the only ones that are worth it(price and time wise)

I think I'm misunderstanding - they're converting video into their representation which was bootstrapped with LIDAR, video and other sensors. I feel you're alluding to Tesla, but Tesla could never have this outcome since they never had a LIDAR phase.

(edit - I'm referring to deployed Tesla vehicles, I don't know what their research fleet comprises, but other commenters explain that this fleet does collect LIDAR)

The purpose of lidar is to prove error correction when you need it most in terms of camera accuracy loss.

Humans do this, just in the sense of depth perception with both eyes.

That is still important for safety reasons in case someone uses a LiDAR jamming system to try to force you into an accident.

I think there are two steps here: converting video to sensor data input, and using that sensor data to drive. Only the second step will be handled by cars on road, first one is purely for training.

Autonomous cars need to be significantly better than humans to be fully accepted especially when an accident does happen. Hence limiting yourself to only cameras is futile.

They may be trying to suggest that, that claim does not follow from the quoted statement.

I've always wondered... if Lidar + Cameras is always making the right decision, you should theoretically be able to take the output of the Lidar + Cameras model and use it as training data for a Camera only model.

The idea is that, over time, the quality and accuracy of world-model outputs will improve. That, in turn, lets autonomous driving systems train on a large amount of “realistic enough” synthetic data.

For example, we know from experience that Waymo is currently good enough to drive in San Francisco. We don’t yet trust it in more complex environments like dense European cities or Southeast Asian “hell roads.” Running the stack against world models can give a big head start in understanding what works, and which situations are harder, without putting any humans in harm’s way.

We don’t need perfect accuracy from the world model to get real value. And, as usual, the more we use and validate these models, the more we can improve them; creating a virtuous cycle.

I don't think you say "ok now the car is ball bearing proof."

Think of it more like unit tests. "In this synthetic scenario does the car stop as expected, does it continue as expected." You might hit some false negatives but there isn't a downside to that.

If it turns out your model has a blind spot for albino cows in a snow storm eating marshmallows, you might be able to catch that synthetically and spend some extra effort to prevent it.

Looks like they need to blackouts and parades to that simulator...

https://www.yahoo.com/news/articles/waymo-paralyzed-parade-b...

Isn't that true for any scenario previously unencountered, whether it is a digital simulation or a human? We can't optimize for the best possible outcome in reality (since we can't predict the future), but we can optimize for making the best decisions given our knowledge of the world (even if it is imperfect).

In other words it is a gradient from "my current prediction" to "best prediction given my imperfect knowledge" to "best prediction with perfect knowledge", and you can improve the outcome by shrinking the gap between 1&2 or shrinking the gap between 2&3 (or both)

seems like the obvious answer to that is you cover a patch of road with 5mm ball bearings, and send a waymo to drive across it. if the ball bearings behave the way the simulation says they would, and the car behaves the way the simulation said it would, then you've validated your simulation.

do that for enough different scenarios, and if the model is consistently accurate across every scenario you validate, then you can start believing that it will also be accurate for the scenarios you haven't (and can't) validate.

> from a tornado to a casual encounter with an elephant

A sims style game with this technology will be pretty nice!

You could train it in simulation and then test it in reality.

They probably just look at the results of the generation.

I mean would I like a in-depth tour of this? Yes.

But it's a marketing blog article, what do you expect?

As someone who lives in the Bay Area we already have trains, and they're literally past the point of bankruptcy because they (1) don't actually charge enough maintain the variable cost of operations, (2) don't actually make people pay at all, and (3) don't actually enforce any quality of life concerns short of breaking up literal fights. All of this creates negative synergies that pushes a huge, mostly silent segment of the potential ridership away from these systems.

So many people advocate for public transit, but are unwilling to deal with the current market tradeoffs and decisions people are making on the ground. As long as that keeps happening, expect modes of transit -- like Waymo -- that deliver the level of service that they promise to keep exceeding expectations.

I've spent my entire adult life advocating for transportation alternatives, and at every turn in America, the vast majority of other transit advocates just expect people to be okay with anti-social behavior going completely unenforced, and expecting "good citizens" to keep paying when the expected value for any rational person is to engage in freeloading. Then they point to "enforcing the fare box" as a tradeoff between money to collect vs cost of enforcement, when the actually tradeoff is the signalling to every anti-social actor in the system that they can do whatever they want without any consequences.

I currently only see a future in bike-share, because it's the only system that actually delivers on what it promises.

Trains need well behaved people, otherwise they are shit.

I don't want to hear tiktok or full volume soap operas blasting at some deaf mouth breather.

I don't want to be near loud chewing of smelly leftovers.

I don't want to be begged for money, or interact with high or psychotic people.

The current culture doesn't allow enforcement of social behaviour: so public transport will always be a miserable containment vessel for the least functional, and everyone with sense avoids the whole thing.

No matter what, people are going to still use cars because they are an absolute advantage over public transportation for certain use cases. It is better that the existing status quo is improved to reduce death rates, than hope for a much larger scale change in infrastructure (when we have already seen that attempts at infrastructure overhaul in the US, like high-speed rail, is just an infinitely deep money pit)

Even though the train system in Japan is 10x better than the US as a whole, the per-capita vehicle ownership rate in Japan is not much lower than the US (779 per 1000 vs 670 per 1000). It would be a pipe dream for American trains/subways to be as good as Japan, but even a change that significant would lead to a vehicle ownership share reduced by only about 13%.

Me too but given our extensive car brain culture, Waymo is an amazing step to getting less drivers & cars off the road, and to further cement future generations not ever needing to drive or own cars

Ski lifts man, ski lifts all over the city

I think future generations will resent us for bureaucratizing our way out of the California HSR.

Isn't a vehicle that goes from anywhere to anywhere on your own schedule, safely, privately, cleanly, and without billions in subsidies better?

Enough with the trains. I’m all for trains but theyre good for in city or 1-3 hour journeys. Taking a train across the US would take a day even with high speed trains.

I’d much rather have my own vehicle than share my space with a bunch of people.

This is the real story buried under the simulation angle. If you can generate reliable 3D LiDAR from 2D video, every dashcam on earth becomes training data. Every YouTube driving video, every GoPro clip, every security camera feed.

Waymo's fleet is ~700 cars. The internet has millions of hours of driving footage. This technique turns the entire internet into a sensor suite. That's a bigger deal than the simulation itself.

3d from moving 2d images has been a thing for decades.

It's not unheard of, there are a handful [0] of metric monodepth methods that output data that's not unlike a really inaccurate 3D lidar, though theirs certainly looks SOTA.

[0] https://github.com/YvanYin/Metric3D

All this simulation buys a single vehicle that drives better. That failure was a fleet-wide event (overloading the remote assistance humans).

That is, both are true: this high-fidelity simulation is valuable and it won't catch all failure modes. Or in other words, it's still on Waymo for failing during the power outage, but it's not uniquely on Waymo's simulation team.

They've also been seen driving directly into flood waters, with one driving through the middle of a flooded parking lot.

https://www.reddit.com/r/SelfDrivingCars/comments/1pem9ep/hm...

That's a very interesting way of looking at it. Yes, you start with simulating something simpler than the real world. Then you use the real world. Then you need to go back to simulations for real-world things that are too rare in the real world to train with.

Seems like there ought to be a name for this, like so-and-so's law.

Regardless of the corporate structure DeepMind is a lot more than just another Alphabet subsidiary at this point considering Demis Hassabis is leading all of Google AI.

I don't think there's a conflict. If you can predict the world you understand it.

Ideally it would learn a relationship between the sensor input and the correct actions, even if the sensor input is not realistic for the GG in reality.

Racing and street driving are completely different. Racing involves detailed knowledge of vehicle dynamics and grip. Street driving is mainly obstacle recognition and avoidance. No waymo ever operates anywhere close to the limit of grip, which is where you are all the time when racing.

You're forgetting that they are also training with real data from the 100+ million miles they've driven on real roads with riders, and using that data to train the world model AI.

> there's probably no examples in the training data where the car is behind a stopped car, and the driver pulls over to another lane and another car comes from behind and crashes into the driver because it didn't check its blindspot

This specific scenario is in the examples: https://videos.ctfassets.net/7ijaobx36mtm/3wK6IWWc8UmhFNUSyy...

It doesn't show the failure mode, it demonstrates the successful crash avoidance.

While there most likely is going to be some bias in the training of those kinds of models, we can also hope that transfer learning from other non-driving videos will at least help generate something close enough to the very real but unusual situations you are mentioning. We could imagine an LLM serving as some kind of fuzzer to create a large variety of prompts for the world model, which as we can see in the article seems pretty capable at generating fictive scenarios when asked to.

As always tho the devil lies in the details: is an LLM based generation pipeline good enough? What even is the definition of "good enough"? Even with good prompts will the world model output something sufficiently close to reality so that it can be used as a good virtual driving environment for further training / testing of autonomous cars? Or do the kind of limitations you mentioned still mean subtle but dangerous imprecisions will slip through and cause too poor data distribution to be a truly viable approach?

My personal feeling is that this we will land somewhere in between: I think approaches like this one will be very useful, but I also don't think the current state of AI models mean we can have something 100% reliable with this.

The question is: is 100% reliability a realistic goal? Human drivers are definitely not 100% reliable. If we come up with a solution 10x more reliable than the best human drivers, that maybe has some also some hard proof that it cannot have certain classes of catastrophic failure modes (probably with verified code based approaches that for instance guarantees that even if the NN output is invalid the car doesn't try to make moves out of a verifiably safe envelope) then I feel like the public and regulators would be much more inclined to authorize full autonomy.

As a Londoner who used to have to ride up Abbey Road at least once per week there are people on that crossing pretty much all day every day reproducing that picture. So now Waymo are in Beta in London[1] they have only to drive up there and they'll get plenty of footage they could use for taht.

[1] I've seen a couple of them but they're not available to hire yet and are still very rare.

Not at all. It's not the counter-factual they're generating, it's the "too rare to capture often enough to train a response to" they're generating.

They're implying that without the model having knowledge, even approximate, of a scene to react to, it simply doesn't react at all; it simply "yields" to the situation until it passes. In my experience taking Waymo's almost daily this holds.

I would rather not have the Waymo yield to a tornado, rising flood-waters, or charging elephant...

Driving is always a balance between speed and safety. If you want ultimate safety you just sit in the driveway. But obviously that isn't useful. So functionally one of the most important things a self-driving system will decide is "how fast is it safe to drive right now". Slower is not always better and it has to balance safety with productivity.

Not entering a roundabout when it's clearly safe to do so is a mark against you at a driving exam. So would be always driving at 5mph. It's just not that simple.

Well it certainly helps,doesn't it? This system is going to encounter more edge cases than a single human ever would. Hopefully the lessons from known unknowns generqlise to unknowns. And once they've been seen once they took can become part of the corpus.

I don’t think coming up with novel situations is all that hard. LLMs already do it in text form all the time.

"The Filipino driver thing" is simply that there's a manual override ability when this profoundly complex and marvelously novel technology gets trapped in edge cases.

Once it gets unstuck, it runs autonomously.

As someone who half-learned to drive in Manila, the idea that they would use Filipino drivers as backups is ironic.

For context, my "driver's test" was going to the back of the office, and driving some old car backwards and forwards a few meters.

2. Yes, a Waymo exec described it in a Congressional hearing.

https://news.ycombinator.com/item?id=46918043

I am very pro public transit. But there is still a place for cars (ideally mostly taxis). Going to more rural areas or when you need to carry more stuff. I think an ideal society would have both urban transit, inter-city transit and taxis for the other trips and going out into the country.

Filipino driver is false. Filipino guidance person is true.

(2) I really don't understand why people are surprised that Waymo has fallbacks? The fact that they had a team ready to take over as necessary was well known. I've seen a bunch of comments about this and it seems like people are confused.

America is not europe, how would public transport work for the last 1/2miles

They are not trying to "bury" remote assistance at all. They wrote a white paper about it in 2020 and a blog post about it in 2024.

Anyway you can think it's a waste but they're wasting their money, not yours. If you want a train in your town, go get one. Waymo has only spent, cumulatively, about 4 months of the budgets of American transit agencies. If you had all that money it wouldn't amount to anything.

My view on Waymo and autonomous taxis in general is they will eventually make public transit obsolete. Once there is a robotaxi available to pick up and drop off every passenger directly from a to b, the whole system could be made to be super efficient. It will take time to get there though.

But eventually I think we will get there. Human drivers will be banned, the roads will be exclusively used by autonomous vehicles that are very efficient drivers (we could totally remove stoplights, for example. Only pedestrian crossing signs would be needed. Robo-vehicles could plug into a city-wide network that optimizes the routing of every vehicle.) At that point, public transit becomes subsidized robotaxi rides. Why take a subway when a car can take you door to door with an optimized route?

So in terms of why it isn’t a waste of time, it’s a step along the path towards this vision. We can’t flip a switch and make this tech exist, it will happen in gradual steps.

It seems inevitable that they'll soon be used as the starting points for developing almost all video game environments.

Not for the rendering (that's still way too expensive), but for the initial world generation that gets iteratively refined and then still ultimately gets converted into textured triangles.

It's easier to build trust for such a safety-critical service when you're more open about how it works an performs. For the complete opposite approach, see Tesla.

Given the announcement from a few days ago of google trying to get external investment, this is their follow up, showing what that investment is good for. Also, it’s pretty light on details that are of much use to competitors. “We made an accurate simulation system to test our system in before deployment” would be pretty mundane if you were talking about any other field of engineering.

Maybe to distract from the story that they use remote drivers after one of their cars hit a kid? [1]

[1] https://people.com/waymo-exec-reveals-company-uses-operators...

edit: fixed kill -> hit

San Francisco isn't uniformly dense and narrow, but it does have both, and it's run remarkably well so far.

What would be an example city? Waymo just announced they're ramping up in Boston: https://waymo.com/blog/?modal=short-back-to-boston

"we’re excited to continue effectively adapting to Boston’s cobblestones, narrow alleyways, roundabouts and turnpikes."

I live in such an area. The route to my house involves steep topography via small windy streets that are very narrow and effectively one-way due to parked cars.

Human drivers routinely do worse than Waymo, which I take 2 or 3 times a week. Is it perfect? No. Does it handle the situation better than most Lyft or Uber drivers? Yes.

As a bonus: unlike some of those drivers the Waymo doesn't get palpably angry at me for driving the route.

Yes, something like Ho Chi Minh or Mumbai in a peak hour! With lots of bike riders, pedestrians, and livestock at the same roundabout.

Like London? https://www.youtube.com/watch?v=KvctCbVEvwQ

Waymo cars are driving around London right now.

Not taking paying passengers yet though!

They're being trialled in London right now.

Napoli

Old Delhi is the the final boss.

Does it, though? Maybe Dhaka will never get Waymo. The same way you can’t get advanced gene therapy there.

The article is about using the world model to generate simulations, not for controlling the vehicle.

This is cool, but they are still not going about it the right way.

This is legit hilarious to read from some random HN account.

Completely false: https://x.com/i/status/2019213765506670738

Listen to the statement.

The operators help when the Waymo is in a "difficult situation".

Car drives itself 99% of the time, long tail of issues not yet fixed have a human intervene.

Everyone is making out like it's an RC car, completely false.

Why is this relevant at all?

Having humans in the loop at some level is necessary for handling rare edge cases safely.

Interesting question. If the Waymo was driving aggressively to remove us from the situation but relatively safely I might stay in it.

This does bring up something, though: Waymo has a "pull over" feature, but it's hidden behind a couple of touch screen actions involving small virtual buttons and it does not pull over immediately. Instead, it "finds a spot to pull over". I would very much like a big red STOP IMMEDIATELY button in these vehicles.

I can! If the Waymo got you into one on the way home because Google didn’t integrate with watch duty yet, that’s plausible

The model generates camera and Lidar data. As if it was a Waymo car that drove through the simulated scenario with its cameras running. This synthetic training data can then be used to train the driving models.

At least 6 radar units: https://support.google.com/waymo/answer/9190838?hl=en

>> get them to comply with the same laws that limit what people can do

I think you meant, "Attempt" to limit what people can do.

Driving in SF (for example) provides many opportunities to see "free will" exerted in the most extreme ways -- laws be damned.

Can you explain? I lived in PH, and my guess is that you mean navigating and modeling the unending and constantly changing chaos of the street systems (and lack thereof) is going to be a monumental task which I completely agree with. It would be an impressive feat if possible.

Edit: or are you talking about the allegations of workers in the Philippines controlling the Waymos: https://futurism.com/advanced-transport/waymos-controlled-wo... I guess both are valid.

What was the admission? That they use cheap labor to provide the waymo clarity when it is confused? That has been known for a long time.

My understanding is that support is basically playing an RTS (point and click), not a 1P driving game. Which makes sense, if they were directly controlling the vehicles they'd put support in central America for better latency, like the food delivery bot drivers

This isn't news, they've always acknowledged that they have remote navigators that tell the cars what to do when they get stuck or confused. It's just that they don't directly drive the car.

Yeah I have some videos of these drivers in action, I think the sensors are as assistance and but not the whole story, so yeah there’s models lidars etc etc but human factor is there, unfortunately this means we should see soon many cobitics are teleopetated remotely from India and Philippines and the likes to satisfy the greed of these companies to pay peanuts to operate them.

This is not false, but gives the wrong idea that foreigners are driving them in real time.

> After being pressed for a breakdown on where these overseas operators operate, Peña said he didn’t have those stats, explaining that some operators live in the US, but others live much further away, including in the Philippines.

> “They provide guidance,” he argued. “They do not remotely drive the vehicles. Waymo asks for guidance in certain situations and gets an input, but the Waymo vehicle is always in charge of the dynamic driving tasks, so that is just one additional input.”

This is quite misleading... From the article:

“When the Waymo vehicle encounters a particular situation on the road, the autonomous driver can reach out to a human fleet response agent for additional information to contextualize its environment,” the post reads. “The Waymo Driver [software] does not rely solely on the inputs it receives from the fleet response agent and it is in control of the vehicle at all times.” [from Waymo's own blog https://waymo.com/blog/2024/05/fleet-response/]

What's the problem with this?

Have you read the article ? The guys in the Philippines are providing high level executive indications, they don't drive remotely the car or have any low level control of the car.

Dig deep enough into any "AI" idea and you'll find the bottom end of the scam looks exactly like this.

We've simply relabeled the "Mechanical Turk" into "AI."

The rest is built on stolen copyrighted data.

The new corporate model: "just lie the government clearly doesn't give a shit anymore."

Yes, let's stop all progress and roll-back all automation to keep hypothetical angry people with guns happy.

Waymo has been operating since 2004 (22 years ago), and replacing drivers on the road will take many more decades. Nothing is happening "overnight".

If Waymo's history is any guide, it's not going to happen overnight. Even in San Francisco, their market share is only 20-30%.

this sounds like a major benefit.

i dont want my uber driver bragging anout how theyre going to shoot me before i get out of the car

> What's going to happen to all the millions of drivers who will lose their job overnight? In a country with 100 million guns, are we really sure we've thought this through?

Same was said about electricity, or the internet.

Reminds me of the history or radio and the absolute uproar that someone played a record on the radio rather than live performances!!

same thing that happened during the industrial revolution, you pay enough of them to 'protect the law' vs the rest.

UBI or war, or both

what did all the farmers do when the first tractors rolled into the fields?

I don't think Uber goes out of business. There is probably a sweet spot for Waymo's steady state cars, and you STILL might want 'surge' capabilities for part time workers who can repurpose their cars to make a little extra money here and there.

Those are rookie numbers. The US has 400 million guns. https://www.theglobalstatistics.com/united-states-gun-owners...

As to the revolt, America doesn't do that any more. Years of education have removed both the vim and vigor of our souls. People will complain. They will do a TikTok dance as protest. Some will go into the streets. No meaningful uprising will occur.

The poor and the affected will be told to go to the trades. That's the new learn to program. Our tech overlords will have their media tell us that everything is ok (packaging it appropriately for the specific side of the aisle).

Ultimately the US will go down hill to become a Belgium. Not terrible, but not a world dominating, hand cutting entity it once was.