In an effort to prove that robot drivers are safer than humans, Waymo has simulated dozens of fatal accidents that have occurred in Arizona for nearly a decade. The Google spin-off discovered that replacing one of the vehicles in a two-engine accident with its robot-guided minibuses would eliminate almost all deaths, according to data released today.
The results are meant to bolster Waymo’s case that autonomous vehicles work safer than people being driven. As millions of people worldwide die in car accidents each year, AV operators are increasingly relying on this safety issue to encourage regulators to pass legislation that allows more autonomous vehicles on the road.
But it was difficult to prove, thanks to the very limited number of autonomous vehicles operating on public roads today. To provide more argument for his argument, Waymo turned to counter-facts, or “what if?” scenarios, which are intended to show how the robotic vehicles would react in real situations.
Last year, the company published 6.1 million miles of driving data in 2019 and 2020, including 18 accidents and 29 near-collisions. In the incidents where safety operators took control of the vehicle to prevent an accident, Waymo’s engineers simulated what would have happened if the driver had not disconnected the vehicle’s self-driving system to cause a counterattack. The company also made some of its data available to academic researchers.
This work in counter-facts continues in this most recent release of data. Through a third party, Waymo gathered information about every fatal accident that occurred in Chandler, Arizona, a suburban community outside of Phoenix, between 2008 and 2017. Focus only on the accidents that occurred within its operational design domain, or the about 100- Waymo area in which the company has its cars driven, Waymo identified 72 accidents to reconstruct in simulation to determine how its autonomous system would react in similar situations.
Some of these accidents involved one vehicle, while most involved two. For two-vehicle accidents, Waymo conducted separate experiments simulating its autonomous vehicles in the role of each vehicle – first replacing the vehicle that started the accident and then replacing the vehicle that responded to the other vehicle’s actions. For one-vehicle accidents, Waymo only simulated the single vehicle. This left a total of 91 simulations.
The company has reconstructed these accidents and systematically aligned the vehicle’s lane to ensure that its Waymo vehicles are exposed to a similar situation as in the actual fatal accident. Waymo uses the same simulation platform he uses to train and evaluate his autonomous vehicles on virtual roads in ordinary operations.
The results show that Waymo’s autonomous vehicles would “avoid or mitigate” 88 of the 91 total simulations, said Trent Victor, director of safety research and best practices at Waymo. In addition, Waymo’s vehicles would have reduced the chance of serious injuries by 1.3 to 15 times for the accidents that were mitigated, Victor said.
“This means that, even if it did not completely avoid the accident, it acted to reduce the severity of the impact,” Victor said. “If the severity is reduced, the driver is less likely to die.”
When Waymo replaced the vehicle on the instigator of the accident, it found that it “completely avoided” 82 percent of the simulated accidents – the vast majority of them without having to brake hard or avoid evasive action. In another ten percent in response, the urgent ways of the Waymo vehicle helped reduce the severity of the crash. The accidents all took place at an intersection, where the other vehicle turned left or cut across the Waymo vehicle, leaving little time to react, the company said.
“We are not saying we will eliminate all deaths, but we are saying that the best way to reduce the chance of serious injury is to take an evasive maneuver if possible,” Victor said. “And in all these accident simulations, the Waymo driver undertook evasive maneuvers.”
There have been three incidents of a person dying after being chased by another vehicle. In simulation, Waymo was unable to avoid these accidents when his vehicle was the one driven backwards. “In the case of rear-end accidents, there’s not much the answer role can do,” said Matthew Schwall, head of field safety at Waymo. “The Waymo driver therefore really suffers from the same challenge as people do in such situations, that it is difficult to predict quickly enough to be able to make an evasive maneuver.”
Twenty simulated accidents caused a pedestrian or cyclist to be hit by a driver. Waymo’s autonomous vehicles avoided 100 percent of the accidents in simulation, the company said.
There is no standard approach to evaluating AV safety. A recent study by RAND concluded that customers are unlikely to trust the government, although U.S. regulators are happy to allow the private sector to determine what is safe. In this vacuum, Waymo hopes that policymakers, researchers and even other businesses can begin to take on the task of developing a universal framework by publishing this data.
To be sure, Waymo did not subject his findings to a peer-reviewed analysis for publication in an academic or scientific journal, although it would be open for publication in the future, a spokesman said. The simulations were not performed independently of the company, nor were they checked by any third party to verify them before the company made them public.
The company did share its findings with a select group of academic experts to get feedback. Daniel McGehee, director of the National Advanced Driving Simulator Laboratories at the University of Iowa, said Waymo is pushing safety analyzes and transparency ‘to a new level.’
Jonas Bargman, associate professor of vehicle safety at Chalmers University of Technology in Sweden, said Waymo’s use of simulation-based assessment is ‘the scientific cutting edge’, but only one of the components needed to assess the safety of automotive vehicles.
After reviewing the report, Bargman concluded that the ‘Waymo simulation platform is very sophisticated’, referring to the company’s 3D sensor survey models. He singled out the use of counter-factual axle-axis scenarios as ‘extremely relevant to the introduction of Waymo self-driving vehicles to the public’.