Chinese internet titan Baidu has snatched Google's crown in one of the few widely recognized measures of progress in autonomous driving technology -- but the race to develop a truly self-driving car is still wide open.
Autonomous driving is a relatively opaque industry, as companies keep the details of their proprietary technologies under wraps and there are few if any established authorities capable of evaluating the nascent field.
That is why the California Department of Motor Vehicles' annual autonomous vehicle disengagement report, a detailed road testing record of almost all major driverless car players, is watched so closely as a gauge of technological progress.
By law, all companies testing their self-driving vehicles in California must report every instance in which a human safety driver had to take control of a self-driving car being tested on public roads in the state.
Fewer disengagements per mile suggests the car requires less human intervention, which some argue is an indication of a safer and more advanced autonomous vehicle.
For years, Google's self-driving unit Waymo had the lowest disengagement per mile. But last year, that crown was taken by Apollo, the autonomous driving venture owned by Chinese search engine giant Baidu, according to the latest California DMV autonomous vehicle disengagement report released on Thursday.
Is this proof that Baidu's self-driving car is better than Google's? Both companies say no.
"Waymo's position from the start -- and a growing industry consensus -- is that disengagements are not the best or only measure of the progress of self-driving tech," a Waymo spokesperson told the Nikkei Asian Review.
Focus too heavily on disengagement per mile "does little more than creating a horse race on a muddy track in our industry," which can become a distraction from true progress and improvement, the spokesperson added.
Waymo's criticism of the measurement of disengagement rate might not come as a surprise given that it just lost its lead for the first time, but even Baidu is not placing too much importance on the report.
"The DMV report can be used as a reference, but the ranking is not meaningful," a Baidu Apollo spokesperson told Nikkei.
"Miles per intervention is an internal reflection of the speed of technical iterations, but the comparison between companies is not that meaningful [because] business needs vary with each autonomous driving company," the spokesperson added.
General Motors-owned Cruise, which came at the third place in the 2019 disengagement per mile ranking, piled on the criticism of the disengagement measurement.
"The idea that disengagements give a meaningful signal about whether an autonomous vehicle is ready for commercial deployment is a myth," said Cruise's co-founder and CTO Kyle Vogt in a blogpost. "This data is really great for giving the public a sense of what's happening on the roads. ... But it's woefully inadequate for most uses beyond those of the DMV."
Baidu's Apollo logged a record-breaking 0.06 disengagements per thousand miles traveled in 2019, while Waymo and Cruise's disengagement rates averaged 0.076 and 0.08 times, respectively, according to the California DMV report.
However, both Waymo and Cruise logged significantly more testing miles with many more test cars deployed on the road than Baidu did in 2019.
Cruise had 227 vehicles on the road that traveled a combined 831,039.9 miles. Waymo tested 147 cars, which recorded 1,453,137.3 miles on the road. Baidu, on the other hand, deployed only four vehicles and traveled a total of 108,300 miles in California public road tests.
Two self-driving startups from China -- AutoX and Pony.ai -- came in fourth and fifth place, meaning three out of the top five companies in the ranking are Chinese.
"The Chinese have been and continue to invest hard in this area and are seeing some of the fruits of that investment," Richard Windsor, founder of Radio Free Mobile, wrote in a research note.
However, it is impossible to distinguish between the top Chinese and American players "given the weaknesses in this data and the fact that many of these companies are also testing in other locations [outside California]," according to Windsor.
A Silicon Valley-based venture investor who specializes in the automobile sector agreed that it is difficult to draw conclusions based only the California report.
"Chinese and American self-driving companies are basically playing in two totally different fields now as they are all focusing on their respective domestic market, " the auto investor said. "I don't think we can truly get a sense of whose technology is better until companies from the two countries start competing for the same market."
If that is the case, it might be a while until the winner in the driverless car race emerges. Baidu, AutoX and Pony.ai have all told the Nikkei Asian Review in multiple separate interviews that they are not interested in the U.S. market for the time being, though California will remain a crucial testing ground.
"At present, Apollo's main theater is still China, where a more complicated environment needs larger-scale road testing and further technical iterations," a Baidu Apollo spokesperson said.
