Over the past few years, the automotive industry has seen a huge uptick in partially automated vehicles. Nearly every major automaker has some form of Level 2 driver assistance, whether that be Tesla's Autopilot, GM's Super Cruise, or even the Honda Sensing suite. Consumer Reports recently tested every large player on the market to evaluate performance, reliability, and overall usefulness. To everyone's surprise, it wasn't a legacy automaker who came out on top—it was a small California-based startup called Comma AI.
Before we jump into how Comma ranked so highly on Consumer Reports' scale, it's important to understand how the organization chose to evaluate each automotive offering.
The objective of Consumer Reports'testing was to evaluate the major Advanced Driver Assistance Systems on the market today and determine how well the systems performed when stacked up against each other. It's worth noting that the testers indicated that not all systems have complete feature parity, as the systems may be designed with different intentions and goals in mind by the manufacturer. However, a large emphasis on safety was taken when evaluating the different systems.
Of all the solutions, Comma AI's Comma Two (which is an Android-based device capable of running an open-source application called OpenPilot) came out on top. This was the only aftermarket system that Consumer Reports tested and, as predicted by Alex Roy in 2016, it seemingly embarrassed the rest of the auto industry by simply outperforming what automakers have been building into their cars for years.
In a close second was GM's Super Cruise, followed by Tesla's AutoPilot, and then Ford's Co-Pilot 360. At the bottom of the list sits Land Rover's Driver Assist, GM's Driver Confidence, and Mazda's i-ActivSense.
OpenPilot, when installed on a $999 Comma Two, uses existing factory-installed vehicle safety hardware to improve a vehicle's ability to center itself in a lane. This includes using the forward-facing monocular camera to help track lane lines and the vehicle's radar unit to determine how far away other cars might be. The Comma Two also adds a layer of driver monitoring to ensure that someone is paying attention to the road at all times, and even doubles as a dashcam. It's important to recognize it as a supplemental driver assistance system instead of a factory baked-in solution.
I'm no stranger to OpenPilot. I actually purchased Comma's products for my Honda Odyssey, simply due to the improvement over Honda's stock package. However, it's important to note that the capability and performance can vary drastically from car to car. For example, a stock Honda vehicle may greatly limit torque whereas some Toyota or Hyundai vehicles are much better suited to adhering to tight lane curvature. In Consumer Reports' test, a Toyota Corolla LE was driven, which arguably will provide a better overall experience than, say, a Honda Accord.
This report was met with a good bit of skepticism across the internet. People are more concerned about how well the system performed rather than the additional categories vetted during testing. After all, surely a system like Tesla's Autopilot can perform better since it uses multiple cameras around the vehicle to help make partially automated driving decisions, right?
Well, it's not just the performance that Consumer Reports was evaluating. The outlet's testing also measured how well it kept the driver engaged, along with general ease of use. Additional tests were performed to gauge how clearly the system indicated when its features were safe to use, as well as how it handled an unresponsive driver.
A large takeaway from the testing is that automakers may not make it clear enough when its systems are safe to use. Consumer Reports gave a larger number of sub-par ratings to nearly every ADAS system on the market with the exception of Super Cruise and, to a lesser extent, OpenPilot. Perhaps this is an indication that automakers need to take a step back and look at how to communicate its intended Operational Design Domain (ODD) to drivers, as activating the system outside of the ODD can cause the system to perform outside of the user's expectations and cause a false sense of driver security.
OpenPilot also scored well on its driver engagement thanks to its sophisticated Driver Monitoring System (DMS). The Comma Two uses its driver-facing camera to determine the position of the driver's head and eyes. If it determines that the driver is distracted, it activates a visual and audible alert. To ensure that the driver is also tracked at night, a series of infrared LEDs are used to illuminate a dark cabin. An inattentive driver will be handed back control of the vehicle.
It's almost surprising to see Comma rank so highly among seasoned automakers, especially pioneers like Tesla and General Motors who are undoubtedly seen as leaders in the self-driving segment. But all things considered, the small startup has done quite a few things right that some automakers could use as a lesson on. Now the battle will be staying ahead as automakers breeze on to higher levels of automation.