Although it's comforting that this exact situation shouldn't happen again in an Uber autonomous car... there is no mechanism to share that learning with the other car companies. There seriously fucking needs to be a consortium for exactly this purpose: sharing system failures.
Also my problem with this is that a human death is functionally treated as finding edge cases that are missing a unit test, and progressing the testing rate of the code... and that really bothers me somehow. We need to avoid treating deaths as progress in the pursuit of better things
> We need to avoid treating deaths as progress in the pursuit of better things
Au contraire. Go read building codes some time. There's a saying that they're "written in blood" - every bit, no matter how obvious or arbitrary seeming, was earned through some real-world failure.
The death itself isn't progress, of course. But we owe it to the person to who died to learn from what happened.
Federal Aviation Administration's Joint Order 7110.65 is exactly what you are talking about. It is the manual that Air Traffic Controllers live by. To include situations that involve diverging aircraft or taxiing instructions and how to handle said situations. The entire manual was practically written from real-world experience.
Not just building codes, but a lot of car safety regulations and practices too. Stuff like crumple zones, ABS, breakaway road signs, safety glass, etc.
Clearly I think Uber is a benevolent entity that has all of our best interests at heart. Also, I eat babies.
Or, you know, don't jump on comments for their not explicitly addressing the hobby horse you're riding. Frankly I just wanted to express a better engineering context around the loss of life without getting into the political bullshit for once.
>There seriously fucking needs to be a consortium for exactly this purpose: sharing system failures.
This implies they're using the same systems and underlying models. If one model hit a pedestrian because of a weakness in training data plus a sub-optimal model hyperparameter, and therefore was classified a pedestrian in that specific pose as some trash on the street, how do you share that conclusion with other companies models?
I guess it depends on the data available, but I don't think the hyperparameters are what's needed. You just need to see what the car sensed and how it responded. Then the other car companies can try to replicate the similar circumstances and see what their models would do.
I don't think all self-driving cars have the same sensors.
If you have LIDAR model X, and they were using LIDAR model Y, will your system "magically figure it out?
If your car has cameras at 5ft high, and the data is from a camera 6ft high?
Sure, someone could release the data, but will it screw up your models more than it fixes them?
(I totally agree the data should be released, I'm just not sure other self-driving cars will directly benefit. Certainly they can indirectly benefit from it.)
Is the design of self driving cars so limited that we will have to go through this every time carmakers must redesign hardware and/or vehicles themselves? Will the experiences of a sedan be impossible to transport to the experience of a semi truck? And vice versa?
The idea you present is possible, but I have to wonder how viable it makes the idea of self driving cars.
I understand, but the error may be in how it interpreted what it sensed. This is callous language, but if the models interpreted the pedestrian as trash on the street, then how it responded (driving over it) is not inappropriate.
> I understand, but the error may be in how it interpreted what it sensed.
It may, it also might not. If sharing data fails, further methods would be needed - but it's a good start for figuring out what data should be recorded for comparison. If the data is entirely incompatible, then we should have regulation to require companies to at minimum transcribe the data into a consumable format after an event such as this.
> ... if the models interpreted the pedestrian as trash on the street, then how it responded (driving over it) is not inappropriate.
If the models saw anything at all, they should not have driven over it. Even human drivers are taught that driving over road debris is dangerous. At minimum it puts the car/occupants at risk - in extreme cases, the driver may not recognize what it is they are driving over.
If this isn't a case where the car was physically unable to stop - it's more likely the telemetry didn't identify the person as an obstacle to avoid at all.
I would expect nowadays you could fairly trivially produce near life-like footage and allow self driving cars to 'play a video game' that tests tens of thousands of situations as a pre-requisite for certification.
The great thing about computer generated test cases is you don't need real footage of every hypothetical awful thing that could happen i.e. a truck losing control and rolling over sideways. These could be a stage 1 test -- a pre-requisite to real testing. Like the hazards test before you are allowed to sit the drive test with a real drive test employee [1] to make sure you're not a retard.
I think there is likely a common need for a set of standard training models. This may hinder innovation for some time, but it's a cost we should accept when releasing a potentially dangerous technology to the public. It would have the added benefit of multiple companies contributing to a common self-driving standard which could accelerate its development.
That being said, when the first real cars were introduced to the world and later improved upon there were many more fatalities than we're likely to experience with self-driving technology.
Every technology is dangerous. Every technology costs lives to some extent when spread across billions of people. I'm sure forks take more lives each year than self driving cars.
Weigh this against the potential lives saved. I posit you'd be killing more people with drunk drivers by slowing innovation than you'd be saving via luddism.
Just sensor data to replay the scenario as fully as possible should be sufficient. Whatever it looks like to humans, there's clearly something in there that can be difficult for AI systems and so everyone should be using it as a regression test.
Well, for one thing, I think I have misgivings in part because Uber hasn't really demonstrated an attitude that makes me think they'll be very careful to reduce accidents. (Also, the extent to which they've bet the farm on autonomous driving presents a lot of temptation to cut corners)
Circumstances basically force them into betting the farm. Whether they want SDCs or not, if someone scoops them on SDCs, it's an existential threat to their business.
Sure, and also paying the drivers is a big part of the reason they aren't making money yet. Nevertheless, I don't think it changes the fact that there is a big temptation for someone to start cutting corners if it's not moving along fast enough.
Oh yeah, I'm not disagreeing with you. I think it actually makes the temptation much worse. They've built this business with billions in revenue, and they will probably lose it all if this SDC doesn't succeed.
What's more, a lot of people seem to think Waymo's tech is further along. So not only does this project have to succeed, it is also the underdog. So no wonder they're aggressive.
It makes me uncomfortable too. I think it's because it's a real world trolley problem.
We all make similar decisions all of our lives, but nearly always at some remove. (With rare exceptions in fields like medicine and military operations.) But autonomous vehicles are a very visceral and direct implementation. The difference betweeen the trolley problem and in autonomous vehicles is in time delay and the amount of effort and skill required in execution.
Plus, we're taking what is pretty clearly a moral decision and putting it into a system that doesn't do moral decision-making.
Although the FAA is of course involved, it's generally the NTSB that is the lead investigator in airplane crashes. The NTSB already has jurisdiction to investigate highway accidents.
There's also NHTSA (which is indeed part of the DoT).
It looks to me like we don't need any new agency at all, just a very small expansion of NHTSA's mandate to specifically address the "self-driving" part of self-driving cars.
Thinking about different scenarios as unit tests it shouldn’t be hard for them to simulate all sorts of different scenarios and share those tests. Perhaps that would become part of a new standard for safety measures in addition to crash tests with dummies.
In fact, I really think this will become the norm in the near future. It might even be out there already in some form.
> We need to avoid treating deaths as progress in the pursuit of better things
Then by all means lets stay at home and avoid putting humans in rockets ever again because if you think space exploration will be done without deaths you are in for a surprise.
Notoriously criminal company killing pedestrians with robo-cars to make money for themselves != Space exploration by volunteer scientists and service members
Also my problem with this is that a human death is functionally treated as finding edge cases that are missing a unit test, and progressing the testing rate of the code... and that really bothers me somehow. We need to avoid treating deaths as progress in the pursuit of better things