Most of my posts up until now have been about autonomous vehicles (AVs) because they have been at the front of the news, and thus in the minds of regulators. However, there are a variety of other autonomous systems operating out there, and some of them are doing things that will almost certainly require more oversight than they currently have.

I’m going to focus in this series of articles on how a robot can cross a legal or ethical boundary, whether by accident or by design. Let’s start with a type of robot that was in the news last December – security robots.

Last December, the news broke that a Knightscope security robot was being rented and used by the SPCA in San Francisco to deter homeless people from settling in the area. Knightscope disputed the allegation, claiming that their robot was only being used to help secure the SPCA employees and environment. This didn’t prevent the SF government from threatening a $1000 per day fine if the robot continued to operate on the sidewalk without a permit, nor did it stop headlines like “Robots Are Being Used Against San Francisco’s Homeless.” The vox pop comments on the issue were very strongly against the use of a robot to chase away homeless people even if it helped the employees feel safer and lowered the rate of car break-ins and vandalism. With the backlash growing, and the robot being attacked by people on the street, the SPCA decided to return their K5 unit.

More thoughtful articles about the ethics of surveillance and security robots followed in the wake of the events. Their points are well taken. The Knightscope robot was unarmed and unable to take positive action. It wasn’t even allowed to get closer than a few feet without triggering proximity alerts which made it try to avoid collision. So then, why was it such a controversial tool if all it did was trundle around sending video back to the security people in SPCA?

Well, for one thing it had recognition algorithms. The main use case presented by Knightscope is that it’s supposed to be used to detect whether cars have been on the property for too long, or to compare against a list of people either allowed or barred from being on the property being patrolled. This means that it was ‘remembering’ who it saw wherever it went, creating a database of individuals doing business with, passing by, or otherwise using the space around the SPCA building. Creating a visual library of who goes where is always a tricky thing to do, especially when the SPCA said they were sharing their data with the police department – albeit only when a crime was committed.

Second, it was a symbol of surveillance. Similar to having a marked car patrolling the street or a camera on every corner, these robots were flat-out showing that everyone and everything near the building was being watched. It wasn’t necessarily intimidating just because it was a robot – although its size and shape have been compared to a smartphone crossed with a Dalek – but instead as a reminder that someone was watching. Not just watching but recording and taking notes.

Image Courtesy of Knightscope Gallery

Ultimately, this situation was less about the robot and more about the people. Example: The thing about some of the algorithms onboard the robot is that they’re also supposed to be used to ‘predict’ when crimes will occur. There’s been a wealth of studies and literature about how criminal justice AIs are biased. The bias is because they have to be trained by humans or use human data and so they form the same biases the humans do. Thus a criminal justice AI will be about as accurate as a human doing the same work because the conclusions programmed into it by humans as a basis for its decision making lead it to come to similar conclusions in the future. This has been seen in the use of AIs to predict recidivism in recently released prisoners. If you tell the AI to prevent recidivism, you increase the number of wrong decisions because it will bias in favor of recommending innocent people who have been convicted in the past be placed on stricter parole. Tell it to decrease false positives and you’ll get more people slip through the net and re-offend. These robots and AIs, then, just expand an existing system flaws and all.

What the examples show us is that people don’t trust robots to enforce the law and provide security in public spaces. It makes them feel watched, untrusted, and afraid of what the robot might do. The flipside is that the robots patrolling private spaces seem to be better regarded. Possibly because they expect more surveillance indoors and possibly because the robots seem to be under control if they’re not roaming the streets. One Knightscope robot that fell in a fountain even got a memorial put up by the people who worked nearby.

Let’s take this a step further. Robots that actively enforce rather than watch and catalog. A study is ongoing which puts people in situations where an armed robot asks them to hand over an object like a weapon in order to help maintain peace in a simulated environment. The study so far has shown that the more human-like the robot, the more likely someone will comply with the robot, which suggests to me that people don’t trust robots that remind them that they’re robots.

Looking at an existing robot, there’s the Chinese AnBot, a robot similar to those fielded by Knightscope. It has the ability to identify fugitives, decide where to patrol using data gathered during prior patrols, a button people can press to signal an emergency, and is armed. Its weapon is an electroshock probe used in riot control and currently requires a human in the loop to authorize its use. Still, it’s only a minor software change away from being able to enforce laws autonomously.

So far the security robots haven’t crossed any major boundaries, and when they crossed the minor boundary of being too intrusive they were sharply curtailed.  They’ve acted as extensions of a human security guard and fulfilled the same function that a multiplicity of highly visible cameras could.  One that’s armed and programmed to use force when its internal algorithm predicts it needs to crosses several. Ignoring the ethics of a ‘justice machine’, there’s the big question of legality. In many places, security guards are required to pass certifications and be licensed to use force – and strictly on the premises they’ve been hired to guard.  Restricting ourselves to municipalities where security guards are authorized to use force – since if a human can’t, we can assume a machine can’t either – it’s still a big question. The first robot that physically assaults someone with the intent of upholding civil security will face immense scrutiny. What does it count as?

I should note that I’m relying mostly on US case law because it’s what I’m most familiar with.  Countries with significantly different cultures may have different mores and legal structures.  Still, I think that we can consider the USA to be middle of the road or even somewhat favorable when it comes to methods by which private property is defended.

The Supreme Court of California has ruled against a ‘a device that is without discretion’ in home defense.  A section of their ruling is

“Allowing persons, at their own risk, to employ deadly mechanical devices imperils the lives of children, firemen and policemen acting within the scope of their employment, and others. Where the actor is present, there is always the possibility he will realize that deadly force is not necessary, but deadly mechanical devices are without mercy or discretion. Such devices ‘are silent instrumentalities of death. They deal death and destruction to the innocent as well as the criminal intruder without the slightest warning. The taking of human life (or infliction of great bodily injury) by such means is brutally savage and inhuman.”

But these robots do have discretion.  Their algorithms are sophisticated enough to tell one human from another and to know who is a visitor and who has unrestricted access to a property.  So an autonomous guard is not a booby trap.

Is the robot equivalent to a guard dog?  The robot may use force, it can distinguish between individuals, though arguably it’s better at recognizing specific people and acting predictably in repeated scenarios.  In the UK, a guard dog must be tied up or under control of a handler and the US has a different law in every State.  The general expectation, however, is that dogs can attack trespassers but the trespasser may be able to recover damages depending on the municipality and whether there were signs posted that the dog was there.  I don’t think a robot is analogous enough to a dog since a robot is capable of clearer communication and more predictable behavior.

So the answer is what you might expect: a robot is a robot.  It might sometimes seem like a booby trap since it’s a mechanical contrivance and sometimes like a guard dog because it roams around and can take independent action.  It’s something unique, however, because of the form its intelligence takes.  It certainly isn’t a human security guard because it lacks flexible discretion.  It does resemble all three, however, in that it only operates in a set space and (presumably) cannot exercise its authority outside of the premises it guards.  Even without specific laws regarding security robots, the case of the SPCA robot clearly shows that governments won’t allow robots to leave their designated zones.

A situation: Someone is in a place of business and is looking at a sign on the wall. They’re holding a water bottle in one hand and feeling a cramp they stretch their arm. A security robot sees this and correlates the action and object with previous situations in which vandals spray painted the wall. It doesn’t contextualize, just sees a human with a cylindrical object reaching up while looking at a wall. It goes over and tells the human to move along. The human is confused and says they’re there for legitimate business. The robot is no longer paying attention to the object, just the human not following its command. A human security guard would be contextually aware even while approaching and would have realized by now that the human visitor wasn’t doing anything wrong. The robot follows a pre-planned script and threatens force. The human might leave, angry and intending to complain or withdraw their business. Or they might stand fast and the robot – still following the script – threatens force and then uses it, pushing the human or using some other means of restraint or forced removal.

It’s a simplified case, but a possible one. The robot would only be looking for behaviors that correlate with criminal activity. In this case it notes pose, object in hand, location, and that the human is not on its list of permanent building occupants. The robot wrongly concludes criminal intent and goes through a pre-planned list of actions based on the human’s responses. Who, then, is liable? The manufacturer, the person who trained the robot, the owner, or someone else? The robot has assaulted someone without reasonable cause. It’s crossed a legal and social boundary based on a mis-classification in its algorithm-based decision tree. A human security guard would be fired or disciplined, and the company might have to pay damages. Would the robot face the same or just be reprogrammed? A robot cannot be penalized the same way a human can. In the case of a human guard, punishment is shared between the guard and company. Is the guard’s portion ignored or does the company face greater punishment since the offender is company owned equipment?

It is likely that many governments will simply ban security robots from being allowed to use force as soon as the issue comes up. There’s little demonstrated need, and legislation of what constitutes a situation in which force is allowed looks mechanistic on paper but often relies on human ‘sense’ to prevail. A robot would almost certainly be programmed to do precisely what is legal, and that often fails to be humane. Additionally, while a robot might technically be ‘well trained’ from a certification standpoint (and once one robot can demonstrate all the training material, they all can), and even if they could demonstrate that they would only use force when appropriate, that probably won’t be enough. What we saw in the above examples made it clear that robots in the real world aren’t trusted to patrol and deter crime. Governments won’t be willing to allow armed robots in civil situations if there’s no demonstrated need and a high chance of public outcry. Unlike AVs, the public doesn’t want a robot that tells them what to do and where they can go.

In the public consciousness, robots exist to serve. When they tell us what to do, we get annoyed and when they use force to make us comply then we’re angry and afraid. Our stories are filled with examples of the creation destroying or enslaving its creator, and that’s what most people will think of when they see an automaton enforcing the law. They’ll feel as if they are being dominated by the robot even if that robot is doing exactly what another human has instructed them to. Until people are used to seeing passive security robots, they’re unlikely to be willing to accept active ones. Even then, the boundary being crossed – that of a non-human having physical authority over a human – may engender resentment even after we’ve gotten used to seeing the surveillance bots patrolling our buildings and streets.

In the last three sections of this article, we followed the events of the first autonomous vehicle fatality involving a pedestrian.  We covered how an autonomous Uber struck and killed a pedestrian in Tempe, Arizona, and then how the AV industry decided to join the voices condemning Uber rather than go to their aid.  In the third section, both the governor of Arizona and Uber themselves were attacked repeatedly and their relationships exposed as close, to say the least, with preferential treatment seeming to be the standard practice for Governor Ducey.  We end here with the advent of the preliminary report which I used to lay out the events of the night, and how Uber turned the situation around to the point where they are now back to testing on public roads – albeit with fewer cars and drivers than before.

The preliminary report is where the driver’s culpability started to surface again in media reports.  At first, it was just a hint buried in the report that the local media outlets – always on the lookout for more details about the Tempe crash – found and ran with.  The report’s suggestion wasn’t anything particularly new, just the first official confirmation that the AV driver was responsible for braking – and thus responsible for anything that happened if the car didn’t brake – in an emergency.  It was the first hint, and the beginning of a turn in the investigation that has given Uber a slim chance of salvaging the tragedy and going on with development.

It was the Tempe police department that uncovered that the driver had been streaming Hulu on her phone at the approximate time of collision.  This discovery completely changed the interpretation of what the internal camera had captured.  Instead of looking at the AV interface as the driver had claimed at the time, it appeared that she was watching a show on her phone.  All of a sudden the blame and media attention focused squarely on the driver, especially since the investigations suggested that had the driver braked when the pedestrian became visible, the pedestrian would have made it to the other side of the street before the car could hit her.

Uber grabbed the opportunity.  They stressed the training where operators of their AVs spend three weeks on a closed course learning how to safely operate the car, and also their strict policy of firing anyone found to be using their phone while driving the car.  This was their opportunity to improve their training and safety culture while placing blame on an individual, and thus being able to take action without admitting being at fault.

Slight problem: the internal camera.  One of Uber’s main ways of determining whether a driver was distracted was random spot checks of the internal camera to see what the driver was doing during a test.  In this crash, the internal camera was not only reviewed by Uber, but by hundreds of media outlets, their readers, and authorities.  At no point before the Hulu stream records were obtained was the driver accused of using her phone.  If one driver could make it look like they were monitoring the AV interface when in actuality watching a video on their phone, then how many other drivers were on the road doing exactly the same thing?  Uber wasn’t out of the woods yet, and they evidently decided that they needed to take more drastic action than a safety review.

Less than a month after the Hulu records were published, Uber fired almost all of its AV operators.  Those drivers were put at the head of the line to re-apply for the job of ‘mission specialist‘ – although there were only 55 mission specialist positions open as opposed to the 100+ AV drivers who’d just been laid off.  The plan appeared to be to scale back testing to the closed tracks again, with the mission specialists being better trained and more technically focused than the drivers.  The change would mean focusing more on testing with specific circumstances and issues rather than sending the AV out into the wild to find problems.  The mission specialists would understand their vehicles a lot better, and the cars themselves would be intensively tested in controlled conditions before a limited revival of testing on public roads.

The clock was ticking, though.  Waymo and GM plan on having their AVs on the road and providing taxi services later this year and in 2019 respectively, while Ford is aiming for a 2021 rollout.  Uber was first to market, but the setback here was threatening to lose them that advantage.  Waymo has the backing of Alphabet and GM is a well known and trusted brand.  If they hit the road and made a big success before Uber had captured market share, Uber might get pushed out completely.  Ever the company best known for fast and decisive action, Uber had its mission specialists on the roads of Pittsburgh less than two weeks after the restructure.  The closed course shakedown appeared successful; the AVs once again had collision avoidance and emergency braking controlled by the computer, and new driver attentiveness monitoring was reportedly integrated with existing systems.  They’d also decided that it was time to focus their own attention.  They shut down their autonomous truck division, likely shifting all technical work in that group to their cars.  As of now, Uber appears to be determined to regain its competitive advantage of being first on the ground and first into people’s hands.  The public awareness of the Tempe fatality is fading from the public’s interest, which is good for Uber’s executives, given all that they have to deal with.  Just in the last couple weeks, they’ve been facing regulation in New York City and in Spain which would cap the number of cars they’re allowed to have on the streets.  The changes to their operations resulting from being the first AV company with a pedestrian death attributed to their system are likely to continue.

For now, it appears as if the worst thing the industry as a whole faces is federal gridlock and no surety as to what the Department of Transportation will eventually settle on as regulatory status quo nationwide.  That could easily change as more companies field much larger fleets with the rollout of AV taxis across the country.  One death is a tragedy and a fluke; two or three could swing the pendulum the other way, and given how favorable regulations have been so far, the pendulum could sweep a lot of companies off the board completely.  A recent survey shows declining trust for AVs among consumers,  This suggests that the negative news surrounding AVs has had an effect.  The industry will have to work hard to regain that lost ground, and if Uber wants to be first to market, they’ll have the responsibility of working hardest at proving that their technology is beneficial and dependable.  Otherwise companies with better reputations like Waymo or more trust like GM will take away that market share, and Uber’s strategy of being first will have been what ends their ambitions of being at the forefront of public AV use.  If consumers aren’t convinced that AVs are necessary and safe, then regulators may decide to tighten restrictions on them rather than face the same kind of backlash governor Ducey did.

In the last two sections of this article, we saw how an autonomous Uber struck and killed a pedestrian in Tempe, Arizona, and then how the AV industry decided to join the voices condemning Uber rather than go to their aid.  Technology does not develop in a vacuum, however, and as the days passed, the governor of Arizona began to get hit almost as badly as Uber.

By the 26th, 8 days after the crash and a week after press coverage began, the staunchly pro-industry governor of Arizona, Doug Ducey, had taken all the beating he could stand. He suspended Uber’s permission to test their AVs in his State – a somewhat empty gesture since they’d already halted testing everywhere. The message was that he was looking out for Arizona’s citizens, but investigative reporting suggested that he was bracing for the inevitable revelations about his relationship with Uber. Barely two days after the executive order, The Guardian broke the story and got the evidence, and the news spread to major outlets.

Doug Ducey had been very friendly to Uber during their time in his State. A month after his swearing in in 2014, Governor Ducey met with members of Uber’s leadership and in April of 2015 he and the State legislature legalized ride-sharing throughout the State, overriding any city or town regulations. During the legalization ceremony, Uber even got the schedule changed so that one of their drivers introduced the governor instead of the other way around, though they failed to convince him to wear one of their shirts for the event. In June, Ducey had signed an executive order (with help from Uber) making it legal for them to test their AVs on public roads as long as they had a driver, and on university campuses without a driver. The EO also provided for an oversight committee. Uber wanted to have a representative, but didn’t get that. However, the committee was made up of 8 members all appointed by Ducey and only one of whom was an expert – a professor of systems engineering at the University of Arizona. The committee met no more than twice, and never took any noticeable actions.

In 2016, the governor’s office reportedly leaned on the Phoenix city council to allow Uber to pick up riders at Sky Harbor airport. The fact that Uber sent out a tweet thanking the governor for something that the Phoenix city council did when the plan was implemented suggests some truth to this. In addition, the following month Ducey himself tweeted a commercial written for him that praised Uber Eats and provided a link to their service. Uber then began testing its AVs in Arizona’s public streets – something it hadn’t actually begun until then – but chose to maintain publicity only for Pittsburgh, even going so far as to ask that someone ‘discreet’ be informed in the Arizona highway patrol. There was no public reporting requirement in Arizona law, so Uber was able to do its AV testing without informing anyone, and chose only to tell the governor’s office and have them decide who to pass the message to. They also offered office space to Ducey’s deputy chief of staff when the deputy chief visited California that same month.  The Arizona government disputed the Guardian’s assertion that they hadn’t informed anyone, pointing out that an article in August of 2016 in the Capital Times which stated that both Waymo and Uber were testing on Arizona roads. It is true that it’s hard to ‘keep it quiet’ if it’s in a newspaper, but it’s also true that the governor apparently made little of the 2016 tests in public statements.

December 2016 was when the California DMV made it clear that Uber was not welcome after it tested in San Francisco without a permit. When it chose to move all the AVs it had been testing in CA to AZ, Ducey issued a glowing public statement. He followed this up with a January state of the State which called CA’s regulations ‘nutty’ and promised to repeal even more regulations to make his State friendlier still to AVs. No mention was made that Uber had been testing in AZ for months prior to its CA fleet being added to the existing AZ fleet.

All of this was good news for Democrats in Arizona, and bad news for Uber. Little that they did was more than the type of behavior one would expect out of a high powered lobbyist, but it was fodder for opponents and competitors in the wake of this high profile death. One of the the core principles of influencing public policy is that the more you pull the pendulum your way, the further it will swing against you if things go wrong. It’s always important to plan your fallback positions to be at least as strong as your main thrust of policy influence, and Uber’s fallbacks were proving ineffective. Especially since Ducey seemed willing to throw Uber under the bus, but Uber had not appeared to plan out a counterthrust should their political allies turn on them. Speaking purely from my own opinion now, I’d say they ought to have had a policy document in their back pocket which would have provided tough regulations and a panopticon of oversight ready to be proposed the moment something like this happened.

The coverage also resulted in the Phoenix New Times – an alternative newspaper which has several strong editorial opinions on Arizona’s politics – to abandon its fairly balanced coverage in favor of a piece slamming the governor. Leading with what appears to be a picture of a car with a goblin head on it, the article harshly criticizes the governor, and some of their zeal spills over onto the AV industry. They point out that the accident rate of AVs is currently higher than that of conventionally piloted vehicles, somewhat disingenuously. It is true that – as they quote from a professor in AZ – conventional cars go 100 million miles annually per fatality versus AVs current rate of 10 million. It is also true that the number in 1925 was approximately 18 per hundred million VMT, almost 20 years after the Ford Model T was released. AVs may not be doing as well as we are now, but we took a long time to reach that 1 per 100 million. We didn’t even get to the 10 per 100 million until the 40s.

It’s fair to say that Uber got hammered far worse than Ducey in the overall press coverage. As the revelations about his communications with Uber came out, so did another round of articles about issues with the AVs and their drivers and even an attempt to settle with the victim’s family backfired. The previously mentioned New York Times piece had been the rumbles, and now the avalanche had arrived. Reports came out that drivers had felt deep misgivings about the technology and the testing routines of the company. While it’s true that in a company the size of Uber, it’s not hard to find someone with a negative opinion about something after a disaster occurs, the drivers’ allegations were very troubling, though in hindsight make perfect sense.

It was no secret that Uber had mostly swapped from taking fares to just trying to get as many miles on the systems as possible. It was a good publicity event when they had a driver and a technician on board to also pick up passengers, but once they went over to one driver they also decided that gathering data for the algorithm was the highest priority. To keep up public interests, some of the AVs still stopped for passengers, but most were out sucking up the data for the software. Hence why the driver in the crash was on their second loop around a predefined route. The predictability made the machines’ jobs easier, but added to the boredom factor for the drivers. They spent hours monitoring the system, alone and with little stimulation. Being on predefined routes, there wasn’t much new to see, and they had to remain alert in case the system encountered something it couldn’t deal with. As we saw earlier, visual and auditory warnings were lacking in several areas, so they had to watch the road. It’s one thing to do this with someone in the car to talk to, or through unfamiliar roads, but they had neither. The drivers say they were encouraged to get in x number of miles per day, though Uber contends there was no quota and an expectation that the drivers would take breaks.

Many articles also made mention of Uber’s difficulties with their AV algorithm. They weren’t going nearly as many miles per intervention as Waymo. Waymo’s engineers have gone on record to say that the distance an AV goes before a driver has to intervene is an unreliable metric for how well the AV works and tends to reflect how much the algorithm is being pushed than how good it is. That is to say, the lower the number, the more aggressive the development speed of the algorithm.

That said, they were also struggling with false positives causing frequent and uncomfortable braking and swerving. It makes sense as the reason why the emergency brake system had been disabled and the algorithm took so long to classify the woman with her bike. Uber was testing with a high threshold of confidence on their classifier and prioritizing a smooth ride during these tests, requiring the algorithm to be very sure before deciding that something in the road was an obstacle or person instead of a cloud of steam or a plastic bag. Both of these things have been cited as difficult things for the system to classify. In May reports suggested that investigators were leaning towards this as the ultimate cause of the crash, though since then the blame has shifted more towards the driver who appeared distracted.

Like with Tesla Autopilot, it was down to the driver to make sure the system was working safely at all times. The element of trust has been picked up in other publications and in interviews with the Uber AV drivers. It’s even more dangerous in a system that’s pushing into level 3 because the driver can safely tune out most of the time. Their miles per intervention must have also been improving because the drivers described going hours without touching the wheel – though no mention of the brakes. Uber’s AV may or may not have required frequent intervention in the daytime, but at night with little but the well known road itself and the traffic signals, a driver could go an entire 8 hour night shift without ever hitting a snag. It’s easy as to lose track and trust the AV implicitly as it would be if they were being driven by a human.

Uber announced that they were conducting a safety review in the beginning of May – shortly after the reports of the investigators’ belief that the algorithm played a major role. They’d resume testing of AVs on public roads ‘in a few months’, suggesting that this was when they expected their internal review to end. This did little to turn the media’s focus away from the disabling of automatic brakes and the algorithm’s apparent lack of ability to handle something that happens most nights in any city. The day before the release of the NTSB preliminary report that I used to lay out the events of the collision itself, Uber announced that they would not be doing any testing in Arizona, even after they resumed testing elsewhere. Pittsburgh remained a major hub of development and testing, and they remained hopeful that they could make arrangements with California to return to testing there. Coverage of the preliminary report was similar to coverage so far, pointing out the braking issue, and some including a mention of Waymo’s future plans to have ride-hailing AVs ready in AZ by the end of 2018.

Next time, we wrap up with Uber’s attempts to recover their competitive advantage of being first on the road, including how they found something they’d needed throughout the investigation – a scapegoat.

Wrapping up (for now) the series on autonomous vehicle collisions, we have the incident I’ve spent the most time studying and thinking about because it’s had such far-reaching implications.  It was such a big story with so many twists and turns that I’ve decided it would be better to release in sections, so here’s section 1.

The final post in this series (until something else happens) will come as no surprise to anyone with even a cursory interest in AVs.  The pedestrian fatality in Tempe; the only confirmed pedestrian death involving an AV.  I’m going to skip ahead briefly to the NTSB’s preliminary report to present the facts as they appeared on the night of the accident, and then go back to the beginning of coverage and discussion.  By going week by week and sometimes day by day, I’ll walk through the evolution of how a tragic death of a pedestrian turned into a whirlwind of change for one of the most prominent AV developers – Uber.

It was about 10 at night in March of this year when a woman was walking her bicycle across Mill Avenue in Tempe, Arizona.  Ignoring signs to use the crosswalk 100 meters away she walked across Mill Avenue in an area between the streetlights.  An autonomous Uber with a test operator was driving a preplanned test loop along the Tempe streets and had already completed one full circuit.  The car had been in autonomous mode for 19 minutes when it struck the pedestrian, killing her.  There was minimal damage to the Uber and no harm to the test driver.  One of the outward facing cameras captured the moments before the collision and an inward facing camera shows the operator looking down frequently at something before the crash occurred.  When the police arrived, the driver said she was monitoring the AV’s interface – a requirement for the job so that drivers can flag mistakes the car makes so the engineers can improve the algorithm.

Uber AVs have significant sensor coverage, including a top-mounted LIDAR.  The car’s algorithm went through several classifications of what it saw in the road, including another car, an unknown object, and finally a bicycle.  This is relatively normal since as it gets closer, the sensor image becomes more clear, and so classification becomes more accurate.  At 1.9 seconds before collision, the car designated a need for emergency braking.  Emergency braking having been disabled by Uber as a nuisance in autonomous mode and no alert system having been implemented, the driver was not warned and the car did not have time to initiate emergency evasion maneuvers which it has the authority to engage on its own. The car struck the pedestrian at 39 mph and the operator braked after hitting her, although she did try to pull the wheel over less than a second before impact.

So, those were the facts as they appeared to be on the day.  As the story unfolded, other developments changed the story, but this is what would have been known if all the data could have been extracted from the car on that night.  Before I go on, however, I should make it clear what kind of system the Uber AV is.  I’ve spent several posts now on Tesla Autopilot which is solidly in level 2 of autonomy where the human driver is still absolutely necessary at all times.  Uber’s system is level 3 – in theory – and so the driver is expected to engage in an emergency and in unexpected situations, but otherwise is not involved in the activity of driving.  As such, they have to pay attention, but rarely actually do anything.  In theory.  However, taking the action of braking (or at least, emergency braking) away from the AV and not having an alert may arguably kick the AV back down to a very high level 2 since it’s a key function that has not been given over to the AV system.

Now to the timeline.

The fatality occurred the night of March 18th.  By the 19th, the articles were already surprisingly vehement – in contrast to the articles written about Tesla collisions.  Some of this might be because it was a pedestrian that was killed, and some may be because Uber has a bit of a different reputation in business.  In the last couple of weeks, the shine has been taken off of Elon Musk, but four months ago Tesla was very well thought of and its leadership pretty well respected.  Uber, on the other hand, was respected technically, but in the wider media had taken a beating from reports of harassment and discrimination in the workplace, bans in multiple countries and municipalities over allegations of unfair trade practices, and disputes over whether Uber drivers are being given fair compensation for their work.  Uber was a high tech company with private investment funding and a stupendously high burn ratio.  It progressed fast, hard, and energetically to try to keep ahead and it often needed to attract more investment to keep going.  Its highest overhead was its drivers and it was going into AV in a big way to try to get ahead of Tesla, Google, and the big automakers.  In the new economy where owning a vehicle was no longer necessary, Uber wanted to be the go-to app to call a car.  That said, Uber had gotten some good press in the past after a collision in Tempe which flipped one of its cars over.  In this crash, it was the other vehicle that was at fault.  This spawned at least one opinion piece in Wired praising AVs and opining that we needed them as soon as possible to prevent more accidents like the one the Uber AV had been involved in.

Unlike with Tesla, many of the articles that came out in the aftermath of this fatality focused on regulation as much as technology and the facts of the case.  Rightly or wrongly the media narrative for Uber had gone poorly in the past, and being in the position of the villain in previous stories meant that they were in for mixed press at best even before much was known about the circumstances of the incident.  The day after the collision, the New York Times published an article which included a section on AZ regulation and the reaction of the Phoenix government.  TechCrunch included a quote from the California DMV, a subtle dig given the shared history with Uber.  Uber had previously been testing in CA, but had moved to AZ when they were ordered to stop testing on city streets without a permit.  Including quotes from the regulatory agency which had censured and blocked them in the past sends a message that Uber was going to have to prove it hadn’t acted recklessly and that California wasn’t right to push them out.

On the same day as these articles were being published, a joint letter from a variety of consumer watchdogs, disability advocacy organizations, and prominent advocates was sent to the committees considering the AV START Act.  The Act is a Senate bill that was introduced in September of 2017, and passed unanimously through the Commerce committee in October.  The House version – the SELF DRIVE Act – had passed already when this committee was having its hearing.  The letter was sent in March of this year and urged the committees and the Senate in general to delay further consideration until the NTSB had completed its investigation in Tempe.  It went on to ask that they make changes to the Act to tighten the rules on exemptions and require minimum design and reporting standards.  The letter also brought up the fact that the AV START Act would preempt State regulation on passing, even though the regulations of the Act would require time to draw up and implement, leaving AVs effectively unregulated for the period it would take the Department of Transportation to promulgate its rules.  The swiftness of this letter might have shown just how big the incident was in the public consciousness.  It wasn’t just Uber that was in trouble; it was the entire AV industry.

However, it might have also been a hasty rewrite, since on March 14, four days before the crash, five US Senators lead by the Diane Feinstein had written a letter explaining why they were voting against the Act.  This vote meant it would not be fast-tracked through the Senate and instead need debate before it could be passed and likely presaged an attempt to make changes to the Act to increase oversight and regulatory powers.  It was only to be expected; Senator Feinstein has expressed her conservatism (with respect to AVs) many times over the last couple years.  It was unlikely that AV START would pass easily, and with the Uber crash and the recall of Takata airbags that was happening at the same time, it became likely that it would not pass for months if ever.

This might seem like a setback for industry, but given that many States (including Arizona) have quite permissive regulatory structures, it’s hard to say.  The current situation puts the responsibility more in the hands of industry, so if something goes wrong they’re the ones on the hook.  Some States – like California – regulate AVs more than others, but the fact remains that most State governments are hands-off.  The flipside is that should the DOT be given the tools to regulate, it’s unclear what they would do with them.  The current administration is difficult to predict, and industry may be hoping the Act will pass later on when they can be more certain of what degree of regulation they’ll get.

Less than two days after the collision, Stanford posted an interview with a law professor about the liability and regulations surrounding the incident.  He noted that

“…if the safety driver failed to exercise reasonable care in avoiding the accident, Uber would be responsible for the driver’s negligence. If the automated features of the AV failed to note the presence of the victim, the manufacturer of the vehicle, as well as Uber, could be held responsible under product liability principles.”

He went on to point out that the pedestrian might share partial fault if they crossed in an unsafe manner.

The heat was turning up and Uber did the best thing it could in the situation.  It suspended testing on all public roads in the four cities they’d rolled out to so far.  This likely helped their image – as well as giving the engineers at Uber Advanced Technology Group (ATG) the chance to go over the incident with a fine-toothed comb without more data piling up.  On the other hand, more data for the algorithmic engineers might not have mattered since it was unlikely that they were the department that was about to get a lot more work.  It is a well known principle of machine learning algorithms that the negative set is usually nowhere near as large as the positive set.  That is to say, the examples of things to do right has to be many times more numerous than examples of what to do wrong.  In that way, humans and AIs are very similar.  It is very likely that this was not the first time an AV – or even an Uber – had been in this situation, and given the facts so far, this was already shaping up to be not as much an algorithmic as a human-robot interface issue.  After all, the car knew something was up – albeit only a second or so before the collision – but there was no means for it to warn the driver despite it being the driver who was expected to do something about it.

Tune in next time when the AV industry looks at one of their beleaguered colleagues suffering a situation that might shatter trust in everything they’ve worked years to develop … and says “who, us?”