Monday, August 15, 2016

Repost of Michael Barr's Bookout victory comments

Well-encrusted, but far too ornate and heavy

Here below is a frank blog post by Michael Barr, the leader of the embedded systems experts who examined Toyota's source code under conditions of tight security imposed by Toyota to protect the secrecy of the software it called its "Crown Jewel."
After Barr gave expert testimony that Toyota's ETC failsafe was like a "house of cards," I think Toyota has ceased describing its software with those words. Not a queen of diamonds there.

Barr's concerns about the source code's poor quality echo those of the Toyota engineers, as they wrote in the documents.  Here is Barr's blog post:

An Update on Toyota and Unintended Acceleration
Saturday, October 26th, 2013 by Michael Barr
(downloaded on 29 October 2013)
Michael Barr
Michael Barr is an expert on the design of software-powered medical
devices and other embedded computer systems. (full bio)

In early 2011, I wrote a couple of blog posts (here and here) as well as a later article
(here) describing my initial thoughts on skimming NASAs official report on its
analysis of Toyotas electronic throttle control system. Half a year later, I was
contacted and retained by attorneys for numerous parties involved in
suing Toyota for personal injuries and economic losses stemming from incidents of
unintended acceleration. As a result, I got to look at Toyotas engine source code
directly and judge for myself.
From January 2012, Ive led a team of seven experienced engineers, including three
others from Barr Group, in reviewing Toyotas electronic throttle and some other
source code as well as related documents, in a secure room near my home in
Maryland. This work proceeded in two rounds, with a first round of expert reports and
depositions issued in Summer 2012 that led to a billion-dollar economic loss
settlement as well as an undisclosed settlement of the first personal injury case set
for trial in U.S. Federal Court. The second round began with an over 800 page formal
written expert report by me in April 2013 and culminated this week in an Oklahoma
jurys decision that the multiple defects in Toyotas engine software directly caused a
September 2007 single vehicle crash that injured the driver and killed her passenger.
Dont be misled by much of the mainstream coverage of the Oklahoma verdict. While
it is true this was the first time Toyota has lost an unintended acceleration case in
court, it is more significant that this was the first and only jury so far to hear any
opinions about Toyotas software defects. Each of the earlier cases either predated
our source code access, applied a non-software theory, or was settled by Toyota for
an undisclosed sum.
In our analysis of Toyotas source code, we built upon the work that NASA had done.
First, we looked more closely at more lines of the source code for more vehicles for
more man months. And we also did a lot of things that NASA didnt have time to do,
including reviewing Toyotas operating systems internals, reviewing the source code
for Toyotas monitor CPU (which even Toyota hadnt ever done before! (!)),
performing an independent worst-case stack depth analysis, running portions of the
main CPU software including the RTOS in a processor simulator, and
demonstratingin exemplar Toyota Camry vehiclesa link between loss of throttle
control and the numerous defects we found in the software.
In a nutshell, the team led by Barr Group found what the NASA team sought but
couldnt find: a systematic software malfunction in the Main CPU that opens the
throttle without operator action and continues to properly control fuel injection and
ignition that is not reliably detected by any fail-safe. To be clear, NASA never
concluded software wasnt at least one of the causes of Toyotas high complaint rate
for unintended acceleration; they just said they werent able to find the specific
software defect(s) that caused unintended acceleration. We did.
Now its your turn to judge for yourself. Though I dont think you can find my 800
page expert report outside the Court system, heres the trial transcript [*] of my expert
testimony to the Oklahoma jury in Bookout, v. Toyota.
Note that the jury in Oklahoma went with the software defects and found that Toyota
owed each victim $1.5 million in compensatory damages and also found reckless
disregard. The latter legal standard meant the jury was headed toward deliberations
on additional punitive damages when Toyota finally called the plaintiffs to settle (for
yet another undisclosed amount). I understand there are about 500 personal injury
cases still working their way through various courts, including one set for trial in
November in U.S. District Court in Santa Ana, California.
2 Responses to An Update on Toyota and Unintended Acceleration
1. Miro Samek says:
October 28, 2013 at 4:49 pm
Hi Michael,
Thank you for posting the link to your court deposition. I found it fascinating
and couldnt stop reading late into the night
There is no doubt in my mind that exposing the inadequacies in the Toyota
firmware is a very important development for the whole embedded software
It is also interesting to see old mistakes repeated time and time again. For
example a timed task degenerating into a kitchen sink.
I also bet my shirt that there were no assertions in the Toyota firmware.
Assertions in software work like fuses in electrical systems and beyond
certain density of assertions in the code all failures (including hardware
failures) manifest themselves as assertion violations. Im sure that this could
have saved the day (besides making software development so much faster).
Anyway, there are tons of valuable lessons to learn here. From now on I will
imagine that all my software is on trial
2. David W. Gilbert, Ph.D. says:
October 28, 2013 at 10:25 pm
Dear Mr. Barr,
Nicely done! I found your testimony very interesting, and while I am not a
software expert, I can certainly verify the inability of Toyota vehicles to detect
certain malfunctions in the electronic throttle controls. And few malfunctions
are more apparent than tin whiskers growing inside the APP sensors!
Since my 2010 testimony in the Washington Toyota hearings, I have learned
much. Your testimony certainly adds to that knowledge and I am pleased that
it has received much needed media attention.
Maybe our paths will cross someday.
[*] Trial transcript is available on the Safety Research & Strategies website