Well-encrusted, but far too ornate and heavy
Here below is a frank blog post by Michael Barr, the leader of the embedded systems experts who
examined Toyota's source code under conditions of tight security imposed by
Toyota to protect the secrecy of the software it called its "Crown
Jewel."
After Barr gave expert testimony that Toyota's ETC failsafe was like a
"house of cards," I think Toyota has ceased describing its software
with those words. Not a queen of diamonds there.
Barr's concerns about the source code's poor quality echo those of the Toyota engineers, as they wrote in the documents. Here is Barr's blog post:
An Update on Toyota and
Unintended Acceleration
Saturday, October 26th, 2013
by Michael Barr
http://embeddedgurus.com/barr-code/2013/10/an-update-on-toyota-and-unintended-acceleration/
(downloaded on 29 October 2013)
Michael Barr
Michael Barr is an expert on the design of
software-powered medical
devices and other embedded computer systems. (full bio)
In early 2011, I wrote a couple of blog posts (here and
here) as well as a later article
(here) describing my initial thoughts on skimming NASA’s official report on its
analysis of Toyota’s electronic throttle control
system. Half a year later, I was
contacted and retained by attorneys for numerous
parties involved in
suing Toyota for personal injuries and economic losses
stemming from incidents of
unintended acceleration. As a result, I got to look at
Toyota’s engine source code
directly and judge for myself.
From January 2012, I’ve led a team of seven
experienced engineers, including three
others from Barr Group, in reviewing Toyota’s electronic throttle and some other
source code as well as related documents, in a secure
room near my home in
Maryland. This work proceeded in two rounds, with a
first round of expert reports and
depositions issued in Summer 2012 that led to a
billion-dollar economic loss
settlement as well as an undisclosed settlement of the
first personal injury case set
for trial in U.S. Federal Court. The second round began
with an over 800 page formal
written expert report by me in April 2013 and
culminated this week in an Oklahoma
jury’s decision that the multiple
defects in Toyota’s engine software directly
caused a
September 2007 single vehicle crash that injured the
driver and killed her passenger.
Don’t be misled by much of the
mainstream coverage of the Oklahoma verdict. While
it is true this was the first time Toyota has lost an
unintended acceleration case in
court, it is more significant that this was the first
and only jury so far to hear any
opinions about Toyota’s software defects. Each of
the earlier cases either predated
our source code access, applied a non-software theory,
or was settled by Toyota for
an undisclosed sum.
In our analysis of Toyota’s source code, we built upon
the work that NASA had done.
First, we looked more closely at more lines of the
source code for more vehicles for
more man months. And we also did a lot of things that
NASA didn’t have time to do,
including reviewing Toyota’s operating system’s internals, reviewing the
source code
for Toyota’s “monitor CPU” (which even Toyota hadn’t ever done before! (!)),
performing an independent worst-case stack depth
analysis, running portions of the
main CPU software including the RTOS in a processor
simulator, and
demonstrating–in exemplar Toyota Camry
vehicles–a link between loss of throttle
control and the numerous defects we found in the
software.
In a nutshell, the team led by Barr Group found what
the NASA team sought but
couldn’t find: “a systematic software
malfunction in the Main CPU that opens the
throttle without operator
action and continues to properly control fuel injection and
ignition” that is not reliably detected by any fail-safe. To be clear, NASA never
concluded software wasn’t at least one of the causes
of Toyota’s high complaint rate
for unintended acceleration; they just said they weren’t able to find the specific
software defect(s) that caused unintended acceleration.
We did.
Now it’s your turn to judge for
yourself. Though I don’t think you can find my 800
page expert report outside the Court system, here’s the trial transcript [*] of my expert
testimony to the Oklahoma jury in Bookout, et.al. v.
Toyota.
Note that the jury in Oklahoma went with the software
defects and found that Toyota
owed each victim $1.5 million in compensatory damages
and also found “reckless
disregard”. The latter legal standard
meant the jury was headed toward deliberations
on additional punitive damages when Toyota finally
called the plaintiffs to settle (for
yet another undisclosed amount). I understand there are
about 500 personal injury
cases still working their way through various courts,
including one set for trial in
November in U.S. District Court in Santa Ana,
California.
***********************************************************
2 Responses to “An Update on Toyota and
Unintended Acceleration”
1. Miro Samek says:
October 28, 2013 at 4:49 pm
Hi Michael,
Thank you for posting the link to your court
deposition. I found it fascinating
and couldn’t stop reading late into the
night…
There is no doubt in my mind that exposing the
inadequacies in the Toyota
firmware is a very important development for the whole
embedded software
profession.
It is also interesting to see old mistakes repeated
time and time again. For
example a timed task degenerating into a kitchen sink.
I also bet my shirt that there were no assertions in
the Toyota firmware.
Assertions in software work like fuses in electrical
systems and beyond
certain density of assertions in the code all failures
(including hardware
failures) manifest themselves as assertion violations.
I’m sure that this could
have saved the day (besides making software development
so much faster).
Anyway, there are tons of valuable lessons to learn
here. From now on I will
imagine that all my software is on trial…
–Miro
2. David W. Gilbert, Ph.D. says:
October 28, 2013 at 10:25 pm
Dear Mr. Barr,
Nicely done! I found your testimony very interesting,
and while I am not a
software expert, I can certainly verify the inability
of Toyota vehicles to detect
certain malfunctions in the electronic throttle
controls. And few malfunctions
are more apparent than tin whiskers growing inside the
APP sensors!
Since my 2010 testimony in the Washington Toyota
hearings, I have learned
much. Your testimony certainly adds to that knowledge
and I am pleased that
it has received much needed media attention.
Maybe our paths will cross someday.
DWG
***************************************************
[*] Trial transcript is available on the Safety
Research & Strategies website
