Well-encrusted, but far too ornate and heavy
Here below is a frank blog post by Michael Barr, the leader of the embedded systems experts who
examined Toyota's source code under conditions of tight security imposed by
Toyota to protect the secrecy of the software it called its "Crown
Jewel."
After Barr gave expert testimony that Toyota's ETC failsafe was like a "house of cards," I think Toyota has ceased describing its software with those words. Not a queen of diamonds there.
Barr's concerns about the source code's poor quality echo those of the Toyota engineers, as they wrote in the documents. Here is Barr's blog post:
An Update on Toyota and Unintended Acceleration
Saturday, October 26th, 2013 by Michael Barr
http://embeddedgurus.com/barr-code/2013/10/an-update-on-toyota-and-unintended-acceleration/
(downloaded on 29 October 2013)
Michael Barr
Michael Barr is an expert on the design of software-powered medical
devices and other embedded computer systems. (full bio)
In early 2011, I wrote a couple of blog posts (here and here) as well as a later article
(here) describing my initial thoughts on skimming NASA’s official report on its
analysis of Toyota’s electronic throttle control system. Half a year later, I was
contacted and retained by attorneys for numerous parties involved in
suing Toyota for personal injuries and economic losses stemming from incidents of
unintended acceleration. As a result, I got to look at Toyota’s engine source code
directly and judge for myself.
From January 2012, I’ve led a team of seven experienced engineers, including three
others from Barr Group, in reviewing Toyota’s electronic throttle and some other
source code as well as related documents, in a secure room near my home in
Maryland. This work proceeded in two rounds, with a first round of expert reports and
depositions issued in Summer 2012 that led to a billion-dollar economic loss
settlement as well as an undisclosed settlement of the first personal injury case set
for trial in U.S. Federal Court. The second round began with an over 800 page formal
written expert report by me in April 2013 and culminated this week in an Oklahoma
jury’s decision that the multiple defects in Toyota’s engine software directly caused a
September 2007 single vehicle crash that injured the driver and killed her passenger.
Don’t be misled by much of the mainstream coverage of the Oklahoma verdict. While
it is true this was the first time Toyota has lost an unintended acceleration case in
court, it is more significant that this was the first and only jury so far to hear any
opinions about Toyota’s software defects. Each of the earlier cases either predated
our source code access, applied a non-software theory, or was settled by Toyota for
an undisclosed sum.
In our analysis of Toyota’s source code, we built upon the work that NASA had done.
First, we looked more closely at more lines of the source code for more vehicles for
more man months. And we also did a lot of things that NASA didn’t have time to do,
including reviewing Toyota’s operating system’s internals, reviewing the source code
for Toyota’s “monitor CPU” (which even Toyota hadn’t ever done before! (!)),
performing an independent worst-case stack depth analysis, running portions of the
main CPU software including the RTOS in a processor simulator, and
demonstrating–in exemplar Toyota Camry vehicles–a link between loss of throttle
control and the numerous defects we found in the software.
In a nutshell, the team led by Barr Group found what the NASA team sought but
couldn’t find: “a systematic software malfunction in the Main CPU that opens the
throttle without operator action and continues to properly control fuel injection and
ignition” that is not reliably detected by any fail-safe. To be clear, NASA never
concluded software wasn’t at least one of the causes of Toyota’s high complaint rate
for unintended acceleration; they just said they weren’t able to find the specific
software defect(s) that caused unintended acceleration. We did.
Now it’s your turn to judge for yourself. Though I don’t think you can find my 800
page expert report outside the Court system, here’s the trial transcript [*] of my expert
testimony to the Oklahoma jury in Bookout, et.al. v. Toyota.
Note that the jury in Oklahoma went with the software defects and found that Toyota
owed each victim $1.5 million in compensatory damages and also found “reckless
disregard”. The latter legal standard meant the jury was headed toward deliberations
on additional punitive damages when Toyota finally called the plaintiffs to settle (for
yet another undisclosed amount). I understand there are about 500 personal injury
cases still working their way through various courts, including one set for trial in
November in U.S. District Court in Santa Ana, California.
***********************************************************
2 Responses to “An Update on Toyota and Unintended Acceleration”
1. Miro Samek says:
October 28, 2013 at 4:49 pm
Hi Michael,
Thank you for posting the link to your court deposition. I found it fascinating
and couldn’t stop reading late into the night…
There is no doubt in my mind that exposing the inadequacies in the Toyota
firmware is a very important development for the whole embedded software
profession.
It is also interesting to see old mistakes repeated time and time again. For
example a timed task degenerating into a kitchen sink.
I also bet my shirt that there were no assertions in the Toyota firmware.
Assertions in software work like fuses in electrical systems and beyond
certain density of assertions in the code all failures (including hardware
failures) manifest themselves as assertion violations. I’m sure that this could
have saved the day (besides making software development so much faster).
Anyway, there are tons of valuable lessons to learn here. From now on I will
imagine that all my software is on trial…
–Miro
2. David W. Gilbert, Ph.D. says:
October 28, 2013 at 10:25 pm
Dear Mr. Barr,
Nicely done! I found your testimony very interesting, and while I am not a
software expert, I can certainly verify the inability of Toyota vehicles to detect
certain malfunctions in the electronic throttle controls. And few malfunctions
are more apparent than tin whiskers growing inside the APP sensors!
Since my 2010 testimony in the Washington Toyota hearings, I have learned
much. Your testimony certainly adds to that knowledge and I am pleased that
it has received much needed media attention.
Maybe our paths will cross someday.
DWG
***************************************************
[*] Trial transcript is available on the Safety Research & Strategies website
After Barr gave expert testimony that Toyota's ETC failsafe was like a "house of cards," I think Toyota has ceased describing its software with those words. Not a queen of diamonds there.
Barr's concerns about the source code's poor quality echo those of the Toyota engineers, as they wrote in the documents. Here is Barr's blog post:
An Update on Toyota and Unintended Acceleration
Saturday, October 26th, 2013 by Michael Barr
http://embeddedgurus.com/barr-code/2013/10/an-update-on-toyota-and-unintended-acceleration/
(downloaded on 29 October 2013)
Michael Barr
Michael Barr is an expert on the design of software-powered medical
devices and other embedded computer systems. (full bio)
In early 2011, I wrote a couple of blog posts (here and here) as well as a later article
(here) describing my initial thoughts on skimming NASA’s official report on its
analysis of Toyota’s electronic throttle control system. Half a year later, I was
contacted and retained by attorneys for numerous parties involved in
suing Toyota for personal injuries and economic losses stemming from incidents of
unintended acceleration. As a result, I got to look at Toyota’s engine source code
directly and judge for myself.
From January 2012, I’ve led a team of seven experienced engineers, including three
others from Barr Group, in reviewing Toyota’s electronic throttle and some other
source code as well as related documents, in a secure room near my home in
Maryland. This work proceeded in two rounds, with a first round of expert reports and
depositions issued in Summer 2012 that led to a billion-dollar economic loss
settlement as well as an undisclosed settlement of the first personal injury case set
for trial in U.S. Federal Court. The second round began with an over 800 page formal
written expert report by me in April 2013 and culminated this week in an Oklahoma
jury’s decision that the multiple defects in Toyota’s engine software directly caused a
September 2007 single vehicle crash that injured the driver and killed her passenger.
Don’t be misled by much of the mainstream coverage of the Oklahoma verdict. While
it is true this was the first time Toyota has lost an unintended acceleration case in
court, it is more significant that this was the first and only jury so far to hear any
opinions about Toyota’s software defects. Each of the earlier cases either predated
our source code access, applied a non-software theory, or was settled by Toyota for
an undisclosed sum.
In our analysis of Toyota’s source code, we built upon the work that NASA had done.
First, we looked more closely at more lines of the source code for more vehicles for
more man months. And we also did a lot of things that NASA didn’t have time to do,
including reviewing Toyota’s operating system’s internals, reviewing the source code
for Toyota’s “monitor CPU” (which even Toyota hadn’t ever done before! (!)),
performing an independent worst-case stack depth analysis, running portions of the
main CPU software including the RTOS in a processor simulator, and
demonstrating–in exemplar Toyota Camry vehicles–a link between loss of throttle
control and the numerous defects we found in the software.
In a nutshell, the team led by Barr Group found what the NASA team sought but
couldn’t find: “a systematic software malfunction in the Main CPU that opens the
throttle without operator action and continues to properly control fuel injection and
ignition” that is not reliably detected by any fail-safe. To be clear, NASA never
concluded software wasn’t at least one of the causes of Toyota’s high complaint rate
for unintended acceleration; they just said they weren’t able to find the specific
software defect(s) that caused unintended acceleration. We did.
Now it’s your turn to judge for yourself. Though I don’t think you can find my 800
page expert report outside the Court system, here’s the trial transcript [*] of my expert
testimony to the Oklahoma jury in Bookout, et.al. v. Toyota.
Note that the jury in Oklahoma went with the software defects and found that Toyota
owed each victim $1.5 million in compensatory damages and also found “reckless
disregard”. The latter legal standard meant the jury was headed toward deliberations
on additional punitive damages when Toyota finally called the plaintiffs to settle (for
yet another undisclosed amount). I understand there are about 500 personal injury
cases still working their way through various courts, including one set for trial in
November in U.S. District Court in Santa Ana, California.
***********************************************************
2 Responses to “An Update on Toyota and Unintended Acceleration”
1. Miro Samek says:
October 28, 2013 at 4:49 pm
Hi Michael,
Thank you for posting the link to your court deposition. I found it fascinating
and couldn’t stop reading late into the night…
There is no doubt in my mind that exposing the inadequacies in the Toyota
firmware is a very important development for the whole embedded software
profession.
It is also interesting to see old mistakes repeated time and time again. For
example a timed task degenerating into a kitchen sink.
I also bet my shirt that there were no assertions in the Toyota firmware.
Assertions in software work like fuses in electrical systems and beyond
certain density of assertions in the code all failures (including hardware
failures) manifest themselves as assertion violations. I’m sure that this could
have saved the day (besides making software development so much faster).
Anyway, there are tons of valuable lessons to learn here. From now on I will
imagine that all my software is on trial…
–Miro
2. David W. Gilbert, Ph.D. says:
October 28, 2013 at 10:25 pm
Dear Mr. Barr,
Nicely done! I found your testimony very interesting, and while I am not a
software expert, I can certainly verify the inability of Toyota vehicles to detect
certain malfunctions in the electronic throttle controls. And few malfunctions
are more apparent than tin whiskers growing inside the APP sensors!
Since my 2010 testimony in the Washington Toyota hearings, I have learned
much. Your testimony certainly adds to that knowledge and I am pleased that
it has received much needed media attention.
Maybe our paths will cross someday.
DWG
***************************************************
[*] Trial transcript is available on the Safety Research & Strategies website