Tuesday, February 17, 2009

How not to build an embedded critical application

Guess what? According to Diebold "Nobody protects against viruses like Diebold". That's true, I guess, but not in a good way. It's like saying "Nobody protects against bone-shattering accidents like Evel Knievel. Wears a helmet every time!" Here's a better idea. Don't do crazy stupid things in the first place!

Crazy stupid thing: Using an overly complex, constantly under-attack and in-flux operating system in an embedded critical application. An OS for which you can't ever see the source code, but which millions of 15-years-olds can find out how to hack.

If you are planning to build software for an embedded application (this means, there's a computer in it but it's only going to run a specific application and it only has to run on hardware you design) you need ask yourself these questions about an OS before you decide to use it:

- What does it do well, compared to the other options.

- What does it do badly

- Which of the above are important in my application

If you are building an ATM, or a voting machine, and you did the above and decided to use Windows, go back and do it again. If you did it again and got the same result: resign, apologize to the American people for wasting their tax dollars, and go find another line of work. Above all, if your solution includes installing third-party anti-virus software, and you don't think there's anything wrong with that -- seriously, get a job where your bad decisions can't hurt people.

Maybe you said "nobody who works here understands anything but Windows, so that's our only option." NO. Imcompetence doesn't make it right. Hire some people that understand this stuff, and they can help you out and educate the rest of the staff as needed.

Let's try it:

What does Windows do well that other options can't?
1. Nifty multiple overlapped windowed gui which supports flash and 3D and all kinds of things.
2. Supports a large range of display hardware, sound cards etc.
3. Supports a wide range of 3rd party applications
4. Has network support
5. Lots of development tools available

How much of this is a useful advantage in an ATM or voting machine?
1. No, I actually want to keep the display really simple; this is an ATM or a voting machine.
2. No, I only need it to work on the hardware I build it in.
3. NO, I DON'T want people installing extra stuff in the field. Anti-useful.
4. Yes, but there are plenty of other options with this. Disqualified.
5. Umm, mostly these are to help me build complex apps or apps with 3D, etc, See point 1. No advantage against other options.

What does it do badly? This category includes, but is not limited to:

1. Only runs on one processor architecture (which happens to be one of the least efficient, in terms of mops/Watt, around); limits your hardware choices.
2. Very complex; large amounts of storage, RAM & disk, required to accomplish a given simple task. Raises hardware costs.
3. Very very homogeneous code base. Flaws are regularly found which affect many or all builds of the OS, old and new.
4. The entire OS is basically targeted at desktop & office use, is intended to run on a fairly narrow spectrum of hardware capability (i.e. you need a faster CPU just to run the OS, nothing to do with your app); and the embedded application market is a pretty small sliver of the whole pie. In other words, you are starting with a square peg in a round hole. The vendor has their own priorities, their own tradeoffs they are working out in a different market, and they don't care much about you. You get what you get (this sounds pretty grim .. good thing there are so many things in the '+' column .. oh wait, there aren't any).

All of these are detrimental in an embedded system. 3. Is so bad, that your system will actually catch viruses without even being targeted; your system will get compromised by accident without anyone specifically saying "I want to attack this particular manufacturer". This actually happened to Diebold ATMS on a closed network.


Put it more simply .. if you are running a customized OS on an unusual processor, you can't catch a 'standard' virus, plain and simple. Biodiversity is strength. I'm not saying you'll be perfectly attack-proof; but (a) you will not catch anything by accident, and (b) you will be much more secure than a Windows system running standard "Let's try to close that barn door whenever we detect a pattern of horses leaving" anti-virus measures.

There seems to be this idea that "I'm running it on the desktop, so why not run it in the product?" I don't know.. I have no answer to this .. why should you? You drive a cement mixer at work, so why not drive it to the cottage? If there's an advantage there, make it part of a rational decision process, not the whole thing.

There was a spate of people doing stupid things with DOS in the 90's. I heard a horror story about a pacemaker running DOS, which I'm afraid may have been true.

Let's try the OS evaluation process again. Strengths and weaknesses of DOS in pacemaker app:

(1) File system. Check! DOS has one, it basically works. That's what the 'D' stands for.
(2) Process management.... err, not really no. DOS 'stays out of the way' so that the processes can manage themselves around it. That's the best I can put it.
(3) Time management. Not much at all. Hey bill, the time & date aren't really two different things, you know. Think about it.

What would you need in a pacemaker OS?

(1) File system. No, thanks, don't need that.
(2) Process management. Yes please. And watchdogs and so forth.
(3) Time management. Yes, please!

So for instance DOS vs. No OS at all? roughly comparable, "nothing" has a lower license cost.

The voting-machine arm of Diebold has changed its name to "Premier Election Solutions", but ATMs are still made by Diebold, so that just means there are two brand names to mistrust.

Further reading:

(1) The comic that inspired me to vent my spleen on this issue: http://xkcd.com/463/

(2) From 2007: A single key opens the memory card door on every Diebold voting machine. Only qualified individuals can order the key from Diebold. Or you can get the photo of the key from Diebold's web page, get a blank from the hardware store, and file the blank to match the photo and that will open the door on every machine. How can these guys stay in business?

(3) Pot Luck: http://www.google.com/search?q=diebold+virus

No comments: