Linux Security Controversy
Dan O’Dowd, CEO of Green Hills Software, Inc.

Part IV
Linux in Defense: Free Software is Just Too Expensive


Many people are under the misconception that Linux is free (or at least cheap). But the true cost of using Linux in defense systems is far higher than the cost of using Green Hills Software’s proprietary INTEGRITY operating system. In addition, the business model used by companies providing commercial support for embedded Linux is not sustainable. On the other hand, our proven proprietary software business model has enabled us to provide uninterrupted support for over 20 years, longer than Linux and the “open source” and “free software” movements have existed.


 


EE Times article

» "Linux: unfit for national security?"
Green Hills Software links
» Home page
» More on Linux
  » INTEGRITY operating system
  » Development tools for Linux (pdf)
  Contact Us



Linux is More Expensive than Proprietary Operating Systems

It is a common misconception that Linux is “free,” so it will save the government money relative to “proprietary” operating systems such as our INTEGRITY real-time operating system. But MontaVista Software, the #1 embedded Linux vendor, charges over $8,000 per developer per year for embedded Linux and GNU support. That is more than it costs to license Green Hills Software’s “proprietary” INTEGRITY real-time operating system and software development tools, including support!

MontaVista claims to be growing rapidly. This proves that more and more embedded Linux users are finding that supporting embedded Linux in-house costs even more than commercial support from MontaVista. It is understandable that in-house Linux support operations can’t compete with MontaVista on price, because MontaVista outsources its support to Russia, China, and India. Paying U.S. wages to support Linux in-house is clearly the most expensive operating system option. Whether the U.S. military or defense contractors use commercial vendors or in-house support, Linux costs more than Green Hills Software’s “proprietary” INTEGRITY real-time operating system.

For systems that require high reliability, the cost of using Linux skyrockets. As explained in Part I of this series of white papers, a Federal Aviation Administration DO-178B Level A safety approval should be required for any defense system upon which many lives depend. As outlined in Part III of this series of white papers, the cost of a DO-178B Level A certification is about $1000 per line of source code. For Linux, this would cost billions of dollars. Green Hills Software’s INTEGRITY-178B operating system has already been approved as satisfying DO-178B Level A safety objectives, so for high reliability systems, Linux is far more expensive than INTEGRITY-178B.


Linux Increases Development Time and Development Cost

Surveys of embedded Linux developers consistently report that embedded Linux developers’ biggest complaint about Linux is the poor quality of embedded Linux software development tools. A March, 2003 survey by Evans Data Corporation (http://www.evansdata.com/survey_embedded_03_1_topical.shtml), reported that 62% of embedded Linux users rated Linux software development tools as “not very good” or “adequate.” Just 24% said Linux software development tools were “good” or “excellent.” And 14% said software development tools didn’t matter! By what standards are these developers rating Linux software development tools? By the standards of commercial software development tools for proprietary operating systems. Poor Linux software development tools slow down software development. Since the primary cost of software development is the cost of the programmers, increasing the development time increases the cost of using Linux proportionately.

It shouldn’t be surprising that Linux development tools aren’t very good for embedded development. They weren’t designed for embedded development. They don’t provide features that are expected by most embedded developers, including integrated system- and task-level debugging, real-time event analysis, resource utilization analysis, code coverage analysis, and integration with hardware debug probes. All of these essential capabilities are provided by Green Hills Software’s MULTI development environment, which is bundled with our INTEGRITY operating system. The most difficult bugs to find are intermittent bugs that pop up at random. It requires luck or genius for a developer using Linux development tools to find the most difficult intermittent bugs, so most systems are shipped with intermittent bugs that cause random glitches and crashes. Our TimeMachine debugger enables developers to find the most outrageously difficult intermittent bugs in minutes, an innovation that has earned the TimeMachine debugger three industry awards (http://www.ghs.com/products/timemachine.html).

Another reason that embedded Linux development is more time consuming and costly than using our INTEGRITY operating system is the inferior support available for embedded Linux. Linux vendors do not actually employ the people who developed most of the Linux operating system. So when you call for support, you are often talking to people who are ill-equipped to help you. Our INTEGRITY operating system is supported by its original developers, who fully understand its design and implementation.

Some people believe that access to source code and support from the Linux community will free them from dependence on a proprietary solution. But depending on the charity of strangers to support software for defense systems doesn’t seem wise. The free support that is available on the Internet is for standard Linux distributions running on standard Pentium-based desktop systems. There is no free support for non-Pentium systems with custom hardware configured for defense systems. Besides, accepting any code for a defense system, be it bug fixes or device drivers, over the Internet from an unknown source is too risky (Part II of this series of white papers). That “free” support may come from foreign intelligence agencies or terrorists!

The poor software development tools and support available for embedded Linux compared to proprietary operating systems, such as INTEGRITY, further increases the cost of using Linux in defense systems.


Linux Increases Hardware Costs

Linux also increases the cost of manufacturing a defense system. Linux requires 10 to 20 times more memory than the INTEGRITY operating system, directly increasing hardware costs. Linux is not only big, it is slow. Linux’s worst-case response time to external events is between 100 and 1,000 times slower than the INTEGRITY operating system, meaning that a defense system using Linux must employ a faster, more expensive processor, increasing hardware costs even more.

The compiler that is used to develop a program determines the performance of that program and the amount of memory that the program will require. The Green Hills compilers consistently make programs much faster and smaller than the GNU compilers that come with Linux. When we compiled the Linux kernel with a Green Hills compiler, Linux was 35% smaller than when it was compiled with the GNU compiler. The additional memory that using the Linux compilers entails increases hardware costs even more.

The EEMBC benchmarks (www.eembc.org) are used by all of the embedded microprocessor vendors as the industry standard means of comparing the performance of their microprocessors. To produce the highest possible performance benchmarks for their microprocessors, microprocessor vendors (including IBM, Motorola/Freescale, NEC, and Toshiba) choose Green Hills compilers far more often than GNU compilers.

Over the last 22 years, our compilers have been used in the development of thousands of applications. I have never seen a customer application program that we could not make run 20% faster than when compiled with the GNU compiler. This means that to get the same amount of processing done with Linux requires a 20% higher performance microprocessor than if you use Green Hills compilers, further increasing the hardware cost of using Linux.

The faster processor and additional memory required by Linux directly increase manufacturing costs, but they also require greater power consumption, which in turn increases power supply costs, cooling costs, battery size, weight, and device size, all of which further increase hardware costs.

The additional hardware costs imposed by using Linux often far exceeds the entire cost of licensing a proprietary operating system, such as INTEGRITY.


Recall and Rework Expenses

There are many critical security bugs found in Linux every year. In a traditional desktop or server environment, the open source methodology makes it easy to patch Linux security holes soon after they are found. But patching security holes is a nightmare for the suppliers of embedded computer-based defense systems.

As soon as a new critical security vulnerability is announced on a public website, every Linux system in the world is vulnerable to attack by any foreign intelligence agency or terrorist that is monitoring the website. Every announcement of a critical Linux security vulnerability will create a national security emergency until every affected defense system can be patched.

If the Linux operating system is burned into ROM or flash memory in a missile, a bomb, or a tank, the system will have to be recalled to the factory, or technicians will have to be deployed into the field at great cost to retrofit every system. Systems that are buried underground, or under the sea, or in space, or in foreign countries will be especially difficult and expensive to recall and retrofit. If these systems are deployed in a war zone, soldiers under fire are going to be very unhappy when you take away their weapons or defenses to patch security holes.

DO-178B Level A operating systems, such as Green Hills Software’s INTEGRITY-178B, design in reliability and security from the start, so they don’t need to depend on patching to maintain security.

Every defense contractor will have to add to the cost of using Linux the future cost of recalls and retrofits required to patch every critical Linux security hole that is discovered in the future.


Linux Doesn’t Offer the Long Term Support Model Required by Defense Systems

Many of the objections to my assertion that Linux is not safe for defense systems were based on the misconception that open source operating system vendors can provide the necessary long-term support, while proprietary operating system vendors can’t. These objections mostly come from people who have been burned by proprietary software vendors who went out of business. These people failed to consider the most important factor in choosing any software vendor: whether the vendor has a long history of profitability. The purpose of a business is to make money. If a company can’t make money, it has no reason to exist, so it probably won’t exist much longer.

The proprietary software business model is not unstable, it is vendors who are not profitable that are unstable. A proprietary vendor that has a long history of profitability is likely to continue in the same business for a long time in the future doing the same thing so that they can make lots more money. When selecting an operating system vendor, pick a successful one, a company with ten or, better yet, twenty years of profitability, like Green Hills Software. We have been supporting some products for over 20 years. That is longer than the life of most defense systems. It is also longer than the life of Linux and the “open source” and “free software” movements, let alone any vendor of open source software or support.

Don’t ever depend on a long term commitment from any vendor that can’t make money consistently. None of the Linux software or support vendors has a history of profitability. In fact, most of them have never made money. Linux vendors have been dropping like flies. If you accept a custom version of Linux from a commercial Linux vendor, when they go out of business you will be left holding the bag. There will be no one to support your custom version of Linux or update it in the future.

No one has ever established a profitable open source business model, because no company can sustain an exploitable proprietary advantage. The nature of open source is that every proprietary advantage must be returned to the public domain. The talents of individual engineers can’t be retained because the engineers can just quit and take all of their knowledge with them to apply in their next job. The open source process drives the profit out of Linux businesses leading to their eventual demise.

Green Hills Software’s highly profitable proprietary software business model long ago demonstrated that it delivers the long-term support that defense systems require. No Linux vendor will ever be able to provide the long-term support that defense systems need.


Green Hills Software’s Business Success

Many people are under the misconception that I have criticized Linux because Linux is hurting Green Hills Software’s business. Nothing could be further from the truth. Our earnings in 2003 were up over 100% from 2002. In the first quarter of 2004, our earnings were up more than 50% over the first quarter of 2003. We haven’t lost money in any of the last 21 years. In fact, we have never had an operating profit margin below 14% in any of the last 21 years, even during the last two recessions. And we have grown an average of about 30% per year for the last 20 years without acquisitions.

When our INTEGRITY operating system competes against Linux for defense applications, INTEGRITY wins. All we have to do is go through the security, reliability, and cost arguments. Unfortunately, we don’t find out about every defense application before the developers make their operating system selection. Many developers of defense applications have heard from Linux promoters that Linux is the best operating system for defense applications because it offers better security, reliability, and cost than any proprietary operating system. I have discovered recently that some critical defense systems have selected Linux without being aware of its severe security, reliability, and cost problems. That is what concerns me.

I am calling attention to the serious security problems of Linux so that everyone who is responsible for selecting operating systems for critical defense applications will be able to evaluate all of the information available about Linux security and the alternatives. When they are fully informed, I believe that they will avoid Linux, and chose a “proprietary” operating system that doesn’t have the security problems of Linux.


Linux: Too Expensive for Defense Systems

Linux is probably the most expensive operating system for defense systems. It not only costs far more to acquire and support Linux than our proprietary INTEGRITY operating system, but using Linux also substantially increases manufacturing and development costs. But worst of all, there are no consistently profitable commercial embedded Linux vendors upon whom you can depend for long term support. If you choose Linux for a defense system, you must be prepared for the enormous expense of supporting an orphaned Linux variant in-house indefinitely.

Choosing the INTEGRITY operating system means lower acquisition and support costs from a reliable long-term partner, lower manufacturing and development costs, and the reliability and security that defense systems need.

Next week in part V of this series of white papers, “Linux in Defense: An Urgent Threat to National Security,” I will show why Linux poses an urgent and immediate threat to national security and why we must act now to stop the spread of Linux through our defense systems.



Other Linux security white papers:   For more information:
» Part I: FAA Certified Operating Systems Deliver the Reliability & Security Defense Systems Require; Linux Does Not
» Part II: "Many Eyes" - No Assurance Against Many Spies
» Part III: Linux Security: Unfit for Retrofit - Green Hills Software
» Part IV: Linux in Defense: Free Software is Just Too Expensive - Green Hills Software
» Part V: Linux in Defense: An Urgent Threat to National Security - Green Hills Software
 
 
» Contact us
» EE Times article - "Linux: unfit for national security?" (pdf)
» Green Hills Software home page
» More on Linux
» INTEGRITY operating system
» Development tools for Linux (pdf)