Why Open Source Software / Free Software (OSS/FS)? Look at the Numbers!

Open Source Software / Free Software (OSS/FS) has risen to prominence. If you're not sure what these terms mean, you can get an explanation of these terms and related information from my list of Open Source Software / Free Software (OSS/FS) references at http://www.dwheeler.com/oss_fs_refs.html. Briefly, OSS/FS programs are programs whose licenses permit users the freedom to run the program for any purpose, to modify the program, and to redistribute the original or modified program (without payment or restriction on who they can redistribute the program to).

However, you shouldn't base a decision to use OSS/FS on a few anecdotes, which is all some sites provide. Instead, this paper emphasizes quantitative measures (such as experiments and market studies) on why using OSS/FS products is, in a number of circumstances, a reasonable or even superior approach. I should note that while I find much to like about OSS/FS, I'm not a rabid advocate; I use both proprietary and OSS/FS products myself. Vendors of proprietary products often work hard to find numbers to support their claims; this page provides a useful antidote of hard figures to aid in comparing these proprietary products to OSS/FS.

Note that this paper's goal is not to show that all OSS/FS is better than all proprietary software. There are those who believe this is true from ethical, moral, or social grounds, but no numbers could prove such a broad statement. Instead, I'll simply compare commonly-used OSS/FS software with commonly-used proprietary software, to show that, at least in certain situations and by certain measures, OSS/FS is at least as good or better than its proprietary competition.

I'll emphasize the GNU/Linux operating system (which some abbreviate as ``Linux'') and the Apache web server, since these are some of the most visible OSS/FS projects. I'll also primarily compare it to Microsoft Windows, since Windows has significant market share and Microsoft is one of proprietary software's strongest proponents. I'll mention Unix systems in passing as well, though the situation with Unix is more complex; many Unix systems include a number of OSS/FS components or software primarily derived from OSS/FS components. Thus, comparing proprietary Unix systems to OSS/FS systems (when examined as entire systems) is often not as clear-cut. Both Unix and GNU/Linux are ``Unix-like'' systems.

Below is data involving market share, reliability, performance, scaleability, security, and total cost of ownership. I close with a brief discussion of non-quantitative issues and a list of other sites providing other related information.

Market Share

Many people believe that a product is only a winner if it has significant market share. This is lemming-like, but there's some rationale for this: products with big market shares get applications, trained users, and momentum that reduces future risk. Some writers argue against OSS/FS or GNU/Linux as ``not being mainstream'', but if their use is widespread then such statements reflect the past, not the present. There's excellent evidence that OSS/FS has significant market share in several markets:

1. Apache is the #1 web server on the public Internet, and has been since April 1996. Netcraft's statistics on webservers have consistently shown Apache (an OSS/FS web server) dominating the public Internet web server market ever since Apache became the #1 web server in April 1996. For example, in September 2001, Netcraft polled all the web sites they could find (totaling 32,398,046 sites), and found that of the active sites, Apache had 60.86% of the market; the next two largest market shares were were Microsoft's (30.00%) and iPlanet's (aka Netscape's, 2.06%). Here is Netcraft's data over time:
   Market Share for Top Servers Across All Domains August 1995 - September 2001

   

   The same overall result has been determined independently by E-soft - their report published July 1, 2001 surveyed 3,178,197 web servers and found that Apache was #1 (59.84%), with Microsoft IIS being #2 (28.42%).

2. GNU/Linux is the #2 web serving operating system on the public Internet (counting by IP address), according to a study surveying March and June 2001. Some of Netcraft's surveys have also included data on operating systems; two 2001 surveys (their June 2001 and September 2001 surveys) found that GNU/Linux is the #2 operating system for web servers when counting by IP address (and has been consistently gaining market share since February 1999). Every connection to a network accessible to the public is given a unique number called the IP address, and this survey uses IP addresses (their web server survey counts by name, not by IP address). A quick aside: Netcraft dates their information based on the web server surveys (not the publication date), and they only report operating system summaries from an earlier month. Thus, the survey dated ``June 2001'' was published in July and covers operating system survey results of March 2001, while the survey dated ``September 2001'' was published in October and covers the operating system survey results of June 2001. Here's a summary of Netcraft's study results:

   OS group	Percentage (March)	Percentage (June)	Composition
   Windows	49.2%	49.6%	Windows 2000, NT4, NT3, Windows 95, Windows 98
   Linux	28.5%	29.6%	Linux
   Solaris	7.6%	7.1%	Solaris 2, Solaris 7, Solaris 8
   BSD	6.3%	6.1%	BSDI BSD/OS, FreeBSD, NetBSD, OpenBSD
   Other Unix	2.4%	2.2%	AIX, Compaq Tru64, HP-UX, IRIX, SCO Unix, SunOS 4 and others
   Other non-Unix	2.5%	2.4%	MacOS, NetWare, proprietary IBM OSs
   Unknown	3.6%	3.0%	not identified by Netcraft operating system detector

   Much depends on what you want to measure. Several of the BSDs (FreeBSD, NetBSD, and OpenBSD) are OSS/FS as well; so at least a portion of the 6.1% for BSD should be added to Linux's 29.6% to determine the percentage of OSS/FS operating systems being used as web servers. Thus, it's likely that approximately one-third of web serving computers use OSS/FS operating systems. There are also regional differences, for example, GNU/Linux leads Windows in Germany, Hungary, the Czech Republic, and Poland.

   If you really want to know about the market breakdown of ``Unix vs. Windows,'' you can find that also in this study. All of the various Windows operating systems are rolled into a single number (even Windows 95/98 and Windows 2000/NT4/NT3 are merged together, although they are fundamentally very different systems). Merging all the Unix-like systems in a similar way produces a total of 44.8% for Unix-like systems (compared to Windows' 49.2%) in March 2001.

   Note that these figures would probably be quite different if they were based on web addresses instead of IP addresses; in such a case, the clear majority of web sites are hosted by Unix-like systems. As stated by Netcraft, ``Although Apache running on various Unix systems runs more sites than Windows, Apache is heavily deployed at hosting companies and ISPs who strive to run as many sites as possible on a single computer to save costs.''

3. GNU/Linux is the #1 server operating system on the public Internet (counting by domain name), according to a 1999 survey of primarily European and educational sites. The first study that I've found that examined GNU/Linux's market penetration is a survey by Zoebelein in April 1999. This survey found that, of the total number of servers deployed on the Internet in 1999 (running at least ftp, news, or http (WWW)) in a database of names they used, the #1 operating system was GNU/Linux (at 28.5%), with others trailing. It's important to note that this survey, which is the first one that I've found to try to answer questions of market share, used existing databases of servers from the .edu (educational domain) and the RIPE database (which covers Europe , the Middle East, parts of Asia, and parts of Africa), so this isn't really a survey of ``the entire Internet'' (e.g., it omits ``.com'' and ``.net''). This is a count by domain name (e.g., the text name you would type into a web browser for a location) instead of by IP address, so what it's counting is different than the Netcraft June 2001 operating system study. Also, this study counted servers providing ftp and news services (not just web servers).

   Here's how the various operating systems fared in the study:

   Market Share	Operating System	Composition
   GNU/Linux	28.5%	GNU/Linux
   Windows	24.4%	All Windows combined (including 95, 98, NT)
   Sun	17.7%	Sun Solaris or SunOS
   BSD	15.0%	BSD Family (FreeBSD, NetBSD, OpenBSD, BSDI, ...)
   IRIX	5.3%	SGI IRIX

   A portion of the BSD family is also OSS/FS, so the OSS/FS operating system total is even higher; if over 2/3 of the BSDs are OSS/FS, then the total share of OSS/FS would be about 40%. Advocates of Unix-like systems will notice that the majority (around 66%) were running Unix-like systems, while only around 24% ran a Microsoft Windows variant.

4. GNU/Linux is the #2 server operating system sold in 1999 and 2000, and is the fastest-growing. According to a June 2000 IDC survey of 1999 licenses, 24% of all servers (counting both Internet and intranet servers) installed in 1999 ran GNU/Linux. Windows NT came in first with 36%; all Unixes combined totaled 15%. Again, since some of the Unixes are OSS/FS systems (e.g., FreeBSD, OpenBSD, and NetBSD), the number of OSS/FS systems is actually larger than the GNU/Linux figures. Note that it all depends on what you want to count; 39% of all servers installed from this survey were Unix-like (that's 24%+15%), so ``Unix-like'' servers were actually #1 in installed market share once you count GNU/Linux and Unix together.

   IDC released a similar study on January 17, 2001 titled ``Server Operating Environments: 2000 Year in Review''. On the server, Windows accounted for 41% of new server operating system sales in 2000, growing by 20% - but GNU/Linux accounted for 27% and grew even faster, by 24%. Other Unixes had 13%, and Novell declined (to 17%).

   These measures do not measure all systems installed that year; some Windows systems are not paid for (they're illegally pirated), and GNU/Linux is often downloaded and installed for multiple systems (since it's legal and free to do so).

5. Microsoft sponsored its own research to ``prove'' that GNU/Linux isn't as widely used, but this research has been shown to be seriously flawed. Microsoft sponsored a Gartner Dataquest report claiming only 8.6% of servers shipped in the U.S. during the third quarter of 2000 were Linux-based. However, it's worth noting that Microsoft (as the research sponsor) has every incentive to create low numbers, and these numbers are quite different from IDC's research in the same subject. IDC's Kusnetzky commented that the likely explanation is that Gartner used a very narrow definition of "shipped"; he thought the number was "quite reasonable" if it only surveyed new servers with Linux , "But our research is that this is not how most users get their Linux. We found that just 10 to 15 percent of Linux adoption comes from pre-installed machines... for every paid copy of Linux, there is a free copy that can be replicated 15 times." Note that it's quite difficult to buy a new x86 computer without a Microsoft operating system (Microsoft's contracts with computer makers ensure this), but that doesn't mean that these operating systems are used. Gartner claimed that it used interviews to counter this problem, but its final research results (when compared to known facts) suggest that Gartner did not really counter this effect. For example, Gartner states that Linux shipments in the supercomputer field were 'zero'. In fact, Linux is widely used on commodity parallel clusters at many scientific sites, including a number of high-profile sites. Many of these systems were assembled in-house, showing that Gartner's method of defining a ``shipment'' does not appear to correlate to working installations. The Register's article, ``No one's using Linux'' (with its companion article ``90% Windows..'') discusses this further. In short, Microsoft-sponsored research reported low numbers, but these numbers are quite suspect.

6. GNU/Linux had 80% as many client shipments in 1999 as Apple's MacOS. According to the June 2000 IDC survey of 1999 licenses (5.0% for Mac OS, 4.1% for GNU/Linux), there were almost as many client shipments of GNU/Linux as there were of MacOS - and no one doubts that MacOS is a client system.

   Currently, GNU/Linux is a relatively new contender in the client operating system (OS) market, and has relatively little market penetration. This should not be surprising; unlike the wealth of server and developer applications, GNU/Linux has relatively few OSS/FS client applications and many of those client applications are still maturing. There are commercial client applications (e.g. Corel's Word Perfect), but they aren't OSS/FS and are available on other platforms as well. Without strong OSS/FS client applications (in particular an office suite), GNU/Linux has no strong advantage over other client platforms. After all, GNU/Linux combined with a proprietary office suite still lacks the freedoms and low cost of purely OSS/FS systems, and it has to compete with established proprietary systems which have more applications available to them. Microsoft Office is essentially a monopoly controlling the office suite arena, and it isn't available on GNU/Linux (the second-largest player in office suites is Corel; since Corel is partly owned by Microsoft, Corel cannot really be considered a competitor to Microsoft). StarOffice has recently been released as open source (and renamed OpenOffice), and this has suddenly made it possible to have an OSS/FS office suite. However, its user interface is uncommon (which many dislike), and simply changing its license hasn't suddenly made its interface more like others' user interfaces. Other OSS/FS office application programs, such as AbiWord, Gnumeric, and the KOffice suite, are just now maturing. In addition, the cost of switching all those desktops in the face of compatibility issues with an entrenched monopoly makes it difficult to use anything other than Windows and Office on clients.

   Nevertheless, in spite of weaker OSS/FS client applications in several key areas at the time, an IDC study determined that GNU/Linux captured 4% of the market in 1999. More generally, IDC determined that there were 98.6 million shipments of client operating systems in 1999, of which Windows 3.x, 95 and 98 accounted for 66%, Windows NT had 21%, MacOS had 5%, and GNU/Linux 4%. It will be interesting to see if these numbers begin to change around 2002-2003, when I anticipate OSS/FS client applications (e.g., AbiWord) will be available that will begin to rival their proprietary competitors in both useability and in the functionality that people need. There's already some evidence that others anticipate this; Richard Thwaite, director of IT for Ford Europe, stated in 2001 that an open source desktop is their goal, and that they expect the industry to eventually go there (he controls 33,000 desktops, so this would not be a trivial move). It could be argued that this is just a ploy for negotiation with Microsoft - but such ploys only work if they're credible.

7. Businesses plan to increase their use of GNU/Linux. A Zona Research study found that more than half of the large enterprise respondents expected increases of up to 25% in the number of GNU/Linux users in their firm, while nearly 20% expected increases of more than 50%. In small companies, more than one third felt that GNU/Linux usage would expand by 50%. The most important factors identified that drove these decisions were reliability, lower price, speed of applications, and scaleability. Here are the numbers:

   Expected GNU/Linux Use	Small Business	Midsize Business	Large Business	Total
   50% increase	21.0%	16%	19.0%	19%
   10-25% increase	30.5%	42%	56.5%	44%
   No growth	45.5%	42%	24.5%	36%
   Reduction	3.0%	0%	0%	1%

   You can see more about this study in ``The New Religion: Linux and Open Source'' (ZDNet) and in InfoWorld's February 5, 2001 article ``Linux lights up enterprise: But concerns loom about OS vendor profitability.''

8. The global top 1000 Internet Service Providers expect GNU/Linux use to increase by 154%, according to Idaya's survey conducted January through March 2001. A survey conducted by Idaya of the global top 1000 ISPs found that they expected GNU/Linux to grow a further 154% in 2001. Also, almost two thirds (64%) of ISPs consider the leading open source software meets the standard required for enterprise level applications, comparable with proprietary software. Idaya produces OSS/FS software, so keep that in mind as a potential bias.

9. IBM found a 30% growth in the number of enterprise-level applications for GNU/Linux in the six month period ending June 2001. At one time, it was common to claim that ``Not enough applications run under GNU/Linux'' for enterprise-level use. However, IBM found there are more than 2,300 Linux applications (an increase in 30% over 6 months) available from IBM and the industry's top independent software vendors (ISVs).

10. A survey in the second quarter of 2000 found that 95% of all reverse-lookup domain name servers (DNS) used bind, an OSS/FS product. The Internet is built from many mostly-invisible infrastructure components. This includes domain name servers (DNSs), which take human-readable machine names (like ``yahoo.com'') and translate them into numeric addresses. Publicly accessible machines also generally support ``reverse lookups'', which convert the numbers back to names; for historical reasons, this is implemented using the hidden ``in-addr.arpa'' domain. By surveying the in-addr domain, you can gain insight into how the entire Internet is supported. Bill Manning has surveyed the in-addr domain and found that 95% of all name servers (in 2q2000) performing this important Internet infrastructure task are some version of ``bind.'' Bind is an OSS/FS program.

Reliability

There are a lot of anecdotal stories that OSS/FS is more reliable, but finally there is quantitative data confirming that mature OSS/FS programs are more reliable:

1. Equivalent OSS/FS applications are more reliable, according to a 1995 study. The 1995 ``Fuzz Revisited'' paper measured reliability by feeding programs random characters and determining which ones resisted crashing and freeze-ups. Some researchers scoff at this measure, since this approach is unlikely to find subtle failures, but the study authors note that their approach still manages to find many errors in production software and is a useful tool for finding software flaws.

   OSS/FS had higher reliability by this measure. It states in section 2.3.1 that:

   It is also interesting to compare results of testing the commercial systems to the results from testing "freeware" GNU and Linux. The seven commercial systems in the 1995 study have an average failure rate of 23%, while Linux has a failure rate of 9% and the GNU utilities have a failure rate of only 6%. It is reasonable to ask why a globally scattered group of programmers, with no formal testing support or software engineering standards can produce code that is more reliable (at least, by our measure) than commercially produced code. Even if you consider only the utilities that were available from GNU or Linux, the failure rates for these two systems are better than the other systems.

   There is evidence that Windows NT applications have similar reliability to the proprietary Unix software (e.g., less reliable than the OSS/FS software). A later paper, ``An Empirical Study of the Robustness of Windows NT Applications Using Random Testing'', found that with Windows NT GUI applications, they could crash 21% of the applications they tested, hang an additional 24% of the applications, and could crash or hang all the tested applications when subjecting them to random Win32 messages. Thus, there's no evidence that proprietary Windows software is more reliable than OSS/FS by this measure.

   Now be careful: OSS/FS is not magic pixie dust; beta software of any kind is still buggy! However, the 1995 experiment measured mature OSS/FS to mature proprietary software, and the OSS/FS software was more reliable under this measure.

2. GNU/Linux is more reliable than Windows NT, according to a 10-month ZDnet experiment. ZDnet ran a 10-month test for reliability to compare Caldera Systems OpenLinux, Red Hat Linux, and Microsoft's Windows NT Server 4.0 with Service Pack 3. All three used identical (single-CPU) hardware, and network requests were sent to each server in parallel for standard Internet, file, and print services. The result: NT crashed an average of once every six weeks, each taking about 30 minutes to fix; that's not bad, but neither GNU/Linux server ever went down. This ZDnet article also does a good job of identifying GNU/Linux weaknesses (e.g., desktop applications and massive SMP).

3. GNU/Linux is more reliable than Windows NT, according to a one-year Bloor Research experiment. Bloor Research had both operating systems running on relatively old Pentium machines. In the space of one year, GNU/Linux crashed once because of a hardware fault (disk problems), which took 4 hours to fix, giving it a measured availability of 99.95 percent. Windows NT crashed 68 times, caused by hardware problems (disk), memory (26 times), file management (8 times), and a number of odd problems (33 times). All this took 65 hours to fix, giving an availability of 99.26 percent. It's intriguing that the only GNU/Linux problem and a number of the Windows problems were hardware-related; it could be argued that the Windows hardware was worse, or it could be argued that GNU/Linux did a better job of avoiding and containing hardware failures. In any case, file management failure can be blamed on Windows, and the ``odd problems'' appear to be caused by Windows as well, indicating that GNU/Linux is far more reliable than Windows. Gnet summarized this as saying ``the winner here is clearly Linux.''

4. Sites using Microsoft's IIS web serving software have more than double the time offline (on average) than sites using the Apache software, according to a 3-month Swiss evaluation. These are the results of Syscontrol AG's analysis of website uptime (announced February 7, 2000) They measured over 100 popular Swiss web sites over a three-month period, checking from 4 different locations every 5 minutes (it'd be interesting to see what a larger sample would find!). You can see their report (in German), or a Babelfish (machine) translation of the report. Here's their entire set of published data on ``average down-time per hour per type of server'', plus a 3-month average that I've computed:

   Downtime	Apache	Microsoft	Netscape	Other
   September	5.21	10.41	3.85	8.72
   October	2.66	8.39	2.80	12.05
   November	1.83	14.28	3.39	6.85
   Average	3.23	11.03	3.35	9.21

   It's hard not to notice that Apache (the OSS web server) had the best results over the three-month average (and with better results over time, too). Indeed, Apache's worst month was better than Microsoft's best month. I believe the difference between Netscape and Apache is statistically insignificant - but this still shows that the freely-available OSS/FS solution (Apache) has a reliability at least as good as the most reliable proprietary solution. The report does note that this might not be solely the fault of the software's quality, since there were several Microsoft IIS sites that had short interruptions at the same time each day (suggesting regular restarts). However, this still begs the question -- why did the IIS sites require so many more regular restarts than the Apache sites? Every outage, even if preplanned, results in a service loss (and for e-commerce sites, a potential loss of sales).

5. According to a separate uptime study by Netcraft, OSS/FS does very well; as of August 3, 2001, of the 50 sites with the highest uptimes, 92% use Apache and 50% run on OSS/FS operating systems. Netcraft keeps a track of the 50 often-requested sites with the longest uptimes at http://uptime.netcraft.com/. Looking at the August 3, 2001 uptime report, I found that 92% (46/50) of the sites use Apache; one site's web server was unknown, and three others were not Apache. Of those three, only one reported to be Microsoft IIS, and that one instance is suspicious because its reported operating system is BSD/OS (this apparant inconsistency can be explained in many ways, e.g., perhaps there is a front-end BSD/OS system that ``masks'' the IIS web site, or perhaps the web server is lying about its type to confuse attackers). In this snapshot, 50% (25/50) ran on an open source operating system, and only Unix-like operating systems had these large uptimes (no Windows systems were reported as having the best uptimes).

   As with all surveys, this one has weaknesses, as discussed in Netcraft's Uptime FAQ. Their techniques for identifying web server and operating systems can be fooled. Only systems for which Netcraft was sent many requests were included in the survey (so it's not ``every site in the world''). Any site that is requested through the ``what's that site running'' query form at Netcraft.com is added to the set of sites that are routinely sampled; Netcraft doesn't routinely monitor all 22 million sites it knows of for performance reasons. Many operating systems don't provide uptime information and thus can't be included; this includes AIX, AS/400, Compaq Tru64, DG/UX, MacOS, NetWare, NT3/Windows 95, NT4/Windows 98, OS/2, OS/390, SCO UNIX, Sony NEWS-OS, SunOS 4, and VM. Thus, this uptime counter can only include systems running on BSD/OS, FreeBSD (but not the default configuration in versions 3 and later), recent versions of HP-UX, IRIX, Linux 2.1 kernel and later (except on Alpha processor based systems), MacOS X, recent versions of NetBSD/OpenBSD, Solaris 2.6 and later, and Windows 2000. Note that Windows NT systems cannot be included in this survey (because their uptimes couldn't be counted), but Windows 2000 systems are included in this survey. Again, no Windows system actually made it into the top 50 for uptimes in this snapshot. Note that HP-UX, (many versions of) Linux, Solaris and recent releases of FreeBSD cycle back to zero after 497 days, exactly as if the machine had been rebooted at that precise point. Thus it is not possible to see an HP-UX, (most) Linux, or Solaris system with an uptime measurement above 497 days, and in fact their uptimes can be misleading (they may be up for a long time, yet not show it). Still, this survey does compare Windows 2000, Linux (up to 497 days usually), FreeBSD, and several other operating systems, and a vast number of web servers, and OSS/FS does quite well.

Of course, there are many anecdotes about Windows reliability vs. Unix. For example, the Navy's ``Smart Ship'' program caused a complete failure of the entire USS Yorktown ship in September 1997. Anthony DiGiorgio (a whistle-blower) stated that Windows is ``the source of the Yorktown's computer problems.'' Ron Redman, deputy technical director of the Fleet Introduction Division of the Aegis Program Executive Office, said ``there have been numerous software failures associated with [Windows] NT aboard the Yorktown.'' Redman also said ``Because of politics, some things are being forced on us that without political pressure we might not do, like Windows NT... If it were up to me I probably would not have used Windows NT in this particular application. If we used Unix, we would have a system that has less of a tendency to go down.''

One problem with reliability measures is that it takes a long time to gather data on reliability in real-life circumstances. Nevertheless, the available evidence suggests that OSS/FS has a significant edge in reliability.

Performance Data

Comparing GNU/Linux and Windows NT/2000 performance on equivalent hardware has a history of contentious claims and different results based on different assumptions. I think that OSS/FS has at least shown that it's often competitive, and in many circumstances it beats the competition.

Performance benchmarks are very sensitive to the assumptions and environment, so the best benchmark is one you set up yourself to model your intended environment. Failing that, you should use unbiased measures, because it's so easy to create biased measures.

First, here are a few recent studies suggesting that some OSS/FS systems beat their proprietary competition in at least some circumstances:

1. GNU/Linux with TUX has produced better SPEC values than Windows/IIS in several cases, even when given inferior drive configurations. One organization that tries to develop unbiased benchmarks is the SPEC Consortium, which develops and maintains a whole series of benchmarks. We can compare Microsoft Windows versus GNU/Linux by comparing SPECweb99 results (which measure web server performance) on identical hardware if both have undergone the same amount of performance optimization effort. Alas, things are not so simple; rarely are the same basic hardware platforms tested with both operating systems, and even when that occurs, as of July 13, 2001 no exactly identical configurations have been tested (they differ in ways such as using a different number of hard drives, or including some faster hard drives). Using all results available by July 13, 2001, there were three hardware configurations, all from Dell, which ran both GNU/Linux (using the TUX web server/accelerator) and Windows (using IIS) on exactly the same underlying hardware. Here are the SPECweb99 results as of July 13, 2001 (larger is better), noting configuration differences:

   System	Windows SPEC Result	Linux SPEC Result
   Dell PowerEdge 4400/800, 2 800MHz Pentium III Xeon	1060 (IIS 5.0, 1 network controller)	2200 (TUX 1.0, 2 network controllers)
   Dell PowerEdge 6400/700, 4 700MHz Pentium III Xeon	1598 (IIS 5.0, 7 9GB 10KRPM drives)	4200 (TUX 1.0, 5 9GB 10KRPM drives)
   Dell PowerEdge 8450/700, 8 700MHz Pentium III Xeon	7300/NC (IIS 5.0, 1 9Gb 10KRPM and 8 16Gb 15KRPM drives) then 8001 (IIS 5.0, 7 9Gb 10KRPM and 1 18Gb 15KRPM drive)	7500 (TUX 2.0, 5 9Gb 10KRPM drives)

   The first row (the PowerEdge 4400/800) doesn't really prove anything. The IIS system has lower performance, but it only had one network controller and the TUX system has two - so while the TUX system had better performance, that could simply be because it had two network connections it could use.

   The second entry (the PowerEdge 6400/700) certainly suggests that GNU/Linux plus TUX really is much better - the IIS system had two more disk drives available to it (which should increase performance), but the TUX system had more than twice the IIS system's performance.

   The last entry for the PowerEdge 8450/700 is even more complex. First, the drives are different - the IIS systems had at least one drive that revolved more quickly than the TUX systems (which should give IIS higher performance overall, since the transfer speed is almost certainly higher). Also, there were more disk drives (which again should give IIS still higher performance). When I originally put this table together showing all data publicly available in April 2001 (covering the third quarter of 1999 through the first quarter of 2001), IIS 5.0 (on an 8-processor Dell PowerEdge 8450/700) had a SPECweb99 value of 7300. Since that time, Microsoft changed the availability of Microsoft SWC 3.0, and by SPECweb99 rules, this means that those test results are ``not compliant'' (NC). This is subtle; it's not that the test itself was invalid, it's that Microsoft changed what was available and used the SPEC Consortium's own rules to invalidate a test (possibly because the test results were undesirable to Microsoft). A retest then occurred, with yet another disk drive configuration, at which point IIS produced a value of 8001. However, both of these figures are on clearly better hardware - and in one circumstance the better hardware didn't do better.

   Thus, in these configurations the GNU/Linux plus TUX system was given inferior hardware yet still sometimes won on performance. Since other factors may be involved, it's hard to judge - there are pathological situations where ``better hardware'' can have worse performance, or there may be another factor not reported that had a more significant effect. Hopefully in the future there will be many head-to-head tests in a variety of identical configurations.

   Note that TUX is intended to be used as a ``web accelerator'' for many circumstances, where it rapidly handles simple requests and then passes more complex queries to another server (usually Apache). I've quoted the TUX figures because they're the recent performance figures I have available. As of this time I have no SPECweb99 figures or other recent performance measures for Apache on GNU/Linux, or for Apache and TUX together; I also don't have TUX reliability figures. I expect that such measures will appear in the future.

2. In performance tests by Sys Admin magazine, GNU/Linux beat Solaris (on Intel), Windows 2000, and FreeBSD. The article ``Which OS is Fastest for High-Performance Network Applications?'' in the July 2001 edition of Sys Admin magazine examined high-performance architectures and found that GNU/Linux beat its competition when compared with Solaris (on Intel), FreeBSD (an OSS/FS system), and Windows 2000. They intentionally ran the systems ``out of the box'' (untuned), except for increasing the number of simultaneous TCP/IP connections (which is necessary for testing multi-threaded and asynchronous applications). They used the latest versions of operating systems and the exact same machine. They reported (by operating system) the results of two different performance tests.

   The FreeBSD developers complained about these tests, noting that FreeBSD by default emphasizes reliability (not speed) and that they expected anyone with a significant performance need would do some tuning first. Thus, Sys Admin's re-did the tests for FreeBSD after tuning FreeBSD. One change they made was switching to ``asynchronous'' mounting, which makes a system faster (though it increases the risk of data loss in a power failure) - this is the GNU/Linux default and easy to change in FreeBSD, so this was a very small and reasonable modification. However, they also made many other changes, for example, they found and compiled in 17 FreeBSD kernel patches and used various tuning commands. The other operating systems weren't given the chance to ``tune'' like this, so comparing untuned operating systems to a tuned FreeBSD isn't really fair.

   In any case, here are their two performance tests:

   1. Their ``real-world'' test measured how quickly large quantities of email could be sent using their email delivery server (MailEngine). Up to 100 simultaneous sends there was no difference, but as the number increased the systems began showing significant differences in their hourly email delivery speed. By 500 simultaneous sends GNU/Linux was clearly faster than all except FreeBSD-tuned, and GNU/Linux remained at the top. FreeBSD-tuned had similar performance to GNU/Linux when running 1000 or less simultaneous sends, but FreeBSD-tuned peaked around 1000-1500 simultaneous connections with a steady decline not suffered by GNU/Linux, and FreeBSD-tuned had trouble going beyond 3000 simultaneous connections. By 1500 simultaneous sends, GNU/Linux was sending 1.3 million emails/hour, while Solaris managed approximately 1 million, and Windows 2000 and FreeBSD-untuned were around 0.9 million.
   2. Their ``disk I/O test'' created, wrote, and read back 10,000 identically-sized files in a single directory, varying the size of the file instances. Here Solaris was the slowest, with FreeBSD-untuned the second-slowest. FreeBSD-tuned, Windows 2000, and GNU/Linux had similar speeds at the smaller file sizes (in some cases FreeBSD-tuned was faster, e.g., 8k and 16k file size), but when the file sizes got to 64k to 128k the operating systems began to show significant performance differences; GNU/Linux was the fastest, then Windows 2000, then FreeBSD. At 128k, FreeBSD was 16% worse than Windows 2000, and 39% worse than GNU/Linux; all were faster than FreeBSD-untuned and Solaris. When totaling these times across file sizes, the results were GNU/Linux: 542 seconds, Windows 2000: 613 seconds, FreeBSD-tuned: 630 seconds, FreeBSD-untuned: 2398 seconds, and Solaris: 3990 seconds.

3. Low-level benchmarks by IBM found that GNU/Linux had better pipe performance than Windows. Ed Bradford (manager of Microsoft Premier Support for IBM Software group) published in October 2001 the study Pipes in Linux, Windows 2000, and Windows XP. In this study he examined the the performance of pipes, a common low-level mechanism for communicating between program processes. He found the pipes in Red Hat 7.1 (with Linux kernel 2.4.2 kernel) had a peak I/O rate of around 700 MB/sec, with a steady state at near 100 MB/sec for very large block sizes. In constrast, Windows 2000 peaked at 500 MB/sec, with a large block steady state of 80 MB/sec. Windows XP Professional (evaluation version) was especially disappointing; its peak I/O rate was only 120 MB/sec, with a stead state of 80 MB/sec, all on the same platform and all running a GUI.

4. PC Magazine's November 2001 performance tests for file servers found that Linux with Samba significantly outperformed Windows 2000 PC Magazine's article Performance Tests: File Server Throughput and Response Times found that Linux with Samba significantly outperformed Windows 2000 Server when used as a file server for Microsoft's own network file protocols. This was true regardless of the number of simultaneous clients (they tested a range up to 30 clients), and it was true on the entire range on computers they used (PII/233Mhz with 128Mb RAM, PIII/550 with 256Mb RAM, and PIII/1GHz with 512Mb RAM). Indeed, as the machines became more capable the absolute difference became more pronounced; on fastest hardware and largest number of clients, Linux's throughput was about 130 Mb/sec vs. Windows' 78 Mb/sec, so Linux was 78% faster in that case.

All operating systems in active development are in a constant battle for performance improvements over their rivals. The history of comparing Windows and GNU/Linux helps put this in perspective:

1. Ziff-Davis found that GNU/Linux with Apache beat Windows NT 4.0 with IIS by 16%-50% depending on the GNU/Linux distribution. Ziff-Davis compared Linux and Windows NT's performance at web serving. They found that ``Linux with Apache beats NT 4.0 with IIS, hands down. SuSE, the least effective Linux, is 16% faster than IIS, and Caldera, the leader, is 50% faster.''

2. Mindcraft released a report in April 1999 that claimed that Microsoft Windows NT Server 4.0 is 2.5 times faster than Linux (kernel 2.2) as a File Server and 3.7 times faster as a Web Server when running on a 4-CPU SMP system. Several people and organizations, such Linux Weekly News (LWN) and Dan Kegel, identified serious problems with this study. An obvious issue was that NT was specially tuned by Microsoft's NT experts, at Microsoft, while GNU/Linux was not tuned at all. Another issue is that the price/performance wasn't considered (nor was total expenditure kept constant - for the same amount of money, the GNU/Linux system could have had better hardware). Mindcraft claimed they asked for help, but they didn't use the documented methods for getting help nor did they purchase a support contract. Many were especially offended that even though this study was funded by Microsoft (one of the contestants) and held at their facility, neither Mindcraft's initial announcement nor its paper made any mention of this conflict-of-interest - and it could be easily claimed that their configuration was designed to put GNU/Linux at a disadvantage. Their configuration was somewhat bizarre - it assumed all web pages were static (typical big sites tend to use many dynamically generated pages) and that there were 100 or so clients connected via 100baseT (in 1999 a more typical situation would be that most clients are using slower 28.8 or 56 Kbps modems).

   Careful examination of the benchmark did find some legitimate Linux kernel problems, however. These included a TCP bug, the lack of ``wake one'' semantics, and SMP bottlenecks (see Dan Kegel's pages for more information). The Linux kernel developers began working on the weaknesses identified by the benchmark.

3. PC Week confirmed that Windows did indeed do better in this less probable configuration. In June 30, 1999, Mindcraft released their Open Benchmark in conjunction with PC Week. While this didn't excuse Mindcraft's biases, it did make a convincing case that there were legitimate problems in the Linux kernel and Apache that made GNU/Linux a poorer-performing product in this somewhat improbable configuration (serving static web pages to clients with high-speed connections). Note that this configuration was considerably different than Ziff-Davis's, so the benchmarks don't necessarily conflict; it's merely that different assumptions can produce different results (as I've already stressed).

4. Network Computing found that GNU/Linux with Samba ran at essentially the same speed as Windows for file serving. In their article ``Is it Time for Linux'', Network Computing compared Red Hat Linux v5.2 running Samba 2.0.3 against Microsoft Windows NT Server Enterprise Edition on a Pentium II-based HP NetServer LPr. They used Coffee Computing Corp.'s FileMetric 1.0 benchmarking tool (www.coffeecomputing.com), stressing the machine with multiple reads and writes of small, medium and large files over the course of several hours, examining ramp-ups, massive file transfers and saturating the network.

   For file serving, they discovered only ``negligible performance differences between the two for average workloads... [and] depending on the degree of tuning performed on each installation, either system could be made to surpass the other slightly in terms of file-sharing performance.'' Linux slightly outperformed NT on file writes, but NT edged out Linux on massive reads. Note that their configuration was primarily network-limited; they stated ``At no point were we able to push the CPUs much over 50-percent utilization--the single NIC, full duplex 100BASE-T environment wouldn't allow it.''

   They also noted that ``examining the cost difference between the two licenses brings this testing into an entirely new light... the potential savings on licenses alone is eye-opening. For example, based on the average street price of $30 for a Windows NT client license, 100 licenses would cost around $3,000, plus the cost of an NT server license (around $600). Compare this to the price of a Red Hat Linux CD, or perhaps even a free download, and the savings starts to approach the cost of a low-end workgroup server. Scale that up to a few thousand clients and you begin to see the savings skyrocket.''

5. The German magazine c't found that web sites with NT was better at static content and dual network connections, but GNU/Linux was better for sites with dynamic content and single connections. Their article Mixed Double: Linux and NT as Web Server on the Test Bed examined Windows NT with IIS against GNU/Linux (kernel 2.2.9) with Apache on a machine with four Pentium II Xeon CPUs. They found that the performance winner depended on the situation (by now that should not be a surprise). If the web server primarily served static web pages through two high-performance network cards, NT's performance was better. However, they also noted that in sophisticated web sites this result didn't apply, because such sites tend to have primarily dynamic content, and that few sites had this kind of dual-network connection (when only one network board was available, GNU/Linux generally had an edge). See their paper for more figures and background.

6. The Linux developers' various efforts to improve performance appear to have paid off. In June 2000, Dell measured the various SPECweb99 values noted above.

There are other benchmarks available, but I've discounted them on various grounds:

1. A more recent set of articles from eWeek on June 2001, shows some eye-popping performance numbers for GNU/Linux with TUX. However, although they compare it to Microsoft IIS, they don't include Microsoft's SWC (Scaleable Web Cache), Microsoft's response to TUX - and omitting it makes this comparison unfair in my opinion. You can read more at ``Tux: Built for Speed'', ``Smart Coding pays off Big'', and Kegel's detailed remarks.

2. The ZDNet article Take that! Linux beats MS in benchmark test, loudly trumpeted that GNU/Linux was the May 2001 performance leader in the TPC-H decision support (database) benchmark (``100Gb'' category). However, this result should not be taken very seriously; the hardware that Linux ran on was more powerful than that of the runner-up (Windows 2000). Frankly, the more surprising fact than its top score (which can be easily explained by the hardware) is its mere measurement at all with this benchmark - traditionally only Microsoft's numbers were reported for this benchmark at this range. For more information, see the TPC results.

You can get a lot more information on various benchmarks from Kegel's NT vs. Linux Server Benchmark Comparisons, and many more benchmarks from SPEC. You can also see the dmoz entry on benchmarking.

Remember, in benchmarking everything depends on the configuration and assumptions that you make. Many systems are constrained by network bandwidth; in such circumstances buying a faster computer won't help at all. Even when network bandwidth isn't the limitation, neither Windows nor GNU/Linux do well in large-scale symmetric multiprocessing (SMP) configurations; if you want 64-way CPUs with shared memory, neither are appropriate (Sun Solaris, which is not OSS/FS, does much better in this configuration). On the other hand, if you want massive distributed (non-shared) memory, GNU/Linux does quite well, since you can buy more CPUs with a given amount of money. If massive distribution can't help you and you need very high performance, Windows isn't even in the race; today Windows 2000 only runs on Intel x86 compatible chips, while GNU/Linux runs on much higher performance processors as well as the x86.

Scaleability

1. GNU/Linux and NetBSD (both OSS/FS) support a wider range of hardware platforms and performance than any other operating system. Many people mean by ``scaleability'' to answer the question, ``can you use the same software system for both small and large projects?'' Often the implied issue is that you'd like to start with a modest system, but have the ability to grow the system as needs demand without expensive modifications. Here OSS/FS is unbeatable; because many people can identify scaleability problems, and because its source code can be optimized for its platform, the scaleability of many OSS/FS products is amazing. Let's specifically look at GNU/Linux. GNU/Linux works on PDAs (including the Agenda VR3), obsolete hardware (so you needn't throw the hardware away), common modern PC hardware, over a dozen different chipsets (not just Intel x86s), mainframes, massive clusters, and a number of supercomputers. GNU/Linux can be used for massive parallel processing; a common approach for doing this is the Beowulf architecture. Sandia's ``CPlant'' runs on a set of systems running GNU/Linux, and it's the forty-second most powerful computer in the world as of June 2001 (number 42 on the TOP 500 Supercomputer list, June 2001). There's even a prototype implementation of GNU/Linux on a wrist watch, And GNU/Linux runs on a vast number of different CPU chips, including the x86, Intel Itanium, ARM, Alpha, IBM AS/400 (midrange), SPARC, MIPS, 68k, and Power PC. Another operating system that widely scales to many other hardware platforms is NetBSD.

   Thus, you can buy a small GNU/Linux or NetBSD system and grow it as your needs grow; indeed, you can replace small hardware with massively parallel or extremely high-speed processors or very different CPU architectures without switching operating systems. Windows CE/ME/NT scales down to small platforms, but not to large ones, and it only works on x86 systems. Many Unix systems (such as Solaris) scale well to specific large platforms but not as well to distributed or small platforms. These OSS/FS systems are some of the most scaleable programs around.

2. OSS/FS development processes can scale to develop large software systems. At one time it was common to ask if the entire OSS/FS process is ``scaleable,'' that is, if OSS/FS processes could really develop large-scale systems. Bill Gates' 1976 ``Open Letter to Hobbyists'' asked rhetorically, ``Who can afford to do professional work for nothing? What hobbyist can put three man-years into programming, finding all bugs, documenting his product, and distribute it for free?'' He presumed these were unanswerable questions. He was wrong; I developed a quantitative examination of the content of a GNU/Linux distribution. Feel free to see my reports estimating GNU/Linux's size. For Red Hat Linux 6.2, I estimate the size to be over 17 million source lines of code (SLOC). Done traditionally it would have taken 4,500 person-years and over $600 million to implement. For Red Hat Linux 7.1, I found it to have over 30 million SLOC, representing 8,000 person-years or $1 billion (a ``Gigabuck''). Most developers ascribe to the design principle that components should be divided into smaller components where practical - a practice also applied to GNU/Linux - but some components aren't easily divided, and thus some components are quite large themselves (e.g., over 2 million lines of code for the kernel, mostly in device drivers). Thus, it's no longer reasonable to argue that OSS/FS cannot scale to develop large systems -- because it clearly can.

Security

1. J.S. Wurzler Underwriting Managers' ``hacker insurance'' costs 5-15% more if Windows is used instead of Unix or GNU/Linux for Internet operation. At least one insurance company has indicated that Windows NT is less secure than Unix or GNU/Linux systems, resulting in higher premiums for Windows-based systems. It's often difficult to find out when a company has been successfully cracked; companies often don't want to divulge such information to the public for a variety of reasons. Indeed, if consumers or business partners lost trust in a company, the resulting loss might be much greater than the original attack. However, insurance companies that insure against cracking can require that they get such information (as a condition of coverage), and can compute future premiums based on that knowledge. According to CNET, Okemos, Mich.-based J.S. Wurzler Underwriting Managers, one of the earliest agencies to offer ``hacker insurance'' (and thus more likely to have historical data for premium calculation), has begun charging its clients anywhere from 5 to 15 percent more if they use Microsoft's Windows NT software instead of Unix or GNU/Linux for their Internet operations. Walter Kopf, senior vice president of underwriting, said that ``We have found out that the possibility for loss is greater using the NT system.'' He also said the decision is based on findings from hundreds of security assessments the company has done on their small and midsize business clients over the past couple of years.

2. Most defaced web sites are hosted by Windows, and Windows sites are disproportionately defaced more often than explained by its market share. Another way to look at security is to look at the operating system used by defaced web sites, and compare them to their market share. A ``defaced'' web site is a site that has been broken into and has its content changed (usually in a fairly obvious way, since subtle modifications are often not reported). The advantage of this measure is that unlike other kinds of security break-ins (which are often ``hushed up''), it's often very difficult for victims to hide the fact that they've been successfully attacked. Historically, this information was maintained by Attrition.org. A summary can be found in James Middleton's article, with the actual data found in Attrition.org's web site. Attrition.org's data showed that 59% of defaced systems ran Windows, 21% Linux, 8% Solaris, 6% BSD, and 6% all others in the period of August 1999 through December 2000. Thus, Windows systems have has nearly 3 times as many defacements as GNU/Linux systems. This would make sense if there were 3 times as many Windows systems, but no matter which figures you use, that's simply not true.

   Of course, not all sites are broken through their web server and OS - many are broken through exposed passwords, bad web application programming, and so on. But if this is so, why is there such a big difference in the number of defacements based on the operating system? No doubt some other reasons could be put forward (this data only shows a correlation not a cause), but this certainly suggests that OSS/FS can have better security.

   Attrition.org has decided to abandon keeping track of this information due to the difficulty of keeping up with the sheer volume of broken sites, and it appeared that tracking this information wouldn't be possible. However, defaced.alldas.de has decided to perform this valuable service. Their recent reports show that this trend has continued; on July 12, 2001, they report that 66.09% of defaced sites ran Windows, compared to 17.01% for GNU/Linux, out of 20,260 defaced websites.

3. The Bugtraq vulnerability database suggests that the least vulnerable operating system is OSS/FS, and that all the OSS/FS operating systems in its study were less vulnerable than Windows in 1999-2000. One approach to examining security is to use a vulnerability database; an analysis of one database is the Bugtraq Vulnerability Database Statistics page. As of September 17, 2000, here are the total number of vulnerabilities for some leading operating systems:

   OS	1997	1998	1999	2000
   Debian GNU/Linux	2	2	30	20
   OpenBSD	1	2	4	7
   Red Hat Linux	5	10	41	40
   Solaris	24	31	34	9
   Windows NT/2000	4	7	99	85

   You shouldn't take these numbers very seriously. Some vulnerabilities are more important than others (some may provide little if exploited or only be vulnerable in unlikely circumstances), and some vulnerabilities are being actively exploited (while others have already been fixed before exploitation). Open source operating systems tend to include many applications that are usually sold separately in proprietary systems (including Windows and Solaris) - for example, Red Hat 7.1 includes two relational database systems, two word processors, two spreadsheet programs, two web servers, and a large number of text editors. In addition, in the open source world, vulnerabilities are discussed publicly, so vulnerabilities may be identified for software still in development (e.g., ``beta'' software). Those with small market shares are likely to have less analysis. The ``small market share'' comment won't work with GNU/Linux, of course, since we've already established that GNU/Linux is the #1 or #2 server OS (depending on how you count them). Still, this clearly shows that the three OSS/FS OSs listed (Debian GNU/Linux, OpenBSD, and Red Hat Linux) did much better by this measure than Windows in 1999 and (so far) in 2000. Even if a bizarre GNU/Linux distribution was created explicitly to duplicate all vulnerabilities present in any major GNU/Linux distribution, this intentionally bad GNU/Linux distribution would still do better than Windows (it would have 88 vulnerabilities in 1999, vs. 99 in Windows). The best results were for OpenBSD, an OSS/FS operating system that for years has been specifically focused on security. It could be argued that its smaller number of vulnerabilities is because of its rarer deployment, but the simplest explanation is that OpenBSD has focused strongly on security - and achieved it better than the rest.

   This data is partly of interest because one journalist completely distorted these numbers in an attempt to show that GNU/Linux was worse. He took these numbers and then added the GNU/Linux ones so each Linux vulnerability was counted at least twice (once for every distribution it applied to plus one more). By using these nonsensical figures he declared that GNU/Linux was worse than anything. If you read his article, you also need to read the rebuttal by the manager of the Microsoft Focus Area at SecurityFocus to understand why the journalist's article was so wrong.

   Indeed, as noted in Bruce Schneier's Crypto-gram of September 15, 2000, vulnerabilities are affected by other things such as how many attackers exploit the vulnerability, the speed at which a fix is released by a vendor, and the speed at which they're applied by administrators. Nobody's system is invincible.

   A more recent analysis by John McCormick in Tech Republic compared Windows and Linux vulnerabilities using numbers through September 2001. This is an interesting analysis, showing that although Windows NT lead in the number of vulnerabilities in 2000, using the 2001 numbers through September 2001, Windows 2000 had moved to the ``middle of the pack'' (with some Linux systems having more, and others having fewer, vulnerabilities). However, it appears that in these numbers, bugs in Linux applications have been counted with Linux, while bugs in Windows applications haven't - and if that's so, this isn't really a fair comparison. As noted above, typical Linux distributions bundle many applications that are separately purchased from Microsoft.

4. Red Hat (an OSS/FS vendor) responded more rapidly than Microsoft or Sun to advisories; Sun had fewer advisories to respond to yet took the longest to respond. Another data point is that SecurityPortal has compiled a list of the time it takes for vendors to respond to vulnerabilities. They concluded that:

   How did our contestants [fare]? Red Hat had the best score, with 348 recess days on 31 advisories, for an average of 11.23 days from bug to patch. Microsoft had 982 recess days on 61 advisories, averaging 16.10 days from bug to patch. Sun proved itself to be very slow, although having only 8 advisories it accumulated 716 recess days, a whopping three months to fix each bug on average.

   Their table of data for 1999 is as shown:

   1999 Advisory Analysis Vendor Total Days of Hacker Recess Total Advisories Days of Recess per Advisory Red Hat 348 31 11.23 Microsoft 982 61 16.10 Sun 716 8 89.50

   Clearly this table uses a different method for counting security problems than the previous table. Of the three noted here, Sun's Solaris had the fewest vulnerabilities, but it took by far the longest to fix security problems identified. Red Hat was the fastest at fixing security problems, and placed in the middle of these three in number of vulnerabilities. It's worth noting that the OpenBSD operating system (which is OSS/FS) had fewer reported vulnerabilities than all of these. Clearly, having a proprietary operating system doesn't mean you're more secure - Microsoft had the largest number of security advisories, by far, using either counting method.

5. Apache has a better security record than Microsoft's IIS, as measured by reports of serious vulnerabilities. Eweek's July 20, 2001 article ``Apache avoids most security woes'' examined security advisories dating back to Apache 1.0. They found that Apache's last serious security problem (one where remote attackers could run arbitrary code on the server) was announced in January 1997. A group of less serious problems (including a buffer overflow in the server's logresolve utility) was announced and fixed in January 1998 with Apache 1.2.5. In the three and a half years since then, Apache's only remote security problems have been a handful of denial-of-service and information leakage problems (where attackers can see files or directory listings they shouldn't).

   In contrast, in the article ``IT bugs out over IIS security,'' eWeek determined that Microsoft has issued 21 security bulletins for IIS from January 2000 through June 2001. Determining what this number means is a little difficult, and the article doesn't discuss these complexities, so I've examined Microsoft's bulletins myself to find their true significance. Not all of the bulletins have the same significance, so just stating that there were ``21 bulletins'' doesn't give the whole picture. However, it's clear that several of these bulletins discuss dangerous vulnerabilities that allow an external user to gain control over the system. I count 5 bulletins on such highly dangerous vulnerabilities for IIS 5.0 (in the period from January 2000 through June 2001), and previous to that time, I count 3 such bulletins for IIS 4.0 (in the period of June 1998 through December 1999). Feel free to examine the bulletins yourself; they are MS01-033, MS01-026, MS01-025, MS01-023, MS00-086, MS99-025, MS99-019, and MS99-003. The Code Red worm, for example, exploited a vast number of IIS sites through the vulnerabilities identified in the June 2001 security bulletin MS01-033.

   In short, by totaling the number of reports of dangerous vulnerabilities (that allow attackers to execute arbitrary code), I find a total of 8 bulletins for IIS from June 1998 through June 2001, while Apache had zero such vulnerabilities for that time period. Apache's last such report was in January 1998, and that one affected the log analyzer not the web server itself. As was noted above, the last such dangerous vulnerability in Apache itself was announced in January 1997.

   The article claims there are four reasons for Apache's strong security, and three of these reasons are simply good security practices. Apache installs very few server extensions by default (a ``minimalist'' approach), all server components run as a non-privileged user (supporting ``least privilege''), and all configuration settings are centralized (making it easy for administrators to know what's going on). However, the article also claims that one of the main reasons Apache is more secure than IIS is that its ``source code for core server files is well-scrutinized,'' a task that is made much easier by being OSS/FS, and it could be argued that OSS/FS encourages the other good security practices.

   Simple counts of vulnerability notices aren't necessarily a good measure, of course. A vendor could intentionally release fewer bulletins - but since Apache's code and its security is publicly discussed, it seems unlikely that Apache is releasing fewer notices. Fewer vulnerability notices could result if the product isn't well scrutinized or is rarely used - but this simply isn't true for Apache. Even the trend line isn't encouraging - using the months of the bulletins (2/99, 6/99, 7/99, 11/00, three in 5/01, and 6/01), I find the time in months between new major IIS vulnerability announcements to be 4, 1, 18, 6, 0, 0, 1, and 3 as of September 2001; this compares to 12 and 44 as of September 2001 for Apache. Given these trends, it looks like IIS's security is slowly improving, but it has little likelihood of meeting Apache's security in the near future. Indeed, these vulnerability counts are corroborated by other measures such as the web site defacement rates.

   The issue here isn't whether or not a particular program is invincible (what nonsense!) - the issue here is which is more likely to resist future attacks, based on past performance. It's clear that Apache has much a better security record than IIS, so much so that Gartner Group decided to make an unusual recommendation (described next).

6. The Gartner Group is recommending that businesses switch from Microsoft IIS to Apache or iPlanet due to IIS's poor security track record, noting that enterprises had spent $1.2 billion simply fixing Code Red (IIS-related) vulnerabilities by July 2001. Microsoft's IIS has such a bad security record that in September 2001, Gartner Group announced a recommendation that ``businesses hit by both Code Red and Nimda immediately investigate alternatives to IIS, including moving Web applications to Web server software from other vendors such as iPlanet and Apache. Although those Web servers have required some security patches, they have much better security records than IIS and are not under active attack by the vast number of virus and worm writers.'' Microsoft is sometimes a Gartner Group customer, so this announcement is especially surprising.

   In a background document by Gartner, they discuss Code Red's impacts further. By July 2001, Computer Economics (a research firm) estimated that enterprises worldwide had spent $1.2 billion fixing vulnerabilities in their IT systems that Code Red could exploit (remember, Code Red is designed to only attack IIS systems; systems such as Apache are immune). To be fair, Gartner correctly noted that the problem is not just that IIS has vulnerabilities; part of the problem is that enterprises using IIS are not keeping their IT security up to date, and Gartner openly wondered why this was the case. However, Gartner also asked the question, ``why do Microsoft's software products continue to provide easily exploited openings for such attacks?'' This was prescient, since soon after this the ``Nimba'' attack surfaced which attacked IIS, Microsoft Outlook, and other Microsoft products.

   A brief aside is in order here. Microsoft spokesman Jim Desler tried to counter Gartner's recommendation, trying to label it as ``extreme'' and saying that ``serious security vulnerabilities have been found in all Web server products and platforms.. this is an industrywide challenge.'' While true, this isn't the whole truth. As Gartner points out, ``IIS has a lot more security vulnerabilities than other products and requires more care and feeding.'' It makes sense to select the product with the best security track record, even if no product has a perfect record.

7. The majority of CERT/CC's ``most frequent, high-impact types of security incidents and vulnerabilities'' only apply to Microsoft's products, and not to OSS/FS products. Some security vulnerabilities are more important than others, for a variety of reasons. The CERT Coordination Center (CERT/CC) is federally funded to study security vulnerabilities and perform related activities such as publishing security alerts. I sampled their list of ``current activity'' of the most frequent, high-impact security incidents and vulnerabilities on September 24, 2001, and found yet more evidence that Microsoft's products have poor security compared to others (including OSS/FS). Four of the six most important security vulnerabilities were specific to Microsoft: W32/Nimda, W32/Sircam, cache corruption on Microsoft DNS servers, and ``Code Red'' related activities. Only one of the six items primarily affected non-Microsoft products (a buffer overflow in telnetd); while this particular vulnerability is important, it's worth noting that many open source systems (such as Red Hat 7.1) normally don't enable this service (telnet) in the first place and thus are less likely to be vulnerable. The sixth item (``scans and probes'') is a general note that there is a great deal of scanning and probing on the Internet, and that there are many potential vulnerabilities in all systems. Thus, 4 of 6 issues are high-impact vulnerabilities are specific to Microsoft, 1 of 6 are vulnerabilities primarily affecting Unix-like systems (including OSS/FS operating systems), and 1 of 6 is a general notice about scanning. Again, it's not that OSS/FS products never have security vulnerabilities - but they seem to have fewer of them.

8. According to a Network Security evaluation, an OSS/FS vulnerability scanner (Nessus) was found to be the best (most effective). On January 8, 2001, Network Computing's article Vulnerability Assessment Scanners. reported an evaluation of 9 network scanning tools, most of them proprietary. The tools were Axent Technologies' NetRecon, BindView Corp.'s HackerShield, eEye Digital Security's Retina, Internet Security Systems' Internet Scanner, Network Associates' CyberCop Scanner, and two open-source products: Nessus Security Scanner and Security Administrator's Research Assistant (SARA). One product, World Wide Digital Security's System Analyst Integrated Network Tool (SAINT), is open source, with a commercial reporting tool.

   In their evaluation, they set up systems with 17 of the most common and critical vulnerabilities, and evaluated how well each tool found the vulnerabilities. Sadly, not one product detected all vulnerabilities; the best scanner was the OSS/FS program Nessus Security Scanner, which found 15 of the 17 (which also received their top total score); the next best was Internet Security Systems' Internet Scanner, which found 13.5 out of 17.

   In their words,

   Some of us were a bit skeptical of the open-source Nessus project's thoroughness until [Nessus] discovered the greatest number of vulnerabilities. That's a hard fact to argue with, and we are now eating our words ... [Nessus] got the highest overall score simply because it did more things right than the other products.

   I agree with the authors that ideally a vulnerability scanner should find every well-known vulnerability, and that ``even one hole is too many.'' Still, perfection is rare in the real world. More importantly, a vulnerability scanner should only be part of the process to secure an organization - it shouldn't be the sole activity. Still, this evaluation suggests that an organization will be more secure, not less secure, by using an OSS/FS program. It could be argued that this simply shows that this particular OSS/FS program had more functionality - not more security - but in this case, the product's functionality was to improve security.

Now it should be obvious from these figures that OSS/FS systems are not magically invincible from security flaws. Indeed, some have argued that making the source code available gives attackers an advantage (because they have more information to make an attack). While OSS/FS gives attackers more information, this thinking ignores an opposing force: having the source code makes it possible for defenders to examine their code and improve it. For a longer description of these issues, see my discussion on open source and security (part of my book on writing secure software). However, from these figures, it appears that OSS/FS systems are often better - not just equal - in their resistance to attacks.

Total Cost of Ownership (TCO)

Indeed, whatever product you use or support, you can probably find a study to show it has the lowest TCO for some circumstance. Not surprisingly, both Microsoft and Sun provide studies showing that they have the lowest TCO (but see my comments later about Microsoft's study). Xephon has a study determining that Mainframes are the cheapest per-user (due to centralized control) at £3450 per user per year; Centralized Unix cost £7350 per user per year, and a decentralized PC environment costs £10850 per user per year. Xephon appears to be a mainframe-based consultancy, though, and would want the results to come out this way.

In short, what has a smaller TCO depends on your environment and needs. To determine TCO you have to identify all the important cost drivers (the ``cost model'') and estimate their costs. Don't forget ``hidden'' costs, such as administration costs, upgrade costs, technical support, end-user operation costs, and so on. However, OSS/FS has a number of strong cost advantages in various categories that, in many cases, will result in its having the smallest TCO.

1. OSS/FS costs less to initially acquire. OSS/FS costs much less to get initially. OSS/FS isn't really ``free'' in the monetary sense to get; the ``free'' in ``free software'' refers to freedom, not price (usually summarized as ``free speech, not free beer''). You'll still spend money for paper documentation, support, training, system administration, and so on, just as you do with proprietary systems. Many distributions' executable programs can be acquired freely by downloading them (linux.org provides some pointers on how to get distributions). However, most people (especially beginners) will want to pay a small fee to a distributor for a nicely integrated package with CD-ROMs, paper documentation, and support. Even so, OSS/FS is far less expensive.

   For example, look at some of the price differences when trying to configure a server (say a public web server or an intranet file and email server, in which you'd like to use C++ and an RDBMS for some portions of it). This is an example, of course; different missions would involve different components. I used the prices from ``Global Computing Supplies'' (Suwanee, GA), September 2000, and rounded to the nearest dollar. Here's a quick summary of some costs:

   	Microsoft Windows 2000	Red Hat Linux
   Operating System	$1510 (25 client)	$29 (standard), $76 deluxe, $156 professional (all unlimited)
   Email Server	$1300 (10 client)	included (unlimited)
   RDBMS Server	$2100 (10 CALs)	included (unlimited)
   C++ Development	$500	included

   Basically, Microsoft Windows 2000 (25 client) costs $1510; their email server Microsoft Exchange (10-client access) costs $1300, their RDBMS server SQL Server 2000 costs $2100 (with 10 CALs), and their C++ development suite Visual C++ 6.0 costs $500. Red Hat Linux 6.2 (a widely-used GNU/Linux distribution) costs $29 for standard (90 days email-based installation support), $76 for deluxe (above plus 30 days telephone installation support), or $156 for professional (above plus SSL support for encrypting web traffic); in all cases it includes all of these functionalities (web server, email server, database server, C++, and much more). A public web server with Windows 2000 and an RDBMS might cost $3610 ($1510+$2100) vs. Red Hat Linux's $156, while an intranet server with Windows 2000 and an email server might cost $2810 ($1510+$1300) vs. Red Hat Linux's $76.

   Both packages have functionality the other doesn't have. The GNU/Linux system always comes with an unlimited number of licenses; the number of clients you'll actually use depends on your requirements. However, this certainly shows that no matter what, Microsoft's server products cost thousands of dollars more per server than the equivalent GNU/Linux system. See Jimmo's Cost Comparison for another set of data points. Obviously, the price difference depends on exactly what functions you need for a given task.

2. Upgrade costs are typically far less. Long-term upgrade costs are far less for OSS/FS systems. For example, upgrading a Microsoft system will typically cost around half the original purchase. What's worse, you are essentially at their mercy for long-term pricing, because there is only a single supplier (see Microsoft Turns the Screws). In contrast, the GNU/Linux systems can be downloaded (free), or simply re-purchased (generally for less than $100), and the single upgrade be used on every system. This doesn't include technical support, but the technical support can be competed (a situation that's not practical for proprietary software). If you don't like your GNU/Linux supplier (e.g., they've become too costly), you can switch.

3. OSS/FS can often use older hardware more efficiently than proprietary systems, yielding smaller hardware costs and sometimes eliminating the need for new hardware. OSS/FS runs faster on faster hardware, of course, but many OSS/FS programs can use older hardware more efficiently than proprietary systems, resulting in lower hardware costs - and in some cases requiring no new costs (because ``discarded'' systems can suddenly be used again). For example, the minimum requirements for Microsoft Windows 2000 Server (according to Microsoft) are a Pentium-compatible CPU (133 MHz or higher), 128 Mb of RAM minimum (with 256Mb the ``recommended minimum''), and a 2 GB hard drive with at least 1.0 Gb free. According to Red Hat, Red Hat Linux 7.1 (a common distribution of GNU/Linux) requires at a minimum an i486 (Pentium-class recommended), 32Mb RAM (64Mb recommended), and 650Mb hard disk space (1.2 Gb recommended).

   In Scientific American's August 2001 issue, the article The Do-It-Yourself Supercomputer discusses how the researchers built a powerful computing platform with a large number of obsolete, discarded computers and GNU/Linux. The result was dubbed the ``Stone Soupercomputer''; by May 2001 it contained 133 nodes, with a theoretical peak performance of 1.2 gigaflops.

4. As the number of systems and hardware performance increases, this difference in initial and upgrade costs becomes even more substantial. As the number of servers increases, proprietary solutions become ever more expensive. First, many proprietary systems (including Microsoft) sell per-client licenses; this means that even if your hardware can support more clients, you'll have to pay more to actually use the hardware you've purchased. Secondly, if you want to use more computers, you have to pay for more licenses in proprietary systems. In contrast, for most GNU/Linux distributions, you can install as many copies as you like for no additional fee, and there's no performance limit built into the software. There may be a fee for additional support, but you can go to competing vendors for this support.

   According to Network World Fusion News, Linux is increasingly being used in healthcare, finance, banking, and retail because of its cost advantages when large numbers of identical sites and servers are built. According to their calculations for a 2,000 site deployment, SCO UnixWare would cost $9 million, Windows would cost $8 million, and Red Hat Linux costs $180.

5. There are many other factors; their effect varies on what you're trying to do. There are many other factors in TCO, but it's difficult to categorize their effects in general, and it's generally difficult to find justifiable numbers for these other effects. Windows advocates claim that system administrators are cheaper and easier to find than Unix/Linux administrators, while GNU/Linux and Unix advocates argue that fewer such administrators are needed (because administration is easier to automate and the systems are more reliable to start with). Some GNU/Linux advocates have told me that GNU/Linux lends itself to hosting multiple services on a single server in cases where Windows installations must use multiple servers. License compliance administration can be costly for proprietary systems (e.g., time spent by staff to purchase CALS, keep track of licenses, and undergo audits) - a cost that simply isn't relevant to OSS/FS.

6. For many circumstances, the total cost savings can be substantial; for example, savings exceeding $250,000 per year were reported by 32% of the CTOs surveyed in a 2001 InfoWorld survey; 60% of these CTOs saved more than $50,000 annually. The August 27, 2001 InfoWorld (pages 49-50) reported on a survey of 40 CTOs who were members of the InfoWorld CTO network. In this survey, 32% using OSS reported savings greater than $250,000; 12% reported savings between 100,001 and $250,000; and 16% reported saving between $50,001 and $100,000. Indeed, only 8% reported annual savings less than $10,000 (so 92% were saving $10,000 or more annually). A chief benefit of OSS, according to 93% of the CTOs, was reduced cost of application development or acquisition; 72% said that a chief benefit was reduced development or implementation time (multiple answers were allowed). The CTOs reported using or planning to use OSS for web servers (65%), server operating systems (63%), web-application servers (45%), application development testing (45%), and desktop operating system (38%), among other uses. InfoWorld summarized it this way: ``in early 2000, it seemed as if no one was using open-source software for business-critical tasks... a vast majority of today's corporate IT executives are now using or plan to use OSS operating systems and web servers for their enterprise applications.''

Microsoft's TCO study (mentioned earlier) is probably not useful as a starting point for estimating your own TCO. Their study reported the average TCO at sites using Microsoft products compared to the average TCO at sites using Sun systems, but although the Microsoft systems cost 37% less to own, the Solaris systems handled larger databases, more demanding applications connecting to those databases, 63% more concurrent connections, and 243% more hits per day. In other words, the Microsoft systems that did less work were less expensive. This is not a useful starting point if you're using TCO to help determine which system to buy -- to make a valid comparison by TCO, you need to compare the TCOs of systems that both perform the job you need to do. A two-part analysis by Thomas Pfau (see part 1 and part 2) identifies this and many other flaws in the study.

Again, it's TCO that matters, not just certain cost categories. However, given these large differences, in many situations OSS/FS has a smaller TCO than proprietary systems. At one time it was claimed that OSS/FS installation took more time, but nowadays OSS/FS systems can be purchased pre-installed and automatic installers result in equivalent installation labor. Some claim that system administration costs are higher, but studies like Sun's suggest than in many cases the system administration costs are lower, not higher, for Unix-like systems (at least Sun's). For example, on Unix-like systems it tends to be easier to automate tasks (because you can, but do not need, to use a GUI) - thus over time many manual tasks can be automated (reducing TCO). Retraining costs can be significant - but now that GNU/Linux has modern GUI desktop environments, there's anecdotal evidence that this cost is actually quite small (I've yet to see serious studies quantitatively evaluating this issue). In short, it's often hard to show that a proprietary solution's purported advantages really help offset their demonstrably larger costs in other categories when there's a competing mature OSS/FS product for the given function.

The paper Linux as a Replacement for Windows 2000 is an example of an analysis comparing Red Hat Linux 7.1 to Windows 2000; in this customer's case, using Linux instead of Windows 2000 saved $10,000. The reviewer came from a Windows/DOS background, and after performing an intensive hands-on Linux project lasting several months, determined that ``you will be stunned by the bang for the buck that ... open source software offers.''

Does this mean that OSS/FS always have the lowest TCO? No! As I've repeatedly noted, it depends on its use. But the notion that OSS/FS always has the larger TCO is simply wrong.

Non-Quantitative Issues

In fairness, I must note that not all issues can be quantitatively measured, and to many they are the most important issues. These issues include freedom, protection from license litigation, and flexibility.

1. OSS/FS protects its users from the risks and disadvantages of single source solutions. While ``free software'' advocates use the term ``freedom,'' and some businesses emphasize different terms such as ``multiple sources,'' the issue is the same: users do not want to be held hostage by any one vendor. Businesses often prefer to buy products in which there is a large set of competing suppliers, because it reduces their risk; they can always switch to another supplier if they're not satisfied or the original supplier goes out of business. This translates into an effect on the products themselves: if customers can easily choose and switch between competing products, the products' prices go down and their quality goes up. Conversely, if there is a near monopoly for a given product, over time the vendor will raise the cost to use the product and limit its uses to those that benefit the monopolist.

   Historically, proprietary vendors eventually lose to vendors selling products available from multiple sources, even when their proprietary technology is (at the moment) better. Sony's Betamax format lost to VHS in the videotape market, and IBM's microchannel architecture lost to ISA in the PC architecture market, because customers prefer the reduced risk (and eventually reduced costs) of non-proprietary products. This is sometimes called ``commodification'', a term disparaged by proprietary vendors and loved by users. Since users spend the money, users eventually find someone who will provide what they want.

   With OSS/FS, users can choose between distributors, and if a supplier abandons them they can switch to another supplier. As a result, I believe OSS/FS tends to lower risks, lower costs, and higher quality products. Users can even band together and maintain the product themselves (this is how the Apache project was founded), making it possible for bands of users to protect themselves from abandonment.

2. OSS/FS protects its users from licensing management and litigation. Proprietary vendors make money from the sale of licenses, and are imposing increasingly complex mechanisms on consumers to manage these licenses. For example, Microsoft's Windows XP requires product activation - a scheme that means that an accumulation of hardware changes requires a new activation code. Proprietary vendors also litigate against those who don't comply with their complex licensing management requirements, creating increased legal risks for users. In contrast, OSS/FS users have no fear of litigation from the use and copying of OSS/FS. Licensing issues do come up when OSS/FS software is modified and then redistributed, but to be fair, proprietary software essentially forbids this action (so it's a completely new right). Even in this circumstance, redistributing modified OSS/FS software generally requires following only a few simple rules (depending on the license), such as giving credit to previous developers and releasing modifications under the same license as the original program.

3. OSS/FS has greater flexibility. OSS/FS users can tailor the product as necessary to meet their needs in ways not possible without source code. Users can tailor the product themselves, or hire whoever they believe can solve the problem (including the original developer). Some have claimed that this creates the ``danger of forking,'' that is, of multiple incompatible versions of a product. This is ``dangerous'' only to those who believe competition is evil - we have multiple versions of cars as well. And in practice, the high cost of maintaining software yourself has resulted in a process in which the change is contributed back to the community. If it's not contributed (e.g., it solves a problem that needed solving but only for a particular situation), then it's still a win for the user - because it solved a user's problem which would not have been solved otherwise.

While I cannot quantitatively measure these issues, these are actually the most important issues to many.

Other Information

Here are some other related information sources:

1. Microsoft has been trying to claim that open source is somehow dangerous, and indeed is its leading critic, yet the Wall Street Journal's Lee Gomes found that ``Microsoft Uses Open-Source Code Despite Denying Use of such Software.'' Here are some interesting quotes from his article:

   ... But Microsoft's statements Friday suggest the company has itself been taking advantage of the very technology it has insisted would bring dire consequences to others. ``I am appalled at the way Microsoft bashes open source on the one hand, while depending on it for its business on the other,'' said Marshall Kirk McKusick, a leader of the FreeBSD development team.

   More recently Microsoft has particularly targeted the GPL license, but ``it hasn't previously suggested that there were benign forms of open-source software, and while singling out Linux for special criticism, has tended to criticize all open-source with the same broad brush.'' The article closes with this statement:

   In its campaign against open-source, Microsoft has been unable to come up with examples of companies being harmed by it. One reason, said Eric von Hippel, a Massachusetts Institute of Technology professor who heads up a research effort in the field, is that virtually all the available evidence suggests that open source is ``a huge advantage'' to companies. ``They are able to build on a common standard that is not owned by anyone,'' he said. ``With Windows, Microsoft owns them.''

2. Some have commented about this paper that ``yes, but with OSS/FS you give up your right to sue if things go wrong.'' This paper is about numbers, so this is really outside its scope, but the obvious retort is that essentially all proprietary software licenses also forbid lawsuits - so this isn't a difference at all! Also, see ``A Senior Microsoft Attorney Looks at Open-Source Licensing'', where Bryan Pfaffenberger argues that ``With open-source software... you are, in principle, walking into the deal with your eyes wide open. You know what you're getting, and if you don't, you can find someone who does. Open-source licenses enable the community of users to inspect the code for flaws and to trade knowledge about such flaws, which they most assuredly do. Such licenses allow users to create derivative versions of the code that repair potentially hazardous problems the author couldn't foresee. They let users determine whether the program contains adequate safeguards against safety or security risks. In contrast, the wealthy software firms pushing UCITA are asking us to buy closed-source code that may well contain flaws, and even outright hazards attributable to corporate negligence -- but they won't let us see the code, let alone modify it. You don't know what you're getting.''
3. Several studies examine developers (instead of the programs they write), including ``A Quantitative Profile of a Community of Open Source Linux Developers'', Herman, Hertel and Niedner's study (based on questionnaires), and the Who Is Doing It (WIDI) study.
4. There are several general information sites about OSS/FS or Unix that might be of interest, such as the Free Software Foundation (FSF), the Open Source Initiative website, the Linux.org site, and the Unix versus NT site,
5. Recommendations of the Panel on Open Source Software For High End Computing; this is the report of a panel created by the (U.S.) President's Information Technology Advisory Committee (PITAC). It recommends that the ``Federal government should encourage the development of open source software as an alternate path for software development for high end computing''.
6. Large-scale roll-outs suggest that OSS/FS really is viable for enterprise deployments. Many retailer cash registers are switching to GNU/Linux, according to Information Week; for example, Home Depot plans to roll out 90,000 terminals running Linux by 2003.
7. Several documents were written to counter Microsoft's statements in "Linux Myths". This included LWN's response and Jamin Philip Gray's response, and the FUD-counter site. The shared source page argues that Microsoft's ``shared source'' idea is inferior to open source. The letter Free Software Leaders Stand Together argues against a number of statements by Craig Mundie.
8. ``NT Religious Wars: Why Are DARPA Researchers Afraid of Windows NT?'' found that, in spite of strong pressure by paying customers, computer science researchers strongly resisted basing research on Windows. Reasons given were: developers believe Windows is terrible, Windows really is terrible, Microsoft's highly restrictive non-disclosure agreements are at odds with researcher agendas, and there is no clear technology transition path for OS and network research products built on Windows (since only Microsoft can distribute changes to its products).
9. ``How Big Blue Fell For Linux'' is an article on how IBM transitioned to becoming a major backer. Now IBM plans to invest $1 Billion in GNU/Linux, and it's just one company. See the IBM annual report.
10. ``Open Source-onomics: Examining some pseudo-economic arguments about Open Source'' by Ganesh Prasad counters ``several myths about the economics of Open Source.''
11. For a scientifically unworthy but really funny look at what people who use the various operating systems say, take a look at the Operating System Sucks-Rules-O-Meter. It counts how many web pages make statements like ``Linux rocks''. It's really just an opinion poll, but if nothing else it's great for a laugh.
12. A general-purpose site that tries to compare all operating systems (with a bent towards Apple's MacOS) is http://www.osdata.com/; it has a lot of interesting information, though it tends towards testimonials and such instead of quantitative information.
13. The book The Cathedral and the Bazaar by Eric Raymond, available via the Cathedral-Bazaar web site, examines OSS/FS development processes and issues.
14. Microsoft inadvertently advocated OSS/FS in its leaked internal documents, called the "Halloween" documents.
15. Other evaluations include the Gartner Group's and GNet's evaluations.
16. For more general information on OSS/FS, see my list of Open Source Software / Free Software (OSS/FS) references at http://www.dwheeler.com/oss_fs_refs.html

Conclusions

OSS/FS has significant market share, is often the most reliable software, and in many cases has the best performance. OSS/FS scales, both in problem size and project size. OSS/FS software generally has far better security, particularly when compared to Windows. Total cost of ownership for OSS/FS is often far less than proprietary software, particularly as the number of platforms increases. These statements are not merely opinions; these effects can be shown quantitatively, using a wide variety of measures. This doesn't even consider other issues that are hard to measure, such as freedom from control by a single source, freedom from licensing management (with its accompanying litigation), and increased flexibility. I believe OSS/FS options should be carefully considered any time software or computer hardware is needed.

You can view this page at http://www.dwheeler.com/oss_fs_why.html. Feel free to see my home page at http://www.dwheeler.com/.