UserLinux: Repairing the Economic Paradigm of Enterprise Linux

Bruce Perens <bruce@perens.com>, Perens LLC

DRAFT. Recent changes are in red.

Subscribe to the discussion list.

Copyright 2003 Perens LLC. You may translate, excerpt, and reformat to fit your presentation, and you may republish the result, but you may not edit the material to change my opinion or take my statements out of context.

Note: I am going through the list and my own mailbox for suggestions to integrate to the paper. I am up to message 391 on the list, bringing me current with postings as of the morning of December 9, and haven't done my mailbox yet. - Bruce

The Problem

Enterprise users have embraced GNU/Linux. But the very aspects that make Linux desirable, its low cost, Open Source nature, and the way it gives customers more control over their software, are under attack by Linux vendors bent on increasing shareholder value. Businesses are paying more as Linux distributions demand a per-seat cost and service lock-in for software that they didn't develop and that others support. Many of the early adopters of Linux are small but profitable industries with extremely sophisticated needs, and commercial Linux distributors simply can't afford to pay much attention to them while larger markets are waiting.

This has hampered the adoption of Linux. For example, a very large multinational bank recently informed me that they had called off a 10,000-system Linux deployment because "Linux is now more expensive than Windows". An ISP complained that the cost of Enterprise Linux is greater than the annual profit of one of his servers.

We, the Free Software developers, created this software to empower everyone, and for everyone to share. But today's Enterprise Linux is a lock-in play, designed to draw the customer into expensive subscriptions and single-vendor service. Customers are made to agree not to pass service bulletins on to others. While this is within the letter of the licenses that we crafted for our software, it's outside of their spirit. We have no problem with payment for service, when service is rendered. But the $1000 per year or greater that many customers now pay for their Linux systems goes not for service, but for a brand and the endorsement of a few application providers like Oracle.

The economics of Open Source work worst for commercial Linux distributions. They are attempting to generate profit from a product that they don't own, and to which they can't add much value without departing from the factors that make Linux desirable. This has forced even the best of them to depart from the ethos of Open Source with lock-in plays or pay-per-seat proprietary content. And the worst of them used to be called Caldera.

The Solution

Linux distribution works poorly as a profit-center. But today's business Linux users have an economic interest in funding the engineering of a GNU/Linux system on a cost-sharing basis. To this end, I am currently in negotiation with an industry group that proposes to fund between $1million or more annually to pay for the engineering of a fully supported and certified GNU/Linux system, without a per-seat fee, that meets the specific needs of their industry. That group represents approximately 50,000 desktop or server units - do the math and you'll see that they would save tremendously over Enterprise Linux. Obviously, there is room for the participation of many additional industry groups, companies, educational institutions, and individual developers.

To satisfy these users, we need to provide several factors that differentiate the commercial Linux distributions from the best of the non-commercial ones:

But we must do this without perpetuating the lock-in situations that exist today. To do this, I am proposing a new structure built upon existing components that I call UserLinux.

UserLinux would be a system for both desktop and server use in businesses of all sizes. The User in the name stands for the fact that the expense of producing UserLinux is supported by its users on an engineering and service fee basis, rather than through software sales. It is not an attempt to produce a "Linux for Grandma", but is usable by the millions of people with the technical skill to run GNU/Linux systems at home, many of whom do so to facilitate their business or employment. Of course we'll take advantage of improvements in user friendliness as they are developed, however it's not our intent to lead their development.

UserLinux is intended to be given away via free download and sold at retail in a user-friendly package. The data for the retail package design, manual, and CD will be made available for download, so that anyone can duplicate and sell the retail package.

Structure

At the core of UserLinux is a not-for-profit entity in charge of the Linux distribution, with engineering-by-meritocracy as in the Linux kernel. Surrounding that non-profit are for-profit companies that are in the business of providing service and engineering for the UserLinux distribution. Some of them may specialize in providing service to a particular vertical market or geographical region, others might be more general. Many of these companies already exist and have developed their customer base, and would be adding UserLinux to their existing services. Each of those businesses would reside upon a level playing field, with the same access to the software as any of the others. The non-profit core would be the entity that would be branded and certified as an enterprise-quality system. Customers would have their choice of service company, and would be able to select from multiple, competing vendors who service and improve the same system.

There will be a service organization built on top of all of the individual service companies, so that customers can have a "large entity" to call that can provide service anywhere, but customers will also be able to go directly to individual service vendors.

Trust in a brand will come easier when the brand is one developed for an industry, by that industry.

Companies that participate in UserLinux will be able to drive vendor support through their existing relationships with their vendors.

The service companies and their customers will drive development of the umbrella service organization and certification by standards organizations.

This structure establishes economic limits on the service companies. While they can make a living, they can't make a killing. These businesses would be "body-shops", which means they would scale linearly with the addition of engineering talent - each body brings in n additional dollars of profit. Venture capitalists aren't very interested in funding businesses that scale linearly, but fortunately the cost-of-entry for these businesses will be low. Many consulting firms are quite profitable, even if they are limited by the body-shop equation. A well-known Linux distribution company that I consulted tells me that they are already doing 70% of their business as service, and will be happy to live in the structure I have outlined.

One of the areas in which UserLinux could probably differentiate is the ability to actually provide service to a large number of customers. Red Hat boasts that it employs 300 engineers, but few of those engineers are in customer-contact positions. Their support organization is surprisingly small. Our multi-company effort has the potential to be able to offer more service, even by simple metrics like head-count, reasonably early in its existence. It can provide better-localized service because of the potential for involvement by service companies in many regions. And we can provide better quality, and lower-cost service, due to the fact that our service providers will compete with each other for business.

There are examples of the sort of structure I propose for UserLinux already existing, if on a smaller scale: Apache is a non-profit software core with a number of for-profit companies that service and engineer it, including HP and IBM. The same could be said for the Linux kernel.

Prospective Members

This project will be of interest to many companies that would like an inexpensive and reliable GNU/Linux system, with payment for service and engineering rather than "seats".

I don't expect the large computer vendors to come in on this project right away. My experience with the Desktop Linux Consortium was that IBM held back to see if the project was "real", and agreed to have their speaker present at our conference very late in the process.

I can't mention some of the companies who have approached me, either because they might not want to be publicly associated with this proposal, or because I'm in a contract negotiation with them.

Conectiva and Voxel have shown interest. Mandrake sent an inquiry and we don't yet know how they'd fit. I discussed the project with Nat Friedman of Novell.

There are a number of Debian-derivative distributions that are naturals for this project. Notable are Skolelinux ("School Linux"), a project supported by governments and educational institutions of several European nations, the non-commercial projects Knoppix and Morphix, and the commercial Debian derivatives Progeny, Xandros, Libranet, and perhaps Lindows.

Business Issues

The Service Provider Organization and Marketing Programs

There are several business issues that we'll have to address collectively, for example building our brand and cultivating ISVs. These tasks take money, thus I propose a membership organization for the service providers (the "service provider organization"), that would grant them "official" status and referrals from our global service phone number in exchange for their meeting our technical standards and making a financial contribution.  Financial contributions would be on a sliding scale based on the size of the company, and would be in two forms: a straight membership fee, and a percentage of  new business referred by the service provider organization. These contributions would be used for marketing programs, including building the UserLinux brand, and cultivating ISVs.

It's important to build a balance into the service provider organization, so that the service providers would not be able to dictate the project's technical direction to their own benefit - we'd prefer not to hear "Don't do a desktop system, my company is doing that for a fee". Thus, technical direction of the project must remain a pure meritocracy, not the slave of business management, while service providers would have more of a say over marketing programs.

Building the UserLinux Brand

Brand-building will be necessary if we want business people to trust us. This requires a good logo and other product design, advertising, a presence at trade shows, public relations and publicity, a well-done web site, etc. These things will be paid for by the service provider organization.

Cultivating Software ISVs

There is a perception in the business world that there are only two "real" brands of Linux: Red Hat and SuSE. Much of this perception is due to the cultivation of software ISVs, particularly Oracle, by those two brands. It is in the interest of ISVs to commoditize everything except the part of a solution that their own business sells, and an involvement with UserLinux is consistent with this goal. ISVs are not generally wed to one platform, although they would prefer not to support a hundred of them. ISVs will generally go where their customers ask, and the best way to reach them is through their customers. But ISVs are used to being catered to by the platform providers. They generally want hand-holding while porting their products, they want the assurance that they will be supported, and they don't want to pay for any of this. Thus it will probably be necessary for us to arrange to have a porting lab for the use of ISVs, where they could come to do their work with the support of an expert in our system, and for them to have free call-in support on issues related to supporting their products on our platform. These things would be paid for by the service provider organization.

Cultivating Hardware IHVs

Unlike software ISVs, hardware vendors generally pay their own engineering costs. Several factors attract an IHV to a operating system platform: the ecology of solution providers around that platform, the customer demand for that platform, and the cost of providing that platform on their hardware. We can offer essentially zero cost per copy of our platform to the hardware vendor, which is what other platform vendors should be offering. Our desirability to them is thus dependent upon our success in gaining software ISVs, and the extent to which the customer requests our platform.

User Feedback

We need to listen to users. Thus, we'd operate a suggestions mailing list, monitor our peer-support lists (which are operated by users) and operate focus groups among our customer groups.

Hardware Compatibility List and Certification Program

The project would maintain a hardware compatibility list naming devices that were compatible with the free core. Individual service providers can support proprietary-driver-only hardware if their customers require it. We can also maintain a branding program for certified-compatible hardware, this should set some standards regarding the posting of programming information, etc.

Support Lifetime

Service providers should commit to a support lifetime for releases. 5 years is suggested, as the number offered by MS and Red Hat and the planning horizon for upgrades of many IS managers. A minimum of 3 is probably necessary. A price premium for support upon releases more than a year out of date might be a good mechanism to motivate the customer to upgrade.

Technical Plan

This is an early pass at rough technical goals. Obviously, the technical plan will become more detailed as we talk with prospective service providers and customers.

Make it Free Software

The core UserLinux system should be 100% Free Software. The service providers will provide proprietary software according to the demand of their particular customers.

Build on Debian Base

I propose to work with the Debian distribution, integrating our changes directly into Debian, rather than creating a separate distribution.

Debian could be the world's largest Free Software project by many metrics. It boasts over 1000 developers all over the world, and 12884 Free Software packages in the official version of their system. The project has responded to over 200,000 bug reports, and keeps its bug database accessible to the public. Debian's dependency system works properly for all of these packages. Dependencies are resolved, and their packages installed, automatically, avoiding the most oft-voiced complaint of Red Hat users. Debian had an automatic, network package feed years before "Red Hat Network", and they have maintained it as a free service to this day. Debian has thought through the entire distribution process over its 10-year existence, resulting in hundreds of pages of developer policy documentation. The recent Debian security compromise was reported and solved in an open and timely manner that has never been duplicated by a commercial Linux distribution. And most important: Debian has a fair and democratic structure, an equal partnership between all participants, and a legal non-profit organization.

The goals of UserLinux are compatible with Debian's Social Contract, which I created.

The Debian group will admit any developer to their "core team" after a security check, and will allow that developer to add packages to the system as long as they follow the distribution's policies. Working with Debian, rather than creating a new distribution from the ground up, will be much better accepted by the Linux and Open Source developer community, whose cooperation we need.

I am a past Debian project leader, currently an elected director on the organization's non-profit board, and Debian's representative to W3C and other organizations. I understand the Debian organization, and can guide a fruitful collaboration with them.

A Netcraft survey reports that Debian is the number two Linux distribution found on web servers, beating all distributions but Red Hat. A European Union Government-sponsored survey names Debian as the leading distribution used by Linux and Open Source developers (rather than end-users). In the 2003 D.H. Brown Linux Function Review, Debian came in behind Red Hat and SuSE only in categories where support from proprietary application vendors was important, and ahead of them in other categories. Debian includes more software packages in their system than Red Hat or any for-profit distribution. They release for more architectures (11 architectures are officially supported) than any other Linux distribution, and their quality is generally held to be better than the commercial Linux distributions.

A number of people have suggested that we base the system upon Red Hat 9 so that we can ride upon Red Hat's branding success. Those people underestimate the cost of maintaining a Linux distribution. Debian has already mobilized 1000 people to carry out this task, and has worked out the problems of such a project over a 10-year period. Debian's participation in this project is critical. With them, we are faced with the formidable tasks of mobilizing a global service organization and helping Debian make the relatively small changes necessary to make their distribution more acceptable to enterprise users. Without them, we need to do all of that as well as build an entire (paid) staff to maintain a Linux distribution, and take care of all of the mundane details of the distribution instead of working on the things that directly interest our customers. A top-flight distribution is not maintained by less than two hundred people. Only the Debian organization has succeeded in coming close to the quality and the huge volunteer corps we need.

There have been suggestions regarding Linux platforms other than Red Hat and Debian, which I have classified as partisan.

Play Favorites

I think it makes sense for an enterprise project to make choices among the two complete GUIs available on Debian, the dozen web servers, and so on. Having a bounded set of packages to collectively support and improve is important, especially at the beginning. Additions to that list can be driven by what customers are willing to pay for. Expect the initial choosing to be painful. Of course any of the service providers can make their own support choices from the full set of software in Debian, for their own paying customers, overriding our choices.

It would be desirable to provide a platform that customers and ISVs can develop (and distribute) for without any additional licenses. If we were to go about this, it would require that we use the alternative (non-GPL) libraries to provide access to the MySQL server, and it would make the KDE vs. GNOME decision for us (Qt requires a developer royalty).

Even if we decide not to go with KDE (or GNOME) there would probably be a sufficient amount of people interested in maintaining an alternative GUI on the platform, and in Debian as they do today, including support companies whose customers have already made their GUI decision.

Remember, whatever choices we make apply only to what we choose to support as a group. Our choices don't cause the alternatives to be removed from Debian, they don't constrain what a service provider can support to their own customers, they don't discourage service providers from forming their own sub-group of the project dedicated to maintaining one of the alternatives that wasn't chosen.

I am anticipating being flamed to the ground for some of these choices, it's part of the job. Here's my prototype list of selections:
GUI desktop: GNOME. The fact that it comes with a license to develop and distribute proprietary applications is the sole reason for this decision. A long discussion on the mailing list has made it clear that GNOME and KDE are similar in technical merit and commercial acceptance at this time, leaving only the licensing issue as a basis for this decision. If you like KDE, it's still there in Debian, please go ahead and support it, and form a sub-group with everyone else who wants to. However, I think that it is better for the overall group to make a decision than to support two GUIs.

Database: MySQL with non-GPL libraries. This departs from the standard MySQL release in order to facilitate commercial development and distribution without forcing the user to purchase a MySQL developer license. Pay MySQL for training and service. I am not a database pro, and will listen to argument about the merits of other database servers.

Web Server: Apache 2. Apache 1 is more popular, probably because it has more working plug-ins at the moment. I think the plug-in issue will be resolved reasonably soon.

Mail transfer agent: Postfix. Best balance of performance, simplicity, security, acceptable licensing.

Interpretive language: Python. I sob at this not being Ruby, as I admire its refinement over Python, but Ruby has not managed to develop a large community outside of Japan and thus there is a lack of good library code - at least with documentation many of us can read. No doubt I will get flack from Perl partisans.

Java-like environment: It sounds as if GCJ/Classpath is in the lead among free Java-like implementations, but is not up to Java 2.0 and even misses the Java 1.3 standard. There is an implementation of Swing, and Eclipse can now be built . Some of the service providers will need to provide a Sun-derived JDK as an option.

Print spooler: CUPS. It's where all of the development is going, and seems to work reasonably well.

Web browser: Mozilla. I am not betting on Thunderbird attaining maturity in time for us, and on the various GNOME-and-Gecko-based browsers having working JavaScript and Java and all of the plugins and configurabilty of Mozilla on our time-line. But we will have to deal with Mozilla's using its own widget set and thus having consistency issues vs. the rest of the GUI.

Office Suite: OpenOffice for Word, Presenter. Is GNUMeric better for the spreadsheet? OpenOffice will have GUI consistency issues, as it, too, uses its own widget set.

I am interested in argument about choices for facilities not mentioned here, especially remote management and backup. I am interested in seeing the GUI argument end, as I've just read all of the postings in it and didn't learn much during those several hours.

Drive The Certification Process, Support Software Vendors

Debian is currently coming into compliance with the Linux Standard Base, we will help drive certification to that standard. We will use the Linux Professional Institute certification program to certify staff for our platform. We will collaborate on vendor certification driven by enterprise users of our system, using their existing vendor relationships. We won't approach Common Criteria until we're big enough. We can, however, do OpenCOE, especially since the COE folks had to wedge apt into Red Hat to get it to work to their satisfaction. And we have a FIPS 140 certification for OpenSSL that applies to Debian.

Many businesses now insist on ISO 9000 or 9001 certification. Some businesses with European Government contracts are required to use it. ISO 9001 is a subset of ISO 9000 that requires documentation of the quality assurance process, and is not so onerous as it sounds. It would be desirable to drive ISO 9001 certification of Debian's bug reporting and response process, and the service process used by our service providers.

Common Criteria is a security certification driven by 18 national governments. It's big. Most likely we won't be allowed to leverage upon the Common Criteria certification that Red Hat is currently going through with funding from IBM and others. This should probably be left until the organization is of sufficient size to tackle it.

Provide Standard Configurations

Every business has its own preferred software configuration, so the configurations we generate must allow further modification by the business (or one of us, working for the business). We will produce two configurations that can be installed non-interactively: one for the workstation, and one for the server. A configuration is a list of software package selections and a list of configuration database entries, these are used to generate a disk image for the installer.

Provide Automatic First-Time Installation

Debian currently supports individual installation by CD or network, with System Imager and FAI for automated installation. Debian needs a lot of help with installation, for both individual systems and clusters. It must support a fully automatic option for typical configurations.

Provide Cluster Management Facilities

There are a number of remote-management and cluster-management products in Open Source such as Ganglia, SIS, C3. We  would select some of these tools for improvement and integration.

Provide Retail Package

The UserLinux retail package would be patterned upon the Official Debian CD process that I designed years ago. Debian doesn't manufacture CDs. They just produce the ISO image files used by CD manufacturers. The ISO files are downloaded by CD manufacturers for manufacturing. If the manufacturers use Debian's ISO files without modification, and provide the source-code CDs as well as the binary ones, they are allowed to call the product "Official Debian". There are hundreds of CD vendors all over the world selling Official Debian CDs for just a few dollars, and the Debian project is spared the work of manufacturing CDs and licking stamps.

For UserLinux, this process would be extended. Besides ISO images, we would produce a manual, CD labels and product packaging - both economical CD sleeve packaging and a box with book-and-CD design. We could leverage upon one of the Open Source Debian books to produce a UserLinux book that could be printed by CD vendors without a royalty or fee. We would use a trademark to control how duplicators brand this material, as Debian does with "Official Debian".

My publisher, Prentice Hall PTR, could be interested in producing additional UserLinux material as part of my book series. All books in the Bruce Perens' Open Source Series are published under an Open Source license.

Alternative Proposals

Red Hat's Proposal

Red Hat has already proposed an answer to this problem, but I think it's the wrong answer. Their Fedora project is obviously intended to look like Debian. But unlike Debian, Fedora is an extremely unequal partnership. "Fedora" is where the community developers are supposed to build Red Hat's product, while the certifications and vendor endorsements are held back for the high-priced "Red Hat Enterprise Linux" brand. This is especially obvious in recent certification announcements: the Common Criteria certification will go to "Red Hat Enterprise Linux", not "Fedora". And of course the entire steering board of the Fedora project are Red Hat employees. Red Hat recently announced a second draft of the leadership structure for Fedora, in which they have eliminated voting, expressing the need to keep control in the hands of Red Hat's management.

But the most ludicrous aspect of the Fedora project is that with Fedora, Red Hat seeks to achieve what Debian did long ago. Because they can't (and shouldn't) control Debian, they decided to re-invent the wheel. It would take them years to achieve a fraction of what Debian already has.

But let's not lose sight of the fact that Red Hat has helped us tremendously. They contributed a lot of Free Software development, and they have helped gain popularity for our software. Unfortunately, they are now taking Enterprise Linux in an unhealthy direction. We should not stand by while that happens, but fortunately, we don't need to fight them. We need only provide a good alternative. We, the developers and customers, should choose where we think our resources will help the most, and what paradigm of Linux distribution is in our own long-term best interests.

How much would I advise a volunteer, community developer to contribute to Fedora? I think it makes sense to make Fedora packages for your own software - that way, you have some assurance that it will be packaged correctly. Beyond that, because of the vastly unequal partnership, I fear that a volunteer developer would be making himself an unpaid employee of Red Hat rather than a member of a real community. I guess that's OK if you think they'll hire you eventually. But I'd advise volunteer developers to concentrate their work on projects where the partnerships are demonstrably equal.

For the business participating in Fedora, the equation is even worse. I know of one business that invested millions into developing the IA-64 Linux system, with a marked absence of help from Red Hat. Now, that business is forced to buy their own software back from Red Hat at a high per-unit price, to package with their own products.

United Linux

United Linux might have been considered to be another proposal to arrive at a similar structure to UserLinux. But it ended up being a competitive play against Red Hat by only three distributions, and was never able to integrate the important non-commercial distributions like Debian into their project. The project ended up being poisoned by the fact that SCO was one of the three members. Given SCO's habit of suing its partners, the work of UnitedLinux ends up being a liability for the other two.

Linux Standard Base

I originally proposed the Linux Standard Base as a binary base system that would be shared and collectively maintained by all Linux distributions. This proposal echoes some of what was discussed back then. Because of the protests of others, LSB ended up being a paper standard rather than a collection of binary packages. Although the Free Standards Group has created many important standards, application vendors persist in certifying their applications for specific commercial Linux distributions, rather than for Linux Standard Base compliant systems. The reason is simple: the larger commercial distributions didn't want to promote LSB instead of their own brand. Hopefully the UserLinux structure can be more effective in promoting the importance of LSB.

The Name

I'm not stuck on the UserLinux name, and would listen to alternatives - but please try not to flood the mailing list with them, as we have more important tasks to discuss. Also, please do a search at www.uspto.gov for a trademark on the name you propose in a computer-related category before you propose one. I've seen a few dozen already-trademarked names so far.

I proposed gnUserLinux, but RMS didn't like it! He feels that having the GNU up front would signify that it's an FSF official project. UserGNULinux doesn't roll off of the tongue quite as easily. 

Credits

I first discussed this project at a Denver LUG meeting in September. Thanks to the Denver LUG, and to IBM, who paid my speaker's fee to appear at their Denver sales meeting. Joe "Zonker" Brockmeier reported that meeting in LWN, which resulted in the industry group contacting me.

In October, The Desktop Linux Consortium graciously allowed me to discuss this project during my opening keynote at their Desktop Linux Conference in Boston. That was widely reported in the press.

I have pasted some text from Zenaan Harkness' recent exposition on this subject at http://debian-enterprise.org/ .

I have not mentioned some of the people I've talked with, as they might not wish to be publicly associated with my proposal. Tell me if you feel you've been left out.

This version incorporates suggestions from the various subscribers to the discuss@lists.userlinux.com mailing list, and emails sent to me.