by Dan Kohn, Linux Foundation, June 2008
Linux today supports more hardware devices than any other operating system in the history of the world. It does this using a development model significantly different from the familiar Windows device driver model. The Linux development process continues to evolve to better support the needs of Independent Hardware Vendors (IHVs), distributions, and other members of the community, and the advantages of the Linux model are increasing with time. While Linux will not provide a stable source or binary interface for driver developers, IHVs should familiarize themselves with a number of useful projects, many sponsored by the Linux Foundation, that ease driver development, including the Hardware NDA program, the Linux Drivers Project, and the Driver Backport Workgroup. When IHVs engage with the Linux community, they almost invariably find that the Linux driver model provides significant benefits that lower their costs while producing better drivers.
A fundamental purpose for operating systems (OSes) is to serve as an abstraction layer between applications and hardware to enable interoperability. An IHV wants their hardware to be able to make use of all the relevant features of an OS, and an OS wants to take full advantage of the hardware it's running on. Since both the OS and the hardware tend to add and to rearrange features over time, it is a dynamic interaction. What everybody wants is for the hardware to “Just Work” without hassles or support calls.
Today, Linux works with more devices than any other OS in the history of the world. However, this is no comfort when there's no support for the device you need to use. In the past, Windows has benefited from the fact that IHVs expend significantly more resources on their Windows drivers. A review of the strengths and weaknesses of the Windows and Linux driver models helps explain the relative situation today, and provides the context to understand how Linux device support is finally overtaking Windows in both quantity and quality of device support.
Many vendors are most familiar with the Windows model. The theory behind the Windows model is that Microsoft commits to a stable set of Application Binary Interface (ABI) calls that can be used by driver developers. These interfaces represent a set of services that the core OS makes available to device drivers. IHVs test and certify their binary drivers against the released binary OS. Microsoft promises to provide the same binary interfaces over time so that IHVs can rely on those interfaces.
The reasoning behind this approach is to enable driver development to be decoupled from the OS development. The idea is that Microsoft controls their closed source OS and the IHV develops their closed source driver, with the stable ABI being the only interface between the two. This also means that drivers are external, second class citizens compared to the core operating system. Driver developers do not have access or visibility into the core OS beyond this ABI, although a buggy driver can still crash the entire OS.
IHVs do not make available the source code for their Windows drivers as the drivers are only released in binary form. IHVs can independently release and upgrade their drivers and have users install newer versions. This has trained users to expect to install a CD-ROM to get their new hardware working. (Since those drivers are frequently being updated by the IHV, savvy users today will ignore the CD-ROM and instead download the newest driver from the IHV website).
One downside of this approach is that a stable device driver ABI requires the OS to lock into obsolete and non-optimal interfaces in ways that make the OS less reliable and secure. As an example, both Windows and Linux needed to rewrite their internal USB implementations multiple times in order to take advantage of all of the capabilities of different devices. Linux was able to upgrade the interfaces and the drivers utilizing them simultaneously, and then to deprecate and remove the old interfaces.1 By contrast, Windows must continue to support the obsolete interfaces indefinitely, which has repeatedly caused development headaches and can lead to bugs and security holes.
In addition, binary drivers tend to be buggier, because bugs can generally only be identified and fixed by the IHV, not by the community of system vendors, developers, and users. Another key disadvantage of the closed nature of Windows drivers is that large amounts of important subsystem code is duplicated (in different forms) across different Windows drivers. This duplication is also due to the small number of people who have access to the code: because no one person can see all the code, it is very difficult to factor out and optimize common subsystem code.
But the biggest problem with the Windows model is that stable device driver ABIs do not actually remain stable. Microsoft has modified the Windows ABI in every Windows release, resulting in a relentless succession of hardware support issues. Any change in the ABI can cause hardware to stop working correctly, and can even crash the entire OS. Fixing the problems requires IHVs to rewrite to the newest ABI, but updated drivers can take years to be released. Some IHVs have decided not to support Vista drivers for older hardware at all, in order to force users into purchasing upgrades. And every new Windows release (even service packs) provides new opportunities for driver failures.
As PC World recently pointed out: “Microsoft's debacle with Windows Vista device drivers malfunctioning after an upgrade to Service Pack 1 is an expected, almost inevitable result of the strategic path it took with Windows' initial release more than two decades ago.”2
The Linux driver model is different. For users, the goal is to provide the “Just Works” experience. The Linux model is that IHVs get the source code for their driver accepted into the mainline kernel. This entails a public peer review process to ensure that the driver code is of sufficient quality and does not have obvious bugs or security risks.3 Linux has neither a stable binary driver ABI nor a stable source-code driver Application Programming Interface (API). That is, there is no guarantee that an interface provided in one version of the kernel will be available in the next version, and portions of the ABI and API change in every kernel release.
By contrast, the Linux kernel does provide a stable userspace interface for Linux applications. These applications essentially have a contract with the Linux kernel that the userspace binary interfaces they rely on will continue to work consistently over time. That's why a pre-compiled Linux application can run correctly on multiple distributions and multiple versions. The underlying implementation of the userspace binary interfaces can and does change, but even an application compiled for pre-1.0 Linux will run correctly on the latest kernel. This is the opposite of device drivers, which have no guarantee whatsoever that any interface they rely on, whether binary or source, will remain consistent between versions of the kernel.
Counterintuitive though it might be from a proprietary viewpoint, this lack of internal kernel interface stability is preferable because both the kernel code and all of the drivers relying on it are open source. In fact, driver code is an integral part of the Linux operating system, not a second-class add-on. Once a driver is accepted into the mainline kernel, it will be maintained over time as internal kernel interfaces change. That is, when a subsystem maintainer accepts a patch to make an incompatible change to a kernel interface, that patch will simultaneously upgrade every driver that relies on the interface. And, new drivers and any upgrades to them automatically flow downstream from the mainline kernel to all Linux distributions.
The key strength of this approach from the user's viewpoint is that, in happy contrast with proprietary operating systems like Windows Vista, once a device is working on a given version of Linux, that support continues through all future versions. (Devices are generally only removed when they have become so rare that no users can be found.) In Linux, hardware support only gets better; it never gets worse.
From the IHV's point of view, the big benefit is that an IHV's driver is maintained over time by the community, meaning that other people fix, tune, and add features to the driver. When internal kernel interfaces change in each new OS release, IHVs don't need to write and release a new driver; their driver is upgraded automatically. Obsolete interfaces can be deprecated and removed rather than being maintained indefinitely. Common subsystems can be factored out of drivers, enabling leaner, less buggy device drivers while adding more functionality for all hardware. This improves the stability, security, and maturity of both the OS and the driver.
In addition, the Linux model enables cross-architecture driver support nearly for free. Even when an IHV only tests their driver on one chip architecture, interested developers ensure that the driver works with every architecture that Linux supports, which is more than any other OS in history. The strength of this approach has been especially apparent over the last decade as many chip architectures have moved from the 32 to 64 bits. Nearly all Linux drivers were quickly updated to support these newer architectures, while driver support for 64-bit Windows Vista even on the highest volume x86 architecture remains extremely poor today.
The biggest hurdle of the Linux driver model for some IHVs is the need to open source their driver code, which a small (but thankfully dwindling) number have been reluctant to do. Also, once a driver is accepted into the mainline, it can take up to 18 months to be deployed into an enterprise distro. It has not until now been convenient to backport the driver to existing distros, but that is improving with the Driver Backport Workgroup.
Having hardware reliably supported by Linux means getting the driver accepted into the mainline kernel. Supporting an out-of-mainline open source driver creates significant, never-ending support costs for the IHV, as new versions constantly need to be released as the kernel API changes. Supporting an out-of-mainline binary driver means even bigger, never-ending support costs for the IHV, and directly contradicts the recent statement by a large number of kernel developers that binary drivers are “undesirable”4.
The Linux driver model is different from the Windows model many IHVs are used to. But it is a consistent and compelling approach, and has been successful at supporting nearly the entire universe of computer hardware. Moreover, the vast majority of all IHVs have adapted to Linux and have thriving businesses that work with the Linux driver development model. For the very small number of remaining IHVs that do not support mainline Linux drivers, it is time to embrace the Linux development model. Accommodating this Linux model means lower costs, more stable drivers, and brings the IHV into the mainstream of Linux development, the largest and most successful collaborative development project in history.
Several programs are helping to make the device development process easier for IHVs:
IHVs can use the Non-Disclosure Agreement (NDA) program to confidentially share specs and hardware with kernel developers. IHVs sign an NDA with the Linux Foundation (LF) covering LF contractors. With agreement from the IHV, one or more kernel developers sign contractor agreements with the LF to be covered under that NDA. The NDAs are available on the LF website.5 The LF is also working to establish liaison relationships with PCI and other consortia on behalf of the kernel community.
IHVs can get their driver written for free by volunteer kernel developers. The IHV provides a specification that describes how their device works, the email address of an engineer willing to answer questions, and (ideally) a few sample devices. An expert developer team of volunteers led by Greg Kroah-Hartman returns a complete and working Linux driver that is added to the mainline Linux kernel source tree.
The driver (like all Linux drivers) is automatically kept up to date and working through all Linux kernel API changes. The driver will work with all of the different CPU types supported by Linux, the largest number of CPU types supported by any operating system ever. At the IHV's option, the work can be done under LF NDA program. More info is on the Linux Driver Project website.6
There's a recent call from Linus Torvalds and the subsystem maintainers for IHVs to merge new drivers early, as long as the new driver:
The goal is to get hardware working ASAP and to improve collaborative development.7 In addition, the linux-staging tree was recently created specifically to hold new drivers that are not mature enough to be merged into the mainline kernel.8
As described above, drivers get developed, tested, and integrated into the mainline kernel. However, most users use consumer or enterprise distributions rather than the mainline kernel. When a new driver is accepted into the mainline, it can take up to 18 months for that driver to be shipped in an enterprise distro, and can be another year or two before it is purchased and installed. For users of existing distros, there needs to be a way to backport drivers from the mainline kernel to the kernel they are using.
Today, there are numerous, incompatible ways to backport driver code from the latest kernel to work in older distros. With active support from Novell, Red Hat, Ubuntu, IBM, Dell, and HP, the Linux Foundation has created a Driver Backport workgroup to standardize the process for distributing new driver code to older versions of the OS. The workgroup is providing a packaging solution so that older kernels in Linux distributions can make use of upstream driver code from the latest mainline kernel. The goal is to enable Linux distributions to automatically download and install the newest versions of relevant device drivers. More info is on the workgroup page.9
Membership in the Linux Foundation is not required for an IHV to support their devices on Linux. But membership ensures that your company's investment in Linux is protected from a legal, technical and promotional perspective. Membership provides access to events and summits where the brightest minds in the Linux ecosystem, both corporate and community, meet to collaborate. It enables the Linux Foundation to continue supporting Linus Torvalds and providing legal, financial, and logistic support for the rest of the Linux development community. If Linux is strategic to your business, you should be a member of the Linux Foundation.10
Linux has achieved an unprecedented level of hardware support. It has done so using a development model that is quite different from the established Windows model, but that offers significant benefits in stability, capability, and security. Like the rest of the Linux OS, the driver model leverages the strengths of open source development to provide better code that improves over time. With several new initiatives from the kernel community and the Linux Foundation, and through cooperation from the relevant IHVs, Linux should soon be able to fully support the small number of remaining devices.