Most parts of the operating system (from low kernel, through to libraries, all the way up to X, and then even to applications) use fairly obvious interface layers, where the "communication protocols" or "argument passing" mechanisms (ie. APIs) can be understood by any developer who takes the time to read the free code. Device drivers pose an additional and significant challenge though: because many vendors refuse to document the exact behavior of their devices. The devices are black boxes. And often they are surprisingly weird, or even buggy.

When vendor documentation does not exist, the development process can become extremely hairy. Groups of developers have found themselves focused for months at a time, figuring out the most simple steps, simply because the hardware is a complete mystery. Access to documentation can ease these difficulties rapidly. However, getting access to the chip documentation from vendors is ... almost always a negotiation. If we had open access to documentation, anyone would be able to see how simple all these devices actually are, and device driver development would flourish (and not just in OpenBSD, either).

When we proceed into negotiations with vendors, asking for documentation, our position is often weak. One would assume that the modern market is fair, and that selling chips would be the primary focus of these vendors. But unfortunately a number of behemoth software vendors have spent the last 10 or 20 years building political hurdles against the smaller players.

A particularly nasty player in this regard has been the Linux vendors and some Linux developers, who have played along with an American corporate model of requiring NDAs for chip documentation. This has effectively put Linux into the club with Microsoft, but has left all the other operating system communities -- and their developers -- with much less available clout for requesting documentation. In a more fair world, the Linux vendors would work with us, and the device driver support in all free operating systems would be fantastic by now.

• Blobs can be 'de-supported' by vendors
  at any time.
  
  
• Blobs cannot be supported by developers.
  
  
• Blobs cannot be fixed by developers.
  
  
• Blobs cannot be improved.
  
  
• Blobs cannot be audited.
  
  
• Blobs are specific to an architecture, thus
  less portable.
  
  
• Blobs are quite often massively bloated.
  

Take Adaptec for instance. Before the 3.7 release we disabled support for the aac(4) Adaptec RAID driver because negotiations with the Adaptec had failed. They refused to give us documentation. Without documentation, support for their controller had always been poor. The driver had bugs (which affected some users more than others) which caused crashes, and of course there was no RAID management support. Apparently most of these bugs are because the Adaptec controllers have numerous buggy firmware issues which require careful workarounds; without documentation we cannot solve these issues.

The driver was written by an OpenBSD developer, who cribbed parts of it from a FreeBSD driver written by an ex-Adaptec employee. But no public documentation exists, and Adaptec has dozens of cards with different firmware issues. All of this adds up to a very desperate development model -- it becomes very hard for the principle of "quality" to show its head.

RAID devices have two main qualities that people buy them for:

• Redundancy
• Repair

Some vendors (or like the above Adaptec case, ex-employee) have sometimes given us some documentation so that we could write drivers, so that their devices could support Redundancy. But these vendors have never given us any documentation for performing Repairs.

Instead these vendors have tried to pass out non-free RAID management tools. These are typically gigantic Linux binaries, or some crazy thing, that is supposed to work through a bizarre interface in the device driver, which we are apparently supposed to write code for without any documentation.

And since we refuse to accept our users being forced into depending on vendor binaries, we have reverse engineered the management interface for the AMI controllers.

There is no great "intellectual property" in this stuff, it is all rather simple primitives. This is all that we need to implement basic RAID management:

• SCSI transactions on the back-side busses
• Discovering which drives are in which volumes
• Being able to silence the buzzer
• Marking a new drive as a Hot-Spare

The AMI driver needed to support these small primitive operations. And once we had that, we rely on something else which we know: Almost all the RAID controllers would need the same primitives.

Thus armed, we were able to write a generic framework which would later work on other vendors' RAID cards, that is, once we get documentation or do some reverse engineering for their products.

But having been ignored for so long by these vendors, it is not clear when (if ever) we will get around to writing that support for Adaptec RAID controllers now. And Adaptec has gone and bought ICP Vortex, which may mean we can never get documentation for the gdt(4) controllers. The "Open Source Friendly liar" IBM owns Mylex, and Mylex has told us we would not get documentation, either. 3Ware has lied to us and our users so many times they make politicians look saintly.

Until other vendors give us documentation, if you want reliable RAID in OpenBSD, please buy LSI/AMI RAID cards. And everything will just work.

While not exactly bait-and-switch, this is something which has been causing the community continual grief, and therefore we decided to honour a few of the projects that have decided to go non-free. After all.. having gone non-free, no one is going to remember them in the end.

This song is dedicated to a few worthy groups who have made this Free-to-Non-Free transition with their offerings in the last few years:

• David Dawes worked for years with a team of developers to make a free X11 distribution for us to use, called XFree86, 98% of which was based on entirely free code from MIT. Suddenly, one day, he decided that we must give him more credit (ie. advertise his name) or stop using it. Within about 4 months every project had told him to get stuffed, and the community has created a replacement effort. Now his team cannot even keep their web pages up to date...

• OpenBSD was the first operating system to integrate a packet filter, and it was the ipf codebase from Darren Reed that we chose. But a few years later he told us that we were not free to make changes to the code. So we deleted ipf, and our new packet filter far exceeds the capabilities of the one he wrote. And other projects are switching too...

• The Apache group started from the humble beginnings of just being 'a patchy' set of changes to a completely free web server of dubious quality. But the years have changed them, and what they supply is now quite non-free... released under a license so entangled in legalese that we have absolutely no doubt that there are encumbrances hidden within. Legal terms protect. Who are they protecting? Not your freedom.

We've been working a few years now on our packet filtering software pf(4) and it became time to add failover. We want to be able to set up pf firewalls side by side, and exchange the stateful information between them, so that in case of failure another could take over 'keep state' sessions. Our pfsync(4) protocol solves this problem. However, on both sides of the firewall, it is also necessary to have all the regular hosts not see a network failure. The only reliable way to do this is for both firewall machines to have and use the same IP and MAC addresses. But the only real way to do that is to use multicast protocols.

The IETF community proposed work in this direction in the late 90's, however in 1997 Cisco informed them that they believed some of Cisco's patents covered the proposed IETF VRRP (Virtual Router Redundancy Protocol); on March 20, 1998 they went further and specifically named their HSRP "Hot Standby Router Protocol" patent. Reputedly, they were upset that IETF had not simply adopted the flawed HSRP protocol as the standard solution for this problem. Despite this legal pressure, the IETF community forged ahead and published VRRP as a standard even though there was a patent in the space. Why? There was much deliberation at all levels of the IETF, and unfortunately for all of us the politicians within eventually decided to allow patented technology in standards -- as long as the patented technology is licensed under RAND (Reasonable And Non Discriminatory) terms. As free software programmers, we therefore find ourselves in the position that these RAND standards must not be implemented by us, and we must deviate from the standard. We find all this rather Unreasonable and Discriminatory and we *will* design competing protocols. Some standards organization, eh?

Due to some HSRP flaws fixed by VRRP and for compatibility with the (HSRP-licensed) VRRP implementations of their competitors, Cisco in recent times has largely abandoned HSRP and now relies on VRRP instead -- a protocol designed for and by the community, but for which they claim patent rights.

On August 7 2002, after many communications, Robert Barr (Cisco's lawyer) firmly informed the OpenBSD community that Cisco would defend its patents for VRRP implementations -- meaning basically that it was impossible for a free software group to produce a truly free implementation of the IETF standard protocol. Perhaps this is because Cisco and Alcatel are currently engaged in a pair of patent lawsuits; a small piece of which is Cisco attempting to use the HSRP patent against Alcatel for their use of VRRP. Some IETF working group members took note of our complaints, however an attempt in April 2003 to have the IETF abandon the use of patented technology failed to "reach consensus" in the IETF.

A few years ago, the W3C, who designs our web protocols, tried to move to a RAND policy as well (primarily because of pressure from Microsoft and Apple), but the community outrage was so overpowering that they backed down. Some standards groups use this policy, while others avoid it -- the one differentiation being the amount of corporate participation. In the IETF, the pro-RAND agents work for AT&T, Alcatel, IBM, Cisco, Microsoft, and other large companies. Since IETF is an open forum, they can blend in as the populace, and vote just like all others, except against the community.

Translation: In failing to "reach consensus", the companies who benefit from RAND won, and the community lost again.

Left with little choice, we proceeded to reinvent the wheel or, more correctly, abandon the wheel entirely and go for a "hovercraft". We designed CARP (Common Address Redundancy Protocol) to solve the same problem that these other protocols are designed for, but without the same technological basis as HSRP and VRRP. We read the patent document carefully and ensured that CARP was fundamentally different. We also avoided many of the flaws in HSRP and VRRP (such as an inherent lack of security). And since we are OpenBSD developers, we designed it to use cryptography.

The combination of pf(4), pfsync(4), and carp(4) has permitted us to build highly redundant firewalls. To date, we have built a few networks that include as many as 4 firewalls, all running random reboot cycles. As long as one firewall is alive in a group, traffic through them moves smoothly and correctly for all of our packet filter functionality. Cisco's low end products are unable to do this reliably, and if they have high end products which can do this, you most certainly cannot afford them.

As a final note of course, when we petitioned IANA, the IETF body regulating "official" internet protocol numbers, to give us numbers for CARP and pfsync our request was denied. Apparently we had failed to go through an official standards organization. Consequently we were forced to choose a protocol number which would not conflict with anything else of value, and decided to place CARP at IP protocol 112. We also placed pfsync at an open and unused number. We informed IANA of these decisions, but they declined to reply.

As we did for the 3.3 release, we have once again tried making release artwork and music which are allegorical of recent happenings.

Two years ago we became involved with the University of Pennsylvania and DARPA, who were funding us to do security research and development .. on things that we were already intending to do. We provided ideas, wrote papers, and deployed cutting-edge technology; DARPA provided finances and reaped a share of the credit, and the University of Pennsylvania acted as a middle-man. We accepted funding based on the promise that our freedom to operate as we wished was unaffected. To us, freedom is more important than funding -- heck, we were dealing with the evil forces of government, and needed to be careful.

A few months prior to this release, DARPA suddenly and without warning decided to withdraw that funding; they also aggressively backed out of contractual obligations. Many articles in the press followed regarding this sudden maneuver. Apparently this hoopla happened because an OpenBSD-related article in the Canadian newspaper The Globe & Mail had quoted Theo de Raadt making anti-war statements regarding Iraq and the theft of oil.

The only answer given (to major media reporters) by a DARPA spokesperson (Jan Walker) was this:

"As a result of the DARPA review of the project, and due to world events and the evolving threat posed by increasingly capable nation-states, the Government on April 21 advised the University to suspend work on the "security fest" portion of the project."

That almost toes the line of calling us terrorists! We had lost financial support, but the release of the statement above suddenly made us very happy to be free of any perceived obligation to such crazy people.

Since the termination came near natural contract termination (about 4 months remained), less damage than expected was sustained by the project. Sponsors stepped forward and helped us make up the missing funds we needed to run our "Hackathon", and the event proceeded as planned. We even had T-shirts made with "Workstations of Mass Development" artwork for those developers who attended (sorry, they are not for sale).