Open Sources 2.0/Beyond Open Source: Collaboration and Community/Everything Is Known
In Gabriel Garcia Marquez's epic novel, One Hundred Years of Solitude, Aureliano Buendia spends his days and nights in an old alchemical laboratory, isolated from the rest of the world, his nose buried in books. While Aureliano mostly keeps to himself and says little, he occasionally surprises members of his family or his small group of friends with a fact or insight they would have expected from someone more worldly. When they ask him how he could possibly have known about something, he responds simply, "Everything is known."
It's a fascinating thesis, one that not only highlights our individual ignorance, but also suggests a different approach to pursuing knowledge. When we are faced with something that is new to us, we often forge ahead blindly, learning and creating as we go along. We usually call this process "innovation." Our use of this word suggests that we place high value on uniqueness, on things we think have not happened before. However, how often are we qualified to determine whether something is new? How much do most of us know about our past or about the past of others? More importantly, if something is profound and relevant and significant, who cares if it's new? It's far more interesting if it's been discovered repeatedly in many different contexts. We should take note of important new ideas, but we should cherish ideas that prove their importance over and over again.
Many people find the emergent, collaborative aspects of free and open source software fascinating and compelling. However, before we start labeling anything as "innovative," we ought to seek what is already known by examining emergent collaboration in other contexts and by identifying recurring patterns there. If we can find patterns that occur over and over again in all these contexts, we can apply this knowledge to other areas. Simply put, these patterns can improve our ability to work together regardless of the domain, a skill on which the future of our world depends.
This chapter examines two stories: the PACT compiler project in the mid-1950s—the first collaborative software effort to transcend organizational boundaries—and the recovery effort at the site of the World Trade Center following the tragedy on September 11, 2001. On the surface, these stories seem very different, but closer examination reveals several common patterns, patterns that are also found in successful free and open source software projects. These patterns suggest principles for facilitating emergent collaboration in many different contexts, from software development to grass-roots politics.
The PACT Project
If you examine open source projects closely, you will discover a deeply embedded culture of collaboration, a culture that is embodied in the community's tools and practices. Programmers gather in a variety of contexts, both physically (conferences, user group meetings, code sprints) and virtually (mailing lists, IRC channels). Their toolkits are stocked with software for sharing, documenting, and collectively authoring code. The very notion of reusable source code is an implicit invitation to collaborate. Most programmers take this culture for granted, but these tools and norms did not always exist. They can be traced as far back as 1954, to an early collaborative compiler project known as PACT.
The onset of the Cold War in the late 1940s created a burgeoning aerospace industry in Southern California, and the companies that emerged suffered from a common affliction: a lack of good programmers. Computers were barely a decade old at this point, and programming was difficult and expensive. The notion of sharing source code was laughable, partially because the notion of source code barely existed. There were no high-level programming languages. Most people wrote software in machine language, which meant that software written for one type of machine could not be transferred to another. Although some people wrote homegrown assemblers and interpreters to ease their programming burden, there were no standards, so code sharing was still impossible. Without high-level languages, programming was expensive. Without the ability to share code, collaborating on software projects outside of one's organization was technically infeasible.
To make matters worse, there were only about 1,000 programmers in the United States at the time (Campbell-Kelly, 193). As a result, companies regularly raided each other's talent, which led to greater secrecy within companies about their computer-related work and a general tightfistedness with their prized programmers. While the technical barriers to collaboration were large, the culture of secrecy was the biggest impediment. R. Blair Smith, an IBM salesman, observed this culture firsthand in the early 1950s. He watched each of his customers struggle with the same expense and difficulty of developing software, and he decided that the overlap in effort was silly. If these companies would just set aside their differences and work together to simplify programming for everyone, everyone would win. As obvious as this seemed to Smith, he was realistic. Collaboration made sense, but it wasn't going to happen unless the culture of secrecy shifted radically.
Smith decided to throw a party. He invited all of his customers to dinner on November 15, 1952, at the Santa Inez Inn in Santa Monica, California (Mapstone, 367). The evening did not begin well. His guests were not used to interacting with their competitors, and the tension was thick. Smith decided something drastic was in order, so he decided to do something that neither he nor any of his IBM colleagues had ever done before. He bought drinks for his guests, compliments of IBM. Several rounds later, the tension had been replaced by good spirits, and the party began in earnest.
Little was gained other than a new sense of camaraderie, consensus regarding a second gathering, and the largest expense sheet Smith ever submitted to IBM. Smith stopped picking up the dinner and drinks tabs, but the meetings continued. The group was expanded, and a name was chosen—the Digital Computers Association (DCA). Speakers were invited to discuss various technical topics, but the real value of the meetings stemmed from the camaraderie. The atmosphere was jovial and irreverent, and the predinner cocktails quickly became the most important aspect of the meeting. As with the first gathering, little of tangible value was accomplished. However, two important things did happen. First, the members made a tacit gentleman's agreement not to raid each other's programmers (Carlson, 66). Second, bringing people with similar interests from different companies together made them recognize the commonality of their computing problems and the possible benefits of intercompany collaboration.
The first to test the possibility of these benefits were two DCA stalwarts, Jack Strong and Frank Wagner, both from North American Aviation. Concerned as always with the high cost of programming and the performance overhead of interpretive systems, Strong and Wagner wanted to explore automatic programming as a way of addressing these issues. On November 16, 1954, Strong and Wagner organized a meeting held at Douglas Aircraft Company's El Segundo plant (Melahn, 267). With representatives from five different companies—North American Aviation, Douglas Aircraft Company, IBM, Ramo-Woolridge, and the RAND Corporation—in attendance, Strong and Wagner proposed a cooperative effort to develop an automatic coding system for the IBM 701. The group agreed to establish two committees to undertake the project, the Policy Committee and the Working Committee, with each participating company committing one full-time representative to the project. Calling themselves Project for the Advancement of Coding Techniques, or PACT, the group decided to meet and work at the RAND Corporation, which was considered neutral territory among these competing interests.
The working committee immediately ran into trouble with language and culture. Wesley Melahn, a participant from RAND, wrote, "People didn't really know their neighbors' systems well enough to be able to talk about them intelligently, much less to understand the subtle pressures that made small points seem important enough to argue about" (Melahn, 268). Paul Armer, another important contributor, added:
The members of the working committee of PACT spent several weeks in mutual education, because at first they had to overcome the "our way is best" attitude and also a serious language problem. That this mutual education led to mutual admiration and respect for other people's abilities is testified to by the final report of the PACT-I working committee. I quote from their Primary recommendation: "The spirit of cooperation between member organizations and their representatives during the formulating of PACT-I has been one of the most valuable resources to come from the project. It is essential that this spirit of cooperation continue with future project plans" (Armer, 125).
The collaboration ultimately resulted in the first software developed cooperatively by employees of several different companies: a working compiler that went through two versions, PACT-I and PACT-II. The group also delivered a series of papers at the 1955 Meeting of the Association of Computing Machinery (ACM) in Philadelphia. More importantly, PACT was a watershed. As a standard compiler that many companies used, it removed the technical barrier to collaborating on software projects. PACT also demonstrated the feasibility of a cooperative coding project and affirmed the value of cooperation, transforming the culture of business computing and creating a climate conducive to further collaboration.
Some of the similarities between PACT and today's free and open source projects are obvious. For example, neutral space is critical for encouraging emergent collaboration. R. Blair Smith's party would not have had the same effect had he worked for one of those aerospace firms rather than for IBM. Similarly, it was no accident that PACT participants chose to meet at RAND. Neutral space manifests itself in a number of ways in free and open source projects. When Linus Torvalds, the creator of the wildly successful Linux operating system, graduated from the University of Helsinki, he chose not to accept a job at one of the many emerging Linux distribution companies to maintain his neutrality. Although companies are not required to give up ownership of their code to make their software open source, many choose to transfer their copyright to neutral bodies, such as the Free Software Foundation and the Apache Software Foundation.
For collaboration to emerge, there must be a culture of collaboration. Prior to PACT, that culture did not exist. Fostering it required many gatherings and a large drink bill, courtesy of IBM. Those gatherings did not result in concrete deliverables, but they helped establish shared understanding and shared language. The culture of collaboration within successful free and open source projects is so deeply engrained among their participants, it's often taken for granted. Projects that simply release source code under an open source license in the absence of this culture usually fail.
The World Trade Center Recovery Effort
R. Blair Smith's dinner party in 1952 helped catalyze a shift in culture among the business computing community, but it still took two years before it manifested in a concrete project. The terrorist attacks on the morning of September 11, 2001 caused a far more immediate and visceral transformation. The collapse of the World Trade Center not only killed 3,000 people, it also brought fear and uncertainty to the entire United States. Nobody knew who was responsible, how it had happened, or worst of all, what to expect next. The only thing we knew was that there had been an attack and that our lives would never be the same.
That was all the residents of New York needed to know. As soon as the towers collapsed, those who were at the site began a concerted rescue and recovery effort. The impulse to help after such a horrific tragedy was not surprising, but the collaborative process that emerged from this initial impulse was remarkable. William Langewiesche, a correspondent for The Atlantic, was on hand from the beginning, and in his book, American Ground, he described the intense motivation shared among those who participated in the recovery:
Throughout the winter and into the spring the workers rarely forgot the original act of aggression, or the fact that nearly 3,000 people had died there, including the friends and relatives of some who were toiling in the debris. They were reminded of this constantly, not only by the frequent discovery of human remains, and the somber visits from grieving families, but also by the emotional response of America as a whole, and the powerful new iconography that was associated with the disaster—these New York firemen as tragic heroes, these skeletal walls, these smoking ruins as America's hallowed ground. Whether correctly or not, the workers believed that an important piece of history was playing out, and they wanted to participate in it—often fervently, and past the point of fatigue. From the start that was the norm (Langewiesche, 9).
The recovery effort consisted of two parts: recovering the remains of the dead, and restoring the site. This latter task was literally and figuratively enormous. Each tower had stood 110 stories high, almost a quarter of a mile each. The collapse had created a 1.5-million-ton labyrinth of steel and concrete over 17 acres, with hazards hidden everywhere. The six-story-deep foundation and the entire New York subway system was in danger of being flooded by the Hudson River, thanks to a damaged subway tube and a fragile slurry wall protecting the foundation. Most dangerous of all was the underground chiller plant, which had cooled all of the buildings and which the collapse had rendered inaccessible. The plant contained 168,000 gallons of freon, and nobody knew the status of that gas. If the freon containers had somehow survived the collapse and if the workers inadvertently damaged it in the recovery process, the gas would immediately escape, displacing the air and suffocating those working underground. Additionally, if that gas caught fire, it would become a mustard gas-like chemical that would pose a threat to the entire city. As if navigating through the wreckage carefully for their own safety and the safety of the city weren't enough, the workers also had to sift through it carefully for remains of the victims and for potential evidence that could shed light on the tragedy.
Thousands of volunteers had converged onto the scene immediately following the attacks, looking to help in whatever way they could. However, those volunteers could do little, and after the first few days, the vast majority of them were turned away. The only volunteers who were allowed to stay were members of the Red Cross and Salvation Army who fed the workers. The recovery effort required expertise and plenty of heavy machinery. There were national and city plans for responding to disasters of all sorts, including terrorism, that prescribed the procedures and hierarchies for requisitioning the necessary resources. However, those organizational charts were scrapped, and the plans were never executed. Langewiesche explained:
The problems that had to be solved were largely unprecedented. Action and invention were required on every level, often with no need or possibility of asking permission. As a result, within the vital new culture that grew up at the Trade Center site even the lowliest laborers and firemen were given power. Many of them rose to it, and some of them sank. Among those who gained the greatest influence were people without previous rank who discovered balance and ability within themselves, and who in turn were discovered by others (Langewiesche, 11).
Kenneth Holden and Michael Burton, top officials of New York's Department of Design and Construction (DDC), quickly emerged as the leaders of the recovery effort. The afternoon following the collapse, the two met and started enlisting resources within the DDC to help with the rescue effort. Langewiesche wrote:
Holden and Burton responded tactically, with no grand strategy in mind. At the police headquarters they discovered a telephone in a room off the temporary command center—a chaotic hall filled with officials struggling to get organized—and they began making calls. No one asked them to do this, or told them to stop. One of the deputy mayors there had formally been given the task of coordinating the construction response, but with little idea of how to proceed, he had so far done nothing at all. Holden and Burton themselves were operating blind, groping forward through the afternoon with only the vaguest concept of the realities on the ground. The DDC's previous experience with emergencies had been limited to a sinking EMS station in Brooklyn, caused by a water-main break, and a structural failure at Yankee Stadium, one week before baseball's opening day (Langewiesche, 88).
The fire and police departments and other government agencies were also frantically coordinating their own efforts, with little communication between each group. It was clear that these groups could not continue working in isolation, and that a collaborative process would be in order. However, the city decided to scrap its emergency response plan in favor of the process that was already emerging. At that point, the DDC had already bypassed the standard emergency response procedure, and Holden and Burton had already in many ways taken charge of the effort. The city made it official, giving the DDC, the fire department, and the police department joint control over the cleanup effort.
As with the PACT project, neutral territory played an important role in defusing traditional politics and allowing new roles to emerge. Burton commandeered a kindergarten classroom at a nearby school and held twice-daily meetings there. Because of the urgency of their task, they did not keep minutes or write memos. Those meetings were the primary nexus of communication, and everyone involved—about 20 government agencies and several contractors—had representatives there. Recording devices were banned to encourage participants to voice their opinions freely. According to Langewiesche, "Some of the participants were accomplished people with impressive resumes, but within the inner world of the Trade Center site it hardly mattered what they had done before. However temporarily, there was a new social contract here, which everyone seemed to understand. All that counted about anyone was what that person could provide now" (Langewiesche, 113).
The recovery effort was largely complete after six months, although it did not officially end until a few months later. The entire 1.5 million tons of wreckage had been properly examined and transported to a landfill on Staten Island. The site was no longer hazardous, and the city was ready to build something new. Incredibly, despite all of the dangers, not a single person died during the recovery process.
Comparing the World Trade Center recovery to free and open source projects may seem far-fetched at first, but there are important similarities. For instance, most of the recovery effort workers were financially compensated, although the plan for doing so was not predetermined. Once the recovery process became apparent, city officials made sure the appropriate people were paid. Similarly, writing software requires tremendous expertise, and for a project to be sustainable, it must be able to retain its experts. With free and open source projects, the methods of compensation are not always explicit, nor are they always monetary, but they are there.
The most evident pattern from the World Trade Center recovery was an orientation among participants toward action. In the free and open source software community, there is a common saying: "scratch your own itch." That attitude also pervaded the recovery effort. No one waited for instructions from above. People simply did what had to be done. At the same time, the recovery effort was not devoid of process. Not only were the participants action oriented, but they also were well coordinated, thanks to the twice-daily meetings and culture of openness. Similarly, successful free and open source projects employ effective and open lines of communication and coordination.
Facilitating Emergent Collaboration
The similarities among PACT, the World Trade Center recovery, and free and open source projects reveal several important patterns and lessons for facilitating emergent collaboration. I say facilitate, not create, because emergence implies an element of surprise and a lack of intention and control. Process and organization exist, but are not imposed. Patterns such as neutral territory create an environment that is permissive, not prescriptive, and participants who are action oriented thrive in these environments.
The defining characteristic of the World Trade Center recovery was that action trumped everything else. The problem was constantly changing, and thus, the process had to evolve as well. Asking for permission didn't make sense in this environment, because those on the ground understood the problem better than anyone holding traditional authority. City leaders recognized this and deliberately chose to rubber-stamp the process that emerged, instead of imposing a written plan that was clearly not suited to the problem. Similarly, PACT did not happen because the leaders of the different companies came together and decided to collaborate on a project. PACT happened because those who would most benefit from collaboration—the software developers—began to trust each other and identified a common need. These same developers agreed on a representative governance model with a consensus-based decision-making approach. While not as freewheeling as the Trade Center recovery process, it still emphasized exploration by all of its members, and its leaders were especially proactive. The PACT process had to be slower and more deliberative by design, because the motivation to collaborate was nascent. Participants had to see collaboration work before they became comfortable with it, which meant that the process evolved slowly.
Another prerequisite for emergent collaboration is an emphasis on open, effective communication. Effective communication begins with shared language. When the PACT project began, different participants had different understandings of the same words, and communication was rendered ineffective. Before the PACT participants could even start thinking about software, they had to understand each other's world views and language. Shared language was not an issue with the World Trade Center recovery. Everyone had a vivid picture of what had happened, and what needed to be done. However, while the recovery effort was highly individualistic, coordination was imperative, and hence, those twice-daily meetings among all participants were critical. Banning recording devices encouraged participants to express their feelings openly. Good communication is the hallmark of the best free and open source projects. Anyone may participant in a forum, and core participants respond to questions quickly. Active projects often summarize online discussion on a regular basis so that others can follow high-volume discussions without being overwhelmed. Most importantly, there is a shared language among those who participate, some of which is embodied in the various free and open source licenses.
All of these elements are necessary for collaboration to emerge, but they are not sufficient. Culture is critical. The strong desire to collaborate among those who participated in the World Trade Center recovery was a direct result of its unique circumstances and scale. With PACT, the incentives and environment were not powerful enough to establish a culture of collaboration, and rightfully so. You cannot reasonably expect a software project to inspire the same emotion as an act of terror and the tragic loss of life. Instead, the instigators of PACT had to nurture that culture every step of the way, beginning by bringing the groups together to break bread and to discover commonalities for themselves. The seedlings of community—camaraderie among those with shared interests and goals—catalyzed a culture of collaboration, and every instance of successful collaboration further reinforced that culture.
We have limited control over emergent collaboration, and we must adjust our expectations accordingly. However, the patterns we discover by examining stories such as PACT and the World Trade Center recovery help us understand how we can facilitate emergent collaboration. These stories were very different, yet their commonalities are striking, especially in light of what we already know about free and open source software development. Perhaps everything is not known, but we can learn much from knowing what is. The most important lesson is that the best way to understand collaboration is to collaborate.
Thanks to Chris Dent, H. Jessica Kim, and Mark Stone for their insightful comments.
Paul Armer, "SHARE—A Eulogy to Cooperative Effort." Annals of the History of Computing 2, no. 2, April 1980: 122-129.
Martin Campbell-Kelly and William Aspray, Computer: A History of the Information Machine (New York, NY: BasicBooks, 1996).
Walter M. Carlson, "The Life and Times of the Digital Computers Association (DCA)." Annals of the History of Computing 18, no. 2, Summer 1996: 63-66.
William Langewiesche, American Ground: Unbuilding the World Trade Center (New York, NY: North Point Press, 2002).
Bobbi Mapstone and Morton I. Bernstein, "The Founding of SHARE: NCC '80 Pioneer Day." Annals of the History of Computing 2, no. 4, Fall 1980: 363-372.
Wesley S. Melahn "A Description of a Cooperative Venture in the Production of an Automatic Coding System." Journal of the ACM 3, Oct. 1956: 266-271.