Social Networks : Decentralized Intelligence

Duncan Watts is associate professor of sociology at Columbia University, where he directs the Collective Dynamics Group (http://cdg.columbia.edu), and an external faculty member of the Santa Fe Institute. His research on the structure of social networks and the dynamics of network processes has appeared in journals such as Nature, Science, and the American Journal of Sociology. He is also the author of two books—Six Degrees: The Science of a Connected Age (W.W. Norton, 2003) and Small Worlds: The Dynamics of Networks between Order and Randomness (Princeton University Press, 1999). He holds a B.Sc. in Physics from the University of New South Wales, and Ph.D. in Theoretical and Applied Mechanics from Cornell University. Originally Posted on the Slate website on Thursday, Aug. 5, 2004 The most publicized recommendation of the 9/11 commission—and one President Bush and Sen. John Kerry have raced to endorse—is that the United States create a national director of intelligence. Centralizing is an understandable response to the pre-9/11 intelligence fiasco. But as organizational science and history show, it's also a misguided one. When organizations fail, our first reaction is typically to fall into "control mode": One person, or at most a small, coherent group of people, should decide what the current goals of the organization are, and everyone else should then efficiently and effectively execute those goals. Intuitively, control mode sounds like nothing so much as common sense. It fits perfectly with our deeply rooted notions of cause and effect ("I order, you deliver"), so it feels good philosophically. It also satisfies our desire to have someone made accountable for everything that happens, so it feels good morally as well. But when a failure is one of imagination, creativity, or coordination—all major shortcomings of the various intelligence branches in recent years—introducing additional control, whether by tightening protocols or adding new layers of oversight, can serve only to make the problem worse. To understand this, we need to step outside government bureaucracy for a moment and take a look at the world of industrial organization, which has had some valuable experience in recovering from major mistakes. In 1997, the Toyota group suffered what seemed like a catastrophic failure in its production system when a key factory—the sole source of a particular kind of valve essential to the braking systems of all Toyota vehicles—burned to the ground overnight. Because of their much-vaunted just-in-time inventory system, the company maintained only three days of stock, while a new factory would take six months to build. In the meantime Toyota's production of over 15,000 cars a day would grind to an absolute halt. This was the kind of disaster with the potential to wreck not just the company itself, but the entire Japanese automotive industry. Clearly, then, Toyota, along with the more than 200 other companies that are members of the extended Toyota group, had ample incentives to find a solution. The big question was: How? How does one rapidly regenerate large quantities of a complex component, in several different varieties, without any specialized tools, gauges, and manufacturing lines (almost all of which were lost), with barely any relevant experience (the company that made them was highly specialized), with very little direction from the original company (which was quickly overwhelmed), and without compromising any of their other production tasks? Well, actually it's not clear that one could do it at all, nor was it clear at the time to any of the senior managers of the Toyota group. After all, if this was the kind of disaster that their risk management executives had considered, they would never have left themselves vulnerable to it in the first place. Nevertheless, they succeeded, but not in the way one might have expected. Rather than relying on the guidance and coordination of an inspired leader (control mode), the response was a bewildering display of truly decentralized problem solving: More than 200 companies reorganized themselves and each other to develop at least six entirely different production processes, each using different tools, different engineering approaches, and different organizational arrangements. Virtually every aspect of the recovery effort had to be designed and executed on the fly, with engineers and managers sharing their successes and failures alike across departmental boundaries, and even between firms that in normal times would be direct competitors. Within three days, production of the critical valves was in full swing, and within a week, production levels had regained their pre-disaster levels. The kind of coordination this activity required had not been consciously designed, nor could it have been developed in the drastically short time frame required. The surprising fact was that it was already there, lying dormant in the network of informal relations that had been built up between the firms through years of cooperation and information sharing over routine problem-solving tasks. No one could have predicted precisely how this network would come in handy for this particular problem, but they didn't need to—by giving individual workers fast access to information and resources as they discovered their need for them, the network did its job anyway. Much the same kind of recovery happened in lower Manhattan in the days after Sept. 11, 2001. With much of the World Trade Center in rubble and several other nearby buildings closed indefinitely, nearly 100,000 workers had no place to go on Sept. 12. In addition to the unprecedented human tragedy of lost friends and colleagues, dozens of firms had to cope with the sudden disappearance of their offices along with much of their hardware, data, and in some cases, critical members of their leadership teams. Yet somehow they survived. Even more dramatically, almost all of them were back in business within a week—an achievement that even their own risk management executives viewed with amazement. Once again, the secret to their success was not so much that any individual had anticipated the need to build up emergency problem-solving capacities or was able to design and implement these capacities in response to the particular disaster that struck. Rather, the collective ability of firms and individuals alike to react quickly and flexibly was a result of unintentional capabilities, based on informal and often accidental networks that they had developed over years of socializing together and collaborating on unrelated and routine—even trivial—problems. When talking about their recovery efforts, manager after manager referred, often with puzzlement and no small sense of wonder, to the importance of informal relationships and the personal knowledge and understanding that these relationships had engendered. Perhaps the most striking example of informal knowledge helping to solve what would appear to be a purely technical problem occurred in a particular company that lost all its personnel associated with maintaining its data storage systems. The data itself had been preserved in remote backup servers but could not be retrieved because not one person who knew the passwords had survived. The solution to this potentially devastating (and completely unforeseeable) combination of circumstances was astonishing, not because it required any technical wizardry or imposing leadership, but because it did not. To access the database, a group of the remaining employees gathered together, and in what must have been an unbearably wrenching session, recalled everything they knew about their colleagues: the names of their children; where they went on holidays; what foods they liked; even their personal idiosyncrasies. And they managed to guess the passwords. The knowledge of seemingly trivial factoids about a co-worker, gleaned from company picnics or around the water cooler, is not the sort of data one can feed into a risk-management algorithm, or even collate into a database—in fact, it is so banal that no one would have thought to record it, even if they could. Yet it turned out to be the most critical component in that firm's stunning return to trading only three days after the towers fell. So, how does one make this kind of magic happen? Unfortunately, no one is quite sure. Different organizations, from business firms to research communities to the military, have tried to address their collective problem-solving needs in a variety of ways. Some militaries make a point of training their officers in joint-service academies and staff colleges, both of which serve the purpose of building friendships and professional relationships across otherwise frosty institutional boundaries. Academic researchers, for their part, organize interdisciplinary conferences and working groups that serve to introduce disciplinary specialists who turn out to have complementary knowledge or skill sets, but who otherwise would never have had occasion to meet. And business firms from the automotive to high-tech and finance industries deliberately cross-pollinate their intellectual capital by fostering worker exchanges across divisions or even firms, building problem-solving teams around tasks rather than departments, emphasizing informal agreements and collaborations over formal contracts, and organizing sophisticated team-building exercises for geographically dispersed junior executives. None of these methods, however, come with any guarantee of success, and all of them come with their associated (and usually far more tangible) costs. As a result, not everyone in either private or public sectors explicitly encourages informal network building, nor is it always effective when they do. Even from a purely theoretical perspective, no one has figured out what is required to build organizations that are not only efficient at performing routine, familiar tasks, but also good at adapting to the exceptional and the unanticipated. And even where we have apparently clear examples of success, it isn't clear that what works, say, for a car manufacturer or finance firm is going to work for the CIA and the FBI. In the world of intelligence, the kind of information that, in less-clandestine businesses, tends to flow along informal social networks must necessarily be subject to greater constraints. So harnessing the power of social networks for innovation, creativity, and rapid adaptation is a trickier business for intelligence organizations than arguably for any other kind. What should be clear, however, is that combining the many different agencies involved in intelligence gathering and analysis at a single point—that of the director of intelligence—is almost certain not to succeed in delivering the kind of ambiguous yet essential functionality that everyone wants. So, some other kind of connectivity, along with a more creative approach, is required—one that incorporates not only the sharing of information across agency boundaries (a recommendation of the commission's that has received relatively little attention), but active collaboration, joint training, and the development of long term personal relationships between agencies as well. Creative intelligence analysis has a lot in common with other kinds of problem-solving activities: thinking outside the box, challenging deeply held assumptions, and combining different, often seemingly unrelated, kinds of expertise and knowledge. By understanding how innovative and successful organizations have been able to solve large-scale, complex problems, without anyone "at the top" having to micromanage the process, the intelligence community could learn some valuable lessons that might help it escape the mistakes of the past. Source: leader-values.com