...so take it easy.

The full title of this 1973 U.C. Berkeley public planning book (recommended by A Better Oakland) is formidable: Implementation: How Great Expectations In Washington Are Dashed In Oakland; Or, Why It's Amazing That Federal Programs Work At All, Economic Development Administration As Told By Two Sympathetic Observers Who Seek To Build Morals On a Foundation Of Ruined Hopes. It seems significant that all the illustrations are excerpts of Rube Goldberg machines.

I bought this book because of the Oakland connection, but there's a lot in here that's relevant to any form of project planning and completion, especially for software developers and designers (like me) trying to figure out why it's so easy to start, and so hard to finish, a project. Other writers in the development world have touched on this before, and there's an entire discipline called Agile that seeks to cut through impediments to completion like a Gordian Knot.

There's an undercurrent of misery and pathos to the book - nothing ruins dinner like 200+ pages on fucked-out-of-the-gate early 1970's social welfare programs in a city you love. The historical framing is an EDA jobs program for the hardcore unemployed that sought to deliver funding to projects and businesses which would in turn employ economically disadvantaged Oakland residents. The late-1960's urgency behind the project stemmed from a desire to nip in the bud further urban race riots like those that had taken place in Watts and elsewhere. Oakland, home of the Black Panthers, was viewed as a potential trouble spot. Rapid flows of federal money aimed at helping the unemployed was identified as the solution. As you might expect from the title, things didn't turn out as planned: money and time were generally wasted, and few people received the promised help. The program fit the general pattern of the past 50 years: splashy introduction, front page news, energy and excitement at the outset, slow leakage of enthusiasm, and an eventual page 10 notice of cancellation several years later.

This is going to get wildly relevant in the coming years, especially in light of Obama's recently-alluded-to New New Deal: "We'll be working out the details in the weeks ahead, but it will be a two-year, nationwide effort to jumpstart job creation in America and lay the foundation for a strong and growing economy." Part of me is saying "uh oh", but a bigger, louder part of me is saying "hellz yeah, where do I sign up to help?"

The core question that authors Aaron Wildavsky and Aaron Pressman hope to answer is: why is the road to hell paved with good intentions? Is there a difference between policy and implementation that can be somehow bridged, or at least described more precisely? "Implementation" refers to that often-overlooked part of the project that happens after the ideas, funding, and excitement, but before any tangible results.

Page 87, on the complexity of joint action:

When we say that programs have failed, this suggests we are surprised. If we thought from the beginning that they were unlikely to be successful, their failure to achieve stated goals or to work at all would not cry out for any special explanation. If we believed that intense conflicts of interests were involved, if people who had to cooperate were expected to be at loggerheads, if necessary resources were far beyond those available, we might wonder rather more why the programs were attempted instead of expressing amazement at their shortcomings. The problem would dissolve, so to speak, in the statement of it.

I love this idea, and it slots in neatly with the commercial world's wisdom that successful companies create room for mistakes, if those mistakes can be used to gain experience and learn. From a project point of view, no one wants to be on the job that fails. From a societal point of view, failed experiments that are adequately described point the way toward eventual success.

Page 98, on mismatched means and ends:

When programs are not being implemented, it is tempting to conclude that the participants disagreed about the special ends they sought rather than the ordinary means for attaining them. Thinking about means and ends in isolation, however, imposes an artificial distinction, especially when more than one program is involved. One participant's ends, such as a training facility, may be another actor's means. Once innumerable programs are in operation, the stream of transactions among people who are simultaneously involved in them may evidence neither clear beginning nor end but only an ebb and flow. As the managers of each program try to impose their preferred sequence of events on the others, their priorities for the next step, which differs for each one and cannot be equally important to all, may conflict. The means loom larger all the time because they are what the action is about. Actually, it is easier to disagree about means because they are there to provoke quarrels, while ends are always around the corner.

The difference between success and failure seems to be the difference between turbulent and laminar flow. Each participant has the same end in mind, but one is relaxed while another is in a hurry, one wants to start here and another there. Even with all actors ostensibly moving in the same direction, turbulence and chaotic flow result from these seemingly-small differences in chosen velocity.

Page 113, on delay:

What had looked like a relatively simple, urgent, and direct program - involving one federal agency, one city, and a substantial and immediate funding commitment - eventually involved numerous diverse participants and a much longer series of decisions that was planned. None of the participants actually disagreed with the goal of providing jobs for minority unemployed, but their differing perspectives and senses of urgency made it difficult to translate broad substantive agreement into effective policy implementation. It was not merely the direction of their decisions - favorable or unfavorable - but the time orientation of the participants - fast or slow, urgent or indolent - that determined the prospects of completion. When so many future decisions depend on past actions, delay in time may be equivalent to defeat in substance.

Much of the process methodology behind Agile seems to recognize that priority-setting is the critical point for most friction: people must agree on what the next most important task is, and this is where most negotiation is designed to take place.

Page 133, on the need for bureaucracy:

If one wishes to assure a reasonable prospect of program implementation, he had better begin with a high probability that each every actor will cooperate. The purpose of bureaucracy is precisely to secure this degree of predictability. Many of its most criticized features, such as the requirement for multiple and advance clearances and standard operating procedures, serve to increase the ability of each participant to predict what the others will do and to smooth over differences. The costs of bureaucracy - a preference for procedure over purpose or seeking the lowest common denominator - may emerge in a different light when they are viewed as part of the price paid for predictability of agreement over time among diverse participants. The price may be too high, but the cost of accomplishing little or nothing otherwise must be placed against it.

Big, dumb bureaucracy has a lubricating effect here. Things take a long time because processes are designed to insulate actors from each others' instabilities. The computation metaphor that seems appropriate here is boundedness: CPU or I/O? What exactly are you waiting for at any given time, and how can project management help participants understand that some given task or responsibility is simply going to take a while, and maybe you should find something else to do?

Page 134, on coordination:

When one bureaucrat tells another to coordinate a policy, he means that it should be cleared with other official participants who have some stake in the matter. This is a way of sharing the blame in case things go wrong (each initial on the documents being another hostage against retribution) and of increasing the predictability of securing each agreement needed for further action. Since other actors cannot be coerced, their consent must be obtained. Bargaining must take place to reconcile the differences, with the result that the policy may be modified, even to the point of compromising its original purpose. Coordination in this sense is another word for consent.

Telling another person to coordinate, therefore, does not tell him what to do. He does not know whether to coerce or bargain, to exert power or secure consent. Here we have one aspect of an apparently desirable trait of antibureaucratic administration that covers up the very problems - conflict versus cooperation, coercion versus consent - its invocation is supposed to resolve.

Everyone wants coordination on his own terms.

This is the part where I criticize unilateral approaches like 37 Signals' Getting Real. The core tenets of Getting Real seem to essentially boil down to a pathological aversion to commitment: commitment to people ("small teams"), to goals ("flexible scope"), and to details ("ignore details", "it doesn't matter"). Generally speaking, people who believe this will have already put themselves in a position to live it: it's no accident that Stamen is seven people. The act of externalizing Getting Real makes it a process, one that's spectacularly bad at addressing coordination. Fine for small projects where everyone starts on roughly the same page, but disastrous for any situation where other actors need to give consent: managers, clients, investors, customers. The universe of Getting Real is a cramped, airless one populated by to-do list managers and communication software for tiny teams.

Where someone needs to be convinced, coerced, or seduced into cooperating with you, process gives way to sub-rational animal instinct.

Pages 165-166, on implementation-as-control:

In this view, for instance, implementers must know what they are supposed to do in order to be effective. Yet, "street level" bureaucrats are notorious for being too busy coping with their day-to-day problems to recite to themselves the policies they are supposed to apply. ... Writing about the administrative process in the regulatory commissions of the New Deal era, James Landis recalls how "one of the ablest administrators that it was my good fortune to know, I believe, never read, at least more than casually, the statutes that he translated into reality. He assumed that they gave him power to deal with the broad problems of an industry and, upon that understanding, he sought his own solutions."

The planning model recognizes that implementation may fail because the original plan was infeasible. But it does not recognize the important point that many, perhaps most, constraints remain hidden in the planning stage, and are only discovered in the implementation process.

This is what I think Agile seeks to address: the idea that requirements change because they flex and respond to previous requirements already met.

Pages 167-168, on implementation as interaction:

This view is strangely reminiscent of old syndicalist doctrines summarized in once-popular slogans like "The Railroads to the Railroadmen" and "The Mines to the Miners." The syndicalists' demand for "industrial democracy" actually concealed a view of production as an end in itself rather than a means of satisfying consumers' wants. We feel the emphasis on consensus, bargaining, and political maneuvering can easily lead (and has, in fact, led) to the conception that implementation is its own reward.

The interaction model of implementation carries interesting evolutionary overtones. The results are not predictable, an element of surprise is maintained, and the outcomes are likely to be different from those sought by any single participant.

This is where I think Agile falls apart: the manifesto promises to do away with process, but introduces process of its own. In particular, the process it introduces is fundamentally introspective, a kind of "Programming for the Programmers" frame of mind that seems to focus on the needs of the development team over the needs of the broader project. The outcomes are likely to have been bent or twisted somewhat along the way.

Page 215, on implementation as adaptation:

In a world of flux, it is only through continuous negotiation between administrators, implementers, and decision makers that any "congruence between program design and program implementation" (mentioned as essential in the literature) can ever be achieved.

"Adaptation" is Pressman and Wildavsky's final watchword for a useful view of implementation. It encodes ideas of flexibility, negotiation, while still leaving room for a deeper goal. This is not willy-nilly natural selection, but a process of constant self-evaluation. There's a lot more on this topic in a future post on Arthur Bentley's The Process Of Government.

Page 228, on learning from error:

In reaction to what is widely perceived as a dismal record, students of implementation, like the evaluators before them, have sought to guard themselves against failure. Instead of learning from error as it is occurring, they hope to prevent future failure before it takes place. Since there can be little learning without mistakes to learn from, however, the field of implementation is caught in a double bind: too much error suggests incompetence and too little inhibits learning.

Earlier tonight I briefly met Spencer Salazar from Smule, the makers of the iPhone Ocarina. They have a small suite of like Sonic Boom ("turns your phone into a virtual firecracker"), Sonic Vox ("the real-time voice shifter"), and Sonic Lighter ("Sonic Lighter is a lighter") that are mostly technology gimmicks. Spencer admitted as much but I'm still completely smitten with the fact that 75% of their applications have a simple globe view that uses the network features of the phone to show you what other people, all around the world, are doing with each app right now. You can hear other people's clumsy ocarina playing, watch little explosions when other people use Sonic Boom, and see who's using the lighter app with some sense of how those people are related to you based on flame-passing connections.

We've seen this all before, in Twittervision and other such globetrotting applications. These Smule globes seem strangely different and much more interesting, largely I think because you hold the phone in your hand instead of the laptop or monitor on your desk. It's a more personal, touched engagement with the screen that makes visualizing an earth-spanning army of phone lighters and flute blowers more physically personal. In particular, the Sonic Boom visualization is like watching television: no reading, no place names, just tiny explosions with audio all over the world with the same unmediated appearance as old top-down resource gathering games like War Craft I.

Having just read Teeming Void's Against Information (a critique of "data art"), I'm thinking about direct perception of data as a way of making it more visceral. The Golan Levin and Jonathan Harris pieces referenced in the paper all suffer from various forms of indirection: Levin makes breaking up look like math and physics, while Harris jumps to all sorts of crazy conclusions based on faulty language parsing and excessively abstract visual metaphor. How can a visual representation of data make itself felt right there, in your hand? Pictures help. Sound helps.

I've been expanding the georeferenced collection of Oakland maps that Gem and I started back in May. Recently, I purchased a 1967 Standard Oil map of Oakland for a few bucks from EBay. I was looking for late 1960's / early 1970's, because that's when the freeway structure here really started to take shape. Previously, we looked at a switch from rail to roads. Through the 50's and 60's, the switch was accelerated with the construction of massive highways through what had formerly been residential neighbhorhoods.

Particularly interesting is the Cypress Viaduct, a raised connection between highways 880, 580 and 80 running through West Oakland. When built, it was sharply criticized for splitting the neighborhood and further isolating it from downtown Oakland. The current site of the viaduct was where I made some of my first edits to OpenStreetMap. The structure was destroyed in the 1989 Loma Prieta earthquake shortly after my family moved to California, but on this map it's a fresh addition to the landscape:

The 19th anniversary of the quake was October 17th, one month ago.

The new 1967 map is a striking constrast to the previous 1952 map. The various freeways connected to Interstate 80 are one major difference, but the cartography is also a big contrast. This map is similar to the other Gousha-designed map from 1936 in its choice of bright colors, but it also features topographic shading up in the hills and orange highlights around freeway exits. A significant piece of infrastructure still under construction at this point is the 980 / 24 connector from downtown Oakland up into the hills toward the Caldecott Tunnel. The construction areas for the southern stretch are marked, while the northern route is still a whispy dotted line through miles of backyards.

Are you a talented developer interested in supporting our data visualization and mapping practice from the server end of things? Are you interested in a full-time gig at our San Francisco office?

You'll be working with a small team of designers and engineers who will be looking to you to make their ideas feasible. You're excited by the possibility of cutting and bending data to fit it through the thin straw of the internet. You can look at a source of information and model it as resources, rows and columns, messages and queues.

...

Our technology choices lean towards open source databases, unix-flavored operating systems, and scripting languages like Python and PHP. You'll be expected to know these things, and bring something new and unexpected besides.

Read the rest of the job description and let us know! (watch the spamarrest response on that e-mail address)

Earlier this morning, I returned from a four-day spin through Tokyo, my second visit there, to speak at Web Directions East. This trip was wholly unexpected, as I was pinch-hitting for Jeff Veen who had to cancel at the last minute and suggested me as a replacement. Fortunately it was possible to book a last-minute flight, take over a hotel reservation, and write an hour-long keynote talk in just a few days. It was an surprise honor to even be asked, and the experience was smooth sailing all the way, including the miniature hotel room.

I'm not going to post exact talk notes, but I outlined a general overview of data visualization and then focused on a range of Stamen's projects from the past four years and how they illustrate some deeper trends. Elements of this should be familiar to anyone who's heard me, Eric, Shawn or Tom speak before:

• Mark Newman's 2004 election cartograms. I learned from Karl that they'd been freshly updated to reflect last week's results.
• Examples of flow-management including 538 and John Allspaw's Flickr capacity-planning ideas.
• Our 2004 MoveOn Virtual Town Hall application, for the first time in what seemed like years. It would be amazing to revisit this project again somehow, possibly in a non-political context.
• Digg Labs, in the context of "liveness", from our Live/Vast/Deep iron triangle of visualization.
• Trulia Hindsight and Snapshot, as examples of "vastness".
• The hot, fresh, new SFMOMA ArtScope as an example of "deepness". This also served as a nice digression about map tiles, something I've been thinking and talking about a bunch.
• Oakland Crimespotting to round out some ideas about stable linking and data on the web. I also heavily name-checked Tom Coates's Native To A Web Of Data and Matt Biddulph's Designing Data For Reuse here.

A few people asked after the source of a particular background photo from my slides - it was an image of the Shanghai Urban Planning Museum's city model, possibly gaffled from blog.360dgrs.nl.

In addition to meeting the other speakers and organizers, John, Oli, Andy, Jeremy, Eric, Dan, and Doug, I also had a chance to take an excellent nighttime Tokyo bicycle and beer ride with Craig and Verena that made me wish SF and Oakland maintained the pavements a bit more effectively. I now desperately want to add a third bike to my collection, a mama charion granny bike.