Our aspiration should be to fail well.
Keywords: Markup Languages
This talk was transcribed from tape through the kind offices of Kate Hamilton, who supplied the descriptions of audience reactions. The author has supplied some text where the tape was indistinct and has taken the opportunity to correct some ungainly phrasings and some factual errors, but the text is otherwise unimproved.
The title listed in your program for this talk is “Failing Upward”. I have wondered several times over the course of the last few days whether I was going to talk about that subject ... or instantiate that subject. I'll leave that to you to judge.
One thing has become clear during the preparation of the talk: failing is harder than it looks.
It's hard to get failure right.
Many of you will have read the series of books by Henry Petroski, a civil engineer from Duke University who has written about the pencil and about bridge design and about a variety of civil engineering failures.1 Petroski is very taken with the idea — and he's infected me with it — that in interesting ways failure is an engine of progress. You can chart the progress of various branches of civil engineering as a series of alternations of successes — successful building of a new type of bridge, and successful reallocation of that new bridge-building technique to longer and longer spans and lower and lower coasts, as material savings were expected — until finally the design limits of that particular technique are exceeded and you have a spectacular failure which so traumatizes an entire generation of civil engineers that they stay well away from those design limits and you have a generation without failures — until people who are not yet building bridges, or who were not building bridges when the last set of design failures happened — and you have another bridge or major structure that exceeds the design limits of its technique.
Many of you will have read in Petroski or in other places — engineers love to talk about this2 — about the story of John Roebling, the great engineer who built the Brooklyn Bridge, at a time when engineers were well aware that bridges had to withstand not just the static stress of the weight of the structure and the weight of the traffic they must bear, but also the forces imposed by crosswinds, which have a sort of aerodynamic effect causing it to lift or to flex. The outstanding example of this is the Tacoma Narrows Bridge, which shook itself to death in 1940 because it wasn't stiff enough to withstand the crosswinds that caused it to shake. Its shaking was well-known at the time: that bridge had the nickname Galloping Gertie. There was a film crew at the bridge one day to take shots of it shaking in the breeze; it happened to be the day that the breeze got a little stronger, and the bridge shook a little harder, the harmonics were just right, and the bridge shook itself to pieces. A striking engineering failure precisely because of these aerodynamic forces.
Roebling knew about those forces. He knew he didn't have a clue how to model them mathematically, but he knew they existed. He did do a static stress analysis, and built the bridge six times as strong as the bridge needed to be in order to provide a margin of safety, or margin of error. As a result, of course, the Brooklyn Bridge still stands, still carries traffic, to this day. Other bridges, built before and after, that did not give themselves so large a margin of error, collapsed; Galloping Gertie is only the most famous, because it was captured on film.
Petroski will point out that typically you learn more from a failure than from a success. What do you learn from the Brooklyn Bridge? If you build a bridge with a huge margin of error, you have a chance of building a bridge that succeeds. If you have a safety factor of six, you have succeeded in narrowing down the estimate of the amount of strength or stiffness that a bridge built in that way needs, to a little less than an order of magnitude. That's better than nothing; but it would be nice to have something a little more precise. It wouldn't be so bad if the people who pay for bridges didn't mind the difference between one times and six times a particular dollar figure; but they do. And so civil engineers always want to give an adequate margin of error but not an excessive margin of error.
Some people looking at this glass will say it's half full, and some will say it's half empty. Some engineers will say it's a 100% safety margin; others will say it's over-engineered by a factor of two: it has twice as much capacity as you currently need.
You don't learn nearly as much from success as you might hope; whereas for any given failure you have a concrete example of a failure mode that you can trace, and I'm sure that civil engineers learned a lot more about the mathematics needed to predict the aerodynamic forces on bridges from Galloping Gertie than they did from the Brooklyn Bridge.
There are a lot of people in this room who fundamentally identify with the goal of research: an increase of knowledge. And yet if you were a civil engineer, which bridge would you rather have on your resume: the Brooklyn Bridge or Galloping Gertie? [laughter] As a contribution to knowledge they may not be comparable, but we would all rather succeed than fail.
As I said, failure is hard to get right. If we all succeed, we don't learn nearly as much. Our engineering memory is better if we do. Every rule in a building code, I sometimes think, is a memory of some long-ago disaster, just as every rule in an accounting system is a memory of some long-ago fraud.
What would markup systems be like if we learned from our failures? If we feared failure the way bridge builders fear failure?
First of all, markup languages would feel a lot more grown-up. We would be much more in line with the responsibilities that markup systems carry as they become more and more ubiquitous. With ubiquity comes responsibility. It's one thing to have casual failures of this kind or another when you're building casual systems for amusement; it's something else when you're building patient records, or airline security systems.
In order to do better, you need analysis of failures, and that's difficult for a number of reasons.
It's hard because the word “failure” can be rhetorically divisive. I was once talking with a colleague, collaborating on a project, and they were saying: “We need to change things, this isn't working.” I was in a hurry, and in an engineering mindset; in order to cut to the chase I said: “Okay, what are the indices of failure that you're seeing?” So that we can understand what's going on and see what it takes to fix those. I hadn't seen any indications of failure; perhaps my colleague had seen something I didn't. When I said the word “failure” my colleague — who, like me, was trained as a humanist and wasn't at that moment thinking like an engineer — was immediately offended, and said, “I — I — I didn't say there was a failure. I just said we had to talk.” We then spent the rest of our time analyzing the rhetoric of the word “failure” instead of analyzing and addressing the problem in our joint project.
The rhetorical divisiveness is part of the problem. And of course the loss of face that comes when any of us is associated with the development of a system which is later analyzed as having caused or suffered a failure is another part of it. I notice that when I speak with some people in the room about supporting, say, overlap in markup systems, it's very convenient to talk about the shortcomings or failures of SGML or XML in this or that way. And I noticed when I was talking on that topic with Joe Gangemi that introducing it that way shut off the conversation very rapidly. He felt just enough responsibility for SGML that identifying it has having failed was putting him in an extremely uncomfortable position.3
Those of you who were on a debate team in high school will remember that we were taught two different ways to argue for a proposition. For a long time, academic scholastic debate, at least in North America, has had a very conventional form. There is a proposition, an affirmative team and a negative team. The affirmative team argues in favor of the proposition; the negative team argues against. The usual structure of the affirmative case begins by identifying a need for some change in the status quo. You argue for the proposition by arguing that the status quo is untenable: there is a problem, we have to solve it, there is a need for change. Then you identify a plan, how to fix the problem, and then the advantages that flow out of adoption of the plan. This is the need-plan-advantage case.
Some time after WWII some theorists of academic debate said, When things aren't obviously broken, it's difficult to make the case that there's a need for change. When you're trying to sell someone a Chevrolet, it's actually counterproductive to argue that their Ford is about to blow up, or is not a good idea: they are, after all, invested in that Ford: they chose it. What you need to argue is that there's nothing intrinsically wrong with the car you bought, but the one I would like to sell you is a little better, comparatively speaking. This is the comparative advantage plan. It doesn't assign failure to the status quo, it only says things could be better.
Of course, my high school debate coach absolutely refused to allow us to make any comparative-advantage case. This new-fangled style of case was too weak; you really needed a strong indictment of the status quo. He may have been right. Those of you who did musicals in high school will remember that Professor Harold Hill in The Music Man does not come in and tell the citizens of the small town: “Life will be slightly better if you have a music program in your public schools.” He comes in and says: “You have trouble! Right here in River City!” [laughter]
Whether we put the rhetorical emphasis on the problems with the status quo or the comparative advantages of something else, however, it's essential that we be able to talk about failures and shortcomings without loss of face and without putting people in an emotionally uncomfortable situation in which they are resisting analysis of the problem in order to save face.
Many of our most successful engineering environments have cultivated a practice of always calling a spade a spade. The NASA flight controller Gene Kranz published his memoirs a few years ago, under the title Failure is not an option. Kranz was one of the flight controllers who controlled the Gemini and the Apollo flights of the 1960s. He opens his memoirs with a striking anecdote about a simulation; they are running into some problems with the propellant tanks on the command module, and Kranz calls for a particular mission-abort option. As the simulation proceeds, it becomes clear that he's made the wrong choice: instead of coming down in the Atlantic, the capsule comes down on land. The simulation controller comes on the intercom and says, “Flight, not only did you put the spacecraft down on land, you killed the crew. Because the mountains you brought them down in are above 10,000 feet. And that's where the parachutes should open.” By calling a spade a spade, that engineering culture tried to make it possible to grapple with the problem instead of with the engineers' egos. That's an absolute requirement for going forward.
If you're like me, when you saw the title Failure is not an option, or when you heard it just now, you will have said, “Wait: if you think failure is not a strong possibility, you're not paying attention.” It took me a long time to give him more credit for his word choice. He didn't say “failure is not a possibility,” but failure is not an outcome he wished to accept willingly. Later on in this talk, though, I'm going to argue that we should increase our rate of failures, which amounts to saying that sometimes failure should be an option.
If we ask, “What would markup systems be like if we analyzed failure the way engineers analyze failure?”, we would be looking for the failure modes of the projects we support or the projects we help our clients with. What can go wrong in a markup project, in the design of a vocabulary or its deployment? We've had some papers that shed light on this — on ways of building systems to increase, for example, software reliability. John Clark talking about software assurance [Clark 2006], or David Dubin talking about just how bad things can get — the failures that can arise, the semantic degradation, after a series of individually fairly rational decisions [Dubin/Plutchak 2006]. After that talk there was some discussion about whether there actually is a failure: I think some people in the audience think there was at least one failure in that chain of decisions; some people think there was not a failure but there was an opportunity for improvement. There is at least a rhetorical difference there, but maybe it's just a rhetorical difference.
It's difficult — as the Dubin/Plutchak paper makes clear — to make sure you've identified the failure at the correct layer. Whenever you have a complicated system, the failure may be in the fact that one layer doesn't do what the overall design says it should do vis-a-vis some other layer; in that case, it may be unambiguous. Or, you may not actually be clear what the interaction between those two layers or those two parts of the system should have been, and you may be discovering that even if you have a clear boundary between the two parts of the system, the practical experience may be teaching you that it's not the failure of this layer or that layer but the failure of the overall design: you didn't draw the lines where you needed to. You need to make it easier for the layers to succeed individually and in their interactions with each other. That can be complicated.
At least one person here is interested in overlap, and in markup languages that support overlap at a very basic level, in part because when we have developed systems for handling overlap at the level of vocabularies, such as the TEI Guidelines, people have said, “That's a kludge: you're just trying to work around a fundamental problem in XML by having some higher-level thing. You're just escaping to another level, but the place to solve that is at the XML level, or at the general markup-syntax level.” At least one person in the audience said to me this morning, “Gee, why is he trying to solve this problem at the markup-language level? All you have to do is define the presentation semantics of a particular vocabulary for handling this, and then the fundamental layers of the system don't have to deal with the complications you're trying to introduce at the basic level”! Which level is actually responsible for handling all that? Is it “supposed to be” at the markup level, or “supposed to be” at the application level? The buck passes through; we hand it on.
The problem is that the boundary between the markup language and the vocabulary is not God-given: it's one that we devise, that we specify. We don't have enough chances to do experiments — to experiment with a lot of different boundaries. You have to make a certain commitment — an annealing process — a commitment here, a commitment there, and then you've defined this process. But if you make slightly different decisions, or in a different order, you would assign responsibility for the problem in an entirely different place. People criticizing XML Schema, for example, will say, “The problem with Schema is the way wildcards work: I can't have wildcards where I'd like to have them.” (I can talk about wildcards for hours - but I won't.) So I ask: “Why do you want wildcards everywhere?” And they say, “I want every 2.0 message of my vocabulary to be able to have new child elements — I don't know what I want it to be yet, but I know I'll want to add something to my 1.0 messages, and I'll want a 2.0 message to be valid against the 1.0 schema.” And I say, “Why do you want every 2.0 message to be valid against the 1.0 schema?” “When we deploy servers that send these messages, we generate objects from the schema. They will assess the messages against the schema, and if they're not valid they won't open them. Because we have deployed software supporting 1.0, when we roll out 2.0, 2.0 messages will be sent around, and they'll go to 1.0 servers, and the 1.0 server needs to be able to support them.” And I say, “Well, I think what you've just described is this: You're using data-binding tools to generate classes, and the guarantee offered by the data-binding tool is that the classes it generates will accept any message valid against the schema that the class is generated from. If you have validated the data upstream, and it's valid, these classes will handle everything. You're now deploying those classes without a validator in front of them, so that they're receiving both valid and invalid data. And the problem, you say, is in the schema...? I would have thought the problem is that you're deploying software outside of its design specs, or possibly that your data-binding tools are not giving you what you wanted, which is classes that will accept and correctly process all valid messages and will not roll over when they see invalid messages.”
Failure turns out to be surprisingly psychological in nature.
This surprised me, at first, because when you think of failure in terms of bridges falling down, or the treads separating from a tire, or a fuse blowing, the difference between failure and not failure seems fairly — crisp, shall we say. But the American Society for Civil Engineers has a Technical Council on Forensic Engineering that defines failure as “an unacceptable difference between expected and observed performance.”4 Notice there are two wide-open doors through which psychology can drive into this definition. If you have too many failures or too few, you can change your decisions about what are unacceptable deviations from expected performance, or you can change your expectation for that component.
So failure is a little bit like the definition of a weed: A weed is an unwanted plant. That's what the dictionaries say. When my wife tells me to weed the yard, I go out and say, “I love you all.” [laughter] “I want you just where you are.” And I go in and say, “We have no weeds.” [laughter] Experience has taught me that a truly descriptive lexicography would gloss “weed” in English as a plant that is not desired by the woman in the house. The man has nothing to say about it. [not so much laughter]
Notice that the acceptability of the deviation may seem at first to be very simple and straightforward. Things can perform better than expected, and that's not necessarily a failure.
Unless you are Henry Ford, who, as you may know, sent people out to haunt the junkyards of the U.S. for Model Ts that had gone out of service, to see which parts were still good. If there was still a working carburetor in every Ford that had failed for some other reason, the conclusion was simple: we're spending too much money on carburetors. If our goal is to reduce the overall cost, there is no point in building a carburetor that will last 100 years if the rest of the car will only last some shorter period of time. Henry Ford's ideal was the wonderful one-hoss shay described by Oliver Wendell Holmes, in which everything was just as strong as everything else: it lasted 100 years and then collapsed into dust, because everything failed at exactly the same moment.
Exceeding expectations can also be a form of failure. You can see this in the stock market every now and then. If a company exceeds its expected revenue by a large enough amount, the conclusion is not that the company is healthy but that they don't know what they're doing, and the stock may fall just as readily as it falls if they come in under expectations. It's a little like ISO 9000, which turns out not to require — in terms of quality control — that you have a low failure rate, but that you have a constant failure rate. This still surprises me. But it relates to the notion of failure as a deviation from expectations.
What that means is that for a lot of things you get a choice. Is delayed adoption of a spec a failure? Was CSS a failure because it wasn't adopted as early as we would like? Most people would now say CSS is a success — except for those who say, “CSS 3 has been out for six months and I don't see it in the browsers yet!” I don't know what the turnover rate for the population of browsers is. Jon Bosak was telling me the turnover rate for automobiles for this country is about 17 years: if everybody started buying electrics today, we'd still have gasoline-powered vehicles on the road in 17 years. Shifting to another energy source will not happen quickly if it happens at all; and lord knows shifting to another browser generation doesn't happen quickly. How many people work for institutions whose web standards require that your web page be viewable in Netscape 3.5? Yes, I see there are at least a couple die-hard organizations.
Another aspect of the psychological aspect of failure and success is that people who have worked with the semiconductor industry tell me that when a new chip is designed and prototyped, what the companies first do is find out what it does. It may or may not do what the designer was trying to make it do, but whether it does or not, it will have some interesting properties, and it is the job of the marketing department to find out what those are and find out who wants to buy them. Like the old cartoon in which the cowboy is saying, “Before I get around to describing the fine qualities of this horse I want to sell you, just what fine qualities are you interested in?”
So — we get to choose our rate of failure, by adjusting our expectations. If you're in an environment in which there's been nothing but success after success, you will observe that you risk you and your colleagues becoming complacent, and you may find it necessary to increase your failure rate by increasing your expectations — by cranking it up a notch, so that not everything is a guaranteed lead-pipe cinch. You want a certain failure rate, if only to motivate yourself. In that case, it's clear that the civil engineers who followed John Roebling analyzed, in some sense, the Brooklyn Bridge as a failure: If we can get by with less, we can build the bridge for less, and that is also a success in civil engineering terms. Until, of course, you get to the point when that doesn't work.
Validation has always struck me as being one of the success stories of markup systems. There is a clear failure mode for most software. Software will fall over when it is confronted with a statement that it does not expect, and validation allows us to define very clearly the set of inputs for which a program should be prepared. You can check the software against a valid set of documents and make sure it does the right thing; if it doesn't, you know who to blame. In the convenience-store industry, I'm told, if the cash register sends a message to the gas pump — or vice versa — and the recipient throws it on the ground — the system stops working, and the convenience-store owner calls both vendors and says, “I have a problem.” They then call each other and say, “You have a problem.” Eventually somebody goes out to the convenience store, gets a copy of the message, and validates it against the schema. They do have agreements — if the cash register and gas pump are from different companies, they will have agreed to send and receive messages that are conformant to a schema. If the message conforms to the schema, then the maker of the device that rejects that message gets the joy of figuring out what's wrong with that device, fixing it, and deploying it, there and anywhere else they have this faulty software. Whereas if the message is not valid, it is the sender's job to fix it. This works very well. Some of us might think it would work even better if they validated the stuff routinely, but that's a question of optimization. There's a lot of optimistic non-validation in the world today.
But in another sense validation is a failure story for markup languages. Those of us who, like me, are obsessed with validation and talk about it all the time, give to a lot of people the impression that people interested in markup and especially in formal markup languages are just tight-assed control freaks who wear belts and suspenders at the same time because they're afraid something might happen that we're not prepared for. [laughter] We put those people off.
I think this is one reason that the microformats community we were talking about the other day in connection with Liam Quin's paper [Quin 2006] mark up their HTML documents the way they do is precisely that validators won't object to it and there are no rules, no one to come along and say, “Hey! That document's no good!” There's a threat in having your document, in which you invested, identified as invalid. There's a certain shame, a loss of face; it's a failure on the part of the document. Some document authors identify enough with their documents to feel responsible. One wants to be responsible within limits; but we'd prefer they fix the problem rather than just being depressed. [laughter]
I am pleased to say the conference has had a number of examples of this failure-as-an-option kind of papers — problems that we identify by ratcheting up our expectations a little bit. Why, for example, should we not demand that XSLT be a fully functional programming language [Novatchev 2006]? If you make that the standard of success, you can identify what is otherwise clearly a very successful language as having a problem, and then we can improve things.
Why not demand or expect that it have a much more compact notation, so that you get much more of your algorithm onto a single screen, and it can look like a normal programming language [Wilmott 2006]? Well, there are reasons - and I talked to Sam Wilmott at some length about them afterwards — but still: it is a perfectly legitimate example of failure as an option. Let us demand that our data be re-usable between RDF and Topic Maps, and let those two forms just be exchangeable garments between which we can shift the information, which is what we really care about, rather than the garment that it's dressed in at a given moment [Pepper et al. 2006].
Let us demand that XML be easy to write, both in the editor-interface sense that Peter Flynn was talking about the other day [Flynn 2006] and semantically in the way that Yves Marcoux was talking about this morning [Marcoux 2006]. In that connection I was very struck with something that Maria Bilde Rasmussen said in connection with her poster, about author-centered XML. The goal for an editing system for lexicographers is that when a lexicographer looks at the screen he should not say, “Oh, okay: this is an XML document that represents a lexicographic entry.” You want the lexicographer to look at the screen and say: “This is a lexicographic entry.” Period. If absolutely necessary it may be ok if they say, “This a lexicographic entry represented in XML” - but not “This is primarily an XML document.” You want them to see through the XML, to the information. Why not, then, similarly demand that it be easy to inch up to the semantic content of XML by using microformats? Why not find ways to increase the reliability of microformats, by finding ways to validate them without making the suppliers of the microformatted information feel that they have lost face? We need more failures; we need to choose our failures; we need to identify potential failures that we can adopt and make our own. In this we can imitate the best of our mathematical colleagues, some of whom are famous as being able to identify interesting unsolved problems. Alfred Tarski, about whom I read a biography recently, was apparently famous among his students and colleagues because already in the first weeks of introductory courses he could identify interesting problems that were unsolved, that some of his students then became interested in and went on to solve.
The cycle of success and failure can in unhappy cases just become a sort of vicious circle, when as you are trying to solve a particular problem, your solution takes on aspects of the problem that you were trying to solve. This is not unique to information technology: we can all think of institutions fighting enemies of human rights who fall into the trap of prosecuting that fight by tending to neglect human rights in their own right.
What we want, though, is to fail upwards: we want our failures to prompt us to solutions that leave us — certainly with more problems to solve — but ideally in a situation where things are slightly more positive. We would like not just a circle but a helix.
So let us fail boldly. Let us choose failures — choose things to improve. The only failure we should really be afraid of is the failure to push our systems and our demands hard enough. The failure to set our sights high enough. The failure to imagine and work for a better world. [grand applause]
In essentials, my description of Roebling and the Brooklyn Bridge and Galloping Gertie follows that of Jon Bentley in a column entitled “The back of the envelope”, included in [Bentley 1986]. I'm recounting it from memory, though, and may have gotten some details wrong.
Joe Gangemi, who served on the ISO working group that defined ISO 8879, SGML, was at Extreme to give a polemical talk on the shortcomings of XML for publishing: [Gangemi 2006]. My colleagues have had occasion to point out to me that I tend to become defensive with certain formulations of the overlap problem, objecting to them as unfairly negative towards XML and counter-productive; they particularly enjoy pointing it out when the formulation in question is one I have used myself in earlier work.
The definition is quoted in [Petroski 2006], page 51. As his source he cites Kenneth L. Carper, “Construction pathology in the United States” Structural engineering international 1 (1996): 57.
[Bentley 1986] Jon Bentley. Programming pearls. Reading, Mass.: Addison-Wesley, 1986.
[Clark 2006] John L. Clark. “Structured software assurance.” Talk delivered 8 August 2006 at Extreme markup Languages 2006, an IDEAlliance conference, Montréal. http://www.idealliance.org/papers/extreme/proceedings/html/2006/Clark01/EML2006Clark01.html.
[Dubin/Plutchak 2006] David Dubin and Joel Plutchak. “Metadata enrichment for digital preservation.” Talk delivered 9 August 2006 at Extreme markup Languages 2006, an IDEAlliance conference, Montréal. http://www.idealliance.org/papers/extreme/proceedings/html/2006/Dubin01/EML2006Dubin01.html.
[Feferman and Feferman 2006] Anita Burdman Feferman and Solomon Feferman. Alfred Tarski: Life and logic. Cambridge: Cambridge University Pres, 2004.
[Flynn 2006] Peter Flynn. “If XML is so easy, how come it's so hard to write?” Talk delivered 8 August 2006 at Extreme markup Languages 2006, an IDEAlliance conference, Montréal. http://www.idealliance.org/papers/extreme/proceedings/html/2006/Flynn01/EML2006Flynn01.html.
[Gangemi 2006] Joseph V. Gangemi. “XML for publishing.” Daily polemic delivered 8 August 2006 at Extreme markup Languages 2006, an IDEAlliance conference, Montréal. http://www.idealliance.org/papers/extreme/proceedings/html/2006/Gangemi01/EML2006Gangemi01.html.
[Kranz 2000] Gene Kranz. Failure is not an option: mission control from Mercury to Apollo 13 and beyond. New york: Simon & Schuster, 2000.
[Marcoux 2006] Yves Marcoux. “A natural-language approach to modeling: Why is some XML so difficult to write?” Talk delivered 11 August 2006 at Extreme markup Languages 2006, an IDEAlliance conference, Montréal. http://www.idealliance.org/papers/extreme/proceedings/html/2006/Marcoux01/EML2006Marcoux01.html.
[Novatchev 2006] Dimitre Novatchev. “Higher-order functional programming with XSLT 2.0 and FXSL.” Talk delivered 8 August 2006 at Extreme markup Languages 2006, an IDEAlliance conference, Montréal. http://www.idealliance.org/papers/extreme/proceedings/html/2006/Novatchev01/EML2006Novatchev01.html.
[Ormerod 2005] Paul Ormerod. Why most things fail: Evolution, extinction & economics. New York: Pantheon, 2005.
[Pepper et al. 2006] Steve Pepper, Valentina Presutti, Lars Marius Garshol, and Fabio Vitali. “Reusing data across RDF and Topic Maps.” Talk delivered 8 August 2006 at Extreme markup Languages 2006, an IDEAlliance conference, Montréal. http://www.idealliance.org/papers/extreme/proceedings/html/2006/Pepper01/EML2006Pepper01.html.
[Petroski 1985] Petroski, Henry. To Engineer Is Human: The Role of Failure in Successful Design. New York: St. Martin's Press, 1985.
[Petroski 1990] Petroski, Henry. The Pencil: A History of Design and Circumstance. New York: Alfred A. Knopf, 1990.
[Petroski 1994] Petroski, Henry. Design Paradigms: Case Histories of Error and Judgment in Engineering. New York: Cambridge University Press, 1994.
[Petroski 1995] Petroski, Henry. Engineers of Dreams: Great Bridge Builders and the Spanning of America. New York: Alfred A. Knopf, 1995.
[Petroski 1996] Petroski, Henry. Invention by Design: How Engineers Get from Thought to Thing. Cambridge, Mass.: Harvard University Press, 1996.
[Petroski 2004] Petroski, Henry. Pushing the limits: New adventures in engineering. New York: Knopf, 2004.
[Petroski 2006] Petroski, Henry. Success through failure: the paradox of design. Princeton: Princeton University Press, 2006.
[Quin 2006] Liam Quin. “Microformats: contaminants or ingredients? Introducing MDL and asking questions.” Talk delivered 10 August 2006 at Extreme markup Languages 2006, an IDEAlliance conference, Montréal. http://www.idealliance.org/papers/extreme/proceedings/html/2006/Quin01/EML2006Quin01.html.
[Wilmott 2006] Sam Wilmott. “Rethinking XSLT.” Talk delivered 9 August 2006 at Extreme markup Languages 2006, an IDEAlliance conference, Montréal. http://www.idealliance.org/papers/extreme/proceedings/html/2006/Wilmott01/EML2006Wilmott01.htmlWilmott01.