The art of the fail in engineering, same for food

The 1985 book, To Engineer is Human: The Role of Failure in Successful Design, is one of the most influential I’ve read.

I’m a huge fan of the author, Henry Petroski, the engineer and professor who at 70, is still going strong in his new book, To Forgive Design: Understanding Failure (I really need to restart writing books), reviewed in tomorrow’s New York Times.

“In May 1987 the Golden Gate Bridge had a 50th birthday party. The bridge was closed to automobile traffic so people could enjoy a walk across the spectacular span.

“Organizers expected perhaps 50,000 pedestrians to show up. Instead, by some estimates, as many as 800,000 thronged the bridge approaches. By the time 250,000 were on the bridge engineers noticed something ominous: the roadway was flattening under what turned out to be the heaviest load it had ever been asked to carry. Worse, it was beginning to sway.

“Though crowds of people do not generally walk in step, if the bridge beneath them begins to move sideways — for whatever reason — the people on it instinctively tend to fall into step the better to keep their balance,” Henry Petroski writes. “This in turn exacerbates the sideways motion of the structure, and a positive feedback loop is developed,” making matters worse and worse.

“This time disaster was averted. The authorities closed access to the bridge and tens of thousands of people, caught in pedestrian gridlock, made their way back to land, a process that for some took hours.

“The story is one of scores in “To Forgive Design: Understanding Failure,” a book that is at once an absorbing love letter to engineering and a paean to its breakdowns.

“Dr. Petroski writes, “no matter what the technology is, our best estimates of its success tend to be overly optimistic.”

“Failure is what drives the field forward.

Sometimes devices fail because they are subjected to unexpected stress, like the Golden Gate under the weight of all those people, who collectively applied far more stress than the ordinary automobile traffic the bridge was supposed to carry.

“Then there are the insulating O rings on the booster that launched the Challenger, which stiffened on an unusually cold morning in Cape Canaveral, Fla. Engineers alarmed by this issue recommended that the launch be postponed; managers overruled them.

"After the 1907 collapse of a bridge under construction in Quebec, engineers in Canada instituted a ceremony by which new graduates entering the profession received iron rings meant to remind them of their responsibilities. A variation of this practice is spreading in the United States, even as this country struggles to enhance its engineering success in the world economy."

Which would explain when I ask would-be engineers from around the world about their iron-ring, they look at me funny. In Canada, engineers proudly wear their iron rings on their pinky. That’s how you know they’re engineers.

Food safety has enough failures. Learn from failure.

Ignoring the alarm

Matthew Wald writes in the NY Times this morning that “when an oil worker told investigators on July 23 that an alarm to warn of explosive gas on the Transocean rig in the Gulf of Mexico had been intentionally disabled months before, it struck many people as reckless.

“Reckless, maybe, but not unusual. On Tuesday, the National Transportation Safety Board said that a crash last year on the Washington subway system that killed nine people had happened partly because train dispatchers had been ignoring 9,000 alarms per week. Air traffic controllers, nuclear plant operators, nurses in intensive-care units and others do the same.”

These are problems of human behavior and design in complex systems — like in a meat processing plant that collects lots of listeria samples but doesn’t act when an increase seems apparent.

If consumers and retailers have food safety recall fatigue, do producers and processors have alarm fatigue – learning to ignore rather than investigate data that may highlight a problem?

In the Maple Leaf 2008 listeria outbreak that killed 22 Canadians, an investigative review found a number of environmental samples detected listeria in the culprit plant months before the public was alerted to possible contamination and that the company failed to recognize and identify the underlying cause of a sporadic yet persistent pattern of environmental test results that were positive for Listeria spp.

Alarms and monitoring systems are established to alert humans – with all their failings – that something requires attention.

Mark R. Rosekind, a psychologist who is a member of the National Transportation Safety Board, told the Times,

“The volume of alarms desensitizes people. They learn to ignore them.”

Wald further writes,

“On the oil rig and in the Guam control tower, the operators were annoyed by false alarms, which sometimes went off in the middle of the night. At the refinery and the reactor, the operators simply did not believe that the alarms would tell them anything very important.

Wald says, “… the alarms conveyed no more urgency to these operators than the drone of a nagging spouse — or maybe the shepherd boy in Aesop’s fable, who cried “Wolf!”

So what to do? The warning systems need to be better designed delivered and continually debated throughout any organization that values a safety culture. Engineers have known this for decades when designing fail-safe systems (sic). The food sector has a lot to learn.

What engineers can teach food safety types, learn from failure

After bombing out as a genetics grad student and dabbling in journalism, I re-entered academia teaching risk analysis to engineering students at the University of Waterloo (that’s in Canada, down the road from Guelph). I taught a course called Science, Technology and Values to about 100 engineering undergrads twice a year.

I loved it.

We got to examine in real-time the assessment, management and communication failures of the 1994 Intel chip melt-down, which is now being repeated with the Apple iPhone. Engineers are big on failure analysis and figuring out ways to prevent future accidents.

The causes are usually cultural rather than technological failures.

As William J. Broad writes in the New York Times this morning, disasters teach more than successes.

While that idea may sound paradoxical, it is widely accepted among engineers.

They say grim lessons arise because the reasons for triumph in matters of technology are often arbitrary and invisible, whereas the cause of a particular failure can frequently be uncovered, documented and reworked to make improvements.

Disaster, in short, can become a spur to innovation.

Henry Petroski, a historian of engineering at Duke University and author of “Success Through Failure,” a 2006 book, said,

“It’s a great source of knowledge — and humbling, too — sometimes that’s necessary. Nobody wants failures. But you also don’t want to let a good crisis go to waste.”

What’s baffled me is that the food industry seems immune to such lessons. Or it takes forever. It took 29 outbreaks involving leafy greens before the California industry had a tipping point and decided to get serious about food safety? The same mistakes are repeated over and over and over and it’s boring (and really dangerous).

Canadian greats, The Tragically Hip, who are not engineers, just dudes from Kingston (that’s in Ontario) summed it up in their 1994 song, Titanic Terrarium:

An accident’s sometimes the only way
To worm our way back to bad decisions