Computers, Automation and the Danger of Forgetting How Things Really Work

A recent article by newspaper columnist D. L. Stewart ("Computer Reliance Pecks at Basic Skills", Tribune Media Services, as published in The Intelligencer, Doylestown, PA, July 2, 1995) described his experience in purchasing item costing $1.97 using a 25 cent off coupon. The checkout computer did not have a button to press to handle this computation, and the clerk was unable to subtract 25 from 197 and get the correct answer. The clerk finally found a button that he thought might do the job, and told Mr. Stewart that the cost was $1.85. This story reminded me of a similar experience I once had when I went into a fast food restaurant on the day when the cash registers with the pictures of hamburgers and French fries weren't working. The scene was rather chaotic because many of the workers could not do the calculations manually.

However, this commentary is not intended to lament the sorry state of education or to complain that young people today can't even do the simple math that we could all do in our heads in the "good old days". I can actually understand the problems that the clerks had when the computer they relied upon failed. I believe that these stories represent simple examples of a serious issue which should concern us all in the operation of automated manufacturing plants, and in the use of computer models in plant design. The key question is "does the user of the computer tool understand what the computer is really doing?" This knowledge is essential to understanding when an answer doesn't make sense, and to knowing when the use of the particular model embedded in the software may be inappropriate.

Here are a few examples which I have personally observed or which have been reported to me by colleagues:

• The results of a toxic gas dispersion analysis were reported to 16 decimal places.

• A newspaper article described an emergency response system that allows a company to "leave the written plans on the shelf and rely on the computer to spell out the appropriate course of action." Does anybody know what to do on the day the computer fails? Does anybody understand the models the computer is using to spell out its recommendations?

• The probability of rupture of a storage tank was estimated by fault tree analysis to be about 1 in 10¹⁸ years. Compare this to the estimated age of the universe of about 15x10⁹ years; I am confident that all of the rupture scenarios which were excluded from the analysis were orders of magnitude more important than those that were included and analyzed.

• Emergency planners tried to use the results of a toxic gas dispersion model to precisely determine where people would be impacted, ignoring the fact that the plant is at the base of a 2000 foot high mountain, and the models are for flat terrain.

• A process operator watched a computer controlled process cycle through a batch distillation in 20 minutes and go on to the next step, even though the distillation normally took 12 hours (a vapor temperature sensor used to control the distillation was miscalibrated.)

The common thread to all of these examples, including Mr. Stewart's and my experiences with store clerks, is that a person is using a computer tool without understanding what the tool is actually doing. The user is capable of entering data, telling the computer to manipulate that data and observing an answer. He does not understand how the tool really works, and therefore does not understand its limitations and assumptions. He is also incapable of recognizing and questioning an answer that "doesn't make sense" to someone who understands how the computer tool works.

How many users of computer tools are in the same position as the native fireman on an African river steamboat as described by Marlowe, the narrator and protagonist in Joseph Conrad's Heart of Darkness (1899):

"A few months of training had done for that really fine chap....He was useful because he had been instructed; and what he know was this - that should the water in that transparent thing disappear, the evil spirit inside the boiler would get angry through the greatness of his thirst, and take a terrible vengeance." (1)

Because he does not understand the operation of the boiler, the fireman's ability to recognize dangers is limited to the few areas where he has been trained. He is unable to respond to a new and unique situation.

I will acknowledge that for most of my examples the same error could have been made with manual calculations or operations. But I really believe that it is less likely. In these examples the availability of an "easy" computer tool allowed a user to do something that he really did not understand. And most of the time he could probably do it successfully, because the tool was being used within its design parameters. This could even create an illusion of greater understanding than actually exists. But if the user is not aware of the design constraints, he will not be able to recognize when he has gone beyond them. In the case of manual calculations or operations, the continuing interaction between the user and the tool will lead to learning and understanding of the model. Furthermore, the magnitude of the calculations would be likely to discourage the user from attempting to do calculations or operations which he does not really understand.

This is not a new concern, but it is increasing in importance as computers become cheaper and more powerful, and software becomes easier to use and more "user friendly". The American Society of Civil Engineers expressed this concern in 1984 (as quoted by Henry Petroski in his 1984 book To Engineer is Human):

"Many have predicted that the engineering failures of the future will be attributed to the use or misuse of computers. Is it becoming easy to take on design work outside of the engineer's area of expertise simply because a software package is available? How can civil engineers guarantee the accuracy of the computer program, and that the engineer is qualified to use it properly?"

The concern applies equally to chemical engineering and all other engineering fields.

I do not have the answer to these issues. Obviously, computer design and process control tools offer tremendous advantages in capability, efficiency and productivity, and I would never advocate going back to the old, manual tools. One thing that is clear to me is that education and training become even more important when you are using an "easy" computer design tool or plant control system. Manual design (or plant operation) provides a significant "on the job" training as the engineer (operator) works through the design (operation) of the plant. This "hands on" training is lost when the computer does much of the actual work. Furthermore, we must recognize that professional responsibility and ethics require that we understand our design tools and their limitations, and that we do not attempt design projects outside our expertise and knowledge.

(1) The "evil spirit" in steam boilers has "taken a terrible vengeance" many times since 1899. One notable occasion was on March 28, 1979 at Three Mile Island, when loss of feed water to the steam generator was one of the events that lead to the accident.