Reflections on "The Illusion of Validity" on the birthday of the Federal Reserve System

Courtesy of Woodrow Wilson Birthplace Foundation Painting by Wilbur G. Kurtz, Sr. of President Wilson signing the Federal Reserve Act in 1913.

"Statistics and simple arithmetic tell us more about ourselves than expert intuition." -- Freeman Dyson

Happy 98th Birthday, Federal Reserve System?

President Woodrow Wilson signed the Federal Reserve Act on December 23, 1913, creating a central bank for the United States.

The Fed, of course, has become a source of great national controversy, eliciting books such as End the Fed by Congressman Ron Paul. Why?

World-renowned physicist and public intellectual Freeman Dyson recently composed a profound essay, How to Dispel Your Illusions, in the New York Review of Books. In it, Dyson reflects on the recent book by Nobel Economics laureate Daniel Kahneman's Thinking, Fast and Slow. Dyson implicitly targets the fatal flaw in the well-intended foundation of the Federal Reserve. Dyson:

In 1955, when Daniel Kahneman was twenty-one years old, he was a lieutenant in the Israeli Defense Forces. He was given the job of setting up a new interview system for the entire army. The purpose was to evaluate each freshly drafted recruit and put him or her into the appropriate slot in the war machine. The interviewers were supposed to predict who would do well in the infantry or the artillery or the tank corps or the various other branches of the army. The old interview system, before Kahneman arrived, was informal. The interviewers chatted with the recruit for fifteen minutes and then came to a decision based on the conversation. The system had failed miserably. When the actual performance of the recruit a few months later was compared with the performance predicted by the interviewers, the correlation between actual and predicted performance was zero.

Kahneman had a bachelor's degree in psychology and had read a book, Clinical vs. Statistical Prediction: A Theoretical Analysis and a Review of the Evidence by Paul Meehl, published only a year earlier. Meehl was an American psychologist who studied the successes and failures of predictions in many different settings. He found overwhelming evidence for a disturbing conclusion. Predictions based on simple statistical scoring were generally more accurate than predictions based on expert judgment. ...

Kahneman knew how to improve the Israeli army interviewing system. ... Statistics and simple arithmetic tell us more about ourselves than expert intuition.

Reflecting fifty years later on his experience in the Israeli army, Kahneman remarks in Thinking, Fast and Slow that it was not unusual in those days for young people to be given big responsibilities. The country itself was only seven years old. “All its institutions were under construction,” he says, “and someone had to build them.” He was lucky to be given this chance to share in the building of a country, and at the same time to achieve an intellectual insight into human nature. He understood that the failure of the old interview system was a special case of a general phenomenon that he called “the illusion of validity.” At this point, he says, “I had discovered my first cognitive illusion.”

Cognitive illusions are the main theme of his book. A cognitive illusion is a false belief that we intuitively accept as true. The illusion of validity is a false belief in the reliability of our own judgment. The interviewers sincerely believed that they could predict the performance of recruits after talking with them for fifteen minutes. Even after the interviewers had seen the statistical evidence that their belief was an illusion, they still could not help believing it. Kahneman confesses that he himself still experiences the illusion of validity, after fifty years of warning other people against it. He cannot escape the illusion that his own intuitive judgments are trustworthy.

Dyson continues, drawing from his own experience:
An episode from my own past is curiously similar to Kahneman's experience in the Israeli army. I was a statistician before I became a scientist. At the age of twenty I was doing statistical analysis of the operations of the British Bomber Command in World War II. ... Air Vice Marshal Sir Ralph Cochrane was the commander of 5 Group, the most independent and the most effective of the groups. Our bombers were then taking heavy losses, the main cause of loss being the German night fighters.

Cochrane said the bombers were too slow, and the reason they were too slow was that they carried heavy gun turrets that increased their aerodynamic drag and lowered their operational ceiling. Because the bombers flew at night, they were normally painted black. Being a flamboyant character, Cochrane announced that he would like to take a Lancaster bomber, rip out the gun turrets and all the associated dead weight, ground the two gunners, and paint the whole thing white. Then he would fly it over Germany, and fly so high and so fast that nobody could shoot him down. Our commander in chief did not approve of this suggestion, and the white Lancaster never flew.

The reason why our commander in chief was unwilling to rip out gun turrets, even on an experimental basis, was that he was blinded by the illusion of validity. This was ten years before Kahneman discovered it and gave it its name, but the illusion of validity was already doing its deadly work. All of us at Bomber Command shared the illusion. We saw every bomber crew as a tightly knit team of seven, with the gunners playing an essential role defending their comrades against fighter attack, while the pilot flew an irregular corkscrew to defend them against flak. An essential part of the illusion was the belief that the team learned by experience. As they became more skillful and more closely bonded, their chances of survival would improve.

When I was collecting the data in the spring of 1944, the chance of a crew reaching the end of a thirty-operation tour was about 25 percent. The illusion that experience would help them to survive was essential to their morale. After all, they could see in every squadron a few revered and experienced old-timer crews who had completed one tour and had volunteered to return for a second tour. It was obvious to everyone that the old-timers survived because they were more skillful. Nobody wanted to believe that the old-timers survived only because they were lucky.

At the time Cochrane made his suggestion of flying the white Lancaster, I had the job of examining the statistics of bomber losses. I did a careful analysis of the correlation between the experience of the crews and their loss rates, subdividing the data into many small packages so as to eliminate effects of weather and geography. My results were as conclusive as those of Kahneman. There was no effect of experience on loss rate. So far as I could tell, whether a crew lived or died was purely a matter of chance. Their belief in the life-saving effect of experience was an illusion.

The demonstration that experience had no effect on losses should have given powerful support to Cochrane's idea of ripping out the gun turrets. But nothing of the kind happened. As Kahneman found out later, the illusion of validity does not disappear just because facts prove it to be false. Everyone at Bomber Command, from the commander in chief to the flying crews, continued to believe in the illusion. The crews continued to die, experienced and inexperienced alike, until Germany was overrun and the war finally ended.

Kahneman's mordant recognition of "the illusion of validity" mirrors, of course, one of the key insights of another Nobel economics laureate: Hayek. In his prize acceptance speech Hayek famously indicted "the pretense of knowledge." Hayek:

We have indeed at the moment little cause for pride: as a profession we have made a mess of things. ...

The theory which has been guiding monetary and financial policy during the last thirty years, and which I contend is largely the product of such a mistaken conception of the proper scientific procedure, consists in the assertion that there exists a simple positive correlation between total employment and the size of the aggregate demand for goods and services; it leads to the belief that we can permanently assure full employment by maintaining total money expenditure at an appropriate level. Among the various theories advanced to account for extensive unemployment, this is probably the only one in support of which strong quantitative evidence can be adduced. I nevertheless regard it as fundamentally false, and to act upon it, as we now experience, as very harmful.

The conflict between what in its present mood the public expects science to achieve in satisfaction of popular hopes and what is really in its power is a serious matter because, even if the true scientists should all recognize the limitations of what they can do in the field of human affairs, so long as the public expects more there will always be some who will pretend, and perhaps honestly believe, that they can do more to meet popular demands than is really in their power. It is often difficult enough for the expert, and certainly in many instances impossible for the layman, to distinguish between legitimate and illegitimate claims advanced in the name of science. ...

But it is by no means only in the field of economics that far-reaching claims are made on behalf of a more scientific direction of all human activities and the desirability of replacing spontaneous processes by "conscious human control." If I am not mistaken, psychology, psychiatry, and some branches of sociology, not to speak about the so-called philosophy of history, are even more affected by what I have called the scientistic prejudice, and by specious claims of what science can achieve. ...

If man is not to do more harm than good in his efforts to improve the social order, he will have to learn that in this, as in all other fields where essential complexity of an organized kind prevails, he cannot acquire the full knowledge which would make mastery of the events possible. He will therefore have to use what knowledge he can achieve, not to shape the results as the craftsman shapes his handiwork, but rather to cultivate a growth by providing the appropriate environment, in the manner in which the gardener does this for his plants. There is danger in the exuberant feeling of ever-growing power which the advance of the physical sciences has engendered and which tempts man to try, "dizzy with success," to use a characteristic phrase of early communism, to subject not only our natural but also our human environment to the control of a human will. The recognition of the insuperable limits to his knowledge ought indeed to teach the student of society a lesson of humility which should guard him against becoming an accomplice in men's fatal striving to control society — a striving which makes him not only a tyrant over his fellows, but which may well make him the destroyer of a civilization which no brain has designed but which has grown from the free efforts of millions of individuals.

Dyson: "The white Lancaster never flew." " The crews continued to die...." "The illusion of validity is a false belief in the reliability of our own judgment. The interviewers sincerely believed that they could predict the performance of recruits after talking with them for fifteen minutes. Even after the interviewers had seen the statistical evidence that their belief was an illusion, they still could not help believing it. Kahneman confesses that he himself still experiences the illusion of validity, after fifty years of warning other people against it. He cannot escape the illusion that his own intuitive judgments are trustworthy." Hayek: "...against becoming an accomplice in men's fatal striving to control society — a striving which makes him not only a tyrant over his fellows, but which may well make him the destroyer of a civilization...."

The Greeks taught us that after Hubris -- overweening pride -- comes the goddess Nemesis: retribution. Dyson, Kahneman, Hayek reteach this tragic lesson. Lehrman, whose Institute hosts this website, for over half a century been quietly preaching the humility -- a humble reliance on something very like "statistics and simple arithmetic," eschewing the "illusion of validity" reposed in expert judgment -- that endows the gold standard with its great benevolent power to create economic security and foster prosperity.

Now, as noted here, here, and here, the Bank of England itself has demonstrated beyond a shadow of a doubt that the world dollar standard is vastly inferior -- in terms both of stability and prosperity -- than even the dilute form of the gold standard established at Bretton Woods -- the monetary experts still cannot "escape the illusion that [their] own intuitive judgments are trustworthy." Happy Birthday, Federal Reserve. High time for your transformation into an institution much like the National Institute of Standards and Technology, one devoutly devoted entirely to keeping the value of the dollar defined to a weight of gold fixed, in precision, to the atomic level, by following the "rules of the game as codified" in Lewis E. Lehrman's The True Gold Standard -A Monetary Reform Plan without Official Reserve Currencies.