No decision about me without me
From G. Gigerenzer, Risk Savvy. Penguin (Draft, cite only with permission from the author)
If you haven’t had a mammogram,
you need more than your breast examined.
American Cancer Society, 1980s campaign poster
We want the principle of ‘shared decision making’ to
become the norm: no decision about me without me.
Department of Health, UK
While running for president of the United States, former New York City mayor Rudy Giuliani said in a 2007 campaign advertisement:
I had prostate cancer, 5, 6 years ago. My chance of surviving prostate cancer – and thank God, I was cured of it – in the United States? Eighty-two percent. My chance of surviving prostate cancer in England? Only 44 percent under socialized medicine.
For Giuliani, this meant that he was lucky to be living in New York and not in York, since his chances of surviving prostate cancer appeared to be twice as high. That was big news. It was also a big mistake. Despite the impressive difference in survival, the number of men who died of prostate cancer was about the same in the US and the UK. How can survival be so different when mortality is the same?
The answer is that when it comes to screening, differences in survival rates don’t tell us anything about differences in mortality rates. In fact, over the past 50 years, changes in 5-year survival for the most common solid tumors had no connection with changes in mortality. There are two reasons.
How Rudi Giuliani Was Misled
The first reason is called lead-time bias. Imagine two groups of men with invasive prostate cancer. The first do not attend screening for prostate-specific antigens (PSA); the second do. All die of cancer at age 70. The first group could consist of men in Britain, where PSA testing is not routinely used and most cancer is diagnosed by symptoms, and the second of men in the US, where using the test began in the late 1980s and spread rapidly, despite the lack of evidence that it saves lives.
In the British group, prostate cancer is detected by symptoms, say at age 67 (Figure 10-1 top). All of these men die at age 70. Everyone survived only 3 years, so the 5-year survival is zero percent. In the U.S. group, prostate cancer is detected early by PSA tests, say at age 60, but they too die at age 70 (Figure 10-1 bottom). According to the statistics, everyone in that group survived ten years and thus their 5-year survival rate is 100%. The survival rate has improved dramatically, although nothing has changed about the time of death: Whether diagnosed at age 67 or at age 60, all patients die at age 70. Survival rates are inflated by setting the time of diagnosis earlier. No life is prolonged or saved.
The second reason why survival rates tell us nothing about living longer is overdiagnosis bias. Overdiagnosis happens when doctors detect abnormalities that will not cause symptoms or early death. For instance, a patient might correctly be diagnosed with cancer but because the cancer develops so slowly, the patient would never have noticed it. These cancers are called slow-growing or non-progressive cancers. PSA screening detects both kinds of cancer. But the test, like most other cancer screening tests, cannot tell the difference. Figure 10-2 (top) shows 1,000 British men with progressive cancer who do not undergo screening. After 5 years, 440 are still alive, which results in a survival rate of 44%. Figure 10-2 (bottom) shows 1,000 Americans who participate in PSA screening and have progressive cancer. The test, however, also finds 2,000 people with non-progressive cancers – meaning that they will not die from them. By adding these 2,000 to the 440 who survived progressive cancer, the survival rate leaps to 81%. Even though the survival rate increases dramatically, the number of men who die remains exactly the same.
Talking about survival can be useful for surgery or other medical treatments (where there is no possibility of setting the time of diagnosis earlier or of overdiagnosing patients) but in the context of screening, survival is always a misleading message. Misunderstanding what survival means has unnecessarily turned healthy lives into troubled ones and people into patients. Many a man whose non-progressive cancer is detected undergoes unnecessary and harmful surgery or radiation treatment. The treatment is unnecessary because he would never have noticed the small cancer in his lifetime. And it is harmful because every other man ends up incontinent or impotent for the rest of his life. The number of men who spend the rest of their lives wearing Pampers is legion. Many have been misled to believe that incontinence is the price they had to pay for survival, and that survival means living longer.
More Men Die With Prostate Cancer Than From It
Prostate cancer is not a deadly strike out of the blue. In fact, it is highly common. Consider a group of five American men in their 50s. One of them is likely to have some form of prostate cancer (Figure 10-3). When these men are in their 60s and 70s, two or three are expected to have prostate cancer, and when they are in their 80s, four of the five will. Almost every man lucky enough to live a long life will eventually get it. Yet most will never notice because the cancer grows only slowly or not at all. Fortunately, the lifetime risk of dying from prostate cancer is only about 3 percent. More men die with prostate cancer than from it.
Giuliani is not the only politician who failed to appreciate the difference between survival rates and mortality rates. A report by the U.K. Office for National Statistics on cancer survival trends noted that 5-year survival for colon cancer was 60% in the US compared to 35% in Britain. Experts dubbed this finding ‘‘disgraceful’’ and called for government spending on cancer treatment to be doubled. In response, then-Prime Minister Tony Blair set a target to increase survival rates by 20% over the next 10 years: ‘‘We don’t match other countries in its prevention, diagnosis and treatment.’’ Despite these large differences in 5-year survival, the mortality rate for colon cancer in Britain is about the same as the rate in the US. Higher survival does not mean longer life or a better health system. Nevertheless, Giuliani himself appears to be absolutely convinced that the operation saved his life, as do other U.S. politicians, including John Kerry. In the late 1990s, Congress initiated a postal stamp featuring “Prostate Cancer Awareness,” which promoted “annual checkups and tests.” Giuliani and the U.S. Postal Service were obviously of one mind. To this day, celebrities recount in advertisements their personal stories about how early detection saved their lives.
Is there a way to reach people with the facts? There is. It’s a simple tool called the icon box.
Screening for Prostate Cancer
An icon box brings transparency into health care. It can used to communicate facts about screening, drugs, or any other treatment. The alternative treatments are placed directly next to each other in columns and the benefits and the harms listed row by row. Most importantly, no misleading statistics such as survival rates and relative risks are allowed to enter the box. All information is given in plain frequencies. The aim is not to tell people what to do, but to provide the main facts so that everyone can make an informed decision. An icon box brings light into the darkness of persuasion and is an antidote to paternalism.
Let’s contrast Giuliani’s statement with an icon box for prostate cancer. The box summarizes the results from all medical studies using the best evidence from randomized trials. In these trials, about half of the men were randomly assigned to a group who were screened for prostate cancer with PSA testing and digital rectal exam, the other half to the control group who received no screening. If a cancer was detected in the first group by screening or in the control group by symptoms, then the patients typically underwent cancer treatment. The box shows what happened to the men after 10 years.
What’s the Benefit?
Let’s begin with the positive aspect, the benefit. First, is there any evidence that early detection reduces the number of deaths from prostate cancer? The answer is no. No matter whether men went for screening or not, there was no difference in the number of men who died of prostate cancer. Second, is there evidence that detecting cancer at an early stage reduces the total number of deaths from any cause whatsoever? Again no. In the course of ten years, a fifth of the men who did not undergo screening died, as did a fifth of those who faithfully went for screening. Simply stated, there is no evidence that early detection saves lives; it reduced neither prostate cancer mortality nor total mortality.
Why then do many men, like Giuliani, believe that early detection saved their lives? They are likely among the two out of every 100 men in Figure 10-4 (right side) who have non-progressive cancer, that is, a form of cancer that would never have caused symptoms. After surgery or radiation therapy, every one of these men might think that their lives were saved by the test. But without the test and treatment they would in fact also be alive – and in better health.
What’s the Harm?
The icon box also shows the harms that occurred to men who went for screening. There are two kinds of harms: for men without prostate cancer and for men with prostate cancer that is non-progressive. When a man without cancer repeatedly has a high PSA level, doctors typically do a biopsy. But unlike a mammogram, a PSA test does not tell the doctor where to insert the needle. As a result, men are often subjected to the nightmare of multiple needle biopsies in search of a tumor that is not there in the first place. These false alarms occur frequently because many men without cancer have high PSA levels. Out of 100 men who participated in screening, about 18 experienced one or more false alarms with biopsy.
Men with non-progressive prostate cancer suffered even more. If a biopsy showed any signs of cancer, most were pushed into unnecessary treatments, such as prostatectomy and radiation therapy. As the icon box shows, out of every 100 men who underwent screening, 2 were treated unnecessarily. Giuliani is likely one of those. About half of them ended up incontinent or impotent. Altogether, 20 men are harmed by unnecessary biopsies, surgery, or radiation. For these unfortunate men, early detection actually lowered the quality of their lives, without extending its length.
After a talk I gave to a health organization, a man approached me and introduced himself as the CEO of a bank management institution. “The story you told is my story. My doctor always advised me not to take PSA tests, and so did my father’s doctor. But then there was a substitute doctor and he did a PSA test, cancer was detected, and then what happened was exactly what you said. Now I’m 60, and I’m running to the bathroom all the time.” An icon box can save you from incontinence and impotence.
We are told again and again that if you find prostate cancer early, your life might be saved. A plausible story, but it’s on shaky ground. Because there is no evidence that lives are saved and strong evidence that some men are harmed, the National Cancer Institute explicitly recommend that men without any symptoms not be screened routinely, and its website (www.cancer.gov) states that men should consider benefits and harms before making a decision. Nor does the U.S. Preventive Services Task Force recommend routine PSA tests or digital rectal exams. Recall that Richard Ablin, the discoverer of PSA, himself spoke out against routine PSA screening. Even so, many hospitals and doctors adopt a policy of automatic screening or persuade men into screening. They do this for three familiar reasons, the SIC syndrome: Many doctors and clinics protect themselves against patients who might sue by recommending screening. Others do not know the medical evidence. Out of a random sample of 20 Berlin urologists, only 2 knew the benefits and harms of PSA screening. And the profit that can be made is substantial. In the US alone, some $3 billion are spent every year on performing the test and follow-ups; if the entire U.S. male population were screened, the first year would cost taxpayers $12 to $28 billion. For comparison, a WHO expert panel concluded that getting rid of measles worldwide would cost only half as much.
Prostate cancer screening is a prime example of how time and money are wasted in health care. Doctors could use the time for helping instead of harming patients. The billions wasted could be used for saving lives instead. When I presented the icon box on PSA screening during a talk at the annual conference of private health insurers, the head of one company approached me afterwards. “You are spoiling our business plan,” he said, decidedly annoyed. “To get an edge over the state-run health insurance plans, we pay for PSA tests. Now you show evidence that it’s for no good.” Why don’t they offer their clients something that helps rather then hurts them?
Let’s have a closer look at two of the reasons why this game persists: doctors’ flawed understanding of health statistics and money-driven conflicts of interest.
Do Doctors Understand Survival Rates?
Do doctors understand survival rates, or are they fooled just like Rudi Giuliani? As far as I can tell, no study ever asked this question. For that reason Odette Wegwarth, I,and some other coleagues of mine conducted a national sample of 412 physicians in the United States. These doctors practiced family medicine, general medicine, and internal medicine. Most of them had ten to twenty years experience. The first question was:
Which of the following prove that a screening test “saves lives” from cancer?
1. More cancers are detected in screened populations than in unscreened populations.
2. Screen-detected cancers have better 5-year survival rates than cancers detected because of symptoms.
3. Mortality rates are lower among screened persons than unscreened persons in a randomized trial.
What do you think? Almost half (47%) of the doctors mistakenly believed that (1), detecting more cancers, proves that the test saves lives. What they didn’t consider is that all screening tests of any value on the market will detect cancers. And, as the icon box for prostate cancer screening demonstrates, early detection is not always a godsend for patients. Although a few kinds of cancer screening such as pap smears can save lives, most save none and harm large numbers of people; some can even have fatal results. Within one month after lung cancer surgery, five percent of patients are no longer alive. That does not stop advertising campaigns for lung cancer screening, such as with spiral CTs. The point is not that spiral CTs are no good. In fact, they are too good. They are so accurate that they detect about as many lung cancers in non-smokers as in smokers! In other words, they detect non-progressive cancers in non-smokers – abnormalities that, technically speaking, are cancer but do not lead to symptoms. Detecting more cancers is not proof that lives are saved. Every doctor should know this elementary fact.
Like Rudi Giuliani, 76 percent of the doctors falsely believed that (1), higher 5-year survival rates, proves that lives are saved. The majority of them (81 percent) also believed that lower mortality rates prove that lives are saved; this time they finally got it right. But if most doctors believe that both survival and mortality rates prove the same thing, there must be some confusion. We tested this in our next question.
A result can be phrased either as a survival or a mortality rate. Will this influence what doctors recommend to patients? Doctors were asked:
Imagine that a 55-year-old healthy patient asks you about a screening for Cancer X. Answer the following questions based on data for patients age 50 to 69, which come from a large trial of U.S. adults that lasted about 10 years.
Without screening test
With screening test
|5-year survival rate||
Would you recommend this screening test to your patient?
The majority of physicians (nearly 70%) said that they would definitely recommend the test to their patients. Later, doctors were asked the same question about screening for Cancer Z, which was in fact based on the numbers for Cancer X. But now the information was provided in mortality rates:
Without screening test
With screening test
|Mortality: Risk of dying from Cancer Z over 5 years||
2 deaths per 1000 people
1.6 deaths per 1000 people
This time, only 23% said they would recommend the test (Figure 10-5). It was disturbing to see that the majority of doctors in the US can be easily persuaded by survival rates.
Are doctors in Germany any better? When we asked 65 German internal medicine physicians, they too could be easily manipulated into recommending screening.  We also asked these physicians to explain what lead-time bias is (see Figure 10-1). Only two of the 65 could do it. And when we asked them about overdiagnosis (see Figure 10-2), not a single one could explain it.
All in all, most of the doctors in Germany and the US drew wrong conclusions from survival rates. In addition, almost half of the U.S. doctors falsely believed that detecting more cancers proves that lives are saved. Under the influence of their confusion, they would recommend screening to patients. To change this to the better, icon boxes should be standard in reports on medical treatments.
How Prestigious Institutions Fool You
Comparing Apples and Pears
As long as nobody notices, institutions can twist information to influence doctors and patients. What would you do to win over patients? One trick is shown in Figure 10-6. See if you can catch it before continuing on. The ad is for MD Anderson, one of the most prestigious cancer centers in the US. According to it, survival has steadily increased at the clinic over the years in comparison to the much lower nationwide rates. The Center claims: “As national mortality rates for prostate cancer fluctuated between 1960 and 1990, five year survival rates for prostate cancer among MD Anderson patients continued to improve.” This increase in survival is presented as an “overall increase in longevity.” And there we have it: Survival is equated with living longer, and the increase in survival rates is compared to nationwide mortality rates. But unlike us, many of the ad’s readers who have never heard of lead-time bias and overdiagnosis will be fooled into believing that the Center has made considerable progress in the war against cancer. We have already encountered this commonly used trick in another form, double-tonguing. Here it means reporting survival rates to make one’s own success look amazing, and then reporting mortality rates to make others’ success look small or nil.
Conflicts of Interest
Why is the public being so blatantly deceived? I think there are two quite different groups to blame. The first consists of true believers in screening who don’t want to see the scientific evidence or don’t understand it. The second group is commercially motivated and has a business plan that puts patients at risk for unnecessary treatments. These two groups often cohabit peacefully: If the one avoids the evidence, the other has a better conscience about its business plan. One scheme is to offer screening at sharply reduced costs or even for free in order to make it a “loss leader.” It’s the same strategy supermarkets and other stores use: to price an item well below costs, in order to attract customers and stimulate more profitable sales later. In the words of oncologist Dr. Otis Brawley, now Chief Medical Officer of the American Cancer Society, in his former position as director of the Georgia Cancer Center at Emory:
We at Emory have figured out that if we screen 1,000 men at the North Lake Mall this coming Saturday, we could bill Medicare and insurance companies for $4.9 million in health care costs [for biopsies, tests, prostatectomies, etc.]. But the real money comes later – from the medical care the wife will get in the next three years because Emory cares about her man, and from the money we get when he comes to Emory’s emergency room when he gets chest pain because we screened him three years ago. …
We don’t screen anymore at Emory, once I became head of Cancer Control. It bothered me, though, that my P.R. and money people could tell me how much money we would make off screening, but nobody could tell me if we could save one life. As a matter of fact, we could have estimated how many men we would render impotent … but we didn’t. It’s a huge ethical issue.
We need more directors like Dr. Brawley. But we also need more risk savvy patients. A clinic that offers PSA tests to uninformed men without providing an icon box can only win. If the test result is negative, the patient is reassured. If the test result is positive, the patient is grateful that the clinic detected the cancer early, even if he soon ends up incontinent. As Dr. Brawley noted, profiting from uninformed patients is unethical. Medicine should not be a money game.
What Do Men Know?
How knowledgeable are people about PSA testing? To find out, we asked 5,000 men in nine European countries: Austria, France, Germany, Italy, the Netherlands, Poland, Spain, the UK, and the European part of Russia.
The British came up last among the Europeans: Almost 99 percent of British men overestimated the benefit vastly or did not know (Figure 10-7). In fact, a fifth of them believed that 200 of every 1,000 men would be saved (!). Their huge miscalculation is likely due to a widely publicized study on screening that was touted around the globe as having found a “20 percent mortality reduction” from prostate cancer. Yet this impressive 20 percent is nothing but a reduction from 3.7 to 3.0 in every 1,000 men. Quite a number of British men seem to have fallen into the trap. Once again, the relative-risk trick that was played on British women in the contraceptive pill scare (covered in Chapter 1) was successful. For the men, it generated vast hopes, for the women huge fear.
In France, the Netherlands, Poland, and Germany, a somewhat higher percentage of men were well informed. But these numbers were extremely low as well: A maximum of six percent give realistic estimates of the benefit. Across all European countries, 89 percent of men overestimated the benefits of prostate cancer screening. The country that did best was Russia – “only” 77 percent of Russian men overrated the benefit. The reason is probably not that Russians get more information, but that they get less misleading information.
Did those who more often consulted their GPs, health pamphlets, or the Internet for medical information make better estimates? Not at all. The Internet did not help; it offers easy access to too much misleading information. Frequent consultation of doctors and pamphlets also failed to improve understanding and was even slightly associated with people’s overestimating the benefit.
I don’t know of a similar study in North America or Asia, but I would be surprised if men elsewhere were better informed. In fact, a national telephone survey of U.S. adults reported that the majority were “extremely confident” in their decision about prostate, colorectal, and breast screening, believed they were well-informed, but could not correctly answer a single knowledge question. Few men are in a position to make informed decisions about screening; most behave like the economists in Chapter 4 and just do whatever their doctors or wives tell them.
Screening for Breast Cancer
I have chosen prostate cancer screening to demonstrate the incredible mass of misinformed patients and doctors. It is unfortunately the rule rather than the exception. Most people have equally inflated beliefs about the effectiveness of other cancer screening, drugs, and treatments. The tricks are always the same: survival, relative risks, double-tonguing, and others. In each case, icon boxes can work to get patients’ heightened expectations back on solid ground. Consider breast cancer screening. In the good old days of male chauvinism, the American Cancer Society could simply say:
“If you haven’t had a mammogram, you need more than your breast examined”.
Today, that’s fortunately no longer acceptable. But although the Society has stopped making overt jokes about female brains, the paternalist attitude towards women hasn’t changed much. Many women complain to me about the emotional pressure they get from their doctors. “You still don’t go screening? Be responsible for your body and think of your children.” And across all Western countries, women are still often treated like children: told what to do, but not given the facts needed for an informed decision. A delicate pink health leaflet by the Arkansas Foundation for Medical Care explains:
Why should I have a mammogram? Because you’re a woman.
After this deeply informative answer, women get zero information in the rest of this leaflet about benefits and harms that would let them make up their own minds. Instead they are told to tell every other woman to do what they were told to do: “That’s why you should ask your mothers, sisters, daughters, grandmothers, aunts, and friends to have mammograms, too.”
How can women be empowered to make an informed decision? A fact box is one means.  Unlike the icon box for prostate cancer screening, it uses numbers instead of icons. These are all stated as simple frequencies, no relative risks or misleading 5-year survival rates. The fact box is based on the evidence from available randomized trials, here with women age 50 and older who were divided at random into two groups: One group was screened for breast cancer regularly, the other was not screened. The box shows what happened to the women after 10 years.
What’s the Benefit?
Let’s first look at the positive side, the potential benefits. There are two questions a woman might ask: First, is there evidence that mammography screening reduces my chance of dying from breast cancer? The answer is yes. Out of every 1,000 women who did not participate in screening, about 5 died of breast cancer, while this number was 4 for those who participated. In statistical terms that is an absolute risk reduction of 1 in 1,000. But if you find this information in a newspaper or brochure, it is almost always presented as a “20 percent risk reduction” or more. Second, is there evidence that mammography screening reduces my chance of dying from any kind of cancer, including breast cancer? The answer is no. The studies showed that of every 1,000 women who participated in screening, about 21 died of any kind of cancer, and the number was the same for women who did not participate.
In plain words, there is no evidence that mammography saves lives. One less woman in 1,000 dies with the diagnosis breast cancer, but one more dies with another cancer diagnosis. Some women die with two or three different cancers, where it’s not always clear which of these caused death. That is why the total cancer mortality (including breast cancer) is the more reliable figure. That’s the number you will have a hard time finding. After all, what would women say if they learned that they had been asked for years to attend mammography screening without any evidence that it saves lives?
What’s the Harm?
Now let’s look at the harms. First, women who do not have breast cancer can experience false alarms and unnecessary biopsies. This happened to about 100 out of every 1,000 women who participated in screening. Legions of women have suffered from this procedure and the related anxieties. After false alarms, many of them worried for months, developing sleeping problems and changed relationships with family and friends. Second, women who do have breast cancer, but a non-progressive or slowly growing form that they would never have noticed during their lifetime, often undergo lumpectomy, mastectomy, toxic chemotherapy, or other interventions that have no benefit for them, leading only to a lower quality of life. This happened to about 5 women out of 1,000 who participated in screening. Apart from temporary harms such as baldness, nausea, and anemia, chemotherapy can lead to long-term fatigue, premature menopause, and heart damage.
The fact box on mammography screening allows women to make informed decisions based on their personal values. When I showed the box to a gynecologist, he stopped his paternalistic treatment of women and began to use it. He told me that a third of his patients looked at the facts and said “no way,” another third said “not now, but let’s discuss this again in five years,” and the others decided to participate in screening. While the fact box clearly shows that there is no good reason to push women into screening, my point is exactly not to replace the old paternalistic message with a new one by telling women not to go to screening. Every woman who wants to make her own decision should get the facts she needs – without being told what to do.
The fact box is not like a package insert that lists all possible side effects in miniscule print just to protect the company from being sued. It is simple and transparent. There are other potential harms the fact box does not list because there is no reliable evidence for them. For instance, a few women might get cancer from the X-rays; crude estimates are between 1 and 5 in 10,000. And many women also experience pain from the squeezing of their breasts while being X-rayed.
Does Every Gynecologist Understand the Benefit of Screening?
The benefit of screening is typically presented as a “25 percent reduction of breast cancer mortality.” This is the same as the 20 percent figure mentioned above, only rounded up. In another CME session I asked a group of 150 gynecologists what this figure actually means.
How many fewer women are likely to die of breast cancer? What is the best answer?
1. 1 [66%]
2. 25 [16%]
3. 100 [ 3%]
4. 250 [15%]
The numbers in the brackets show the percentage of gynecologists who chose that answer. Two thirds understood that the best answer is 1 in 1,000. Yet the others did not. Most of these believed that 250 fewer women die, which happens when relative risks are confused with absolute risks, or that 25 will be saved, which results from an additional error in arithmetic. After the CME session, in which they learned the difference between relative and absolute risks, everyone understood that 25 percent means 1 in 1,000 – all except one doctor, who stubbornly insisted on Answer #4.
Some years ago, I was interviewed by the editor of a medical journal who was also the head of a teaching hospital in Switzerland. He was surprised to hear that some doctors don’t understand the benefit of mammography screening. After the interview, he asked the 15 gynecologists in his own department what the 25% figure means. One physician thought that the figure meant 2.5 out of 1,000, another 25 out of 1,000; the total range was between 1 and 750 out of 1,000. Relative risks confuse not only the general public but some doctors as well.
Do female gynecologists themselves participate in mammography screening? In that particular CME group, they were divided. Among those in their 40s and 60s, around half of them did, while 82 percent of those in their 50s participated. When I checked their knowledge about the harms, most did not know the results of the scientific studies and had never seen a fact box.
The writer Barbara Ehrenreich expressed her unhappiness about the infantilizing breast-cancer industry with its pink ribbons, teddy bears, and relentless cheerfulness. Has feminism been replaced by the pink-ribbon breast cancer cult? she asked. When First Lady Laura Bush traveled to Saudi Arabia in 2007, what vital issue did she take up with the locals? Not women’s right to drive, vote, or leave the house without a man, but “breast cancer awareness.” When the House of Representatives passed the Stupak amendment, which restricts abortion rights, the response of American women ranged from muted to inaudible. A few weeks later, when the U.S. Preventive Services Task Force recommended that regular screening mammography not start until age 50, all hell broke loose. Once upon a time, committed women challenged the establishment by burning their bras. Now, far too many do what they are told to do and don’t ask questions about evidence. And when they do ask, many are taken in with just four tricks. You will recognize these by now.
Trick #1: Don’t mention that mammography screening doesn’t reduce the chance of dying from cancer. Only talk about the reduction in dying from breast cancer.
Trick #2: Tell women that screening reduces breast cancer mortality by 20 percent or more. Don’t reveal that this is the same as an absolute risk reduction of 1 in 1,000, which would sound less impressive.
Trick #3: Tell women about increased survival. For instance, “If you participate in screening and breast cancer is detected, your survival rate is 98 percent.” Don’t mention mortality.
Trick #4: Don’t tell women about unnecessary surgery, biopsies, and other harms from overtreatment. If you are asked, play these down.
These four tricks have been highly successful in shaping many women’s attitude towards mammography screening. Trick #1 is ubiquitous. Hardly a health brochure dispenses with it. Trick #2 is used in the majority of brochures worldwide. Most websites use it too, although at least one out of five does provide information in transparent absolute numbers. Tricks #3 and #4 can be found in many magazines, leaflets, and websites all across the world, as the following three examples illustrate.
Tricking Women in Three Languages
In the Spanish edition of Newsweek, Julio Frenk, former Minister of Health of Mexico and currently Dean of the Harvard School of Public Health, and his wife Felicita Knaul, economist and author of a book on breast cancer, wrote:
Only early detection prevents women from dying of breast cancer. In rich countries, where screening is common, detection is early, so treatment is more effective. The probability of five-year survival after early-state diagnosis is 98 percent.
This number is impressive. But as I explained before, high survival rates don’t tell us anything about whether or not lives are saved. It is painful to see Trick #3 used by the Dean of Harvard School of Public Health, who should know better. The proper figure would be an absolute risk reduction of 1 in 1,000, as shown in the fact box.
Susan G. Komen for the Cure is one of the largest, best-funded and most-trusted breast cancer organizations in the United States. Its logo is – take a guess – a pink ribbon. Komen has in fact dominated the pink ribbon marketplace; for instance, it paired with M&M’s to sell pink-coated candies high in sugar and fat, and with the fast food restaurant chain KFC to promote fried and grilled chicken sold in pink branded buckets, both foods possibly causing obesity and cancer. It has invested about $2 billion for breast cancer research education and advocacy, with some 100,000 volunteers working worldwide. One of its recommendations is that all women from the age of 40 onwards get a yearly mammogram. In a promotion, Komen shows the face of a woman next to a vertical red arrow on which is written: “What’s key to surviving breast cancer? You.” The arrow points to what women should do: “GET SCREENED NOW”. Under the arrow, the text says:
LESS TALK. MORE ACTION. Early detection saves lives. The 5-year survival rate for breast cancer when caught early is 98%. When it’s not? 23%.
The promotion provides no other information about benefits or harms. Similarly, a widely distributed German health brochure combined Tricks #2 and #3:
Women whose tumor has been detected in an early stage have a 5-year survival rate of over 98 percent after the operation. Studies showed that among women age 50 to 69, mammography screening reduces mortality from breast cancer by up to 30 percent.
Here, the reduction from 5 to 4 in every 1,000 women, typically presented as 20 percent, is generously rounded up to 30 percent.
Trick #4 is also played routinely. Harms are rarely mentioned, with one exception. As described in Chapter 7, Germans and Austrians are more anxious than almost everyone else about radiation, from cell phones to nuclear power plants to mammography. The potential harms from X-rays are mentioned in every Austrian pamphlet and about half of the German pamphlets, but women are assured that these are negligible.
What Do Women Know?
How does this biased information policy affect women? 5,000 women in the nine European countries mentioned earlier were asked about the benefits of mammography screening.
Once again, the French, Dutch, and Germans were among the least informed and ranked lowest among the nine countries: About 98 percent of women overestimated the benefit by a factor of ten, one hundred, or even more, or did not know (Figure 10-7). Are more British women informed? Yes, almost 100 percent more! In absolute numbers, however, that’s not even four percent of the British women. In fact, a fourth of them believed that 200 of every 1,000 women would be saved (!). Their huge miscalculation is likely due to the fact that the absolute risk reduction – from 5 to 4 in 1,000 women – has been presented to them as a “20 percent mortality reduction” from breast cancer (Trick #2). Few women knew that the right answer is around 1 in 1,000. As with PSA screening, Russians were again least prone to overestimation. And once again, those who consulted doctors and health pamphlets more frequently tended to choose a higher number. Altogether in the nine countries, 92 percent of women overestimated the benefit. Thus, few European women are in a position to make an informed choice about breast cancer screening.
Women have a right to be informed in a transparent way, but they aren’t.
[Box: Facts and fiction about cancer screening]
The Best Defense is Prevention, Not Early Detection
As the facts show, the war against cancer is not won by screening. The best defense is prevention and developing better therapy. Some screening programs misleadingly advertise themselves as cancer prevention. But prevention is not the same as early detection (screening), although the terms are often mixed up. Early detection means detecting a cancer that is already there, while prevention means reducing the chance that cancer will occur.
Since President Nixon declared the war on cancer, billions have been invested into the development of drugs. Every month, the exciting discovery of a new wonder drug is proclaimed by the media. Yet the actual results are disappointing. Drugs typically prolong life by a few weeks or months, or may even shorten it. And their side effects can dramatically decrease the quality of the last months or years of one’s life. Moreover, an analysis of 53 “landmark” publications in top journals on cancer drugs revealed that the positive effects of most (47) studies could not be replicated. In one case, the scientists who tried to replicate an effect over and over again without any success contacted the study’s lead researcher, who admitted having done the experiment six times and getting the result only once. But he put that one result into the paper because it made for a good story. Publishing only the positive result fosters exciting but shoddy science. The reason for this violation of scientific honesty is a system with the wrong incentives: publishing in prestigious journals pays, whether or not the results are correct. Replication does not.
Is there any hope that lives can be saved from cancer? Or is cancer human destiny? Some think that cancer is purely genetic. However, about half of all cancers have their roots in behavior. This is best illustrated by t he fact that immigrants tend to get the local cancer of the country they move to. For example, Japanese in Osaka are much less likely than Americans to get prostate and breast cancer, but the moment they migrate to the Hawaii, the gap narrows substantially (Figure 10-9). It’s not that lifestyle or the environment in Hawaii are conducive to ill health in general, but to specific cancers. In their home country, Japanese have substantially more stomach cancers than Americans, but after they move to Hawaii, stomach cancer largely disappears. Here are the major behaviors that cause cancer, with estimates for the US:
Cigarette smoking accounts for about 25 to 30 percent of all cancers. At the beginning of the 20th century, lung cancer was almost unknown. It was so rare that Isaac Adler, who wrote the first book-length review in 1912, apologized for wasting so much ink on such a trivial topic. At that time, people smoked pipes and cigars, which cause other cancers than lung cancer. Sigmund Freud, for one, suffered from mouth cancer the last sixteen years of his life as a result of his addiction to cigar smoking. Not until World War I did cigarettes become popular. Today, lung cancer is the leading cause of cancer death for American men and women. In total, one to two out of every ten smokers will develop it. It kills about 435,000 U.S. citizens each year, which is more than HIV, traffic accidents, homicide, suicide, and terrorism combined. But lung cancer is not the only cancer due to smoking. When a person smokes for 30 or 40 years, carcinogens are disposed in the urine and enter the bladder; before they are disposed from the body, genes in bladder mutate, which results in cancer of the bladder.
Obesity and diet account for 10 to 20 percent of all cancers. The American Cancer Society, in 2003, estimated that excess weight could be the cause of one in seven cancers, specifically cancers of the breast, cervix, colon, gall bladder, kidney, liver, oesophagus, ovary, pancreas, prostate, stomach (in men), and uterus. This number is on the rise.
Alcohol abuse accounts for 10 percent of all cancers in men (3 percent in women). Four percent of all newly diagnosed cases of breast cancer are attributed to alcohol abuse. Heavy intake of alcohol (more than 50 -70 g/day) is also a risk factor for liver and colorectal cancer. Some 13,000 cancers result every year from British drinking habits. Most of these occur in the upper digestive tract, mouth and throat, colorectum, and breast. These cancers could be avoided if men had no more then two drinks a day, and women no more than one.
CT scans account for about 2 percent of all cancers. As mentioned before, CT scans typically have a hundred times the radiation dose of a chest X-ray. Some scans are in the interest of the patient, but others are in the interest of those who profit and market them.
The fact that about half of all cancers are due to behavior shows where the war against cancer can be won: by focusing on prevention, not screening. In general, preventing a disease is more effective and costs less than detecting it an early stage. That’s why healthy lifestyle, not screening, is the stronger weapon against cancer. Physical activity during leisure time can lower the risk of getting breast cancer more than screening mammograms, and it does not matter much whether it’s walking, jogging, dancing, or gardening. Even when cancer is present, lifestyle changes can be as effective as expensive drugs that decrease quality of life. In a study with three thousand nurses who had breast cancer, those who walked three to five hours per week died less often from cancer than those who did not. Lifestyle can go a long way towards protection from cancer.
The mighty opponents in this war are the tobacco industry and other industries that make huge profits from carcinogenic products causing cancer. What is a human life worth to a cigarette manufacturer? Cigarettes cause about one death per million cigarettes smoked with a latency of 25 years, and tobacco companies earn about a penny in profit for each cigarette. One million times a penny is $10,000. That is the price tag on a person who died from smoking. While more and more people are becoming aware of the real hazards of smoking, tobacco companies are moving from rich to poor countries, ensuring that the death toll will continue.
Lifestyle changes require risk-savvy people who take their lives into their own hands. But there is a more profitable kind of prevention from the industry’s perspective: medication.
A Dream Drug?
A drug that could prevent cancer without harming the patient would be a dream. Some time back a full-page advertisement appeared in mass magazines and reached about 41 million readers on its first appearance. The headline, stretched over the back of a young woman in a lacy bra, reads: “If you care about breast cancer, care more about being a 1.7 than a 36B” (Figure 10-10). What is this mysterious number? Below the headline we learn: “Know your breast cancer risk assessment number.” The ad contains a toll-free number that every woman can call to find out whether she is at risk. But what exactly does 1.7 mean? You might think, it’s high risk, and if you are at high risk, then you should take tamoxifen (Nolvadex). What it actually means is a 1.7 percent chance of being diagnosed as having breast cancer in the next five years, in other words, a 98.3 percent chance of not developing breast cancer. That should be good news rather than a fright. The average 60-year-old American woman has a risk of 1.7. It is based on the so-called Gail model and includes factors such as early first birth that go along with lower risk of breast cancer.
The eye-catching bra distracts from the two questions one should ask: What are the benefits and what are the harms of tamoxifen? The text below the ad lists some but nowhere states how big they are, not even in relative risks. But it does clarify that the drug does not prevent breast cancer, nor does it increase “survival.” So what then does it do? A fact box sheds light on the question.
The fact box shows that out of every 1,000 women who took tamoxifen for 5 years, 17 developed breast cancer, compared to 33 who took a sugar pill. That is, 16 fewer women developed breast cancer. Yet the study could not show a reduction of mortality from breast cancer or from other causes. At the same time, among every 1,000 women who took tamoxifen, there were 5 additional cases of blood clots in their legs or lungs, and 6 more women got invasive uterine cancer. Besides these life-threatening side effects, hundreds of women had hot flashes and vaginal discharges, and a few needed cataract surgery. Unlike the ad, the fact box allows each woman to make an informed decision herself. If you care about breast cancer, care more about a fact box than being a 1.7 or 36B.
Many of us are restless at night, and lie in bed waiting to finally fall asleep. Insomnia can cause poor concentration, irritability spikes, and increased risk of a car accident. Should we stop drinking coffee in the evening, begin walking every day for half an hour, or go to bed at a regular time? The alternative might be a sleeping pill. But does it help and if so, how much earlier will it send us to sleep?
Patients and doctors will rarely if ever find the answer in drug ads and package inserts. Drug ads provide messages designed to create enthusiasm for the product. Inserts generally do not even provide the most fundamental information consumers need: how well the drug works. Direct-to-consumer advertisement is a prime vehicle for overstating benefits and downplaying harms. In the US, pharmaceutical companies spend about $5 billion a year, more than twice the budget of the FDA, on colorful messages that omit the essentials. Instead, they simply assert that the pill works very well, or pay a celebrity to declare “it works for me.”
LUNESTA is a popular sleeping pill that sold more than $600 million in 2011. All that its package insert says about how well it works is that the pill is superior to a placebo. A poetic ad with the headline “Peaceful, restful sleep” and a picture of a man snoozing contentedly in bed provides no concrete information either. The website simply assures you that “LUNESTA can help you find a restful night.”
When companies apply for drug approval, they submit the results of two or more randomized trials to regulatory agencies such as the FDA. The staff experts evaluate the findings and if they feel that benefits outweigh harms, approve the drug. Lisa Schwartz and Steven Woloshin from Dartmouth Medical School took the pains to go through the 403-page FDA review on LUNESTA and condense it into a fact box (Figure 10-12).
As in all fact boxes, the two alternatives are placed side by side: people who take a sugar pill (placebo) and people who take LUNESTA. Here it is made clear exactly how big the benefits are and what the harms are. With Lunesta, people fell asleep 15 minutes earlier on average. They also slept longer, resulting in a total of 37 minutes more sleep. It is also clear what harms to expect, and how often. No life-threatening harms, that’s the good news. But LUNESTA leads to a number of side effects, from unpleasant taste in the mouth to nausea. The fact box lets each of us make an informed decision about whether or not to take the sleeping pill. If you don’t like the idea of feeling nauseated in exchange for an extra half hour of sleep, then just take a walk before heading for bed.
How to Make Patients Smart
Does John Q Public understand a fact box? When I listen to doctors who tell me that their patients lack intelligence, the answer seems to be no. To test this claim, one group of people were shown two actual drug ads consisting of a picture and a brief text. A second group received the same pictures, but the text was replaced by a fact box. The effect was stunning. For instance, less than 10 percent of the first group correctly identified a proton-pump inhibitor for treating heartburn to be a lot more effective than a histamine-2 blocker, but 70 percent of the fact box group did so. Similarly, less than 10 percent of the first group correctly understood the absolute risk reduction of a statin, but in the fact box group, this increased to over 70 percent. Fact boxes improve consumers’ knowledge of benefits and side effects, and also result in better choices of drugs. They can also fight the daily barrage of medical news and advertisement faced by both doctors and patients.
Schwartz and Woloshin presented the results of several fact box studies to the FDA’s Risk Communication Advisory Committee, which voted unanimously that the FDA should adopt the fact box as a standard. But the recommendation was non-binding. Eventually, after The New York Times covered the presentation, two senators submitted a bill to Congress calling on the FDA to adopt fact boxes for drugs. The bill was incorporated into the health care reform bill signed into law in 2010. Wonderful, we might think, it’s about time. Unfortunately, the bill does not clarify two basic issues: who writes the fact boxes and where the boxes should appear. This is why we still have no fact boxes in package inserts or drug ads.
Who and where? The answer appears obvious. The FDA should write the fact boxes, since it has the data and has already written a report on every drug it approves. This fact box should then be part of inserts and ads. And they should be available in every doctor’s waiting room. Then patients would finally have easy access to clean information.
Let me end on a positive note. Can we do anything about the tricks played to patients? Yes. One bargaining chip is reputation. For years, I have argued that biased reporting in health pamphlets is a major cause of the bulk of misinformed patients and doctors. Such pamphlets are distributed by trusted organizations around the world. One of these was the Deutsche Krebshilfe, the largest German cancer care organization, which receives about 100 million euros a year in donations. When speaking to other medical organizations, I noted that the Krebshilfe risked losing its credibility (and, implicitly, its funding) for its misleading pamphlets. Eventually, the press speaker flew from Bonn to Berlin to ask whether I had a personal vendetta against the organization. I said, “Not at all. On the contrary, I’d be happy to help rewrite your pamphlets so that everyone can understand the evidence.” It turned out that the press speaker was not aware of the tricks used in its many cancer brochures. I took my time to explain what transparent facts look like. The organization agreed to my proposal, and there is now an entirely new generation of pamphlets for various cancers. All misleading relative risks and 5-year survival rates have been axed and replaced by absolute numbers. For the first time, the potential harms of screening are mentioned, including how often they occur. This laudable move will rightly secure the organization the trust of the public. The next step is to provide fact boxes, which are even simpler to understand.
We can all change things, each one in his or her own way. People can question their doctors about pros and cons rather than only about what to do. Doctors can provide fact boxes in their waiting rooms rather than only Cosmopolitan and Newsweek. Organizations can begin to inform patients in an understandable way rather than only trying to increase participation rates.
Raising taxes or rationing care is often viewed as the only alternative to increasing health care costs. I argue that there is a third option: by promoting health literacy of doctors and patients, we can get better care for less money.
Facts and fiction about cancer screening
Note that screening is for people without symptoms.
- If I participate in screening, I will reduce my chance of getting cancer. No, screening is not prevention. Just as accident insurance does not reduce the chance of an accident, screening does not reduce incidence; it can only detect cancers that are already there.
- A positive test means that I have cancer. No, most people with positive mammograms, PSA tests, or fecal occult blood tests do not have cancer.
- If the test is negative, I can be sure that I don’t have cancer. You can be more confident, but not sure. Misses happen; no test is perfect.
- If I get cancer, I will die from it. Cancer is not cancer. Most men with prostate cancer don’t die from it – they might not even notice it. But pancreatic cancer kills most in a short time.
- Early detection saved my life because after treatment, I am still alive. This conclusion may or may not be true. It is not for those who have been “overdiagnosed.” Tests can detect tiny tumors that are non-progressive and would have never affected your health. As a consequence, people suffer from needless appointments, needless tests, and needless drugs and surgery.
- Isn’t the fact that mortality rates decline over the years proof that screening works? No, the proof is in randomized trials (see fact boxes). For instance, the mortality rates for stomach cancer have declined since the 1930s in Western countries, before there was any screening. The reason is probably better conservation of food.
- Why is total mortality (or total cancer mortality) more relevant to understanding the benefit than cancer-specific mortality? First, some patients have multiple cancers, and it is difficult to determine which of them caused death Second, surgery following screening can kill as many as it cures, or more. These unlucky patients are included in the total mortality rate, but not in the cancer-specific rate. Third, consider the seemingly paradoxical fact that cigarette smoking reduces breast-cancer mortality by 1 in 1,000 women (same effect as for mammography screening). The reason is that smoking kills earlier so that some women don’t live long enough to get breast cancer. Here, smoking appears to reduce breast cancer mortality but in fact increases total mortality.
- Wouldn’t I be best off by screening for all cancers? No, you wouldn’t because some lead to more harm than gain. For instance, the U.S. Preventive Services Task Force explicitly recommends against screening for cancers of the prostate, lungs, pancreas, ovaries, bladder, and thyroid. Pap smear screening for cervical cancer, in contrast, appears to save lives; this has not yet been tested in a randomized trial.
- What can I do against cancer? Because about half of cancers are due to behavior, prevention has much more potential than early detection. Avoid smoking, obesity, wrong diet, and excessive alcohol consumption, and increase physical activity such as walking for three to five hours a week. Lifestyle change also has benefits for health in general.
- Where can I get reliable information about screening, and health in general? Sources for reliable information include:
Cochrane Library www.thecochranelibrary.com
U.S. Preventive Services Task Force www.ahrq.gov
Bandolier Oxford, UK www.medicine.ox.ac.uk/bandolier/
Foundation for Informed Medical Decision Making www.informedmedicaldecisions.org/
Agency for Healthcare Research and Quality www.ahrq.gov
Harding Center for Risk Literacy, Max Planck Institute for Human Development: www.harding-center.de
Watchdog group: healthnewsreview.org
 Reported in Michael Dobbs, “Rudy wrong on cancer survival chances,” Washington Post, October 30, 2007. Giuliani apparently used data from the year 2000, when 49 British men per 100,000 were diagnosed with prostate cancer, 28 of whom died within 5 years—about 44%. More recent figures (which differ from those cited by Giuliani) are 98% 5-year survival in the United States versus 71% in Britain.
 About 26 prostate cancer deaths per 100,000 American men versus 27 per 100,000 in Britain (Shibata & Whittemore, 2001).
 The correlation coefficient is exactly 0.0. Welch et al. (2000).
 Scientists have begun to uncover biological mechanisms that halt the progression of cancer (Folkman & Kalluri 2004; Mooi & Peeper 2006).
 These estimates are based on autopsies. Delongchamps et al. (2006) provide even slightly higher estimates for White and Black U.S. Americans. Note that estimates based on incidence rates are lower and less reliable because they do not include men with smaller or undetected cancers or those who did not attend screening.
 Steimle (1999, p. 1189).
 The icon box is based on Djulbegovic et al. (2010). Arkes & Gaissmaier (2012) provide a version with 1,000 people. To simplify, non-significant differences are represented by the same number. All numbers are “abouts.”
 Welch et al. (2011, p. 50).
 Stiftung Warentest 2004.
 Enserink (2010, p. 1738).
 Wegwarth et al. (2012).
 See Welch et al. (2011, Chapter 5).
 Wegwarth et al. (2011). Unlike the U.S. sample, the German doctors were a convenience sample.
 Welch et al. (2011, p. 156).
 Gigerenzer, Mata, & Frank (2009).
 This study is covered by the icon box: When one looks at all studies even this small effect disappears. To err on the conservative side, I have counted both “0 in 1,000” and “1 in 1,000” as a realistic estimate in Figure 10-7 (for both prostate and breast cancer, see below). The response alternatives were 0, 1, 10, 50, 100, 200 in 1,000, and “don’t know.”
 Hoffman et al. (2010).
 An ACS campaign poster from the 1980s: http://comedsoc.org/Breast_Cancer_Screening.htm?m=66&s=447
 “Why should I have a mammogram,” brochure from the Arkansas Department of Human Services and Arkansas Foundation of Medical Care.
 Fact boxes were developed and tested by two brilliant researchers from Dartmouth Medical School, Lisa Schwartz and Steven Woloshin. See Schwartz et al. (2009) and Schwartz & Woloshin (2011)
 Based on the Cochrane Review by Gøtzsche & Nielsen (2011). The authors distinguished between better (adequately randomized) studies and suboptimal ones. The better studies found only a non-significant absolute reduction of 0.3 deaths from breast cancer in every 1,000 women, while the suboptimal studies found a reduction of 2.1 in 1,000. Favoring the better studies, the authors estimated an overall reduction of 0.5 in 1,000. My more generous estimate in the fact box is 1 in 1,000, which is the middle (rounded) of reductions reported by the better and suboptimal studies. This is also consistent with Nyström (2002), who reported a reduction from 5.0 to 3.9 deaths from breast cancer in 1,000 women. The size of the reduction varies slightly from study to study. It is often said that modern technology and new treatment save more lives and that these estimates are therefore overly pessimistic. However, the benefits actually get smaller the more recent the study and the better the mammography equipment is. Total costs for breast cancer screening in the US amount to around 3 billion dollars and in Germany 300 to 400 million euros per year, or 0.1 of the total expenditure of the health system (news.doccheck.com/de/article).
 Schüssler 2005.
 Ehrenreich 2010.
 Gigerenzer, Gaissmaier, et al. 2007.
 Knaul & Frenk 2010.
 Posted by Gary Schwitzer on http://www.healthnewsreview.org : What doctors don’t know and journalists don’t convey about screening may harm patients. (March 8, 2012). Wikipedia lists other sources of funding in its entry on Susan G. Komen for the Cure.
 Deutsche Krebshilfe in their 2007 brochure Brustkrebs (pp. 15-16). After I pointed out this state of affairs to the Deutsche Krebshilfe, they reacted and rewrote their brochures with the help of my colleague Odette Wegwarth. The resulting new generation of cancer brochures has no misleading survival rates and relative risks, and informs in transparent frequencies. This admirable reaction is rare among cancer organizations.
 Gigerenzer, Gaissmaier, et al. (2007, p. 79).
 Take Avastin, the world’s best-selling cancer medicine with annual sales of $6 billion in 2010. It is used for the treatment of advanced cancers of colon, breast, lung, and kidney, among others. An analysis of 16 trials with more than 10,000 people showed that when Avastin was added to chemotherapy, more people died than when receiving chemotherapy alone (Rampura, Hapani, & Wu, 2011). Thus, not only did the drug not prolong lives of hopeful patients for a few weeks or months, it in fact shortened them. Given the huge amount of money at stake for the pharmaceutical industry (Avastin treatment costs up to $57,000 per year for one patient), we are fed false hopes and expensive drugs that can do us more harm than good.
 Begley 2012. On lack of replication see Ioannidis 2005.
 Brandt 2007.
 Willyard 2011.
 Willyard 2011.
 Schütze et al. 2011.
 Longnecker et al. 1995.
 Tengs et al. 1995.
 Holmes et al. 2005.
 Proctor 2012.
 Schwartz & Woloshin 2011.
 Schwartz et al 2009.
 Schwartz & Woloshin 2011.
 Gigerenzer, Gaissmaier, et al. 2007, Table 1.