variability in medical diagnoses -- 9/1/21
Today's selection -- from Noise: A Flaw in Human Judgment by Daniel Kahneman, Olivier Sibony, and Cass R. Sunstein. Medical professionals can come to significantly different conclusions even when presented with the same data;
"Consider just a few other findings from the literature, drawn from areas in which the volume of noise [decision variability] seems especially noteworthy. We describe these findings not to give authoritative statements about the current state of medical practice, which continues to evolve and improve (in some cases rapidly), but to convey a general sense of the pervasiveness of noise, both in the relatively recent past and in the present.
"1. Heart disease is the leading cause of death in both men and women in the United States. Coronary angiograms, a primary method used to test for heart disease, assess the degree of blockage in the heart's arteries in both acute and nonacute settings. In nonacute settings, when a patient presents with recurrent chest pain, treatment -- such as stent placement -- is often pursued if more than 70% of one or more arteries is found to be blocked. However, a degree of variability in interpreting angiograms has been documented, potentially leading to unnecessary procedures. An early study found that 31 % of the time, physicians evaluating angiograms disagreed on whether a major vessel was more than 70% blocked. Despite widespread awareness by cardiologists of potential variability in reading angiograms, and despite continuing efforts and corrective steps, the problem has yet to be solved.
"2. Endometriosis is a disorder in which endometrial tissue, normally lining the inside of the uterus, grows outside the uterus. The disorder can be painful and lead to fertility problems. It is often diagnosed through laparoscopy, in which a small camera is surgically inserted into the body. Digital videos of laparoscopies in three patients, two of whom had endometriosis of varying degrees of severity and one of whom did not, were shown to 108 gynecological surgeons. The surgeons were asked to judge the number and location of endometriotic lesions. They disagreed dramatically, with weak correlations on both number and location.
"3. Tuberculosis (TB) is one of the most widespread and deadly diseases worldwide -- in 2016 alone, it infected more than 10 million people and killed almost 2 million. A widely used method for detecting TB is a chest X-ray, which allows examination of the lungs for the empty space caused by the TB bacteria. Variability in diagnosis of TB has been well documented for almost seventy-five years. Despite improvements over the decades, studies have continued to find significant variability in diagnosis of TB, with 'moderate' or just 'fair' interrater agreement. There is also variability in TB diagnoses between radiologists in different countries.
"4. When pathologists analyzed skin lesions for the presence of melanoma -- the most dangerous form of skin cancer -- there was only 'moderate' agreement. The eight pathologists reviewing each case were unanimous or showed only one disagreement just 62% of the time. Another study at an oncology center found that the diagnostic accuracy of melanomas was only 64%, meaning that doctors misdiagnosed melanomas in one of every three lesions. A third study found that dermatologists at New York University failed to diagnose melanoma from skin biopsies 36% of the time. The authors of the study conclude that 'the clinical failure to diagnose melanoma correctly has grievous implications for survival of patients with that potentially fatal disease.'
"5. There is variability in radiologists' judgments with respect to breast cancer from screening mammograms. A large study found that the range of false negatives among different radiologists varied from 0% (the radiologist was correct every time) to greater than 50% (the radiologist incorrectly identified the mammogram as normal more than half of the time). Similarly, false-positive rates ranged from less than 1% to 64% (meaning that nearly two-thirds of the time, the radiologist said the mammogram showed cancer when cancer was not present). False negatives and false positives, from different radiologists, ensure that there is noise.
"These cases of interpersonal noise dominate the existing research, but there are also findings of occasion noise. Radiologists sometimes offer a different view when assessing the same image again and thus disagree with themselves (albeit less often than they disagree with others). When assessing the degree of blockage in angiograms, twenty-two physicians disagreed with themselves between 63 and 92% of the time. In areas that involve vague criteria and complex judgments, intrarater reliability, as it is called, can be poor.
"These studies offer no clear explanation of this occasion noise.
"But another study, not involving diagnosis, identifies a simple source of occasion noise in medicine -- a finding worth bearing in mind for both patients and doctors. In short, doctors are significantly more likely to order cancer screenings early in the morning than late in the afternoon. In a large sample, the order rates of breast and colon screening tests were highest at 8 a.m., at 63.7%. They decreased throughout the morning to 48.7% at 11 a.m. They increased to 56.2% at noon -- and then decreased to 47.8% at 5 p.m. It follows that patients with appointment times later in the day were less likely to receive guideline-recommended cancer screening.
"How can we explain such findings? A possible answer is that physicians almost inevitably run behind in clinic after seeing patients with complex medical problems that require more than the usual twenty-minute slot. We already mentioned the role of stress and fatigue as triggers of occasion noise, and these elements seem to be at work here. To keep up with their schedules, some doctors skip discussions about preventive health measures. Another illustration of the role of fatigue among clinicians is the lower rate of appropriate handwashing during the end of hospital shifts. (Handwashing turns out to be noisy, too.)
"It would be a major contribution not only to medicine but also to human knowledge to provide a comprehensive account of the existence and magnitude of noise in the context of different medical problems."