Reliability
If two observers examine the same patients independently this can be displayed in a 2 x 2 table. Observer A finds a positive sign in w1 patients, and negative sign in w2 patients. Observer B finds a positive sign in y1 patients and negative sign in y2 patients. The two observers agree that the sign if positive in a patients, and is negative in d patients. The observed agreement (po) is calculated:
- po= (a + d)/N
To calculate the ฮบ-statistic the first step is calculating the agreement that would have occurred simply due to chance. Observer A found that among all patients W1/N have the sign. Observer B finds that the patients y1 have the sign, and among y1 patients Observer A would find the sign by chance in (w1y1/N) patients. This number represents the number of patients in which both observers agree by chance. Likewise, both observers would agree that the sign is negative by chance in (w2y2/N) patients.
The expected chance of agreement (pe) is the sum of these two values, divided by N.
- pe = (w1y1 + w2y2)/N2
If both observers agree that a finding is rare (approaching 0) or very common (approaching N) then the expected chance of agreement (pe) approaches 100%
The ฮบ-statistic is the difference in observed agreement (po) and the expected chance of agreement (pe) x (po - pe), divided by the maximal increment that could have been observed if the observed agreement (po) was 100% (1 - pe)
- ฮบ = (po โ pe)/(1 โ pe)
The sample problem in the figure represents a fictional study of 100 patients with ischial tuberosity tenderness. Both observers agree that tenderness is present in 5 patients and absent in 75 patients.
- Observed agreement (po) is (5 + 75)/100 = 0.8.
- By chance the observers would agree the sign was present in ((5+5) x (15+5))/100 patients = 2 patients
- By chance the observers would agree the sign was not present in ((75+15) x (75+5))/100 patients = 72 patients.
- Expected chance of agreement (pe) is (2 + 72)/100 patients = 0.74.
- The ฮบ-statistic is (0.80 โ 0.74)/(1 โ 0.74) = (0.06)/(0.26) = 0.23.
Bibliography
- McGee, Steven R. Evidence-based physical diagnosis. Philadelphia: Elsevier/Saunders, 2012.