Reliability

This article is a stub.

Interobserver Agreement and the κ (Kappa) Statistic. Top: Conventional 2 x 2 table for Kappa calculation. Bottom: Sample problem, with observed agreement of 80%, chance agreement of 74%, and Kappa statistic of 0.23

If two observers examine the same patients independently this can be displayed in a 2 x 2 table. Observer A finds a positive sign in w1 patients, and negative sign in w2 patients. Observer B finds a positive sign in y1 patients and negative sign in y2 patients. The two observers agree that the sign if positive in a patients, and is negative in d patients. The observed agreement (p_o) is calculated:

p_o= (a + d)/N

To calculate the κ-statistic the first step is calculating the agreement that would have occurred simply due to chance. Observer A found that among all patients W1/N have the sign. Observer B finds that the patients y1 have the sign, and among y1 patients Observer A would find the sign by chance in (w1y1/N) patients. This number represents the number of patients in which both observers agree by chance. Likewise, both observers would agree that the sign is negative by chance in (w2y2/N) patients.

The expected chance of agreement (p_e) is the sum of these two values, divided by N.

p_e = (w₁y₁ + w₂y₂)/N²

If both observers agree that a finding is rare (approaching 0) or very common (approaching N) then the expected chance of agreement (p_e) approaches 100%

The κ-statistic is the difference in observed agreement (p_o) and the expected chance of agreement (p_e) x (p_o - p_e), divided by the maximal increment that could have been observed if the observed agreement (p_o) was 100% (1 - p_e)

κ = (p_o − p_e)/(1 − p_e)

The sample problem in the figure represents a fictional study of 100 patients with ischial tuberosity tenderness. Both observers agree that tenderness is present in 5 patients and absent in 75 patients.

Observed agreement (p_o) is (5 + 75)/100 = 0.8.

By chance the observers would agree the sign was present in ((5+5) x (15+5))/100 patients = 2 patients

By chance the observers would agree the sign was not present in ((75+15) x (75+5))/100 patients = 72 patients.

Expected chance of agreement (p_e) is (2 + 72)/100 patients = 0.74.

The κ-statistic is (0.80 – 0.74)/(1 – 0.74) = (0.06)/(0.26) = 0.23.

Bibliography

• McGee, Steven R. Evidence-based physical diagnosis. Philadelphia: Elsevier/Saunders, 2012.