Reliability

This article is a stub.

Interobserver Agreement and the κ (Kappa) Statistic. Top: Conventional 2 x 2 table for Kappa calculation. Bottom: Sample problem, with observed agreement of 80%, chance agreement of 74%, and Kappa statistic of 0.23

If two observers examine the same patients independently this can be displayed in a 2 x 2 table. Observer A finds a positive sign in w1 patients, and negative sign in w2 patients. Observer B finds a positive sign in y1 patients and negative sign in y2 patients. The two observers agree that the sign if positive in a patients, and is negative in d patients. The observed agreement (PO) is calculated:

PO =

.mw-parser-output .sfrac{white-space:nowrap}.mw-parser-output .sfrac.tion,.mw-parser-output .sfrac .tion{display:inline-block;vertical-align:-0.5em;font-size:85%;text-align:center}.mw-parser-output .sfrac .num,.mw-parser-output .sfrac .den{display:block;line-height:1em;margin:0 0.1em}.mw-parser-output .sfrac .den{border-top:1px solid}.mw-parser-output .sr-only{border:0;clip:rect(0,0,0,0);height:1px;margin:-1px;overflow:hidden;padding:0;position:absolute;width:1px}(a + d)/N

To calculate the κ-statistic the first step is calculating the agreement that would have occurred simply due to chance. Observer A found that among all patients W1/N have the sign. Observer B finds that the patients y1 have the sign, and among y1 patients Observer A would find the sign by chance in (w1y1/N) patients. This number represents the number of patients in which both observers agree by chance. Likewise, both observers would agree that the sign is negative by chance in (w2y2/N) patients.

The expected chance of agreement (PE) is the sum of these two values, divided by N.

PE =

(w 1 y 1 + w 2 y 2) / N 2

If both observers agree that a finding is rare (approaching 0) or very common (approaching N) then the expected chance of agreement (PE) approaches 100%

The κ-statistic is the difference in observed agreement (PO) and the expected chance of agreement (PE) (PO-PE), divided by the maximal increment that could have been observed if the observed agreement (PO) was 100% (1 - PE)

κ =

(PO - PE) / (1 - PE)

The sample problem in the figure represents a fictional study of 100 patients with ischial tuberosity tenderness. Both observers agree that tenderness is present in 5 patients and absent in 75 patients.

Observed agreement (PO) is (5 + 75)/100 = 0.8.

By chance the observers would agree the sign was present in ((5+5) x (15+5))/100 patients = 2 patients

By chance the observers would agree the sign was not present in ((75+15) x (75+5))/100 patients = 72 patients.

Expected chance of agreement (PE) is (2 + 72)/100 patients = 0.74.

The κ-statistic is (0.80 – 0.74)/(1 – 0.74) = (0.06)/(0.26) = 0.23.

Bibliography

McGee, Steven R. Evidence-based physical diagnosis. Philadelphia: Elsevier/Saunders, 2012.