Evaluating evaluation: assessment of the American Board of Internal Medicine Resident Evaluation Form

Area of Science:

• Medical Education Research
• Psychometrics in Healthcare
• Internal Medicine Training

Background:

• The American Board of Internal Medicine recommends a standardized rating form for residents.
• The form assesses nine dimensions of clinical competence on a 1-9 scale.
• Concerns exist regarding the psychometric properties of such evaluation tools.

Purpose of the Study:

• To evaluate the reliability and validity of a specific internal medicine residency rating form.
• To determine if the form can reliably distinguish between residents with high performance scores.
• To assess the psychometric evidence supporting the use of the nine-dimensional rating scale.

Main Methods:

• Analysis of 1,039 ratings from 135 attendings for 85 residents.
• Application of factor analysis to examine correlations among the nine rating dimensions.
• Calculation of inter-rater reliability using the method of James to assess agreement.

Main Results:

• A single global factor accounted for 86% of the variance, with high correlations (0.72-0.92) among dimensions.
• 95.6% of ratings fell between 6 and 9, indicating a potential ceiling effect.
• Average inter-rater reliability was weak (0.64), suggesting poor agreement among evaluators for high-scoring residents.

Conclusions:

• The rating form demonstrates low reliability and fails to discriminate between different dimensions of clinical care.
• The tool has limited ability to reliably distinguish among residents who are rated as competent.
• Psychometric limitations suggest the current rating method may not be effective for detailed resident assessment.