Impact of Clinical Performance Examination on Incoming Interns' Clinical Competency in Differential Diagnosis of Headache

Seong-Min Park; Yun-Mi Song; Bo-Kyoung Kim; Hyoeun Kim

doi:10.4082/kjfm.2014.35.2.56

Korean J Fam Med > Volume 35(2); 2014 > Article

Park, Song, Kim, and Kim: Impact of Clinical Performance Examination on Incoming Interns' Clinical Competency in Differential Diagnosis of Headache

Original Article

Korean Journal of Family Medicine 2014;35(2):56-64.

Published online: March 24, 2014

DOI: https://doi.org/10.4082/kjfm.2014.35.2.56

Impact of Clinical Performance Examination on Incoming Interns' Clinical Competency in Differential Diagnosis of Headache

Seong-Min Park, Yun-Mi Song, Bo-Kyoung Kim, Hyoeun Kim

Department of Family Medicine, Samsung Medical Center, Sungkyunkwan University School of Medicine, Seoul, Korea.

Corresponding Author: Yun-Mi Song.
Tel: +82-2-3410-2442, Fax: +82-2-3410-0388, yunmisong@skku.edu

Received August 24, 2012 Accepted February 21, 2014

(open-access, http://creativecommons.org/licenses/by-nc/3.0/):

This is an open-access article distributed under the terms of the Creative Commons Attribution Non-Commercial License (http://creativecommons.org/licenses/by-nc/3.0/) which permits unrestricted noncommercial use, distribution, and reproduction in any medium, provided the original work is properly cited.

Abstract

Background

In Korea, clinical performance examination (CPX) has been included in license examination for medical doctors since 2009 in order to improve clinical performance of medical students. This study aimed to evaluate the contribution of CPX to medical education.

Methods

Clinical competency in the differential diagnosis of secondary headache was compared between the incoming interns in 2009 unexposed to CPX and the incoming interns in 2010 exposed to CPX, using the data of patients who visited the emergency department due to headache (181 patients seen by 60 CPX non-exposed interns and 150 patients seen by 50 CPX-exposed interns). We obtained the data by reviewing electronic medical records and nominal lists of doctors. Clinical competency was assessed by sensitivity and specificity between the diagnostic impression by interns and the final diagnosis. The association between CPX exposure and clinical competency in secondary headache diagnosis was evaluated using multiple logistic regression analysis.

Results

When we assessed clinical competency on the basis of all listed diagnostic impressions, sensitivity and specificity were 67.9% and 80.0%, respectively, for headaches seen by CPX-exposed interns, and 51.7%, and 71.7%, respectively, for headaches seen by CPX non-exposed interns. Multivariable adjusted logistic regression analysis showed exposure to CPX was not associated with increased competency for identifying secondary headache.

Conclusion

Exposure to CPX as a part of the medical license examination was not effective for the improvement of clinical competency of interns in identifying secondary headache.

Keywords: Competency-Based Education; Clinical Performance Examination; Headache; Internship and Residency; Medical Student

INTRODUCTION

Headache is one of the most common symptoms with great disease burden worldwide and thus, primary care physicians frequently encounter patients with headache in daily practice.^1,²⁾ Headache develops from various causes, some of which pertain to fatal secondary headache.³⁾ Therefore, it is very important for physicians to accurately identify secondary headache,^3,^4,⁵⁾ which raises the importance of training medical students to make an accurate differential diagnosis of headache.⁶⁾

In Korea, the clinical performance examination (CPX) was included in the license examination for qualified medical doctors in 2009 for the first time in order to improve clinical competence of medical students.⁷⁾ Accordingly, incoming interns in 2010 should have passed CPX as an essential step to acquire a medical license.^7,⁸⁾ CPX is a test that requires great change in the medical education system.⁹⁾ Therefore, it seems necessary to evaluate whether the introduction of CPX to the Korean medical education system has contributed to the improvement of clinicians' quality of performance as expected.

There have been studies evaluating the impact of CPX on the improvement of interviewing skills of medical students. Han et al.¹⁰⁾ reported a significant correlation between CPX score and patient-doctor relationship (r = 0.707). Jang et al.¹¹⁾ also showed a positive correlation between CPX score and the attitude of interviewing with patients (r = 0.191). However, no study has evaluated the impact of CPX on the clinician's diagnostic accuracy of diseases.

In this regard, we aimed to assess the impact of CPX on incoming interns' clinical competency in differential diagnosis of headache. Headache has been included in the list of essential items of CPX in license examinations for qualified Korean medical doctors from the beginning. We hypothesized that interns who have been exposed to CPX (CPX-exposed interns) would show higher competency compared with interns who have not been exposed (CPX non-exposed interns).

METHODS

1. Study Subjects

In order to evaluate the impact of CPX, we compared the clinical performance of identifying secondary headache using the data of patients with headache who visited emergency department (ED) of a tertiary hospital located in Seoul, Korea between March 1st, 2009 and January 31st, 2010 (seen by 69 CPX non-exposed interns), and between March 1st, 2010 and January 31st, 2011 (seen by 60 CPX-exposed interns). All medical records were reviewed by one of the authors.

Initially, we reviewed the electronic medical records of 542 headache patients (290 patients seen by CPX non-exposed interns and 252 patients seen by CPX-exposed interns), respectively. Among the 542 patients, a total of 211 were excluded from the study for the following reasons: headache associated with trauma (18 patients), having discharged from the ED on his/her own (1 patient), medical records lacking information on intern's differential diagnosis (57 patients), unavailable information about final diagnosis due to not showing at recommended follow-up visit (135 patients). Thus, 331 headache cases (181 enrolled in 2009 and 150 enrolled in 2010) seen by 110 interns (60 in 2009 and 50 in 2010) were finally included in our study. This study was approved by the institutional review board of Samsung Medical Center.

2. Study Variables

We collected data on the demographic and clinical characteristics of patients through the review of electronic medical records: age, sex, time of ED visit, characteristics of headache, presence of red flag signs, past medical history, name of duty intern, list of diagnostic impressions for headache given by intern, and final diagnosis for headache. In addition, we collected the working information of duty doctor for each patient: occurrence of patient hand-off, duration of clinical experience of duty intern, and on-duty time of duty intern (more than 16 hours or not).

We collected demographic characteristics of interns by reviewing electronic medical records and the nominal lists of doctors enrolled in the residency program, including sex, exposure to CPX, and enrollment into residency program after completing internship training.

We identified the presence of red flag signs of headache according to the Scottish Intercollegiate Guidelines Network guidelines on diagnosis and management of headache in adults as follows:¹²⁾ onset of new headache after age ≥50, change in headache frequency or characteristics, headache worsened by postural change or Valsalva's maneuver, neck stiffness, focal neurologic deficit, associated systemic symptoms, previous history of secondary headache or cancer, having risk factors for cerebral venous sinus thrombosis, jaw claudication, and being infected by human immunodeficiency virus. Focal neurologic deficit was defined as the presence of the following features (≥1); mental status change, disorientation, abnormal findings in cranial nerve examination, abnormal findings in motor power or sensory examination, increase (>grade 2) or decrease (<grade 2) in deep tendon reflex, presence of Babinski or Chaddok sign, abnormal cerebellar function test, and abnormal findings in Romberg's test, gait, or tandem gait.

We categorized the final diagnosis of headache into two groups (primary or secondary) according to International Headache Society (IHS) classification.¹³⁾ Secondary headache was defined as a headache related with infection, neoplasm, vascular disease, cranial nerve pathology, psychiatric cause, musculoskeletal cause, iatrogenic, and other ill-defined causes. Remaining types of headache were categorized as a primary headache.

We also categorized diagnostic impression by interns into two groups (primary headache or secondary headache) according to IHS classification, not only on the basis of primary diagnostic impression but also on the basis of all listed diagnostic impressions. On the basis of primary diagnostic impression, we classified the diagnostic impression as secondary headache when the diagnostic impression in advance of others was ascertained to be on the lists of secondary headaches of IHS classification. On the basis of all listed diagnostic impressions, we classified the diagnostic impression as secondary headache when any of the listed diagnostic impressions was ascertained to be on the lists of secondary headaches of IHS classification.

3. Statistical Analysis

We compared characteristics between the patients seen by CPX non-exposed interns and the patients seen by CPX-exposed interns by chi-square test and t-test. We also compared the characteristics between the CPX non-exposed interns and the CPX-exposed interns.

In order to assess clinical competency, we calculated sensitivity, specificity, positive predictive value, and negative predictive value for identifying secondary headache. We repeatedly assessed the statistics based on both the primary diagnostic impression and all listed diagnostic impressions.

We defined sensitivity as the proportion of suspected secondary headache cases over finally-diagnosed secondary headache cases, and specificity as a proportion of suspected primary headache cases over finally-diagnosed primary headache cases.¹⁴⁾ Positive predictive value was defined as a proportion of finally-diagnosed secondary headache cases over suspected secondary headache cases.¹⁴⁾ Negative predictive value was defined as a proportion of finally-diagnosed primary headache cases over suspected primary headache cases.¹⁴⁾

In order to evaluate the association between CPX exposure and clinical competency in secondary headache diagnosis, we conducted multiple logistic regression analysis with an adjustment for provable covariates: sex of intern,¹⁵⁾ patient handoff,¹⁶⁾ enrollment into residency program after completing internship training, duration of clinical experience of duty intern, duration of working at ED, sex of patient, age of patient, patient encounter by intern who was on duty for more than 16 hours,¹⁷⁾ presence of red flag sign (≥1), first-onset headache at the age of 50 or older, abrupt-onset headache, presence of focal neurologic deficits, headache aggravated by postural change or by Valsalva' s maneuver, presence of neck stiffness, presence of systemic symptoms (fever, dizziness, and general weakness), and history of previous secondary headache. Covariates related with clinical characteristics of patient were selected because they have been commonly considered to be significant clinical factors suggesting secondary headache.¹²⁾

We conducted all the analyses under the assumption that all measurements were independent of each other. The P-value of significance level was set at 0.05 for all statistical assessment. All the analyses were performed using SAS ver. 9.3 (SAS Institute Inc., Cary, NC, USA).

RESULTS

Table 1 shows general characteristics of interns. The mean number ± SD of headache patients per one intern was 3.02 ± 1.97 for CPX non-exposed interns and 3.00 ± 2.14 for CPX-exposed interns. The proportion of male interns and the interns who were enrolled in a residency program after completing internship training did not differ between the two groups.

In Table 2, demographic and clinical characteristics of patients and related working information of the duty doctor were presented. We compared those characteristics between patients seen by CPX non-exposed interns and CPX-exposed interns. Males occupied a higher proportion among patients seen by CPX-exposed interns (46.0%) than among patients seen by CPX nonexposed interns (35.3%) with borderline statistical significance. The proportion of patients experiencing hand-off of duty intern was higher among cases (10.7%) seen by CPX-exposed interns compared to that among cases (5%) seen by CPX non-exposed interns (P = 0.05). The proportion of patients seen by duty interns with longer clinical experience in internship training (≥7 months) was higher among cases seen by CPX-exposed interns (58.0%) compared to that in patients seen by CPX non-exposed interns (44.8%) (P = 0.02). For other characteristics, there was no difference between the patients seen by CPX non-exposed interns and patients seen by CPX-exposed interns.

When we repeated the comparison in patients who were finally excluded from the present study, no significant difference was found between the patients seen by CPX-exposed interns and patients seen by CPX non-exposed interns for all selected characteristics.

Table 3 shows a list of ascertained final diagnoses for headaches among patients. In both groups, more than half of cases were ascertained as primary headache, and migraine was the most prevalent diagnosis. The proportion of patients with each ascertained diagnosis did not show a significant difference between both groups (P > 0.05 by chi-square test).

Table 4 shows the estimated accuracy of differential diagnosis for headache on the basis of primary diagnostic impression. All the statistics assessing diagnostic accuracy tended to be slightly higher for the headache cases seen by CPX non-exposed interns than that for the headache cases seen by CPX-exposed interns.

When we repeated the same analysis for estimating the diagnostic accuracy for headaches on the basis of all listed diagnostic impressions, we observed similar findings (Table 5).

Table 6 shows the multivariable-adjusted associations between CPX exposure and accurate identification of secondary headache with an adjustment for various characteristics of patients and interns. Exposure to CPX was not associated with an improvement in correct identification of secondary headache regardless of whether the estimation was conducted on the basis of primary diagnostic impression or all listed diagnostic impressions.

DISCUSSION

The present study is the first Korean study to assess the impact of CPX on incoming interns' clinical competency by comparing the diagnostic accuracy of identifying secondary headache between CPX-exposed interns and CPX non-exposed interns. Although we hypothesized that CPX-exposed interns would show higher competency than CPX non-exposed interns, findings of our study did not support our hypothesis.

Regarding the null effect of CPX on the improvement of clinical competency as seen in the present study, several possible reasons exist. Firstly, the interns who were selected for the present study had taken CPX for the first time in Korea with very scarce experience with CPX. Thus, their clinical competency might have been barely influenced by CPX. Secondly, the evaluation of clinical competency of interns on the basis of headache diagnosis might have resulted in underestimation of the impact of CPX. A wide range of clinical problems should have been evaluated in CPX⁸⁾ in order to assess clinical competency more accurately, given that clinical performance consists not only of diagnostic accuracy but also procedure skills, patient education, and communication skills.⁷⁾ Thirdly, the impact of CPX might have been masked by other unmeasured factors that could have greater influence on clinical competency of individual interns.^16,^17,^18,^19,^20,²¹⁾

Examinations testing clinical competency such as objective structured clinical examination (OSCE) or CPX have been adopted not only to the Korean medical license examination (KMLE) but also to board certifying examinations of plastic surgeons, neurologists, and family physicians. There were studies that provided evidence for positive aspects of the examinations testing clinical competency. In board examinations of plastic surgeons, there was a weak association between written test scores and OSCE scores (r = 0.32).²²⁾ In board examinations of neurologists, similar findings were revealed.²³⁾ A study using a questionnaire survey reported that levels of satisfaction regarding adopting of CPX into the KMLE were high for physician faculties, nurses, patients, and examinees.⁹⁾

Meanwhile, a criticism against the United States Medical Licensing Examination step 2 Clinical Skills (USMLE step 2CS), one type of CPX, has been raised²⁴⁾ such as inappropriately high rate of success (up to 97%),²⁵⁾ lack of adequate supervision or feedback,²⁴⁾ and low cost-effectiveness.²⁴⁾ A study by Clauser et al.²⁶⁾ showed that significant inconsistency may exist regarding the assessment of examinees in the USMLE step 2 CS. In addition, the USMLE step 2 score reflected residency directors' rating for interns less accurately than undergraduate grades did.²⁷⁾ Therefore, further studies evaluating the effect of CPX on clinical competency of medical students and doctors need to be conducted.

Several factors have been suggested to have an association with clinical skill examination scores or clinical performance.^16,^17,^18,^19,^20,²¹⁾ Ogunyemi and Taylor-Harris¹⁸⁾ reported that undergraduate written test scores, evaluation grade by faculty, and age of examinee were significantly correlated with scores of the USMLE step 2, while sex and race were not. In another study in Canada, clinical experience, case volume, and socioeconomic factors were related with physicians' ability to solve acute health problems.¹⁹⁾ Reduced workload has also been found to contribute to completeness of residents' discharge summaries,²⁰⁾ which provided a basis for the Institute of Medicine's recommendation that restricts duty hours of a resident to less than 16 hours.¹⁷⁾ However, we could not find factors that were consistently associated with clinical competency in identifying secondary headache.

The present study has several limitations. Firstly, we collected data by reviewing the medical record, which may have led to biased results. Secondly, one of the authors who knew the hypothesis underlying this study performed data collection. This un-blinded method of data collection may have resulted in overestimation of the beneficial impact of CPX. However, this seems unlikely to have occurred, given that rather lower diagnostic accuracy was observed for CPX-exposed interns. Thirdly, selection bias may have occurred because too many cases were excluded from the final analysis and a higher proportion of patients with primary headache who need no further evaluation or follow up might have been excluded. However, we do not believe this distorted our findings, because no significant difference was found between the patients seen by CPX-exposed interns and the patients seen by CPX non-exposed interns for all selected characteristics among the excluded cases. Fourthly, it may not be easy to generalize the findings from the present study because our study focused on one specific clinical entity, headache. Fifthly, although we made an adjustment for the enrollment of studied individual interns into residency program after completing internship training, duration of clinical experience of duty intern, and duration of working at ED, we could not take account for other aspects of individual professional competency of interns including KMLE score, performance level in medical school, or other unmeasured individual clinical abilities due to a lack of information. In addition, as we could not control the number of patients seen by each intern equally, the competency level of individual interns might also have influence the study results. These limitations may have resulted in under- or over-estimation of the impact of the Clinical Performance Examination on incoming interns' clinical competency in differential diagnosis of headache.

In conclusion, this study on incoming interns found that exposure to CPX as a part of the KMLE was not effective for the improvement of diagnostic competency for secondary headache. However, given the limitations of our study, further evaluations of the long-term effect of CPX over more diverse clinical entities are needed.

Notes

No potential conflict of interest relevant to this article was reported.

References

1. Stovner Lj, Hagen K, Jensen R, Katsarava Z, Lipton R, Scher A, et al. The global burden of headache: a documentation of headache prevalence and disability worldwide. Cephalalgia 2007;27:193-210. PMID: 17381554.

2. Solomon GD, Cady RK, Klapper JA, Ryan RE Jr. National Headache Foundation. Standards of care for treating headache in primary care practice. Cleve Clin J Med 1997;64:373-383. PMID: 9223767.

3. Goldstein JN, Camargo CA Jr, Pelletier AJ, Edlow JA. Headache in United States emergency departments: demographics, work-up and frequency of pathological diagnoses. Cephalalgia 2006;26:684-690. PMID: 16686907.

4. Clinch CR. Evaluation of acute headaches in adults. Am Fam Physician 2001;63:685-692. PMID: 11237083.

5. Edlow JA, Panagos PD, Godwin SA, Thomas TL, Decker WW. American College of Emergency Physicians. Clinical policy: critical issues in the evaluation and management of adult patients presenting to the emergency department with acute headache. Ann Emerg Med 2008;52:407-436. PMID: 18809105.

6. Korean Association of Medical Colleges. Manual for basic clinical practice: a guideline for medical students. Seoul: Academia press; 2010.

7. Park H. Clinical skills assessment in Korean Medical Licensing Examination. Korean J Med Educ 2008;20:309-312.

8. Kim KS. Introduction and administration of the clinical skill test of the medical licensing examination, republic of Korea (2009). J Educ Eval Health Prof 2010;7:4PMID: 21179230.

9. Park HK. The impact of introducing the Korean Medical Licensing Examination clinical skills assessment on medical education. J Korean Med Assoc 2012;55:116-123.

10. Han JJ, Lee MJ, Im HJ. Comprehension of patient-physician interaction through analysis of relationships between domains in clinical performance examination. Korean J Med Educ 2010;22:177-184.

11. Jang SS, Seo JH, Cho GJ, Hong SC, Woo HO. Correlation of communication skills for emotional empathy and academic achievement on clinical performance examinations. Korean J Med Educ 2010;22:121-130.

12. Duncan CW, Watson DP, Stein A. Guideline Development Group. Diagnosis and management of headache in adults: summary of SIGN guideline. BMJ 2008;337:a2329PMID: 19022842.

13. Headache Classification Subcommittee of the International Headache Society. The International Classification of Headache Disorders: 2nd edition. Cephalalgia 2004;24(Suppl 1):9-160. PMID: 14979299.

14. Hulley SB, Browner WS, Grady DG, Newman TB. Designing clinical research. 3rd ed. Philadelphia: Lippincott Williams & Wilkins; 2006.

15. Swygert KA, Cuddy MM, van Zanten M, Haist SA, Jobe AC. Gender differences in examinee performance on the Step 2 Clinical Skills data gathering (DG) and patient note (PN) components. Adv Health Sci Educ Theory Pract 2012;17:557-571. PMID: 22041870.

16. Aylward MJ, Rogers T, Duane PG. Inaccuracy in patient handoffs: discrepancies between resident-generated reports and the medical record. Minn Med 2011;94:38-41. PMID: 22372047.

17. Wagner MJ, Wolf S, Promes S, McGee D, Hobgood C, Doty C, et al. Duty hours in emergency medicine: balancing patient safety, resident wellness, and the resident training experience: a consensus response to the 2008 Institute of Medicine resident duty hours recommendations. J Emerg Med 2010;39:348-355. PMID: 20634017.

18. Ogunyemi D, Taylor-Harris D. Factors that correlate with the U.S. Medical Licensure Examination Step-2 scores in a diverse medical student population. J Natl Med Assoc 2005;97:1258-1262. PMID: 16296216.

19. Wenghofer EF, Williams AP, Klass DJ. Factors affecting physician performance: implications for performance improvement and governance. Healthc Policy 2009;5:e141-e160. PMID: 21037818.

20. Coit MH, Katz JT, McMahon GT. The effect of workload reduction on the quality of residents' discharge summaries. J Gen Intern Med 2011;26:28-32. PMID: 20697967.

21. Volpp KG, Rosen AK, Rosenbaum PR, Romano PS, Even-Shoshan O, Canamucio A, et al. Mortality among patients in VA hospitals in the first 2 years following ACGME resident duty hour reform. JAMA 2007;298:984-992. PMID: 17785643.

22. Kim SW, Lim SY, Mun GH, Shin JS. Introduction of OSCE (objective structured clinical examination) to board licensure examination of plastic surgeons. Korean J Med Educ 2006;18:153-160.

23. Kim BJ, Sung JJ, Park HK, Seo DW, Chung CS, Yoon BW. Clinical performance examination utilizing standardized patients in board examination: based on the board examination of Korean Neurological Association for three Years. Korean J Med Educ 2011;23:127-135.

24. Mehta NP, Kramer DB. A critique of the USMLE clinical skills examination. MedGenMed 2005;7:76.

25. Whelan G. High-stakes medical performance testing: the clinical skills assessment program. JAMA 2000;283:1748PMID: 10755506.

26. Clauser BE, Harik P, Margolis MJ, Mee J, Swygert K, Rebbecchi T. The generalizability of documentation scores from the USMLE Step 2 Clinical Skills examination. Acad Med 2008;83(10 Suppl):S41-S44. PMID: 18820498.

27. Taylor ML, Blue AV, Mainous AG 3rd, Geesey ME, Basco WT Jr. The relationship between the National Board of Medical Examiners' prototype of the Step 2 clinical skills exam and interns' performance. Acad Med 2005;80:496-501. PMID: 15851465.