Using Simulation Modeling Approach to Predict USMLE Steps 1 and 2 Performances

Authors: Chau-Kuang Chen, John Hughes, Jr., A. Dexter Samuels

Abstract:

The prediction models for the United States Medical Licensure Examination (USMLE) Steps 1 and 2 performances were constructed by the Monte Carlo simulation modeling approach via linear regression. The purpose of this study was to build robust simulation models to accurately identify the most important predictors and yield the valid range estimations of the Steps 1 and 2 scores. The application of simulation modeling approach was deemed an effective way in predicting student performances on licensure examinations. Also, sensitivity analysis (a/k/a what-if analysis) in the simulation models was used to predict the magnitudes of Steps 1 and 2 affected by changes in the National Board of Medical Examiners (NBME) Basic Science Subject Board scores. In addition, the study results indicated that the Medical College Admission Test (MCAT) Verbal Reasoning score and Step 1 score were significant predictors of the Step 2 performance. Hence, institutions could screen qualified student applicants for interviews and document the effectiveness of basic science education program based on the simulation results.

Keywords: Prediction Model, Sensitivity Analysis, Simulation Method, USMLE.

Digital Object Identifier (DOI): doi.org/10.5281/zenodo.1126203

References:

[1] S.R. Poulter, Monte Carlo Simulation in Environmental Risk Assessment – Science, Policy, And Legal Issues, Risk: Health, Safety & Environment 7 (Winter 1998).
[2] M. J. O'Donnell, et al, Background essential to the proper use of results of step 1 and step 2 of the USMLE, Journal of Academic Medicine, Vol. 68, 1993.
[3] J. B. Erdmann, Guidance for the use of the USMLE in medical education Settings, Journal of Academic Medicine, Vol 68, 732-733, 1993.
[4] D. B. Swanson and D. R. Ripkey, S. M. Case, and the 1994-95 Validity Study Group for USMLE Step 1/2 Pass/Fail Standards, preliminary study of the accurate of the old and new medical college admission tests for predicting performance on USMLE step 1, Journal of Academic Medicine, Vol. 71, No. 1, 1996.
[5] C. Elam, NBME Part I versus USMLE Step 1: Predicting scores based on preadmission and medical school performances, Journal of Academic Medicine, Vol. 69, 1994.
[6] B. Silver and C. S. Hodgson, Evaluating GPAs and MCAT scores as predictors of NBME-I and clerkship performances based on students' data from one undergraduate institution, Journal of Academic Medicine, Vol. 72; 394-396, 1997.
[7] C.K. Chen, V.C. Campbell, and A. Suleiman. Predicting Student Performances at a Minority Professional School. AIR 2001 Annual Forum Paper. Available at http://files.eric.ed.gov/fulltext/ED457714.pdf
[8] E. R. Julian, Validity of the medical college admission test for predicting medical school performance, Journal of Academic Medicine, Vol. 80(10); 910-917, 2005.
[9] W. T. Basco Jr, D. P. Way, G. E. Gilbert, and A. Hudson, Undergraduate institutional MCAT scores as predictors of USMLE Step 1 performance, Journal of Academic Medicine, Vol. 77(10); S13-S16, 2002.
[10] C. A. Peterson and R. P. Tucker, Medical gross anatomy as a predictor of performance on the USMLE Step 1. The Anatomical Record Part B: The New Anatomist, Vol 283(1); 5-8, 2005.
[11] J. A. Koenig and A. Wiley, The validity of the medical college admission test for predicting performance in the first two years of medical school, Journal of Academic Medicine, Vol. 71; S83-S85, 1996.
[12] D. B. Swanson, et al, Relationship between achievement in basic science coursework and performance on 1994 USMLE step 1 test administrations, Journal of Academic Medicine, Vol. 71, 1998.
[13] A. Sawhill, et al, Using the NBME self-assessments to project performance on USMLE step 1 and step 2: impact of test administration conditions, Academic Medicine, Vol. 79, No. 10, October Supplement, 2004.
[14] D. Fone, et al, Systematic review of the use and value of computer simulation modeling in population health and health care delivery, Journal of Public Health Medicine Vol. 25, No. 4, pp. 325–335, DOI: 10.1093/pubmed/fdg075, 2003.
[15] P. Peduzzi, J. Concato, E. Kemper, T. Holford, and A. Feinstein, A Simulation study of the number of events per variable in logistic regression analysis, J Clin Epidemiol Vol. 49, No. 12, pp. 1373-1379, 1996.
[16] S. Poulter, Monte Carlo simulation in environmental risk assessment — science, policy and legal issues, 9 Risk: Health, Safety & Environment 7, Winter 1998.
[17] IBM Corporation, Better decision making under uncertain condition using Monte Carlo Simulation, Monte Carlo simulation and risk analysis techniques in IBM SPSS Statistics, IBM Software, Business Analytics, IBM SPSS Statistics, 2013.
[18] J. Kleshinski, S. A. Khuder, J. I. Shapiro, and J. P. Gold, Impact of preadmission variables on USMLE step 1 and step 2 performance, Advances in Health Sciences Education, 14(1), 69-78, 2009.
[19] C. L. Elam, and M. M. Johnson, Using preadmission and medical school performances to predict scores on the USMLE step 2 examination, Academic Medicine, 69(10), 852, 1994.
[20] Health Sciences Reasoning Test (HSRT), 2016. Available at: http://www.insightassessment.com/Products/ProductsSummary/Critical-Thinking-Skills-Tests/Health-Sciences-Reasoning-Test-HSRT.
[21] S.R. Simon, A. Bui, S. Day, D. Berti, & K. Volkan, The relationship between second‐year medical students’ OSCE scores and USMLE Step 2 scores. Journal of evaluation in clinical practice, 13(6), 901-905, 2007.