Part I: Red Flags on Residency Applications - Diving into the Data

In my work, many applicants applying for residency have at least one red flag, which they often hesitantly report to me during our initial coaching session. It's important to understand exactly what these flags are and know how to handle these potentially negative aspects of one's application professionally and competently.    

I will address this subject in two parts.

• Part I will provide data about red flags to ensure you understand how Program Directors perceive them.

• Part II will provide a roadmap detailing how to address and overcome them in your application and interview. 

Red flags typically constitute the following:

• Multiple USMLE /COMLEX Attempts or Low USMLE/COMLEX Score(s)

• Issues during medical school related to failures, repeat coursework or unexpected breaks in one's academic record, and professionalism concerns.

• Year of graduation ≥ specified number of years from current date

• Time gaps (may be related to financial issues, personal or family illness, extenuating circumstances in one's country [civil unrest/war] or other). 

I would also mention here that plagiarism on one's ERAS application and/or deceptive or fabricated statements could easily merit your exclusion from consideration, although they are not generally listed as red flags. And it goes without saying that a prior NRMP Match violation or misdemeanor or felony conviction—generally DUI or other drug-related—would be highly likely to land your application in the rejection pile. 

Other red flags relate to applicants who have been terminated or resigned from their programs and are re-applying for residency, but these are uncommon and always describe very personal circumstances and will not be addressed here.

THE DATA

What does available data inform us about what program directors think about red flags? Fortunately, there is some valuable information to share. Understanding the weight given to some of these factors by program directors is not intended to discourage any of you as match aspirants but instead to highlight red-flag challenges. I'll later reinforce the importance of appropriately addressing these in one's residency application.

USMLE Scores, Medical School Performance and LORs/MSPE

The NRMP's 2020 Program Director Survey (1) provides the following information regarding failed attempts on the USMLE and COMLEX exams. Program Directors were asked if they would consider extending invitations to applicants with first-attempt failures. For both the USMLE and COMLEX exams, program directors were more "forgiving" of early failures (Step 1 and COMLEX 1) than later failures, particularly Step 2 and COMLEX 3). IMG applicants sometimes take USMLE Step 3 as a strategy to partially mitigate a non-competitive Step 2 CK result. It is, therefore, important for those with such red flags considering such a tactic to fully recognize that a subsequent Step 3 failure would likely further reduce their residency opportunities. 

Finally, It is critical to recognize that this is aggregated data across specialties with some specialties having greater willingness to consider an applicant with a standardized test failure. Below I have compiled data from Program Directors in various specialties in regards to consideration of applicants who have failed Step I or Step 2 CK exams. As you can see, all things are not equal.

Internal Medicine residency program directors' screening practices were examined by Angus et al. in Academic Medicine (2). Red flags considered the most important exclusion criteria for interview selection were failures on Step 2 CK and unprofessional behavior noted on a candidate's LORs or MSPE. Of the program directors who responded (n=231), 47% indicated that they "absolutely would not invite" a failed attempt on Step 2 CK, and 68% stated the same for those candidates with unprofessional behavior and judgment.     

However, there is another way to interpret this data. One can take away the message that 66% of program directors who encounter someone with a first-attempt failure on USME Step I might still grant them an interview. An initial Step 2 CK failure, while looked upon with greater concern, also did not uniformly eliminate all applicants from the running. It is worth noted that a professionalism lapse is perceived to be a far worse red flag (and in my perspective for good reason), than a standardized testing failure. Such information signals a willingness of some program directors to look past certain data points in an applicant's file, especially in programs that have embraced a more comprehensive holistic approach to residency application review. 

A narrative review by Hartman et al. (3) examined the most common and highest-rated factors used to select applicants for interviews. The presence of a failed USMLE/COMLEX exam was used by 70% of program directors as an interview selection criterion and was rated as 4.5 out of 5.0 in terms of overall importance. Other application factors utilized relevant to this discussion were professionalism and ethics considerations (percent used 65%; importance 4.5) and applicant with match violation (37%; 4.8), 

In a another questionnaire sent to Internal medicine program directors (4), they were asked to indicate which one out of predefined criteria was the most common data point used to sort/filter applicants if they employed such filters.   While Step 2 CK score was reported as the most common data point selected to filter applicants (32%), the presence of a failing clerkship grade in medicine was listed by only 9% of PDs. 

In 2023, Hemrajani et al. (5) found that "flags" from either the interview or file review were statistically significant (p <.003; OR 2.82) in terms of their association with subsequent performance-related difficulties during residency. Although such flags were not explicitly defined, they note that typical areas of concern include course/clerkship failures, legal issues, and portfolio inconsistencies.

In a study of Anesthesia program directors from 2020 (6) , 84% considered red flags very important when selecting an applicant for an interview. The four most common "red flags" were: failure of USMLE exams, failure of a course or clinical rotation, gaps in education/missing time from school without explanation, and any felonies/other criminal history".

Strasser et al. (7) evaluated the importance of applicant factors by program directors and found that passing the USMLE was considered to be "extremely or very important" by 88% of respondents. An applicant's grades on core clerkships were also deemed highly important, and it is reasonable to infer that failing grades or repeated rotations would raise strong concerns.

While this is not an exhaustive list of all studies, there is considerable consistency present. Furthermore, at least in terms of test performance, ample literature supports the fact that USMLE scores are predictors of residency performance, at least in certain domains.

Sharma et al., (8) looked at post-match residency multimodal performance (standardized testing, individual clinical assessments). They found that in multivariate analysis, the USMLE Step 2 score was "the most predictive of performance in all residency performance domains measured."   In contrast, USMLE Step 1 scores were associated only with subsequent in-training examination scores. 

In a meta-analysis of 80 studies with 41,704 participants (9), the authors found that "standardized examination performance and medical school grades showed the strongest associations with current measure of doctor performance. 

However, Lipman JM et al. in more recent 2023 study published in the Journal of Graduate Medical Education (10), which aimed to summarize the evidence linking elements in the ERAS application with selection and training outcomes, concluded that "the association of Step 1 and Step 2 CK scores with performance metrics are widely mixed." 

Finally, each Program Director will harbor his or her beliefs regarding standardized test scores and their association with the subsequent ability of their graduates to pass specialty boards. For instance, Rayamajhi et al. (11) found that USMLE Step 1 (before becoming P/F), Step 2 CK, and Step 3, along with the In-service exam during the third year of residency, predicted the results on the Internal Medicine Certifying exam (ABIM). 

The Accreditation Council for Graduate Medical Education (ACGME) has specific cut-offs for each specialty regarding the percentage of graduates who must pass their specialty boards (rolling average over a specified number of years). Failure to achieve these benchmarks can have ramifications for programs including ACGME program citations and applicants' rank order list decisions.   It is also worth noting that a resident who fails to pass Step 3 during residency cannot graduate and can risk dismissal.

Year of Graduation (YOG) and Time Gaps

In terms of graduation year, programs that exclude older applicants indicate such clearly on the   Freida™ AMA Residency database by specifying their YOG requirements. Programs can use a candidate's YOG as a screening criterion through ERAS.     While older candidates usually apply to programs that do not list YOG restrictions or have more liberal inclusion criteria, these cut-off dates are not always set in stone. I will tell you that I have coached candidates with YOG > 10 years who did not meet the prescribed cut-offs yet still interviewed and matched.  

While the literature on the impact of a "gap" year for recent grads between undergraduate education and medical school is readily available, there is no comparable data that I could find regarding intervals between medical school and residency application in the US. Applicants with sizable gaps are almost all IMGs, each with a different story. The Freida™ database indicates which programs will consider those with gaps. As with YOG, in my experience, these rules are generally not written in stone. 

The 2024 NRMP Program Director Survey (aggregated for all specialties) does indicate that 43% of all program directors who responded reported using "evidence of continuous medical education without gaps" as one of many criteria for granting interviews with a mean importance rating of 3.9 (scale 1-5). Only 30% of program directors considered this criterion when they subsequently turned to the ranking process (mean importance at 4.0).

Summary of Data

• Failures on USMLE or COMLEX exams clearly represent red flags, but for many can be overcome. For almost all subspecialties, a failure should NOT be construed as an insurmountable obstacle to residency.

• Failures carry more weight progressively on both USMLE and COMLEX exam as one moves from the preclinical exams to the clinical exam content. This is particularly important to understand when developing an application strategy which may include taking Step 3 or COMLEX-3 to offset an earlier poor score or exam failure.

• The effect of exam failure on one's application can be expected to vary from specialty to specialty.

• In terms of "application harm", evidence of professionalism lapses are harder to overcome than first-time standardized exam failures.

• Please note that this information applies to single exam failures and does not necessarily apply to those with multiple failures (on either one part or multiple parts) of the USMLE or COMLEX exam series.

Part II of this series provides detailed information regarding potential ways to effectively respond to any red flags which your application may harbor. 

_____________________________

(1)   NRMP Program Director Survey 2022 - www.nrmp.org

(2)   Steven V Angus et al. Internal Medicine Residency Program Directors’ Screening Practices and Perceptions About Recruitment Challenges. Acad Med. 2020 Apr; 95(4):582-589.

(3)   Nicholas D. Hartman, Cedric W. Lefebvre, David E. Manthey; A Narrative Review of the Evidence Supporting Factors Used by Residency Program Directors to Select Applicants for Interviews. J Grad Med Educ 1 June 2019; 11 (3): 268–273.

(4)   Adam M. Garber, Brian Kwan, Christopher M. Williams, Steven V. Angus, T. Robert Vu, Matthew Hollon, Marty Muntz, Arlene Weissman, Anne Pereira; Use of Filters for Residency Application Review: Results from the Internal Medicine In-Training Examination Program Director Survey. J Grad Med Educ 1 December 2019; 11 (6): 704–707. 

(5)   Reena Hemrajani et al. Association of Interview and Holistic Review Metrics with Resident Performance-Related Difficulties in an Internal Medicine Residency Program. J Grad Med Educ. 2023 Oct;15(5):564-571.

(6)   Vinagre R et al. Red Flags, Geography, Exam Scores, and Other Factors Used by Program Directors in Determining Which Applicants Are Offered an Interview for Anesthesiology Residency. Cureus. 2020 Nov 18;12(11): e11550. 

(7)   Strausser et al. Importance of residency applicant factors based on specialty and demographics:  a national survey of program directors.  BMC Medical Education (2024) 24:275.

(8)   Sharma et al., USMLE Step 2 CK: Best Predictor of Multimodal Performance in an Internal Medicine Residency. J Grad Med Educ. 2019 Aug; 11(4): 412–419.

(9)  Kenney S. et al., Association between residency selection strategies and doctor performance: A meta-analysis.  Medical Education 2013: 47: 790–800

(10)   Lipman, Jeremy M. et al. A Systematic Review of Metrics Utilized in the Selection and Prediction of Future Performance of Residents in the United States.  J Grad Med Educ (2023) 15 (6): 652–668.

(11)   Rayamajhi, S., Dhakal, P., Wang, L. et al. Do USMLE steps, and ITE score predict the American Board of Internal Medicine Certifying Exam results?  BMC Med Educ 20, 79 (2020).