Pre-Health Advisors Share Concerns About SJTs

Last Updated on October 6, 2023 by Laura Turner

The ascent of Situational Judgment Tests (SJTs) in admissions, including Casper and AAMC PREview, has spurred frustration and concern among health professional school applicants and their advisors. In June 2023, the National Association of Advisors in the Health Professions (NAAHP) unveiled a report shedding light on pre-health advisors’ insights and reflections about these assessments, echoing the need for clarity and transparency regarding using test results in admissions. The report voices concerns over how the assessments are shaping the admissions landscape.

NAAHP Testing Task Force

The NAAHP convened a Testing Task Force (TTF) in September 2022 following discussions at the 2022 NAAHP National Conference in Denver in June and on their list-serv. The TTF was charged to query the NAAHP membership about their concerns and to present their findings to the NAAHP Executive Committee and external stakeholders. The TTF solicited feedback from advisor members via a survey and conducted follow-up listening sessions. 54 NAAHP members responded to the survey, and 56 participated in the listening sessions. Additionally, 32 patron members belonging to health professional programs and professional organizations provided additional feedback at the listening sessions.

Advisor and Program Concerns About SJTs

The final report from the Testing Task Force identified multiple concerns about Situational Judgment tests:

1. Technical concerns, including proctoring issues with AAMC PREview (Meazure Learning platform) and responses to Casper scenarios
2. Transparency concerns about how scores are used in admissions processes and access to scores by advisors and students
3. Ethical concerns related to the lack of transparency about how scores are used and the pressure applicants feel to take SJTs even if only “recommended,” as well as concerns that schools that are using SJTs for research are pushing the burden of cost onto applicants
4. Concern that SJTs are increasing, rather than decreasing, barriers to access as they add cost and stress to the process and as there are race and demographic differences between groups that could potentially add bias to the admissions process
5. Concerns about monetary gain by test prep companies, despite Casper and PREview test administrators claiming that test prep outside of their tutorials does not increase scores
6. Concerns about the lack of evidence that SJTs can correctly predict success in medical school
7. Concerns about the rise of SJTs in medical residency selection despite uncertainties about the correlation between medical school admission and residency selection SJT scores

Admissions officers told the TTF they used SJTs to manage a large applicant pool and identify candidates with future professionalism issues. The TTF also heard from programs that had discontinued the use of SJTs, who stated that they were concerned about the additional barriers to admissions that the tests may cause, the introduction of bias, and redundancy with other parts of the application or interview process.

Interestingly, of all the health professional school associations, the TTF report states that only the AAMC has endorsed using SJTs in the admissions process. The AAMC administers the PREview test, charging applicants $100 to complete the test on the applicant’s own device using online proctoring. This fee is waived for students who qualify for the AAMC Financial Assistance Program.

Advisor SJT Concerns Echo Issues Raised by Students, Others

The concerns raised by the NAAHP TTF mirror those raised by applicants required to take the exams and third-party observers.

In a Situational Judgment Test Experience survey conducted by the Health Professional Student Association (publishers of SDN) in September 2022, respondents reported that their Casper score reflected their typing ability rather than the attributes it was meant to assess. Students also raised concerns about the “invasive” proctoring required for PREview, also mentioned in the TTF report.

The self-reported scores in the survey also indicated that the tests may increase bias, as the PREview scores showed a skew towards lower quartiles for underrepresented minority applicants. Casper results were more evenly distributed in the survey results.

Survey respondents also noted that their test results often changed widely between administrations of the exam, which belies the assertions of test developers that the assessments measure innate qualities and scores do not change with preparation. Student Doctor Network expert gyngyn, an admissions committee member, counseled a student worried about a disparity in PREview and Casper scores that “my experience with these tests shows no inter-test reliability. The same person taking the same test (PREview) a second time is as likely to get a score two quartiles away as not. Your result is but another example of how little they can be relied upon.”

Likewise, in a letter to the editor of Medical Education, medical educators from the University of Oxford cited research that coaching medical school applicants brought about a 0.5 standard deviation improvement in SJT scores.

SJTs: What’s the Evidence?

Research on SJTs has been published for over a decade (Patterson et al., 2012). Although SJTs have been used to assess and predict professionalism competencies of first-year (Wolcott et al., 2019) and second-year (Smith et al., 2022) pharmacy students, SJTs have also been used to identify potential professionalism issues among medical residency trainees. Surgery residency directors found SJTs to have better reliability with first-year residency performance results than emotional intelligence (“EQ”) tests (Gardner et al., 2017). Since then, SJTs and other non-cognitive assessments have been considered for selecting residents as the number of applicants per program increases (Cullen et al., 2020; Cullen et al., 2022; Takacs and Tracy, 2022). 

Acuity Insights’ website lists several case studies that claim Casper results correlate with specific measures gauging professionalism in health professions programs, claiming over 90% of all US “medical school” applicants take Casper. Casper has also been required for a few dental schools (Case Western Reserve University, University of Louisville), including international dentist programs (University of Colorado).

Designed for medical school admissions, the PREview exam’s reliability and validity were studied at eight allopathic medical schools in 2017 (AAMC Using PREview data, 2023), but these results have not been publicly published. Unlike the MCAT, examinees cannot challenge a PREview question or request a PREview rescore (AAMC PREview Essentials, 2023), raising their anxieties about the exam further.

Many are skeptical about the validity of the Casper studies, citing issues including a lack of score reproducibility, insufficient transparency (rubrics, scenario design, expertise of third-party raters), and concerns about conflicts of interest. Although SJTs offer a standardized way to compare preprofessional competencies among applicants, examinees and advisors remain frustrated when SJT scores do not confirm their evaluations as documented in pre-health committee letters. Applicants worry that their dreams of becoming a health professional will be crushed by a bad result from a “recommended” SJT, and our 2022 survey results suggest that non-medical health professional programs likely prefer candidates with high-quartile Casper results. The TTF found none of the schools had discussed sharing insights about using Casper in admissions decisions among their peers. 

NAAHP Testing Task Force Recommendations

The NAAHP Testing Task Force made several recommendations for the test developers and programs that utilize these SJTs. 

Test Developer Recommendations

For test developers, the TTF identified an obligation to ensure tests are used appropriately and that the testing calendar is set up to make data available earlier in the application cycle. They also identified a need to increase transparency in several areas:

• Score reporting, so that advisors and applicants receive the same score reports that programs receive
• Requirements for tests, including the ability to carry over results from prior years
• What qualifies applicants for testing accommodations and prompt responses to requests for accommodations

Health Professional Program Recommendations

The TTF recommended that programs ensure that they interpret and utilize SJT scores equitably and transparently, providing clear guidance to students about how and at what stage these scores are used in admissions. For programs that are using scores for “research purposes,” they recommended that programs bear the burden of the costs of tests and scores rather than pushing these costs to applicants. Finally, programs that do not use PREview results should hide PREview information from their admissions committee members in AMCAS to ensure that members are not biased by the scores.

Some Schools Run Their Own SJTs

Needing more non-cognitive data for holistic applicant review, many health professions programs have found platforms to conduct their own Casper-like SJTs that use their own questions and faculty evaluators. Applicants to many dental, pharmacy, physical therapy, veterinary, and other health professional programs have been surprised by a Kira Talent assessment requirement this cycle. In addition, Washington University of St. Louis School of Medicine included a Standardized Video Interview in the 2023-2024 cycle. The admissions committees can better control the number of questions applicants see and can directly evaluate responses rather than trusting third-party raters (which is the case for Casper). Furthermore, the results from these SJTs are confidential data for the admissions committee, so – like MMI interview scores – these results will not be shared with the advising community or subject to the above recommendations. Consequently, applicants taking these school-run SJTs will likely receive sparse feedback to improve their application.

Conclusion: Prepare for Future SJTs

SJTs have become integral to the application process for health professions programs and residencies. Anyone seeking to become a healthcare professional should prepare to navigate at least one SJT. While the test constructors Acuity Insights and AAMC/Meazure Learning respond to student and advisor concerns, applicants must remain hopeful that these assessments are designed with their well-being and the welfare of future patients in mind. 

While you cannot prepare for the specific content of SJTs, refining the decision-making skills essential in personal, academic, and clinical settings early is paramount. The integration of SJTs puts situational judgment on par with scientific knowledge, recognizing it as a foundational component of clinical skills. We must trust that SJTs will reward competent and diverse future clinicians.

Additional References

Association of American Medical Colleges. “2023 PREview Essentials, July 26, 2023,”https://students-residents.aamc.org/media/13296/download, accessed September 26, 2023.

Association of American Medical Colleges. “Using AAMC PREview Data in 2023 Medical School Student Selection, September 2022,” https://www.aamc.org/media/61226/download, accessed September 26, 2023. 

Cullen, M. J., Zhang, C., Marcus-Blank, B., Braman, J. P., Tiryaki, E., Konia, M., Hunt, M. A., Lee, M. S., Van Heest, A., Englander, R., Sackett, P. R., & Andrews, J. S. (2020). Improving Our Ability to Predict Resident Applicant Performance: Validity Evidence for a Situational Judgment Test. Teaching and learning in medicine, 32(5), 508–521. https://doi.org/10.1080/10401334.2020.1760104. 

Cullen, M. J., Zhang, C., Sackett, P. R., Thakker, K., & Young, J. Q. (2022). Can a Situational Judgment Test Identify Trainees at Risk of Professionalism Issues? A Multi-Institutional, Prospective Cohort Study. Academic medicine: journal of the Association of American Medical Colleges, 97(10), 1494–1503. https://doi.org/10.1097/ACM.0000000000004756. 

Gardner AK, Dunkin BJ. Evaluation of Validity Evidence for Personality, Emotional Intelligence, and Situational Judgment Tests to Identify Successful Residents. JAMA Surg. 2018;153(5):409-416. https://doi:10.1001/jamasurg.2017.5013.  

Patterson, F., Ashworth, V., Zibarras, L., Coan, P., Kerrin, M., & O’Neill, P. (2012). Evaluations of situational judgement tests to assess non-academic attributes in selection. Medical education, 46(9), 850–868. https://doi.org/10.1111/j.1365-2923.2012.04336.x. 

Smith, K. J., Neely, S., Dennis, V. C., Miller, M. M., & Medina, M. S. (2022). Use of Situational Judgment Tests to Teach Empathy, Assertiveness, Communication, and Ethics. American journal of pharmaceutical education, 86(6), 8761. https://doi.org/10.5688/ajpe8761. 

Takacs, E. B., & Tracy, C. R. (2022). Evaluating the Whole Applicant: Use of Situational Judgment Testing and Personality Testing to Address Disparities in Resident Selection. Current urology reports, 23(11), 309–318. https://doi.org/10.1007/s11934-022-01115-8. 

“What is a situational judgment test?” Acuity Insights Blog, June 2023. https://acuityinsights.com/admissions/guide-to-situational-judgment-tests-2/, accessed September 26, 2023.

Wolcott, M. D., Lupton-Smith, C., Cox, W. C., & McLaughlin, J. E. (2019). A Five-Minute Situational Judgment Test to Assess Empathy in First-Year Student Pharmacists. American journal of pharmaceutical education, 83(6), 6960. https://doi.org/10.5688/ajpe6960.