Experienced Summer Internship: Psychometrics and Data Science – Remote 2025 Summer Research Program

About NBME and the Summer Internship Program

National Board of Medical Examiners (NBME) is a leading organization in the field of medical education and assessment. Our mission is to advance assessment science and improve the quality of medical education and healthcare. We are committed to innovation, excellence, and collaboration, and we are seeking highly motivated and talented individuals to join our team as summer interns.

Role Overview

We are offering multiple full-time remote internship positions for the Summer of 2025. As a summer intern, you will have the opportunity to collaborate with our staff and interact with fellow graduate students as you complete a research project. The expected deliverable is an internal research presentation. Specific projects for the summer of 2025 will be discussed with applicants as part of the interview process.

Responsibilities

• Operational Research: This research supports NBME's current portfolio of assessments. Utilizing already defined capabilities and focusing on established constructs, this foundational programmatic research improves our existing measurement capabilities or gathers validity evidence for our assessments.
• Project Examples:
  • Modeling Item Revisit Behavior: The Hierarchical Speed Accuracy Revisit Model
  • Using Process Data to Improve Examinee Ability Estimation Precision
  • Evaluation of a Virtual Clinical Reasoning Performance Assessment
  • The Utility of Nonfunctioning Distractors for Credentialing Tests
  • Modeling the Conditions That Influence Response Time and Performance
  • Examining USMLE Step 1 Feature Usage: A Response Process Validity Study
  • The Association between USMLE Step 2 CK Scores and Patient Mortality: A Validity Study
• Innovation Research: This research expands NBME's capabilities to provide more insight into our assessments and examinee behavior. For example, NBME now has a dedicated natural language processing (NLP) team that has its own line of research and supports various other initiatives in Operational Research and Innovation Research.
• Project Examples:
  • Automated Short Answer Question Scoring with Adversarial Training
  • Automated distractor generation for clinical MCQs
  • Automatic Item Generation for Medical Assessment with Large Language Models: Challenges of Accuracy and Novelty Coupl
  • Using NLP to Explore Item Language Patterns as Part of Test Bias and Test Validity
  • Clinical Reasoning Scoring and Analysis
  • Automatic Enemy Item Detection Using Natural Language Processing: Latent Semantic Analysis

Qualifications

To be eligible for this internship, you must meet the following qualifications:

• Eligibility: Must be eligible to work June 2-July 25, 2025. We are unable to shift the timeline of the internship program.
• Education: Active enrollment in a doctoral program in measurement, statistics, computer science, cognitive science, medical education, or related field.
• Experience: Completion of at least one year of your PhD program by Summer 2025.
• Skills: Experience or coursework in one or more of the following: statistics, measurement theory, natural language processing, data science, research design, IRT, and test development.
• Software: Working knowledge of statistical software (e.g., R, Python, C#).
• Availability: Interns will be assigned to one or more mentors and must be able to work independently.
• Authorization: Must be authorized to work in the US for any employer. If selected, F-1 holders will need to apply for Curricular Practical Training authorization through their school's international student office and have a social security number for payroll purposes.
• Availability: Must not have any competing employment throughout the duration of the internship.
• Work Schedule: Must be available between NBME core hours (10am-3pm Eastern Time).

About NBME

NBME offers a versatile selection of high-quality assessments and educational services for students, professionals, educators, regulators, and institutions dedicated to the evolving needs of medical education and healthcare. To ensure our assessments meet the highest standards of quality, stay relevant, and align to the current curriculum in medical schools and training programs, we rely on a wide network of collaborators. These include the volunteers who help develop our exam questions, the committees and panels who represent various groups within the medical education community, external researchers, and health profession organizations.

Benefits and Compensation

We offer a competitive salary of $45 per hour, amounting to a total stipend of $12,600 for full-time interns (35 hours per week). Additionally, all interns are eligible to receive up to $1,000 to support their attendance at a conference. This is not conditional on presenting at the conference.

Work Environment and Culture

NBME is a remote-friendly workplace that values diversity, equity, and inclusion. We are committed to creating a work environment that is inclusive, respectful, and supportive of all employees. Our benefits package includes healthcare, dental, prescription, and vision plans, 401(k) with match, tuition reimbursement plan, commuter benefit, and more.

How to Apply

If you are a motivated and talented individual who is passionate about medical education and assessment, we encourage you to apply for this internship. Please submit your application, including a cover letter outlining your relevant experience and specific research programs or topics of interest, by Friday, January 31, 2025. We look forward to reviewing your application and discussing how you can become a vital part of our success story.