Software Engineer in Test, LLM AI & Automation Focus @ Natera
POSITION LOCATION: San Carlos, CA, Austin, TX, or Remote USA POSITION SUMMARY: We are seeking a Software Engineer in Test ( AI & Automation Focus) to join a forward-thinking engineering team responsible for building and evaluating AI-powered systems, including a sophisticated voice assistant platform to help patients get the help they need fast. This is not a traditional SQE role - this is a unique opportunity to blend test automation, machine learning evaluation, and data analysis into one impactful position.You'll design test automation frameworks, evaluate large language model (LLM) performance, and develop custom tooling to analyze, monitor, and continuously improve product quality in both traditional software and AI-driven components. KEY RESPONSIBILITIES: Design, build, and maintain robust, data-driven test automation frameworks JavaScript/TypeScript and Python. Build automated evaluation pipelines and prompt-based tests for LLM and conversational AI systems. Analyze large-scale, unstructured logs and telemetry data to identify issues and user experience gaps. Create custom scripts to support test automation, log parsing, and performance evaluation of machine learning models. Develop visual dashboards and reports to track quality, system health, and model behavior over time. Support test environment setup and integrate with CI/CD pipelines for continuous testing. Perform root cause analysis, debugging, and reporting of product issues, collaborating closely with Developers and SQEs across the SDLC. Drive innovation by introducing new testing methodologies, tools, and best practices for both deterministic and non-deterministic systems. Partner with global, cross-functional teams including ML engineers, product managers, and infrastructure leads. QUALIFICATIONS: BS degree in Computer Science or related technical field. 3+ years of experience in test automation, ML evaluation, data analysis, or related roles. KNOWLEDGE, SKILLS, AND ABILITIES: Proficient in JavaScript/TypeScript and Python; strong scripting experience, especially for automation and log parsing. Experience with tools such as Jest, Playwright, Cypress, or Selenium for UI and API automation. Strong… Apply To This Job