QA Engineer – Quality Engineer

• Build and maintain the platform’s automated test suite, including E2E, integration, and regression tests
• Create mechanisms to evaluate responses generated by AI agents
• Develop golden datasets, scoring criteria, and regression tests for LLM-generated artifacts
• Determine whether changes to prompts or models have improved or degraded results
• Define Quality Gates and the criteria that may block a release
• Ensure the reliability of CI pipelines
• Identify and fix flaky tests and slow pipelines
• Perform exploratory and release testing
• Triage defects, assign severity, and track fixes through final validation
• Turn failure patterns identified during testing into insights for product and roadmap evolution

Requirements

• 3+ years of experience in QA, SDET, or Test Engineering
• Hands-on experience with test automation for products in production
• Strong knowledge of Python and/or TypeScript
• Experience with modern E2E testing frameworks such as Playwright, Cypress, or equivalent
• Experience with API testing
• Solid knowledge of CI/CD, including pipeline creation and maintenance
• Ability to test systems with non-deterministic responses
• Experience evaluating LLM or Machine Learning outputs, or the ability to develop a consistent approach to doing so
• Ability to accurately assess defect severity and impact
• Use of AI tools in day-to-day work
• Self-directed, with the ability to establish processes in environments that do not yet have a defined model
• Previous experience at early-stage companies, preferably having helped build the QA function or processes from the ground up
• Experience evaluating LLM-based features or AI agents in production
• Knowledge of evaluation design (evals), hallucination detection, and Human-in-the-Loop (HITL) review processes
• Familiarity with enterprise software ecosystems, especially SAP, Salesforce, ERP, and CRM
• Experience with SAP testing tools such as Tricentis or Worksoft
• Experience with SAP testing cycles and UAT
• Knowledge of performance, load, or security testing
• Sufficient programming ability to not only identify a bug but also potentially fix it
• Professional to fluent English

Core Competencies

Demonstrates expertise in test automation, particularly in building and maintaining automated test suites for AI-driven products. Proficient in evaluating LLM outputs and ensuring the reliability of CI/CD pipelines while effectively managing defect triage and testing processes.

Highest-signal resume keywords

• Test Automation
• Python
• E2E Testing Frameworks
• CI/CD Pipeline Management
• LLM Evaluation

ATS Optimization Keywords

Hard Skills

• Test Automation
• Python
• TypeScript
• E2E Testing
• API Testing
• CI/CD
• Defect Severity Assessment
• Performance Testing
• Load Testing
• Security Testing

Soft Skills

• Self-Directed
• Process Establishment
• Insight Generation

Industry Keywords

• QA
• SDET
• Test Engineering
• Human-in-the-Loop
• Evaluation Design
• Hallucination Detection

Tools & Technologies

• Playwright
• Cypress
• Tricentis
• Worksoft
• SAP
• Salesforce
• ERP
• CRM

#J-18808-Ljbffr