QA Engineer (AI + backend)

Engagement: We are partnering with a US-based health-tech company on the takeover of a production AI-powered mobile coaching platform. Real users, real production, real releases - not a greenfield project.

Role summary: Own quality on the two hardest surfaces of a production AI coaching platform:

the LLM agents that answer differently every run, and the API layer everything else depends on.

As AI & Backend QA Engineer you own the quality of what the model says and what the APIs return. On the AI side that means building the evaluation practice: golden sets, quality metrics, and regression detection that catch a degraded answer before a user does. On the backend side it means API test suites and consumer-driven contract testing with Pact - agreed with the client's US engineering team as a standing requirement for this role, not an optional technique.

The defining skill is localization: when an answer is wrong, you can tell whether the problem sits in the model, in the service code, or in the input data - and you bring the evidence, not a guess.

What you'll be testing

• AI core - a Python service with LangGraph agents producing streaming, non-deterministic coaching responses; this
• role's primary surface
• Backend APIs - a NestJS REST gateway behind Auth0, plus the services behind it
• Mobile app - React Native, shipping to iOS and Android app stores; AI features surface directly in the app that
• members use
• Data layer - MongoDB, Redis, and Postgres behind the services
• Process - Agile/Scrum, Jira for work tracking and defects, TestRail for test management
• You join the Polish QA function alongside our Manual QA Engineer and QA Automation Engineer, and work in a shared workflow with the client's US-based engineering and QA teams.

What you'll do

• Build and run LLM evaluation. Curate datasets and golden sets, define response quality metrics, and catch quality regressions between prompt and model versions before they reach production
• Define acceptance criteria for non-deterministic output. Assertions on properties and contracts - structure, constraints, invariants - rather than exact wording
• Treat prompts as tested artifacts. Change a prompt, measure the effect of that change; probe systematically for hallucinations and prompt injection
• Own API testing. REST endpoints, JSON payloads, auth token flows, error handling - driven through Postman and Swagger/OpenAPI specs
• Implement contract testing with Pact. Consumer-driven contracts between the mobile app, the gateway, and the backend services, verified on every change
• Test with security in mind. Authorisation boundaries, token misuse, malformed input - the API-level abuse cases nobody writes a happy-path test for
• Localise failures across the stack. Logs, traces, and database queries to pin a defect to the model, the service code, or the input data
• Help wire evaluations into CI/CD. Evaluation runs in the pipeline and quality gates on pull requests, so a bad prompt change cannot merge quietly
• Keep the record straight. Results mapped to test cases in TestRail, defects logged and driven to closure in Jira, findings written up so the team can act on them

Must-have skills

• 3+ years in technical QA or test automation, including API testing on production systems
• Python - writing and maintaining test code, pytest, fixtures
• Strong HTTP and API fundamentals - REST, JSON, status codes, headers, auth tokens
• Contract testing with Pact - consumer-driven contracts in practice; this is a hard requirement for the role, agreed with the client's engineering team
• Security-aware API testing - authorisation boundaries, token handling, malformed and hostile input
• Comfort with non-determinism - defining acceptance criteria when there is no single correct output: assertions on properties and contracts rather than exact wording
• LLM evaluation - datasets and golden sets, response quality metrics, catching quality regressions between prompt and model versions
• Prompt engineering - the ability to change a prompt and measure the effect of that change; awareness of hallucinations and prompt injection
• Backend savvy - HTTP and REST, JSON, tokens, logs, database queries; you can tell whether a problem sits in the model, in the service code, or in the input data
• Postman and Swagger/OpenAPI as daily tools
• Git in a team workflow - branches, pull requests, reviews
• TestRail for test-case management and run tracking
• Mobile product context (Android/iOS) - comfortable testing features that ship in a mobile app, including apprelease timing and store builds
• Strong English - you work in a shared workflow with US colleagues
• Must work hours that overlap US Mountain Time afternoons (~14:00-18:00 PL time)

Nice-to-haves

• Langfuse or equivalent (LangSmith, Phoenix) - working with traces and attaching evaluations to them
• LangGraph - testing an agent as a state graph, both a single node and the full flow
• CI/CD - running evaluations in a pipeline and quality gates on pull requests
• Python with Appium - mobile automation exposure alongside the API work
• Testing other AI or LLM-driven features in production
• Experience taking over quality ownership of someone else's product

Offer

• Location: 100% remote
• Engagement: B2B contract
• Duration: Initial 3-month contract, expected to extend
• Rate: upt to 160 PLN/h
• Start: asap