Advance QA roadmap
LLM Testing
LLM testing evaluates prompts, retrieval, hallucination risk, safety, structured output, and regression behaviour.
It helps testers bring evidence and repeatability to systems that do not behave deterministically.
Roadmap
Beginner
- Learn the purpose, vocabulary, and everyday QA situations where LLM Testing is used.
- Practise with small examples, clear acceptance criteria, and simple evidence notes.
- Create one reusable checklist or template that can be applied on a real feature.
Intermediate
- Apply LLM Testing across realistic product flows, edge cases, and release risks.
- Connect the skill to defects, traceability, test data, environments, and reporting.
- Review output with another tester or developer and tighten the evidence.
Advanced
- Turn LLM Testing into a repeatable workflow that supports delivery decisions.
- Automate or standardise the parts that repeat without hiding human judgement.
- Use metrics, examples, and lessons learned to improve the team process.
Practical checklist
- Define what good LLM Testing evidence looks like before starting.
- Confirm the feature, risk, user, environment, and data scope.
- Cover happy paths, negative paths, boundaries, and realistic user behaviour.
- Record assumptions, gaps, blockers, and follow-up questions.
- Share results in a format developers and stakeholders can act on.
Common mistakes
- Treating LLM Testing as a document task instead of a thinking workflow.
- Testing only the happy path and missing risk-heavy conditions.
- Using vague pass/fail notes that do not explain impact or evidence.
- Ignoring maintainability, repeatability, and stakeholder readability.
Interview questions
- How would you explain LLM Testing to a non-technical stakeholder?
- What risks would make LLM Testing more important on a release?
- How do you decide what to test first when time is limited?
- What evidence would you include in a QA sign-off summary?