LLM-as-a-judge in QA terminology
If you’re thinking about transitioning from QA to ML engineer, you should learn the main concepts of large language models (LLM) and the evaluation of their output. One of the key conceptions is evaluating a smaller model’s work by a bigger model, instead of or together with a human evaluation. This bigger model is called LLM-as-a-Judge.
So, simply put, you use a more capable LLM (like a high-end model) to act as an automated test assessor. For AI output, traditional AQA assertions don’t work here because of 2 reasons:
1. The output is non-deterministic, and every test will be flaky.
2. It is quite complicated to define the criteria of a test pass/fail if we deal with, for example, textual similarity.
How does LLM-as-a-Judge usage look in practice?
- Define the Rubric: You define strict criteria (e.g., “Is the answer factually correct? Does it contain hallucinations?”). This acts as your test spec.
- Judge Prompting: You write a prompt for the Judge LLM with the rubric and provide it with the target model’s output and the original prompt. On this stage, automation is typically used, and an evaluation pipeline is created or updated. Python is the best choice, since it has a lot of libraries for LLM-as-a-judge automation.
- Execution: The Judge runs through the dataset, applying the rubric to each response. Better to do it multiple times for each dataset, to have the statistics, because of a non-deterministic output.
- Reporting: The Judge returns a score or a structured JSON response (Pass/Fail + Reasoning), which you can aggregate to calculate your general model quality.
Practically, you are testing for “Is the model’s logic compliant with our brand voice and safety guidelines?” across thousands of variations in minutes.
This is how LLM-as-a-Judge is mapped to a QA terminology:
- The Judge LLM (the source of truth) –> Test Oracle
- Human-in-the-loop (HITL) spot-checking (Re-checking the result by a human after LLM-as-a-Judge evaluation) –> Manual Review
- Prompts + Expected behavior guidelines –> Test cases
- The Rubric (e.g., “Score 1-5 on helpfulness/accuracy”) –> Acceptance criteria
- LLM scoring or classification of the output –> Assertion
If you are a QA engineer looking at the AI space, don’t feel like you’re starting over. You’re just upgrading your toolset.
