The idea for this post came to me during a regular meeting with my fellow mentors, SDETs from several international companies. We were discussing the future of the QA Automation market and reached some rather interesting conclusions.
If you are using LLMs to evaluate other LLMs, it is a bad idea to use adjectives for your evals - better to use Rubrics instead. Let's consider an example:
If you are using LLMs to evaluate other LLMs, it is always better to have a golden standard to compare with. Otherwise, your LLM-as-a-judge relies on its own memory and can hallucinate much more, especially if your texts are professional.
Why the traditional QA "A + B = C" rule is broken for AI application testing and what to do? Traditional QA, armed with AI tools, essentially does not differ from testing without them.
ML evaluation take-home task: how does it look like and what do employers expect?
If you're thinking about transitioning from QA to ML engineer, you should learn the main concepts of large language models (LLM) and the evaluation of their output. One of the key conceptions is evaluating a smaller model's work by a bigger model, instead of or together with a human evaluation.
If you think ML Evaluation is just "testing of AI applications," come spend a day in my shoes.09:30 - 10:30 The Architectural ShiftStarted the day with a sync on our AI agentic workflow. The development team is introducing a new Agent.
There is a growing terminology crisis in the QA market, and it is bleeding into hiring decisions and team structures. Three distinct roles are being lumped under the same "AI Testing" umbrella, despite having almost nothing in common at the technical level.
The shift from automating deterministic systems to evaluating probabilistic models is not a lateral career move — it's a fundamental change in how you define "correctness." A test that asserts `expected == actual` becomes meaningless when the system under test produces non-deterministic outputs, and traditional pass/fail assertions give way to statistical thresholds, distribution analysis, and metric-based evaluation. Here's a condensed breakdown of what the transition actually required:
