Ask HN: How does manual QA fit into your process?

We're seeing increasing efforts to improve developer cadence - LLM-improved auto-complete, agentic coding, agentic code review and agent-generated automated testing.

So how does Manual QA still exist in your process? Is it a blocker on your release process? Are QA's being flooded with tickets? Does every ticket get QA'd or is it more of a high level pass?

Or are you entirely reliant on automated testing to catch issues? What happens to manual QA in the short/long term?

1 points | by philipwhiuk 2 hours ago

2 comments

  • gojkoa 2 hours ago
    We keep manual QA for non-deterministic, exploratory stuff, things that do not have an expected input and output and where spotting secondary information and insights (e.g. "why is the chrome network console logging so much when I click on this button?"). things that have a deterministic input and expected output get automated. This whole area is a terminology minefield, but one term I like from the context-driven testing community for category 2 (deterministic) is "checking". The term I like for category 1 is "exploring". If a computer can "check" a post condition given some preconditions then a human does not need to waste time on it. If you need up to go after sidequests and look for unknown factors, it requires a human to explore. Having said that, there is a role for automation to speed up exploration (e.g. record a flow through the app for some key scenario, so a human can play it at 10x speed, prefill data to bring up an app into some well known state for exploration etc), but the judgement stays human. If the QA is flooded with tickets for exploration, giving them tooling to speed up data setups, cleanups, reviews etc might help.

    I've picked up a framework for deciding what/how much to test manually from James Whittaker's ACC matrix a while ago, and it's been incredibly helpful to create a policy around when something requires manual testing, how much and what the manual testing should actually cover. It's covered reasonably well in the book "How google tests software". Having a policy around that means that there's no misunderstanding what gets automated and by whom, and what's left for exploring and by whom.

  • stephan-repeato 2 hours ago
    I think the work of QAs is going to shift from executing the test cases themselves to instructing agents to do the work and checking if the execution was done properly.
    [-]
    • stephan-repeato 2 hours ago
      Of course that's already happening. But I think this is going to happen more and more