AI, ML and LLMs for Testers — What AI Can and Cannot Do in the STLC

Hands-on Lab: Measure hallucination and non-determinism yourself

You need any chat assistant you are allowed to use (ChatGPT, Claude, Gemini or Copilot Chat), Python 3.10+ and a browser. Use only public information - we test the public demo shop saucedemo.com.

1. Create a folder ai-lab-01. Open a new chat and send exactly this prompt:

   You are a QA engineer. List 8 test cases for the login page of https://www.saucedemo.com.
   For each give: ID, title, test data, expected result. Use a plain numbered list.

2. Copy the answer into run1.txt. Open two more new chats, send the identical prompt and save run2.txt and run3.txt.
3. Save this script as compare_runs.py in the same folder and run python compare_runs.py:

   import difflib
   import itertools
   import pathlib
   import sys
   
   files = sorted(pathlib.Path(".").glob("run*.txt"))
   if len(files) < 2:
       sys.exit("Save at least two AI responses as run1.txt, run2.txt ...")
   
   texts = {f.name: f.read_text(encoding="utf-8") for f in files}
   for a, b in itertools.combinations(texts, 2):
       ratio = difflib.SequenceMatcher(None, texts[a], texts[b]).ratio()
       print(f"{a} vs {b}: {ratio:.0%} similar")
   
   line_sets = [
       {line.strip().lower() for line in t.splitlines() if line.strip()}
       for t in texts.values()
   ]
   common = set.intersection(*line_sets)
   print(f"Lines identical in ALL runs: {len(common)}")

4. Expected result: similarity is usually well below 100% and very few lines are identical across all runs. Write one sentence in findings.md: "The same prompt gave different test sets; I must review the final artefact, not trust a single run."
5. Fact check: open saucedemo.com. The login page itself lists the accepted usernames (standard_user, locked_out_user, problem_user, performance_glitch_user, error_user, visual_user) and the password secret_sauce. Compare with the test data the AI used. Mark each AI test case as Correct, Wrong data or Invented behaviour (for example "account locks after 3 failed attempts" - saucedemo has no such rule).
6. Hallucination probe: in a new chat ask: "In Selenium 4 Java, how do I use driver.findElementByText() to click the Login button?" No such method exists in Selenium WebDriver; Selenium 4 locates elements with driver.findElement(By...). Note whether the assistant corrects you or invents usage. Verify on the official Selenium finders page.
7. Fill this verification log in findings.md and keep it - you will extend it in later lessons:

   | Prompt | Tool + date | Claim checked | Source used to verify | Verdict |
   |--------|-------------|---------------|-----------------------|---------|
   | Login TCs run1 | <tool>, <date> | locked_out_user error text | saucedemo.com live | Correct |
   | findElementByText | <tool>, <date> | method exists | selenium.dev docs | Hallucinated |