How ML Systems Differ — Lifecycle, Risks and a QA Test Strategy — ML Model & Data Testing — Data Quality, Bias, Drift & AI Regulations

Why this matters on the job

Picture a sprint review at a Bengaluru fintech. The data science team has a new fraud model for UPI payments. It scored 98% accuracy in a notebook, and the product manager wants it live before the festive sale. The QA lead is asked one question: is it safe to ship? The usual checklist of UI flows, API status codes and regression packs does not answer it. A model can return HTTP 200 with a perfectly valid JSON body and still block thousands of genuine customers, or let fraud through because the training data was stale.

Companies hiring testers for AI projects - Indian IT services firms delivering for US, UK and EU banks, and product companies in fintech, e-commerce, insurance and healthtech - now expect QA to own data quality, model evaluation, fairness, robustness and monitoring as a test discipline. Regulators expect the same: the EU AI Act treats credit scoring of individuals as high-risk, and Indian regulators such as RBI have published responsible-AI expectations for financial services. This lesson gives you the mental model and a test strategy you can reuse in every later lab.

Concepts

1. Traditional software vs machine-learning systems

In traditional software, a developer writes the rules and you test that the code follows them. In ML, the developer writes a training procedure, and the rules are learned from data. That changes what can go wrong and what a tester must check.

Aspect	Traditional software	ML system
Where logic lives	Source code written by developers	Model weights learned from training data
Expected output	Exact and known for each input (test oracle exists)	Probabilistic; often no single correct answer (oracle problem)
Main defect sources	Code bugs, wrong requirements	Bad or biased data, leakage, wrong metric, wrong threshold, drift
Pass/fail	Assert equals	Assert metric above a gate, assert behaviour relations hold
Change over time	Only when code changes	Degrades silently when real-world data changes
Reproducibility	Same code, same result	Needs fixed seeds, versioned data, pinned libraries
Who is harmed by defects	Users of the broken feature	Whole groups of people, sometimes unfairly and invisibly

Two ideas from the table come up again and again. The oracle problem: for a single transaction you usually cannot say what the "right" fraud score is. So we test aggregate quality (metrics on a labelled test set) and relations between inputs and outputs (behavioural and metamorphic tests). And silent failure: a model never throws an exception when the world changes - it just gets worse.

2. The ML lifecycle and where QA plugs in

Most teams follow some version of this lifecycle. Learn the stages, because test plans, interview questions and audit reports are organised around them.

Problem framing - what decision does the model support, what is a positive class, which errors are expensive? QA asks for written acceptance criteria here, not after training.
Data collection and labelling - sources, consent, label definitions (when is a transaction "fraud": customer complaint, chargeback, or analyst decision?).
Data preparation - cleaning, joining, feature engineering. Most defects are introduced here.
Training and tuning - algorithm choice, hyperparameters, cross-validation.
Evaluation - metrics on a held-out test set, slices, fairness, robustness, behaviour.
Deployment - packaging, scoring API, shadow mode, canary or A/B rollout.
Monitoring and retraining - data drift, prediction drift, delayed-label performance, incident handling.

A useful rule: every stage produces an artefact you can test - a label guide, a dataset, a feature table, a trained model file, a metrics report, an API, a dashboard. Your test plan lists those artefacts and the checks for each.

3. The ML test pyramid

The classic test pyramid still works, with new layers:

Layer	What you test	Typical tools	Runs when
Data tests	Schema, nulls, ranges, duplicates, label rate, leakage	pandas, Pandera, Great Expectations	Every pipeline run
Code unit tests	Feature functions, preprocessing, metric code	pytest	Every commit
Model quality tests	Metrics vs gates, slices, champion vs challenger	scikit-learn, pytest, MLflow	Every training run
Behavioural tests	Invariance, directional, minimum functionality, metamorphic	pytest, CheckList-style suites	Every training run
Fairness and robustness	Group metrics, noise, edge cases, evasion	Fairlearn, custom perturbation code	Release candidates
System and API tests	Scoring endpoint contract, latency, fallbacks	pytest + requests, Playwright, k6	Every deploy
Production monitoring	Drift, delayed-label performance, alerts	Evidently, dashboards	Daily or weekly

Notice that the expensive human review (fairness sign-off, audit report) sits on top, and cheap automated checks run all the time. That is exactly how you will build things in Lesson 8.

4. Four example systems and their risks

System	Positive class	Costly error	Extra QA focus
UPI / card fraud detection	Fraud transaction	Missed fraud (money lost) and false blocks (angry genuine users)	Class imbalance, threshold choice, evasion, delayed labels
Credit scoring / loan approval	Good or bad credit risk	Wrongly rejecting good applicants, especially from one group	Fairness, explainability, regulation (EU AI Act high-risk, RBI expectations, US adverse action rules)
Product recommendation	Item the user will click or buy	Irrelevant or repetitive suggestions, popularity bias	Ranking metrics, cold start, regional and language coverage
Image classification (KYC document, medical scan, defect detection)	Defect / class label	Missing a real defect or a forged document	Robustness to blur, lighting, rotation, low-end phone cameras

5. A QA test strategy for an ML feature

Write the strategy as a short, versioned file in the repo so it is reviewed like code. Here is a realistic example for the fraud model:

# ml-test-strategy.yaml - one page, agreed with data science before the sprint starts
system: upi-fraud-scoring
model_owner: data-science
qa_owner: qa-ml
risk_level: high            # blocks real payments, affects customers directly
layers:
  data:
    - schema and range checks on every training and scoring batch
    - duplicate, null and label-rate checks
    - train/test leakage checks (shared IDs, post-event columns)
  model:
    - offline metrics vs agreed gates (PR-AUC, recall at chosen threshold)
    - slice metrics: new users, high amount, night-time, each bank / PSP
    - behavioural tests: invariance, directional, minimum functionality
    - fairness checks where people are affected (age band, region, gender if available)
    - robustness: noise, missing fields, extreme values, evasion attempts
  system:
    - API contract tests for the scoring endpoint (latency, schema, errors)
    - fallback rule when the model is down or times out
  production:
    - data drift and prediction drift dashboards, weekly
    - delayed-label performance (chargebacks arrive 30-90 days later)
exit_criteria:
  - all P1 data checks pass
  - PR-AUC >= 0.25 and recall >= 0.70 on frozen test set
  - no slice with recall below 0.55
  - model card and test report attached to the release ticket

The key shift is in exit criteria: they are numbers agreed in advance, not "looks good". If the data science team cannot agree on numbers, that is your first finding.

6. Non-determinism: see it for yourself

The same code with a different random split or seed gives different metrics. A tester who sees one number from one run has not seen the model's real quality - they have seen one sample of it.

from sklearn.datasets import load_breast_cancer
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score, recall_score

X, y = load_breast_cancer(return_X_y=True)   # bundled with scikit-learn, no download
# In this dataset 0 = malignant, 1 = benign. Missing a malignant case is the costly error.
for seed in [0, 1, 2, 3, 4]:
    X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25, stratify=y, random_state=seed)
    model = RandomForestClassifier(n_estimators=100, random_state=seed).fit(X_tr, y_tr)
    pred = model.predict(X_te)
    print(f"seed={seed} accuracy={accuracy_score(y_te, pred):.3f} "
          f"malignant_recall={recall_score(y_te, pred, pos_label=0):.3f}")

On my run, accuracy moved between about 0.94 and 0.98 just by changing the seed. That spread is why later labs use cross-validation, fixed seeds and tolerances instead of exact equality.

Hands-on Lab: Map the risks of a real ML feature

Setup: Python 3.11+ and pip install scikit-learn. You will use the breast cancer dataset bundled with scikit-learn (no download, no personal data).

Create a folder ml-qa-course and a virtual environment: python -m venv .venv, then activate it and run pip install scikit-learn pandas.
Save the seed experiment above as seed_experiment.py and run it. Record the minimum, maximum and spread of accuracy and malignant recall.
Extend the loop to 20 seeds and print the mean and standard deviation. Decide what tolerance a regression test on accuracy should use (for example mean minus two standard deviations).
Pick one of the four systems in the table above (fraud, credit, recommendation, image). Write its positive class, the two error types in plain words and which one costs more.
Copy the YAML strategy and adapt it to your chosen system: data checks, model checks, slices, fairness groups, robustness cases, monitoring signals and exit criteria with numbers.
List every artefact produced in the lifecycle for your system (label guide, dataset, features, model file, report, API, dashboard) and at least one test per artefact.
Review your strategy against the ML test pyramid: is anything only tested manually that could be automated?

Common mistakes

Treating the model as a black-box API and only testing status codes and JSON shape - the model can be badly wrong while every API test is green.
Accepting a single headline metric such as accuracy from one notebook run without asking about class balance, seeds and the test set.
Joining the project after training is done; most ML defects are baked in during problem framing and data preparation.
Writing exit criteria as "model performs well" instead of numbers agreed before training.
Forgetting production: no plan for drift, delayed labels or a fallback when the model fails.
Pasting real customer data into public AI tools to "quickly analyse" it - always work with synthetic or properly anonymised data and follow the client's data policy.

Real-world assignment

Choose a public ML-style product you use - a food delivery app's restaurant ranking, a bank app's spending categoriser, or a photo app's face grouping. Write a two-page ML test strategy for it: decision supported, positive class, error costs, data risks, the slices you would test (for example Hindi vs English users, metro vs tier-2 cities, low-end Android phones), fairness concerns, robustness cases, monitoring signals and numeric exit criteria. Add it to your GitHub portfolio as ML_TEST_STRATEGY.md; it is a strong conversation starter in interviews for AI QA roles.

Key takeaways

In ML, the logic is learned from data, so data and evaluation become first-class test targets.
There is often no exact oracle; test aggregate metrics against gates and test input-output relations.
Every lifecycle stage produces an artefact you can test - list them in your strategy.
Models fail silently when the world changes, so monitoring is part of QA scope.
Exit criteria must be numeric and agreed before training, including slice and fairness criteria.

How ML Systems Differ — Lifecycle, Risks and a QA Test Strategy