ML Model & Data Testing — Data Quality, Bias, Drift & AI Regulations
Learn to test machine-learning systems like fraud detection and credit scoring end to end - data quality, model evaluation, fairness, robustness, drift and CI quality gates - and map your evidence to the AI rules clients in India, the EU, the US and the UK care about.
✅ This course is included in the Unlimited & Global plan →About this course
<p>This course is for manual and automation testers, SDETs and QA leads who are moving into AI projects - in Indian IT services, fintech and product companies, or remote roles for EU, UK and US clients. You will learn how machine-learning systems fail differently from normal software and how to test them: data quality, model evaluation, behaviour, fairness, robustness, drift and MLOps quality gates.</p><p>Every lab uses free tools and public or synthetic data - scikit-learn, Pandera, Fairlearn, Evidently, MLflow and pytest - with code tested on current library versions. You will build a UPI fraud-model test suite that runs in GitHub Actions, and finish with a complete test and audit report for a credit-scoring model.</p><p>You will also learn the AI regulations clients ask about - the EU AI Act (including the 2026 timeline changes), NIST AI RMF, ISO/IEC 42001, India's DPDP Act and AI Governance Guidelines, and RBI's FREE-AI framework - and how to turn your tests into audit-ready evidence. These skills are in demand for AI QA, model validation and responsible-AI testing roles in India and abroad.</p>
What you will be able to do
- Design a numeric, risk-based test strategy for ML systems such as UPI fraud detection and credit scoring
- Write data contracts with Pandera that catch schema, null, range, duplicate and leakage defects
- Evaluate classifiers honestly with baselines, cross-validation, PR-AUC and cost-based decision thresholds
- Build CheckList-style behavioural and metamorphic test suites in pytest
- Measure and document group fairness with Fairlearn, including proxies and mitigation trade-offs
- Test robustness to noise, invalid inputs and evasion, and detect data and prediction drift with Evidently and PSI
- Run ML quality gates in CI with GitHub Actions and track evidence in MLflow
- Map test evidence to the EU AI Act, NIST AI RMF, ISO/IEC 42001, India's DPDP Act and AI governance guidelines in an audit-ready report
Syllabus
-
▶1. How ML Systems Differ — Lifecycle, Risks and a QA Test Strategy Understand why ML systems fail differently from normal software, map the ML lifecycle and write a numeric, reusable QA test strategy.
-
🔒2. Data Quality Testing — Schemas, Nulls, Ranges, Duplicates and Leakage with Pandera Profile ML training data, write a Pandera data contract that catches real pipeline defects, and detect label leakage before it fools everyone.
-
🔒3. Model Evaluation — Metrics, Cross-Validation and Thresholds for UPI Fraud Detection Evaluate a fraud model honestly with baselines, cross-validation, confusion matrices, PR-AUC and a cost-based decision threshold.
-
🔒4. Behavioural Testing of Models — Invariance, Directional, MFT and Metamorphic Tests Write CheckList-style minimum functionality, invariance and directional tests plus metamorphic relations in pytest, and catch a real non-monotonic defect.
-
🔒5. Bias and Fairness Testing — Demographic Parity and Equalized Odds with Fairlearn Measure group fairness on the public Adult census data with Fairlearn, find proxy variables, try a mitigation and document the trade-offs.
-
🔒6. Robustness and Adversarial Testing — Noise, Perturbations, Edge Cases and Evasion Measure how image and fraud models behave under noise, shifts, invalid inputs and evasion attempts, and protect the service with validated inputs.
-
🔒7. Data and Model Drift — Detection and Monitoring with Evidently and PSI Detect data and prediction drift with Evidently 0.7 and PSI, turn the results into automated monitoring gates and write a drift runbook.
-
🔒8. ML Pipelines in CI — pytest for ML, Quality Gates, Reproducibility and MLflow Build a CI pipeline that trains a candidate model, runs data, quality, behaviour and robustness gates, compares with production and tracks runs in MLflow.
-
🔒9. AI Regulations and Governance for Testers — EU AI Act, NIST AI RMF, ISO/IEC 42001, DPDP and India's AI Guidelines Understand the AI rules clients in India, the EU, the US and the UK care about, and turn your tests into audit-ready evidence with traceability matrices and model cards.
-
🔒10. Capstone — Complete Test and Audit Report for a Credit-Scoring Model Audit a credit-scoring model on the public German Credit data end to end - data, performance, fairness, behaviour, robustness, monitoring and regulatory mapping - and deliver a release verdict.
