Testing LLM, Chatbot & RAG Apps — Evaluation, Safety & Guardrails
Learn to test AI chatbots and RAG assistants end to end - golden datasets, LLM-as-a-judge metrics, Promptfoo, DeepEval, Ragas, red teaming, guardrails and production monitoring - using free local models.
✅ This course is included in the Unlimited & Global plan →About this course
<p>This course is for manual and automation testers, SDETs and QA leads who are being asked to test AI features - customer-support chatbots, RAG knowledge assistants and LLM-powered product features. You will learn why LLM apps fail (non-determinism, hallucination, retrieval misses, prompt injection, bias, cost blow-ups) and how to turn those risks into measurable, automated tests.</p><p>Everything runs on free, open-source tools and local models through Ollama, so you need no paid API keys. Across ten lessons you build a demo UPI support bot and a RAG assistant, then test them with golden datasets, LLM-as-a-judge metrics, Promptfoo, DeepEval, Ragas, promptfoo redteam, garak, Locust and Langfuse - finishing with a full CI-driven test suite and a GO / NO-GO report.</p><p>LLM evaluation and AI safety testing are among the fastest-growing QA skills, with demand from Indian product companies, GCCs and IT services firms as well as US, UK and EU employers hiring remotely. The course covers Hindi and Hinglish users, India's DPDP Act 2023 and the EU AI Act, so your portfolio speaks to both Indian and global clients.</p>
What you will be able to do
- Design a risk-based test strategy and release gate for LLM, chatbot and RAG features
- Build versioned golden datasets covering intents, multi-turn flows and Hindi/Hinglish users
- Choose and validate metrics - deterministic checks, semantic similarity, LLM-as-a-judge and human rubrics
- Automate prompt and model regression tests with Promptfoo and DeepEval in CI
- Measure RAG retrieval quality and groundedness with hit rate, MRR and Ragas metrics
- Red team LLM apps against the OWASP Top 10 for LLM Applications 2025 with Promptfoo and garak
- Test bias, toxicity and guardrails and map evidence to DPDP Act and EU AI Act requirements
- Load test LLM endpoints, track tokens and cost, and monitor quality with Langfuse
Syllabus
-
▶1. How LLM Apps Work and Fail — Test Strategy and Risk Matrix Understand the LLM app pipeline, non-determinism, hallucination and context limits, and turn them into a risk-based test strategy.
-
🔒2. Golden Datasets for Chatbots — Intents, Multi-turn and Hinglish Cases Design a versioned golden dataset covering intents, multi-turn flows, Hindi/Hinglish users and risk levels, and replay it against a local bot.
-
🔒3. Evaluation Metrics — Similarity, LLM-as-a-Judge and Human Rubrics Choose and validate metrics for free-text answers: deterministic checks, embedding similarity, rubric-based LLM judges and human review agreement.
-
🔒4. Promptfoo — Assertions, Test Matrices and CI Regression Build a Promptfoo suite with deterministic, similarity and LLM-graded assertions across prompts and local models, then gate pull requests in CI.
-
🔒5. DeepEval with pytest — LLM Unit Tests and Custom G-Eval Metrics Write pytest-style LLM tests with DeepEval: parametrised goldens, G-Eval rubrics, relevancy, safety and multi-turn checks using a local Ollama judge.
-
🔒6. RAG Testing — Retrieval Quality, Context Precision/Recall and Groundedness with Ragas Separate retrieval from generation failures: compute hit rate and MRR, then Ragas faithfulness, context precision and recall on a local RAG bot.
-
🔒7. LLM Security — OWASP Top 10, Prompt Injection and Red Teaming Test a local bot against the OWASP Top 10 for LLM Applications with a canary-based attack pack, promptfoo redteam, garak and PII checks.
-
🔒8. Bias, Toxicity and Guardrails — DPDP Act and EU AI Act Basics Test fairness with counterfactuals, evaluate toxicity and guardrails as classifiers, and map test evidence to DPDP Act 2023 and EU AI Act requirements.
-
🔒9. Performance, Cost and Monitoring — Load Testing LLM Endpoints and Tracing with Langfuse Measure TTFT, latency percentiles, tokens and cost, load test an LLM endpoint with Locust, handle rate limits and trace production-style calls in Langfuse.
-
🔒10. Capstone — Full Test Suite for a Local RAG Chatbot with Ollama Build, automate and report a complete evaluation, security, bias and performance test suite for a FastAPI + Ollama RAG chatbot with a GO / NO-GO decision.
