Semester 4 · Language models
LLM Applications: Retrieval, RAG and Agents
https://epochschool.com/subjects/llm-applications-rag-agents
Language models only become useful products when they can find the right facts and act through tools safely.
You build search from first principles (inverted indexes, BM25, dense dual encoders, vector indexes such as IVF, product quantisation and HNSW, hybrid fusion and reranking), then put it to work in retrieval-augmented generation that cites its sources and knows when to say "I don't know", and in tool-using agents that plan, remember and are evaluated with the statistics of reliability. You finish by securing these systems against prompt injection and data exfiltration and by deploying them with caching, cost and latency controls; every lab runs on one CPU core with deterministic stand-ins for the language model, so what you build and measure is the engineering around the model.
- Prerequisites
- Required: Finish this subject before you start. The subject builds directly on it.Recommended: Helpful background. You can start without it if you are willing to fill the gaps.
What you’ll be able to do
- LO1Build lexical retrieval systems from first principles (inverted and positional indexes, Boolean and phrase queries, TF-IDF, BM25 and query likelihood) and explain how top-k scoring is made fast at scale.
- LO2Design retrieval evaluations (test collections, graded relevance judgements, held-out query sets) and compute, interpret and statistically compare ranking metrics (precision and recall at k, MRR, MAP, nDCG).
- LO3Turn contrastively trained text encoders into dense retrievers (shared or asymmetric towers, mined hard negatives and false negatives, evaluation against the full corpus) and explain when dense, sparse, late-interaction or hybrid representations win.
- LO4Implement and tune approximate nearest-neighbour indexes (LSH, IVF, product quantisation, HNSW) and reason quantitatively about recall, latency, memory, filtering and updates in vector databases.
- LO5Build multi-stage ranking pipelines with score and rank fusion, reranking (cross-encoders and learning to rank), query rewriting and diversification, and justify their cost.
- LO6Design, build and evaluate retrieval-augmented generation pipelines: chunking, context assembly, grounded prompting with citations and abstention, and metrics for retrieval quality, faithfulness, answer correctness and abstention.
- LO7Specify reliable interfaces to language models: prompts as versioned and tested code, structured output with schema validation and constrained decoding, and function calling.
- LO8Design, implement and evaluate tool-using agents and multi-agent systems (the agent loop, ReAct, planning, memory, orchestration) using state-based task checks and reliability metrics such as pass^k with confidence intervals.
- LO9Threat-model and secure LLM applications against prompt injection, data exfiltration and excessive agency, and deploy them with sandboxing, caching, cost and latency controls and observability.
Chapters
The chapter and lab list for this subject is being finalised and will be published here.
Before you start
Required
Finish this subject before you start. The subject builds directly on it.
Sequence Models & Transformers
Neural Networks & Deep Learning
Recommended
Helpful background. You can start without it if you are willing to fill the gaps.
Building and Training LLMs from Scratch
Where it leads
Subject 4 of 4 in the Language models track.
How it’s taught
You read short sections written for the page. Each ends with a check, and in-browser exercises let you practise. Graded labs run on your own cloud lab machine, CPU-friendly.
No videos. No live classes. No AI tutor.