Jason Liu – Systematically Improving RAG Applications

Stop building RAG systems that impress in demos but disappoint in production

Transform your retrieval from “good enough” to “mission-critical” in weeks, not months. Most RAG systems stall in prototype purgatory: they demo well, but fail on complex queries—eroding trust and wasting engineering time. The difference isn’t just better tech, but a systematic mindset.

With the RAG Flywheel, you’ll:
✅ Pinpoint failures with synthetic evals
✅ Fine-tune embeddings for 20–40% gains
✅ Collect 5x more user feedback
✅ Segment queries to target high-impact fixes
✅ Build multimodal indices for docs, tables, images
✅ Route queries to the best retriever automatically

Week by week, you move from vague “make it better” to clear metrics, focused improvements, and compounding value. Real-world results include +20% accuracy from re-ranking, +14% with cross-encoders, and $50M revenue boosts from better search.

Join 400+ engineers applying this framework in production. Instructor Jason Liu has built multimodal retrieval and recommendation systems at Facebook, Stitch Fix, and through consulting—experience that shaped this practical, battle-tested approach.

What you’ll learn

Follow a repeatable process to continually evaluate and improve your RAG application

Analyze and Diagnose RAG System Performance

• Evaluate retrieval quality using precision, recall, and MRR metrics to identify system weaknesses
• Differentiate between leading metrics (experiments run) and lagging metrics (customer satisfaction) to drive actionable improvements
• Design synthetic data generation pipelines that enable rapid experimentation without waiting for user data

Construct Data-Driven Improvement Frameworks

• Create comprehensive evaluation datasets using LLMs to generate realistic query-answer pairs
• Establish baselines using tools like LanceDB to benchmark different retrieval implementations

Design Specialized Search Systems

• Develop multimodal retrieval systems that handle documents, images, tables, and structured data
• Synthesize lexical (BM25), semantic (embeddings), and metadata-based search for optimal results

Optimize Query Understanding & Routing

• Extract structured information from diverse data sources to enable precise filtering
• Classify queries using domain expertise and few-shot classifiers to improve routing accuracy