The model was confidently wrong about the syllabus

The first generated learning path I looked at properly was for CSIR NET chemical sciences, and it was good. Sensible progression, reasonable pacing, decent quizzes. It also contained a unit that hasn't been on that syllabus for years.

I only caught it because I've been running an exam-prep company since 2018 and that particular topic is one I have opinions about. A student wouldn't have caught it. That's the problem with these systems in one line: the failures don't look like failures, they look like content.

Why the obvious pipeline does so little

The naive version is about forty lines. Chunk the corpus, embed it, embed the user's topic, pull the top five chunks, paste them into the prompt. It demos beautifully. It took me longer than I'd like to admit to articulate why it wasn't working.

Question answering is extractive. The answer sits in the corpus somewhere and retrieval's job is to find the paragraph. Curriculum generation is not that. "Build me a 12-week path for CSIR NET chemical sciences" has no answer sitting in any document. It's a structural task, and the model is genuinely good at structure: outlines, progression, prerequisite ordering. What it's bad at is knowing whether a topic is still examinable this year.

Retrieving against the whole request, then, is retrieving against nothing. No single passage answers it, and the top-k you get back is a grab bag that makes the prompt longer without making it truer.

Plan first, ground second

We split the job in two, and that change mattered more than everything else combined.

The first pass produces structure only. No retrieval, and a hard constraint that it emit topic titles and prerequisites, no content. The second pass walks that tree and writes each node separately, and that's where retrieval happens: one query per node, scoped to the node's topic, with the outline path as context.

LangGraph earned its place here, though not for the reason I expected. The appeal wasn't orchestration. It was that each node gets its own retrieval, its own generation and its own validity check, so a node that fails can be retried or dropped without collapsing the run. When you're generating forty lessons, the gap between "one lesson was rejected" and "the request failed" is the gap between a product and a demo.

async def write_lesson(state: LessonState) -> LessonState:
    query = build_query(state.topic, state.outline_path, state.exam)
    hits = await retrieve(query, k=30)
    context = rerank(query, hits)[:6]

    if max(h.score for h in context) < GROUNDING_FLOOR:
        return state.copy(status="uncovered", lesson=None)

    lesson = await generate(state.topic, context)
    return state.copy(lesson=lesson, sources=[c.id for c in context])

status="uncovered" is the line I'd argue for hardest. A node that can't find support for its topic returns nothing and says so. It doesn't write the lesson anyway.

Chunking

We used fixed token windows at first, because that's what every tutorial does. Moving to heading-aware chunks improved retrieval more than any model change we made: split on document structure, keep sections whole where they fit, and prepend the heading path to the chunk text before embedding.

The heading path is the actual trick. A chunk beginning "Unit 4 › Electrochemistry › Nernst equation" embeds somewhere meaningfully different from the same paragraph naked, and it matches a query built from an outline node almost by construction. It costs nothing. It is string concatenation.

Sibling context helped too. Each chunk stores the ids of the ones before and after it, so a retrieved chunk can pull its neighbours in. Textbooks and syllabus documents are full of sentences that only mean anything alongside the paragraph above them.

The eval set

For about three weeks we improved the pipeline by reading outputs and going "hmm, better". That isn't improvement, it's mood.

Building an eval set fixed it and was much less work than I'd feared. Around 120 queries taken from real topic titles, each labelled with the corpus sections that ought to come back for it. Two of our subject teachers did the labelling in an afternoon and a half. Then one number to watch: recall@10.

We started near 0.62 and got to about 0.88. Heading-aware chunking was the biggest single jump. Hybrid search was next, and it wasn't close: dense embeddings are weak on exactly the terms that matter here, a scheme name, an author, a specific reaction, while BM25 nails those and is hopeless at paraphrase, so running both and fusing the rankings beat either alone. Then a cross-encoder rerank from the top 30 down to 6, slower per query and worth it, because the generation prompt now receives six chunks that are about the topic rather than six that are vaguely nearby.

None of that involved touching the vector database. The vector database is the least interesting component in a RAG system and it takes up a wildly disproportionate share of the conversation about them.

Retrieval only fixes what's retrievable

This is the honest limit, and it's where the syllabus bug came from.

Good retrieval hands the model the right context when the right context exists. Where the corpus doesn't cover something, no amount of retrieval work helps. The model writes a fluent, plausible, unsupported lesson that reads exactly like the correct ones.

So there's a verification pass. After a lesson is generated we extract its factual claims and check each against the chunks retrieved for it, using a small cheap model for entailment. Claims that aren't supported get the lesson marked for review instead of published.

It isn't clever and it doesn't catch everything. What it catches is the specific failure I care about most, content that drifted from the source material while sounding entirely reasonable. A learner can't detect that. Neither could I, on topics I don't personally know.

The other half is abstention. When a topic genuinely isn't in the corpus, the right output is to say so, and getting a language model to do that reliably is mostly about making it structurally possible: the uncovered branch in the graph, a floor on retrieval score, and a prompt that offers refusal as a real option rather than an apology. Models will refuse when refusal is a shape the system can accept. They won't when the only path out is a lesson.

Making a minute feel like seconds

A full path runs to roughly forty lessons. Sequentially that's minutes, and nobody waits minutes.

Node generation runs in parallel behind a bounded semaphore, and retrieval is cached per topic, which matters more than it sounds because paths for related exams overlap heavily. The outline streams to the UI as soon as it exists, so a learner watches their path assemble while lessons fill in behind it. The wait they actually experience is the outline, a few seconds, not the run.

At around 50 concurrent learners the bottleneck was never our infrastructure. It was provider rate limits, which meant a queue with per-user fairness so that one person generating three paths doesn't stall everybody else.

If I were starting again

I'd build the eval set before the pipeline. Before choosing a database, before any prompt engineering. A hundred labelled queries is an afternoon of somebody's time and it converts every later decision from an argument into a measurement, which is worth more than it sounds when you have three people with three opinions about chunk size.

The other thing is less about engineering. Get a domain expert to read the first hundred outputs properly, not as a spot check. The bug that started all of this was a single line in a lesson that was otherwise completely fine, and no metric we had was ever going to find it.