Adding Conversation Memory to a RAG Chatbot Without Wrecking Retrieval

Until yesterday, POST /chat on this project took a single question: str. Every request was answered in total isolation - the LLM had no idea what was said two messages ago, because nothing about the previous turn ever reached it. Ask "what's the pricing?" and then "is there a free tier?" and the second question would get retrieved and answered as if it were the first message anyone had ever sent.

The fix sounds trivial - "just pass the whole conversation" - until you notice that a RAG chatbot has two things reading the conversation, not one: the retriever and the LLM. Feed both the same raw transcript and you'll fix the LLM's amnesia while quietly breaking retrieval.

The naive version breaks retrieval

Vector search works by embedding the query and finding chunks near it in vector space. If you embed the entire conversation transcript as the retrieval query, every prior turn - including ones about a completely different subtopic - gets folded into that embedding. Ask about pricing, then about the free tier, then about SSO, and by the third message your retrieval query is a blurry average of three topics. The chunks that come back are less relevant to what the user is actually asking right now, not more.

Retrieval needs a sharp, single-topic query. Conversation history is the opposite of that by construction.

Two consumers, two inputs

The actual fix: retrieval and prompting are separate concerns, and they should read from the conversation differently.

ChatRequest moved from a single field to an array:

class Message(BaseModel):
    role: Literal["user", "assistant"]
    content: str

class ChatRequest(BaseModel):
    messages: list[Message]

The router 422s early if the array is malformed - empty, or not ending on a user turn:

if not body.messages or body.messages[-1].role != "user":
    raise HTTPException(
        status_code=422,
        detail="messages must be non-empty and end with a user message",
    )

That guarantee matters downstream: everything in rag.py can safely assume messages[-1] is the current question.

rag.answer (and its streaming twin answer_stream) then splits that one array into two different inputs:

def answer(
    messages: list[dict],
    collection_name: str,
    top_k: int = 5,
    llm_config: dict | None = None,
) -> dict:
    cfg = llm_config or {}
    instructions = cfg.get("system_prompt", _DEFAULT_INSTRUCTIONS)
    temperature = cfg.get("temperature", _DEFAULT_TEMPERATURE)
    max_tokens = cfg.get("max_tokens", _DEFAULT_MAX_TOKENS)
    question = messages[-1]["content"]
    history = _format_history(messages)

    start = time.time()
    query_vector = embed([question])[0]
    retrieved = vector_search(collection_name, query_vector, top_k=top_k)
    context = build_context(retrieved)
    prompt = _build_prompt(instructions, context, question, history)
    response = _text(_bound_llm(temperature, max_tokens).invoke(prompt))
    ...

embed([question]) only ever sees messages[-1]["content"] - the last user turn, nothing else. Retrieval quality doesn't degrade as the conversation gets longer, because the query it searches with never grows. It's always exactly as sharp as a single, isolated question would be.

Everything else in the array - every turn except the last - goes into _format_history, which renders it as plain text:

def _format_history(messages: list[dict]) -> str:
    """Render all but the last message as a transcript for prompt context."""
    role_labels = {"user": "User", "assistant": "Assistant"}
    lines = [
        f"{role_labels.get(m['role'], m['role'])}: {m['content']}"
        for m in messages[:-1]
    ]
    return "\n".join(lines)

That transcript never touches the embedding model or the vector store. It has exactly one destination: the prompt.

Where history sits in the prompt

_build_prompt gained an optional history argument, and where it gets inserted is the other half of the design:

def _build_prompt(
    instructions: str, context: str, question: str, history: str = ""
) -> str:
    history_block = f"Conversation so far:\n{history}\n\n" if history else ""
    return (
        f"{instructions}\n\n{history_block}Context:\n{context}\n\n"
        f"Question: {question}\nAnswer:"
    )

The order is: system instructions, then conversation history, then retrieved context, then the current question. History sits before the retrieved chunks, not mixed into them. The model reads it as "here's what's already been discussed" before it reads "here's the grounding material for this specific answer" - the two stay visually and structurally distinct in the prompt, matching how they were sourced. History gives the model enough to resolve "what about the free tier?" into "the free tier of the pricing plan we were just discussing." The retrieved context still answers strictly to the current question, because that's the only thing that produced it.

When there's no history yet - the first turn in a conversation - history_block is just an empty string, so a single-message request produces exactly the same prompt shape this project had before the change. No special-casing needed at the call site.

The frontend just needs to build the array

None of this leaked into the UI layer. chat-input.tsx and widget-view.tsx are untouched - they still call sendMessage(question: string) exactly as before. The hook is where the array gets assembled, from state it already had:

async function sendMessage(question: string) {
  const history = messages.map((m) => ({
    role: m.role === "bot" ? "assistant" : "user",
    content: m.content,
  }))

  // ...

  body: JSON.stringify({
    messages: [...history, { role: "user", content: question }],
  }),
}

useStreamingChat's existing ChatMessage[] state already tracked every turn with a role: "user" | "bot" field for rendering - the only translation needed was "bot" → "assistant", since that's the vocabulary the backend's Message schema expects. The new user turn gets appended at the end, satisfying the router's "must end with a user message" check. No new state, no new fetch logic, no changes to the components that actually render the chat.

Why this generalizes

The specific mechanism here - embed the last message, render everything else as text, splice the text into the prompt ahead of retrieved context - is almost incidental. The pattern underneath is what's worth taking to other RAG systems: retrieval query and prompt context are different consumers of the same conversation, with different tolerances for noise. A vector search wants one clean signal; an LLM wants as much disambiguating context as the window allows. Forcing both to read the same blob - the full transcript, always - optimizes neither. Splitting them lets each one get exactly what it needs: retrieval keeps embedding the actual current question, and the model still gets the memory it needs to make sense of "what about the free tier?" without that memory ever showing up in a vector search.

Wrap-up

Multi-turn memory in a RAG chatbot isn't "concatenate the conversation and embed it" - that's the version that quietly degrades retrieval as conversations get longer. It's "figure out which parts of the conversation each downstream consumer actually needs." Retrieval gets the last message, unblurred. The prompt gets everything, but as text, positioned so the model can use it for context without confusing it for the current question. Same messages array, two different reads.