CogniVault Backend Explained, Part 3 · How a Question Becomes a Cited Answer

All abbreviations are fully explained in the appendix at the bottom of the page.

You type a question. A few seconds later you get an answer with footnotes — the exact documents and pages it came from. This part walks through everything that happens in between.

In Part 2 we built the knowledge base: every document chunked, embedded, and indexed. Now we get to use it — and this is where CogniVault stops being a pipeline and starts being interesting.

Two librarians, because one keeps failing you

Imagine a library with one librarian who organises everything by vibe. Ask her about “server downtime procedures” and she’s brilliant — she understands what you mean and finds documents that discuss the concept, whatever words they use. But ask her for “Error Code 404B” and she shrugs, handing you general networking guides. She doesn’t do exact strings.

Down the hall is a second librarian with a card catalogue. He finds the exact string “404B” instantly — but ask him a conceptual question phrased differently from the source text, and he finds nothing at all.

These are the two halves of search:

• Semantic search (FAISS) — your question is embedded into a vector, and the index finds chunks whose vectors point the same way (technically: cosine similarity — how closely two arrows align). Great for meaning, blind to exact identifiers.
• Keyword search (BM25) — a scoring formula that rewards chunks containing your exact words, weighted by how distinctive those words are. Great for identifiers, blind to synonyms.

CogniVault asks both librarians every time, then merges their answers with Reciprocal Rank Fusion (RRF) — a formula that combines ranked lists using only the positions:

score(chunk) = sum over both lists of  1 / (60 + rank)

A chunk ranked highly by either librarian scores well; a chunk both of them liked floats to the top. The elegance is what’s missing: you never have to reconcile FAISS’s similarity scores with BM25’s completely different scale, because ranks are the only input. The constant 60 comes straight from the original 2009 research paper, and yes, it’s cited in the code.

A few implementation details worth knowing: both searches deliberately over-fetch (at least 20 candidates each) so the fusion has material to work with; very weak semantic matches are dropped, but a keyword-perfect chunk can still be rescued through fusion; and the final answer uses the top 7 chunks. I benchmarked this whole setup against pure vector search in Hybrid Retrieval in Practice if you want the war stories.

The agent: a model that decides for itself

Here’s the second idea that trips up beginners: CogniVault’s chat is not “paste chunks into a prompt, get an answer.” It’s an agent — a model running in a loop where it can choose to call tools, read their results, and only then answer.

Built with the Strands Agents SDK, the agent gets six tools:

There is no hard-coded routing. The model reads your question and decides which tools to call, guided by its system prompt. Ask “compare the two contracts on termination clauses” and it reaches for compare_documents; ask “what’s 15% of 2,340” and it uses the calculator instead of hallucinating arithmetic.

Two safety details I want beginners to notice, because they’re the difference between a toy and a product: a fresh agent is constructed for every request (no shared state bleeding between concurrent chats), and the document-analysis tools call the model directly rather than through the agent — otherwise an agent calling a tool that calls the agent could recurse forever.

Watching the model think

When you send a message, the response streams back as NDJSON (Newline-Delimited JSON — each line of the stream is its own small JSON object). And it arrives in two phases:

Phase 1 — thinking. Gemma’s reasoning chain streams first, rendered in the collapsible panel above the answer. It’s deliberately best-effort: if it fails for any reason, the answer still comes.

Phase 2 — the agent answer. Tools run, citations appear in the Sources panel the moment the search completes — before the answer finishes writing — and the answer text streams in.

Each line in the stream is typed: thinking, metadata (a citation), text (answer), memory (how full the conversation budget is), or error. The frontend just reads lines and routes them to the right panel. I dissected this design — and why thinking comes before the tool calls — in Two-Phase Streaming: Showing the Model Think Before It Acts.

A memory budget, not a bottomless pit

Gemma’s context window (the amount of text the model can consider at once) is 128K tokens, but CogniVault doesn’t let conversation history sprawl across all of it. Each chat session gets a budget of 48,000 characters — roughly 12,000 tokens. Exceed it, and the oldest question-answer pair quietly drops out first, keeping the bulk of the window free for what matters: your current question and the retrieved chunks.

Two resilience touches worth stealing for your own projects:

• Restart survival. In-memory history dies with the process. So the first message in a session after a backend restart rebuilds its history from the chat log the frontend persists. Multi-turn memory survives reboots.
• Edit and regenerate. Editing an earlier message rewinds the stored history to that point before re-asking — the model genuinely forgets the timeline that no longer exists.

Scope: pinning the AI to specific documents

One last feature, and a lesson about small local models. You can pin a chat to specific files or a category. The filter travels with the request and a mandatory-search instruction is injected into both the system prompt and the user message itself.

Why both? Because small models sometimes skip instructions that live only in the system prompt — but they can’t ignore what’s inside the question. Belt and braces. When you work with 4-billion-parameter models instead of frontier ones, you learn to make instructions impossible to miss rather than hoping they’re followed.

The takeaway

A cited answer is four systems cooperating: two retrievers covering each other’s blind spots, a fusion formula that needs nothing but ranks, an agent that picks its own tools, and a stream that shows its work. None of the four is exotic on its own — the product is the cooperation.

Appendix: Abbreviations in this post

Next up: Part 4 · Study Tools, Progress, and the Privacy Receipts — the same machinery pointed at generating quizzes, workshops, flashcards, and mindmaps, plus a table of every byte the app stores and exactly where it lives.