AI |

Gemma CogniVault

Mon, 25 May 2026 00:00:00 +0000

Overview

Gemma CogniVault is a 100% local, privacy-first AI study companion. Your documents stay on your hardware. Inference runs via Ollama on localhost. No telemetry, no embeddings sent to third parties, no exceptions. A live Privacy Vault Audit Panel confirms zero external connections at runtime.

It’s also genuinely capable — Gemma 4’s full surface (completion, vision, tools, reasoning) running on your laptop, wrapped in an app that turns your documents into quizzes, multi-lesson workshops, flashcard decks, and visual mindmaps, with a learning-progress dashboard and 25 achievement badges.

What’s inside

Layer	Technology
LLM & Embeddings	Ollama · `gemma4:e4b` · `embeddinggemma`
Agent Framework	Strands Agents SDK
Backend	FastAPI · Python 3.10+ · Pydantic
Vector Search	FAISS IndexFlatIP + BM25Okapi · Reciprocal Rank Fusion
Document Parsing	pypdf · python-docx · python-pptx · openpyxl · trafilatura
OCR	pytesseract · pymupdf · Pillow
Audio	faster-whisper
Workflow Engine	DBOS + PostgreSQL
Frontend	React 19 · TypeScript · Vite · Tailwind v4 · Framer Motion · TanStack Query

Four sections

Section	What it’s for
💬 Chat	Ask anything about your documents. Cited answers, scope filter, voice, attachments.
📚 Knowledge Base	Upload, categorise, and manage your documents. SHA-256 change detection on re-upload.
🎓 Study Hub	Four AI-powered study modes: Quiz · Workshop · Flashcards · Mindmaps.
📊 Dashboard	Total study time, current streak, 25 achievement badges, 90-day activity heatmap.

Highlights

🧠 Thinking Mode — collapsible reasoning panel streams Gemma 4’s chain of thought before the answer
🔍 Hybrid Retrieval — FAISS dense + BM25 keyword fused with Reciprocal Rank Fusion
🖼️ Multimodal — attach images, PDFs, and DOCX inline in chat
🛟 Durable workflows — DBOS-checkpointed ingestion; crash-safe and resumable
🏆 25 achievement badges — auto-tracked across chat, quizzes, workshops, flashcards, mindmaps
🔒 Vault Audit Panel — live “zero external connections” indicator

Writing about it

I’m publishing a series of posts unpacking the engineering decisions behind CogniVault — privacy framing, the retrieval stack, the agent loop, ingestion durability, getting JSON out of a local model, drawing mindmaps without a graph library, the gamification layer, and how the test suite avoids needing any infrastructure to run.

See the for the full series.

Try it

git clone https://github.com/ndimoforaretas/local-gemma-rag.git
cd local-gemma-rag
./scripts/setup.sh # one-time
./scripts/start.sh

Then open .

Part 8 · Testing a Local-AI App: 351 Tests, Zero Infrastructure

Mon, 25 May 2026 00:00:00 +0000

Part of a series on building . Previously: . All abbreviations are fully explained in the appendix at the bottom of the page.

CogniVault has 351 tests across 22 files (at the time of writing — the suite grows with the app). None of them need Ollama. None of them need Postgres. None of them need a real PDF, a microphone, or an internet connection. The whole suite runs in about three seconds on my laptop.

That’s not because there isn’t much to test — the surface is wide. It’s because the test suite is built around one principle: mock at the edge, real everywhere else. This post is about what “the edge” means in a local-AI app, and how to draw the line so the suite stays useful instead of decorative.

The 22 test files

File	What it covers
`test_api.py`	The HTTP endpoints (upload, ingest, RAG, history, KB browsing)
`test_tools.py`	Calculator, clock, KB search tool
`test_thinking.py`	Two-phase stream, thinking tokens, session isolation
`test_chat_attachments.py`	Multi-file attach, PDF/DOCX extraction, size limits
`test_chat_memory.py`	Session history budget, trimming, restart rebuild
`test_doc_scope_filter.py`	Per-request ContextVar isolation, search filtering
`test_doc_tools.py`	`list_documents`, `analyze_document`, `compare_documents`
`test_edit_regenerate.py`	History rewind, trim_history_to_turns validation
`test_structure_chunking.py`	Markdown header splits, CSV row batches, doc types
`test_ocr_fallback.py`	OCR trigger threshold, graceful degradation
`test_new_formats.py`	PPTX, XLSX, HTML extractors, extension routing
`test_docx_url.py`	DOCX ingestion and URL import (with the SSRF guard)
`test_reingest.py`	SHA-256 change detection, idempotency
`test_vector_db.py`	BM25, FAISS, RRF fusion, hybrid search
`test_audio.py`	Whisper transcription endpoint
`test_progress.py`	Sessions, daily aggregation, achievement criteria
`test_prompts.py`	The prompt-template loader and custom overrides
`test_vault_stats.py`	The Privacy Vault Audit numbers
`test_quiz.py` / `test_workshop.py` / `test_flashcards.py` / `test_mindmaps.py`	Per-mode parsing, endpoints, achievements

Everything that can be tested in isolation is tested in isolation. Everything that needs to be tested through the FastAPI layer is, but the only things mocked are the calls that cross the process boundary.

What gets mocked, what doesn’t

The single most important question in a project like this: where do you stub?

[ React frontend ] ←─ not in scope for backend tests
 │
 ▼
[ FastAPI handlers ] ←─ tested directly with TestClient
 │
 ▼
[ services/ ] ←─ tested directly (vector_db, rag_agent, generators)
 │
 ├─► [ FAISS + BM25 ] ←─ real, in-memory, fast
 ├─► [ SQLite ] ←─ real, against a tmp_path file
 ├─► [ DBOS ] ←─ patched (no launch, no Postgres)
 ├─► [ Ollama ] ←─ patched at each service's import site
 └─► [ Whisper ] ←─ stubbed (no 145 MB model load)

The rule of thumb: anything that crosses a process or network boundary, mock. Anything in-process, run for real.

FAISS and BM25 are real because they’re libraries we link into the test process. SQLite is real because it’s a file. DBOS is patched because launching it expects a Postgres connection, and that’s network. Ollama is patched because it’s HTTP. Whisper is stubbed because loading a 145 MB model in a unit test is silly.

That principle keeps the test suite fast (no I/O the OS can’t handle in milliseconds) and meaningful (the real code paths through retrieval, chunking, parsing, scope filtering all execute).

Mocking Ollama

Most CogniVault tests need some model output, but they don’t care what model produced it. Each service imports the ollama module directly, so the tests patch that reference at the service’s own import site:

# Real pattern from test_quiz.py
from unittest.mock import patch
from backend.services import quiz_generator

def test_quiz_parses_questions():
 fake = {"message": {"content": json.dumps({"questions": [VALID_MCQ] * 5})}}
 with patch.object(quiz_generator, "ollama") as mock_ollama:
 mock_ollama.chat.return_value = fake
 result = quiz_generator.generate_quiz(
 difficulty="beginner", num_questions=5, question_types=["mcq"],
 )
 assert len(result.questions) == 5

A streaming variant feeds chunk sequences instead of a single response, used by the RAG and thinking tests. The key property: one patch.object against the module the service actually uses. No deep mock hierarchies, no fragile string paths into third-party internals. Easy to read in a code review, easy to debug when a test fails.

Mocking DBOS

DBOS expects launch() to connect to Postgres. The shared client fixture in conftest.py simply patches the dbos instance before the app is exercised:

# Real pattern from conftest.py
@pytest.fixture()
def client():
 """A FastAPI TestClient with DBOS launch mocked out — no Postgres needed."""
 with patch("backend.services.ingest.dbos") as mock_dbos:
 mock_dbos.launch = MagicMock()
 from backend.main import app
 with TestClient(app) as c:
 yield c

The decorated workflow steps still execute as ordinary Python functions — we lose the durability semantics, but the tests aren’t testing durability, they’re testing the business logic inside the steps (hash detection, extraction, chunking). The durability layer has its own tests upstream, in DBOS’s own suite.

There’s a second isolation layer that runs on every test automatically: an autouse fixture points the docs folder, FAISS index, and metadata file at a per-test tmp_path via environment variables, so no test can ever touch real data on disk.

Real SQLite, with one override

Progress tracking, achievements, quiz storage, deck CRUD — all SQLite. The progress tracker exposes a single test seam: a module-level path override.

# Real pattern from test_quiz.py
@pytest.fixture(autouse=True)
def _isolate_progress_db(tmp_path, monkeypatch):
 monkeypatch.setattr(progress_tracker, "_db_path_override",
 str(tmp_path / "progress_test.db"))

Every test gets a fresh database file; the schema auto-creates on first use. No connection pooling drama, no leaked state between tests, no in-memory :memory: gymnastics. Just a temp file per test.

This is the kind of test that catches bugs an SQL-level mock would never see — a missing index, a botched migration, a constraint that doesn’t fire. SQLite is fast enough on every machine I’ve ever owned that “use the real database” isn’t even a trade-off.

The TestClient pattern

For HTTP tests, FastAPI’s TestClient runs the app in-process. The upload, the validation, the chunking, the vector-store update, the response serialisation — every layer runs for real. Only the calls that would leave the process (the Ollama embedding call inside ingestion, the model call inside generation) are patched. That’s the right line: the test verifies the integration of those layers, but doesn’t depend on an external service.

The streaming endpoint tests use a slightly different style — they iterate the response body and parse each NDJSON line (one JSON envelope per line, as described in ) — but the principle is identical.

Coverage gaps I accept

Three things the test suite doesn’t cover:

The frontend. No React testing in this suite — that’s a separate concern. Most failures show up in API tests anyway, because the frontend is a thin client over a typed API.
Real Ollama prompt quality. Whether gemma4:e4b actually produces useful quiz questions is not a thing tests can answer. That’s evaluation, not testing. It belongs in a separate harness with a real model running.
Race conditions across DBOS workflow restarts. The resume path is exercised at the logic level, but the full state space of “what happens if Postgres goes away at this exact instant” is too large to enumerate.

These are conscious gaps. The test suite is for catching regressions in code I wrote; it’s not a replacement for evaluation, integration testing, or actual chaos engineering.

What the suite is actually for

Two things, in order:

Refactor confidence. When I rip out the agent loop and put a new one in, do the tests still pass? If yes, the API contracts I care about haven’t drifted.
PR review surface. Every PR runs the suite in CI. A green run is a precondition for merge. The suite is loud enough that a real regression makes the noise.

Notice what it isn’t for: proving the model works. It can’t. Tests can pin behaviour but they can’t pin quality. That’s a different muscle, and it belongs in a different harness.

What’s worth borrowing

If you’re building a local-AI app and your tests need Ollama running:

Patch the ollama module at each service’s import site with patch.object(service_module, "ollama") — one seam per service, no shims required.
Give your DB layer a path override and run against a tmp_path SQLite file.
Use an autouse fixture to redirect every on-disk artefact (docs folder, index files) to tmp_path, so no test can touch real data even by accident.
For each external service (model, audio, workflow engine), draw the seam at the process boundary. Test everything above it with real code.

The result is a suite where every test runs in any environment, finishes in milliseconds, and exercises the actual integration of every layer of code you wrote. 351 tests in about three seconds isn’t an optimisation, it’s a side-effect of mocking only at the edges.

Appendix: Abbreviations in this post

Abbreviation	Full form	Meaning
CI	Continuous Integration	Automatically running the test suite on every push/PR
PR	Pull Request	A proposed code change — merged only when the suite is green
API	Application Programming Interface	The HTTP surface the TestClient exercises in-process
HTTP	HyperText Transfer Protocol	The protocol the (in-process) endpoint tests speak
RAG	Retrieval-Augmented Generation	The retrieval-then-answer pipeline under test
KB	Knowledge Base	The indexed document collection
FAISS	Facebook AI Similarity Search	Real in tests — it’s an in-process library
BM25	Best Match 25	The keyword index — also real in tests
RRF	Reciprocal Rank Fusion	The rank-merging formula covered by `test_vector_db.py`
SQLite / SQL	(SQL = Structured Query Language)	The real, file-based database every progress test runs against
DBOS	Database-Oriented Operating System	The durable-workflow library — patched so no Postgres is needed
OCR	Optical Character Recognition	The scanned-PDF fallback with its own trigger-threshold tests
SSRF	Server-Side Request Forgery	The URL-import attack class covered in `test_docx_url.py`
NDJSON	Newline-Delimited JSON	The streaming format the endpoint tests parse line by line
SHA-256	Secure Hash Algorithm, 256-bit	The content fingerprint behind the re-ingest tests
CRUD	Create, Read, Update, Delete	The basic storage operations for decks, quizzes, and maps
PDF / DOCX / PPTX / XLSX / HTML	Portable Document Format / Word / PowerPoint / Excel / HyperText Markup Language	The extractor formats with dedicated tests

That’s the series. Eight posts on the parts of I’m most proud of — and a handful I’d build differently. If any of it was useful to you, the code is open source at , and the is on YouTube.

Your data. Your hardware. Your AI. Your vault.

Part 5 · Getting Reliable JSON Out of a Local LLM

Sun, 10 May 2026 00:00:00 +0000

Part of a series on building . Previously: .

All abbreviations are fully explained in the appendix at the bottom of the page.

CogniVault’s Study Hub generates four kinds of structured artefacts from your documents: quizzes, multi-lesson workshops, flashcard decks, and mindmaps. All four need the model to return structured JSON, not prose. All four ride on Gemma 4 running locally via Ollama. And all four would fail far too often if I trusted the model to “just return JSON.”

Here’s the defensive pattern that brings that failure rate close to zero — and what to do about the cases that still get through.

The pattern

1. Retrieve → hybrid search restricted by user-selected scope
2. Prompt → strict schema-by-example with explicit count + shape rules
3. Generate → ollama.chat with format="json" (grammar-constrained)
4. Parse → json.loads, tolerant of object / array / fenced shapes,
 with a trailing-comma repair pass
5. Validate → drop malformed items rather than fail the whole batch
6. Retry → the workshop outline retries once with a stronger prompt
7. Persist → SQLite (progress.db) so the user can come back later

Every generator in CogniVault follows it. The interesting moves are 2, 4, and 5.

Step 3: `format="json"` does real work

Ollama exposes a format="json" option that puts the model under a grammar constraint during sampling. The decoder won’t emit tokens that would make the output invalid JSON. It’s not perfect — schemas are bigger than “valid JSON,” and the model can still produce well-formed garbage — but it eliminates the entire class of “the model started writing prose before the closing brace” failures.

If your local-LLM stack supports a grammar option (Ollama, llama.cpp, vLLM, etc.), turn it on. It’s not free (sampling is slightly slower) but the failure-mode improvement is enormous. Without it, you’ll spend most of your error budget on truncated objects.

Step 2: schema-in-prompt that the model can actually obey

format="json" guarantees the shape of the output is JSON. It says nothing about whether the JSON matches your domain schema. That’s the prompt’s job.

The pattern that works for me: instead of dumping a formal JSON Schema and saying “obey this,” include a filled-in example that shows the model the exact shape, plus explicit counts. Here’s the heart of CogniVault’s real quiz template (it lives as an editable Markdown file in backend/prompts/quiz.md):

Output ONLY a single JSON object — no prose, no markdown fences,
no text outside the JSON.

NUMBER OF QUESTIONS: EXACTLY $num_questions. This is a hard requirement.

OUTPUT SCHEMA:
{
 "questions": [
 {
 "type": one of [$types_csv],
 "question": the question text (string, no leading numbering),
 "options": array of strings (length 4 for mcq, length 2 for true_false),
 "correct_index": integer index into options (0-based),
 "explanation": 1-2 sentence explanation of the correct answer
 },
 ... exactly $num_questions entries
 ]
}

A few choices that matter:

Show the shape, don’t describe it. “Each item has a type field” gets ignored more often than the literal example.
Pin the count. “EXACTLY 10” — repeated, in capitals, as a hard requirement — is much more reliable than “around 10.”
Index, don’t repeat. The correct answer is correct_index, an integer pointing into options — not the answer text again. Repeated text invites paraphrase drift (“Paris” vs “Paris, France”), and then your grading comparison breaks.
One artefact per call. I tried generating a full workshop (outline + every lesson) in one call. The model’s quality degrades sharply as the response grows. Splitting into outline-first, lesson-on-demand is the two-pass strategy below.

Step 4: parse, tolerantly

Even with format="json", two parsing problems survive in practice.

The shape surprise. This one bit me in production: I’d assumed the model would return a bare JSON array of questions. With format="json", Gemma consistently returns an object — {"questions": [...]} — and for a while the parser only accepted the array. Result: a 502 on every quiz generation until I found it. The fix is a parser that meets the model where it is:

# Simplified from backend/services/quiz_generator.py
def extract_items(raw: str) -> list | None:
 for candidate in (raw, extract_json_object(raw), extract_json_array(raw)):
 if candidate is None:
 continue
 data = load_json_lenient(candidate)
 if isinstance(data, list):
 return data # bare array
 if isinstance(data, dict):
 items = data.get("questions") # the expected object shape
 if isinstance(items, list):
 return items
 return None

Lexical glitches. Occasionally a trailing comma slips through. The repair is deliberately narrow — one regex pass, then give up:

def load_json_lenient(text: str):
 try:
 return json.loads(text)
 except json.JSONDecodeError:
 repaired = re.sub(r",(\s*[\]}])", r"\1", text) # strip trailing commas
 try:
 return json.loads(repaired)
 except json.JSONDecodeError:
 return None

I don’t try to balance brackets, complete truncated strings, or guess at missing fields. Either the output is fixable with a trailing-comma pass and some substring extraction, or it isn’t, and we move to step 5.

Step 5: drop malformed items, don’t fail the batch

This is the call that took me a while to make peace with.

When the model returns 10 quiz questions but #7 is missing its options field, the temptation is to error out and regenerate the whole batch. Don’t. Validate each item independently and drop the ones that fail.

# CogniVault does this with explicit field checks into a dataclass;
# pydantic works just as well.
questions = []
for raw_item in parsed_items:
 q = validate_item(raw_item, allowed_types) # returns None if malformed
 if q is not None:
 questions.append(q)

The user gets 9 questions instead of 10. They don’t notice. Re-running the whole generation to fix question #7 takes 30 seconds and might introduce new failures in questions 1-6. The dropped-item approach is strictly better UX. (The model also sometimes overshoots the count — the validated list is simply trimmed back to what was asked for.)

Step 6: the outline retries once

Workshops are the exception that proves the rule. A workshop is a structured outline (title, summary, lesson list) plus each lesson’s content. The outline must parse — there’s no partial success for a table of contents — so a parse failure there triggers exactly one retry, with the prompt re-sent plus a stern reminder: “Your previous response was unparseable. Output ONLY a single valid JSON object.” If the second attempt fails too, the user gets a clear error suggesting a narrower scope.

One retry, not three. Three retries when the model is consistently confused is just wasted seconds and watts.

The lessons themselves, interestingly, are not JSON at all. A lesson body is prose — forcing it into a JSON string would buy nothing and cost escaping headaches. Lessons are generated as plain Markdown, then run through a small cleanup pass that strips chat-isms the model sometimes adds despite instructions (“I hope this helps!”, “Let me know if…”). Different output, different contract.

Two-pass: outline first, lessons on demand

Workshops use a two-pass generation pattern:

Pass 1 — generate outline: {"title": ..., "lessons": [{"title": ...}, ...]} (cheap, JSON)
Pass 2 — for each lesson: a full Markdown lesson body (on demand)

The outline is fast and lets the user see the shape of the workshop immediately. Each lesson is generated when the user opens it — meaning the user is reading lesson 1 while deciding whether they even want lesson 5. The total wall-clock time to “first useful content” is small even for a 10-lesson workshop.

This is the same architectural move the chat side makes with : split a slow operation into a tiny fast part and a larger slow part, hand the user the fast part immediately.

What I learned so far putting those generators together

A few principles distilled from the four generators:

Use the grammar option in your inference stack. Don’t try to coax JSON out of a free-form decoder.
Pin every quantifier in the prompt. “Exactly 10,” “exactly 4 options,” “one or two sentences.” Vague counts = inconsistent output.
Don’t assume the top-level shape. Grammar-constrained Gemma likes objects; your code might expect arrays. Accept both — the parser is cheaper than relying on the model to return the expected shape.
Drop, don’t fail. Lossy success beats brittle perfection.
One retry, never more. If two tries can’t produce valid output, the prompt is wrong, not the model.
Split large generations. Outline + lessons. Skeleton + body. Two small calls beat one big one almost every time. And if a part of the output is naturally prose, let it be prose.

Local LLMs in 2026 are good enough that structured generation is genuinely usable for production-shaped features. They are not so good that you can skip the defensive scaffolding. The scaffolding above is maybe 80 lines of code total across all four generators, and it’s the difference between “demo-quality” and “I trust this enough to ship.”

Appendix: Abbreviations in this post

Abbreviation	Full form	Meaning
JSON	JavaScript Object Notation	The structured text format the generators must produce
LLM	Large Language Model	A neural network trained on huge amounts of text that can read and generate language
AI	Artificial Intelligence	Software performing tasks that normally need human intelligence
MCQ	Multiple-Choice Question	One of the two quiz question types (the other is true/false)
UX	User Experience	Why 9 valid questions beat a regeneration error
SQLite	(SQL = Structured Query Language)	The single-file database where generated artefacts persist
DBOS	Database-Oriented Operating System	The durable-workflow library from the previous post
HTTP 502	Bad Gateway (HyperText Transfer Protocol status code)	The error my array-only parser produced until I accepted Gemma’s object shape

Next up: — what hand-rolling an SVG radial layout taught me, and why version two uses React Flow anyway.

Part 3 · Two-Phase Streaming: Showing the Model Think Before It Acts

Thu, 30 Apr 2026 00:00:00 +0000

Part of a series on building . Previously: . All abbreviations are fully explained in the appendix at the bottom of the page.

When I first wired up Gemma 4 with inside CogniVault, the chat felt slow. Not laggy — slow in a way that’s worse than laggy. The user types a question. The cursor sits there. Then, eventually, an answer drops out of the void.

The model wasn’t idle. It was thinking. Gemma 4 has a chain-of-thought mode that produces a (sometimes long) reasoning trace before its final reply. With a single-phase agent stream, all of that thinking is happening inside the agent loop — silently — before any tool calls run, before any tokens get emitted to the UI.

So I split the call into two phases.

The shape

POST /rag
 │
 ├── Phase 1 — Direct Ollama call, thinking enabled
 │ stream: {"type":"thinking","data":"..."} (reasoning tokens)
 │
 └── Phase 2 — Strands Agent (thinking disabled)
 stream: {"type":"metadata","data":{...}} (citations, as soon as search runs)
 stream: {"type":"text","data":"..."} (answer tokens)
 stream: {"type":"memory","data":{...}} (end-of-stream: session memory usage)

The endpoint streams newline-delimited JSON (NDJSON): each line of the response body is one self-contained JSON envelope with a type and a data. The frontend dispatches on type and renders accordingly: a collapsible reasoning panel for the thinking tokens, the main message bubble for the text tokens, a sidebar card per citation.

The user sees the model start thinking immediately. Latency to first byte drops from “long enough to wonder if it crashed” to “instant.” Total time to final answer doesn’t change. Perceived speed does.

Phase 1 — Thinking only

Phase 1 is a single direct call to Ollama with thinking enabled. It gets exactly what Phase 2 will see — the same system prompt, the current question, and any attached images — so the reasoning reflects reality. Only the reasoning tokens are consumed; whatever answer text Phase 1 starts to produce is discarded, because we don’t want a half-formed answer competing with the real one.

# Simplified from backend/services/rag_agent.py
client = ollama.AsyncClient(host=settings.ollama_host)
stream = await client.chat(
 model=settings.llm_model,
 messages=[
 {"role": "system", "content": system_prompt},
 {"role": "user", "content": query, "images": images},
 ],
 options={"thinking": True},
 stream=True,
)
async for chunk in stream:
 if chunk.message.thinking:
 yield envelope("thinking", chunk.message.thinking)

Phase 1 is deliberately best-effort: any failure here is swallowed and logged, and the stream moves straight on to Phase 2. A broken reasoning panel should never cost the user their answer.

Phase 2 — Agent with tools

Phase 2 builds a fresh Strands Agent per request — no shared mutable state between concurrent chats — restores the session’s conversation history into it, and runs the tool loop with six tools registered:

Tool	Purpose
`search_knowledge_base(query)`	Hybrid FAISS + BM25 search, top-7, RRF fusion. Scope-filter-aware.
`list_documents()`	Inventory of every indexed file with type and chunk count.
`analyze_document(filename)`	Inner Gemma call → structured summary (topics, entities, key facts).
`compare_documents(doc_a, doc_b, question)`	Inner Gemma call answering across two documents.
`calculator(expression)`	Safe AST evaluator — no `eval()`, no arbitrary code.
`current_time()`	Timestamp for time-aware queries.

The agent decides which tools to call and in what order. There’s no hard-coded router; the system prompt explains what’s available and Strands handles the loop. For most document questions the path is: search_knowledge_base → answer. For comparisons: compare_documents → answer. For “what files do I have?”: list_documents → answer. For greetings and arithmetic, the system prompt tells the agent it may skip search entirely. The model picks.

Two details that took debugging to get right:

Phase 2 runs with thinking explicitly disabled. Without that flag, Gemma’s default behaviour can leak <think>…</think> tags into the visible answer, and everything before the closing tag gets swallowed by the Markdown renderer. One model option — options={"thinking": False} — fixed a “truncated responses” bug that looked much scarier than it was.
Citations are flushed before the first answer token. Tools run before text deltas arrive, so by the time the first visible token streams, every source the search found is already in the sidebar. The accumulator is a request-local ContextVar the search tool appends to.

# Simplified — the real loop reads Strands' raw event dicts
async for event in agent.stream_async(user_input):
 delta = event["event"].get("contentBlockDelta", {}).get("delta", {}).get("text")
 if delta:
 for doc in new_citations(): # drain the ContextVar accumulator
 yield envelope("metadata", doc)
 yield envelope("text", delta)

Why this matters more than it sounds

You could implement similar behaviour with one agent call that interleaves thinking events with text events. The reasons I split it anyway:

The thinking model and the tool model can be different. Right now they’re both gemma4:e4b, but the architecture lets me swap a smaller, faster model in for Phase 1 reasoning and keep the big one for Phase 2 tool use. I’m not doing that yet — but I want the option.
Phase 1 always streams immediately. A pure agent loop only starts producing tokens after the model has decided what to say. Two-phase guarantees the user sees activity almost as soon as they press Enter, regardless of how complex the Phase 2 tool work gets.
Failures isolate. If Phase 2 falls over (Ollama timeout, tool error), Phase 1’s reasoning is still visible — the user can see what the model was trying to do, which makes the error far less frustrating than a blank “something went wrong.”

ContextVar isolation, again

The same ContextVar trick that scopes retrieval in carries here. At the start of each /rag stream, the handler sets two request-local variables: the document-scope filter and the citation accumulator. The agent’s tools read and write them implicitly. Conversation history itself lives in a per-session store guarded by per-session asyncio locks, so two concurrent requests in the same chat can’t corrupt each other either.

Tested with two browser tabs open on the same backend, scoped to different document categories, sending overlapping queries simultaneously. Zero cross-contamination. The test suite covers this explicitly in test_thinking.py and test_doc_scope_filter.py — see for the broader story.

The frontend side of the contract

A detail that tripped me up: this is a POST endpoint, so the browser’s EventSource API (which only does GET) is out. The frontend uses fetch and reads the response body incrementally, splitting on newlines and parsing each line as JSON:

// Simplified from useRagStream.ts
const res = await fetch("/rag", {
 method: "POST",
 body: JSON.stringify(payload),
});
const reader = res.body!.getReader();
const decoder = new TextDecoder();
let buffer = "";

while (true) {
 const { done, value } = await reader.read();
 if (done) break;
 buffer += decoder.decode(value, { stream: true });
 const lines = buffer.split("\n");
 buffer = lines.pop()!; // keep the trailing partial line
 for (const line of lines) {
 if (!line.trim()) continue;
 const { type, data } = JSON.parse(line);
 switch (type) {
 case "thinking":
 appendThinking(data);
 break;
 case "text":
 appendText(data);
 break;
 case "metadata":
 addCitation(data);
 break;
 case "memory":
 updateMemoryMeter(data);
 break;
 }
 }
}

The reasoning panel starts collapsed, with a small pulsing indicator while thinking tokens are still streaming — enough to signal “the model is working” without shoving a wall of chain-of-thought at the user. One click expands the full trace, during or after the stream.

What I’d revisit

Phase 1 reasons toward a full answer, and we throw the answer part away. A dedicated “plan your approach, don’t answer yet” prompt for Phase 1 would make the reasoning trace tighter and cheaper. Today it shares the main system prompt — simpler, but the trace can ramble.
No interrupt yet. Once Phase 1 starts, it runs to completion. If the user types a follow-up mid-stream we let it finish. A real cancel button would mean wiring an abort signal through Ollama’s HTTP client — feasible, not yet done.
Phase 1 occasionally over-thinks. Greetings and trivial questions still produce a paragraph of reasoning. A “should I think?” gate (probably a tiny classifier or even a heuristic on query length) would skip Phase 1 entirely for those cases.

Takeaway

Streaming is not just an optimisation. It’s a UX primitive. Two-phase streaming buys you a free property: the visible part of the interaction starts before the slow part does. The user gets to watch the model think, which is — genuinely — more interesting than watching a spinner.

If your agent app feels slow even though the answers are fast, look at when tokens start flowing. The fix often isn’t a faster model.

Appendix: Abbreviations in this post

Abbreviation	Full form	Meaning
NDJSON	Newline-Delimited JSON	A stream where each line is its own complete JSON object — what `/rag` emits
JSON	JavaScript Object Notation	The universal text format for structured data
UX	User Experience	How the product feels to use — the real beneficiary of two-phase streaming
UI	User Interface	The visible surface the stream renders into
FAISS	Facebook AI Similarity Search	The dense half of hybrid retrieval (previous post)
BM25	Best Match 25	The keyword half of hybrid retrieval (previous post)
RRF	Reciprocal Rank Fusion	The rank-only formula that merges the two result lists
AST	Abstract Syntax Tree	The parsed form of an expression — how the calculator evaluates maths without `eval()`
HTTP	HyperText Transfer Protocol	The protocol carrying the stream
SSE	Server-Sent Events	The browser’s built-in GET-only streaming format — notably not usable here, because `/rag` is a POST
API	Application Programming Interface	The boundary the frontend calls

Next up: — how CogniVault re-ingests edited PDFs without re-embedding everything, and survives a kill -9 mid-pipeline.

Part 2 · Hybrid Retrieval in Practice: FAISS + BM25, Fused with RRF

Sat, 25 Apr 2026 00:00:00 +0000

Part of a series on building , a fully local AI study companion. Previous: .

All abbreviations are fully explained in the appendix at the bottom of the page.

The first version of CogniVault used pure dense retrieval — embed the query with embeddinggemma, search a FAISS index, pass the top-7 chunks to the model. It worked. It worked beautifully — until a user uploaded a PDF containing some German legal text and asked for “§3 Absatz 2.”

The model couldn’t find it.

The chunk was right there. The PDF was indexed. But “§3 Absatz 2” doesn’t embed into anything semantically meaningful — it’s a token-level identifier, not a concept. The dense vector for the query landed nowhere near the dense vector for the chunk, even though the chunk literally contains the string the user asked for.

That bug killed pure dense retrieval for me. This post is about what replaced it.

Two kinds of “similar”

You already use both kinds of search every day. When Spotify builds a “song radio” from a track you like, it’s matching feel — tempo, mood, genre — and it will happily play you a song whose title shares no words with the original. But when you type Bohemian Rhapsody remastered 2011 into the search box, you don’t want feel. You want that exact string, and “a similar operatic rock epic” is a wrong answer.

Search systems formalise that split into two notions of similarity:

Lexical similarity — “do these strings share rare words?” This is what TF-IDF and BM25 model. They thrive on identifiers, names, code, technical terminology, and direct quotes.
Semantic similarity — “do these passages talk about the same idea, even with different words?” This is what embeddings model. They thrive on paraphrase, conceptual queries, and natural-language questions.

Neither subsumes the other. A user asking “how is the practical exam structured?” needs semantic search — the document doesn’t say “structure of practical exam.” A user asking "§3 Absatz 2" needs lexical search — there’s no concept to embed, just a literal string.

Production RAG has to do both. CogniVault does both, and then fuses the result lists with Reciprocal Rank Fusion (RRF).

The stack

Query
 ├── embed via embeddinggemma ──► FAISS IndexFlatIP ──► top-K dense
 └── tokenize + lowercase ──► BM25Okapi ──► top-K sparse
 │
 Reciprocal Rank Fusion ◄──┘
 │
 top-7 fused chunks

Both indexes live in memory, fronted by a VectorDB singleton. FAISS does inner-product search over normalised embeddings (so dot product = cosine). BM25 is rank_bm25’s BM25Okapi, fed the same chunks tokenised by a simple lowercase-and-split tokenizer.

The corpora are kept in lockstep: soft-deleting a file’s chunks triggers a BM25 rebuild over the remaining active chunks, and the singleton reloads both indexes from vector_store.faiss + vector_store.json (chunk metadata + raw text) after every ingestion run and on app start.

Why FAISS `IndexFlatIP`, not HNSW or IVF?

IndexFlatIP is brute-force exact search. It scans every vector, every query. For tens of thousands of chunks that’s fine — sub-millisecond on a laptop. CogniVault is a single-user, local-first app; the index is never going to be billions of vectors. Trading recall for speed via HNSW or IVF would buy nothing here and lose the “exact” guarantee. Boring, correct, fast enough.

When the corpus grows large enough that brute-force gets sticky, switching is a one-line change. Until then, the simplest index wins.

Reciprocal Rank Fusion

The naive way to combine two ranked lists is to score them and add. That sounds reasonable until you remember FAISS returns inner-product scores in some bounded range and BM25 returns scores in an unbounded one — they aren’t comparable without normalisation, and any normalisation you pick is somewhat arbitrary.

RRF sidesteps the problem entirely. It only looks at ranks, not scores. For each result list, an item at rank r contributes 1 / (k + r) to its final score (with k = 60 by convention — large enough to flatten the tail, small enough that the top items still dominate). Items that appear in both lists get summed.

# Simplified — the real implementation also de-duplicates chunks
# by (source, chunk_id, page) before scoring.
def reciprocal_rank_fusion(result_lists, k=60):
 scores = defaultdict(float)
 for results in result_lists:
 for rank, chunk_id in enumerate(results, start=1):
 scores[chunk_id] += 1.0 / (k + rank)
 return sorted(scores.items(), key=lambda kv: kv[1], reverse=True)

That’s the whole algorithm. No tuning, no calibration, no per-corpus weights. A chunk that’s #1 in BM25 and #4 in FAISS easily beats a chunk that’s #2 in only one of them. A chunk that both indexes agree on rises to the top deterministically.

The result for the “§3 Absatz 2” query: BM25 finds the literal match and lands it at rank 1. FAISS finds nothing useful (its top hits are about exam regulations in general). RRF surfaces the BM25 hit at the top of the fused list. Problem solved.

Scope filtering with ContextVar isolation

One detail that’s easy to get wrong: the retriever has to be scope-aware. CogniVault lets users limit a question to a single category or specific files. The scope is set by the request, but the search is called from deep inside the Strands agent loop, which is called from a streaming FastAPI handler, possibly with multiple concurrent requests in flight per worker.

Threading the scope through every function call would be ugly. A global is unsafe. The right primitive is Python’s , which gives you per-task isolated state that asyncio and threads both respect.

from contextvars import ContextVar

_doc_scope: ContextVar[DocScope | None] = ContextVar("doc_scope", default=None)

def set_doc_scope(scope: DocScope | None) -> None:
 _doc_scope.set(scope)

def current_doc_scope() -> DocScope | None:
 return _doc_scope.get()

The /rag request handler sets the scope at the very start of each streaming response; the search tool reads it; because the value is task-local, it dies with the request. No globals, no parameter drilling, no race conditions across concurrent users.

This is one of those design choices that looks like over-engineering until you have two browser tabs open and realise that without it, tab A’s scope filter would leak into tab B’s question.

Chunking choices that pay off downstream

Hybrid retrieval is only as good as the chunks. CogniVault uses a RecursiveCharacterTextSplitter with 1,000 characters, 100 overlap for unstructured text — small enough to keep retrieval precise, large enough to carry context for the model.

For structured formats it switches strategy:

Markdown → MarkdownHeaderTextSplitter emits one chunk per H1/H2/H3 section with the heading hierarchy prepended as a breadcrumb (“Privacy > Vault Audit > Indicators”). BM25 loves breadcrumbs — they make heading-keyword queries match cleanly.
CSV → header row + 20-row batches per chunk, so a query for a column name lands on the right block.
PPTX → one chunk per slide, title and body text together.
XLSX → header + row batches, per sheet, with a [Sheet: name] prefix.

Tiny fragments get filtered: unstructured text needs at least 100 characters to become a chunk, while the structured formats drop the bar to 20 — a two-line Markdown section or a header-only sheet is short but still meaningful. The recursive splitter is well-trodden territory, but the per-format strategies matter much more than people give them credit for.

What I’d do differently

A few things I’d revisit if I were starting over:

Stop tokenising for BM25 with str.split(). It’s fine, but a real tokenizer that handles punctuation and German compounds would meaningfully improve recall on the legal docs.
Add a small reranker. RRF gets the right set, but a cross-encoder rerank on the top 20 would polish the order. Locally-served, of course — there are good small ones now.
Query expansion for thin queries. Two-word questions like “§3 exam” could be expanded via a quick gemma4 call before retrieval. Latency cost, recall gain.

None of those are in the box yet. RRF over FAISS + BM25 is already so much better than either alone that I haven’t felt the pull to optimise further.

The takeaway

If your retrieval is “embed + cosine + top-k,” it will fail in exactly the way mine did — on the queries that contain literal identifiers your model has no embedding for. The fix isn’t a better embedding model. It’s a second retriever that doesn’t pretend everything is a concept.

FAISS for ideas. BM25 for strings. RRF to decide which one was right today.

Appendix: Abbreviations in this post

Abbreviation	Full form	Meaning
RAG	Retrieval-Augmented Generation	Retrieve relevant passages from your own documents first; let the model answer from them
FAISS	Facebook AI Similarity Search	Meta’s library for storing vectors and finding the most similar ones fast
BM25	Best Match 25	A keyword-ranking formula — the 25th ranking function developed in the Okapi information-retrieval system
RRF	Reciprocal Rank Fusion	Merges ranked lists using only ranks: each item scores `Σ 1/(k + rank)` across lists
TF-IDF	Term Frequency–Inverse Document Frequency	BM25’s ancestor: score words by how often they appear here vs how rare they are everywhere
IP (in `IndexFlatIP`)	Inner Product	The similarity measure FAISS computes; on normalised vectors it equals cosine similarity
HNSW	Hierarchical Navigable Small World	A popular approximate vector-index structure — deliberately not used here
IVF	Inverted File Index	Another approximate FAISS index type — also deliberately not used
AEVO	Ausbildereignungsverordnung	The German trainer-aptitude regulation whose “§3 Absatz 2” query broke pure dense retrieval
CSV / PPTX / XLSX	Comma-Separated Values / PowerPoint / Excel (Office Open XML)	Structured formats with their own chunking strategies
H1/H2/H3	Heading levels 1–3	The Markdown heading tiers used to split sections

Next up: — how CogniVault’s /rag endpoint streams Gemma 4’s thinking before any tool calls run.

Part 1 · Why I Built a Local-First RAG

Mon, 20 Apr 2026 00:00:00 +0000

All abbreviations are fully explained in the appendix at the bottom of the page.

I’ve spent the last few years in front of virtual classrooms full of career-changers in Germany, walking them through programming basics, web development, and introductory AI courses. Most of the information we deal with is fine to paste into cloud-based AI tools. Some of it really isn’t.

Exam materials under confidentiality. A trainee’s portfolio with personal details. Other private documents that should never end up training someone else’s model.

So I built — a fully local AI study and productivity tool. No cloud. No telemetry. No “we may use this data to improve our service.” Just Gemma 4 running on Ollama, on my laptop, talking to my files.

The leaky abstraction

The pitch for cloud AI is great: a giant model, available instantly, billed by the token. The fine print is where it gets uncomfortable:

Where does the data physically live during inference?
Whose jurisdiction governs that hardware this afternoon?
Does the audit trail stop at the API boundary, or can you actually trace what happened to your bytes?
When you tick “do not train on my data,” are you trusting a control, a contract, or both?

For most consumer use cases, those questions are fine to wave away. For education, healthcare, finance, legal, public administration — the answer “trust us” isn’t an answer.

What “local-first” actually means here

Lots of products say “private.” I wanted three concrete properties:

The model lives on your machine. Gemma 4 (gemma4:e4b) and embeddinggemma are pulled via Ollama. Inference is a localhost HTTP call.
Your documents never leave. Vectors, chunks, chat history, study sessions, achievements — all on disk on your computer.
You can verify it. Gemma CogniVault ships a Privacy Audit Panel that shows a live “zero external connections” indicator alongside document counts and the Ollama host. It’s not a promise — it’s a status light.

If a future build of Gemma CogniVault ever made an outbound call, that panel would be the first thing to scream.

What you get back

Going local sounds like a trade-off — surely you lose the magic of the giant frontier models? In practice, with Gemma 4 you get more than enough:

Thinking mode — Gemma 4’s chain-of-thought streams into a collapsible panel before the answer. Watching the model reason about your documents is genuinely useful as a teaching tool.
Tool use — through the , the model decides when to search the knowledge base, summarise a document, compare two files, or check the time.
Vision — attach images and PDFs straight into a chat turn.
Generation that’s actually structured — quizzes, multi-lesson workshops, flashcard decks, and interactive mindmaps, generated with format="json" so the output parses reliably.

Cognivault doesn’t try to be a giant ecosystem. It’s a single-purpose tool that does one thing well: use your own documents with a capable local model in a private environment. I must admit that it was inspired to a great extent by , which I’ve found incredibly useful but not private enough for my needs.

The shape of the app

CogniVault is split into four sections that map to how I actually work with information on cloud-based AI tools:

Section	What it’s for
Chat	Ask anything about your documents. Cited answers, scope filter, voice in.
Knowledge Base	Upload, categorise, manage. SHA-256 detects edits on re-upload.
Study Hub	Quiz · Workshop · Flashcards · Mindmaps — four ways to drill into the source.
Dashboard	Total study time, streak, 25 badges, GitHub-style 90-day heatmap.

Everything reachable from a sidebar that remembers where you left off, on a stack that fits in your ~/Documents folder.

What comes next

This is the first in a short series. Over the next few posts I’ll dig into the parts I’m most proud of — and a few I’d build differently next time:

Hybrid retrieval — why FAISS and BM25, fused with Reciprocal Rank Fusion
Two-phase streaming with Gemma 4 and Strands Agents
Crash-resumable ingestion with DBOS, hash-aware re-ingest, OCR fallback
Getting reliable JSON out of a local LLM (and what to do when it fails)
The mindmap renderer — what hand-rolling SVG taught me, and why v2 uses React Flow
Gamifying learning — 25 badges, idle-gap sessions, 90-day heatmap
Testing a local-AI app with 350+ tests and zero infrastructure

If you want to skip ahead, the code is open source at , and there’s a .

Your data. Your hardware. Your AI. Your vault.

Appendix: Abbreviations in this post

Abbreviation	Full form	Meaning
RAG	Retrieval-Augmented Generation	Retrieve relevant passages from your own documents first; let the model answer from them instead of from training memory
AI	Artificial Intelligence	Software performing tasks that normally need human intelligence
LLM	Large Language Model	A neural network trained on huge amounts of text that can read and generate language
HTTP	HyperText Transfer Protocol	The protocol browsers and APIs use to exchange requests and responses
API	Application Programming Interface	The boundary where you call someone else’s software — and where cloud audit trails stop
IHK	Industrie- und Handelskammer	The German Chamber of Commerce and Industry, which administers trainer certification
AEVO	Ausbildereignungsverordnung	The German trainer-aptitude regulation — the exam material that motivated this project
FAISS	Facebook AI Similarity Search	Meta’s vector-search library (covered in the next post)
BM25	Best Match 25	A classic keyword-ranking formula (also next post)
SDK	Software Development Kit	A library of building blocks — here, Strands, which provides the agent loop
JSON	JavaScript Object Notation	The universal text format for structured data
PDF	Portable Document Format	One of the eight-plus file types CogniVault ingests
SHA-256	Secure Hash Algorithm, 256-bit	A content fingerprint used to detect edited files on re-upload
OCR	Optical Character Recognition	Turning pictures of text (scans) into machine-readable text
DBOS	Database-Oriented Operating System	The durable-workflow library behind crash-resumable ingestion
SVG	Scalable Vector Graphics	The browser’s built-in vector drawing format

AI |

Gemma CogniVault

Overview

What’s inside

Four sections

Highlights

Writing about it

Try it

Part 8 · Testing a Local-AI App: 351 Tests, Zero Infrastructure

The 22 test files

What gets mocked, what doesn’t

Mocking Ollama

Mocking DBOS

Real SQLite, with one override

The TestClient pattern

Coverage gaps I accept

What the suite is actually for

What’s worth borrowing

Appendix: Abbreviations in this post

Part 5 · Getting Reliable JSON Out of a Local LLM

The pattern

Step 3: format="json" does real work

Step 2: schema-in-prompt that the model can actually obey

Step 4: parse, tolerantly

Step 5: drop malformed items, don’t fail the batch

Step 6: the outline retries once

Two-pass: outline first, lessons on demand

What I learned so far putting those generators together

Appendix: Abbreviations in this post

Part 3 · Two-Phase Streaming: Showing the Model Think Before It Acts

The shape

Phase 1 — Thinking only

Phase 2 — Agent with tools

Why this matters more than it sounds

ContextVar isolation, again

The frontend side of the contract

What I’d revisit

Takeaway

Appendix: Abbreviations in this post

Part 2 · Hybrid Retrieval in Practice: FAISS + BM25, Fused with RRF

Two kinds of “similar”

The stack

Why FAISS IndexFlatIP, not HNSW or IVF?

Reciprocal Rank Fusion

Scope filtering with ContextVar isolation

Chunking choices that pay off downstream

What I’d do differently

The takeaway

Appendix: Abbreviations in this post

Part 1 · Why I Built a Local-First RAG

The leaky abstraction

What “local-first” actually means here

What you get back

The shape of the app

What comes next

Appendix: Abbreviations in this post

Step 3: `format="json"` does real work

Why FAISS `IndexFlatIP`, not HNSW or IVF?