DBOS |

CogniVault Backend Explained, Part 2 · From File to Searchable Knowledge

Fri, 12 Jun 2026 00:00:00 +0000

All abbreviations are fully explained in the appendix at the bottom of the page.

An LLM cannot “open” your PDF. That sentence surprises a lot of newcomers, so let’s sit with it for a second: when you chat with your documents in CogniVault, the model never touches the original files. Something has to happen between “I dropped a file into the browser” and “the AI just quoted page 47 back at me.”

That something is ingestion, and it’s the subject of this part. In we drew the whole map; today we zoom into one region — the conveyor belt that turns files into searchable knowledge.

The conveyor belt

Think of ingestion as a four-station assembly line:

Extract the text out of each file — even scanned ones.
Chunk it into pieces small enough to fit into a prompt.
Embed each chunk — turn it into a vector (a list of numbers that captures its meaning) so similar ideas land near each other in vector space.
Store vectors and metadata so they can be searched later.

flowchart TD A["Upload
POST /upload
saved to docs/"] --> B subgraph WF["DBOS durable workflow"] B["Step 1
Which files changed?
SHA-256 fingerprints"] --> C["Step 2
Extract text
per-format + OCR fallback"] C --> D["Chunk
1000 chars, 100 overlap"] D --> E["Step 3
Embed
embeddinggemma, batches of 5"] E --> F["Step 4
Save
FAISS index + metadata JSON"] end F --> G["Reload in-memory index
instantly searchable"]

Simple enough. The interesting engineering is in the failure cases — so let’s start there.

The factory ledger: why the pipeline can’t lose work

Embedding a large library takes minutes. What happens when your laptop goes to sleep at page 800 of a 1,000-page manual? With a plain Python script: everything restarts from page 1.

CogniVault instead writes the pipeline as a DBOS durable workflow. Picture a factory where every station stamps a permanent ledger the moment it finishes a box. If the power cuts out, nobody rebuilds finished boxes — the workers read the ledger and resume at the first unstamped entry.

DBOS is that ledger, and PostgreSQL is the book it’s written in. Each pipeline station is a checkpointed step; on restart, completed steps return their recorded results instantly and execution continues from the first unfinished one. A failed embedding batch is simply retried.

This is also what powers the live progress timeline in the UI: starting an ingestion returns a workflow_id, and the frontend polls a status endpoint that reports which steps have completed, which are running, and which are still waiting.

I wrote a whole deep dive on this mechanism — including what happens when you kill -9 the process mid-ingest — in .

Fingerprints, not faith: SHA-256 change detection

Re-embedding your whole library every time you add one file would be wasteful. So before any work happens, the pipeline computes each file’s SHA-256 hash (a content fingerprint — change one character in the file and the fingerprint changes completely) and compares it to the fingerprint stored with the file’s existing chunks:

Never seen before → ingest it.
Fingerprint changed → the old chunks are soft-deleted and the file is re-ingested.
Fingerprint identical → skip it entirely.

Why “soft”-deleted? Because the FAISS index type CogniVault uses cannot remove individual vectors. Stale chunks are just marked deleted: true in the metadata; their vectors stay in the index but every search filters them out. It’s an honest, boring solution — and it never corrupts the index.

Every format gets its own treatment

Here’s a detail that separates a demo from a product. A naive pipeline extracts “all the text” and calls it a day. CogniVault gives each format an extractor that preserves the structure that retrieval will need later:

Format	Strategy
PDF	Page by page, keeping page numbers (those become citations later). Any page yielding fewer than 50 characters is presumed scanned and sent to OCR
Scanned page	The page is rendered to an image at roughly 144 dpi, then Tesseract OCR (Optical Character Recognition — reading text out of images) extracts the words
Markdown	Split on headings; each section chunk gets a breadcrumb prefix like `[Section: Intro > Setup]` so its embedding carries the document hierarchy
CSV	Rows grouped 20 per chunk — and every chunk is prefixed with the header row, so the model always knows the column names
Excel	Same row-group idea per sheet, prefixed `[Sheet: name]`
PowerPoint	One chunk per slide
Word	Paragraphs plus table cells
Web pages	Fetched on request and stripped to clean article text — behind an SSRF guard (Server-Side Request Forgery protection: the server refuses to fetch private or internal addresses)

Ask yourself why the CSV detail matters. If chunk 14 of a spreadsheet is just twenty naked rows of numbers, no search will ever connect it to the question “what was the Q3 budget?” Prefix it with the header row, and the chunk knows it contains budget columns. Structure is retrieval fuel.

Chunking: 1,000 characters with a 100-character safety overlap

Long text is split into pieces of about 1,000 characters, with neighbouring pieces overlapping by 100. The overlap is insurance: a sentence sliced at a chunk boundary still appears whole in one of the two neighbours, so no idea falls into the gap between chunks.

Embedding and saving

Chunks are embedded by embeddinggemma (via Ollama) in batches of five — each chunk becomes one vector. The vectors are normalised and appended to a FAISS index; alongside it, a JSON file records each chunk’s source filename, page number, category, fingerprint, and the text itself. The index holds the numbers; the JSON holds the meaning.

One choice worth highlighting for beginners: this is an exact index, not an approximate one. Many vector databases use ANN (Approximate Nearest Neighbour) shortcuts that trade a little accuracy for speed at massive scale. At personal-library scale you don’t need the trade — CogniVault checks every vector on every search and is still fast.

The whole journey, end to end

%%{init: {'sequence': {'actorFontSize': 28, 'messageFontSize': 24, 'loopTextFontSize': 22, 'noteFontSize': 22}}}%% sequenceDiagram actor U as You participant F as Frontend participant B as FastAPI participant W as DBOS Workflow participant O as Ollama (embeddinggemma) participant V as FAISS + metadata U->>F: Drag and drop a file, pick a category F->>B: POST /upload B->>B: Validate type and size, save to docs/ F->>B: POST /ingest B->>W: Start durable workflow B-->>F: workflow_id loop Poll status F->>B: GET /ingest/status/{workflow_id} B-->>F: Step list (drives the progress timeline) end W->>W: SHA-256 change detection W->>W: Extract text (per format, OCR if scanned) W->>W: Chunk (1000 chars / 100 overlap) W->>O: Embed in batches of 5 O-->>W: Vectors W->>V: Append vectors + metadata B-->>F: SUCCESS — index reloaded F-->>U: "Knowledge Sync Complete"

The takeaway

Ingestion is where most RAG quality is actually won or lost — long before any clever prompting. Page numbers preserved, headers carried into every spreadsheet chunk, scans rescued by OCR, and a ledger that makes the whole thing crash-proof: none of it is glamorous, all of it shows up later as answers that cite the right page.

Appendix: Abbreviations in this post

Abbreviation	Full form	Meaning
LLM	Large Language Model	A neural network trained on huge amounts of text that can read and generate language
DBOS	Database-Oriented Operating System	The library that checkpoints workflow steps in PostgreSQL so crashed jobs resume
SHA-256	Secure Hash Algorithm, 256-bit	A content fingerprint — change one byte of a file and the hash changes completely
OCR	Optical Character Recognition	Reading text out of images — the rescue path for scanned PDF pages
SSRF	Server-Side Request Forgery	An attack where a server is tricked into fetching internal URLs; the URL importer blocks it
FAISS	Facebook AI Similarity Search	The vector index the embeddings are appended to
ANN	Approximate Nearest Neighbour	The accuracy-for-speed shortcut CogniVault deliberately does not take
dpi	Dots Per Inch	Image resolution — scanned pages are rendered at ~144 dpi before OCR
JSON	JavaScript Object Notation	The format of the chunk-metadata file beside the FAISS index
PDF / CSV	Portable Document Format / Comma-Separated Values	Two of the eight-plus supported file formats
API	Application Programming Interface	The endpoints (`/upload`, `/ingest`, `/ingest/status/…`) driving the flow

Next up: — hybrid retrieval, the six-tool agent, and the two-phase stream that shows the model think before it answers.

Part 2 · CogniVault Architecture: Durable Ingestion with DBOS

Tue, 02 Jun 2026 00:00:00 +0000

All abbreviations are fully explained in the appendix at the bottom of the page.

In a basic local AI setup, adding documents to your database is usually just a simple Python script. You open a PDF, chop the text into chunks, turn those chunks into math (embeddings), and save them.

This works great for a five-page essay. But what happens when you are ingesting a 1,000-page technical manual and your laptop goes to sleep at page 800?

The script dies. When you wake your laptop up, you have to start all over from page 1, wasting time and compute power. A simple script wasn’t going to cut it for CogniVault. We needed a Durable Workflow.

The Factory Ledger (DBOS)

Think of data ingestion like a factory assembly line. If the power goes out, the workers shouldn’t have to rebuild every product from scratch. They should just look at a permanent ledger, see exactly which box they were packing when the lights went out, and resume from there.

CogniVault uses a framework called DBOS (Database-Oriented Operating System) backed by a PostgreSQL database to act as this ledger.

Every step of the ingestion process records its completion in Postgres. If the server crashes mid-way, nothing dramatic happens in the moment — the magic is on restart: DBOS reads the ledger, sees which steps already finished, replays their recorded results instantly, and resumes from the first unfinished step.

One important boundary: Postgres holds only the ledger — which steps ran and what they returned. Your documents, chunks, and vectors never live there. They go to a FAISS index plus a JSON metadata file on disk.

SHA-256 Hashing: The Idempotency Trick

The system also needs to be smart about re-uploads. If you fix a typo in a massive document and upload it again, you don’t want the system to waste 10 minutes re-embedding the whole thing.

CogniVault achieves Idempotency (the ability to run the same operation multiple times without changing the result beyond the initial application) with the workflow’s very first step: it scans the docs/ folder and generates a SHA-256 hash (a unique digital fingerprint) for every file.

If the hash is new, it processes the file.
If the hash has changed (because you edited the file), it soft-deletes the old text chunks and only re-embeds the new version.
If the hash is identical, it skips the file entirely.

We can see here how this flows logically:

graph TD Raw[📄 Uploaded Document] --> DBOS[🐘 DBOS Workflow Starts] subgraph Durable Ingestion Pipeline DBOS -->|Step 1| Hash{Hash Check SHA-256} Hash -->|Unchanged| Skip[Skip Processing] Hash -->|New / Changed| Extract[✂️ Step 2: Extract Text per Document] Extract --> Chunk[Chunk: 1000 chars, 100 overlap] Chunk -->|Step 3, batches of 5| Embed[🔢 embeddinggemma Embeddings] Embed -->|Step 4| Save[(💾 FAISS Index + Metadata JSON)] end Save -->|Workflow Complete| Done[✅ Ready for Search]

(A detail for the curious: the checkpointed steps are the scan, the per-document extraction, each embedding batch, and the save. The chunking in between is fast pure-Python work, so it simply re-runs as part of the workflow body — checkpointing it would cost more than redoing it.)

What’s Next?

By wrapping the ingestion pipeline in DBOS, the system transforms from a fragile script into a resilient, production-grade state machine.

Now that our data is safely ingested, how do we deploy this entire pipeline without melting our laptop’s GPU? Read Part 3: Why We Keep Ollama Out of Docker

You can also explore the DBOS implementation directly in the backend/services/ingest.py file of the .

Appendix: Abbreviations in this post

Abbreviation	Full form	Meaning
DBOS	Database-Oriented Operating System	A library that checkpoints workflow steps in a database so crashed jobs resume instead of restarting
SHA-256	Secure Hash Algorithm, 256-bit	A fingerprint function: any file maps to a unique 64-character hash; change one byte and the hash changes completely
PDF	Portable Document Format	The document format whose text (and scans) the pipeline extracts
FAISS	Facebook AI Similarity Search	Meta’s vector-search library — where the embeddings actually live
JSON	JavaScript Object Notation	The text format used for the chunk-metadata file stored next to the FAISS index
AI	Artificial Intelligence	Software performing tasks that normally need human intelligence
GPU	Graphics Processing Unit	The hardware that makes local model inference fast — the subject of Part 3

Part 4 · Crash-Resumable Ingestion: DBOS, SHA-256, and Surviving a kill -9

Tue, 05 May 2026 00:00:00 +0000

Part of a series on building . Previously: .

All abbreviations are fully explained in the appendix at the bottom of the page.

There are two things you absolutely don’t want your RAG ingestion pipeline to do:

Re-embed a 200-page PDF because you fixed a typo on page 12.
Lose its progress if you close the laptop lid halfway through.

The first wastes time and compute resources. The second leads to distrust in the system. Both have the same root: ingestion is treated like a fire-and-forget function, when it’s actually a long-running pipeline with intermediate state worth preserving.

CogniVault treats ingestion as a durable workflow. Specifically, a workflow checkpointed in Postgres, with content hashing for incremental work. This post walks through both pieces.

The pipeline

1. Scan docs/ → SHA-256 hash per file
 ├── New file → queue for embedding
 ├── Changed file → soft-delete old chunks, re-embed
 └── Unchanged → skip (idempotent)

2. Extract text → per-format extractor (PDF/OCR, DOCX, PPTX, XLSX, MD, CSV, TXT, HTML)
3. Chunk → RecursiveCharacterTextSplitter (1000 chars, 100 overlap)
4. Embed → embeddinggemma via Ollama, batches of 5
5. Save → append to FAISS IndexFlatIP + JSON metadata on disk

The heavy stages run as DBOS steps inside one parent workflow, each one checkpointed: if the process dies between steps, the next start picks up at the last completed one.

SHA-256 as the source of truth

The naive approach is to track ingestion by filename. That breaks the first time someone edits a file in place. Filename is the same; content isn’t. The vector store quietly carries stale chunks.

The fix is content-addressed: hash the file bytes, store the hash alongside the chunks. Every ingestion run:

current_hash = hashlib.sha256(file_bytes).hexdigest()
stored_hash = chunk_metadata_for(filename).get("file_hash")

if stored_hash is None:
 schedule_ingest(filename) # new file
elif stored_hash == current_hash:
 skip(filename) # unchanged
else:
 soft_delete_chunks_for(filename) # changed
 schedule_ingest(filename)

This gives ingestion an idempotent property that’s worth its weight in gold: running the pipeline twice in a row does almost nothing the second time. That’s not just an optimisation — it’s what makes the next section possible.

DBOS workflows

is a Python library that turns regular functions into checkpointed workflows backed by Postgres. The model is dead simple: decorate a function with @DBOS.workflow(), mark each long-running call inside it as a @DBOS.step(), and DBOS records each step’s input, output, and status in Postgres as it runs.

If the workflow crashes — process killed, OS reboot, Postgres connection drop — the next start sees there’s an unfinished workflow with the same ID, replays the recorded step outputs from Postgres (without re-running them), and resumes from the first incomplete step.

Here’s the actual step structure (slightly simplified from backend/services/ingest.py):

@DBOS.workflow()
def ingest_workflow() -> int:
 filenames = list_document_files() # @DBOS.step — scan + hash check
 docs = []
 for name in filenames:
 docs += process_single_document(name) # @DBOS.step — extract text, one file each
 chunks = chunk(docs) # plain Python — fast, re-runs freely
 embeddings = []
 for batch in batches_of_5(chunks):
 embeddings += embed_batch(batch) # @DBOS.step — the slow one, retried on failure
 save_vector_store(embeddings, chunks) # @DBOS.step — append to FAISS + metadata
 return len(chunks)

The granularity of @DBOS.step is the granularity of crash recovery, and it’s chosen deliberately. Extraction is one step per file, so a crash during file 9 of 10 doesn’t re-read the first eight. Embedding is one step per batch of five chunks, for one specific reason: embed_batch is the slow one. If the laptop dies during embeddings, we resume the embedding loop at the failed batch, not at PDF extraction.

Notice what isn’t a step: chunking. Splitting text is fast pure-Python work — checkpointing it would cost more ledger bookkeeping than simply redoing it on a resume.

There’s a related sizing trick hiding in the batch number. DBOS records each step’s output in Postgres, and embed_batch returns its vectors — so each ledger entry contains five embeddings’ worth of floats. Small batches keep each checkpoint record small and each retry cheap. One giant “embed everything” step would mean one giant ledger row and zero resume granularity.

The format extractors

Step 2 (process_single_document) is a dispatch on file extension. Each extractor is small and obvious; the interesting choices are in the chunking strategy each one feeds downstream.

Format	Library	Chunking note
PDF	`pypdf` page-by-page; `pytesseract` OCR fallback for image-only pages	Recursive splitter, 1000/100
DOCX	`python-docx` (paragraphs + table rows joined as text)	Recursive splitter
PPTX	`python-pptx`	One chunk per slide (title + body text)
XLSX	`openpyxl`	Header + 20-row batches, per sheet
MD	`MarkdownHeaderTextSplitter`	One chunk per H1/H2/H3 section, breadcrumb prepended
CSV	manual reader	Header row + 20-row batches
TXT	raw UTF-8 read	Recursive splitter
HTML	`trafilatura` clean text	Recursive splitter

The OCR fallback is the one worth pausing on. PDFs come in two flavours: ones with a real text layer, and ones that are basically scanned images wearing a PDF costume. pypdf returns nothing useful for the second kind, but it doesn’t raise — it just hands back empty strings. Without a fallback, your “ingestion succeeded” log is lying to you.

The detector is a heuristic: if pypdf returns fewer than 50 characters for a page, route the page through pymupdf → Pillow → pytesseract OCR. Slower, but at least produces text. The threshold is tuned to be sensitive enough to catch scanned pages while not punishing legitimately short pages (a chapter cover, a colophon).

Soft delete, not hard delete

When a file changes and we re-ingest, the old chunks need to go. The temptation is to physically remove them from the FAISS index, but FAISS IndexFlatIP doesn’t support efficient delete — you’d have to rebuild.

Soft delete instead: changed files get their old chunks marked with a deleted: true flag in the metadata; new chunks are appended without it. Search filters on the flag at query time, so stale vectors sit harmlessly in the index. If enough dead weight ever accumulates, the escape valve is obvious — rebuild the index from active chunks only — but in practice I haven’t needed it.

This is the same pattern most append-only systems use. It pairs naturally with content hashing — flag-and-append is much cheaper than remove-and-rebuild. One subtlety: the keyword index has to follow suit. CogniVault’s VectorDB.delete_by_source() flips the flags and rebuilds BM25 over the remaining active chunks, so the two retrievers never disagree about what exists.

What the user sees

Starting an ingestion (POST /ingest) returns a workflow_id, and the frontend polls GET /ingest/status/{workflow_id} to draw a live timeline of the workflow’s steps — scanning, per-file extraction (“Reading pages… 3 of 21”), embedding (“Calibrating batch 4 of 12”), saving. If the user closes the tab mid-ingest, comes back five minutes later, and reopens — the workflow finished in the background regardless. The next call to GET /api/vault/stats reflects the new chunk count. No “click to resume” button, no manual recovery dance.

The first time I closed the lid mid-embedding and watched the workflow pick itself up from the next step on resume, I’ll admit I was a little smug. That’s exactly the property I wanted, with surprisingly little code.

Pitfalls and edges

A few things I had to learn the hard way:

Don’t make embed_batch too big. Ollama isn’t great at backpressure. Batches of 5 are a sweet spot for embeddinggemma on a 16 GB machine — bigger batches stall on memory, smaller ones waste round-trip overhead. (And as noted above, the batch size doubles as your checkpoint-record size.)
Be careful with file deletion. Soft-deleted chunks must also disappear from BM25’s corpus, or keyword search will keep returning text that dense search no longer sees. Rebuilding BM25 inside delete_by_source() keeps the two in lockstep.
OCR is slow. A 50-page scan can take a minute or more. Surface that latency to the user; otherwise they think it’s hanging.

Takeaway

Durable workflows aren’t only for distributed systems. A single-user local app benefits from them in exactly the same ways: incremental work, crash recovery, idempotent retries. DBOS makes the cost of opting in trivially low — decorate your function, run Postgres locally, and you get a pipeline that survives lid-closes, OS updates, and your own Ctrl-C.

Combined with content-addressed hashing, ingestion stops being a thing you avoid touching for fear of having to wait 20 minutes. It becomes a thing you re-run whenever you feel like it — because re-running is free when nothing has changed.

Appendix: Abbreviations in this post

Abbreviation	Full form	Meaning
DBOS	Database-Oriented Operating System	A library that checkpoints workflow steps in Postgres so crashed jobs resume instead of restarting
SHA-256	Secure Hash Algorithm, 256-bit	A content fingerprint: change one byte of a file and the hash changes completely
RAG	Retrieval-Augmented Generation	Retrieve relevant passages from your own documents first; let the model answer from them
OCR	Optical Character Recognition	Turning pictures of text (scanned pages) into machine-readable text
FAISS	Facebook AI Similarity Search	The vector index the embeddings are appended to
IP (in `IndexFlatIP`)	Inner Product	FAISS’s similarity measure; equals cosine similarity on normalised vectors
BM25	Best Match 25	The keyword index that must stay in lockstep with FAISS on deletes
PDF / DOCX / PPTX / XLSX / MD / CSV / TXT / HTML	Portable Document Format / Word / PowerPoint / Excel / Markdown / Comma-Separated Values / plain text / HyperText Markup Language	The formats the per-extension extractors handle
JSON	JavaScript Object Notation	The format of the chunk-metadata file next to the FAISS index
UTF-8	Unicode Transformation Format, 8-bit	The text encoding used when reading plain-text files
OS	Operating System	What reboots underneath you mid-ingest

Next up: — what happens after Gemma 4 enthusiastically returns {"questions": [{"text": "..."},}].