Part 4 · Crash-Resumable Ingestion: DBOS, SHA-256, and Surviving a kill -9

Part of a series on building Gemma CogniVault. Previously: Two-phase streaming with Strands Agents.

All abbreviations are fully explained in the appendix at the bottom of the page.

There are two things you absolutely don’t want your RAG ingestion pipeline to do:

1. Re-embed a 200-page PDF because you fixed a typo on page 12.
2. Lose its progress if you close the laptop lid halfway through.

The first wastes time and compute resources. The second leads to distrust in the system. Both have the same root: ingestion is treated like a fire-and-forget function, when it’s actually a long-running pipeline with intermediate state worth preserving.

CogniVault treats ingestion as a durable workflow. Specifically, a DBOS workflow checkpointed in Postgres, with content hashing for incremental work. This post walks through both pieces.

The pipeline

1. Scan docs/      → SHA-256 hash per file
                     ├── New file     → queue for embedding
                     ├── Changed file → soft-delete old chunks, re-embed
                     └── Unchanged    → skip (idempotent)

2. Extract text    → per-format extractor (PDF/OCR, DOCX, PPTX, XLSX, MD, CSV, TXT, HTML)
3. Chunk           → RecursiveCharacterTextSplitter (1000 chars, 100 overlap)
4. Embed           → embeddinggemma via Ollama, batches of 5
5. Save            → append to FAISS IndexFlatIP + JSON metadata on disk

The heavy stages run as DBOS steps inside one parent workflow, each one checkpointed: if the process dies between steps, the next start picks up at the last completed one.

SHA-256 as the source of truth

The naive approach is to track ingestion by filename. That breaks the first time someone edits a file in place. Filename is the same; content isn’t. The vector store quietly carries stale chunks.

The fix is content-addressed: hash the file bytes, store the hash alongside the chunks. Every ingestion run:

current_hash = hashlib.sha256(file_bytes).hexdigest()
stored_hash = chunk_metadata_for(filename).get("file_hash")

if stored_hash is None:
    schedule_ingest(filename)              # new file
elif stored_hash == current_hash:
    skip(filename)                         # unchanged
else:
    soft_delete_chunks_for(filename)       # changed
    schedule_ingest(filename)

This gives ingestion an idempotent property that’s worth its weight in gold: running the pipeline twice in a row does almost nothing the second time. That’s not just an optimisation — it’s what makes the next section possible.

DBOS workflows

DBOS is a Python library that turns regular functions into checkpointed workflows backed by Postgres. The model is dead simple: decorate a function with @DBOS.workflow(), mark each long-running call inside it as a @DBOS.step(), and DBOS records each step’s input, output, and status in Postgres as it runs.

If the workflow crashes — process killed, OS reboot, Postgres connection drop — the next start sees there’s an unfinished workflow with the same ID, replays the recorded step outputs from Postgres (without re-running them), and resumes from the first incomplete step.

Here’s the actual step structure (slightly simplified from backend/services/ingest.py):

@DBOS.workflow()
def ingest_workflow() -> int:
    filenames = list_document_files()          # @DBOS.step — scan + hash check
    docs = []
    for name in filenames:
        docs += process_single_document(name)  # @DBOS.step — extract text, one file each
    chunks = chunk(docs)                       # plain Python — fast, re-runs freely
    embeddings = []
    for batch in batches_of_5(chunks):
        embeddings += embed_batch(batch)       # @DBOS.step — the slow one, retried on failure
    save_vector_store(embeddings, chunks)      # @DBOS.step — append to FAISS + metadata
    return len(chunks)

The granularity of @DBOS.step is the granularity of crash recovery, and it’s chosen deliberately. Extraction is one step per file, so a crash during file 9 of 10 doesn’t re-read the first eight. Embedding is one step per batch of five chunks, for one specific reason: embed_batch is the slow one. If the laptop dies during embeddings, we resume the embedding loop at the failed batch, not at PDF extraction.

Notice what isn’t a step: chunking. Splitting text is fast pure-Python work — checkpointing it would cost more ledger bookkeeping than simply redoing it on a resume.

There’s a related sizing trick hiding in the batch number. DBOS records each step’s output in Postgres, and embed_batch returns its vectors — so each ledger entry contains five embeddings’ worth of floats. Small batches keep each checkpoint record small and each retry cheap. One giant “embed everything” step would mean one giant ledger row and zero resume granularity.

The format extractors

Step 2 (process_single_document) is a dispatch on file extension. Each extractor is small and obvious; the interesting choices are in the chunking strategy each one feeds downstream.

The OCR fallback is the one worth pausing on. PDFs come in two flavours: ones with a real text layer, and ones that are basically scanned images wearing a PDF costume. pypdf returns nothing useful for the second kind, but it doesn’t raise — it just hands back empty strings. Without a fallback, your “ingestion succeeded” log is lying to you.

The detector is a heuristic: if pypdf returns fewer than 50 characters for a page, route the page through pymupdf → Pillow → pytesseract OCR. Slower, but at least produces text. The threshold is tuned to be sensitive enough to catch scanned pages while not punishing legitimately short pages (a chapter cover, a colophon).

Soft delete, not hard delete

When a file changes and we re-ingest, the old chunks need to go. The temptation is to physically remove them from the FAISS index, but FAISS IndexFlatIP doesn’t support efficient delete — you’d have to rebuild.

Soft delete instead: changed files get their old chunks marked with a deleted: true flag in the metadata; new chunks are appended without it. Search filters on the flag at query time, so stale vectors sit harmlessly in the index. If enough dead weight ever accumulates, the escape valve is obvious — rebuild the index from active chunks only — but in practice I haven’t needed it.

This is the same pattern most append-only systems use. It pairs naturally with content hashing — flag-and-append is much cheaper than remove-and-rebuild. One subtlety: the keyword index has to follow suit. CogniVault’s VectorDB.delete_by_source() flips the flags and rebuilds BM25 over the remaining active chunks, so the two retrievers never disagree about what exists.

What the user sees

Starting an ingestion (POST /ingest) returns a workflow_id, and the frontend polls GET /ingest/status/{workflow_id} to draw a live timeline of the workflow’s steps — scanning, per-file extraction (“Reading pages… 3 of 21”), embedding (“Calibrating batch 4 of 12”), saving. If the user closes the tab mid-ingest, comes back five minutes later, and reopens — the workflow finished in the background regardless. The next call to GET /api/vault/stats reflects the new chunk count. No “click to resume” button, no manual recovery dance.

The first time I closed the lid mid-embedding and watched the workflow pick itself up from the next step on resume, I’ll admit I was a little smug. That’s exactly the property I wanted, with surprisingly little code.

Pitfalls and edges

A few things I had to learn the hard way:

• Don’t make embed_batch too big. Ollama isn’t great at backpressure. Batches of 5 are a sweet spot for embeddinggemma on a 16 GB machine — bigger batches stall on memory, smaller ones waste round-trip overhead. (And as noted above, the batch size doubles as your checkpoint-record size.)
• Be careful with file deletion. Soft-deleted chunks must also disappear from BM25’s corpus, or keyword search will keep returning text that dense search no longer sees. Rebuilding BM25 inside delete_by_source() keeps the two in lockstep.
• OCR is slow. A 50-page scan can take a minute or more. Surface that latency to the user; otherwise they think it’s hanging.

Takeaway

Durable workflows aren’t only for distributed systems. A single-user local app benefits from them in exactly the same ways: incremental work, crash recovery, idempotent retries. DBOS makes the cost of opting in trivially low — decorate your function, run Postgres locally, and you get a pipeline that survives lid-closes, OS updates, and your own Ctrl-C.

Combined with content-addressed hashing, ingestion stops being a thing you avoid touching for fear of having to wait 20 minutes. It becomes a thing you re-run whenever you feel like it — because re-running is free when nothing has changed.

Appendix: Abbreviations in this post

Next up: Getting reliable JSON out of a local LLM — what happens after Gemma 4 enthusiastically returns {"questions": [{"text": "..."},}].