Agentic Financial Research
97% of answers traceable to the filing they came from, up from 63%
A financial research team needed dense filings turned into direct, checkable answers. We recommended an agentic retrieval loop with citation-level grounding and a scoped refusal boundary, benchmarked against the base models before it shipped.
- Client
- Atlas
- Documents
- 10-Ks, filings, contracts
- Build
- Python, Gemini, vector store
- Service areas
- Five of six
The result — benchmarked against the base models
- Answers grounded in source
- 97%
- Overall pass rate
- +22%
- Correct refusals, from zero
- 67%
01 — Brief
The problem
The client wanted an agentic platform that could read dense financial documents — annual reports, 10-Ks, filings, contracts — and put what is inside them in front of a reader as plainly as possible. The requirement was not a summary of the document; it was a direct answer to a specific question, with the passage it came from attached.
The work being replaced
An analyst opening a three-hundred-page filing and hunting for the figure by hand.
What replaced it
One question, one answer, and the page it was read off — checkable without leaving the reply.
02 — Constraint
Why the obvious path doesn’t work here
The standard LLM pipeline — extract the text, chunk it, embed it, retrieve once, answer — breaks on financial filings at three separate stages, and the failures compound.
Stage 1
ExtractionThe document is already wrong before the model sees it.
Ordinary extraction reads a two-column filing straight across the page, interleaving two unrelated sentences into one line, and fixed-window chunking cuts tables so figures arrive stripped of the row and column headers that told you what they measured. Everything downstream is reasoning over text the filing never said.
Stage 2
RetrievalOne similarity search can’t answer a financial question.
A normal pipeline embeds the question once and answers from whatever the top matches return. “How much did total assets grow across the years” needs figures from three separate pages, and a single search returns three passages about the same one. Embeddings are near-blind to the literals that decide the answer, too — the 2024 paragraph and the 2023 paragraph are almost identical vectors and completely different numbers.
Stage 3
AnsweringNothing in the chain can say “not in here.”
Handed passages that don’t contain the answer, the model answers anyway, filling the gap from what it knows about banks in general. Benchmarked on the client’s own documents, the base models correctly declined an out-of-scope question 0% of the time — and a fabricated figure arrives formatted, confident, and indistinguishable from a real one.
03 — Recommendation
Our solution
Build vs buy
A document-chat product cannot be steered into citation-level grounding it was not designed for, and the refusal behaviour the client needed is a property of the retrieval loop, not the model. The call was to build.
The loop, end to end
AI Strategy & Roadmap
Build an agentic retrieval loop rather than a chatbot wrapped around a PDF. The agent decides when to go back to the documents and what to look for, instead of answering from a single pass of context — which is what turns “here is roughly what the filing says” into “here is the figure, and here is the page it is on.” Scoped from the start to financial documents only, so the boundary of what it will answer is a design decision rather than an afterthought.
AI Stack
A custom Python service over a managed vector store rather than an off-the-shelf document-chat tool: column-aware extraction that crops each column of a two-column filing separately, page-as-parent chunking so a retrieved fragment is swapped back for its full page before the model reasons over it, and hybrid retrieval combining vector similarity with keyword search — because a financial question is half semantic and half an exact ticker, line item, or year. Gemini runs the reasoning loop; a knowledge graph built at ingestion handles the questions that span a whole document rather than sitting in one passage.
AI Governance
Every answer carries citations back to the exact passages it was drawn from, surfaced in the interface rather than buried — the reader checks the source without leaving the answer. The agent’s reasoning trace is exposed alongside each reply, so a wrong answer can be diagnosed instead of merely disbelieved, and the agent is scoped to refuse questions its documents do not cover. Full tracing on every model call and every request means a disputed answer can be reconstructed after the fact.
AI Evaluation
Bench the agentic loop against the base models before committing to it, across six dimensions rather than a single accuracy score. That evaluation is what established the loop was earning its complexity, and it stays as the regression suite for every change after.
GroundingRefusalHallucinated figuresOutput cleanlinessIn-scope answeringOverall passAI Security
Authenticated calls on every hop, and visitor-supplied model keys held only for the life of the session rather than persisted server-side. Ingestion is content-hashed and idempotent, so re-uploading a document cannot quietly duplicate or corrupt what is already in the store.
More case studies
Multi-Language AI Translation
100k+ Bible verses translated across five languages using AI
Yahuah Bible Platform — word-for-word Hebrew and Greek text alongside translations for 50K+ readers.
Multilingual Scripture AI
4x increase in subscriptions after adding a verse-aware assistant in six languages
Horeb Reference Bible — a Vertex AI assistant answering on the verse the reader names, evaluated in each of six languages.