On this page · 10 sections
Summary. Retrieval quality decides whether your internal-docs assistant works, and most builds get it wrong. A 2026 benchmark of ten retrieval strategies on financial text-and-table documents measured dense-only retrieval at Recall@5 of 0.587; adding BM25 and a reranking pass moved that to 0.816, a 39.0% relative gain. The model layer is the cheap part: OpenAI charged $0.02 per million tokens for text-embedding-3-small and $2.50 per million input tokens for gpt-5.6-terra as of July 2026, while Pinecone's Standard plan carries a $50/month minimum and Amazon OpenSearch Serverless bills $0.24 per OCU-hour. The expensive part is everything else. Stanford RegLab and HAI researchers tested LexisNexis and Thomson Reuters RAG products and found Lexis+ AI produced incorrect information more than 17% of the time and Westlaw's AI-Assisted Research more than 34% of the time, which is why "we used RAG" is not a quality claim. McKinsey's 2025 State of AI survey found more than 80% of respondents saw no tangible EBIT impact from generative AI. In India, DPDP Rules notified on 13 November 2025 set a hard compliance date of 13 May 2027, with penalties reaching ₹250 crore. This is what a working build costs, what it looks like, and where it breaks.
What breaks first is retrieval, not the model
Teams budget for tokens and discover the bill was never the problem. The assistant answers confidently and wrongly, users try it twice, and the project quietly dies.
The mechanism is simple. A RAG system does two things: it retrieves candidate documents, then it generates an answer from them. If the right document is not in the retrieved set, no model can recover. Generation quality is bounded by retrieval quality, and the 2026 benchmark of retrieval strategies over the FinQA, ConvFinQA and TAT-DQA datasets measured that bound directly: the correlation between Recall@5 and final answer accuracy was r > 0.99. Better retrieval, better answers, almost linearly.
That study also quantified the ceiling problem. Its authors note that the original benchmark found "the best method (Hybrid BM25) reached only 41% Number Match against an oracle-context ceiling of 72-79%, a gap of more than 30 percentage points." Even when you hand the model the correct context, accuracy tops out below 80% on hard numerical questions. Retrieval failure explains the other thirty-plus points.
The Stanford RegLab and HAI study makes the same point from the buyer's side. Its researchers built a pre-registered dataset of over 200 open-ended legal queries and ran it against products marketed on the promise of "hallucination-free" citations. Lexis+ AI and Ask Practical Law AI produced incorrect information more than 17% of the time; Westlaw's AI-Assisted Research hallucinated more than 34% of the time. GPT-4, with no retrieval at all, was worse. RAG helped. RAG did not solve it.
Daniel E. Ho, the William Benjamin Scott and Luna M. Scott Professor of Law and director of the RegLab at Stanford University, co-authored the study, which concludes: "Based on what we know, legal hallucinations have not been solved."
The study also separates two failure types worth naming in your own acceptance criteria. A response can be plainly incorrect, or it can be misgrounded: the answer is right but the citation does not support it. The second kind is harder to catch and more dangerous, because the citation exists and looks authoritative. As the authors put it, "even RAG systems are not hallucination-free."
Retrieval architectures, benchmarked
The 2026 arXiv benchmark evaluated ten retrieval methods on 23,088 queries over text-and-table financial documents. These are the measured numbers, not vendor claims.
| Retrieval method | Recall@5 | MRR@3 | nDCG@10 |
|---|---|---|---|
| Dense only (text-embedding-3-large) | 0.587 | 0.351 | 0.466 |
| HyDE (hypothetical document embeddings) | 0.544 | not reported | not reported |
| BM25 (lexical only) | 0.644 | not reported | not reported |
| Multi-query + RRF | 0.640 | 0.397 | 0.506 |
| CRAG (corrective RAG) | 0.658 | not reported | not reported |
| Hybrid (BM25 + dense, RRF) | 0.695 | 0.433 | 0.551 |
| Hybrid + Cohere Rerank | 0.816 | 0.605 | 0.683 |
Four things in that table are worth your attention.
BM25 beat dense retrieval. Plain lexical search at 0.644 Recall@5 outperformed OpenAI's text-embedding-3-large at 0.587 on this corpus, and beat it on every metric except Recall@20. Financial documents are full of exact tokens, company names, reporting periods, line-item labels, and lexical matching handles those better than semantic similarity. If your corpus is contracts, tickets, SKUs or policy documents, assume the same until you measure otherwise.
Hybrid fusion beat both. Reciprocal Rank Fusion over BM25 and dense results reached 0.695, improving on both constituent methods across every metric and every dataset subset. The paper's recommendation is blunt: "We recommend hybrid retrieval as the minimum viable baseline for any RAG deployment." RRF works on rank positions rather than scores, so it sidesteps score normalisation entirely. It is roughly twenty lines of code.
Reranking was the single largest gain. Adding a cross-encoder reranking pass on top of hybrid took Recall@5 from 0.695 to 0.816, a 17.4% relative improvement, and MRR@3 from 0.433 to 0.605, up 39.7%. Against dense-only, hybrid plus reranking is 39.0% better. No prompt engineering produces gains at that scale.
The clever techniques underperformed. HyDE scored 0.544, below plain dense retrieval, because LLM-generated hypothetical documents hallucinate plausible-but-wrong figures and drag the embedding away from the real context. Multi-query RAG-Fusion gave 0.640 against BM25's 0.644, effectively nothing. CRAG triggered its correction pathway on 63% of queries (14,569 of 23,088) and still landed at 0.658, below simple hybrid fusion. Sophistication is not the same as recall.
Contextual Retrieval was the exception that paid: prepending short LLM-generated context summaries to each chunk at indexing time added 2.8 percentage points of Recall@5 to dense and 2.2 points to hybrid. You pay for that once, at index time, not per query.
Where retrieval actually fails
The same study analysed the 7,188 queries (31.1%) where the gold document never appeared in the hybrid top-5, sampling 100 and categorising them.
| Failure mode | Share of sampled failures |
|---|---|
| Table structure mismatch | 73% |
| Numerical reasoning | 20% |
| Vocabulary mismatch | 5% |
| Multi-hop reasoning | 1% |
| Long document | 1% |
Seventy-three percent of failures were tables. The answer sat in a table whose markdown representation does not embed well as running text, so a query like "What was net income in 2019?" cannot match a row where "net income" and "2019" live in separate cells. Among all failures, 71.0% of gold documents appeared in neither the dense nor the BM25 top-5, which means these were genuinely hard retrieval cases and not artifacts of fusion.
The engineering lesson is unglamorous and worth more than any model upgrade: how you parse and chunk tables determines most of your accuracy. Most enterprise corpora are worse than this benchmark, not better, because they mix PDFs, scanned documents, spreadsheets and wiki pages. The real cost is usually the ingestion pipeline, not the code that calls the model.
What it actually costs to run
Vendor list prices, retrieved from primary sources in July 2026.
| Component | Vendor and rate (July 2026) | Notes |
|---|---|---|
| Embeddings | OpenAI text-embedding-3-small, $0.02 / 1M tokens | text-embedding-3-large is $0.13 / 1M; Batch API halves both |
| Generation | OpenAI gpt-5.6-terra, $2.50 / 1M input, $15.00 / 1M output | Cached input $0.25 / 1M; gpt-5.6-luna is $1.00 / $6.00 |
| Vector storage | Pinecone Standard, $50/month minimum | Query costs 1 RU per 1 GB of namespace, minimum 0.25 RU |
| Managed search | Amazon OpenSearch Serverless, $0.24 per OCU-hour | Next-gen collections scale to zero |
| Hosted retrieval | OpenAI file search, $2.50 / 1k calls + $0.10 / GB / day | First 1 GB of storage free |
Work an example. A 200,000-page internal corpus at roughly 500 tokens per chunk after cleaning is about 100 million tokens. Embedding it once with text-embedding-3-small costs $2.00. Re-embedding the whole corpus every month costs $24.00 a year. The embedding line item is a rounding error, and teams routinely spend three months arguing about it.
Storage sizing is arithmetic, and Pinecone documents the formula: records multiplied by ID size plus metadata size plus dimensions times 4 bytes. Their own worked example puts 1,000,000 records at 1,536 dimensions with 1,000 bytes of metadata at 7.15 GB; 500,000 records at 768 dimensions with 500 bytes of metadata is 1.79 GB. Note what dimensionality does to your bill. Halving dimensions from 1,536 to 768 roughly halves storage, and because a Pinecone query costs 1 RU per GB of namespace, it roughly halves query cost too.
Namespace design is the lever most teams miss. Query cost scales linearly with namespace size, not with how many results you ask for. The top_k, include_metadata and include_values parameters do not change RU cost at all. A 100 GB namespace costs 100 RUs per query; the same data split across departmental namespaces of 5 GB costs 5 RUs per query. That is a 20x difference from a partitioning decision made in week two.
Generation dominates once you have users. At $2.50 per million input tokens, an assistant that stuffs 8,000 tokens of retrieved context into each call and returns 500 tokens costs about $0.0275 per question. Ten thousand questions a month is roughly $275. Cached input at $0.25 per million cuts the input side by 90% where your system prompt and common context repeat, which in a knowledge assistant is most of the time.
Now compare that to the platform floor. Amazon OpenSearch Serverless bills $0.24 per OCU-hour, so holding two OpenSearch Compute Units around the clock costs about $350 a month (2 x $0.24 x 730 hours) whether or not anyone asks a question. AWS announced general availability of next-generation OpenSearch Serverless on 28 May 2026, with scale-to-zero, pay-per-usage pricing, auto-scaling 20x faster than its predecessor, and savings of "up to 60% compared to the cost of provisioning Opensearch clusters for peak loads." If you priced managed retrieval before mid-2026 and rejected it on that idle floor, the arithmetic has changed.
Pinecone's plan minimums as of July 2026: Starter $0/month, Builder $20/month flat, Standard $50/month, Enterprise $500/month. On Builder, usage beyond the plan limits is blocked rather than billed. HIPAA compliance is included at Enterprise; on Standard it is a $190/month add-on with a six-month minimum. Egress allowances run 1 GB free, 10 GB on Builder, 100 GB on Standard and Enterprise.
Build versus buy
| Dimension | Hosted assistant (vendor SaaS) | Managed retrieval (OpenSearch Serverless, file search) | Custom build |
|---|---|---|---|
| Time to first answer | Days | 2-4 weeks | 8-12 weeks |
| Monthly floor | Per-seat, scales with headcount | $0-$350 depending on collection type | $50-$500 infra plus engineering |
| Table and PDF handling | Vendor's parser, no control | Vendor's parser, limited control | Yours, tunable against the 73% failure mode |
| Access control granularity | Usually folder or group level | IAM plus metadata filters | Row-level, matched to your existing permissions |
| Data residency | Vendor regions | Regional endpoints, 10% uplift on eligible OpenAI models | Your choice of region |
| Evaluation harness | Rarely exposed | Bring your own | Built in from week one |
The honest answer for most organisations is that the hosted route is right until it is not, and the thing that breaks it is almost always permissions or parsing. If your documents carry per-user access rules, a folder-level ACL will leak. If a third of your answers live in tables, a generic parser caps you at the 0.587 tier and no amount of prompt work moves it.
India-specific considerations
The compliance clock is real and dated. India's DPDP Rules 2025 were notified on 13 November 2025. Provisions establishing the Data Protection Board of India took effect immediately. Consent-manager provisions take effect on 13 November 2026, twelve months from notification. The substantive compliance obligations take effect on 13 May 2027, eighteen months from notification, and that date is the hard enforcement line. Penalties for major violations reach ₹250 crore.
For a RAG assistant, three provisions matter more than the rest.
Purpose limitation cuts against the instinct to index everything. An assistant trained on the whole document store will surface personal data that was collected for a different purpose. Scope the corpus deliberately, and record why each source is in it.
The Significant Data Fiduciary designation carries localisation risk. The Rules empower the government to require that specified categories of personal data not leave India. The detailed list has not been notified as of July 2026. If you are in BFSI, healthcare or government-adjacent work, the safe architecture assumption is regional data residency, which means your vector store and your inference endpoint both need an India option or an on-premise path.
Data-principal rights need a deletion story. If a person exercises erasure, you must remove them from the source system, the chunk store, the vector index and any cached context. Teams that treat the index as a derived artifact they can rebuild later discover that "rebuild later" is not a compliance answer.
The market context is worth stating plainly. NASSCOM reports that 67% of Indian enterprises allocate less than 10% of IT budget to AI, and that India hosts 2,117 GCCs across 3,728 units employing about 2.36 million professionals as of FY26, with $98.4 billion in market revenue. The budgets are modest and the engineering capacity is enormous. That combination rewards a scoped, measurable build over a platform bet.
How we scope a build
eCorpIT is a Gurugram-based technology consulting organisation, founded in 2021, assessed at CMMI Level 5 and MSME certified, with partnerships including AWS, Microsoft and Google. Our senior engineering teams run RAG builds on a twelve-week shape, and we run them against numbers rather than demos.
Weeks 1-2 are corpus triage. We inventory the sources, measure what fraction of answers live in tables, and build the evaluation set before we build the pipeline. A hundred real questions with known-correct source documents, collected from the people who will use the thing. Without this you cannot tell an improvement from a regression, and the r > 0.99 relationship between Recall@5 and answer accuracy means this set is the only instrument that matters.
Weeks 3-5 are ingestion. Parsing, table extraction, chunking, metadata enrichment. This is where the 73% failure mode is won or lost, so it gets the most senior person on the team. Contextual Retrieval summaries go in here, at index time, for their 2.2-2.8 point recall gain.
Weeks 6-8 are retrieval. Hybrid BM25 plus dense with RRF as the baseline, measured against the week-1 evaluation set, then a reranking pass. We expect to land near the benchmark's 0.695 to 0.816 band. If we do not, the answer is in the ingestion layer, and we go back.
Weeks 9-10 are generation and grounding. Answer synthesis, citation rendering, and explicit refusal behaviour when retrieved evidence does not support an answer. We design applications aligned with DPDP requirements, including access filters that mirror your existing permissions rather than approximating them.
Weeks 11-12 are hardening and handover. Load behaviour, namespace partitioning for query cost, caching for the input-token line, monitoring on retrieval metrics rather than just uptime, and a runbook your team owns.
The engagement model is fixed-scope for the twelve weeks with a named senior team, then an optional support retainer. We publish the evaluation numbers at each phase gate. If retrieval does not clear the bar, you see that in week eight, not in month six.
For the broader governance picture around enterprise AI programmes, see our enterprise generative AI strategy guide. If the assistant is also meant to make your content visible to AI search engines, our guide to AEO, GEO and SEO covers how retrieval systems outside your firewall pick sources, and our 2026 SEO guide covers the ranking side. More about how we work is on our about page.
FAQ
How eCorpIT can help
We build retrieval systems that are measured, not demoed. Our senior engineering teams start from your corpus and your evaluation set, get hybrid retrieval and reranking to a recall number you can see, and design the access controls and deletion paths aligned with DPDP requirements before the assistant reaches a single user. eCorpIT has been building software from Gurugram since 2021, is assessed at CMMI Level 5, and partners with AWS, Microsoft and Google. If you want to know what your documents would actually cost to turn into an answer engine, contact us and we will scope it against real numbers.
References
Last updated: 16 July 2026.
