4 RAG embedding models compared in 2026: Nemotron 3 Embed vs OpenAI, Cohere and open-weight

Summary. On 15 July 2026 NVIDIA released Nemotron 3 Embed, and the 8B checkpoint now leads the RTEB (beta) retrieval leaderboard with a score of 78.46, alongside 75.45 on MMTEB Retrieval. It outputs 4096-dimensional vectors, reads up to 32,768 tokens, and ships under the commercial-friendly OpenMDW-1.1 license. That headline hides the real decision. OpenAI text-embedding-3-small still costs $0.02 per million tokens, text-embedding-3-large costs $0.13, and Cohere Embed v4 lists around $0.12 for text with a 128,000-token context. Open-weight Qwen3-Embedding-8B (Apache 2.0) held the top MTEB multilingual spot at 70.58. For most teams the winning model is not the highest benchmark; it is the one that matches your data-residency rules, your token volume, and the dimension count your vector database can afford to store. This guide puts the four options side by side with benchmarks, prices, self-host break-even math, and working code.

Retrieval-augmented generation lives or dies on the retrieval step. If the embedding model returns the wrong passages, no amount of prompt engineering on the generator recovers the answer. So the embedding model is the highest-use component in a RAG stack, and in 2026 the choices split into three camps: closed APIs from OpenAI and Cohere, NVIDIA's newly released open Nemotron 3 Embed collection, and community open-weight models led by Qwen3-Embedding. The differences that matter are accuracy on realistic retrieval tasks, vector dimensionality, context length, license, and total cost at your volume.

The 2026 embedding landscape in one table

The table below lists the current flagship options and the specifications that drive an engineering decision. Prices are per million input tokens as of July 2026; embeddings bill on input only, with no separate output charge.

Three things stand out. First, the two open 8B models, Nemotron and Qwen3, both emit 4096-dimensional vectors, which is double text-embedding-3-large and roughly 2.7 times Cohere's default 1536. That storage difference compounds fast in a large index. Second, Cohere Embed v4 reads 128,000 tokens in a single pass, far past the 8,191 OpenAI accepts and the 32,768 the open 8B models handle, which matters for long-document embedding. Third, the open models carry no per-token fee but shift the cost to GPUs you have to run and keep busy.

What the benchmarks actually say

Benchmark scores are only useful if the benchmark resembles your workload. The Massive Text Embedding Benchmark (MTEB) and its multilingual extension (MMTEB) have been the reference for years, but they mix clustering, classification, and retrieval tasks, and several leaderboards became saturated as models trained on their public test sets. The Retrieval Embedding Benchmark (RTEB), introduced by the MTEB maintainers in 2026, focuses only on retrieval accuracy across real-world tasks and mixes open and held-out datasets to resist overfitting. For a RAG system, RTEB is the closer proxy.

NVIDIA published the following NDCG@10 figures for the Nemotron 3 Embed checkpoints on its Hugging Face model card, evaluated with the sequence length set to 4096.

The jump from the previous 1B generation (60.47 RTEB) to Nemotron-3-Embed-1B (72.38) is larger than the gap between the new 1B and 8B checkpoints. That is the practical finding: the 1B model recovers most of the accuracy at a fraction of the compute, which is why NVIDIA positions it for production-scale deployment while the 8B model chases the leaderboard. NVIDIA reports the 8B model as the leader on the multilingual RTEB (beta) leaderboard as of July 2026, and it was evaluated across 34 languages including Hindi, Hinglish, Marathi, Tamil, and Telugu.

On the open-weight side, Qwen3-Embedding-8B took the number-one MTEB multilingual position at 70.58 when it launched, with a 4096-dimensional hidden state, a 36-block transformer, and support for more than 100 languages. It is worth remembering that MTEB multilingual and RTEB are different scales, so 70.58 and 78.46 are not a like-for-like ranking; they measure different task mixes. Read each score against the benchmark it came from, not across benchmarks.

The closed APIs report strong MTEB numbers too, but OpenAI and Cohere publish fewer directly comparable RTEB figures. In practice, the sensible move is to run your own retrieval evaluation on a few hundred labelled query-passage pairs from your domain before you commit. A public leaderboard cannot tell you how a model handles your product names, your acronyms, or your support tickets.

Accuracy is not the whole story: dimensions and truncation

A 4096-dimensional vector carries more information than a 1536-dimensional one, but it costs more to store and search. At float32, one 4096-dim vector is 16 kilobytes; a 1536-dim vector is 6 kilobytes. Across 10 million chunks that is 160 GB versus 60 GB before index overhead. Larger vectors also slow approximate-nearest-neighbour search and raise memory bills on a hosted vector database. This is where dimension flexibility earns its keep.

OpenAI text-embedding-3-large supports Matryoshka representation learning, so you can request a shorter vector (for example 1024 or 256 dims) and keep most of the accuracy while cutting storage. text-embedding-3-small accepts any size from 256 to 1536 through the same dimensions parameter. Cohere Embed v4 offers 256, 512, 1024, and 1536 outputs plus byte and binary quantization. Qwen3-Embedding lets you set any output size from 32 to 4096. The open Nemotron 3 Embed checkpoints emit a fixed 4096-dim vector, so if storage is tight you truncate and re-normalise downstream rather than at the API.

The engineering rule is simple: pick the smallest dimension that still passes your retrieval evaluation. Many production RAG systems run well at 1024 dims. Paying to store 4096-dim vectors only helps if your evaluation shows a real recall gain on your data. If you want a deeper treatment of where those vectors live and what they cost to query, see our guide on pgvector versus a dedicated vector database.

The cost question: API price versus self-host break-even

This is where most comparisons go wrong. They quote the per-token API price and stop, or they quote a GPU hourly rate and stop. You need both, at your volume, with utilization included.

Start with the API side. Embedding pricing bills input tokens only. At July 2026 list prices, a workload that embeds 150 million tokens in a month (a realistic mid-size RAG: a one-time index of a few hundred thousand documents plus ongoing queries and new content) costs the following.

The self-host rows use on-demand GPU rates reported for July 2026: roughly $0.79 per hour for an L40S at specialised providers and about $2.01 per hour for an H100 PCIe, which work out to about $569 and $1,447 per month at continuous use. An 8B model in BF16 needs about 16 GB for weights, so it fits comfortably on a single 48 GB L40S with room for batching.

Now the break-even. A self-hosted L40S at $569 per month only undercuts text-embedding-3-large once you sustain about 4.4 billion tokens per month, and it does not beat text-embedding-3-small until roughly 28 billion tokens per month. Against Cohere Embed v4 the L40S break-even is near 4.7 billion tokens per month. Below those volumes, the managed API is cheaper and you carry no operational load.

The honest conclusion runs against the usual open-weight enthusiasm: on raw price, OpenAI text-embedding-3-small is very hard to beat, and self-hosting rarely wins on dollars per token until you are at large, steady scale with a GPU you keep saturated. The real reasons to self-host are not the token price. They are data residency, latency and throughput control, freedom from rate limits, and the ability to fine-tune. If your volume is spiky, a rented GPU sitting idle destroys the math, because you pay for 720 hours a month whether you use them or not. For a broader view of routing spend across models, our LLM cost and routing framework and the AI cloud cost playbook for Indian teams go further.

Code: calling an API and self-hosting an open model

Here is how each path looks in practice. First, an OpenAI call with a reduced dimension count to control storage.

            from openai import OpenAI
client = OpenAI()

resp = client.embeddings.create(
    model="text-embedding-3-large",
    input=["How do I reset my DPDP consent preferences?"],
    dimensions=1024,  # Matryoshka truncation: 3072 -> 1024
)
vector = resp.data[0].embedding  # length 1024
          

Cohere Embed v4 takes an input_type so queries and documents get the correct treatment, and lets you set the output dimension directly.

            import cohere
co = cohere.ClientV2()

docs = co.embed(
    model="embed-v4.0",
    texts=["Nemotron 3 Embed reads up to 32,768 tokens."],
    input_type="search_document",
    output_dimension=1024,
    embedding_types=["float"],
).embeddings.float
          

Self-hosting Nemotron 3 Embed with vLLM gives an OpenAI-compatible endpoint you run inside your own network.

            pip install "vllm==0.25.0"
vllm serve nvidia/Nemotron-3-Embed-8B-BF16 --host 0.0.0.0 --port 8000
# retrieval endpoint: POST /v2/embed with input_type "query" or "document"
          

For local batch work, Sentence Transformers reads the model's saved query and document prompts. Nemotron expects a query: prefix for questions and a passage: prefix for documents, and the vectors are L2-normalised, so cosine similarity and dot product are equivalent.

            from sentence_transformers import SentenceTransformer
import torch

model = SentenceTransformer(
    "nvidia/Nemotron-3-Embed-8B-BF16",
    device="cuda",
    model_kwargs={"dtype": torch.bfloat16},
)
model.max_seq_length = 32768

q = model.encode_query(["How is eczema distinguished from other rashes?"])
d = model.encode_document(["Eczema causes itchy, dry, inflamed patches of skin."])
score = model.similarity(q, d)  # single normalised cosine score
          

The open path adds an obligation the API hides: you own tokenisation, pooling, batching, GPU drivers, and uptime. That work is worth it when residency or scale demand it, and wasteful when they do not.

Data residency and India: DPDP and self-hosting

For teams handling Indian personal data, the model choice is partly a compliance decision. India's Digital Personal Data Protection Act, 2023 (DPDP) sets consent and processing obligations, and many enterprises prefer to keep document text and query logs inside infrastructure they control rather than sending them to a third-party embedding API abroad. Embedding requests carry the raw content of every document and every user query, so they are exactly the traffic a data-protection review scrutinises.

This is the strongest argument for the open models. Running Nemotron 3 Embed or Qwen3-Embedding on a GPU in an Indian region, or on-premises, keeps the text inside your boundary and removes a cross-border transfer from the review entirely. A single L40S at roughly ₹47,000 to ₹49,000 per month of continuous use can serve a substantial internal knowledge base, and the OpenMDW-1.1 and Apache 2.0 licenses both permit commercial use. We design applications aligned with DPDP requirements rather than claiming certification, and for cost-sensitive teams the residency benefit often justifies the GPU bill even when a managed API would be cheaper per token. Our note on DPDP compliance cost for Indian startups covers the wider obligations, and private LLM deployment covers the serving side.

How to choose: a decision guide

Match the model to the constraint that dominates your project.

Whatever you shortlist, run a retrieval evaluation on your own labelled pairs before you commit, keep the vector dimension as small as your recall target allows, and re-check prices, because vendors move them. The model that wins on a public leaderboard is rarely the one that wins on your data, at your volume, under your compliance rules.

FAQ

How eCorpIT can help

eCorpIT designs and ships production RAG systems for teams that need accuracy, cost control, and data residency at the same time. We benchmark embedding models on your own labelled data, size the vector database, and stand up self-hosted or API-backed retrieval that fits your DPDP posture and your budget. If you are choosing between Nemotron 3 Embed, a managed API, or an open-weight model, talk to our engineering team and we will scope a retrieval evaluation and a deployment plan for your workload.

References

1. NVIDIA Nemotron-3-Embed-8B-BF16 model card, Hugging Face

1. NVIDIA Nemotron 3 Embed ranks #1 on RTEB, Hugging Face blog

1. Retrieval Embedding Benchmark (RTEB), Hugging Face blog

1. MTEB leaderboard (RTEB and MMTEB), Hugging Face Spaces

1. MMTEB: Massive Multilingual Text Embedding Benchmark, arXiv 2502.13595

1. OpenAI new embedding models and API updates

1. OpenAI text-embedding-3-large model documentation

1. Cohere Embed models: dimensions and context length

1. Cohere pricing

1. Qwen3-Embedding-8B model card, Hugging Face

1. Qwen3 Embedding technical report, arXiv 2506.05176

1. Spheron GPU cloud pricing comparison 2026

1. Lambda GPU cloud pricing

_Last updated: 18 July 2026._