Swapping your vector index after a model upgrade

The upgrade that quietly breaks retrieval

You swap in a newer embedding model because it scores better on some leaderboard, or because your provider deprecated the old one and gave you a migration deadline. You update one line in your config, redeploy, and your RAG pipeline keeps running without a single error. Then a week later someone notices the chatbot is pulling irrelevant chunks for queries that used to work fine. Nothing crashed. Nothing logged a warning. The retrieval just got worse, and it took a while to notice because embedding similarity scores still come back looking plausible.

This is the failure mode that catches people off guard: changing the embedding model is not a config change, it’s a data migration. If you treat it like the former, you end up with a vector index that’s silently corrupted.

Why you can’t just point the same index at a new model

Every embedding model defines its own vector space. The numbers a model outputs aren’t measuring some universal, model-agnostic notion of “meaning.” They’re coordinates that only make sense relative to how that specific model was trained. Cosine similarity between a vector from model A and a vector from model B is not meaningless in the sense of throwing an error, it’s meaningless in the sense of returning a number that looks like a similarity score but isn’t measuring anything real.

So the moment you re-embed new documents with model B while your index still holds old vectors from model A, you have a mixed index. Queries embedded with model B will compare against a pile of vectors that live in a different space. Some comparisons will still land close by coincidence, which is exactly why this bug is hard to spot from the outside. Retrieval degrades gradually and unevenly instead of failing outright.

The fix isn’t complicated in concept: every vector in your index needs to come from the same model as the one embedding your queries. The complication is entirely in the mechanics of getting there without downtime, without a huge one-time bill, and without silently shipping a worse index than the one you started with.

Dimension mismatches will stop you cold, which is a mercy

If the new model outputs a different vector dimension than the old one, most vector databases will refuse the write outright. Pinecone indexes are locked to a fixed dimension at creation. pgvector columns are typed with a fixed dimension. FAISS indexes are built for one dimension. You’ll get an error immediately, and honestly, that’s the easy case. It forces you to build a new index instead of quietly poisoning the old one.

The dangerous case is when the new model happens to share the same dimension as the old one. Plenty of models output 1536 or 3072 dimensions, and if you swap between two models that share a dimension count, nothing stops you from writing new vectors right into the old index. This is exactly the scenario where you need a manual check, not just an assumption that “the write succeeded so it’s fine.”

The actual migration: re-embed everything

There’s no partial upgrade path. You have to re-embed your entire corpus with the new model, from raw source text, not from the old vectors. This means every chunk you’ve ever indexed goes through the new model’s API or local inference again. For a small knowledge base of a few thousand chunks this is trivial. For a corpus of a few million chunks, it’s a real cost and a real amount of wall-clock time, since most embedding APIs rate-limit you, and self-hosted models are bound by whatever GPU throughput you have.

Budget for this before you decide to switch models. If your embedding costs are billed per token, re-embedding is the same order of cost as your original ingestion run, not a marginal cost. If you’re chunking documents at ingestion time, you also need to decide whether to keep the same chunk boundaries or re-chunk. Some newer embedding models handle longer context windows well and produce better vectors with larger chunks than the model you’re replacing was ever able to use effectively. If you’re upgrading models anyway, it’s worth checking whether your chunking strategy was tuned around the old model’s limitations.

Build the new index alongside the old one, don’t overwrite in place

The safest pattern is a shadow index: create a completely separate index for the new model’s vectors, run your full re-embedding job into it, and leave the old index serving production traffic the entire time. This costs you double the storage for the migration window, which for most teams is a rounding error next to the API cost of re-embedding, but it buys you the ability to compare old and new retrieval side by side before anyone depends on the new one.

Once the new index is fully populated, run your actual query set (or a sample of real production queries if you have logging for that) against both indexes and look at what comes back. You’re checking for retrieval quality regressions, not embedding math. Did the top-k results change in ways that make sense? Are you pulling the same source documents for known good queries? This is where you catch problems like a new model that’s worse at your specific domain vocabulary, even if it does better on general benchmarks, since a leaderboard number doesn’t tell you what happens to your acronyms and product names.

Cut traffic over only after this check passes. Keep the old index alive for a rollback window. Deleting it the same day you cut over is how a bad migration turns into a bad incident with no way back.

Watch the similarity threshold, it moves

If your application has a hardcoded similarity cutoff, for example “only use retrieved chunks above 0.75 cosine similarity,” that number was tuned against the old model’s output distribution. Different embedding models don’t just place vectors differently, they also produce different overall score distributions. A 0.75 threshold that filtered noise well with one model might reject good matches or admit garbage with another. Re-tune the threshold against the new index rather than carrying the old number forward on the assumption that it’s some universal constant. It isn’t. It was fit to one model’s behavior.

The same applies to any reranking step downstream. If you have a reranker trained or calibrated against the retrieval patterns of the old embedding model, check that its inputs still make sense after the swap.

Normalize consistently, especially across providers

Some models output normalized vectors by default, some don’t, and whether you should use cosine similarity or dot product depends on that. If you switch from a model that returns normalized vectors to one that doesn’t, and your index config assumes normalization, similarity scores will be wrong even though nothing errors out. This is a small detail that’s easy to miss because it doesn’t show up until you’re comparing scores across the old and new index and wondering why the numbers look so different. Check your new model’s documentation for whether outputs are normalized, and match your index’s distance metric to that.

What actually justifies the migration

None of this is worth doing on a whim. Re-embedding a large corpus costs real money and takes real engineering time to verify. The honest reason to do it is a genuine improvement in retrieval quality for your specific data, which you can only confirm by running your own queries against both indexes, or a forced deprecation where your current model is being sunset and you have no choice. “The new model is newer” isn’t a reason on its own. If you’re going to pay the re-embedding bill, pay it because you’ve checked that it actually helps your use case, not because a release announcement told you it should.

If you’re building or maintaining a RAG pipeline and want more of this kind of grounded, no-nonsense breakdown, check out the rest of what we cover at AI Tool Gazette.