Embedding dimension is a parameter people tend to inherit rather than choose, because the model dictates it. Yet it is what sets the memory bill for your index, and therefore its running cost.
The basic arithmetic
A vector of dimension d stored as 32-bit floats takes 4 × d bytes. For d = 768, that is a little over 3 KB per vector — roughly 3 GB for a million vectors, before any index at all.
On top of that sits the search structure. An HNSW graph keeps a neighbour list per node per layer; with an M parameter of 16, you typically store around 32 identifiers for the base layer, plus upper layers whose population decays geometrically. At 4 bytes per identifier, the graph overhead lands somewhere between 150 and 200 bytes per vector. That stays modest next to raw data at dimension 768, but it becomes dominant at low dimensions or after aggressive quantization.
The order of magnitude to remember: memory for a dense index is driven first by d, then by M.
What dimension buys you, and where it stops
Raising d gives the model more capacity to separate neighbouring concepts. The gain is not linear: past a certain point, the extra dimensions encode variance that matters less and less for the search task, while memory cost keeps growing in proportion.
A subtler effect appears in high dimensions: distance concentration. As d grows, distances between points drawn from the same distribution cluster around a mean value, and the contrast between the nearest neighbour and the runners-up narrows. Learned embeddings partly escape this, because they occupy a manifold whose intrinsic dimension is far below d, but it explains why doubling the dimension does not mechanically improve recall.
Shrinking without retraining
Two levers cut cost without changing models.
Truncation, first, when the model was trained for it. Matryoshka representation models order information by decreasing importance: you can cut the vector at the first k components and renormalize, with graceful recall degradation. On a model without that property, truncating removes information at random and the degradation is abrupt.
Quantization, second. Going from float32 to an 8-bit integer per component divides memory by four for a generally small loss of precision. Binary quantization, which keeps only the sign of each component, reaches a factor of 32 and allows comparison by Hamming distance; in exchange it requires a reranking pass over the shortlisted candidates using full-precision vectors to recover acceptable recall.
A way to decide
Assemble an evaluation set of a few hundred queries with their expected documents. Compute exact recall@k by exhaustive search — that is your ceiling. Then compare the configurations you are considering — dimension, quantization, index parameters — on that same set, and keep the cheapest one whose recall stays inside your loss budget.
That budget is a product decision, not a technical one. For a RAG pipeline that retrieves twenty passages before reranking, a few points of recall@20 are often invisible downstream. For deduplication, where you are looking for a single near-identical neighbour, the tolerance is far tighter.