ALGOTECHNIC
All articles
8 min read

Vector Search as Application Infrastructure: Embeddings, ANN Indexes, Filters, and Operations

Embeddings become application infrastructure only when index design, metadata filters, retrieval quality and operational ownership are engineered as one system.

  • Vector search
  • Embeddings
  • Search infrastructure
Model-produced vectors flow through an approximate index and metadata controls into operated search results

A vector index does not turn a model into a search service. It sits between two contracts: the model must produce comparable vectors, and the application must receive results that satisfy its relevance and access rules. The index makes proximity cheap enough to query, but it cannot repair a weak representation or decide which documents a user may see.

Treat vector search as application infrastructure, then, rather than as a new database field. The design joins representation, candidate generation, filtering and operations. Each choice changes the others: a new embedding model can invalidate an index; a selective tenant filter can defeat an approximate query; a memory-saving code can alter recall.

Start with the representation contract

An embedding model maps an input to a fixed-dimensional vector, trained so that a chosen distance or similarity function places useful neighbours close together. Sentence-BERT illustrates the mechanism for text: siamese or triplet training and pooling produce sentence vectors that can be compared with cosine similarity, avoiding a pairwise transformer evaluation for every possible match.

The stored numbers have no meaning without that model context. Record the document and query encoder versions, preprocessing, pooling, normalisation, dimension and metric as one contract. Where the same encoder serves documents and queries, use the same deployed artefact at both stages. Models designed with distinct encoding paths need their corresponding pair instead. A model upgrade creates a new coordinate system; mixing its vectors with those from the old model makes proximity unreliable.

Chunking belongs to this contract too. A passage vector represents the text the encoder actually received, not the whole parent document. Long documents split into passages can improve the match between query and evidence, but they require stable parent identifiers and rules for returning either the passage or its source.

Distance is not relevance. Cosine similarity, inner product and Euclidean distance impose different rankings, and normalisation changes how some of them relate. Even the exact nearest neighbour under the correct metric can be off-topic, stale or too broad. Exact product names and rare identifiers may also favour lexical search. OpenSearch 2.9 supports combining BM25 and neural retrieval, allowing an application to cover both behaviours rather than forcing one ranking signal to do every job.

Choose an index by what it avoids scanning

Exact search compares the query with every stored vector. It returns the true nearest neighbours under the configured metric, needs no graph construction or quantiser training, and provides the baseline against which approximate results should be measured. Its predictable work also makes it suitable for modest collections and small filtered subsets. That work grows with the collection, so exactness is not a scalable default.

Approximate nearest-neighbour (ANN) indexes reduce the candidates or the cost of comparing them. FAISS supplies several distinct mechanisms rather than one universal index:

  • Inverted file (IVF) partitions vectors around coarse centroids. A query examines only selected inverted lists. Probing more lists usually recovers more neighbours while scanning more data. A neighbour assigned to an unvisited list cannot be returned.
  • Product quantisation (PQ) splits vectors into subvectors and replaces each with a short code. Approximate distances can then be computed from compact codes and lookup tables. IVF and PQ can combine candidate pruning with compression, but their errors are separate design concerns.
  • Hierarchical Navigable Small World (HNSW) builds layered proximity graphs. Sparse upper layers make long moves towards the query region; denser lower layers refine the search. Its candidate-list width controls exploration at query time, while graph connectivity affects memory and construction cost.

These mechanisms expose operating points, not fixed speed rankings. For HNSW, increasing the search candidate list generally buys recall with extra distance computations. For IVF, the equivalent control is how many lists to probe. PQ shortens storage but approximates the vectors themselves. Hardware, dimension, data distribution, update pattern and concurrency can all change the result.

Measure recall@k against an exact top-k result set on representative queries, then measure application relevance separately. High ANN recall means the index found what the metric preferred; it says nothing about whether those neighbours helped the user. Record tail latency, resident memory and index-build time at the same operating point. A lower latency obtained by losing difficult queries is not a free optimisation.

Documents and queries share a versioned embedding contract before filtered retrieval, with quality and resource telemetry around the index
The retrieval controller chooses exact, efficiently filtered ANN or post-filtered search. Model compatibility and measured quality surround every path.

Filtering changes the search problem

Metadata filtering is not a cosmetic clause around nearest-neighbour search. It determines which candidates are eligible and can change the best retrieval strategy.

Post-filtering first finds approximate neighbours across the collection, then removes ineligible results. A restrictive predicate can leave far fewer than the requested k results because eligible documents may never enter the candidate set. Requesting a larger k can reduce that underflow, but it adds work and still provides no completeness guarantee.

Pre-filtering restricts the document set before an exact distance scan. It can answer selective queries precisely, yet a large eligible subset makes the brute-force cost substantial. Applying metadata constraints during ANN traversal offers another path, but its behaviour depends on the engine and algorithm; filtering should not be mistaken for a property that every HNSW implementation shares.

OpenSearch 2.9 makes these distinctions unusually clear. Its efficient filtering supports FAISS with HNSW, alongside the existing Lucene support. The engine considers the collection size, filtered subset size and requested k, then chooses exact search on a pre-filtered subset or approximate search with modified post-filtering. When enough documents match, efficient filtering aims to return all k requested results. Ordinary Boolean and post_filter forms remain post-filters, while a scoring-script filter performs exact search after pre-filtering. These are different query plans, not interchangeable syntax.

Test filters by selectivity and by important combinations, not only with an unfiltered query set. Tenant and authorisation predicates belong in the engine-enforced query, with document metadata kept consistent as permissions change. Filtering later in a presentation layer can expose ineligible records to application code and corrupt result counts and pagination.

OpenSearch joins model serving to retrieval

OpenSearch 2.9 declares neural search production-ready and its ML framework generally available. An ingest pipeline can invoke a text-embedding model and write its output to a mapped vector field; a neural query supplies query text and a model identifier for the corresponding transformation. Model access control governs who can use registered models. This integration removes application glue, but it also makes model deployment part of the search cluster’s operational boundary.

The same release adds FAISS filtering and Lucene byte-sized vectors. The latter store one byte per dimension instead of four, reducing memory and storage with a possible accuracy cost. That option belongs to Lucene, not to every FAISS index, and differs from PQ’s learned compact codes.

For native-library engines such as FAISS and NMSLIB, OpenSearch builds vector indexes for each vector-field and Lucene-segment pair and loads them into native memory under cache management. Capacity planning must therefore include more than the source documents. A raw float-vector estimate of vectors × dimensions × 4 bytes excludes graph links, codes, metadata and replicas.

Give quality and capacity an owner

Operational ownership starts with compatibility. Store the embedding-contract version with the index and make it observable at query time. For a model change, build and backfill a separate index, evaluate it, then switch reads while retaining the previous model and index for rollback. Rolling back only the query encoder leaves the document vectors incompatible.

Freshness needs its own measures: embedding failures, documents awaiting indexing, deletion lag and the age of the newest searchable update. A healthy cluster can still serve stale vectors. Index rebuilds also need resource headroom so that training or backfilling does not displace the live query workload.

Monitor service and retrieval quality together. End-to-end latency includes model inference, candidate search, merging shard results and any reranking. Track percentiles under representative concurrency, with cold and warm caches distinguished where relevant. Add recall by query cohort and filter class, native-memory use, cache behaviour, and shard and index health. OpenSearch 2.9’s shard and index monitors help with the last group; they cannot detect a semantic regression by themselves.

Start with exact search as the correctness baseline. Introduce approximation when measured workload demands it, choose filter strategies from real selectivity, and revisit the operating point when the model or data changes. The durable abstraction is not “top-k vectors”; it is a versioned retrieval service with explicit quality and resource budgets.

Sources and further reading