# Text to embedding

text = "The quick brown fox"

embedding = model.encode(text)

# [0.123, -0.456, 0.789, ..., 0.234] # 384-1536 dimensions

Key Property:

Similar content produces similar vectors. The vector for "The quick brown fox" is closer to "A fast orange fox" than to "Quantum mechanics theory."

Why Vector Databases?

Traditional databases optimize for exact matches:

SELECT * FROM products WHERE name = 'iPhone 15'

Vector databases optimize for similarity:

# Find products most similar to this embedding

results = vector_db.search(query_embedding, top_k=10)

This enables:

• Semantic search (meaning, not keywords)
• Recommendation (similar items)
• RAG (finding relevant context)
• Image similarity
• Anomaly detection

The Technical Challenge

Similarity search is computationally expensive:

Brute Force:

def brute_force_search(query, vectors, k):

distances = [distance(query, v) for v in vectors]

return top_k_by_distance(distances, k)

• Complexity: O(n × d) per query
• For 1M vectors, 768 dimensions: ~768M operations
• Impractical at scale

The Solution:

Approximate Nearest Neighbor (ANN) algorithms trade small accuracy loss for massive speed gains.

How Vector Databases Work

Distance Metrics

Before indexing, we must define "similarity":

Euclidean Distance (L2):

d(a, b) = sqrt(sum((a_i - b_i)^2))

• Actual geometric distance
• Works well for dense vectors
• Scale-sensitive

Cosine Similarity:

sim(a, b) = (a · b) / (||a|| × ||b||)

• Measures angle between vectors
• Scale-invariant
• Common for text embeddings

Dot Product:

dot(a, b) = sum(a_i × b_i)

• Fast computation
• Assumes normalized vectors for similarity
• Common in recommendation

Which to Use:

• Text embeddings: Cosine similarity (most common)
• Normalized embeddings: Dot product (faster)
• Some scientific applications: Euclidean

Indexing Algorithms

The core of vector database performance:

Flat Index (Exact):

• Stores vectors directly
• Brute-force search
• Perfect accuracy
• Only practical for small datasets (<100K)

IVF (Inverted File Index):

Clusters vectors, searches only relevant clusters:

1. Training: K-means clustering on vectors
2. Indexing: Assign each vector to cluster
3. Search: Find nearest clusters, search within them

# Pseudocode

def ivf_search(query, k, nprobe=8):

# Find nearest cluster centroids

nearest_clusters = find_nearest_centroids(query, nprobe)

# Search only within those clusters

candidates = []

for cluster in nearest_clusters:

candidates.extend(cluster.vectors)

# Rank candidates

return top_k(candidates, query, k)

Trade-offs:

• More probes (nprobe): Better accuracy, slower
• Fewer probes: Faster, may miss results
• Works well up to ~10M vectors

HNSW (Hierarchical Navigable Small World):

Builds a multi-layer graph for navigation:

1. Structure: Multiple layers of graphs
2. Upper layers: Sparse, for fast long-range jumps
3. Lower layers: Dense, for local refinement
4. Search: Navigate from top to bottom

# Pseudocode

def hnsw_search(query, k):

# Start at top layer, entry point

current = entry_point

for layer in layers[top:bottom]:

# Greedy search in this layer

while True:

neighbors = get_neighbors(current, layer)

closest = argmin(distance(n, query) for n in neighbors)

if distance(closest, query) >= distance(current, query):

break

current = closest

# Refine at bottom layer

return refine_search(current, query, k)

Characteristics:

• Excellent query performance
• Higher memory usage
• Good for read-heavy workloads
• Industry standard for production

ScaNN (Scalable Nearest Neighbors):

Google's approach combining techniques:

• Learned quantization
• Anisotropic vector quantization
• Very high performance
• More complex to tune

DiskANN:

Microsoft's disk-based approach:

• Stores vectors on SSD
• Memory-efficient
• Good for very large datasets
• Slightly higher latency

Quantization

Reduce memory footprint:

Scalar Quantization:

• Convert float32 to int8
• 4× memory reduction
• Small accuracy loss
• Fast and simple

Product Quantization (PQ):

• Split vector into subvectors
• Quantize each subvector
• 10-50× compression possible
• More accuracy loss

Binary Quantization:

• Convert to binary vectors
• Hamming distance for search
• Massive compression
• Larger accuracy impact

Filtering and Hybrid Search

Real applications need filtering:

Pre-filtering:

results = vector_db.search(

query_embedding,

filter={"category": "electronics", "price": {"$lt": 100}},

top_k=10

)

Challenges:

• Filtering after vector search may return too few results
• Filtering before vector search requires full scan
• Filtering during search is complex

Hybrid Search:

Combine vector and keyword search:

results = db.hybrid_search(

text="comfortable running shoes",

embedding=query_embedding,

alpha=0.7 # 70% semantic, 30% keyword

)

Popular Vector Database Options

Pinecone

Type: Fully managed SaaS

Strengths:

• Easy to use
• No infrastructure management
• Good performance
• Built-in filtering

Limitations:

• SaaS only (vendor lock-in)
• Cost at scale
• Limited control

Best For:

• Teams without infrastructure expertise
• Quick deployment needs
• Moderate scale applications

Pricing:

• Free tier available
• Usage-based pricing
• Can be expensive at scale

Weaviate

Type: Open source, self-hosted or managed

Strengths:

• Feature-rich (modules, hybrid search)
• Active community
• Good documentation
• GraphQL interface

Limitations:

• More complex to operate
• Higher resource usage
• Learning curve

Best For:

• Teams wanting control
• Complex use cases
• Hybrid search requirements

Milvus/Zilliz

Type: Open source (Milvus), managed (Zilliz)

Strengths:

• Very high performance
• Mature and battle-tested
• Rich feature set
• Scalable architecture

Limitations:

• Complex to operate at scale
• Steep learning curve
• Resource intensive

Best For:

• High-performance requirements
• Large-scale deployments
• Technical teams

Qdrant

Type: Open source, self-hosted or managed

Strengths:

• Excellent performance
• Written in Rust
• Good filtering capabilities
• Simple API

Limitations:

• Smaller community
• Fewer integrations
• Newer (less mature)

Best For:

• Performance-critical applications
• Teams comfortable with newer technology
• Filtering-heavy use cases

Chroma

Type: Open source, primarily embedded

Strengths:

• Very simple to start
• Python-native
• Good for prototyping
• LangChain integration

Limitations:

• Not production-scale focused
• Limited clustering support
• Fewer enterprise features

Best For:

• Prototyping
• Small applications
• Learning vector databases

pgvector

Type: PostgreSQL extension

Strengths:

• Use existing PostgreSQL
• ACID transactions
• Familiar SQL interface
• No new infrastructure

Limitations:

• Performance at scale
• Limited ANN options
• Not specialized for vectors

Best For:

• PostgreSQL users
• Smaller datasets
• Simplicity priority

Implementing a Vector Database

Basic Operations

Creating a Collection:

# Pinecone example

import pinecone

pinecone.init(api_key="...")

pinecone.create_index(

name="my-index",

dimension=768,

metric="cosine"

)

Inserting Vectors:

index = pinecone.Index("my-index")

# Upsert vectors with metadata

vectors = [

("id1", [0.1, 0.2, ...], {"text": "doc1", "category": "tech"}),

("id2", [0.3, 0.4, ...], {"text": "doc2", "category": "health"}),

]

index.upsert(vectors)

Querying:

results = index.query(

vector=[0.15, 0.25, ...],

top_k=10,

include_metadata=True,

filter={"category": "tech"}

)

for match in results.matches:

print(f"ID: {match.id}, Score: {match.score}")

Building a RAG System

End-to-End Example:

from openai import OpenAI

import chromadb

# Initialize

openai = OpenAI()

chroma = chromadb.Client()

collection = chroma.create_collection("documents")

def embed(text):

response = openai.embeddings.create(

model="text-embedding-3-small",

input=text

)

return response.data[0].embedding

# Index documents

documents = ["Doc 1 content...", "Doc 2 content...", ...]

for i, doc in enumerate(documents):

collection.add(

ids=[f"doc_{i}"],

embeddings=[embed(doc)],

documents=[doc]

)

# Query function

def query_rag(question):

# Get relevant documents

results = collection.query(

query_embeddings=[embed(question)],

n_results=3

)

context = "\n\n".join(results['documents'][0])

# Generate answer

response = openai.chat.completions.create(

model="gpt-4",

messages=[

{"role": "system", "content": f"Answer based on this context:\n{context}"},

{"role": "user", "content": question}

]

)

return response.choices[0].message.content

Scaling Considerations

Data Size Planning:

Memory per vector ≈ (dimensions × 4 bytes) + metadata

Example: 768 dims, 100 bytes metadata = 3.1 KB per vector

1M vectors ≈ 3.1 GB

10M vectors ≈ 31 GB

100M vectors ≈ 310 GB

Performance Optimization:

• Choose appropriate index type for scale
• Tune index parameters (ef_construction, M for HNSW)
• Consider quantization for memory reduction
• Use batching for bulk operations
• Implement caching for frequent queries

High Availability:

• Replicas for read scaling
• Backup strategies
• Monitoring and alerting
• Disaster recovery planning

Advanced Topics

Multi-tenancy

Serving multiple customers from one database:

Namespace/Partition Approach:

# Separate namespace per tenant

results = index.query(

vector=query_vec,

top_k=10,

namespace="tenant_123"

)

Filter Approach:

# Filter by tenant ID

results = index.query(

vector=query_vec,

top_k=10,

filter={"tenant_id": "tenant_123"}

)

Separate Indexes:

• Best isolation
• More operational overhead
• Best for strict requirements

Embedding Updates

When embeddings change (new model, fine-tuning):

Full Reindex:

• Re-embed all documents
• Replace entire index
• Most straightforward
• Can be expensive

Versioned Indexes:

• Keep old index during transition
• Gradually migrate traffic
• Rollback capability

Hybrid Architectures

Combining vector search with other systems:

Vector + SQL:

-- Vector DB returns IDs

candidate_ids = vector_db.search(query_embedding, k=100)

-- SQL DB filters and ranks

SELECT * FROM products

WHERE id IN (candidate_ids)

AND in_stock = true

AND price < 100

ORDER BY rating DESC

LIMIT 10

Vector + Full-Text:

# Combine scores

vector_results = vector_db.search(query_embedding, k=50)

text_results = elasticsearch.search(query_text, k=50)

# Reciprocal rank fusion

combined = reciprocal_rank_fusion(vector_results, text_results)

Evaluation and Monitoring

Retrieval Quality Metrics:

• Recall@k: Fraction of relevant docs retrieved
• Precision@k: Fraction of retrieved docs that are relevant
• NDCG: Ranking quality measure
• MRR: Mean Reciprocal Rank

System Metrics:

• Query latency (p50, p99)
• Throughput (queries per second)
• Index size
• Memory utilization
• Cache hit rate

Testing:

def evaluate_retrieval(queries, ground_truth, k=10):

recalls = []

for query, relevant_ids in zip(queries, ground_truth):

results = vector_db.search(query, k=k)

retrieved_ids = [r.id for r in results]

recall = len(set(retrieved_ids) & set(relevant_ids)) / len(relevant_ids)

recalls.append(recall)

return sum(recalls) / len(recalls) # Average recall@k