Vector Databases Reads

Milvus Notes - Index/consistency / availability options

#1. Index type - usecase

• IVF_FLAT - High-speed query
• IVF_PQ - Very high-speed query
• HNSW - High-speed query

Inverted File (IVF): An IVF index divides the vector space into several clusters and holds an inverted file for each cluster, recording which vectors belong to the cluster.

IVF Flat: This is a combination of IVF and flat index. It uses the IVF index to partition the data into clusters and then uses the flat index (brute-force search) within each cluster.

Hierarchical Navigable Small World (HNSW): HNSW builds a multi-layer navigation graph to represent the vector space.

#2. Consistency levels - Strong, Bounded, Session or Eventually

• Strong - Most strict
• Bounded staleness - allows data inconsistency during a certain period of time.
• Session - Like dirty reads 
• Eventually - weakest level among the four.

#3. HA - In-memory replicas help Milvus recover faster if a query node crashes.

#4. Vector search & Hybrid Search params offset, limit

offset - Number of results to skip in the returned set

limit - Number of the most similar results to return

How indexing and querying works

• Trees – ANNOY -  Annoy (Approximate Nearest Neighbors Oh Yeah)
• Proximity graphs - HNSW Hierarchical Navigable Small World (HNSW) Graphs
• Clustering - FAISS
• Hashing -  LSH - Locality-Sensitive Hashing (LSH)
• Vector compression - PQ or SCANN. - ScaNN (Scalable Nearest Neighbors). Product Quantization (PQ): PQ index compresses vectors into compact codes and is beneficial for large-scale, high-dimensional data.

More Reads

• Utilizing Few-shot and Zero-shot learning with OpenAI embeddings
• Query Comparison
• Accelerating Similarity Search on Really Big Data with Vector Indexing

Keep Exploring!!!