Vector database - Research Paper Read

Vector database

Key Summary

• Support for high dimensionality and sparsity
• Describe rich data such as texts, images and video in various domains such as recommender systems, similarity search, and chatbots.

• Vector data is in geospatial applications 
• Two dimensional points such as the location of the end-user and points-of-interest may be represented as vectors
• High-dimensional vectors can be used to represent more complex data such as text, image, audio and video features

• VDBMSs typically support similarity search through indexing methods that enable rapid and accurate searching of similar vectors
• Search for vectors that closely resemble a given query vector based on specific distance metrics such as Euclidean distance or cosine similarity.

• In natural language processing, words and phrases are vectorized into vectors in such a way that similar words have similar vector representations.
• Word2vec [7], FastText, and Doc2vec [8] are examples of techniques that create vector embeddings for words in natural language
• From a developer perspective, queries in VDBMSs are more closely related to simple document or keyvalue store queries than to complex queries in relational databases
• Vectors are retrieved using one or several query vectors

Use-cases

• Similarity search in general
• Image and video similarity search
• Voice recognition
• Chatbots and long-term memory

Current challenges

• Balancing between speed and accuracy
• Growing dimensionality and sparsity
• Information security

Keep Exploring!!!

Vector Databases Reads

Milvus Notes - Index/consistency / availability options

#1. Index type - usecase

• IVF_FLAT - High-speed query
• IVF_PQ - Very high-speed query
• HNSW - High-speed query

Inverted File (IVF): An IVF index divides the vector space into several clusters and holds an inverted file for each cluster, recording which vectors belong to the cluster.

IVF Flat: This is a combination of IVF and flat index. It uses the IVF index to partition the data into clusters and then uses the flat index (brute-force search) within each cluster.

Hierarchical Navigable Small World (HNSW): HNSW builds a multi-layer navigation graph to represent the vector space.

#2. Consistency levels - Strong, Bounded, Session or Eventually

• Strong - Most strict
• Bounded staleness - allows data inconsistency during a certain period of time.
• Session - Like dirty reads 
• Eventually - weakest level among the four.

#3. HA - In-memory replicas help Milvus recover faster if a query node crashes.

#4. Vector search & Hybrid Search params offset, limit

offset - Number of results to skip in the returned set

limit - Number of the most similar results to return

How indexing and querying works

• Trees – ANNOY -  Annoy (Approximate Nearest Neighbors Oh Yeah)
• Proximity graphs - HNSW Hierarchical Navigable Small World (HNSW) Graphs
• Clustering - FAISS
• Hashing -  LSH - Locality-Sensitive Hashing (LSH)
• Vector compression - PQ or SCANN. - ScaNN (Scalable Nearest Neighbors). Product Quantization (PQ): PQ index compresses vectors into compact codes and is beneficial for large-scale, high-dimensional data.

More Reads

• Utilizing Few-shot and Zero-shot learning with OpenAI embeddings
• Query Comparison
• Accelerating Similarity Search on Really Big Data with Vector Indexing

Keep Exploring!!!

Milvus, an open-source, cloud-native vector database - Key Architecture Notes

Milvus, an open-source, cloud-native vector database - Key Architecture Notes

• Milvus takes a ground-breaking approach and introduces a publish-subscribe (pub/sub) system for log storage and persistency
• Milvus offers two ways of deployment - standalone or cluster
• Access layer acts as the system's face, exposing the endpoint of the client connection to the outside world
• Coordinator service responsible for cluster topology node management, load balancing, timestamp generation, data declaration, and data management. 
• Worker, or execution, node executing instructions issued by the coordinator service and the data manipulation language (DML) commands initiated by the proxy.

Storage is the cornerstone of Milvus, responsible for data persistence. The storage layer is divided into three parts:

• Meta store: Responsible for storing snapshots of metadata such as collection schema, node status
• Log broker: A pub/sub system that supports playback and is responsible for streaming data persistence, reliable asynchronous query execution
• Object storage: Stores snapshot files of logs, scalar/vector index files, and intermediate query processing results.

Log broker, logs are decoupled from the server, ensuring that Milvus is itself stateless and better positioned to quickly recover from system failure.

Ref - Link

Vector Search 

• Knowhere not only further extends the functions of Faiss but also optimizes the performance
• Built on top of FAISS, Annoy, HNSW

Ref - Link

Consistency Discussions

• GuaranteeTs is configurable in the search request to achieve the level of consistency specified by you. A GuaranteeTs with a large value ensures strong consistency at the cost of a high search latency.

Ref - Link 

Query Mechanism

• Before a query is executed, the data has to be loaded to the query nodes first.
• There are two types of data that are loaded to query node: streaming data from log broker, and historical data from object storage (also called persistent storage below).

Ref - Link 

Milvus - Applications

• Video media: video understanding, video deduplication.
• E-commerce and mobile applications: image understanding, reverse image search.
• Finance/Telecommunications/Retail: AI-aided customer support, QA chatbots.
• Internet: personalized recommender systems, personalized search.
• Autonomous vehicles: automated data labeling and annotation, object detection.
• Biopharmaceutical: virtual compound screening, compound retrosynthetic analysis, protein property prediction, and DNA testing.
• Cybersecurity: malware detection and cyberattack alert.
• Quantitative trading: data analysis and prediction.
• Metaverse: environmental perception and interaction in the virtual world.

Ref - Link

Hnswlib - fast approximate nearest neighbor search

Distance parameter Equation

Squared L2 'l2' d = sum((Ai-Bi)^2)

Inner product ' ip' d = 1.0 - sum(Ai*Bi)

Cosine similarity 'cosine' d = 1.0 - sum(Ai*Bi) / sqrt(sum(Ai*Ai) * sum(Bi*Bi))

• hnswlib uses hnswlib to pre-calculate approximate nearest neighbors.

Do you actually need a vector database?

Vector databases are having their day right now. Vector databases are able to calculate similarity quickly because they have already pre-calculated it

A Gentle Introduction to Vector Databases

• Storing and searching across table-based data such as the one shown above is exactly what relational databases were designed to do.

• vector databases are used for: searching across images, video, text, audio, and other forms of unstructured data via their content rather than keywords or tags (which are often input manually by users or curators). When combined with powerful machine learning models, vector databases have the capability of revolutionizing semantic search and recommendation systems.

Qdrant 

Benchmarking Vector Search Engines

• Qdrant and Milvus are the fastest engines when it comes to indexing time.
• Qdrant achives highest RPS and lowest latencies in almost all scenarios, no matter the precision threshold and the metric we choose.
• Elasticsearch is typically way slower than all the competitors, no matter the dataset and metric.

Towhee is an open-source machine learning pipeline. that helps you encode your unstructured data into embeddings.

Replacing my best friends with an LLM trained on 500,000 group chat messages

Ref - Link

Keep Exploring!!!