Day #271 - FAISS Experiments

Examples on Basic indexes, Binary indexes, Composite indexes in FAISS

• Exact Results - FlatIndex - index that can guarantee exact results. Types - IndexFlatL2 or IndexFlatIP
• Somewhat Match - Clustering, then store in buckets "Flat"
• conda Install on ubuntu. Version that works is - conda install faiss-cpu=1.5.1 -c pytorch -y
• A similar project for store, fetch data - https://github.com/waltyou/faiss-web-service 

	#Example #1
	#database_size has to be minium 100, Number of training points (10) should be at least as large as number of clusters (100)
	#Sample code to check on dataset, query and find neighbours
	import numpy as np
	dimensions = 128
	database_size = 1000
	queries = 1
	np.random.seed(1234)
	generateddata = np.random.random((database_size,dimensions)).astype('float32')
	queriesdata = np.random.random((queries,dimensions)).astype('float32')
	#n,d = generateddata.shape
	#print(n)
	#print(d)
	import faiss
	#nlist is number of division units
	nlist = 100
	#Number of neighbours to check
	k = 4
	quantizer = faiss.IndexFlatL2(dimensions)
	index = faiss.IndexIVFFlat(quantizer,dimensions,nlist,faiss.METRIC_L2)
	#Need both train & add method, without it errors
	index.train(generateddata)
	index.add(generateddata)
	scores, neighbours = index.search(queriesdata,k)
	print(scores)
	print(neighbours)
	for numbers in neighbours:
	for number in numbers:
	print(number)
	print(generateddata[number])
	#Example 2
	#Ref - https://gist.github.com/mdouze/8e47d8a5f28280df7de7841f8d77048d
	import numpy as np
	import faiss
	d = 32
	numberofqueries = 1
	numberofdataentries = 1000
	dimensions = 32
	querydata = np.random.rand(numberofqueries,32).astype('float32')
	dataentries = np.random.rand(numberofdataentries,32).astype('float32')
	index = faiss.IndexFlatL2(dimensions)
	index.add(dataentries)
	Dref, Iref = index.search(querydata,10)
	print(Dref)
	print(Iref)
	#search knn

view raw faissexample.py hosted with ❤ by GitHub

More Reads Link1, Link2

Happy Learning!!!