The last few blogs have introduced the relationships between storage devices and AI workflows; now it is time to focus in on what I consider to be the most exciting form of AI: generative AI. Generative AI is where most end users will interact with AI systems directly. Generative AI uses the large language models (LLMs) built through training to create new content that can be textual, audial, or visual. Generative AI can boost productivity and synthesize answers to complex questions using the tremendous amount of information contained in the training datasets.
The Limits of Large Language Models
The large language models have become very capable; however, they have inherent limitations. They are created at great expense utilizing clusters of tens of thousands of GPUs analyzing the massive amounts of public data contained on the internet and take months of calculations. Therin lies the two biggest limitations of the large language models: they do not have access to private data, and due to the length of time it takes to create them, the data they were trained on may not be up to date. When a large language model provides answers based on incomplete or old data the results are inaccurate, and this condition is euphemistically termed a hallucination. Retrieval augmented generation is a technique that can reduce or eliminate hallucinations, resulting in more accurate and timely results.
How RAG Grounds LLMs in Reality
Retrieval augmented generation works by processing private data and data created after the large language model was trained to build a database of relevant context that may be fed to a large language model. This allows solutions to be built that can leverage the power of large language models on private datasets enabling domain or enterprise specific knowledge and even answers based on real time data. The process of adding supplemental context to reduce hallucinations is termed grounding.
At a high level the process of creating a grounded retrieval augmented generation solution consists of breaking up the private or new data into small pieces termed embeddings and then creating multidimensional vectors to describe the attributes of each of these embeddings. Often the vectors describing each of the embeddings may have hundreds of dimensions, and this high dimensionality leads to a challenge in finding the most relevant embeddings (based on their attributes) to supply to the large language model as context for a user’s query. For large datasets it is computationally impractical to search the entire vector space to find the best context for a given query.
Enter Approximate Nearest Neighbor Search
To enable a more practical search of the vectors describing the embeddings, many vector databases implement what is called an approximate nearest neighbor search (ANNS) algorithm. Instead of considering all of the high dimensionality vectors that are near to the query, a graph-based index of the vector space is created (this process is well suited to GPUs) and a subset of the nearest neighbor vectors is returned by traversal of the graph. It is important to note that while this technique is not guaranteed to return the true nearest neighbor vectors, the trade off in reduced computational complexity has been shown to outweigh the loss of accuracy in large datasets.
HNSW: Powerful but DRAM-Intensive
The most common software used by vector databases for retrieval augmented generation is the hierarchical navigable small worlds (HNSW) package. HNSW uses DRAM to store its vectors and graph-based index. Whereas this provides relatively good performance, the DRAM requirements of HNSW pose a challenge for scalability. If you were to try to implement a 100 billion high-dimensionality vector database in HNSW, you would require over 100 terabytes of DRAM, far beyond the capabilities of today’s servers.
DiskANN: A Scalable Alternative
Microsoft® research created a solution to the scalability problems of HNSW with DiskANN, an open-source project that uses SSDs to store the index as well as the full precision vectors of the vector database, and greatly reduces DRAM requirements by using DRAM only for product quantized vectors; these are much smaller than the full precision vectors but also less accurate.
The immediate reaction most people have to the description of DiskANN is “isn’t that going to be both slower and less accurate than HNSW?” Surprisingly the answer is that at high levels of recall accuracy DiskANN can actually outperform HNSW! This seems counterintuitive; however, Microsoft employed clever techniques to accelerate the search. The first is the use of product quantized vectors, these allow the use of much simpler calculations and fundamentally DiskANN needs to perform less computational work than HNSW. The tradeoff for the reduction in calculation is the need for disk accesses and a loss of precision. With SSDs, the reduced calculations can offset the cost of a disk access at higher levels of recall, but what about the accuracy loss caused by the product quantized vectors? Again, a clever technique was used to overcome this: recall that DiskANN stores the full precision vectors on disk. Although the lower precision quantized vectors are used to determine where to look on disk, the full precision vectors are retrieved and used to re-rank the results and improve accuracy.
With DiskANN Microsoft enables far greater scalability of vector databases; compared to HNSW the DRAM requirements for a 100 billion high-dimensionality vector database would be reduced from over 100 terabytes to about 10 terabytes. This is far more practical and far less expensive, all while achieving comparable accuracy and even better performance at high levels of recall. The techniques utilized by DiskANN are in production in Microsoft’s Azure CosmosDBTM vector database service, and in my next blog I will outline how KIOXIA has taken the concept of DiskANN one step further!
Microsoft is a registered trademark of the Microsoft Corporation.
AZURE COSMOS DB is a trademark of Microsoft Corporation.
All other company names, product names and service names may be trademarks of their respective companies.