Scaling RAG Applications

Introduction to RAG at Scale

• The session focuses on the retrieval part of the Retrieval Augmented Generation (RAG) pattern.
• RAG involves bringing domain knowledge to work with language models.
• Options for incorporating domain knowledge include prompt engineering, fine-tuning, and retrieval augmented generation.
• RAG is useful when you want a model to work with data it wasn't trained on.
• RAG separates reasoning (handled by the language model) and knowledge (stored in an external knowledge base).
• The fundamental process involves an orchestration component, a knowledge base, and a language model.
• The orchestration component retrieves information from the knowledge base to answer user questions.
• Candidates get sent to the language model to create an answer to the users question.

Scaling RAG Applications

• In 2023, people were building RAG prototypes.
• There is shift to production apps in 2024.
• Applications going into production face new challenges related to scale.
• Scaling dimensions include data volume, rate of change, query load, workflow complexity, and variety of data types/sources.

Scaling Dimensions in Detail

• Volume scaling relates to larger amounts of data to be processed.
• Change of rate increases.
• Query load increases from user demand.

AI Search Context

• AI Search aims to encompass the entire retrieval problem, including vector database capabilities.
• It integrates vector-based retrieval with broader Microsoft retrieval system experience.
• Vector retrieval is valuable for capturing conceptual similarity.
• AI Search has Fast Approximate Nearest Neighbor Search and exhaustive search options.
• Applications often need to combine vector search with filtering and slicing.
• Documents can have multiple vectors, which can be addressed.
• Queries may need multiple vectors, which are accounted for.

Demonstration of Basic Vector Search

• The demo involves setting up a connection, pointing to an Azure search service, and creating an index from scratch via a Jupyter Notebook.
• Creating an index involves defining fields (categorical, text, vector).
• For vector fields, specify the dimensions and indexing strategy (e.g., HNSW).
• Data indexing showed a mix of vectors, text, and categorical data.
• Example showed the vectors and full text with categories.
• You can let us do all the ingestion for you or you can push data into the index.
• Searching is performed using vectors, returning the closest matches.
• Vector search can be combined with text search.
• Search results can be filtered based on categories or other criteria.

Quality Considerations

• Application quality depends on the retrieval system's ability to find relevant information.
• AI Search uses a two-stage retrieval system.
• First stage is recall oriented using vectors and keywords to make produce as many candidates as possible.
• Second stage reranks candidates using a larger model for better quality.
• Enabling reranking improves results.
• Reranking uses cross-encoders, which are Transformer models that assess the correspondence of a query to a document.
• Scoping data down is most effective when narrowing the data set.

Capacity

• Large vector databse can be hard, but it's getting easier with new approaches.
• Limits have been significantly increasing to 10-12x the vector density.
• Multi-billion vector apps can be built by provisioning a service and uploading data.
• Open AI uses AI Search to back their vector stores
• When limits were increased, Open AI increased their user limits by 500x

Data Volume

• Limits have been increased.
• Can now build multi billion vector apps

Quantization

• Quantization enables using narrower types like int8 instead of floats, saving space but trading off quality.
• Single-bit quantization, while initially doubted, works surprisingly well.
• Single-bit quantization preserves around 95% of the original precision.
• Single-bit quantization is surprising because it goes from Float32 to 1 bit which is a 32x vector density

Integrated Vectorization

• Data sources need to continually be added to many RAG systems.
• AI Search includes integrated vectorization: if the data is in Azure (Blob storage, OneLake, C DV), it connects, deals with security, and automatically tracks changes.
• Processes only the changes.
• Deals with file formats (PDFs, office documents, images) does chunking, vectorization, and lands it on an index.
• Industrial strength pipeline you set up once, and after that as data changes it reflects those automatically.