Qdrant, a leading provider of vector database solutions, has recently unveiled an innovative search technology called BM42. This new approach promises to revolutionize information retrieval, especially in AI applications that rely on retrieving data as a part of retrieval-augmented generation pipelines that provide context to the language models.
BM42 is a hybrid search algorithm developed by Qdrant. Traditional search engines rely on keyword matching, which has limitations in terms of understanding context and meaning. Vector search, emerging with AI and machine learning, seeks to grasp the semantic meaning behind words and phrases, offering more nuanced results.
Retrieval-Augmented Generation, a method in AI applications for accurate and up-to-date information retrieval, can be significantly enhanced with BM42. This new algorithm’s benefits include improved accuracy, as it combines keyword and contextual search for reliable, relevant information, crucial in fields like medicine or law. Its design for quick results with less computational power reduces costs for AI applications. Additionally, BM42 works with different languages and specialized vocabularies without extensive retraining. It also enhances AI-generated text with more accurate and relevant information.
BM42 combines the precision of traditional keyword search with the intelligence of AI-powered vector search, akin to a super-smart librarian who understands both the exact location and deeper context of your request.
The new algorithm, BM42, uses two main components: sparse vectors handle exact word matching, similar to traditional keyword search, like identifying specific ingredients in a recipe. Dense vectors capture overall meaning and context, similar to how our brains understand language, like grasping the overall flavor profile of a dish. BM42 employs a method called Reciprocal Rank Fusion to blend results from both approaches, ensuring precise matching and contextual understanding.
BM42’s impact is evident in several practical scenarios. For customer support, AI-powered chatbots can find relevant information from knowledge bases, providing faster, more accurate responses. In legal research, lawyers can quickly locate relevant case law and legal documents, understanding both terminology and legal concepts. In medical diagnosis, healthcare systems can search medical literature for studies and case reports matching symptoms and patient context. For content recommendation, streaming services and online retailers can offer more accurate recommendations, understanding user preferences and broader trends.
BM42 represents a significant advancement in search technology. As AI becomes more integral to retrieving and processing data, technologies like BM42 will enhance AI systems’ ability to provide useful and reliable results. Businesses and organizations can expect more efficient operations and better customer service, while individuals will benefit from more accurate AI-powered tools and services.
Qdrant’s new hybrid search algorithm offers several key advantages that help differentiate the company in the competitive vector database and search technology market.
Qdrant claims that BM42 provides an efficient and cost-effective solution compared to existing hybrid search approaches, with the ability to compute Inverse Document Frequency in real-time, allowing for dynamic updates without pre-computation of statistics.
Unlike some competitors, Qdrant’s solution is open-source, allowing for greater transparency and community involvement. By embedding BM42 across its open source, cloud, and hybrid offerings, Qdrant positions itself as a versatile, efficient and forward-thinking option in the vector search market, particularly for organizations looking to implement or improve their RAG and AI applications.
