Jyotishko Biswas, HP, AI Leader – HP Finance
The advent of transformers and large language models (LLMs) has vastly improved the accuracy, relevance and speed-to-market of AI applications. As the core technology behind LLMs, transformers enable LLMs to predict and generate the next word (to be specific tokens) by learning from vast datasets containing trillions of words. This results in significant improvements in precision, suitability and coherence. However, LLMs still have gaps, which is where retrieval-augmented generation (RAG) becomes essential.
How RAG Counters The Gaps In Transformers
Transformers are limited by the data they are trained on. For example, if trained on web data up to 2022, they cannot answer questions about events in 2024. Additionally, transformers can generate non-factual answers, known as hallucinations, which undermine their reliability.
RAG is a technique in which an LLM is connected to an external, updateable database. It addresses the data gaps in transformers by providing domain-specific and up-to-date information to LLMs, enabling them to answer questions about recent events and significantly reducing hallucinations.
Limitations Of Standard Retrieval Augmented Generation
Despite its benefits, standard RAG has its limitations, which are outlined below.
1. It retrieves additional information even when the prompt is simple, and retrieval is not required, leading to higher compute and memory costs.
2. The retrieved information can be irrelevant, risking the quality of the LLM’s output, with no checks for relevance.
3. Only a certain number of top documents are used, leaving out potentially useful information.
4. Similarity checks between the prompt and retrieved documents are often insufficient; the utility of the retrieved documents is more critical.
5. Extracting relevant information from vector databases has limitations.
6. Leakage of private and sensitive information in LLM outputs remains a concern.
Advancements In RAG That Help Overcome These Limitations
Multiple technological advancements have come in the past two to three years to overcome the challenges in standard RAG.
Self-RAG is one of these advancements. It addresses issues with retrieval necessity, relevance of retrieved documents and LLM output quality. It includes a critique LLM that determines if retrieval is needed based on the prompt. For simple prompts, like “What is the capital of the U.S.?”, retrieval might not be necessary.
The critique LLM also evaluates the relevance of retrieved documents, retaining only those that are pertinent. This ensures the main LLM uses relevant information, resulting in more accurate and coherent output.
Also, unlike standard RAG, which retrieves information once per prompt, Self-RAG can perform multiple retrievals per prompt, ensuring more relevant information is provided.
Another advanced RAG approach is called MetRAG (“a Multi–layEred Thoughts enhanced Retrieval-Augmented Generation framework”), which addresses key RAG challenges. This method uses an additional LLM to evaluate the utility of retrieved documents rather than relying solely on similarity.
For example, consider the prompt “Tell me about famous football player Cristiano Ronaldo.” Document D1 states, “Cristiano Ronaldo is a famous football player,” while Document D2 states, “Cristiano Ronaldo is a Portuguese professional footballer, plays as a forward and captains both Saudi Pro League club Al Nassr and the Portugal national team.”
Checking for similarity might rank D1 higher, but D2 contains more useful information. This shows that similarity doesn’t always retrieve the most useful information; hence, document utility is used to identify relevance.
Additionally, in MetRAG, another LLM summarizes the retrieved documents, preventing information loss by ensuring that relevant details from non-top documents are retained. This is unlike standard RAG, where only the top retrieved documents are retained and the rest are discarded. This approach of summarizing all retrieved documents results in a more accurate and comprehensive final output.
Another advanced RAG approach uses knowledge graphs instead of vector databases to store external information. While vector databases struggle with complex, multi-relational data, knowledge graphs excel by storing information as entities and their relationships. For example, in the sentence “Argentina won the 2022 FIFA World Cup,” “Argentina” and “2022 FIFA World Cup” are entities, and “won” is the relationship.
Storing external information in a knowledge graph instead of a vector database allows RAG to retrieve more relevant information. This leads to the main LLM producing more accurate, relevant and coherent outputs.
Protecting Sensitive Data
One significant challenge with LLM models is the potential leakage of sensitive information or their use in training models. To address this, RAG can be enhanced by including checks to identify if retrieved documents contain sensitive and private information.
Another LLM, trained specifically to recognize sensitive and personal data, can be employed to review the retrieved information. If a document contains sensitive information, it is either excluded or anonymized/pseudonymized.
Preventing sensitive information leakage is critical in the healthcare industry, which handles highly sensitive patient data, as discussed in this article from Suresh Martha: “The pharmaceutical industry is experiencing a significant transformation, driven by the integration of artificial intelligence (AI) and more current generative AI (GenAI) into aspects of drug discovery, clinical trials, and patient care. While these advancements promise substantial benefits, from accelerating drug development to providing more personalized medical treatments, the GenAI revolution raises ethical considerations regarding data protection, privacy, and the responsible use of technology.”
Limitations Of Advanced RAG Systems
While advanced RAG systems overcome many challenges of standard RAG, they come with their own set of limitations.
1. Extra processing load and increased latency due to the retrieval process may limit RAG’s use in low-latency applications.
2. Lengthy context from adding retrieved documents to the prompt could restrict the use of LLMs with shorter context lengths.
3. Although advanced RAG systems like Self-RAG and knowledge graph-powered RAG reduce irrelevant document retrieval, further improvements are still needed.
Conclusion
Recent technological advancements in RAG have improved various aspects of RAG-based applications while reducing compute and memory costs. Code for implementing many of these techniques is available on GitHub, Hugging Face and other repositories. However, despite these advancements, there are still gaps, and global research is ongoing to fill them.
Forbes Technology Council is an invitation-only community for world-class CIOs, CTOs and technology executives. Do I qualify?
