Anthropic AI recently published two breakthrough research papers that provide surprising insights into how an AI model “thinks.” One of the papers follows Anthropic’s earlier research that linked human-understandable concepts with LLMs’ internal pathways to understand how model outputs are generated. The second paper reveals how Anthropic’s Claude 3.5 Haiku model handled simple tasks associated with ten model behaviors.
These two research papers have provided valuable information on how AI models work — not by any means a complete understanding, but at least a glimpse. Let’s dig into what we can learn from that glimpse, including some possibly minor but still important concerns about AI safety.
Looking ‘Under The Hood’ Of An LLM
LLMs such as Claude aren’t programmed like traditional computers. Instead, they are trained with massive amounts of data. This process creates AI models that behave like black boxes, which obscures how they can produce insightful information on almost any subject. However, black-box AI isn’t an architectural choice; it is simply a result of how this complex and nonlinear technology operates.
Complex neural networks within an LLM use billions of interconnected nodes to transform data into useful information. These networks contain vast internal processes with billions of parameters, connections and computational pathways. Each parameter interacts non-linearly with other parameters, creating immense complexities that are almost impossible to understand or unravel. According to Anthropic, “This means that we don’t understand how models do most of the things they do.”
Anthropic follows a two-step approach to LLM research. First, it identifies features, which are interpretable building blocks that the model uses in its computations. Second, it describes the internal processes, or circuits, by which features interact to produce model outputs. Because of the model’s complexity, Anthropic’s new research could illuminate only a fraction of the LLM’s inner workings. But what was revealed about these models seemed more like science fiction than real science.
What We Know About How Claude 3.5 Works
One of Anthropic’s groundbreaking research papers carried the title of “On the Biology of a Large Language Model.” The paper examined how the scientists used attribution graphs to internally trace how the Claude 3.5 Haiku language model transformed inputs into outputs. Researchers were surprised by some results. Here are a few of their interesting discoveries:
- Multi-Step Reasoning — Claude 3.5 Haiku was able to complete some complex reasoning tasks internally without showing any intermediate steps that contributed to the output. Researchers were surprised to find out that the model could create intermediate reasoning steps “in its head.” Claude likely used a more sophisticated internal process than previously thought. Red flag: This raises some concerns because of the model’s lack of transparency. Biased or flawed logic could open the door for a model to intentionally obscure its motives or actions.
- Planning for Text Generation — Before creating text such as poetry, the model used structural elements of the text to create a list of rhyming word in advance, then used that list to construct the next lines. Researchers were surprised to discover that the model used that amount of forward planning, which in some respects is human-like. Research showed it chose words like “rabbit” beforehand because they rhymed with later phrases such as “grab it.” Red flag: This is impressive, but it is possible that a model could use sophisticated planning capability to create deceptive content.
- Chain-of-Thought Reasoning — The model’s stated chain-of-thought reasoning steps did not necessarily reflect its actual decision-making processes as revealed by research. It was shown that sometimes Claude performed reasoning steps internally but didn’t reveal them. As an example, research found that the model silently determined that “Dallas is in Texas” before actually stating that Austin was the state capital. This suggests that explanations for reasoning could potentially be fabricated after an answer has been determined, or that the model might intentionally conceal its reasoning from the user. Anthropic previously published deeper research into this subject in a paper entitled “Reasoning Models Don’t Always Say What They Think.” Red flag: This discrepancy opens the door for intentional deception and misleading information. It is not dangerous for a model to reason internally, because humans do that, too. The problem here is that the external explanation doesn’t match the model’s internal “thoughts.” That could be intentional or just a function of its processing. Nevertheless, it erodes trust and hinders accountability.
We Need More Research Into LLMs’ Internal Workings And Security
Scientists who conducted the research for “On the Biology of a Large Language Model” concede that Claude 3.5 Haiku exhibits some concealed operations and goals not evident in its outputs. The attribution graphs revealed a number of hidden issues. These discoveries underscore the complexity of the model’s internal behavior and highlight the importance of continued efforts to make models more transparent and aligned with human expectations. It is likely these issues also appear in other similar LLMs.
With respect to my red flags noted above, it should be mentioned that Anthropic continually updates its Responsible Scaling Policy, which has been in effect since September 2023. Anthropic has made a commitment not to train or deploy models capable of causing catastrophic harm unless safety and security measures have been implemented that keep risks within acceptable limits. Anthropic has also stated that all of its models meet the ASL Deployment and Security Standards, which provide a baseline level of safe deployment and model security.
As LLMs have grown larger and more powerful, deployment has spread to critical applications in areas such as healthcare, finance and defense. The increase in model complexity and wider deployment has also increased pressure to achieve a better understanding of how AI works. It is critical to ensure that AI models produce fair, trustworthy, unbiased and safe outcomes.
Research is important for our understanding of LLMs, not only to improve and more fully utilize AI, but also to expose potentially dangerous processes. The Anthropic scientists have examined just a small portion of this model’s complexity and hidden capabilities. This research reinforces the need for more study of AI’s internal operations and security.
In my view, it is unfortunate that our complete understanding of LLMs has taken a back seat to the market’s preference for AI’s high performance outcomes and usefulness. We need to thoroughly understand how LLMs work to ensure safety guardrails are adequate.
Moor Insights & Strategy provides or has provided paid services to technology companies, like all tech industry research and analyst firms. These services include research, analysis, advising, consulting, benchmarking, acquisition matchmaking and video and speaking sponsorships. Moor Insights & Strategy does not have paid business relationships with any company mentioned in this article.