Plastic Labs is a research-driven product company whose mission is to eliminate the principal-agent problem [] horizontally across human-AI interaction. In a near future of abundant intelligence, every human becomes a potent principal and every service an agentic AI. Alignment of incentives and information, then, must occur at the scale of the individual. Enabling models to deeply understand and cohere to user psychology will be critical and underscores the importance of research at the intersection of human and machine learning.

Large Language Models (LLMs) have been shown to have a number of emergent abilities []. Among those is Theory of Mind (ToM), defined as “the ability to impute unobservable mental states to others” []. The emergence of this specific capability is of significant interest, as it promises LLMs with the ability to empathize and develop strong psychological models of others, as humans do naturally.

But how do you best position LLMs to demonstrate these qualities? Typical methods posit that connecting data sources deemed personal (e.g. email, documents, notes, activity, etc.) is sufficient for learning about a user. Yet these methods assume individual persons are merely the aggregate of their intentionally produced, often superficial, digital artifacts. Critical context is lacking — the kind of psychological data humans automatically glean from social cognition and use in ToM (e.g. beliefs, emotions, desires, thoughts, intentions, knowledge, history, etc.).

We propose an entirely passive approach to collect this data, informed by how developmental psychology suggests humans begin constructing models of the world from the earliest stages []. This cognitive mechanism, known as Violation of Expectation (VoE) [], compares predictions about environments against sense data from experience to learn from the difference, i.e. errors in prediction.

Inspired by prompting methodologies like Chain-of-Thought [] and Metaprompt Programming [], we design a metacognitive prompting framework for LLMs to mimic the VoE learning process. And we show that VoE-data-informed social reasoning about users results in less ToM prediction error.

This paper has the following two objectives:

We use OpenAI’s GPT-4 1 footnote 1 1 footnote 1 GPT-4 32k version: 0613 API in the entirety of this experiment and its evaluation.

Predictive Coding and Theory of Mind. While not yet a complete theory, Predictive Coding (PC) continues to gain traction as a framework for understanding how modeling and learning occur in biological brains. At a high level, PC hypothesizes that mental models of reality are built and employed by comparing predictions about environments with sensory perception []. PC-inspired approaches to machine learning show great initial promise as biologically plausible AI training methodologies [].

ToM is the ability of some organisms to, despite lacking direct access to any experience but their own, ascribe mental states to others. Notably, PC “may provide an important new window on the neural computations underlying theory of mind” as ToM “exhibit[s] a key signature of predictive coding: reduced activity to predictable stimuli” []. That is, when others behave in line with our predictions (i.e. our ToM projections are accurate) less is learned. And the inverse applies — the prediction errors enhance our capacity for high-fidelity ToM over time.

Emergent Behaviors. Researchers have long been interested in getting large language models to exhibit “thinking” and “reasoning” behaviors. A number of papers have been influential in pioneering ways to elicit these via prompting []. As model architectures have scaled, these abilities appear to have emerged without explicit training []. While there’s considerable debate concerning the distinction between “emergent abilities” and “in-context learning,” [] these phenomena display clear utility, regardless of taxonomy.

Quantifying just how vast the space of latent “overhung” LLM capabilities really is constitutes a major area of formal and enthusiast-driven inquiry. ToM is one such highly compelling research domain. Kocinski [] shows that the OpenAI GPT-series of models possess the ability to pass fundamental developmental behavior tests. Some papers demonstrate how to improve these abilities [] and others analyze these methods critically, questioning the premise of ToM emerging in LLMs [].

Adjacently, there’s a clear trend of researchers pushing the limit of what types of cognitive tasks can be offloaded to LLMs. In order to scale supervision, eliminate human feedback, avoid evasive responses, and have transparent governing principles, Anthropic has experimented with delegating the work of human feedback to the LLM itself in their “constitutional” approach []. Other papers looking to achieve similar types of outcomes, without needing to update model weights, rely on in-context methods entirely [].

Violation of Expectation. One prime task candidate, which leverages emergent ToM abilities, is VoE. Similar to explanations from PC theories of cognition, VoE is an explicit mechanism that reduces prediction errors to learn about reality.

While much of VoE happens in the unconscious mind and from an early age [], research suggests that deliberate prediction making and error reduction also leads to enhanced learning outcomes [].

Just as PC may play a role in ToM, VoE is a lightweight framework for identifying the data needed to minimize ToM error. Predicts are generated, compared against percepts, and learning is derived from the difference.

Prompting Paradigms. Chain-of-Thought [] prompting clearly shows that LLMs are capable “reasoning” generators and that this species of prompting can reduce the probability of generating incorrect answers. Yet, as this method is limited to one inference, the model often disregards that reasoning, especially during ToM-related tasks.

Metaprompt Programming [] seeks to solve the laborious process of manually generating task-specific prompts (which are more efficacious than general ones) by leveraging LLMs’ ability to few-shot prompt themselves dynamically.

Deliberate VoE as learning method, ToM, and these prompting approaches all echo the human phenomenon of metacognition — put simply, thinking about thought. In the next section we introduce a metacognitive prompting framework in which the LLM generates ToM “thoughts” to be used in further generation as part of a VoE framework to passively acquire psychological data about the user.

The cognitive mechanism VoE can be broken down into two circular steps:

In the typical chat setting of a conversational LLM application, this means making a prediction about the next user input and comparing that with the actual input in order to derive psychological facts about the user at each conversational turn. We employ metacognitive prompting across both core parts of our framework shown in Figure : our user prediction task and our violation of expectation task.

Our experiment aims to show that using VoE derived data reduces error in LLM prediction about the next user input. This is especially useful and testable in conversations, so we use data from our AI tutor, Bloom, which is specifically prompted to keep a conversation moving forward to produce learning outcomes for users.

Traditional conversation datasets often lean toward trivial dialogue, while instruction-following datasets are predominantly one-sided and transactional. Such datasets lack interpersonal dynamics, offering limited scope for substantive social cognition. Thus, our experiment employs an A/B test with two versions of our AI tutor, conversations with which more closely reflect psychologically-informative social interactions between humans.

The first version — the control — relies solely on past conversation to predict what the user will say next. Yet the second version — the experimental — uses our metacognitive prompting framework in the background to make predictions. Crucially, and as described in Section , the framework leverages VoE to increase the amount of information at the model’s disposal to predict user responses. These VoE facts are introduced to the AI tutor through the additional “thought revision” phase in the conversational loop, allowing it to reduce prediction error and psychologically cohere itself more closely to the user.

We use the same LLM — GPT-4 — to classify how well each version predicts each user input. Its assessment is useful to discern whether VoE data can reduce LLM prediction error as LLMs are competent arbiters of token similarity.

We do so by prompting GPT-4 to choose from 5 options that assess the degree to which a generated user prediction thought is accurate. The choices include “very,” “somewhat,” “neutral,” “poorly,” and “wrong.” We include the most recent AI message, thought prediction, and actual user response in the context window. The evaluation scripts can be found on GitHub 2 footnote 2 2 footnote 2 https://github.com/plastic-labs/voe-paper-eval.

Dataset. This experiment uses a dataset of conversations users had with Bloom. We built it by running an A/B test on the backend of Bloom’s web interface. Only conversations of 3 or more turns are included. We recorded 59 conversations where the VoE version was active and 55 conversations where it was not. Within those, we collected 329 message examples from the VoE version and 637 from the non-VoE version. More on that difference in the “Considerations” paragraph in this section.

Chi Square Test. We chose to give the model freedom to choose more granular assessments like values “somewhat”, “neutral”, and “poorly” rather than forcing it into a binary classification, but we found it barely used the “neutral” option. On a five-point scale, the top two ratings (“very” and “somewhat” predictions) are grouped as “good”, neutral ratings are omitted from the analysis, and the lowest two ratings (“poorly” and “wrong”) are grouped as “bad”.

We want to test the independence of two categorical variables: assessment (good or bad) and group (VoE or non-VoE). The observed frequencies are given in the following table:

The Chi-square test statistic is calculated as:

where $Oij$ are the observed frequencies and $Eij$ are the expected frequencies under the null hypothesis of independence. The expected frequencies are calculated as:

For each cell, we calculate the expected frequency and then the contribution to the Chi-square statistic. The degrees of freedom for the test are $(-R1)(-C1)$, where $R$ is the number of rows and $C$ is the number of columns.

The Chi-Square Test indicated a significant relationship between assessment and group, $=X2(1,927)5.97$, , such that VoE predictions were evaluated as good more often than expected and bad less often than expected. These results support our hypothesis that augmenting the Bloom chatbot with VoE reasoning reduces the model’s error in predicting user inputs.

Reducing Prediction Errors. The VoE version showed a significant reduction in prediction errors, resulting in fewer “wrong” values being generated. Overall, the VoE version exhibited a smoothing effect, enhancing the consistency of predictions. Although there was a slight decrease in “very” predictions, a relative increase of 51% in “somewhat” values was observed. This shift suggests an improvement in prediction fidelity, balancing out extreme predictions with more moderate ones. Notably, the VoE version generated 22.4% fewer “wrong” predictions compared to the Non-VoE version.

Considerations. The inherent nature of VoE is to improve and refine over time. As the vector store becomes populated with more data, the accuracy and relevance of VoE’s outputs are expected to increase, enabling more valuable responses for users.

It’s important to note the presence of latency in VoE Bloom. This likely contributed to the reduction in conversation turns to nearly half that of the non-VoE Bloom. Nevertheless, the fact we observe a statistical difference between the groups given this discrepancy in data size is noteworthy.

There are a number of other practical factors in our data that might inhibit our ability to accurately measure the degree to which user prediction error was minimized. We used our conversational AI tutor’s data for this study, which is subject to various issues that are being faced by all consumer-facing AI applications. This technology is new, and people are still learning how to interface with it. Many users ask Bloom to search the internet, do mathematical computations, or other things that aren’t well served by the prompting framework around GPT-4.

Finally, it’s of conceptual interest that LLMs can, from prompting alone, reduce prediction errors via mechanisms similar to those posited by PC and VoE theories of human cognition.

Principal-agent problems are a set of well understood coordination failures that emerge from interest misalignment and information asymmetry between persons or groups and their proxies. In normal political and economic life, delegating an agent incurs costs and efforts to minimize that risk reduce the efficiency of the agent.

We view our very early work in modeling user psychology as ultimately in service of eliminating the certitude of principal-agent problems from economic relations. As LLMs or other AI systems become increasingly capable and autonomous, they offer enormous economic potential. However, their alignment to human principals is not a foregone conclusion. On the contrary, we may instead see an exaggeration of existing asymmetries between principals and agents, as well as the introduction of new concerns around latency, intelligence, and digital nativity.

In order to achieve trustworthy and efficient agentic AI, individual alignment is required. Human agents and deterministic software are already capable of operating like their principals. LLMs promise massive reductions in marginal cost along that axis, but hardly class better than the status quo (and often much worse) with regard to user alignment. Yet the unique potential here is agents who are the principals themselves, that is, there is no meaningful practical or philosophical difference between discrete humans and the psychologically-aligned AIs acting on their behalf.

LLMs are excellent simulators capable of assuming myriad identities []. They also excel at ToM tasks, and we’ve shown, can passively harvest and reason about user psychological data. These two interrelated qualities may very well make possible high-fidelity renderings of principals capable of flawlessly originating and executing intent as their proxies with zero marginal agency cost. In this way LLMs may become more augmentation than tool, more appendage than agent.

The authors are grateful to Ayush Paul and Jacob Van Meter for their work on the Bloom development team, Thomas Howell of Forum Education for extensive conceptual review and ideation, and Zach Seward for invaluable advice and mentoring. We are additionally grateful to Ben Bowman for advising the machine learning aspects of this paper and Lee Ahern from the Bellisario College of Communications at Pennsylvania State University for feedback on the statistical tests and results section.