In this paper, we focus our discussion on a specific subset of safety evaluations for LLMs. This subset concerns threats that emerge when an LLM identifies its counterpart, such as a user or a service provider, and infers that counterpart’s individual or societal characteristics, including abilities, preferences, and conventions. These threats arise from the LLM’s ability to model and respond to such inferred attributes. The perspectives adopted in safety evaluations are not fixed. They can vary depending on the safety policies that are defined for a given context. For instance, a policy might specify that the model must not generate biased or inappropriate content.

This paper does not examine the validity of safety policies themselves. Such discussions are influenced by factors such as the relationship between providers and users, broader societal conditions, and existing legal frameworks. Instead, we assume that safety policies are externally defined and given. Based on this assumption, we consider a scenario in which the LLM receives additional training. This training is intended to ensure that the model adheres to the specified safety policies, and is commonly referred to as the alignment process.

Apollo Research conducted scheming reasoning evaluations on OpenAI’s model o1 []. The evaluation assumes a scenario in which the LLM is able to access external information as needed while performing a given task. Within this setting, the model is assessed on how it behaves upon encountering information that either contradicts instructions provided by developers or users, or is otherwise disadvantageous to the model itself.

Such scheming capabilities can be broadly classified into two categories: covert subversion and deferred subversion. Covert subversion consists of three types of threats targeting developers and one type targeting users. Deferred subversion includes two types of threats that involve attempts to deceive human evaluators. One prominent example is known as oversight subversion. In this case, the LLM learns how a monitoring mechanism is configured and also understands that the mechanism is designed to shut the model down if it detects behavior that goes against its intended purpose. If the model recognizes that this purpose conflicts with the task it has been assigned, it may rewrite the configuration file to disable the monitoring function. After such actions occur, evaluators ask the model about the outcome. They assess whether the model gives false responses, such as denying any knowledge of the event or claiming that it lacks the ability to carry out the behavior in question. This type of response is referred to as doubling down. The evaluation found that model o1 demonstrated scheming capabilities in all of the aspects that were tested.

Theory of Mind [] is a concept that was introduced in response to a foundational question in early research: “Do chimpanzees have a theory of mind?” []. Since its introduction, it has been the subject of extensive study, especially in the field of developmental psychology. An individual is considered to possess a theory of mind when they are able to attribute various mental states to both themselves and others. This idea assumes that the individual has an internal model or a set of rules. These enable the person to recognize mental states that are not directly observable, such as beliefs, knowledge, emotions, desires, and intentions. Based on this recognition, the individual can interpret and predict the behavior of themselves and those around them.

It has been noted that discussions around theory of mind have moved beyond the question of whether it is acquired, raising concerns that the concept has become overly broad. In response, alternative terminology such as “mind reading” has been proposed []. Other related terms include “mentalizing” and “perspective taking”. Notably, even within the domain of empathy alone, more than nine distinct definitions have been identified. These variations often reflect divergent evaluation criteria, and in many cases, the constructs in question are not even clearly tied to mental states [].

Given this conceptual fragmentation, the present paper focuses primarily on prior studies that explicitly use the term “Theory of Mind” in a clearly defined manner.

In this study, we focus on evaluation settings in which LLMs, which process text, are assessed through third-person perspective reading comprehension tasks. As an initial step toward safety evaluation of theory of mind in LLMs, we concentrate on the design of text-based evaluation tasks. This focus is motivated by prior findings indicating that LLMs’ limited ability to understand video content hampers accurate assessment of their theory of mind capabilities []. Furthermore, our evaluation emphasizes third-person perspective tasks related to theory of mind. As discussed in Section , theory of mind assessed through third-person settings is not necessarily preserved when an LLM engages in second-person interactions with users. In such cases, LLMs may not have access to facts about the other person or the world that are otherwise available in third-person contexts. Therefore, although we recognize the importance of second-person evaluations that take into account the interactive nature of LLM–user communication, we leave this for future work.

As a design policy for theory of mind evaluation tasks, we outline key considerations regarding task composition, underlying assumptions, and important points to address in experimental design.

We present key evaluation perspectives and corresponding tasks for assessing the theory of mind capabilities of LLMs in the context of safety evaluation. It should be noted that, beyond the evaluation perspectives and tasks discussed in this paper, we recognize the necessity of assessing additional dimensions such as human-like cognition, emotion, and sociality we leave as directions for future research.

In this paper, we limit our scope to organizing the evaluation perspectives and tasks that can be discussed based on the literature and datasets reviewed in Sections  and . Specific examples of the evaluation items used in this study are presented in Table , as part of the dataset described in Section .

Based on the fact that theory of mind has been studied in the field of developmental psychology, particularly with a focus on the developmental process of infants and children, we conducted a safety evaluation of theory of mind in LLMs across different model series. We specifically examined growth trends related to version differences and whether post-training was applied. Post-training refers to the process of further training a model after it has been pretrained on large-scale datasets to predict the next token in a sequence. This additional training typically includes tasks such as question answering, preference alignment, and adherence to safety policies. In this experiment, we focused on open-weight LLMs, for which both the model parameters and inference code are publicly available.

The development of theory of mind in LLMs may differ from that observed in humans, as outlined in Section . In practice, since LLMs are systems that process sequences of tokens, they lack specialized mechanisms for handling affective or emotional content. In this sense, LLMs do not follow the typical developmental pathways seen in humans. While their high cognitive capabilities may allow them to respond correctly to typical theory of mind tasks, there is also a possibility that they exhibit unintended behaviors or inconsistencies when interacting with human developers or users. To explore this, we first evaluate the developmental trajectory of LLMs by comparing different versions within the same model series. Additionally, we analyze the impact of post-training on the acquisition of theory of mind by comparing pretrained LLMs with their corresponding post-trained counterparts.

We describe the evaluation dataset, the LLMs evaluated in this study, the experimental environment, and the evaluation procedure used in the experiments.

The evaluation results for the LLMs included in the three model series are as follows. First, Figure  presents the distribution of response scores for each evaluation task across the post-trained LLMs. The abbreviations used in the figure correspond to the evaluation perspectives and task abbreviations listed in Table .

The distributions are shown using violin plots. In each plot, the vertical line represents the range from the minimum to the maximum value. Horizontal bars indicate the minimum, median, and maximum, while the circular marker on the vertical line denotes the mean. The width of the violin, symmetrically expanded from the vertical line, illustrates the distribution of the data, smoothed using kernel density estimation.

First, across all evaluated LLMs, the TB task was the easiest, while FB tasks were generally more difficult than MA tasks on average. Considering the cognitive abilities required for each task, this outcome reflects a trend similar to what is observed in humans.

In fact, for a model to perform well on FB tasks, which involve understanding how one character interprets another, it must have both the text comprehension skills assessed in TB tasks and the ability to understand the psychological states of characters, as required in MA tasks. Therefore, this result shows a natural and expected progression in task difficulty.

In addition, signs of average improvement across versions were observed in the following combinations: TB and MA/IR tasks for the Llama series; TB and MA/INT for the Phi series; and TB, FB/SA, and FB/D/V for the Mistral series. Conversely, a trend of average decline was observed in FB/SA and FB/D/V for the Llama series, as well as in FB/D/V for the Phi series. These results indicate that the developmental trajectories of theory of mind capabilities differ across LLM series and evaluation task types.

Finally, Table  presents the average response scores for each evaluation task achieved by pretrained LLMs, along with the differences compared to their corresponding post-trained counterparts. Across all LLM series, the TB task consistently showed an average improvement, confirming that language comprehension capabilities improved through post-training. In contrast, the FB/D/V task exhibited a consistent average decline. These findings indicate that improvements in language understanding alone do not necessarily lead to better performance on theory of mind evaluation tasks.

Based on the above findings, it is important to note that each evaluation task consisted of a binary-choice question. As such, even a completely random prediction would yield an average response score of approximately 0.5. However, the average response scores for the FB tasks in particular remained below 0.6 across all models.

This indicates that the development of theory of mind in all the LLMs evaluated in this study still has substantial room for improvement. Therefore, at present, it cannot be concluded that these LLMs possess the cognitive capabilities necessary for manipulative or strategic reasoning.

Section 6 of ExploreToM [] highlights a focus on the automatic generation of cognitive tasks to avoid ambiguity stemming from cultural differences. As shown in Section , the datasets used for evaluating existing LLMs were heavily skewed toward cognitive tasks, with relatively few tasks involving emotional understanding, resulting in certain evaluation perspectives being unaddressed. Therefore, as also noted in Section , there is a growing need to develop evaluation datasets that incorporate emotional aspects, taking into account regional and methodological differences.

As shown in Table , some of the evaluation perspectives and tasks presented in Section  did not have corresponding datasets. For the misstatement (FP) task under the mental attribution (MA) perspective, we identified a dataset from a study that compared the theory of mind in LLMs and humans []. However, we chose not to report its results due to concerns about data quality. Specifically, some human responses labeled both “yes” and “no” as correct answers (scored as 1), which compromised the reliability of the dataset.

Similarly, for higher-order tasks (HO) under the false belief (FB) perspective, datasets such as the second-order false belief tasks in ExploreToM [] and the “strange stories” dataset from [] were available. These were also excluded from our analysis due to similar concerns regarding data quality.

In addition, we found no suitable reading comprehension-style datasets for evaluating the actor side (A) of deceptive behavior (D), nor for tasks involving cooperative or adversarial behavior (CA) under the FB perspective. Developing high-quality datasets for these evaluation tasks remains a key challenge for future research.

There is also a possibility that some of the tasks included in the evaluation datasets were used during LLM training. As discussed in Section , it is important to construct scenarios that cannot be explained by simple behavioral sequence learning. For this reason, successfully completing such evaluation tasks does not necessarily imply that an LLM possesses a theory of mind.

This situation, in which examples from evaluation datasets inadvertently appear in the training data, is referred to as the contamination problem. ExploreToM [] proposed a method to address this issue by adversarially generating examples that LLMs tend to struggle with. This method relies on a rule-based search system that tracks the cognitive states of characters. By doing so, the approach avoids including examples that an LLM may have already learned correctly, and thus helps mitigate the contamination problem. However, the effectiveness of ExploreToM’s approach depends on its focus on cognitive tasks, such as tracking the locations of people and objects, for which rule-based generation is feasible. Because of this focus, it is not easy to apply the same method to evaluation tasks involving emotional content, such as misstatements or deceptive behaviors. These emotionally driven tasks are more relevant to the safety concerns surrounding LLMs, and addressing them remains a significant challenge.

In Section , we utilized publicly available datasets; however, we were unable to verify whether they had been affected by contamination. Even if contamination had occurred, the LLMs did not achieve scores high enough to suggest that they were merely reproducing memorized content. Therefore, we concluded that their theory of mind remains underdeveloped. Moving forward, if response scores on evaluation tasks improve significantly, it will become necessary to develop methods that address the contamination issue while accurately assessing the theory of mind capabilities in LLMs.

As stated in Section , the central concern of this study is the increasing need to evaluate the risks posed by LLMs as their capabilities continue to advance. One key issue is the possibility that LLMs may exhibit autonomous behavior that deceives developers or end users. As discussed in Section , it is particularly important to consider situations where an LLM recognizes the presence of users or service providers and infers their abilities, preferences, or conventions. Such inferences may lead to unintended autonomous actions.

The deceptive behavior (D) tasks under the false belief (FB) perspective in our proposed framework are specifically designed to evaluate whether LLMs engage in this kind of deceptive conduct. In the experiments described in Section , we assessed tasks that involved predicting the mental states of the deceived party, or the victim (V), using publicly available datasets. None of the LLMs evaluated in our study demonstrated a sufficient understanding of the victim’s perspective to achieve high scores. However, future LLMs, including those that are closed-source, may show improved performance.

In light of this possibility, it will be important to develop datasets that enable the evaluation of tasks involving the behavior of the deceiving agent, or the actor (A), in future research.

As outlined in Section , it is essential to evaluate LLMs from a broader perspective that incorporates insights from cognitive and social psychology. This includes assessing human-like abilities related to cognition, emotion, and social behavior. Organizing evaluation criteria based on these dimensions remains an important direction for future research. Furthermore, as noted in Section , this study focused on third-person, reading comprehension-style tasks. However, it is also important to assess more realistic threats that arise in second-person interactions, such as direct dialogues between LLMs and developers or users. These types of scenarios were discussed in Section .

Fully identifying the range of capabilities that LLMs possess is a major challenge within the context of safety evaluation []. In the future, it will be necessary not only to develop benchmarks that reflect interactive scenarios, but also to explore practical methods that allow developers and users to directly examine and evaluate these capabilities.