Negation is an important construct in language for reasoning over the truth of propositions ( ) , garnering interest from philosophy ( ) , psycholinguistics ( ) , and natural language processing (NLP) ( ) . While transformer language models (TLMs) ( ) have achieved impressive performance across many NLP tasks, a great deal of recent work has found that they do not process negation well, and often make predictions that would be trivially false in the eyes of a human ( ) .

In developmental psychology, there has likewise been a great deal of interest in how a child’s ability to comprehend negation emerges in the early years of life ( ) . Unlike in NLP, which typically treats negation as representing a single monolithic competency, this research has long understood that there are many kinds of negation used in everyday interactions ( ) . This ranges from using negation to express a child’s rejection of something to clarifying a child’s knowledge. These “developmental” categories of negation do not emerge simultaneously; children tend to start using certain kinds before others ( ) .

Given that these categories represent some of the earliest uses of negation among humans, understanding how well TLMs can master them is important for building more human-like models of language processing. Understanding how well models perform on different categories will indicate whether they have mastery of some forms of negation, while also helping to identify failure points. Another interesting question is whether the proficiency of TLMs on these categories is at all related to competencies in human children (e.g., is the category which models consistently perform the best on the same that children most frequently employ?). However, to our knowledge, no prior work in NLP has focused on how well models perform on the forms of negation of interest to developmental psychology.

In this short paper, we investigate how well a suite of TLMs can process developmental negation, 2 footnote 2 2 footnote 2 By which we mean the forms of negation studied in development psychology. by framing the problem as a natural language inference (NLI) task. We develop a rule-based parser to extract problems from existing NLI datasets, and evaluate our models on each category, in order to determine (i) whether certain categories are more solvable by our models than others, and (ii) what relationships exist among the categories. We find that models can consistently achieve stronger performance only on certain categories, and that training on combinations or sequences of these categories does not substantially improve a model’s downstream performance.

Negation is known to be frequently used in everyday conversation. While this includes its logical form, we primarily focus on negation’s psycholinguistic forms, especially those that have been studied in the context of developmental psychology. Negation emerges early in child development, with ‘no’ sometimes being a child’s first word ( ) , and even infants appear to understand forms of negation ( ) . Preschool children use at least three different kinds of negation ( ) , but possibly as many as nine ( ) . As noted by ( ) , one of the first categories children use is rejection, where a child rejects an object or activity. This is later followed by existence, where a child might express the lack of an object, and later still denial, which a child uses to deny the truth of a claim. Larger scale studies of child-directed speech have found that truth-functional kinds of negation tend to emerge later ( ) , but individual children do vary in their specific order of acquisition ( ) . It is unknown whether this ordering reflects any deeper dependencies among the different categories, or whether the ordering is reflected in how artificial language models (LMs) learn negation.

In NLP, methods from psycholinguistics have been used to probe the reasoning capabilities of LMs. Results from some studies have indicated that TLMs are not human-like in their processing of negation ( ) . A similar line of work has used the NLI task to probe a model’s ability to process negation and found that TLMs will often alter their predictions when negation is inserted or removed, even when the negation does not alter the entailment relationship ( ) . As argued by ( ) , part of the challenge of modeling purely logical negation is that a predicate often occurs in very similar contexts regardless of whether it is being negated. They argue that we should view negation as being a “graded similarity function”, and show that distributional models can predict human plausibility judgments quite well, even in the presence of negation. These works show that it is unclear how well distributional models, especially TLMs, are actually processing negation. We contribute to this literature from a new perspective, by studying how well models can reason over forms of negation common in developmental psychology.

We use the NLI task to study the negation reasoning capabilities of our models. NLI problems consist of two sentences: a premise ($p$) and hypothesis ($h$), and solving such a problem involves assessing whether $p$ textually entails $h$. The generic structure of the NLI task makes it suitable for studying a variety of underlying reasoning skills, including negation. We specifically use the SNLI ( ) and MNLI ( ) datasets.

To automatically identify questions that contain a specific kind of negation, we rely on the work by ( ) which studied how frequently different kinds of developmental negation occur in child-directed speech, using the data from the CHILDES corpus ( ) . To do this, they created a simple rule-based parser to automatically tag each sentence in CHILDES with the type of negation it contained (if any). We re-implement their parser, in some cases tweaking the rules slightly to better suit the structure of the NLI task. For each example across all the splits of both datasets, we first obtain a dependency parse of both $p$ and $h$ using the diaparser package ( ) , and check if either contains an explicit negation marker (“no”, “not”, or “n’t”). If one span contains negation, we check if the syntactic structure obeys the rules of any of our categories. If the span falls into a category, we mark it as belonging to that category. We use these questions as the diagnostic set for our experiments, splitting out 1/3 of the questions in each category as a diagnostic test set, and leaving the remainder as a diagnostic train set (and we will refer to them as such). We place the remaining NLI questions containing no negation in a separate $NLItrain$ set, giving us about 730,000 examples we use to finetune our models on the NLI task. We split out 9,000 questions from this train set at random to use as a $NLIdev$ set, balanced for each label. In the following, we describe the precise rules used to determine which category a negated example should be assigned to:

Using the curated dataset, we performed a series of exploratory experiments to help us understand how well TLMs process each of the negation categories. We use BERT ( ) , and RoBERTa ( ) , two popular transformer LMs that have demonstrated impressive results on a variety of language understanding tasks. We also examine MiniBERTa ( ) and BabyBERTa ( ) , which are both based on the RoBERTa architecture but were pre-trained on a much smaller number of tokens (10 million and 5 million respectively), which is more realistic to the amount of language a child is exposed to in the first few years of life. We use the Huggingface implementation of all models ( ) , and use both the base and large version of BERT and RoBERTa, which differ only in the number of trainable parameters.

In this paper, we have explored how well transformers process categories of developmental negation. We find that performance rankings across categories are generally consistent, but that the categories seem to test for orthogonal skills in the majority of LMs. In BabyBERTa, we see significant similarities with the order of negation acquisition in children. Two of the best performing categories are $R$ and $L$, while two of the worst are $EX$ and $PR$, which aligns quite well to the order observed by ( ) . It thus seems that TLMs do at least partially reflect the order of negation acquisition observed in children, although more experiments would be needed to understand the extent of this correlation. That we found category rankings to generally be consistent across LMs may have interesting implications, and understanding why LMs struggle with certain categories may help to improve the ability of LMs to process negation.

Future work can build on these experiments in several ways. In Experiments 2 and 3, we modeled interactions among the negation categories in either a pairwise or sequential fashion, which is unlikely to reflect how children are exposed to negation. More experiments, mixing all of the categories at once in various proportions, might yield a more realistic model of cognitive development. Our approach also requires that each category fits into a specific structure, which limits the amount of examples that can be extracted. Future work will need to expand our ruleset to include more variations in the negated utterances covered. Finally, while we primarily focus on finetuning, pre-training is likely to impact the proficiency of our models on the categories as well. Future work should precisely control the prevalence of each category in the pre-training corpus, to observe what effect this has on downstream performance.