Decomposition of logographs into sub-units is necessary to locate the sub-units hinting at the phonetic or semantic information. Logographs are decomposed recursively using a rule-based parser. The substitution rules used by the parser are defined by human experts from the Ideographic Rapporteur Group. A substitution rule defines a mapping from one logograph to sub-units and a geometric operator (Ideographic Description Character) which denotes the relative position of the sub-units. The output of the parser is a binary tree as shown in Figure .

At the start, there is only the root node, which is the logograph itself. The parser extends the tree by recursively replacing nodes in the tree with sub-trees defined by the substitution rules. The root of the sub-trees are the geometric operator and the children of the sub-trees are the sub-units. This process is repeated until there is no more node in the tree that can be further decomposed.

Figure  shows how the structure (represented as a binary tree) is constructed for the logograph {CJK*}UTF8bsmi仕. At step 1, there is a root node {CJK*}UTF8bsmi仕 with no children. At step 2, using the rule in red, the node {CJK*}UTF8bsmi仕 is replaced by a geometric operator (horizontal stacking) and two children nodes. At step 3, using the rules in red, the nodes are further simplified into by {CJK*}UTF8bsmi人, {CJK*}UTF8bsmi十 and {CJK*}UTF8bsmi一. The process terminates at step 4 where there are four leaf nodes with three distinct values {CJK*}UTF8bsmi人, {CJK*}UTF8gkai丨, and {CJK*}UTF8bsmi一 which cannot be simplified further. There are 505 sub-units which can be leaf nodes. These sub-units are not hand-picked, thus whether or not the representation of the sub-units carries phonetic or semantic information is automatically learned during training. Hence, the hierarchical embeddings can be used in different tasks. The phonetic and semantic sub-units can be at depth 1 (children of the root node) or they can reside deeper in the trees.

To construct logograph embeddings from trees, one can use bag-of-words models, sequence models, or tree-structured models []. Since the ordering of sub-units within logographs is important in determining the logographs’ pronunciation and meaning, order-agnostic models such as bag-of-words models are sub-optimal for constructing logograph embeddings. Since sequence models and tree-structured models are sensitive to the ordering of sub-units, they can be used to construct logograph embeddings. Sequence models such as recurrent neural networks (RNNs), in particular LSTM [], can be used with tree inputs by first linearizing the trees into sequences. In contrast, recursive neural networks, such as treeLSTM, can consume tree inputs directly to yield the logograph embeddings. We compared LSTM and bi-directional LSTM (biLSTM), which are structure-agnostic, and treeLSTM, which is innate for modeling tree structures.

At each position $t$ in a linearized tree of length $T$, $xt$, $ct$, $ht$ are the input, cell value, and hidden state of the LSTM respectively. The last hidden state, $hT$, is used as the logograph embedding. Figure  shows the LSTM model.

The biLSTM consists of two LSTMs, the forward LSTM and the backward LSTM, which read the linearized trees in opposite direction. The logograph embedding, $hT$, is formed by concatenating the last hidden states of the backward and forward LSTMs, i.e. $hbT$ and $hfT$.

The model structure is similar to that of []. The input to the model are also sequences formed by linearizing the trees of logographs. The CNN model consists of 7 parallel 1D convolutional layers with kernel size from 1 to 7. Each convolutional layer has 200 filters. The convolutional layers are followed by max-pooling layers. After that, the outputs are concatenated and fed through a fully-connected layer. The output of the fully-connected layer is the logograph embedding.

At each node $n$ of the binary tree with two children $l$ and $r$, $xn$, $cn$, $hn$ are the input, cell value, and hidden state of the treeLSTM respectively. $i$, $fl$, $fr$, $o$ are the input gate, left forget gate, right forget gate, and output gate respectively. The forward pass of a treeLSTM unit is given by:

The hidden state of the root node, $hroot$, is considered as the representation of the entire tree. Figure  shows the treeLSTM model.

One problem with tree-structured models is that training is very slow []. It is hard to batch the training samples as they might have different tree shapes []. As a result, training with batch size of one is very common for tree-structured model [], which fails to maximize parallel computation and thus leads to slow training. Instead, we used dynamic batching to speed up training and inference. Dynamic batching in Pytorch 3 3 footnote 3 https://devblogs.nvidia.com/recursive-neural-networks-pytorch/ has been used to create batches of nodes on the fly to speed up the SPINN model [] training and inference. In our experiments, using a batch size of 128 results in more than 10 times faster training and inference. Besides, we only considered binary trees and converted any ternary nodes (nodes with three children) to two nested binary nodes. This is done to reduce the number of parameters that the model has to learn therefore improving the learning efficiency. The tree representation is sensitive to the order of the children nodes as swapping the left child and the right child in a tree results in a character with potentially different meaning and pronunciation. Thus, we need separate weight matrices for each of the children. As such, modeling both binary and ternary nodes would require from 3 to 5 weight matrices whereas modeling binary trees only requires 2 weight matrices. Since the amount of data is limited, we preferred models with fewer parameters and thus we converted all ternary nodes to binary nodes.

The data was extracted from UniHan database 4 4 footnote 4 https://www.unicode.org/charts/unihan.html, which is a pronunciation database of characters of Han logographic languages. Each entry consists of a character and its pronunciations in various languages such as Cantonese and Mandarin. For entry with multiple pronunciations, since the dominant pronunciation is not indicated, we randomly picked one of the variants. For this task, the input is the logographic character and output is the Cantonese pronunciation. The pronunciation includes onset, nucleus, and coda. As far as we know, lexical tones are not directly determined by logographic structures so we did not include lexical tones as prediction targets.

There are two types of logographs used in Cantonese, namely traditional and simplified characters. Simplified characters, as the name implies, are derived from their traditional counterparts by removing or replacing some complex sub-units with simpler ones. Non-simplified characters include both traditional characters and the subset of Chinese characters that are identical for traditional and simplified counterparts. Hence, simplified and traditional Chinese characters are quite different in terms of unique sub-units and their complexity.

A common weakness of deep learning models is that they often merely memorize patterns and do not generalize well on unseen data []. LSTM has the same weakness as it performs well when there is abundant training data and test distribution is the same as the training distribution []. When the test and training distributions are different, LSTM does not perform as well. Strong generalization requires models to extrapolate to out-of-distribution data points rather than to interpolate using data points within distribution [].

To test the generalizability of standard LSTM and treeLSTM, the original UniHan dataset was split into training and test sets in three different scenarios described in Table . In the first scenario, the training and test set’s distribution were homogeneous: both contained traditional and simplified characters. In the second scenario, the test set only contained simplified characters and the training set contained non-simplified characters. In the third scenario, the distributions were different and the training data was limited: the test set contained only simplified characters while the training sets contained corresponding traditional characters.

The third scenario is inspired by the fact that humans being able to predict pronunciations of simplified characters given the corresponding traditional characters, although they may rely on word context. Given that human performance is high, it should not be impossible for models to generalize to simplified characters even when trained solely on traditional characters. By contrasting results obtained from scenario 1 and 2, we could determine whether the models merely memorized patterns or they learned the underlying rules to predict pronunciation as humans, since models that merely memorize patterns would do well in scenario 1 but not scenario 2. In addition, contrasting scenario 2 and 3 would hint at how models perform in low-resource scenarios of limited training data as well as whether the bias induced by the logographic structures is useful for improving model generalization. It should be noted that scenario 1 is the ideal case in which one is very careful in collecting data and performs data normalization. If data is collected indiscriminately, one can end up in scenario 2.

The task-specific layer uses the logograph embedding to predict the logograph’s pronunciation, which includes onset, nucleus and coda. Probability of each sub-syllabic unit’s pronunciation is given by:

where $WCD$, $WNU$ and $WON$ are weights of the fully-connected layer specific to each sub-syllabic unit.

The setup for treeLSTM to predict a logograph’s pronunciation using hierarchical embeddings is shown in Figure .

We evaluated models’ performance using string error rate (SER) and token error rate (TER). A wrongly predicted phoneme (onset, nucleus or coda) was counted as one token error. An output containing at least one token error was counted as one string error. We used modified Obuchowski statistical test [] to assess the difference in predictive differences.

The size of hidden layers is fixed as 256. We used dropout [] on input and hidden layers to prevent overfitting. We optimized the models using the Adam [] optimizer. The batch size was 128. For each of the model, we searched for the best learning rates and dropout rates using grid-search. The learning rate ranges from $\times 310-2$ to $\times 110-4$. The drop out rate ranges from 0.0 to 0.5.

Since there are multiple ways to linearize trees into sequences, in this section, we investigated what is the optimal linearization order for the models. We compared three different schemes namely: in-order, pre-order, post-order linearization. We paired each of the models (there are five models in total) with the 3 different linearization schemes. This resulted in fifteen different combinations. For each combination, we conducted hyperparameter search on the development set. The lowest TER for each of the combination is reported in Table .

For all the models, the difference in performance between different linearization schemese is quite small. However, across all models, the pre-order linearization is slightly better than the post-order and the in-order linearization. Hence, for the subsequent experiments, we use pre-order linearization to convert from trees to sequences.

Table  shows the prediction results by LSTM, biLSTM, and treeLSTM for three experimental scenarios listed in Table . In scenario 1 and 2, biLSTM performed slightly worse than LSTM so we only compared LSTM against treeLSTM. In scenario 1 where the training and test distributions were the same, treeLSTM yields 1.8% $(=p2e-4)$ and 2.0% $(=p6e-5)$ lower absolute TER (5.4% and 6.0% relative TER) than 1-layer and 2-layer LSTM respectively. treeLSTM also yields 1.6% $(=p0.06)$ and 0.6% $(=p0.4)$ lower absolute SER (2.7% and 1.0% relative SER) than 1-layer and 2-layer LSTM respectively. The trends are similar when individual output units (i.e., onset, nucleus, coda) are considered. This result is unlikely due to treeLSTM having a higher capacity since the 2-layer LSTM had more parameters than treeLSTM.

When training and test distributions are different (scenario 2), models that have better inductive bias should perform better []. For example, the convolution operation in convolutional neural network (CNN) has translation equivariant bias []. This bias enforces that the representation of an object is the same regardless of its position in an image. This bias makes CNN generalize much better and require few training samples than fully-connected neural networks. For logographs, the inductive bias is that the interaction between sub-units is local in space. This inductive bias is enforced in the treeLSTM model since a child node only interacts with its sibling. The result is that the hierarchical embeddings is much more data-efficient than the LSTM. The result shown in Table  indicates that treeLSTM can generalize better than LSTM models even when the test set has out-of-distribution samples. treeLSTM yields 4.7% and 2.0% $(=p6e-4)$ lower absolute TER (9.6% and 4.3% relative TER) than 1-layer LSTM and 2-layer LSTM respectively. Besides, treeLSTM yields 3.9% $(=p3e-6)$ and 1.7% $(=p3e-2)$ lower absolute SER (5.3% and 2.3% relative SER) than 1-layer LSTM and 2-layer LSTM respectively. The trends are similar when individual sub-syllabic classes (i.e., onset, nucleus, coda) are considered.

When training and test distributions are different and the amount of training data is limited, good inductive biases are even more important to obtain good generalization. Comparing scenario 2 and 3, treeLSTM is less affected than LSTM by the limited training data. In the limited training data regime, 2-layer LSTM clearly overfits badly compared to 1-layer LSTM and treeLSTM. It is interesting to note that although the CNN model is the most competitive baseline in scenario 3 although it is worse than the LSTM and biLSTM when there is more data (scenario 1 and 2). However, compared to the CNN, the treeLSTM still has lower SER $(=p0.07)$ and TER $(=p0.5)$.

We conducted ablation experiments to see how much the models depend on the composition operators. Without the operators, the LSTM, biLSTM, and CNN cannot discern the hierarchical grouping of sub-units. On the other hand, even without the composition operations, the treeLSTM model still receives some structural information from ordering of the sub-units in a tree.

In order to implement the case where there is no composition operators in the input, for the LSTM, biLSTM, and CNN models, the operators were removed from the input sequences. For the treeLSTM model, all the $V$ terms were removed from the equation of the inner nodes. We searched the best hyperparameters for each of the model using the development set of scenario 1. We picked scenario 1 because it is the most common way to split data into training/validation/test sets, i.e. standard split. The result is shown in Table . It can be seen that the composition operators do provide salient information for the task since taking them out results in worse performance across all models (as reflected by increases in error rates). However, the treeLSTM performance does not degrade by much, it is more certain that treeLSTM learns to compose the sub-units chiefly from the tree structure.

The phonetic subunits in Chinese characters usually predict nucleus and coda more reliably than onset. This trend can be seen in Figure  whereby all the nuclei and codas are the same across the first four characters which share the same phonetic subunit. However, the most effective ordering of input and output in machine learning is may not align with human intuition. For example, reversing the order of the input sentence boosted the performance of machine translation [], while swapping the order of onsets and nuclei in Thai syllables boosted the performance of English-to-Thai transliteration []. We adopted the Coda-Nucleus-Onset prediction order in this paper as shown in Section . However, we also tried using a different prediction order which is Onset-Nucleus-Coda. We replaced the task-specific layer of the proposed model and searched for the optimal hyperparameters. The model with the best hyperparameters is then applied on the test set. Empirically, we observed little difference in performance between the two orders.

As the characters (logographs) in the output of language models are not independent, it is difficult to design meaningful statistical tests to evaluate the effectiveness of our proposed approach. Instead we chose a wide variety of five different datasets, consisting of three datasets using simplified characters (Chinese Penn Treebank (CTB) Version 5.1 [], Beijing University (PKU) dataset [], and Microsoft Research (MSR) dataset []) and two datasets using traditional characters (City University of Hong Kong (CITYU) dataset [] and Academia Sinica (AS) dataset []). If we can show consistent improvements across these datasets, it implies the proposed hierarchical embeddings are effective. Table  shows the data split for each of the datasets. Data splits for CTB and PKU datasets are taken from []. 5 5 footnote 5 https://s3.eu-west-2.amazonaws.com/k-kawakami/seg.zip

We used AWD-LSTM (ASGD Weight-Dropped LSTM) model [] as the core in the language modeling experiment. The input to AWD-LSTM is either hierarchical embeddings (Figure ) or standard character embeddings (Figure ). We considered the standard character embeddings as the baseline. We trained the model using the training set for a fixed number of epochs and used the validation set to select the best model. The best model performance was evaluated on the test set after training finished.

We evaluated models’ performance using perplexity (PPL) and bits-per-character (BPC). BPC is a standard evaluation metric for character-level LMs [].

where $x$ is the whole corpus, $xt$ is the character at position $t$, and $|x|$ is the length of the corpus.

The same hyperparameters are used across the datasets. We optimized the models using the Adam [] optimizer for 300 epochs. The learning rate was set at 0.002 and is divided by 10 after 250 epochs. The size of hidden layer is fixed as 1000. The size of the embedding is fixed as 200. The AWD-LSTM has three hidden layers with sizes 1000, 1000, 200 respectively. We used dropout [] on input and hidden layers to prevent overfitting. Dropout rates were set as 0.1, 0.1, and 0.25 for the input, hidden and output layers of the AWD-LSTM. L2 weight decay was set as $\times 1.210-6$. Weight dropout was set at 0.5. The batch size was 100. To improve computational speed, only embeddings of the characters appearing in the training batch were updated. During testing, the embeddings were constructed once and then cached, hence using hierarchical embedding was nearly as fast as standard embeddings. The caching technique was similar to [].

Table  shows the prediction results. We also report results on the CTB and PKU datasets from [] and results on the MSR dataset from []. The results from [] can be compared with our results since the results are evaluated on the same data splits. However, direct comparison is unfair for [] because our models are bigger than theirs. The results from [] cannot be compared with our results as the data splits are different because their data split is not publicly available. Thus, we reimplemented the glyph embedding for a fairer comparison. The glyph embedding model architecture is similar to that used in the original paper []. We only include the results for the Segmental Neural LM model for reference and did not reimplement this model because it depends on multitask training which is different from the other models. Our result agrees with the conclusion from [] that the glyph embeddings are slightly worse than standard embeddings regardless of the baseline (GRU or AWD-LSTM). The hierarchical embeddings outperformed the standard embeddings in all datasets, regardless of whether the datasets use simplified or traditional characters.

Exploiting recursive structures has been shown to be beneficial in many NLP tasks such as sentiment analysis [], text simplification [], and machine translation []. These models are usually trained using human annotated structures but may be tested on structures annotated automatically using parsers when human annotation is not available. This mismatch in annotation quality could worsen the performance of these models and could partially explain why exploiting structures in NLP tasks have not always led to better results. For example, recursive models [] were not as good as the biLSTM in sentiment analysis task []. To address the mismatch in annotation quality, new models which can both produce and exploit structures have been introduced []. For our case, annotation quality is consistent across the training and test set, thus, better ways of modeling structures led to better results.

In languages like Mandarin, Japanese or Cantonese, logographs are characters and the number of characters are in the range of thousands. In contrast, alphabetic languages usually have far fewer characters (e.g. 26 characters for English). The large number of characters in languages with logographic origin makes character-level modeling inefficient and worsens the problem of out-of-vocabulary words and characters. However, alphabetic languages and languages with logographic origin are often treated the same way, disregarding their intrinsically marked differences []. Modeling logograph sub-units can alleviate these issues since there are fewer sub-units and they can be used to construct out-of-vocabulary words and characters. This is consistent with how learners of languages with logographic origin can comprehend the meaning or pronunciation of a logograph from its constituent sub-units []. Hence, leveraging structures of logographs can be useful in capturing semantic [] or phonological information [].

There are many prior work on building embeddings of logographs. The first approach is to apply convolutional neural network (CNN) on the visual rendering of logographs []. The second approach is to combine sub-unit embeddings with the logograph embeddings. Sub-units embeddings can be learned independently of logograph embeddings [] using Skip-Gram or CBOW models [] or learned jointly with logograph embeddings []. The third approach is to apply CNN or RNN on the sequence of sub-units [].

Our work is most similar to the third approach. However, while our approach exploits the recursive structures of logographs, most work in this area ignores structures or only consider the structures implicitly.

In languages like English, popular models to learn word embeddings assign a distinct vector to each word, ignoring word morphology (how characters, word’s sub-units, form a word). This approach uses solely the context surrounding words to learn the embeddings which may be a limitation in languages with a large vocabulary and many rare words since the context may be insufficient to learn good embeddings. Building logographic (character) embeddings in languages of logographic origin has the same difficulty since there a lot of logographs (characters) and many of them are rare characters.

To incorporate morphology into word embeddings learning, [] proposed building word embeddings by averaging bags of character n-grams. This method may be agnostic to the order of characters if the n-gram length is short. Others have used RNN [] or CNN [] to better incorporate word morphology information into words embeddings. Unlike English words which are linear sequences of characters, logographs are recursive structures of of sub-units. Hence, using models operating on sequences such as RNN or LSTM may not be optimal.

Rare word/character embeddings can be improved by leveraging similarity in morphology between rare words and common words. [] proposed building embeddings of new words from pre-trained embeddings by learning mapping from characters to embeddings. However, in this line of approach, the embeddings are fixed, which may not be useful for tasks that require information not captured in the embeddings pre-trained via unsupervised language modeling. In work from [], the embeddings are learned jointly with the task models so that the embeddings contain useful information for the task. Our hierarchical embeddings can be trained on task-specific data, making it potentially useful for many different tasks.

More than 80% of frequently used Han logographs are semantic-phonetic compounds []. These compounds consist of sub-units that might contain phonetic or semantic information []. Pronunciation of these compounds could conceivably be predicted from the phonetic sub-units. Amongst semantic-phonetic compounds, logographs with the left-right arrangement (in which the semantic sub-unit is on the left and the phonetic sub-unit is on the right) are the most common. For logographs with the left-right arrangement, a good model for logograph’s pronunciation should prefer the right child (the likely phonetic sub-unit) of a root node for making pronunciation prediction. To check whether the hierarchical embeddings prefer the left child or the right child, we compared the norm of the left forget gate against the norm of the right forget gate. The right child is preferred if the norm of the right forget gate is larger.

In Table , the second column shows the number of logographs following the left-right arrangement for different scenarios. The third column shows the number of logographs following the left-right arrangement in which the right child is preferred over the left child. The hierarchical embeddings prefer the right child most of the time (close to 100%) in all three scenarios. Thus, the learned hierarchical embeddings consider the right sub-units to be more relevant for pronunciation prediction for the majority of compound logographs with the left-right arrangement. This is consistent with human intuition. Since human depends on this intuition to infer pronunciation and it seems to work well, this suggests that the hierarchical embeddings might have learned a general solution that works well.

By overfitting to common patterns at the expense of more difficult, infrequent samples that require deeper understanding, statistical models can perform well as measured by some aggregate metrics []. A common pattern useful for predicting pronunciation is that phonetic sub-units usually occur at the end of the linearized sequences. A general model would be able to find where the phonetic sub-units are in the sequences. A model that only attends to the end of sequences would make wrong prediction when the phonetic sub-units are not at the end of the sequences.

To determine how the models predict, we visualize the hidden states of LSTM and treeLSTM. The visualization for biLSTM is not shown since it performed worse than LSTM. For both LSTM and treeLSTM, the last hidden state (e.g. $h15$ in Figure ) is considered the logograph embedding. The intermediate embeddings (e.g. $h1$ to $h14$) are embeddings of the subsequence of sub-units for LSTM and embeddings of the subtrees of sub-units for treeLSTM. The hidden states (embeddings) evolve to contain more phonetic information with more sub-units as indicated by generally increasing magnitude of the hidden states (corresponding to darker bands). When the magnitude of the hidden states are small (corresponding to faint bands), the hidden states do not have enough information to predict pronunciation confidently. We also obtained the prediction corresponding to each hidden state by feeding the hidden states ($h1$ to $h15$) to the task-specific layer in order to determine at which step did the embeddings contain phonetic information to make the correct pronunciation prediction.

Figure  shows how the models predict the pronunciation of the logograph {CJK*}UTF8bkai賄 (bribery). This is a common example as the phonetic sub-units are on the right (corresponding to end of the linearized sequence). While both models predict correctly, they used the logograph structural representation differently. LSTM had to observe the whole sequence to predict correctly, as suggested by the build-up in magnitude of the embeddings until the end of the sequence. For treeLSTM, the pattern of the embeddings’ magnitude is consistent with the hierarchical structure of the input logograph with two subtrees {CJK*}UTF8bkai貝 and {CJK*}UTF8bkai有. Specifically, not only was the final pronunciation prediction of {CJK*}UTF8bkai賄 correct (“f ui #”), but pronunciation of the subtrees ({CJK*}UTF8bkai貝 and {CJK*}UTF8bkai有) were also correct (“b ui #” and “j au #” respectively).

Figure  shows a rare example where the phonetic sub-units are not at the end of the sequence. LSTM made the correct prediction after observing the relevant parts (up to the second last input token) but soon forgot the correct prediction as it might focus more on the end of the sequence. This mistake indicates that LSTM might have learned a heuristic instead of the general strategies. On the other hand, treeLSTM predicted the pronunciation correctly by seemingly focusing on the relevant part ({CJK*}UTF8bkai如) of the logograph and ignoring the less relevant tokens. Thus, imposing a prior on the mapping from logographs to embeddings by using recursive network seems to lead to a solution that may generalize better to more challenging cases.

Hierarchical embeddings could learn better representations of infrequent characters than standard embeddings could since the latter ignores the morphology within characters. Using the learned embeddings in the language modeling experiments, we looked for characters that are most similar (nearest neighbors) to the infrequent characters in the embedding space. If the nearest neighbors are semantically or phonologically close then we are more certain that the learned embeddings are sensible. The distance between embedding vectors is calculated using cosine similarity. Table  showed that the infrequent characters and their nearest neighbors are relatively close in meaning when using hierarchical embeddings.

For standard embeddings, infrequent characters and their neighbors are generally unrelated. For example, spider are unrelated to a plant, drawer, or scold. It is possible that with little training data, the infrequent characters’ embedding stay close to the original random initialized values and hence are far away from related characters in embedding space. For hierarchical embeddings, infrequent characters are more related to their neighbors. The relatedness between infrequent characters and their neighbors can be semantic or phonological. For example, the first row in Table  shows characters ({CJK*}UTF8bkai蛛,蛐,蚶,蚜) that share the same sematic sub-units (shown in red). Accordingly, spider is semantically related to cricket, ark clam, and louse since they are all insects. The second row in Table  shows another example in which characters ({CJK*}UTF8bkai砦,柴,紫,雌) have the same phonetic sub-units (shown in blue). Correspondingly, “zh ai #” is phonologically related (having similar pronunciation) to “ch ai #”, “z i #”, and “c i #”.

However, the hierarchical embeddings are not always accurate and the last row of Table  shows an interesting failure. The cosine distance suggests that (celery, “q i n”) and (axe, “f u #”) are related. Although both characters has a common sub-unit, the sub-unit carries phonological information (color-coded as blue) in the case of (celery, “q i n”) while the sub-unit carries semantic information (color-coded as red) in the case of (axe, “f u #”).

The granularity of the input features derived from logographs could have a major impact on model performance. The input features could be as granular as individual strokes, which results in a small vocabulary. Different permutations of the strokes can form unique ideographs and expand the vocabulary. The choice of the vocabulary set has a major impact on sequential models like RNN, as a big vocabulary makes training slow and makes it hard for the model to generalize. On the other hand, a small vocabulary leads to longer sequences and makes it harder for models to learn. [] showed that a big vocabulary yields lower perplexity for language modeling of Japanese, while big vocabulary implies that each token is a meaningful unit that carries semantic information []. Moreover, different sub-unit granularities might be more suitable for different logographic languages. For example, ideographs are more suitable as input tokens for Chinese, while individual strokes are better suited for Japanese []. [] chose to extract visual features of logographs instead of symbolic features to avoid specifying the level of granularity when decomposing a character.

In Figure  and , LSTM treats all input features with relatively equal importance, evident by relatively high activation values across most hidden states. On the contrary, the structural constraints imposed by treeLSTM resulted in a more automated selection of input features, in which most of the high activation concentrate at the hidden states of sub-trees’ roots. In other words, treeLSTM seemed to have learned to build representations relevant to the task at the right level of granularity. Learning the right features via structures instead of delicate feature engineering is an advantage that should be explored further for RNN models.

Various work suggested that incorporating syntactic structures is tricky and does not always improve results. For example, in subject-verb agreement modeling, a model could easily ignore syntactic information from the input data and so syntactic constraints must be explicitly injected into the model’s architecture []. Doing so would make it easier for the model to discern certain relationship of interest (subject-verb agreement) by shortening the path between relevant sub-units (subject and verb in a sentence) []. Hence, being explicit in modeling structures may be the key to obtaining performance gain. Similarly, our work showed that modeling structures explicitly (using treeLSTM) is better than implicitly (using LSTM) in terms of model performance.

Models that learn task-specific trees from data could be better than models that use conventional parsers to obtain the trees []. However, the learned trees are usually shallow and hard to interpret []. Shallow trees make the paths between related tokens shorter but they do not always result in better performance. For the binary trees of logographs, the tree depth is unlikely to account for the improved performance because the trees are not balanced binary trees (which are shallowest). The improved performance is more likely due to the inductive bias using logographic structures. We showed that by exploiting structures like human intuition, treeLSTM could arrive at the general and correct solution in a more data-efficient and effective manner for pronunciation prediction and language modeling tasks concerning logographs (Chinese characters). Better interpretability due to the model following human intuition provides some confidence that the model is general and is not exploiting statistical biases in the data.

To tackle the out-of-vocabulary problem, it is common to apply pre-processing steps such as replacing infrequent characters or characters unseen during training with the UNK token. However, these pre-processing steps could potentially remove information stemmed from the usage of the infrequent characters. Hierarchical embeddings enable modeling Chinese text directly without these pre-processing steps. By treating Chinese characters as recursive structures of common sub-units instead of independent tokens, hierarchical embeddings make it possible for model to have a much bigger vocabulary. Hierarchical embeddings also make learning representations of infrequent characters easier through leveraging the similarity between structures of infrequent characters and of common characters. Thus, models that use hierarchical embeddings may be able to capture the intention behind the usage of infrequent characters. Furthermore, hierarchical embeddings can also be used to model Japanese Kanji which are logographs created using the same principles as Chinese logographs.

We used hierarchical embeddings in the pronunciation prediction task and language modeling task. However, other NLP tasks may also benefit from using hierarchical embeddings as previous work exploiting logograph structures have shown promising results in tasks such as machine translation [] or textual error detection []. In particular, hierarchical embeddings may be useful in named entity recognition (NER) where infrequent characters may be used in names or in poetry generation where characters need to rhyme.

The current work can be extended in a few different ways. Although treeLSTM was used to construct the hierarchical embeddings, it is possible that self-attention models such as Transformer [] might lead to even better performance as they could learn patterns in trees that the current models could not. However, since self-attention models usually have a lot of parameters they may overfit given the small amount of training data. It would be interesting to see how well a bigger and more powerful model such as Transformer can model logographic structures given limited data. For language modeling, using hierarchical embeddings does not constrain the choice of the language model. The AWD-LSTM can be replaced by a more powerful model such as Transformer [] or BERT []. It is possible that hierarchical embeddings may have little benefit for models such as Transformer or BERT which can exploit contextual information to deduce the representations of the infrequent characters. However, exploiting both contextual information and structural similarity to obtain better embeddings for infrequent characters should theoretically be better than just relying on contextual information.