Aligned RGB face images of size $\times 2242243$ are passed through a ResNet-50 backbone, with the first 44 parameters tensors frozen and the rest fine-tuned. The output feature vector $\in fimgR2048$ captures global semantic information and is refined by an attention module.

Facial graphs are constructed from 468 landmarks with edges defined by facial geometry (jawline, eyebrows, eyes, mouth). A 3-layer Graph Convolutional Network (GCN) extracts relational features, and the pooled 128-dimensional embedding $\in fgeomR128$ is further refined with attention.

The concatenated feature vector $fjoint=[\parallel fimgfgeom]\in R2176$ is passed through a series of dense layers with dropout and LayerNorm. Emotion class probabilities are predicted using a softmax layer.

Model training minimizes KL divergence between predicted scores $s\theta$ and calibrated targets $~y$:

where $\in i\{1,\dots ,C\}$ indexes emotion classes.

We leverage a pre-trained ResNet-50 backbone. ResNet-50 backbone extracts high-level features from facial images, incorporates residual learning through skip connections. We used the standard ResNet-50 architecture [], comprising four residual stages with bottleneck blocks. The original ResNet-50 uses Batch Normalization, ReLU activations, and identity skip connections within its residual blocks to facilitate residual learning. However, in our architecture, we additionally apply a Layer Normalization step after the attention module to stabilize the reweighted feature distribution before fusion with the GCN branch. The final FC layer is removed, and the rest of the network is retained up to the Global Average Pooling (GAP) layer. This transforms ResNet-50 into a strict feature extractor, with the GAP layer producing a 2048-dimensional feature vector for each input image.

We adopt partial fine-tuning by specifically freezing first 44 parameter tensors while the remaining tensors are fine-tuned, which enable learning domain-specific features relevant to autism-oriented emotion data.

To further enhance the discriminative capacity of the extracted features, a lightweight attention module is appended after ResNet-50. This module comprises two fully connected layers with ReLU and Sigmoid activations. The resulting output is a learned attention weight vector that reweights the 2048-dimensional features, emphasizing the most informative components.

The feature map $\in FCNNR2048$ is refined using an attention module applied on the feature vector:

Here, $FCNN$ is the 2048‑dimensional raw feature vector from the last ResNet layer, $W1$ and $W2$ are learned fully‑connected weight matrices, ReLU is the rectified linear activation, $\sigma$ is the element‑wise sigmoid function (squeezing values to [0,1]), and $ACNN$ is the attention weight vector (the same size as $FCNN$).

We represent each face as a fixed-topology graph $=G(V,E)$ where $=|V|468$, and edges are manually constructed based on facial geometry (jawline, eyebrows, eyes, and mouth), partially following the the MediaPipe topology (i.e., edge-index). A 3-layer GCN computes node embeddings:

Here, $H(\ell )$ is the node-feature matrix output by layer $\ell$, $E$ represents the graph’s edge list or adjacency matrix, and the GCNConv operator, originating from Kipf and Welling’s seminal GCN model [] and implemented in PyTorch Geometric [] performs the graph convolution. $X$ is the initial $\times 4683$ matrix of landmark coordinates. ReLU activation is applied in the first two GCN layer, while the third produces the final 128-D embeddings.

Stacking the 3 GCN layers enables each landmark to gather information from its neighbors and neighbors-of-neighbors. A try-except block is implemented to handle cases where the GCN fails. In such cases, a zero vector of dimension-128 is filled in to maintain consistency.

Mean-pooled, then attention-refined yields:

Here, $Hi(3)$ denotes the 128‑D embedding of landmark $i$ after three GCN layers, $Attn(\cdot )$ is a small fully‑connected attention module applied on the pooled global embedding and $N$ is the total number of landmarks (468). Layer Normalization is applied prior fusion.

While CNN and GCN features are concatenated for representational purposes, the fused representation $\in [\parallel FattnCNNFGCN]R2176$ is passed through the classification head. Both the CNN and GCN branches contribute to the final prediction.

Here, $\in W1R\times 5122176$ and $\in W2R\times 256512$ are learned weight matrices, $\in W3R\times 7256$ is the final linear projection, $h1$ and $h2$ are intermediate 512-dimensonal and 256-dimensional hidden vectors, respectively. ReLU is the rectified-linear activation function, LN denotes layer normalization as introduced by Ba et al. [], $FattnCNN$ is the 2048-dimensional attention-refined CNN feature vector and $^y$ is the predicted probability vector for seven emotion classes. Both CNN and GCN branches contribute complementary information to the fused representation. This process has been illustrated in Fig. .

[h] Classifier Head Pseudo‑Algorithm \lx@orig@algorithmic[1] \REQUIRE$\in XR\times B2176$ Fused feature matrix (batch size $B$) $\in W1R\times 2176512,\in b1R512$ $\in W2R\times 512256,\in b2R256$ $\in W3R\times 2567,\in b3R7$ \ENSURE$\in logitsR\times B7$ Pre-softmax scores for each emotion class \FOR$\leftarrow i1$ to $B$ \STATEFC1: $\leftarrow Z1+X[i]W1b1$ \STATELN1: $\leftarrow N1LayerNorm(Z1)$ \STATEReLU1: $\leftarrow A1ReLU(N1)$ \STATEDrop1: $D1\leftarrow Dropout(A1,p=0.325)$ \STATEFC2: $\leftarrow Z2+D1W2b2$ \STATELN2: $\leftarrow N2LayerNorm(Z2)$ \STATEReLU2: $\leftarrow A2ReLU(N2)$ \STATEDrop2: $D2\leftarrow Dropout(A2,p=0.275)$ \STATEFC3: $\leftarrow logits[i]+D2W3b3$ \ENDFOR

While CNNs excel at modeling texture and color, they fail to capture geometric expressiveness, especially in ambiguous or flattened affect. GCNs, while geometrically robust, miss texture semantics. Fusion-N effectively combines both modalities, enhancing generalizability and interpretability in real-world ASD settings.

To validate our ensemble-based emotion labeling framework for ASD contexts, we used an external dataset of autistic children curated by Dr. Fatma M. Talaat []. A representative subset of 100 images was selected with regards to maintaining a balance between the emotions and to match our cohort’s age and maximize ethnic diversity, reflecting the cross-cultural variance emphasized in [].

Each image was annotated by a licensed clinical psychologist after which 61 total images were finally analysed (some were removed on the account of the image being a little difficult to label as per and to avoid confusions) and compared against predictions from our ensemble fusion pipeline, which integrates multiple pre-trained models. The approach achieved 90.16% accuracy relative to expert labels, demonstrating high reliability and reducing the annotation burden typical in ASD datasets.Compared to DeepFace(Mini-Xception) (67.07%), FER (71.95%), and their average-fused variant (73.17%), our ensemble showed superior accuracy shown in Table IV, reinforcing its robustness and suitability for real-world clinical deployment.

Several prior works have explored emotion recognition models tailored for autistic children. Alhakbani [] developed a CNN trained on ASD facial images across five emotion classes, achieving 75% accuracy, reflecting the challenges of affect recognition in this population. Smitha and Vinod [] proposed a PCA-based system deployed on FPGA; though it reached 94.1% on JAFFE, performance dropped to 82.3% on real-world ASD data, underscoring domain-specific limitations. Wang et al. [] introduced a multimodal CVT architecture combining facial and speech inputs, where the facial-only branch achieved 79.12% and the fused model reached 90%, highlighting the benefits of cross-modal integration.

These unimodal facial expression systems (75%, 82.3%, 79.12%) offer directly comparable baselines to evaluate our model, as summarized in Table V. In contrast, our architecture built on ResNet-50 and GCN backbones was trained exclusively on an in-house ASD-specific dataset and achieved 96.2% accuracy. This improvement demonstrates the advantage of residual feature fusion for capturing subtle affective cues often missed by traditional CNNs or hand-crafted methods.

Our preprocessing component analyzed 48,891 frames from NAO-mediated child–robot interaction videos, recorded in a naturalistic, unconstrained environment without head fixation or behavioral restrictions. Of these, 1,600 were discarded due to blurriness and 20,170 due to missed detections, leaving 19,322 valid face crops obtained through our two-stage pipeline, corresponding to a 39.5% face detection success rate. The comparatively low yield is consistent with the free-play setup, in which the NAO robot called the child’s name 12 times across sessions involving toys and spontaneous movement. The total preprocessing duration was 40,453.52 seconds ($\approx$ 11.2 hours). A summary of these statistics is provided in Table VI.

Each child participated in a 200-second interaction session, with video recorded at 15 frames per second, yielding a high number of frames per participant. These were processed through our facial landmark extraction and hybrid deep learning classification pipeline.

Fig. 8 presents the distribution of emotion labels obtained via our weighted ensemble method. Most frames were classified as neutral (8,969) and happy (5,309), suggesting a predominance of non-negative affective states during the interaction. Moderate representation was observed for angry (1,822), surprise (1,605), and sad (1,386), while disgust (152) and fear (79) were rare, likely due to the controlled experimental setting.

Our results are consistent with psychological research on emotion expression in ASD, where children with developmental or emotional difficulties possess an innate bias toward positive expressions in interactive and observational situations. In our dataset, 73.3 % of the children exhibited a positive emotional dominance, represented by happy and surprise. An interesting minority (26.7%), however, manifested a negative dominance, namely sad, disgust, and angry, seen among participants 2, 5, 6, and 7. This diversity highlights the importance of individualized, emotion-sensitive interventions since children with the overarching negative affect can be helped through specialized intervention in affective learning environments. Furthermore, these findings verify the viability of using robotic stimuli like NAO to examine and perhaps augment autistic children’s emotional expressivity, and demonstrate the potential of emotion-aware robotics as a tool in affective computing and autism therapy.