In ME analysis, the frame sequence from the onset frame to the apex frame effectively reflects the dynamic characteristics of facial muscle movements and their evolving trends. However, these sequences often exhibit inconsistent frame lengths, ranging from 9 to over 100 frames. For sequences with a higher frame count, direct downsampling may result in unsmooth motion trends and potential loss of critical motion information. Conversely, sequences with insufficient frames require effective upsampling strategies to reconstruct their motion information. To address these issues, we employs a keyframe-based video frame interpolation (VFI) algorithm, designed to achieve two primary objectives: (i) generating standardized-length frame sequences, and (ii) preserving the temporal dynamics of MEs, thereby producing smoother and more continuous motion patterns that are crucial for capturing subtle ME features. Specifically, we utilize the VFI model proposed by Zhang et al. []. This model innovatively restructures the information processing mechanism of inter-frame attention, enhancing appearance feature representation through attention maps and effectively capturing motion dynamics. In implementation, the onset frame $Io$ and apex frame $Ia$ are used as inputs to the VFI model to produce fixed-length ME sequences of 11 frames. The interpolation process can be formulated as:

where $It$ represents the interpolated frame at time step $t$, and $VFI(\cdot )$ denotes the frame interpolation function.

The selection of 11 frames strikes a balance between capturing high-resolution motion features and maintaining computational efficiency. Furthermore, this length selection is informed by the successful practices of previous studies []. This approach not only preserves the spatiotemporal information of the original ME videos and eliminates redundant frames but also accentuates the motion dynamics of the peak frame, providing robust input data for subsequent feature extraction and classification. The impact of this hyperparameter on model performance is analyzed in Section .

For the fixed-length ME frame sequences, we compute inter-frame optical flow and frame difference features as handcrafted inputs for the model. Optical flow features capture pixel-level motion between frames, while frame difference features represent the intensity variations of each pixel between consecutive frames. These features are crucial for capturing the subtle muscular movements essential for MER and are inherently complementary. In this study, we employ the deep learning-based optical flow estimation method, FlowNet 2.0 2 footnote 2 2 footnote 2 https://github.com/NVIDIA/flownet2-pytorch, which has proven effective in detecting subtle motions in videos, to extract optical flow features. The optical flow between two consecutive frames $It$ and $I+t1$ is computed as a displacement vector field $(ut,vt)$, where $ut$ and $vt$ represent the horizontal and vertical displacements, respectively. This can be expressed as:

The resulting optical flow features for the entire sequence are represented as a tensor $\in OR\times 128128210$, where 2 corresponds to the horizontal and vertical displacement components, and 10 denotes the number of inter-frame pairs. The frame difference features are computed as the pixel-wise intensity difference between consecutive frames for each RGB channel. For a given channel $\in c\{R,G,B\}$, the frame difference $\Delta Itc$ at time $t$ is calculated as:

The resulting frame difference features for the entire sequence are represented as a tensor $\in DR\times 128128310$, where 3 corresponds to the RGB channels. To integrate the optical flow features $O$ and frame difference features $D$, we concatenate them along the channel dimension, resulting in a fused feature tensor $\in FR\times 128128510$:

where $Concat(\cdot )$ denotes the concatenation operation along the channel dimension. This fused feature combines both motion dynamics (via optical flow) and intensity variations (via frame differences), providing a comprehensive representation of the ME sequence for subsequent analysis. The extraction of optical flow and frame difference features is shown in Fig. . Notably, as the ME progresses, both features become increasingly pronounced.

In this study, we integrate the I3D architecture with CA Block to construct the backbone of our feature encoder, termed CA-I3D, for precise spatiotemporal feature extraction of MEs. The I3D model is an extension of 2D convolutional networks, which introduces a temporal dimension by extending traditional 2D convolutional and pooling kernels into 3D forms, thereby enabling the modeling of dynamic information within video sequences. Meanwhile, the CA Block enhances feature representation by effectively capturing channel relationships and long-range dependencies through precise positional information. The overall architecture of the proposed CA-I3D model is illustrated in Fig. . The CA-I3D model comprises multiple 3D convolutional layers, max-pooling layers, and 3D CA-Inception v1 modules []. It takes the fused feature tensor $F$ as input and produces the deep feature vector $f\theta (x)$ for a given sample $x$, as defined below:

To better align with the requirements of MER tasks, we optimized the original I3D network architecture. The model begins with a 3$\times$3$\times$3 convolutional layer for spatial feature extraction, followed by a 1$\times$3$\times$3 max-pooling layer with strides of (1, 2, 2), which performs pooling along the height and width dimensions while preserving the channel and temporal dimensions. The 3D CA-Inception v1 module employs multiple convolutional filters of varying sizes (e.g., 1$\times$1$\times$1 and 3$\times$3$\times$3) to capture diverse spatial patterns across different scales. It consists of multiple parallel convolutional branches to extract scale-specific features. The CA module is integrated after the concatenation layer of the 3D Inception v1 module within the I3D architecture. Another 1$\times$3$\times$3 max-pooling layer with strides of (1, 2, 2) is applied to downsample the feature map while preserving essential information. An additional 3D CA-Inception v1 module is then incorporated to further enhance the network’s ability to capture complex spatial patterns. The network concludes with a linear layer to facilitate nonlinear transformations. All convolutional layers use rectified linear unit (ReLU) activation, and network weights are initialized randomly following a standard normal distribution with a mean of 0 and a variance of 1. To prevent the loss of low-level image features typically associated with pooling operations, the first max-pooling layer was removed. Additionally, the final average pooling layer was eliminated, retaining only the convolutional layers. This modification not only reduces the number of parameters but also preserves global image information, enhancing the network’s robustness. Furthermore, to mitigate the risk of overfitting, the number of Inception modules was reduced from nine to two.

The pre-training of the GFE is achieved through prior learning based on a triplet network, enabling the GFE to acquire the capability of extracting general features for MER. This capability allows the model to effectively distinguish similarities and differences between samples of different categories. Based on the assumption that ME samples from the same category should form tight clusters in the embedding space, we constructed a triplet network to ensure that samples with the same label are closely positioned in the embedding space, while samples with different labels are positioned farther apart. The triplet network consists of three CA-I3D components that share the same architecture and parameters. The input of the network is a series of triple samples defined as $[xa,xp,xn]$, which consists of an anchor sample $xa$, a positive sample $xp$ from the same category, and a negative sample $xn$ from a different category. During the training process, for an input triplet sample, the model will output three feature vectors $f(xa)$, $f(xp)$ and $f(xn)$ after the processing of CA-I3D modules. We employ a triplet loss function to learn discriminative feature embedding, such that the embedded distance of the positive pair (images of the same class) is closer than that of the negative pair (images of different classes) by a distance margin. The triplet loss is presented as follows:

where $\alpha$ is a hyperparameter that controls the margin between the distances of $xp$ and $xn$. We use the Euclidean distance between ME features as a metric, which is described as follows:

After several epochs of training on the ME dataset, the network reached a fixed error over the triplet comparisons. Then we use the parameters of the convolutional layer of the trained CA-I3D model to initialize our GFE to extract the generic features of MEs.

The pretrained GFE is capable of extracting general features of MEs. However, relying solely on these general features is insufficient for accurately modeling the fine-grained information required for MER tasks. To address this limitation, we designed and pretrained an additional feature encoder, termed the AFE, to enhance the model’s ability to capture advanced features essential for MER. These advanced features include subtle facial movement patterns, complex texture characteristics, and fine-grained motion details. The pretraining of the AFE is based on a prior learning strategy that utilizes a balanced, motion-amplified ME dataset. Specifically, we employed the video magnification model proposed by Tae-Hyun et al. [], using frame sequences from the onset to the apex frames of MEs as input. To address the imbalance in sample distribution across categories, we dynamically adjusted the magnification factors based on each category’s sample size. Categories with smaller sample sizes were assigned a greater range of magnification factors, allowing for the generation of more synthetic samples and improving dataset balance. This approach not only enhances the representation of subtle motions in videos but also effectively mitigates sample imbalance within the dataset. As defined by Wu et al. [] in their work on motion magnification, a single frame within a continuous video can be described as:

where $\delta (x,t)$ is the motion field at position $x$ and time $t$. By performing motion magnification on the original image, we can obtain image $Imag$:

where $\phi$ is the magnification factor.

The sample balancing process of the ME dataset is shown in Table . It is evident that the original ME dataset suffers from significant class imbalance. For instance, in the CASME II dataset, the “negative” class contains 88 samples, while the “surprise” class has only 25 samples. Similarly, in the SAMM dataset, the “negative” class consists of 92 samples, while the “disgust” class contains just 15 samples. By applying different combinations of magnification factors, we achieved three key objectives: (i) significantly increased the sample size (from 442 samples to 4022 samples), (ii) effectively mitigated the class imbalance issue, and (iii) enhanced the visibility of facial muscle movements. Experimental results indicate that excessive magnification (especially beyond a factor of 15) leads to severe facial distortions and reduced image quality. Therefore, we set the optimal maximum magnification factor to $\phi$ = 14 in this study. Subsequently, we trained the AFE on the augmented ME dataset to improve its ability to capture more advanced and abstract ME features.

After training the GFE and AFE, we integrate the encoders with complementary prior knowledge in parallel within a metric-based meta-learning framework, forming MPFNet-P. The architectural details of MPFNet-P are presented in Fig. . This model adopts a parallel architecture design, where the GFE and AFE operate independently, extracting general and advanced features from the input data, respectively. This design enables the parallel processing of multi-perspective feature representations. The overall workflow of MPFNet-P is as follows:

(i) Data preprocessing module: The input to this module consists of image pairs formed by the onset and apex frames of ME samples. A frame interpolation method is applied to generate ME frame sequences of normalized length. Subsequently, optical flow and frame difference-based fusion features are computed to enhance the representation of spatiotemporal information. The position of this module within the workflow is illustrated in Fig. (a), with further details provided in Section . (ii) Prior learning based on a triplet network (PLTN): This module enables the model to learn generic feature extraction capabilities by constructing triplet ME samples and assessing their similarity. The acquired prior knowledge is stored in frozen parameters and subsequently used to initialize the GFE, as shown in Fig. (b). (iii) Prior learning based on sample-balanced motion-amplified MEs (PLSM): This module involves training a CA-I3D model on a sample-balanced and augmented ME dataset to improve its ability to capture advanced ME features. Upon completion of training, the learned parameters are frozen and transferred to the AFE, as depicted in Fig. (c). Additional details regarding these two prior learning processes can be found in Section .

(iv) Multi-prior fusion meta-learning module: Within the metric-based meta-learning framework, this module integrates the complementary prior knowledge of GFE and AFE to enhance the model’s MER capability, as illustrated in Fig. (d). In this module, we first encode the deep feature vectors of the samples in both the support set $S$ and the query set $Q$. Then, we average the feature vectors of samples within the same class in the support set $S$ to obtain the mean feature vector, which serves as the class centroid in the embedding space. The specific calculation method is as follows:

where $f\theta$ represents the feature encoder, $f\theta (xs)$ denotes the deep feature vector of a data sample $xs$ in the support set $S$, $Sc$ is the sample cluster of the $c$-$th$ class, and $wc$ is the centroid of the $c$-$th$ class.

Then, we employ a standard metric-based meta-learning process to compute the cosine similarity distance between each sample $f\theta (xq)$ in the query set $Q$ and the centroid of each class $wc$ in the support set $S$ within the embedding space. The calculation is as follows:

where $wcG$ and $wcA$ represent the centroids of each class in the support set $S$, computed from the deep features extracted using the GFE and the AFE, respectively. Similarly, $f\theta G(xq)$ and $f\theta A(xq)$ denote the deep feature vectors of the samples in the query set $Q$, encoded by the GFE and AFE, respectively. $dGFEi$ and $dAFEi$ denote the cosine similarity distances between features encoded by the GFE and AFE, respectively, and their corresponding centroids.

The two data streams are subsequently combined to enhance MER performance. Inspired by Gong et al. [], the calculated distances of the dual-streams are added using a weighted-sum:

where $dGFEi$ and $dAFEi$ denote the cosine similarity distances between the feature of the $i$-$th$ sample, encoded by GFE and AFE, respectively, and their corresponding centroids. The weighted distance $dsumi$ is then utilized to identify the nearest neighbour and predict the ME class label. Here, $\gamma$ represents the weighting coefficient, whose optimal value is determined through experimental validation as detailed in Section .

(v) The classification module. The classification process involves computing the Euclidean distance between the feature representations of the query set and the centroid vectors of the support set. A nearest-neighbor approach is then employed to classify the samples in the query set, as illustrated in Fig. (e). Further details are provided in Section .

MPFNet-C retains the same five core modules as MPFNet-P. The primary distinction lies in its multi-prior fusion module, where the GFE and AFE are structured in a cascaded architecture. Furthermore, a residual structure is integrated to facilitate feature transmission and fusion, enhancing information flow across encoding layers, reducing potential feature loss, and improving representational capacity. The architectural details are illustrated in Fig. . In MPFNet-C, the support set samples undergo data preprocessing to obtain the initial features $xs$. Subsequently, $xs$ is encoded by the GFE, producing the general feature representation $f\theta G(xs)$ for MER. Next, we concatenate $f\theta G(xs)$ with the original feature $xs$ along the channel dimension to form the intermediate feature representation $f\theta \prime (xs)$. This representation is then fed into the AFE for further encoding, yielding the advanced ME feature $f\theta A(xs)$. The process can be expressed as follows:

Next, for the feature vectors $f\theta A(xs)$ in the support set $S$, we compute the mean of features belonging to the same class to obtain the centroid $wc$ for each class, as defined in Equation . Similarly, the feature vectors $f\theta G(xq)$ of the samples in the query set $Q$ are computed in the same manner. The distance is then determined based on the cosine similarity between these vectors and the centroids of each class in the embedded space of the support set $S$, as follows:

The SDE task involves conducting experiments on each of the three datasets using their original emotion labels. Specifically, the SMIC dataset contains three emotion categories, while the CASME II and SAMM datasets share a common set of five emotion categories.

In the MEGC2019, the original emotion labels of the three datasets were consolidated into three broad categories: negative, positive, and surprise. Following this scheme, we first conducted three-class classification experiments separately on each dataset. Subsequently, to achieve a more comprehensive evaluation, we merged the three datasets into a composite dataset, MEGC2019-CD, and performed further experimental analyses.

To independently evaluate each participant’s samples, we employ the Leave-One-Subject-Out (LOSO) cross-validation method to assess MPFNet’s performance. During each iteration, the model was optimized solely on the training set (i.e., all data except that of the current test subject), and hyperparameter tuning was conducted only on a subset of the training data (i.e., the validation set). Importantly, the test subject’s data was never involved in the hyperparameter tuning process. As a result, the final test outcomes remain unaffected by the tuning process, ensuring the validity and reliability of our evaluation. The training process of both the GFE and AFE was also conducted using LOSO cross-validation. For the SDE task, we use accuracy (Acc) and F1-score (F1) for evaluation, with the F1-score providing a more objective and persuasive measure due to its robustness to class imbalance, especially in the CASME II and SAMM datasets. For the CDE task, we follow MEGC 2019 and use the unweighted F1-score (UF1) and unweighted average recall (UAR) to assess model performance. In fact, UF1 is commonly referred to as the macro-averaged F1-score, while UAR represents the “balanced accuracy". The calculation methods for these metrics are detailed as follows:

where $TPc$, $FPc$, and $FNc$ are the numbers of true positives, false positives, and false negatives for the $c$-$th$ class, respectively. $Nc$ is the number of samples in the $c$-$th$ class.

We design multiple experimental conditions by progressively incorporating prior knowledge. These conditions include without prior learning (w/o PL), where the encoder is trained from scratch; prior learning based on triplet network (PLTN); prior learning based on sample-balanced motion-amplified MEs (PLSM); and the fusion of both prior learning methods within the MPFNet-P or MPFNet-C framework. The experimental results are presented in Table . It is evident that the model performance significantly improves with the gradual incorporation of prior knowledge, especially with the MPFNet model that adopts a multi-prior fusion strategy, which achieves the best performance. Furthermore, MPFNet-C outperforms MPFNet-P across all evaluation metrics, indicating that the cascade feature encoder structure is more effective for MER tasks than the parallel feature encoder structure.

Optical flow features serve as a crucial motion representation method, effectively capturing pixel-level motion information between video frames. Consequently, they have been widely applied in MER. Meanwhile, frame difference features, which quantify pixel intensity variations between consecutive frames, provide complementary visual cues. This study combines both features to form a comprehensive visual representation, as described in Section . Few studies have evaluated the relative contributions of these two visual features in MER tasks. To fill this gap, we conducted an ablation experiment to identify the dominant feature. We performed extensive experiments on both the MPFNet-P and MPFNet-C architectures across three feature configurations: (i) optical flow features only, (ii) frame difference features only, and (iii) integrated features combining both modalities. As demonstrated in Table , our experimental results reveal two key findings: First, optical flow features consistently outperform frame difference features across both model architectures. Second, the feature fusion strategy yields significant performance improvements over single-feature approaches. Specifically, for the three-class classification task on the CASME II dataset, MPFNet-C achieves an accuracy of 0.865 with optical flow features alone, compared to 0.662 using only frame difference features. Notably, the integrated feature approach boosts accuracy to 0.924, demonstrating the complementary nature of these feature modalities. These results confirm the dominant role of optical flow features in MER tasks, while also demonstrating that incorporating frame difference features effectively improves the comprehensive representation capability of visual features. This study provides empirical evidence for understanding the contributions of different visual features in MER.