DiM-Gestor: Co-Speech Gesture Generation with Adaptive Layer Normalization Mamba-2 (2024)

IV-A1 Mamba-2 Fuzzy Feature Extractor

This module employs a fuzzy inference strategy, which does not rely on explicit and manual classification inputs. Instead, it provides implicit, continuous, and fuzzy feature information, automatically learning and inferring the global style and specific details directly from raw speech audio. Illustrated in Figure 2, this module is a dual-component extractor comprising both global and local extractors. The local extractor utilizes the Chinese Hubert Speech Pretrained Model [23] to process the audio sequence into discrete tokens. This pre-trained model, for its proficiency in capturing the complex attributes of speech audio, allows it to effectively represent universal Chinese speech audio latent features, denoted as $Z_{x}$.

We implement a Mamba-2 [14] global style extractor framework. In the Mamba architecture, the module scans the entire sequence of $Z_{x}$ to capture the style feature. The last output token, $z_{s}$, is considered crucial as it encompasses the global style feature contained within the speech audio. This feature is then broadcast to align with the local fuzzy features, ensuring that the global context influences the local gesture synthesis. This process allows the model to maintain a holistic understanding of the style and emotional context throughout the gesture generation process. This unified latent representation is then channeled to the downsampling module for further refinement.

The downsampling module, crucial for aligning each latent representation with its corresponding sequence of encoded gestures, is seamlessly integrated into the condition extractor. We implement a Conv1D layer with a kernel size of 201 within our architecture[5]. The size of the kernel in the gesture synthesis model suggests that the preceding gesture is influenced not only by the current semantics but also by the prior and subsequent semantic contexts [24]. This kernel size thus plays a crucial role in capturing the temporal dynamics across sequences, allowing for a more coherent and contextually integrated gesture generation that aligns with the natural flow of speech and its semantic shifts. The output of this module, $C=[c_{t}]_{t=1}^{t=T}$, serves as a unified latent representation that encapsulates both encoded acoustic features and the diffusion time step $n$, ensuring a coherent and accurate gesture generation process.

IV-A2 AdaLN Mamba-2

The AdaLN’s fundamental purpose is to incorporate a conditional mechanism [6, 5] that uniformly applies a specific function across all tokens. It offers a more sophisticated and nuanced approach to modeling, enabling the system to capture and articulate the complex dynamics between various input conditions and their corresponding outputs. Consequently, this improves the model’s predictive accuracy and ability to generate outputs more aligned with the given conditions.

Diffusion Transformers (DiTs) exemplify a sophisticated advancement in diffusion model architectures, incorporating an AdaLN-infused transformer framework primarily for text-to-image synthesis. The utility of DiTs has recently expanded to include text-conditional video generation, illustrating their versatility. Furthermore, DiTs have shown potential in co-speech gesture generation [5], marking a significant step in applying these models to sequence-based tasks. However, the inherent quadratic space complexity associated with Transformers results in substantial memory consumption and slower inference speeds.

The AdaLN architecture involves regressing the dimension-wise scale and shift parameters ($\gamma\in(T,D)$ and $\beta\in(T,D)$), derived from the fuzzy feature extractor output $C$, rather than directly learning these parameters, as depicted in Figure 2 and algorithm 1. In each AdaLN Mamba-2 stack, a latent feature $z^{n}_{1:T,m}$ is generated, combining condition information and gesture using AdaLN and the Mamba-2 architecture. The index $m$ ranges from 1 to $M$, where $M$ represents the total number of AdaLN Mamba-2 stacks. Furthermore, as illustrated in Figure 2, the final layer utilizes the same fuzzy features, supplemented by a scale and shift operation to fine-tune the gesture synthesis.

Mamba-2[14], an evolution within the Structured State Space Duality (SSD) framework, refines Mamba’s selective state-space model (SSM), markedly enhancing computational efficiency. This architecture is engineered to supplant the traditional attention mechanism in Transformers, specifically utilizing structured semiseparable matrices to optimize facets such as training speed and memory consumption. Its robust performance on modern hardware, versatility across varying sequence lengths, and the implementation of tensor parallelism firmly establish Mamba-2 as a potent alternative to traditional attention-based models, offering substantial efficiency gains and reduced operational costs.

The dual form of Structured State Space Duality (SSD) is characterized by a quadratic computation closely related to the attention mechanism. It can be defined as follows:

where $a_{i}$ are input-dependent scalars bounded within the interval $[0,1]$. These scalars represent the normalized attention weights in the structured state-space model, modulating the influence of each input token on the computed output. For indexing of $a$, $i:j$ refers to the range $(i,i+1,\dots,j-1)$ when $i<j$ and $(i,i-1,\dots,j+1)$ when $i>j$.

Let Structured Masked Attention (SMA) $M=(L\circ(QK^{T}))$, we chose 1-semi separable for structured Mask $L$ for constructing the 1-SS Structured Attention, as shown in Figure 3. So the Mamba-2 output is $Ma_{n}=M\cdot V$, Where the subscript $n$ represents the n-th blocks of the AdaLN Mamba-2.

Compared to standard (self-)attention mechanisms, the Structured State Space (SSD) model with 1-semi separable Structured Masked Attention (SMA) introduces significant optimizations. SSD eliminates the softmax normalization step, effectively reducing the requisite state size of the model from a linear to a constant scale, thereby enhancing computational efficiency from quadratic to linear. Additionally, SSD incorporates a distinct elementwise mask matrix that is applied multiplicatively, further refining the model’s efficiency and operational dynamics. This alteration not only simplifies the computational process but also improves the scalability and speed of the model, making it more adept at handling larger datasets and complex tasks without compromising performance. In contrast to the Persona-Gestor [5], which utilizes a causal mask, the primary SSD model employs a 1-Semiseparable Structured Masked Attention (1-SS mask), offering a more structured approach to attention that enhances both computational efficiency and performance in sequence modeling tasks.

IV-A3 Gesture Encoder and Decoder

The architecture of the gesture encoder and decoder is meticulously designed to process gesture sequences, as illustrated in Fig. 2.

Gesture Encoder employs a Convolution1D layer with a kernel size of 3 to encode the initial sequence of gestures, denoted as $X$, into a hidden state $H_{x}=[h_{t}]_{t=1}^{t=T}$. Experimental results reveal that using a kernel size of 1 often leads to animation jitter, likely due to insufficient spatial-temporal feature capture. Conversely, the kernel size 3 effectively mitigates this issue by capturing spatial-temporal relationships across adjacent frames, ensuring smoother gesture dynamics.

The decoder transforms the high-dimensional output from the AdaLN Mamba-2 layer to the original dimension, corresponding to the skeletal joint channels. This step involves generating the predicted noise ($\epsilon_{\theta}$), a critical component for gesture reconstruction. Utilizing a 1D kernel of size 1 in the decoder enables the model to extract relevant features and correlations between adjacent joint channels, thereby improving the overall quality and coherence of the generated gestures.

V-A1 Datasets

We have constructed Chinese Co-Speech Gestures, CCG dataset, a high-quality, synchronized motion capture and speech audio dataset comprising 391 monologue sequences. The dataset features performances by 12 female and 5 male actors, all conducted in Chinese and covering 6 distinct emotion styles with 5 different senses. These styles were carefully selected to comprehensively represent various postures, hand gestures, and head movements. The total length of the dataset amounts to 958.3 minutes. Table I and Figure 4 illustrate the time distribution of the different motion styles across the various scenes within the dataset.

The style labels for each sequence were assigned according to predefined actor directives. However, it is essential to note that these labels may not always align with the subjective interpretations of independent external annotators regarding the observed movement styles. The motion capture data was recorded using the Noitom PNS system¹¹1https://www.noitom.com.cn/, which employs inertial motion capture technology.

The full-body motion capture was conducted at a frame rate of 100 frames per second (fps), with the motion data encoded using a skeletal model comprising j=59 joints. Concurrently, the audio data was recorded at a sampling rate of 44,100 Hz.

V-A2 Speech Audio Data Process

Due to the Chinese Hubert Speech Pretrained Model being pre-trained on speech audio sampled at 16 kHz, we uniformly resampled all audio down from 44.1 kHz to match this frequency, ensuring compatibility and optimal performance.

V-A3 Gesture Data Process

We concentrate exclusively on full-body gestures, employing the data processing techniques detailed by Alexanderson et al. [29]. This includes capturing translational and rotational velocities to accurately delineate the root’s trajectory and orientation. The datasets are uniformly downsampled to a frame rate of 20 fps. To ensure precise and continuous representation of joint angles, we utilize the exponential map technique [30]. All data are segmented into 20-second clips for training and validation purposes.

V-D1 Subjective Evaluation

We utilize three distinct metrics for comprehensive subjective evaluations: human-likeness, appropriateness, and style-appropriateness. Human-likeness evaluates the naturalness and resemblance of the generated gestures to authentic human movements, independent of speech. Appropriateness assesses the temporal alignment of gestures with the speech’s rhythm, intonation, and semantics to ensure a seamless and natural interaction. Style-appropriateness quantifies the similarity between the generated gestures and their original human counterparts, ensuring fidelity to the intended gesture style.

We executed a user study utilizing pairwise comparisons by the methodology outlined by [33]. During each session, participants were shown two 20-second video clips. These clips, generated by different models, including the Ground Truth (GT), were presented for direct comparative analysis. Participants were instructed to select the preferred clip based on predefined evaluation criteria. Preferences were quantified on a scale ranging from 0 to 2, with the non-selected clip in each pair receiving a corresponding inverse score. A score of zero was used to indicate a neutral preference. More details are described in [33, 5].

Given the extensive array of styles within the datasets, an individual evaluation of each style was considered unfeasible. A random selection methodology was employed to address this. Each participant was assigned a subset of five styles for assessment. Critically, none of the audio clips selected for evaluation were used in the training or validation sets. This strategy ensured a broad yet manageable coverage of the dataset’s diversity in a controlled and unbiased manner.

We invited 30 native Chinese volunteers—14 males and 16 females aged between 18 and 35 for user study.

One-way ANOVA and post hoc Tukey, multiple comparison tests, were conducted to determine if there were statistically significant differences among the models’ scores across the three evaluation aspects. The results are presented in Figure 9 and Table II, offering detailed insights into the performance variances observed among models regarding human-likeness, appropriateness, and style-appropriateness.

Regarding the Human Likeness metric, our proposed model, DiM_m2_s_m2, achieves a score of $0.653\pm 1.323$. This result closely approximates the top-performing PG-12blocks model ($0.668\pm 1.192$) and the Ground Truth (GT) benchmark ($0.694\pm 1.430$). Statistical analysis revealed no significant differences among these three models, as Figure 10 illustrates. This indicates that the DiM_m2_s_m2 model performs comparably to the human baseline regarding perceived naturalness. In contrast, alternative methods such as DSG+ ($-1.494\pm 0.794$), GDC ($-0.467\pm 1.399$), and LDA ($-1.069\pm 1.099$) exhibit significantly lower scores on the Human Likeness metric.

In terms of the Appropriateness metric, DiM_m2_s_m2 attains a score of $0.682\pm 1.326$, demonstrating high competitiveness and being nearly on par with PG-12blocks, the top-performing synthetic model with a score of $0.687\pm 1.247$. The Ground Truth (GT) establishes the benchmark at a score of $0.710\pm 1.268$. Additionally, no significant difference exists among these three models. The models’ significant differences are visually depicted in Figure 10. Models such as DSG+, Taming, and GDC, which record scores within the negative range, evidently have difficulty synchronizing gestures with the speech context.

The model DiM_m2_s_m2 exhibits significant superiority in the Style Appropriateness metric, achieving the highest score of $1.30\pm 0.770$ among all evaluated models. This metric underscores our method’s capability to generate gestures stylistically congruent with the various Chinese speech styles. In contrast, the PG-12blocks model attains a lower score of $0.664\pm 1.216$, while the other methods, including DSG+, GDC, and LDA, exhibit negative scores in this category. These findings emphasize the distinct advantage of DiM_m2_s_m2 in producing gestures that align closely with the intended stylistic attributes of the spoken language.

In conclusion, DiM_m2_s_m2 outperforms alternative models in Style Appropriateness and achieves highly competitive results in Human-Likeness and Appropriateness. These findings suggest that DiM_m2_s_m2 effectively generates perceptually realistic gestures and is well-aligned with speech-driven gesture synthesis’s contextual and stylistic requirements. This highlights the strength of our Mamba-2 approach in addressing the multi-dimensional challenges in this domain, setting a new standard for synthetic gesture quality compared to traditional transformer methods.

V-D2 Objective Evaluation

We employ three objective evaluation metrics to assess the quality and synchronization of generated gestures: Fréchet Gesture Distance (FGD) in both feature and raw data spaces[34], and BeatAlign[35]. Inspired by the Fréchet Inception Distance (FID)[36], FGD evaluates the quality of generated gestures and has shown moderate correlation with human-likeness ratings, surpassing other objective metrics[37]. BeatAlign, on the other hand, assesses gesture-audio synchrony by calculating the Chamfer Distance between audio beats and gesture beats, thus providing insights into the temporal alignment of gestures with speech rhythms.

Table III provides a comparison of the objective evaluation metrics for several models in speech-driven gesture synthesis, including our proposed method, DiM_m2_s_m2, alongside GT, DSG+, GDC, LDA, Taming, PG-6blocks, and PG-12blocks. The metrics evaluated are the Fréchet Gesture Distance (FGD) in both feature space and raw data space, as well as the BeatAlign score. Lower FGD values and higher BeatAlign scores indicate better performance.

Our proposed model, DiM_m2_s_m2, achieves the lowest FGD on the feature space among all synthetic models with a score of 17.716, significantly outperforming the other methods. By comparison, PG-12blocks and PG-6blocks, which also performed well on other perceptual metrics, scored 100.899 and 108.913, respectively. High FGD values in DSG+ (42329.558) and LDA (34678.668) indicate poor alignment with the target gesture distribution, highlighting our approach’s substantial improvements in naturalness and similarity to natural gestures.

Regarding the BeatAlign metric, DiM_m2_s_m2 scores competitively with a BeatAlign value of 0.670. While this is close to the highest scores achieved by PG-12blocks (0.669) and DiM_m2_s_conv (0.678), it demonstrates the overall balance of our model across both spatial and temporal metrics. Models such as DSG+ (0.423) and LDA (0.428) scored significantly lower, indicating suboptimal temporal synchronization.

In summary, DiM_m2_s_m2 achieves superior performance across multiple objective metrics, with the lowest FGD values in both feature and raw data spaces, indicating a close resemblance to natural gestures. Although its BeatAlign score is slightly lower than the top-performing PG-12blocks, the overall results validate DiM_m2_s_m2 as a highly effective approach for generating temporally and spatially consistent gestures in the domain of speech-driven gesture synthesis.

V-E1 Effect of Fuzzy Feature Extractor

Table II and Figure 9 present the performance differences when utilizing mamba-1, mamba-2, and convolution 1D [5] as the fuzzy feature extractor across various model configurations. In all instances, the AdaLN module is integrated with the Mamba-2 architecture.

In the Human-likeness metric, both DiM_m2_s_m2 (0.653 ± 1.323) and DiM_m2_s_conv (0.593 ± 1.222) achieve high scores, with no statistically significant difference ($p=1.0>0.05$) observed between the two. These findings suggest that Mamba-2 and 1D Convolution represent competitive alternatives for capturing human-like gestures. However, the model DiM_m2_s_m1, which employs Mamba-1, achieves a significantly lower score of $0.243\pm 1.348$.

Similarly, in the Appropriateness metric, there is no significant difference ($p=2.1e-1>0.05$) between DiM_m2_s_m2 (0.682 ± 1.326) and DiM_m2_s_conv (0.425 ± 1.319). Both models align well with the contextual requirements of speech-driven gestures. However, DiM_m2_s_m1 achieves a much lower score (0.106 ± 1.420).

In Style Appropriateness, DiM_m2_s_m2 outperforms the alternatives, achieving the highest score of 1.30 ± 0.770. This result underscores the superiority of Mamba-2 in capturing stylistic nuances and generating gestures that are contextually relevant and visually coherent with the speech content. In comparison, DiM_m2_s_conv scores moderately at 0.549 ± 1.111, while DiM_m2_s_m1 achieves a significantly lower score of 0.230 ± 1.221.

The objective evaluation results in Table III highlight the benefits of using the mamba-2 fuzzy feature extractor in achieving superior alignment and detail in gesture synthesis. Specifically, DiM_m2_s_m2 records the lowest Fréchet Gesture Distance (FGD) in feature space at 17.716, outperforming alternative configurations like DiM_m2_s_conv (44.282) and DiM_m2_s_m1 (120.053).

For the BeatAlign metric, which measures temporal synchronization with speech, DiM_m2_s_conv achieves the highest score of 0.678, slightly outperforming DiM_m2_s_m2 (0.670) and DiM_m2_s_m1 (0.669). While mamba-2 in DiM_m2_s_m2 remains competitive.

The ablation study of fuzzy feature extractor demonstrates that our proposed model, DiM_m2_s_m2, utilizing Mamba-2 for the fuzzy feature extractor, consistently outperforms the Mamba-1 configurations, particularly in Style Appropriateness. Although no significant difference was observed between DiM_m2_s_m2 and DiM_m2_s_conv in Human Likeness and Appropriateness, DiM_m2_s_m2 demonstrates a clear advantage in Style Appropriateness. These findings validate the effectiveness of Mamba-2 in achieving high-quality gesture synthesis that balances naturalness, contextual alignment, and stylistic coherence.

V-E2 Effect of adaLN mamba Architecture

This experiment investigates the impact of employing different adaLN Mamba architectures (adaLN Mamba-1 and adaLN Mamba-2) on the generation effect, as shown in Table II and Table III. In the Human Likeness metric, DiM_m2_s_m2 achieves a high score of 0.653 ± 1.323, comparable to DiM_m1_s_conv (0.651 ± 1.279) and DiM_m1_s_m1 (0.641 ± 1.236). Statistical analysis reveals no significant difference between these models in this metric (Figure 9). However, DiM_m2_s_m1, which uses the Mamba-1 architecture, scores considerably lower at 0.243 ± 1.348. This suggests that while both architectures can produce human-like gestures, the Mamba-2 architecture in DiM_m2_s_m2 exhibits slight improvements in capturing nuanced human motion.

Style Appropriateness highlights a more pronounced distinction between the models. Our proposed model, DiM_m2_s_m2, achieves the highest score of 1.30 ± 0.770, indicating superior stylistic coherence and visual appeal in gesture synthesis. Both DiM_m1_s_conv (0.384 ± 1.186) and DiM_m1_s_m1 (0.323 ± 1.136) perform moderately, while DiM_m2_s_m1 again scores poorly (0.230 ± 1.221).

Quantitatively, Table III highlights that models with adaLN mamba-2 architectures, like DiM_m2_s_m2, consistently achieve lower FGD scores, underscoring their alignment with natural gestures. For instance, DiM_m2_s_conv records an FGD of 44.282 in feature space, while DiM_m1_s_m1, using adaLN mamba-1, records a significantly higher FGD of 130.119. This suggests that adaLN mamba-2 enhances alignment with the target gesture distribution.

The ablation study confirms that using mamba-2 for both the fuzzy feature extractor and adaLN mamba architecture provides optimal results. Our proposed model, DiM_m2_s_m2, outperforms all other configurations in perceptual and quantitative metrics, highlighting the combined benefits of the mamba-2 configurations in generating realistic, contextually appropriate, and stylistically aligned gestures in speech-driven gesture synthesis.

Interestingly, experiments that mixed Mamba-1 and Mamba-2 architectures resulted in a noticeable decrease in performance. Conversely, utilizing a consistent Mamba architecture throughout substantially enhanced the scores. The best results were achieved using Mamba-2, underscoring its superior effectiveness in this application context.