MotionCLR: Motion Generation and Training-free Editing via Understanding Attention Mechanisms

Ling-Hao Chen1, 2, Wenxun Dai1, Xuan Ju3, Shunlin Lu4, Lei Zhang2🤗

🤗Corresponding author.
1Tsinghua University , 2International Digital Economy Academy (IDEA),
3The Chinese University of Hong Kong, 4The Chinese University of Hong Kong, Shenzhen
arXiv Video Code Demo v1-preview Blogpost

📖 Abstract

This research delves into the problem of interactive editing of human motion generation. Previous motion diffusion models lack explicit modeling of the word-level text-motion correspondence and good explainability, hence restricting their fine-grained editing ability. To address this issue, we propose an attention-based motion diffusion model, namely MotionCLR (/ˈmoʊʃn klɪr/), with CLeaR modeling of attention mechanisms. Technically, MotionCLR models the in-modality and cross-modality interactions with self-attention and cross-attention, respectively. More specifically, the self-attention mechanism aims to measure the sequential similarity between frames and impacts the order of motion features. By contrast, the cross-attention mechanism works to find the fine-grained word-sequence correspondence and activate the corresponding timesteps in the motion sequence. Based on these key properties, we develop a versatile set of simple yet effective motion editing methods via manipulating attention maps, such as motion (de-)emphasizing, in-place motion replacement, and example-based motion generation, etc. For further verification of the explainability of the attention mechanism, we additionally explore the potential of action-counting and grounded motion generation ability via attention maps. Our experimental results show that our method enjoys good generation and editing ability with good explainability.

Demo Video

MotionCLR Interactive Demo Video

🤗 MotionCLR v1-preview Demo

📖 Introducing MotionCLR

MotionCLR supports versatile motion generation and editing. The blue and red figures represent original and edited motions. (a) Motion deemphasizing and emphasizing via adjusting the weight of ''jump''. (b) In-place replacing the action of ''walks'' with ''jumps'' and ''dances''. (c) Generating diverse motion with the same example motion. (d) Transferring motion style referring to two motions (style and content reference). (e) Editing the sequentiality of a motion.

Inplace Motion Replacement

a man jumps. → a man walks.

a man walks. → a man dances.

a man runs and then sits. → a man jumps and then sits.

Motion (de)-emphasizing

Adjusting the weight of the word ''jumps'' to (de-)emphasize the motion.

Motion Erasing

Erasing the second jumping action.

Erasing the second waving action.

Motion Shifting

Shifitng the waving and walking actions.

Shifitng the standing and squatting actions.

Shifitng the plain walking and stair walking actions.

Motion Style Transferring

🌹 Acknowledgement

The author team would like to acknowledge Dr. Jingbo Wang from Shanghai AI Laboratory and Dr. Xingyu Chen from Peking University for his constructive suggestions and discussions on downstream applications. We also would like to acknowledge Mr. Hongyang Li and Mr. Zhenhua Yang from SCUT for their detailed discussion on some technical details and writing. Mr. Bohong Chen from ZJU also provided us with insightful feedback on the evaluation and the presentations. We convey our thanks to all of them.

Citation

@article{motionllm,
  title={MotionCLR: Motion Generation and Training-free Editing via Understanding Attention Mechanisms},
  author={Chen, Ling-Hao and Dai, Wenxun and Ju, Xuan and Lu, Shunlin and Zhang, Lei},
  journal={arxiv:2410.18977},
  year={2024}
}

The website template was adapted from HumanMAC Project.