cogvideox_paper.pdf

CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer Zhuoyi Yang⋆ Jiayan Teng⋆ Wendi Zheng Ming Ding Shiyu Huang Jiazheng Xu...

CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer Zhuoyi Yang⋆ Jiayan Teng⋆ Wendi Zheng Ming Ding Shiyu Huang Jiazheng Xu Yuanming Yang Wenyi Hong Xiaohan Zhang Guanyu Feng Da Yin arXiv:2408.06072v1 [cs.CV] 12 Aug 2024 Xiaotao Gu Yuxuan Zhang Weihan Wang Yean Cheng Ting Liu Bin Xu Yuxiao Dong Jie Tang Zhipu AI Tsinghua University Abstract We introduce CogVideoX, large-scale diffusion transformer models designed for generating videos based on text prompts. To efficently model video data, we pro- pose to levearge a 3D Variational Autoencoder (VAE) to compress videos along both spatial and temporal dimensions. To improve the text-video alignment, we pro- pose an expert transformer with the expert adaptive LayerNorm to facilitate the deep fusion between the two modalities. By employing a progressive training technique, CogVideoX is adept at producing coherent, long-duration videos characterized by significant motions. In addition, we develop an effective text-video data processing pipeline that includes various data preprocessing strategies and a video captioning method. It significantly helps enhance the performance of CogVideoX, improving both generation quality and semantic alignment. Results show that CogVideoX demonstrates state-of-the-art performance across both multiple machine metrics and human evaluations. The model weights of both the 3D Causal VAE and CogVideoX are publicly available at https://github.com/THUDM/CogVideo. 1 Introduction The rapid development of text-to-video models has been phenomenal, driven by both the Transformer architecture (Vaswani et al., 2017) and diffusion model (Ho et al., 2020). Early attempts to pretrain and scale Transformers to generate videos from text have shown great promise, such as CogVideo (Hong et al., 2022) and Phenaki (Villegas et al., 2022). Meanwhile, diffusion models have recently made exciting advancements in multimodal generation, including video generation (Singer et al., 2022; Ho et al., 2022). By using Transformers as the backbone of diffusion models, i.e., Diffusion Transformers (DiT) (Peebles & Xie, 2023), text-to-video generation has reached groundbreaking levels, as evidenced by the impressive Sora showcases (OpenAI, 2024b). Despite these rapid advancements in DiTs, it remains technically unclear how to achieve long-term consistent video generation. Challenges such as efficiently modeling video data, effectively aligning videos with text semantics, as well as constructing the high-quality text-video pairs for model training have thus far been largely unaddressed. In this work, we train and introduce CogVideoX, a set of large-scale diffusion transformer models designed for generating long-term, temporally consistent videos. We address the challenges mentioned above by developing a 3D variational Autoencoder (VAE), an expert Transformer, and a video data *Equal contributions. Core contributors: Zhuoyi, Jiayan, Wendi, Ming, and Shiyu. {yangzy22,tengjy24}@mails.tsinghua.edu.cn, {yuxiaod,jietang}@tsinghua.edu.cn Technical Report  Figure 1: The performance of openly-accessible text-to-video models in different aspects. filtering and captioning pipeline, respectively. First, to efficiently consume video data, we design and train a 3D causal VAE that compresses the video along both spatial and temporal dimensions. Compared to unfolding a video into a one-dimensional sequence in the pixel space, this strategy helps significantly reduce the sequence length and associated training compute. Unlike previous video models (Blattmann et al., 2023) that often use a 2D VAE to encode each frame separately, the 3D VAE helps prevent flicker in the generated videos, that is, ensuring continuity among frames. Second, to improve the alignment between videos and texts, we propose an expert Transformer with expert adaptive LayerNorm to facilitate the fusion between the two modalities. To ensure the temporal consistency in video generation and capture large-scale motions, we propose to use 3D full attention to comprehensively model the video along both temporal and spatial dimensions. Third, as most video data available online lacks accurate textual descriptions, we develop a video captioning pipeline capable of accurately describing video content. This pipeline is used to generate new textual descriptions for all video data, which significantly enhances CogVideoX’s ability to grasp precise semantic understanding. In addition, we adopt and design progressive training techniques, including mixed-duration training and resolution progressive training, to further enhance the generation performance and stability of CogVideoX. Furthermore, we propose Explicit Uniform Sampling, which stablizes the training loss curve and accelerates convergence by setting different timestep sampling intervals on each data parallel rank. Both machine and human evaluations suggest that CogVideoX outperforms well-known public models. Figure 1 shows the performance of CogVideoX in different aspects. CogVideoX is an ongoing attempt to advance text-to-video generation. To facilitate further devel- opments, we open-source the model weight of part of CogVideoX and the 3D VAE, and we plan to release future and larger models as well. Now open-sourced CogVideoX is capable of generating 720×480 videos of six seconds with eight frames per second. It can be publicly accessed from https://github.com/THUDM/CogVideo. 2 The CogVideoX Architecture In the section, we present the CogVideoX model. Figure 2 illustrates the overall architecture. Given a pair of video and text input, we design a 3D causal VAE to compress the video into the latent space, and the latents are then patchified and unfolded into a long sequence denoted as zvision. Simultaneously, we encode the textual input into text embeddings ztext using T5 (Raffel et al., 2020). Subsequently, 2 ztext and zvision are concatenated along the sequence dimension. The concatenated embeddings are then fed into a stack of expert transformer blocks. Finally, the model output are unpatchified to restore the original latent shape, which is then decoded using a 3D causal VAE decoder to reconstruct the video. We illustrate the technical design of the 3D causal VAE and expert transfomer in detail. Figure 2: The overall architecture of CogVideoX. 2.1 3D Causal VAE Videos encompass not only spatial information but also substantial temporal information, usually resulting in orders of magnitude more data volumes than images. To tackle the computational challenge of modeling video data, we propose to implement a video compression module based on 3D Variational Autoencoders (3D VAEs) (Yu et al., 2023). The idea is to incorporate three- dimentional convolutions to compress videos both spatially and temporally. This can help achieve a higher compression ratio with largely improved quality and continuity of video reconstruction when compared to previous image VAEs (Rombach et al., 2022; Esser et al., 2021). Figure 3 (a) shows the structure of the proposed 3D VAE. It comprises an encoder, a decoder and a latent space regularizer. The Gaussian latent space is constrained by a Kullback-Leibler (KL) regularizer. The encoder and decoder consist of four symmetrically arranged stages, respectively performing 2× downsampling and upsampling by the interleaving of ResNet block stacked stages. The first two rounds of downsampling and the last two upsampling involve both the spatial and temporal dimensions, while the last round only applies spatial sampling. This enables the 3D VAE to achieve a 4× compression in the temporal dimension and an 8×8 compression in the spatial dimension. In total, this achieves a 4×8×8 compression from pixels to the latents. We adopt the temporally causal convolution (Yu et al., 2023), which places all the paddings at the beginning of the convolution space, as shown in Figure 3 (b). This ensures the future information not to influence the present or past predictions. Given that processing videos with a large number of frames introduces excessive GPU memory usage, we apply context parallel at the temporal dimension 3 Figure 3: (a) The structure of the 3D VAE in CogVideoX. It comprises an encoder, a decoder and a latent space regularizer, achieving a 4×8×8 compression from pixels to the latents. (b) The context parallel implementation on the temporally causal convolution. for 3D convolution to distribute computation among multiple devices. As illustrated by Figure 3 (b), due to the causal nature of the convolution, each rank simply sends a segment of length k − 1 to the next rank, where k indicates the temporal kernel size. This results in relatively low communication overhead. During actual implementation, we first train a 3D VAE on lower resolutions and fewer frames to save computation. We observe that the encoding of larger resolution generalizes naturally, while extending the number of frames to be encoded does not work as seamlessly. Therefore, we conduct a two-stage training process by first training on short videos and finetuning by context parallel on long videos. Both stages of training utilize a weighted combination of the L2 loss, LPIPS (Zhang et al., 2018) perceptual loss, and GAN loss from a 3D discriminator. 2.2 Expert Transformer We introduce the design choices in Transformer for CogVideoX, including the patching, positional embedding, and attention strategies for handling the text-video data effectively and efficiently. Patchify. The 3D causal VAE encodes a video latent vector of shape T × H × W × C, where T represents the number of frames, H and W represent the height and width of each frame, C represents the channel number, respectively. The video latents are then patchified along the spatial dimensions, generating sequence zvision of length T · H W p · p. Note that, we do not patchify along the temporal dimension in order to enable joint training of images and videos. 3D-RoPE. Rotary Position Embedding (RoPE) (Su et al., 2024) is a relative positional encoding that has been demonstrated to capture inter-token relationships effectively in LLMs, particularly excelling in modeling long sequences. To adapt to video data, we extend the original RoPE to 3D-RoPE. Each latent in the video tensor can be represented by a 3D coordinate (x, y, t). We independently apply 1D-RoPE to each dimension of the coordinates, each occupying 3/8, 3/8, and 2/8 of the hidden states’ channel. The resulting encoding is then concatenated along the channel dimension to obtain the final 3D-RoPE encoding. We empirically examine the use of RoPE. Figure 4 (a)shows the comparison between 3D RoPE and sinusoidal absolute position encoding. We can observe that the loss curve using 3D RoPE converges significantly faster than that with sinusoidal encoding. We further compare the use of 3D RoPE alone against the combination of 3D RoPE and learnable absolute position embedding. Figure 4 (b) 4 indicates that the loss curves of both methods converge almost identically. Therefore, we choose to use 3D RoPE alone for simplicity. (a) RoPE vs. Sinusoidal (b) RoPE vs. RoPE + Learnable (c) Expert Ada. LN vs. Expert Ada. LN + MLP (d) Uniform Sampling vs. No Uniform Sampling Figure 4: Training loss curve of different ablations. Expert Transformer Block. We concatenate the embeddings of both text and video at the input stage to better align visual and semantic information. However, the feature spaces of these two modalities differ significantly, and their embeddings may even have different numerical scales. To better process them within the same sequence, we employ the Expert Adaptive Layernorm to handle each modality independently. As shown in Figure 2, following DiT (Peebles & Xie, 2023), we use the timestep t of the diffusion process as the input to the modulation module. Then, the Vision Expert Adaptive Layernorm (Vison Expert AdaLN) and Text Expert Adaptive Layernorm (Text Expert AdaLN) apply this modulation mechanism to the vision hidden states and text hidden states, respectively. This strategy promotes the alignment of feature spaces across two modalities while minimizing additional parameters. To verify the adoption of Expert Adaptive Layernorm, we experiment with different ways of incorpo- rating experts: expert LayerNorm and MLP, and expert Layernorm only. Our experiments find that adding expert MLP does not effectively accelerate the model’s convergence (Cf. Figure 4 (c)). To reduce the model parameters, we only choose to use Expert Adaptive Layernorm. 3D Full Attention. Previous works (Singer et al., 2022; Guo et al., 2023) often employ separated spatial and temporal attention to reduce computational complexity and facilitate fine-tuning from text-to-image models. However, as illustrated in Figure 5, this separated attention approach requires extensive implicit transmission of visual information, significantly increasing the learning complexity and making it challenging to maintain the consistency of large-movement objects. Considering the great success of long-context training in LLMs (AI@Meta, 2024; Bai et al., 2024; Xiong et al., 2023) and the efficiency of FlashAttention (Dao et al., 2022), we propose a 3D text-video hybrid attention mechanism. This mechanism not only achieves better results but can also be easily adapted to various parallel acceleration methods. 5 Figure 5: The separated spatial and temporal attention makes it challenging to handle the large motion between adjacent frames. In the figure, the head of the person in frame i + 1 cannot directly attend to the head in frame i. Instead, visual information can only be implicitly transmitted through other background patches. This can lead to inconsistency issues in the generated videos. Figure 6: The diagram of mixed-duration training and Frame Pack. To fully utilize the data and enhance the model’s generalization capability, we train with videos of different durations within the same batch. 3 Training CogVideoX We mix images and videos during training, treating each image as a single-frame video. Additionally, we employ progressive training from the resolution perspective. For the diffusion setting, we adopt v-prediction (Salimans & Ho, 2022) and zero SNR (Lin et al., 2024), following the noise schedule used in LDM (Rombach et al., 2022). During diffusion training for timestep sampling, we also employ an explicit uniform timestep sampling method, which benefits training stability. 3.1 Frame Pack Previous video training methods often involve joint training of images and videos with fixed number of frames (Singer et al., 2022; Blattmann et al., 2023). However, this approach usually leads to two issues: First, there is a significant gap between the two input types using bidirectional attention, with images having one frame while videos having dozens of frames. We observe that models trained this way tend to diverge into two generative modes based on the token count and not to have good generalizations. Second, to train with a fixed duration, we have to discard short videos and truncate long videos, which prevents full utilization of the videos of varying number of frames. To address these issues, we choose mixed-duration training, which means training videos of different lengths together. However, inconsistent data shapes within the batch make training difficult. Inspired by Patch’n Pack (Dehghani et al., 2024), we place videos of different lengths into the same batch to ensure consistent shapes within each batch, a method we refer to as Frame Pack. The process is illustrated in Figure 6. 6 3.2 Resolution Progressive Training The training pipeline of CogVideoX is divided into three stages: low-resolution training, high- resolution training, and high-quality video fine-tuning. Similar to images, videos from the Internet usually include a significant amount of low-resolution ones. Progressive training can effectively utilize videos of various resolutions. Moreover, training at low resolution initially can equip the model with coarse-grained modeling capabilities, followed by high-resolution training to enhance its ability to capture fine details. Compared to direct high-resolution training, staged training can also help reduce the overall training time. Figure 7: The comparison between the initial generation states of extrapolation and interpolation when increasing the resolution with RoPE encoding. Extrapolation tends to generate multiple small, clear, and repetitive images, while interpolation generates a blurry large image. Extrapolation of Position Code. When adapting low-resolution position encoding to high- resolution, we consider two different methods: interpolation and extrapolation. We show the effects of two methods in Figure 7. Interpolation tens to preserve global information more effectively, whereas the extrapolation better retains local details. Given that RoPE is a relative position encoding, We chose the extrapolation to maintain the relative position between pixels. High-Quality Fine-Tuning. Since the filtered pre-training data still contains a certain proportion of dirty data, such as subtitles, watermarks, and low-bitrate videos, we selected a subset of higher quality video data, accounting for 20% of the total dataset, for fine-tuning in the final stage. This step effectively removed generated subtitles and watermarks and slightly improved the visual quality. However, we also observed a slight degradation in the model’s semantic ability. 3.3 Explicit Uniform Sampling Ho et al. (2020) defines the training objective of diffusion as √ √ 2 Lsimple (θ) := Et,x0 ,ϵ ϵ − ϵθ ( ᾱt x0 + 1 − ᾱt ϵ, t) , (1) where t is uniformly distributed between 1 and T. The common practice is for each rank in the data parallel group to uniformly sample a value between 1 and T , which is in theory equivalent to Equation 1. However, in practice, the results obtained from such random sampling are often not sufficiently uniform, and since the magnitude of the diffusion loss is related to the timesteps, this can lead to significant fluctuations in the loss. Thus, we propose to use Explicit Uniform Sampling to divide the range from 1 to T into n intervals, where n is the number of ranks. Each rank then uniformly samples within its respective interval. This method ensures a more uniform distribution of timesteps. As shown in Figure 4 (d), the loss curve from training with Explicit Uniform Sampling is noticeably more stable. In addition, we compare the loss at each diffusion timestep alone between the two methods for a more precise comparison. We find that after using explicit uniform sampling, the loss at all timesteps decreased faster, indicating that this method can accelerate loss convergence. 7 3.4 Data We construct a collection of relatively high-quality video clips with text descriptions through video filters and recaption models. After filtering, approximately 35M single-shot clips remain, with each clip averaging about 6 seconds. Video Filtering. Video generation models need to learn the dynamic information of the world, but unfiltered video data is of highly noisy distribution, primarily due to two reasons: First, videos are human-created, and artificial editing may distort the authentic dynamic information; Second, the quality of videos can significantly drop due to issues during filming, such as camera shakes and substandard equipment. In addition to the intrinsic quality of the videos, we also consider how well the video data supports model training. Videos with minimal dynamic information or lacking connectivity in dynamic aspects are considered detrimental. Consequently, we have developed a set of negative labels, which include: Editing: Videos that have undergone obvious artificial processing, such as re-editing and special effects, causing degradation of the visual integrity. Lack of Motion Connectivity: Video segments with image transitions lacking motion connectivity, commonly seen in videos artificially spliced or edited from images. Low Quality: Poorly shot videos with unclear visuals or excessive camera shake. Lecture Type: Videos focusing primarily on a person continuously talking with minimal effective motion, such as educational content, lectures, and live-streamed discussions. Text Dominated: Videos containing a substantial amount of visible text or primarily focusing on textual content. Noisy Screenshots: Noisy videos recorded from phone or computer screens. We sample 20,000 video data samples and label the presence of negative tags in each of them. By using these annotations, we train several filters based on video-llama (Zhang et al., 2023b) to screen out low-quality video data. In addition, we calculate the optical flow scores and image aesthetic scores of all training videos and dynamically adjust the threshold ranges during training to ensure the fluency and aesthetic quality of the generated videos. Video Caption. Typically, most video data does not come with corresponding descriptive text, so it is necessary to convert the video data into textual descriptions to provide the essential training data for text-to-video models. Currently, there are some video caption datasets available, such as Panda70M (Chen et al., 2024b), COCO Caption (Lin et al., 2014), and WebVid Bain et al. (2021). However, the captions in these datasets are usually very short and fail to describe the video’s content comprehensively. To generate high-quality video caption data, we establish a Dense Video Caption Data Generation pipeline, as detailed in Figure 8. The idea is to generate video captions from image captions. First, we use the Panda70M video captioning model (Chen et al., 2024b) to generate short captions for the videos. Then, we employ the image recaptioning model CogVLM (Wang et al., 2023a) used in Stable Diffusion 3 (Esser et al., 2024) and CogView3 (Zheng et al., 2024a) to create dense image captions for each of the frames within a video. Subsequently, we use GPT-4 to summarize all the image captions to produce the final video caption. To accelerate the generation from image captions to video captions, we fine-tune a Llama2 model (Touvron et al., 2023) using the summary data generated by GPT-4 (Achiam et al., 2023), enabling large-scale video caption data generation. Additional details regarding the video caption data generation process can be found in Appendix C. The pipeline above generates the caption data that is used to trained the CogVideoX model introduced in this report. To further accelerate video recaptioning, we also fine-tune an end-to-end video understanding model CogVLM2-Caption, based on the CogVLM2-Video1 and Llama3 (AI@Meta, 2024), by using the dense caption data generated from the aforementioned pipeline. The video caption 1 The CogVLM2-Video model weight is openly available at https://github.com/THUDM/CogVLM2. 8 Figure 8: The pipeline for dense video caption data generation. In this pipeline, we generate short video captions with the Panda70M model, extract frames to create dense image captions, and use GPT-4 to summarize these into final video captions. To accelerate this process, we fine-tuned a Llama 2 model with the GPT-4 summaries. data generated by CogVLM2-Caption is used to train the next generation of CogVideoX. Examples of video captions generated by this end-to-end CogVLM2-Caption model are shown in Appendix D. In Appendix E, we also present some examples of video generation where a video is first input into CogVLM2-Caption to generate captions, which are then used as input for CogVideoX to generate new videos, effectively achieving video-to-video generation. 4 Empirical Evaluation In this section, we present the performance of CogVideoX through two primary methods: automated metric evaluation and human assessment. We train the CogVideoX models with different parameter sizes. We show results for 2B and 5B for now, larger models are still in training. To facilitate the development of text-to-video generation, we open-source the model weight at https://github.com/THUDM/CogVideo. 4.1 Automated Metric Evaluation Baselines. We choose openly-accessible top-performing text-to-video models as baselines, includ- ing T2V-Turbo (Li et al., 2024), AnimateDiff (Guo et al., 2023), VideoCrafter2 (Chen et al., 2024a), OpenSora (Zheng et al., 2024b), Show-1 (Zhang et al., 2023a), Gen-2 (runway, 2023), Pika (pik, 2023), and LaVie-2 (Wang et al., 2023b). Evaluation Metrics. To evaluate the text-to-video generation, we employed several metrics from VBench (Huang et al., 2024): Human Action, Scene, Dynamic Degree, Multiple Objects, and Appearance Style. VBench is a suite of tools designed to automatically assess the quality of generated videos. We have selected certain metrics from VBench, excluding others that do not align with our evaluation needs. For example, the color metric, intended to measure the presence of objects corresponding to specific colors across frames in the generated video, assesses the model’s quality by calculating the probability. However, this metric may mislead video generation models that exhibit greater variation, thus it is not to include it in our evaluation. For longer-generated videos, some models might produce videos with minimal changes between frames to obtain higher scores, but these videos lack rich content. Therefore, a metric for evaluating the dynamism of the video becomes more important. To address this, we employ two video evaluation tools: Dynamic Quality from Devil (Liao et al., 2024) and GPT4o-MTScore from ChronoMagic (Yuan et al., 2024), which focus more on the dynamic characteristics of videos. Dynamic Quality is defined by the integration of various quality metrics with dynamic scores, mitigating biases arising from negative correlations between video dynamics and video quality. ChronoMagic, for instance, 9  Table 1: Evaluation results. Human Dynamic Multiple Appearance Dynamic GPT4o-MT Models Scene Action Degree Objects Style Quality Score T2V-Turbo 95.2 55.58 49.17 54.65 24.42 – – AnimateDiff 92.6 50.19 40.83 36.88 22.42 – 2.62 VideoCrafter-2.0 95.0 55.29 42.50 40.66 25.13 43.6 2.68 OpenSora V1.2 85.8 42.47 47.22 58.41 23.89 63.7 2.52 Show-1 95.6 47.03 44.44 45.47 23.06 57.7 – Gen-2 89.2 48.91 18.89 55.47 19.34 43.6 2.62 Pika 88.0 44.80 37.22 46.69 21.89 52.1 2.48 LaVie-2 96.4 49.59 31.11 64.88 25.09 – 2.46 CogVideoX-2B 88.0 39.94 63.33 53.70 23.67 57.7 3.09 CogVideoX-5B 96.8 55.44 62.22 70.95 24.44 69.5 3.36 introduces GPT4o-MTScore, a metric designed to measure the metamorphic amplitude of time-lapse videos, such as those depicting physical, biological, and meteorological changes. This metric using GPT-4o (OpenAI, 2024a) to score the degree of change, providing a fine-grained assessment of video dynamism. Results. Table 1 provides the performance comparison of CogVideoX and other models. CogVideoX achieves the best performance in five out of the seven metrics and shows competi- tive results in the remaining two metrics. These results demonstrate that the model not only excels in video generation quality but also outperforms previous models in handling various complex dynamic scenes. In addition, Figure 1 presents a radar chart that visually illustrates the performance advantages of CogVideoX. 4.2 Human Evaluation In addition to automated scoring mechanisms, a comparative analysis between the Kling (Team, 2024) and CogVideoX is conducted with manual evaluation. One hundred meticulously crafted prompts are used for human evaluators, characterized by their broad distribution, clear articulation, and well-defined conceptual scope. We randomize videos for blind evalution. A panel of evaluators is instructed to assign scores for each detail on a scale from zero to one, with the overall total score rated on a scale from 0 to 5, where higher scores reflect better video quality. To better complement automated evaluation, human evaluation emphasizes the instruction-following capability: the total score cannot exceed 2 if the generated video fails to follow the instruction. The results shown in Table 2 indicate that CogVideoX wins the human preference over Kling across all aspects. More details about human evaluation are shown in Appendix F. Table 2: Human evaluation between CogVideoX and Kling. Sensory Instruction Physics Cover Total Model Quality Following Simulation Quality Score Kling 0.638 0.367 0.561 0.668 2.17 CogVideoX-5B 0.722 0.495 0.667 0.712 2.74 5 Conclusion In this paper, we present CogVideoX, a state-of-the-art text-to-video diffusion model. It leverages a 3D VAE and an Expert Transformer architecture to generate coherent long-duration videos with significant motion. By implementing a comprehensive data processing pipeline and a video re- captioning method, we significantly improve the quality and semantic alignment of the generated videos. Our progressive training techniques, including mixed-duration training and resolution progressive training, further enhance the model’s performance and stability. Our ongoing efforts focus on refining the CogVideoX’s ability to capture complex dynamics and ensure even higher 10 quality in video generation. We are also exploring the scaling laws of video generation models and aim to train larger and more powerful models to generate longer and higher-quality videos, pushing the boundaries of what is achievable in text-to-video generation. Acknowledgments We would like to thank all the data annotators, infrastructure operators, collaborators, and partners. We also extend our gratitude to everyone at Zhipu AI and Tsinghua University who have provided support, feedback, or contributed to the CogVideoX, even if not explicitly mentioned in this report. We would also like to greatly thank BiliBili for technical discussions. References Pika beta. 2023. URL https://pika.art/home. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023. AI@Meta. Llama 3 model card. 2024. URL https://github.com/meta-llama/llama3/blob/ main/MODEL_CARD.md. Yushi Bai, Xin Lv, Jiajie Zhang, Yuze He, Ji Qi, Lei Hou, Jie Tang, Yuxiao Dong, and Juanzi Li. Longalign: A recipe for long context alignment of large language models. arXiv preprint arXiv:2401.18058, 2024. Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman. Frozen in time: A joint video and image encoder for end-to-end retrieval. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 1728–1738, 2021. James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, et al. Improving image generation with better captions. Computer Science. https://cdn. openai. com/papers/dall-e-3. pdf, 2(3):8, 2023. Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15127, 2023. Haoxin Chen, Yong Zhang, Xiaodong Cun, Menghan Xia, Xintao Wang, Chao Weng, and Ying Shan. Videocrafter2: Overcoming data limitations for high-quality video diffusion models, 2024a. Tsai-Shien Chen, Aliaksandr Siarohin, Willi Menapace, Ekaterina Deyneka, Hsiang-wei Chao, Byung Eun Jeon, Yuwei Fang, Hsin-Ying Lee, Jian Ren, Ming-Hsuan Yang, et al. Panda-70m: Captioning 70m videos with multiple cross-modality teachers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13320–13331, 2024b. Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory- efficient exact attention with io-awareness. Advances in Neural Information Processing Systems, 35:16344–16359, 2022. Mostafa Dehghani, Basil Mustafa, Josip Djolonga, Jonathan Heek, Matthias Minderer, Mathilde Caron, Andreas Steiner, Joan Puigcerver, Robert Geirhos, Ibrahim M Alabdulmohsin, et al. Patch n’pack: Navit, a vision transformer for any aspect ratio and resolution. Advances in Neural Information Processing Systems, 36, 2024. Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 12873–12883, 2021. Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first International Conference on Machine Learning, 2024. 11 Yuwei Guo, Ceyuan Yang, Anyi Rao, Zhengyang Liang, Yaohui Wang, Yu Qiao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text-to-image diffusion models without specific tuning. arXiv preprint arXiv:2307.04725, 2023. Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840–6851, 2020. Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, et al. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:2210.02303, 2022. Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang. Cogvideo: Large-scale pretraining for text-to-video generation via transformers. arXiv preprint arXiv:2205.15868, 2022. Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianxing Wu, Qingyang Jin, Nattapol Chanpaisit, Yaohui Wang, Xinyuan Chen, Limin Wang, Dahua Lin, Yu Qiao, and Ziwei Liu. VBench: Comprehensive benchmark suite for video generative models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024. Jiachen Li, Weixi Feng, Tsu-Jui Fu, Xinyi Wang, Sugato Basu, Wenhu Chen, and William Yang Wang. T2v-turbo: Breaking the quality bottleneck of video consistency model with mixed reward feedback. arXiv preprint arXiv:2405.18750, 2024. Mingxiang Liao, Hannan Lu, Xinyu Zhang, Fang Wan, Tianyu Wang, Yuzhong Zhao, Wangmeng Zuo, Qixiang Ye, and Jingdong Wang. Evaluation of text-to-video generation models: A dynamics perspective, 2024. URL https://arxiv.org/abs/2407.01094. Shanchuan Lin, Bingchen Liu, Jiashi Li, and Xiao Yang. Common diffusion noise schedules and sample steps are flawed. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pp. 5404–5411, 2024. Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision– ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp. 740–755. Springer, 2014. OpenAI. Gpt-4o. 2024a. OpenAI. Sora. 2024b. URL https://openai.com/index/sora/. William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4195–4205, 2023. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1–67, 2020. Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 10684–10695, 2022. runway. Gen-2. 2023. URL https://runwayml.com/ai-tools/gen-2-text-to-video. Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. arXiv preprint arXiv:2202.00512, 2022. Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et al. Make-a-video: Text-to-video generation without text-video data. arXiv preprint arXiv:2209.14792, 2022. Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063, 2024. Kuaishou AI Team. Kling. 2024. URL https://kling.kuaishou.com/en. 12 Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017. Ruben Villegas, Mohammad Babaeizadeh, Pieter-Jan Kindermans, Hernan Moraldo, Han Zhang, Mohammad Taghi Saffar, Santiago Castro, Julius Kunze, and Dumitru Erhan. Phenaki: Variable length video generation from open domain textual descriptions. In International Conference on Learning Representations, 2022. Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, et al. Cogvlm: Visual expert for pretrained language models. arXiv preprint arXiv:2311.03079, 2023a. Yaohui Wang, Xinyuan Chen, Xin Ma, Shangchen Zhou, Ziqi Huang, Yi Wang, Ceyuan Yang, Yinan He, Jiashuo Yu, Peiqing Yang, et al. Lavie: High-quality video generation with cascaded latent diffusion models. arXiv preprint arXiv:2309.15103, 2023b. Wenhan Xiong, Jingyu Liu, Igor Molybog, Hejia Zhang, Prajjwal Bhargava, Rui Hou, Louis Martin, Rashi Rungta, Karthik Abinav Sankararaman, Barlas Oguz, et al. Effective long-context scaling of foundation models. arXiv preprint arXiv:2309.16039, 2023. Lijun Yu, José Lezama, Nitesh B Gundavarapu, Luca Versari, Kihyuk Sohn, David Minnen, Yong Cheng, Agrim Gupta, Xiuye Gu, Alexander G Hauptmann, et al. Language model beats diffusion– tokenizer is key to visual generation. arXiv preprint arXiv:2310.05737, 2023. Shenghai Yuan, Jinfa Huang, Yongqi Xu, Yaoyang Liu, Shaofeng Zhang, Yujun Shi, Ruijie Zhu, Xinhua Cheng, Jiebo Luo, and Li Yuan. Chronomagic-bench: A benchmark for metamorphic evaluation of text-to-time-lapse video generation. arXiv preprint arXiv:2406.18522, 2024. David Junhao Zhang, Jay Zhangjie Wu, Jia-Wei Liu, Rui Zhao, Lingmin Ran, Yuchao Gu, Difei Gao, and Mike Zheng Shou. Show-1: Marrying pixel and latent diffusion models for text-to-video generation. arXiv preprint arXiv:2309.15818, 2023a. Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction-tuned audio-visual language model for video understanding. arXiv preprint arXiv:2306.02858, 2023b. Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 586–595, 2018. Wendi Zheng, Jiayan Teng, Zhuoyi Yang, Weihan Wang, Jidong Chen, Xiaotao Gu, Yuxiao Dong, Ming Ding, and Jie Tang. Cogview3: Finer and faster text-to-image generation via relay diffusion. arXiv preprint arXiv:2403.05121, 2024a. Zangwei Zheng, Xiangyu Peng, Tianji Yang, Chenhui Shen, Shenggui Li, Hongxin Liu, Yukun Zhou, Tianyi Li, and Yang You. Open-sora: Democratizing efficient video production for all, March 2024b. URL https://github.com/hpcaitech/Open-Sora. 13 Figure 9: Text to video showcases. The displayed prompt will be upsampled before being fed into the model. The generated videos contain large motion and can produce various video styles. 14 Figure 10: Text to video showcases. 15 A Image To Video Model We finetune an image-to-video model from the text-to-video model. Drawing from the (Blattmann et al., 2023), we add an image as an additional condition alongside the text. The image is passed through 3D VAE and concatenated with the noised input in the channel dimension. Similar to super-resolution tasks, there is a significant distribution gap between training and inference( the first frame of videos vs. real-world images). To enhance the model’s robustness, we add large noise to the image condition during training. Some examples are shown in 11, 12. CogVideoX can handle different styles of image input. 16 Figure 11: Image to video showcases. The displayed prompt will be upsampled before being fed into the model. 17 Figure 12: Image to video showcases. 18 B Caption Upsampler To ensure that text input distribution during inference is as close as possible to the distribution during training, similar to (Betker et al., 2023), we use a large language model to upsample the user’s input during inference, making it more detailed and precise. Finetuned LLM can generate better prompts than zero/few-shot. For image-to-video, we use the vision language model to upsample the prompt, such as GPT4V, CogVLM(Wang et al., 2023a). Zero-shot prompt for Text Upsampler You are part of a team of bots that create videos. You work with an assistant bot that will draw anything you say in square brackets. For example, outputting \" a beautiful morning in the woods with the sun peaking through the trees \" will trigger your partner bot to output a video of a forest morning, as described. You will be prompted by people looking to create detailed, amazing videos. The way to accomplish this is to take their short prompts and make them extremely detailed and descriptive. There are a few rules to follow : You will only ever output a single video description per user request. When modifications are requested, you should not simply make the description longer. You should refactor the entire description to integrate the suggestions. C Dense Video Caption Data Generation In the pipeline for generating video captions, we extract one frame every two seconds for image captioning. Ultimately, we collected 50,000 data points to fine-tune the summary model. Below is the prompt we used for summarization with GPT-4: Prompt for GPT-4 Summary We extracted several frames from this video and described each frame using an image understanding model, stored in the dictionary variable ‘image_captions: Dict[str: str]‘. In ‘image_captions‘, the key is the second at which the image appears in the video, and the value is a detailed description of the image at that moment. Please describe the content of this video in as much detail as possible, based on the information provided by ‘image_captions‘, including the objects, scenery, animals, characters, and camera movements within the video. \n image_captions={new_captions}\n You should output your summary directly, and not mention variables like ‘image_captions‘ in your response. Do not include ‘\\n’ and the word ’video’ in your response. Do not use introductory phrases such as: \"The video presents\", \"The video depicts\", \"This video showcases\", \"The video captures\" and so on.\n Please start the description with the video content directly, such as \"A man first sits in a chair, then stands up and walks to the kitchen....\"\n Do not use phrases like: \"as the video progressed\" and \"Throughout the video\".\n Please describe 19  the content of the video and the changes that occur, in chronological order.\n Please keep the description of this video within 100 English words. D Video Caption Example Below we present some examples to compare the performance of the Panda-70M video captioning model and our CogVLM2-Caption model: Caption Generated by Panda-70M A close up of a white dragon with blue eyes. Caption Generated by CogVLM2-Caption A meticulously crafted white dragon with a serene expression and piercing blue eyes is showcased in various close-ups, its scales detailed with golden highlights. Initially, the dragon’s horns are curved back, but as time passes, they change to spiraling gold, adding elegance. The creature’s demeanor shifts from tranquil majesty to a more formidable presence, with its mouth agape revealing sharp teeth at times. Its expressions alternate between calmness and fierce determination, maintaining an air of mythical serenity throughout. The consistent neutral background focuses attention on the dragon’s textured features and expressive eyes. Caption Generated by Panda-70M A crab is walking on the beach with a light bulb on its back. Caption Generated by CogVLM2-Caption A hermit crab with a dark, glossy shell and reddish-brown legs is seen carrying an illuminated light bulb on its back across the sandy terrain of a beach at night. The scene transitions from a soft glow to a more pronounced illumination as the crab moves, highlighting its delicate limbs against the smooth sand 20 and tranquil sea backdrop. This surreal tableau blends natural beauty with human ingenuity, creating a serene yet whimsical atmosphere that emphasizes the crab’s unique adaptation and the contrast between nature and technology in this quiet nocturnal setting. Caption Generated by Panda-70M A young black man is sitting on a cloud and reading a book with a blue sky in the background. Caption Generated by CogVLM2-Caption A young Black man with an afro hairstyle and a neatly trimmed beard is seen sitting cross-legged on fluffy white clouds, deeply engrossed in reading a book with a red cover. He wears a plain white T-shirt and dark pants against a vivid blue sky dotted with cumulus clouds. Throughout the scenes, his expression remains one of deep concentration and peaceful contemplation, highlighting a moment of intellectual pursuit amidst nature’s grandeur. The imagery suggests a serene atmosphere that emphasizes solitude and introspection, with no other people or objects around him. 21 E Video to Video via CogVideoX and CogVLM2-Caption In this section, we present several examples of video-to-video generation using CogVideoX and CogVLM2-Caption. Specifically, we first input the original video into CogVLM2-Caption to obtain the video’s caption, and then feed this caption into the CogVideoX model to generate a new video. From the examples below, it can be seen that our pipeline achieves a high degree of fidelity to the original video: 22 23 F Human Evaluation Details Sensory Quality: This part focuses mainly on the perceptual quality of videos, including subject consistency, frame continuity, and stability. Table 3: Sensory Quality Evaluation Criteria. Score Evaluation Criteria 1 High sensory quality: 1. The appearance and morphological features of objects in the video are completely consistent 2. High picture stability, maintaining high resolution consistently 3. Overall composition/color/boundaries match reality 4. The picture is visually appealing 0.5 Average sensory quality: 1. The appearance and morphological features of objects in the video are at least 80% consistent 2. Moderate picture stability, with only 50% of the frames maintaining high resolution 3. Overall composition/color/boundaries match reality by at least 70% 4. The picture has some visual appeal 0 Poor sensory quality: large inconsistencies in appearance and morphology, low video resolution, and composition/layout not matching reality Instruction Following: This part focuses on whether the generated video aligns with the prompt, including the accuracy of the subject, quantity, elements, and details. Table 4: Instruction Following Evaluation Criteria. Score Evaluation Criteria 1 100% follow the text instruction requirements, including but not limited to: elements completely correct, quantity requirements consistent, elements complete, features accu- rate, etc. 0.5 100% follow the text instruction requirements, but the implementation has minor flaws such as distorted main subjects or inaccurate features. 0 Does not 100% follow the text instruction requirements, with any of the following issues: 1. Generated elements are inaccurate 2. Quantity is incorrect 3. Elements are incomplete 4. Features are inaccurate Physics Simulation: This part focuses on whether the model can adhere to the objective law of the physical world, such as the lighting effect, interactions between different objects, and the realism of fluid dynamics. Table 5: Physics Simulation Evaluation Criteria. Score Evaluation Criteria 1 Good physical realism simulation capability, can achieve: 1. Real-time tracking 2. Good action understanding, ensuring dynamic realism of entities 3. Realistic lighting and shadow effects, high interaction fidelity 4. Accurate simulation of fluid motion 0.5 Average physical realism simulation capability, with some degradation in real-time track- ing, dynamic realism, lighting and shadow effects, and fluid motion simulation. Issues include: 1. Slightly unnatural transitions in dynamic effects, with some discontinuities 2. Lighting and shadow effects not matching reality 3. Distorted interactions between objects 4. Floating fluid motion, not matching reality 0 Poor physical realism simulation capability, results do not match reality, obviously fake 24 Cover Quality: This part mainly focuses on metrics that can be assessed from single-frame images, including aesthetic quality, clarity, and fidelity. Table 6: Cover Quality Evaluation Criteria. Score Evaluation Criteria 1 Image is clear, subject is obvious, display is complete, color tone is normal. 0.5 Image quality is average. The subject is relatively complete, color tone is normal. 0 Cover image resolution is low, image is blurry. 25