Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models
NVIDIA Research has developed a new method for generating high-resolution videos with latent diffusion models. The method, called Video LDM, first generates sparse keyframes at low frame rates. These keyframes are then temporally upsampled twice by another interpolation latent diffusion model. This process results in temporally coherent videos at high resolution.
In addition to generating high-resolution videos, Video LDM can also be used for video prediction. By conditioning on starting frames, Video LDM can generate long videos in an autoregressive manner. This makes it possible to create videos that are both realistic and temporally consistent.
Video LDM has been shown to achieve state-of-the-art performance on both driving video synthesis and text-to-video tasks. For driving video synthesis, Video LDM can generate temporally coherent, multiple minute long videos at resolution 512x1024. For text-to-video, Video LDM can synthesize short videos of several seconds lengths with resolution up to 1280x2048.
Video LDM is a powerful new tool for generating high-resolution videos. It is capable of generating realistic and temporally consistent videos for a variety of tasks. Video LDM is a promising new technology that has the potential to revolutionize the way we create and consume videos.
Here are some of the key features of Video LDM:
It can generate high-resolution videos at up to 1280x2048 resolution.
It can generate temporally coherent videos.
It can be used for both driving video synthesis and text-to-video tasks.
It has been shown to achieve state-of-the-art performance on both tasks.
Video LDM is a powerful new tool for generating high-resolution videos. It is capable of generating realistic and temporally consistent videos for a variety of tasks. Video LDM is a promising new technology that has the potential to revolutionize the way we create and consume videos.