[Model] SD + AnimateDiff 图片转视频AnimateDiff 是一个基于扩散模型实现从文本生成图像

AnimateDiff 是什么

AnimateDiff

GitHub - guoyww/AnimateDiff: Official implementation of AnimateDiff. AnimateDiff 是一个基于扩散模型实现从文本生成图像动画化的机器学习方案.

开源AI文本生成视频AnimateDiff模型下载安装使用教程-openAI维基百科

Animatediff 安装

前置条件

Git
Conda
cmder Windows Terminal 开发环境 - 掘金 (juejin.cn)

Model 下载

guoyww/animatediff at main (hf-mirror.com)

AnimateDiff v3 Model Zoo

Name	HuggingFace	Type	Storage Space	Description
`v3_adapter_sd_v15.ckpt`	Link	Domain Adapter	97.4 MB
`v3_sd15_mm.ckpt.ckpt`	Link	Motion Module	1.56 GB
`v3_sd15_sparsectrl_scribble.ckpt`	Link	SparseCtrl Encoder	1.86 GB	scribble condition
`v3_sd15_sparsectrl_rgb.ckpt`	Link	SparseCtrl Encoder	1.85 GB	RGB image condition

AnimateDiff SDXL-Beta Model Zoo

Name	HuggingFace	Type	Storage Space
`mm_sdxl_v10_beta.ckpt`	Link	Motion Module	950 MB

AnimateDiff v2 Model Zoo

Name	HuggingFace	Type	Parameter	Storage Space
mm_sd_v15_v2.ckpt	Link	Motion Module	453 M	1.7 GB
v2_lora_ZoomIn.ckpt	Link	MotionLoRA	19 M	74 MB
v2_lora_ZoomOut.ckpt	Link	MotionLoRA	19 M	74 MB
v2_lora_PanLeft.ckpt	Link	MotionLoRA	19 M	74 MB
v2_lora_PanRight.ckpt	Link	MotionLoRA	19 M	74 MB
v2_lora_TiltUp.ckpt	Link	MotionLoRA	19 M	74 MB
v2_lora_TiltDown.ckpt	Link	MotionLoRA	19 M	74 MB
v2_lora_RollingClockwise.ckpt	Link	MotionLoRA	19 M	74 MB
v2_lora_RollingAnticlockwise.ckpt	Link	MotionLoRA	19 M	74 MB

AnimateDiff v1 Model Zoo

Name	HuggingFace	Parameter	Storage Space
mm_sd_v14.ckpt	Link	417 M	1.6 GB
mm_sd_v15.ckpt	Link	417 M	1.6 GB

Animatediff 项目环境

下载 Animatediff 源码

git clone --depth 1  https://github.com/guoyww/AnimateDiff.git

创建 conda python 虚拟环境

conda env create -f environment.yaml
conda activate animatediff

如果安装提示 xformers 无法安装, 需要手动修改项目根目录下的environment.yaml文件, 移动 xformers 到 pip: 后面
如果 torch｜torchvision｜torchaudio 安装的是非 GPU 版本, 卸载重新安装 GPU 版本, 否则无法以 GPU 方式运行.

pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118

[可选方式]在已有虚拟环境中安装依赖新建 requirements_versions.txt, 编辑 python 依赖清单, 使用 pip 安装依赖

torch==1.12.1
torchvision==0.13.1
torchaudio==0.12.1
xformers
diffusers[torch]==0.11.1
transformers==4.25.1
imageio==2.27.0
gdown
einops
omegaconf
safetensors

pip install -e requirements_versions.txt

Animatediff 模型

StableDiffusion: 从 huggingface下载基础模型到 models/StableDiffusion 目录,大小 37.6GB.

git lfs install
git clone https://huggingface.co/runwayml/stable-diffusion-v1-5 models/StableDiffusion/

Motion_Module: 通过download_basescript/0-MotionModule.sh 脚本下载模型到 models/Motion_Module 目录. 大小 3.11GB

bash download_bashscripts/0-MotionModule.sh

DreamBooth_LoRA: 通过 download_basescript 中其余 8 个脚本下载模型到 models/DreamBooth_LoRA 目录, 大小 27.6GB

bash download_bashscripts/1-ToonYou.sh
bash download_bashscripts/2-Lyriel.sh
bash download_bashscripts/3-RcnzCartoon.sh
bash download_bashscripts/4-MajicMix.sh
bash download_bashscripts/5-RealisticVision.sh
bash download_bashscripts/6-Tusun.sh
bash download_bashscripts/7-FilmVelvia.sh
bash download_bashscripts/8-GhibliBackground.sh

Animatediff 运行

使用不同的模型，需要加载不同的yaml文件，官方内置了8个Prompt文件，可以帮你快速生成想要的内容。下面使用 1-ToonYou.yaml 为例, 在执行后会在 samples 目录下生成 gif 图片

λ python -m scripts.animate --config configs/prompts/1-ToonYou.yaml --pretrained_model_path models/StableDiffusion --L 16 --W
512 --H 512
loaded temporal unet's pretrained weights from models/StableDiffusion\unet ...
### missing keys: 560;
### unexpected keys: 0;
### Temporal Module Parameters: 417.1376 M
Some weights of the model checkpoint at openai/clip-vit-large-patch14 were not used when initializing CLIPTextModel:

sampling best quality, masterpiece, 1girl, looking at viewer, blurry background, upper body, contemporary, dress ...
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████| 25/25 [08:44<00:00, 21.00s/it] 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████| 16/16 [00:03<00:00,  4.65it/s] save to samples/1-ToonYou-2023-08-13T21-00-19/sample/best-quality,-masterpiece,-1girl,-looking-at-viewer,-blurry-background,-upper.gif
current seed: 6520604954829636163
sampling masterpiece, best quality, 1girl, solo, cherry blossoms, hanami, pink flower, white flower, spring season, wisteria, petals, flower, plum blossoms, outdoors, falling petals, white hair, black eyes, ...
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████| 25/25 [08:48<00:00, 21.13s/it] 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████| 16/16 [00:03<00:00,  4.69it/s] save to samples/1-ToonYou-2023-08-13T21-00-19/sample/masterpiece,-best-quality,-1girl,-solo,-cherry-blossoms,-hanami,-pink-flower,.gif
current seed: 6519455744612555650
sampling best quality, masterpiece, 1boy, formal, abstract, looking at viewer, masculine, marble pattern ...
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████| 25/25 [08:48<00:00, 21.14s/it] 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████| 16/16 [00:03<00:00,  4.66it/s] save to samples/1-ToonYou-2023-08-13T21-00-19/sample/best-quality,-masterpiece,-1boy,-formal,-abstract,-looking-at-viewer,-masculine,.gif
current seed: 16372571278361863751
sampling best quality, masterpiece, 1girl, cloudy sky, dandelion, contrapposto, alternate hairstyle, ...
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████| 25/25 [1:05:09<00:00, 156.38s/it] 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████| 16/16 [00:03<00:00,  4.72it/s] save to samples/1-ToonYou-2023-08-13T21-00-19/sample/best-quality,-masterpiece,-1girl,-cloudy-sky,-dandelion,-contrapposto,-alternate-hairstyle,.gif

0-best-quality,-masterpiece,-1girl,-looking-at-viewer,-blurry-background,-upper.gif

3-best-quality,-masterpiece,-1girl,-cloudy-sky,-dandelion,-contrapposto,-alternate-hairstyle,.gif

SD Animatediff 插件

GitHub - continue-revolution/sd-webui-animatediff: AnimateDiff for AUTOMATIC1111 Stable Diffusion WebUI

2024-03-31 02:13:02,518 - AnimateDiff - INFO - AnimateDiff process start.                                                            
2024-03-31 02:13:02,518 - AnimateDiff - INFO - Loading motion module mm_sd_v15_v2.ckpt from E:\vendor\cv\stablediffusion\sdwebui\sdwe
bui-src\extensions\sd-webui-animatediff\model\mm_sd_v15_v2.ckpt                                                                      
Calculating sha256 for E:\vendor\cv\stablediffusion\sdwebui\sdwebui-src\extensions\sd-webui-animatediff\model\mm_sd_v15_v2.ckpt: 69ed
0f5fef82b110aca51bcab73b21104242bc65d6ab4b8b2a2a94d31cad1bf0                                                                         
2024-03-31 02:13:06,006 - AnimateDiff - INFO - Guessed mm_sd_v15_v2.ckpt architecture: MotionModuleType.AnimateDiffV2                
2024-03-31 02:13:10,378 - AnimateDiff - INFO - Injecting motion module mm_sd_v15_v2.ckpt into SD1.5 UNet middle block.               
2024-03-31 02:13:10,378 - AnimateDiff - INFO - Injecting motion module mm_sd_v15_v2.ckpt into SD1.5 UNet input blocks.               
2024-03-31 02:13:10,379 - AnimateDiff - INFO - Injecting motion module mm_sd_v15_v2.ckpt into SD1.5 UNet output blocks.              
2024-03-31 02:13:10,379 - AnimateDiff - INFO - Setting DDIM alpha.                                                                   
2024-03-31 02:13:10,400 - AnimateDiff - INFO - Injection finished.                                                                   
2024-03-31 02:13:10,400 - AnimateDiff - INFO - AnimateDiff + ControlNet will generate 16 frames.                                     
INFO:sd_dynamic_prompts.dynamic_prompting:Prompt matrix will create 16 images in a total of 1 batches.                               
  0%|                                                                                                        | 0/20 [00:00<?, ?it/s]2
024-03-31 02:13:11,265 - AnimateDiff - INFO - inner model forward hooked                                                             
100%|███████████████████████████████████████████████████████████████████████████████████████████████| 20/20 [01:01<00:00,  3.10s/it] 
2024-03-31 02:14:17,732 - AnimateDiff - INFO - Restoring DDIM alpha.████████████████████████████████| 20/20 [00:56<00:00,  2.94s/it] 
2024-03-31 02:14:17,733 - AnimateDiff - INFO - Removing motion module from SD1.5 UNet input blocks.                                  
2024-03-31 02:14:17,734 - AnimateDiff - INFO - Removing motion module from SD1.5 UNet output blocks.                                 
2024-03-31 02:14:17,734 - AnimateDiff - INFO - Removing motion module from SD1.5 UNet middle block.                                  
2024-03-31 02:14:17,735 - AnimateDiff - INFO - Removal finished.                                                                     
2024-03-31 02:14:17,741 - AnimateDiff - INFO - Saving output formats: GIF, PNG                                                       
2024-03-31 02:14:21,448 - AnimateDiff - INFO - AnimateDiff process end.                                                              
Total progress: 100%|███████████████████████████████████████████████████████████████████████████████| 20/20 [01:04<00:00,  3.22s/it] 
Total progress: 100%|███████████████████████████████████████████████████████████████████████████████| 20/20 [01:04<00:00,  2.94s/it]

[Model] SD + AnimateDiff 图片转视频