AnimateDiff 是什么

AnimateDiff GitHub - guoyww/AnimateDiff: Official implementation of AnimateDiff. AnimateDiff 是一个基于扩散模型实现从文本生成图像动画化的机器学习方案.

Animatediff 安装

前置条件

Git
Conda
cmder Windows Terminal 开发环境 - 掘金 (juejin.cn)

Animatediff 项目环境

下载 Animatediff 源码

git clone --depth 1  https://github.com/guoyww/AnimateDiff.git

创建 conda python 虚拟环境

conda env create -f environment.yaml
conda activate animatediff

如果安装提示 xformers 无法安装, 需要手动修改项目根目录下的environment.yaml文件, 移动 xformers 到 pip: 后面
如果 torch｜torchvision｜torchaudio 安装的是非 GPU 版本, 卸载重新安装 GPU 版本, 否则无法以 GPU 方式运行.

pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118

[可选方式]在已有虚拟环境中安装依赖新建 requirements_versions.txt, 编辑 python 依赖清单, 使用 pip 安装依赖

torch==1.12.1
torchvision==0.13.1
torchaudio==0.12.1
xformers
diffusers[torch]==0.11.1
transformers==4.25.1
imageio==2.27.0
gdown
einops
omegaconf
safetensors

pip install -e requirements_versions.txt

Animatediff 模型

StableDiffusion: 从 huggingface下载基础模型到 models/StableDiffusion 目录,大小 37.6GB.

git lfs install
git clone https://huggingface.co/runwayml/stable-diffusion-v1-5 models/StableDiffusion/

Motion_Module: 通过download_basescript/0-MotionModule.sh 脚本下载模型到 models/Motion_Module 目录. 大小 3.11GB

bash download_bashscripts/0-MotionModule.sh

DreamBooth_LoRA: 通过 download_basescript 中其余 8 个脚本下载模型到 models/DreamBooth_LoRA 目录, 大小 27.6GB

bash download_bashscripts/1-ToonYou.sh
bash download_bashscripts/2-Lyriel.sh
bash download_bashscripts/3-RcnzCartoon.sh
bash download_bashscripts/4-MajicMix.sh
bash download_bashscripts/5-RealisticVision.sh
bash download_bashscripts/6-Tusun.sh
bash download_bashscripts/7-FilmVelvia.sh
bash download_bashscripts/8-GhibliBackground.sh

Animatediff 运行

使用不同的模型，需要加载不同的yaml文件，官方内置了8个Prompt文件，可以帮你快速生成想要的内容。下面使用 1-ToonYou.yaml 为例, 在执行后会在 samples 目录下生成 gif 图片

λ python -m scripts.animate --config configs/prompts/1-ToonYou.yaml --pretrained_model_path models/StableDiffusion --L 16 --W
512 --H 512
loaded temporal unet's pretrained weights from models/StableDiffusion\unet ...
### missing keys: 560;
### unexpected keys: 0;
### Temporal Module Parameters: 417.1376 M
Some weights of the model checkpoint at openai/clip-vit-large-patch14 were not used when initializing CLIPTextModel:

sampling best quality, masterpiece, 1girl, looking at viewer, blurry background, upper body, contemporary, dress ...
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████| 25/25 [08:44<00:00, 21.00s/it] 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████| 16/16 [00:03<00:00,  4.65it/s] save to samples/1-ToonYou-2023-08-13T21-00-19/sample/best-quality,-masterpiece,-1girl,-looking-at-viewer,-blurry-background,-upper.gif
current seed: 6520604954829636163
sampling masterpiece, best quality, 1girl, solo, cherry blossoms, hanami, pink flower, white flower, spring season, wisteria, petals, flower, plum blossoms, outdoors, falling petals, white hair, black eyes, ...
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████| 25/25 [08:48<00:00, 21.13s/it] 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████| 16/16 [00:03<00:00,  4.69it/s] save to samples/1-ToonYou-2023-08-13T21-00-19/sample/masterpiece,-best-quality,-1girl,-solo,-cherry-blossoms,-hanami,-pink-flower,.gif
current seed: 6519455744612555650
sampling best quality, masterpiece, 1boy, formal, abstract, looking at viewer, masculine, marble pattern ...
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████| 25/25 [08:48<00:00, 21.14s/it] 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████| 16/16 [00:03<00:00,  4.66it/s] save to samples/1-ToonYou-2023-08-13T21-00-19/sample/best-quality,-masterpiece,-1boy,-formal,-abstract,-looking-at-viewer,-masculine,.gif
current seed: 16372571278361863751
sampling best quality, masterpiece, 1girl, cloudy sky, dandelion, contrapposto, alternate hairstyle, ...
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████| 25/25 [1:05:09<00:00, 156.38s/it] 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████| 16/16 [00:03<00:00,  4.72it/s] save to samples/1-ToonYou-2023-08-13T21-00-19/sample/best-quality,-masterpiece,-1girl,-cloudy-sky,-dandelion,-contrapposto,-alternate-hairstyle,.gif

0-best-quality,-masterpiece,-1girl,-looking-at-viewer,-blurry-background,-upper.gif

3-best-quality,-masterpiece,-1girl,-cloudy-sky,-dandelion,-contrapposto,-alternate-hairstyle,.gif

windows 下安装 AnimateDiff