你还不会用么？MiniGemini重磅来袭

老规矩，先贴项目地址：github.com/dvlab-resea…

环境准备[这里特别容易有问题]

git clone https://github.com/dvlab-research/MiniGemini.git
conda create -n minigemini python=3.10 -y
conda activate minigemini
cd MiniGemini
pip install --upgrade pip deepspeed open-clip-torch  # enable PEP 660 support
pip install -e .
pip uninstall transformers
pip install transformers diffusers

conda install paddlepaddle-gpu==2.6.1 cudatoolkit=11.7 -c https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/Paddle/ -c conda-forge
pip install "paddleocr>=2.0.1" # Recommend to use version 2.0.1+

预训练权重下载

不需要科学上网也可以超速下载，参考链接

mkdir model_zoo
cd model_zoo
hdf.sh openai/clip-vit-large-patch14-336 --tool aria2c -x 10
hdf.sh laion/CLIP-convnext_large_d_320.laion2B-s29B-b131K-ft-soup --tool aria2c -x 10
hdf.sh --token xxxxx google/gemma-2b-it --tool aria2c -x 10
hdf.sh lmsys/vicuna-7b-v1.5 --tool aria2c -x 10
hdf.sh lmsys/vicuna-13b-v1.5 --tool aria2c -x 10
hdf.sh mistralai/Mixtral-8x7B-Instruct-v0.1 --tool aria2c -x 10
hdf.sh NousResearch/Nous-Hermes-2-Yi-34B --tool aria2c -x 10

下载需要一段时间，下载好之后调整文件夹位置

MiniGemini
├── minigemini
├── scripts
├── work_dirs
│   ├── Mini-Gemini
│   │   ├── Mini-Gemini-2B
│   │   ├── ...
├── model_zoo
│   ├── LLM
│   │   ├── gemma
│   │   │   ├── gemma-2b-it
│   │   ├── vicuna
│   │   │   ├── 7B-V1.5
│   │   │   ├── 13B-V1.5
│   │   ├── mixtral
│   │   │   ├── Mixtral-8x7B-Instruct-v0.1
│   │   ├── Nous-Hermes-2-Yi-34B
│   ├── OpenAI
│   │   ├── clip-vit-large-patch14-336
│   │   ├── openclip-convnext-large-d-320-laion2B-s29B-b131K-ft-soup

启动

启动一个controller

python -m minigemini.serve.controller --host 0.0.0.0 --port 10000

启动一个gradio web server

python -m minigemini.serve.gradio_web_server --controller http://localhost:10000 --model-list-mode reload

启动一个model worker

CUDA_VISIBLE_DEVICES=0,1 python -m minigemini.serve.model_worker --host 0.0.0.0 --controller http://localhost:10000 --port 40000 --worker http://localhost:40000 --model-path work_dirs/Mini-Gemini/Mini-Gemini-13B-HD

Enjoy