千问的github仓库被清空了，不知道具体是什么个情况
环境准备
直接使用docker来准备基础环境，可以参考教程
需要注意，在容器内，把torch版本升级到最新以避免一些torch相关的问题
pip install -U torch torchvision
transformers库当中仍有一些bug在修复中，可以直接使用对应的commit来安装
pip install -U git+https://github.com/huggingface/transformers@f95700bcda9b7d95d10d7495a377e36c48871310
安装千问工具包
pip install qwen-vl-utils
pip install accelerate>=0.26.0
运行代码
from transformers import Qwen2VLForConditionalGeneration, AutoTokenizer, AutoProcessor
from qwen_vl_utils import process_vision_info
from modelscope import snapshot_download
model_dir = snapshot_download("qwen/Qwen2-VL-7B-Instruct")
# default: Load the model on the available device(s)
model = Qwen2VLForConditionalGeneration.from_pretrained(
model_dir, torch_dtype="auto", device_map="auto"
)
# We recommend enabling flash_attention_2 for better acceleration and memory saving, especially in multi-image and video scenarios.
# model = Qwen2VLForConditionalGeneration.from_pretrained(
# model_dir,
# torch_dtype=torch.bfloat16,
# attn_implementation="flash_attention_2",
# device_map="auto",
# )
# default processer
processor = AutoProcessor.from_pretrained(model_dir)
# The default range for the number of visual tokens per image in the model is 4-16384. You can set min_pixels and max_pixels according to your needs, such as a token count range of 256-1280, to balance speed and memory usage.
# min_pixels = 256*28*28
# max_pixels = 1280*28*28
# processor = AutoProcessor.from_pretrained(model_dir, min_pixels=min_pixels, max_pixels=max_pixels)
messages = [
{
"role": "user",
"content": [
{
"type": "image",
"image": "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen-VL/assets/demo.jpeg",
},
{"type": "text", "text": "Describe this image."},
],
}
]
# Preparation for inference
text = processor.apply_chat_template(
messages, tokenize=False, add_generation_prompt=True
)
image_inputs, video_inputs = process_vision_info(messages)
inputs = processor(
text=[text],
images=image_inputs,
videos=video_inputs,
padding=True,
return_tensors="pt",
)
inputs = inputs.to("cuda")
# Inference: Generation of the output
generated_ids = model.generate(**inputs, max_new_tokens=128)
generated_ids_trimmed = [
out_ids[len(in_ids) :] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)
]
output_text = processor.batch_decode(
generated_ids_trimmed, skip_special_tokens=True, clean_up_tokenization_spaces=False
)
print(output_text)
运行结果
['The image depicts a serene beach scene with a woman and her dog enjoying a moment together. The woman is sitting on the sand, facing the ocean, and appears to be engaging in a playful activity with her dog. She is wearing a plaid shi
rt and dark pants, and her hair is long and flowing. Her dog, which seems to be a large breed, possibly a Labrador Retriever, is sitting beside her on the sand. The dog is wearing a harness and has its front paws raised, seemingly givi
ng the woman a high-five or pawing at her hand. The woman is smiling, indicating a joyful interaction.\n\nThe background']
