LLM from Huggingface Start-up - 一些有用的工具
- (下载模型)First , we need to know how to download the model. Before I download the model throught download and copy, it costs me lots of time. So please follow this tutorial to download using huggingface-cli huggingface-cli if you find this error:
Use mirrors:
export HF_ENDPOINT=https://hf-mirror.com
Then, if you want to downlaod model, you can follow this tutorial: download from china
Using this command to download model:
huggingface-cli download --resume-download (模型名称) --local-dir (模型存放位置) --local-dir-use-symlinks False --resume-download
2. (下载数据）How to download dataset from the huggingface.
First, you should login in your huggingface account following the command above.I will give you an easy example. Let us dowanload a image classification called ethz/food101 from huggingface. tape the following command:
huggingface-cli download --repo-type dataset --resume-download ethz/food101 --cache-dir . --local-dir-use-symlinks **False**
Some explainations:
- --local-dir-use-symlinks False 参数可选，因为huggingface的工具链默认会使用符号链接来存储下载的文件，导致--local-dir指定的目录中都是一些“链接文件”，真实模型则存储在~/.cache/huggingface下，如果不喜欢这个可以用 --local-dir-use-symlinks False取消这个逻辑。
- --resume-download: 这个选项意味着如果下载过程中被中断，那么再次执行命令时会从上次中断的地方继续下载，而不是从头开始。
