|
|
@@ -0,0 +1,489 @@
|
|
|
+{
|
|
|
+ "cells": [
|
|
|
+ {
|
|
|
+ "cell_type": "markdown",
|
|
|
+ "metadata": {},
|
|
|
+ "source": [
|
|
|
+ "本文介绍在 ModelScope 社区下载模型的三种方式:",
|
|
|
+ "",
|
|
|
+ "1. 使用命令行工具下载",
|
|
|
+ "2. 使用 SDK 下载",
|
|
|
+ "3. 通过 Git 下载",
|
|
|
+ "",
|
|
|
+ "# 模型下载默认存放地址",
|
|
|
+ "",
|
|
|
+ "无论是使用命令行还是ModelScope SDK,模型会下载到`~/.cache/modelscope/hub`默认路径下。如果需要修改 cache 目录,可以手动设置环境变量:`MODELSCOPE_CACHE`,完成设置后,模型将下载到该环境变量指定的目录中。",
|
|
|
+ "",
|
|
|
+ "# 使用命令行工具下载",
|
|
|
+ "",
|
|
|
+ "**`modelscope download` 参数说明**",
|
|
|
+ "",
|
|
|
+ "| 参数 | 简写 | 类型 | 默认值 | 说明 |",
|
|
|
+ "|------|------|------|--------|------|",
|
|
|
+ "| `repo_id` | - | str | - | 位置参数,仓库 ID(可选,也可通过 `--model` 指定) |",
|
|
|
+ "| `files` | - | str | - | 位置参数,指定要下载的文件(支持多个) |",
|
|
|
+ "| `--model` | - | str | None | 模型 ID(与 `--dataset` 互斥) |",
|
|
|
+ "| `--dataset` | - | str | None | 数据集 ID(与 `--model` 互斥) |",
|
|
|
+ "| `--repo-type` | - | choice | `model` | 仓库类型(model/dataset),与位置参数 repo_id 配合使用 |",
|
|
|
+ "| `--revision` | - | str | None | 版本/分支/tag |",
|
|
|
+ "| `--cache_dir` | - | str | None | 缓存目录 |",
|
|
|
+ "| `--local_dir` | - | str | None | 本地目录(优先于 cache_dir) |",
|
|
|
+ "| `--include` | - | list | None | 包含的文件 glob 模式 |",
|
|
|
+ "| `--exclude` | - | list | None | 排除的文件 glob 模式 |",
|
|
|
+ "| `--token` | - | str | None | 访问令牌(私有模型需要) |",
|
|
|
+ "| `--endpoint` | - | str | None | ModelScope 服务端点 |",
|
|
|
+ "| `--max-workers` | - | int | 默认 | 最大并发下载线程数 |",
|
|
|
+ "",
|
|
|
+ "## 使用示例",
|
|
|
+ "",
|
|
|
+ "命令示例(以[Qwen2-7B](https://www.modelscope.cn/models/Qwen/Qwen2-7b))模型为例",
|
|
|
+ "",
|
|
|
+ "### 下载整个模型repo(到默认cache地址)",
|
|
|
+ "",
|
|
|
+ "",
|
|
|
+ ""
|
|
|
+ ]
|
|
|
+ },
|
|
|
+ {
|
|
|
+ "cell_type": "code",
|
|
|
+ "execution_count": null,
|
|
|
+ "metadata": {},
|
|
|
+ "outputs": [],
|
|
|
+ "source": [
|
|
|
+ "modelscope download --model 'Qwen/Qwen2-7b'"
|
|
|
+ ]
|
|
|
+ },
|
|
|
+ {
|
|
|
+ "cell_type": "markdown",
|
|
|
+ "metadata": {},
|
|
|
+ "source": [
|
|
|
+ "",
|
|
|
+ "",
|
|
|
+ "### 下载整个模型repo到指定目录",
|
|
|
+ "",
|
|
|
+ "",
|
|
|
+ ""
|
|
|
+ ]
|
|
|
+ },
|
|
|
+ {
|
|
|
+ "cell_type": "code",
|
|
|
+ "execution_count": null,
|
|
|
+ "metadata": {},
|
|
|
+ "outputs": [],
|
|
|
+ "source": [
|
|
|
+ "modelscope download --model 'Qwen/Qwen2-7b' --local_dir 'path/to/dir'"
|
|
|
+ ]
|
|
|
+ },
|
|
|
+ {
|
|
|
+ "cell_type": "markdown",
|
|
|
+ "metadata": {},
|
|
|
+ "source": [
|
|
|
+ "",
|
|
|
+ "",
|
|
|
+ "### 指定下载单个文件(以'tokenizer.json'文件为例)",
|
|
|
+ "",
|
|
|
+ "",
|
|
|
+ ""
|
|
|
+ ]
|
|
|
+ },
|
|
|
+ {
|
|
|
+ "cell_type": "code",
|
|
|
+ "execution_count": null,
|
|
|
+ "metadata": {},
|
|
|
+ "outputs": [],
|
|
|
+ "source": [
|
|
|
+ "modelscope download --model 'Qwen/Qwen2-7b' tokenizer.json"
|
|
|
+ ]
|
|
|
+ },
|
|
|
+ {
|
|
|
+ "cell_type": "markdown",
|
|
|
+ "metadata": {},
|
|
|
+ "source": [
|
|
|
+ "",
|
|
|
+ "",
|
|
|
+ "### 指定下载多个个文件",
|
|
|
+ "",
|
|
|
+ "",
|
|
|
+ ""
|
|
|
+ ]
|
|
|
+ },
|
|
|
+ {
|
|
|
+ "cell_type": "code",
|
|
|
+ "execution_count": null,
|
|
|
+ "metadata": {},
|
|
|
+ "outputs": [],
|
|
|
+ "source": [
|
|
|
+ "modelscope download --model 'Qwen/Qwen2-7b' tokenizer.json config.json"
|
|
|
+ ]
|
|
|
+ },
|
|
|
+ {
|
|
|
+ "cell_type": "markdown",
|
|
|
+ "metadata": {},
|
|
|
+ "source": [
|
|
|
+ "",
|
|
|
+ "### 指定下载某些文件",
|
|
|
+ "",
|
|
|
+ "",
|
|
|
+ ""
|
|
|
+ ]
|
|
|
+ },
|
|
|
+ {
|
|
|
+ "cell_type": "code",
|
|
|
+ "execution_count": null,
|
|
|
+ "metadata": {},
|
|
|
+ "outputs": [],
|
|
|
+ "source": [
|
|
|
+ "modelscope download --model 'Qwen/Qwen2-7b' --include '*.safetensors'"
|
|
|
+ ]
|
|
|
+ },
|
|
|
+ {
|
|
|
+ "cell_type": "markdown",
|
|
|
+ "metadata": {},
|
|
|
+ "source": [
|
|
|
+ "",
|
|
|
+ "### 过滤指定文件",
|
|
|
+ "",
|
|
|
+ "",
|
|
|
+ ""
|
|
|
+ ]
|
|
|
+ },
|
|
|
+ {
|
|
|
+ "cell_type": "code",
|
|
|
+ "execution_count": null,
|
|
|
+ "metadata": {},
|
|
|
+ "outputs": [],
|
|
|
+ "source": [
|
|
|
+ "modelscope download --model 'Qwen/Qwen2-7b' --exclude '*.safetensors'"
|
|
|
+ ]
|
|
|
+ },
|
|
|
+ {
|
|
|
+ "cell_type": "markdown",
|
|
|
+ "metadata": {},
|
|
|
+ "source": [
|
|
|
+ "",
|
|
|
+ "### 指定下载cache\\_dir",
|
|
|
+ "",
|
|
|
+ "",
|
|
|
+ ""
|
|
|
+ ]
|
|
|
+ },
|
|
|
+ {
|
|
|
+ "cell_type": "code",
|
|
|
+ "execution_count": null,
|
|
|
+ "metadata": {},
|
|
|
+ "outputs": [],
|
|
|
+ "source": [
|
|
|
+ "modelscope download --model 'Qwen/Qwen2-7b' --include '*.json' --cache_dir './cache_dir'"
|
|
|
+ ]
|
|
|
+ },
|
|
|
+ {
|
|
|
+ "cell_type": "markdown",
|
|
|
+ "metadata": {},
|
|
|
+ "source": [
|
|
|
+ "",
|
|
|
+ "模型文件将被下载到`'cache_dir/Qwen/Qwen2-7b'`。",
|
|
|
+ "",
|
|
|
+ "### 指定下载local\\_dir",
|
|
|
+ "",
|
|
|
+ "",
|
|
|
+ ""
|
|
|
+ ]
|
|
|
+ },
|
|
|
+ {
|
|
|
+ "cell_type": "code",
|
|
|
+ "execution_count": null,
|
|
|
+ "metadata": {},
|
|
|
+ "outputs": [],
|
|
|
+ "source": [
|
|
|
+ "modelscope download --model 'Qwen/Qwen2-7b' --include '*.json' --local_dir './local_dir'"
|
|
|
+ ]
|
|
|
+ },
|
|
|
+ {
|
|
|
+ "cell_type": "markdown",
|
|
|
+ "metadata": {},
|
|
|
+ "source": [
|
|
|
+ "",
|
|
|
+ "模型文件将被下载到`'./local_dir'`。",
|
|
|
+ "",
|
|
|
+ "如果`cache_dir`和`local_dir`参数同时被指定,`local_dir`优先级高,`cache_dir`将被忽略。",
|
|
|
+ "",
|
|
|
+ "## 下载私有模型需要登录",
|
|
|
+ "",
|
|
|
+ "### 通过login命令",
|
|
|
+ "",
|
|
|
+ "当下载私有模型时,您需要先登陆。通过 CLI 方式登陆的命令为`modelscope login`,详细使用说明如下:",
|
|
|
+ "",
|
|
|
+ " usage: modelscope <command> [<args>] login [-h] --token TOKEN",
|
|
|
+ " ",
|
|
|
+ " options:",
|
|
|
+ " -h, --help show this help message and exit",
|
|
|
+ " --token TOKEN The Access Token for modelscope.",
|
|
|
+ "",
|
|
|
+ " modelscope login --token YOUR_MODELSCOPE_ACCESS_TOKEN",
|
|
|
+ "",
|
|
|
+ "您可以在 [我的访问令牌](https://modelscope.cn/my/myaccesstoken) 页面获取**访问令牌**。",
|
|
|
+ "",
|
|
|
+ "# 使用 ModelScope SDK 下载",
|
|
|
+ "",
|
|
|
+ "## 下载整个模型仓库",
|
|
|
+ "",
|
|
|
+ "您可以使用`snapshot_download`下载整个模型仓库,示例如下:",
|
|
|
+ "",
|
|
|
+ ""
|
|
|
+ ]
|
|
|
+ },
|
|
|
+ {
|
|
|
+ "cell_type": "code",
|
|
|
+ "execution_count": null,
|
|
|
+ "metadata": {},
|
|
|
+ "outputs": [],
|
|
|
+ "source": [
|
|
|
+ "from modelscope.hub.snapshot_download import snapshot_download",
|
|
|
+ "",
|
|
|
+ "model_dir = snapshot_download('iic/nlp_xlmr_named-entity-recognition_viet-ecommerce-title')"
|
|
|
+ ]
|
|
|
+ },
|
|
|
+ {
|
|
|
+ "cell_type": "markdown",
|
|
|
+ "metadata": {},
|
|
|
+ "source": [
|
|
|
+ "",
|
|
|
+ "",
|
|
|
+ "**参数说明**",
|
|
|
+ "",
|
|
|
+ "| **字段名** | **必填** | **类型** | **描述** |",
|
|
|
+ "| ------------------- | :---------: | :----------: |------------------------|",
|
|
|
+ "| model_id | 是 | str | 模型ID |",
|
|
|
+ "| revision | 否 | str | 模型的Git版本,分支名或tag |",
|
|
|
+ "| cache_dir | 否 | str,Path | 指定模型本次下载缓存目录,给定后下载的具体模型文件将会被存储在cache_dir/model_id/THE_MODEL_FILES |",
|
|
|
+ "| allow_patterns | 否 | str,List | 指定要下载的文件模式,如文件名或文件扩展名 |",
|
|
|
+ "| ignore_patterns | 否 | str,List | 指定要忽略下载的文件模式,如文件名或文件扩展名|",
|
|
|
+ "| local_dir | 否 | str | 指定模型的下载存放目录,给定后本次下载的模型文件将会被存储在local_dir/THE_MODEL_FILES|",
|
|
|
+ "",
|
|
|
+ "如果`cache_dir`和`local_dir`参数同时被指定,local_dir优先级高,cache_dir将被忽略;更多参数使用说明可以参见开源代码的接口文档。如需指定下载或过滤下载某种/某类文件模式,可以使用 `allow_patterns`或`ignore_patterns`参数,示例如下:",
|
|
|
+ "",
|
|
|
+ "- **指定下载某些文件**",
|
|
|
+ "",
|
|
|
+ "以指定下载`Qwen/QwQ-32B-GGUF`中`q4_k_m`量化版本到`path/to/local/dir`目录下为例。",
|
|
|
+ "",
|
|
|
+ "",
|
|
|
+ ""
|
|
|
+ ]
|
|
|
+ },
|
|
|
+ {
|
|
|
+ "cell_type": "code",
|
|
|
+ "execution_count": null,
|
|
|
+ "metadata": {},
|
|
|
+ "outputs": [],
|
|
|
+ "source": [
|
|
|
+ "from modelscope.hub.snapshot_download import snapshot_download",
|
|
|
+ "",
|
|
|
+ "model_dir = snapshot_download('Qwen/QwQ-32B-GGUF',allow_patterns='qwq-32b-q4_k_m.gguf',local_dir='path/to/local/dir')"
|
|
|
+ ]
|
|
|
+ },
|
|
|
+ {
|
|
|
+ "cell_type": "markdown",
|
|
|
+ "metadata": {},
|
|
|
+ "source": [
|
|
|
+ "",
|
|
|
+ "- **过滤指定文件**",
|
|
|
+ "",
|
|
|
+ "以将`deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B`仓库除`figures`子目录外的所有文件下载到指定的`path/to/local/dir`目录为例。",
|
|
|
+ "",
|
|
|
+ "",
|
|
|
+ ""
|
|
|
+ ]
|
|
|
+ },
|
|
|
+ {
|
|
|
+ "cell_type": "code",
|
|
|
+ "execution_count": null,
|
|
|
+ "metadata": {},
|
|
|
+ "outputs": [],
|
|
|
+ "source": [
|
|
|
+ "from modelscope.hub.snapshot_download import snapshot_download",
|
|
|
+ "",
|
|
|
+ "model_dir = snapshot_download('deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B',ignore_patterns='figures/',local_dir='path/to/local/dir')"
|
|
|
+ ]
|
|
|
+ },
|
|
|
+ {
|
|
|
+ "cell_type": "markdown",
|
|
|
+ "metadata": {},
|
|
|
+ "source": [
|
|
|
+ "",
|
|
|
+ "",
|
|
|
+ "## 下载模型指定文件",
|
|
|
+ "",
|
|
|
+ "您也可以使用`model_file_download`下载模型指定文件。示例如下:",
|
|
|
+ "",
|
|
|
+ "",
|
|
|
+ ""
|
|
|
+ ]
|
|
|
+ },
|
|
|
+ {
|
|
|
+ "cell_type": "code",
|
|
|
+ "execution_count": null,
|
|
|
+ "metadata": {},
|
|
|
+ "outputs": [],
|
|
|
+ "source": [
|
|
|
+ "from modelscope.hub.file_download import model_file_download",
|
|
|
+ "",
|
|
|
+ "model_dir = model_file_download(model_id='Qwen/QwQ-32B-GGUF',file_path='qwq-32b-q4_k_m.gguf')"
|
|
|
+ ]
|
|
|
+ },
|
|
|
+ {
|
|
|
+ "cell_type": "markdown",
|
|
|
+ "metadata": {},
|
|
|
+ "source": [
|
|
|
+ "",
|
|
|
+ "",
|
|
|
+ "**参数说明**",
|
|
|
+ "",
|
|
|
+ "| **字段名** | **必填** | **类型** | **描述** |",
|
|
|
+ "| ------------------- | :---------: | :----------: |------------------------|",
|
|
|
+ "| model_id | 是 | str | 模型ID |",
|
|
|
+ "| file_path | 是 | str | 待下载文件在远程模型仓库的相对路径|",
|
|
|
+ "| revision | 否 | str | 模型的Git版本,分支名或tag |",
|
|
|
+ "| cache_dir | 否 | str,Path | 指定模型本次下载缓存目录,给定后下载的具体模型文件将会被存储在cache_dir/models/model_id/THE_MODEL_FILES |",
|
|
|
+ "| local_dir | 否 | str | 指定模型的下载存放目录,给定后本次下载的模型文件将会被存储在local_dir/THE_MODEL_FILES|",
|
|
|
+ "",
|
|
|
+ "如果`cache_dir`和`local_dir`参数同时被指定,`local_dir`优先级高,`cache_dir`将被忽略;更多参数使用说明可以参见开源代码的接口文档。",
|
|
|
+ "",
|
|
|
+ "## 下载私有模型 ",
|
|
|
+ "",
|
|
|
+ "当下载非公开模型及申请制模型时,您需要先登陆,然后可参考前述文档步骤下载模型。此处以通过 snapshot_download 下载非公开模型为例:",
|
|
|
+ "",
|
|
|
+ "",
|
|
|
+ ""
|
|
|
+ ]
|
|
|
+ },
|
|
|
+ {
|
|
|
+ "cell_type": "code",
|
|
|
+ "execution_count": null,
|
|
|
+ "metadata": {},
|
|
|
+ "outputs": [],
|
|
|
+ "source": [
|
|
|
+ "from modelscope import HubApi",
|
|
|
+ "from modelscope import snapshot_download",
|
|
|
+ "",
|
|
|
+ "# login to ModelScope",
|
|
|
+ "api=HubApi()",
|
|
|
+ "api.login('YOUR_MODELSCOPE_ACCESS_TOKEN')",
|
|
|
+ "",
|
|
|
+ "# download your model, the model_path is downloaded model path.",
|
|
|
+ "model_path =snapshot_download(model_id='the_model_id')"
|
|
|
+ ]
|
|
|
+ },
|
|
|
+ {
|
|
|
+ "cell_type": "markdown",
|
|
|
+ "metadata": {},
|
|
|
+ "source": [
|
|
|
+ "",
|
|
|
+ "ModelScope SDK提供了多种登陆方式,您也可以通过命令行等其他方式完成登陆。",
|
|
|
+ "",
|
|
|
+ "## 通过加载模型触发下载 ",
|
|
|
+ "",
|
|
|
+ "除了直接下载模型文件外,当使用ModelScope SDK加载模型时,也会自动触发模型下载。如果模型和ModelScope SDK绑定,则只需要几行代码即可加载模型,同时 ModelScope 还支持通过 AutoModel 等接口来加载模型。此处以使用 `Model` 加载模型为例:",
|
|
|
+ "",
|
|
|
+ "",
|
|
|
+ ""
|
|
|
+ ]
|
|
|
+ },
|
|
|
+ {
|
|
|
+ "cell_type": "code",
|
|
|
+ "execution_count": null,
|
|
|
+ "metadata": {},
|
|
|
+ "outputs": [],
|
|
|
+ "source": [
|
|
|
+ "from modelscope.models import Model",
|
|
|
+ "model = Model.from_pretrained('iic/nlp_xlmr_named-entity-recognition_viet-ecommerce-title', revision='v1.0.1')",
|
|
|
+ "# revision为可选参数,不指定版本会取模型默认版本,默认版本,默认版本为ModelScope library发布前最后一个版本",
|
|
|
+ "",
|
|
|
+ "# 如何得到发布时间",
|
|
|
+ "import modelscope",
|
|
|
+ "print(modelscope.version.__release_datetime__)"
|
|
|
+ ]
|
|
|
+ },
|
|
|
+ {
|
|
|
+ "cell_type": "markdown",
|
|
|
+ "metadata": {},
|
|
|
+ "source": [
|
|
|
+ "",
|
|
|
+ "",
|
|
|
+ "# 使用 GIT 下载模型",
|
|
|
+ "",
|
|
|
+ "ModelScope服务端的模型都是通过Git存储,所以也可以在安装Git LFS后,通过git clone的方式在本地下载模型。",
|
|
|
+ "",
|
|
|
+ "",
|
|
|
+ ""
|
|
|
+ ]
|
|
|
+ },
|
|
|
+ {
|
|
|
+ "cell_type": "code",
|
|
|
+ "execution_count": null,
|
|
|
+ "metadata": {},
|
|
|
+ "outputs": [],
|
|
|
+ "source": [
|
|
|
+ "# 公开模型下载",
|
|
|
+ "git lfs install",
|
|
|
+ "git clone https://www.modelscope.cn/<owner_name>/<model-name>.git",
|
|
|
+ "# 例如: git clone https://www.modelscope.cn/iic/ofa_image-caption_coco_large_en.git",
|
|
|
+ "",
|
|
|
+ "# 私有模型下载,前提是您有响应模型权限 方法1",
|
|
|
+ "git lfs install",
|
|
|
+ "git clone http://oauth2:your_access_token@www.modelscope.cn/<owner_name>/<model-name>.git",
|
|
|
+ "# 方法2",
|
|
|
+ "git clone http://your_user_name@www.modelscope.cn/<owner_name>/<model-name>.git",
|
|
|
+ "# Password for 'http://your_user_name@modelscope.cn':",
|
|
|
+ "# input access token"
|
|
|
+ ]
|
|
|
+ },
|
|
|
+ {
|
|
|
+ "cell_type": "markdown",
|
|
|
+ "metadata": {},
|
|
|
+ "source": [
|
|
|
+ "",
|
|
|
+ "",
|
|
|
+ "如果**希望跳过LFS大文件的下载**,可以在git clone命令前添加`GIT_LFS_SKIP_SMUDGE=1`,来只获取`LFS`指针,而不下载实际的大文件:",
|
|
|
+ "",
|
|
|
+ "",
|
|
|
+ ""
|
|
|
+ ]
|
|
|
+ },
|
|
|
+ {
|
|
|
+ "cell_type": "code",
|
|
|
+ "execution_count": null,
|
|
|
+ "metadata": {},
|
|
|
+ "outputs": [],
|
|
|
+ "source": [
|
|
|
+ "GIT_LFS_SKIP_SMUDGE=1 https://www.modelscope.cn/<namespace>/<model-name>.git"
|
|
|
+ ]
|
|
|
+ },
|
|
|
+ {
|
|
|
+ "cell_type": "markdown",
|
|
|
+ "metadata": {},
|
|
|
+ "source": [
|
|
|
+ "",
|
|
|
+ "",
|
|
|
+ "",
|
|
|
+ "# 如何获取访问令牌",
|
|
|
+ "",
|
|
|
+ "用您的账号登录<https://www.modelscope.cn> ,在个人中心->访问令牌,新建并拷贝访问令牌.",
|
|
|
+ "<!--  -->",
|
|
|
+ "",
|
|
|
+ "",
|
|
|
+ "",
|
|
|
+ "",
|
|
|
+ ""
|
|
|
+ ]
|
|
|
+ }
|
|
|
+ ],
|
|
|
+ "metadata": {
|
|
|
+ "kernelspec": {
|
|
|
+ "display_name": "Python 3",
|
|
|
+ "language": "python",
|
|
|
+ "name": "python3"
|
|
|
+ }
|
|
|
+ },
|
|
|
+ "nbformat": 4,
|
|
|
+ "nbformat_minor": 2
|
|
|
+}
|