lxylxy123321 4 часов назад
Родитель
Сommit
d10846b737
2 измененных файлов с 44 добавлено и 36 удалено
  1. 1 0
      backend/app/services/dataset_service.py
  2. 43 36
      result.txt

+ 1 - 0
backend/app/services/dataset_service.py

@@ -431,6 +431,7 @@ def _download_via_msdataset(dataset_id: str, ds_dir: Path) -> tuple[list[dict],
     from modelscope.msdatasets import MsDataset
     from PIL import Image
     import shutil
+    import os
 
     namespace, ds_name = dataset_id.split("/", 1) if "/" in dataset_id else ("", dataset_id)
     images_dir = ds_dir / "images"

+ 43 - 36
result.txt

@@ -1,36 +1,43 @@
-lq@lq:~/Fine-tuning$ cp backend/scripts/test_ms_api.py backend/data/ && sudo docker exec -it finetune-backend python3 /root/Fine-tuning/backend/data/test_ms_api.py tany0699/carBrands50
-数据集: tany0699/carBrands50
-
-=== 用 MsDataset.load() 下载 ===
-2026-05-28 07:26:26,648 - modelscope - INFO - No subset_name specified, defaulting to the default
-2026-05-28 07:26:27,880 - modelscope - WARNING - Reusing dataset dataset_builder (/root/.cache/modelscope/hub/datasets/tany0699/carBrands50/master/data_files)
-2026-05-28 07:26:27,880 - modelscope - INFO - Generating dataset dataset_builder (/root/.cache/modelscope/hub/datasets/tany0699/carBrands50/master/data_files)
-2026-05-28 07:26:27,880 - modelscope - INFO - Reusing cached meta-data file: /root/.cache/modelscope/hub/datasets/tany0699/carBrands50/master/data_files/a6ade4dacefa0beffa564bf1f50f7ffd
-split='train' 成功, 共 4397 条
-
-=== 处理前 5 条数据(复制图片) ===
-Record 0: image:FILE -> copied Bentley_025.jpg (6811 bytes) as 000000.jpg
-Record 1: image:FILE -> copied Bentley_004.jpg (12710 bytes) as 000001.jpg
-Record 2: image:FILE -> copied Bentley_038.jpg (6993 bytes) as 000002.jpg
-Record 3: image:FILE -> copied Bentley_011.jpg (8629 bytes) as 000003.jpg
-Record 4: image:FILE -> copied Bentley_006.jpg (7090 bytes) as 000004.jpg
-
-=== 写入 JSONL ===
-写入 5 条记录到 /tmp/ms_test_download/data.jsonl
-
-=== JSONL 内容 ===
-0: {"image:FILE": "images/000000.jpg", "category": 4}
-1: {"image:FILE": "images/000001.jpg", "category": 4}
-2: {"image:FILE": "images/000002.jpg", "category": 4}
-3: {"image:FILE": "images/000003.jpg", "category": 4}
-4: {"image:FILE": "images/000004.jpg", "category": 4}
-
-=== images 目录 ===
-  000000.jpg (6811 bytes)
-  000001.jpg (12710 bytes)
-  000002.jpg (6993 bytes)
-  000003.jpg (8629 bytes)
-  000004.jpg (7090 bytes)
-
-=== 测试完成! ===
-数据集目录: /tmp/ms_test_download
+INFO:     Started server process [1]
+INFO:     Waiting for application startup.
+2026-05-28 07:52:40 | INFO     | peft-platform | JobQueue started with 2 workers
+INFO:     Application startup complete.
+INFO:     Uvicorn running on http://0.0.0.0:8010 (Press CTRL+C to quit)
+INFO:     172.20.0.4:35472 - "GET /api/v1/models/ HTTP/1.0" 200 OK
+INFO:     172.20.0.4:35474 - "GET /api/v1/datasets/ HTTP/1.0" 200 OK
+INFO:     172.20.0.4:35500 - "GET /api/v1/training/jobs HTTP/1.0" 200 OK
+INFO:     172.20.0.4:35510 - "GET /api/v1/models/ HTTP/1.0" 200 OK
+INFO:     172.20.0.4:35514 - "GET /api/v1/datasets/ HTTP/1.0" 200 OK
+INFO:     127.0.0.1:46858 - "GET /health HTTP/1.1" 200 OK
+2026-05-28 07:52:46 | INFO     | peft-platform | Deleted dataset directory: /root/Fine-tuning/backend/data/processed/ms_tany0699_carBrands50
+2026-05-28 07:52:46 | INFO     | peft-platform | Deleted dataset: tany0699/carBrands50
+INFO:     172.20.0.4:58764 - "DELETE /api/v1/datasets/98dd637f-879b-4fc4-b7ea-f64238110c25 HTTP/1.0" 200 OK
+INFO:     172.20.0.4:58766 - "GET /api/v1/datasets/ HTTP/1.0" 200 OK
+2026-05-28 07:52:51 | INFO     | peft-platform | Dataset download task started: tany0699/carBrands50 (task_id=035d9cad-3c35-467b-9409-01de28520fb3)
+INFO:     172.20.0.4:58774 - "POST /api/v1/datasets/download HTTP/1.0" 200 OK
+INFO:     172.20.0.4:57858 - "GET /api/v1/datasets/download/035d9cad-3c35-467b-9409-01de28520fb3 HTTP/1.0" 200 OK
+INFO:     172.20.0.4:57866 - "GET /api/v1/datasets/download/035d9cad-3c35-467b-9409-01de28520fb3 HTTP/1.0" 200 OK
+2026-05-28 07:53:00,756 - modelscope - INFO - No subset_name specified, defaulting to the default
+INFO:     172.20.0.4:57876 - "GET /api/v1/datasets/download/035d9cad-3c35-467b-9409-01de28520fb3 HTTP/1.0" 200 OK
+2026-05-28 07:53:02,365 - modelscope - WARNING - Reusing dataset dataset_builder (/root/.cache/modelscope/hub/datasets/tany0699/carBrands50/master/data_files)
+2026-05-28 07:53:02,365 - modelscope - INFO - Generating dataset dataset_builder (/root/.cache/modelscope/hub/datasets/tany0699/carBrands50/master/data_files)
+2026-05-28 07:53:02,366 - modelscope - INFO - Loading meta-data file ...
+4398it [00:00, 40120.82it/s]
+INFO:     172.20.0.4:38498 - "GET /api/v1/datasets/download/035d9cad-3c35-467b-9409-01de28520fb3 HTTP/1.0" 200 OK
+9% INFO:     172.20.0.4:38514 - "GET /api/v1/datasets/download/035d9cad-3c35-467b-9409-01de28520fb3 HTTP/1.0" 200 OK
+40% INFO:     172.20.0.4:38520 - "GET /api/v1/datasets/download/035d9cad-3c35-467b-9409-01de28520fb3 HTTP/1.0" 200 OK
+61% INFO:     172.20.0.4:38522 - "GET /api/v1/datasets/download/035d9cad-3c35-467b-9409-01de28520fb3 HTTP/1.0" 200 OK
+73% INFO:     127.0.0.1:43340 - "GET /health HTTP/1.1" 200 OK
+84% INFO:     172.20.0.4:47586 - "GET /api/v1/datasets/download/035d9cad-3c35-467b-9409-01de28520fb3 HTTP/1.0" 200 OK
+100% INFO:     172.20.0.4:47596 - "GET /api/v1/datasets/download/035d9cad-3c35-467b-9409-01de28520fb3 HTTP/1.0" 200 OK
+INFO:     172.20.0.4:47600 - "GET /api/v1/datasets/download/035d9cad-3c35-467b-9409-01de28520fb3 HTTP/1.0" 200 OK
+2026-05-28 07:53:22 | INFO     | peft-platform | MsDataset.load() loaded split 'train': 4397 records
+2026-05-28 07:53:22 | WARNING  | peft-platform | MsDataset.load() failed for tany0699/carBrands50: name 'os' is not defined, falling back to CLI
+2026-05-28 07:53:22 | INFO     | peft-platform | Fallback CLI: modelscope download --dataset tany0699/carBrands50 --local_dir /root/Fine-tuning/backend/data/processed/ms_tany0699_carBrands50
+INFO:     172.20.0.4:43150 - "GET /api/v1/datasets/download/035d9cad-3c35-467b-9409-01de28520fb3 HTTP/1.0" 200 OK
+2026-05-28 07:53:27 | INFO     | peft-platform | CLI downloaded 9 files to /root/Fine-tuning/backend/data/processed/ms_tany0699_carBrands50
+2026-05-28 07:53:27 | INFO     | peft-platform | Selected data file: /root/Fine-tuning/backend/data/processed/ms_tany0699_carBrands50/train.csv (size=140505)
+2026-05-28 07:53:27 | INFO     | peft-platform | Dataset downloaded: tany0699/carBrands50 (4397 records)
+INFO:     172.20.0.4:43156 - "GET /api/v1/datasets/download/035d9cad-3c35-467b-9409-01de28520fb3 HTTP/1.0" 200 OK
+INFO:     172.20.0.4:43160 - "GET /api/v1/datasets/ HTTP/1.0" 200 OK
+INFO:     172.20.0.4:43172 - "GET /api/v1/datasets/4eb51319-31aa-4f8c-a770-0b94294d0b5a/preview?rows=10 HTTP/1.0" 200 OK