Главная Обзор Помощь
Регистрация Вход
CRBC-MaaS-Platform-Project
/
LQDeployConfig
4
0
Ответвить 0
Файлы Задачи 0 Запросы на слияние 0 Вики
Ветка: master
Ветки Метки
LQDeployConfig
/
prod
/
models
/
vllm
/
logs
/
qwen3-embedding-8b-server.log

qwen3-embedding-8b-server.log 15 KB
Постоянная ссылка История Исходник

1234567891011121314151617181920212223242526272829303132333435363738394041424344454647484950515253545556575859606162636465666768697071727374757677787980818283 
  1. (APIServer pid=8) INFO 03-28 12:07:05 [utils.py:297] 
    2. 

  2. (APIServer pid=8) INFO 03-28 12:07:05 [utils.py:297]        █     █     █▄   ▄█
    3. 

  3. (APIServer pid=8) INFO 03-28 12:07:05 [utils.py:297]  ▄▄ ▄█ █     █     █ ▀▄▀ █  version 0.18.0
    4. 

  4. (APIServer pid=8) INFO 03-28 12:07:05 [utils.py:297]   █▄█▀ █     █     █     █  model   /model/Qwen3-Embedding-8B
    5. 

  5. (APIServer pid=8) INFO 03-28 12:07:05 [utils.py:297]    ▀▀  ▀▀▀▀▀ ▀▀▀▀▀ ▀     ▀
    6. 

  6. (APIServer pid=8) INFO 03-28 12:07:05 [utils.py:297] 
    7. 

  7. (APIServer pid=8) INFO 03-28 12:07:05 [utils.py:233] non-default args: {'model_tag': '/model/Qwen3-Embedding-8B', 'host': '0.0.0.0', 'port': 30000, 'api_key': ['lq123456'], 'model': '/model/Qwen3-Embedding-8B', 'convert': 'embed', 'trust_remote_code': True, 'gpu_memory_utilization': 0.45}
    8. 

  8. (APIServer pid=8) INFO 03-28 12:07:10 [config.py:821] Found sentence-transformers modules configuration.
    9. 

  9. (APIServer pid=8) INFO 03-28 12:07:10 [config.py:848] Found pooling configuration.
    10. 

  10. (APIServer pid=8) INFO 03-28 12:07:10 [model.py:824] Resolved `--runner auto` to `--runner pooling`. Pass the value explicitly to silence this message.
    11. 

  11. (APIServer pid=8) INFO 03-28 12:07:10 [model.py:533] Resolved architecture: Qwen3ForCausalLM
    12. 

  12. (APIServer pid=8) INFO 03-28 12:07:10 [model.py:1582] Using max model len 40960
    13. 

  13. (APIServer pid=8) INFO 03-28 12:07:10 [scheduler.py:231] Chunked prefill is enabled with max_num_batched_tokens=8192.
    14. 

  14. (APIServer pid=8) INFO 03-28 12:07:10 [vllm.py:754] Asynchronous scheduling is enabled.
    15. 

  15. (APIServer pid=8) WARNING 03-28 12:07:10 [vllm.py:916] Pooling models do not support full cudagraphs. Overriding cudagraph_mode to PIECEWISE.
    16. 

  16. (EngineCore pid=403) INFO 03-28 12:07:16 [core.py:103] Initializing a V1 LLM engine (v0.18.0) with config: model='/model/Qwen3-Embedding-8B', speculative_config=None, tokenizer='/model/Qwen3-Embedding-8B', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=40960, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=/model/Qwen3-Embedding-8B, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=PoolerConfig(pooling_type=None, seq_pooling_type='LAST', tok_pooling_type='ALL', use_activation=True, dimensions=None, enable_chunked_processing=False, max_embed_len=None, logit_bias=None, step_tag_id=None, returned_token_ids=None), compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_endpoints': [8192], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.PIECEWISE: 1>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256, 272, 288, 304, 320, 336, 352, 368, 384, 400, 416, 432, 448, 464, 480, 496, 512], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 512, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []}
    17. 

  17. (EngineCore pid=403) INFO 03-28 12:07:16 [parallel_state.py:1395] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://172.19.0.6:54393 backend=nccl
    18. 

  18. (EngineCore pid=403) INFO 03-28 12:07:16 [parallel_state.py:1717] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A, EPLB rank N/A
    19. 

  19. (EngineCore pid=403) INFO 03-28 12:07:17 [gpu_model_runner.py:4481] Starting to load model /model/Qwen3-Embedding-8B...
    20. 

  20. (EngineCore pid=403) INFO 03-28 12:07:17 [cuda.py:317] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION'].
    21. 

  21. (EngineCore pid=403) INFO 03-28 12:07:17 [flash_attn.py:598] Using FlashAttention version 3
    22. 

  22. (EngineCore pid=403) <frozen importlib._bootstrap_external>:1301: FutureWarning: The cuda.cudart module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.runtime module instead.
    23. 

  23. (EngineCore pid=403) <frozen importlib._bootstrap_external>:1301: FutureWarning: The cuda.nvrtc module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.nvrtc module instead.
    24. 

  24. (EngineCore pid=403) 
Loading safetensors checkpoint shards:   0% Completed | 0/4 [00:00<?, ?it/s]
    25. 

  25. (EngineCore pid=403) INFO 03-28 12:07:18 [adapters.py:204] Mapping weights to Qwen3Model as they are relative to this model instead of Qwen3ForEmbedding.
    26. 

  26. (EngineCore pid=403) 
Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:00<00:00, 681.53it/s]
    27. 

  27. (EngineCore pid=403) 
    28. 

  28. (EngineCore pid=403) INFO 03-28 12:07:19 [default_loader.py:384] Loading weights took 1.55 seconds
    29. 

  29. (EngineCore pid=403) INFO 03-28 12:07:20 [gpu_model_runner.py:4566] Model loading took 14.11 GiB memory and 2.213984 seconds
    30. 

  30. (EngineCore pid=403) INFO 03-28 12:07:26 [backends.py:988] Using cache directory: /root/.cache/vllm/torch_compile_cache/0e0e010fc1/rank_0_0/backbone for vLLM's torch.compile
    31. 

  31. (EngineCore pid=403) INFO 03-28 12:07:26 [backends.py:1048] Dynamo bytecode transform time: 6.46 s
    32. 

  32. (EngineCore pid=403) INFO 03-28 12:07:32 [backends.py:371] Cache the graph of compile range (1, 8192) for later use
    33. 

  33. (EngineCore pid=403) INFO 03-28 12:07:37 [backends.py:387] Compiling a graph for compile range (1, 8192) takes 10.06 s
    34. 

  34. (EngineCore pid=403) INFO 03-28 12:07:38 [decorators.py:627] saved AOT compiled function to /root/.cache/vllm/torch_compile_cache/torch_aot_compile/cd886c2cde0130c8e5277a4021e7c3918b28d805d8174b25b39ef6f0cdc42624/rank_0_0/model
    35. 

  35. (EngineCore pid=403) INFO 03-28 12:07:38 [monitor.py:48] torch.compile took 18.33 s in total
    36. 

  36. (EngineCore pid=403) INFO 03-28 12:07:39 [monitor.py:76] Initial profiling/warmup run took 1.26 s
    37. 

  37. (EngineCore pid=403) INFO 03-28 12:07:40 [kv_cache_utils.py:826] Overriding num_gpu_blocks=0 with num_gpu_blocks_override=512
    38. 

  38. (EngineCore pid=403) INFO 03-28 12:07:40 [gpu_model_runner.py:5607] Profiling CUDA graph memory: PIECEWISE=51 (largest=512)
    39. 

  39. (EngineCore pid=403) INFO 03-28 12:07:41 [gpu_model_runner.py:5686] Estimated CUDA graph memory: 0.32 GiB total
    40. 

  40. (EngineCore pid=403) INFO 03-28 12:07:41 [gpu_worker.py:456] Available KV cache memory: 47.32 GiB
    41. 

  41. (EngineCore pid=403) INFO 03-28 12:07:41 [gpu_worker.py:490] In v0.19, CUDA graph memory profiling will be enabled by default (VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=1), which more accurately accounts for CUDA graph memory during KV cache allocation. To try it now, set VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=1 and increase --gpu-memory-utilization from 0.4500 to 0.4523 to maintain the same effective KV cache size.
    42. 

  42. (EngineCore pid=403) INFO 03-28 12:07:41 [kv_cache_utils.py:1316] GPU KV cache size: 344,576 tokens
    43. 

  43. (EngineCore pid=403) INFO 03-28 12:07:41 [kv_cache_utils.py:1321] Maximum concurrency for 40,960 tokens per request: 8.41x
    44. 

  44. (EngineCore pid=403) 2026-03-28 12:07:41,897 - INFO - autotuner.py:262 - flashinfer.jit: [Autotuner]: Autotuning process starts ...
    45. 

  45. (EngineCore pid=403) 2026-03-28 12:07:41,908 - INFO - autotuner.py:268 - flashinfer.jit: [Autotuner]: Autotuning process ends
    46. 

  46. (EngineCore pid=403) 
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):   0%|          | 0/51 [00:00<?, ?it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):   4%|▍         | 2/51 [00:00<00:03, 13.26it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):   8%|▊         | 4/51 [00:00<00:03, 13.02it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  12%|█▏        | 6/51 [00:00<00:03, 13.05it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  16%|█▌        | 8/51 [00:00<00:03, 13.41it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  20%|█▉        | 10/51 [00:00<00:02, 13.87it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  24%|██▎       | 12/51 [00:00<00:02, 14.23it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  27%|██▋       | 14/51 [00:00<00:02, 14.75it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  31%|███▏      | 16/51 [00:01<00:02, 15.23it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  37%|███▋      | 19/51 [00:01<00:01, 16.76it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  41%|████      | 21/51 [00:01<00:01, 17.48it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  45%|████▌     | 23/51 [00:01<00:01, 17.60it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  49%|████▉     | 25/51 [00:01<00:01, 18.01it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  55%|█████▍    | 28/51 [00:01<00:01, 18.90it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  61%|██████    | 31/51 [00:01<00:01, 19.61it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  67%|██████▋   | 34/51 [00:02<00:00, 20.66it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  73%|███████▎  | 37/51 [00:02<00:00, 21.61it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  78%|███████▊  | 40/51 [00:02<00:00, 22.38it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  84%|████████▍ | 43/51 [00:02<00:00, 23.29it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  90%|█████████ | 46/51 [00:02<00:00, 24.05it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  96%|█████████▌| 49/51 [00:02<00:00, 24.98it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 51/51 [00:02<00:00, 19.04it/s]
    47. 

  47. (EngineCore pid=403) INFO 03-28 12:07:45 [gpu_model_runner.py:5746] Graph capturing finished in 4 secs, took 0.35 GiB
    48. 

  48. (EngineCore pid=403) INFO 03-28 12:07:45 [gpu_worker.py:617] CUDA graph pool memory: 0.35 GiB (actual), 0.32 GiB (estimated), difference: 0.03 GiB (7.9%).
    49. 

  49. (EngineCore pid=403) INFO 03-28 12:07:45 [core.py:281] init engine (profile, create kv cache, warmup model) took 25.76 seconds
    50. 

  50. (EngineCore pid=403) INFO 03-28 12:07:46 [vllm.py:754] Asynchronous scheduling is enabled.
    51. 

  51. (APIServer pid=8) INFO 03-28 12:07:46 [api_server.py:576] Supported tasks: ['token_embed', 'embed']
    52. 

  52. (APIServer pid=8) WARNING 03-28 12:07:46 [utils.py:140] To make v1/embeddings API fast, please install orjson by `pip install orjson`
    53. 

  53. (APIServer pid=8) INFO 03-28 12:07:46 [io_processor.py:52] Loaded prompt prefixes for input_type: ['query', 'document']
    54. 

  54. (APIServer pid=8) INFO 03-28 12:07:46 [api_server.py:580] Starting vLLM server on http://0.0.0.0:30000
    55. 

  55. (APIServer pid=8) INFO 03-28 12:07:46 [launcher.py:37] Available routes are:
    56. 

  56. (APIServer pid=8) INFO 03-28 12:07:46 [launcher.py:46] Route: /openapi.json, Methods: GET, HEAD
    57. 

  57. (APIServer pid=8) INFO 03-28 12:07:46 [launcher.py:46] Route: /docs, Methods: GET, HEAD
    58. 

  58. (APIServer pid=8) INFO 03-28 12:07:46 [launcher.py:46] Route: /docs/oauth2-redirect, Methods: GET, HEAD
    59. 

  59. (APIServer pid=8) INFO 03-28 12:07:46 [launcher.py:46] Route: /redoc, Methods: GET, HEAD
    60. 

  60. (APIServer pid=8) INFO 03-28 12:07:46 [launcher.py:46] Route: /tokenize, Methods: POST
    61. 

  61. (APIServer pid=8) INFO 03-28 12:07:46 [launcher.py:46] Route: /detokenize, Methods: POST
    62. 

  62. (APIServer pid=8) INFO 03-28 12:07:46 [launcher.py:46] Route: /load, Methods: GET
    63. 

  63. (APIServer pid=8) INFO 03-28 12:07:46 [launcher.py:46] Route: /version, Methods: GET
    64. 

  64. (APIServer pid=8) INFO 03-28 12:07:46 [launcher.py:46] Route: /health, Methods: GET
    65. 

  65. (APIServer pid=8) INFO 03-28 12:07:46 [launcher.py:46] Route: /metrics, Methods: GET
    66. 

  66. (APIServer pid=8) INFO 03-28 12:07:46 [launcher.py:46] Route: /v1/models, Methods: GET
    67. 

  67. (APIServer pid=8) INFO 03-28 12:07:46 [launcher.py:46] Route: /ping, Methods: GET
    68. 

  68. (APIServer pid=8) INFO 03-28 12:07:46 [launcher.py:46] Route: /ping, Methods: POST
    69. 

  69. (APIServer pid=8) INFO 03-28 12:07:46 [launcher.py:46] Route: /invocations, Methods: POST
    70. 

  70. (APIServer pid=8) INFO 03-28 12:07:46 [launcher.py:46] Route: /pooling, Methods: POST
    71. 

  71. (APIServer pid=8) INFO 03-28 12:07:46 [launcher.py:46] Route: /v1/embeddings, Methods: POST
    72. 

  72. (APIServer pid=8) INFO 03-28 12:07:46 [launcher.py:46] Route: /v2/embed, Methods: POST
    73. 

  73. (APIServer pid=8) INFO 03-28 12:07:46 [launcher.py:46] Route: /score, Methods: POST
    74. 

  74. (APIServer pid=8) INFO 03-28 12:07:46 [launcher.py:46] Route: /v1/score, Methods: POST
    75. 

  75. (APIServer pid=8) INFO 03-28 12:07:46 [launcher.py:46] Route: /rerank, Methods: POST
    76. 

  76. (APIServer pid=8) INFO 03-28 12:07:46 [launcher.py:46] Route: /v1/rerank, Methods: POST
    77. 

  77. (APIServer pid=8) INFO 03-28 12:07:46 [launcher.py:46] Route: /v2/rerank, Methods: POST
    78. 

  78. (APIServer pid=8) INFO:     Started server process [8]
    79. 

  79. (APIServer pid=8) INFO:     Waiting for application startup.
    80. 

  80. (APIServer pid=8) INFO:     Application startup complete.
    81. 

  81. (APIServer pid=8) INFO 03-28 12:08:27 [loggers.py:259] Engine 000: Avg prompt throughput: 0.9 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%
    82. 

  82. (APIServer pid=8) INFO:     172.19.0.1:50770 - "POST /v1/embeddings HTTP/1.1" 200 OK
    83. 

  83. (APIServer pid=8) INFO 03-28 12:08:37 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%
    84.