Pārlūkot izejas kodu

修复dpo报错

lxylxy123321 1 dienu atpakaļ
vecāks
revīzija
75c46db7e1
1 mainītis faili ar 1 papildinājumiem un 1 dzēšanām
  1. 1 1
      backend/app/engines/text_engine.py

+ 1 - 1
backend/app/engines/text_engine.py

@@ -650,7 +650,7 @@ class TextEngine(BaseEngine):
             }
 
         hf_dataset = HFDataset.from_list(raw_data)
-        tokenized = hf_dataset.map(tokenize_fn, batched=True, remove_columns=hf_dataset.column_names)
+        tokenized = hf_dataset.map(tokenize_fn, batched=True)
         return tokenized