Token indices sequence length is longer than the specified maximum sequence length for this model (60022 > 32768). Running this sequence through the model will result in indexing errors

[ADIOCLASSMATE]

我们在8卡A100 80G 等效算力上复现预训练, 发现:
问题1:

[WARNING|tokenization_utils_base.py:3945] 2025-09-05 13:49:03,241 >> Token indices sequence length is longer than the specified maximum sequence length for this model (60022 > 32768). Running this sequence through the model will result in indexing errors

问题2:

[WARNING][veomni.data.dynamic_batching:110] 09/05/2025 13:49:06 >> DynamicBatchDataset iter data exception:  
Traceback (most recent call last):
  File "/inspire/hdd/project/embodied-multimodality/liuxiaoran-240108120089/projects_zhuying/Open-dLLM/veomni/data/dynamic_batching.py", line 107, in batch_data_generator
    processing_item = next(self._data_iter)
                      ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torchdata/stateful_dataloader/stateful_dataloader.py", line 450, in __next__
    return super().__next__()
           ^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/utils/data/dataloader.py", line 733, in __next__
    data = self._next_data()
           ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torchdata/stateful_dataloader/stateful_dataloader.py", line 1407, in _next_data
    raise StopIteration
StopIteration

可能是由问题一导致.

问题3:

Epoch 1/1:   0%|                | 3/953675 [10:20<54448:05:23, 205.54s/it, loss: 29.20, grad_norm: 1280.00, lr: 9.44e-08]

训练速度过慢, 请问是否正常?

训练命令:

export TOKENIZERS_PARALLELISM=false

NNODES=${NNODES:=1}
NPROC_PER_NODE=$(nvidia-smi --query-gpu=name --format=csv,noheader | wc -l)
NODE_RANK=${NODE_RANK:=0}
MASTER_ADDR=${MASTER_ADDR:=[0.0.0.0](http://0.0.0.0/)}

MASTER_PORT=${MASTER_PORT:=12345}

WANDB_MODE=offline torchrun --nnodes=$NNODES --nproc-per-node $NPROC_PER_NODE --node-rank $NODE_RANK \
  --master-addr=$MASTER_ADDR --master-port=$MASTER_PORT tasks/train_torch.py \
  configs/pretrain/qwen2_5_coder_500M.yaml \
  --data.train_path=data/test_data \
  --train.ckpt_manager=dcp \
  --train.micro_batch_size=16 \
  --train.global_batch_size=512 \
  --train.output_dir=logs/Qwen2.5-Coder-0.5B_mdm \
  --train.save_steps=10000

[pengzhangzhi]

hey,

I'm using English to make it accessible to non-Chinese speakers.

question 1 and 2 are normal. The data processing makes a batch of sequences by max #tokens; it's just showing u something is not perfectly under the max #tokens. And issue 2 is the data loading logic that keeps loading data when it hits the end, basically, it will raise a warning and then start over. None of them are errors.

issue3: the training will employ gradient accumulation global_batch_size/(micro_batch_size * #gpus), which will be 512/(16*8) = 4. Therefore u will accumulate 4 steps to update one training step. it might be the reason why it looks slow.

Hope this helps!! And let us know if u have further questions : )

[rangehow]

你训练的数据集数据量和模型参数量多大呢？时间其实可以通过total flops和卡的算力有一个大概预估的。8卡A100如果1B模型过100B token的话正常应该10天就能完成。

How large is your training dataset and model parameter count? The training time can actually be roughly estimated based on the total FLOPs and the GPU performance. For example, with 8 A100 GPUs, training a 1B model on over 100B tokens should normally take about 10 days to complete.

[huchenz1]

My training pipeline stalls with 0% GPU utilization during data loading. Increasing num_workers does not solve the issue. Is it for my SSD's limitation? Could owner share the more training details, like num_workers and prefetch fator? Thanks!

[pengzhangzhi]

你训练的数据集数据量和模型参数量多大呢？时间其实可以通过total flops和卡的算力有一个大概预估的。8卡A100如果1B模型过100B token的话正常应该10天就能完成。

How large is your training dataset and model parameter count? The training time can actually be roughly estimated based on the total FLOPs and the GPU performance. For example, with 8 A100 GPUs, training a 1B model on over 100B tokens should normally take about 10 days to complete.

hmm it was trained using multiple different setting. I'll suggest total tokens per batch 4B, and train for 500B steps, def monitor perf. per 100B step.

[pengzhangzhi]

My training pipeline stalls with 0% GPU utilization during data loading. Increasing num_workers does not solve the issue. Is it for my SSD's limitation? Could owner share the more training details, like num_workers and prefetch fator? Thanks!

to debug, use num workers = 0, prefetch factor = 0? normally i do 2 and 2

[pharaohcaptain]

你训练的数据集数据量和模型参数量多大呢？时间其实可以通过total flops和卡的算力有一个大概预估的。8卡A100如果1B模型过100B token的话正常应该10天就能完成。
How large is your training dataset and model parameter count? The training time can actually be roughly estimated based on the total FLOPs and the GPU performance. For example, with 8 A100 GPUs, training a 1B model on over 100B tokens should normally take about 10 days to complete.

hmm it was trained using multiple different setting. I'll suggest total tokens per batch 4B, and train for 500B steps, def monitor perf. per 100B step.

不好意思，我不是很理解具体的训练数据量。这里说的每个batch 4B什么意思，一般batch全局的话也没这么大。一般像dream训练了0.6T=600B数据，我想请教您一共用了多少B数据呢？另外数据有没有重复训练呢？

[pengzhangzhi]

不太记得 total training tokens了，training setting 是follow open-coder里的，max length 4096， micro batch size 128？应该是有重复训练的。

[skywalker0523]

不太记得 total training tokens了，training setting 是follow open-coder里的，max length 4096， micro batch size 128？应该是有重复训练的。

请问您当时是用多少卡训练了大概多久呢？我在四卡H800也遇到了训练很慢的问题：Epoch 1/1: 0%| | 1/476838 [03:35<28488:35:53, 215.08s/it, loss: 35.31, grad_norm: 2752.00, lr: 6.29e-08]，似乎用4卡H800完成训练的话要200天，不确定是为什么要这么久. 以及我这里尝试发现micro batch size只能设到8，超过的话80G H800的显存就不够了，不确定为什么要这么大的显存，您那边micro batch size如果设置成128似乎需要很大显存的gpu才行

[pengzhangzhi]

it's normal lol micro 是8 global 512

[skywalker0523]

不太记得 total training tokens了，training setting 是follow open-coder里的，max length 4096， micro batch size 128？应该是有重复训练的。

Thank you for your response. Additionally, I couldn't find the parameter to set the max length in your scripts and config files.

[pengzhangzhi]

Open-dLLM/configs/pretrain/qwen2_5_coder_500M.yaml

Line 10 in 109e475

max_seq_len: 4096