[Bug]: online-rl sampling is different from offline-sampling

[yyht]

Your current environment

``` ENV: openrlhf-latest version vllm==0.7.2 ```

🐛 Describe the bug

I tried to run ppo/reinforce++ using openrlhf.
The dataset and reward-func is same to https://github.com/Open-Reasoner-Zero/Open-Reasoner-Zero/

However, when i tried to filter data before training, I use the exact the same ENV and sampling-params, the accuracy and format-accuracy is different from the first step of online-sampling.

Since the first step of online-sampling should be identical to offline-sampling, the format-accuracy should be similar. However, there exists a huge gap.
The sampling-params:

sampling_params = SamplingParams(temperature=1.0, 
                                 top_p=1.0,
                                 top_k=-1,
                                 max_tokens=16000+4096,
                                 min_tokens=1,
                                 stop=[tokenizer.eos_token, "User:", "Human:", "Assistant:", "</answer>"],
                                 skip_special_tokens=False,
                                 include_stop_str_in_output=True,
                                 n=1)

online-sampling while offline-sampling, the format-rewards is nearly 15%, while online-sampling is 57%.

When I remove </answer> from the stop-token, online-sampling of the first step is similar to offline-sampling.
It's so odd.

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.