Upgrade gymnasium to 1.0.0

[sdpkjc]

Description

Upgrade gymnasium to 1.0.0

• gymnasium classic control

  • c51.py
  • c51_jax.py
  • dqn.py
  • dqn_jax.py
  • ppo.py
  • pqn.py

• gymnasium mujoco

  • ddpg_continuous_action.py
  • ddpg_continuous_action_jax.py
  • td3_continuous_action.py
  • td3_continuous_action_jax.py
  • sac_continuous_action.py
  • ppo_continuous_action.py
  • rpo_continuous_action.py

• gymnasium atari (EpisodicLifeEnv conflicts with gymnasium v1.0.0's RecordEpisodeStatistics and will be fixed later.)

  • c51_atari.py
  • c51_atari_jax.py
  • dqn_atari.py
  • dqn_atari_jax.py
  • qdagger_dqn_atari_impalacnn.py
  • qdagger_dqn_atari_jax_impalacnn.py
  • sac_atari.py
  • ppo_atari.py
  • ppo_atari_lstm.py
  • ppo_atari_multigpu.py

• envpool

  • ppo_rnd_envpool.py
  • pqn_atari_envpool_lstm.py
  • pqn_atari_envpool.py
  • ppo_atari_envpool.py
  • ppo_atari_envpool_xla_jax.py
  • ppo_atari_envpool_xla_jax_scan.py

• other

  • ppg_procgen.py
  • ppo_pettingzoo_ma_atari.py
  • ppo_procgen.py
  • ppo_trxl.py
  • ppo_continuous_action_isaacgym.py

Types of changes

• Bug fix
• New feature
• New algorithm
• Documentation

Checklist:

• I've read the CONTRIBUTION guide (required).
• I have ensured pre-commit run --all-files passes (required).
• I have updated the tests accordingly (if applicable).
• I have updated the documentation and previewed the changes via mkdocs serve.
  • I have explained note-worthy implementation details.
  • I have explained the logged metrics.
  • I have added links to the original paper and related papers.

If you need to run benchmark experiments for a performance-impacting changes:

• I have contacted @vwxyzjn to obtain access to the openrlbenchmark W&B team.
• I have used the benchmark utility to submit the tracked experiments to the openrlbenchmark/cleanrl W&B project, optionally with --capture_video.
• I have performed RLops with python -m openrlbenchmark.rlops.
  • For new feature or bug fix:
    • I have used the RLops utility to understand the performance impact of the changes and confirmed there is no regression.
  • For new algorithm:
    • I have created a table comparing my results against those from reputable sources (i.e., the original paper or other reference implementation).
  • I have added the learning curves generated by the python -m openrlbenchmark.rlops utility to the documentation.
  • I have added links to the tracked experiments in W&B, generated by python -m openrlbenchmark.rlops ....your_args... --report, to the documentation.

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
cleanrl	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Mar 4, 2025 9:24am

[pseudo-rnd-thoughts]

Thanks for starting this @sdpkjc

Ale-py should be updated to v0.10.1

And the auto reset mode of the vector environment should be updated

[pseudo-rnd-thoughts]

Can this be specified as v1.1.0, if sb3 is the limitation then I think see if I can update it

[sdpkjc]

Yes, sb3 depends on <=1.0.0.

[pseudo-rnd-thoughts]

If I remember correctly, SB3 is used for the replay buffer and the atari wrappers. IMO, those features can probably be shifted in cleanrl_utils directly however this should happen in a separate later PR

[pseudo-rnd-thoughts]

This should probably be called SameStepModeSyncVectorEnv or we just shift to gymnasium v1.1.0

[pseudo-rnd-thoughts]

https://farama.org/Vector-Autoreset-Mode

[pseudo-rnd-thoughts]

What is limiting us increasing this?

[pseudo-rnd-thoughts]

I suspect we will have a new release of sb3 with support for gymnasium v1.1.0 as no changes seem to be required on their end (DLR-RM/stable-baselines3#2095)