ra

.github/issue_template.md

@@ -3,7 +3,8 @@
 
 ## Checklist
 - [ ] I have installed dependencies via `poetry install` (see [CleanRL's installation guideline](https://docs.cleanrl.dev/get-started/installation/).
-- [ ] I have checked that there is no similar [issue](https://github.com/vwxyzjn/cleanrl/issues) in the repo (required)
+- [ ] I have checked that there is no similar [issue](https://github.com/vwxyzjn/cleanrl/issues) in the repo.
+- [ ] I have checked the [documentation site](https://docs.cleanrl.dev/) and found not relevant information in [GitHub issues](https://github.com/vwxyzjn/cleanrl/issues).
 
 ## Current Behavior
 <!--- Tell us what happens instead of the expected behavior -->

.github/pull_request_template.md

@@ -11,22 +11,23 @@
 ## Checklist:
 <!--- Go over all the following points, and put an `x` in all the boxes that apply. -->
 <!--- If you're unsure about any of these, don't hesitate to ask. We're here to help! -->
-- [ ] I've read the [CONTRIBUTION](https://github.com/vwxyzjn/cleanrl/blob/master/CONTRIBUTING.md) guide (**required**).
+- [ ] I've read the [CONTRIBUTION](https://docs.cleanrl.dev/contribution/) guide (**required**).
 - [ ] I have ensured `pre-commit run --all-files` passes (**required**).
-- [ ] I have updated the documentation and previewed the changes via `mkdocs serve`.
 - [ ] I have updated the tests accordingly (if applicable).
-
-If you are adding new algorithms or your change could result in performance difference, you may need to (re-)run tracked experiments. See https://github.com/vwxyzjn/cleanrl/pull/137 as an example PR. 
-- [ ] I have contacted [vwxyzjn](https://github.com/vwxyzjn) to obtain access to the [openrlbenchmark W&B team](https://wandb.ai/openrlbenchmark) (**required**).
-- [ ] I have tracked applicable experiments in [openrlbenchmark/cleanrl](https://wandb.ai/openrlbenchmark/cleanrl) with `--capture-video` flag toggled on (**required**).
-- [ ] I have added additional documentation and previewed the changes via `mkdocs serve`.
+- [ ] I have updated the documentation and previewed the changes via `mkdocs serve`.
     - [ ] I have explained note-worthy implementation details.
     - [ ] I have explained the logged metrics.
-    - [ ] I have added links to the original paper and related papers (if applicable).
-    - [ ] I have added links to the PR related to the algorithm.
-    - [ ] I have created a table comparing my results against those from reputable sources (i.e., the original paper or other reference implementation).
-    - [ ] I have added the learning curves (in PNG format with `width=500` and `height=300`).
-    - [ ] I have added links to the tracked experiments.
-    - [ ] I have updated the overview sections at the [docs](https://docs.cleanrl.dev/rl-algorithms/overview/) and the [repo](https://github.com/vwxyzjn/cleanrl#overview)
-- [ ] I have updated the tests accordingly (if applicable).
+    - [ ] I have added links to the original paper and related papers.
+
+If you need to run benchmark experiments for a performance-impacting changes:
+
+- [ ] I have contacted @vwxyzjn to obtain access to the [openrlbenchmark W&B team](https://wandb.ai/openrlbenchmark).
+- [ ] I have used the [benchmark utility](/get-started/benchmark-utility/) to submit the tracked experiments to the [openrlbenchmark/cleanrl](https://wandb.ai/openrlbenchmark/cleanrl) W&B project, optionally with `--capture-video`.
+- [ ] I have performed RLops with `python -m openrlbenchmark.rlops`.
+    - For new feature or bug fix:
+        - [ ] I have used the RLops utility to understand the performance impact of the changes and confirmed there is no regression.
+    - For new algorithm:
+        - [ ] I have created a table comparing my results against those from reputable sources (i.e., the original paper or other reference implementation).
+    - [ ] I have added the learning curves generated by the `python -m openrlbenchmark.rlops` utility to the documentation.
+    - [ ] I have added links to the tracked experiments in W&B, generated by `python -m openrlbenchmark.rlops ....your_args... --report`,  to the documentation.
 

.github/workflows/pre-commit.yml

@@ -1,10 +1,7 @@
 name: pre-commit
 
 on:
-  push:
-    branches: [ master ]
-  pull_request:
-    branches: [ master ]
+  push
 jobs:
   build:
     runs-on: ubuntu-latest
@@ -22,3 +19,5 @@ jobs:
         with:
           python-version: ${{ matrix.python-version }}
       - uses: pre-commit/[email protected]
+        with:
+          extra_args: --hook-stage manual --all-files

.github/workflows/tests.yaml

@@ -5,18 +5,13 @@ on:
       - '**/README.md'
       - 'docs/**/*'
       - 'cloud/**/*'
-  pull_request:
-    paths-ignore:
-      - '**/README.md'
-      - 'docs/**/*'
-      - 'cloud/**/*'
 jobs:
   test-core-envs:
     strategy:
       fail-fast: false
       matrix:
         python-version: [3.8]
-        poetry-version: [1.2]
+        poetry-version: [1.3]
         os: [ubuntu-22.04, macos-latest, windows-latest]
     runs-on: ${{ matrix.os }}
     steps:
@@ -31,19 +26,22 @@ jobs:
 
       # classic control tests
       - name: Install core dependencies
-        run: poetry install --with pytest
+        run: poetry install -E pytest
       - name: Downgrade setuptools
         run: poetry run pip install setuptools==59.5.0
       - name: Run core tests
         run: poetry run pytest tests/test_classic_control.py
       - name: Install jax
         if: runner.os == 'Linux' || runner.os == 'macOS'
-        run: poetry install --with jax
+        run: poetry install -E "pytest jax"
       - name: Run core tests with jax
         if: runner.os == 'Linux' || runner.os == 'macOS'
         run: poetry run pytest tests/test_classic_control_jax.py
+      - name: Run gae tests with jax
+        if: runner.os == 'Linux' || runner.os == 'macOS'
+        run: poetry run pytest tests/test_jax_compute_gae.py
       - name: Install tuner dependencies
-        run: poetry install --with optuna
+        run: poetry install -E "pytest optuna"
       - name: Run tuner tests
         run: poetry run pytest tests/test_tuner.py
 
@@ -52,7 +50,7 @@ jobs:
       fail-fast: false
       matrix:
         python-version: [3.8]
-        poetry-version: [1.2]
+        poetry-version: [1.3]
         os: [ubuntu-22.04, macos-latest, windows-latest]
     runs-on: ${{ matrix.os }}
     steps:
@@ -67,14 +65,14 @@ jobs:
 
       # atari tests
       - name: Install atari dependencies
-        run: poetry install --with pytest,atari
+        run: poetry install -E "pytest atari"
       - name: Downgrade setuptools
         run: poetry run pip install setuptools==59.5.0
       - name: Run atari tests
         run: poetry run pytest tests/test_atari.py
       - name: Install jax
         if: runner.os == 'Linux' || runner.os == 'macOS'
-        run: poetry install --with jax
+        run: poetry install -E "pytest atari jax"
       - name: Run core tests with jax
         if: runner.os == 'Linux' || runner.os == 'macOS'
         run: poetry run pytest tests/test_atari_jax.py
@@ -84,7 +82,7 @@ jobs:
       fail-fast: false
       matrix:
         python-version: [3.8]
-        poetry-version: [1.2]
+        poetry-version: [1.3]
         os: [ubuntu-22.04, macos-latest, windows-latest]
     runs-on: ${{ matrix.os }}
     steps:
@@ -99,9 +97,9 @@ jobs:
 
       # pybullet tests
       - name: Install core dependencies
-        run: poetry install --with pytest
+        run: poetry install -E pytest
       - name: Install pybullet dependencies
-        run: poetry install --with pybullet
+        run: poetry install -E "pytest pybullet"
       - name: Downgrade setuptools
         run: poetry run pip install setuptools==59.5.0
       - name: Run pybullet tests
@@ -112,7 +110,7 @@ jobs:
       fail-fast: false
       matrix:
         python-version: [3.8]
-        poetry-version: [1.2]
+        poetry-version: [1.3]
         os: [ubuntu-22.04, macos-latest, windows-latest]
     runs-on: ${{ matrix.os }}
     steps:
@@ -127,19 +125,18 @@ jobs:
 
       # procgen tests
       - name: Install core dependencies
-        run: poetry install --with pytest,procgen
+        run: poetry install -E "pytest procgen"
       - name: Downgrade setuptools
         run: poetry run pip install setuptools==59.5.0
       - name: Run pybullet tests
         run: poetry run pytest tests/test_procgen.py
 
-
   test-mujoco-envs:
     strategy:
       fail-fast: false
       matrix:
         python-version: [3.8]
-        poetry-version: [1.2]
+        poetry-version: [1.3]
         os: [ubuntu-22.04]
     runs-on: ${{ matrix.os }}
     steps:
@@ -153,32 +150,83 @@ jobs:
           poetry-version: ${{ matrix.poetry-version }}
 
       # mujoco tests
-      - name: Install core dependencies
-        run: poetry install --with pytest
-      - name: Install pybullet dependencies
-        run: poetry install --with pybullet
-      - name: Install mujoco dependencies
-        run: poetry install --with mujoco
-      - name: Install jax dependencies
-        run: poetry install --with jax
+      - name: Install dependencies
+        run: poetry install -E "pytest mujoco dm_control"
       - name: Downgrade setuptools
         run: poetry run pip install setuptools==59.5.0
       - name: install mujoco dependencies
+        run: |
+          sudo apt-get update && sudo apt-get -y install libgl1-mesa-glx libosmesa6 libglfw3
+      - name: Run mujoco tests
+        continue-on-error: true # MUJOCO_GL=osmesa results in `free(): invalid pointer`
+        run: poetry run pytest tests/test_mujoco.py
+
+  test-mujoco-envs-windows-mac:
+    strategy:
+      fail-fast: false
+      matrix:
+        python-version: [3.8]
+        poetry-version: [1.3]
+        os: [macos-latest, windows-latest]
+    runs-on: ${{ matrix.os }}
+    steps:
+      - uses: actions/checkout@v2
+      - uses: actions/setup-python@v2
+        with:
+          python-version: ${{ matrix.python-version }}
+      - name: Run image
+        uses: abatilo/[email protected]
+        with:
+          poetry-version: ${{ matrix.poetry-version }}
+
+      # mujoco tests
+      - name: Install dependencies
+        run: poetry install -E "pytest mujoco dm_control"
+      - name: Downgrade setuptools
+        run: poetry run pip install setuptools==59.5.0
+      - name: Run mujoco tests
+        run: poetry run pytest tests/test_mujoco.py
+
+
+  test-mujoco_py-envs:
+    strategy:
+      fail-fast: false
+      matrix:
+        python-version: [3.8]
+        poetry-version: [1.3]
+        os: [ubuntu-22.04]
+    runs-on: ${{ matrix.os }}
+    steps:
+      - uses: actions/checkout@v2
+      - uses: actions/setup-python@v2
+        with:
+          python-version: ${{ matrix.python-version }}
+      - name: Run image
+        uses: abatilo/[email protected]
+        with:
+          poetry-version: ${{ matrix.poetry-version }}
+
+      # mujoco_py tests
+      - name: Install dependencies
+        run: poetry install -E "pytest pybullet mujoco_py mujoco jax"
+      - name: Downgrade setuptools
+        run: poetry run pip install setuptools==59.5.0
+      - name: install mujoco_py dependencies
         run: |
           sudo apt-get update && sudo apt-get -y install wget unzip software-properties-common \
             libgl1-mesa-dev \
             libgl1-mesa-glx \
             libglew-dev \
             libosmesa6-dev patchelf
-      - name: Run mujoco tests
-        run: poetry run pytest tests/test_mujoco.py
+      - name: Run mujoco_py tests
+        run: poetry run pytest tests/test_mujoco_py.py
 
   test-envpool-envs:
     strategy:
       fail-fast: false
       matrix:
         python-version: [3.8]
-        poetry-version: [1.2]
+        poetry-version: [1.3]
         os: [ubuntu-22.04]
     runs-on: ${{ matrix.os }}
     steps:
@@ -193,7 +241,7 @@ jobs:
 
       # envpool tests
       - name: Install envpool dependencies
-        run: poetry install --with pytest,envpool,jax
+        run: poetry install -E "pytest envpool jax"
       - name: Downgrade setuptools
         run: poetry run pip install setuptools==59.5.0
       - name: Run envpool tests
@@ -204,7 +252,7 @@ jobs:
       fail-fast: false
       matrix:
         python-version: [3.8]
-        poetry-version: [1.2]
+        poetry-version: [1.3]
         os: [ubuntu-22.04]
     runs-on: ${{ matrix.os }}
     steps:
@@ -219,7 +267,7 @@ jobs:
 
       # atari multigpu tests
       - name: Install atari dependencies
-        run: poetry install --with pytest,atari
+        run: poetry install -E "pytest atari"
       - name: Downgrade setuptools
         run: poetry run pip install setuptools==59.5.0
       - name: Run atari tests
@@ -230,7 +278,7 @@ jobs:
       fail-fast: false
       matrix:
         python-version: [3.8]
-        poetry-version: [1.2]
+        poetry-version: [1.3]
         os: [ubuntu-22.04, macos-latest]
     runs-on: ${{ matrix.os }}
     steps:
@@ -245,10 +293,10 @@ jobs:
 
       # pettingzoo tests
       - name: Install pettingzoo dependencies
-        run: poetry install --with pytest,pettingzoo,atari
+        run: poetry install -E "pytest pettingzoo atari"
       - name: Downgrade setuptools
         run: poetry run pip install setuptools==59.5.0
       - name: Install ROMs
         run: poetry run AutoROM --accept-license
       - name: Run pettingzoo tests
-        run: poetry run pytest tests/test_pettingzoo_ma_atari.py
+        run: poetry run pytest tests/test_pettingzoo_ma_atari.py

.github/workflows/utils_test.yaml

@@ -11,7 +11,7 @@ jobs:
       fail-fast: false
       matrix:
         python-version: [3.8]
-        poetry-version: [1.2]
+        poetry-version: [1.3]
         os: [ubuntu-22.04]
     runs-on: ${{ matrix.os }}
     steps:
@@ -25,9 +25,9 @@ jobs:
           poetry-version: ${{ matrix.poetry-version }}
 
       - name: Install test dependencies
-        run: poetry install --with pytest
+        run: poetry install -E pytest
       - name: Install cloud dependencies
-        run: poetry install --with cloud
+        run: poetry install -E "pytest cloud"
       - name: Downgrade setuptools
         run: poetry run pip install setuptools==59.5.0
       - name: Run utils tests

.gitignore

@@ -1,3 +1,4 @@
+runs
 balance_bot.xml
 cleanrl/ppo_continuous_action_isaacgym/isaacgym/examples
 cleanrl/ppo_continuous_action_isaacgym/isaacgym/isaacgym

.gitpod.Dockerfile

@@ -9,10 +9,10 @@ RUN sudo apt-get update && \
 
 # install python dependencies
 RUN mkdir cleanrl_utils && touch cleanrl_utils/__init__.py
-RUN pip install poetry
+RUN pip install poetry --upgrade
 RUN poetry config virtualenvs.in-project true
 
-# install mujoco
+# install mujoco_py
 RUN sudo apt-get -y install wget unzip software-properties-common \
     libgl1-mesa-dev \
     libgl1-mesa-glx \