Memory Gym: Partially Observable Challenges to Memory-Based Agents in Endless Episodes

Prior works

Memory Gym: Partially Observable Challenges to Memory-Based Agents in Endless Episodes

Marco Pleines, M. Pallasch, F. Zimmer, M. Preuss
2023

Memory Gym: Towards Endless Tasks to Benchmark Memory Capabilities of Agents

Marco Pleines, M. Pallasch, Frank Zimmer, Mike Preuss
2023

Task Aware Dreamer for Task Generalization in Reinforcement Learning

Chengyang Ying, Zhongkai Hao, Xinning Zhou, Hang Su, Songming Liu, J. Li, Dong Yan, Jun Zhu
2023

Generalization, Mayhems and Limits in Recurrent Proximal Policy Optimization

Marco Pleines, M. Pallasch, F. Zimmer, M. Preuss
2022

Reward Informed Dreamer for Task Generalization in Reinforcement Learning

Chengyang Ying, Zhongkai Hao, Xinning Zhou, Hang Su, Songming Liu, J. Li, Dong Yan, Jun Zhu
2023

Obstacle Tower Without Human Demonstrations: How Far a Deep Feed-Forward Network Goes with Reinforcement Learning

Marco Pleines, J. Jitsev, M. Preuss, Frank Zimmer
2020

Sample Factory: Egocentric 3D Control from Pixels at 100000 FPS with Asynchronous Reinforcement Learning

Aleksei Petrenko, Zhehui Huang, T. Kumar, G. Sukhatme, V. Koltun
2020

NAPPO: Modular and scalable reinforcement learning in pytorch

Albert Bou, G. D. Fabritiis
2020

Structured State Space Models for In-Context Reinforcement Learning

Chris Xiaoxuan Lu, Yannick Schroecker, Albert Gu, Emilio Parisotto, J. Foerster, Satinder Singh, Feryal M. P. Behbahani
2023

Mastering Memory Tasks with World Models

Mohammad Reza Samsami, Artem Zholus, Janarthanan Rajendran, Sarath Chandar
2024

Description

Memory Gym introduces a unique benchmark designed to test Deep Reinforcement Learning agents, specifically comparing Gated Recurrent Unit (GRU) against Transformer-XL (TrXL), on their ability to memorize long sequences, withstand noise, and generalize. It features partially observable 2D environments with discrete controls, namely Mortar Mayhem, Mystery Path, and Searing Spotlights.

These originally finite environments are extrapolated to novel endless tasks that act as an automatic curriculum, drawing inspiration from the car game “I packed my bag”. These endless tasks are not only beneficial for evaluating efficiency but also intriguingly valuable for assessing the effectiveness of approaches in memory-based agents. Given the scarcity of publicly available memory baselines, we contribute an implementation driven by TrXL and Proximal Policy Optimization.

This implementation leverages TrXL as episodic memory using a sliding window approach. In our experiments on the finite environments, TrXL demonstrates superior sample efficiency in Mystery Path and outperforms in Mortar Mayhem. However, GRU is more efficient on Searing Spotlights. Most notably, in all endless tasks, GRU makes a remarkable resurgence, consistently outperforming TrXL by significant margins.