Exploiting morphological symmetries in offline reinforcement learning
Reinforcement learning has enabled robotic agents to learn complex tasks, from locomotion to manipulation. While this usually requires interaction with the environment, such interaction can be costly or impractical. In these cases, offline reinforcement learning (ORL) allows agents to learn from pre...
| Autor: | |
|---|---|
| Tipo de recurso: | tesis de maestría |
| Fecha de publicación: | 2025 |
| País: | España |
| Institución: | Universitat Politècnica de Catalunya (UPC) |
| Repositorio: | UPCommons. Portal del coneixement obert de la UPC |
| Idioma: | inglés |
| OAI Identifier: | oai:upcommons.upc.edu:2117/445043 |
| Acceso en línea: | https://hdl.handle.net/2117/445043 |
| Access Level: | acceso abierto |
| Palabra clave: | Group theory Reinforcement learning Aprenentatge per reforçament Robòtica Teoria de grups Augment de dades Xarxa neuronal equivariant Aprenentatge per reforçament fora de línia Simetries de MDP Data augmentation Equivariant neural network Offline reinforcement learning Symmetry Morphological symmetries MDP symmetries Grups, Teoria de Aprenentatge per reforç Àrees temàtiques de la UPC::Informàtica::Intel·ligència artificial::Aprenentatge automàtic |
| Sumario: | Reinforcement learning has enabled robotic agents to learn complex tasks, from locomotion to manipulation. While this usually requires interaction with the environment, such interaction can be costly or impractical. In these cases, offline reinforcement learning (ORL) allows agents to learn from pre-collected data instead. However, this paradigm introduces challenges such as extrapolation error and the inability to explore beyond the dataset, making sufficient and diverse data essential. Many robots also exhibit structural regularities that preserve system dynamics under transformations. We refer to these as morphological symmetries, which can be formalized with group theory, applied with representation theory, and interpreted as symmetries of the underlying MDP. In this thesis, we explore how exploiting morphological symmetries can improve data efficiency, motion consistency, and generalization in ORL. Specifically, we investigate two complementary approaches: (1) data augmentation via symmetry transformations and (2) equivariant neural architectures based on invariant and equivariant MLPs. We evaluate their performance across multiple robotic environments and datasets of varying quality, and propose an extension of TD3+BC, RAISymE(TD3+BC), that mitigates mean-seeking behavior arising from dataset multimodality introduced through symmetry-based augmentation. Our results show that, when the behavior policy induces an overlapping support across symmetric regions of the state space, exploiting morphological symmetries leads to consistent performance gains in data-scarce scenarios and promotes more symmetric policies. |
|---|