Q-Learning for Online PID Controller Tuning in Continuous Dynamic Systems: An Interpretable Framework for Exploring Multi-Agent Systems
[EN] This study proposes a discrete multi-agent Q-learning framework for online tuning of PID controllers in continuous dynamic systems with limited observability. Each PID gain (Kp, Ki, Kd) is adjusted by an independent learning agent operating in a discrete state space defined by its own gain valu...
| Autores: | , , |
|---|---|
| Tipo de recurso: | artículo |
| Fecha de publicación: | 2025 |
| País: | España |
| Institución: | Universitat Politècnica de València (UPV) |
| Repositorio: | RiuNet. Repositorio Institucional de la Universitat Politécnica de Valéncia |
| Idioma: | inglés |
| OAI Identifier: | oai:riunet.upv.es:10251/231762 |
| Acceso en línea: | https://riunet.upv.es/handle/10251/231762 |
| Access Level: | acceso abierto |
| Palabra clave: | Q-learning Multi-agents Proportional–Integral–Derivative (PID) Online controller Interpretable control |
| Sumario: | [EN] This study proposes a discrete multi-agent Q-learning framework for online tuning of PID controllers in continuous dynamic systems with limited observability. Each PID gain (Kp, Ki, Kd) is adjusted by an independent learning agent operating in a discrete state space defined by its own gain value. At each decision step, agents choose among three actions: decrease, keep, or increase the gain, and they act simultaneously at fixed decision intervals. This design helps preserve quasi-stationary conditions from the agents’ perspective and supports convergence. Coordination is achieved through a shared cumulative global reward that combines system-performance terms with time and control-effort penalties, as well as stability incentives, guiding exploration toward control objectives. Implemented in Python, the framework is validated on two nonlinear control problems: a water-tank system and an inverted pendulum (cart-pole). The agents reach initial convergence after approximately 300 and 500 episodes, respectively, and obtain overall success rates of 49.6% and 46.2% over 5,000 training episodes. Results show sustained learning toward effective PID parameter settings capable of stabilizing both systems without requiring explicit dynamic models. These findings support the feasibility of a low-complexity, discrete reinforcement-learning approach for online adaptive PID tuning, producing interpretable and reproducible control policies and providing a foundation for future hybrid schemes combining classical control theory with reinforcement-learning agents. |
|---|