Q-Learning for Online PID Controller Tuning in Continuous Dynamic Systems: An Interpretable Framework for Exploring Multi-Agent Systems

[EN] This study proposes a discrete multi-agent Q-learning framework for online tuning of PID controllers in continuous dynamic systems with limited observability. Each PID gain (Kp, Ki, Kd) is adjusted by an independent learning agent operating in a discrete state space defined by its own gain valu...

Descripción completa

Detalles Bibliográficos
Autores: Ibarra-Pérez, Davor Matías Samuel, Garcia-Nieto, Sergio|||0000-0002-2722-742X, Sanchís Saez, Javier|||0000-0001-9697-2696
Tipo de recurso: artículo
Fecha de publicación:2025
País:España
Institución:Universitat Politècnica de València (UPV)
Repositorio:RiuNet. Repositorio Institucional de la Universitat Politécnica de Valéncia
Idioma:inglés
OAI Identifier:oai:riunet.upv.es:10251/231762
Acceso en línea:https://riunet.upv.es/handle/10251/231762
Access Level:acceso abierto
Palabra clave:Q-learning
Multi-agents
Proportional–Integral–Derivative (PID)
Online controller
Interpretable control
id ES_018a6bcb0d36d2b40fb0220db278dab8
oai_identifier_str oai:riunet.upv.es:10251/231762
network_acronym_str ES
network_name_str España
repository_id_str
spelling Q-Learning for Online PID Controller Tuning in Continuous Dynamic Systems: An Interpretable Framework for Exploring Multi-Agent SystemsIbarra-Pérez, Davor Matías SamuelGarcia-Nieto, Sergio|||0000-0002-2722-742XSanchís Saez, Javier|||0000-0001-9697-2696Q-learningMulti-agentsProportional–Integral–Derivative (PID)Online controllerInterpretable control[EN] This study proposes a discrete multi-agent Q-learning framework for online tuning of PID controllers in continuous dynamic systems with limited observability. Each PID gain (Kp, Ki, Kd) is adjusted by an independent learning agent operating in a discrete state space defined by its own gain value. At each decision step, agents choose among three actions: decrease, keep, or increase the gain, and they act simultaneously at fixed decision intervals. This design helps preserve quasi-stationary conditions from the agents’ perspective and supports convergence. Coordination is achieved through a shared cumulative global reward that combines system-performance terms with time and control-effort penalties, as well as stability incentives, guiding exploration toward control objectives. Implemented in Python, the framework is validated on two nonlinear control problems: a water-tank system and an inverted pendulum (cart-pole). The agents reach initial convergence after approximately 300 and 500 episodes, respectively, and obtain overall success rates of 49.6% and 46.2% over 5,000 training episodes. Results show sustained learning toward effective PID parameter settings capable of stabilizing both systems without requiring explicit dynamic models. These findings support the feasibility of a low-complexity, discrete reinforcement-learning approach for online adaptive PID tuning, producing interpretable and reproducible control policies and providing a foundation for future hybrid schemes combining classical control theory with reinforcement-learning agents.This research was funded by Vicerrectorado de Investigación de la Universitat Politècnica de València (PAID-11-24) and supported by Agencia Nacional de Investigación de Chile ANID BECAS/DOCTORADO EN EL EXTRANJERO 72230436-2023.MDPI AGDepartamento de Ingeniería de Sistemas y AutomáticaEscuela Técnica Superior de Ingeniería Aeroespacial y Diseño IndustrialInstituto Universitario de Automática e Informática IndustrialEscuela Técnica Superior de Ingeniería IndustrialUNIVERSIDAD POLITECNICA DE VALENCIAAgencia Nacional de Investigación y Desarrollo de ChileRepositorio Institucional de la Universitat Politècnica de València Riunet20252025-10-01journal articlehttp://purl.org/coar/resource_type/c_6501VoRhttp://purl.org/coar/version/c_970fb48d4fbd8a85info:eu-repo/semantics/articleapplication/pdfhttps://riunet.upv.es/handle/10251/231762reponame:RiuNet. Repositorio Institucional de la Universitat Politécnica de Valénciainstname:Universitat Politècnica de València (UPV)InglésengUniversitat Politècnica de València https://doi.org/10.13039/501100004233 PAID-11-24 A novel proposal for the design, control and optimization of a zero-emission multigeneration architecture based on renewable energy and green H2 (NET-MULTIGENERA)Agencia Nacional de Investigación y Desarrollo, Chile ANID 72230436-2023open accesshttp://purl.org/coar/access_right/c_abf2Reconocimiento (by)http://creativecommons.org/licenses/by/4.0/info:eu-repo/semantics/openAccessoai:riunet.upv.es:10251/2317622026-06-13T07:49:27Z
dc.title.none.fl_str_mv Q-Learning for Online PID Controller Tuning in Continuous Dynamic Systems: An Interpretable Framework for Exploring Multi-Agent Systems
title Q-Learning for Online PID Controller Tuning in Continuous Dynamic Systems: An Interpretable Framework for Exploring Multi-Agent Systems
spellingShingle Q-Learning for Online PID Controller Tuning in Continuous Dynamic Systems: An Interpretable Framework for Exploring Multi-Agent Systems
Ibarra-Pérez, Davor Matías Samuel
Q-learning
Multi-agents
Proportional–Integral–Derivative (PID)
Online controller
Interpretable control
title_short Q-Learning for Online PID Controller Tuning in Continuous Dynamic Systems: An Interpretable Framework for Exploring Multi-Agent Systems
title_full Q-Learning for Online PID Controller Tuning in Continuous Dynamic Systems: An Interpretable Framework for Exploring Multi-Agent Systems
title_fullStr Q-Learning for Online PID Controller Tuning in Continuous Dynamic Systems: An Interpretable Framework for Exploring Multi-Agent Systems
title_full_unstemmed Q-Learning for Online PID Controller Tuning in Continuous Dynamic Systems: An Interpretable Framework for Exploring Multi-Agent Systems
title_sort Q-Learning for Online PID Controller Tuning in Continuous Dynamic Systems: An Interpretable Framework for Exploring Multi-Agent Systems
dc.creator.none.fl_str_mv Ibarra-Pérez, Davor Matías Samuel
Garcia-Nieto, Sergio|||0000-0002-2722-742X
Sanchís Saez, Javier|||0000-0001-9697-2696
author Ibarra-Pérez, Davor Matías Samuel
author_facet Ibarra-Pérez, Davor Matías Samuel
Garcia-Nieto, Sergio|||0000-0002-2722-742X
Sanchís Saez, Javier|||0000-0001-9697-2696
author_role author
author2 Garcia-Nieto, Sergio|||0000-0002-2722-742X
Sanchís Saez, Javier|||0000-0001-9697-2696
author2_role author
author
dc.contributor.none.fl_str_mv Departamento de Ingeniería de Sistemas y Automática
Escuela Técnica Superior de Ingeniería Aeroespacial y Diseño Industrial
Instituto Universitario de Automática e Informática Industrial
Escuela Técnica Superior de Ingeniería Industrial
UNIVERSIDAD POLITECNICA DE VALENCIA
Agencia Nacional de Investigación y Desarrollo de Chile
Repositorio Institucional de la Universitat Politècnica de València Riunet
dc.subject.none.fl_str_mv Q-learning
Multi-agents
Proportional–Integral–Derivative (PID)
Online controller
Interpretable control
topic Q-learning
Multi-agents
Proportional–Integral–Derivative (PID)
Online controller
Interpretable control
description [EN] This study proposes a discrete multi-agent Q-learning framework for online tuning of PID controllers in continuous dynamic systems with limited observability. Each PID gain (Kp, Ki, Kd) is adjusted by an independent learning agent operating in a discrete state space defined by its own gain value. At each decision step, agents choose among three actions: decrease, keep, or increase the gain, and they act simultaneously at fixed decision intervals. This design helps preserve quasi-stationary conditions from the agents’ perspective and supports convergence. Coordination is achieved through a shared cumulative global reward that combines system-performance terms with time and control-effort penalties, as well as stability incentives, guiding exploration toward control objectives. Implemented in Python, the framework is validated on two nonlinear control problems: a water-tank system and an inverted pendulum (cart-pole). The agents reach initial convergence after approximately 300 and 500 episodes, respectively, and obtain overall success rates of 49.6% and 46.2% over 5,000 training episodes. Results show sustained learning toward effective PID parameter settings capable of stabilizing both systems without requiring explicit dynamic models. These findings support the feasibility of a low-complexity, discrete reinforcement-learning approach for online adaptive PID tuning, producing interpretable and reproducible control policies and providing a foundation for future hybrid schemes combining classical control theory with reinforcement-learning agents.
publishDate 2025
dc.date.none.fl_str_mv 2025
2025-10-01
dc.type.none.fl_str_mv journal article
http://purl.org/coar/resource_type/c_6501
VoR
http://purl.org/coar/version/c_970fb48d4fbd8a85
dc.type.openaire.fl_str_mv info:eu-repo/semantics/article
format article
dc.identifier.none.fl_str_mv https://riunet.upv.es/handle/10251/231762
url https://riunet.upv.es/handle/10251/231762
dc.language.none.fl_str_mv Inglés
eng
language_invalid_str_mv Inglés
language eng
dc.relation.none.fl_str_mv Universitat Politècnica de València https://doi.org/10.13039/501100004233 PAID-11-24 A novel proposal for the design, control and optimization of a zero-emission multigeneration architecture based on renewable energy and green H2 (NET-MULTIGENERA)
Agencia Nacional de Investigación y Desarrollo, Chile ANID 72230436-2023
dc.rights.none.fl_str_mv open access
http://purl.org/coar/access_right/c_abf2
Reconocimiento (by)
http://creativecommons.org/licenses/by/4.0/
dc.rights.openaire.fl_str_mv info:eu-repo/semantics/openAccess
rights_invalid_str_mv open access
http://purl.org/coar/access_right/c_abf2
Reconocimiento (by)
http://creativecommons.org/licenses/by/4.0/
eu_rights_str_mv openAccess
dc.format.none.fl_str_mv application/pdf
dc.publisher.none.fl_str_mv MDPI AG
publisher.none.fl_str_mv MDPI AG
dc.source.none.fl_str_mv reponame:RiuNet. Repositorio Institucional de la Universitat Politécnica de Valéncia
instname:Universitat Politècnica de València (UPV)
instname_str Universitat Politècnica de València (UPV)
reponame_str RiuNet. Repositorio Institucional de la Universitat Politécnica de Valéncia
collection RiuNet. Repositorio Institucional de la Universitat Politécnica de Valéncia
repository.name.fl_str_mv
repository.mail.fl_str_mv
_version_ 1869402591183503360
score 15,812429

Similares en LA Referencia