CM3 framework for deep multi-agent reinforcement learning in football

Collaboration amongst agents in various multi-agent cooperative and mixed environments has been extensively studied in the field of Deep Multi-Agent Reinforcement Learning. This cooperative behavior and roles emerging out of such cooperation could be beneficial for the agents collectively when they...

Descripción completa

Detalles Bibliográficos
Autor: Patel, Shivani
Tipo de recurso: tesis de maestría
Fecha de publicación:2023
País:España
Institución:Universitat Politècnica de Catalunya (UPC)
Repositorio:UPCommons. Portal del coneixement obert de la UPC
Idioma:inglés
OAI Identifier:oai:upcommons.upc.edu:2117/407134
Acceso en línea:https://hdl.handle.net/2117/407134
Access Level:acceso abierto
Palabra clave:Multiagent systems
Reinforcement learning
Deep Multi-Agent RL
Curriculum Learning
Unity ML-Agents
Sistemes multiagent
Aprenentatge per reforç
Àrees temàtiques de la UPC::Informàtica::Intel·ligència artificial
id ES_bfbe2701a34862a7a4c8ed3e2585ce05
oai_identifier_str oai:upcommons.upc.edu:2117/407134
network_acronym_str ES
network_name_str España
repository_id_str
spelling CM3 framework for deep multi-agent reinforcement learning in footballPatel, ShivaniMultiagent systemsReinforcement learningDeep Multi-Agent RLCurriculum LearningUnity ML-AgentsSistemes multiagentAprenentatge per reforçÀrees temàtiques de la UPC::Informàtica::Intel·ligència artificialCollaboration amongst agents in various multi-agent cooperative and mixed environments has been extensively studied in the field of Deep Multi-Agent Reinforcement Learning. This cooperative behavior and roles emerging out of such cooperation could be beneficial for the agents collectively when they align their individual objectives towards a common goal, share resources effectively, and communicate efficiently to optimize their combined efforts. Research spans across various sub-areas, namely communication in MARL (Comm-MARL), intrinsic rewards, exploration in MARL, curriculum learning, reward shaping, and emergent behavior. Cooperative Multi-Goal Multi-Stage Multi-Agent RL, abbreviated as CM3 is one such framework that uses curriculum learning and a specialized policy function to tackle the issues of efficient exploration and credit assignment respectively. It has been tested on 3 multi-agent environments to demonstrate its power by learning significantly faster than direct adaptations of existing algorithms. As part of this thesis, we have hypothesized if the domain of football from a multi-agent perspective benefits from CM3. Taking notes from the intersection of reinforcement learning and football, and some of the current state-of-the-art football algorithms, such as TiKick and WeKick which are based primarily on PPO, we see how actor-critic algorithms like A2C and PPO compare when used in our multi-agent environment. For this demonstration, we have leveraged a modified version of the Unity ML-Agents' SoccerTwos environment. We also propose an additional enhancement to the original CM3 framework by extending the training further to a 3rd stage when the reward is independent of the goal. We hypothesized that it could enhance coordination because there'd be a single common, collective goal for the team - to win the match - as opposed to the individual goals of scoring or saving.Universitat Politècnica de CatalunyaRio Doval, AnnaMorales, MiguelIsbell, Charles20232023-10-1920242024-04-25master thesishttp://purl.org/coar/resource_type/c_bdccNAhttp://purl.org/coar/version/c_be7fb7dd8ff6fe43info:eu-repo/semantics/masterThesisapplication/pdfapplication/pdfhttps://hdl.handle.net/2117/407134reponame:UPCommons. Portal del coneixement obert de la UPCinstname:Universitat Politècnica de Catalunya (UPC)Inglésengopen accesshttp://purl.org/coar/access_right/c_abf2info:eu-repo/semantics/openAccessoai:upcommons.upc.edu:2117/4071342026-05-27T15:37:01Z
dc.title.none.fl_str_mv CM3 framework for deep multi-agent reinforcement learning in football
title CM3 framework for deep multi-agent reinforcement learning in football
spellingShingle CM3 framework for deep multi-agent reinforcement learning in football
Patel, Shivani
Multiagent systems
Reinforcement learning
Deep Multi-Agent RL
Curriculum Learning
Unity ML-Agents
Sistemes multiagent
Aprenentatge per reforç
Àrees temàtiques de la UPC::Informàtica::Intel·ligència artificial
title_short CM3 framework for deep multi-agent reinforcement learning in football
title_full CM3 framework for deep multi-agent reinforcement learning in football
title_fullStr CM3 framework for deep multi-agent reinforcement learning in football
title_full_unstemmed CM3 framework for deep multi-agent reinforcement learning in football
title_sort CM3 framework for deep multi-agent reinforcement learning in football
dc.creator.none.fl_str_mv Patel, Shivani
author Patel, Shivani
author_facet Patel, Shivani
author_role author
dc.contributor.none.fl_str_mv Rio Doval, Anna
Morales, Miguel
Isbell, Charles
dc.subject.none.fl_str_mv Multiagent systems
Reinforcement learning
Deep Multi-Agent RL
Curriculum Learning
Unity ML-Agents
Sistemes multiagent
Aprenentatge per reforç
Àrees temàtiques de la UPC::Informàtica::Intel·ligència artificial
topic Multiagent systems
Reinforcement learning
Deep Multi-Agent RL
Curriculum Learning
Unity ML-Agents
Sistemes multiagent
Aprenentatge per reforç
Àrees temàtiques de la UPC::Informàtica::Intel·ligència artificial
description Collaboration amongst agents in various multi-agent cooperative and mixed environments has been extensively studied in the field of Deep Multi-Agent Reinforcement Learning. This cooperative behavior and roles emerging out of such cooperation could be beneficial for the agents collectively when they align their individual objectives towards a common goal, share resources effectively, and communicate efficiently to optimize their combined efforts. Research spans across various sub-areas, namely communication in MARL (Comm-MARL), intrinsic rewards, exploration in MARL, curriculum learning, reward shaping, and emergent behavior. Cooperative Multi-Goal Multi-Stage Multi-Agent RL, abbreviated as CM3 is one such framework that uses curriculum learning and a specialized policy function to tackle the issues of efficient exploration and credit assignment respectively. It has been tested on 3 multi-agent environments to demonstrate its power by learning significantly faster than direct adaptations of existing algorithms. As part of this thesis, we have hypothesized if the domain of football from a multi-agent perspective benefits from CM3. Taking notes from the intersection of reinforcement learning and football, and some of the current state-of-the-art football algorithms, such as TiKick and WeKick which are based primarily on PPO, we see how actor-critic algorithms like A2C and PPO compare when used in our multi-agent environment. For this demonstration, we have leveraged a modified version of the Unity ML-Agents' SoccerTwos environment. We also propose an additional enhancement to the original CM3 framework by extending the training further to a 3rd stage when the reward is independent of the goal. We hypothesized that it could enhance coordination because there'd be a single common, collective goal for the team - to win the match - as opposed to the individual goals of scoring or saving.
publishDate 2023
dc.date.none.fl_str_mv 2023
2023-10-19
2024
2024-04-25
dc.type.none.fl_str_mv master thesis
http://purl.org/coar/resource_type/c_bdcc
NA
http://purl.org/coar/version/c_be7fb7dd8ff6fe43
dc.type.openaire.fl_str_mv info:eu-repo/semantics/masterThesis
format masterThesis
dc.identifier.none.fl_str_mv https://hdl.handle.net/2117/407134
url https://hdl.handle.net/2117/407134
dc.language.none.fl_str_mv Inglés
eng
language_invalid_str_mv Inglés
language eng
dc.rights.none.fl_str_mv open access
http://purl.org/coar/access_right/c_abf2
dc.rights.openaire.fl_str_mv info:eu-repo/semantics/openAccess
rights_invalid_str_mv open access
http://purl.org/coar/access_right/c_abf2
eu_rights_str_mv openAccess
dc.format.none.fl_str_mv application/pdf
application/pdf
dc.publisher.none.fl_str_mv Universitat Politècnica de Catalunya
publisher.none.fl_str_mv Universitat Politècnica de Catalunya
dc.source.none.fl_str_mv reponame:UPCommons. Portal del coneixement obert de la UPC
instname:Universitat Politècnica de Catalunya (UPC)
instname_str Universitat Politècnica de Catalunya (UPC)
reponame_str UPCommons. Portal del coneixement obert de la UPC
collection UPCommons. Portal del coneixement obert de la UPC
repository.name.fl_str_mv
repository.mail.fl_str_mv
_version_ 1869418417513037824
score 15.301603

Similares en LA Referencia