The effects of weight quantization on online federated learning for the IoT: a case study

Many weight quantization approaches were explored to save the communication bandwidth between the clients and the server in federated learning using high-end computing machines. However, there is a lack of weight quantization research for online federated learning using TinyML devices which are rest...

Descripción completa

Detalles Bibliográficos
Autores: Llisterri Giménez, Nil, Lee, Junkyu, Freitag, Fèlix|||0000-0001-5438-479X, Vandierendonck, Hans
Tipo de recurso: artículo
Fecha de publicación:2024
País:España
Institución:Universitat Politècnica de Catalunya (UPC)
Repositorio:UPCommons. Portal del coneixement obert de la UPC
Idioma:inglés
OAI Identifier:oai:upcommons.upc.edu:2117/402637
Acceso en línea:https://hdl.handle.net/2117/402637
https://dx.doi.org/10.1109/ACCESS.2024.3349557
Access Level:acceso abierto
Palabra clave:Machine learning
Internet of things
TinyML
Approximate computing
Federated learning
IoT
Aprenentatge automàtic
Internet de les coses
Àrees temàtiques de la UPC::Informàtica::Intel·ligència artificial::Aprenentatge automàtic
id ES_730747bb4e845cbfe144981d138ea3f1
oai_identifier_str oai:upcommons.upc.edu:2117/402637
network_acronym_str ES
network_name_str España
repository_id_str
spelling The effects of weight quantization on online federated learning for the IoT: a case studyLlisterri Giménez, NilLee, JunkyuFreitag, Fèlix|||0000-0001-5438-479XVandierendonck, HansMachine learningInternet of thingsTinyMLApproximate computingFederated learningIoTAprenentatge automàticInternet de les cosesÀrees temàtiques de la UPC::Informàtica::Intel·ligència artificial::Aprenentatge automàticMany weight quantization approaches were explored to save the communication bandwidth between the clients and the server in federated learning using high-end computing machines. However, there is a lack of weight quantization research for online federated learning using TinyML devices which are restricted by the mini-batch size, the neural network size, and the communication method due to their severe hardware resource constraints and power budgets. We name Tiny Online Federated Learning (TinyOFL) for online federated learning using TinyML devices in the Internet of Things (IoT). This paper performs a comprehensive analysis of the effects of weight quantization in TinyOFL in terms of accuracy, stability, overfitting, communication efficiency, energy consumption, and delivery time, and extracts practical guidelines on how to apply the weight quantization to TinyOFL. Our analysis is supported by a TinyOFL case study with three Arduino Portenta H7 boards running federated learning clients for a keyword spotting task. Our findings include that in TinyOFL, a more aggressive weight quantization can be allowed than in online learning without FL, without affecting the accuracy thanks to TinyOFL’s quasi-batch training property. For example, using 7-bit weights achieved the equivalent accuracy to 32-bit floating point weights, while saving communication bandwidth by 4.6× . Overfitting by increasing network width rarely occurs in TinyOFL, but may occur if strong weight quantization is applied. The experiments also showed that there is a design space for TinyOFL applications by compensating for the accuracy loss due to weight quantization with an increase of the neural network size.This work was supported in part by the European Union’s Horizon 2020 Research and Innovation Program (ASSIST-IoT) under Grant 957258, in part by the Spanish Government (DiPET CHIST-ERA) under Contract PID2019-106774RB-C21 and Contract PCI2019-111850-2, and in part by the Generalitat de Catalunya as Consolidated Research Group under Grant 2021-SGR-01059.Peer ReviewedInstitute of Electrical and Electronics Engineers (IEEE)20242024-01-0420242024-02-22journal articlehttp://purl.org/coar/resource_type/c_6501VoRhttp://purl.org/coar/version/c_970fb48d4fbd8a85info:eu-repo/semantics/articleapplication/pdfhttps://hdl.handle.net/2117/402637https://dx.doi.org/10.1109/ACCESS.2024.3349557reponame:UPCommons. Portal del coneixement obert de la UPCinstname:Universitat Politècnica de Catalunya (UPC)InglésengEuropean Commission http://doi.org/10.13039/100010661 Horizon 2020 Framework Programme 957258 Architecture for Scalable, Self-*, human-centric, Intelligent, Secure, and Tactile next generation IoTAgencia Estatal de Investigación http://doi.org/10.13039/501100011033 Plan Estatal de Investigación Científica y Técnica y de Innovación 2017-2020 PID2019-106774RB-C21 SISTEMAS INFORMATICOS Y DE RED DESCENTRALIZADOS CON RECURSOS DISTRIBUIDOSAgencia Estatal de Investigación http://doi.org/10.13039/501100011033 Plan Estatal de Investigación Científica y Técnica y de Innovación 2017-2020 PCI2019-111850-2 PROCESAMIENTO DE FLUJO DISTRIBUIDO EN SISTEMAS DE NIEBLA Y BORDE MEDIANTE COMPUTACION TRANSPRECISAopen accesshttp://purl.org/coar/access_right/c_abf2Attribution-NonCommercial-NoDerivatives 4.0 Internationalhttp://creativecommons.org/licenses/by-nc-nd/4.0/info:eu-repo/semantics/openAccessoai:upcommons.upc.edu:2117/4026372026-05-27T15:37:01Z
dc.title.none.fl_str_mv The effects of weight quantization on online federated learning for the IoT: a case study
title The effects of weight quantization on online federated learning for the IoT: a case study
spellingShingle The effects of weight quantization on online federated learning for the IoT: a case study
Llisterri Giménez, Nil
Machine learning
Internet of things
TinyML
Approximate computing
Federated learning
IoT
Aprenentatge automàtic
Internet de les coses
Àrees temàtiques de la UPC::Informàtica::Intel·ligència artificial::Aprenentatge automàtic
title_short The effects of weight quantization on online federated learning for the IoT: a case study
title_full The effects of weight quantization on online federated learning for the IoT: a case study
title_fullStr The effects of weight quantization on online federated learning for the IoT: a case study
title_full_unstemmed The effects of weight quantization on online federated learning for the IoT: a case study
title_sort The effects of weight quantization on online federated learning for the IoT: a case study
dc.creator.none.fl_str_mv Llisterri Giménez, Nil
Lee, Junkyu
Freitag, Fèlix|||0000-0001-5438-479X
Vandierendonck, Hans
author Llisterri Giménez, Nil
author_facet Llisterri Giménez, Nil
Lee, Junkyu
Freitag, Fèlix|||0000-0001-5438-479X
Vandierendonck, Hans
author_role author
author2 Lee, Junkyu
Freitag, Fèlix|||0000-0001-5438-479X
Vandierendonck, Hans
author2_role author
author
author
dc.subject.none.fl_str_mv Machine learning
Internet of things
TinyML
Approximate computing
Federated learning
IoT
Aprenentatge automàtic
Internet de les coses
Àrees temàtiques de la UPC::Informàtica::Intel·ligència artificial::Aprenentatge automàtic
topic Machine learning
Internet of things
TinyML
Approximate computing
Federated learning
IoT
Aprenentatge automàtic
Internet de les coses
Àrees temàtiques de la UPC::Informàtica::Intel·ligència artificial::Aprenentatge automàtic
description Many weight quantization approaches were explored to save the communication bandwidth between the clients and the server in federated learning using high-end computing machines. However, there is a lack of weight quantization research for online federated learning using TinyML devices which are restricted by the mini-batch size, the neural network size, and the communication method due to their severe hardware resource constraints and power budgets. We name Tiny Online Federated Learning (TinyOFL) for online federated learning using TinyML devices in the Internet of Things (IoT). This paper performs a comprehensive analysis of the effects of weight quantization in TinyOFL in terms of accuracy, stability, overfitting, communication efficiency, energy consumption, and delivery time, and extracts practical guidelines on how to apply the weight quantization to TinyOFL. Our analysis is supported by a TinyOFL case study with three Arduino Portenta H7 boards running federated learning clients for a keyword spotting task. Our findings include that in TinyOFL, a more aggressive weight quantization can be allowed than in online learning without FL, without affecting the accuracy thanks to TinyOFL’s quasi-batch training property. For example, using 7-bit weights achieved the equivalent accuracy to 32-bit floating point weights, while saving communication bandwidth by 4.6× . Overfitting by increasing network width rarely occurs in TinyOFL, but may occur if strong weight quantization is applied. The experiments also showed that there is a design space for TinyOFL applications by compensating for the accuracy loss due to weight quantization with an increase of the neural network size.
publishDate 2024
dc.date.none.fl_str_mv 2024
2024-01-04
2024
2024-02-22
dc.type.none.fl_str_mv journal article
http://purl.org/coar/resource_type/c_6501
VoR
http://purl.org/coar/version/c_970fb48d4fbd8a85
dc.type.openaire.fl_str_mv info:eu-repo/semantics/article
format article
dc.identifier.none.fl_str_mv https://hdl.handle.net/2117/402637
https://dx.doi.org/10.1109/ACCESS.2024.3349557
url https://hdl.handle.net/2117/402637
https://dx.doi.org/10.1109/ACCESS.2024.3349557
dc.language.none.fl_str_mv Inglés
eng
language_invalid_str_mv Inglés
language eng
dc.relation.none.fl_str_mv European Commission http://doi.org/10.13039/100010661 Horizon 2020 Framework Programme 957258 Architecture for Scalable, Self-*, human-centric, Intelligent, Secure, and Tactile next generation IoT
Agencia Estatal de Investigación http://doi.org/10.13039/501100011033 Plan Estatal de Investigación Científica y Técnica y de Innovación 2017-2020 PID2019-106774RB-C21 SISTEMAS INFORMATICOS Y DE RED DESCENTRALIZADOS CON RECURSOS DISTRIBUIDOS
Agencia Estatal de Investigación http://doi.org/10.13039/501100011033 Plan Estatal de Investigación Científica y Técnica y de Innovación 2017-2020 PCI2019-111850-2 PROCESAMIENTO DE FLUJO DISTRIBUIDO EN SISTEMAS DE NIEBLA Y BORDE MEDIANTE COMPUTACION TRANSPRECISA
dc.rights.none.fl_str_mv open access
http://purl.org/coar/access_right/c_abf2
Attribution-NonCommercial-NoDerivatives 4.0 International
http://creativecommons.org/licenses/by-nc-nd/4.0/
dc.rights.openaire.fl_str_mv info:eu-repo/semantics/openAccess
rights_invalid_str_mv open access
http://purl.org/coar/access_right/c_abf2
Attribution-NonCommercial-NoDerivatives 4.0 International
http://creativecommons.org/licenses/by-nc-nd/4.0/
eu_rights_str_mv openAccess
dc.format.none.fl_str_mv application/pdf
dc.publisher.none.fl_str_mv Institute of Electrical and Electronics Engineers (IEEE)
publisher.none.fl_str_mv Institute of Electrical and Electronics Engineers (IEEE)
dc.source.none.fl_str_mv reponame:UPCommons. Portal del coneixement obert de la UPC
instname:Universitat Politècnica de Catalunya (UPC)
instname_str Universitat Politècnica de Catalunya (UPC)
reponame_str UPCommons. Portal del coneixement obert de la UPC
collection UPCommons. Portal del coneixement obert de la UPC
repository.name.fl_str_mv
repository.mail.fl_str_mv
_version_ 1869410779081474048
score 15,300719

Similares en LA Referencia