Self-Supervised High-Order Information Bottleneck Learning of Spiking Neural Network for Robust Event-Based Optical Flow Estimation

Event cameras form a fundamental foundation for visual perception in scenes characterized by high speed and a wide dynamic range. Although deep learning techniques have achieved remarkable success in estimating event-based optical flow, existing methods have not adequately addressed the significance...

Descripción completa

Detalles Bibliográficos
Autores: Yang, Shuangming, Linares-Barranco, Bernabé, Wu, Yuzhu, Chen, Badong
Tipo de recurso: artículo
Estado:Versión publicada
Fecha de publicación:2025
País:España
Institución:Consejo Superior de Investigaciones Científicas (CSIC)
Repositorio:DIGITAL.CSIC. Repositorio Institucional del CSIC
OAI Identifier:oai:digital.csic.es:10261/402210
Acceso en línea:http://hdl.handle.net/10261/402210
https://api.elsevier.com/content/abstract/scopus_id/86000425314
Access Level:acceso abierto
Palabra clave:Event-based vision
Neuromorphic computing
Optical flow estimation
Self-supervised learning
Spiking neural network
id ES_0f9aac9d1efda9155d51204e2a7dfab7
oai_identifier_str oai:digital.csic.es:10261/402210
network_acronym_str ES
network_name_str España
repository_id_str
spelling Self-Supervised High-Order Information Bottleneck Learning of Spiking Neural Network for Robust Event-Based Optical Flow EstimationYang, ShuangmingLinares-Barranco, BernabéWu, YuzhuChen, BadongEvent-based visionNeuromorphic computingOptical flow estimationSelf-supervised learningSpiking neural networkEvent cameras form a fundamental foundation for visual perception in scenes characterized by high speed and a wide dynamic range. Although deep learning techniques have achieved remarkable success in estimating event-based optical flow, existing methods have not adequately addressed the significance of temporal information in capturing spatiotemporal features. Due to the dynamics of spiking neurons in SNNs, which preserve important information while forgetting redundant information over time, they are expected to outperform analog neural networks (ANNs) with the same architecture and size in sequential regression tasks. In addition, SNNs on neuromorphic hardware achieve advantages of extremely low power consumption. However, present SNN architectures encounter issues related to limited generalization and robustness during training, particularly in noisy scenes. To tackle these problems, this study introduces an innovative spike-based self-supervised learning algorithm known as SeLHIB, which leverages the information bottleneck theory. By utilizing event-based camera inputs, SeLHIB enables robust estimation of optical flow in the presence of noise. To the best of our knowledge, this is the first proposal of a self-supervised information bottleneck learning strategy based on SNNs. Furthermore, we develop spike-based self-supervised algorithms with nonlinear and high-order information bottleneck learning that employs nonlinear and high-order mutual information to enhance the extraction of relevant information and eliminate redundancy. We demonstrate that SeLHIB significantly enhances the generalization ability and robustness of optical flow estimation in various noise conditions. In terms of energy efficiency, SeLHIB achieves 90.44% and 45.70% cut down of energy consumption compared to its counterpart ANN and counterpart SNN models, while attaining 33.78% lower AEE (MVSEC), 5.96% lower RSAT (ECD) and 6.21% lower RSAT (HQF) compared to the counterpart ANN implementations with the same sizes and architectures.This work was supported by the National Natural Science Foundation of China under Grant 62088102, Grant 62376185, and Grant U21A20485. Recommended for acceptance by C. Fermuller.Peer reviewedInstitute of Electrical and Electronics EngineersNational Natural Science Foundation of ChinaYang, Shuangming [0000-0002-8044-0860]Linares-Barranco, Bernabé [0000-0002-1813-4889]Chen, Badong [0000-0003-1710-3818]Consejo Superior de Investigaciones Científicas [https://ror.org/02gfc7t72]202520252025info:eu-repo/semantics/articlehttp://purl.org/coar/resource_type/c_6501Publisher's versioninfo:eu-repo/semantics/publishedVersionapplication/pdfhttp://hdl.handle.net/10261/402210https://api.elsevier.com/content/abstract/scopus_id/86000425314reponame:DIGITAL.CSIC. Repositorio Institucional del CSICinstname:Consejo Superior de Investigaciones Científicas (CSIC)Ingléshttps://doi.org/10.1109/TPAMI.2024.3510627Síinfo:eu-repo/semantics/openAccessoai:digital.csic.es:10261/4022102026-05-22T06:33:51Z
dc.title.none.fl_str_mv Self-Supervised High-Order Information Bottleneck Learning of Spiking Neural Network for Robust Event-Based Optical Flow Estimation
title Self-Supervised High-Order Information Bottleneck Learning of Spiking Neural Network for Robust Event-Based Optical Flow Estimation
spellingShingle Self-Supervised High-Order Information Bottleneck Learning of Spiking Neural Network for Robust Event-Based Optical Flow Estimation
Yang, Shuangming
Event-based vision
Neuromorphic computing
Optical flow estimation
Self-supervised learning
Spiking neural network
title_short Self-Supervised High-Order Information Bottleneck Learning of Spiking Neural Network for Robust Event-Based Optical Flow Estimation
title_full Self-Supervised High-Order Information Bottleneck Learning of Spiking Neural Network for Robust Event-Based Optical Flow Estimation
title_fullStr Self-Supervised High-Order Information Bottleneck Learning of Spiking Neural Network for Robust Event-Based Optical Flow Estimation
title_full_unstemmed Self-Supervised High-Order Information Bottleneck Learning of Spiking Neural Network for Robust Event-Based Optical Flow Estimation
title_sort Self-Supervised High-Order Information Bottleneck Learning of Spiking Neural Network for Robust Event-Based Optical Flow Estimation
dc.creator.none.fl_str_mv Yang, Shuangming
Linares-Barranco, Bernabé
Wu, Yuzhu
Chen, Badong
author Yang, Shuangming
author_facet Yang, Shuangming
Linares-Barranco, Bernabé
Wu, Yuzhu
Chen, Badong
author_role author
author2 Linares-Barranco, Bernabé
Wu, Yuzhu
Chen, Badong
author2_role author
author
author
dc.contributor.none.fl_str_mv National Natural Science Foundation of China
Yang, Shuangming [0000-0002-8044-0860]
Linares-Barranco, Bernabé [0000-0002-1813-4889]
Chen, Badong [0000-0003-1710-3818]
Consejo Superior de Investigaciones Científicas [https://ror.org/02gfc7t72]
dc.subject.none.fl_str_mv Event-based vision
Neuromorphic computing
Optical flow estimation
Self-supervised learning
Spiking neural network
topic Event-based vision
Neuromorphic computing
Optical flow estimation
Self-supervised learning
Spiking neural network
description Event cameras form a fundamental foundation for visual perception in scenes characterized by high speed and a wide dynamic range. Although deep learning techniques have achieved remarkable success in estimating event-based optical flow, existing methods have not adequately addressed the significance of temporal information in capturing spatiotemporal features. Due to the dynamics of spiking neurons in SNNs, which preserve important information while forgetting redundant information over time, they are expected to outperform analog neural networks (ANNs) with the same architecture and size in sequential regression tasks. In addition, SNNs on neuromorphic hardware achieve advantages of extremely low power consumption. However, present SNN architectures encounter issues related to limited generalization and robustness during training, particularly in noisy scenes. To tackle these problems, this study introduces an innovative spike-based self-supervised learning algorithm known as SeLHIB, which leverages the information bottleneck theory. By utilizing event-based camera inputs, SeLHIB enables robust estimation of optical flow in the presence of noise. To the best of our knowledge, this is the first proposal of a self-supervised information bottleneck learning strategy based on SNNs. Furthermore, we develop spike-based self-supervised algorithms with nonlinear and high-order information bottleneck learning that employs nonlinear and high-order mutual information to enhance the extraction of relevant information and eliminate redundancy. We demonstrate that SeLHIB significantly enhances the generalization ability and robustness of optical flow estimation in various noise conditions. In terms of energy efficiency, SeLHIB achieves 90.44% and 45.70% cut down of energy consumption compared to its counterpart ANN and counterpart SNN models, while attaining 33.78% lower AEE (MVSEC), 5.96% lower RSAT (ECD) and 6.21% lower RSAT (HQF) compared to the counterpart ANN implementations with the same sizes and architectures.
publishDate 2025
dc.date.none.fl_str_mv 2025
2025
2025
dc.type.none.fl_str_mv info:eu-repo/semantics/article
http://purl.org/coar/resource_type/c_6501
Publisher's version
info:eu-repo/semantics/publishedVersion
format article
status_str publishedVersion
dc.identifier.none.fl_str_mv http://hdl.handle.net/10261/402210
https://api.elsevier.com/content/abstract/scopus_id/86000425314
url http://hdl.handle.net/10261/402210
https://api.elsevier.com/content/abstract/scopus_id/86000425314
dc.language.none.fl_str_mv Inglés
language_invalid_str_mv Inglés
dc.relation.none.fl_str_mv https://doi.org/10.1109/TPAMI.2024.3510627
Sí
dc.rights.none.fl_str_mv info:eu-repo/semantics/openAccess
eu_rights_str_mv openAccess
dc.format.none.fl_str_mv application/pdf
dc.publisher.none.fl_str_mv Institute of Electrical and Electronics Engineers
publisher.none.fl_str_mv Institute of Electrical and Electronics Engineers
dc.source.none.fl_str_mv reponame:DIGITAL.CSIC. Repositorio Institucional del CSIC
instname:Consejo Superior de Investigaciones Científicas (CSIC)
instname_str Consejo Superior de Investigaciones Científicas (CSIC)
reponame_str DIGITAL.CSIC. Repositorio Institucional del CSIC
collection DIGITAL.CSIC. Repositorio Institucional del CSIC
repository.name.fl_str_mv
repository.mail.fl_str_mv
_version_ 1869403463404748800
score 15.81155

Similares en LA Referencia