Deep and wide neural networks covariance estimation
It has been recently shown that a deep neural network with i.i.d. random parameters is equivalent to a Gaussian process in the limit of infinite network width. The Gaussian process associated to the neural network is fully described by a recursive covariance kernel determined by the architecture of...
| Autores: | , , |
|---|---|
| Formato: | artículo |
| Fecha de publicación: | 2020 |
| País: | España |
| Recursos: | Universitat Politècnica de Catalunya (UPC) |
| Repositorio: | UPCommons. Portal del coneixement obert de la UPC |
| Idioma: | inglés |
| OAI Identifier: | oai:upcommons.upc.edu:2117/407308 |
| Acesso em linha: | https://hdl.handle.net/2117/407308 https://dx.doi.org/10.1007/978-3-030-61609-0_16 |
| Access Level: | acceso abierto |
| Palavra-chave: | Neural Networks Deep neural networks Gaussian process Kernels Hermite polynomials Xarxes neuronals (Informàtica) Àrees temàtiques de la UPC::Informàtica |
| Resumo: | It has been recently shown that a deep neural network with i.i.d. random parameters is equivalent to a Gaussian process in the limit of infinite network width. The Gaussian process associated to the neural network is fully described by a recursive covariance kernel determined by the architecture of the network, and which is expressed in terms of expectation. We give a numerically workable analytic expression of the neural network recursive covariance based on Hermite polynomials. We give explicit forms of this recursive covariance for the cases of neural networks with activation function the Heaviside, ReLU and sigmoid. |
|---|