Late residual neural networks: an approach to combat the dead ReLU problem
Date
2022
Authors
Ernst, Matthew Frederick, author
Whitley, Darrell, advisor
Anderson, Chuck, committee member
Buchanan, Norm, committee member
Journal Title
Journal ISSN
Volume Title
Abstract
The rectified linear unit (ReLU) activation function has been a staple tool in deep learning to increase the performance of deep neural network architectures. However, the ReLU activation function has trade-offs with its performance, specifically the dead ReLU problem caused by vanishing gradients. In this thesis, we introduce "late residual connections" a type of residual neural network with connections from each hidden layer connected directly to the output layer of a network. These residual connections improve convergence for neural networks by allowing more gradient flow to the hidden layers of a network.
Description
Rights Access
Subject
residual connections
dead ReLU