Same data, same features: modern ImageNet-trained convolutional neural networks learn the same thing
Date
2020
Authors
McNeely-White, David G., author
Beveridge, J. Ross, advisor
Anderson, Charles W., committee member
Seger, Carol A., committee member
Journal Title
Journal ISSN
Volume Title
Abstract
Deep convolutional neural networks (CNNs) are the dominant technology in computer vision today. Much of the recent computer vision literature can be thought of as a competition to find the best architecture for vision within the deep convolutional framework. Despite all the effort invested in developing sophisticated convolutional architectures, however, it's not clear how different from each other the best CNNs really are. This thesis measures the similarity between ten well-known CNNs, in terms of the properties they extract from images. I find that the properties extracted by each of the ten networks are very similar to each other, in the sense that any of their features can be well approximated by an affine transformation of the features of any of the other nine. In particular, there is evidence that each network extracts mostly the same information as each other network, though some do it more robustly. The similarity between each of these CNNs is surprising. Convolutional neural networks learn complex non-linear features of images, and the architectural differences between systems suggest that these non-linear functions should take different forms. Nonetheless, these ten CNNs which were trained on the same data set seem to have learned to extract similar properties from images. In essence, each CNN's training algorithm hill-climbs in a very different parameter space, yet converges on a similar solution. This suggests that for CNNs, the selection of the training set and strategy may be more important than the selection of the convolutional architecture.
Description
Rights Access
Subject
convolutional neural networks
feature space
machine learning
feature mapping
computer vision
ImageNet