Visible & thermal imaging and deep learning based approach for automated robust detection of potholes to prioritize highway maintenance
Date
2023
Authors
Chen, Wei-Hsiang, author
Jia, Gaofeng, advisor
Guo, Yanlin, committee member
Chen, Haonan, committee member
Journal Title
Journal ISSN
Volume Title
Abstract
Potholes are a primary pavement distress that can compromise safety and cause expensive damage claims. Potholes are results of deterioration of pavements due to aging, weather and traffic overloads and are common problems across the U.S. Potholes are even more common in the Mountain Plains region due to the snow and freeze/thaw effect. Identifying and repairing potholes is one critical aspect of highway maintenance. Accurate, robust and fast detection of potholes is critical to enabling timely and cost-effective pavement maintenance. Recently, there has been growing interest and research in using machine learning techniques for pothole detection using different views of visible images. However, quality of potholes detection using only visible images may be significantly compromised due to poor lighting, weather conditions, low contract to surrounding pavement. On the other hand, thermal images are more robust to lighting and weather conditions. Although thermal images may lack the texture details of visible images, they can offer additional unique features compared to visible images, e.g., temperature difference between pothole and surrounding pavement, which can be potentially used for pothole detection. However, so far, the great potential and effectiveness of integrating both visible and thermal images as well as using fused images to enable accurate and robust pothole detection have not been investigated. This research aims to develop an automated deep learning based pothole detection and mapping tool for highway maintenance using the fusion of visible and thermal images. First, a unique and valuable database of geotagged and labeled trios of visible, thermal and fused images is established for training pothole detection algorithms. This is done through collecting pothole images using a low-cost FLIR ONE thermal camera connected to a smart phone. These data are used to train the machine learning algorithms for pothole detection. To establish an accurate pothole detection algorithm, we proposed and compared the performances of three machine learning algorithms, i.e., Anisotropic Diffusion Fusion (ADF) + Mask R-CNN, RTFNet, and RTFNet with Enhancement Parameters (EPs). These algorithms differ in how the visible and thermal images are fused and used for pothole detection. We achieved the best F1-score of 93.7% in the daytime scenario by the RTFNet method and 90.9% in the nighttime scenario by the RTFNet with EPs method. To best leverage the information from the thermal images, in the end we developed a Bright-Dark detector to determine the lighting conditions of candidate testing images, and then feed the images to the respective algorithms for pothole detection. For images with potholes detected, we also developed a mapping tool to map the location of the pothole using GPS information of the images. In the end, the trained overall algorithm is packaged as a tool with graphical user interface (GUI) to facilitate its adoption by highway maintenance team. As more images are collected, the overall algorithm can be continuously improved to further increase the pothole detection accuracy.
Description
Rights Access
Subject
pothole
machine learning