Weather forecasting automation error type, reliability, and transparency affect use and corresponding attitudes

Abstract

In two experiments, 208 and 163 participants completed a series of trials in which they were to decide if a school should remain open or close due to expected snowfall. These experiments differed in type of error that automation made (errors due to the challenge of predicting a noisy environment in Experient 1 and errors due to algorithm miscalculations in Experient 2). Participants were given a weather forecast automation prediction of snowfall whose predictions were either 70% or 90% reliable and were either accompanied by raw data (transparency) or not. Participants self-reported trust, and outcome measures of dependence and accuracy were also recorded. Overall, participants reported high trust of weather forecasts, regardless of the presence of transparency or level of reliability. Increasing reliability increased trust, dependence, and accuracy. We found trends that transparency is most helpful at lower reliability and that participants do not tend to depend on highly reliable automation as much as they should. Further, there are implications regarding the amount of uncertainty with a prediction decision by the user that automation does not account for regarding decision making.