Enabling autoscaling for in-memory storage in cluster computing framework
Date
2019
Authors
Shrestha, Bibek Raj, author
Pallickara, Sangmi Lee, advisor
Pallickara, Shrideep, committee member
Hayne, Stephen C., committee member
Journal Title
Journal ISSN
Volume Title
Abstract
IoT enabled devices and observational instruments continuously generate voluminous data. A large portion of these datasets are delivered with the associated geospatial locations. The increased volumes of geospatial data, alongside the emerging geospatial services, pose computational challenges for large-scale geospatial analytics. We have designed and implemented STRETCH , an in-memory distributed geospatial storage that preserves spatial proximity and enables proactive autoscaling for frequently accessed data. STRETCH stores data with a delayed data dispersion scheme that incrementally adds data nodes to the storage system. We have devised an autoscaling feature that proactively repartitions data to alleviate computational hotspots before they occur. We compared the performance of S TRETCH with Apache Ignite and the results show that STRETCH provides up to 3 times the throughput when the system encounters hotspots. STRETCH is built on Apache Spark and Ignite and interacts with them at runtime.
Description
Rights Access
Subject
geospatial analysis
scalable analytics
in-memory cluster computing
dynamic partitioning