Improving performance and sustainability in Big Data Clouds

Abstract

Big Data analytics and Cloud Computing have become two increasingly important IT paradigms to enhance business agility and productivity. As Cloud Computing offers a cost-effective way to support Big Data analytics, it exhibits many key characteristics. First, the user-perceived resource quantity and quality can vary significantly over time due to various interferences in the Cloud. Second, different types of Big Data applications, such as MapReduce and Spark, require distinct resource management schemes in the Cloud. Finally, sustainability is another concern for Cloud operators when consumers and organizations aggressively adopt cloud-based computing models. This thesis presents adaptive resource provisioning approaches on improving the performance and sustainability in Big Data Cloud systems. From the perspective of performance, we proposed and developed adaptive approaches for optimizing performance of Big Data applications in MapReduce and Spark. These approaches provide quality-of- service guarantees to the clients and improve the resource efficiency. They provide responsive and stable resource control in the presence of workload heterogeneity and Cloud dynamics. Specifically, we designed and developed independent but complementary approaches at middleware levels to reduce performance variations of in-cloud applications without compromising other performance objectives. From the perspective of sustainability, we proposed and developed autonomic resource provisioning and scheduling techniques for sustainable computing. These techniques adaptively manage application demands on power and improve energy efficiency without compromising reliability and performance. Specifically, we developed adaptive resource provisioning and workload scheduling techniques for sustainable cloud computing via applying renewable energy sources and improving energy efficiency through IT consolidation and workload management. We implemented and evaluated the proposed techniques in private clouds and clusters with PUMA and BigDataBench benchmark suites. Results demonstrated the effectiveness and efficiency of the proposed techniques. The primary contributions of this thesis lie in novel and effective systematic solutions that improve the performance and sustainability for Big Data Cloud systems.