Repository logo
 

Real time stream processing for Internet of things and sensing environments

Simple item page
dc.contributor.authorHewa Raga Munige, Thilina, author
dc.contributor.authorPallickara, Shrideep, advisor
dc.contributor.authorPallickara, Sangmi, committee member
dc.contributor.authorTurk, Daniel, committee member
dc.date.accessioned2016-01-11T15:13:39Z
dc.date.available2016-01-11T15:13:39Z
dc.date.issued2015
dc.description.abstractImprovements in miniaturization and networking capabilities of sensors have contributed to the proliferation of Internet of Things (IoT) and continuous sensing environments. Data streams generated in such settings must keep pace with generation rates and be processed in real time. Challenges in accomplishing this include: high data arrival rates, buffer overflows, context-switches during processing, and object creation overheads. We propose a holistic framework that addresses the CPU, memory, network, and kernel issues involved in stream processing. Our prototype, Neptune, builds on the Granules cloud runtime and leverages its support for scheduling packets and communications based on publish/subscribe, peer to peer, and point-to-point. The framework maximizes bandwidth utilization in the presence of small messages via the use of buffering and dynamic compactions of packets based on their entropy. Our use of thread-pools and batched processing reduces context switches and improves effective CPU utilizations. The framework alleviates memory pressure that can lead to swapping, page faults, and thrashing through efficient reuse of objects. To cope with buffer overflows we rely on flow control and throttling the preceding stages of a processing pipeline. Our correctness criteria included deadlock/livelock avoidance, and ordered and exactly-once processing. Our benchmarks demonstrate the suitability of the Granules/Neptune combination and we contrast our performance with Apache Storm, the dominant stream-processing framework developed by Twitter. At a single node, we are able to achieve a processing rate of ~2 million stream packets per-second. In a distributed cluster setup, we are able to achieve a processing rate of ~100 million stream packets per-second with a near-optimal bandwidth utilization.
dc.format.mediumborn digital
dc.format.mediummasters theses
dc.identifierHewaRagaMunige_colostate_0053N_13272.pdf
dc.identifier.urihttp://hdl.handle.net/10217/170303
dc.languageEnglish
dc.language.isoeng
dc.publisherColorado State University. Libraries
dc.relation.ispartof2000-2019
dc.rightsCopyright and other restrictions may apply. User is responsible for compliance with all applicable laws. For information about copyright law, please see https://libguides.colostate.edu/copyright.
dc.subjecthigh performance
dc.subjectreal time processing
dc.subjectstream processing
dc.subject.lcshInternet of things
dc.titleReal time stream processing for Internet of things and sensing environments
dc.typeText
dcterms.rights.dplaThis Item is protected by copyright and/or related rights (https://rightsstatements.org/vocab/InC/1.0/). You are free to use this Item in any way that is permitted by the copyright and related rights legislation that applies to your use. For other uses you need to obtain permission from the rights-holder(s).
thesis.degree.disciplineComputer Science
thesis.degree.grantorColorado State University
thesis.degree.levelMasters
thesis.degree.nameMaster of Science (M.S.)

Files

Original bundle
Now showing 1 - 1 of 1
Thumbnail Image
Name:
HewaRagaMunige_colostate_0053N_13272.pdf
Size:
1.72 MB
Format:
Adobe Portable Document Format
Download

Collections

2000-2019
Theses and Dissertations