A heartbeat mechanism and its application in gigascope

Theodore Johnson, S. Muthukrishnan, Vladislav Shkapenyuk, Oliver Spatscheck

Research output: Chapter in Book/Report/Conference proceedingConference contribution

51 Scopus citations

Abstract

Data stream management systems often rely on ordering properties of tuple attributes in order to implement non-blocking operators. However, query operators that work with multiple streams, such as stream merge or join, can often still block if one of the input stream is very slow or bursty. In principle, punctuation and heartbeat mechanisms have been proposed to unblock streaming operators. In practice, it is a challenge to incorporate such mechanisms into a high-performance stream management system that is operational in an industrial application. In this paper, we introduce a system for punctuation-carrying heartbeat generation that we developed for Gigascope, a high-performance streaming database for network monitoring, that is operationally used within AT&T's IP backbone. We show how heartbeats can be regularly generated by low-level nodes in query execution plans and propagated upward unblocking all streaming operators on its way. Additionally, our heartbeat mechanism can be used for other applications in distributed settings such as detecting node failures, performance monitoring, and query optimization. A performance evaluation using live data feeds shows that our system is capable of working at multiple Gigabit line speeds in a live, industrial deployment and can significantly decrease the query memory utilization.

Original languageEnglish (US)
Title of host publicationVLDB 2005 - Proceedings of 31st International Conference on Very Large Data Bases
Pages1079-1088
Number of pages10
StatePublished - 2005
EventVLDB 2005 - 31st International Conference on Very Large Data Bases - Trondheim, Norway
Duration: Aug 30 2005Sep 2 2005

Publication series

NameVLDB 2005 - Proceedings of 31st International Conference on Very Large Data Bases
Volume3

Other

OtherVLDB 2005 - 31st International Conference on Very Large Data Bases
CountryNorway
CityTrondheim
Period8/30/059/2/05

All Science Journal Classification (ASJC) codes

  • Engineering(all)

Fingerprint Dive into the research topics of 'A heartbeat mechanism and its application in gigascope'. Together they form a unique fingerprint.

Cite this