Time and Ordering in a Streaming Distributed System
In the world of stream processing, distributed systems are composed from basic building blocks: coordination, transport, storage, and checkpointed compute services. Working together harmoniously, systems remain consistent, resilient and reliable. However, there is something far more fundamental at play: time and ordering; something experienced so tangibly, yet so difficult to solve for.
Over the past years, we at Netflix have learned to build an efficient, scalable and fault tolerant stream processing platform atop a microservices architecture. In this talk, we’ll try to examine the concepts of time, ordering, and causality in distributed systems. Wielding the knowledge of fundamental physics and computer science, we’ll look into how Einstein's Special Theory of Relativity, Lamport’s famous time clock paper, details of time synchronization, etc and their corresponding implications on streaming system design. Followed by evaluations of assumptions from some pre-streaming systems, then we’ll share the learnings from our hardest quests towards correct and consistent stream processing in the absence of ordering guarantees.
Who is this for: Anyone with some background or interest in stream processing, distributed systems, or anyone simply curious enough to keep digging for answers.
