(Stream|Pipe|Message Queue|Event Processing)

> (Data|State) Management and Processing > (Data Processing|Data Integration) > (Stream|Pipe|Message Queue|Event Processing)

1 - About

We should have some ways of coupling programs like garden hose–screw in another segment when it becomes when it becomes necessary to massage data in another way. This is the way of IO also.

An Stream is a combination of:

  • an input source
  • an output destination.
  • and a sequence of data (that may be a byte of an object). In the general sense, it's called a message.
  • that then may be chained to form a pipeline

From an abstract point of view, a stream can be seen as an infinite iteration and is then associated to realtime processing.

Stream processing lets model systems that have state without ever using assignment or mutable data.

The data structures involved in streams application are then:

  • and a queue to store messages.

Streams are a flexible system to implement a pipeline of data transformations.

An application that handle this message is called a messaging system.

In a stream architecture, the following rules: Something happened (Event), subscribe to it (Streams).

In Unix/Linux, streams are implemented by the shell with pipes.

A table is a stream with an infinite windows.

The world is concurrent. Things in the world don’t share data. Things communicate with messages. Things fail.

Event sourcing describes a process as a sequence of event.


3 - Streaming concepts

4 - Other definition

A stream is derivative of state over time. The product rule, (uv)' = u'v + uv', is analogous to the rule for joining streams.

5 - Architecture

A messaging technology needs to have the following characteristics:

  • Replayable
  • Persistent
  • Capable of high performance at large scale

5.1 - Vision

Real-time Mapreduce Event-driven microservices
Storm, Spark Streaming, Flink Kafka Stream API
Central cluster Embedded library in any Java app
Custom packaging, deployment & monitoring Just Kafka and your app
Suitable for analytics-type use cases Makes stream processing accessible to any use case

6 - Event Centric

7 - Example

Streams of data

  • user activity on a website
  • sensor readings from devices (IOT)
  • order delivery

8 - Documentation / Reference