(Stream|Pipe|Message Queue|Event Processing)
Table of Contents
1 - About
We should have some ways of coupling programs like garden hose–screw in another segment when it becomes when it becomes necessary to massage data in another way. This is the way of IO also. Malcolm Douglas McIlroy - Inventor of pipes and other wonders.
An Stream is a combination of:
- an input source
- an output destination.
- and a sequence of data (that may be a byte of an object). In the general sense, it's called a message.
- that then may be chained to form a pipeline
From an abstract point of view, a stream can be seen as an infinite iteration.
The data structures involved in streams application are then:
- a message
- and a queue to store messages.
Streams are a flexible system to implement a pipeline of data transformations.
An application that handle this message is called a messaging system.
In a stream architecture, the following rules: Something happened (Event), subscribe to it (Streams).
In Unix/Linux, streams are implemented by the shell with pipes.
A table is a stream with an infinite windows.
The world is concurrent. Things in the world don’t share data. Things communicate with messages. Things fail. Joe Armstrong, father of Erlang
Event sourcing describes a process as a sequence of event.
2 - Articles Related
3 - Streaming concepts
- characteristics of unbounded streams,
- and state
4 - Other definition
5 - Architecture
A messaging technology needs to have the following characteristics:
- Capable of high performance at large scale
5.1 - Vision
6 - Event Centric
7 - Example
Streams of data
- user activity on a website
- sensor readings from devices (IOT)
- order delivery