Table of Contents
1 - About
We should have some ways of coupling programs like garden hose–screw in another segment when it becomes when it becomes necessary to massage data in another way. This is the way of IO also. Malcolm Douglas McIlroy - Inventor of pipes and other wonders.
An Stream is a combination of:
- an input source
- an output destination.
- and a sequence of data (that may be a byte of an object). In the general sense, it's called a message.
- that then may be chained to form a pipeline
From an abstract point of view, a stream can be seen as an infinite iteration.
The data structures involved in streams application are then:
- a message
- and a queue to store messages.
Streams are a flexible system to implement a pipeline of data transformations.
An application that handle this message is called a messaging system.
In a stream architecture, the following rules: Something happened (Event), subscribe to it (Streams).
In Unix/Linux, streams are implemented by the shell with pipes.
A table is a stream with an infinite windows.
The world is concurrent. Things in the world don’t share data. Things communicate with messages. Things fail. Joe Armstrong, father of Erlang
2 - Articles Related
3 - Other definition
4 - Example
Streams of data
- user activity on a website
- sensor readings from devices (IOT)
- order delivery
5 - Stream vs Batch
|Process must be prioritized||Yes||No||When the load becomes to high, choice must be made on a batch platform whereas on a streaming platform, you will just adjust cap the number of running process|
|IO||Few||A lot||In a batch architecture, the CPU is busy moving a lot of data whereas on a stream platform only the delta is taken into account|
|CPU Load||Incremental||Not predicable||In an batch architecture, it's really difficult to forecast the future load as one big batch is enough to got the CPU high.|
Typical Batch Orchestration CPU metrics