(Stream|Pipe|Message Queue)

1 - About

We should have some ways of coupling programs like garden hose–screw in another segment when it becomes when it becomes necessary to massage data in another way. This is the way of IO also. Malcolm Douglas McIlroy - Inventor of pipes and other wonders.

An Stream is a combination of:

  • an input source
  • an output destination.
  • and a sequence of data (that may be a byte of an object). In the general sense, it's called a message.
  • that then may be chained to form a pipeline

From an abstract point of view, a stream can be seen as an infinite iteration.

Stream processing lets model systems that have state without ever using assignment or mutable data.

The data structures involved in streams application are then:

  • and a queue to store messages.

Streams are a flexible system to implement a pipeline of data transformations.

An application that handle this message is called a messaging system.

In a stream architecture, the following rules: Something happened (Event), subscribe to it (Streams).

In Unix/Linux, streams are implemented by the shell with pipes.

A table is a stream with an infinite windows.

The world is concurrent. Things in the world don’t share data. Things communicate with messages. Things fail. Joe Armstrong, father of Erlang

3 - Other definition

A stream is derivative of state over time. The product rule, (uv)' = u'v + uv', is analogous to the rule for joining streams. Martin Kleppmann and Julian Hyde

4 - Example

Streams of data

  • user activity on a website
  • sensor readings from devices (IOT)
  • order delivery

5 - Stream vs Batch


Feature Stream Batch Description
Process must be prioritized Yes No When the load becomes to high, choice must be made on a batch platform whereas on a streaming platform, you will just adjust cap the number of running process
IO Few A lot In a batch architecture, the CPU is busy moving a lot of data whereas on a stream platform only the delta is taken into account
CPU Load Incremental Not predicable In an batch architecture, it's really difficult to forecast the future load as one big batch is enough to got the CPU high.

Typical Batch Orchestration CPU metrics

  • Bookmark "(Stream|Pipe|Message Queue)" at del.icio.us
  • Bookmark "(Stream|Pipe|Message Queue)" at Digg
  • Bookmark "(Stream|Pipe|Message Queue)" at Ask
  • Bookmark "(Stream|Pipe|Message Queue)" at Google
  • Bookmark "(Stream|Pipe|Message Queue)" at StumbleUpon
  • Bookmark "(Stream|Pipe|Message Queue)" at Technorati
  • Bookmark "(Stream|Pipe|Message Queue)" at Live Bookmarks
  • Bookmark "(Stream|Pipe|Message Queue)" at Yahoo! Myweb
  • Bookmark "(Stream|Pipe|Message Queue)" at Facebook
  • Bookmark "(Stream|Pipe|Message Queue)" at Yahoo! Bookmarks
  • Bookmark "(Stream|Pipe|Message Queue)" at Twitter
  • Bookmark "(Stream|Pipe|Message Queue)" at myAOL
stream/stream.txt · Last modified: 2017/08/14 17:35 by gerardnico