Spark - Engine

> Database > Spark > Spark - Engine

1 - About

In Spark, the Spark engine is a SQL’s optimized execution engine and understand as input:

and is therefore sometimes known as the SQL Engine.

In Spark, functions are pipelined around their data structure.

A function will take a Spark data structure (data type) as input and output the same data structure (data type).

It results of a builder pattern where the last method performs an action whereas the previous one set parameters.

Advertising

3 - Interface

You are not manipulating the engine directly but via interfaces:

4 - Documentation / Reference