Spark - Engine

About

In Spark, the Spark engine is a SQL’s optimized execution engine and understand as input:

• Spark - Sql
• or Spark - DataSet

and is therefore sometimes known as the SQL Engine.

In Spark, functions are pipelined around their data structure.

A function will take a Spark data structure (data type) as input and output the same data structure (data type).

It results of a builder pattern where the last method performs an action whereas the previous one set parameters.

Articles Related

Interface

You are not manipulating the engine directly but via interfaces:

• Spark - Sql
• Spark - DataSet (and/of Spark DataSet - Data Frame

Documentation / Reference

• https://databricks.com/blog/2015/04/13/deep-dive-into-spark-sqls-catalyst-optimizer.html
• https://docs.azuredatabricks.net/spark/latest/gentle-introduction/gentle-intro.html