Whether you are experienced or thinking about getting your hands on Apache Spark, this Apache Spark tutorial will guide you through:
- downloading and running Spark
- launching Spark’s consoles
- Spark’s basic architecture
- Spark’s language APIs
- DataFrames and SQL
- Spark’s Toolset
What is Apache Spark?
Apache Spark is a unified computing engine and a set of libraries for parallel data processing on computer clusters. Spark is the most actively developed open-source engine for this task; making it the de facto tool for any developer or data scientist interested in big data. Spark supports multiple widely-used programming languages (Python, Java, Scala and R), includes libraries for diverse tasks ranging from SQL to streaming and machine learning, and runs anywhere from a laptop to a cluster of thousands of servers. This makes it an easy system to start with and scale up to big data processing or an incredibly large scale.
Here’s a simple illustration of all that Spark has to offer an end user.
Let’s break down our description of Apache Spark – a unified computing engine and set of libraries for big data – into its key components.
1. Unified: Spark’s key driving goal is to offer a unified platform for writing big data applications. Spark is designed to support a wide range of data analytics tasks, ranging from simple data loading and SQL queries to machine learning and streaming computation, over the same computing engine and with a consistent set of APIs.
2. Computing Engine: At the same time that Spark strives for unification, Spark carefully limits its scope to a computing engine. By this, we mean that Spark only handles loading data from storage systems and performing computation on it, not permanent storage as the end itself.
3. Libraries: Spark’s final component is its libraries, which build on its design as a unified engine to provide a unified API for common data analysis tasks. Spark supports both standard libraries that ship with the engine, and a wide array of external libraries published as third-party packages by the open-source communities.
Spark’s Language APIs
Spark’s language APIs allow you to run Spark code from other languages. For the most part, Spark presents some core “concepts” in every language and these concepts are translated into Spark code that runs on the cluster of machines.
Scala
Spark is primarily written in Scala, making it Spark’s “default” language. This book will include Scala code examples wherever relevant.
Java
Even though Spark is written in Scala, Spark’s authors have been careful to ensure that you can write Spark code in Java. This book will focus primarily on Scala but will provide Java examples where relevant.
Python
Python supports nearly all constructs that Scala supports. This book will include Python code examples whenever we include Scala code examples and a Python API exists.
SQL
Spark supports ANSI SQL 2003 standard. This makes it easy for analysts and non-programmers to leverage the big data powers of Spark. This book will include SQL code examples wherever relevant
R
Spark has two commonly used R libraries, one as a part of Spark core (SparkR) and another as an R community driven package (sparklyr).
Download the full free Apache Spark tutorial here.
Editor’s note: Article includes introductory information about Apache Spark from the Databricks free ebook: “A Gentle Introduction to Apache Spark”
