Set up an Apache Spark cluster and integrate with Jupyter Notebook

Set up an Apache Spark cluster and integrate with Jupyter Notebook

What is Spark?

Spark is a framework to make computations with large amounts of data. Why do you need something like Spark? Think for example about a small dataset that fit easily into memory, let’s say some Gb maximum. You will probably load the entire  dataframe using Pandas, R or your tool of choice and after some quick cleaning and visualization you will be almost done  with no major hassles related with computing performance if you are using a proper computer (or cloud infrastructure).

Now think that you have to process a 1Tb (or bigger) dataset and train a ML algorithm on it. Even with a powerful computer  it is crazy. Spark gives you two features you need to handle these data monsters:

  • Parallel computing: you use not one but many computers to speed your calculations.
  • Fault tolerance: you must be able to recover if one of your computers hangs in the middle of the process.

How Spark works internally is out  of the scope of this tutorial and I will assume you are already familiar with that. Anyway you will need little  knowledge about Spark’s internals to set up and run you own cluster at home.

What is a Spark cluster and what does ‘standalone’ mean?

Spark clusters

A Spark cluster is just some computers running Spark and working together. A cluster consist on:

  • Master: is one of the computers that orchestrate how everything works. It distributes the work and take care of everything.
  • Slaves: these are the computers that get the job done. They process chunks of your massive datasets following the Map  Reduce paradigm. A computer can be master and slave at the same time.


It just mean that Spark is installed in every computer involved in the cluster. The cluster manager in use is provided  by Spark. There are other cluster managers like Apache Mesos and Hadoop YARN.


To follow this tutorial you need:

  • A couple of computers (minimum): this is a cluster.
  • Linux: it should also work for OSX, you have to be able to run shell scripts. I have not seen Spark running on native  windows so far.

For this tutorial I have used a MacBook Air with Ubuntu 17.04 and my desktop system with Windows 10 running  Linux Subsystem for Windows (yeah!) with Ubuntu 16.04 LTS.

If you don’t meet these simple requirements, please don’t panic, follow this steps and you are done:

And that’s all, you have 2 Linux machines to run your cluster.


Step 1: Install Java

Note: you will have to perform this step for all machines involved.

Spark needs Java to run. My recommendation is going with Open JDK8. Go to your Terminal and write the following commands:

$ sudo apt-get update
$ sudo apt-get upgrade
$ sudo apt-get install openjdk-8-jdk

Test your Java installation typing:

$ java -version

You should see the following output:

openjdk version "1.8.0_131" OpenJDK Runtime Environment (build 1.8.0_131-8u131-b11-2ubuntu1.17.04.3-b11) OpenJDK 64-Bit Server VM (build 25.131-b11, mixed mode)

Step 2: Install Spark

Download Spark. I have used 2.2.0 pre-built for  this tutorial. You can go to Spark download page and download it from there  or in case you don’t have access to a graphical desktop and have to use command line just run:

$ curl -O

After that you will have the compressed source of Spark in your home directory. Extract the code and move the resulting folder to your home (recommended but not necessary) in case you downloaded it in Downloads folder.


$ tar zxvf spark-2.2.0-bin-hadoop2.7.tgz

And move (optional):

$ mv spark-2.2.0-bin-hadoop2.7 ~

Finally you will set the needed environment variable SPARK_HOME.

Go to your home directory and open with a text editor the .bashrc file:

$ cd ~ $ sudo nano .bashrc

Add the following lines at the end of the file. It will set up the variable properly when your turn on your computer or start a session. Take into account that folder name will change in the future and will not match the one written here,  but the procedure is similar.

# Spark export SPARK_HOME="/home/<your_username>/spark-2.2.0-bin-hadoop2.7/"

Save the file. If you are using nano just do ctrl+x, write y and press return to get  it done.

Now you can check your Spark installation. Go to Spark folder and execute pyspark:

$ cd spark-2.2.0-bin-hadoop2.7 $ bin/pyspark

If everything is properly installed you should see an output similar to this:

Python 3.6.1 |Anaconda custom (64-bit)| (default, May 11 2017, 13:09:58) 
[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
Using Spark's default log4j profile: org/apache/spark/
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
17/08/17 20:39:03 WARN Utils: Your hostname, david-MacBookAir resolves to a loopback address:; using instead (on interface wlp2s0)
17/08/17 20:39:03 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address
17/08/17 20:39:13 WARN ObjectStore: Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 1.2.0
17/08/17 20:39:13 WARN ObjectStore: Failed to get database default, returning NoSuchObjectException
17/08/17 20:39:14 WARN ObjectStore: Failed to get database global_temp, returning NoSuchObjectException
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /__ / .__/\_,_/_/ /_/\_\   version 2.2.0

Using Python version 3.6.1 (default, May 11 2017 13:09:58)
SparkSession available as 'spark'.

Up to this point, you may see this warning at Spark initialization:

WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable".

To solve this problem, we will have to install Hadoop. This is optional because Spark is going to run anyway, but I guess  there must be some performance improvements of using native Hadoop over some kind of adapters.

Step 3 (Optional): Installing Hadoop

The process of installing Hadoop is pretty much the same as Spark, and I will go over it quickly. This is an optional step  and is not strictly necessary to run your cluster. You can also decide to do it later. The steps for installing Hadoop are:

  • Download Hadoop source here. You are free  to install whatever version you like, but make sure you choose a version > 2.7 because it’s a Spark’s requirement. This   is pretty much the same as we did before with Spark. Feel free to use equivalent command line expressions.
  • Extract in your home directory.
  • Add the following lines at the end of your .bashrc file:
# Hadoop export HADOOP_HOME="/home/<your_username>/hadoop-2.8.0" export LD_LIBRARY_PATH=$HADOOP_HOME/lib/native:$LD_LIBRARY_PATH
  • Test your Hadoop installation by starting pyspark and make sure you don’t see the warning.

Step 4: Launch the master server

Previous to launch the master server, you must check your ip to pass it as the host parameter. Type the following command:

$ ifconfig

You should see an output similar to this:

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet  netmask
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 3735  bytes 7174785 (7.1 MB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 3735  bytes 7174785 (7.1 MB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

wlp2s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet  netmask  broadcast
        inet6 xxxx::xxxx:xxxx:xxxx:xxxx  prefixlen 64  scopeid 0x20<link>
        ether 14:10:9f:f3:a0:5e  txqueuelen 1000  (Ethernet)
        RX packets 67543  bytes 60399180 (60.3 MB)
        RX errors 0  dropped 0  overruns 0  frame 119733
        TX packets 52684  bytes 11712399 (11.7 MB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        device interrupt 17  

Copy the inet value inthe second text block. This is going to be <your_master_ip>. In this example you should copy

Now, go to your Spark installation directory and type:

$ ./sbin/ -h <your_master_ip_>

You should see an output similar to:

starting org.apache.spark.deploy.master.Master, logging to /home/<your_username>/spark-2.2.0-bin-hadoop2.7//logs/spark-<your_username>-org.apache.spark.deploy.master.Master-1-<your_username>-MacBookAir.out

Now you can go to localhost:8080 in the same computer where you started the master server, or to <your_master_ip>:8080 in  case you don’t have access to a web browser but have other devices connected to the same private network (e.g. a phone). In that URL you have access to the master server web user interface.

In this web you can see just behind Spark logo an URL parameter similar to spark://<your_master_ip>:7077. This URL is  very important because is the one you are going to need when connecting slaves to your cluster and I will name it <your_master_url>.

Step 5: Connect your slaves

Now you have a master server running, is time to start a couple of slave servers to get the job done. To start a slave  server you have to type the following command running from your Spark installation folder, using the URL you copied from  master server web interface:

$ ./sbin/ <your_master_url>

You should see an output very similar to the master’s one.

Perform this operation in every computer you want to connect to the cluster. In this tutorial I am going to run 2 slaves.

After that, if you go to the master server web interface again you should see several rows in the ‘Workers’ section where,  one per slave server you started. There is also useful info like worker’s memory, status, cores used and ip.

You can also click the links in the Spark web app and go to worker’s page and tasks page. As you don’t have any running  app connected to the cluster, you can’t access running and completed tasks page.

So far you have a fully working Spark cluster running.

Step 6: Integration with Jupyter

To get the most out of Spark is a good idea integrating with some interactive tool like Jupyter. If you already have  Jupyter installed and running, skip the following lines where I explain how to set up a local Jupyter server.

Installing Jupyter

The easiest way to install Jupyter is probably using conda  (package, dependency and environment management). If you have Anaconda Python distribution,  conda is already installed in your computer. If not, I strongly recommend you giving Anaconda a try, it really worth it.  In case you don’t want to install the full Anaconda Python (it includes a lot of libraries and needs about 350 Mb of disk)  you can opt for Miniconda, a lighter version which only includes Python and conda.

If you are using command line, just download the installation file (shell script) using curl and execute it with './'

Once you have conda installed in your computer, let’s create a conda virtual environment called jupyter to avoid  messing the root one:

$ conda create -n jupyter

Now have a clean virtual environment to install Jupyter on it. To activate this environment type:

$ source activate jupyter

With the environment activated you should see (jupyter) at the beginning of your command prompt. Something like this:

(jupyter) david@david-MacBookAir:~$

It means that your environment is activated and all changes you do hereinafter (installing libraries, etc.) will be  applied to this conda virtual environment.

To install Jupyter, type the following command:

(jupyter) $ conda install notebook

This command will install Jupyter notebook and all its dependencies (IPython, Jupyter, etc.) so you don’t have to worry  about setting all these things up (thanks to conda package manager!).

Now it’s time to launch a Jupyter notebook and test your installation. Type:

(jupyter) $ jupyter notebook

If Jupyter is properly installed you should be able to go localhost:8888/tree URL in a web browser and see Jupyter folder tree.

Installing findspark

findspark is a Python library that automatically allow you to import and use PySpark as any other Python library. There  are other options to make the integration (create a jupyter profile for Spark) but up to date findspark is imho the  faster and simpler one.

To install findspark run the following command:

(jupyter) $ pip install findspark

Now you have findpark installed in your jupyter virtual environment. Let’s create your first application.

Create your first Spark application

To create your first Spark app and start to make cool things with data, run the following script in a Jupyter cell:

import findspark 

import pyspark 
sc = pyspark.SparkContext(master='<your_master_url>',

This script just initialize findspark so you can import PySpark as a library, and create an instance of Spark Context with  your master URL and app name (up to you) as parameters.

To test everything works well, you can display sc in your Jupyter notebook and should see an output like this:

You can click the link and go to your app web ui, which is very interesting while you a running long tasks.

You should also connect to your master server web interface (localhost:8080 or <your_master_ip>:8080) and see a new row  in Running Applications section, like this:


To turn off your master and slaves servers, use the following commands from your Spark installation directory:

$ ./sbin/ $ ./sbin/

To deactivate your jupyter conda virtual environment, just run the following command:

(jupyter) $ source deactivate jupyter

And that’s all!

There are many other things you can tweak and configure about Spark! If people is interested, I will create more content  about this amazing technology in the future as I am still learning new concepts everyday.

Hope you found this tutorial useful. If you see any error, bug, errand or whatever, just tell me or leave a comment and I  will try to fix it asap.