Geeky ramblings

by Ovidiu Beldie

Running a Spark/Mesos cluster on your laptop using Docker (part II) — 13 September, 2015

Running a Spark/Mesos cluster on your laptop using Docker (part II)

After we went through a crash course on Docker in the first part of this series, it’s time to start installing our first services. In this second part we’ll cover Mesos and Hadoop. But before that, we’ll warm up by creating our first Docker image, which will serve as our base for all the others.

2. The Ubuntu base image

Here is the Dockerfile for our base image:

FROM ubuntu:14.04
# Creates /etc/ and runs this script as entry point
RUN echo "#!/bin/bash" > /etc/ \
&& chmod 777 /etc/
# This entry point will be inherited by all derived images
ENTRYPOINT ["/etc/"]

All we’re doing here is specifying that we want to start with a base Ubuntu 14.04 image (which will be downloaded from the repository if necessary) and then we create an empty shell script called /etc/ that is defined as an entrypoint for this image.

In order to build the image,  we have to run the docker build command:

docker build -t acme/ubuntu_base_14_04 bases/ubuntu_14_04

As described in the first part of the tutorial, the -t argument defines the tag for the newly created image and the last argument is the path to the Dockerfile to be used. In our case, the tag has a namespace – acme – representing a fictitious company, Acme Inc. This tag is what we’ll use in the FROM directive for the images that will be derived from the one we just built.

3. Mesos

3.1 Overview

Mesos is a cluster manager, i.e. a service that coordinates the distribution of a cluster’s computing resources among  tasks spawned by one or more applications. It is one of several managers that Spark can currently use, the others being Spark’s own manager – called standalone – and YARN. Without going into the details here, the reasons why I chose Mesos over the others are that it seems to be the more flexible and future-proof choice.

3.2 The Docker images

3.2.1 The Mesos base image

This section will walk you through the creation of the Mesos base image and give you a bit of background why the Dockerfile looks a bit intimidating.

Currently, if you want to install Mesos from the project’s site, the only possibilty is to download the source code and compile it. I did that initially, but then I ran into some issues when running the master. Since I didn’t have time to figure out what the problem was, I discovered in the Spark docummentation that Mesos can also be installed from binaries provided by Mesosphere.

Now that you know the story behind it, here is the Dockerfile:

FROM acme/base_ubuntu_14_04
### Install mesos from mesosphere
# Set up key
RUN apt-key adv --keyserver --recv E56151BF
# Add the repository
RUN DISTRO=$(lsb_release -is | tr '[:upper:]' '[:lower:]') \
&& CODENAME=$(lsb_release -cs) \
&& echo "deb${DISTRO} ${CODENAME} main" | \
tee /etc/apt/sources.list.d/mesosphere.list
RUN apt-get -y update
#Install mesos
RUN apt-get -y install mesos
#Set mesos env
ENV MESOS_WORK_DIR=/var/lib/mesos
ENV MESOS_LOG_DIR=/var/log/mesos

All we’re really doing is adding Mesosphere’s repository and then installing Mesos using apt-get.

3.2.2 The Mesos master image

Here is the Dockerfile for the Mesos master image:

FROM acme/mesos_base
COPY /opt/
#Add this layer's bootstrap to /etc/
RUN cat /opt/ >> /etc/
RUN rm /opt/

The file is a one liner which simply launches the Mesos master:
/usr/bin/mesos master

And finally, here is the script I use to run the container:

docker run -d -e "MESOS_IP=${_MESOS_MASTER_IP}" -e "MESOS_HOSTNAME=${_MESOS_HOSTNAME}" --name "mesos_master" --net="host" acme/mesos_master

Through trail and error, I discovered that only 2 among many environment variables where mandatory to run the Mesos master: MESOS_IP and MESOS_HOSTNAME. This post pointed me in the right direction:

The syntax of the docker run command was presented in the first part of the tutorial. Also in the first part, I promised that I will provide more details about how we can configure a container to reuse the Docker host’s IP address instead of a randomly allocated one. That goal is acheived by supplying the --net="host" argument to the docker run command.

As aluded in the fist post, in my intial iteration I used $_DOCKER_IP instead of $_MESOS_MASTER_IP (which are both set to the IP address of the Docker host) and boot2docker (a string, not a var) instead of $_MESOS_HOSTNAME. Both approaches work but the second will allow a smoother transition for a multi-host cluster.

3.2.3 The Mesos slave image

The slave’s Dockerfile is identical to the master’s so I won’t include it again.

The file is as follows:

/usr/bin/mesos slave $1

This script lauches the mesos slave and expects the first argument passed to the script to define the master node.

Let’s turn to the run script to see how this argument is passed in:

docker run -d --name "mesos_slave1" acme/mesos_slave --master="${_MESOS_MASTER_IP}":5050

If you recall from the first part, any arguments that follow the name of the image are supplied as arguments to the entrypoint, which in our case is the bootstrap script we just saw.

3.3 Testing

Now that you have a master and slave containers running, you can open the master’s web UI running on your docker host on port 5050 and check that it’s running and that in the left column you have one active slave.

4. Hadoop

4.1 Relation to Spark

I have to admit that the relation between Spark and Hadoop wasn’t initially clear to me. On one side, I was learning that Spark didn’t require Hadoop to run, but on the other, when downloading Spark, I had to choose a Hadoop version. As it turns out, the reason for this dependency is Spark’s support of a lot of input sources, one of the prominent ones being HDFS due to its ubiquity in big data environments.

Another use for HDFS can be to host the archive that contains the Spark executor, which is the framework that runs on the Spark nodes and executes the tasks. This archive will be downloaded by each Mesos slave in order to assume the role of a Spark node. We’ll see the details of how this is configured when we’ll get to the configuration of Spark.

4.2 The Docker images

4.2.1 The Hadoop base image

Let’s start with the Dockerfile:

FROM acme/base_ubuntu_14_04
RUN apt-get update -y
RUN apt-get install -y wget ssh rsync openjdk-7-jre
#current version: hadoop-2.7.1
COPY hadoop.tar.gz /opt/hadoop.tar.gz
RUN cd /opt \
&& tar -xzf hadoop.tar.gz \
&& rm hadoop.tar.gz \
&& mv hadoop-* hadoop
ENV JAVA_HOME /usr/lib/jvm/java-1.7.0-openjdk-amd64
ENV HADOOP_HOME /opt/hadoop
RUN useradd hadoop
RUN echo 'hadoop:password' | chpasswd
RUN mkdir /home/hadoop
RUN chown -R hadoop:hadoop /opt/hadoop
RUN chown -R hadoop:hadoop /home/hadoop
COPY core-site.xml /opt/hadoop/etc/hadoop/core-site.xml
COPY hdfs-site.xml /opt/hadoop/etc/hadoop/hdfs-site.xml
COPY /opt/
#Add this layer's bootstrap to /etc/
RUN cat /opt/ >> /etc/
RUN rm /opt/

We need to install ssh, rsync, the JVM (OpenJDK 7) and finally Hadoop 2.7.1. We also need to set the JAVA_HOME and HADOOP_HOME variables and to create an unprivileged user and make it the owner of the Hadoop installation directory because we can’t run Hadoop as the root user (which is the default for Docker containers).

As can be seen in the Dockerfile, we need to supply a couple of configuration files in order to run Hadoop. The first one is hdfs-site.xml (I skipped the initial boilerplate):

  <!-- Don't do reverse DNS checks (for the test cluster) -->

What’s important to note in this file is that:

  • we’re storing the Hadoop data files in /home/hadoop so we had to create this folder in the Dockerfile
  • I had to set ip-hostname-check to false, otherwise Hadoop would attempt to resolve the datanodes’ addresses to hostnames (and fail since we didn’t bother with DNS)

The second configuration file is core-site.xml:


In this file, the string _HADOOP_NAMENODE_IP is a placeholder, which will be replaced with the IP address of the namenode in the bootstrap script:

sed -i "s/_HADOOP_NAMENODE_IP/$_HADOOP_NAMENODE_IP/g" /opt/hadoop/etc/hadoop/core-site.xml

The HADOOP_NAMENODE_IP env var must be passed in when calling docker run on the namenode or datanode images and in this scenario will be the IP of the Docker host.

4.2.2 The Hadoop namenode image

The Dockerfile for the namenode should look familiar by now:

FROM acme/hadoop_base
COPY /opt/
#Add this layer's bootstrap to /etc/
RUN cat /opt/ >> /etc/
RUN rm /opt/

In the we provide the commands to format HDFS and start the namenode:

$HADOOP_HOME/bin/hdfs namenode -format spark_test_cluster
$HADOOP_HOME/sbin/ start namenode
while true; do sleep 1000; done

The reason for the last line in the script was given in the first part of the tutorial.

Lastly, here are the contents of the docker run file:

docker run -d -u hadoop -e "USER=hadoop" --name "hadoop_namenode" \
--net=host -e "_HADOOP_NAMENODE_IP=${_HADOOP_NAMENODE_IP}" acme/hadoop_namenode

Specific to this command are the options to set the user to “hadoop” and also the USER environment variable (which doesn’t happen automatically as I assumed). We also need to set the _HADOOP_NAMENODE_IP variable.

4.2.3 The Hadoop datanode image

The datanode’s Dockerfile is identical to the namenode’s one.

In we start the datanode and we copy the Spark archive to HDFS. This is the Spark tarball that you can download from the Spark project’s site, I just removed the version from the file name since I’m experimenting with different versions.

$HADOOP_HOME/sbin/ start datanode
/opt/hadoop/bin/hadoop fs -put /volume/spark.tgz /spark.tgz
while true; do sleep 1000; done

Finally, here is the docker run command:

docker run -d -u hadoop -e USER=hadoop --name "hadoop_datanode1" \
-v "$_DOCKER_SHARED_DIR/spark:/volume" --net="host" \
-e "_HADOOP_NAMENODE_IP=${_HADOOP_NAMENODE_IP}" acme/hadoop_datanode

What’s notable with this command is that we’re mounting a folder in the image (as /volume) and that we’re using again the host’s network stack. In my initial version of the run script I went with the default network configuration and that worked fine on a single machine pseudo-cluster but it didn’t work when I switched to a physical cluster.

4.3 Testing

If you’ve launched the Hadoop namenode and datanode containers, you can check the Hadoop web UI on the Docker host machine on port 50070 to check that the nodes are up and that the Spark tarball was copied to HDFS.

With this second part of the tutorial completed, we got everything in place to install and run Spark on our virtualized cluster. That will be the subject of the 3rd part of this series.

Running a Spark/Mesos cluster on your laptop using Docker (part I) — 6 September, 2015

Running a Spark/Mesos cluster on your laptop using Docker (part I)

This is the first in a series of posts in which I describe the steps I had to take in order to run a prototype Spark cluster on my development machine (which happens to be a MacBook). I will cover first how to use Docker to create a virtualized cluster infrastructure, then how to install Mesos, Spark, Hadoop and Cassandra and how to test that the ensemble works as expected.

1. Docker

1.1 Overview

Since we’ll be using Docker to create our virtualized infrastructure, this section will give a brief overview of this technology for those who aren’t yet familiar with it.

The core concept of Docker is the container, which can be viewed as a lightweight virtual machine. This means that we have the experience of using virtualized Linux machines, where we can install and run software, but the container doesn’t virtualize the entire operating system. This is possible because the containers share a lot of resources with the host OS and only certain OS capabilities like the file system or the network interfaces are indeed virtualized. The major benefit of this reuse is that, on a single host, we can run significantly more Docker containers in parallel than VMs, which in turn means that those containers can be made more granular and composable.

VM vs Docker container
Figure 1: Implementation of virtual machines vs Docker containers

In Docker terminology, a container is in fact a running instance of a Docker image, which corresponds to what we usually call a virtual machine. An image is usually built from a base image that we either created previously or downloaded from a repository and then we customized by adding software and altering configuration options. The best practice when creating Docker images is to use a Dockerfile to describe the steps that need to be performed to create the image. To those familiar with Vagrant, a Dockerfile is a very similat concept to a Vagrantfile.

1.2 Specificities

Docker has a few specificities and limitations that we must be aware of:

  • there is a single entry point (i.e. initial running process) in a container, although this can be a script that launches multiple other processes. 
  • containers are allocated a random IP address on every run. The quick workaround if we require static IP addresses is to configure the container to use the host’s network stack which ensures that the container has the same IP address as the host.
  • Docker doesn’t run natively on Mac OS or Windows since it’s based on Linux containers. In order to get around this limitation we can use boot2docker, which can be used to run Docker containers within a lightweight VirtualBox Linux VM. After launching boot2docker you can use the docker commands seamlessly from your host’s command line terminal.
  • with release 1.8, boot2docker was deprecated in favor of docker-machine. I recently tried docker-machine and, although I didn’t have any problem initially, when I attempted to test that the Spark cluster still worked the test failed. I will explain the reason why this happened in the appropriate section (and I think it’s just a configuration issue), but I do want to make you aware that it happened and I reverted to using boot2docker.
  • one issue I ran into was that my boot2docker VM ran out of memory when running too many containers in parallel. I followed the steps described in this post to resize my VM: 
  • update 1: if you install Docker on ubuntu, follow the instructions on the docker site instead of using apt-get (e.g. apt-get install I ran into issues when building images by installing the package.

1.3 Commands

Docker comes with a command line tool called (you guessed it) docker, which is used to create images, start and stop containers etc. Here is a shortlist of the most important Docker commands and a some of the arguments they support:

1. docker build [-t <tag>] <path>

This command builds a Docker image based on the Dockerfile located at the specified path. The convention for the tag is to specify a namespace and an image name separated by a slash (e.g. my_company/spark).The tags you specify when building an image are used in Dockerfiles of derived images (in the FROM directive).

2. docker ps [-a]

The ps command shows the containers that are currently running. The -a argument shows all containers, whether running or not.

3. docker run [-it] [-d] [-e <ENV_VAR>=<value>] [-v <path_on_host:path_in_container>] [--rm] [--name <container_name>] 
--entrypoint <image_name> [<command>] [<args>]

This command starts a container. It takes many parameters so I listed the ones I use more often:

  • -i and -t are usually used together when we want to interact with the container (e.g. through a bash shell). I normally use these arguments when I’m “debugging” an image (along with --rm)
  • -d is used to run the container in the background which basically means that you won’t see any output from the container. I use this option when I’ve figured out the container’s configuration and I don’t need a console access to it
  • --rm is used to automatically remove the container after it exits and is mutually exclusive with -d
  • -e is used to set up environment variables in the container
  • -v is used to mount a local directory (if you’re using boot2docker note that this is a directory on your machine and not the boot2docer VM) as a directory in the container
  • --name is used to set a name for the container which can be used subsequently in other docker commands (else a name is randomly generated)
  • after the argument list we need to specify the name of the Docker image that we want to run
  • optionally we can specify a command to run (such as bash). Note that if an entry point was specified for the container, all parameters following the name of the image are passed in as arguments for the entry point
  • finally we can pass in some arguments for the container’s command or entry point.
4. docker stop

This command stops a running container.

5. docker exec [-it] <image_name> <command>

This command allows us to run commands in a running container. I usually use it to run a bash shell.

6. docker start <container_name>

This command is used to start a stopped container (whereas ‘docker run’ creates a container). This is useful when we want to preserve the state of a container from one run to the next. 

There’s obviously a lot more to Docker than I could squeeze in this tutorial. Some of the most useful links when working with Docker are the following:

1.4 Conventions

Here are some conventions that I adopted while working on this prototype: 

a) I’m using an ubuntu 14.04 image as a base for all my images (downloaded from the Docker repository)

b) in terms of bootstrapping the containers, I’m usingusing a script called /etc/ as an entry point and each new derived image appends to this bootstrap file (I discovered this method while examining SequenceIQ‘s images on the Docker repository)

c) for the services that have more than one instance type (e.g. Mesos, which has master and slave instances), I usually create a base image which contains the artifacts and then one image for each instance type which is derived from base. The base image can be compared to a base class in OOP terminology from which we can derive concrete class, which are meant to be instantiated. I also created an overall base image for all my images which contains the (empty) bootstrap script and defines it at the entry point

d) for each image I also usually create a script to capture the docker run command’s arguments. So, to summarize, for each image I have a dedicated directory containing the Dockerfile, the bootstrap script, any files that need to be inserted in the image (as we’ll see later) and a file. Except for the Dockerfile, all the other files are optional. Contrary to the other files, the file may be placed elsewhere.

e) some of the services required for this prototype will need the IP address or the hostname of another service as arguments. Since I’m trying to keep the environment as simple as possible and I don’t want to run a DNS server or to update the hosts file on several containers after each restart, I decided to have all the services that need to be exposed to use the host’s IP address and hostname (which is the boot2docker VM if you use that). I will explain how this is done when we encounter the first such case. 

f) tied to the previous item, I defined a few environment variables on my physical machine to simplify the configuration of the containers such as _DOCKER_IP which is the IP address part of DOCKER_HOST and _DOCKER_SHARED_DIRS which is a path to the directories I mount in Docker containers. (DOCKER_HOST is an env variable that you must have set in order to use the docker client). I’m using the underscore prefix because I once chose the same name for a variable that Mesos uses and I lost hours before I figured out what was wrong.

update 2: After I started to deploy the containers on more than one machine, using the _DOCKER_IP variable obvioulsy didn’t make sense anymore as there were more than one Docker hosts. Instead I started defining a variable for each service such as _MESOS_MASTER_IP or _HADOOP_NAMENODE_IP. For the purposes of running the cluster on your development machine both approaches work. I decided however to use the the second one in the rest of this tutorial as it’s more descriptive and prepares the way for deployment on a physical cluster.

g) I installed all software in the /opt folder of my images for consistency (when not using apt).

h) Docker containers stop when the entry point process exits. When running certain containers, they exit immediately, although the process launched from my bootstrap script still runs and I presume this happens because the process is run in the background, which in turn causes the script itself to exit. A trick that I found in SequenceIQ’s scripts that prevents the container’s premature exit is to add the following snippet at the end of the bootstrap script:

while true; do sleep 1000; done

i) there are 2 ways of installing software in Docker images (in addition to apt-get): either download the package, as a step of building the image, or download the package on your machine and use the Dockerfile COPY command to place it in the image. I tend to prefer the 2nd option although it requires to place the files in the Dockerfile’s directory and they can’t even be soft links. The benefit is that if you need to rebuild that image often, you only have to download the package once.

 With these technicalities about Docker out of the way, we can start building our cluster. In the second part of this tutorial I will cover the creation of Mesos and Hadoop images, then we’ll run the containers and make sure that everything is running as expected.