How Does Docker File Read I,age if Npt Imstalled

Tutorial: Running a Dockerized Jupyter Server for Data Scientific discipline

Published: Nov 22, 2015

At Dataquest, we provide an piece of cake to apply environment to start learning data scientific discipline. This environs comes preconfigured with the latest version of Python, well known information science libraries, and a runnable code editor. It allows make new information scientists, and experienced ones, to start running lawmaking right abroad. While nosotros provide a seemless experience to learn on our datasets, when you desire to switch to your own data sets you'll take to move to a local development surroundings.

Sadly, setting upwards your own local environment is the most frustrating experience of being a data scientist. Dealing with inconsistent package versions, lengthy installations that fail due to errors, and obscure setup instructions brand information technology difficult to even offset programming. These problems are exaggerated to a higher degree when working on teams with different operating systems. For many, the setup is the biggest detractor to learning how to lawmaking.

Fortunately, there has been a ascent of technologies that help with these development woes. The one we'll be exploring in this postal service is a containerization tool chosen Docker. Since 2013, Docker has made information technology fast and easy to launch multiple data science environments supporting the infrastructure needs of different projects.

In this tutorial, we're going to show you lot how to set up your ain Jupyter Notebook server using Docker. We'll cover the basics of Docker and containerization, how to install Docker, and how to download and run Dockerized applications. By the end, you should be able to run your own local Jupyter server with the latest information scientific discipline libraries.

The Docker whale is hither to aid

An overview of Docker and containerization

Before we dive into Docker, information technology'due south important to know some preliminary software concepts that led to the rise of technologies similar Docker. In the introduction, nosotros briefly described the difficulty of working on teams with multiple operating systems and installing tertiary-party libraries. These types of problems accept been around since the beginning of software development.

One solution has been the use of virtual machines. Virtual machines allow you to emulate culling operating systems from the ane running on your local motorcar. A common instance is running a Windows desktop with a Linux virtual car. A virtual car is essentially a fully isolated operating system with applications that are run independent of your own system. This is extremely helpful in a team of developers as every member tin run the verbal aforementioned organisation regardless of the Bone on their machine.

Nevertheless, virtual machines are not a panacea. They are hard to gear up up, require significant arrangement resources to run, and accept a long time to boot.

An example of using Windows in a virtual machine on a mac

Edifice on this concept of virtualization, an culling approach to full virtual machine isolation is called containerization. Containers are similar to virtual machines every bit they also run applications in an isolated environment. All the same, instead of running a complete operating system with all of its libraries, a containerized environment is a lightweight process that runs on acme of a container engine.

The container engine runs the containers and provides the ability for the host OS to share read merely libraries and its kernel to the container. This makes a running container MBs in size versus virtual machines that tin exist tens of GBs or more than. Docker is a type of container engine. Information technology allows us to download and run images which contain a preconfigured lightweight Bone, prepare of libraries, and application specific packages. When we run an image, the process spawned by the Docker engine is called a container.

As mentioned earlier, containers eliminate configuration problems and ensure compatibility across platforms, freeing us from the restrictions of underlying operating systems or hardware. Similar to virtual machines, systems built on different technologies (due east.yard. Mac OS X vs. Microsoft Windows) tin deploy completely identical containers. There are multiple advantages to using containers over virtual environments:

• Ability to get started quickly. You don't need to wait for packages to install when y'all just want to jump in and commencement doing analysis.
• Consistent across platforms. Python packages are cross-platform, merely some behave differently on Windows vs Linux, and some have dependencies that can't exist installed on Windows. Docker containers always run in a Linux environs, so they're consequent.
• Power to checkpoint and restore. Yous can install packages into a Docker image, then create a new image of that checkpoint. This gives you lot the ability to apace undo changes or rollback configurations.

A good overview of containerization and the difference betwixt virtual machines tin be institute in the official Docker documentation. In the next department, we're going to encompass how to setup and run Docker on your arrangement.

Installing Docker

At that place's a graphical installer for Windows and Mac that makes installing Docker easy. Here are instructions for each Bone:

• Windows
• Mac Bone
• Linux

For the balance of the tutorial we'll exist covering the Linux and macOS (Unix) instructions of running Docker. The examples nosotros'll provide should be run in your terminal awarding (Unix), or the DOS prompt (Windows). While nosotros'll highlight the Unix beat out commands, the Windows commands should be similar. We recommend checking the official Docker documentation if there is whatsoever discrepancy. To check that the Docker customer is correctly installed, here are a few test commands:

• docker version: Returns information on the Docker version running on your local car.
• docker aid: Returns a listing of Docker commands.
• docker run hullo-globe: Runs the hello-world image and verifies that Docker is correctly installed and functioning.

Running a Docker container from an image

With Docker installed, we can now download and run images. Think that an paradigm contains a lightweight Bone and libraries to run your application, while the running image is called a container. Yous can call up of an image equally the executable file, and the running procedure spawned by that file as the container. Permit'due south beginning by running a basic Docker image.

Enter the docker run command (below) in your shell prompt. Make sure to enter the full control: docker run ubuntu:sixteen.04 If your Docker is installed correctly, you should meet something forth the following output:

            Unable to detect image 'ubuntu:16.04' locally 16.04: Pulling from library/ubuntu 297061f60c36: Downloading [============> ] 10.55MB/43.03MB e9ccef17b516: Download complete dbc33716854d: Download complete 8fe36b178d25: Download complete 686596545a94: Download complete          

Permit'due south break down the previous command. Offset, we started by passing the run argument to the docker engine. This tells Docker that the next statement, ubuntu:16.04 is the image we want to run. The image argument we passed in is equanimous of the image name, ubuntu and a corresponding tag, xvi.04. You can recall of the tag as the prototype version. Furthermore, if you were to leave the prototype tag blank, Docker would run the latest image version (i.e. docker run ubuntu < -> docker run ubuntu:latest).

Once we event the command, Docker starts the run procedure by checking if the image is on your local car. If Docker can't find the epitome, information technology volition bank check Docker hub and download the image. Docker hub is an prototype repository, pregnant information technology hosts open source community built images that are available to download.

Finally, afterwards downloading the image, Docker will and so run information technology as a container. However, observe that when the ubuntu container starts upwardly, it immediately exits. The reason is exits is considering we didn't pass in additional arguments providing context to the running container. Let's effort running another image with some optional arguments to the run command. In the following command, we'll provide the -i and -t flag, starting an interactive session and simulated terminal (TTY).

Run the post-obit to get access to a Python prompt running in a Docker container: docker run -i -t python:3.six This is equivalent to: docker run -it python:3.6

Running the Jupyter Notebook

After running the previous control, you should have entered the Python prompt. Within the prompt, you can write Python lawmaking as normal but the code volition be exectuting in the running Docker container. When you exit the prompt, you'll both quit the Python process and leave the interactive container fashion which shuts downward the Docker container. Then far, we have ran both an Ubuntu and Python epitome. These types of images are great to develop with, but they're not that exciting on their own. Instead, we're going to run a Jupyter paradigm that is an application specific image that is congenital on height of the ubuntu image.

The Jupyter images we'll be using come from Jupyter's evolution community. The blueprint of the images, called a Dockerfile, tin be plant in their Github repo. Nosotros won't cover Dockerfiles in detail this tutorial, so just think of them as the source code for the created epitome. An image's Dockerfile is normally hosted on Github while the built image is hosted on Docker Hub. To brainstorm, let's call the Docker run control on one of the Jupyter images.

We're going to run the minimal-notebook that simply has Python and Jupyter installed. Enter the control below: docker run jupyter/minimal-notebook Using this command, we'll exist pulling the latest paradigm of the minimal-notebook from the jupyter Docker hub account. You'll know information technology has ran successfully if yous see the post-obit output:

            [C 06:14:15.384 NotebookApp] Copy/paste this URL into your browser when you lot connect for the commencement fourth dimension, to login with a token: https://localhost:8888/?token=166aead826e247ff182296400d370bd08b1308a5da5f9f87          

Similar to running a non-Dockerized Jupyter notebook, you'll have a link to the Jupyter localhost server and a given token. Even so, if you endeavor to navigate to the provided link, you won't be able to access the server. On your browser, you'll be presented instead with a "Site Cannot exist Reached" folio.

This is because the Jupyter server is running inside it's ain isolated Docker container. This means that all the ports, directories, or any other files are not shared with your local machine unless explicitly directed. To access the Jupyter server in the Docker container, nosotros need to open the ports between the host and container by passing in the -p <host_port>:<container_port> flag and argument. docker run -p 8888:8888 jupyter/minimal-notebook

This is what you lot should see after navigating to the URL in your browser If you see the screen above, yous're successfully developing within the Docker container. To recap, instead of having to download Python, some runtime libraries, and the Jupyter packet, all that was required was to install Docker, download the official Jupyter image, and run the container. Next, we'll expand on this and learn how to share notebooks from your host machine (local car) with the running container.

Sharing notebooks betwixt the host and container

To begin, nosotros'll offset by creating a directory on our host car where we'll keep all of our notebooks. In your home directory, create a new directory called notebooks. Here'southward one command to do that: mkdir ~/notebooks

Now that nosotros have a defended directory for our notebooks, we can share this directory between the host and container. Similar to opening the ports, nosotros'll need to pass in another additional argument to the run control. The flag for this argument is -five <host_directory>:<container_directory> which tells the Docker engine to mountain the given host directory to the container directory. From the Jupyter Docker documentation, it specifies the working directory of the container as /home/jovyan. Thus, we'll mount our ~/notebooks directory to the container's working directory using the mountain run flag. docker run -p 8888:8888 -5 ~/notebooks:/domicile/jovyan jupyter/minimal-notebook With the directory mounted, get to the Jupyter server and create a new notebook. Rename the notebook from "Unititled" to "Example Notebook". On your host machine, check the ~/notebooks directory. In there, you should see an iPython file: Example Notebook.ipynb!

Installing additional packages

In our minimal-notebook Docker image, there are pre-installed Python packages available for use. One of them we have been using explictiy, the Jupyter notebook, which is the notebook server that we're accessing on the browser. Other packages are implicitly installed, like the requests package, which you lot can import inside a notebook. Notice that these pre-installed packages were bundled in the prototype, we did not install them ourselves.

As we have mentioned, non having to install pacakges is one of the major benefits of using containers for development. But, what if an image is missing a data science package you wanted to use, say something like tensorflow for car learning? One way to install a parcel in your container is to utilize the docker exec command.

The exec control has similar arguments with the run command, simply information technology doesn't get-go a container with the arguments, it executes on an already running container. So, insead of creating a container from an image, like docker run, docker exec requires a running container ID or container name which are chosen container identifiers. To locate a running container's identifiers, you need to call the docker ps commandwhich lists all the running containers and some additional info. For example, here's what our docker ps outputs while the minimal-notebook container is running.

            $ docker ps CONTAINER ID IMAGE Control CREATED STATUS PORTS NAMES 874108dfc9d9 jupyter/minimal-notebook "tini -- starting time-noteb…" Less than a second ago Up four seconds 0.0.0.0:8900->8888/tcp thirsty_almeida          

Now that we have the container ID, nosotros can install Python packages in the container. From the docker exec documentation, we laissez passer in runnable command equally an argument proceeding the identifier which is and then executed in the container. Recall that the command to install Python packages is pip install <package proper name>. To install tensorflow, nosotros'll run the following in our shell. NB:, your container ID will exist different from the provided example. docker exec 874108dfc9d9 pip install tensorflow

If successful, you lot should see the pip installation output logs. One thing you lot'll notice is that the installation of tensorflow within this Docker container is relatively quick (given you take fast internet). If yous've ever installed tensorflow earlier, you'll know that it's quite the tedious setup, then you might exist surprised at how painless this procedure is. The reason it is a quick installation process, is because the minimal-notebook prototype has been written with data science optimization in mind. The C libraries and other Linux arrangement level packages have already been pre-installed based on the Jupyter community'south thoughts on installation best practices. This is the greatest benefit of using open source customs developer Docker images every bit they are usually optimized for the type of developement work yous volition be doing.

Extended Docker images

Up to this point, y'all've installed Docker, ran your get-go container, accessed a Dockerized Jupyter container, and installed tensorflow on a running container. At present, suppose you've finished your day working on the Jupyter container using the tensorflow library, and you lot desire to close the container down to repossess processing speed and memory.

To end the container, you can run either the docker cease or docker rm commands. The next 24-hour interval, yous rerun the container once more, and are ready to commencement hacking on your tensorflow piece of work. However, when y'all go to run the notebook cells, you're blocked by an ImportError. How could did this happen if you already had installed tensorflow the previous mean solar day?

The trouble lies in the docker exec command. Recall that when you run exec, you lot are executing the given command to the running container. The container is merely a running process of the image, where the image is the executable that contains all the pre-installed libraries. And then, when you install tensorflow on the container, it is simply installed for that specific instance.

Therefore, shutting down the container is deleting that example from retentivity, and when you lot restart a new container from the prototype, merely the libraries contained in the paradigm will exist available for utilize again. The simply way to salve tensorflow to the list of installed packages on an paradigm is to either modify the original Dockerfile and build a new image, or to extend the minimal-container Dockerfile and build a new image from this new Dockerfile.

Unfortunately, each of these steps require an understanding of Dockerfiles, a Docker concept that we won't embrace in detail in this tutorial. However, nosotros are using the Jupyter community developed Docker images, so permit'south bank check if at that place is already a congenital Docker image with tensorflow. Looking at the Jupyter github repository again, we tin can see that there is a tensorflow notebook!

Not simply tensorflow, simply there are quite a few other options as well. The post-obit tree diagram from their documentation describes the human relationship extension betwixt the Dockerfiles, and each bachelor image in use. Because tensorflow-notebook is extended from minimal-notebook, nosotros tin the same docker run command from before and only change the name of the image. Here'southward how we can run a Dockerized Jupyter notebook server preinstalled with tensorflow: docker run -p 8888:8888 -five ~/notebooks:/home/jovyan jupyter/tensorflow-notebook On your browser, navigate to the running server using the same method described a few sections ago. There, run import tensorflow in a code cell and yous should come across no ImportError!

Next steps

In this tutorial, nosotros covered the differences between virtualization and containerization, how to install and run Dockerized applications, and the benefits of using open up-source community developer Docker images. We used a containerized Jupyter notebook server equally an example, and showed how painless working on a Jupyter server within a Docker container is. Finishing this tutorial, you lot should experience comfy working with Jupyter community images, and be able to incorporate a Dockerized data science setup in your daily work.

While we covered a lot of Docker concepts, these were only the basics to assist you lot get started. There is a lot more to learn nearly Docker and how powerful of a tool it tin can exist. Mastering Docker will not only aid you lot in your local development fourth dimension, but can salve time and money when working with teams of data scientists. If enjoyed this instance, here are some improvements y'all can add to help learn additional Docker concepts:

1. Run the Jupyter server in detached mode as a background process.
2. Proper noun your running container to keep your processes clean.
3. Create your own Dockerfile that extends the minimal-notebook that contains your necessary data science libraries.
4. Build an paradigm from the extended Dockerfile.

Become a Information Engineer!

Learn the skills you need to work as a data engineer today. Sign up for a complimentary account and become access to our interactive Python information applied science grade content.

Tags

browntinto1988.blogspot.com

Source: https://www.dataquest.io/blog/docker-data-science/

Share this post