Nested Docker containers: Run Docker in a Docker container

• Author
• Recent Posts

Hitesh Jethva

Hitesh is a technical writer and DevOps engineer with more than 15 years of experience in IT. He focuses on cloud computing, DevOps, Linux, Kubernetes, Docker, security, and databases.

Latest posts by Hitesh Jethva (see all)

• Ansible beginner tutorial - Wed, Nov 15 2023
• Nested Docker containers: Run Docker in a Docker container - Fri, Nov 10 2023
• Docker networking: Connect a Docker container - Fri, Oct 27 2023

Use cases for nested Docker containers

Running Docker within a container is generally not recommended for most production environments, although there are scenarios in which nested containers can be beneficial or even necessary.

Development and testing

Running applications, scripts, and tools in an isolated environment is a common use case for Docker containers, especially for development and testing purposes. Docker containers provide a safe and convenient way to test Dockerfiles without affecting the host system. Nested containers can also be useful in training environments or labs where there's a need to simulate complex network configurations or systems.

Continuous Integration/Continuous Deployment (CI/CD)

In CI/CD pipelines such as Jenkins, GitLab CI, or GitHub Actions, nested containers can be used to create isolated, reproducible testing or build environments. Running Docker in Docker allows for building, testing, and packaging containerized applications within a CI/CD workflow.

Isolation and security

Providing an extra layer of isolation between applications can be achieved using nested containers, which may be desirable in certain high-security scenarios. Additionally, running security tools and services in a nested container can help isolate them from the host system and other containers.

This guide explores the concept of Docker-in-Docker (dind) and its use cases, and explains how to run Docker inside a Docker container. We will also discuss why nested docker containers should be avoided and explain some great alternatives.

Docker socket (DooD)

This method is also called a DooD (Docker outside Docker) because it uses the Docker daemon on the host machine from a Docker container. In this method, you mount the socket (/var/run/docker.sock) of the host environment into a container. docker.sock is a Unix socket file used by the Docker daemon to listen for connections from the Docker CLI and other Docker APIs. It acts as a communication channel between the Docker daemon and the clients, enabling commands and data to be sent between them.

The command below sends an HTTP request to the Docker daemon via the Unix socket at /var/run/docker.sock to retrieve version information about the Docker installation.

curl --unix-socket /var/run/docker.sock http://localhost/version

Retrieving the Docker version using a socket

Now, use docker.sock to run Docker inside a Docker container.

docker run -v /var/run/docker.sock:/var/run/docker.sock -ti docker:18.06

This command pulls the latest Docker image from the Docker Hub registry and creates a container. After you execute the command, you will find yourself inside the container with an interactive command prompt from which you can run commands.

Creating a Docker container using the docker.sock method

Here is a more detailed explanation of the above command:

• -v /var/run/docker.sock:/var/run/docker.sock: The Docker daemon socket from the host system (/var/run/docker.sock before the colon) is mounted (-v parameter) inside the container on the same path (/var/run/docker.sock after the colon).
• -ti: The -ti flag in a Docker command is a combination of -t and -i, where -t allocates a pseudo-TTY (terminal) and -i keeps STDIN open (interactive) to allow interaction with the container via the terminal.

Let's pull the latest version of the httpd (Apache HTTP Server) image from the Docker Hub repository to your local machine.

docker pull httpd

Downloading the Apache image inside the container

Verify that the Apache image has been pulled from the Docker repository.

Docker images

Displaying the local image

Now, create a container from the Apache image.

docker container run -dit --name=apache-container httpd

Next, verify that the Apache container has been created.

docker ps

Verifying the status of the Apache container

What you see here is an Apache container running inside a Docker container. In other words, you have created a nested Docker container.

dind tag

This method uses a Docker image with a dind tag. The dind tag in Docker stands for "Docker in Docker." It refers to a Docker image tag that's configured to allow running the Docker daemon inside a Docker container. The dind image contains the necessary configurations and dependencies to start a Docker daemon inside the container it creates.

Utilizing the --privileged option when employing the dind tag is essential due to the nature of Docker-in-Docker operations, which require a higher level of system access than typical containerized processes. We essentially give the nested Docker container the same privileges as the Docker demon running on the host.

This means that the nested Docker daemon within the container can interact with the host system almost as if it were running directly on the host, allowing it to manage containers, configure networking, access devices, and perform other system-level operations that the Docker daemon typically requires.

However, it's important to note that this configuration can pose security risks, as it breaks down the isolation between the container and the host system, potentially exposing the host to malicious actions or misconfigurations from within the container.

The command below creates and starts a new detached Docker container named dind-container with elevated privileges, using the second image to facilitate Docker-in-Docker operations. A detached Docker container is one that runs in the background of your shell session instead of being attached to the terminal's input/output.

docker run --privileged --name dind-container -d docker:stable-dind

Creating a Docker container using the dind method

If you now try to run a container inside this container, you might run into this error message:

docker: Error response from daemon: cgroups: cgroup mountpoint does not exist: unknown.

To avoid this issue, you have to mount a cgroup filesystem, specifically for the systemd controller. Systemd manages services and their associated cgroups. A cgroup (control group) is a Linux kernel feature that allows allocating resources for tasks and their children by partitioning processes into hierarchical groups. Docker provides process-level isolation by making use of cgroups.

In a Docker-in-Docker scenario where you want to run a systemd-based container inside another Docker container, the inner container's systemd process may need to access and manage cgroups. Since the outer Docker container already creates its own cgroup hierarchy, the inner container (with systemd) might not find the cgroup setup it expects by default.

The commands below mount the systemd cgroup subsystem, enabling systemd to function correctly inside the nested container by accessing /sys/fs/cgroup/systemd.

sudo mkdir /sys/fs/cgroup/systemd
sudo mount -t cgroup -o none,name=systemd cgroup /sys/fs/cgroup/systemd

Next, you can connect to the dind-container using the docker exec command.

docker exec -it dind-container /bin/sh

Once you are connected to the dind-container, you will need to create and mount the cgroup again inside the container:

mkdir /sys/fs/cgroup/systemd
mount -t cgroup -o none,name=systemd cgroup /sys/fs/cgroup/systemd

Next, create an Nginx container inside the nested container using the following command:

docker container run -dit --name=nginx-container nginx:latest

Now, verify that the Nginx container is running using the docker ps command.

docker ps

Verifying Nginx container status

Sysbox runtime

Sysbox, built by Nestbox, is a distinctive container runtime that facilitates running system-level workloads within containers, essentially allowing Docker and OCI-based containers to act as virtual hosts. Unlike traditional container runtimes, Sysbox allows the execution of low-level system software in addition to conventional applications within containers.

First, stop and remove all running containers using the commands below. Without these precautions, you might run into this error when you install Sysbox:

dpkg: error processing package sysbox-ce (--configure):

installed sysbox-ce package post-installation script subprocess returned error exit status 1

This command stops all containers:

docker stop $(docker ps -q)

Make sure that you no longer need the containers before you remove them all from your system.

docker rm -f $(docker ps -a -q)

Next, download the Sysbox package file using the wget command.

wget https://downloads.nestybox.com/sysbox/releases/v0.6.2/sysbox-ce_0.6.2-0.linux_amd64.deb

Now install the downloaded package file using the apt command.

sudo apt install ./sysbox-ce_0.6.2-0.linux_amd64.deb

Once you have Sysbox runtime installed on your system, create a Docker container with a Sysbox runtime flag.

docker run --runtime=sysbox-runc --name sysbox-container -d docker:dind

Creating a Docker container using the Sysbox method

Next, connect to the Sysbox-container.

docker exec -it sysbox-container /bin/sh

You can now run any Docker command inside the container. For example, to check which Docker version is installed inside the Docker container, run the following command in the container:

docker info

You will see something like this:

Showing Docker version information

Comparing nested Docker solutions

Docker Socket Mounting (DooD)

Advantages

Simplicity: It's easier to set up compared to true Docker-in-Docker. By bind-mounting the Docker socket into a container, you enable that container to communicate directly with the host's Docker daemon.

Performance: There's no nested layer of Docker daemons; containers started by the inner Docker client run directly on the host Docker daemon.

Compatibility: Generally works with existing Docker tooling without major modifications.

Disadvantages

Security: It's a major concern. The container has full access to the Docker daemon, which potentially allows for unwanted or malicious operations, including accessing or modifying other containers.

State persistence: Changes to Docker (like image pulls) from inside the container will affect the host system and persist even after the container is removed.

Docker-in-Docker (dind tag)

Advantages

Isolation: The inner Docker daemon and its containers are isolated from the host Docker. This ensures that operations inside the nested environment don't interfere with the host's Docker setup.

Clean environment: Useful for CI/CD pipelines where you might want to have a fresh Docker environment for each build or test run.

Disadvantages

Complexity: Introduces additional layers of networking, storage, etc., which can be challenging to configure and manage.

Performance: Running an entire Docker daemon inside a container can have overhead.

Storage: The inner Docker daemon has its own set of image layers and containers, which can consume more disk space.

Sysbox Runtime

Advantages

Security: Designed to provide strong isolation between the container and the host, even when running system-level workloads inside the container.

Flexibility: Can run a variety of system-level workloads inside containers without special setup (such as system services, Docker, Kubernetes, etc.).

Performance: Optimized for system-level workloads, potentially offering better performance than traditional Docker-in-Docker.

Disadvantages

Maturity: Sysbox is newer in comparison to traditional Docker setups, so there might be concerns regarding its maturity or community support.

Compatibility: Requires the Sysbox runtime, so there might be considerations when integrating with existing systems or tools that expect the default Docker runtime.

In conclusion, the method you choose will largely depend on your specific needs, security concerns, and the environment in which you're operating. Each method has its unique strengths and weaknesses, so it's crucial to evaluate based on the criteria most important to your use case.

Downside of nested Docker containers

Nested Docker containers can lead to issues. Here are some reasons why you should avoid this approach:

• Depending on the solution you use to create nested containers, you may expose the system's Docker daemon to vulnerabilities.
• It can cause network-related issues, and containers might not be able to access services running on the host or communicate with other containers.
• Nested containers increase the complexity of troubleshooting and debugging inner containers. It can affect the outer host and any other containers running on it.
• Generally, Docker containers are used to share resources on the host system. When you run Docker in Docker, there might be a possibility of resource conflict errors, downgraded performance, and unpredictable behavior.

Conclusion

In this guide, we explored three different methods to run Docker inside a Docker container: Docker socket (Dood), dind tag, and Sysbox. The method you choose will largely depend on your specific needs, security concerns, and the environment in which you're operating. Each method has unique strengths and weaknesses. Avoid using nested containers in production to reduce complexity and improve performance.