DevOps Docker

Docker Health Check - Monitoring Your Container Health

Health checks are an essential part of a system that ensures that the system works as expected. With the release of Docker 1.12, it supports the enabling of health checks in containers and such checks ensure the containers are running and doing the job right. Running containers in Swarm mode with health-checks makes it possible to remove and reschedule unhealthy containers on itself without external monitoring tools. 

Taking a bare minimum approach, we require a shell command to determine if a container is healthy or not. When the specified command is executed against the container then it is supposed to return an exit code which determines if the container is running (exit-code = 0 = healthy) or not.

As an example, let’s create a small Python Bottle app and see how we can include a health check by running that app as a docker container.

The app directory structure looks like this :

bottle_app
       ├── Dockerfile
       └── app.py

Here is the Dockerfile :

FROM python:3.7-alpine

COPY . /app

WORKDIR /app

RUN apk add --no-cache py3-pip curl \
    && pip3 install --upgrade pip \
    && pip3 install bottle==0.12.18

CMD ["python", "app.py"]

In the end, app.py:

from bottle import Bottle, run

app = Bottle()

@app.route('/')
def home():
    return "Welcome Home!"

if __name__ == '__main__':
    run(app, host='0.0.0.0', port=8080)

Let's build the local image named bottle-app with a tag local :

$ docker build -t bottle-app:local .

It is time to run the container from the newly build image:

$ docker run --rm --name bottle-docker-app -p 8080:8080 bottle-app:local

Now opening URL http://localhost:8080 in your favorite browser will print "Welcome Home!".

Now our goal in this example is to ensure that the bottle-docker-app container is always running and serving traffic on port 8080 and for that, we are going to use the Docker health check mechanism to achieve the same.

There are 4 common places where health checks are enabled in Docker containers:

 

1. Health Check in Docker Run Command:

Docker allows us to specify a health-check command and its options/policies in docker run command.

$ docker run --health-cmd='curl -sS http://localhost:8080 || exit 1' \
    --health-timeout=10s \
    --health-retries=3 \
    --health-interval=5s \
   --name bottle-docker-app \
   -p 8080:8080 \
   bottle-app:local

In the above health check instruction, URL http://localhost:8080 is invoked using curl and if the command fails then exit code 1 is returned to mark the container unhealthy for that attempt.

 As we can see, the option --health-cmd is passed in docker run which takes a shell command. There are other options specified there so let us see what they all represent:

  • --health-cmd: This defines what shell command to run to check the health of the container. The command defined here must be present in the container/image. We are using curl throughout all our examples. 
  • --health-timeout (default 30s): Maximum allowed time for the specified health command to complete. If the command takes longer than the specified time than the container will be considered a failure.
  • --health-retries (default 3): Number of retries health check command will do on failures before marking that container unhealthy.
  • --health-interval (default 30s): This defines a delay between 2 consecutive attempts being made to execute the health-check command.

The common way to check if a container is healthy or not is done by using docker inspect command:

$ docker inspect --format "{{json .State.Health }}" bottle-docker-app

Output:

{
   "Status": "healthy",
   "FailingStreak": 0,
   "Log": [
      {
         "Start": "2020-05-31T14:06:02.159546109Z",
         "End": "2020-05-31T14:06:02.271559993Z",
         "ExitCode": 0,
         "Output": "Welcome Home!"
      }
   ]
}

 

2. Health Check in Dockerfile:

The health check is defined in Dockerfile by HEALTHCHECK instruction and that executes a command checking for exit code.

HEALTHCHECK [OPTIONS] CMD <health-check-cmd>

Where OPTIONS defines a set of other options similar to docker run command (timeout, interval, retries, etc). Now we can open Dockerfile of our bottle app to add either of the below lines right before the last line (CMD statement) and save it.

HEALTHCHECK CMD curl -sS http://localhost:8080 || exit 1

OR

HEALTHCHECK --interval=5s --timeout=2s --retries=12 CMD curl -sS http://localhost:8080 || exit 1

Now we can rebuild and re-run the same bottle-docker-app container with auto health-check monitoring using Dockerfile.

$ docker build -t bottle-app:local .
$ docker run --rm --name bottle-docker-app -p 8080:8080 bottle-app:local

 

3. Health Check in Docker-Compose File

If you use a Docker Compose file, then it is the most recommended place where health-check should be defined. One downside of defining your health check in Dockerfile is that the same health-check is enabled on all types of environments (dev, QA, and prod). Docker-compose is a great way to define the environment-specific health checks.

Continuing our same bottle app example, let us create a docker-compose.yml file and use it for running the container with health-check enabled.

docker-compose.yml with health check option :

---
version: "3.7"
services:
  bottle-app:
    build: 
      context: .
    container_name: bottle-docker-app
    ports:
      - "8080:8080"
    healthcheck:
      test: ["CMD", "curl", "-sS", "http://localhost:8080", "||", "exit 1 "]
      interval: 1m30s
      timeout: 10s
      retries: 3

Running the following commands will build and run a container using docker-compose with auto health check enabled.

$ docker-compose build .
$ docker-compose up

 

4. Health Check in Swarm Mode:

If you run docker containers in Swarm mode with docker service create or docker stack deploy then you can enable health-checks for your containers. Running docker containers in swarm mode with health checks stops and reschedules unhealthy containers itself. This can also remove incoming traffic to unhealthy containers.

Stack Deploy uses a Compose file so you can define health-checks in that file and service create command takes the same arguments like docker run command.

Docker Service Create Command:

If you want to turn your local Docker instance to Swarm mode executing 'docker swarm init' command can do that. To create docker services/containers with health checks enabled you can run:

$ docker service create --health-cmd='curl -sS http://localhost:8080 || exit 1 '\
    --health-timeout=10s \
    --health-retries=3 \
    --health-interval=5s \
    --publish mode=host,target=8080,published=8080 \
    --name bottle-docker-app \
    bottle-app:local

Docker Stack Deploy Command:

This article will not cover the entire process of how to set-up and run docker containers in Swarm mode (with stack deploy), but there is doc available at https://docs.docker.com/engine/swarm/stack-deploy/ that have detailed instructions on that. 

Here is the sample Compose file that can continue our bottle-app example to deploy services/containers in Swarm mode:

---
version: "3.7"
services:
  bottle-app:
    image: <REGISTRY_HOST_HERE>/bottle-app:local
    deploy:
      replicas: 2
      restart_policy:
        condition: on-failure
    ports:
      - "8080:8080"
    healthcheck:
      test: curl -sS http://localhost:8080 || exit 1
      interval: 1m30s
      timeout: 10s
      retries: 3

 

In a Nutshell:

  • The health check instruction tells Docker how to test a container to check that it is still up and running as expected.
  • At the minimum, there are 4 places where health check instruction can be defined.
  • Docker-compose file is the most recommended way of enabling health check for containers as you can define environment-specific health-checks.
  • Running docker containers in Swarm has a self-healing feature that stops and reschedules unhealthy containers.

Share post

  •  
  •  
  •  
  •  

Tilak S.

Technology freak, Open Source lover. Someone trying to understand many things. Wants to make a difference. Life liver and Peripatetic.

Other posts you might like