Docker Journey with Python/Django

After some ups and downs deploying a Django App with Docker, I wanted to share my workflow on a Docker based deployment.

I will start with a classic Django App with some capabilities :

• Django 2.2.x
• LDAP authentification
• Gunicorn as WSGI server
• NGINX as reverse proxy & serving statics
• PostgreSQL DB
• Redis Cache

LDAP authentication is really useful to handle your authentication workflow if your company uses an Active Directory or an equivalent. Look at django-auth-ldap, if you want to use django.

This build is customized to work with a pypi mirror on your corporate network.
You need to update your Certificate Authority (CA) in your build if your mirror is using HTTPS.

Docker image :

Docker Best practices for writing Dockerfile is a good place to start.

I'm using multistage builds to keep the image as light as possible.

Here, this example uses CentOS as base image but you can use Ubuntu or even Alpine if image size is a necessity for you.

The logic will be the same, keep only runtime dependencies & virtual env in the release image & install build dependencies in the build stage.

1. Base stage
   • Install python & runtime dependencies
   • Configure Python & pipenv
2. Build stage
   • Install build dependencies
   • Create virtual env with pipenv
   • Install python dependencies with pipenv
3. Release stage
   • Configure wsgi server (Gunicorn here)
   • Create app folders
   • Copy virtual env from build stage
   • Copy source
   • Create volume & folders
   • Copy script
   • Expose port

# Stage 1/3 Base image
FROM centos:7.6.1810 as base-image

ENV LC_ALL=en_US.UTF-8 LANG=en_US.UTF-8 LANGUAGE=en_US.UTF-8
# Add corporate CA
COPY docker/coporate.crt /etc/pki/ca-trust/source/anchors/corporate.crt
# Update trusted CA
RUN update-ca-trust extract

# Add CentOS mirror repository configuration.
COPY docker/CentOS-Base.repo /etc/yum.repos.d/

ENV UID=10001 GID=10001

# Add user & group
RUN groupadd --gid $GID myapp && useradd --uid $UID --gid $GID myapp

# Install runtime dependencies
RUN set -ex \
    && yum install -y -q \
           gettext \
           postgresql-libs \
           python3-3.6.8 python3-pip \
    && yum -q clean all \
    && rm -rf /var/lib/yum/ \
    && rm -rf /var/cache/yum/ \
    && rm -rf /var/log/

# SSL verification with CA trust store
# PIP & PIPENV configuration
ENV REQUESTS_CA_BUNDLE=/etc/pki/tls/certs/ca-bundle.crt \
    PIP_INDEX=https://<corporate-pypi-mirror>/pypi/ \
    PIP_INDEX_URL=https://<corporate-pypi-mirror>/pypi/simple \
    PIP_NO_CACHE_DIR=off \
    PIPENV_VENV_IN_PROJECT=1 \
    PIPENV_NOSPIN=1 \
    PIPENV_VERBOSITY=-1
# Python configuration
ENV PYTHONDONTWRITEBYTECODE=1 \
    PYTHONIOENCODING=UTF-8 \
    PYTHONUNBUFFERED=1 \
    PYTHON_VERSION=3.6.8
ENV PIPENV_PYPI_MIRROR=${PIP_INDEX_URL}

RUN pip3.6 install pipenv

# Stage 2/3 Build stage
# Install build dependencies & install requirements in virtualenv
RUN set -ex \
    && yum install -y -q \
           gcc gcc-gfortran make \
           openldap-devel \
           postgresql-devel \
           python3-devel-3.6.8 \
    && yum -q clean all

# Install su-exec to change user after running entrypoint
ARG SU_EXEC_VERSION=0.2
ARG SU_EXEC_URL="https://github.com/ncopa/su-exec/archive/v${SU_EXEC_VERSION}.tar.gz"

RUN set -ex \
 && set -o pipefail \
 && curl -sL "${SU_EXEC_URL}" | tar -C /tmp -zxf - \ 
 && make -C "/tmp/su-exec-${SU_EXEC_VERSION}" \
 && cp "/tmp/su-exec-${SU_EXEC_VERSION}/su-exec" /usr/bin \
 && rm -rf "/tmp/su-exec-${SU_EXEC_VERSION}" \
 && yum clean all -q \
 && su-exec nobody true

# Install python dependencies
RUN pip3.6 install -q pipenv

COPY Pipfile Pipfile.lock /app/
WORKDIR /app

RUN pipenv install --deploy

# Stage 3/3 Release stage
FROM base-image as release

# Add the virtualenv to your PATH
ENV PATH="/app/.venv/bin:$PATH"
# Set Django global settings
ENV DJANGO_SETTINGS_MODULE=config.settings.production
ENV STATIC_ROOT=/app/static MEDIA_ROOT=/app/media

RUN mkdir -p /app/media \
 && mkdir -p /app/static
 
VOLUME /app/static /app/media
 
# Use shared memory mount /dev/shm in container for gunicorn heartbeat
# Pipe logs to STDOUT & STDERR
ENV GUNICORN_CMD_ARGS="--bind 0.0.0.0:8000 \
    --workers 2 \
    --threads 4 \
    --worker-tmp-dir /dev/shm \
    --log-level=info \
    --access-logfile=- \
    --error-logfile=-"
 
EXPOSE 8000

# Add scripts & binaries
COPY docker/bin/ /usr/local/bin/
RUN chmod +x /usr/local/bin/*
# Add config script for sourcing configuration when using docker exec bash
COPY docker/bin/config.sh /etc/profile.d/
COPY . /app/

# Copy virtualenv
COPY --from=build /app/.venv /app/.venv
COPY --from=build /usr/bin/su-exec /usr/bin/su-exec

ENTRYPOINT ["docker-entrypoint.sh"]
CMD ["start.sh"]

Logs

It is considered good practice to log to STDOUT/STDERR in a container.

This way, you can debug & get our logs with a simple command
docker logs or docker service logs.

To do so with Gunicorn, you just have to add:
--access-logfile=- --error-logfile=- to your GUNICORN_CMD_ARGS.

Entrypoint

The docker entrypoint for your image must be the place where you must :

• Configure your container (create / fetch configuration)
• Performing tasks on your volume (files/folder) -> like collectstatics for django
• Fix your volume permissions (when using dedicated user)
• Drop root user using gosu or su-exec (if you need root user to perform some tasks like fixing permissions).
• Setting the base command for your container.

In your entrypoint you can source your docker secrets with a bash script that most official docker images are using. (See PostgreSQL, MySQL, RabbitMQ).

I like to source my configuration to retrieve my docker secret from a config script which I use in my entrypoint. This config file can be sourced when debugging or executing commands in your container.

This is really useful if you want to manage django inside your containers or for debugging puposes.

You just have to copy config.sh to /etc/profile.d/, the config script will be executed when using a simple docker exec -it <container_id> bash.

# usage: file_env VAR [DEFAULT]
#    ie: file_env 'XYZ_DB_PASSWORD' 'example'
# (will allow for "$XYZ_DB_PASSWORD_FILE" to fill in the value of
#  "$XYZ_DB_PASSWORD" from a file, especially for Docker's secrets feature)
file_env() {
	local var="$1"
	local fileVar="${var}_FILE"
	local def="${2:-}"
	if [ "${!var:-}" ] && [ "${!fileVar:-}" ]; then
		echo >&2 "error: both $var and $fileVar are set (but are exclusive)"
		exit 1
	fi
	local val="$def"
	if [ "${!var:-}" ]; then
		val="${!var}"
	elif [ "${!fileVar:-}" ]; then
		val="$(< "${!fileVar}")"
	fi
	export "$var"="$val"
	unset "$fileVar"
}

# get secret or env variable
echo "Getting DB & LDAP configuration"
file_env 'DJANGO_SECRET_KEY'
file_env 'POSTGRES_PASSWORD'
file_env 'AUTH_LDAP_BIND_DN'
file_env 'AUTH_LDAP_BIND_PASSWORD'

#!/bin/bash
# Exit if any command fails
set -o errexit
set -o pipefail
set -o nounset

wait-for-it.sh  -h $POSTGRES_HOST -p ${POSTGRES_PORT:-5432} \
-s -t 60 -- echo "DB is up & running."

# Configuration for django (DB, LDAP configuration, secret keys)
. config.sh

# Fix volume permissions
chown -R $UID:$GID /app

# Drop down root user
set -- su-exec $UID:$GID "$@"

exec "$@"

#!/bin/bash

set -o errexit
set -o pipefail
set -o nounset

python manage.py collectstatic --noinput
gunicorn config.wsgi

Infrastructure as Code

To deploy in production our stack you need to create a docker-compose.yml file that will configure your Infrastructure.

Your YAML file will act as the desired state of your infrastructure. Upon deployment, your scheduler (either Kubernetes or Swarm), will take this state & try to match it on your cluster.

You can deploy this on Swarm or Kubernetes using on your CD pipeline:
docker stack deploy -c docker-compose.yml

Swarm deployment enable lots of features like :

• Docker secrets
• Docker services ( = easy scaling)
• Deployments options (replicas, constraints)

See the docker documentation for more details.

Here is an example of compose file compatible with Docker Swarm :

version: '3.6'
services:
  nginx:
    image: nginx:stable-alpine
    ports:
      - ${PORT:-80}:80
    volumes:
      - static_volume:/app/static:ro
      - media_volume:/app/media:ro
    configs:
      - source: nginx
        target: /etc/nginx/conf.d/<appname>.conf
    depends_on:
      - web
  web:
    image: /<docker-repo>/<image_name>:${IMAGE_TAG:-latest}
    environment:
      DJANGO_SECRET_KEY_FILE: /run/secrets/django_secret_key
      DJANGO_ALLOWED_HOSTS: example.com
      AUTH_LDAP_SERVER_URI: ldap://<ldap_server>:389
      AUTH_LDAP_BIND_DN_FILE: /run/secrets/AUTH_LDAP_BIND_DN
      AUTH_LDAP_BIND_PASSWORD_FILE: /run/secrets/AUTH_LDAP_BIND_PASSWORD
      REDIS_URL: redis://redis:6379
      POSTGRES_HOST: db
      POSTGRES_USER: <db_user>
      POSTGRES_PASSWORD_FILE: /run/secrets/db_password
      POSTGRES_PORT: 5432
      POSTGRES_DB: <database_name>
    volumes:
      - static_volume:/app/static
      - media_volume:/app/media
    depends_on:
      - redis
      - db
    secrets:
      - AUTH_LDAP_BIND_DN
      - AUTH_LDAP_BIND_PASSWORD
      - db_password
      - django_secret_key
  db:
    image: postgres:12-alpine
    volumes:
      - data_volume:/var/lib/postgresql
    environment:
      POSTGRES_USER: <db_user>
      POSTGRES_PASSWORD_FILE: /run/secrets/db_password
      POSTGRES_DB: <database_name>
    secrets:
      - db_password
  redis:
    image: redis:5-alpine

volumes:
  data_volume:
    name: <appname>.data
    driver: local
    driver_opts:
      type: nfs
      o: nfsvers=4,addr=<nfs_server_host>,rw
      device: ":/data/nfs/<appname>/data"
  static_volume:
    name: <appname>.static
    driver: local
    driver_opts:
      type: nfs
      o: nfsvers=4,addr=<nfs_server_host>,rw
      device: ":/data/nfs/<appname>/static"
  media_volume:
    name: <appname>.media
    driver: local
    driver_opts:
      type: nfs
      o: nfsvers=4,addr=<nfs_server_host>,rw
      device: ":/data/nfs/<appname>/media"

secrets:
  db_password:
    name: <appname>.db_password
    external: true
  django_secret_key:
    name: <appname>.django_secret_key
    external: true
  AUTH_LDAP_BIND_DN:
    external: true
  AUTH_LDAP_BIND_PASSWORD:
    external: true

configs:
  nginx:
    name: nginx_conf_v1
    file: ./docker/nginx.conf

CI/CD

Now we are one step away from deploying to production.

Just build a CI/CD pipeline with :

• Building our image & pushing it to our private registry.
• Deploying it to our Swarm.