Home » Home Server » 20 Docker Security Best Practices – Hardening Traefik Docker Stack

20 Docker Security Best Practices – Hardening Traefik Docker Stack

Security for Docker Traefik Stack

written by Anand September 30, 2020

With increasing docker applications and images, security for docker containers requires more attention than ever before. These Docker Security best practices will help you harden your docker host and applications.

Starting with our original Docker media server guide, followed by Traefik v1 reverse proxy tutorial, and the current Docker Home Server with Traefik v2, hundreds of thousands of users have successfully built their apps based on Docker. [Read: What is Docker: Docker vs VirtualBox, Home Server with Docker]

All though all these guides have security information scattered around, we have never had one guide describing the best practices for Docker security. And, this guide will address that gap.

I will mention upfront that I am NOT an expert in Docker Security. Rather, the focus of this guide will be to showcase the Docker security measures I have put in place, based my own research, to harden my Traefik Docker stack.

This is a basic docker security checklist and certainly NOT an exhaustive guide for enterprise or DevOps applications.

Docker Security Issues

Write for Us: Competitive pay. Familiar with Smart Home Automation, Media Streaming, HTPC, and Home Server topics? Tinkerers and hobbyists preferred. Writing experience not required. APPLY HERE.

Why do we need to secure Docker?

Well, like any piece of software, Docker has vulnerabilities. If docker security vulnerabilities are not hardened, it can lead to a disaster. Here are some example scenarios:

  • Running a container as root or elevated privileges can open the door for an app to take over your Docker host.
  • Untrusted docker images can have malicious code that could compromise sensitive data or even intentionally expose it.
  • Docker services can intentionally or unintentionally consume your host resources, leading to failure or resources being unavailable to other apps.
  • Exposing docker socket, which is owned by root, to containers can lead to a full-system takeover. For example, Traefik requires access to the docker socket. So if you use Traefik and if Traefik is compromised, then then your system is compromised as well.
  • Improperly protected (eg. weak Authentication system) can compromise your web apps.
  • Docker malware can use your resources for unintended purposes (eg. Crypto mining).

There are a lot more examples. There was a time when Docker was not ready for production. Even now, many believe that to be the case.

Personally, I have implemented several security measures over the last couple of years. This has given me enough confidence to move this WordPress blog from a traditional LEMP stack to a Docker Traefik Stack.

In this guide, I will share some of the best Docker Security practices I have implemented.

My setup is constantly changing/evolving. I will try my best to keep this post up-to-date. For my current setup, always check my GitHub Repo.

Docker Security Best Practices

Securing Docker and its applications can generally be split into several categories. Let us look both simple and some advanced security measures in each of these categories.

SmartHomeBeginner brings in-depth tutorials easy enough to understand even for beginners. This takes considerable amount of work. If this post helps you, please consider supporting us as a token of appreciation:

  • Feeling generous? Buy me a coffee (or two).
  • May be another day? Shop on Amazon using our links. Your prices won't change but we get a small commission.
  • Don't feel like spending? You can still show your support by sharing this post, linking to it in forums, or even commenting below.

Securing the Docker Host

The first category covers all the different things that you can to secure your docker host.

1. Keep Docker Host Up-to-date

Really no explanation needed here. This is the simplest of the Docker security best practices and it literally takes seconds.

Keep your docker host system up-to-date on security updates. In my Linux based Docker Traefik stack, I frequently refresh the packages and update the system using the following commands:

sudo apt-get update
sudo apt-get upgrade

2. Use a Firewall

A good Docker security practice is to block access to unnecessary ports. I do this using Universal Firewall (UFW) on Debian/Ubuntu systems. As shown in the screenshot below, I only allow access on ports 80 and 443, which are used by traefik. There are other ports that I only allow access from the local private network (192.168.0.0/16).

UFW Allowed Ports List - Docker Security

UFW Allowed Ports List

Caution: Docker does not appear to respect IP Table firewall rules (GitHub Issue #690). The developers have not fixed this in more than a year. A workaround described later in this guide is required for firewall rules to work.

If you are using a virtual private server on Cloud, your provider may also offer a firewall. For example, I use (and recommend) Digital Ocean and I have their firewall enabled to allow only certain ports to forward to the droplet.

Inbound Firewall Rules in Digital Ocean

Inbound Firewall Rules in Digital Ocean

3. Use a Reverse Proxy

Many of the docker apps listed in my Docker Traefik guide, including Traefik, require ports to be exposed to the internet to be able to access the UI from anywhere.

The best way is to not expose any ports to the internet and instead VPN into your private network and access the apps locally. But this is too cumbersome and putting apps behind a reverse proxy is a convenient but worse alternative.

You will need to setup port forwarding on your internet gateway to forward certain ports to the Docker host.

Traefik 2 Dashboard

Traefik 2 Dashboard

I strongly recommend not exposing all the docker apps to the internet. Instead, put them behind a reverse proxy. I use, Traefik and expose only ports 80 and 443 to the internet. Even the traefik dashboard which uses port 8080 is behind a reverse proxy.

Securing Docker

The next category is a big one: Docker. Let us go through, what I consider, a Docker security checklist to ensure your setup is protected.

4. Do not Change Docker Socket Ownership

Do not mess with the ownership of Docker Socket (/var/run/docker.sock in Linux). By default the socket is owned by root user and docker group.

Docker Socket Owned by root for additional Docker security

Docker Socket Ownership

For convenience sake, I have recommended adding yourself (your username) to the docker group, in the past.

The benefit is that you can run docker commands without having to use sudo. But this is a security risk. I have moved away from it and do not recommend it.

5. Do not Run Docker Containers as Root

The default behavior is for containers to run as root user inside the container, which gives root privileges. This is a security risk.

One of the best Docker security practices is to run the container as non-root user (UID not 0). Reputed and trusted images use this good security practice while building images. For example, LinuxServer.io provides docker images for several home server apps. Their images allow explicitly specifying the UID and GID as environmental variables.

If you followed my Docker Traefik guide, this should look what is shown in the code block below.

    environment:
      - PUID=$PUID
      - PGID=$PGID

6. Use Privileged Mode Carefully

The default behavior is for Docker containers to run in "unprivileged" mode. This means these containers cannot run a docker daemon inside themselves. This also disallows the use of host devices or certain kernel functions.

This is usually done by adding the following line to services:

    privileged: true

Some services require privileged mode. For example, only four services in my Docker compose file use the privileged mode:

  • Home Assistant - For accessing the Z-wave USB controller
  • Socket Proxy - A requirement for Socket proxy, which enhances the security
  • Glances - For system monitoring
  • APCUPSD - For APC UPS Daemon to communicate with the UPS via USB

In such situations, only use docker images from trusted sources (more on this later). An even better approach is to use docker capabilities.

In addition, adding the following line makes sure that the containers do not gain additional privileges:

    security_opt:
      - no-new-privileges:true

7. Use Trusted Docker Images

When possible, always use images from verified publishers or official sources as shown below.

Official Images on Docker Hub offer Docker container security

Official Images on Docker Hub

This gives immediate trust and ensures Docker container security.

With unpopular images, it is difficult to predict or guess if Docker security best practices were followed/implemented.

Unfortunately, for many home server apps such as Sonarr, Radarr, etc. there are no "official" or "verified" publishers. Therefore, you will have to go based on the popularity of the image (number of downloads/stars).

8. Use Docker Secrets

Specifying all your sensitive information (eg, API keys) in the .env file, /etc/environment, or docker-compose.yml file can be a security risk.

This is exactly why Docker secrets was introduced: to manage sensitive data.

Implementing Docker secrets for your stack is a multistep process.

A. Create Secrets Folder

First, create a secrets folder inside the docker root folder.

As a docker docker best practice, secrets folder must be owned by root

Docker Secrets Folder Permissions

Set permissions of this folder to 600, owned by the user root and group root.

sudo chown root:root ~/docker/secrets
sudo chmod 600 ~/docker/secrets

This makes this folder accessible only to the root user, adding a layer of security while accessing sensitive information.

B. Create Secret Files

Next, you will have to put your sensitive information in a file. As an example, let us define a secret for Cloudflare account email.

Let's create a file inside the secrets folder with the name cloudflare_email. Remember that you will need root permissions to create the file. On my Ubuntu system, I use:

sudo su

followed by:

nano cloudflare_email

You could use any other text editor.

In the file, the only thing that needs to be added is your Cloudflare account email, as can be seen in my GitHub Repo.

Save and exit.

C. Define Secrets in Docker Compose File

Now that the Docker secret is created, let define it in the Docker compose file. This is done using the secrets: block.

The example below shows two secrets: cloudflare_email and cloudflare_api_key.

########################### SECRETS
secrets:
  cloudflare_email:
    file: $SECRETSDIR/cloudflare_email
  cloudflare_api_key:
    file: $SECRETSDIR/cloudflare_api_key

$SECRETSDIR is the environmental variable that contains the path to Docker secrets folder. You can set this up as explained in my Docker Traefik 2 guide.

More examples are shown in my GitHub Repo.

D. Use the Secrets in Docker Services

Once defined globally, we can use the secrets in the docker-compose snippets for individual services. Since we added Cloudflare account details as Docker secrets, let us see how to use them in the docker-compose snippet for Traefik.

First, we have to make the secrets available inside the Traefik container. To do this, you have to add the following block to the docker-compose snippet for Traefik:

    secrets:
      - cloudflare_email
      - cloudflare_api_key

What this does is that it makes the secret file available at /run/secrets folder inside the container.

Next, we can set the environment variables to read sensitive data from these secret files using the environment: block, as shown below:

    environment:
      - CF_API_EMAIL_FILE=/run/secrets/cloudflare_email
      - CF_API_KEY_FILE=/run/secrets/cloudflare_api_key

Notice that the environmental variables now have _FILE appended at the end. Don't miss this or it won't work.

Save and recreate the service (in this case Traefik) and check the logs for any errors. If Traefik is unable to read the secrets correctly, you will see it as an error in the logs.

In order for Docker secrets to work properly, the container's base image must support it. If the image is a reputed/trusted image, the chances are very high that the developers have implemented Docker security best practices, including Docker secrets.

I have moved pretty much all my sensitive information to Docker secrets.

9. Use a Docker Socket Proxy

Any time you expose the Docker socket to a service, you are making it easier for the container to gain root access on the host system.

But, some apps require access to Docker socket and API (eg. Traefik, Glances, Dozzle, Watchtower, etc.).

If Traefik gets compromised, then your host system could be compromised. Traefik's own documentation lists using a Socket Proxy as a solution.

A socket proxy is like a firewall for the docker socket/API. You can allow or deny access to certain API.

I started with Tecnativa's Socket Proxy but recently moved to FluenceLab's Socket Proxy as it provided more granular control.

As always, refer to my GitHub Repo for the current version as I keep my repo more frequently updated than this post.

To add socket proxy, first create a separate network for the proxy:

sudo docker network create socket_proxy

Alternative, you can define the network gateway and subnet to use (this is what I do):

sudo docker network create --gateway 192.168.91.1 --subnet 192.168.91.0/24 socket_proxy

Then, add the network to your compose file (ignore the first line if you already have a networks: block):

networks:
  socket_proxy:
    external:
      name: socket_proxy

And finally, here the docker-compose snippet to add socket proxy for improving docker security:

  # Docker Socket Proxy - Security Enchanced Proxy for Docker Socket
  socket-proxy:
    container_name: socket-proxy
    image: fluencelabs/docker-socket-proxy
    restart: always
    networks:
      socket_proxy:
        ipv4_address: 192.168.91.254 # You can specify a static IP
    privileged: true
    ports:
      - "127.0.0.1:2375:2375"
    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock"
    environment:
      - LOG_LEVEL=info # debug,info,notice,warning,err,crit,alert,emerg
      ## Variables match the URL prefix (i.e. AUTH blocks access to /auth/* parts of the API, etc.).
      # 0 to revoke access.
      # 1 to grant access.
      ## Granted by Default
      - EVENTS=1
      - PING=1
      - VERSION=1
      ## Revoked by Default
      # Security critical
      - AUTH=0
      - SECRETS=0
      - POST=1 # Watchtower
      - DELETE=1 # Watchtower
        # GET Optons
      - BUILD=0
      - COMMIT=0
      - CONFIGS=0
      - CONTAINERS=1 # Traefik, portainer, etc.
      - DISTRIBUTION=0
      - EXEC=0
      - IMAGES=1 # Portainer, Watchtower
      - INFO=1 # Portainer
      - NETWORKS=1 # Portainer, Watchtower
      - NODES=0
      - PLUGINS=0
      - SERVICES=1 # Portainer
      - SESSION=0
      - SWARM=0
      - SYSTEM=0
      - TASKS=1 # Portaienr
      - VOLUMES=1 # Portainer
      # POST Options
      - CONTAINERS_CREATE=1 # WatchTower
      - CONTAINERS_START=1 # WatchTower
      - CONTAINERS_UPDATE=1 # WatchTower
      # DELETE Options
      - CONTAINERS_DELETE=1 # WatchTower
      - IMAGES_DELETE=1 # WatchTower
Caution: Do not ever expose port 2375 to the internet. You will get hacked (here is an example). This is even more important for virtual private servers that typically expose all ports. Enable a firewall to allow only ports 80 and 443 (and block the rest) to passthrough to your server and also implement the Docker IP Tables workaround described later in this guide.

In addition, port 2375 should only ever get exposed to the internal network (127.0.0.1:2375).

In the environment: block we specify the Docker API section that we want to open up or close. I have added comments to describe which services require what API sections. For example, if you do not use WatchTower, you can enter 0 for several of the API sections.

Once the Socket proxy container starts, you can replace the direct access to Docker socket with the Socket Proxy for all the services that require it. This can be done in several ways, depending on how the container image supports it.

For Traefik, replace the following CLI argument (if you use CLI arguments instead of static configurations):

      - --providers.docker.endpoint=unix:///var/run/docker.sock

with

      - --providers.docker.endpoint=tcp://socket-proxy:2375

For other services, you may remove specifying Docker Socket as a volume (the following line under volumes:):

      - /var/run/docker.sock:/var/run/docker.sock:ro

and add the DOCKER_HOST environmental variable as shown below:

      DOCKER_HOST: tcp://socket-proxy:2375

This is what I do for Glances, WatchTower, and Dozzle. If you are lost, check my Docker Compose file to see how this is done.

Recreate your stack and your services should be using the secure Docker socket proxy instead of the docker socket.

10. Change DOCKER_OPTS to Respect IP Table Firewall

I accidentally stumbled upon this issue. I enabled UFW like I always do on my Digital Ocean VPS and blocked everything except 80 and 443. I unintentionally tried to access one of the services using the port number and was shocked that I was connected.

Upon digging further, I came across this open issue on GitHub. Why this has been open for more than a year despite the huge number of people requesting a fix is beyond me.

So if you have Socket proxy enabled and firewall enabled, due to the security flaw in docker, hackers can still hack into your system using the socket proxy port (2375).

Fortunately, there is a workaround. On Ubuntu/Debian based systems, edit /etc/default/docker and add the following line:

DOCKER_OPTS="--iptables=false"

Save the file and restart the docker service.

Try to confirm the fix by accessing one of your services using WAN-IP:PORT.

11. Control Docker Resource Usage

I loved being able set resources for Docker services. Unfortunately, this is only possible with Docker-Compose version 2 or Docker Swarm mode.

If you are using either of those, you can set resource limits for Docker services.

Here is a docker-compose example for setting resource limits in Docker Swarm mode:

    deploy:
      resources:
        limits:
          cpus: '0.50'
          memory: 50M
        reservations:
          cpus: '0.25'
          memory: 20M

By setting resource limits, you can restrict any service that goes rogue and hogs your system resources.

Securing Docker Applications using Traefik

There are things that can be done on Traefik side to harden your stack against malicious attacks. Let us look at some of the Traefik security measures that can be implemented.

I continue to make changes to my setup. For the current set of Traefik security middleware, always check my GitHub Repo.

12. Rate Limit

Rate limiting is quite common to mitigate brute force or denial of service attacks. In my Traefik Docker stack, I have a middleware to define rate-limiting.

    middlewares-rate-limit:
      rateLimit:
        average: 100
        burst: 50

The above generic set of numbers work great for me. It can be customized to your situation using Traefik's documentation on rate-limiting.

13. Traefik Security Headers

Security headers are basic requirements for a website's security. They protect against various attacks, including XSS, click-jacking, code injection, and more.

Explaining the purpose of these headers is beyond the scope of this post.

Here are the Traefik security headers I have defined as middleware:

    middlewares-secure-headers:
      headers:
        accessControlAllowMethods:
          - GET
          - OPTIONS
          - PUT
        accessControlMaxAge: 100
        hostsProxyHeaders:
          - "X-Forwarded-Host"
        sslRedirect: true
        stsSeconds: 63072000
        stsIncludeSubdomains: true
        stsPreload: true
        forceSTSHeader: true
        # frameDeny: true #overwritten by customFrameOptionsValue
        customFrameOptionsValue: "allow-from https:example.com" #CSP takes care of this but may be needed for organizr.
        contentTypeNosniff: true
        browserXssFilter: true
        # sslForceHost: true # add sslHost to all of the services
        # sslHost: "example.com"
        referrerPolicy: "same-origin"
        # Setting contentSecurityPolicy is more secure but it can break things. Proper auth will reduce the risk.
        # the below line also breaks some apps due to 'none' - sonarr, radarr, etc.
        # contentSecurityPolicy: "frame-ancestors '*.example.com:*';object-src 'none';script-src 'none';"
        featurePolicy: "camera 'none'; geolocation 'none'; microphone 'none'; payment 'none'; usb 'none'; vr 'none';"
        customResponseHeaders:
          X-Robots-Tag: "none,noarchive,nosnippet,notranslate,noimageindex,"
          server: ""

There was a bug in Traefik that prevented one from defining security headers in both dynamic and static configuration. That has since been closed.

So it is now possible to add sslForceHost and sslHost to individual services, if you prefer, for additional security.

14. TLS Options

TLS options allow the configuration of TLS connections to secure the connection between the client and your service. More explanation can be found in Traefik's TLS Documentation.

In my setup, I have defined the following TLS options for Traefik:

tls:
  options:
    default:
      minVersion: VersionTLS12
      sniStrict: true
      cipherSuites:
        - TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
        - TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
        - TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305
        - TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
        - TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
        - TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305
        - TLS_AES_128_GCM_SHA256
        - TLS_AES_256_GCM_SHA384
        - TLS_CHACHA20_POLY1305_SHA256
      curvePreferences:
        - CurveP521
        - CurveP384

15. Multifactor Authentication

This one is becoming more and more obvious/must-have. If you have not protected your docker apps with multi-factor authentication, do it right now. I have tested and used two authentication systems: Google OAuth and Authelia.

Google OAuth for Docker Apps

Refer to our detailed guide on setting up Google OAuth for Docker.

I started with Google OAuth, which worked great and there was minimal maintenance.

Authelia Self-Hosted MFA for Docker Apps

Mid-2020 (during the COVID-19 pandemic), I made the switch to Authelia. Authelia offers a lot more control but that also means more maintenance. [Read: Authelia Tutorial – Protect your Docker Traefik stack with Private MFA]

As mentioned before, this website now runs on Docker Traefik 2 stack with multifactor authentication from Authelia with Duo Push notification. I love it. But I have also run into minor issues and had to recreate Authelia to get back in business.

CloudFlare Settings

Not everybody uses Cloudflare. In my Docker Traefik guide, I recommended using Cloudflare for all the very nice features it offers even in the free plan.

In Cloudflare settings for Docker post, I described all the different Cloudflare settings and how to optimize them for Docker security and performance.

16. Secure Docker Containers Using Cloudflare

For details, review the post linked above. Here is a summary of the key Cloudflare settings to enhance the security of Docker containers when exposed to the internet.

  • Cloudflare Proxy - Enabled to utilize Cloudflare's security and performance enhancements.
    Cloudflare DNS Entries

    Cloudflare DNS Entries

  • SSL Mode - Full or Strict. This encrypts the connection between origin server to Cloudflare and from Cloudflare to client.
  • Edge Certificates:
    • Always Use HTTPS: ON
    • HTTP Strict Transport Security (HSTS): Enable (Be Cautious)
    • Minimum TLS Version: 1.2
    • Opportunistic Encryption: ON
    • TLS 1.3: ON
    • Automatic HTTPS Rewrites: ON
    • Certificate Transparency Monitoring: ON
  • Firewall Rules - Create rules to allow or deny certain traffic (eg. I only allow traffic from US to my private apps as I access it only from the United States).
    Cloudflare Firewall Rules

    Cloudflare Firewall Rules

  • Firewall Settings:
    • Security Level: High
    • Bot Fight Mode: ON
    • Challenge Passage: 30 Minutes
    • Browser Integrity Check: ON

Those are the Docker security-related Cloudflare settings. To optimize performance-related settings, refer to my Cloudflare settings for Docker post.

SmartHomeBeginner brings in-depth tutorials easy enough to understand even for beginners. This takes considerable amount of work. If this post helps you, please consider supporting us as a token of appreciation:

  • Feeling generous? Buy me a coffee (or two).
  • May be another day? Shop on Amazon using our links. Your prices won't change but we get a small commission.
  • Don't feel like spending? You can still show your support by sharing this post, linking to it in forums, or even commenting below.

Other Security Improvements for Docker Traefik Stack

All the above docker security best practices are what I have implemented so far. But there are more and I strongly recommend exploring the following for added security.

17. Fail2ban

Fail2ban scans your log files and bans IP address that shows malicious intent (eg. looking for exploits, password failures, etc.). When a suspicious activity is found, it updates the firewall rules to block the IP address for a specified amount of time.

The reason I have not given this a priority is that I have Authelia authentication system, which has a built-in login limits. But this is still something I plan to implement in the future for improving the security of Docker apps as well as other services.

18. Docker Bench Security

The Docker Bench for Security is a script that checks for dozens of common best-practices around deploying Docker containers in production.

This is quite easy to implement and I will add it to my stack in the near future.

19. RBAC

RBAC is role-based access control. If you are an enterprise or have multiple users, this is a must-have. It can be quite expensive to implement but portainer makes it super easy (as a sidenote FunkyPenguin's cookbook is awesome...check it out if you haven't) for a nominal fee.

20. Container Vulnerability Scanner

The last on the list of best practices for docker security is a vulnerability scanner. There are a few examples here but I will list just one: Clair.

Clair is an open-source project for the static analysis of vulnerabilities in application containers.
It uses the Clair API to index the container images and then matches it against known docker security vulnerabilities.

This is also something I would love to implement at some point in the future.

Final Thoughts on Docker Security Tips

As I said at the beginning of the guide, I am not an expert on Docker or Security. As with everything on this site, I research, learn, try, and then share what I learned with the community.

Docker security is an important subject that requires serious consideration. This Docker security guide is meant to be a starting point for beginners and by no means exhaustive. I strongly suggest continuing to look for additional security measures or even enlist professional help to harden your docker stack.

The docker container security measures I have put in place have worked well for me so far. I will continue to explore more and keep this guide updated. Meanwhile, I hope that the Docker security best practices listed in this guide serves as Docker security checklist and strengthens your setup.

Write for Us: Competitive pay. Familiar with Smart Home Automation, Media Streaming, HTPC, and Home Server topics? Tinkerers and hobbyists preferred. Writing experience not required. APPLY HERE.
Did this post help you?
SmartHomeBeginner brings in-depth tutorials easy enough to understand even for beginners. If you are reading this, please consider buying us a coffee (or two) as a token of appreciation.

Related Articles