Quick.Work

The scenario: there are a few situations when we need to get host ip address to pass into docker container, recently I have this requirement to develop a simple web app using Python Flask and it is packaged as a docker container, here on known as Client (requester), client will send requests to a remote server to complete a long task, a big job which may need more than 10 seconds to complete it, lets call this Server (worker to produce big output).

The requirement: must avoid network timeout while Client waiting for Server to complete its job and to avoid Server holding too many long connections (which may resulted in too many active threads and maybe crashed).

The simple solution: Server will only do a quick check of the request parameter, if everything okay then return response of 202: Accepted without data, the actual output result data will be sent back to Client in a separate callback (asynchronous process), Client just need to create a URL endpoint to receive result from Server, Server just need to know Client's real IP address, and as Server is a REST API server therefore it is easy to get Client's IP address, unfortunately this does not work if Server is packaged as a docker container and running in MacOS, there is known issue that container in MacOS can not get real client IP address also known issue in MS Windows.

Even if Server can get Client IP address, how to get Client's port number and endpoint URL? Server can pre-store this callback full URL ('Callback') in database such as: http://192.168.0.111:8234/result/, or Server can avoid to always update database every time Client IP address is changed by storing http://client:8234/result/, and use OS 'hosts' file to dynamically translate hostname 'client' to an ip address, either way Server will be able to send result to Client's callback full URL without problem.

hosts file location in MS Windows

c:\windows\system32\drivers\etc\hosts

hosts file location in Linux and MacOS

/etc/hosts

Changing/updating hosts file is troublesome and human-error-prone, also in some cases this may not be possible due to network configuration or system permission, so we need to find a better solution.

The extended requirement: Server want to allow multiple different Clients and each Client have different callback full URLs (http[s]://[ip-address]:[port-number]/[endpoint])

The extended solution: each Client must send this callback full URL information to Server in all request's parameter along with the job description.

The final problem: if Client is packages as a docker container then how Client can get its host IP addres?

The solution for this final problem is the main info to share in this article, to solve this problem, the first thing we normally attempt is to manually inserting the hardcoded Client IP address value into docker-compose.yml, it is error-prone (either forgot to set or client host IP is changed) and not efficient (always need to check).

I've tried a few different ways and found a small script which work and tested for both MacOS and Linux, the way to start the Client docker image, instead of using the common docker-compose up -d, we can this simple script.

start_my_web_app.sh

#!/bin/sh

# Get HOST IP address automatically and forward it to docker compose argument

<< 'comments'
Tested in MacOS Catalina (10.15.5), Ubuntu 18.04, CentOS 7.8

NOTE: 
1. Make sure to only return a single value, multiple values will be invalid, test before use !! 
2. Default is to make exception for 127.0.0.1 (localhost) and 172.* (Docker network IP)
3. If there are multiple WIFI or multiple NIC then please add exception.
   * add more exceptions if necessary by adding 's/[except]//;'

TODO:
1. Make a script to support deployment in MS Windows by detect MS Windows OS then retrieve host IP address.
comments

export AUTOMATIC_HOST_IP=$(ifconfig | sed -En 's/127.0.0.1//;s/172.*//;s/.*inet (addr:)?(([0-9]*\.){3}[0-9]*).*/\2/p')

echo "------------"
echo "my host ip is:" $AUTOMATIC_HOST_IP
echo "------------"

docker-compose up -d

echo "--------------------------------------------------"
echo "--- Viewing web console log output ---------------"
echo "--- to exit console log, please press 'ctrl-c' ---"
echo "--------------------------------------------------"

docker-compose logs -f

Make sure start_my_web_app.sh is executable by invoking

$ sudo chmod +x start_my_web_app.sh

The host environment AUTOMATIC_HOST_IP can be retrieved by docker compose using the following docker-compose.yml

# docker engine >= 19.03.0
# compose >= 1.25.5
version: '3.8' 

####################################################################
### SUGGESTION: do not use variable here, eg: ${MY_VAR} ############
### so .env file does not need to be included in release package ###
####################################################################

services:
    myapp:
        # optional, specify a custom container name, rather than a generated default name.
        container_name: my_rest_server
        
        # tag the image, so we can later push this image to repository
        image: my_rest_server:2.4.0.1-b2

        # https://docs.docker.com/compose/environment-variables/
        # if there is no 'value' defined, then the 'value' will be retrieved from host environment (eg: 'export')
        environment: 
            # Pass host environment variables to container
            - AUTOMATIC_HOST_IP
            # Set new environment variables into container
            #- MANUAL_HOST_IP=192.168.0.111

        # if docker deamon is restarted then make sure this service also restarted automatically
        # if use manual "docker kill container" then this service will not be restarted
        restart: always

        # in release package, remove this 'build' block, because no source code to build!
        build:
            # path to source code
            context: . 
            # path to Dockerfile, may be outside source code working directory or inside sub-directory.
            dockerfile: Dockerfile

        ports:
            - 8234:8234
        
        # make sure to get client's REAL IP
        # tested, with or without "network_mode: host", the real Client IP can be retrieved in Linux based host OS!
        # WARNING: if defined "ports" above THEN do NOT set "network_mode:host"
        # because in MacOS, docker container will run inside 'Linux VM', so "network_mode: host" will be using "host" of Linux, not MacOS!
        #network_mode: host

        volumes:
            # TIPS: use 'path-based-volumes' (developer manage/define host location) instead of 'named-volumes' (Docker manage host location, random)
            # volume for html static files, on-the-fly change without rebuild docker
            # NOTE: if 'host' location is not exist then will be created with 'root' permission or similar to 'docker-compose' 
            
            # must have:
            - ./data/logs/:/data/logs/
            - ./data/db/:/data/db/
            - ./data/html/static/:/data/html/static/
            
            # optionals (add/remove if necessary):
            - ./test/:/test/

Inside compose-docker.yml file above, we may uncomment the line to define MANUAL_HOST_IP value which will be taken as higher priority than AUTOMATIC_HOST_IP, we need to change it before start the Client's container, if container was running then we need to restart it after change this value.

The way to get these value inside the python is shown in code snippet below

import os

def get_host_ip():
    # if not exist then return None
    manual_ip = os.getenv('MANUAL_HOST_IP')

    # priority is the manual_ip
    if manual_ip is not None and len(manual_ip.strip()) > 0:
        return manual_ip
    
    return os.getenv('AUTOMATIC_HOST_IP')

That's it, a simple way to bring host IP address into docker container, this is just a temporary workaround, hopefully in the near future Docker can provide an official (better) way than this but in the mean time we can use this small trick, enjoy~