Simplify application management and align with other docker applications

The notion of application "instances" is dropped in favor of a simpler,
single-instance approach. Default values are provided for all configuration
variables so allow the app to run without additional manual configuration.

A health check and auto-heal script are added to automatically restart the
application in case it becomes unavailable during operation.

README.md

@@ -12,9 +12,7 @@ Several defaults can be overridden with command line options or by setting appro
 ## Installing
 1. extract the generated archive file in a temporary folder on the target system
 2. run `<package-name>/install.sh`
-3. add a `instance.conf` file at the root of the install directory (see notes below)
+3. (optional) configure an `app.conf` file for integration with Signalytic services (e.g. dbsync)
-4. (optional) restart the systemd service if running on a system with systemd installed: `systemctl restart streamline-emr`
-5. (optional) configure an `app.conf` file for integration with Signalytic services (e.g. dbsync)
 
 ## Running
 ### Production (systemd)
@@ -32,85 +30,75 @@ systemctl stop streamline-emr
 systemctl restart streamline-emr
 ```
 
-## Monitoring
-The script `monitor` can be used at any time to query the current status of all instances. If the `bgsystemlog` program is found, status data will be written to the Signalytic system log under the `streamline_emr` location (json format).
-
 ### Docker
-In a docker environment, the application must be started manually. This is done by running the `entry` script, for example:
+In a docker environment, the application must be started manually. This is done by running the `start` script, for example:
 ```
-/var/signalytic/clientapps/streamline/streamline-emr/entry
+/var/signalytic/clientapps/streamline/streamline-emr/start
 ```
-Note that the entry script will block until the first service stops or until interrupted.
 
-If multiple instances are run from the same machine, additional configuration may be required to differentiate the instances. This is generally the case when using the "sibling container" method for running docker from within a docker container. See the `Configuration` section for more details.
+Similarly, to stop the application:
+```
+/var/signalytic/clientapps/streamline/streamline-emr/stop
+```
+
+### Manual
+During development or initial setup, it may be convenient to manually control the application.
+Note that manually managing the application requires that the systemd service be disabled:
+```
+# check status
+systemctl status streamline-emr
+
+# stop service
+systemctl stop streamline-emr
+```
+
+From here, the same `start` and `stop` scripts used during docker development can be used to manage the application.
+
+## Monitoring
+The script `monitor` can be used at any time to query the current status and usage of the application. Two queries are made:
+- service status: current status of each docker service including health, up-time, etc.
+- resource usage: reporting on cpu, memory, disk, and network usage for each container
+
+If the `bgsystemlog` program is found, data will be written to the Signalytic system log under the `streamline_emr_status` and `streamline_emr_usage` locations using json format.
 
 ## Configuration
-Certain configuration options can be adjusted by setting environment variables or by defining these variables in a `.env` file stored in the same directory as the `docker-compose.yml` file. These variables are:
+Certain configuration options can be adjusted by setting environment variables or by defining these variables in a `.env` file stored in the same directory as the `docker-compose.yml` file. A template file `template.env` provides an example with all values set to their defaults.
-`APP_PORT`: port mapping for the web app server
-`CONTAINER_NAME`: name of the application container
-`DATA_DIR`: directory to be mounted for peristant data
-`COMPOSE_PROJECT_NAME`: assign the project name associate with the compose file - this avoids conflicts when the "same" compose file is used to manage multiple instances (e.g. when using docker sibling containers)
-`IMAGE`: optional, overrides the default `streamline-emr:2.0` image
-
-Each instance of the app uses its own `.env` file to specify its configuration. This is automated through the `start` script using information stored in a file `instances.conf`. This file defines two lists:
-`INSTANCES`: a list of unique instance names
-`APP_PORTS`: a corresponding list of port numbers to map to the local host
 
 Important note when using "sibling containers": the sibling container method uses the host docker socket to provide containers with access to docker. As a result, port mappings and volume mount points all refer to the host system rather than the container from which the docker command originates.
 
-### Example Configuration Files
-The following snippet shows an `instances.conf` file with 3 instances defined. Instance `test0001-_default` will use port `3001`, `test0002-_default` will use `3002`, etc.
-```
-INSTANCES="test0001-_default" "test0002-_default" "test0003-_default"
-APP_PORTS=3001 3002 3003
-```
-
 ### App Configuration
-An `app.conf` file is required for integration with other Signalytic services. A template is provided which demonstrates integration with the `dbsync` service.
+An `app.conf` file is required for integration with other Signalytic services. A template is provided which demonstrates integration with the `dbsync` service. Note that all empty fields must be assigned values before use.
 
-## Scripts
+## Implementation Notes
+
+### Scripts
 Several scripts are used to manage app instances, including:
-`entry`: main entry script used to start all configured instances - does not return until a service stops or the process is interrupted.
+`start`: main entry point - calls either `setup` or `docker compose up -d`
-`start`: loads configuration details from `instances.conf`, then configures and starts each application instance in a dedicated subfolder
+`setup`: run once - loads default data for the facility and applies initial database configuration then starts the application
-`stop`: locates and stops all instances, even if no longer defined in `instances.conf`
+`stop`: stops the application
-`monitor`: checks the status of each instance service and sends data to `bgsystemlog` if possible (loc: `streamline_emr`)
+`autoheal`: managed by `start` and `stop` scripts, checks for and restarts any services that become `unhealthy`
-
+`monitor`: checks the status of each instance service and sends data to `bgsystemlog` if installed
-## Docker Images
 
+### Docker Images
 We want to use our own copy of the Docker images hosted on our GitLab Registry. The `docker` folder contains a set of Dockerfiles that should be used for generating images.
 
-### Automated
+If not already configured, ensure appropriate docker credentials are available:
-The script `build-image` can be used to automatically build, tag, and push images with standard naming conventions. To build and push all images, simple run `build-image -p`. To simply build images and skip pushing to a registry, omit the `-p` flag.
+```bash
+# Login to the GitLab registry (Signalytic packages)
+docker login registry.gitlab.com
+
+# Login to an external Docker Hub registry
+docker login -u <user> -p <credentials>
+```
+
+The script `docker/build-image` can be used to automatically build, tag, and push images with standard naming conventions. To build and push all images, simple run `build-image -p`. To simply build images and skip pushing to a registry, omit the `-p` flag.
 
 Building for another platform (e.g. building arm64 from amd64 host) may fail if the Dockerfile requires running certain commands. To build only specific images, dockerfiles may be specified with `build-image -f <DOCKERFILE> [ -f <DOCKERFILE> ... ]`
 
 Other options can be specified to override the image name or version. See the help menu for full details: `build-image -h`
 
-### Manual
+#### Conventions
-1. Make sure you are logged in to the Signalytic GitLab Registry:
+- We are building for two platforms: `linux/amd64` (x86_64) and `linux/arm64` (aarch64)
-
+- One images is currently required: `streamline-emr`
-    ```bash
+- Images will be stored to GitLab with the name: `registry.gitlab.com/signalytic/client-external/streamline/streamline-emr/<image>/<arch>:<tag>`
-    # Login to the registry
+  - example: `registry.gitlab.com/signalytic/client-external/streamline/streamline-emr/streamline-emr/arm64:latest`
-    docker login registry.gitlab.com
-    ```
-
-2. Build an image (note: we're using platforms `linux/amd64` and `linux/arm64`)
-
-    ```bash
-    # Build an image
-    docker build -f ${DOCKERFILE} --platform ${PLATFORM} -t ${IMAGE_TAG} .
-    ```
-
-3. Make a copy of an existing image:
-
-    ```bash
-    # Create a tag
-    docker tag ${IMAGE_ID} registry.gitlab.com/signalytic/client-external/streamline/streamline-emr/${NAME}:${TAG}
-    ```
-
-3. Push the image to our GitLab Registry:
-
-    ```bash
-    # Push the image
-    docker push registry.gitlab.com/signalytic/client-external/streamline/streamline-emr/${NAME}:${TAG}
-    ```

docker/Dockerfile.streamline-emr.amd64

@@ -3,7 +3,6 @@
 # COPY my.cnf /etc/mysql/my.cnf
 # COPY mysql-init.sh /var/www/html/.docker/mysql/mysql-init.sh
 
-
 FROM php:8.2-fpm-alpine
 
 # Install necessary packages and cleanup   

package.sh

@@ -37,7 +37,7 @@ IMAGES_LIST=${IMAGES_LIST:-${PROJECT_ROOT}/images.csv}
 
 
 function print_usage {
-  echo "Kenya EMR packaging utility"
+  echo "Streamline EMR packaging utility"
   echo
   echo "Usage: package.sh [ OPTIONS ]"
   echo

root/etc/systemd/system/streamline-emr.service

@@ -1,14 +1,16 @@
 [Unit]
 Description=Run Streamline EMR
-After=network.target
+Requires=docker.service
-StartLimitIntervalSec=0
+After=docker.service
 
 [Service]
-Type=simple
+Type=oneshot
-Restart=always
+RemainAfterExit=yes
-RestartSec=10
 User=root
-ExecStart=/var/signalytic/clientapps/streamline/streamline-emr/entry
+WorkingDirectory=/var/signalytic/clientapps/streamline/streamline-emr
+ExecStart=/var/signalytic/clientapps/streamline/streamline-emr/start
+ExecStop=/var/signalytic/clientapps/streamline/streamline-emr/stop
+TimeoutStartSec=0
 
 [Install]
 WantedBy=multi-user.target

root/var/signalytic/clientapps/streamline/streamline-emr/autoheal

@@ -0,0 +1,45 @@
+#!/bin/bash
+
+LOCATION="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
+
+INTERVAL=${INTERVAL:-60}
+
+# track pid with a file `autoheal.pid`. If a previous instance is still running, exit early.
+function cleanup() {
+  rm -f ${LOCATION}/autoheal.pid
+}
+trap cleanup EXIT
+kill -0 $(cat ${LOCATION}/autoheal.pid 2> /dev/null) 2> /dev/null && echo "autoheal script already running" && exit 1
+echo $$ > ${LOCATION}/autoheal.pid
+
+# exit on SIGINT
+trap exit INT
+
+# run at fixed interval, checking for and restarting any service marked as 'unhealthy'
+pushd $LOCATION > /dev/null
+  while [ 1 ]; do
+
+    # if setup is not running, make sure containers are active and running
+    if ! kill -0 $(cat ${LOCATION}/setup.pid 2> /dev/null) 2> /dev/null; then
+
+      # get current state (running or not) and status (healthy, unhealthy, etc)
+      app_state=$(docker compose ps --format '{{.State}}' streamline-emr)
+      app_status=$(docker compose ps --format '{{.Status}}' streamline-emr)
+
+      # check if app is running and healhty
+      if [ "${app_state}" = "running" ]; then
+        if [[ "${app_status}" = *\(unhealthy\) ]]; then
+          echo "Unhealthy service detected. Restarting 'streamline-emr'"
+          docker compose restart streamline-emr
+        fi
+      # app should be running - restart
+      else
+        echo "Container not running. Starting 'streamline-emr'"
+        docker compose up -d streamline-emr
+      fi
+    fi
+
+    # wait a little while before checking again
+    sleep $INTERVAL
+  done
+popd > /dev/null

root/var/signalytic/clientapps/streamline/streamline-emr/configure

@@ -1,49 +0,0 @@
-#!/bin/bash
-
-LOCATION="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
-
-CONF_FILE=${LOCATION}/instances.conf
-INSTANCES_DIR=${LOCATION}/instances
-COMPOSE_FILE=${LOCATION}/docker-compose.yml
-SETUP_FILE=${LOCATION}/setup
-ENV_TEMPLATE=${LOCATION}/template.env
-VERSION_FILE=${LOCATION}/version
-
-version=$(head -n 1 ${VERSION_FILE} 2>/dev/null)
-if [ $? -ne 0 ];then
-  version="unknown version"
-fi
-echo "Configuring Kenya EMR: $version"
-
-if [ ! -f "${CONF_FILE}" ]; then
-  echo "missing conf file: ${CONF_FILE}"
-  exit 1
-fi
-
-echo "processing config file: ${CONF_FILE}"
-
-# load instance names and port numbers from conf file
-instances=($(sed -n "s/INSTANCES=//p" ${CONF_FILE}))
-db_ports=($(sed -n "s/DB_PORTS=//p" ${CONF_FILE}))
-http_ports=($(sed -n "s/HTTP_PORTS=//p" ${CONF_FILE}))
-https_ports=($(sed -n "s/HTTPS_PORTS=//p" ${CONF_FILE}))
-for ((i=0;i<${#instances[@]};i++)); do
-
-  # prepare directory and config files for instance
-  instance=$(echo ${instances[$i]} | tr -d '"')
-  db_port=$(echo ${db_ports[$i]} | tr -d '"')
-  http_port=$(echo ${http_ports[$i]} | tr -d '"')
-  https_port=$(echo ${https_ports[$i]} | tr -d '"')
-  echo
-  echo "configuring instance $((i+1)) / ${#instances[@]}"
-  echo "  name: ${instance}"
-  echo "  db_port: ${db_port}"
-  echo "  http_port: ${http_port}"
-  echo "  https_port: ${https_port}"
-  mkdir -p ${INSTANCES_DIR}/${instance}/data
-  sed -e "s/{{INSTANCE_NAME}}/${instance}/g" -e "s/{{INSTANCE_DB_PORT}}/${db_port}/g" -e "s/{{INSTANCE_HTTP_PORT}}/${http_port}/g" -e "s/{{INSTANCE_HTTPS_PORT}}/${https_port}/g" ${ENV_TEMPLATE} > ${INSTANCES_DIR}/${instance}/.env
-  cp ${COMPOSE_FILE} ${INSTANCES_DIR}/${instance}/
-  cp ${SETUP_FILE} ${INSTANCES_DIR}/${instance}/
-done
-
-echo "done."

root/var/signalytic/clientapps/streamline/streamline-emr/docker-compose.yml

@@ -1,9 +1,18 @@
-version: '3.9'
+version: "3.8"
 services:
   streamline-emr:
-    image: ${IMAGE:-streamline-emr:latest}
+    image: ${APP_IMAGE:-streamline-emr:latest}
-    container_name: ${CONTAINER_NAME}
+    container_name: ${APP_CONTAINER:-streamline-emr}
     volumes:
-        - "${DATA_DIR}:/var/lib/mysql"
+        - "${DATA_DIR:-/var/signalytic/clientapps/streamline/streamline-emr/data}:/var/lib/mysql"
     ports:
-      - "${APP_PORT}:80"
+      - "${APP_PORT:-3000}:80"
+    dns: 0.0.0.0
+    healthcheck:
+      test: ["CMD", "curl", "-f", "http://localhost:80"]
+      timeout: 30s
+      start_period: 3600s
+      start_interval: 10s
+      interval: 30s
+      retries: 5
+

root/var/signalytic/clientapps/streamline/streamline-emr/entry

@@ -1,27 +0,0 @@
-#!/bin/bash
-
-LOCATION="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
-
-function cleanup {
-  ${LOCATION}/stop
-  echo "done."
-}
-
-trap cleanup EXIT
-
-# configure instances
-${LOCATION}/configure
-
-# start all instances
-${LOCATION}/start
-
-# wait for any service to stop
-pids=()
-for f in $(find ${LOCATION} -mindepth 2 -maxdepth 3 -name docker-compose.yml); do
-  pushd $(dirname $f) > /dev/null
-  docker compose wait streamline-emr &
-  pids+=($!)
-  popd > /dev/null
-done
-echo "waiting on processes: ${pids[@]}"
-wait -n ${pids[@]} > /dev/null

root/var/signalytic/clientapps/streamline/streamline-emr/monitor

@@ -2,20 +2,42 @@
 
 LOCATION="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
 
-for f in $(find ${LOCATION} -mindepth 2 -maxdepth 3 -name docker-compose.yml); do
+SYSTEMLOG=${SYSTEMLOG:-1}
-  pushd $(dirname $f) > /dev/null
+if ! command -v bgsystemlog &> /dev/null; then
+  SYSTEMLOG=0
+fi
+
+# get container info
+APP_CONTAINER=${APP_CONTAINER:-streamline-emr}
+if [ -f ${LOCATION}/.env ]; then
+  APP_CONTAINER=$(sed -n 's/APP_CONTAINER=//p' ${LOCATION}/.env)
+fi
+
+pushd $LOCATION > /dev/null
+  # get process status for each service
+  echo "checking streamline-emr service status"
   for service in $(docker compose ps --services); do
     status=$(docker compose ps --format '{"name":"{{.Names}}","created":"{{.CreatedAt}}","elapsed":"{{.RunningFor}}","state":"{{.State}}","status":"{{.Status}}"}' ${service})
     if [ "${status}" ]; then
       echo $status
-        if command -v bgsystemlog &> /dev/null
+      if [ "$SYSTEMLOG" -eq 1 ]; then
-        then
+        bgsystemlog -j streamline_emr_status -m ${status}
-          bgsystemlog -j streamline_emr -m ${status}
       fi
     else
-        echo "no services found"
+      echo "no status results found for service ${service}"
     fi
   done
+  # get resource usage
+  echo "checking streamline-emr resource usage"
+  status=$(docker stats --no-stream --format '{"name":"{{.Name}}","cpu_p":"{{.CPUPerc}}","mem_p":"{{.MemPerc}}","mem_u":"{{.MemUsage}}","net_io":"{{.NetIO}}"},"block_io":"{{.BlockIO}}"}' ${APP_CONTAINER})
+  if [ "${status}" ]; then
+    echo $status
+    if [ "$SYSTEMLOG" -eq 1 ]; then
+      bgsystemlog -j streamline_emr_usage -m ${status}
+    fi
+  else
+    echo "no usage results found for container ${APP_CONTAINER}"
+  fi
 popd > /dev/null
-done
+
 echo "done."

root/var/signalytic/clientapps/streamline/streamline-emr/start

@@ -2,40 +2,24 @@
 
 LOCATION="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
 
-CONF_FILE=${LOCATION}/instances.conf
-INSTANCES_DIR=${LOCATION}/instances
 VERSION_FILE=${LOCATION}/version
-
 version=$(head -n 1 ${VERSION_FILE} 2>/dev/null)
 if [ $? -ne 0 ];then
   version="unknown version"
 fi
 echo "Starting Streamline EMR: $version"
 
-if [ ! -f "${CONF_FILE}" ]; then
+# work from the install root
-  echo "missing conf file: ${CONF_FILE}"
+pushd $LOCATION > /dev/null
-  exit 1
+  # make sure we're always working from a clean state
-fi
+  ./stop
 
-echo "processing config file: ${CONF_FILE}"
+  # start the application
+  echo "starting services"
+  docker compose up -d
 
-# load instance names and port numbers from conf file
+  # start autoheal to restart unhealthy containers
-instances=($(sed -n "s/INSTANCES=//p" ${CONF_FILE}))
+  ./autoheal &
-for ((i=0;i<${#instances[@]};i++)); do
-
-  # prepare directory and config files for instance
-  instance=$(echo ${instances[$i]} | tr -d '"')
-  echo
-  echo "starting instance $((i+1)) / ${#instances[@]}"
-
-  pushd ${INSTANCES_DIR}/${instance} > /dev/null
-
-    # setup instance
-    ${INSTANCES_DIR}/${instance}/setup
-
-    # start app
-    docker compose up -d --force-recreate
 popd > /dev/null
-done
 
 echo "done."

root/var/signalytic/clientapps/streamline/streamline-emr/stop

@@ -2,10 +2,11 @@
 
 LOCATION="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
 
-echo "stopping all instances"
+echo "stopping streamline emr"
-for f in $(find ${LOCATION} -mindepth 2 -maxdepth 3 -name docker-compose.yml); do
+pushd $LOCATION > /dev/null
-  pushd $(dirname $f) > /dev/null
+  if [ -f autoheal.pid ];then
+    kill $(cat autoheal.pid)
+  fi
   docker compose down
 popd > /dev/null
-done
 echo "done."

root/var/signalytic/clientapps/streamline/streamline-emr/template.app.conf

@@ -17,6 +17,12 @@
       "username": "",
       "password": ""
     },
-    "instances": {}
+    "instances": {
+      "streamline-emr": {
+        "source": "streamline-emr",
+        "node": "",
+        "location": "_default"
+      }
+    }
   }
 }

root/var/signalytic/clientapps/streamline/streamline-emr/template.env

@@ -1,5 +1,5 @@
-COMPOSE_PROJECT_NAME=streamline-emr-{{INSTANCE_NAME}}
+COMPOSE_PROJECT_NAME=streamline-emr
-CONTAINER_NAME=streamline-emr-{{INSTANCE_NAME}}
+DATA_DIR=/var/signalytic/clientapps/streamline/streamline-emr/data
-DATA_DIR=/var/signalytic/clientapps/streamline/streamline-emr/instances/{{INSTANCE_NAME}}/data
+APP_IMAGE=streamline-emr:latest
-APP_PORT={{INSTANCE_PORT}}
+APP_CONTAINER=streamline-emr
-#IMAGE=streamline-emr:latest
+APP_PORT=3000

root/var/signalytic/clientapps/streamline/streamline-emr/template.instances.conf

@@ -1,2 +0,0 @@
-INSTANCES=
-APP_PORTS=