sq.

Diagnose a stack without resetting it

Inspect configuration, process state, routes, logs, and disk space before changing anything.

Published:

Use the practice directory. These commands inspect it; they do not delete data. Logs can contain private values in real applications. Review and redact them before sharing.

sudo docker compose config --quiet
sudo docker compose ps -a
sudo docker compose logs --tail 50 web gateway metrics
sudo docker stats --no-stream
sudo docker compose port gateway 8080
df -h

Read the first error, not just the final “failed” line. Separate the layers:

  • Configuration rejected: check indentation, file paths, and missing variables before starting containers.
  • Exited or restarting: inspect that service's logs. Wrong permissions, missing settings, incompatible database versions, or memory exhaustion need different fixes.
  • Connection refused: check the process and published port. A hostname change cannot start a stopped listener.
  • 404: check the requested hostname and route. It may also be an application response.
  • 502 or 504: the proxy could not get a valid backend response. Check its service name, port, network, readiness, and logs.
  • Login loop: examine application base URL, cookies, HTTPS, and trusted-proxy settings. Do not disable authentication to hide it.
  • No space left: compare df -h, df -i, and sudo docker system df. Identify the growing data before removing anything. Never run a volume prune as a routine fix.

Break only the practice application

This intentionally stops the disposable web service. It must not be run in a production project.

sudo docker compose stop web
curl --max-time 20 -i --resolve lab.test:18080:127.0.0.1 http://lab.test:18080/
sudo docker compose logs --tail 20 gateway
sudo docker compose start web
curl --fail --resolve lab.test:18080:127.0.0.1 http://lab.test:18080/

While stopped, expect a gateway error or timeout rather than the HTML page. After restart, retry after a few seconds: the page should return. Prometheus may still report the gateway as up=1 during this outage. Its process is scrapeable even when the application is unavailable. This is why uptime checks complement metrics.

If a container was killed, inspect its State.OOMKilled field using docker inspect with a narrow --format, and compare its memory limit with measured use. Do not publish a full inspect dump: it may contain credentials.

Record the symptom, relevant error, one change, and result. That record is more useful next time than a list of restarts. Next: backup and restore.

All notes →