nginx — Troubleshooting & Support

A support runbook, not a tutorial. Work top to bottom: know the version and layout, validate before you reload, and treat every reload/restart as a deliberate act. Commands and output below were run against nginx 1.31.4 in Docker.

nginx ships as a single 1.x line — there is no nginx 7/8/9. The version axis that matters is stable (even minor: 1.28, 1.26) vs mainline (odd minor: 1.31, 1.27), and open-source vs the commercial tier (which adds a live status/config API). Diagnose against the running binary, not an assumption.

0. Identify what you are on, and how it is deployed

nginx -v                 # nginx version: nginx/1.31.4
nginx -V 2>&1 | tr ' ' '\n' | grep -E 'conf-path|error-log|http-log|modules-path|--with-'
#   --conf-path=/etc/nginx/nginx.conf
#   --error-log-path=/var/log/nginx/error.log
#   --http-log-path=/var/log/nginx/access.log

nginx -V is the first move on an unfamiliar box: it prints the compiled-in paths (so you stop guessing where the config and logs are) and the modules the binary was built with (so you know whether, e.g., --with-http_v2_module is even present).

Deployment: establish it with the detect-platform skill (bare metal vs VM vs container vs pod, OS/package family, init system). This skill assumes that profile and keys §1 off it.

1. Where things are, per deployment

Docker

docker ps --filter ancestor=nginx --format '{{.Names}}\t{{.Image}}\t{{.Status}}'
docker exec -it <container> nginx -t          # validate the config that is actually mounted
docker logs --tail 200 -f <container>         # nginx logs to stdout/stderr in the official image

In the image, access/error logs are symlinked to /dev/stdout and /dev/stderr — read them with docker logs, not by catting a file.

systemd (bare metal or VM)

systemctl status nginx
journalctl -u nginx -n 200 --no-pager         # startup/exit errors land here
nginx -t                                       # syntax + can-it-open-its-files check
tail -f /var/log/nginx/error.log               # the runtime log the config points at

Bare metal — the host limits that bite nginx

# Open files: nginx needs ~1 fd per connection; worker_rlimit_nofile must fit under the OS limit.
cat /proc/$(pgrep -o -x nginx)/limits | grep 'open files'
ulimit -n
# Listen backlog: a burst past net.core.somaxconn is dropped before nginx sees it.
sysctl net.core.somaxconn                      # raise to match `listen … backlog=`
# Ephemeral ports for an upstream-heavy proxy: exhaustion shows as intermittent 502s.
sysctl net.ipv4.ip_local_port_range
# SELinux (RHEL): denies nginx opening a non-standard port or proxying out by default.
getenforce 2>/dev/null    # if Enforcing, check `ausearch -m avc -ts recent` and semanage/setsebool

Kubernetes

kubectl get pods -l app.kubernetes.io/name=nginx
kubectl exec -it <pod> -- nginx -t
kubectl logs <pod> --tail 200 -f

If nginx is a config-driven ingress, the live nginx.conf is generated — edit the source (ConfigMap / the controller's resources), not the file inside the pod, which is overwritten.

2. The commands and arguments you will reach for

nginx -t                 # test config: syntax AND whether referenced files/certs open. ALWAYS before reload.
nginx -T                 # test, then dump the FULL effective config (all includes merged) to stdout.
nginx -V                 # version + build flags + compiled paths (stderr).
nginx -s reload          # graceful: re-read config, start new workers, drain old ones. No dropped conns.
nginx -s reopen          # reopen log files — pair with logrotate so a rotated log keeps being written.
nginx -s quit            # graceful stop (finish in-flight); `stop` is immediate.
nginx -c /path/nginx.conf   # run with a specific config file.
nginx -p /path/prefix       # set the prefix directory (where relative paths resolve).
nginx -g 'directive;'       # inject a global directive (e.g. `daemon off;` in containers).

Verified: nginx -t returns syntax is ok / test is successful; nginx -s reload logs signal process started and exits 0.

3. Diagnostics — read-only first

nginx -t                                  # does the current config even load?
nginx -T | grep -nE 'listen|server_name|root|proxy_pass|ssl_certificate|include'
ps -o pid,ppid,cmd -C nginx               # one master + N workers; workers dying = crash loop
ss -ltnp | grep nginx                     # is it actually listening on the port you expect?
tail -n 100 /var/log/nginx/error.log      # the level matters: [error]/[crit] vs [warn]

Read the error log at the right level — error_log … notice; (as in the default) is noisy; grep for [error], [crit], [alert]. Each 502/504 usually has a matching upstream line.

4. Common problems → resolution

Won't start or reload

• nginx -t first, always. It reports the exact file and line, and it catches the second most common cause after syntax: a referenced cert/log/root path nginx cannot open (permissions, or a path that does not exist yet).
• bind() to 0.0.0.0:80 failed (98: Address already in use) — another process (often an old nginx master, or Apache) holds the port. ss -ltnp | grep :80 finds it.
• After editing, a reload that "did nothing": you edited a file not included by nginx.conf. nginx -T | grep <your directive> proves whether it is in the effective config.

502 Bad Gateway nginx reached the upstream and the upstream failed or refused. It is almost never nginx.

• Check the upstream is up and listening on the address in proxy_pass; from the nginx host, curl -v http://<upstream> reproduces it without nginx.
• connect() failed (111: Connection refused) in the error log = upstream down or wrong port. 13: Permission denied on a socket = SELinux (setsebool -P httpd_can_network_connect 1 on RHEL) or file-mode on a unix socket.

504 Gateway Timeout The upstream accepted but did not answer in time. Raise the relevant timeout only after confirming the upstream is genuinely slow, not deadlocked: proxy_connect_timeout, proxy_read_timeout, proxy_send_timeout.

403 Forbidden / 404 Not Found

• 403: file permissions (nginx worker user cannot read root), or index/autoindex with no index file, or an SELinux label on the docroot (chcon -R -t httpd_sys_content_t).
• 404 on a path you expect: root vs alias confusion, or a try_files that falls through. nginx -T shows the root/location that actually matched.

Dropped connections under load

• worker_connections × worker_processes is the ceiling; a proxy uses ~2 connections per client (client + upstream). Raise worker_connections, but raise worker_rlimit_nofile and the OS ulimit -n with it or nginx logs Too many open files.

TLS handshake fails

• nginx -T | grep ssl_certificate and confirm the chain file (fullchain, not just the leaf) and that the key matches: openssl x509 -noout -modulus -in cert | openssl md5 vs the key's.
• From outside: openssl s_client -connect host:443 -servername <name> shows what is actually served for that SNI — a mismatch means the wrong server block is the default.

5. Before you run anything that changes serving

reload, restart, and config edits change what users get, and on a nanoinfra deployment they resolve to a mutate.remote capability — approval interactively, a standing grant unattended.

• nginx -t (or -T) before every reload — a reload of a broken config on modern nginx keeps the old workers, but a restart of one exits; validating first turns "site is down" into "reload refused".
• nginx -s reload over restart: reload drains old workers with zero dropped connections; restart drops in-flight requests.
• Editing the live nginx.conf on a generated/ingress deployment is reverted on the next reconcile — change the source.