Skill
nginx

nginx · current version v2
Troubleshoot and support an nginx web server / reverse proxy — connect, validate config, and resolve problems on Docker, bare metal, systemd, or Kubernetes. Covers nginx -t/-T/-V, reload signals, config and log paths, worker limits, and the common 403/404/502/504/SSL failures. Use when someone reports nginx will not start or reload, returns 502/504/403, drops connections, serves the wrong site, or fails a TLS handshake.
13 downloads · published 2026-09-02
What this grants

skill nginx
Skill Card

License or terms: check the skill's own repository for license details (nginx on GitHub).
Security Audits

NanoInfra Scanner PASS no issues found
VirusTotal PASS no engines flagged this file (full report)
Version history

Version	Published	Status
v2	2026-09-02	published
v1	2026-09-02	published
Files

SKILL.md (8494 bytes)
SKILL.md

raw | preview
---
name: nginx
description: Troubleshoot and support an nginx web server / reverse proxy — connect, validate config, and resolve problems on Docker, bare metal, systemd, or Kubernetes. Covers nginx -t/-T/-V, reload signals, config and log paths, worker limits, and the common 403/404/502/504/SSL failures. Use when someone reports nginx will not start or reload, returns 502/504/403, drops connections, serves the wrong site, or fails a TLS handshake.
---

# nginx — Troubleshooting & Support

A support runbook, not a tutorial. Work top to bottom: know the version and layout, **validate
before you reload**, and treat every reload/restart as a deliberate act. Commands and output
below were run against nginx 1.31.4 in Docker.

nginx ships as a single **1.x** line — there is no nginx 7/8/9. The version axis that matters is
**stable** (even minor: 1.28, 1.26) vs **mainline** (odd minor: 1.31, 1.27), and open-source vs
the commercial tier (which adds a live status/config API). Diagnose against the running binary,
not an assumption.

## 0. Identify what you are on, and how it is deployed

```bash
nginx -v                 # nginx version: nginx/1.31.4
nginx -V 2>&1 | tr ' ' '\n' | grep -E 'conf-path|error-log|http-log|modules-path|--with-'
#   --conf-path=/etc/nginx/nginx.conf
#   --error-log-path=/var/log/nginx/error.log
#   --http-log-path=/var/log/nginx/access.log
```
`nginx -V` is the first move on an unfamiliar box: it prints the **compiled-in paths** (so you
stop guessing where the config and logs are) and the modules the binary was built with (so you
know whether, e.g., `--with-http_v2_module` is even present).

Deployment: establish it with the **detect-platform** skill (bare metal vs VM vs container vs
pod, OS/package family, init system). This skill assumes that profile and keys §1 off it.

## 1. Where things are, per deployment

**Docker**
```bash
docker ps --filter ancestor=nginx --format '{{.Names}}\t{{.Image}}\t{{.Status}}'
docker exec -it <container> nginx -t          # validate the config that is actually mounted
docker logs --tail 200 -f <container>         # nginx logs to stdout/stderr in the official image
```
In the image, access/error logs are symlinked to `/dev/stdout` and `/dev/stderr` — read them
with `docker logs`, not by catting a file.

**systemd (bare metal or VM)**
```bash
systemctl status nginx
journalctl -u nginx -n 200 --no-pager         # startup/exit errors land here
nginx -t                                       # syntax + can-it-open-its-files check
tail -f /var/log/nginx/error.log               # the runtime log the config points at
```

**Bare metal — the host limits that bite nginx**
```bash
# Open files: nginx needs ~1 fd per connection; worker_rlimit_nofile must fit under the OS limit.
cat /proc/$(pgrep -o -x nginx)/limits | grep 'open files'
ulimit -n
# Listen backlog: a burst past net.core.somaxconn is dropped before nginx sees it.
sysctl net.core.somaxconn                      # raise to match `listen … backlog=`
# Ephemeral ports for an upstream-heavy proxy: exhaustion shows as intermittent 502s.
sysctl net.ipv4.ip_local_port_range
# SELinux (RHEL): denies nginx opening a non-standard port or proxying out by default.
getenforce 2>/dev/null    # if Enforcing, check `ausearch -m avc -ts recent` and semanage/setsebool
```

**Kubernetes**
```bash
kubectl get pods -l app.kubernetes.io/name=nginx
kubectl exec -it <pod> -- nginx -t
kubectl logs <pod> --tail 200 -f
```
If nginx is a config-driven ingress, the live `nginx.conf` is generated — edit the source
(ConfigMap / the controller's resources), not the file inside the pod, which is overwritten.

## 2. The commands and arguments you will reach for

```bash
nginx -t                 # test config: syntax AND whether referenced files/certs open. ALWAYS before reload.
nginx -T                 # test, then dump the FULL effective config (all includes merged) to stdout.
nginx -V                 # version + build flags + compiled paths (stderr).
nginx -s reload          # graceful: re-read config, start new workers, drain old ones. No dropped conns.
nginx -s reopen          # reopen log files — pair with logrotate so a rotated log keeps being written.
nginx -s quit            # graceful stop (finish in-flight); `stop` is immediate.
nginx -c /path/nginx.conf   # run with a specific config file.
nginx -p /path/prefix       # set the prefix directory (where relative paths resolve).
nginx -g 'directive;'       # inject a global directive (e.g. `daemon off;` in containers).
```
Verified: `nginx -t` returns `syntax is ok` / `test is successful`; `nginx -s reload` logs
`signal process started` and exits 0.

## 3. Diagnostics — read-only first

```bash
nginx -t                                  # does the current config even load?
nginx -T | grep -nE 'listen|server_name|root|proxy_pass|ssl_certificate|include'
ps -o pid,ppid,cmd -C nginx               # one master + N workers; workers dying = crash loop
ss -ltnp | grep nginx                     # is it actually listening on the port you expect?
tail -n 100 /var/log/nginx/error.log      # the level matters: [error]/[crit] vs [warn]
```
Read the error log at the right level — `error_log … notice;` (as in the default) is noisy;
grep for `[error]`, `[crit]`, `[alert]`. Each 502/504 usually has a matching upstream line.

## 4. Common problems → resolution

**Won't start or reload**
- `nginx -t` first, always. It reports the exact file and line, and it catches the second most
  common cause after syntax: a referenced cert/log/root path nginx cannot open (permissions, or
  a path that does not exist yet).
- `bind() to 0.0.0.0:80 failed (98: Address already in use)` — another process (often an old
  nginx master, or Apache) holds the port. `ss -ltnp | grep :80` finds it.
- After editing, a reload that "did nothing": you edited a file not included by `nginx.conf`.
  `nginx -T | grep <your directive>` proves whether it is in the effective config.

**502 Bad Gateway**
nginx reached the upstream and the upstream failed or refused. It is almost never nginx.
- Check the upstream is up and listening on the address in `proxy_pass`; from the nginx host,
  `curl -v http://<upstream>` reproduces it without nginx.
- `connect() failed (111: Connection refused)` in the error log = upstream down or wrong port.
  `13: Permission denied` on a socket = SELinux (`setsebool -P httpd_can_network_connect 1` on
  RHEL) or file-mode on a unix socket.

**504 Gateway Timeout**
The upstream accepted but did not answer in time. Raise the relevant timeout only after
confirming the upstream is genuinely slow, not deadlocked:
`proxy_connect_timeout`, `proxy_read_timeout`, `proxy_send_timeout`.

**403 Forbidden / 404 Not Found**
- 403: file permissions (nginx worker user cannot read `root`), or `index`/`autoindex` with no
  index file, or an SELinux label on the docroot (`chcon -R -t httpd_sys_content_t`).
- 404 on a path you expect: `root` vs `alias` confusion, or a `try_files` that falls through.
  `nginx -T` shows the `root`/`location` that actually matched.

**Dropped connections under load**
- `worker_connections` × `worker_processes` is the ceiling; a proxy uses ~2 connections per
  client (client + upstream). Raise `worker_connections`, but raise `worker_rlimit_nofile` and
  the OS `ulimit -n` with it or nginx logs `Too many open files`.

**TLS handshake fails**
- `nginx -T | grep ssl_certificate` and confirm the chain file (fullchain, not just the leaf)
  and that the key matches: `openssl x509 -noout -modulus -in cert | openssl md5` vs the key's.
- From outside: `openssl s_client -connect host:443 -servername <name>` shows what is actually
  served for that SNI — a mismatch means the wrong `server` block is the default.

## 5. Before you run anything that changes serving

`reload`, `restart`, and config edits change what users get, and on a nanoinfra deployment they
resolve to a `mutate.remote` capability — approval interactively, a standing grant unattended.

- **`nginx -t` (or `-T`) before every reload** — a reload of a broken config on modern nginx
  keeps the old workers, but a *restart* of one exits; validating first turns "site is down" into
  "reload refused".
- `nginx -s reload` over `restart`: reload drains old workers with zero dropped connections;
  restart drops in-flight requests.
- Editing the live `nginx.conf` on a generated/ingress deployment is reverted on the next
  reconcile — change the source.