Apache HTTP Server (httpd) — Troubleshooting & Support

A support runbook, not a tutorial. Work top to bottom: find which packaging you are on (this is Apache's biggest trap), configtest before you reload, and treat every reload/restart as a deliberate act. Commands and output below were run against Apache 2.4.68.

Apache is on the 2.4 line and has been for years — there is no Apache 7/8/9, and 2.2 is long EOL. The axes that actually matter for support are the packaging (below) and the MPM (prefork / worker / event), which decides the whole process/threading model.

0. The packaging trap — identify it first

Run the detect-platform skill first — the OS/package family it reports (Debian vs RHEL) is exactly what decides the packaging below, and its SELinux/AppArmor finding drives the permission diagnostics later.

The same server ships three ways, with different binary names, control commands, and config layouts. Getting this wrong is why a documented command "does not exist":

| Packaging | Binary / control | Config root | Enable a module/site | |---|---|---|---| | Debian/Ubuntu | apache2, apache2ctl | /etc/apache2/ (sites-enabled/, mods-enabled/) | a2enmod / a2ensite + reload | | RHEL/Fedora | httpd, apachectl | /etc/httpd/ (conf.d/, conf.modules.d/) | drop a .conf, reload | | Upstream Docker image | httpd, apachectl at /usr/local/apache2/bin/ | /usr/local/apache2/conf/httpd.conf | edit httpd.conf |

Detect which:

command -v apache2ctl apachectl httpd apache2 2>&1     # which names exist here
httpd -V 2>/dev/null || apache2ctl -V                  # -V prints HTTPD_ROOT + SERVER_CONFIG_FILE

-V is authoritative — it printed HTTPD_ROOT="/usr/local/apache2" and SERVER_CONFIG_FILE="conf/httpd.conf" on the upstream image, and it also prints the MPM.

Below, apachectl means "your control binary" — substitute apache2ctl on Debian/Ubuntu.

1. Version, MPM, modules

httpd -v                 # Server version: Apache/2.4.68 (Unix)
httpd -V | grep -E 'Server MPM|HTTPD_ROOT|SERVER_CONFIG_FILE'
#   Server MPM:     event          <- event | worker (threaded) | prefork (process-per-request)
httpd -M                 # loaded modules; "(shared)" = DSO via LoadModule, "(static)" = built in

The MPM is a frequent root cause. prefork (still forced by old mod_php) is memory-heavy and caps concurrency at MaxRequestWorkers processes; event/worker are threaded. A box "running out of memory under light load" is often prefork with a high MaxRequestWorkers.

2. Where things are, per deployment

Docker (upstream image)

docker exec -it <container> httpd -t
docker exec -it <container> httpd -S            # vhost map (see §3)
docker logs --tail 200 -f <container>           # logs go to /proc/self/fd/1,2 -> docker logs

systemd (bare metal or VM)

systemctl status apache2 2>/dev/null || systemctl status httpd
journalctl -u apache2 -n 200 --no-pager 2>/dev/null || journalctl -u httpd -n 200 --no-pager
apachectl configtest                            # alias for httpd -t
tail -f /var/log/apache2/error.log 2>/dev/null || tail -f /var/log/httpd/error_log

Bare metal — host limits and SELinux

# SELinux is the number-one RHEL Apache surprise: it blocks non-standard ports, proxying out,
# and serving from a non-default docroot.
getenforce
# a denied action leaves an AVC:
ausearch -m avc -ts recent 2>/dev/null | tail
# the usual fixes are booleans/ports/labels, not disabling SELinux:
#   setsebool -P httpd_can_network_connect 1        # allow mod_proxy to reach a backend
#   semanage port -a -t http_port_t -p tcp 8080     # allow Listen 8080
#   chcon -R -t httpd_sys_content_t /srv/www        # label a custom docroot
# Open files / worker ceiling:
cat /proc/$(pgrep -o httpd 2>/dev/null || pgrep -o apache2)/limits | grep 'open files'

Kubernetes

kubectl get pods -l app=httpd
kubectl exec -it <pod> -- httpd -t
kubectl logs <pod> --tail 200 -f

3. The commands and arguments you will reach for

httpd -t              # configtest: syntax check. ALWAYS before a reload. (apachectl configtest)
httpd -S              # dump the VirtualHost map + ServerRoot + DocumentRoot + ErrorLog + ports.
httpd -M              # list loaded modules.
httpd -V              # version, MPM, compile-time paths and -D defines.
httpd -D DUMP_VHOSTS  # same vhost map as -S; -D DUMP_MODULES == -M; -D DUMP_RUN_CFG for run config.
apachectl graceful    # reload: finish in-flight requests, then re-read config. No dropped conns.
apachectl graceful-stop   # stop, but let in-flight requests finish.
apachectl restart     # hard restart: drops in-flight requests.
apachectl -k start|stop|restart|graceful     # the -k form, identical verbs.

Verified: httpd -t → Syntax OK; httpd -k graceful exits 0. The AH00558: Could not reliably determine the server's fully qualified domain name notice is harmless — set ServerName globally to silence it; it is not the cause of a failure.

4. Diagnostics — read-only first

httpd -t                                  # loads? names the file:line on failure
httpd -S                                  # which vhost wins for which name:port — the #1 "wrong site" tool
httpd -M | grep -E 'ssl|proxy|rewrite|php|mpm'   # is the module the config needs actually loaded?
ps -o pid,ppid,rss,cmd -C httpd           # (or -C apache2) parent + children; RSS to spot prefork bloat
ss -ltnp | grep -E 'httpd|apache'         # listening where you expect?
tail -n 100 /var/log/apache2/error.log    # or /var/log/httpd/error_log — the AH##### code is greppable

Every Apache error carries an AHxxxxx code; grep the code, not the prose — it maps to an exact cause in the docs.

5. Common problems → resolution

Won't start / reload

• httpd -t names the file and line. A common one: AH00526 (syntax) or a LoadModule for a module not installed — on Debian that is a missing a2enmod, on RHEL a missing package.
• (98)Address already in use: AH00072: make_sock: could not bind to address 0.0.0.0:80 — another process holds the port; ss -ltnp | grep :80.
• Started but a directive "does nothing": it is in a file not Included, or inside a <VirtualHost> that does not match the request. httpd -S shows the vhost that actually serves a given name:port.

403 Forbidden

• Filesystem: the Apache user (www-data on Debian, apache on RHEL) cannot read the docroot.
• Config: missing Require all granted in the <Directory> (2.4 denies by default), or Options without an index file.
• SELinux (RHEL): the docroot is unlabeled — chcon -R -t httpd_sys_content_t <dir>; confirm with ausearch -m avc.

500 Internal Server Error

• The app behind Apache errored, or a broken .htaccess/rewrite. The error log has the real cause (a PHP fatal, a mod_rewrite loop AH00124). A 500 with nothing in Apache's log = the handler/CGI/FPM logged it instead.

502 / 503 (as a reverse proxy)

• 502 AH00898/AH01102 from mod_proxy: the backend refused or spoke bad HTTP. curl the backend directly from the Apache host. On RHEL, setsebool -P httpd_can_network_connect 1.
• 503 Service Unavailable: mod_proxy_balancer has all workers in error state, or MaxRequestWorkers is exhausted (every worker busy) — check ps count vs the directive and the server-status page if mod_status is enabled.

Processes/memory climbing

• prefork MPM with a high MaxRequestWorkers is process-per-request: each child holds its memory. Either lower MaxRequestWorkers to fit RAM, or move off prefork (possible once nothing requires it, e.g. PHP via FPM instead of mod_php).

TLS handshake fails

• httpd -M | grep ssl — is mod_ssl even loaded? Then check SSLCertificateFile / SSLCertificateKeyFile (and the chain) match: openssl x509 -noout -modulus -in cert | openssl md5 vs the key. openssl s_client -connect host:443 -servername <name> shows what is served per SNI.

6. Before you run anything that changes serving

graceful/restart and config edits change what users get, and on a nanoinfra deployment they resolve to a mutate.remote capability — approval interactively, a standing grant unattended.

• httpd -t / apachectl configtest before every reload. A restart on a broken config exits the server (site down); validating first turns that into a refused reload.
• apachectl graceful over restart: graceful drains in-flight requests; restart drops them.
• On RHEL, reach for SELinux booleans/labels before setenforce 0 — disabling SELinux to "fix" a 403 hides the problem and removes the protection.
• On Debian, a2enmod/a2ensite/a2dissite edit symlinks under mods-enabled/sites-enabled — the reload still has to follow, and it is still a deliberate change.