Fix Certbot ACME HTTP-01 Challenge Failed Errors on Nginx

You run certbot --nginx or certbot certonly --webroot, and Let’s Encrypt immediately spits out an Invalid response ... 404 or Connection refused error on the ACME challenge path. Let’s dig into why Nginx is failing to serve that temporary validation token and get your SSL certificate issued.

I hit this exact wall last week spinning up a reverse proxy on Ubuntu 24.04. Let’s Encrypt looks for a specific validation token at http://yourdomain.com/.well-known/acme-challenge/ over plain HTTP. If Nginx pushes that request into an overzealous HTTPS redirect, passes it to an app backend that knows nothing about it, or drops it on a permission error, validation blows up instantly.

How the HTTP-01 Challenge Actually Works

When you request a certificate, Let’s Encrypt needs proof that you control the domain. The HTTP-01 challenge is the standard way to prove it.

Here is what happens under the hood during validation:

• Certbot generates a temporary token and drops a validation file into your local webroot directory (typically /var/www/html/.well-known/acme-challenge/).
• Certbot pings the Let’s Encrypt ACME server to say the challenge is ready.
• Let’s Encrypt sends an HTTP request to http://yourdomain.com/.well-known/acme-challenge/ on port 80.
• Nginx has to catch that request on port 80 and return the raw token with a 200 OK.
• Once Let’s Encrypt verifies the payload matches, validation passes, and Certbot pulls down your certificates to /etc/letsencrypt/live/.

If anything blocks port 80, mangles the URL path, looks in the wrong folder, or returns a 403, 404, or 502, Certbot throws the dreaded Certbot failed to authenticate some domains (HTTP-01) error.

Checking the ACME Log to Find the Real Cause

Do not guess what went wrong. Check Certbot’s console output or tail the full log at /var/log/letsencrypt/letsencrypt.log.

I always run Certbot with -v when debugging so I can see the exact handshake:

sudo certbot certonly --webroot -w /var/www/html -d example.com -d www.example.com -v

Almost every failure boils down to one of these four scenarios:

• 404 Not Found: Nginx caught the request on port 80, but checked the wrong directory, or a blanket proxy_pass forwarded the path to an app that returned 404.
• Connection Refused / Timeout: Let’s Encrypt could not reach port 80 at all—usually UFW, a cloud security group, or Nginx not listening on port 80.
• 403 Forbidden: File permissions prevent www-data from reading the challenge token in the webroot directory.
• Wrong IP Address: DNS still points to an old server, an inactive floating IP, or an edge proxy misrouting the path.

Fix 1: Add a Dedicated Location Block for .well-known

If you run a reverse proxy for Node.js, Python, or Go (like the setup in our guide on deploying Node.js on a VPS with Nginx), your catch-all location / block forwards everything to http://127.0.0.1:3000. Your Node app has no route for /.well-known/acme-challenge/, so it returns a 404.

The cleanest fix is adding a dedicated location block in Nginx that intercepts ACME challenge requests and routes them directly to a static directory before any proxying or redirects take place.

Open your Nginx configuration file:

sudo nano /etc/nginx/sites-available/example.com

Make sure your port 80 server block has an explicit handler for the challenge path:

server { listen 80; listen [::]:80; server_name example.com www.example.com; # Dedicated challenge directory location ^~ /.well-known/acme-challenge/ { default_type "text/plain"; root /var/www/html; allow all; } # Catch-all redirect to HTTPS (after SSL is setup) or reverse proxy location / { return 301 https://$host$request_uri; }
}

Notice the ^~ modifier. That tells Nginx: if the URI starts with /.well-known/acme-challenge/, stop evaluating other location blocks or regexes and serve the static file straight from /var/www/html.

Make sure the physical directory exists and has proper permissions:

sudo mkdir -p /var/www/html/.well-known/acme-challenge
sudo chown -R www-data:www-data /var/www/html
sudo chmod -R 755 /var/www/html

Test your Nginx syntax and reload:

sudo nginx -t
sudo systemctl reload nginx

Fix 2: Open Port 80 and Check Firewalls

A frequent gotcha is assuming you only need port 443 open because you are configuring SSL. Let’s Encrypt’s HTTP-01 challenge always starts on plain HTTP over port 80. If port 80 is closed, you will get a Connection refused or Timeout during connect error.

If you run UFW on Ubuntu or Debian, verify that port 80 is open:

sudo ufw status verbose

If port 80 is missing from your active rules, enable both HTTP and HTTPS:

sudo ufw allow 'Nginx Full'
sudo ufw delete allow 'Nginx HTTP' # clean up redundant rules
sudo ufw reload

If you are on AWS (Security Groups), GCP (VPC Firewall), or Oracle Cloud, ensure your ingress rules allow inbound TCP on port 80 from 0.0.0.0/0. Let’s Encrypt validates from multiple vantage points globally, so you cannot whitelist specific IP ranges.

Confirm Nginx is actively listening on port 80:

sudo ss -tulpn | grep ':80'

You should see nginx bound to 0.0.0.0:80 and [::]:80. If another process (like an orphaned Apache instance or Docker container) grabbed port 80 first, you will need to stop it, much like the port conflict checks in our Docker Compose VPS deployment guide.

Fix 3: Check DNS Records and CDN Proxies

Let’s Encrypt resolves your domain via public DNS. If your A record points to an old IP address or a stale server, the challenge request never lands on your current Nginx instance.

Check what public DNS returns using dig or curl:

dig +short example.com A
curl -4 ifconfig.me

Compare the IP from dig against your server’s actual public IP from ifconfig.me. If they differ, fix the A record at your registrar. If you are managing custom glue records, review how to create custom nameservers and glue records to make sure authoritative nameservers respond properly.

If you are using Cloudflare with proxying turned on (orange cloud), the ACME request hits Cloudflare first. While Cloudflare usually forwards ACME requests, strict page rules, HTTPS redirects, or WAF settings can block them.

If you suspect CDN interference, temporarily toggle your DNS record to DNS Only (grey cloud), wait a minute for DNS to propagate, and re-run Certbot. If you see SSL negotiation errors after turning proxying back on, see our walkthrough on fixing 502 bad gateway and SSL handshake errors between CDN and origin.

Fix 4: Remove Conflicting Default Server Blocks

Nginx matches incoming requests against server_name. But if Ubuntu’s default site configuration is still enabled with listen 80 default_server;, it can swallow requests intended for your custom domain block.

Check if the default site symlink is still active:

ls -la /etc/nginx/sites-enabled/

If default is sitting in sites-enabled alongside your real config, remove it:

sudo rm /etc/nginx/sites-enabled/default
sudo nginx -t
sudo systemctl reload nginx

Next, search for any duplicate server_name definitions across all configuration files:

grep -rnw '/etc/nginx/sites-enabled/' -e 'example.com'

Make sure your domain only appears in one active port 80 block. Duplicate server_name entries cause Nginx to pick an arbitrary config, routing the validation request to the wrong root directory.

Testing the ACME Path Manually Before Hitting Rate Limits

Let’s Encrypt enforces strict rate limits (check the official Let’s Encrypt Rate Limits documentation so you do not lock yourself out for a week). Test the static route manually with curl before firing Certbot again.

Drop a dummy test file into your challenge directory:

mkdir -p /var/www/html/.well-known/acme-challenge
echo "test-token-ok" | sudo tee /var/www/html/.well-known/acme-challenge/test.txt

Fetch that file over plain HTTP from your local machine or an external terminal:

curl -i http://example.com/.well-known/acme-challenge/test.txt

Look at the response headers. You want a clean 200 OK:

HTTP/1.1 200 OK
Server: nginx
Content-Type: text/plain test-token-ok

If you get a 301 Moved Permanently, check that your redirect preserves the full URI path and does not redirect to HTTPS before a certificate is installed. If you get a 404, check the root path in your Nginx config. Once it returns 200, clean up the test file:

sudo rm /var/www/html/.well-known/acme-challenge/test.txt

Issuing the Certificate and Verifying Auto-Renewal

Once Nginx serves the ACME challenge folder properly, run Certbot using the webroot plugin:

sudo certbot certonly --webroot -w /var/www/html -d example.com -d www.example.com

Certbot places your new certificate files in /etc/letsencrypt/live/example.com/:

• fullchain.pem: The full certificate chain (your cert plus intermediate certs).
• privkey.pem: Your private key (keep this secure).

Reference these files inside your Nginx port 443 block:

server { listen 443 ssl http2; listen [::]:443 ssl http2; server_name example.com www.example.com; ssl_certificate /etc/letsencrypt/live/example.com/fullchain.pem; ssl_certificate_key /etc/letsencrypt/live/example.com/privkey.pem; ssl_protocols TLSv1.2 TLSv1.3; ssl_ciphers HIGH:!aNULL:!MD5; location / { proxy_pass http://127.0.0.1:3000; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; }
}

Reload Nginx to start serving traffic over HTTPS:

sudo nginx -t && sudo systemctl reload nginx

Certbot sets up a systemd timer to handle renewals automatically. Test that renewal works end-to-end using the Certbot dry-run flag:

sudo certbot renew --dry-run

If the dry run reports success, your SSL setup is fully automated and renewals will run smoothly in the background.

Frequently Asked Questions

Can I use Certbot standalone mode while Nginx is running?

No. The certbot certonly --standalone command spins up its own internal web server on port 80. If Nginx is already running, you will get an error that port 80 is already bound. Either use --webroot with Nginx running, or temporarily stop Nginx with sudo systemctl stop nginx while running standalone.

Does Let’s Encrypt support HTTP-01 validation on non-standard ports?

No. Per the official Let’s Encrypt challenge specification, HTTP-01 challenges must start on standard port 80. If your provider blocks port 80, you will need to switch to DNS-01 challenge validation instead.

Why does Certbot fail when redirecting HTTP to HTTPS?

Let’s Encrypt actually follows HTTP-to-HTTPS redirects just fine, even if the destination has a self-signed or expired cert. But if your redirect strips the URI path, drops the token, or points to an invalid host, validation fails. Adding a dedicated location ^~ /.well-known/acme-challenge/ block before any HTTPS redirect prevents this completely.

How do I make Nginx reload automatically after certificate renewal?

Add a renewal hook script. Create /etc/letsencrypt/renewal-hooks/deploy/reload-nginx.sh containing systemctl reload nginx, make it executable with chmod +x, and Certbot will automatically execute it every time certificates renew successfully.