Custom domains & certificate sync

How a tenant-owned domain (register.furrycon.org) becomes a live HTTPS endpoint serving a Midsummer event — DNS verification, Let’s Encrypt issuance, nginx config generation, and how all of it stays consistent across a multi-pod Kubernetes deployment.

For the step-by-step deployment runbook (load balancer, PVCs, env vars), see the setup guide. This page explains the design.

The moving parts

Every pod in the deployment is a single container running three supervisord programs (supervisord.conf):

  • api — entrypoint.sh → Hypercorn (Django) on :8082.

  • proxy — nginx (start-nginx.sh → nginx.conf), listening on :8080 (HTTP) and :8443 (TLS with per-domain certs, SNI).

  • custom-domain-cron — cron -f running cron/midsummer-custom-domains.

Two load balancers sit in front:

 *.midsummer.cloud ──▶ LB1 (TLS termination, wildcard cert) ──▶ nginx :8080 ──▶ Hypercorn :8082

 custom domains ─────▶ LB2 (TCP passthrough, NO cert)
                          ├─ :80  ──▶ nginx :8080   (ACME challenges + 301→https)
                          └─ :443 ──▶ nginx :8443   (per-domain LE cert via SNI)

nginx’s :8443 server blocks live in /etc/nginx/conf.d/custom-domains.conf, which is generated from the database by tenant/services/custom_domain_nginx.py — one SSL server block per CustomDomain with status SSL_ISSUED, plus one shared :8080 block that serves ACME challenges and redirects HTTP→HTTPS. Unknown hosts on :8443 hit a snakeoil default_server that returns 421.

Data model & lifecycle

tenant.models.CustomDomain (public schema) drives everything:

dns_pending ──verify──▶ dns_verified ──issue──▶ ssl_pending ──certbot──▶ ssl_issued
                                                     │                       │
                                                     └──────▶ failed ◀───────┘ (re-issue possible)

  1. Add — a tenant superuser/developer adds the domain in tenantui (tenant/views.py CustomDomainViewSet). They get CNAME instructions pointing at CUSTOM_DOMAIN_CNAME_TARGET (default custom.midsummer.cloud, an A record on LB2’s IP).

  2. Verify DNS — verify_cname() in tenant/services/custom_domain_cert.py resolves the CNAME and checks it targets us → dns_verified.

  3. Issue — the UI enqueues tenant.tasks.issue_custom_domain_task on the certmgr Celery queue; a cron also retries all dns_verified/ssl_pending/ failed domains every 30 minutes (customdomain_issue --all-pending). issue_certificate() runs certbot certonly --webroot; on success it creates the routing Domain row (so TenantMainMiddleware resolves the host to the tenant) and regenerates the local nginx config.

  4. Serve — EventSetupMiddleware maps the request’s CustomDomain (status ssl_issued) to request.current_event — see Request resolution.

  5. Renew — daily cron (customdomain_renew) runs certbot renew; a deploy hook reloads local nginx, and cert expiry metadata is refreshed on each CustomDomain.

  6. Delete — the viewset’s destroy removes the Domain row + the CustomDomain, regenerates nginx config, and certbot deletes the lineage.

The multi-pod problem

With one replica, none of this is hard. With N replicas there are three consistency problems:

  1. Cert material — certbot writes /etc/letsencrypt on whichever pod (or Celery worker) ran it. Every pod’s nginx needs those files.

  2. ACME challenges — Let’s Encrypt validates over plain HTTP through LB2, which routes to a random pod. The challenge token must be servable from every pod, not just the one running certbot.

  3. Reload propagation — after issuance/renewal, every pod must regenerate its custom-domains.conf and reload nginx.

The design solves these with shared storage + pull-based reconcile:

Shared RWX volume (infrastructure)

/etc/letsencrypt (cert material) and /var/www/letsencrypt (ACME webroot) are mounted from a ReadWriteMany PVC on every pod of the deployment and the certmgr Celery worker. That solves problems 1 and 2 at the storage layer: certbot can run anywhere, and any pod can answer any challenge.

/etc/nginx/conf.d stays pod-local — the conf is cheap to regenerate from the DB and each pod manages its own nginx.

Pull-based reconcile (code)

Problem 3 is solved without any pod-to-pod communication. Every pod runs customdomain_regen_nginx once a minute from cron (scripts/sync_custom_domains.sh). regenerate_nginx_config() in custom_domain_nginx.py:

  1. Renders custom-domains.conf from the DB.

  2. Computes a fingerprint (compute_sync_state()): sha256 over the rendered conf text plus the hash of each issued domain’s fullchain.pem. Cert files are fingerprinted because a renewal changes the cert without changing the conf text — the conf alone can’t tell a sibling pod that it must reload.

  3. Compares against /etc/nginx/conf.d/.custom-domains.state — the fingerprint this pod’s nginx last successfully loaded (deliberately pod-local). Match → exit; the whole minute-cron tick costs two SELECTs and a few file hashes, no subprocesses.

  4. On change: write the conf atomically (tempfile + os.replace, with a .bak of the previous version), then:

    • nginx -t fails → restore the backup and raise. A broken config is never left in place.

    • nginx -s reload fails (e.g. the Celery worker, where nginx isn’t running) → keep the valid conf, log a warning, and don’t record the state file — the next cron tick retries the reload.

So issuance, renewal, and deletion converge on every pod within ~60 seconds of the shared volume/DB changing, no matter where the change happened. New pods converge at boot (entrypoint.sh runs the command after migrate_schemas) or, at the latest, on their first cron tick.

Cross-pod certbot lock

Every pod runs the same issue/renew crons on the same schedule, and a user can trigger issuance from the UI concurrently. certbot’s own lock file only serializes within a pod (fcntl locks over NFS-backed RWX volumes are not trustworthy), so custom_domain_cert.py guards all certbot entry points with a Postgres advisory lock (certbot_cluster_lock(), pg_try_advisory_lock with a fixed key):

  • issue_certificate() polls for up to 120s; if the lock stays busy it skips without changing the domain’s status, so the domain remains ssl_pending and the 30-minute retry cron picks it up.

  • renew_all_certificates() try-locks once and skips on this pod if another pod is already renewing. After a successful renewal it runs the reconcile itself so the winning pod reloads immediately.

The lock is session-scoped: if the holding pod dies mid-run, Postgres releases it when the connection drops.

Design history

Until 2026-07 this used a push model: after issuance, the issuing pod ran kubectl exec against sibling pods to trigger regeneration. That was removed — it targeted only one pod, required kubectl + RBAC in the image, could recurse across pods, and could never work anyway while certs lived on per-pod disks. Don’t reintroduce pod-to-pod orchestration here; the reconcile loop is the mechanism.

Ops quick reference

Thing

Where

Issue one domain / retry all

manage.py customdomain_issue --domain <host> / --all-pending (--force to re-issue an ssl_issued domain)

Renew everything

manage.py customdomain_renew (daily cron, 06:17 UTC)

Force conf regen + reload on a pod

manage.py customdomain_regen_nginx --force

Reconcile cadence

every minute per pod (cron/midsummer-custom-domains)

Generated conf

/etc/nginx/conf.d/custom-domains.conf (+ .bak, .custom-domains.state)

Cert material (shared)

/etc/letsencrypt — RWX PVC

ACME webroot (shared)

/var/www/letsencrypt — RWX PVC

Staging switch

ACME_STAGING env (keep true until E2E verified)

Where to look

  • Config generation + reconcile: tenant/services/custom_domain_nginx.py

  • Issuance/renewal + cluster lock: tenant/services/custom_domain_cert.py

  • Model + viewset: tenant/models.py (CustomDomain), tenant/views.py

  • Celery task: tenant/tasks.py (certmgr queue)

  • Cron + scripts: cron/midsummer-custom-domains, scripts/*.sh

  • nginx skeleton (:8080/:8443 defaults): nginx.conf

  • Pure tests: tenant/tests.py (ComputeSyncStateTests, RegenerateNginxConfigTests, CertbotClusterLockTests)