Custom domains

Note

Canonical source: guides/custom-domains-setup.md. Transcluded here for the docs site. Edit the source file if anything is wrong.

(DigitalOcean Kubernetes / Rancher)

This guide walks through deploying the custom domain feature on a DO Kubernetes cluster managed via Rancher. It covers infrastructure provisioning, Kubernetes manifests, environment configuration, and the end-to-end verification checklist.

Read the design doc first: .plans/custom-domain-support.md

Architecture Recap

                    ┌──────────────────────────────────────────────┐
 *.midsummer.cloud │ LB1 (EXISTING)  TLS-term wildcard  HTTP:8080  │──▶ nginx :8080 ──▶ Hypercorn :8082
                    └──────────────────────────────────────────────┘

                    ┌──────────────────────────────────────────────┐
 custom domains    │ LB2 (NEW)  TLS passthrough  TCP 80+443        │──▶ nginx :8080 (ACME + 301→https)
                    └──────────────────────────────────────────────┘──▶ nginx :8443 (SSL, SNI per-domain cert) ──▶ Hypercorn :8082

  • LB1 (existing): Wildcard *.midsummer.cloud cert, terminates TLS, forwards HTTP to pod :8080. Untouched.

  • LB2 (new): TLS passthrough — no cert on the LB. Traffic reaches nginx :8443 which picks the per-domain LE cert via SNI. Port 80 forwarded to pod :8080 for ACME HTTP-01 challenges and 301→https redirects.

Step 1 — Provision Load Balancer 2 (LB2)

In the DigitalOcean console (or via doctl):

doctl compute load-balancer create \
  --name midsummer-custom-lb \
  --region <YOUR_REGION> \
  --forwarding-rules entry-protocol:tcp,entry-port:443,target-protocol:tcp,target-port:8443 \
  --forwarding-rules entry-protocol:tcp,entry-port:80,target-protocol:tcp,target-port:8080 \
  --health-check protocol:http,port:8080,path:/ping/,check-interval-seconds:10,response-timeout-seconds:5,healthy-threshold:2,unhealthy-threshold:3 \
  --tag k8s:<YOUR_CLUSTER_ID>

Or via the DO cloud console:

  1. Networking → Load Balancers → Create

  2. Name: midsummer-custom-lb

  3. Region: same as your cluster

  4. Forwarding Rules:

    • Rule 1: TCP:443 → TCP:8443 (TLS passthrough for custom domains)

    • Rule 2: TCP:80 → TCP:8080 (ACME HTTP-01 + redirect)

  5. Health Check: HTTP:8080/ping/ every 10s, threshold 2/3

  6. Sticky Sessions: Disabled

  7. TLS: None — this LB must NOT terminate TLS (passthrough only)

Important: Do NOT add a certificate to this LB. TLS termination happens inside nginx on port 8443 using per-domain Let’s Encrypt certs.

Note the LB’s public IP (e.g. 203.0.113.50). You’ll need it for DNS.

Step 2 — DNS Records

In your DNS provider (managing midsummer.cloud):

Record

Type

Name

Value

A

A

custom.midsummer.cloud

<LB2_PUBLIC_IP>

This custom.midsummer.cloud hostname is what tenants will CNAME their custom domains to. It’s configured via CUSTOM_DOMAIN_CNAME_TARGET in the app’s environment.

Tenants will create their own DNS records:

CNAME  register.furrycon.org  →  custom.midsummer.cloud

Step 3 — Shared Volumes for Certificates and the ACME Webroot

Certs in /etc/letsencrypt/ must survive pod restarts and be visible to every pod — certbot runs on whichever pod (or the certmgr Celery worker) picks up the work, and every pod’s nginx serves the resulting certs. Likewise the ACME webroot /var/www/letsencrypt must be shared: Let’s Encrypt validates over plain HTTP through LB2, which routes to a random pod, so any pod must be able to serve any challenge token.

Both volumes therefore need ReadWriteMany (RWX) access mode. DO Block Storage (do-block-storage) is ReadWriteOnce only — see Provisioning RWX storage on DigitalOcean below for what to use instead.

kustomize/resources/custom-domains-pvc.yaml (or equivalent Helm values)

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: midsummer-letsencrypt
  namespace: midsummer
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 1Gi
  storageClassName: nfs-rwx-storage   # OpenEBS NFS Provisioner (see below); or longhorn — NOT do-block-storage
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: midsummer-acme-webroot
  namespace: midsummer
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 1Gi
  storageClassName: nfs-rwx-storage

Update the Deployment to mount them

Add to your Deployment/Pod spec (whether you use a Helm chart, Kustomize, or Rancher UI):

volumeMounts:
  - name: letsencrypt
    mountPath: /etc/letsencrypt
  - name: acme-webroot
    mountPath: /var/www/letsencrypt

volumes:
  - name: letsencrypt
    persistentVolumeClaim:
      claimName: midsummer-letsencrypt
  - name: acme-webroot
    persistentVolumeClaim:
      claimName: midsummer-acme-webroot

Note on /etc/nginx/conf.d: deliberately pod-local (the container filesystem — no volume needed). custom-domains.conf is cheap to regenerate from the database: entrypoint.sh regenerates it at boot, and an in-pod cron reconciles it every minute (see Step 9), so each pod converges on its own. Do NOT share this directory across pods — the per-pod .custom-domains.state file tracks what that pod’s nginx has loaded.

Provisioning RWX storage on DigitalOcean

DOKS ships with only do-block-storage (RWO). Three ways to get an RWX storage class, in order of fit for this stack:

Option B — Longhorn (if you want replicated storage)

Since this cluster is managed via Rancher, Longhorn installs from Rancher → Apps → Charts → Longhorn and gives you a replicated longhorn StorageClass with RWX support (an internal NFSv4 share-manager per RWX volume, no root squash). Set storageClassName: longhorn on the PVCs.

Extra setup on DOKS: Longhorn needs open-iscsi and an NFSv4 client on every node, and DOKS node images are managed — deploy Longhorn’s longhorn-iscsi-installation and longhorn-nfs-installation DaemonSets first (see the Longhorn OS/distro docs). More moving parts than Option A, in exchange for no single-pod SPOF.

Option C — DigitalOcean managed Network File Storage (NOT drop-in)

DO’s managed NFS product looks like the obvious choice but does not work with this stack as-is:

  • Root squashing is enforced and cannot be disabled. Root can read but gets permission denied on writes. The Midsummer image runs certbot (via root cron) and nginx as root, so cert issuance would fail writing to /etc/letsencrypt. You’d have to rework the image to run the whole cert workflow as a non-root user first.

  • Only available in select regions (ATL/NYC/AMS at the time of writing), and the share must be in the same VPC as the cluster.

  • Minimum share size 50 GiB at $0.15/GiB/month ≈ $7.50/month per share — for a few megabytes of certs.

If you do go this route after de-rooting the image, it’s static provisioning (no CSI driver): create the share in the control panel, then a PersistentVolume with an nfs: source (server: + path: from the share’s details) and a PVC with storageClassName: "" — see DO’s DOKS + NFS how-to.

DIY escape hatch: a Droplet NFS server (1-click Droplet) plus a static nfs: PV also works and gives you full control over export options (including no_root_squash) — at the cost of a Droplet to patch and monitor yourself.

Step 4 — Update the Kubernetes Service

Your existing Service likely only exposes 8080 and 8082. Add port 8443:

apiVersion: v1
kind: Service
metadata:
  name: midsummer
  namespace: midsummer
spec:
  type: LoadBalancer   # or ClusterIP + Ingress if you're using an Ingress controller
  ports:
    - name: http
      port: 80
      targetPort: 8080
    - name: https-custom
      port: 443
      targetPort: 8443
    - name: app
      port: 8082
      targetPort: 8082
  selector:
    app: midsummer

If you use an Ingress controller (nginx-ingress, Traefik, etc.) instead of a raw LB, the TLS passthrough setup differs — you’d use TCP passthrough via an Ingress Service of type LoadBalancer for port 443 → 8443. The DO LB approach above is simpler.

Step 5 — Environment Variables

Add these to your Deployment’s environment (Rancher UI → Workloads → your deployment → Environment Variables, or in your Helm values / Kustomize overlay):

Variable

Value

Notes

CUSTOM_DOMAIN_CNAME_TARGET

custom.midsummer.cloud

The hostname tenants CNAME to. Must resolve to LB2’s IP.

ACME_EMAIL

ops@midsummer.cloud

Let’s Encrypt registration email.

ACME_STAGING

true

Start with true! Flip to false after E2E verification.

ACME_WEBROOT

/var/www/letsencrypt

Must match nginx.conf ACME location.

CUSTOM_DOMAIN_QUEUE

certmgr

Celery queue name for cert tasks. Create a worker for it (see Step 7).

Other existing vars (MIDSUMMER_PROD, MIDSUMMER_DB_URL, CELERY_BROKER_URL, etc.) remain unchanged.

Removed (2026-07): RELOAD_ALL_PODS, DEPLOYMENT_NAME, POD_NAMESPACE, and POD_NAME belonged to the old kubectl-exec fan-out and are no longer read. If they’re set on an existing deployment they’re harmless, but you can delete them — along with any pods/exec RBAC that was granted for it. Pod-wide sync is now handled by the per-minute reconcile cron (see Step 11).

Step 6 — Build & Deploy the Updated Image

The Dockerfile already includes the certbot/cron/ssl-cert installs and EXPOSEs port 8443. Build and push:

# From the project root
docker build -t your-registry/midsummer:custom-domains .
docker push your-registry/midsummer:custom-domains

Update your Deployment’s image tag to custom-domains (or whatever your tag strategy is).

Step 7 — Celery Worker for Certificate Management

The tenant.tasks.issue_custom_domain_task is routed to the certmgr queue. You need at least one Celery worker listening on that queue:

celery -A midsummer worker -Q certmgr --loglevel=info

In Kubernetes, add this as a sidecar container in the same pod (shares the /etc/letsencrypt volume), or as a separate Deployment with the same image + volume mount. In Rancher, you can add a sidecar via the workload UI.

Example sidecar in the Deployment:

- name: celery-certmgr
  image: your-registry/midsummer:custom-domains
  command: ["celery", "-A", "midsummer", "worker", "-Q", "certmgr", "--loglevel=info"]
  envFrom:
    - configMapRef:
        name: midsummer-env
    - secretRef:
        name: midsummer-secrets
  volumeMounts:
    - name: letsencrypt
      mountPath: /etc/letsencrypt
    - name: acme-webroot
      mountPath: /var/www/letsencrypt
    - name: nginx-confd
      mountPath: /etc/nginx/conf.d

Step 8 — Startup Procedure

No init container is needed. entrypoint.sh (the supervisord api program) runs manage.py customdomain_regen_nginx at boot, right after migrate_schemas, so /etc/nginx/conf.d/custom-domains.conf is populated from the database on every pod start. If nginx isn’t up yet at that moment the reload is skipped and the per-minute reconcile cron (Step 9) picks it up — worst case a fresh pod serves custom domains ~1 minute after boot.

Since /etc/nginx/conf.d is pod-local (Step 3), an init container in a separate container filesystem couldn’t populate it anyway — don’t add one.

Step 9 — Cron Setup

The Dockerfile installs cron, supervisord runs cron -f in every pod, and the cron file cron/midsummer-custom-domains is copied to /etc/cron.d/. Three jobs run in every pod:

Schedule

Job

Purpose

every minute

sync_custom_domains.sh → customdomain_regen_nginx -v 0

Reconcile: regenerate custom-domains.conf from the DB and reload nginx if certs or domains changed on the shared volume. Cheap no-op otherwise (a fingerprint comparison — no subprocesses). This is how issuance/renewal on one pod propagates to all pods within ~60s.

daily 06:17 UTC

renew_custom_domains.sh → customdomain_renew

Renew all issued certs.

every 30 min

issue_pending_custom_domains.sh → customdomain_issue --all-pending

Retry pending/failed issuances.

All pods firing the renew/issue jobs at the same moment is safe by design: both commands take a cluster-wide Postgres advisory lock before running certbot, so exactly one pod does the ACME work and the rest skip. The reconcile job needs no lock — it only touches pod-local state.

Stick with in-pod cron. The per-minute reconcile must run inside each pod (it writes that pod’s /etc/nginx/conf.d and reloads that pod’s nginx), so a Kubernetes-native CronJob cannot replace it. Renew/issue could technically run as K8s CronJobs (mounting the two RWX volumes), but the advisory lock already makes the in-pod versions race-free, so there’s nothing to gain.

Step 10 — End-to-End Verification Checklist

Perform these steps in order to verify the full flow. Keep ACME_STAGING=true until all steps pass, then flip to false.

10.1 — Infrastructure

  • LB2 is provisioned with TLS passthrough on 443→8443 and TCP 80→8080

  • custom.midsummer.cloud A record resolves to LB2’s public IP: dig custom.midsummer.cloud

  • Pod starts successfully; nginx -t passes (check logs: KUBECTL logs <pod> -c proxy)

  • Health check at http://<pod-ip>:8080/ping/ returns pong

  • Port 8443 is reachable from LB2: curl -k https://custom.midsummer.cloud:443/ returns 421 (snakeoil default)

10.2 — Add a Custom Domain (Staging)

  1. In the tenantui, go to Domains → Add Custom Domain

  2. Enter a test subdomain you control (e.g., test.yourdomain.com)

  3. Select mode “Event” and pick an event

  4. Click Add Domain — confirm the DNS instructions dialog appears

  5. In your DNS provider, create a CNAME: test.yourdomain.com → custom.midsummer.cloud

  6. Wait 1-5 minutes for DNS propagation

  7. Click Verify DNS — status should change to DNS Verified

10.3 — Issue SSL Certificate (Staging)

  1. Click Issue SSL — this enqueues the Celery task

  2. Watch the celery-certmgr logs: kubectl logs <pod> -c celery-certmgr — look for Certificate issued for test.yourdomain.com

  3. Or run manually: kubectl exec <pod> -c api -- python manage.py customdomain_issue --domain test.yourdomain.com

  4. Verify in tenantui: status changes to SSL Issued, cert expiry date populated

  5. Check the cert: kubectl exec <pod> -- certbot certificates --staging

  6. Check the nginx config was regenerated: kubectl exec <pod> -- cat /etc/nginx/conf.d/custom-domains.conf

  7. Verify HTTPS works: curl -vI https://test.yourdomain.com/ — should show the Let’s Encrypt staging cert and a 301→HTTPS or 200 response

10.4 — Switch to Production Let’s Encrypt

  1. Update the deployment environment: ACME_STAGING=false

  2. Redeploy / restart the pod

  3. Force-reissue the test domain: kubectl exec <pod> -c api -- python manage.py customdomain_issue --domain test.yourdomain.com

    • This will issue a production LE cert (the staging cert will be replaced)

  4. Verify: curl -vI https://test.yourdomain.com/ — should show a valid (non-staging) Let’s Encrypt certificate issued by R3

  5. Verify the tenant’s site loads correctly on the custom domain

10.5 — Renewal

  1. Test renewal: kubectl exec <pod> -c api -- python manage.py customdomain_renew

    • For staging: add --staging flag (or ensure ACME_STAGING=true)

  2. Verify the cron logs: kubectl logs <pod> -c custom-domain-cron — should show the daily renewal entry

  3. Check that the customdomain_renew management command runs without error

10.6 — Cleanup

  • Remove the test domain from tenantui: Delete → confirm the Domain routing row is also removed

  • Verify the domain returns 421 on port 8443 (or no longer resolves)

Troubleshooting

certbot fails with “Failed authorization procedure”

  • Confirm DNS CNAME is in place: dig test.yourdomain.com CNAME

  • Confirm port 80 is reachable from the internet to the pod: curl http://test.yourdomain.com/.well-known/acme-challenge/test

  • If using ACME_STAGING=true, the cert will be a fake “Fake LE Intermediate X1” — this is expected

  • Check certbot logs: kubectl exec <pod> -- cat /var/log/letsencrypt/letsencrypt.log

nginx -t fails after customdomain_regen_nginx

  • Check the generated config: kubectl exec <pod> -- cat /etc/nginx/conf.d/custom-domains.conf

  • Check for missing cert files: ls -la /etc/letsencrypt/live/<domain>/ inside the pod

  • If cert dirs are missing, the Domain was marked ssl_issued but the cert wasn’t actually issued — reset its status: python manage.py shell → CustomDomain.objects.filter(domain='<domain>').update(status='failed') and re-issue

Pod restart loses nginx config

  • Expected: /etc/nginx/conf.d is pod-local. entrypoint.sh regenerates the config from the DB at boot, and the per-minute reconcile cron covers the case where nginx wasn’t up yet during boot — a fresh pod serves custom domains within ~1 minute

  • To force it immediately: kubectl exec <pod> -- python manage.py customdomain_regen_nginx --force

A cert was issued/renewed but some pods still serve the old cert

  • Wait one reconcile tick (up to 60s), then re-check

  • Confirm the shared volume is actually RWX and mounted on that pod: kubectl exec <pod> -- ls /etc/letsencrypt/live/<domain>/

  • Check the reconcile log on the lagging pod: kubectl exec <pod> -- tail /var/log/cron-custom-domains.log

  • Check the recorded state: kubectl exec <pod> -- cat /etc/nginx/conf.d/.custom-domains.state — if it never updates, nginx reload is failing on that pod (see supervisord/proxy logs)

Custom domain shows 404 or “No Event matched”

  • Verify the Domain row exists: Domain.objects.filter(domain='<domain>')

  • Verify the CustomDomain row has status='ssl_issued'

  • Verify EventSetupMiddleware finds the event: check that CustomDomain.event is set (not null for mode='event')

  • Check request.current_custom_domain in a Django shell to confirm the middleware is routing correctly

Apex domain (e.g., furrycon.org) doesn’t work with CNAME

  • Some DNS providers don’t support CNAME at the apex. Use an ALIAS/ANAME record instead (Cloudflare, DNSimple, Route53 support this)

  • Alternative: advise tenants to use a subdomain (e.g., events.furrycon.org) which always supports CNAME

  • The UI already shows an “apex note” in the DNS instructions dialog

Rancher-Specific Notes

  • Workload → Deployments → midsummer → Environment Variables — add the custom domain vars here

  • Workload → Deployments → midsummer → Add Sidecar — add the celery-certmgr container with the same image + celery -A midsummer worker -Q certmgr command

  • Storage → PersistentVolumeClaims — create the midsummer-letsencrypt and midsummer-acme-webroot PVCs (1Gi each, RWX — nfs-rwx-storage or longhorn, see Step 3; NOT do-block-storage)

  • Apps → Charts — if you choose Longhorn for RWX (Step 3 Option B), install it from here

  • Service Discovery → Services — ensure the midsummer service exposes port 8443

  • Load Balancing → Load Balancers — create LB2 with the forwarding rules from Step 1

The Rancher UI makes it straightforward to add environment variables, volumes, and sidecars to an existing Deployment without hand-editing YAML.

Step 11 — How Multi-Pod Sync Works

Full design rationale: developer-docs/concepts/custom-domains.md (“Custom domains & certificate sync” on the developer docs site).

With multiple replicas there are three consistency problems, all handled by the combination of shared storage (Step 3) and the reconcile cron (Step 9) — no pod-to-pod communication, no kubectl, no RBAC:

  1. Cert material — certbot runs on whichever pod (or the certmgr Celery worker) wins the Postgres advisory lock, and writes to /etc/letsencrypt on the shared RWX volume. Every pod sees the same certs.

  2. ACME challenges — LB2 routes Let’s Encrypt’s HTTP-01 validation request to a random pod; the shared /var/www/letsencrypt webroot means any pod can serve any challenge token.

  3. Reload propagation — every pod’s per-minute reconcile compares a fingerprint (rendered config text + hash of each issued domain’s fullchain.pem) against what its own nginx last loaded, and rewrites + reloads only on change. A cert issued or renewed anywhere converges on all pods within ~60 seconds. The cert-file hashes are what let a sibling pod notice a renewal, which changes no config text.

The only multi-pod requirement on your side is the RWX storage from Step 3, mounted on the app Deployment and the certmgr Celery worker.

History: before 2026-07 this section described a kubectl exec fan-out (RELOAD_ALL_PODS, DEPLOYMENT_NAME, pod/exec RBAC, rolling restarts). That mechanism was broken and has been removed from the codebase — if your manifests still carry those env vars or the pod-exec-role RBAC, they can be deleted.