Graceful Shutdown

When to use

Every production GoFr deployment on Kubernetes should be configured for graceful shutdown. Without it, rolling updates and node drains return 502/504s for any request that is mid-flight when a pod is terminated, and Pub/Sub consumers can lose un-committed messages.

How GoFr handles signals

App.Run sets up a signal-aware context:

ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGINT, syscall.SIGTERM)

When that context is canceled, a goroutine creates a timeout context using SHUTDOWN_GRACE_PERIOD (default 30s) and calls App.Shutdown. The order is fixed by the framework — see pkg/gofr/gofr.go — and Shutdown joins errors from each step:

1. httpServer.Shutdown(ctx) — stops accepting new connections, waits for in-flight handlers
2. grpcServer.Shutdown(ctx) — drains active streams
3. Cron stop — halts the scheduler and waits for jobs already running, bounded by the shutdown deadline
4. metricServer.Shutdown(ctx) — stops /metrics
5. mcpServer.Shutdown(ctx) — stops the MCP server
6. container.ShutdownMetrics(ctx) and container.Close() — closes SQL pools, Redis clients, Pub/Sub consumers, and other registered datasources
7. Logger close — if the logger implements io.Closer, its Close() is called last

The container's Close is what commits Pub/Sub offsets and lets SQL drivers finish in-progress queries. It runs second-to-last on purpose: every producer of datasource traffic — handlers, streams, cron jobs — is drained before the connections they use are torn down. Application code does not need to coordinate this order.

OnStart hooks vs shutdown hooks

GoFr exposes OnStart hooks for synchronous startup work (cache warmup, seeding). There is no public OnShutdown hook today; App.Shutdown is what gets called and it operates on the framework's own resources. If you need cleanup on exit for resources you own (custom goroutines, file handles, third-party clients), use context-cancellation: pass a context.Context derived from signal.NotifyContext(...) into your goroutines and have each goroutine defer its own cleanup when that context is cancelled. The framework's App.Shutdown runs concurrently with this, so total wind-down stays within SHUTDOWN_GRACE_PERIOD.

The Kubernetes termination flow

When kubelet decides to evict a pod, it executes this sequence:

1. Pod's status flips to Terminating; endpoints controllers begin removing the pod from Service Endpoints.
2. preStop hook runs (if configured).
3. SIGTERM is sent to PID 1.
4. After terminationGracePeriodSeconds (default 30s), SIGKILL is sent.

Steps 1 and 3 race: kube-proxy on every node needs time to update iptables/IPVS rules. A pod can still receive new traffic for a second or two after SIGTERM. The fix is a preStop sleep that delays shutdown long enough for endpoint removal to propagate.

spec:
  terminationGracePeriodSeconds: 60
  containers:
    - name: api
      image: ghcr.io/example/orders-api:1.4.2
      lifecycle:
        preStop:
          exec:
            command: ["/bin/sh", "-c", "sleep 5"]
      env:
        - name: SHUTDOWN_GRACE_PERIOD
          value: "45s"
      readinessProbe:
        httpGet:
          path: /.well-known/health
          port: 8000
      livenessProbe:
        httpGet:
          path: /.well-known/alive
          port: 8000

Sizing the grace period

Set the values so preStop + SHUTDOWN_GRACE_PERIOD is comfortably less than terminationGracePeriodSeconds. A useful starting point:

• preStop: 5s (covers endpoint propagation on most clusters)
• SHUTDOWN_GRACE_PERIOD: P99 request latency × 2, plus headroom for Pub/Sub commits
• terminationGracePeriodSeconds: preStop + SHUTDOWN_GRACE_PERIOD + 10s buffer

For a service with 2s P99, that's 5s + 30s + 10s = 45–60s.

Per-datasource behavior

• SQL. database/sql waits for active queries to finish on Close(). Long-running transactions can extend shutdown — keep request timeouts shorter than SHUTDOWN_GRACE_PERIOD.
• Redis / NoSQL. Clients close idle connections immediately and wait for in-flight commands.
• Pub/Sub. GoFr's subscription manager respects the shutdown context — consumers stop polling and commit current offsets where the broker supports it (Kafka, NATS JetStream).
• Cron jobs. App.Shutdown stops the cron scheduler and waits for jobs that are already running before it closes datasource connections, so a job mid-write is not cut off by container.Close(). The wait is bounded by the shutdown deadline (SHUTDOWN_GRACE_PERIOD): if a job is still running when that expires, Shutdown returns and the job is abandoned. The jobs themselves run with context.Background(), so a job's own ctx is never cancelled — anything that needs to be interruptible has to watch something else. For cron work that routinely outlasts SHUTDOWN_GRACE_PERIOD, run it as a Kubernetes CronJob in its own pod rather than inside the service.

Verification

Trigger a rolling restart and watch the logs:

kubectl rollout restart deployment/orders-api -n prod
kubectl logs -f -l app=orders-api -n prod --previous

You should see Shutting down server with a timeout of 30s followed by Application shutdown complete on each terminating pod, with no connection reset errors on the client side. From a load-test client running during the restart, error rate should stay below 0.1%.