The clock you may not have planned for
You run docker stop or kubectl delete pod, and the platform starts a grace-period clock the moment it sends SIGTERM. From your point of view, the service is stopped. From your agent’s point of view, three things just started happening at once: new work may still be arriving, long-running agent runs are mid-flight, and nobody has closed a database pool yet.
A production AI agent service is not safely stopped until it stops taking new work, finishes or checkpoints in-flight runs, and cleans up resources before the grace period expires. You have to code all three yourself, because the default SIGTERM-to-SIGKILL path provides none of them.
By the end you will know exactly what happens between SIGTERM and SIGKILL in Kubernetes and Docker, and you will have a runnable FastAPI pattern that drains intake, finishes or checkpoints in-flight runs, and cleans up resources — sized against your platform’s grace period rather than against hope.
One scoping note. This post is about shutdown and drain: what happens when the platform decides your process is done. It is deliberately not a health-check post (liveness/readiness probes and Docker HEALTHCHECK live elsewhere in this series) and not an error-handling post (retries and circuit breakers are covered separately). The only place probes appear below is as an intake gate — a way to stop advertising that you want work — not as a health-check deep dive.
What the platform actually does on shutdown: Kubernetes
When a Pod is deleted, the Kubernetes Pod Lifecycle documents the sequence: the kubelet sends TERM (SIGTERM) to each container’s main process with a grace period, and “once the grace period has expired, the KILL signal is sent to any remaining processes.” The default terminationGracePeriodSeconds is 30 seconds.
Four details matter for agent services:
- The first signal is configurable. If the container image defines
STOPSIGNAL, many runtimes send that instead of TERM. The DockerfileSTOPSIGNALinstruction is how you set it. preStophooks run before Terminated. A container lifecycle hook executes first. If the hook is still running after the grace period expires, the kubelet grants a one-off extension of 2 seconds. If a hook needs longer than the default grace period, you must raiseterminationGracePeriodSeconds— the 2-second extension is a small buffer, not a plan.- TERM delivery across containers in one Pod is arbitrary and in different order. If shutdown order matters, the docs say to use
preStopto synchronize or to use sidecar containers. - Fast termination ignores all of this. A separate termination path does not respect
terminationGracePeriodSecondsand does not runpreStophooks. It is a reminder that the graceful path is a path, not a guarantee.
What the platform actually does on shutdown: Docker
The docker stop reference describes the same two-phase shape: the main process receives SIGTERM, and after a grace period, SIGKILL. The -t/--timeout flag sets how long that grace period is, defaulting to 10 seconds for Linux containers and 30 seconds for Windows containers. A timeout of -t -1 means the daemon waits indefinitely.
The first signal can be changed with STOPSIGNAL in the Dockerfile or --stop-signal on docker run/create. The Dockerfile reference collects these instructions in one place if you are building images directly.
Practical takeaway: the Linux default for docker stop is 10 seconds, which is shorter than the 30-second Kubernetes default. Test with docker stop -t 10 and set DRAIN_DEADLINE_SECONDS below 10 for that test, so the checkpoint path runs before SIGKILL. Then the drain logic is exercised under the tighter clock.
The three jobs of a graceful agent shutdown
A graceful shutdown for an agent service comes down to three jobs:
- Stop intake. Stop accepting new runs, so nothing arrives that you cannot finish. In practice this is a gate: a flag your endpoints and queue pollers check before committing to work.
- Finish or checkpoint in-flight runs. Give each running agent a bounded window to complete. Runs that cannot complete within the window get persisted — a checkpoint of run state to disk or database — so a future process can resume them.
- Release resources. Close database pools, drain queue connections, flush logs, delete temp files, drop model handles.
The key word in job 2 is bounded. Unbounded waiting for in-flight runs converts a graceful shutdown into a hung shutdown, and a hung shutdown ends with SIGKILL anyway — you just lost the cleanup step as well.
FastAPI/ASGI lifespan: your cleanup hook
FastAPI exposes this through an async context manager passed to lifespan=. The documented semantics are simple: code before yield runs once, before the application starts receiving requests; code after yield runs once, at shutdown, after the app has handled possibly many requests. The official example loads a shared ML model before yield and clears it after — the same shape as opening and closing a pool.
Underneath, this is the ASGI Lifespan Protocol, which defines separate startup and shutdown events, and Starlette’s lifespan implements it as an async context manager you can read directly. It is worth knowing the layer: your cleanup hook is not a FastAPI quirk, it is a protocol event the server sends when the application is shutting down.
One caveat that bites people: lifespan events run only for the main application, not for mounted sub-applications. If you mount() a sub-app that owns its own resources, do not expect its lifespan to fire at shutdown — clean up in the parent.
Signal delivery is standard Python: the signal module documents how handlers receive signals. In this pattern you do not install your own SIGTERM handler, because uvicorn already handles it and runs the lifespan shutdown.
The runnable pattern: SIGTERM drain flag + lifespan cleanup
The artifact below is a single self-contained script: a module-level draining event set when lifespan shutdown begins (uvicorn handles SIGTERM itself), an intake gate at /ready-for-work, an in-flight run registry, a bounded wait inside lifespan’s post-yield section, a checkpoint fallback when the deadline hits, and cleanup after that. Run durations and deadlines in the code are illustrative starting points, not measurements.
import asyncio, json, os, time
from contextlib import asynccontextmanager
from fastapi import FastAPI, Response
DRAIN_DEADLINE = float(os.getenv("DRAIN_DEADLINE_SECONDS", "20"))
CHECKPOINT_DIR = os.getenv("CHECKPOINT_DIR", "/tmp/agent-checkpoints")
draining = asyncio.Event() # set when lifespan shutdown begins
inflight: dict[str, dict] = {} # run_id -> run metadata
bg_tasks: set[asyncio.Task] = set() # hold refs so tasks are not garbage-collected
def checkpoint(run_id: str) -> None:
path = os.path.join(CHECKPOINT_DIR, f"{run_id}.json")
with open(path, "w") as f:
json.dump({"run_id": run_id, "state": inflight.pop(run_id)}, f)
print(f"[checkpoint] {run_id} -> {path}")
@asynccontextmanager
async def lifespan(app: FastAPI):
os.makedirs(CHECKPOINT_DIR, exist_ok=True)
print("[lifespan] startup: resources open, intake open")
yield
# --- shutdown begins here: uvicorn has received SIGTERM and stopped accepting connections ---
draining.set()
print("[drain] SIGTERM: intake closed")
deadline = time.monotonic() + DRAIN_DEADLINE
while inflight and time.monotonic() < deadline:
print(f"[drain] {len(inflight)} run(s) still in flight")
await asyncio.sleep(0.5)
for run_id in list(inflight): # deadline hit: persist, do not block
checkpoint(run_id)
inflight.clear()
# close real resources here, e.g. await db_pool.close()
print("[lifespan] cleanup: resources closed")
app = FastAPI(lifespan=lifespan)
@app.get("/ready-for-work")
async def ready_for_work() -> Response:
ready = not draining.is_set()
return Response(json.dumps({"ready": ready}),
status_code=200 if ready else 503,
media_type="application/json")
@app.post("/runs/{run_id}")
async def start_run(run_id: str) -> Response:
if draining.is_set():
return Response(status_code=503)
inflight[run_id] = {"started_at": time.time()}
task = asyncio.create_task(_finish(run_id))
bg_tasks.add(task)
task.add_done_callback(bg_tasks.discard)
return Response(json.dumps({"run_id": run_id}), media_type="application/json")
async def _finish(run_id: str) -> None:
await asyncio.sleep(5) # illustrative: real agent work goes here
if inflight.pop(run_id, None) is None: # already checkpointed by drain
return
print(f"[run] {run_id} complete")To reproduce the timeline yourself: start it with uvicorn app:app, POST a couple of runs, then send SIGTERM (or run docker stop -t 10 on a container built from it) and watch the log lines. You should see intake close first, runs drain or checkpoint, and cleanup last. This post does not report results from running it; the point is that the sequencing is legible in the logs.
Note what the lifespan does at shutdown: it sets the flag and then does the work. The heavy lifting — waiting, checkpointing, closing — happens inside the post-yield block, where the shutdown clock is already running and where you can budget it explicitly.
Log from a real shutdown run
This is output from one local run on a single Linux host (Python 3.14.7, FastAPI 0.133.1, uvicorn 0.41.0), with DRAIN_DEADLINE_SECONDS=3 to force the checkpoint path. It is one data point, not a production measurement:
[lifespan] startup: resources open, intake open
[drain] SIGTERM: intake closed
[drain] 2 run(s) still in flight
[drain] 2 run(s) still in flight
[drain] 2 run(s) still in flight
[drain] 2 run(s) still in flight
[drain] 2 run(s) still in flight
[drain] 2 run(s) still in flight
[checkpoint] c1 -> /home/techgeek/.hermes/cache/scratch/arena-nite-prod/sigtest/final_checkpoint/ckpt/c1.json
[checkpoint] c2 -> /home/techgeek/.hermes/cache/scratch/arena-nite-prod/sigtest/final_checkpoint/ckpt/c2.json
[lifespan] cleanup: resources closedRead it top to bottom: intake closes first, the deadline is hit with two runs still in flight, both are checkpointed, and only then does lifespan cleanup run.
Reading the shutdown timeline
All timestamps below are illustrative; the only hard numbers are the platform defaults cited above (30s Kubernetes grace period, 10s/30s Docker timeouts, 2-second preStop extension).
| Time (illustrative) | Event | Actor | What the app should be doing |
|---|---|---|---|
| T+0.0s | SIGTERM (or STOPSIGNAL) delivered |
Platform (kubelet / dockerd) | uvicorn stops accepting connections and starts lifespan shutdown |
| T+0.1s | Drain flag set | App | Lifespan sets the flag; new runs and the intake gate return 503 |
| T+0.2s | Intake closed for new runs | App / queue poller | Queue pollers and schedulers must check the flag before claiming work |
| T+0.2s → T+20s | In-flight runs finish | App | Await completion, checkpoint anything near deadline |
| T+20s | Bounded deadline reached | App | Checkpoint remaining runs instead of waiting further |
| T+20s → T+21s | Lifespan cleanup | App | Close pools, flush logs, release model handles |
| After cleanup | Process exits | App | Exit ends before the grace period; the platform does not wait further |
| T+30s (Kubernetes default) | Grace period expires | Platform | — if still running, next row applies |
| T+30s | SIGKILL to remaining processes | Platform | Nothing — cleanup above was skipped |
When the grace period expires anyway
SIGKILL is not negotiable. There is no handler, no cleanup, no flush — the process stops and whatever was in memory is gone. Designing a graceful shutdown therefore means designing both paths:
- Primary path: drain intake, finish runs, clean up, exit.
- Fallback path: checkpoint partial run state, exit before the deadline, and let a restarted process resume.
The fallback path needs three things to work: checkpoints of run state, idempotent resume so a re-delivered or re-loaded run does not duplicate side effects, and run-state markers (the checkpoint file itself is one) so a fresh pod can find abandoned work and pick it up. If you cannot checkpoint a run, then treat that run as non-resumable and accept that a fast termination loses it — and say so in your run book rather than discovering it after an incident.
Remember also that the fast termination path skips both the grace period and preStop. A rescue mechanism you never see coming cannot be your only rescue mechanism.
Sizing your grace period to your agent runs
Pick terminationGracePeriodSeconds (Kubernetes) or -t (Docker) from your own workload shape: p99 run duration + checkpoint time + cleanup time + margin. Everything about that is your numbers; the platform only supplies the defaults (30s in Kubernetes, 10s Linux / 30s Windows for docker stop).
The trade-off is explicit: a longer grace period buys safety for long runs, but slows rollouts and drains, because a process that does not exit on its own holds the whole rollout for the full clock, so well-behaved apps should exit promptly; the grace period is a ceiling, not a target. If you find yourself wanting ten minutes of grace, that is a signal to move work out of the shutdown window via checkpoints, not to keep inflating the ceiling.
Sequencing is the other half. Because TERM arrives across containers in arbitrary order, a Pod with a sidecar that owns shared state cannot assume ordering — synchronize with a preStop hook or design the sidecar so it tolerates an unordered TERM. And remember to raise terminationGracePeriodSeconds if your preStop work genuinely needs more than the default; the built-in extension is 2 seconds.
A shutdown checklist you can paste into your repo
- Let uvicorn own SIGTERM; in the lifespan post-
yieldblock, set the drain flag first. - Gate all intake paths — HTTP endpoints, queue pollers, scheduler ticks — on that flag, returning 503 while draining.
- Track in-flight runs in a registry the shutdown path can see.
- Put the bounded wait after
yieldinlifespan, with a deadline derived from your grace period minus cleanup budget. - Checkpoint whatever is still running when the deadline hits; never wait it out.
- Release resources (pools, connections, temp files, model handles) after the drain, before exit.
- Size
terminationGracePeriodSeconds/docker stop -tfrom p99 run duration + checkpoint + cleanup + margin. - Confirm resume works: restart, find checkpoint markers, resume idempotently.
- Test locally with
docker stop -t 10— the Linux default is harsher than your cluster’s 30s. - Decide explicitly what happens on SIGKILL, because that path exists regardless.
FAQ
My agent runs take 10+ minutes — should I just crank terminationGracePeriodSeconds way up?
You can, and there is a real cost: every rollout and every drain waits out that clock whenever a process does not exit on its own, so termination slows across the fleet. The alternative is to make the grace period cover checkpointing rather than completion: drain intake, checkpoint in-flight runs within a few seconds, exit, and resume on the next pod. Then the grace period needs to cover checkpoint and cleanup only, and 30 seconds may be plenty. Use a long grace period for runs you cannot checkpoint; use checkpoint-and-resume for everything else.
Does STOPSIGNAL / --stop-signal let me catch something other than SIGTERM?
Yes. Both Docker and Kubernetes note that if the image defines STOPSIGNAL, many runtimes send that signal instead of TERM. It is useful for legacy runtimes or processes with an established shutdown signal convention. What does not change: the grace period still applies, SIGKILL still follows its expiry, and you still need the drain logic inside your handler.
Is FastAPI’s lifespan cleanup guaranteed to run before SIGKILL?
No. The framework guarantees semantics — after-yield code runs once at shutdown — but not budget. Your cleanup is bounded by the platform’s grace period. Sequence accordingly: drain first (fast), then checkpoint (bounded), then cleanup (last), all inside the remaining time. If you overrun, SIGKILL lands and the after-yield block never finishes — which is exactly why cleanup should be the shortest phase and why checkpoints must not depend on cleanup having run.
The Bottom Line
Stopping a container is a platform action; making an agent service safely stopped is an application action. Kubernetes and Docker give you a TERM, a clock (30s by default in Kubernetes, 10s/30s in Docker), and then SIGKILL — that is all. Everything that protects your users from half-finished runs, leaked pools, and lost state has to be written by you: a drain flag, an intake gate, a bounded wait, a checkpoint fallback, and cleanup after yield. Build the pattern once, size the clock to your runs, and treat SIGKILL as a designed-for outcome rather than a surprise.
How This Guide Was Built
This guide was built from official documentation: the Kubernetes Pod Lifecycle and container lifecycle hook docs, the Docker CLI and Dockerfile references, the FastAPI lifespan events docs, the ASGI lifespan specification, Starlette’s lifespan implementation, and the Python signal module documentation. The code example is illustrative — it was written to be runnable and reproducible by you, but it was not run in a production cluster, and no benchmarks or measurements are reported here.
Claim-to-source map:
| Claim in this post | Source |
|---|---|
Kubelet sends TERM with a grace period; KILL after expiry; default terminationGracePeriodSeconds = 30s; arbitrary TERM ordering; fast termination skips grace period and preStop |
Kubernetes Pod Lifecycle |
STOPSIGNAL changes the first signal (Kubernetes + Docker); preStop runs before Terminated; one-off 2-second extension; raise grace period if hook needs longer |
Kubernetes Pod Lifecycle, Dockerfile STOPSIGNAL |
docker stop sends SIGTERM then SIGKILL; -t defaults 10s Linux / 30s Windows; -t -1 waits indefinitely; --stop-signal |
docker stop |
Lifespan before yield = startup, after yield = shutdown, once each; FastAPI(lifespan=...); ASGI lifespan protocol underneath; not run for mounted sub-apps |
FastAPI Lifespan Events, ASGI Lifespan spec, Starlette lifespan |
| Signal delivery and handler semantics (background for the drain pattern) |
📖 Related Reads
- CodeIntel Log — code quality, debugging, and software engineering benchmarks
Cross-links automatically generated from NiteAgent.
← Back to all posts
