Running Temporal Workers on Kubernetes Without Dropping Activities

Temporal workers look like stateless web services. They are not, and treating them like one is how activities get dropped.

A worker process holds in-flight activity executions, a sticky cache of workflow histories, and long-poll connections to the Temporal server. Kill it carelessly and the work it was carrying either stalls until a timeout or replays expensively on some other pod. None of this is visible in the tutorials, which mostly end at "hello world, workflow complete." This post is the setup I actually run on Kubernetes with ArgoCD: deployment topology, task queue routing, rollouts, schedules, and the failures that page you at 3 a.m. rather than the ones that show up in conference talks.

Deployment topology: one deployment per task queue

The task queue is the routing unit in Temporal, so the deployment should mirror it. I run one Kubernetes Deployment per task queue, and I split workflow workers from activity workers the moment their resource profiles diverge.

Workflow workers are memory-bound in a specific way. The sticky cache holds deserialized workflow state so replays stay cheap, and the whole performance model quietly depends on that cache being warm. Activity workers are a different animal: their profile depends on what the activity does. An activity that calls an LLM API is I/O-bound and wants high concurrency per pod. An activity that parses documents is CPU-bound and wants few. Put both behind one HorizontalPodAutoscaler and you have signed up for the worst scaling decision for each, simultaneously.

Rules I follow:

• One task queue per capability domain (billing, ingestion, notifications), not per workflow type. Per-type queues multiply your deployments and your failure modes.
• Never autoscale workflow workers on CPU. Scale them on workflow-task schedule-to-start latency. CPU on a workflow worker mostly reflects cache misses, and scaling on it creates the feedback loop I describe below.
• Activity workers scale on activity schedule-to-start latency. The SDKs expose worker task slot metrics. When schedule-to-start climbs while slots sit near zero, add replicas. When it climbs with slots free, your bottleneck is the server, the network, or a downstream dependency, and more pods will accomplish nothing.

Task queue routing

A single worker binary can listen on many queues. I still keep it to one queue per deployment.

The reason is drain behavior. When a deployment rolls, I want to reason about exactly one queue's in-flight work, not hold a mental model of five. Shared binaries also couple your blast radius: a bad activity deploy takes down workflow progress for every domain the process serves, and you get to explain that in the incident review.

Queue names are namespace-scoped, so encode the domain and nothing else. Environment separation belongs in namespaces and clusters, not in queue name suffixes that application code has to reconstruct at 2 a.m.

Rollouts that do not drop activities

Two mechanisms work together here. One handles the code, one handles the pod.

Worker versioning for code. Temporal's worker versioning pins each workflow execution to the build that started it. Set a build ID from your image tag, enable versioning, and default new executions to the current build. In-flight workflows keep executing on the old build until they complete, which is exactly what you want during a deploy. Skip this and old workflow histories replay against new code. The first change that reorders a command produces a non-determinism error, and the workflow halts at whatever hour your deploy happened to run.

Graceful drain for the pod. On SIGTERM, the SDK stops polling, finishes in-flight workflow tasks, and waits for in-flight activities. Your job is to make Kubernetes patient enough for that to matter:

• Set terminationGracePeriodSeconds above your longest normal activity duration.
• RollingUpdate with maxUnavailable: 0 and a surge, so the new ReplicaSet is healthy before the old one drains.
• A PodDisruptionBudget with maxUnavailable: 1 on workflow workers, so voluntary disruptions (node upgrades, rebalances) happen one at a time instead of all at once.

What about activities that legitimately run longer than any sane grace period? Set a heartbeat timeout and call heartbeat() with progress. If the pod dies anyway, the activity retries on another worker with the last heartbeat details attached, instead of looking dead until its start-to-close expires.

The rollout sequence in ArgoCD goes like this: the new ReplicaSet registers with the new build ID, new executions land there, pinned executions keep draining on the old workers, and the old version comes down when its pinned count reaches zero. Watch the pinned count, not the pod count. Deleting the old deployment early is the quiet way to replay a week of long-running workflows, and the replay does not send a warning first.

Schedules: let the server own them

Use Temporal Schedules instead of CronJobs. A CronJob that starts a workflow re-introduces the exact reliability problem Temporal exists to solve: if the pod's schedule window collides with a node drain, your run is gone, and Kubernetes will not apologize. A Temporal Schedule lives on the server, survives every worker outage, and gives you overlap policies, catchup windows, pause, and backfill for free.

Two settings matter more than the rest.

• Overlap policy. SKIP for reports and rollups, where a delayed run makes the previous one irrelevant. BUFFER_ONE when runs must not be lost but must never overlap. ALLOW_ALL is almost never right for batch work.
• Catchup window. Cap it. After a long namespace outage, the default behavior will happily fire every missed run at once. That is a self-inflicted load spike against your own downstreams, timed for the exact moment you are recovering.

Manage schedules as code. A small idempotent bootstrap works fine: a Terraform resource, or a short program run from an ArgoCD PostSync hook, creating or updating schedule definitions from the repo. Schedules created by hand in the UI are configuration drift with a cron expression.

What actually breaks in prod

1. Non-determinism after a deploy. Code changed without versioning. This is the number one Temporal incident and it is fully preventable with pinned builds.
2. Grace period shorter than the longest activity. The pod dies mid-activity, and without a heartbeat timeout the task looks stuck until start-to-close expires. Set heartbeat timeouts to tens of seconds for anything long.
3. Sticky cache thrash. Autoscale workflow workers on CPU and watch what happens: new pods start with cold caches, replays spike CPU, the autoscaler adds more pods, those pods also start cold. A positive feedback loop that masquerades as a traffic surge. Fix it by scaling on schedule-to-start and capping the HPA.
4. Poller overload against the server. Too many pollers per pod against a namespace RPS budget produces rising schedule-to-start with idle capacity everywhere. The fix is fewer pollers, not more pods. Counterintuitive, and it cost me an afternoon the first time.
5. A database connection per activity. Bursty activity workers exhaust the pool. One pooled client per worker process, sized for max concurrent activities.
6. Schedule catchup storms after downtime, covered above.
7. ArgoCD ordering. Self-hosted Temporal server upgrades and schema migrations must land before workers roll. Use sync waves: migrations first, server second, workers last. And pin images by tag or digest, never latest. ArgoCD correctly sees no diff when latest moves, and your GitOps history becomes fiction.

The short version

One deployment per task queue. Pinned worker versioning on every build. Grace periods that respect your slowest activity, heartbeats on anything long, schedules on the server, and autoscaling driven by schedule-to-start. Six settings. The difference between Temporal as infrastructure and Temporal as a recurring incident.