Skip to content
All posts
temporaldurable-executionai-agentsplatform-engineering

Temporal Best Practices: Durable Execution, Agent Loops, and the Antipatterns That Bite

A practitioner's reference for Temporal in production: the current client landscape, the three patterns worth memorizing with code in Python, Go, Java, and Rust, how agentic workloads map onto durable execution, and the eleven antipatterns that actually cause incidents.

Most Temporal content stops where production starts. The tutorials get you a workflow that prints a greeting. They do not prepare you for the Tuesday where a long-running workflow halts because someone reordered two lines of code, or for the agent loop that quietly burned four figures of tokens because nobody set a retry budget.

I have run Temporal across several platforms now, including the AI workload cases everyone is suddenly discovering, and the practices that matter are stable across all of them. This post is the reference I wish existed when I started: the current client landscape, the three patterns I reach for weekly, how agentic workloads map onto durable execution, and the antipatterns that cause real incidents. Everything below is checked against the SDK versions and documentation current as of August 2026, with sources at the end.

Client landscape, August 2026

Versions move fast enough that any undated advice should be treated with suspicion. Here is what I verified while writing this:

  • Python SDK 1.31.0, per PyPI. The most batteries-included of the SDKs, and the one Temporal's own agentic tutorials use.
  • Go SDK v1.47.0, per the Go SDK developer guide. Recent releases added poller autoscaling enrollment and graduated user metadata fields out of experimental.
  • Java SDK 1.38.0, per Maven Central. The 1.38 line added Spring Boot 4 support and local enforcement of activity heartbeat timeouts.
  • Rust SDK 0.6.0, published days ago on crates.io. This is the real news. The Rust SDK left the old sdk-core-only world and is now a proper high-level SDK in public preview, with typed workflows, signals, queries, child workflows, and schedules, per the API docs. Public preview means the API still breaks between minor versions; the release notes say so in all caps. Usable, promising, not yet boring.

The examples below target these versions. Where the Rust API is still settling, I say so in the code.

The one mental model that matters

Everything in Temporal follows from one mechanism: replay. A workflow execution is rebuilt by re-running your workflow code from the top, feeding it the recorded event history instead of re-doing the work. The code must make the same decisions given the same history. That single constraint explains nearly every rule in this post.

Two consequences fall out. First, workflow code is not application code. It is a deterministic state machine interpreter, and anything that touches the outside world (network, disk, wall clock, randomness) belongs in an activity, whose result gets recorded once and reused on replay. Second, the event history is both your superpower and your budget. It is capped at 51,200 events or 50 MB, with warnings starting at 10,240 events or 10 MB, per the workflow execution limits. Every activity, timer, signal, and child workflow spends from that budget.

Hold those two ideas and the rest of this post is commentary.

Pattern: an activity with a retry policy that respects the failure

Activities retry by default with a 1 second initial interval, 2.0 backoff, a 100 second cap, and unlimited attempts, per the retry policy reference. Unlimited is the part to question. For an LLM endpoint, unlimited retries on a 400-class error is money burned politely. Set a budget, and mark permanent failures as non-retryable.

Python:

@workflow.defn
class AgentWorkflow:
    @workflow.run
    async def run(self, prompt: str) -> str:
        return await workflow.execute_activity(
            call_llm,
            prompt,
            start_to_close_timeout=timedelta(minutes=2),
            retry_policy=RetryPolicy(
                initial_interval=timedelta(seconds=1),
                backoff_coefficient=2.0,
                maximum_interval=timedelta(minutes=1),
                maximum_attempts=5,
                non_retryable_error_types=["InvalidRequest"],
            ),
        )

Go:

func AgentWorkflow(ctx workflow.Context, prompt string) (string, error) {
	ao := workflow.ActivityOptions{
		StartToCloseTimeout: 2 * time.Minute,
		RetryPolicy: &temporal.RetryPolicy{
			InitialInterval:        time.Second,
			BackoffCoefficient:     2.0,
			MaximumInterval:        time.Minute,
			MaximumAttempts:        5,
			NonRetryableErrorTypes: []string{"InvalidRequest"},
		},
	}
	ctx = workflow.WithActivityOptions(ctx, ao)
	var answer string
	err := workflow.ExecuteActivity(ctx, CallLLM, prompt).Get(ctx, &answer)
	return answer, err
}

Java:

public class AgentWorkflowImpl implements AgentWorkflow {
	private final LlmActivities llm = Workflow.newActivityStub(
		LlmActivities.class,
		ActivityOptions.newBuilder()
			.setStartToCloseTimeout(Duration.ofMinutes(2))
			.setRetryOptions(RetryOptions.newBuilder()
				.setInitialInterval(Duration.ofSeconds(1))
				.setBackoffCoefficient(2.0)
				.setMaximumInterval(Duration.ofMinutes(1))
				.setMaximumAttempts(5)
				.setDoNotRetry("InvalidRequest")
				.build())
			.build());

	@Override
	public String run(String prompt) {
		return llm.call(prompt);
	}
}

Rust, against the 0.6.0 preview API. Activity options are a builder, and retry config is a Rust-native RetryPolicy:

#[workflow]
struct AgentWorkflow;

#[workflow_methods]
impl AgentWorkflow {
	#[run]
	async fn run(ctx: &mut WorkflowContext<Self>, prompt: String) -> WorkflowResult<String> {
		let opts = ActivityOptions::with_start_to_close_timeout(Duration::from_secs(120))
			.retry_policy(RetryPolicy {
				maximum_attempts: 5,
				..Default::default()
			})
			.build();
		ctx.execute_activity(AgentActivities::call_llm, prompt, opts).await
	}
}

Note what is consistent across all four: a start-to-close timeout that bounds one attempt, a retry budget that bounds the total, and a distinction between failures worth retrying and failures that are just expensive to repeat. That shape is the pattern; the syntax is decoration.

Pattern: human-in-the-loop with signals

Agents that can spend money, send email, or merge code need a pause button that survives restarts. Temporal's answer is a signal: an asynchronous write into a running workflow, recorded in history, per the message passing reference. The workflow blocks on a condition, not on a thread, so a three-day approval wait costs nothing but one timer event.

Python:

@workflow.defn
class ReviewWorkflow:
    def __init__(self) -> None:
        self._decision: str | None = None

    @workflow.signal
    def approve(self, approved: bool) -> None:
        self._decision = "approved" if approved else "rejected"

    @workflow.query
    def status(self) -> str:
        return self._decision or "pending"

    @workflow.run
    async def run(self, draft: str) -> str:
        try:
            await workflow.wait_condition(
                lambda: self._decision is not None,
                timeout=timedelta(hours=24),
            )
        except asyncio.TimeoutError:
            return "escalated"
        return self._decision

Go:

func ReviewWorkflow(ctx workflow.Context, draft string) (string, error) {
	var decision string
	sig := workflow.GetSignalChannel(ctx, "approve")
	selector := workflow.NewSelector(ctx)
	selector.AddReceive(sig, func(c workflow.ReceiveChannel, more bool) {
		var approved bool
		c.Receive(ctx, &approved)
		if approved {
			decision = "approved"
		} else {
			decision = "rejected"
		}
	})
	selector.AddFuture(workflow.NewTimer(ctx, 24*time.Hour), func(f workflow.Future) {
		if decision == "" {
			decision = "escalated"
		}
	})
	for decision == "" {
		selector.Select(ctx)
	}
	return decision, nil
}

Java:

public class ReviewWorkflowImpl implements ReviewWorkflow {
	private String decision;

	@Override
	public String run(String draft) {
		Workflow.await(Duration.ofHours(24), () -> decision != null);
		return decision == null ? "escalated" : decision;
	}

	@Override
	public void approve(boolean approved) {
		decision = approved ? "approved" : "rejected";
	}

	@Override
	public String status() {
		return decision == null ? "pending" : decision;
	}
}

Rust. Signal handlers are synchronous and receive a SyncWorkflowContext; the macro attributes are #[signal], #[query], and #[run], per the macros reference. Exact signatures are still moving pre-1.0, so treat this as the shape rather than gospel:

#[workflow]
struct ReviewWorkflow {
	decision: Option<String>,
}

#[workflow_methods]
impl ReviewWorkflow {
	#[run]
	async fn run(ctx: &mut WorkflowContext<Self>, _draft: String) -> WorkflowResult<String> {
		ctx.wait_condition(|wf: &Self| wf.decision.is_some()).await?;
		Ok(ctx.state(|wf: &Self| wf.decision.clone()).unwrap())
	}

	#[signal]
	fn approve(&mut self, _ctx: &SyncWorkflowContext<Self>, approved: bool) {
		self.decision = Some(if approved { "approved".into() } else { "rejected".into() });
	}
}

One judgment call worth stating: use a signal when fire-and-forget is fine, and an update when the caller needs a validated, synchronous answer. Approval buttons in UIs are usually updates, because the user wants to know the click landed. Signals are for webhooks and external systems that will not wait.

Pattern: fan-out with child workflows

A single workflow should not spawn 100,000 activities; its history cannot hold them. The standard decomposition is a parent that fans out to children, each of which owns its own history. The child workflow guidance suggests keeping a parent under roughly 1,000 children and reaching for activities first when in doubt. Fan-out across documents, repos, or tenants is where children earn their overhead, because each child gets its own retry boundary, its own identity, and its own place in the UI.

Python:

@workflow.defn
class IndexRepoWorkflow:
    @workflow.run
    async def run(self, documents: list[str]) -> list[str]:
        handles = [
            await workflow.start_child_workflow(
                IndexDocumentWorkflow.run,
                doc,
                id=f"index-doc-{i}",
            )
            for i, doc in enumerate(documents)
        ]
        return await asyncio.gather(*handles)

Go:

func IndexRepoWorkflow(ctx workflow.Context, docs []string) ([]string, error) {
	selector := workflow.NewSelector(ctx)
	results := make([]string, len(docs))
	for i, doc := range docs {
		i, doc := i, doc
		cwo := workflow.ChildWorkflowOptions{WorkflowID: fmt.Sprintf("index-doc-%d", i)}
		ctx = workflow.WithChildOptions(ctx, cwo)
		future := workflow.ExecuteChildWorkflow(ctx, IndexDocumentWorkflow, doc)
		selector.AddFuture(future, func(f workflow.Future) {
			_ = f.Get(ctx, &results[i])
		})
	}
	for range docs {
		selector.Select(ctx)
	}
	return results, nil
}

Java, using Async.function to start children concurrently:

public class IndexRepoWorkflowImpl implements IndexRepoWorkflow {
	@Override
	public List<String> run(List<String> docs) {
		List<Promise<String>> pending = new ArrayList<>();
		for (int i = 0; i < docs.size(); i++) {
			String doc = docs.get(i);
			IndexDocumentWorkflow child = Workflow.newChildWorkflowStub(
				IndexDocumentWorkflow.class,
				ChildWorkflowOptions.newBuilder()
					.setWorkflowId("index-doc-" + i)
					.build());
			pending.add(Async.function(child::run, doc));
		}
		return pending.stream().map(Promise::get).collect(Collectors.toList());
	}
}

Rust, using the typed child workflow API with the deterministic join_all the SDK provides for workflow code:

#[run]
async fn run(ctx: &mut WorkflowContext<Self>, docs: Vec<String>) -> WorkflowResult<Vec<String>> {
	let mut started = Vec::new();
	for (i, doc) in docs.into_iter().enumerate() {
		let opts = ChildWorkflowOptions::builder()
			.workflow_id(format!("index-doc-{i}"))
			.build();
		started.push(ctx.start_child_workflow(IndexDocumentWorkflow::run, doc, opts).await?);
	}
	let results = temporalio_sdk::workflows::join_all(
		started.into_iter().map(|child| child.result()),
	)
	.await;
	results.into_iter().collect()
}

Two rules I enforce on fan-out. Give every child a deterministic, human-readable workflow ID derived from the unit of work, so restarts dedupe naturally and the UI stays navigable. And do not collect results by passing megabytes back through the parent; have children write to storage and return references, for reasons the next section makes concrete.

Agentic workloads are just distributed systems

The AI industry spent two years rediscovering that an agent loop is a long-running distributed process with flaky dependencies. Temporal's own durable AI agent tutorial makes the mapping explicit, and it is the mapping I use:

  • The agent loop is a workflow. Conversation state, tool call history, and budget counters live in workflow state, rebuilt by replay after any crash. A loop that runs for six hours and survives three deploys is a workflow, not a script with a retry wrapper.
  • Tool calls and LLM calls are activities. They are non-deterministic, they fail, and they cost money. Activities give each call its own retry policy, timeout, and recorded result. If the worker dies between the model call and the next step, the result is replayed from history instead of billed twice.
  • Human approval is a signal or update, as above. This is the difference between an agent that pauses gracefully for three days and one that holds a connection open and dies in a pod reschedule.
  • Retry of flaky model endpoints is a retry policy problem, not a while True problem. Rate limits retry with backoff; invalid requests and content-policy rejections are marked non-retryable. For long generation calls, set a heartbeat timeout so a dead worker is detected in seconds rather than at start-to-close expiry, per the activity failure detection docs.

Two practices deserve their own paragraphs because I have seen both ignored at cost.

Idempotency is your job, not Temporal's. Temporal guarantees an activity is retried until it succeeds, which means it can execute more than once in the presence of timeouts. Any activity that charges a card, sends a message, or writes a row needs an idempotency key derived from the workflow ID plus activity ID, and the downstream must honor it. The retry semantics give you at-least-once; you build exactly-once on top.

Token budgets belong in workflow state. Track cumulative tokens as a workflow field, check the budget before each model activity, and use cancellation to stop the loop when the budget or the user says stop. Cancellation propagates to in-flight activities, and because it flows through history, the partial state is exactly where you left it when you resume or continue-as-new. A budget enforced in the workflow survives restarts; a budget enforced in a wrapper process dies with the pod.

Schedules round out the picture for recurring agent work: nightly summarization, periodic re-indexing, heartbeat checks against external systems. Server-side schedules with an explicit overlap policy beat cron-in-a-pod for the same reason everything else here does: the schedule survives the outage that killed your workers.

Antipatterns, and the fix for each

These are ordered roughly by how often I have seen them cause an incident.

Non-deterministic workflow code. What it looks like: datetime.now(), random.random(), uuid4(), or iterating a set inside workflow code. Why it bites: replay produces different commands than the recorded history, the workflow task fails with a non-determinism error, and the workflow halts until you fix the code. The fix: use the SDK's deterministic substitutes (workflow.now(), workflow.uuid4()), keep ordering stable, and run replay tests in CI against captured histories before any workflow change ships.

I/O in workflow code. What it looks like: an HTTP call or a database read inline in the workflow, often "just a quick config fetch." Why it bites: the call re-executes on every replay, returns different data, and you are back to non-determinism; or it blocks the workflow task and stalls everything on that worker. The fix: move it to an activity. There are no exceptions small enough to be worth it.

Unbounded histories. What it looks like: an entity workflow that accumulates an event per operation forever, or a polling loop that sleeps and repeats inside one run. Why it bites: histories warn at 10,240 events and hard-stop at 51,200 events or 50 MB, and large histories also make replays slow and sticky-cache eviction expensive. The fix: continue-as-new at a sensible interval, checking is_continue_as_new_suggested(), and move periodic work into child workflows or schedules so each run stays small.

Giant payloads through history. What it looks like: passing a document, a diff, or a conversation transcript as an activity result. Why it bites: individual payloads are limited to 2 MB and each gRPC message to 4 MB; oversize inputs can terminate the workflow outright, per the payload size troubleshooting guide. The fix: the claim check pattern. Store the bytes in object storage, pass the reference, and consider the SDKs' external storage support once you are doing this routinely. This applies double for agent loops, where conversation history grows monotonically.

Heartbeat misuse. Two opposite sins. The first: long activities with no heartbeat timeout, so a dead worker looks alive until start-to-close expires, which might be hours. The second: heartbeating on a tight interval with no payload, treating heartbeats as keepalive noise. Why it bites: heartbeats exist to detect worker death fast and to carry progress details so a retry resumes mid-work; spamming them just writes RPC traffic. The fix: heartbeat with real progress details (offset, count, checkpoint) on any activity longer than a minute, and set the heartbeat timeout to tens of seconds.

Over-granular workflows. What it looks like: a workflow per row of a CSV import, tens of thousands of executions for a job that is one batch. Why it bites: every workflow execution is persisted state, visibility entries, and history events; child workflows cost more events than activities for the same unit of work. The fix: the official advice is blunt, "when in doubt, use an activity." Batch rows into activities, and reserve workflows for units that need their own identity, durability boundary, or message handlers.

Tight signal loops. What it looks like: streaming progress into a workflow via a signal per item, thousands per hour. Why it bites: every signal is a history event, and pending signals are capped at 2,000 per execution by default. A chatty producer blows the history budget while accomplishing nothing a query could not. The fix: batch updates, have producers write progress to storage and signal only state transitions, or use an update when the caller needs an acknowledgment.

Versioning mistakes. What it looks like: reordering activities or changing a branch in workflow code, deploying, and watching in-flight workflows fail on replay. Why it bites: old histories must replay against new code, and any behavioral divergence is a non-determinism error. The fix: worker versioning with pinned builds for anything long-running, and the patching protocol (patched(), then deprecate_patch(), then removal) for the gap cases. Never remove a patch marker while any history that recorded it is still within retention.

Polling activities where a signal belongs. What it looks like: an activity that loops calling an external status endpoint every five seconds, for days. Why it bites: you pay for a worker slot and RPC traffic around the clock to learn something the external system could have told you once. The fix: invert it. Have the external system call back and signal the workflow when the state changes, or complete the activity asynchronously via its task token. Polling is a fallback for systems that genuinely cannot call you.

Retry policy footguns. What it looks like: default unlimited retries against an endpoint returning permanent 422s; a start-to-close timeout shorter than the work, so every attempt times out and retries forever; a retry policy on the workflow itself. Why it bites: unlimited retries turn a bad request into a meter running; workflow-level retries replay the same deterministic failure, which the retry policy docs explicitly recommend against. The fix: cap attempts or schedule-to-close, mark known-permanent errors non-retryable at the throw site, size start-to-close for the p99 of real work, and let workflows fail so a human looks at them.

Saga without compensation discipline. What it looks like: a multi-step provisioning workflow with compensations registered after each step, or compensations that assume the forward step fully ran. Why it bites: an activity can fail after its side effect landed (timeout after commit), so the compensation runs against partial state; register compensations before the step and write them defensively, per Temporal's saga writeup, which is really Garcia-Molina and Salem's 1987 saga paper with better ergonomics. The fix: compensations first, idempotent and conditional ("put the bowl away if it is out"), and shield compensation runs from cancellation so cleanup finishes even when the workflow is being torn down.

The short version

Durable execution is not complicated, but it is unforgiving about two things: determinism and history size. Keep workflow code pure, keep histories small, put every outside call in an activity with an explicit retry budget, use signals for humans and updates for clicks, fan out with named children, and treat agent loops as the distributed systems they are. Do those and Temporal fades into the background, which is the highest compliment infrastructure can earn.

The operational half of this story, how to deploy the workers themselves without dropping in-flight activities, is in my earlier post on running Temporal workers on Kubernetes.

Sources