In this piece
Google AX: Suspend/Resume, Restart Costs and Self-Hosting
An agent spends $30. Its process disappears. You pay again for work it already did. That is a recovery problem. Paying for agents that spend most of their time waiting is an idle-capacity problem. Google AX is relevant to both, but the same headline does not solve both.
AX, also called Agent Executor, is Google’s Apache-2.0-licensed orchestrator for sandboxed agent tasks. It runs above Agent Substrate and uses Kubernetes-like YAML and commands. The useful idea is to manage agent execution as infrastructure, not to invent another reasoning framework. Read the pinned AX project overview and development warning.
Reviewed on : AX 56bd0fc55574 and Agent Substrate v1.0.0. This is a source and architecture review, not a deployment test or throughput benchmark. The $30 story is an illustrative scenario, not an AX result we measured.
What does Google AX orchestrate now?
The current API has three core resources: Task, Workspace and Model. Older four-resource explanations include a Gateway that is no longer part of this reviewed AX schema. A resource defines configuration; it does not establish every operational guarantee around it.
| Resource | What it declares | What it does not prove |
|---|---|---|
| Task | Image, command, environment, CPU and memory, workspace bindings and debugging. | A job-wide spending cap or successful business outcome. |
| Workspace | Repositories, files, MCP tools and skills prepared for a task. | Automatic persistence of every agent session or external operation. |
| Model | Provider, model identifier, generation settings and a credential reference. | Local model weights or automatic integration with every agent SDK. |
A task can represent a whole agent job or one delegated operation. The development schema binds at most one workspace through spec.workspace. Its path becomes the command’s working directory. AX’s current resource and lifecycle concepts.
What changed since the September AX examples?
Do not deploy an old four-resource manifest unchanged. This refresh replaces the previous Gateway example and updates the execution contract. Pin the CLI, control plane and runner together before migrating configuration.
| Area | Current contract | Action |
|---|---|---|
| Gateway | No AX Gateway resource or Task gateway binding in the current schema. | Configure and test the actual substrate/network policy layer. |
| Workspaces | One optional spec.workspace binding with a name and mount path. | Create the workspace first in the same atespace and verify where the task writes session files. |
| Task changes | Tasks are created as immutable resources. | Use a new task name for changed execution settings; preserve state before replacing a task. |
| Control plane | Direct reconciliation with Substrate and PostgreSQL-backed state and leases. | Check the schema, storage and compatible versions before moving from the released stack. |
| Metadata | Loopback metadata listener, by default 127.0.0.1:8099. | Read AX_METADATA_URL instead of assuming metadata is exposed on port 80. |
Read the Task schema for workspace bindings, execution options and the lifecycle status fields. Inspect the current AX API schema.
AX v0.3.1 remains the latest tagged release. It uses Redis and a separate controller. This November development revision uses PostgreSQL and pins Substrate v1.0.0. Match the CLI, server and runner to the same AX ref. For an upgrade, validate restoration and rollback separately before moving existing tasks. Read the AX installation and version notes.
Is AX really Kubernetes for AI agents?
It is a useful analogy, not a Kubernetes replacement. Your harness manages reasoning and tool decisions. AX manages task declarations and lifecycle. Agent Substrate supplies sandbox execution, while Kubernetes manages the underlying worker infrastructure.
The current AX server reconciles directly with Substrate under distributed locks and stores resource state in PostgreSQL. It does not create a Kubernetes custom resource for every short-lived task. Its design also explicitly describes tasks as immutable. Read the updated AX control-plane design.
The November server writes JSON logs and serves Prometheus metrics by default in the bundled deployment. OTLP export requires configuration. Verify that a disposable task’s lifecycle event reaches your logging system, and keep separate application records for completed external actions. AX observability documentation.
The README’s millions-of-tasks language is a design target. It does not demonstrate millions of simultaneously active agents in your cluster. Size for active concurrency, state size, model latency and bursts. For model loops, tool handling and feedback, use our agent harness engineering guide.
What survives AX suspend and resume?
For the documented AX runner, workspace files survive; the command returns in a fresh process. Stop and suspend signal PID 1 with SIGTERM. The runner forwards shutdown to the command so it can flush state. This orderly path is not the same event as a machine disappearing mid-call. The pinned runner contract describes the persistence and shutdown boundary.
| State | Documented boundary or application responsibility |
|---|---|
| Files under /workspace | The runner contract describes snapshot and restore into a fresh container. |
| In-process variables | The AX contract describes a new process tree. Do not depend on its RAM surviving. |
| Conversation and tool results | Your harness must persist them in a restorable format and load them on startup. |
| An email, payment or database write | An external side effect is not undone by restoring local files. Reconcile the operation before retrying it. |
| Command success | The runner can stay alive after the command exits; the documented control plane does not read back its exit status. |
Make retries idempotent: give an operation a durable identity so repeating the request cannot silently repeat its effect. Also record an explicit success or failure result. Ready, WorkspaceReady and a healthy runner are not substitutes for an accepted deliverable.
Crawler recovery also depends on which extracted records survive alongside the request queue. Our Scrapling checkpoint and output guide follows those stores through a pause and resume.
There is an important layer distinction. Agent Substrate describes full-state snapshots that preserve RAM and filesystem state, and demonstrates approximately 250 actors multiplexed across eight pods. That does not override AX’s specific runner contract. Test the exact runner and backend pair you will deploy. Substrate’s snapshot capabilities, demonstration and deployment documentation.
Does sub-second resume mean zero cold start and zero idle cost?
No. Fast activation is a latency claim, not an end-to-end billing guarantee. Substrate’s README reports resumes below 500 ms and more than 500 suspend/resume activations per second. These are project-reported capabilities, not a Wavect benchmark or a latency promise for every AX workload.
A first launch, unavailable worker, large snapshot, dependency installation or slow model request can still delay useful work. Measure both resume-to-ready and resume-to-first-useful-result, including p95 latency, rather than treating infrastructure activation as the whole user experience.
Releasing CPU and RAM from one suspended actor can improve fleet utilization. It does not make worker pools, PostgreSQL, snapshot storage, networking or operations free. An already admitted model request may still be billable while the local agent waits. Infrastructure idle cost and token cost are different lines in the ledger.
Also distinguish explicit lifecycle commands from automation. AX documents ax suspend and ax resume; automatic idle detection and suspension still appear on its roadmap, alongside token/timeout budgets and approval policies. Do not assume an idle policy is active just because a command exists. Check which AX capabilities remain roadmap items.
How much could restart recovery actually save?
Measure the work repeated after an interruption, not the entire original job. A useful checkpoint can reduce replay. It cannot refund inference already consumed, and a suspend feature alone does not prove crash recovery.
Take an illustrative month with 1,000 jobs. Suppose 10% suffer exactly one interruption. If each affected job repeats $30 of model and tool work, replay costs $3,000. If tested recovery reduces repeated work to $3, replay falls to $300. That is $2,700 avoided replay, not $2,700 guaranteed savings from installing AX.
If the additional runtime, snapshot and operating costs total $600, the scenario leaves $2,100 before any other changed costs. These are illustrative USD assumptions, not provider prices or customer results. The improvement must be established under abrupt failure, not inferred from a successful pause.
For the complete method, including human review and accepted outcomes, use our AI agent cost-per-action guide. Use the same ledger to decide whether recovery improvements reduce the total cost of an accepted outcome.
How can you test AX file recovery without spending model tokens?
Start with a disposable workspace and a harmless sleeping process. This example follows the current manifest fields. It contains no model call or repository credential. It still consumes infrastructure resources. Compare the example with the current manifest reference.
Prerequisites are an installed, compatible AX/Substrate stack and CLI with authorized cluster access. Save this as task.yaml. debug: true is deliberate for this isolated diagnostic task because SSH exposes execution and file access. Do not use it as a public or production default.
apiVersion: ax.io/v1alpha1
kind: Workspace
metadata:
name: recovery-lab
spec:
files:
- path: README.txt
content: "Disposable recovery test. No model calls."
---
apiVersion: ax.io/v1alpha1
kind: Task
metadata:
name: recovery-lab-001
spec:
workspace:
name: recovery-lab
path: /workspace/recovery-lab
command: ["python3", "-c", "import time; time.sleep(3600)"]
debug: trueApply the manifest, confirm the task starts Suspended, then resume it with capacity in the Substrate WorkerPool. Wait for Ready and use Ctrl+C to stop observing the task.
ax apply -f task.yaml
ax resume task recovery-lab-001
ax watch task recovery-lab-001Write the marker after startup, then suspend and resume. The task command does not recreate this file, so reading it later checks persistence rather than repeated initialization.
ax ssh recovery-lab-001 -- sh -c 'printf "%s\n" "checkpoint-before-suspend" > /workspace/recovery-lab/recovery-marker.txt'
ax suspend task recovery-lab-001
ax resume task recovery-lab-001
ax watch task recovery-lab-001Wait for Ready again, stop the watch with Ctrl+C and read the marker:
ax ssh recovery-lab-001 -- cat /workspace/recovery-lab/recovery-marker.txtThe expected output is checkpoint-before-suspend. Passing shows that this file survived an orderly suspend/resume cycle. It does not prove RAM restoration, abrupt-crash durability or safe retries of external actions. We have not run this cluster test here.
Copy any evidence you need, then delete only the disposable resources created above. These commands remove the test task and its workspace definition:
ax delete task recovery-lab-001
ax delete workspace recovery-labWhere does network control live after AX Gateway removal?
Do not keep applying the old AX Gateway example. The reviewed Task schema has no gateway binding. Design outbound restrictions in the actual execution/network layer and verify they apply to the task’s actor.
Substrate v1.0.0 filters HTTP(S) and DNS by policy. Its default proxy path blocks WebSocket, forward-proxy CONNECT and other TCP traffic. Verify that the installed policy rejects unapproved destinations and inspect what your monitoring records for allowed TLS traffic. Review Substrate’s egress traffic rules.
Test allowed model and tool endpoints, denied destinations, redirects, direct provider access and credential revocation. Keep request-level authorization on allowed servers: a permitted hostname is not permission to access every tenant or document behind it. These are proposed checks, not vulnerabilities we reproduced. The agent sandbox security checklist covers the wider boundary.
Does AX automatically stop a runaway token bill?
The reviewed Task schema does not declare a cumulative money or token cap. CPU and memory limits apply to the sandbox. Model admission and accounting belong in the caller’s infrastructure.
Use an independently controlled model-admission layer. Attribute setup, subtasks and retries to one durable job ID. Reserve enough budget before admitting concurrent calls, reconcile usage afterwards and stop new admissions at the limit. Already admitted requests can still incur charges, so define a bounded overshoot rather than promising an instantaneous zero-overrun stop.
The controller that can revoke credentials or block new requests should sit outside the agent’s writable workspace. A prompt asking an agent to respect its budget is not a spending control.
Can you self-host AX without Google Cloud?
A non-GCP development path exists, but production portability needs its own test. Substrate documents local kind and GKE quickstarts. AX needs a reachable Substrate control API, its own control plane and PostgreSQL, and suitable runner images. A local runtime can still use remote inference.
AX still warns about breaking changes. Pin a tested pair and record the AX ref, Substrate release and runner digest together.
Is Antigravity required for every task?
The development Task schema specifies a command and an optional workspace. The recovery test uses a sleeping process with no inference. The runner exposes metadata on loopback at AX_METADATA_URL. Read the current sandbox startup and metadata contract.
What about Pi or a custom agent harness?
Treat it as an integration to validate, not a ready-made AX adapter established by this review. A custom image must supply /usr/local/bin/ax-task-runner, read AX_TASK_YAML and AX_WORKSPACE_YAML, prepare durable workspaces, expose readiness and supervise shutdown. Verify the harness’s session location and resume behavior independently.
Owning the repository is only one part of owning your AI. You should be able to inspect execution, export state, choose inference providers, revoke authority and leave the deployment. Cloud location alone does not answer those questions.
What must an AX pilot prove before production?
Use synthetic tasks and disposable systems first. These are our proposed acceptance gates, not tests AX has passed in our environment.
| Test | Evidence to collect |
|---|---|
| Orderly suspend | A marker written after startup survives; the harness resumes its saved session. |
| Abrupt failure | Kill the disposable worker mid-step and measure the lost checkpoint window and replay cost. |
| Ambiguous tool completion | A simulated timeout after an external write produces one effect, not two, after recovery. |
| Budget exhaustion | Parallel subtasks stop new admissions at a shared limit with measured in-flight overshoot. |
| Completion reporting | Success and failure reach the job supervisor even while the runner remains healthy. |
| Network and credentials | Denied routes stay denied and revocation blocks new privileged requests. |
| Resume under contention | Record p50/p95 resume-to-ready and first-useful-result latency with a busy worker pool. |
| Portability and upgrade | Export, restore and rollback succeed for the specific versions and infrastructure you intend to operate. |
Record the AX/Substrate revisions, runner digest, model route, policy configuration and outcome for every run. Choose AX when measured idle-capacity and recovery gains justify the platform work. A small fleet with cheap, short jobs may not need this extra infrastructure yet.
Wavect’s AI engineering services cover integration and production hardening. Our Twinsoft AI case study is separate delivery experience, not an AX reference deployment. Use the pre-launch QA checklist to structure acceptance evidence, or discuss your agent restart costs and runtime requirements.
Google AX: suspend/resume and deployment questions
What is Google AX Agent Executor?
AX is an Apache-2.0-licensed orchestrator for sandboxed agent tasks above Agent Substrate. Its current core resources are Task, Workspace and Model. It manages execution infrastructure, not the agent’s reasoning loop.
Does the current AX API still include Gateway?
No Gateway resource or Task gateway binding appears in the reviewed November 2026 schema. Replace old AX Gateway examples with configuration for the actual runtime/network policy layer, then verify outbound denial and authorization.
Does AX resume the exact in-memory agent state?
The documented AX runner restores the workspace into a fresh container and process tree. Substrate separately describes full-state snapshot capabilities. Validate the specific integration; do not infer RAM recovery from the lower-level runtime’s feature list.
Will AX automatically recover a $30 agent job after a crash?
Do not assume that. Orderly suspension can preserve workspace state, but abrupt failure requires durable checkpoints and tested recovery. External actions also need reconciliation and idempotency so a retry does not repeat the effect.
Does sub-second resume mean there is no idle cost?
No. Substrate reports resumes below 500 ms, but infrastructure activation is not complete application latency. Worker capacity, control-plane services, snapshot storage and operations still cost money; model charges are a separate category.
Does AX automatically suspend every idle task?
The reviewed documentation provides explicit suspend and resume commands. Automatic idle detection and suspension still appear in the AX roadmap. Verify the policy and implementation in your installed version rather than assuming it is enabled.
Does AX enforce a cumulative token budget?
The reviewed Task schema does not declare a job-wide token or money cap. Use an external admission control that counts setup, subtasks and retries, with bounded in-flight overshoot.
Can AX run without Google Cloud?
Substrate documents local kind development as well as GKE deployment. AX still needs a compatible Substrate control API, its own control plane and suitable runner images. This is a development option, not a verified production deployment on arbitrary infrastructure.
Why does ax ssh refuse to connect?
The task must explicitly enable spec.debug: true for guest execution and file services. Those services are off by default. Use debugging only in a controlled environment; immutable tasks need a deliberate replacement plan when their settings change.
Can a Pi agent or another harness run inside AX?
Treat this as a custom-runner integration to validate. The image must satisfy AX’s runner contract, including workspace preparation, readiness, metadata and shutdown. This review does not establish an official, ready-made Pi adapter.
Final thoughts
AX is worth evaluating when agent fleets waste meaningful capacity or repeat expensive work. Keep the opportunity and the guarantees separate. Prove what survives, what can repeat, what still costs money and who can stop the system. Own the recovery and spending controls, not just the YAML.
