On 3rd October, my astro-coach agent tried to fix a production Supabase environment problem. It kept retrying until a model usage limit stopped it. The tempting summary is “the agent tried to fix production.”
That summary hides most of the engineering.
|
Step in the 3 October astro-coach run |
Who did it |
What it handed on |
|---|---|---|
|
Diagnose the production Supabase environment problem |
Person |
A diagnosis and a specific fix to attempt |
|
Bake the environment settings into the application build |
Code |
A build artifact whose configuration needed verification |
|
Encounter a hang |
Code/tool execution |
An unresolved execution state the agent could not read |
|
Decide to retry the environment fix |
Model, acting through the agent |
Another invocation, without resolving the previous attempt’s state |
|
Keep retrying until a model usage limit stopped the run |
Model choosing retries; code executing them |
An interrupted workflow, without a confirmed outcome |
|
Confirm the login message and whether users could actually log in |
Person required |
The evidence needed to accept the fix or keep the incident open |
This was a workflow on my astro-coach repository, now home to Jyotish Coach. It is tempting to summarize it as “the agent tried to fix production.”
That description hides most of the engineering.
A person supplied the diagnosis. Code performed the environment bake. The model chose to repeat the operation. A person still needed to confirm the result.
The agent’s consequential contribution was the decision at the boundary: run that step again.
Most of what we call an agent workflow is a chain of ordinary software operations and human decisions, with a model choosing some of the transitions. If you ship agents, those transitions deserve as much attention as the model.
Where the loop actually lived
The environment bake was a code operation. Given the intended configuration and a source revision, the build system should produce an artifact and report what happened.
The repository makes this distinction concrete. Its Dockerfile passes NEXT_PUBLIC_SUPABASE_URL and NEXT_PUBLIC_SUPABASE_ANON_KEY into the environment before running the application build.
Its operations notes also warn that, in this deployment setup, changing those variables and triggering a redeploy can reuse the previous image. The documented check is to look for an actual application build.
That means “the variable changed,” “the deployment completed,” and “the browser received the intended configuration” are separate claims.
After the hang in the 3rd October run, the agent could not read the execution state. It kept attempting the production environment fix until a model usage limit stopped it.
The loop lived around the code operation: invoke, lose visibility, invoke again.
A hang does not tell you whether an operation failed before making a change, completed without returning a response, or remained in progress. Repeating the command does not resolve that uncertainty. It can add another attempt whose relationship to the first is also unclear.
The usage limit ended the activity. It did not establish that production was correct.
Name the handoffs
“Agent fixed it” is too coarse to review. Each handoff needs an input, a record of the state left behind, and a next owner.
For this workflow, I would name them explicitly:
|
Handoff |
Input crossing the boundary |
State left behind |
Next owner |
|---|---|---|---|
|
Diagnosis → preparation |
The person’s diagnosis, target environment, and intended configuration change |
A proposed operation with a defined scope |
Model prepares the action; person reviews its scope |
|
Preparation → build |
Approved inputs and source revision |
An identified build attempt and its logs |
Code |
|
Build → deployment approval |
Artifact identity and configuration checks |
A report of success, failure, or unresolved execution |
Person |
|
Approval → production change |
Sign-off tied to that artifact and target |
A deployment attempt with a recorded identity |
Code |
|
Interrupted execution → reconciliation |
Attempt identity and last known status |
A read-only determination of what happened, or an explicit unknown |
Code checks state; person owns unresolved cases |
|
Deployment → login acceptance |
The deployed revision and technical checks |
Evidence from the actual sign-in flow |
Person |
This is a proposed ownership structure, not a claim that the October run followed it. The run exposed why it was needed.
The weakest handoff was interrupted execution to the next decision. The agent had an incomplete observation, but retained the ability to initiate another production attempt.
That is where uncertainty became action.
A durable attempt record would make a different next step possible: inspect the existing operation. Without that record, the model is being asked to reason about state it cannot observe.
Make the environment bake a script that runs once and reports
The structural fix is to take the repeated environment bake out of the model’s decision loop.
I would make it a script that runs once for a specified operation, then reports. Its inputs would identify the source revision, target environment, and intended configuration. Its output would identify the build attempt, resulting artifact, verification results, and final status. It should report configuration checks without exposing secret values.
“Runs once” needs enforcement in code. An operation identifier should prevent a second invocation from silently starting duplicate work. If the caller loses its connection, the next action should retrieve that operation’s status.
The script also needs to preserve uncertainty honestly. A timeout is an unresolved result until the underlying build or deployment state has been checked.
The model can help prepare inputs and interpret the report. The execution contract belongs in code.
Before a production change, a person signs off on the concrete artifact, target, and expected effect. That approval must refer to what will actually be deployed. If the artifact changes, the earlier approval no longer describes the action.
This gives the workflow a stopping condition tied to the task: a reported result, a verified state, or an unresolved case handed to a person. A model usage limit should never have to serve as the incident’s stopping rule.
The step I would not delegate
The final handoff is deployment evidence to user acceptance.
A build can succeed with the wrong configuration. A deployment can be healthy while authentication fails. A sign-in message can appear successful without giving the user a usable session.
The repository’s authentication environment code documents one such failure mode: sign-in can appear to work while subsequent API calls return unauthorized responses. That is why a reassuring message is insufficient evidence.
Automated checks should cover login. They should verify session creation and access to protected routes. But for closing this production incident, I would keep a person responsible for the acceptance decision.
I would open the deployed site, use the relevant sign-in method, read the message, and confirm that the session lets me reach the protected experience.
The model can assemble evidence. Code can test the path. I would not delegate the final confirmation that users can actually log in.