AIForum.

What to Do When an AI API Timeout Still Completes the Request

On this page

THE SHORT ANSWER

A timeout and a successful completion are not mutually exclusive: a request can hit a deadline on one layer while the work finishes on another. The fix is to stop inferring success from partial text and start requiring the provider's terminal completion event, while reading the run outcome from the layer that actually recorded it.
Quick facts

A timeout and a successful completion are not mutually exclusive. A request can blow past a deadline on one layer while the work itself finishes on another, and the response your client receives may reflect only one of those facts. The practical rule is short: a stream without its provider’s terminal completion event is incomplete, not successful. Require that event, and read the run outcome from the layer that actually recorded it.

This guide is for developers and automation builders calling OpenAI-compatible Chat Completions and Responses endpoints, or Anthropic-style Messages endpoints, who have seen a truncated answer arrive looking like a normal success. It covers how to tell which deadline fired, how to make a client fail loudly when it should, and how one open-source gateway fixed the same class of bug at the server side.

Why a timeout and a completion can both be true

Timeouts are not one thing. ModelDial’s timeout, streaming, and retry guide separates three independent deadlines that a request can hit: a connection timeout before a usable response is established, an idle-read timeout after the response starts but no bytes arrive for too long, and an overall deadline that stops a request even while chunks continue to arrive. Raising one value does not repair the other two.

The third case is where the confusion lives. Chunks keep flowing, so nothing looks dead — but an absolute cap enforced by a client, reverse proxy, edge platform, or provider can still end the request. As ModelDial puts it, regular events can prevent an idle connection from looking dead, but they cannot extend an absolute request cap.

A parallel split exists on the server side. An agent or job can carry its own run budget that is separate from the HTTP connection. When that budget expires, the work stops — but the HTTP layer may already have delivered partial text and may have nothing left to report except a normal-looking response. That is precisely the bug fixed in openclaw pull request #133275.

What the openclaw gateway fix changed

The openclaw project merged fix(gateway): report timed-out HTTP agent runs as failures on 2026-08-30 (merge commit a8c8c82708361d2f859f820562ae1b00cf162e93), closing issue #133274. Before the change, callers of the OpenAI-compatible Chat Completions and Responses endpoints received a successful completion when the whole-agent run budget expired. Streaming callers could receive partial text followed by success without ever seeing the timeout.

The mechanism is the part worth copying. Both command return paths already recorded the final run outcome after delivery and cleanup. All four HTTP projections now read that recorded fact instead of inferring failure from optional meta.error or stopReason: "error" fields. The change deleted a duplicate classifier rather than adding a new one; timeout and cancellation precedence, retry policy, existing error envelopes, and disconnect handling were left unchanged. Production code shrank by 9 net lines (+8/−17) while tests grew by 68 net lines (+94/−26).

Two design decisions stand out. First, the fix did not add timeout-string matching or a second HTTP-specific classifier — the pull request explicitly calls that out as duplicating the existing owner policy. Second, it needed no new timing state, because both SSE endpoints already wait for the returned command result before finalizing. After the fix, expired agent runs report an error through JSON or a failed streaming terminal instead of presenting an incomplete answer as successful. Already-delivered partial content still reaches the caller; successful and genuinely recovered runs keep their normal completion behavior.

Diagnose which deadline actually fired

Before changing any configuration, establish which deadline ended the request. The three deadlines behave differently and have different fixes.

DeadlineWhen it firesWhat the client typically sees
Connection timeoutBefore a usable response is establishedNo headers, no body, transport-level error
Idle-read timeoutAfter the response starts, when no bytes arrive for too longHeaders and some events, then a stall, then a close
Overall deadlineAt an absolute wall-clock cap, even while chunks keep arrivingEvents flowing normally, then an abrupt end

To distinguish them, log the shape of the request rather than a single failure label. ModelDial recommends recording when the request started, when headers arrived, when the first event arrived, and when the last event arrived, and keeping the endpoint, protocol, model, effort, status code, and attempt number with that timing record. It also warns against reducing every transport failure to “network error” — the recovery action depends on the stage.

The following is an illustrative logging pattern, not a tested implementation. Adapt it to your own client:

# Illustrative logging pattern — not a tested implementation
started       = now()
headers_at    = None
first_event_at = None
last_event_at  = None
terminal_seen  = False

for event in stream:
    if headers_at is None:
        headers_at = now()
    if first_event_at is None:
        first_event_at = now()
    last_event_at = now()
    if event.type in TERMINAL_EVENTS:
        terminal_seen = True

log({
    "endpoint": endpoint, "protocol": protocol, "model": model,
    "effort": effort, "status": status_code, "attempt": attempt,
    "started": started, "headers_at": headers_at,
    "first_event_at": first_event_at, "last_event_at": last_event_at,
    "terminal_seen": terminal_seen,
})

Two patterns in that data are diagnostic. If requests repeatedly end at nearly the same wall-clock time, an absolute cap is likely in play somewhere in the chain — inspect the client, reverse proxy, edge platform, and provider limits in that order. If the last event arrives long before the close, an idle-read timeout is the better hypothesis. And if terminal_seen is false, the run is incomplete regardless of the HTTP status code.

Make the client require a terminal event

Text deltas are useful for display and recovery, but they do not prove the request completed. ModelDial is explicit about the terminal markers to wait for: OpenAI Responses emits a response.completed event for a successful completed response, and Anthropic Messages ends a normal SSE sequence with message_stop. If the socket closes before that marker, preserve the partial text for diagnosis but mark the evaluation attempt as failed.

  1. Parse the stream as a state machine. Treat the run as pending until a terminal event arrives. Do not resolve the promise on socket close.
  2. Separate display from evaluation. Render partial text to the user if that is useful, but keep the attempt’s status distinct from the text you displayed.
  3. Record the reason for the close. Distinguish “terminal event received” from “socket closed,” “idle timeout,” and “overall deadline exceeded.”
  4. Fail loudly. Surface a failed streaming terminal or an error envelope rather than a success response carrying incomplete content.

Scoring a truncated answer as if it were complete silently rewards transport failures, which is why the openclaw fix reports an error rather than a success when the run budget expires.

Retry rules that do not hide the problem

ModelDial’s guidance is to retry only a bounded number of times and to record each attempt. A bounded retry with per-attempt logging keeps the failure visible while still recovering from transient network blips.

Two caveats matter. First, a retry does not fix an absolute cap: if every attempt ends at the same wall-clock mark, more attempts will not help. Second, if the run actually completed server-side after your client gave up, a blind retry can duplicate work or cost. Where the API supports idempotency keys or request identifiers, use them so a retry can be recognized rather than repeated.

Verify that the fix worked

After changing a client or gateway, confirm the original symptom is gone rather than assuming it. A minimal verification loop:

  1. Force a short overall deadline that you know the run will exceed.
  2. Confirm the client reports a failure — a failed streaming terminal or an error response — not a success with truncated text.
  3. Confirm the partial content you already received is still available for diagnosis.
  4. Run a request that completes normally and confirm it still returns a success with its terminal event intact.
  5. Check the timing log for the four timestamps and the attempt number, so the next failure is diagnosable without re-instrumenting.

If you are patching a gateway or proxy rather than a client, the openclaw change is a useful template: have the component that owns the run record its final outcome, then let every HTTP projection read that recorded outcome instead of guessing from optional fields.

What not to do

  • Do not match on timeout strings. The openclaw pull request deliberately avoided timeout-string matching and a second HTTP-specific classifier, describing both as duplication of the existing owner policy.
  • Do not infer failure from optional fields. meta.error and stopReason: "error" may be absent even when the run failed. Read the authoritative run outcome instead.
  • Do not raise one timeout value and expect a fix. Connection, idle-read, and overall deadlines are separate; raising one does not repair the other two.
  • Do not treat a closed socket as a completed answer. A stream without its terminal event is incomplete.

When you cannot determine the cause

If repeated failures end at inconsistent times with no terminal event and no clear proxy involvement, the missing diagnostic information is usually the four-timestamp log and the per-attempt record. Without those, you cannot distinguish an idle-read timeout from an overall deadline, and you cannot tell whether the server finished the work after your client disconnected. Collect that data first; the fix usually becomes obvious once the timeline is visible.

If you have hit this in your own stack, share your endpoint, protocol, and the four timestamps with the community — a concrete timeline is far more useful than a description of the symptom.

FAQ

Can an AI API request time out and still have completed on the server?

Yes. The HTTP connection and the server-side run can have separate budgets. If the run budget expires, the work stops even though partial text may already have been delivered; if your client's deadline expires first, the server may still finish the job after you have disconnected. In both cases the client's view of success or failure may not match what actually happened.

Why does raising my timeout value not fix the problem?

Connection, idle-read, and overall deadlines are independent. Raising one does not repair the other two. If requests repeatedly end at nearly the same wall-clock time, an absolute cap enforced somewhere else in the chain — a reverse proxy, edge platform, or the provider — is the more likely cause.

Should I retry after a partial stream?

Retry only a bounded number of times and record each attempt. Preserve the partial text for diagnosis but mark the attempt as failed. Be aware that a retry cannot overcome an absolute cap, and that if the run completed server-side after your client gave up, a blind retry may duplicate work or cost — use idempotency keys where the API supports them.

How do I know whether the server recorded a failure?

Read the authoritative run outcome from the component that owns the run, rather than inferring from optional fields. The openclaw gateway fix moved all four HTTP projections to read the recorded final run outcome instead of checking optional meta.error or stopReason: "error" fields.

References

  1. https://github.com/openclaw/openclaw/pull/133275
  2. https://modeldial.com/en/guides/ai-api-timeout-streaming-retry

Related terms

Terminal event
The provider-specific marker that confirms a streamed response completed, such as response.completed for OpenAI Responses or message_stop for Anthropic Messages. Its absence means the run is incomplete.
Idle-read timeout
A deadline that fires after a response has started but no bytes arrive for too long. Distinct from a connection timeout and from an overall deadline.
Overall deadline
An absolute wall-clock cap that stops a request even while chunks continue to arrive. It can be enforced by the client, a reverse proxy, an edge platform, or the provider.
Run budget
A server-side limit on how long an agent or job may run, tracked separately from the HTTP connection. When it expires, the work stops even if the HTTP layer already delivered partial output.
SSE (Server-Sent Events)
The streaming transport used by many chat completion endpoints. Events arrive incrementally, and the sequence is only complete once the provider's terminal event is received.

Leave a Reply

Your email address will not be published. Required fields are marked *