feat(agent-proxy): segment streaming responses at ~54s to bypass Netlify Edge's ~60s response cap
Netlify Edge Functions empirically cap a single streaming response at
~60s wall-clock, regardless of activity. Confirmed against Netlify's
own canonical SSE example (edge-functions-examples.netlify.app/sse)
which also cuts at +60.1s. The Anthropic Managed Agent session is
fine for several minutes; the cap is per-HTTP-response.
This commit splits a long brief across multiple HTTP responses, while
keeping the UX of one continuous stream:
netlify/edge-functions/agent-proxy.ts
- Accept either a new-session payload {stationName, stationLocation,
stationWebsite} or a resume payload {sessionId, lastEventId,
startedAt}.
- On resume, skip session creation + user.message send. Just reopen
the live SSE stream and backfill via
GET /v1/sessions/{id}/events?after_id=lastEventId (deduped by
event.id), then keep tailing.
- Single AbortController per segment. A 54s timer aborts the upstream,
the for-await loops exit, and we write one final NDJSON line:
{type:'segment_end', sessionId, lastEventId, startedAt}.
- The 20-min OVERALL_BUDGET_MS is enforced via Date.now() - startedAt
so it spans across all segments.
- Refactor main loop so every iteration is openStream + backfill +
tail. Cleaner than the previous initial-stream + reconnect-only-on-
drop pattern.
src/App.jsx
- readStream() now returns a {sessionId, lastEventId, startedAt}
payload if it saw a segment_end, or null if the stream ended
cleanly.
- handleSubmit() loops, reopening /api/agent-proxy with the resume
payload until readStream returns null. Spinner/status state stays
on across segments so the UI shows one continuous stream.
README.md
- Document the segmented-streaming protocol and why it exists.
Co-Authored-By: alex <alex@semipublic.co>
This commit is contained in:
@@ -34,27 +34,49 @@ Open http://localhost:8888.
|
||||
1. The browser POSTs the form to `/api/agent-proxy`.
|
||||
2. A Netlify Edge Function (Deno runtime) creates an Anthropic Managed
|
||||
Agent session, opens the upstream SSE event stream, and sends the
|
||||
user message. It then tails the stream and re-opens it (with an
|
||||
user message. It tails the stream and re-opens it (with an
|
||||
`events.list` backfill, deduped by event id) whenever it drops mid
|
||||
session, until the session reports a terminal `session.status_*`
|
||||
event or a 20-minute wall-clock budget is hit.
|
||||
3. Downstream to the browser the function emits newline-delimited JSON
|
||||
(`application/x-ndjson`) — `text`, `status`, `heartbeat`, `done`,
|
||||
`error` — one object per line.
|
||||
(`application/x-ndjson`) — `text`, `status`, `heartbeat`,
|
||||
`segment_end`, `done`, `error` — one object per line.
|
||||
4. The React app reads `response.body.getReader()`, splits on `\n`, and
|
||||
parses each line. `text` lines append to the brief; `status` lines
|
||||
drive a separate “what the agent is doing now” banner.
|
||||
5. A `Thinking` flag stays `true` until the first chunk arrives, then
|
||||
flips to a streaming state with a pulsing cursor.
|
||||
|
||||
Edge Functions are required here — the previous v2 Node Function ran on
|
||||
AWS Lambda and got killed at ~27 s, well before any reconnect could
|
||||
fire. Edge Functions run on Deno Deploy with no streaming-duration cap
|
||||
as long as the function keeps writing to the response body.
|
||||
### Why segmented streaming
|
||||
|
||||
Netlify Edge Functions empirically cap a single streaming response at
|
||||
~60 s wall-clock (Netlify's own canonical SSE example cuts at the same
|
||||
mark, even though the docs imply "indefinite" streaming). The Anthropic
|
||||
Managed Agent session itself is fine for several minutes; the limit is
|
||||
per-HTTP-response. So the proxy segments the stream:
|
||||
|
||||
- Just before ~54 s the function writes one final NDJSON line —
|
||||
`{"type":"segment_end","sessionId":"sess_…","lastEventId":"evt_…","startedAt":…}` —
|
||||
and closes the response.
|
||||
- The React side sees `segment_end`, immediately POSTs to
|
||||
`/api/agent-proxy` again with `{sessionId, lastEventId, startedAt}`
|
||||
in the body.
|
||||
- The function recognises this as a resume payload: it does **not**
|
||||
create a new session or re-send the user message. It just reopens
|
||||
the live SSE stream, backfills via
|
||||
`GET /v1/sessions/{id}/events?after_id=lastEventId` (deduped by
|
||||
`event.id`), and keeps tailing.
|
||||
|
||||
The 20-minute overall budget is enforced via `Date.now() - startedAt`
|
||||
so it spans across all segments. The previous v2 Node Function (on AWS
|
||||
Lambda) was killed at ~27 s, well before any reconnect could fire —
|
||||
moving to Edge Functions extended that to ~60 s per response, and the
|
||||
segment loop extends it the rest of the way.
|
||||
|
||||
## Files
|
||||
|
||||
- [src/App.jsx](src/App.jsx) — form, NDJSON streaming reader, UI states.
|
||||
- [src/App.jsx](src/App.jsx) — form, NDJSON streaming reader,
|
||||
segment-end resume loop, UI states.
|
||||
- [netlify/edge-functions/agent-proxy.ts](netlify/edge-functions/agent-proxy.ts)
|
||||
— Managed Agent proxy with reconnect, backfill, and NDJSON wire
|
||||
protocol.
|
||||
— Managed Agent proxy with reconnect, backfill, segmentation, and
|
||||
NDJSON wire protocol.
|
||||
|
||||
Reference in New Issue
Block a user