2026/11/20 - Amazon SageMaker Gateway Runtime - 2 new api methods
Changes Adds HyperPod Global Inference Gateway APIs to route inference across HyperPod clusters in multiple Regions. Adds the MoveClusterNodes API, which moves running nodes between instance groups, within a cluster or between clusters in the same region, account and subnet, without releasing the instances.
Buffered (non-streaming) inference. Returns the full OpenAI completion body.
Errors carry their origin in the Message prefix. Every typed error's text is in the Message member (JSON key Message), and starts with one of these prefixes:
Global Inference Gateway: -- the gateway rejected or failed the request itself: validation, authorization, throttling, routing (the gateway or model was not found, or no capacity is available), a gateway limit or time limit, or failing to reach the inference gateway ( Global Inference Gateway: could not reach the inference gateway: <reason>, or Global Inference Gateway: timed out waiting for the inference gateway). This includes the 400 for an invalid request and the 413 for a request body that is too large.
inference gateway: -- the inference gateway serving the model returned an error status. The message gives that status code and up to 256 bytes of its error message, for example inference gateway: 503 <message>. A BadGatewayException can instead read inference gateway: response ended unexpectedly, with no status, when the inference gateway's connection failed while it was sending the response.
inference gateway (<component>): -- the same, when the inference gateway reports which of its components produced the error, for example inference gateway (model server): 502 <message>. The component is one of:
config -- the inference gateway's configuration and proxy layer.
scheduler -- the inference gateway's request scheduler.
model selector -- the inference gateway's model selector.
model server -- the model server.
Every error status the inference gateway returns is a typed error, chosen by that status:
400 and any other 4xx not listed here -- a 400 validation error.
401 and 403 -- AccessDeniedException (403).
404 -- ResourceNotFoundException.
408 -- GatewayTimeoutException (504).
429 -- ThrottlingException, with the Retry-After header.
502, and 500, 503 or any other 5xx not listed here -- BadGatewayException (502). A 500 or 503 from the inference gateway comes after the request was dispatched, so it is not modeled as retryable, and the gateway may already have tried another inference gateway. Standard SDK retry modes may still retry a 5xx status; set max attempts to 1 if a repeated inference is unacceptable.
504 -- GatewayTimeoutException.
See also: AWS API Documentation
Request Syntax
client.invoke_global_inference_gateway(
GatewayId='string',
Accept='string',
ContentType='string',
CustomerIdentityToken='string',
Body=b'bytes'|file
)
string
[REQUIRED]
The ID of the Global Inference Gateway to invoke. IAM policies refer to the gateway by its ARN, arn:<partition>:sagemaker:<region>:<account>:global-inference-gateway/<id>.
string
The MIME type you want for the response. It is passed to the model server.
string
[REQUIRED]
The MIME type of the request body. Pass application/json (parameters such as charset=utf-8 are allowed): the body is one OpenAI JSON object. The Global Inference Gateway rejects other content types with a 415 error.
string
Per-caller customer identity token (a JWT), forwarded unchanged to the inference gateway, which validates it. The gateway does not use it to authorize the request. Include it in the SigV4 signed headers so it cannot be altered in transit. Redacted from logs, metrics and traces.
bytes or seekable file-like object
[REQUIRED]
The inference request body: one OpenAI JSON object, at most 10 MiB. Its content is passed to the model server unchanged, except that the request is always sent with "stream": false: this operation returns one JSON document, so a "stream": true in the body is overridden (use InvokeGlobalInferenceGatewayWithResponseStream to stream). A body that is not one JSON object, or that repeats the top-level model key with different values, is rejected with a 400 validation error. A body larger than 10 MiB is rejected with PayloadTooLargeException (413). The gateway reads only model and stream, and does not store the body.
dict
Response Syntax
{
'Body': StreamingBody()
}
Response Structure
(dict) --
The response from InvokeGlobalInferenceGateway.
Body (:class:`.StreamingBody`) --
The model server's OpenAI completion JSON, returned unchanged. A response larger than 10 MiB fails with BadGatewayException (502).
Streaming inference. Returns an AWS event stream of PayloadPart events that carry the model server's server-sent event (SSE) bytes, so SDKs decode the stream into events for you.
The errors listed for this operation, other than ModelStreamError and InternalStreamFailure, are returned before the response starts (before the 200 status), so they end the request, not the stream. None of them can occur after the 200. ModelStreamError and InternalStreamFailure appear only as events in the stream, after the 200. Once the 200 is sent, the stream carries the response as PayloadPart events. If reading the inference gateway's response fails, the stream carries one InternalStreamFailure event and then ends. ModelStreamError is reserved and is not currently sent.
Errors carry their origin in the Message prefix. Every typed error's text is in the Message member (JSON key Message), and starts with one of these prefixes:
Global Inference Gateway: -- the gateway rejected or failed the request itself: validation, authorization, throttling, routing (the gateway or model was not found, or no capacity is available), a gateway limit or time limit, or failing to reach the inference gateway ( Global Inference Gateway: could not reach the inference gateway: <reason>, or Global Inference Gateway: timed out waiting for the inference gateway). This includes the 400 for an invalid request and the 413 for a request body that is too large.
inference gateway: -- the inference gateway serving the model returned an error status. The message gives that status code and up to 256 bytes of its error message, for example inference gateway: 503 <message>.
inference gateway (<component>): -- the same, when the inference gateway reports which of its components produced the error, for example inference gateway (model server): 502 <message>. The component is one of:
config -- the inference gateway's configuration and proxy layer.
scheduler -- the inference gateway's request scheduler.
model selector -- the inference gateway's model selector.
model server -- the model server.
An InternalStreamFailure event's Message uses the same prefixes, with the same text the buffered operation uses for the same failure. Its inference gateway: message carries no status: inference gateway: response ended unexpectedly when the inference gateway's connection failed while it was sending the response.
Every error status the inference gateway returns before the 200 is a typed error, chosen by that status as for InvokeGlobalInferenceGateway:
400 and any other 4xx not listed here -- a 400 validation error.
401 and 403 -- AccessDeniedException (403).
404 -- ResourceNotFoundException.
408 -- GatewayTimeoutException (504).
429 -- ThrottlingException, with the Retry-After header.
502, and 500, 503 or any other 5xx not listed here -- BadGatewayException (502). A 500 or 503 from the inference gateway comes after the request was dispatched, so it is not modeled as retryable, and the gateway may already have tried another inference gateway. Standard SDK retry modes may still retry a 5xx status; set max attempts to 1 if a repeated inference is unacceptable.
504 -- GatewayTimeoutException.
See also: AWS API Documentation
Request Syntax
client.invoke_global_inference_gateway_with_response_stream(
GatewayId='string',
Accept='string',
ContentType='string',
CustomerIdentityToken='string',
Body=b'bytes'|file
)
string
[REQUIRED]
The ID of the Global Inference Gateway to invoke, as for InvokeGlobalInferenceGateway.
string
The MIME type you want for the response. It is passed to the model server.
string
[REQUIRED]
The MIME type of the request body, as for InvokeGlobalInferenceGateway. Pass application/json.
string
The customer identity token (a JWT), as for InvokeGlobalInferenceGateway. It is passed unchanged to the inference gateway, which validates it.
bytes or seekable file-like object
[REQUIRED]
The inference request body, as for InvokeGlobalInferenceGateway, except that the request is always sent with "stream": true: this operation returns an event stream, so a "stream": false in the body (or no stream key) is overridden. A body larger than 10 MiB is rejected with PayloadTooLargeException (413).
dict
The response of this operation contains an :class:`.EventStream` member. When iterated the :class:`.EventStream` will yield events based on the structure below, where only one of the top level keys will be present for any given event.
Response Syntax
{
'Body': EventStream({
'PayloadPart': {
'Bytes': b'bytes'
},
'ModelStreamError': {
'Message': 'string',
'ErrorCode': 'StreamBroken'|'ModelInvocationTimeExceeded'
},
'InternalStreamFailure': {
'Message': 'string'
}
})
}
Response Structure
(dict) --
The response from InvokeGlobalInferenceGatewayWithResponseStream.
Body (:class:`.EventStream`) --
The response as an AWS event stream (application/vnd.amazon.eventstream).
PayloadPart (dict) --
A part of the model server's response.
Bytes (bytes) --
The bytes of this part: a chunk of the model server's server-sent event (SSE) stream.
ModelStreamError (dict) --
Reserved for a model error during the stream. Not currently sent.
Message (string) --
A description of the error.
ErrorCode (string) --
The error code: StreamBroken if the model server's stream broke before it completed, or ModelInvocationTimeExceeded if the model exceeded its allowed invocation time.
InternalStreamFailure (dict) --
A failure after the stream started. The stream ends after this event.
Message (string) --
A description of the failure. It starts with Global Inference Gateway: or inference gateway:, which says where the failure came from.