Configure streaming for LLM responses and other traffic You can now create API Gateway gateways that stream requests and responses instead of buffering them
Share
Services
## Feature
Feature
**Configure streaming for LLM responses and other traffic**
You can now create API Gateway gateways that stream requests and responses instead of buffering them. This Public Preview feature supports incremental response delivery over HTTP/2 or HTTP/1.1 chunked transfer encoding, Server-Sent Events (SSE), WebSockets, and gRPC bidirectional streaming. A common use is streaming token-by-token responses from a large language model (LLM).
To enable streaming, create a gateway with the `--enable-streaming` flag. The streaming mode is fixed when you create the gateway and can't be changed later.
For more information, see [Configure streaming for LLM responses and other traffic](https://cloud.google.com/api-gateway/docs/streaming-configure).
What else is happening at Google Cloud Platform?
Read update
Services
Share
Read update
Services
Share
Google Distributed Cloud (software only) for VMware 1.36.100-gke.144 is now available
about 5 hours ago
Services
Share
Google Distributed Cloud (software only) for bare metal 1.36.100-gke.144 is now available for
about 5 hours ago
Services
Share
BigQuery offers TimesFM 3.0, a built-in time series forecasting model that implements Google Research's open
about 6 hours ago
Services
Share