gRPC and Proxies
Palette uses gRPC to communicate between the management platform and the workload cluster. gRPC is a high-performance, open source universal Remote Procedure Call (RPC) framework. It is used to build distributed applications and services. gRPC is based on Hypertext Transfer Protocol Version 2 (HTTP/2) and uses protocol buffers (protobuf) as the underlying data serialization framework.
Refer to the Network Ports documentation for a detailed network architecture diagram with gRPC and to learn more about the ports used for communication.
gRPC and WebSocket
The Palette agent automatically attempts to connect to the management plane using gRPC through Hypertext Transfer Protocol Secure (HTTPS) using the HTTP/2 protocol. In some environments, the network configuration may not allow gRPC traffic to pass through. A common scenario is when the network is behind a proxy server that does not support HTTP/2. In this scenario, the Palette agent will first attempt to connect to the management plane using HTTP/2. After several failed attempts, the agent will fall first attempts to connect to the management plane using HTTP/2. After several failed attempts, the agent falls back to using WebSocket over HTTPS with HTTP/1.1.
The fallback to WebSocket with transcoding occurs automatically and does not require any additional configuration.
gRPC Transcode
Behind the scenes, when the Palette agent fails to connect with the management plane after a maximum of ten connection attempts, the agent initiates the failover to a WebSocket connection and transcodes the gRPC messages with the HTTP/1.1 protocol.
The Palette agent directs gRPC messages to a freshly started in-memory proxy service, which takes the original gRPC request, transcodes it to HTTP/1.1 protocol, and sends it over the WebSocket connection to the management plane. The management plane's WebSocket handler will then accept the WebSocket message and transcode it back to the HTTP/2 protocol before forwarding it to the gRPC handler. The server will then respond with a gRPC message, which will be transcoded to HTTP/1.1 and sent back to the agent over the WebSocket. The agent's in-memory proxy will read the message and transcode it back to HTTP/2 and pass it to the agent.
Below is a high-level overview of the order of operations when the Palette agent falls back to using WebSocket:
- The agent initiates a new gRPC request to the management plane servers that is picked up by the in-memory proxy service.
- The agent's in-memory proxy creates a WebSocket connection with the management plane servers.
- The management plane server accepts the WebSocket connection
- The agent in-memory proxy transcodes the gRPC request on-demand and sends it via the WebSocket connection.
- The server's WebSocket handler reads the request off the WebSocket connection and forwards it to the server's gRPC handler.
- The gRPC handler processes the request and responds via the same connection. The WebSocket handler sends the response from the gRPC handler back to the agent.
- The agent's in-memory proxy reads the response off the WebSocket connection and transcodes it back to HTTP/2 and passes it to the agent.
A more straightforward way to think about the WebSocket transcoding architecture is that network traffic between the Palette agent and the management plane uses the WebSocket connection and the HTTP/1.1 protocol. The agent and server are still communicating using gRPC, but the messages are transcoded to the HTTP/1.1 protocol between the two entities. Using WebSocket and HTTP/1.1 removes issues due to application firewalls or network proxies not supporting the HTTP/2 protocol. Once the gRPC message is internal to the agent or the server, the HTTP/2 protocol is used for communication.
gRPC and Proxies
The following sections provide information about using gRPC with network proxies. These issues are addressed by using WebSocket and the HTTP/1.1 protocol as a fallback mechanism. However, if you want to better understand the reasons for falling back to a WebSocket connection, the following sections provide more information about challenges with gRPC and network proxies. If you want to learn more about gRPC and transcoding, check out the Red Hat article gRPC Anywhere.
When gRPC is used with network proxies, proxy behavior depends on whether the proxy tunnels the connection (no SSL bump) or intercepts and re-encrypts it (SSL bump). The following table summarizes common proxy scenarios and whether gRPC traffic can pass through. If the gRPC connection fails, the agent automatically falls back to using WebSocket.
| Scenario | Proxy Mode | Description | Proxy Supported |
|---|---|---|---|
| gRPC with HTTP/HTTPS | No SSL bump | gRPC traffic is sent over HTTP/HTTPS, and the proxy does not perform a Secure Socket Layer (SSL) bump. This is universally supported. | ✅ |
| gRPC with HTTP/HTTPS | SSL bump | gRPC traffic is sent over HTTP/HTTPS, and the proxy performs an SSL bump. | Varies by vendor |
| gRPC with Squid | CONNECT HTTP | gRPC can pass through the proxy because the HTTP/2 connection between the Palette agent and the management plane remains intact. | ✅ |
| gRPC with Squid | SSL bump | Squid does not support HTTP/2 for intercepted (SSL-bumped) HTTPS traffic, so gRPC fails when Squid performs an SSL bump. | ❌ |
The following sections provide more information about gRPC and proxies.
Proxy Without SSL Bump
Because gRPC is based on HTTP/2, any proxy server that supports the HTTP CONNECT method can be used to forward gRPC traffic. No configuration is required for this scenario. The exception is when the proxy server performs an SSL bump, discussed in the Proxy With SSL Bump section.
SSL bump is a technique used to decrypt and inspect HTTPS traffic. When SSL bump is enabled, the proxy server terminates the Transport Layer Security (TLS) connection and establishes a new TLS connection to the destination server. In this scenario, the proxy server must support gRPC and may require additional configuration.
Proxy With SSL Bump
Several vendors provide proxy servers that support gRPC. Some of the vendors may require additional configurations or the use of a specific version of the proxy server. We encourage you to review your proxy server documentation for more information.
When you review the vendor documentation, search for information about gRPC and HTTP/2. We provide the following links to some vendors' documentation that addresses HTTP/2 and gRPC support.
Squid Proxy
A common open source proxy server is Squid. Squid is a caching proxy for the web, supporting HTTP, HTTPS, File Transfer Protocol (FTP), and more.
Squid has limited HTTP/2 support. It does not support HTTP/2 for intercepted (SSL-bumped) HTTPS traffic. Because gRPC uses HTTP/2, gRPC traffic does not work when Squid performs an SSL bump. For more information about Squid HTTP/2 support, refer to Feature: HTTP/2.0 support.
If you use Squid without SSL bump, Squid can tunnel the connection using the
CONNECT HTTP method. In this scenario, gRPC can
pass through the proxy because the HTTP/2 connection between the Palette agent and the management plane remains intact.
If gRPC traffic cannot pass through your Squid proxy, the Palette agent automatically falls back to using WebSocket over HTTPS with HTTP/1.1 as described in the gRPC and WebSocket section.