GKE on Bare Metal 1.29.0-gke.1449 is now available for download

## Announcement ### Release 1.29.0-gke.1449 GKE on Bare Metal 1.29.0-gke.1449 is now available for [download](https://cloud.google.com/anthos/clusters/docs/bare-metal/latest/downloads). To upgrade, see [Upgrade clusters](https://cloud.google.com/anthos/clusters/docs/bare-metal/latest/how-to/upgrade). GKE on Bare Metal 1.29.0-gke.1449 runs on Kubernetes 1.29. If you use a third-party storage vendor, check the [GDCV Ready storage partners](https://cloud.google.com/anthos/docs/resources/partner-storage) document to make sure the storage vendor has already passed the qualification for this release of GKE on Bare Metal. ## Announcement **Version 1.15 end of life**: In accordance with the [Version Support Policy](https://cloud.google.com//anthos/docs/support/getting-support#version%5Fsupport%5Fpolicy), version 1.15 (all patch releases) of GKE on Bare Metal has reached its end of life and is no longer supported. ## Feature * **GA**: Support [GKE Identity Service v2 capability](https://cloud.google.com/kubernetes-engine/enterprise/identity/setup/user-access#alternativeuserloginaccess) for an improved security flow when you authenticate with third-party identity solutions. The GA offering of GKE Identity Service v2 has the following requirements and restrictions: * GKE Identity Service v2 now requires ports `11001` and `11002` on the control plane load balancer nodes, instead of `8443` and `8444`. Ensure these ports are open and available before you upgrade a cluster to version 1.29.0-gke.1449 and higher. If the ports aren't open, upgrade preflight checks fail. * GKE Identity Service v2 requires version 1.5.1 or higher of the Anthos Auth gcloud CLI component. If necessary, update the Anthos Auth component (`gcloud components update anthos-auth`). If you use the Google Cloud SDK, updating the SDK (`gcloud components update`) to version 474.0.0 or later also updates the Anthos Auth component to the required version. * GKE Identity Service v2 doesn't work with GKE on Bare Metal clusters with the following configurations: * Clusters with a single control plane node only. * Clusters that use control plane nodes for load balancing. That is, clusters that aren't configured with either a separate load balancing node pool or manual load balancing. * **GA**: Added support for skews of up to two minor versions for [selective node pool upgrades](https://cloud.google.com/anthos/clusters/docs/bare-metal/latest/how-to/upgrade-lifecycle#np%5Fversion%5Frules). * **GA**: Added capability to [pause and resume cluster upgrades](https://cloud.google.com/anthos/clusters/docs/bare-metal/latest/how-to/upgrade#pause%5Fresume). * **GA**: Maintenance mode now uses eviction-based draining for nodes, instead of taint-based draining. Eviction-based draining uses the Eviction API, which honors Pod Disruption Budgets (PDBs). Draining nodes this way provides better protection against workload disruptions. * **Preview**: Added support for node-level private registry configuration for workload images. * **Preview**: Added support for rolling back [select node pool upgrades](https://cloud.google.com/anthos/clusters/docs/bare-metal/latest/how-to/upgrade#select%5Fnp%5Fup). * **Preview**: Added support for admin and hybrid clusters to manage multiple versions user clusters concurrently. * **Preview**: Added support for using an intermediate Certificate Authority (CA) as the cluster root CA. * **Preview**: Added support to route workload logs to a third-party custom Kafka destination. This capability isn't enabled by default. You enable this capability in the cluster `stackdriver` resource spec by adding the `unmanagedKafkaOutputConfig` section. This section lets you specify the IP addresses of Kafka message brokers (`brokers`), topic names (`topics`), and keys to map the topics to partitions (`topicKeys`). * Improved command-line interface errors and [error documentation](https://cloud.google.com/distributed-cloud/docs/reference/gke-error-ref). ## Change **Functionality changes:** * GKE Identity Service v2 now sends extra parameters (`extraParams`) to your OIDC provider. * Extra node viewing permissions are added for accounts specified with the `spec.clusterSecurity.authorization.clusterViewer.gcpAccounts` field in the Cluster resource. * Added `Status.Available` field to `BareMetalMachine` resources to indicate whether the machine is available. * Updated preflight checks add a check for networking kernel modules (`ip_tables` or `np_tables`) and remove the `iptables` package check. * The Google plugin for the GKE Identity Service now caches the public keys based on `max-age` in `cache-control` header. ## Fix **Fixes:** * Fixed an issue where the kubelet doesn't honor shortened, 1-second grace period for pod deletion during eviction-based draining. * Fixed a cluster upgrade issue where the `lifecycle-controller-deployer` Pod was unable to migrate existing GKE on Bare Metal resources to the latest API version. This issue blocked upgrades to earlier version 1.28 releases. * Fixed an issue with configuring a proxy for your cluster that required you to manually set `HTTPS_PROXY` and `NO_PROXY` environment variables on the admin workstation. * Fixed an issue where upgrades are blocked because `cluster-operator` can't delete stale, failing preflight check resources. * Fixed an issue where the network check ConfigMap wasn't updated when nodes were added or removed. ## Fix The following container image security vulnerabilities have been fixed in version 1.29.0-gke.1449: * Critical container vulnerabilities: * [CVE-2021-38297](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-38297) * [CVE-2022-23806](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2022-23806) * [CVE-2023-24538](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-24538) * [CVE-2023-24540](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-24540) * [CVE-2023-25775](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-25775) * [CVE-2023-29402](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-29402) * [CVE-2023-29404](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-29404) * [CVE-2023-29405](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-29405) * High-severity container vulnerabilities: * [CVE-2020-29652](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-29652) * [CVE-2021-29923](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-29923) * [CVE-2021-33195](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-33195) * [CVE-2021-33196](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-33196) * [CVE-2021-33198](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-33198) * [CVE-2021-39293](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-39293) * [CVE-2021-41771](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-41771) * [CVE-2021-41772](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-41772) * [CVE-2021-44716](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-44716) * [CVE-2022-2879](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2022-2879) * [CVE-2022-2880](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2022-2880) * [CVE-2022-21698](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2022-21698) * [CVE-2022-23772](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2022-23772) * [CVE-2022-23773](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2022-23773) * [CVE-2022-24675](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2022-24675) * [CVE-2022-24921](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2022-24921) * [CVE-2022-28131](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2022-28131) * [CVE-2022-28327](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2022-28327) * [CVE-2022-28948](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2022-28948) * [CVE-2022-30580](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2022-30580) * [CVE-2022-30630](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2022-30630) * [CVE-2022-30631](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2022-30631) * [CVE-2022-30632](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2022-30632) * [CVE-2022-30633](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2022-30633) * [CVE-2022-30635](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2022-30635) * [CVE-2022-32189](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2022-32189) * [CVE-2022-41715](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2022-41715) * [CVE-2022-41724](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2022-41724) * [CVE-2022-41725](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2022-41725) * [CVE-2023-5717](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-5717) * [CVE-2023-6040](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-6040) * [CVE-2023-6356](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-6356) * [CVE-2023-6536](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-6536) * [CVE-2023-6606](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-6606) * [CVE-2023-6931](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-6931) * [CVE-2023-6932](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-6932) * [CVE-2023-24534](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-24534) * [CVE-2023-24536](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-24536) * [CVE-2023-24537](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-24537) * [CVE-2023-24539](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-24539) * [CVE-2023-29400](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-29400) * [CVE-2023-29403](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-29403) * [CVE-2023-29499](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-29499) * [CVE-2023-35827](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-35827) * [CVE-2023-46838](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-46838) * [CVE-2023-51780](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-51780) * [CVE-2023-51781](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-51781) * [CVE-2023-51782](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-51782) * [CVE-2023-52436](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-52436) * [CVE-2023-52439](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-52439) * [CVE-2023-52444](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-52444) * [CVE-2023-52445](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-52445) * [CVE-2023-52451](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-52451) * [CVE-2023-52464](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-52464) * [CVE-2023-52469](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-52469) * [CVE-2024-1086](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2024-1086) * [CVE-2024-26586](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2024-26586) * [CVE-2024-26597](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2024-26597) * [CVE-2024-26598](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2024-26598) * Medium-severity container vulnerabilities: * [CVE-2020-29509](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-29509) * [CVE-2020-29511](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-29511) * [CVE-2021-33197](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-33197) * [CVE-2021-34558](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-34558) * [CVE-2021-36221](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-36221) * [CVE-2021-44879](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-44879) * [CVE-2022-1705](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2022-1705) * [CVE-2022-1962](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2022-1962) * [CVE-2022-32148](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2022-32148) * [CVE-2022-41717](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2022-41717) * [CVE-2023-3446](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-3446) * [CVE-2023-3817](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-3817) * [CVE-2023-6004](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-6004) * [CVE-2023-6121](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-6121) * [CVE-2023-6915](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-6915) * [CVE-2023-6918](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-6918) * [CVE-2023-24532](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-24532) * [CVE-2023-29406](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-29406) * [CVE-2023-29409](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-29409) * [CVE-2023-32611](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-32611) * [CVE-2023-32665](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-32665) * [CVE-2023-34324](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-34324) * [CVE-2023-39198](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-39198) * [CVE-2023-39804](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-39804) * [CVE-2023-45863](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-45863) * [CVE-2023-46218](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-46218) * [CVE-2023-46343](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-46343) * [CVE-2023-49290](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-49290) * [CVE-2023-52443](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-52443) * [CVE-2023-52449](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-52449) * [CVE-2023-52470](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-52470) * [CVE-2024-21664](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2024-21664) * [CVE-2024-28085](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2024-28085) * [GHSA-2c7c-3mj9-8fqh](https://github.com/advisories/GHSA-2c7c-3mj9-8fqh) * Low-severity container vulnerabilities: * [CVE-2021-25743](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-25743) * [CVE-2022-30629](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2022-30629) * [CVE-2023-26604](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-26604) * [CVE-2023-2975](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-2975) * [CVE-2023-5178](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-5178) * [CVE-2023-5197](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-5197) * [CVE-2023-6531](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-6531) * [CVE-2023-6817](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-6817) * [CVE-2023-46813](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-46813) * [CVE-2023-46862](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-46862) * [CVE-2023-52438](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-52438) * [CVE-2023-52448](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-52448) * [CVE-2023-52454](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-52454) * [CVE-2023-52456](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-52456) * [CVE-2023-52457](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-52457) * [CVE-2023-52462](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-52462) * [CVE-2023-52463](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-52463) * [CVE-2023-52467](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-52467) * [CVE-2023-52503](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-52503) * [CVE-2023-52513](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-52513) * [CVE-2023-52524](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-52524) * [CVE-2023-52564](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-52564) * [CVE-2023-52573](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-52573) * [CVE-2023-52575](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-52575) * [CVE-2024-0193](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2024-0193) * [CVE-2024-0641](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2024-0641) * [CVE-2024-0646](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2024-0646) * [CVE-2024-24860](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2024-24860) ## Issue **Known issues:** * Clusters that use bundled load balancing with BGP might have [performance degradation as the total number of Services of type LoadBalancer approaches 2,000](https://cloud.google.com/anthos/clusters/docs/bare-metal/latest/troubleshooting/known-issues#perf-degrad-bgp-lb). For information about the latest known issues, see [GKE on Bare Metal known issues](https://cloud.google.com/anthos/clusters/docs/bare-metal/latest/troubleshooting/known-issues) in the Troubleshooting section.