Anthos clusters on bare metal 1.28.0-gke.435 is now available for download

## Feature ### Release 1.28.0-gke.435 Anthos clusters on bare metal 1.28.0-gke.435 is now available for [download](https://cloud.google.com/anthos/clusters/docs/bare-metal/1.28/downloads). To upgrade, see [Upgrading Anthos on bare metal](https://cloud.google.com/anthos/clusters/docs/bare-metal/1.28/how-to/upgrade). Anthos clusters on bare metal 1.28.0-gke.435 runs on Kubernetes 1.28. ## Announcement **Version alignment** For easier identification of the Kubernetes version for a given release, we are[aligning Anthos clusters on bare metal version numbering with GKE version numbering](https://cloud.google.com/anthos/docs/version-alignment). This change starts with this minor release, which is version 1.28\. The version alignment is for major and minor versions only, patch versions are product specific. In addition to this version alignment, the Anthos clusters on bare metal release versions will follow the [GKE semantic versioning scheme](https://cloud.google.com/kubernetes-engine/versioning#versioning%5Fscheme)(x.y.z-gke.N), including the addition of a GKE patch version (-gke.N). Unlike GKE, however, the patch version (z) increments by 100. **Example version numbers for Anthos clusters on bare metal:** * Minor release: 1.28.0-gke.435 * Initial patch release: 1.28.100-gke.27 * Second patch release: 1.28.200-gke.19 This change affects numbering only. Upgrades from 1.16 to 1.28 follow the same process as upgrades between prior minor releases. However, downloads, upgrades, and cluster creation for 1.28 and higher versions require the fully qualified version number, including the GKE patch version. ## Feature * **Preview**: Added support for skews of up to two minor versions for [selective node pool upgrades](https://cloud.google.com/anthos/clusters/docs/bare-metal/latest/how-to/upgrade-lifecycle#np%5Fversion%5Frules). * **Preview**: Added capability to [pause and resume cluster upgrades](https://cloud.google.com/anthos/clusters/docs/bare-metal/latest/how-to/upgrade#pause%5Fresume). * **GA**: Added support for [using custom cluster certificate authorities (CAs)](https://cloud.google.com/anthos/clusters/docs/bare-metal/1.16/how-to/custom-cluster-ca) to enable secure authentication and encryption between cluster components. * **GA**: Added support for using [gkeConnect.location](https://cloud.google.com/anthos/clusters/docs/bare-metal/latest/reference/cluster-config-ref#gkeconnect-location) to specify regional membership for fleets. * **Preview**: Added support for using [controlPlane.apiServerCertExtraSANs](https://cloud.google.com/anthos/clusters/docs/bare-metal/latest/reference/cluster-config-ref#controlplane-apiservercertextrasans) to specify extra subject alternative name (SAN) entries for the Kubernetes API server certificate extra. * **GA**: Added support for [enabling Direct Server Return (DSR) load balancing](https://cloud.google.com/anthos/clusters/docs/bare-metal/latest/how-to/lb-network-mode) for clusters configured with flat-mode networking. In GA, DSR load balancing is enabled with the `clusterNetwork.forwardMode` field in the cluster configuration file. * **GA**: Added support for multiple BGP load balancer (`BGPLoadBalancer`) resources and BGP Community. Multiple BGP load balancer resources provide more flexibility to define which peers advertise specific load balancer nodes and Services. BGP Community support helps you to distinguish routes coming from BGP load balancers from other routes in your network. * **Preview**: Added GKE Identity Service v2 capability for an improved security flow when you authenticate with third-party identity solutions. **Warning:** GKE Identity Service v2 requires ports 8443 and 8444 on the control plane load balancer nodes. Ensure these ports are open and available before you upgrade a cluster to version 1.28.0-gke.435 and higher. If the ports aren't open, upgrade preflight checks fail. ## Change **Functionality changes:** * Configured the local volume provisioner DaemonSet to tolerate all taints. * Updated the SRIOV operator. * To improve logging system integration, updated audit logging to always write a local Kubernetes audit log file, even when Cloud Audit Logging is enabled. * Changed upgrade preflight checks behavior to skip kubeadm job creation check to improve upgrade reliability. * Updated method for providing certificates so that they support the server name indication (SNI) extension of the TLS protocol handshake. * Updated Dataplane V2 to use Cilium v1.13. * Added preflight check for control planes running RHEL 9.2 or Ubuntu 22.04 to check the `fs.inotify` kernel settings. * Removed hardcoded timeout value for `bmctl backup` operation. * Updated certificate management to propagate `private-registry-certs` Secret changes to all machines. * Added support for SSH client certificates in `bmctl backup` and `bmctl restore` commands. * Added the optional `userClaim` field to the ClientConfig custom resource definition bundled with Anthos clusters on bare metal. This change improves support for Azure AD integrations with Anthos Identity Service. * Updated constraint on NodePool `spec.upgradeStrategy.concurrentNodes` to be the smaller of either 15 nodes or 50% of the size of the node pool. ## Change **Supported node pool versions:** If you use [selective worker node pool upgrades](https://cloud.google.com/anthos/clusters/docs/bare-metal/1.28/how-to/upgrade#select%5Fnp%5Fup) to upgrade a cluster to version 1.28.0-gke.435, see [Node pool versioning rules](https://cloud.google.com/anthos/clusters/docs/bare-metal/1.16/how-to/upgrade-lifecycle#supported%5Fnodepool%5Fversions) for a list of the versions that are supported for the worker node pools. ## Fix **Fixes:** * Fixed an issue where the `node-problem-detector` `systemd` service doesn't restart after the node reboots. * Fixed an issue where `CoreDNS` Pods can get stuck in an unready state. * Fixed an issue that caused application metrics to be unavailable in Anthos clusters on bare metal versions 1.16.0 and 1.16.1. * Fixed a memory leak in Dataplane V2. * Fixed an issue that caused file and directory permissions to be set incorrectly after backing up and restoring a cluster. * Added direct dependencies on systemd, containerd, and kubelet over their mount point folders in `/var/lib/`. * Fixed an issue that blocked upgrades to version 1.16 for clusters that have secure computing mode (`seccomp`) disabled. * Fixed an issue where etcd blocked upgrades due to an incorrect initial-cluster-state. * Fixed an issue that sometimes resulted in the upgrade process starting before either all pods have been drained or the draining period has elapsed. * Fixed an issue that resulted in the etcd-events memory request (`resources.requests.memory`) being set incorrectly. ## Fix The following container image security vulnerabilities have been fixed in version 1.28.0-gke-435: * Critical container vulnerabilities: * [CVE-2022-1996](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2022-1996) * [CVE-2023-38408](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-38408) * [CVE-2023-45871](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-45871) * High-severity container vulnerabilities: * [CVE-2017-11468](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2017-11468) * [CVE-2019-11253](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-11253) * [CVE-2019-13509](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-13509) * [CVE-2020-7919](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-7919) * [CVE-2020-8558](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-8558) * [CVE-2020-9283](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-9283) * [CVE-2021-3121](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-3121) * [CVE-2020-16845](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-16845) * [CVE-2020-28362](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-28362) * [CVE-2020-28366](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-28366) * [CVE-2020-28367](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-28367) * [CVE-2021-20206](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-20206) * [CVE-2021-27918](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-27918) * [CVE-2022-39189](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2022-39189) * [CVE-2022-41721](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2022-41721) * [CVE-2023-1380](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-1380) * [CVE-2023-1989](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-1989) * [CVE-2023-2007](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-2007) * [CVE-2023-2124](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-2124) * [CVE-2023-2253](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-2253) * [CVE-2023-3090](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-3090) * [CVE-2023-3111](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-3111) * [CVE-2023-3268](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-3268) * [CVE-2023-3390](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-3390) * [CVE-2023-3609](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-3609) * [CVE-2023-3611](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-3611) * [CVE-2023-3776](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-3776) * [CVE-2023-4206](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-4206) * [CVE-2023-4207](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-4207) * [CVE-2023-4208](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-4208) * [CVE-2023-4244](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-4244) * [CVE-2023-4622](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-4622) * [CVE-2023-4623](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-4623) * [CVE-2023-4921](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-4921) * [CVE-2023-21255](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-21255) * [CVE-2023-27561](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-27561) * [CVE-2023-28840](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-28840) * [CVE-2023-29002](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-29002) * [CVE-2023-34319](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-34319) * [CVE-2023-35001](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-35001) * [CVE-2023-35788](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-35788) * [CVE-2023-40283](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-40283) * [CVE-2023-42753](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-42753) * [GHSA-74fp-r6jw-h4mp](https://github.com/advisories/GHSA-74fp-r6jw-h4mp) * Medium-severity container vulnerabilities: * [CVE-2015-3627](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2015-3627) * [CVE-2019-11250](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-11250) * [CVE-2019-11251](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-11251) * [CVE-2019-11254](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-11254) * [CVE-2019-19794](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-19794) * [CVE-2020-8554](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-8554) * [CVE-2020-8555](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-8555) * [CVE-2020-8561](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-8561) * [CVE-2020-8564](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-8564) * [CVE-2020-8565](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-8565) * [CVE-2020-8569](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-8569) * [CVE-2020-8911](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-8911) * [CVE-2020-14039](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-14039) * [CVE-2020-14040](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-14040) * [CVE-2020-15586](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-15586) * [CVE-2020-21047](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-21047) * [CVE-2020-24553](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-24553) * [CVE-2020-29510](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-29510) * [CVE-2021-3114](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-3114) * [CVE-2021-3507](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-3507) * [CVE-2021-3930](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-3930) * [CVE-2021-20196](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-20196) * [CVE-2021-20329](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-20329) * [CVE-2021-25735](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-25735) * [CVE-2021-25736](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-25736) * [CVE-2022-0216](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2022-0216) * [CVE-2022-2582](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2022-2582) * [CVE-2022-4269](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2022-4269) * [CVE-2022-40982](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2022-40982) * [CVE-2022-46146](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2022-46146) * [CVE-2023-0330](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-0330) * [CVE-2023-1206](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-1206) * [CVE-2023-2002](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-2002) * [CVE-2023-2269](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-2269) * [CVE-2023-3180](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-3180) * [CVE-2023-3212](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-3212) * [CVE-2023-3338](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-3338) * [CVE-2023-3772](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-3772) * [CVE-2023-3863](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-3863) * [CVE-2023-4132](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-4132) * [CVE-2023-4194](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-4194) * [CVE-2023-4273](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-4273) * [CVE-2023-20569](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-20569) * [CVE-2023-20593](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-20593) * [CVE-2023-27593](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-27593) * [CVE-2023-27594](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-27594) * [CVE-2023-27595](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-27595) * [CVE-2023-28841](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-28841) * [CVE-2023-28842](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-28842) * [CVE-2023-30851](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-30851) * [CVE-2023-31084](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-31084) * [CVE-2023-37453](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-37453) * [CVE-2023-39189](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-39189) * [CVE-2023-39192](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-39192) * [CVE-2023-39193](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-39193) * [CVE-2023-39194](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-39194) * [CVE-2023-39347](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-39347) * [CVE-2023-40577](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-40577) * [CVE-2023-41333](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-41333) * [CVE-2023-41913](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-41913) * [CVE-2023-42754](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-42754) * [CVE-2023-42755](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-42755) * [GHSA-2w8w-qhg4-f78j](https://github.com/advisories/GHSA-2w8w-qhg4-f78j) * [GHSA-76wf-9vgp-pj7w](https://github.com/advisories/GHSA-76wf-9vgp-pj7w) * Low-severity container vulnerabilities: * [CVE-2017-16516](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2017-16516) * [CVE-2020-8562](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-8562) * [CVE-2020-8912](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-8912) * [CVE-2020-14394](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-14394) * [CVE-2021-20203](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-20203) * [CVE-2021-25740](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-25740) * [CVE-2021-32292](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-32292) * [CVE-2022-24795](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2022-24795) * [CVE-2022-45886](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2022-45886) * [CVE-2022-45887](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2022-45887) * [CVE-2022-45919](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2022-45919) * [CVE-2022-48554](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2022-48554) * [CVE-2023-1544](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-1544) * [CVE-2023-2156](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-2156) * [CVE-2023-2898](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-2898) * [CVE-2023-3141](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-3141) * [CVE-2023-3301](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-3301) * [CVE-2023-3354](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-3354) * [CVE-2023-3389](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-3389) * [CVE-2023-3610](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-3610) * [CVE-2023-3773](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-3773) * [CVE-2023-3777](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-3777) * [CVE-2023-4004](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-4004) * [CVE-2023-4147](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-4147) * [CVE-2023-6176](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-6176) * [CVE-2023-21400](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-21400) * [CVE-2023-31248](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-31248) * [CVE-2023-33460](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-33460) * [CVE-2023-34242](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-34242) * [CVE-2023-34256](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-34256) * [CVE-2023-35823](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-35823) * [CVE-2023-35824](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-35824) * [CVE-2023-35828](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-35828) * [CVE-2023-35829](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-35829) * [CVE-2023-41332](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-41332) * [CVE-2023-42756](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-42756) * [GHSA-qq97-vm5h-rrhg](https://github.com/advisories/GHSA-qq97-vm5h-rrhg) ## Issue **Known issues:** For information about the latest known issues, see [Anthos clusters on bare metal known issues](https://cloud.google.com/anthos/clusters/docs/bare-metal/1.28/troubleshooting/known-issues) in the Troubleshooting section.