We released an updated version of the Apigee hybrid software, v1.10.0

## Announcement ### hybrid v1.10 On June 30, 2023 we released an updated version of the Apigee hybrid software, v1.10.0. * For information on upgrading, see [Upgrading Apigee hybrid to version 1.10](https://cloud.google.com/apigee/docs/hybrid/v1.10/upgrade). * For information on new installations, see [The big picture](https://cloud.google.com/apigee/docs/hybrid/v1.10/big-picture). ## Feature **Pre-install Cluster Check Kubernetes job** Starting in version 1.10, Apigee hybrid offers a new tool that examines the hybrid cluster before you install the hybrid runtime. See [Step 8: Check cluster readiness](https://cloud.google.com/apigee/docs/hybrid/v1.10/install-check-cluster) . ## Feature **Automated Issue Surfacing (AIS)** Starting with Apigee hybrid 1.10, Apigee hybrid offers a new tool that examines the hybrid runtime and surfaces issues by running a `kubectl` command. If the tool detects errors in the cluster, it returns a detailed error message. The error message contains a link to the troubleshooting guide for that specific error. See [Automated issue surfacing](https://cloud.google.com/apigee/docs/hybrid/v1.10/automated-issue-surfacing) and [Configuration property reference, watcher](https://cloud.google.com/apigee/docs/hybrid/v1.10/config-prop-ref#watcher). ## Feature **Support for AppGroups (preview)** Starting in version 1.10, Apigee hybrid supports AppGroups, which represent a relationship between one or more apps that are managed by the same set of people. For information, see [Using AppGroups to organize app ownership](https://cloud.google.com/apigee/docs/api-platform/publish/organizing-client-app-ownership#using-appgroups-to-organize-app-ownership). AppGroups is in preview as of the Apigee hybrid 1.10 release. See the [AppGroups preview launch announcement](https://cloud.google.com/apigee/docs/release/release-notes#June%5F27%5F2023) for details. ## Feature **Support for environment-level scaling** Starting in version 1.9.3, Apigee hybrid added the following environment configuration properties that enable you to specify environment-specific scaling in the `overrides.yaml` file: * [envs\[\].runtime.replicaCountMax](https://cloud.google.com/apigee/docs/hybrid/latest/config-prop-ref#envs-runtime-max) * [envs\[\].runtime.replicaCountMin](https://cloud.google.com/apigee/docs/hybrid/latest/config-prop-ref#envs-runtime-min) * [envs\[\].synchronizer.replicaCountMax](https://cloud.google.com/apigee/docs/hybrid/latest/config-prop-ref#envs-sync-max) * [envs\[\].synchronizer.replicaCountMin](https://cloud.google.com/apigee/docs/hybrid/latest/config-prop-ref#envs-sync-min) * [envs\[\].udca.replicaCountMax](https://cloud.google.com/apigee/docs/hybrid/latest/config-prop-ref#envs-ucda-max) * [envs\[\].udca.replicaCountMin](https://cloud.google.com/apigee/docs/hybrid/latest/config-prop-ref#envs-ucda-min) **Documentation:** [Environment-based scaling](https://cloud.google.com/apigee/docs/hybrid/latest/scale-and-autoscale#environment-based-scaling) ## Fix | Bug ID | Description | | ---------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **181569522** | **You can now create a new environment with the same name as a deleted environment without needing to perform manual clean-up tasks first.** (Fixed in Apigee hybrid v1.8.5 and v1.7.6) | | **209509030** | **Apigee Ingressgateway cannot access K8s secret from another namespace.** | | **218567150** | **The ingress gateway is now configured to consistently preserve UUID in the x-request-id header.** **Note:** This setting does have some impact on tracing in the ingress gateway. For more information, see _pack\_trace\_reason_ in ["UUID (proto)"](https://www.envoyproxy.io/docs/envoy/latest/api-v3/extensions/request%5Fid/uuid/v3/uuid.proto#extensions-request-id-uuid-v3-uuidrequestidconfig) in the envoy documentation. (Fixed in Apigee hybrid v1.7.6 and v1.8.3) | | **223320630** | **mTLS-related client variables are now set by the Apigee runtime.** (Fixed in Apigee hybrid v1.8.6) | | **245619397** | **In Apigee hybrid, fluentbit support now includes the NO\_PROXY environment variable.** (Fixed in Apigee hybrid v1.8.5, v1.8.6, and v1.9.1) | | **259264961** | **Added support for ASM v1.15.** Please see [Known issue 266452840](https://cloud.google.com/apigee/docs/release/known-issues#266452840) (Fixed in Apigee hybrid v1.7.6) | | **260342163** | **Fixed a narrow scenario where threads in runtime pods ended up consuming 100% CPU.** (Fixed in Apigee hybrid v1.9.1) | | **260372012** | **Requests failed with 500 response and keyvaluemap.service.ErrorDuringDecryption error after upgrade to Hybrid 1.8.** **Note:** Fixed in Apigee hybrid 1.8.4 and newer. (Fixed in Apigee hybrid v1.8.5) | | **262699558** | **The watcher component no longer fails when using Kubernetes Secret to store hybrid service account secret.** (Fixed in Apigee hybrid v1.7.6) | | **263840644** | **Fixed a conflict with an existing ASM on the cluster.** (Fixed in Apigee hybrid v1.8.6) | | **265374889** | **Fixed an issue where in some circumstances the Java Callout would to fail due with the following error:** _Failed to execute JavaCallout. Could not initialize class org.jose4j.jwa.AlgorithmFactoryFactory2_. (Fixed in Apigee hybrid v1.9.1) | | **266411394** | **Add support for Azure Front Door request headers to /healthz health check.** (Fixed in Apigee hybrid v1.8.5 and v1.9.1) | | **266594584** | **Websocket was failing in asm 1.15.** This was due to incompatible capitalization in variable names between the Anthos Service Mesh overlay.yaml file and the and the Envoy filter apigee-envoyfilter.yaml file. (Fixed in Apigee hybrid v1.8.5 and v1.9.1) | | **266814873** | **In certain circumstances, retrieving encrypted KVM entries could fail with an error.** This fix ensures that MART will be able to successfully function for environment-scoped KVM entries, even if the encryption key is used in the Org Env configuration or when the keys contain non-UTF8 characters. There is no change to KVM data. (Fixed in Apigee hybrid v1.8.6 and v1.9.1) | | **266989915266919136** | **In some circumstances, Apigee could return incorrect developer credentials for an app, unless the specific app was selected when requesting the credentials.** (Fixed in Apigee hybrid v1.9.1) | | **267666187** | **When using a custom Kubernetes service for the Apigee ingress gateway, you can disable the creation of a default load balancer.** See [Managing Apigee ingress gateway](https://cloud.google.com/apigee/docs/hybrid/v1.8/managing-ingress#disable-loadbalancer). (Fixed in Apigee hybrid v1.8.6 and v1.9.1) | | **267691299265295406** | **The Apigee controller uses a dedicated apigee-manager Kubernetes service account, instead of using the default SA.** (Fixed in Apigee hybrid v1.8.6 and v1.9.1) | | **268445095** | **The validateOrg flag can be set to false to bypass upgrade validation errors when configuration includes HTTP Forward proxy.** You can use this to avoid upgrade errors caused by HTTP proxy settings. (Fixed in Apigee hybrid v1.7.6) | | **268696297** | **Providing a Kubernetes secret for Cassandra and Redis components is now supported.** See [cassandra.auth.secret](https://cloud.google.com/apigee/docs/hybrid/v1.9/config-prop-ref#cassandra-auth-secret) and [redis.auth.secret](https://cloud.google.com/apigee/docs/hybrid/v1.9/config-prop-ref#redis-auth-secret) in the [Configuration properties reference](https://cloud.google.com/apigee/docs/hybrid/v1.9/config-prop-ref). (Fixed in Apigee hybrid v1.9.1) | | **269451743** | **In certain circumstances, upgrading from Apigee hybrid v1.8.3 to v1.9.0 could fail with an error message when creating the virtual hosts.** (Fixed in Apigee hybrid v1.9.1) | | **269738951** | **The example network policies are now included in the apigeectl/examples/network-policies directory.** see [Configuring Kubernetes network policies](https://cloud.google.com/apigee/docs/hybrid/v1.9/kubernetes-network-policies). (Fixed in Apigee hybrid v1.9.1) | | **270371160** | **In Apigee hybrid v1.8.7, we removed certain insecure TLS ciphers.** Apigee hybrid supports the TLS cipher suites supported by the [Boring FIPS build of Envoy](https://www.envoyproxy.io/docs/envoy/v1.21.5/api-v3/extensions/transport%5Fsockets/tls/v3/common.proto.html). You can now specify specific cipher suites with the [virtualhosts.cipherSuites configuration property](https://cloud.google.com/apigee/docs/hybrid/v1.8/config-prop-ref#virtualhosts) in your overrides. (Fixed in Apigee hybrid v1.8.7) **Note**: Apigee hybrid only supports the RSA ciphers listed. ECDSA ciphers are not supported. | | **270371160** | **In Apigee hybrid v1.9.0, we removed certain insecure TLS ciphers.** Apigee hybrid supports the TLS cipher suites supported by the [Boring FIPS build of Envoy](https://www.envoyproxy.io/docs/envoy/v1.21.5/api-v3/extensions/transport%5Fsockets/tls/v3/common.proto.html). You can now specify specific cipher suites with the [virtualhosts.cipherSuites configuration property](https://cloud.google.com/apigee/docs/hybrid/v1.9/config-prop-ref#virtualhosts) in your overrides. (Fixed in Apigee hybrid v1.9.2) **Note**: Apigee hybrid only supports the RSA ciphers listed. ECDSA ciphers are not supported. | | **271266079** | **Removed port 80 from the default Kubernetes service of Apigee Ingress Gateway.** (Fixed in Apigee hybrid v1.8.6 and v1.9.1) | | **272212164** | **Cassandra CSI backup could clash with Azure default configuration.** The CSI backup script has been fixed to prevent a resource naming issue that could cause backups to fail. (Fixed in Apigee hybrid v1.8.7 and v1.9.2) | | **273561434** | **Some projects were unable to run debug sessions..** (Fixed in Apigee hybrid v1.8.8) | | **274292101** | **In certain circumstances, environment-scoped KVMs in hybrid could cause rollback issues for MART.** (Fixed in Apigee hybrid v1.8.6) | | **274999014** | **Restrict watcher RBAC to a single K8s namespace** | | **278646149** | **In certain circumstances, the logger.livenessProbe.timeoutSeconds configuration property was not working as expected.** See [logger.livenessProbe.timeoutSeconds](https://cloud.google.com/apigee/docs/hybrid/v1.8/config-prop-ref#logger-livenessprobe-timeoutseconds) in the Configuration property reference. (Fixed in Apigee hybrid v1.8.7 and v1.9.2) | | **279053612** | **x-forwarded-client-cert (XFCC) HTTP headers handled with the istiod.forwardClientCertDetails configuration property.** (Fixed in Apigee hybrid v1.8.7 and v1.9.2) See the Configuration properties reference for details: v1.8: [istiod.forwardClientCertDetails](https://cloud.google.com/apigee/docs/hybrid/v1.8/config-prop-ref#istiod-forwardclientcertdetails)v1.9: [istiod.forwardClientCertDetails](https://cloud.google.com/apigee/docs/hybrid/v1.9/config-prop-ref#istiod-forwardclientcertdetails) | | **279193831** | **Envoy has been updated to v1.25.6..** (Fixed in Apigee hybrid v1.8.8) | | **279712107** | **Added the ability to annotate apigee-ingressgateway-manager pods through overrides.yaml file.** (Fixed in Apigee hybrid v1.8.8) | | **280544499** | **Request headers were not seen in debug sessions.** (Fixed in Apigee hybrid v1.8.8) | | **284488296** | **Removed an unneeded Workload Identify on the Cassandra Schema Validation cron job.** (Fixed in Apigee hybrid v1.8.8 and v1.9.3) | ## Breaking | Bug ID | Description | | ------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **270371160** | **In Apigee hybrid v1.9.0, we removed certain insecure TLS ciphers.** Apigee hybrid supports the TLS cipher suites supported by the Boring FIPS build of Envoy.**Note:** Apigee hybrid only supports the RSA ciphers listed. ECDSA ciphers are not supported. | | **271266079** | **Removed port 80 from the default Kubernetes service of Apigee Ingress Gateway.** Port 80 is not supported by Apigee ingress gateway. If you are migrating from ASM to Apigee ingress gateway, and followed the instructions in the [community post](https://www.googlecloudcommunity.com/gc/Cloud-Product-Articles/Apigee-hybrid-ingress-Three-different-options-to-expose-your/ta-p/79149) to enable Port 80, it will not work with Apigee Ingress gateway. (Fixed in Apigee hybrid v1.8.6 and v1.9.1) | ## Security | Bug ID | Description | | ---------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **262576079** | **Security fix for for apigee-envoy.** (Fixed in Apigee hybrid v1.10)This addresses the following vulnerability: [CVE-2022-23806](https://nvd.nist.gov/vuln/detail/CVE-2022-23806) | | **273797045** | **Security fix for for apigee-diagnostics-collector apigee-synchronizer apigee-udca.** (Fixed in Apigee hybrid v1.8.8)This addresses the following vulnerability: [CVE-2021-22573](https://nvd.nist.gov/vuln/detail/CVE-2021-22573) | | **273800345**, **281572616** | **Security fixes for apigee-diagnostics-collector, apigee-mart-server, apigee-mint-task-scheduler, apigee-runtime, apigee-synchronizer, and apigee-udca.** (Fixed in Apigee hybrid v1.8.8 and v1.9.3This addresses the following vulnerabilities: [CVE-2022-3510](https://nvd.nist.gov/vuln/detail/CVE-2022-3510)[CVE-2022-3509](https://nvd.nist.gov/vuln/detail/CVE-2022-3509) [CVE-2022-3171](https://nvd.nist.gov/vuln/detail/CVE-2022-3171) | | **273800717** | **Security fixes for apigee-emulator, apigee-diagnostics-collector, apigee-mart-server, apigee-mint-task-scheduler, apigee-mock-server, apigee-runtime, and apigee-synchronizer.** (Fixed in Apigee hybrid v1.8.7 and v1.9.2)This addresses the following vulnerabilities: [CVE-2022-46364](https://nvd.nist.gov/vuln/detail/CVE-2022-46364) [CVE-2022-46363](https://nvd.nist.gov/vuln/detail/CVE-2022-46363) | | **273800965** | **Security fix for apigee-diagnostics-collector, apigee-mart-server, apigee-mint-task-scheduler, apigee-runtime, and apigee-synchronizer.** (Fixed in Apigee hybrid v1.8.7, v1.9.2, and v1.9.3)This addresses the following vulnerability: [CVE-2019-10172](https://nvd.nist.gov/vuln/detail/CVE-2019-10172) | | **273801301** | **Security fixes for apigee-mart-server and apigee-runtime.**(Fixed in Apigee hybrid v1.8.8 and v1.9.3)This addresses the following vulnerability: [CVE-2020-13936](https://nvd.nist.gov/vuln/detail/CVE-2020-13936) | | **274112103** | **Security fixes to the Apigee Controller and Apigee Watcher.** (Fixed in Apigee hybrid v1.8.6 and v1.9.1)This addresses the following vulnerabilities: [CVE-2022-1996](https://nvd.nist.gov/vuln/detail/CVE-2022-1996) [CVE-2022-27191](https://nvd.nist.gov/vuln/detail/CVE-2022-27191) [CVE-2022-27664](https://nvd.nist.gov/vuln/detail/CVE-2022-27664)[CVE-2022-32149](https://nvd.nist.gov/vuln/detail/CVE-2022-32149)[CVE-2022-41723](https://nvd.nist.gov/vuln/detail/CVE-2022-41723) | | **275002360** | **Security fixes for fluent-bit.** (Fixed in Apigee hybrid v1.8.6 and v1.9.1)This addresses the following vulnerabilities: [CVE-2021-46848](https://nvd.nist.gov/vuln/detail/CVE-2021-46848) [CVE-2022-1304](https://nvd.nist.gov/vuln/detail/CVE-2022-1304)[CVE-2022-2097](https://nvd.nist.gov/vuln/detail/CVE-2022-2097) [CVE-2022-42898](https://nvd.nist.gov/vuln/detail/CVE-2022-42898) | | **277367440** | **Security fixes for Apigee Controller, Watcher, and apigeectl.** (Fixed in Apigee hybrid v1.8.7 and v1.9.2)This addresses the following vulnerabilities: [CVE-2022-41723](https://nvd.nist.gov/vuln/detail/CVE-2022-41723) [CVE-2022-41717](https://nvd.nist.gov/vuln/detail/CVE-2022-41717) [CVE-2022-28948](https://nvd.nist.gov/vuln/detail/CVE-2022-28948) | | **278313047** | **Security fixes for apigee-stackdriver-logging-agent.** (Fixed in Apigee hybrid v1.9.2)This addresses the following vulnerabilities: [CVE-2022-32511](https://nvd.nist.gov/vuln/detail/CVE-2022-32511) [CVE-2022-29181](https://nvd.nist.gov/vuln/detail/CVE-2022-29181) [CVE-2022-24836](https://nvd.nist.gov/vuln/detail/CVE-2022-24836) [CVE-2022-0759](https://nvd.nist.gov/vuln/detail/CVE-2022-0759) [CVE-2021-41817](https://nvd.nist.gov/vuln/detail/CVE-2021-41817) [CVE-2021-41098](https://nvd.nist.gov/vuln/detail/CVE-2021-41098) [CVE-2021-32740](https://nvd.nist.gov/vuln/detail/CVE-2021-32740) [CVE-2021-28965](https://nvd.nist.gov/vuln/detail/CVE-2021-28965) [CVE-2020-8130](https://nvd.nist.gov/vuln/detail/CVE-2020-8130) [CVE-2020-25613](https://nvd.nist.gov/vuln/detail/CVE-2020-25613) [CVE-2019-3881](https://nvd.nist.gov/vuln/detail/CVE-2019-3881) | | **279194142** | **Fixes build issues to achieve FIPS compliance.** (Fixed in Apigee hybrid v1.8.7 and v1.9.2) | | **281561243** | **Security fix for apigee-diagnostics-collector, apigee-mint-task-scheduler, apigee-runtime, and apigee-synchronizer.** (Fixed in Apigee hybrid v1.8.8 and v1.9.3)This addresses the following vulnerability: [CVE-2022-1471](https://nvd.nist.gov/vuln/detail/CVE-2022-1471) | | **283826216** | **Security fixes for apigee-ingressgateway.** (Fixed in Apigee hybrid v1.9.3)This addresses the following vulnerabilities: [CVE-2022-41723](https://nvd.nist.gov/vuln/detail/CVE-2022-41723) [CVE-2022-41721](https://nvd.nist.gov/vuln/detail/CVE-2022-41721) | | **283826785** | **Security fixes for istiod.** (Fixed in Apigee hybrid v1.9.3)This addresses the following vulnerabilities: [CVE-2023-25165](https://nvd.nist.gov/vuln/detail/CVE-2023-25165) [CVE-2022-41723](https://nvd.nist.gov/vuln/detail/CVE-2022-41723) [CVE-2022-41721](https://nvd.nist.gov/vuln/detail/CVE-2022-41721) [CVE-2022-23526](https://nvd.nist.gov/vuln/detail/CVE-2022-23526) [CVE-2022-23525](https://nvd.nist.gov/vuln/detail/CVE-2022-23525) [CVE-2022-23524](https://nvd.nist.gov/vuln/detail/CVE-2022-23524) |