EMR on EKS now supports custom job scheduling
Share
Services
We are excited to announce the addition of Volcano and Apache Yunikorn as job schedulers when running EMR on EKS using Spark operator and spark-submit. [Amazon EMR on EKS](https://aws.amazon.com/emr/features/eks/) enables customers to run open-source big data frameworks such as Apache Spark on Amazon EKS. Using a custom job scheduler for Spark jobs enables fine-grained capacity management and faster pod provisioning at scale.
The default Kubernetes scheduler handles the placement of individual pods, while maintaining constraints such as available capacity, resource requests and limits, and node affinity. However, it does not support scheduling based on jobs. With this new feature, customers have the option to use Apache Yunikorn and Volcano to schedule EMR on EKS Spark jobs with Spark operator and spark-submit. Customers can leverage features like gang scheduling, queue management, preemption, fair-share scheduling in these schedulers, which helps to deliver high scheduling throughput and optimized capacity.
To learn more about this feature, please visit the custom job scheduling section of [our documentation](https://docs.aws.amazon.com/emr/latest/EMR-on-EKS-DevelopmentGuide/tutorial-volcano.html). Volcano and Apache Yunikorn as Kubernetes custom scheduler are supported on Amazon EMR on EKS 6.11 release and above, and available in regions where Amazon EMR on EKS is [currently available](https://docs.aws.amazon.com/emr/latest/EMR-on-EKS-DevelopmentGuide/service-quotas.html).
What else is happening at Amazon Web Services?
Amazon Connect now supports multi-day copy and paste of agent schedules
about 21 hours ago
Services
Share
Amazon Connect can now generate forecast for workloads with as little as one contact
about 21 hours ago
Services
Share
Amazon WorkSpaces now supports file transfer between WorkSpaces sessions and local devices
about 21 hours ago
Services
Share
Amazon EC2 now supports Optimize CPUs post instance launch
about 21 hours ago
Services
Share
Amazon Connect Contact Lens supports new read-only permissions for reports and dashboards
about 21 hours ago
Services
Share