EMR on EKS now supports custom job scheduling
Share
Services
We are excited to announce the addition of Volcano and Apache Yunikorn as job schedulers when running EMR on EKS using Spark operator and spark-submit. [Amazon EMR on EKS](https://aws.amazon.com/emr/features/eks/) enables customers to run open-source big data frameworks such as Apache Spark on Amazon EKS. Using a custom job scheduler for Spark jobs enables fine-grained capacity management and faster pod provisioning at scale.
The default Kubernetes scheduler handles the placement of individual pods, while maintaining constraints such as available capacity, resource requests and limits, and node affinity. However, it does not support scheduling based on jobs. With this new feature, customers have the option to use Apache Yunikorn and Volcano to schedule EMR on EKS Spark jobs with Spark operator and spark-submit. Customers can leverage features like gang scheduling, queue management, preemption, fair-share scheduling in these schedulers, which helps to deliver high scheduling throughput and optimized capacity.
To learn more about this feature, please visit the custom job scheduling section of [our documentation](https://docs.aws.amazon.com/emr/latest/EMR-on-EKS-DevelopmentGuide/tutorial-volcano.html). Volcano and Apache Yunikorn as Kubernetes custom scheduler are supported on Amazon EMR on EKS 6.11 release and above, and available in regions where Amazon EMR on EKS is [currently available](https://docs.aws.amazon.com/emr/latest/EMR-on-EKS-DevelopmentGuide/service-quotas.html).
What else is happening at Amazon Web Services?
Amazon AppStream 2.0 users can now save their user preferences between streaming sessions
December 13th, 2024
Services
Share
AWS Elemental MediaConnect Gateway now supports source-specific multicast
December 13th, 2024
Services
Share
Amazon EC2 instances support bandwidth configurations for VPC and EBS
December 13th, 2024
Services
Share
AWS announces new AWS Direct Connect location in Osaka, Japan
December 13th, 2024
Services
Share
Amazon DynamoDB announces support for FIPS 140-3 interface VPC and Streams endpoints
December 13th, 2024
Services
Share