Amazon EMR releases 6.10.0 and higher support the Kubernetes operator for Apache Spark on EKS. However , it was challenging to deploy our structured stateful streaming apps using Spark Operator.There were a chain of classpath errors and/or class version errors repeating themselves despite multiple attempts such as including additional jars using sbt build tool , adding jars in docker image,adding a few libraries in spark extra classpath,a few jvm options, adding spark operator app's dependencies, permissions settings. At IMG Arena,we have systematically troubleshot this problem by adding debugging Scala code to print out the classloaders and the files loaded. We got a good understanding why the above options were failing and after the changes we are able to migrate all our stateful and real time apps to Spark Operator on EKS in production
Here is a medium blog on the same https://aravind-deva.medium.com/spark-operator-on-aws-eks-systematic-troubleshooting-120bad35de74