In Simply Business we are currently using Snowplow integrated with Spark 1.6 and deployed on EMR-4.6.
Wa are in the process of migrating to Spark 2, EMR 5.2 as it will help us in many areas. However this version of Spark in EMR is built on top of Scala 2.11. Talked with AWS support and they don’t have a similar Spark 2 on Scala 2.10 EMR build they can offer.
So we are migrating our projects to Scala 2.11 in order to be able to run in Spark 2 in EMR. We obviously have a dependency on Snowplow libraries which are all built against Scala 2.10. (We mainly depend on common-enrich)
Before I embark on updating libraries and trial error what goes wrong with every update I do, I was wondering what kind of progress you have done on this area, and if a joint effort (at least for the projects we depend on) would be beneficial. I am aware of this ticket: https://github.com/snowplow/snowplow/issues/2824 but it doesn’t give much detail on roadmap or progress.
Thanks a lot,