Spark 3? #5

The current Docker image uses:

• Scala 2.11
• Spark 2.4.8 + Hadoop 2.7 (spark-2.4.8-bin-hadoop2.7)
• GraphFrames 0.8.1 (graphframes-0.8.1-spark2.4-s_2.11)
• Kafka 2.2.2 (kafka_2.11-2.2.2)
• Java 8 (temurin-8-jdk)
• Python 3.6 (slim-buster)

The main constraint seems to be the supported Scala version:

GraphFrames 0.8.2 only supports up to Spark 3.2.x.

Spark 3.2 supports Java 8/11 and deprecates Python 3.6. (Spark 3.4 supports Java 17.)

Let’s try:

• Spark 3.2.4 + Hadoop 2.7: http://archive.apache.org/dist/spark/spark-3.2.4/spark-3.2.4-bin-hadoop2.7.tgz
• PySpark 3.2.4 (may not need to directly install this—current image doesn’t?): http://archive.apache.org/dist/spark/spark-3.2.4/pyspark-3.2.4.tar.gz
• Kafka 3.5.1 + Scala 2.12: http://archive.apache.org/dist/kafka/3.5.1/kafka_2.12-3.5.1.tgz
• GraphFrames 0.8.2 + Scala 2.12: https://repos.spark-packages.org/graphframes/graphframes/0.8.2-spark3.2-s_2.12/graphframes-0.8.2-spark3.2-s_2.12.jar
• Java 11 (temurin-11-jdk)
• Python 3.11 (slim-buster)