RAVI K YADAV

Data Scientist / ML Engineer

Scala and Spark compatibility issues

One of the major issue I have noticed working around with Spark and Scala is dependency between two packages.

Scala developers are upgrading Scala little faster than folks at Apache Spark can absorb. At the time of writing, Scala 2.12.8 is already out, but Spark’s latest release 2.4.0 is still stuck at Scala 2.11.X. I understand that Spark team is working on upgrading the Spark to framework to be compatible with Scala 2.12. But I think by the time they will release that upgrade, Scala may move to higher version.

So, the lesson for developers is to first get the Spark and then get the compatible version of Scala. Otherwise, you will get errors when trying to run your Scala programs with non compatible Spark (NoSuchMethodError).

Unrelated to the compatibility issue, one other very common issue I noticed when trying to run a spark program on a VM is with failure to bind on ports. It can be fixed by setting SPARK_LOCAL_IP environment variable. If you are running the Spark locally then set it to 127.0.0.1

 

 

Advertisement

Published by

Ravi Kumar (RK) Yadav
I am machine learning researcher. Currently working on building customer navigation solutions to help customer express intent faster and easier. Such systems are designed to make customer journey faster from search to checkout. I spend my spare time working on vision projects. In spring and summer, I spend my time on growing Cayenne pepper. I graduated from North Carolina State University – Raleigh, with a master’s degree in computer science in the Spring of 2016. Prior to this, I completed my bachelor’s degree in electronics and communication engineering.

Leave a Reply

Fill in your details below or click an icon to log in:

Gravatar
WordPress.com Logo

You are commenting using your WordPress.com account. ( Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. ( Log Out /  Change )

Cancel

Connecting to %s

%d bloggers like this: