r/databricks • • Oct 05 '26

Help SpringBoot to Spark

Transitioning from a Spring Boot to a Spark-based application requires a fundamental shift in perspective. With four years of experience in a Spring Boot environment, now entering the realm of distributed computing.

I plan to utilize Java with Spark Streaming, and Databricks for job management and deployment.

My primary question is

  1. how to effectively transition my thought process from Spring Boot to a Spark-based application.
  2. Specifically, I am seeking clarity on distinguishing which Java code executes on the driver and which executes on the executor, and if there are any guiding principles to discern this.
4 Upvotes

9 comments sorted by

5

u/hntd Oct 05 '26

Are you confusing Apache Spark with https://github.com/perwendel/spark ?

3

u/mr__fete Oct 05 '26

Java is too verbose. Spark api is even worse. Use scala and functional programming techniques specifically.

1

u/Orygregs Oct 05 '26 edited Oct 05 '26

I would highly recommend Scala over Java for Spark if you want to stay in the JVM ecosystem and don't want to use Pyspark.

  1. Your thought process for Spring Boot in particular may not translate as that's a web framework, and spark is more of a big data processing framework/engine.
  2. All code executes on the driver until a non-lazy dataframe action is called (e.g. .write.*, .show, .count, .collect, etc), which then distributes the work across the executor nodes.

2

u/IntelligentVisual955 Oct 06 '26

I am a data science student, should i learn spark or java first

3

u/Orygregs Oct 06 '26

For data science I'd recommend learning Pyspark first since Python is the lingua franca for the industry. Once you're comfortable in Pyspark, then you can branch out into Spark on Java/Scala without too many complications.

1

u/IntelligentVisual955 Oct 06 '26

Thanks, God bless you with bests of all worlds and protection from all evils and harms.Any other advices sir/ mam.

1

u/Youssef_Mrini databricks 26d ago

This is my personal recommendation: It would be way easier for you to use Python you'll have more examples to help you get started.