How to Install Spark 3 on Windows 10

 I have been using spark for a long time. It is an excellent, distributed computation framework. I use this regularly at work, and I also have it installed on my local desktop and laptop.

This document is to show the installation steps for installing spark 3+ on Windows 10 in a sudo distributed mode.

Steps:-

1. Install WSL2
1. https://docs.microsoft.com/en-us/windows/wsl/install-win10
2. Install Ubuntu 20.4 LTS from the Microsoft store.
3. Install windows terminal form the Microsoft store. This step is optional. You can use PowerShell or MobaXterm
4. Fire up the Ubuntu from WSL
5. 
   
   
   
   
6. Once logged in, then go to home dir
1. “ cd ~ ”
7. For spark, we need 
1. Python3
2. Java 
3. Latest Scala
4. Spark with Hadoop, zip file
8. Let's download and install all the prerequisite
1. install python
2. sudo apt-get install software-properties-common
3. sudo apt-get install python-software-properties
9. install Java (open JDK)
1. sudo apt-get install openjdk-8-jdk
10. Check the java and javac version
1. java -version
2. javac -version
   
   
   
11. Install Scala
12. get scala binary for unix
1. wget https://downloads.lightbend.com/scala/2.13.3/scala-2.13.3.tgz
2. tar xvf scala-2.13.3.tgz 
13. edit bashrc file to add Scala
1. vi ~/.bashrc
14. add these lines in the end
1. export SCALA_HOME=Path-where-scala-file-is-located#/root/scala-2.13.3
2. export PATH=$PATH:$SCALA_HOME/bin
3. source ~/.bashrc
4. scala -version
15. get the spark
1. I downloaded the spark from the source
2. wget "https://archive.apache.org/dist/spark/spark-3.1.1/spark-3.1.1-bin-hadoop3.2.tgz"
3. tar xvf spark-3.1.1-bin-hadoop3.2.tgz
4. vi ~/.bashrc
5. export SPARK_HOME="/home/sandipan/spark-3.1.1-bin-hadoop3.2"
6. export PATH=$PATH:$SPARK_HOME/bin
7. source ~/.bashrc
16. Start Spark Services
1. cd $SPARK_HOME
2. start the master Server
1. ./sbin/start-master.sh
3. once you start the master server you will get a message saying it had started.
1. you can see the spark status in web console of master using local http://localhost:8080
4. there you will see the master url
1. Mine looks like:- "spark://LAPTOP-7DUT93OF.localdomain:7077"
5. we can start a slave using bellow command
1. SPARK_WORKER_INSTANCES=3 SPARK_WORKER_CORES=2 SPARK_WORKER_MEMORY=7G ./sbin/start-worker.sh spark://LAPTOP-7DUT93OF.localdomain:7077
6. SPARK_WORKER_INSTANCES = how many worker instances you want to start
7. SPARK_WORKER_CORES =  how many cores per instances you want to give. Generally, I give 1 core.
8. SPARK_WORKER_MEMORY = Memory per worker. Be very careful with this parameter. My laptop has 32 GB of memory, so I keep 3GB to 4GB for Windows, 2 GB for the driver program and the rest for the worker node.
9. open the pyspark shell
1. SPARK_HOME/bin/pyspark --master spark://LAPTOP-7DUT93OF.localdomain:7077 --executor-memory 6500mb
2. SPARK_HOME/bin/spark-shell --master spark://LAPTOP-7DUT93OF.localdomain:7077 --executor-memory 6500mb
10. to stop all the workers
1. SPARK_WORKER_INSTANCES=3 SPARK_WORKER_CORES=2 ./sbin/stop-worker.sh spark://LAPTOP-7DUT93OF.localdomain:7077
2. OR TRY
1. kill -9 $(jps -l | grep spark | awk -F ' ' '{print $1}')