Traverse to the spark/ conf folder and make a copy of the spark-env.sh.template file as a spark-env.sh To setup Apache Spark Master configuration, edit spark-env.sh file. Run the following command to make effective changes in the. The above line sets the location (Path) where the spark software file is located to the PATH variable. bashrc file:Īdd the following line to the file and save. To set up the environment for Apache Spark, we need to edit the . $ sudo mv spark-3.1.1-bin-hadoop3.2 /usr/local/spark Run the following command to move the spark files to the spark directory ( /usr/local/bin): Now run the following command to untar the spark tar file: To download spark-3.1.1 with Hadoop-3.2, run the following command:
INSTALL SPARK LOCALLY ON A MAC INSTALL
I will install spark-3.1.1 with Hadoop-3.2. (On all the Virtual Machines – Master and workers)ĭownload the stable version of Apache Spark. You can exit from slave machine by type the command: Run the following command on master to connect to the slaves / workers: Now it’s time to check if everything installed properly. $ ssh-copy-id ssh-copy-id ssh-copy-id user name and IP will be different of your machines. ssh/authorized_keys (all the workers/slaves as well as master). Now copy the content of .ssh/id_rsa.pub form master to. $ cat ~/.ssh/id_rsa.pub > ~/.ssh/authorized_keys Run the following command to authorize the key: For this purpose, run the following command: $ sudo apt-get install openssh-server openssh-client To configure Open SSH server-client, run the following command: Now configure Open SSH server-client on master. To check the version of Scala, run the following command: Install Scala on the all the machines (master and the worker / slaves). #to check version of java, run the following command. Run the following commands on all the Machines (master and workers / slaves).