Breaking News: Grepper is joining You.com. Read the official announcement!

connect to spark cluster

sivakguru answered on September 3, 2021 Popularity 6/10 Helpfulness 4/10

answer connect to spark cluster

connect to spark cluster

Comment

from pyspark.conf import SparkConf
from pyspark.sql import SparkSession, Catalog

spark_conf = SparkConf()
spark_conf.setAll([
    ('spark.master', 'spark://<master-ip>:7077'),
    ('spark.app.name', 'myApp'),
    ('spark.submit.deployMode', 'client'),
    ('spark.ui.showConsoleProgress', 'true'),
    ('spark.eventLog.enabled', 'false'),
    ('spark.logConf', 'false'),
    ('spark.driver.bindAddress', '0.0.0.0'),
    ('spark.driver.host', '<master-ip>')
])

spark = SparkSession.builder.config(conf=spark_conf).getOrCreate()

Popularity 6/10 Helpfulness 4/10 Language python

Source: Grepper

Tags: connect python

Link to this answer
Share Copy Link

Contributed on Sep 03 2021

sivakguru

0 Answers Avg Quality 2/10

Continue with Google

By continuing, I agree that I have read and agree to Greppers's Terms of Service and Privacy Policy.

connect to spark cluster

Contents

More Related Answers

connect to spark cluster

Grepper

Documentation

Social

Legal

Contact

Oops, You will need to install Grepper and log-in to perform this action.