Breaking News: Grepper is joining You.com. Read the official announcement!

sparkcontext pyspark

Innocent Iguana answered on February 12, 2024 Popularity 7/10 Helpfulness 8/10

answer sparkcontext pyspark

sparkcontext pyspark

Comment

# Create a context from SparkSession
spark = SparkSession.builder \
    .appName("example") \
    .getOrCreate()
sc = spark.sparkContext

# Alternative : create spark context explicitly
from pyspark import SparkConf, SparkContext
conf = SparkConf().setAppName("YourAppName").setMaster("local[*]") # Set configuration for SparkContext
sc = SparkContext(conf=conf)


print(sc) # Verify SparkContext
print(sc.version) # Print Spark version
print(sc.pythonVer) # Print Python version
print(sc.master) # Print the spark mode

# Loading data (With specified number of partitions)
numRDD = sc.parallelize(range(10), minPartitions = 6)
fileRDD = sc.textFile("README.md", minPartitions = 6)
fileRDD.getNumPartitions() # See number of broken parts

# Create a SparkSession from SparkContext
spark = SparkSession(sc)

Popularity 7/10 Helpfulness 8/10 Language python

Source: Grepper

Tags: pyspark python

Link to this answer
Share Copy Link

Contributed on Feb 12 2024

Innocent Iguana

0 Answers Avg Quality 2/10

Continue with Google

By continuing, I agree that I have read and agree to Greppers's Terms of Service and Privacy Policy.

sparkcontext pyspark

Contents

More Related Answers

sparkcontext pyspark

Grepper

Documentation

Social

Legal

Contact

Oops, You will need to install Grepper and log-in to perform this action.