Breaking News: Grepper is joining You.com. Read the official announcement!

pytest pyspark spark session example

Allie Ubisse answered on August 1, 2023 Popularity 2/10 Helpfulness 3/10

answer pytest pyspark spark session example

pytest pyspark spark session example

Comment

# conftest.py

from pyspark.sql import SparkSession

import pytest


@pytest.fixture(scope="session")
def spark():
    spark = (
        SparkSession.builder.master("local[1]")
        .appName("local-tests")
        .config("spark.executor.cores", "1")
        .config("spark.executor.instances", "1")
        .config("spark.sql.shuffle.partitions", "1")
        .config("spark.driver.bindAddress", "127.0.0.1")
        .getOrCreate()
    )
    yield spark
    spark.stop()

Popularity 2/10 Helpfulness 3/10 Language python

Source: engineeringfordatascience.com

Tags: pyspark pytest python session

Link to this answer
Share Copy Link