Breaking News: Grepper is joining You.com. Read the official announcement!

data quality with AWS deequ pyspark example

Allie Ubisse answered on December 20, 2023 Popularity 1/10 Helpfulness 1/10

answer data quality with AWS deequ pyspark example

data quality with AWS deequ pyspark example

Comment

from pyspark.sql import SparkSession, Row
import pydeequ

spark = (SparkSession
    .builder
    .config("spark.jars.packages", pydeequ.deequ_maven_coord)
    .config("spark.jars.excludes", pydeequ.f2j_maven_coord)
    .getOrCreate())

df = spark.sparkContext.parallelize([
            Row(a="foo", b=1, c=5),
            Row(a="bar", b=2, c=6),
            Row(a="baz", b=3, c=None)]).toDF()

Popularity 1/10 Helpfulness 1/10 Language python

Source: pydeequ.readthedocs.io

Tags: data-quality pyspark python

Link to this answer
Share Copy Link