Pivot Spark data frame using python

Anuj Sharma answered on June 2, 2022 Popularity 7/10 Helpfulness 2/10

answer Pivot Spark data frame using python

Pivot Spark data frame using python

Comment

from pyspark.sql.functions import avg

flights = (sqlContext
    .read
    .format("csv")
    .options(inferSchema="true", header="true")
    .load("flights.csv")
    .na.drop())

flights.registerTempTable("flights")
sqlContext.cacheTable("flights")

gexprs = ("origin", "dest", "carrier")
aggexpr = avg("arr_delay")

flights.count()
## 336776

%timeit -n10 flights.groupBy(*gexprs ).pivot("hour").agg(aggexpr).count()
## 10 loops, best of 3: 1.03 s per loop

Popularity 7/10 Helpfulness 2/10 Language python

Source: stackoverflow.com

Tags: frame pivot python using

Link to this answer
Share Copy Link

Contributed on Jun 02 2022

Anuj Sharma

0 Answers Avg Quality 2/10

Pivot Spark data frame using python

Contents

More Related Answers

Pivot Spark data frame using python

Grepper

Documentation

Social

Legal

Contact

Oops, You will need to install Grepper and log-in to perform this action.