Breaking News: Grepper is joining You.com. Read the official announcement!

pyspark split dataframe by rows

Glorious Gnu answered on December 6, 2021 Popularity 8/10 Helpfulness 5/10

answer pyspark split dataframe by rows

pyspark split dataframe by rows

Comment

from pyspark.sql.window import Window
from pyspark.sql.functions import monotonically_increasing_id, ntile

values = [(str(i),) for i in range(100)]
df = spark.createDataFrame(values, ('value',))

def split_by_row_index(df, num_partitions=4):
    # Let's assume you don't have a row_id column that has the row order
    t = df.withColumn('_row_id', monotonically_increasing_id())
    # Using ntile() because monotonically_increasing_id is discontinuous across partitions
    t = t.withColumn('_partition', ntile(num_partitions).over(Window.orderBy(t._row_id))) 
    return [t.filter(t._partition == i+1).drop('_row_id', '_partition') for i in range(partitions)]

[i.collect() for i in split_by_row_index(df)]

Popularity 8/10 Helpfulness 5/10 Language python

Source: stackoverflow.com

Tags: dataframe pyspark python rows

Link to this answer
Share Copy Link