Breaking News: Grepper is joining You.com. Read the official announcement!

split spark dataframe into multiple dataframes

amine djeddi answered on September 27, 2022 Popularity 6/10 Helpfulness 7/10

answer split spark dataframe into multiple dataframes

split spark dataframe into multiple dataframes

Comment

#First : add incremental column 

df = df.withColumn("idx", F.monotonically_increasing_id())
w = Window().orderBy("idx")
df = df.withColumn("id", (F.row_number().over(w))).drop('idx')
df.show()


#second step : apply the algo ! 
desired_chunks = 3
cnt = df.count()
res = rows_number//desired_chunks
rem = rows_number%desired_chunks #modulo
iteration_rang = desired_chunks+1

for i in range(1, iteration_rang) :
    if (i == 1):
       #this will only calculate the first chunck
       df_part= df.filter(df.id.between(i, res))
       #process your df_part...
       df_part.show()

    if(i> 1 & i<iteration_rang):
       df_part= df.filter((df.id.between((res*(i-1)+1), res*i)))
       #process your df_part...
       df_part.show()

       if(rem != 0):
          if(i== rng-1):
             df_part= df.filter((df.id.between((res*(i)+1), cnt)))
             # process your df_part...
             df_part.show()
       else :
           continue

Popularity 6/10 Helpfulness 7/10 Language whatever

Source: stackoverflow.com

Tags: spark-dataframe whatever

Link to this answer
Share Copy Link

Contributed on Sep 27 2022

amine djeddi

0 Answers Avg Quality 2/10

split spark dataframe into multiple dataframes

Contents

More Related Answers

split spark dataframe into multiple dataframes

Grepper

Documentation

Social

Legal

Contact

Oops, You will need to install Grepper and log-in to perform this action.