Breaking News: Grepper is joining You.com. Read the official announcement!

calculate time between datetime pyspark

DS in Training answered on March 6, 2022 Popularity 4/10 Helpfulness 2/10

answer calculate time between datetime pyspark

related calculate time between datetime pyspark

calculate time between datetime pyspark

Comment

dates = [("1","2019-07-01 12:01:19.111"),
    ("2","2019-06-24 12:01:19.222"),
    ("3","2019-11-16 16:44:55.406"),
    ("4","2019-11-16 16:50:59.406")
    ]

df = spark.createDataFrame(data=dates, schema=["id","input_timestamp"])

from pyspark.sql.functions import *

#Calculate Time difference in Seconds
df2=df.withColumn('from_timestamp',to_timestamp(col('from_timestamp')))\
  .withColumn('end_timestamp', current_timestamp())\
  .withColumn('DiffInSeconds',col("end_timestamp").cast("long") - col('from_timestamp').cast("long"))
df2.show(truncate=False)

Popularity 4/10 Helpfulness 2/10 Language python

Source: sparkbyexamples.com

Tags: between datetime pyspark python time

Link to this answer
Share Copy Link

Contributed on Mar 06 2022

DS in Training

0 Answers Avg Quality 2/10

Closely Related Answers

calculate time between datetime pyspark

Comment

dates = [("1","2019-07-01 12:01:19.111"),
    ("2","2019-06-24 12:01:19.222"),
    ("3","2019-11-16 16:44:55.406"),
    ("4","2019-11-16 16:50:59.406")
    ]

df = spark.createDataFrame(data=dates, schema=["id","input_timestamp"])

from pyspark.sql.functions import *

#Calculate Time difference in Seconds
df2=df.withColumn('from_timestamp',to_timestamp(col('from_timestamp')))\
  .withColumn('end_timestamp', current_timestamp())\
  .withColumn('DiffInSeconds',col("end_timestamp").cast("long") - col('from_timestamp').cast("long"))
df2.show(truncate=False)

Popularity 4/10 Helpfulness 2/10 Language python

Source: sparkbyexamples.com

Tags: datetime pyspark time between python

Link to this answer
Share Copy Link

Contributed on Mar 06 2022

DS in Training

0 Answers Avg Quality 2/10

calculate time between datetime pyspark

Contents

More Related Answers

calculate time between datetime pyspark

Closely Related Answers

calculate time between datetime pyspark

Grepper

Documentation

Social

Legal

Contact

Oops, You will need to install Grepper and log-in to perform this action.