Breaking News: Grepper is joining You.com. Read the official announcement!

spark delta table merge

Ice Keith answered on September 24, 2022 Popularity 3/10 Helpfulness 1/10

answer spark delta table merge

spark delta table merge

Comment

import io.delta.tables.*

val deltaTable = DeltaTable.forPath(spark, "/data/aggregates")

// Function to upsert microBatchOutputDF into Delta table using merge
def upsertToDelta(microBatchOutputDF: DataFrame, batchId: Long) {
  deltaTable.as("t")
    .merge(
      microBatchOutputDF.as("s"),
      "s.key = t.key")
    .whenMatched().updateAll()
    .whenNotMatched().insertAll()
    .execute()
}

// Write the output of a streaming aggregation query into Delta table
streamingAggregatesDF.writeStream
  .format("delta")
  .foreachBatch(upsertToDelta _)
  .outputMode("update")
  .start()

Popularity 3/10 Helpfulness 1/10 Language scala

Source: docs.delta.io

Tags: delta merge scala

Link to this answer
Share Copy Link

Contributed on Sep 24 2022

Ice Keith

0 Answers Avg Quality 2/10

Continue with Google

By continuing, I agree that I have read and agree to Greppers's Terms of Service and Privacy Policy.

spark delta table merge

Contents

More Related Answers

spark delta table merge

Grepper

Documentation

Social

Legal

Contact

Oops, You will need to install Grepper and log-in to perform this action.