Breaking News: Grepper is joining You.com. Read the official announcement!

pyspark user defined function multiple input

Gifted Grouse answered on August 2, 2020 Popularity 3/10 Helpfulness 2/10

answer pyspark user defined function multiple input

pyspark user defined function multiple input

Comment

>>> from pyspark.sql.types import IntegerType
>>> from pyspark.sql.functions import udf, array
>>> sum_cols = udf(lambda arr: sum(arr), IntegerType())
>>> spark.createDataFrame([(101, 1, 16)], ['ID', 'A', 'B']) \
...     .withColumn('Result', sum_cols(array('A', 'B'))).show()
+---+---+---+------+
| ID|  A|  B|Result|
+---+---+---+------+
|101|  1| 16|    17|
+---+---+---+------+

>>> spark.createDataFrame([(101, 1, 16, 8)], ['ID', 'A', 'B', 'C'])\
...     .withColumn('Result', sum_cols(array('A', 'B', 'C'))).show()
+---+---+---+---+------+
| ID|  A|  B|  C|Result|
+---+---+---+---+------+
|101|  1| 16|  8|    25|
+---+---+---+---+------+

Popularity 3/10 Helpfulness 2/10 Language whatever

Source: stackoverflow.com

Tags: function multiple-input pyspark user-defined whatever

Link to this answer
Share Copy Link