TableValuedFunction.explode

Returns a DataFrame containing a new row for each element in the given array or map. Uses the default column name col for elements in the array and key and value for elements in the map unless specified otherwise.

Syntax

spark.tvf.explode(collection)

Parameters

Returns

pyspark.sql.DataFrame: A DataFrame with a new row for each element.

Examples

Example 1: Exploding an array column

import pyspark.sql.functions as sf
spark.tvf.explode(sf.array(sf.lit(1), sf.lit(2), sf.lit(3))).show()

+---+
|col|
+---+
|  1|
|  2|
|  3|
+---+

Example 2: Exploding a map column

import pyspark.sql.functions as sf
spark.tvf.explode(
    sf.create_map(sf.lit("a"), sf.lit("b"), sf.lit("c"), sf.lit("d"))
).show()

+---+-----+
|key|value|
+---+-----+
|  a|    b|
|  c|    d|
+---+-----+

Example 3: Exploding an array of struct column

import pyspark.sql.functions as sf
spark.tvf.explode(sf.array(
    sf.named_struct(sf.lit("a"), sf.lit(1), sf.lit("b"), sf.lit(2)),
    sf.named_struct(sf.lit("a"), sf.lit(3), sf.lit("b"), sf.lit(4))
)).select("col.*").show()

+---+---+
|  a|  b|
+---+---+
|  1|  2|
|  3|  4|
+---+---+

Example 4: Exploding an empty array column

import pyspark.sql.functions as sf
spark.tvf.explode(sf.array()).show()

+---+
|col|
+---+
+---+

Example 5: Exploding an empty map column

import pyspark.sql.functions as sf
spark.tvf.explode(sf.create_map()).show()

+---+-----+
|key|value|
+---+-----+
+---+-----+