Bemærk
Adgang til denne side kræver godkendelse. Du kan prøve at logge på eller ændre mapper.
Adgang til denne side kræver godkendelse. Du kan prøve at ændre mapper.
User-facing configuration API, accessible through SparkSession.conf.
Supports Spark Connect
Options set here are automatically propagated to the Hadoop configuration during I/O.
Syntax
spark.conf.set(key, value)
spark.conf.get(key)
Properties
| Property | Description |
|---|---|
getAll |
Returns all properties set in this configuration as a dict. |
Methods
| Method | Description |
|---|---|
set(key, value) |
Sets the given Spark runtime configuration property. value can be a string, integer, or boolean. |
get(key, default) |
Returns the value of the configuration property for the given key. Returns default if the key does not exist. Raises an exception if the key is not set and no default is provided. |
unset(key) |
Resets the configuration property for the given key. |
isModifiable(key) |
Returns True if the configuration property for the given key is modifiable in the current session. |
Examples
Set and retrieve a configuration property:
spark.conf.set("key1", "value1")
spark.conf.get("key1")
'value1'
Retrieve a property with a default value:
spark.conf.get("non-existent-key", "my_default")
'my_default'
Unset a configuration property:
spark.conf.set("my_key", "my_value")
spark.conf.unset("my_key")
spark.conf.get("my_key")
pyspark...SparkNoSuchElementException: ... The SQL config "my_key" cannot be found...
Retrieve all configuration properties:
spark.conf.set("key1", "value1")
spark.conf.set("key2", "value2")
spark.conf.getAll
{'key1': 'value1', 'key2': 'value2'}