Configuration reference

This article provides reference for keys supported by Databricks Asset Bundles configuration (YAML). See What are Databricks Asset Bundles?.

For complete bundle examples, see Bundle configuration examples and the bundle-examples GitHub repository.

artifacts

Type: Map

Specifies the artifacts to automatically build during bundle deployments that can be used later in bundle runs. Each key is the name of the artifact, and the value is a Map that defines the artifact build settings.

Added in Databricks CLI version 0.229.0

artifacts:
  <artifact-name>:
    <artifact-field-name>: <artifact-field-value>

Examples

The following configuration builds a Python wheel using Poetry:

artifacts:
  default:
    type: whl
    build: poetry build
    path: .

The following configuration runs tests and builds a wheel. For a complete bundle tutorial that uses artifacts to build a wheel, see Build a Python wheel file using Databricks Asset Bundles.

artifacts:
  default:
    type: whl
    build: |-
      # run tests
      python -m pytest tests/ -v

      # build the actual artifact
      python setup.py bdist_wheel

    path: .

For an example configuration that builds a JAR and uploads it to Unity Catalog, see Bundle that uploads a JAR file to Unity Catalog.

artifacts.name.files

Type: Sequence

The relative or absolute path to the built artifact files. Use source to specify the built artifacts. Paths are relative to the location of the bundle configuration file.

Added in Databricks CLI version 0.229.0

bundle

Type: Map

The bundle attributes when deploying to this target.

A bundle configuration file must contain only one top-level bundle mapping.

This bundle mapping must contain a name mapping that specifies a programmatic (or logical) name for the bundle. The following example declares a bundle with the programmatic (or logical) name hello-bundle.

bundle:
  name: hello-bundle

A bundle mapping can also be a child of one or more of the targets in the top-level targets mapping. Each of these child bundle mappings specify any non-default overrides at the target level.

Added in Databricks CLI version 0.229.0

bundle.databricks_cli_version

The bundle mapping can contain a databricks_cli_version mapping that constrains the Databricks CLI version required by the bundle. This can prevent issues caused by using mappings that are not supported in a certain version of the Databricks CLI.

The Databricks CLI version conforms to semantic versioning and the databricks_cli_version mapping supports specifying version constraints. If the current databricks --version value is not within the bounds specified in the bundle's databricks_cli_version mapping, an error occurs when databricks bundle validate is executed on the bundle. The following examples demonstrate some common version constraint syntax:

bundle:
  name: hello-bundle
  databricks_cli_version: '0.218.0' # require Databricks CLI 0.218.0

bundle:
  name: hello-bundle
  databricks_cli_version: '0.218.*' # allow all patch versions of Databricks CLI 0.218

bundle:
  name: my-bundle
  databricks_cli_version: '>= 0.218.0' # allow any version of Databricks CLI 0.218.0 or higher

bundle:
  name: my-bundle
  databricks_cli_version: '>= 0.218.0, <= 1.0.0' # allow any Databricks CLI version between 0.218.0 and 1.0.0, inclusive

bundle.deployment

Type: Map

The definition of the bundle deployment

Added in Databricks CLI version 0.229.0

bundle.deployment.lock

Type: Map

The deployment lock attributes.

Added in Databricks CLI version 0.229.0

experimental

Type: Map

Defines attributes for experimental features.

Added in Databricks CLI version 0.229.0

include

Type: Sequence

Specifies a list of path globs that contain configuration files to include within the bundle. These path globs are relative to the location of the bundle configuration file in which the path globs are specified. Aside from databricks.yml, you must use the include array to specify all configuration files to include within the bundle.

This include array can appear only as a top-level mapping.

Added in Databricks CLI version 0.229.0

The following example configuration includes three configuration files. These files are in the same folder as the bundle configuration file:

include:
  - 'bundle.artifacts.yml'
  - 'bundle.resources.yml'
  - 'bundle.targets.yml'

The following example configuration includes all files with filenames that begin with bundle and end with .yml. These files are in the same folder as the bundle configuration file:

include:
  - 'bundle*.yml'

permissions

Type: Sequence

Defines the permissions to apply to resources defined in the bundle, where each item in the sequence is a permission for a specific entity. See Set permissions for resources in Databricks Asset Bundles.

Allowed top-level permission levels are CAN_VIEW, CAN_MANAGE, and CAN_RUN.

If you want to apply permissions to a specific resource, see Define permissions for a specific resource.

Added in Databricks CLI version 0.229.0

Example

The following example configuration defines permission levels for a user, group, and service principal, which are applied to all resources defined in resources in the bundle:

permissions:
  - level: CAN_VIEW
    group_name: test-group
  - level: CAN_MANAGE
    user_name: someone@example.com
  - level: CAN_RUN
    service_principal_name: 123456-abcdef

presets

Type: Map

Defines bundle deployment presets. For more information, see Custom presets.

Unless an exception is specified for a preset, if both mode and presets are set, presets override the default mode behavior, and settings of individual resources override the presets.

Added in Databricks CLI version 0.229.0

python

Type: Map

Configures loading of Python code defined with databricks-bundles package. For more information, see Bundle configuration in Python.

Moved from experimental in Databricks CLI version 0.275.0

resources

Type: Map

Defines the resources for the bundle, where each key is the name of the resource, and the value is a Map that defines the resource. For more information about Databricks Asset Bundles supported resources, and resource definition reference, see Databricks Asset Bundles resources.

The resources mapping can appear as a top-level mapping, or it can be a child of one or more of the targets in the top-level targets mapping, and includes zero or one of the supported resource types. Each resource type mapping includes one or more individual resource declarations, which must each have a unique name. These individual resource declarations use the corresponding object's create operation's request payload, expressed in YAML, to define the resource. Supported properties for a resource are the corresponding object's supported fields.

Create operation request payloads are documented in the Databricks REST API Reference, and the databricks bundle schema command outputs all supported object schemas. In addition, the databricks bundle validate command returns warnings if unknown resource properties are found in bundle configuration files.

For more information about resources supported in bundles, as well as common configuration and examples, see Databricks Asset Bundles resources and Bundle configuration examples.

Added in Databricks CLI version 0.229.0

resources:
  <resource-type>:
    <resource-name>:
      <resource-field-name>: <resource-field-value>

Example

The following example configuration defines a job resource:

resources:
  jobs:
    hello-job:
      name: hello-job
      tasks:
        - task_key: hello-task
          existing_cluster_id: 1234-567890-abcde123
          notebook_task:
            notebook_path: ./hello.py

run_as

Type: Map

The identity (user_name or service_principal_name) to use to run Databricks Asset Bundles workflows. It provides the ability to separate the identity used to deploy a bundle job or pipeline from the one used to run the job or pipeline. See Specify a run identity for a Databricks Asset Bundles workflow.

Added in Databricks CLI version 0.229.0

scripts

Type: Map

The scripts that can be run using bundle run. Each named script in the scripts mapping contains content with commands. See Execute scripts.

Added in Databricks CLI version 0.259.0

scripts:
  <script-name>:
    <script-field-name>: <script-field-value>

Examples

scripts:
  my_script:
    content: uv run pytest -m ${bundle.target}

sync

Type: Map

The files and file paths to include or exclude in the bundle.

Added in Databricks CLI version 0.229.0

include and exclude

The include and exclude mappings within the sync mapping specifies a list of files or folders to include within, or exclude from, bundle deployments, depending on the following rules:

• Based on any list of file and path globs in a .gitignore file in the bundle's root, the include mapping can contain a list of file globs, path globs, or both, relative to the bundle's root, to explicitly include.
• Based on any list of file and path globs in a .gitignore file in the bundle's root, plus the list of file and path globs in the include mapping, the exclude mapping can contain a list of file globs, path globs, or both, relative to the bundle's root, to explicitly exclude.

All paths to specified files and folders are relative to the location of the bundle configuration file in which they are specified.

The syntax for include and exclude file and path patterns follow standard .gitignore pattern syntax. See gitignore Pattern Format.

For example, if the following .gitignore file contains the following entries:

.databricks
my_package/dist

And the bundle configuration file contains the following include mapping:

sync:
  include:
    - my_package/dist/*.whl

Then all of the files in the my_package/dist folder with a file extension of *.whl are included. Any other files in the my_package/dist folder are not included.

However, if the bundle configuration file also contains the following exclude mapping:

sync:
  include:
    - my_package/dist/*.whl
  exclude:
    - my_package/dist/delete-me.whl

Then all of the files in the my_package/dist folder with a file extension of *.whl, except for the file named delete-me.whl, are included. Any other files in the my_package/dist folder are also not included.

The sync mapping can also be declared in the targets mapping for a specific target. Any sync mapping declared in a target is merged with any top-level sync mapping declarations. For example, continuing with the preceding example, the following include mapping at the targets level merges with the include mapping in the top-level sync mapping:

targets:
  dev:
    sync:
      include:
        - my_package/dist/delete-me.whl

sync.paths

The sync mapping can contain a paths mapping that specifies local paths to synchronize to the workspace. The paths mapping allows you to share common files across bundles, and can be used to sync files located outside of the bundle root. (The bundle root is the location of the databricks.yml file.) This is especially useful when you have a single repository that hosts multiple bundles and want to share libraries, code files, or configuration.

Specified paths must be relative to files and directories anchored at the folder where the paths mapping is set. If one or more path values traverse up the directory to an ancestor of the bundle root, the root path is dynamically determined to ensure that the folder structure remains intact. For example, if the bundle root folder is named my_bundle then this configuration in databricks.yml syncs the common folder located one level above the bundle root and the bundle root itself:

sync:
  paths:
    - ../common
    - .

A deploy of this bundle results in the following folder structure in the workspace:

common/
  common_file.txt
my_bundle/
  databricks.yml
  src/
    ...

targets

Type: Map

Defines deployment target contexts for the bundle. Each target is a unique collection of artifacts, Azure Databricks workspace settings, and sometimes target-specific resource details.

The targets mapping consists of one or more target mappings, which must each have a unique programmatic (or logical) name. This mapping is optional but highly recommended.

The settings within the targets mapping take precedence over settings specified in the top-level workspace, artifacts, and resources mappings.

A target can also override the values of any top-level variables.

Added in Databricks CLI version 0.229.0

targets:
  <target-name>:
    <target-field-name>: <target-field-value>

targets.name.default

To specify a target default for bundle commands, set the default mapping to true. For example, this target named dev is the default target:

targets:
  dev:
    default: true

If a default target is not configured, or if you want to validate, deploy, and run jobs or pipelines within a specific target, use the -t option of the bundle commands.

The following commands validate, deploy, and run my_job within the dev and prod targets:

databricks bundle validate
databricks bundle deploy -t dev
databricks bundle run -t dev my_job

databricks bundle validate
databricks bundle deploy -t prod
databricks bundle run -t prod my_job

The following example declares two targets. The first target has the name dev and is the default target used when no target is specified for bundle commands. The second target has the name prod and is used only when this target is specified for bundle commands.

targets:
  dev:
    default: true
  prod:
    workspace:
      host: https://<production-workspace-url>

targets.name.mode

To facilitate easy development and CI/CD best practices, Databricks Asset Bundles provides deployment modes for targets that set default behaviors for pre-production and production workflows. Some behaviors are also configurable using targets.name.presets.

For details, see Databricks Asset Bundle deployment modes.

To specify that a target is treated as a development target, add the mode mapping set to development. To specify that a target is treated as a production target, add the mode mapping set to production. For example, this target named prod is treated as a production target:

targets:
  prod:
    mode: production

targets.name.presets

You can customize some of the target deployment mode behaviors using the presets mapping.

For a list of available presets, see Custom presets.

The following example shows a customized production target that prefixes and tags all production resources:

targets:
  prod:
    mode: production
    presets:
      name_prefix: 'production_' # prefix all resource names with production_
      tags:
        prod: true

variables

Type: Map

Defines a custom variable for the bundle. For each variable, set an optional description, default value, whether the custom variable is a complex type, or lookup to retrieve an ID value, using the following format:

variables:
  <variable-name>:
    description: <variable-description>
    default: <optional-default-value>
    type: <optional-type-value> # "complex" is the only valid value
    lookup:
      <optional-object-type>: <optional-object-name>

To reference a custom variable within bundle configuration, use the substitution ${var.<variable_name>}.

For more information on custom variables and substitutions, see Substitutions and variables in Databricks Asset Bundles.

Added in Databricks CLI version 0.229.0

variables.name.lookup

Type: Map

The name of the alert, cluster_policy, cluster, dashboard, instance_pool, job, metastore, pipeline, query, service_principal, or warehouse object for which to retrieve an ID. For information about using lookup, see Retrieve an object's ID value.

Added in Databricks CLI version 0.229.0

workspace

Type: Map

Defines the Databricks workspace for the bundle. The bundle configuration file can contain only one top-level workspace mapping to specify any non-default Azure Databricks workspace settings to use.

Added in Databricks CLI version 0.229.0

workspace Authentication

The workspace mapping can also contain mappings to specify the Databricks authentication mechanism to use. If they are not specified within the top-level workspace mapping, they must be specified in a workspace mapping as a child of one or more of the targets in the top-level targets mapping.

• For OAuth machine-to-machine (M2M) authentication, the mapping client_id is used. Alternatively, you can set this value in the local environment variable DATABRICKS_CLIENT_ID. Or you can create a configuration profile with the client_id value and then specify the profile's name with the profile mapping (or by using the --profile or -p options when running the bundle validate, deploy, run, and destroy commands with the Databricks CLI). See Authorize service principal access to Azure Databricks with OAuth.

  Note

  You cannot specify an Azure Databricks OAuth secret value in the bundle configuration file. Instead, set the local environment variable DATABRICKS_CLIENT_SECRET. Or you can add the client_secret value to a configuration profile and then specify the profile's name with the profile mapping (or by using the --profile or -p options when running the bundle validate, deploy, run, and destroy commands with the Databricks CLI).

• For Azure CLI authentication, the mapping azure_workspace_resource_id is used. Alternatively, you can set this value in the local environment variable DATABRICKS_AZURE_RESOURCE_ID. Or you can create a configuration profile with the azure_workspace_resource_id value and then specify the profile's name with the profile mapping (or by using the --profile or -p options when running the bundle validate, deploy, run, and destroy commands with the Databricks CLI). See Authenticate with the Azure CLI.

• For Azure client secret authentication with service principals, the mappings azure_workspace_resource_id, azure_tenant_id, and azure_client_id are used. Alternatively, you can set these values in the local environment variables DATABRICKS_AZURE_RESOURCE_ID, ARM_TENANT_ID, and ARM_CLIENT_ID, respectively. Or you can create a configuration profile with the azure_workspace_resource_id, azure_tenant_id, and azure_client_id values and then specify the profile's name with the profile mapping (or by using the --profile or -p options when running the bundle validate, deploy, run, and destroy commands with the Databricks CLI). See Authenticate with Microsoft Entra service principals.

  Note

  You cannot specify an Azure client secret value in the bundle configuration file. Instead, set the local environment variable ARM_CLIENT_SECRET. Or you can add the azure_client_secret value to a configuration profile and then specify the profile's name with the profile mapping (or by using the --profile or -p options when running the bundle validate, deploy, run, and destroy commands with the Databricks CLI).

• For Azure managed identities authentication, the mappings azure_use_msi, azure_client_id, and azure_workspace_resource_id are used. Alternatively, you can set these values in the local environment variables ARM_USE_MSI, ARM_CLIENT_ID, and DATABRICKS_AZURE_RESOURCE_ID, respectively. Or you can create a configuration profile with the azure_use_msi, azure_client_id, and azure_workspace_resource_id values and then specify the profile's name with the profile mapping (or by using the --profile or -p options when running the bundle validate, deploy, run, and destroy commands with the Databricks CLI). See Authenticate with Azure managed identities.

• The azure_environment mapping specifies the Azure environment type (such as Public, UsGov, China, and Germany) for a specific set of API endpoints. The default value is PUBLIC. Alternatively, you can set this value in the local environment variable ARM_ENVIRONMENT. Or you can add the azure_environment value to a configuration profile and then specify the profile's name with the profile mapping (or by using the --profile or -p options when running the bundle validate, deploy, run, and destroy commands with the Databricks CLI).

• The azure_login_app_id mapping is non-operational and is reserved for internal use.

workspace.root_path

This workspace mapping can contain a root_path mapping to specify a non-default root path to use within the workspace for both deployments and workflow runs, for example:

workspace:
  root_path: /Workspace/Users/${workspace.current_user.userName}/.bundle/${bundle.name}/my-envs/${bundle.target}

By default, for root_path the Databricks CLI uses the default path of /Workspace/Users/${workspace.current_user.userName}/.bundle/${bundle.name}/${bundle.target}, which uses substitutions.

workspace.artifact_path

This workspace mapping can also contain an artifact_path mapping to specify a non-default artifact path to use within the workspace for both deployments and workflow runs, for example:

workspace:
  artifact_path: /Workspace/Users/${workspace.current_user.userName}/.bundle/${bundle.name}/my-envs/${bundle.target}/artifacts

By default, for artifact_path the Databricks CLI uses the default path of ${workspace.root}/artifacts, which uses substitutions.

workspace.file_path

This workspace mapping can also contain a file_path mapping to specify a non-default file path to use within the workspace for both deployments and workflow runs, for example:

workspace:
  file_path: /Workspace/Users/${workspace.current_user.userName}/.bundle/${bundle.name}/my-envs/${bundle.target}/files

By default, for file_path the Databricks CLI uses the default path of ${workspace.root}/files, which uses substitutions.

workspace.profile

The profile mapping specifies the name of a configuration profile to use to authenticate to this Azure Databricks workspace. This configuration profile maps to the one that you created when you set up the Databricks CLI.

Common objects

git

Type: Map

Defines git version control details. This is useful for propagating deployment metadata that can be used later to identify resources. For example, you can trace the repository origin of a job deployed by CI/CD.

Whenever you run a bundle command such as validate, deploy or run, the bundle command populates the command's configuration tree with the following default settings:

To retrieve or override Git settings, your bundle must be within a directory that is associated with a Git repository, for example a local directory that is initialized by running the git clone command. If the directory is not associated with a Git repository, these Git settings are empty.

Examples

You can override the origin_url and branch settings within the git mapping of your top-level bundle mapping if needed:

bundle:
  git:
    origin_url: <some-non-default-origin-url>
    branch: <some-non-current-branch-name>