Data

This section covers:

1. Adding Data to a Pipeline

2. Removing Data from a Pipeline

3. Replacing Data

4. Using External Data Assets

Adding Data to a Pipeline

The file tree system in a pipeline is similar to a capsule’s file tree. You can add data to your pipeline by attaching a data asset or uploading data. If using uploaded data, files must be placed inside a folder before they can be dragged onto the pipeline building area.

It is best practice to use data assets to reduce runtime and to ensure that the data doesn’t contribute to the size of the pipeline. This also allows the same data asset to be easily attached to multiple pipelines and capsules. For more information see creating data assets.

Once your data is in the file tree system, you can drag and drop it into the pipeline building area and connect it to a capsule.

The pipeline will only use data that is dragged onto the pipeline building area and connected to a capsule.

Removing Data from a Pipeline

To remove a data asset from the pipeline building area you can hover over it and click the garbage can.

Replacing Data

The replace feature can be used to substitute a data asset while maintaining all connections and mappings.

Data assets must be attached in the pipeline's data folder to appear in the Replace menu

Using External Data Assets

To use external data assets in a pipeline, a custom IAM role must be selected in the Pipeline Settings menu. See Pipeline Settings for more information.