Pipelines

Pipelines is an enterprise-grade, asynchronous batch service that ingests large volumes of sensitive data directly into your vault, tokenizes or detokenizes files at scale, and works seamlessly with Skyflow’s de-identification to bulk de-identify sensitive data.

Pipelines automates secure high-volume data transfers and transformations while preserving privacy and without exposing any of this to your backend infrastructure.

Features

When you develop a pipeline, you can tailor the workflow with the following features:

• Sources and destinations: Pipelines acts on files and can connect to S3 buckets, FTPS servers, SFTP servers, and vault file columns to retrieve files for processing.
• Formats: Pipelines support CSV, TSV, PSV, JSON, Metro 2, ACH, and Parquet file formats out of the box.
• Mappings: The content from each of these files can then be mapped into columns in your vault.
• Actions: Various actions like Import, Update, Tokenize, Detokenize, De-identify, Transfer, Export are supported depending on the use case.
• Triggers: Pipelines can be triggered via an API call today. Additional support for different invocations coming soon.

Additionally, pipelines support the following:

• Bring Your Own Tokens (BYOT) for migrating existing tokens
• Token reports
• Error reports
• PGP key encryption

Pipeline actions

Actions determine how the pipeline processes data in your workflow. You can take the following actions:

PGP keys

Sometimes, you might want to encrypt your payload before uploading it to your FTP server. In these instances, pipelines support creation of PGP keys to decrypt the files before the migration. You must have the Vault Owner, Account Admin or Pipeline Manager role to generate a PGP key. In addition to creating PGP Keys, Pipelines also supports SSH_RSA protocol as well for securing access to SFTP servers.

Reports

After a pipeline runs, it generates reports about the run, which you can access via Studio or the Get Pipeline Run and List Pipeline Runs operations. Reports are available for 7 days after the run.

You can calculate the total number of records processed by the pipeline run by totaling the number of records across all the reports.

Token reports

Token reports contain a primary key in the source file that maps to the corresponding Skyflow IDs of the records you insert into the vault and the corresponding vault table.

Import and tokenize reports

After a pipeline runs, it generates a report about the run. These reports contain a primary key in the source file that maps to the corresponding Skyflow IDs of the records you insert into the vault and the corresponding vault table.

If you select the tokenization action, the tokens.csv report also contains token values corresponding to the column names.

Token reports have a limit of 100,000 records. If you have more than 100,000 records, the pipeline run returns multiple report files.

Error reports

For every pipeline run, Skyflow generates error reports even if errors didn’t occur:

• parser-errors.csv reports records that the pipeline couldn’t parse and therefore didn’t attempt to insert into the vault. This can happen if the source file has an invalid format.
• vault-errors.csv contains records that failed to insert into the vault and the associated error.

Use cases

You can use pipelines to

• Migrate data to a vault. Pipelines can migrate millions of records from a source system into a vault. For instance, you can use pipelines to migrate PCI data from a single payments service, like Stripe, and avoid vendor lock-in in a PCI-compliant manner. During migration, your pipeline pulls all the data from the source system into your vault without exposing your infrastructure to sensitive data.

  

• Work with financial institutions. You can use Pipelines to securely handle sensitive files like Metro 2 and ACH files that are generated and consumed by banks, credit bureaus etc.

• Securely de-identify files. You can use Pipelines to de-identify training data on a large scale. See De-identify training data

• Process files stored in a vault. You can use Pipelines to process files already stored in a vault table column. This lets you run pipeline operations like import or tokenize on files that have been uploaded to the vault, referencing them by their Skyflow IDs.

Next steps

Learn how to create a pipeline, explore functions, or assign a role across resources.