Pipelines
The YAML structure that defines every Cannectors pipeline.
A Cannectors pipeline is a YAML file with four required top-level keys. JSON is still parsed (anything that's valid YAML 1.2 is parsed as YAML), but maintained examples and this documentation use YAML throughout.
Required keys
name: sync-orders
input:
type: httpPolling
schedule: "*/15 * * * *"
endpoint: https://source.example.com/api/orders
dataField: orders
filters: []
output:
type: httpRequest
endpoint: https://destination.example.com/api/orders/import
method: POST
requestMode: batch| Key | Purpose |
|---|---|
name | Pipeline identifier shown in logs and execution results. |
input | Source module configuration — see inputs. |
filters | Ordered list of transformations. Use [] for a pass-through chain. |
output | Destination module configuration — see outputs. |
filters is required even when empty. Use filters: [] for a
pass-through pipeline — leaving the key off makes validation fail with
missing required property "filters".
Optional top-level keys
name: sync-orders
version: 1.0.0
description: Forward paid orders to the internal import API.
tags:
- http-polling
- http-output
defaults:
timeoutMs: 15000
onError: log
retry:
maxAttempts: 2
delayMs: 1000
dryRunOptions:
previewLimit: 5
input: …
filters: …
output: …| Key | Purpose |
|---|---|
version | Free-form pipeline version string. Logged on start. |
description | One-line summary. Logged on start, shown in validate --verbose. |
tags | List of free-form tags. Useful for grouping pipelines in tooling. |
defaults | Cross-cutting defaults — see Defaults inheritance. |
dryRunOptions | Tuning for --dry-run — see Dry run. |
The pipeline lifecycle
- Parse — YAML is parsed into an internal config tree.
- Validate — the tree is validated against the JSON Schema for every module type. This catches typos, missing required fields, invalid enum values, and shape mismatches.
- Resolve env vars — every
${VAR}reference is substituted from the process environment. - Build runtime — inputs, filters, and outputs are instantiated.
- Run — the input emits batches of records; each batch flows through the filter chain in declared order; the output consumes what survives.
- Schedule or exit — if the input has a
schedule, the process stays alive and runs on CRON. Otherwise it exits with code 0.
The runtime stops on the first fatal error unless onError: skip or
onError: log is set somewhere along the chain — see
Retry & error handling.
Module shape
Every input, filter, and output has a type field that selects the
module, plus module-specific keys.
filters:
- type: mapping # selects the module
mappings: # module-specific options
- source: id
target: account_idThe full option set for every module is documented in the Modules reference.