Input → Filters → Output

Every pipeline is the same three-stage shape. The input produces records; the filter chain transforms or drops them; the output consumes what survives. Filters run in declared order, top to bottom.

The flow

┌─────────┐   batch    ┌──────────┐   record    ┌──────────┐
│  INPUT  │ ─────────▶ │ FILTERS  │ ──────────▶ │  OUTPUT  │
└─────────┘  of N      └──────────┘   one by    └──────────┘
                                       one

1. The input fetches a batch of records (one HTTP response page, one SQL query result, one webhook delivery).
2. The runtime hands each record to the filter chain, one at a time, in the order filters are declared.
3. Each filter can transform the record, drop it, or pass it through.
4. Records that reach the end of the chain are buffered, then handed to the output.
5. The output sends records either per-record (requestMode: single) or as a batch (requestMode: batch).

What's a "record"?

A record is one row of data — a JSON object, basically. Inputs unwrap the source's payload format to produce a list of records:

• httpPolling reads an array from a JSON response (dataField: orders pulls response.orders).
• webhook treats the POST body as a single record, or unwraps a dataField if configured.
• database returns one record per row.

Records are passed by value between filters — you can mutate them freely without affecting upstream stages. See Records for the access model.

Filters in order

The order in the YAML is the order in execution. This is significant.

filters:
  - type: mapping        # 1. flatten fields, normalize
    mappings: …
  - type: condition      # 2. drop unpaid orders
    expression: "status == 'paid'"
    else:
      - type: drop
  - type: http_call      # 3. enrich the ones that survived
    endpoint: …

If you swap the condition and http_call filters, you'd pay for HTTP enrichments on records you're about to drop. Order matters.

What filters can do

See Filters reference for every option of every filter.

When a filter drops a record

A dropped record is silently removed from the chain. Downstream filters and the output never see it. This is how a condition branch containing drop works, and it's also how a filter that errors out under onError: skip behaves.

The output stats will reflect the drop: it'll only see records that made it all the way through.

Pass-through pipelines

If you want input → output with no transformation, declare filters: []:

filters: []

Cannectors will still run validation, retry, auth, and state for you — you just get no record-level transformations.

Cross-references