Records

A record is the unit of data that moves through a pipeline. It's a JSON object — string, number, boolean, null, array, or nested object as values.

{
  "id": "ord_1Q2W3E",
  "status": "paid",
  "customer": {
    "email": "Hello@Example.com",
    "country": "FR"
  },
  "items": [
    { "sku": "SKU-1", "qty": 2 },
    { "sku": "SKU-2", "qty": 1 }
  ]
}

There's no schema requirement. Cannectors doesn't enforce field presence or types between filters — that's your call, per pipeline.

Where records come from

Field paths

Most modules accept dot-notated paths to point at nested fields:

filters:
  - type: mapping
    mappings:
      - source: customer.email         # reads customer.email
        target: email                  # writes top-level email
        transforms:
          - op: lowercase

Paths work for reading and writing. Writing to a missing intermediate object creates it.

- type: set
  target: metadata.source
  value: cannectors                    # creates `metadata` if absent

Array indexing isn't supported in paths — work on arrays inside a script filter if you need to iterate.

Templates

Request fields (HTTP endpoints, headers, bodies, SOAP envelopes, cache keys) are Jinja templates evaluated once per record, with escaping matched to the field's format applied automatically:

output:
  type: httpRequest
  endpoint: https://api.example.com/customers/{{ record.customerId }}
  method: GET

SQL queries use a dedicated model: the query text carries $1, $2, … placeholders and a parameters list of expressions supplies the values, bound by the driver — never string-spliced into the SQL.

See Templating for the full syntax (filters, {% if %} blocks, contexts, escaping rules).

Mutation between filters

Each filter receives a record, can mutate it, and returns it. The next filter sees the mutated version. Inputs are not consulted again once they've handed off a batch.

This is important for filters like http_call, soap_call, and sql_call with mergeStrategy: merge — they overlay the fetched fields onto the current record before passing it down the chain.

Reserved keys

Two top-level keys belong to the runtime. They are not yours to write, and reading them is how you get at what the runtime knows about a record.

Every filter that writes a record field rejects a target that reaches into _errors — set and mapping through their target, http_call, soap_call and sql_call through their resultKey — and the pipeline fails to start rather than letting a record forge its own success. The check runs when the filter is built, so it surfaces on cannectors run, not on cannectors validate. Only the root segment is reserved: payload._errors is your own field and stays writable.

Both keys are absent when nothing put them there — a record that never failed has no _errors key at all, rather than an empty list. Guard for it with _errors != nil before indexing or counting.

Cross-references