Retry & error handling

Cannectors gives you two knobs for failures: retry (try the same thing again with backoff) and onError (what to do when retries are exhausted).

onError

Every module accepts an onError key. It picks what happens when the module fails after retries.

filters:
  - type: mapping
    onError: skip                  # drop records whose mapping fails
    mappings: …

  - type: http_call
    onError: log                   # keep going, annotate the failures
    endpoint: …

For inputs and outputs, onError: skip is rarely useful — there's no record to skip yet for an input, and the output is the last stage anyway. fail and log are the common choices there.

onError and output batches

An output emits one request per record (requestMode: single), one request for everything (requestMode: batch), or one request per batch of N records (requestMode: batch with batchSize). onError is evaluated per request, so what it skips depends on the mode:

The last row is the one to weigh: splitting a batch trades atomicity for a smaller blast radius. A skip there discards every record of the failing batch, not just the one the destination objected to.

A run that dropped records this way is still a success — skip and log exist precisely so the pipeline finishes. It is not a silent success though: the run reports the records it could not send, so a skip that swallowed a batch is visible in the summary.

✓ Pipeline executed successfully
  Status: success
  Records processed: 70
  Records failed: 50

Records that went through in log mode are counted too, broken down by category — a run where every enrichment failed is a "success" that you still want to see:

✓ Pipeline executed successfully
  Status: success
  Records processed: 120
  Error markers: 12
    server: 9
    validation: 3

The total counts markers, not records: a record that failed an enrichment call and then a persistence call appears in both categories, because both failures happened.

Each marker is counted where it is created, so the count survives whatever the pipeline does with the record afterwards. Routing the rejects out — with drop, or a condition branch that does not forward them — is the point of the marker, and it does not erase the failure that made you route it. Records processed counts what reached the output; Error markers counts what went wrong.

Error markers

A module running with onError: log annotates the record it could not process, under the reserved key _errors. The record continues down the pipeline carrying what went wrong, so later filters and outputs can act on it.

{
  "id": "42",
  "_errors": [
    {
      "module": "http_call",
      "category": "validation",
      "retryable": false,
      "code": "HTTP_CALL_HTTP_ERROR",
      "statusCode": 400,
      "message": "http_call HTTP error 400: …",
      "attempts": 1,
      "recordIndex": 3
    }
  ]
}

Fields

_errors is a list: a record can fail in several filters along the chain, and each failure appends its own marker. The first one is usually the root cause.

Default classification

A status code only appears in the marker when it caused the failure. A 200 whose body turns out to be unparseable leaves a marker without statusCode, so a _errors[0].statusCode >= 400 condition never mistakes it for a success.

Routing on a marker

- type: condition
  expression: "_errors != nil and _errors[0].retryable == false"
  then:
    - type: set
      target: routing.status
      value: REJECTED

Overriding the classification

Some APIs answer 404 for "no match", which is an ordinary outcome rather than an anomaly. errorClassification restates what a status code means for one module:

- type: http_call
  onError: log
  endpoint: https://profiles.example.com/api/customers/{customerId}
  errorClassification:
    functional: [404]        # record as not retryable
    technical: [409]         # record as retryable

An override decides retryable. The category is kept whenever it does not contradict that verdict: a 404 declared functional stays not_found, it does not flatten into validation. It is only rewritten when the two disagree — a 503 declared functional becomes validation, a 409 declared technical becomes server. A code listed in both lists is treated as functional.

Reserved key

_errors belongs to the runtime. Every filter that writes a record field refuses a target that reaches into it, and the pipeline fails to start:

- type: set
  target: _errors          # rejected: reserved for the runtime's error markers
  value: []

- type: sql_call
  mergeStrategy: append
  resultKey: _errors       # rejected for the same reason

That covers set and mapping through their target, and http_call, soap_call and sql_call through their resultKey. A field further down a path is your own data and stays writable — payload._errors is fine.

This one is a startup check, not a schema check: cannectors validate accepts a reserved target, and cannectors run refuses to build the filter. Unlike append without resultKey above, it will not be caught by a validate-only CI gate.

Two filters are deliberate exceptions:

Both are escape hatches. Using them changes what reaches the output — it does not change the summary counters, which count each marker where it is created.

Retry block

For HTTP-shaped and SOAP-shaped modules (httpPolling, http_call, httpRequest, soapPolling, soap_call, soapRequest), the retry block defines how the module retries before giving up and applying onError.

retry:
  maxAttempts: 3                   # total attempts, including the first
  delayMs: 500                     # initial backoff
  backoffMultiplier: 2             # exponential factor
  maxDelayMs: 5000                 # cap on the backoff
  retryableStatusCodes:
    - 429
    - 500
    - 502
    - 503
    - 504
  useRetryAfterHeader: true

Fields

What gets retried

• HTTP status codes in retryableStatusCodes.
• Transport errors (TCP refused, DNS, TLS handshake failures, read timeouts).
• 429 Too Many Requests with a Retry-After header — Cannectors sleeps for the indicated duration before retrying, ignoring the computed backoff.

Everything else fails immediately, no retry.

Defaults inheritance

Drop a retry block into top-level defaults and every HTTP-shaped module inherits it. Module-level retry blocks merge field-by-field with the default: declared module fields override the default, while omitted fields keep the inherited value. See Defaults inheritance for the exact rules.

defaults:
  onError: log
  retry:
    maxAttempts: 3
    delayMs: 500
    backoffMultiplier: 2
    retryableStatusCodes: [429, 500, 502, 503, 504]

What "fail" actually does

When a module fails with onError: fail (or is left as default and fails), Cannectors:

1. Logs the error at ERROR level with context (pipeline, module type, record index, root cause).
2. Stops the pipeline immediately. No further records are fetched or processed.
3. Exits the process with code 3 (runtime errors). For scheduled pipelines, the process exits — your supervisor decides whether to restart.

Exit codes

validate produces 0, 1, or 2. run adds 3 to that.

Cross-references