Retry & error handling

Cannectors gives you two knobs for failures: retry (try the same thing again with backoff) and onError (what to do when retries are exhausted).

onError

Every module accepts an onError key. It picks what happens when the module fails after retries.

filters:
  - type: mapping
    onError: skip                  # drop records whose mapping fails
    mappings: …

  - type: http_call
    onError: log                   # keep going, log warnings on failures
    endpoint: …

For inputs and outputs, onError: skip is rarely useful — there's no record to skip yet for an input, and the output is the last stage anyway. fail and log are the common choices there.

Retry block

For HTTP-shaped and SOAP-shaped modules (httpPolling, http_call, httpRequest, soapPolling, soap_call, soapRequest), the retry block defines how the module retries before giving up and applying onError.

retry:
  maxAttempts: 3                   # total attempts, including the first
  delayMs: 500                     # initial backoff
  backoffMultiplier: 2             # exponential factor
  maxDelayMs: 5000                 # cap on the backoff
  retryableStatusCodes:
    - 429
    - 500
    - 502
    - 503
    - 504
  useRetryAfterHeader: true

Fields

What gets retried

• HTTP status codes in retryableStatusCodes.
• Transport errors (TCP refused, DNS, TLS handshake failures, read timeouts).
• 429 Too Many Requests with a Retry-After header — Cannectors sleeps for the indicated duration before retrying, ignoring the computed backoff.

Everything else fails immediately, no retry.

Defaults inheritance

Drop a retry block into top-level defaults and every HTTP-shaped module inherits it. Module-level retry blocks merge field-by-field with the default: declared module fields override the default, while omitted fields keep the inherited value. See Defaults inheritance for the exact rules.

defaults:
  onError: log
  retry:
    maxAttempts: 3
    delayMs: 500
    backoffMultiplier: 2
    retryableStatusCodes: [429, 500, 502, 503, 504]

What "fail" actually does

When a module fails with onError: fail (or is left as default and fails), Cannectors:

1. Logs the error at ERROR level with context (pipeline, module type, record index, root cause).
2. Stops the pipeline immediately. No further records are fetched or processed.
3. Exits the process with code 3 (runtime errors). For scheduled pipelines, the process exits — your supervisor decides whether to restart.

Exit codes

validate produces 0, 1, or 2. run adds 3 to that.

Cross-references