Next Steps
Where to go after you've run your first pipeline.
You've got the binary installed, run a maintained example, and written your own pipeline. Here's where to go next, depending on what you're trying to do.
Build the right mental model
Cannectors has a small, fixed shape (input → filters → output) and a handful of cross-cutting concerns (auth, retries, state, scheduling). The Concepts section walks through them.
Pipelines
The full YAML structure: name, defaults, input, filters, output.
Input → Filters → Output
How records flow through a pipeline.
Defaults inheritance
Top-level defaults vs module-level overrides.
Retry & error handling
onError, retry policies, retryable codes, Retry-After.
Reach for a specific module
The Modules reference covers every input, filter, and output with all their options, defaults, and pitfalls — generated from the same JSON Schemas that the binary uses to validate.
Inputs
httpPolling, soapPolling, webhook, database — and how to wire each.
Filters
mapping, condition, script, http_call, soap_call, sql_call, set, remove, drop.
Outputs
httpRequest, soapRequest, database — single vs batch, transactions, retries.
Learn the CLI
The CLI is small — validate, run, version — but the flags matter.
CLI reference
Commands, flags, exit codes, examples.
Dry-run mode
--dry-run executes everything except the output side effects.
Crib from real pipelines
The examples/ directory contains 33 maintained YAML pipelines covering
every module and most combinations. Each is small, validated, and
documented.
Run it in production
Cannectors is one process. Use whatever process manager you already use
(systemd, Kubernetes, ECS, Nomad, your CI runner). The Operations
section covers the patterns that matter at scale.
Scheduling in production
Process supervisors, container lifecycles, signal handling.
State management
Durable storage paths, restart safety, cursor backfills.
Secrets management
Environment variables, secret managers, what not to log.
Contribute
Cannectors is Apache 2.0. The codebase is small and well-organized — adding a new module or transform is a matter of implementing one interface.