cannectors / v1.0
01 / declarative data pipelines

Pipelinesdeclared, not coded.On any platform.

A single Go binary and a YAML file for HTTP, webhook, and database pipelines with auth, retries, state, and dry-run previews.

binary
~14 MB
examples
34 maintained
license
apache 2.0
tty · cannectorsrun
fig.01 · pipeline execution tracelive
02 / anatomy

One declarative shape:input → filters → output.

Every pipeline is the same three-block YAML. The shape never changes — only the modules slotted into each block.

01 / input

Pull records.

Polled HTTP or SOAP APIs (page · offset · cursor), webhook listeners (HMAC, queue, rate limit), or SQL databases — all driven by one block of YAML.

  • httpPolling
  • soapPolling
  • webhook
  • database
02 / filters

Compose transforms.

Map fields, route on conditions, iterate arrays, enrich via HTTP, SOAP, or SQL, run JavaScript, set or remove keys. Filters run in declared order, top to bottom.

  • mapping
  • condition
  • loop
  • script
  • http_call
  • soap_call
  • sql_call
  • set
  • remove
  • drop
03 / output

Ship anywhere.

POST to an HTTP endpoint, call a SOAP service, or upsert into a relational database via prepared queries — with retries, transactions, and auth.

  • httpRequest
  • soapRequest
  • database
03 / specifications

Production concerns,wired in from configuration.

Retries, auth, secrets, schedules, state, dry-runs — the parts you'd rebuild by hand every time you wire a new integration.

  • linux · darwin · windows

    One binary, any OS.

    Static Go build for Linux, macOS, and Windows. No runtime, no Python venv, no Docker required — just a CLI you can drop in a CI image.

  • json-schema · cannectors validate

    YAML you can read at 2 AM.

    Pipelines validate against a JSON Schema. cannectors validate gives line-accurate errors before anything runs. Reviewers see intent, not glue code.

  • exponential · jittered · capped

    Retries that match reality.

    Backoff, max delay, retryable status codes, and Retry-After header support — declared once at the defaults level, inherited everywhere unless overridden.

  • oauth2 · bearer · basic · api-key

    Auth out of the box.

    OAuth2 client_credentials, Bearer, Basic, and API key (header or query). Secrets stay in env vars — never in the YAML, never in logs.

  • --dry-run · output preview

    Dry-run before you ship.

    cannectors run --dry-run executes everything up to the output stage and previews what would be sent. Confidence before a destination ever sees a byte.

  • cursor · id · timestamp

    State for incremental loads.

    Cursors, last-seen IDs, and timestamps persist between runs. Restart-safe by default. Schedule on CRON and resume where you left off.

04 / specimen

The whole pipeline,on one printable page.

Real examples from the maintained examples/ set. Annotations explain the parts that aren't obvious yet.

Poll an API on a schedule, transform, ship a batch.

pipeline.yamlview · annotated
name: sync-orders
version: 1.0.0
input:
type: httpPolling
schedule: "*/15 * * * *"
endpoint: https://source.example.com/api/orders
dataField: orders
authentication:
type: oauth2
credentials:
tokenUrl: ${OAUTH_TOKEN_URL}
clientId: ${OAUTH_CLIENT_ID}
clientSecret: ${OAUTH_CLIENT_SECRET}
filters:
- type: mapping
mappings:
- source: order_id
target: id
- source: customer.email
target: email
transforms:
- op: lowercase
- type: condition
expression: "status == 'paid'"
else:
- type: drop
output:
type: httpRequest
endpoint: https://warehouse.example.com/api/orders/import
method: POST
requestMode: batch
retry:
maxAttempts: 3
delayMs: 500
backoffMultiplier: 2
retryableStatusCodes: [429, 500, 502, 503, 504]
05 / catalog

16 modules.Every option documented.

The full module surface, with options, defaults, and pitfalls generated from the same JSON Schemas the binary uses to validate.

06 / ai context

Raw project context for pipeline generation.

The AI context page is a plain text endpoint containing the Cannectors documentation, module options, JSON Schemas, and all maintained YAML examples so online agents can generate valid pipelines with full context.

open ai context
07 / ship

Ship pipelines, not glue code.

Install the CLI, copy one of the 34 maintained examples, run it locally with --dry-run, then schedule it. The whole loop fits inside a coffee break.

go install github.com/alexandrecano/cannectors/cmd/cannectors@latest