Architecture

Cannectors is a single Go process organized as four cooperating layers. Records move through them in one direction.

The four layers

┌─────────────────────────────────────────────────┐
│  cmd/cannectors      CLI entry point            │
│  ↓ parses flags, picks a subcommand             │
├─────────────────────────────────────────────────┤
│  internal/config     YAML → pipeline tree       │
│  ↓ validates against JSON Schema                │
│  ↓ resolves ${ENV} references                   │
├─────────────────────────────────────────────────┤
│  internal/runtime    builds modules, scheduler  │
│  ↓ wires input → filters → output               │
├─────────────────────────────────────────────────┤
│  internal/modules    inputs · filters · outputs │
│     ↓ each module is one Go package             │
└─────────────────────────────────────────────────┘

Each layer depends only on the layer above. The runtime never reaches into cmd/, modules never reach into runtime/.

cmd/cannectors

The CLI. Parses subcommand + flags, opens the YAML, hands off to internal/config. Tiny — under 200 lines.

internal/config

Owns the pipeline tree — the in-memory representation between "YAML on disk" and "running modules". Responsibilities:

• Parse YAML / JSON into a typed config struct.
• Validate against the JSON Schemas under internal/config/schema/*.json.
• Substitute ${ENV} references.
• Apply top-level defaults to module-level configs.
• Surface validation errors with line + column.

The schemas live here, not in the modules, because the validator needs to type-check before any module code runs.

internal/runtime

Turns the validated config into running modules and wires them together. Responsibilities:

• Instantiate one input, N filters, one output from their config.
• Build the scheduler if the input has a schedule.
• Build the per-batch executor that feeds records through the filter chain.
• Manage the StateStore shared between input and executor.
• Handle process signals (SIGINT/SIGTERM) → graceful drain.

The runtime is what gives "input → filters → output" its strong single-direction guarantee. Modules don't know about each other; the runtime is the only place that holds the whole graph.

internal/modules/{input,filter,output}/

The leaf layer. One Go package per module type. Each module implements a small interface (Fetch / Process / Send depending on the slot) plus a Config struct that the validator can populate.

Module packages own:

• Their own retry, auth, cache, and template logic — these are package-private helpers.
• Tests against the real protocol (HTTP servers, sqlmock for SQL).
• Their slice of the JSON Schema is what we vendor into this site.

See Module extensibility for the contract a new module needs to satisfy.

Where to start reading

See also