Logging
JSON log format, shipping, what to monitor.
The runtime emits structured logs to stdout by default. Use
--log-file to redirect, or let your container platform pick stdout
up automatically.
Format
Line-delimited JSON, one event per line.
{"time":"2026-04-21T12:34:56Z","level":"info","pipeline":"sync-orders","module":"httpPolling","msg":"fetched","records":128,"cursor":"eyJpZCI6MTI4fQ"}
{"time":"2026-04-21T12:34:57Z","level":"info","pipeline":"sync-orders","module":"mapping","msg":"processed","records_in":128,"records_out":128,"dropped":0}
{"time":"2026-04-21T12:34:58Z","level":"warn","pipeline":"sync-orders","module":"http_call","msg":"cache miss","key":"cust_42"}Standard fields on every event:
| Field | Meaning |
|---|---|
time | RFC 3339 UTC. |
level | debug, info, warn, error. |
pipeline | The pipeline's name. |
module | The module type that emitted the event. |
msg | Short human label. Stable enough to use in alerts. |
Plus module-specific fields (record counts, cursor values, HTTP status, etc.).
Levels
| Level | When |
|---|---|
debug | Per-record traces. Only emitted with --verbose. |
info | Stage start/end, batch totals, cache hits, scheduled-tick fires. Default. |
warn | Retry attempts, soft failures, deprecated YAML fields. |
error | Stage failed past onError. The runtime is about to abort. |
error events always include the root cause and the record index
that triggered it.
What gets redacted
Logs are shipped, indexed, and retained — often somewhere with a much broader audience than the pipeline itself. Every URL the runtime logs or puts in an error message is stripped first:
| Stripped | Example |
|---|---|
| Query string | ?api_key=s3cret → gone entirely |
| Fragment | #section → gone |
| Embedded credentials | https://user:hunter2@host/x → https://host/x |
A URL that fails to parse is logged as [invalid URL] rather than
printed raw — a malformed value is no reason to leak it.
The query string goes in full, not key by key. Cannectors makes no
attempt to tell an innocuous ?page=2 from a ?token=..., so a
sanitized endpoint in a log line is the path only. When you need the
paging parameters to debug something, read them from the module's own
fields rather than the endpoint.
This covers URLs. It does not scrub your own data: a record field
holding a secret is logged like any other field under --verbose, and a
destination that echoes a token back in an error body will have that
body logged. Keep secrets in environment variables and out of records —
see secrets management.
Shipping
Pipe stdout into your aggregator:
| Aggregator | Pattern |
|---|---|
| Loki | Promtail tailing the container stdout |
| Datadog | Datadog Agent autodiscovery on the container |
| CloudWatch | ECS/EKS log driver, no extra config |
| Vector | Sidecar tailing /var/log/cannectors/*.log |
| OpenTelemetry | otelcol with the filelog receiver |
All of them parse line-delimited JSON without further work — point them at the stream and the fields land as labels/attributes.
What to monitor
Alert on:
level=error— any error event is worth a page. Cannectors doesn't recover from them automatically.msg=retryat high cardinality — sustained retries usually mean the destination is degrading.- No
msg=tickevent in N CRON intervals — the scheduler is stuck or the process is dead.
Don't alert on every level=warn — they cover normal operating
conditions (transient retries, cache misses, etc.).
Local dev
--verbose adds per-record trace lines and is the right mode for
debugging a pipeline locally. --quiet flips it the other way —
nothing on success.
cannectors run --verbose pipeline.yaml | jq -c '.'jq -c keeps the line-delimited format but pretty-prints individual
events as you watch them roll by.