cannectors
ModulesInputs

httpPolling

GET an HTTP endpoint on a CRON schedule, with pagination and state.

The httpPolling input GETs a JSON endpoint, extracts an array of records from the response, and emits each batch into the filter chain. With a schedule, it runs on CRON. Without one, it runs once.

Minimal example

input:
  type: httpPolling
  schedule: "*/15 * * * *"
  endpoint: https://source.example.com/api/orders
  dataField: orders

Options

propertytypedefaultdescription
typerequired
"httpPolling"—Module type discriminator. Must be `httpPolling` for this module.
id
string—Unique identifier within the pipeline.
name
string—Human-readable name.
description
string——
enabled
booleantrueWhether module is active.
tags
array<string>——
onError
string"fail"Default error action. Case-insensitive; normalized to lowercase by the runtime.
endpointrequired
string—HTTP endpoint URL. Templates are accepted; the final URL is validated at runtime after template resolution.
method
object——
headers
map<string, string>——
queryParams
map<string, string>—Query parameters.
body
string—Inline request body. Templates ({{record.field}}) are evaluated at runtime. The body is sent regardless of the HTTP method.
bodyTemplateFile
string—Path to an external template file used as the request body. Mutually exclusive with `body` is not enforced; if both are set, the inline body wins.
authentication
union (4 variants)——
timeoutMs
integer—Request timeout in milliseconds. When omitted, each module applies its own runtime default.
schedule
string—Optional CRON expression for polling. Validated at runtime.
pagination
object——
statePersistence
object——
dataField
string—JSON field path containing the array of records to extract from the response.
retry
object——

Pagination

httpPolling supports three pagination styles. Each requires a different combination of fields.

Page-based

pagination:
  type: page
  param: page              # query param name
  limitParam: per_page
  limit: 100
  totalPagesField: meta.total_pages

Offset-based

pagination:
  type: offset
  param: offset
  limitParam: limit
  limit: 250
  totalField: meta.total

Cursor-based

pagination:
  type: cursor
  param: cursor
  limitParam: per_page
  limit: 100
  nextCursorField: meta.next_cursor

The runtime keeps requesting pages until the source signals there are no more.

limit only reaches the source when limitParam names the query parameter to carry it. limit: 100 on its own is silently ignored — always set both, for all three styles.

Reading nested fields

dataField, nextCursorField, totalPagesField and totalField all accept a dot-separated path, resolved segment by segment from the response root — meta.next_cursor reads next_cursor inside the meta object. There is no escaping, so a response key that literally contains a dot can't be addressed.

A nextCursorField may point at a JSON string or a JSON number; a number is used as-is. Anything else — a boolean, an object, null — reads as "no cursor" and ends the pagination, so don't point the field at a flag like has_more.

State persistence

To resume from where you left off between runs, configure statePersistence. The state file is written after every successful batch and read on startup.

statePersistence:
  timestamp:
    enabled: true
    queryParam: updated_after
  id:
    enabled: true
    field: id
    queryParam: after_id
  storagePath: ./.cannectors-state

See State persistence for the full mental model and per-environment storage recommendations.

Authentication

authentication accepts the standard four schemes — api-key, bearer, basic, oauth2. See Authentication for examples.

Examples

examples/01-http-polling-basic-to-http-batch.yamlview source ↗
01-http-polling-basic-to-http-batch.yaml
name: http-polling-basic-to-http-batch
version: 1.0.0
description: Poll a JSON array from an HTTP API and send it as one batch request.
tags:
  - http-polling
  - http-output
input:
  type: httpPolling
  schedule: "*/15 * * * *"
  endpoint: https://source.example.com/api/orders
  headers:
    Accept: application/json
  dataField: orders
filters: []
output:
  type: httpRequest
  endpoint: https://destination.example.com/api/orders/import
  method: POST
  requestMode: batch
  headers:
    Content-Type: application/json
  success:
    statusCodes:
      - 200
      - 201
      - 202
examples/02-http-polling-page-pagination.yamlview source ↗
02-http-polling-page-pagination.yaml
name: http-polling-page-pagination
version: 1.0.0
description: Poll page-number pagination and normalize fields before output.
tags:
  - http-polling
  - pagination
input:
  type: httpPolling
  schedule: "0 */1 * * *"
  endpoint: https://source.example.com/api/customers
  dataField: data
  pagination:
    type: page
    param: page
    limitParam: per_page
    limit: 100
    totalPagesField: total_pages
filters:
  - type: mapping
    mappings:
      - source: id
        target: customer.id
        transforms:
          - op: toString
      - source: email
        target: customer.email
        transforms:
          - op: trim
          - op: lowercase
      - source: name
        target: customer.name
        onMissing: useDefault
        defaultValue: Unknown
output:
  type: httpRequest
  endpoint: https://destination.example.com/api/customers
  method: POST
  requestMode: batch
examples/03-http-polling-offset-pagination-state.yamlview source ↗
03-http-polling-offset-pagination-state.yaml
name: http-polling-offset-pagination-state
version: 1.0.0
description: Poll offset pagination with timestamp and ID state persistence.
tags:
  - http-polling
  - state
input:
  type: httpPolling
  schedule: "*/10 * * * *"
  endpoint: https://source.example.com/api/events
  dataField: events
  pagination:
    type: offset
    param: offset
    limitParam: limit
    limit: 250
    totalField: total
  statePersistence:
    timestamp:
      enabled: true
      queryParam: updated_after
    id:
      enabled: true
      field: event.id
      queryParam: after_id
    storagePath: ./.cannectors-state
filters:
  - type: set
    target: metadata.source
    value: events-api
  - type: remove
    target:
      - debug
      - internal.notes
output:
  type: httpRequest
  endpoint: https://destination.example.com/api/events
  method: POST
  requestMode: batch
examples/04-http-polling-cursor-oauth2.yamlview source ↗
04-http-polling-cursor-oauth2.yaml
name: http-polling-cursor-oauth2
version: 1.0.0
description: Poll cursor pagination with OAuth2 client credentials.
tags:
  - http-polling
  - oauth2
input:
  type: httpPolling
  schedule: "0 */6 * * *"
  endpoint: https://source.example.com/api/invoices
  dataField: items
  authentication:
    type: oauth2
    credentials:
      tokenUrl: https://source.example.com/oauth/token
      clientId: ${SOURCE_CLIENT_ID}
      clientSecret: ${SOURCE_CLIENT_SECRET}
      scope: invoices.read customers.read
  pagination:
    type: cursor
    param: cursor
    limitParam: limit
    limit: 100
    nextCursorField: next_cursor
filters:
  - type: condition
    expression: status == "paid"
    else:
      - type: drop
output:
  type: httpRequest
  endpoint: https://destination.example.com/api/invoices
  method: POST
  requestMode: batch

Cross-references

Authentication

The auth schemes available for HTTP inputs.

Retry & error handling

Configure how httpPolling reacts to failures.

State persistence

Resume between scheduled runs.

Scheduling

CRON expressions and run-once vs scheduled.

Inputs

Modules that produce records — HTTP polling, SOAP polling, webhooks, databases.

authentication

authentication configuration for the httpPolling module.

On this page

Minimal exampleOptionsPaginationPage-basedOffset-basedCursor-basedReading nested fieldsState persistenceAuthenticationExamplesCross-references