http_call

The http_call filter makes an HTTP request per record, then merges the response back onto the record. Supports path / query / header keys extracted from record fields, LRU + TTL caching, and three merge strategies.

Minimal example

filters:
  - type: http_call
    endpoint: https://profiles.example.com/api/customers/{customerId}
    method: GET
    keys:
      - field: customer_id
        paramType: path
        paramName: customerId
    mergeStrategy: merge

Options

property	type	default	description
typerequired	"http_call"	—	Module type discriminator. Must be `http_call` for this module.
id	string	—	Unique identifier within the pipeline.
name	string	—	Human-readable name.
description	string	—	—
enabled	boolean	true	Whether module is active.
tags	array<string>	—	—
onError	string	"fail"	Default error action. Case-insensitive; normalized to lowercase by the runtime.
endpointrequired	string	—	HTTP endpoint URL. Templates are accepted; the final URL is validated at runtime after template resolution.
method	object	—	—
headers	map<string, string>	—	—
queryParams	map<string, string>	—	Query parameters.
body	string	—	Inline request body. Templates ({{record.field}}) are evaluated at runtime. The body is sent regardless of the HTTP method.
bodyTemplateFile	string	—	Path to an external template file used as the request body. Mutually exclusive with `body` is not enforced; if both are set, the inline body wins.
authentication	union (4 variants)	—	—
timeoutMs	integer	—	Request timeout in milliseconds. When omitted, each module applies its own runtime default.
keys	array<object>	—	List of key configurations for extracting values from records and using them in requests.
cache	object	—	—
mergeStrategy	string	"merge"	How to merge HTTP response with input records. mergereplaceappend
dataField	string	—	Field to extract from HTTP response.
retry	object	—	—

Keys

A key extracts a value from the record and uses it in the outgoing request, in one of three positions:

keys:
  - field: customer.id         # dot path on the record
    paramType: path            # path | query | header
    paramName: customerId

`paramType`	Effect
`path`	Replaces the `{paramName}` placeholder in `endpoint`.
`query`	Appended as `?paramName=value` to the URL.
`header`	Sent as the HTTP header named `paramName`.

Each key is one entry — list multiple if you need more than one parameter.

Caching

Per-record HTTP calls multiply quickly. The built-in cache (LRU with TTL) deduplicates calls that resolve to the same key.

cache:
  enabled: true
  maxSize: 1000              # default 1000
  ttlSeconds: 600            # default 300
  key: "{{record.customerId}}"  # template

Field	Meaning
`enabled`	Master switch.
`maxSize`	LRU capacity. Older entries are evicted past this.
`ttlSeconds`	Per-entry TTL.
`key`	Optional template — if omitted, the resolved URL is the cache key.

If you don't set keys or cache.key, every record resolves to the same URL and you'll observe only one HTTP request per pipeline run — because the cache has one slot for every record. To force a per-record call, configure at least one key or set cache.key to a per-record template.

Merge strategies

What to do with the response, given that it returns alongside an already-shaped record:

Strategy	Effect
`merge` (default)	Deep-merge response fields into the record. Nested objects are merged; response values overwrite conflicts.
`replace`	Overlay response fields onto the record. Existing fields not present in the response are preserved.
`append`	Store the response under `_response` (the filter has no `resultKey` of its own).

mergeStrategy: merge       # response fields overlay on the record

For append, the response sits at _response until you mapping/remove it explicitly.

Examples

examples/14-http-call-get-merge-cache.yaml— GET + merge + cacheview source ↗

14-http-call-get-merge-cache.yaml

name: http-call-get-merge-cache
version: 1.0.0
description: Enrich each record with an HTTP GET response and merge it recursively.
tags:
  - http-call
  - cache
input:
  type: httpPolling
  schedule: "*/20 * * * *"
  endpoint: https://source.example.com/api/orders
  dataField: orders
filters:
  - type: http_call
    endpoint: https://profiles.example.com/api/customers/{customerId}
    method: GET
    keys:
      - field: customer.id
        paramType: path
        paramName: customerId
    dataField: profile
    mergeStrategy: merge
    cache:
      enabled: true
      maxSize: 10000
      ttlSeconds: 900
      key: customer.id
output:
  type: httpRequest
  endpoint: https://destination.example.com/api/orders/enriched
  method: POST
  requestMode: batch

examples/15-http-call-query-header-append.yaml— query + header keys + appendview source ↗

15-http-call-query-header-append.yaml

name: http-call-query-header-append
version: 1.0.0
description: Use query and header keys for an HTTP enrichment call and append the response.
tags:
  - http-call
  - append
input:
  type: httpPolling
  schedule: "*/20 * * * *"
  endpoint: https://source.example.com/api/tickets
  dataField: tickets
filters:
  - type: http_call
    endpoint: https://support.example.com/api/sla
    method: GET
    headers:
      X-Tenant: "{{record.tenantId}}"
    keys:
      - field: priority
        paramType: query
        paramName: priority
      - field: tenantId
        paramType: header
        paramName: X-Tenant-Id
    mergeStrategy: append
    cache:
      enabled: true
      maxSize: 1000
      ttlSeconds: 300
output:
  type: httpRequest
  endpoint: https://destination.example.com/api/tickets
  method: POST
  requestMode: batch

examples/16-http-call-post-template-replace.yaml— POST + body template + replaceview source ↗

16-http-call-post-template-replace.yaml

name: http-call-post-template-replace
version: 1.0.0
description: Use POST enrichment with a body template and replace overlapping response fields.
tags:
  - http-call
  - body-template
input:
  type: httpPolling
  schedule: "0 * * * *"
  endpoint: https://source.example.com/api/addresses
  dataField: addresses
filters:
  - type: http_call
    endpoint: https://geo.example.com/api/normalize
    method: POST
    headers:
      X-Request-Source: cannectors
    bodyTemplateFile: examples/assets/templates/geocode_request.json
    dataField: result
    mergeStrategy: replace
    cache:
      enabled: true
      maxSize: 5000
      ttlSeconds: 86400
      key: address.hash
output:
  type: httpRequest
  endpoint: https://destination.example.com/api/addresses/normalized
  method: POST
  requestMode: batch

http_call

Minimal example

Options

Keys

Caching

Merge strategies

Examples

Cross-references

sql_call filter

Authentication

Retry & error handling

On this page