mapping
Map source fields onto target paths, with chained transforms.
The mapping filter rewrites records: it reads a value from a source
path, optionally runs it through a chain of transforms (trim,
lowercase, dateFormat, …), and writes it to a target path.
Minimal example
filters:
- type: mapping
mappings:
- source: order_id
target: id
- source: customer.email
target: email
transforms:
- op: lowercaseOptions
| property | type | default | description |
|---|---|---|---|
typerequired | "mapping" | — | Module type discriminator. Must be `mapping` for this module. |
id | string | — | Unique identifier within the pipeline. |
name | string | — | Human-readable name. |
description | string | — | — |
enabled | boolean | true | Whether module is active. |
tags | array<string> | — | — |
onError | string | "fail" | Default error action. Case-insensitive; normalized to lowercase by the runtime. |
mappingsrequired | array<object> | — | — |
Field mappings
Each entry in mappings is a fieldMapping — see the
mappings option page for the full per-item shape.
onMissing
What to do when the source path is absent on the record:
| Value | Behavior |
|---|---|
setNull (default) | Set target to null. |
skipField | Leave the target alone — don't write anything. |
useDefault | Use defaultValue. Requires defaultValue to be set. |
fail | Abort the pipeline (or trigger onError). |
- source: customer.region
target: region
onMissing: useDefault
defaultValue: "unknown"Transforms
Each entry in transforms is one operation applied left to right on
the value pulled from source. The shape of each transformOp is
documented on the mappings option page (under
each item's transforms field).
The full list of op values:
op | Notes |
|---|---|
trim | strings.TrimSpace |
lowercase | strings.ToLower |
uppercase | strings.ToUpper |
dateFormat | Reformat with format field — accepts Go layout strings. |
replace | Regex-based, requires pattern and replacement. |
split | Split a string by separator into an array. |
join | Join an array by separator into a string. |
toString, toInt, toFloat, toBool | Coerce types. |
toArray, toObject | Wrap a scalar into a single-item array / single-key object. |
- source: customer.email
target: contact.email
transforms:
- op: trim
- op: lowercase
- source: created_at
target: createdAt
transforms:
- op: dateFormat
format: "2006-01-02T15:04:05Z07:00"Deleting a field
Omit source to delete the target:
- target: internal_notes # field is removed from the recordExamples
examples/10-mapping-transforms-all.yamlview source ↗
name: mapping-transforms-all
version: 1.0.0
description: Demonstrate every mapping transform implemented by the mapping filter.
tags:
- mapping
- transforms
input:
type: httpPolling
schedule: "0 * * * *"
endpoint: https://source.example.com/api/raw-records
dataField: records
filters:
- type: mapping
onError: log
mappings:
- source: name
target: name
transforms:
- op: trim
- op: uppercase
- source: email
target: email
transforms:
- op: trim
- op: lowercase
- source: phone
target: phone
transforms:
- op: replace
pattern: "[^0-9+]"
replacement: ""
- source: created_at
target: createdDate
transforms:
- op: dateFormat
format: YYYY-MM-DD
- source: amount
target: amountInt
transforms:
- op: toInt
- source: score
target: scoreFloat
transforms:
- op: toFloat
- source: active
target: active
transforms:
- op: toBool
- source: tags_csv
target: tags
transforms:
- op: split
separator: ","
- source: tag_list
target: tagString
transforms:
- op: join
separator: "|"
- source: external_id
target: externalId
transforms:
- op: toString
- source: payload
target: payloadArray
transforms:
- op: toArray
- source: attributes
target: attributes
transforms:
- op: toObject
- target: deprecatedField
output:
type: httpRequest
endpoint: https://destination.example.com/api/normalized-records
method: POST
requestMode: batch