mapping

The mapping filter rewrites records: it reads a value from a source path, optionally runs it through a chain of transforms (trim, lowercase, dateFormat, …), and writes it to a target path.

Minimal example

filters:
  - type: mapping
    mappings:
      - source: order_id
        target: id
      - source: customer.email
        target: email
        transforms:
          - op: lowercase

Options

property	type	default	description
typerequired	"mapping"	—	Module type discriminator. Must be `mapping` for this module.
id	string	—	Unique identifier within the pipeline.
name	string	—	Human-readable name.
description	string	—	—
enabled	boolean	true	Whether module is active.
tags	array<string>	—	—
onError	string	"fail"	Default error action. Case-insensitive; normalized to lowercase by the runtime.
mappingsrequired	array<object>	—	—

Field mappings

Each entry in mappings is a fieldMapping — see the mappings option page for the full per-item shape.

`onMissing`

What to do when the source path is absent on the record:

Value	Behavior
`setNull` (default)	Set target to `null`.
`skipField`	Leave the target alone — don't write anything.
`useDefault`	Use `defaultValue`. Requires `defaultValue` to be set.
`fail`	Abort the pipeline (or trigger `onError`).

- source: customer.region
  target: region
  onMissing: useDefault
  defaultValue: "unknown"

Each entry in transforms is one operation applied left to right on the value pulled from source. The shape of each transformOp is documented on the mappings option page (under each item's transforms field).

The full list of op values:

`op`	Notes
`trim`	`strings.TrimSpace`
`lowercase`	`strings.ToLower`
`uppercase`	`strings.ToUpper`
`dateFormat`	Reformat with `format` field — `YYYY`, `YY`, `MM`, `DD`, `HH`, `mm`, `ss`, `SSS` tokens (a raw Go layout also works).
`replace`	Regex-based, requires `pattern` and `replacement`.
`split`	Split a string by `separator` (default `,`) into an array. Each part is trimmed.
`join`	Join an array by `separator` (default `,`) into a string.
`toString`, `toInt`, `toFloat`, `toBool`	Coerce types.
`toArray`	Wrap a scalar into a single-item array; an existing array passes through.
`toObject`	Pass an object through. It does not wrap a scalar — see below.

How transforms treat unexpected types

The ops fall into two groups, and the difference matters when a source field is not the type you expect:

String-only ops — trim, lowercase, uppercase, replace, split — return non-string values unchanged. trim on a number is a silent no-op, not an error, so a wrong source path can pass unnoticed. join behaves the same way for non-arrays.

Converters — toInt, toFloat, toBool, toObject — fail when they cannot convert, which aborts the pipeline unless the filter sets onError:

Input	Op	Result
`"42"`	`toInt`	`42`
`"not-a-number"`	`toInt`	error — cannot parse int from string
`1.5`	`toInt`	error — cannot convert float with fractional part
`"true"`	`toBool`	`true`
`"yes"`	`toBool`	error — only Go boolean literals are accepted (`true`, `false`, `1`, `0`, `t`, `f`)
`{ "k": "v" }`	`toObject`	unchanged
`"text"`	`toObject`	error — cannot convert string to object

dateFormat also errors on a string it cannot parse, but leaves non-string values untouched.

- source: customer.email
  target: contact.email
  transforms:
    - op: trim
    - op: lowercase
- source: created_at
  target: createdAt
  transforms:
    - op: dateFormat
      format: "2006-01-02T15:04:05Z07:00"

Deleting a field

Omit source to delete the target:

- target: internal_notes        # field is removed from the record

Examples

examples/10-mapping-transforms-all.yaml— every transform in one pipelineview source ↗

10-mapping-transforms-all.yaml

name: mapping-transforms-all
version: 1.0.0
description: Demonstrate every mapping transform implemented by the mapping filter.
tags:
  - mapping
  - transforms
input:
  type: httpPolling
  schedule: "0 * * * *"
  endpoint: https://source.example.com/api/raw-records
  dataField: records
filters:
  - type: mapping
    onError: log
    mappings:
      - source: name
        target: name
        transforms:
          - op: trim
          - op: uppercase
      - source: email
        target: email
        transforms:
          - op: trim
          - op: lowercase
      - source: phone
        target: phone
        transforms:
          - op: replace
            pattern: "[^0-9+]"
            replacement: ""
      - source: created_at
        target: createdDate
        transforms:
          - op: dateFormat
            format: YYYY-MM-DD
      - source: amount
        target: amountInt
        transforms:
          - op: toInt
      - source: score
        target: scoreFloat
        transforms:
          - op: toFloat
      - source: active
        target: active
        transforms:
          - op: toBool
      - source: tags_csv
        target: tags
        transforms:
          - op: split
            separator: ","
      - source: tag_list
        target: tagString
        transforms:
          - op: join
            separator: "|"
      - source: external_id
        target: externalId
        transforms:
          - op: toString
      - source: payload
        target: payloadArray
        transforms:
          - op: toArray
      - source: attributes
        target: attributes
        transforms:
          - op: toObject
      - target: deprecatedField
output:
  type: httpRequest
  endpoint: https://destination.example.com/api/normalized-records
  method: POST
  requestMode: batch

mapping

Minimal example

Options

Field mappings

`onMissing`

Transforms

How transforms treat unexpected types

Deleting a field

Examples

Cross-references

set filter

remove filter

script filter

On this page