Batch
Groups items in a source stream into fixed-size arrays.
Not the same node as the Batch under Sequence. The Palette also has a Batch node in the Sequence group, which groups plain messages by count or time interval, or concatenates complete input sequences from configured topics, and does not require a source stream. This page is about the Batch node in the Stream group, which groups items already flowing through a source stream by a fixed Size. Check the group shown when you drag the node onto the canvas if you're not sure which one you have. See Choosing between Source From, Loop, and Split/Join on the Source From page for how streams relate to other iteration approaches in a Flow.
A source stream carries items one at a time, each travelling as its own message. Batch collects them into groups.
Use a Batch node in your Flow when the work downstream is better done in groups than one item at a time —for example, sending records to an API that accepts several per request, writing rows in blocks rather than individually, or reducing the number of calls a stream makes to an external system.
Batch does not close the stream. The grouped arrays continue as stream items, so a Sink Collect node is still needed to finish, and nodes placed after Batch run once per group rather than once per record.
Batch only works inside a stream. It must sit downstream of a node that opens one, such as Source From or a Query Object node with Unfold all pages to source enabled. Batch requires a source stream.
To configure a Batch node in the Flow Editor, drag it from the Stream group in the Palette to the Workspace canvas, then double-click the node to open and configure its behavior.
Properties
Name — optional display name for the node in the editor. By default, the node shows a label of "Batch"; the value you provide here replaces it. The configured size is not shown on the canvas, so naming the node something like Batch of 3 makes a flow easier to read briefly.
Input — the property the node reads from, as a typed input. Defaults to msg.payload.
Size — the number of items to place in each group. Defaults to 2.
Group size and the final group
Items are grouped in the order they arrive. Each group is emitted as its own message, with the group written as an array.
Ten records with Size set to 2 produce five messages, each carrying an array of two.
When the number of items does not divide evenly by Size, no items are discarded — the final group simply holds fewer. Ten records with Size set to 3 produce four messages:
Nodes placed after a Batch node must not assume every group is full. Read the length of the array rather than indexing a fixed number of positions, or the final group will cause errors.
What downstream nodes receive
Batch changes the shape of each stream item, not the fact that items are still flowing. A Map or Filter node placed after a Batch node runs once per group and sees an array, not a single record. Expressions and property paths must be written accordingly.
A Sink Collect node placed after a Batch node collects the groups, producing an array of arrays. Ten records batched by two collect as an array of five, each element an array of two.
To return a flat list of records to a caller, place Sink Collect directly after the stream source and do the grouping elsewhere or flatten the collected result after Sink Collect.
Alternatively, if downstream nodes need to process the batches before returning a flat list, add a Map node after that processing and before Sink Collect. Set Action to Concat, with both from path and to set to payload. This expands the batches back into individual stream items before they are collected.
Batch requires a source stream
A Batch node must sit downstream of a node that opens a source stream. This is enforced by the editor and at runtime. A Batch node with no stream source above it is marked with an error, and the flow reports a lint error. At runtime, it throws Error: source not initialized and emits no output.
The editor's lint error lets you catch the missing source while wiring the flow. Connect the node into a stream and the error clears.
Example
A weather application stores readings as records of the weather app Object Type. The /report4 endpoint returns the readings grouped in pairs rather than as a flat list.
Open a stream
Configure the Query Objects node with Type Id weather app, Query {}, Page Size 10, and Unfold all pages to source enabled. Ten records are emitted, each as its own message.
Configure the Batch node
Leave Input as msg.payload and set Size to 2.
Collect and respond
Add a Sink Collect node to close the stream, then an HTTP Response node.
Confirm the grouping
Place a Log node between Batch and Sink Collect and request /report4. The Debug panel shows five entries, each an array of two records:
A Log node placed after Sink Collect shows a single array[5], each element an array[2].
Changing Size to 3 produces four entries instead: three arrays of three and a final array of one.
Last updated
Was this helpful?