Generators

cross.stream generators use Nushell closures to create streams of data that are emitted as frames into the store.

You can also compose external request-response servers, particularly CLI tools that read requests on stdin and write responses to stdout, using duplex generators.

Basic Usage

To create a generator, append a Nushell script that evaluates to a configuration record with a run closure using the topic <topic>.spawn:

r#'{
  run: {|| ^tail -F http.log | lines }
}'# | .append log.spawn

The generator will:

• Execute the provided Nushell expression
• Output from the pipeline is streamed as log.recv frames. Text pipelines emit one frame per line, while ByteStream pipelines send binary chunks.
• Automatically restarts if it exits until a terminate frame is seen

Lifecycle Events

Generators emit lifecycle events to track their state. See for all component suffixes.

Event	Description
<topic>.running	Generator has started processing
<topic>.recv	Output value from the generator
<topic>.stopped	Generator pipeline has stopped. The `meta.reason` field is a string enum with values finished, error, terminate and update. When finished or error, the pipeline will be restarted automatically; terminate means it was stopped manually and the generator loop will shut down. update indicates the generator reloaded due to a new .spawn frame.
<topic>.parse.error	Script failed to parse
<topic>.shutdown	Generator loop has fully exited; ServeLoop evicts it

All events include source_id which is the ID of the generator instance. When a .stopped frame has meta.reason set to update, it also includes update_id referencing the spawn that triggered the reload. ServeLoop evicts a generator when it receives a <topic>.shutdown frame.

Configuration Options

Option	Type	Default	Description
duplex	boolean	false	Enable sending input to the generator’s pipeline via <topic>.send
return_options	record	—	Customize output frames (see Return Options)

The return_options field controls the suffix and TTL for the .recv frames produced by the generator.

Return Options

The return_options record allows you to customize the output frames produced by the generator:

Option	Type	Default	Description
suffix	string	”.recv”	Custom suffix for output frames (e.g., “.output”)
ttl	string	”forever”	Time-to-live for output frames

TTL Options

• "forever" - Frames never expire (default)
• "ephemeral" - Frames are removed immediately after processing
• "time:<ms>" - Frames expire after specified milliseconds
• "last:<n>" - Keep only the N most recent frames for each topic

Example with custom return options:

r#'{
  run: {|| ^tail -F access.log | lines },
  return_options: {
    suffix: ".line",
    ttl: "last:100"  # Keep only last 100 log lines
  }
}'# | .append logs.spawn

This generator will emit frames with the topic logs.line and automatically maintain only the 100 most recent log entries.

Bi-directional Communication

When duplex is enabled, you can send data into the generator’s input pipeline via <topic>.send frames:

# Create a websocket connection
r#'{
  run: {|| websocat wss://echo.websocket.org | lines },
  duplex: true
}'# | .append echo.spawn

# Send input to the websocket: note the "\n", wss://echo.websocket.org won't
# reply until it sees a complete line
"hello\n" | .append echo.send

When running this generator:

• Lines received from the websocket server are emitted as <topic>.recv frames
• Content from <topic>.send frames is sent to the websocket server

Error Handling

If a generator encounters an error during spawning a <topic>.parse.error frame is emitted with:

• source_id: ID of the failed spawn attempt
• reason: Error message describing what went wrong

The generator does not start and no stop frame is produced.

When a running generator finishes or fails, it automatically restarts after a 1-second delay.

Stopping Generators

To stop a running generator, append a frame with the topic <topic>.terminate. The generator will stop and emit a <topic>.stopped frame with meta.reason set to terminate.

Appending a new <topic>.spawn frame while a generator of the same topic is running reloads it with the new script. If the reload fails to parse, you’ll see a <topic>.parse.error frame, and the previous generator continues running.