Skip to content

Add guide on reassembling multi-segment messages#407

Draft
Zedoraps wants to merge 1 commit into
mainfrom
feat/cisco-ise-ocsf-package
Draft

Add guide on reassembling multi-segment messages#407
Zedoraps wants to merge 1 commit into
mainfrom
feat/cisco-ise-ocsf-package

Conversation

@Zedoraps

@Zedoraps Zedoraps commented Jun 22, 2026

Copy link
Copy Markdown
Contributor

🔍 Problem

  • Some log sources cap transport-message size and split a single logical event into several numbered segments sharing an id.
  • Segments arrive interleaved and ids get reused over time, so the partial messages are useless until reassembled.
  • TQL had no guide documenting how to put such segments back together.

🛠️ Solution

  • New generic guide guides/parsing/reassemble-multi-segment-messages (sidebar: Guides → Parsing).
  • Documents the pattern: frame segments with parse_grok, gather them with an outer window and an inner group keyed on the id, then sort + collect + first to concatenate payloads in order.
  • window outermost (not group) so state is bounded to the open windows rather than one subpipeline per near-unique id; the note explains why and when to invert.
  • Points at work-with-time for supplying an event-time field, and at the cisco::ise::* operators as a real-world implementation.

💬 Review

  • window-outer vs group-outer equivalence verified by running both on interleaved sample data; output is identical.
  • Example pipelines verified to run; markdownlint passes on the changed file.
⚙️ Code PR: tenzir/library#161

@github-actions github-actions Bot added guide How-to guides site Site infrastructure labels Jun 22, 2026
@github-actions

github-actions Bot commented Jun 22, 2026

Copy link
Copy Markdown
Contributor

📦 Preview  ·  View →  ·  🟢 Live

Verified for 4778b83  ·  Auto-updates on push

@Zedoraps Zedoraps marked this pull request as draft June 22, 2026 15:51

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 7ce795c8a1

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread src/content/docs/guides/parsing/reassemble-multi-segment-messages.mdx Outdated
Comment thread src/content/docs/guides/parsing/reassemble-multi-segment-messages.mdx Outdated
Comment thread src/content/docs/guides/parsing/reassemble-multi-segment-messages.mdx Outdated
@Zedoraps Zedoraps force-pushed the feat/cisco-ise-ocsf-package branch 5 times, most recently from 0f1ac73 to 8ae0c10 Compare June 22, 2026 17:08
@github-actions github-actions Bot added the reference Reference documentation label Jun 22, 2026
@Zedoraps Zedoraps force-pushed the feat/cisco-ise-ocsf-package branch 3 times, most recently from be19fae to e7208e4 Compare June 23, 2026 07:37
Document the generic pattern for reassembling a logical message that a
source split across several transport messages: frame each segment, gather
the segments of one message with an outer window and an inner group keyed on
the id, then concatenate the payloads in order. Cast the segment counter
before sorting, and include a delay-based example that shows messages
streaming in over more than the window, plus what happens when one message's
segments outrun it. Add reciprocal See Also links on window and group, the
operators the technique is built on.

Assisted-by: Claude Opus 4.8 (1M context) (Claude Code)
@Zedoraps Zedoraps force-pushed the feat/cisco-ise-ocsf-package branch from e7208e4 to 4778b83 Compare June 23, 2026 07:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

guide How-to guides reference Reference documentation site Site infrastructure

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant