RunReveal Log Processing: Transform, Filter, Drop, Mask, Sample

1 day ago

7 min read

Seven Tools for Shaping Data In Flight

Earlier introduced pipelines as an ordered chain of steps, using one real example that happened to include transform, enrich, filter, detect, and destination steps. What I didn't appreciate until I read through RunReveal's log-processing documentation properly is how much depth sits behind each of those step types individually.

This article covers the ones worth understanding on their own: Transforms, Filtering, Dropping, Masking, Sampling, Enrichments, and Set TTL.

Enrichments — Tagging Events Without Touching Storage Rules

An enrichment matches incoming events against a pattern and attaches supplemental key-value data to them. Rules are comma-delimited rows following match_type, pattern,data, matched against sources you select. The real value isn't the tagging itself — it's what you do with the tag afterward.

The documented patterns for this are genuinely practical, not abstract examples:

Exclude known-good activity from detections — tag events from service accounts or automation systems, then have your detection query check for that tag and skip it, rather than hand-listing every service account inside every detection's WHERE clause.
Tag trusted IP ranges — flag corporate VPN or known cloud infrastructure ranges via CIDR match, so network-based detections can exclude them centrally instead of repeating the same exclusion list everywhere.
Tag events for dropping — rather than hard-coding drop patterns into a Drop step directly, tag candidates with an enrichment first. That keeps drop rules centralized, auditable, and visible before anything actually disappears.
Classify by risk level — tag actors or resources by sensitivity (a production database ARN, an admin IAM role pattern) using regex rules, feeding tiered alerting downstream.

Something worth remembering about how rules get edited: Clicking a rule in the rules table does not open an inline editor. To change one, you delete the existing rule and add a new one with the updated values. If you add a rule with the same pattern as an existing one instead of deleting first, the UI adds a duplicate entry rather than replacing it.

Enrichments are also fully API-manageable — create, update, append rules, get, list, and delete all have real endpoints, which means this is scriptable rather than console-only if you're managing rule sets at any real scale.

Filtering vs. Dropping vs. Sampling — Three Ways to Reduce Volume, and When Each Applies

These three get confused with each other constantly, and RunReveal's documentation actually spells out the distinction cleanly enough that

I'll just relay it directly, because it's the most useful framing I've seen for this problem on any SIEM.

Filtering — reusable, regex on the raw log

A Filter matches a regex pattern against the entire raw log JSON string, not a specific field — so it can catch content anywhere in the event. Filters are managed centrally and can apply globally across all sources or be scoped to specific ones. Dropped events never reach ClickHouse and never show up in Explore.

The tradeoff: because it's a shared, reusable rule set, it's the right tool when you want the same drop logic — health-check traffic, known bot user agents — applied consistently across multiple pipelines.

Dropping — pipeline-specific, precondition-based

A Drop step lives inside one specific pipeline and uses preconditions against normalized fields (eventName regex, an IP CIDR match, a missing-field check) rather than a raw-log regex. Multiple preconditions on the same step AND together.

This is the right tool for a one-off rule scoped to a single pipeline, versus something you want shared everywhere.

Sampling — statistical, not absolute

A Sample step keeps a configured percentage of matching events (1-99%) and discards the rest, while non-matching events pass through untouched. The real-world examples given are telling: sample S3 data events at 10%, but keep VPC Flow Logs' rejected connections and CloudFlare's error responses at full fidelity even while sampling the successful/accepted traffic.

The decision in one line: Filter for shared rules across pipelines. Drop for a one-off, pipeline-specific rule. Sample when you want to keep statistical visibility into high-volume noise rather than delete it outright.

Masking — Protecting Fields Without Losing the Event

Masking mutates a field's value in place — every step and destination after it in the pipeline sees the masked value, not the original. Three modes:

Redact — replace the entire value with a static token.
Hash — replace with a deterministic, non-reversible pseudonym. Equal input values produce equal hashes within a workspace, so a masked field stays joinable and countable in detections even though the real value is gone.
Partial — replace only the regex-matched portion, leaving the rest visible. The documented example: strip the local part of an email address but keep the domain, so you can still group and filter by domain without storing who exactly it was.

One detail worth knowing:

a Mask rule targeting a field that doesn't exist on a given event is silently skipped rather than erroring, which means it's safe to write one mask config that references fields only present on some of your sources.

Mask vs. Drop, in one sentence: mask when you still need the event for analysis but must protect or pseudonymize a specific field; drop when the entire event has zero value and should never be stored at all.

Transforms — Reshaping a Value Before It Lands

A transform rule is a chain of processors run in order against the raw log, always ending in an Output to Field step that writes the final result to a normalized field. Available processors include GJSON-style path extraction, regex find/replace, string prepend/append, trim, substring by index (negative indices allowed, counting from the end), split-by-delimiter, and date format conversion.

Rules are independent of each other, and a processor's output becomes the next processor's input — so "Extract JSON Field" partway through a chain will fail if the prior processor handed it a plain string instead of JSON. That's listed as one of the documented common mistakes, alongside forgetting the mandatory Output to Field step, which just means the rule silently doesn't reach a normalized field at all rather than erroring loudly.

Set TTL — Retention, Per Event

A Set TTL step lets you override retention on a per-event basis. If you never touch it, ingestion applies the platform default of 550 days automatically — this is the same number referenced back in Part 2's retention discussion, now with the mechanism that actually controls it at the event level rather than just the workspace default.

Hands-On: What the API Doesn't Document

Everything above is real, but it's still secondhand — pulled from RunReveal's documentation rather than from actually operating the platform. Since I have live API access to a workspace, it seemed worth actually exercising two of these tools directly rather than just relaying what the docs say about them, and writing down what happened, including the parts that didn't work the way the docs implied.

A transform that's already running, verified against real ingested data

Rather than build a synthetic example, I went looking for a transform rule already active in the workspace and checked its actual output against actual ingested events.

One was already running on a lab source: a processor chain that extracts an "environment" field, uppercases it, and writes the result to a normalized tag field.

A real raw event ingested through that source carries this in its JSON payload:

{"environment":"dev","application":"<internal-app-name>", ...}

And the same event's tags, queried directly from ClickHouse:

tags = {"RR_Pipeline":"Default Pipeline","RR_Topic":"Default Topic", "environment_CAPITAL":"DEV"}

That's the whole point of a transform in one real pair of before/after values: "dev" goes in, "DEV" comes out, on an event nobody hand-crafted for this article.

Building an enrichment rule from scratch — and hitting five undocumented behaviors

The docs describe enrichment rules conceptually ("match_type,pattern,data" rows) but the actual API schema for creating one isn't published anywhere I could find. So I built one by trial and error against the live create endpoint, reading the validation errors back from the server to figure out the real field names.

Five things came out of that exercise that aren't in the documentation:

The source list field is called sources, and it takes source names, not source IDs — passing a valid source ID here fails with "invalid source name," even though every other reference to sources in this API is ID-based.
The match-type field is called type, not matchType or match_type (both of those are silently accepted and silently ignored — you get back the actual error "invalid match type ''" with no hint that the key name itself was wrong). Valid values are exact, regex, or cidr — the docs' own contains-style examples describe the behavior, not the literal parameter name.
A newly created rule comes back with isActive: false by default. It exists, it's saved, and it does nothing until you explicitly flip it on — there's no warning in the create response that the rule is inert.
Updating a rule doesn't accept a partial patch. Sending just {id, isActive: true} returns a bare "unknown error" with no explanation; the update endpoint wants the full object — name, sources, and rules — resent every time, or it fails silently on the missing fields.
The update also regenerates the rule's internal ID rather than editing it in place. That's a nice confirmation, at the API level, of the console-side detail mentioned earlier in this article — there really is no in-place edit happening anywhere in this system, UI or API; every "edit" is a delete-and-recreate under the hood.

The one that actually matters operationally: Activating an enrichment rule does not retroactively tag events that were already ingested before activation, even when their content matches the rule's pattern. I confirmed this directly: queried a set of already-ingested events matching the rule's regex, activated the rule, then re-ran the identical query — the tag was absent both times. Enrichment only applies going forward, at ingest time. If you're rolling out a new tagging rule, don't expect it to backfill your existing data; budget for a re-ingestion or accept the gap.

One more thing worth knowing if you're scripting any of this:

the delete endpoint for enrichments returns a 405 on a plain POST and needs the literal DELETE HTTP method to be accepted at all — and even then, it only clears the rule's contents rather than removing the enrichment record itself, which then lingers, empty and inert, in the list output. That matches a pattern already seen elsewhere in this API: detections and pipelines have the same non-functional delete behavior. Plan for disable-in-place as the real cleanup mechanism here, not delete.

------------------------------------------------------------Dean--------------------------------------