RunReveal Hands-On: Building a Pipeline Drop Rule Through the API
What I Was Trying to Do
Every SIEM bill I've ever looked at has the same problem: a big chunk of what you pay to store is noise nobody will ever query.
RunReveal's answer is the pipeline.
Events are matched, transformed, enriched and, if you want, thrown away before they ever reach storage. One of the part in this series covered this from the console side. For the last part of the series I wanted to do it properly, through the API, and prove it works with real events instead of trusting a green tick in the UI.
The plan was small on purpose:
Build a pipeline with one drop rule: throw away any event whose marker field equals article-drop-me.
Route only webhook traffic into that pipeline, so nothing else in the workspace is touched.
Send two events through a test webhook: one marked article-drop-me, one marked article-keep-me.
Query the logs table. If only the keep-me event is there, the drop rule works.
That isn't what happened. I'm writing it up anyway, because what I hit on the way is more useful than a clean result would have been.
There are four API behaviours here I couldn't find in the docs, and I'd want to know about every one of them before automating anything against this API.
How Routing Actually Works
The docs describe this in prose, but the clearest picture comes from reading a working pipeline back out of the API. The model is:
Source -> Topic (precondition + rank) -> Pipeline (ordered steps) -> destination step -> ClickHouseTopics are checked top to bottom by rank. The first topic whose precondition matches wins.
Each topic points at one pipeline. Anything no topic catches goes to the default pipeline.
A pipeline is an ordered list of steps: transform, enrich, filter, detect, drop, destination.
The destination step is what actually writes the event to storage. A pipeline without one stores nothing.
Our workspace has one catch-all topic (precondition type all) wired to a pipeline that's been running since the workspace was set up. Its steps, as the API returns them:
transform precondition: all settings: {"transform_id": "*"}
enrich precondition: all settings: {"enrichment_id": "*"}
filter precondition: normalized.src.asNumber not_equal … settings: {"filter_id": "*"}
detect precondition: normalized.src.asNumber is_empty
destination precondition: all settings: {"destination_id": "default-clickhouse"}Keep that transform step in mind. My test pipeline didn't have one, and it's one of the two things I still suspect.
Building the Steps: Two Broken Endpoints, One That Works
The API offers three ways to put steps into a pipeline. I tried all three, with payloads copied from the working pipeline above so the shape couldn't be the problem.
POST /pipelines/add-step returned HTTP 500 "unknown error". I tried settings as a JSON object and as base64. Same result both times, again on 13 October.
POST /pipelines/execute-step returned 502 Bad Gateway, even against the working pipeline. That's a gateway error, not a validation error.
POST /pipelines/save-steps worked, first time. It replaces the whole step list in one call.
This is the save-steps call that built my test pipeline:
POST /pipelines/save-steps
{
"id": "<pipeline id>",
"steps": [
{"type": "drop", "rank": 1, "settings": {},
"precondition": [{"field": "marker", "value": "article-drop-me", "type": "equal"}]},
{"type": "destination", "rank": 2,
"settings": {"destination_id": "default-clickhouse"},
"precondition": [{"field": "", "value": "", "type": "all"}]}
]
}Gotcha 1: If you're scripting pipelines, skip add-step and go straight to save-steps. Read the current steps, change the list, and write the whole list back. That's also how the console behaves, as far as I can tell.
Wiring It Up Without Touching Production Traffic
The pipeline does nothing until a topic points at it. I created a topic called webhook-drop-test with a precondition of sourceType equal webhook, then attached the pipeline and moved the topic above the catch-all:
POST /topics/set-pipeline {"id": "<topic id>", "pipeline_id": "<pipeline id>"}
POST /topics/move {"id": "<topic id>", "rank": 1}Both calls returned success. The important check came straight afterwards: our Google Workspace events kept arriving at their normal rate through the catch-all. So the topic precondition did exactly what it should and only caught webhook traffic. The only webhook source in the workspace is an old test source from an internal tool, idle since early July, so nothing real was at risk.
Then the two test events, sent to that source's webhook URL with its bearer token:
{"marker": "article-drop-me", "run": "20261014T191844Z", "message": "pipeline drop test"} -> HTTP 200
{"marker": "article-keep-me", "run": "20261014T191844Z", "message": "pipeline drop test"} -> HTTP 200The Result: Nothing Arrived
I searched the logs table for the run ID, across every source type, for several minutes. Neither event was there. Not the dropped one, which was expected, and not the kept one either, which wasn't.
SELECT receivedAt, sourceType, sourceID, rawLog
FROM logs
WHERE sourceType != 'runreveal-audit'
AND rawLog LIKE '%pipeline drop test%'
-> 0 rowsA small trap while I was doing this:
my first search matched rows, and they were all my own searches. Every query you run is logged to the runreveal-audit source, with the full SQL text in it. Searching rawLog for a string will find your own query unless you exclude that source.
My first theory was the missing transform step. The working pipeline starts with one, mine didn't, and a pipeline that skips transform might be handing the destination an event it can't write.
To test that properly I needed a control: send the same kind of event down the normal path, with my topic out of the way, and confirm it lands. Getting that control set up turned up two more gotchas.
Gotcha 2: topics/move Renumbers, It Does Not Reorder
To get my topic out of the way I moved it to rank 2, the catch-all's slot, expecting the two to swap places. This is what came back:
before: 1 webhook-drop-test 2 test-topic (catch-all)
move webhook-drop-test -> rank 2
after: 2 webhook-drop-test 3 test-topic (catch-all)The catch-all got pushed down to make room. My topic was still checked first, so my control event went through the test pipeline anyway and proved nothing. Rank is a sort value, not a slot. If you want a topic below another one, move it to a rank higher than that topic's current rank, and always read the list back afterwards.
Gotcha 3: topics/update Is a Full Replace
Plan B was to disable my topic instead of moving it. I sent only the field I wanted to change:
PUT /topics/update {"id": "<topic id>", "enabled": false}
-> {"id": "…", "name": "", "enabled": false,
"precondition": [{"field": "", "value": "", "type": ""}], "rank": 2}It disabled the topic, and it also wiped the name and the precondition, because every field I didn't send was overwritten with an empty value. On a disabled test topic that doesn't matter. On a live topic it could be serious: I don't know how the router treats a precondition with an empty type, and the worst case is that it matches everything.
Always send the full object to topics/update, never just the field you're changing.
Gotcha 4, and the reason I have not finished this test: With my topic disabled, the control event went through the normal catch-all path, the same one our Google Workspace data uses every minute. The webhook returned HTTP 200. The event never appeared. So the problem isn't my pipeline, or not only my pipeline. This webhook source is accepting events and storing none of them, and nothing in the API response tells you. The last event it ever stored is from 5 July.
A 200 from a webhook only means the request was received. It doesn't mean the event was stored. If you're running anything that pushes into a RunReveal webhook, it's worth a scheduled query that alerts when a source goes quiet, because the webhook itself will never tell you.
What I Can and Cannot Say Yet
Confirmed, with the calls above:
save-steps builds pipelines reliably. add-step (500) and execute-step (502) are still broken.
A topic with a sourceType precondition isolates traffic cleanly. Other sources were unaffected.
topics/move renumbers rather than reorders, and topics/update overwrites fields you don't send.
This webhook source returns 200 and stores nothing, even on the default path.
Not yet proven:
Whether the drop rule itself works. Nothing reached the pipeline in a way I could measure.
Whether a pipeline without a transform step loses events. That's still my best guess, but the broken source means I can't test it with this setup.
Next is a fresh webhook source built only for this test, a control event to confirm it stores, and then the same two-event run again, with and without a transform step. I'll add the result here when I have it, whichever way it goes.
Closing the Series
That's eleven parts: what RunReveal is, how it's built, searching it, processing logs in flight, getting data in, detections, investigations, scheduled agents, notifications and the MCP server I built, and how it compares with Splunk and Sentinel. If there's one thing that runs through all of it, it's this.
The core idea is good:
your logs in ClickHouse, queried with plain SQL, with detections that are just scheduled queries. The parts around the edges are younger than the core. Most of what I hit in this series, from deletes that don't delete to webhooks that return 200 and store nothing, came from actually using the API rather than reading about it. Test it the same way before you trust it with anything important.
---------------------------------------------------Dean-------------------------------------------------
Comments