RunReveal Data Ingestion: Sources & Connectors
Three Ways Data Gets In
Before any of the architecture matters, data has to actually arrive. RunReveal supports three ingestion methods, and which one a given source uses isn't a style choice — it changes what happens when something goes wrong.
Webhook — push-based. The source sends events to a URL RunReveal gives you. Fast and simple, but if the network hiccups mid-send, that event is gone. No retry, no replay.
Polling — RunReveal reaches out to the source on an interval, roughly every 60 seconds, using encrypted stored credentials. It checkpoints where it left off, so a restart or a blip doesn't lose data — it just resumes.
Object Storage — S3, Azure Blob, GCS, R2, MinIO. A bucket gets a notification when a new object lands, and RunReveal pulls it from there. Good fit for anything that already dumps logs to a bucket as its native export format.
If I were setting up a new source and had a choice between webhook and polling for the same data,
I'd take polling every time — the checkpoint-and-resume behavior alone is worth the extra ~60 seconds of latency for anything that isn't genuinely time-critical.
Forwarders, for Everything Else
For file-tail scenarios, Windows Event Log, and syslog, RunReveal ships Fluent Bit support and its own lightweight forwarder called reveald.
What I didn't expect:
reveald has its own concept of destinations, separate from the platform-level Destinations — it can forward to MQTT, to a literal network printer, to S3 directly, or to RunReveal itself.
The printer option is a strange one to see in 2026, but it tells you reveald was built as a genuinely generic forwarder, not something narrowly scoped to just talk to RunReveal's own backend.
The Connector Catalog
RunReveal advertises around 120 source connectors.
I crawled the documentation for all of them rather than take that number on faith, and it holds up — roughly 114 individual connector doc pages exist, spread across cloud providers (AWS, Azure, GCP), identity platforms, EDR and security tools, SaaS applications, and networking sources, plus generic custom/webhook options for anything without official support.
A caveat worth being upfront about, with real numbers instead of a vague gesture at them: 67 of those 114 connectors — 59% — came back with auth method genuinely unclear from the page text once I structured the crawl into a catalog. Broken down by ingestion type, it's 43 object storage, 34 polling, 18 webhook (push), 4 queue/pubsub, 1 forwarder/syslog, and 14 where even the ingestion method itself wasn't stated plainly.
By category: 50 fall into a catch-all "Other," 17 SaaS, 12 EDR/Security, 10 each for AWS and Networking, 6 Identity, and single digits for Azure, GCP, and generic Custom sources.
Treat any public connector catalog, including this one, as a starting point for discovery — always check the specific connector's live doc page before wiring it up for real.
What a Real Source Looks Like
The workspace I tested against runs Google Workspace ingestion — both the audit log stream and the separate security alerts stream — using polling. Watching real data actually accumulate rather than just reading about the schema was where the earlier ClickHouse architecture stopped being abstract: querying tens of thousands of real events back with sub-second response times is a genuinely different experience than tailing a Splunk search that's still "scanning buckets" thirty seconds in.
Source Lifecycle
Disabling a source stops new ingestion immediately but keeps its configuration and all previously ingested data intact — you're not starting from zero if you re-enable it later. Actually deleting a source's historical data is a support-ticket process, not a self-service button.
Hands-On: Measuring the Polling Lag Instead of Quoting It
"Roughly every 60 seconds" is the documentation's own framing for polling latency, and it's close enough for planning purposes, but I wanted an actual measurement rather than a restated number.
Every Google Workspace audit event carries its own original timestamp inside the raw payload — when Google logged it — separately from receivedAt, RunReveal's own record of when it actually landed in ClickHouse. That gap is the real, end-to-end polling lag, and it's queryable directly.
Eight consecutive real events, timestamp of the underlying Google event versus RunReveal's receivedAt:
event 13:31:53.8 -> received 13:33:09 lag 75.2s
event 13:31:43.0 -> received 13:33:06 lag 83.0s
event 13:32:48.6 -> received 13:33:06 lag 17.4s
event 13:30:36.2 -> received 13:31:56 lag 79.8s
event 13:30:19.3 -> received 13:31:54 lag 94.7s
event 13:30:31.1 -> received 13:31:54 lag 82.9s
event 13:27:45.7 -> received 13:30:21 lag 155.3s
event 13:29:14.9 -> received 13:30:18 lag 63.0s
Min 17 seconds, max 155 seconds, median 83 seconds, average 81 seconds — real variance, not a flat 60-second heartbeat. "Roughly every 60 seconds" is the right mental model for what triggers a poll cycle, but the number that actually matters for anyone setting alert-latency expectations is closer to "budget for up to two and a half minutes, typically around 80 seconds" than "60 seconds."
Worth knowing before you write a detection's schedule assuming data is never more than a minute stale.
Next Part picks up right where the data lands — the detection engine that actually reads all of this on a schedule.
--------------------------------------------------------Dean-----------------------------------
Comments