RunReveal Agents: What I Found Without Pressing Run

1 day ago

8 min read

What Agents Are Supposed to Do

Most of a SIEM is reactive. A detection fires, someone gets paged, someone goes and looks. RunReveal's Agents feature tries to move part of that work onto a schedule. You write a prompt, pick a model, set a cron expression, and an LLM goes off, queries your logs with the same tools you'd use, and sends you what it found.

There are six starting templates in the console:

Weekly Threat Hunt: a recurring hunt that ends in a written report. It doesn't create detections.
Detection Coverage Gap Analysis: compares your sources against your detections and suggests what's missing.
Detection Tuning: looks for noisy detections and proposes changes.
Data Volume Anomaly Detection: watches source volumes for spikes and drops. A source going quiet is often the first sign something broke, or that someone turned logging off.
Data Saving Analysis: points at high-volume, low-value logs you could filter or drop.
Open Investigation Analysis: summarises open cases and flags the ones nobody has touched.

Why I Didn't Run One

Every other part of this series is built on things I actually ran. This one has a gap in the middle, and I'd rather say so up front than dress it up.

An agent needs an AI provider connected to the workspace:

Anthropic, OpenAI, Google AI, Bedrock or Vertex. RunReveal doesn't ship its own model. You bring your own key, or for Bedrock, an IAM role that RunReveal assumes in your AWS account. The workspace I've been testing in is a shared lab, and connecting my personal API key to it would mean my own billing account paying to process company data. I didn't want that, and setting up a proper Bedrock role is somebody else's call. So no provider is connected, and no agent has ever completed a run here.

That turned out to be less of a dead end than I expected. You can learn a lot about an AI feature from everything around the model: what the API insists on, what the defaults are, what instructions the agent gets before it starts, and what gets written to the audit log. That's what the rest of this article cover

What the Create API Actually Insists On

Back in September I created a test agent through the API to see the real request and response shapes. Two of my first attempts failed, and the errors tell you more than the docs do:

POST /agents/create   (no provider)
-> "AI provider is required. Please select a provider (e.g., anthropic, openai, googleai, bedrock, vertex)"

POST /agents/create   (provider, no model)
-> "AI model is required. Please select a model for the chosen provider"

So provider and model are both mandatory, and the provider list is a fixed set of five. What isn't checked is whether the model exists. I passed a model name I made up, not-a-real-model-xyz, and the agent was created without complaint. You'd only find out at the first scheduled run, which could be days later.

The full set of fields create accepts is wider than the console shows: name, displayName, description, prompt, provider, model, schedule (cron), notificationNames, tools, parameters, riskScore, notes, plus two switches worth knowing about: disableRunRevealSystemPrompt and skipPermissions. I didn't test either. The names alone tell you there's a RunReveal system prompt sitting in front of yours, and a permission check that can be turned off.

The Defaults Point at Email, Twice

This is the part I'd want anyone to read before building an agent through the API.

What the API did: I sent "enabled": false and no notification channel. The agent came back with enabled: true, notificationNames: ["email"], and a next run scheduled about two days out. The false I sent was silently ignored, and the email channel was filled in for me.

I caught it because I read the object that came back instead of assuming the request did what I asked. Fixing it took a second call, and that call is a full-replace PUT: my first attempt sent only the fields I wanted to change, and failed:

PUT /agents/update   (just enabled + notificationNames)
-> Key: 'UpdateAgentRequest.DisplayName' Error:Field validation for 'DisplayName' failed on the 'required' tag
   Key: 'UpdateAgentRequest.Prompt' Error:Field validation for 'Prompt' failed on the 'required' tag

You have to fetch the agent, change the two fields, and send the whole thing back.

One more inconsistency while you're there: update wants enabled as a real JSON boolean, while the detections enable endpoint wants it as a string. Same word, opposite types, same API.

The second time email shows up is new since I first wrote about this. The workspace now ships built-in agent skills (more on those below), and one of them is called notifications. This is the instruction it gives the agent, word for word:

"If you are working in a background context and cannot confirm, look for the channel named email.
 Most workspaces should have a channel named email, if not just use the oldest channel with the
 type email."

A scheduled agent is a background context by definition. So if you leave the channel empty, the API fills in email, and if the agent decides to send something on its own, its own instructions also send it to email. In most workspaces that channel goes to real people. In ours it goes to a colleague's inbox.

That's the reason I neutralised the test agent within a minute of creating it, and then deleted it.

My rule for anyone trying this:

create a dedicated test channel first (a Slack channel only you are in, or a webhook you control), set it explicitly on the agent, and read the object back after every create and update. Don't rely on enabled: false to keep anything quiet.

One thing that did work properly: DELETE /agents/delete?id=… really deletes the agent. My test agent was gone from /agents/list straight afterwards. That stood out, because the delete endpoints for detections and pipeline steps I'd hit earlier in this series don't actually remove anything.

Built-in Skills: The Instructions Behind the Agent

Skills are markdown playbooks an agent loads when it needs them. When I checked on 28 September, the skills list in this workspace was empty. Checking again today, there are three, and none of them were written by anyone on our side:

GET /agents/skills/list
  data-analysis        sources -> tables -> schema -> query workflow, ClickHouse syntax, dashboards
  detection-authoring  Sigma vs SQL, lookup order, pre-validation, naming, categories, MITRE
  notifications        choosing a channel, Handlebars templates, NotificationSend

These are worth reading even if you never run an agent, because they're RunReveal telling its own AI how to use the platform properly. A lot of it is good advice for humans too. Some of the specifics I hadn't seen written down anywhere else:

Default to Sigma for single-event matching. Use SQL only for aggregation, correlation, time-window counting or joins. Sigma runs in real time per event, SQL runs on a schedule.
Mandatory pre-validation before creating a detection: read the schema, sample 5 rows from the last 7 days, get the event-name distribution, then test the WHERE clause with LIMIT 5 and check it runs in under 1,500 ms.
Performance targets: a detection on a 15-minute schedule should run in under 1,000 ms, an hourly one under 2,000 ms. Use count(), not count(*).
Naming: displayName is "{Service} {Description In Title Case}" ("Okta Push Fatigue"), the slug is lowercase with hyphens, 3 to 128 characters, no underscores.
Categories go in a fixed order: service, then an OCSF category, then signal or non-signal, then optional tags. Example: [okta, identity-access-management, signal, privilege-escalation].
MITRE fields are validated by regex. TA#### for tactics, T#### or T####.### for techniques. t1078, T1136.1 and TA001 are all rejected.
"Zero results != broken detection." That line made me laugh, given that Part 6 covered a detection that returned zero results because its filter could never match. Both things are true; you have to check which one you're looking at.

The data-analysis skill also has a sensible habit I'd copy: when no timeframe is given, default to the last 24 hours but tell the user you assumed it. And it warns the agent to check how big rawLog is before querying it, because full-table scans time out with a 5xx.

Built-in Resources, and What Sigmalite Changes

Alongside skills there are three reference documents agents can load: a detection engineering guide, a list of console URL routes (so the agent can link you to the right page), and a reference for sigmalite, the Sigma dialect RunReveal actually runs. The sigmalite one is the most useful, because it lists where RunReveal differs from upstream Sigma:

Added: a cidr modifier, for matching IP fields against ranges. Upstream Sigma has no native equivalent.
Not supported: gt, gte, lt, lte, utf16le, utf16be. The rule fails to parse if you use them. Numeric comparisons belong in a SQL detection.
Extra fields: sourcenames and sourcetypes to scope a rule, score for signal chaining, mitre, disabled, and notes.

detection:
  local:
    DestinationIp|cidr:
      - "10.0.0.0/8"
      - "172.16.0.0/12"
      - "192.168.0.0/16"
  condition: not local

If you're porting Sigma rules from a public repo, the unsupported modifiers are the thing to grep for first.

What the Audit Log Shows

RunReveal writes its own audit trail into the same logs table as everything else, so I could query my September test back out of it. Every agent API call is there, failures included, with the error message attached:

SELECT receivedAt, eventName, rawLog
FROM logs
WHERE receivedAt >= now() - INTERVAL 60 DAY
  AND sourceType ILIKE '%runreveal%'
  AND eventName IN ('agents_create','agents_update','agents_delete')
ORDER BY receivedAt

A failed create looks like this (workspace and IP details removed):

{
  "event_name": "agents_create",
  "actor_email": "Testing",
  "readonly": false,
  "supplemental_data": {
    "error": "true",
    "error_message": "AI model is required. Please select a model for the chosen provider",
    "user_agent": "curl/8.5.0"
  }
}

Three things in there are worth knowing if you ever investigate agent activity:

actor_email isn't an email. For API calls it's the name of the API token, here "Testing". If your tokens have vague names, your audit trail will too. Name tokens after the person or the automation that owns them.
There's no request body. The log tells you an agent was created and that it succeeded. It doesn't tell you it was created with enabled: true and an email channel when the caller asked for neither. The silent default from the section above is invisible in the audit trail.
Reads are logged too. agents_list, agent_runs_list, agent_tasks_list, chat_usage_summary and workspace_ai_models_info all show up, so you can see who has been looking at the AI features, not just who changed them.

If you want a detection on this, agents_create and agents_update events where readonly is false and the error field is empty is a reasonable start. An agent is code that runs on a schedule with access to your logs and, by default, the ability to write Sigma rules. New ones deserve the same attention as a new scheduled task on a server.

The Trust Boundary

The tool permissions on a new agent are on by default. The list includes reading schemas, query history, sources and detections, and also creating and updating Sigma rules and sending notifications. Put that next to everything above:

it can be created with enabled: true when you asked for false,
it defaults to emailing a real channel,
its own instructions tell it to fall back to email when it can't ask,
it's allowed to write detection rules,
and there's a skipPermissions switch in the API.

None of that is a vulnerability. It's a product built to be useful out of the box, which is what most people want. But it means the safe configuration is something you build on purpose, not

something you get by default

What I would check before turning one on

A dedicated test notification channel exists and is set explicitly on the agent.
Tool permissions are trimmed to read-only for the first few runs. Turn rule creation on only once you trust the output.
The model name is real, so the first run doesn't fail silently days from now.
The object you get back from create and update matches what you sent.
The API token used has a name that identifies its owner, so the audit trail means something.
Someone has read the built-in skills, since they shape what the agent will do.

Where This Leaves Agents

I can't tell you how good a RunReveal agent's threat hunt report is, because I haven't seen one. What I can tell you is that the scaffolding is more thought through than I expected. The skills carry real detection-engineering rules, the sigmalite reference is honest about what doesn't work, and every call lands in an audit log you can query with SQL. The rough edges are in the defaults, and they all lean the same way: towards running, and towards email.

If a proper provider gets connected to this workspace later, I'll run the Weekly Threat Hunt template against it and write up what comes back, good or bad.

----------------------------------------------------Dean------------------------------------------------