marcredhat-kql/docs/kql_cliffs_explained.md

# Two Kusto performance cliffs explained

Companion deep-dive to [`MPP_vs_KQL.md`](MPP_vs_KQL.md) §6. Two phrases in
the annotated KQL block point at real, well-known performance cliffs that
deserve their own explanation rather than a footnote:

1. `has_any on ProcessCommandLine bypasses the term index`
2. `hint.shufflekey is required to avoid OOM on the cross-table join`

---

## 1. `has_any` on `ProcessCommandLine` bypasses the term index

### What the term index actually does

Kusto builds a **per-shard inverted term index** on string columns. At
ingest time each string value is tokenized into "terms" using a fixed
tokenizer that splits on non-alphanumeric ASCII (whitespace,
punctuation, `\`, `/`, `.`, `-`, etc.) and lowercases. The resulting
tokens are written to the shard's inverted index alongside the columnar
data.

When you write `where Col has "x"`, Kusto:

1. Tokenizes `"x"` the **same way** the indexer did at ingest.
2. Looks up the resulting term in the shard's inverted index.
3. Reads **only the rows in shards whose index says "this term might be
   present here"** — entire shards get skipped.

This is the difference between a 50 ms hunt and a 5-minute one.

### Why `has_any` on `ProcessCommandLine` falls out of that fast path

Three independent reasons compound:

**a) The needle contains characters that the tokenizer treats as separators.**

`ProcessCommandLine` values look like:

```
"C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe" -nop -w hidden -enc JABzAD0A...
```

If you write `has_any ("powershell.exe", "rundll32.exe")` you're not
searching for one token — `powershell.exe` is the **two tokens**
`powershell` and `exe` joined by a `.` (a separator). The index never
stored `powershell.exe` as a single term, so the lookup misses and Kusto
falls back to a row scan.

Quick fix: search for the bare token (`has_any ("powershell", "rundll32")`)
— Kusto's planner will index-prune on each individual token. But
analysts almost never write it that way because they're thinking "the
binary is `powershell.exe`."

**b) `has_any` blows past the term-index cardinality threshold.**

For each candidate term in the `has_any` list, Kusto has to consult the
inverted index, accumulate row-id sets, then union them. The query
optimizer has an internal threshold: above some number of needles (or
above some estimated selectivity), it gives up on index lookups and
just scans, because the OR-merge of many indexed lookups costs more
than the scan would.

The exact threshold is undocumented and changes between versions;
empirically it kicks in fast on `ProcessCommandLine` because that column
has the highest term cardinality in the schema — most of those terms
are unique GUIDs, paths, base64 blobs, hashes, etc. — so the inverted
index is huge and per-term lookup is expensive.

**c) `ProcessCommandLine` itself blows up the indexer's effectiveness.**

Even when you do hit the index, the **selectivity** is terrible. A term
like `powershell` matches a large fraction of all process-creation rows
on a typical workstation fleet. The index tells Kusto "this shard might
contain it" — but every shard *does* contain it, so no shards get
pruned. You still scan everything.

This is the deepest reason of the three: even a perfectly written,
single-token, indexed `has` query on `ProcessCommandLine` gives you the
*index path's CPU cost* on top of *the scan you were going to do anyway*.

### The escape hatch most people don't know about

If you must do this in KQL, push the substring match into a `where`
clause that the planner can convert into a true scan-with-early-exit,
and pre-narrow with something that *is* selective:

```kusto
SecurityEvent
| where TimeGenerated > ago(1h)        // narrow time first — selective
| where EventID == 4688                 // narrow event-id — selective
| where ProcessCommandLine matches regex @"(?i)powershell|rundll32|mshta"
```

Two hours of data and an `EventID` filter is usually enough that the
scan-after-prune is cheap. **At 90 days with no narrowing predicate,
you've lost.** That's the cliff the doc refers to.

### What SDL does instead

No index, so no "did I hit the index?" cliff. Every query is a columnar
scan over the epochs that overlap the time window, with column-level
prefix pruning and run-length compression doing the heavy lifting.
`matches "(powershell|rundll32|mshta)"` on `src.process.cmdline` at 90
days is the same code path as at 1 hour — just more epochs in parallel.

---

## 2. `hint.shufflekey` is required to avoid OOM on the cross-table join

### How Kusto's distributed join works by default

Kusto is distributed. Tables are split into extents (shards), and
extents live on different data nodes. When you write:

```kusto
A | join kind=inner B on Key
```

Kusto picks one of two physical strategies:

- **Broadcast** (the default for "small × large"): take the smaller
  side, replicate it to every node holding the larger side, then do
  local hash joins. Fast when small really is small.
- **Shuffle**: hash both sides on `Key`, send all rows with hash bucket
  *i* to node *i*, then do local hash joins. Needed when both sides are
  big.

The planner chooses based on a **statistics estimate** of how big each
side is *after* the upstream `where` filters apply.

### Where the OOM comes from

For a 90-day cross-source hunt the planner's estimate is almost always
wrong:

1. `bad_dns` after the `has_any (suspect_domains)` filter is **probably**
   small — but if the IOC list has 200 entries or a wildcard sneaks in,
   it can be millions of rows.
2. The planner picks **broadcast** because it estimates `bad_dns` is
   small.
3. At runtime, `bad_dns` turns out to be huge.
4. Kusto tries to ship the entire `bad_dns` payload to every node
   holding `_Im_ProcessEvent` extents (which at 90 d is **every node**).
5. Each node tries to hold the broadcasted copy in memory while
   streaming `_Im_ProcessEvent` past it.
6. `MaxMemoryPerQueryPerNode` (a tenant-level resource governor knob;
   on Sentinel it's a shared, opaque value) gets hit.
7. You get `Request was aborted due to exceeding query memory limits`
   — or worse, partial results with no warning.

The same shape, with a left side just under the broadcast limit, OOMs
intermittently as data volume grows day to day. That's the "silent
regression" that makes 90-day Sentinel hunts unreliable in production.

### What `hint.shufflekey` does

```kusto
A | join kind=inner hint.strategy=shuffle hint.shufflekey=Key (B) on Key
```

You override the planner's estimate and force the shuffle strategy on
`Key`. Both sides get re-hashed by `Key`, sent across the network into
hash buckets, joined locally. No node has to hold all of either side —
each node holds only **its bucket's slice**, so memory grows linearly
with cluster size instead of quadratically with data size.

### Why this is a footgun

1. **You have to know to use it.** The default broadcast looks fine on
   a 1-day window and silently breaks at 90 days.
2. **You have to pick the right key.** If `hint.shufflekey` is something
   with high skew (e.g. one user has 95% of the events), one node still
   OOMs while the others sit idle. You'd then add `hint.num_partitions=N`
   and tune it. Production hunts often have 3+ hints stacked just to
   keep them stable.
3. **You can't compose well.** Two joins in the same query each need
   their own carefully chosen shuffle key. Get one wrong and the second
   join breaks.
4. **The hints are advisory, not contractual.** A future Kusto version
   may ignore your hint if its own cost model thinks broadcast is
   better. Sentinel's update cadence means a query stable today can
   regress on a Tuesday with no warning.

### What SDL does instead

The reduction is distributed **by construction** — there is no
broadcast vs shuffle planner because the engine never moves un-reduced
rows across the network. Each worker filters → projects →
partial-aggregates → partial-joins on its local epochs and sends only
the reduced state up to the coordinator. The 90-day join in the SDL
example in [`MPP_vs_KQL.md`](MPP_vs_KQL.md) §6 needs **zero hints**:
the joiner doesn't need to estimate sizes because it never decides to
broadcast.

---

## TL;DR sentences (drop-in for sidebars)

> **`has_any` cliff.** Kusto's term index tokenizes string columns at
> ingest. `has_any` on `ProcessCommandLine` defeats it three ways at
> once: the needles often contain separator characters (so the indexed
> terms don't match), the OR-merge of many needles exceeds the
> planner's index-vs-scan threshold, and `ProcessCommandLine` has such
> a long-tail term distribution that index lookups rarely prune shards
> anyway. At 90 d the scan that results is the largest single column
> scan in the workspace.

> **`hint.shufflekey` cliff.** Kusto's join planner picks broadcast vs
> shuffle from an estimated cardinality. On a 90-day cross-source hunt
> the estimate is almost always wrong, the planner picks broadcast, and
> the smaller side turns out to be tens of millions of rows. Without
> `hint.strategy=shuffle hint.shufflekey=...` the query OOMs against
> `MaxMemoryPerQueryPerNode`. The hint is required for stability and
> has to be re-tuned per query and per data-volume change — a
> maintenance tax SDL's distributed-reduction engine doesn't impose.