# Two Kusto performance cliffs explained Companion deep-dive to [`MPP_vs_KQL.md`](MPP_vs_KQL.md) §6. Two phrases in the annotated KQL block point at real, well-known performance cliffs that deserve their own explanation rather than a footnote: 1. `has_any on ProcessCommandLine bypasses the term index` 2. `hint.shufflekey is required to avoid OOM on the cross-table join` --- ## 1. `has_any` on `ProcessCommandLine` bypasses the term index ### What the term index actually does Kusto builds a **per-shard inverted term index** on string columns. At ingest time each string value is tokenized into "terms" using a fixed tokenizer that splits on non-alphanumeric ASCII (whitespace, punctuation, `\`, `/`, `.`, `-`, etc.) and lowercases. The resulting tokens are written to the shard's inverted index alongside the columnar data. When you write `where Col has "x"`, Kusto: 1. Tokenizes `"x"` the **same way** the indexer did at ingest. 2. Looks up the resulting term in the shard's inverted index. 3. Reads **only the rows in shards whose index says "this term might be present here"** — entire shards get skipped. This is the difference between a 50 ms hunt and a 5-minute one. ### Why `has_any` on `ProcessCommandLine` falls out of that fast path Three independent reasons compound: **a) The needle contains characters that the tokenizer treats as separators.** `ProcessCommandLine` values look like: ``` "C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe" -nop -w hidden -enc JABzAD0A... ``` If you write `has_any ("powershell.exe", "rundll32.exe")` you're not searching for one token — `powershell.exe` is the **two tokens** `powershell` and `exe` joined by a `.` (a separator). The index never stored `powershell.exe` as a single term, so the lookup misses and Kusto falls back to a row scan. Quick fix: search for the bare token (`has_any ("powershell", "rundll32")`) — Kusto's planner will index-prune on each individual token. But analysts almost never write it that way because they're thinking "the binary is `powershell.exe`." **b) `has_any` blows past the term-index cardinality threshold.** For each candidate term in the `has_any` list, Kusto has to consult the inverted index, accumulate row-id sets, then union them. The query optimizer has an internal threshold: above some number of needles (or above some estimated selectivity), it gives up on index lookups and just scans, because the OR-merge of many indexed lookups costs more than the scan would. The exact threshold is undocumented and changes between versions; empirically it kicks in fast on `ProcessCommandLine` because that column has the highest term cardinality in the schema — most of those terms are unique GUIDs, paths, base64 blobs, hashes, etc. — so the inverted index is huge and per-term lookup is expensive. **c) `ProcessCommandLine` itself blows up the indexer's effectiveness.** Even when you do hit the index, the **selectivity** is terrible. A term like `powershell` matches a large fraction of all process-creation rows on a typical workstation fleet. The index tells Kusto "this shard might contain it" — but every shard *does* contain it, so no shards get pruned. You still scan everything. This is the deepest reason of the three: even a perfectly written, single-token, indexed `has` query on `ProcessCommandLine` gives you the *index path's CPU cost* on top of *the scan you were going to do anyway*. ### The escape hatch most people don't know about If you must do this in KQL, push the substring match into a `where` clause that the planner can convert into a true scan-with-early-exit, and pre-narrow with something that *is* selective: ```kusto SecurityEvent | where TimeGenerated > ago(1h) // narrow time first — selective | where EventID == 4688 // narrow event-id — selective | where ProcessCommandLine matches regex @"(?i)powershell|rundll32|mshta" ``` Two hours of data and an `EventID` filter is usually enough that the scan-after-prune is cheap. **At 90 days with no narrowing predicate, you've lost.** That's the cliff the doc refers to. ### What SDL does instead No index, so no "did I hit the index?" cliff. Every query is a columnar scan over the epochs that overlap the time window, with column-level prefix pruning and run-length compression doing the heavy lifting. `matches "(powershell|rundll32|mshta)"` on `src.process.cmdline` at 90 days is the same code path as at 1 hour — just more epochs in parallel. --- ## 2. `hint.shufflekey` is required to avoid OOM on the cross-table join ### How Kusto's distributed join works by default Kusto is distributed. Tables are split into extents (shards), and extents live on different data nodes. When you write: ```kusto A | join kind=inner B on Key ``` Kusto picks one of two physical strategies: - **Broadcast** (the default for "small × large"): take the smaller side, replicate it to every node holding the larger side, then do local hash joins. Fast when small really is small. - **Shuffle**: hash both sides on `Key`, send all rows with hash bucket *i* to node *i*, then do local hash joins. Needed when both sides are big. The planner chooses based on a **statistics estimate** of how big each side is *after* the upstream `where` filters apply. ### Where the OOM comes from For a 90-day cross-source hunt the planner's estimate is almost always wrong: 1. `bad_dns` after the `has_any (suspect_domains)` filter is **probably** small — but if the IOC list has 200 entries or a wildcard sneaks in, it can be millions of rows. 2. The planner picks **broadcast** because it estimates `bad_dns` is small. 3. At runtime, `bad_dns` turns out to be huge. 4. Kusto tries to ship the entire `bad_dns` payload to every node holding `_Im_ProcessEvent` extents (which at 90 d is **every node**). 5. Each node tries to hold the broadcasted copy in memory while streaming `_Im_ProcessEvent` past it. 6. `MaxMemoryPerQueryPerNode` (a tenant-level resource governor knob; on Sentinel it's a shared, opaque value) gets hit. 7. You get `Request was aborted due to exceeding query memory limits` — or worse, partial results with no warning. The same shape, with a left side just under the broadcast limit, OOMs intermittently as data volume grows day to day. That's the "silent regression" that makes 90-day Sentinel hunts unreliable in production. ### What `hint.shufflekey` does ```kusto A | join kind=inner hint.strategy=shuffle hint.shufflekey=Key (B) on Key ``` You override the planner's estimate and force the shuffle strategy on `Key`. Both sides get re-hashed by `Key`, sent across the network into hash buckets, joined locally. No node has to hold all of either side — each node holds only **its bucket's slice**, so memory grows linearly with cluster size instead of quadratically with data size. ### Why this is a footgun 1. **You have to know to use it.** The default broadcast looks fine on a 1-day window and silently breaks at 90 days. 2. **You have to pick the right key.** If `hint.shufflekey` is something with high skew (e.g. one user has 95% of the events), one node still OOMs while the others sit idle. You'd then add `hint.num_partitions=N` and tune it. Production hunts often have 3+ hints stacked just to keep them stable. 3. **You can't compose well.** Two joins in the same query each need their own carefully chosen shuffle key. Get one wrong and the second join breaks. 4. **The hints are advisory, not contractual.** A future Kusto version may ignore your hint if its own cost model thinks broadcast is better. Sentinel's update cadence means a query stable today can regress on a Tuesday with no warning. ### What SDL does instead The reduction is distributed **by construction** — there is no broadcast vs shuffle planner because the engine never moves un-reduced rows across the network. Each worker filters → projects → partial-aggregates → partial-joins on its local epochs and sends only the reduced state up to the coordinator. The 90-day join in the SDL example in [`MPP_vs_KQL.md`](MPP_vs_KQL.md) §6 needs **zero hints**: the joiner doesn't need to estimate sizes because it never decides to broadcast. --- ## TL;DR sentences (drop-in for sidebars) > **`has_any` cliff.** Kusto's term index tokenizes string columns at > ingest. `has_any` on `ProcessCommandLine` defeats it three ways at > once: the needles often contain separator characters (so the indexed > terms don't match), the OR-merge of many needles exceeds the > planner's index-vs-scan threshold, and `ProcessCommandLine` has such > a long-tail term distribution that index lookups rarely prune shards > anyway. At 90 d the scan that results is the largest single column > scan in the workspace. > **`hint.shufflekey` cliff.** Kusto's join planner picks broadcast vs > shuffle from an estimated cardinality. On a 90-day cross-source hunt > the estimate is almost always wrong, the planner picks broadcast, and > the smaller side turns out to be tens of millions of rows. Without > `hint.strategy=shuffle hint.shufflekey=...` the query OOMs against > `MaxMemoryPerQueryPerNode`. The hint is required for stability and > has to be re-tuned per query and per data-volume change — a > maintenance tax SDL's distributed-reduction engine doesn't impose.