Mirrors/marcredhat-siem-toolkit-patched
1
0
Fork 0
mirror of https://github.com/marcredhat/SIEM-toolkit-patched synced 2026-06-08 20:37:12 +00:00
Code Issues Packages Projects Releases Wiki Activity

Ingest Dashboard: optional background cache pre-warmer

Browse Source 
Adds an asyncio background task that re-runs the heavy Ingest Dashboard
queries every ~4 min (just under the 5 min TTL) so the in-process cache
is always populated. First user hit on any dashboard widget then returns
from cache (single-digit ms) instead of waiting 30-60s for SDL.

Components:
  - backend/services/prewarmer.py: standalone module, opt-in via
    INGEST_PREWARM=1; configurable windows via INGEST_PREWARM_HOURS /
    INGEST_PREWARM_DAYS / INGEST_PREWARM_DAILY_VOLUME_DAYS and interval
    via INGEST_PREWARM_INTERVAL_SECONDS. Logs through the uvicorn logger
    so cycles are visible in 'docker logs'.
  - backend/main.py: spawn the task on FastAPI startup.
  - docker-compose.yml: forward INGEST_PREWARM* env vars to the
    backend service (default off).

Observed on a busy tenant with INGEST_PREWARM=1, default windows:
  top-sources?days=7 first hit after restart: ~39s -> ~8ms (cache warm).

Defaults to disabled (INGEST_PREWARM=0) so existing users see no
behaviour change.
This commit is contained in:
marc
2026-05-22 20:41:36 +02:00
parent bfff0eeec0
commit 7d19c57a5d
3 changed files with 110 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docker-compose.yml
+5
View File
@@ -21,6 +21,11 @@ services:
- ANTHROPIC_API_KEY=${ANTHROPIC_API_KEY}
- DATABASE_URL=postgresql://siem:siem@db:5432/siem
- DETECTIONS_FILE=/app/data/detections.json
- INGEST_PREWARM=${INGEST_PREWARM:-0}
- INGEST_PREWARM_HOURS=${INGEST_PREWARM_HOURS:-1,24,168}
- INGEST_PREWARM_DAYS=${INGEST_PREWARM_DAYS:-7}
- INGEST_PREWARM_DAILY_VOLUME_DAYS=${INGEST_PREWARM_DAILY_VOLUME_DAYS:-5}
- INGEST_PREWARM_INTERVAL_SECONDS=${INGEST_PREWARM_INTERVAL_SECONDS:-240}
depends_on:
db:
condition: service_healthy