nox-project-nox-framework/CHANGELOG.md

# Changelog

All notable changes to NOX are documented here.

## [1.0.5] — 2026-05-06

### Sources
- **Fixed:** `pypi_user` — endpoint was querying the PyPI package API (`/pypi/{target}/json`) instead of user profiles, returning package metadata for any username that matched a package name and 404 for all others. Source now uses the PyPI XML-RPC `user_packages` method, returning the actual list of packages owned by the target username.

### Engine
- **Fixed:** `Detect.qtype` — dotted-quad strings with out-of-range octets (e.g. `255.255.255.256`) were classified as `phone` instead of `username`. Invalid IPs now fall through to `username`.
- **Fixed:** `Record.dedup_key` — records with no identity fields (email, username, password) from different sources all produced the same SHA-256 hash, causing all but the first to be silently dropped from the DB. Source name is now included in the key when identity fields are absent.
- **Added:** `raw_payload` support in `NoxSourceProvider` — POST sources can now send raw string bodies (e.g. XML-RPC) in addition to JSON and form-encoded payloads.
- **Fixed:** `FileSystemProvider._by_regex` — regex-extracted values that are not email addresses (e.g. package names, usernames) are now correctly placed in the `username` field instead of `email`.

## [1.0.4] — 2026-04-22

### Engine
- **Fixed:** `_build_ssl_context` — custom TLS context had zero CA certificates loaded, causing `SSLCertVerificationError` on all HTTPS connections. CA bundle now loaded via `certifi` with `load_default_certs()` fallback.
- **Fixed:** `_NOISE_RE` in reporting — `ssl.`, `aiohttp.`, `asyncio.` were substring-matched, silently zeroing legitimate emails and domains (e.g. `user@ssl.example.com`). Patterns now anchored to start-of-string or whitespace.
- **Fixed:** `COMBO_RE` in `ScrapeEngine` — multiline greedy match produced credentials with embedded newlines in email/password fields. Pattern now excludes newlines from both capture groups.
- **Fixed:** `_in_flight` dict in `AvalancheScanner` — entries were never removed after processing, causing unbounded memory growth on deep scans. Entry is now popped in the `finally` block after the future resolves.
- **Fixed:** `DorkingEngine.__init__` — `ProxyManager.get_proxies()` was called eagerly on every `Orchestrator` instantiation, triggering a proxy fetch and OPSEC warning even for local-only commands (`--crack`, `--list-sources`, `--analyze`). Proxy fetch is now lazy.
- **Fixed:** `ScrapeEngine._fetch_content` — IntelX paste content fetch used `DB.get_key("intelx")` which does not read `apikeys.json`. Replaced with `Vault.get("INTELX_API_KEY")` for consistent key resolution.
- **Fixed:** `AsyncSource._rec` and `RiskEngine.score` — plugin `confidence` values were stored in `NoxSourceProvider._confidence` but never transferred to `Record.source_confidence`. All 124 plugin records received a flat `0.5` confidence regardless of their declared value. `_rec()` now injects `self._confidence` into the record; `RiskEngine.score()` uses `record.source_confidence` as fallback when the source is not in `_SRC_CONFIDENCE`.
- **Fixed:** `ConfigManager.get_key` — `None` results were cached, preventing env vars set after the first lookup from being detected in the same session. Only positive values are now cached.
- **Fixed:** Recursive Avalanche Engine — identifiers extracted from paste content (`paste["patterns"]`) were not being harvested as pivot seeds. All `scrape_res["pastes"]` pattern matches are now fed into `_extract_ids_from_text`.

### Sources
- **Fixed:** `circl_hashlookup` — endpoint hardcoded to `/lookup/md5/{target}` but `input_type=hash` accepts SHA1/SHA256. SHA1 and SHA256 backup endpoints added; engine now routes each hash type to the correct path.
- **Updated:** `crt_sh` — `reliability_score` lowered from 5 to 3, `is_volatile=true` added to reflect documented intermittent availability.

## [1.0.3] — 2026-04-15

### Engine
- **Fixed:** Recursive Avalanche Engine — identifiers extracted from paste content (`paste["patterns"]`) were not being harvested as pivot seeds. Bare emails and other identifiers found in IntelX paste bodies that lacked a `:password` separator were silently dropped from the pivot queue. All `scrape_res["pastes"]` pattern matches are now fed into `_extract_ids_from_text` and reinjected consistently with `credentials`, `telegram`, and `dork_misconfigs`.

## [1.0.2] — 2026-04-14

### Sources
- **Fixed:** `misp_search` — `MISP_URL` added to `api_key_slots` so the instance base URL is resolved at runtime; `health_check_url` corrected from unresolvable placeholder to `https://misp.local`
- **Fixed:** `threatconnect_search` — removed unresolvable `{TC_SIGNATURE}` HMAC placeholder from the `Authorization` header; `reliability_score` lowered to `2`, `is_volatile` set to `true`
- **Fixed:** `spycloud_breach` — endpoint corrected from `breach/data/emails` to `breach/catalog/emails` (standard breach lookup tier)
- **Fixed:** `duckduckgo_api` — primary instance updated to `search.sapti.me`; 5 backup SearXNG instances added to `backup_endpoints` (now consumed by the engine)
- **Fixed:** `gravatar` — endpoint now MD5-hashes the email before URL substitution via new `query_transform: md5_lower` field; raw email was returning 404 on every query
- **Replaced:** `bgpview_ip` → `ripestat_ip` (RIPE Stat prefix-overview API) — BGPView free API decommissioned January 2025; RIPE Stat is free, keyless, and stable (`reliability_score: 5`)
- **Fixed:** `twitter_v2` — marked `is_volatile=true`, `confidence` lowered to `0.1`; free-tier bearer tokens receive HTTP 403 since February 2024
- **Fixed:** `fofa_info` — `qbase64` parameter now receives `base64(domain="<target>")` via `query_transform: fofa_domain`; raw domain was producing malformed queries
- **Fixed:** `pipl_search` — Pipl shut down public REST API in Q3 2024; `reliability_score` lowered to `2`, `confidence` to `0.3`, `is_volatile=true`
- **Fixed:** `spyonweb` — API confirmed unreachable; `reliability_score` lowered to `1`, `confidence` to `0.1`, `is_volatile=true`
- **Fixed:** `hudsonrock_osint` — `is_volatile=true`; `rate_limit` raised from `5.0` to `30.0` to respect Cavalier API throttling (~10 req/hour free tier)
- **Fixed:** `mailboxlayer`, `numverify`, `ipstack`, `ipinfodb` — endpoints and `health_check_url` migrated from `http://` to `https://`; API keys were being transmitted in cleartext before the server-side redirect
- **Added:** `xposedornot` plugin (free, public breach analytics)
- **Added:** `MISP_URL` to service registry and `apikeys.json` — back-filled automatically on first run after upgrade
- Source count: 123 → 124

### Config
- **Fixed:** Duplicate `xposedornot` entry removed from `SERVICE_REGISTRY` in `config_handler.py`

### Engine
- **Fixed:** `_parse_retry_after` helper added — `int()` on an HTTP-date `Retry-After` header raised `ValueError`, causing the retry loop to abort as a hard failure; all 5 call sites in `_get`, `_post`, `Session.get`, and `Session.post` updated
- **Fixed:** `_random_headers` — `Sec-CH-UA` Client Hints were emitted even when a Firefox UA was passed via the `extra` override; guard now evaluates the final `User-Agent` after overrides are applied
- **Fixed:** `HashEngine._hashmob` — Hashmob API v2 changed request field from `"hash"` to `"hashes"` (array) and response schema from `{found, result}` to `{data: [{plaintext}]}`
- **Fixed:** `DeHashEngine` — both `_lookup` and the sync fallback were calling the deprecated `/search` (v1) endpoint; updated to `/v2/search`
- **Fixed:** `DorkEngine.run` — results were labelled with the requested engine name (`google`/`bing`/`ddg`) instead of `SearXNG` which is the actual backend; the 3× request multiplication (one pass per engine name, all hitting the same SearXNG pool) is eliminated
- **Fixed:** `DB.close()` — background event loop was stopped but never closed, leaving the loop object open on process exit
- **Fixed:** `NoxSourceProvider._fetch` — `backup_endpoints` defined in source plugins were parsed but never consumed; primary endpoint failure now falls through to backups in order
- **Fixed:** `_local_crack_sync_blocking` — `hashlib.md5/sha1` now called with `usedforsecurity=False` to prevent hard crash on FIPS-enabled systems (RHEL 9, hardened Kali); Python 3.8 compat guard included

### Codebase
- All internal tracking comments replaced with clean prose throughout `nox.py`, `build_sources.py`, and all helper modules

### Build
- `BUILD_DATE` updated to `2026-04-14`
- `pyproject.toml` version bumped to `1.0.2`; `requests` minimum pin aligned to `>=2.32.3`

## [1.0.1] — 2026-04-13

### Sources
- **Removed:** `cit0day` (HTML fingerprint challenge, no JSON response), `vigilante_pw` (redirects to dehashed.com), `scylla_sh_search` (domain parked, permanently unreachable)
- **Restored:** `proxynova_comb` (live, returns valid JSON — was incorrectly removed)
- **Fixed:** `leak_lookup` now requires API key (`LEAK_LOOKUP_API_KEY`) — provider removed unauthenticated access
- **Fixed:** `intelx_search` two-phase poll implemented — plugin previously submitted the search job but never polled for results, returning 0 records for all queries
- **Fixed:** `hudsonrock_osint` missing `User-Agent` header added — endpoint returns 403 without a browser UA
- **Removed:** `HASHES_API_KEY` registry entry — hashes.org shut down in 2023; `HASHES_COM_API_KEY` is the correct active slot
- **Added:** `LEAK_LOOKUP_API_KEY` to service registry

### Engine
- **Fixed:** `bypass_required` field in source plugins now enforced at runtime — sources declaring `["cloudflare"]` bypass are skipped when `curl_cffi` is absent (previously the field was parsed but never read)
- **Fixed:** Guardian proxy auto-fetch updated to ProxyScrape v3 API (v2 deprecated Q1 2026); `proxy-list.download` replaced with `proxifly` free list

### Dependencies
- **Updated:** `requests>=2.32.3` (CVE fixes)

### README
- Source count updated: 123 active plugins

### Dependencies
- **Updated:** `aiohttp` minimum pin raised to `>=3.13.5` (connection-pool stability fixes under high concurrency)
- **Added:** `zstandard>=0.23.0` — enables native zstd decompression in aiohttp for Cloudflare/Fastly CDN responses

### Engine
- **Updated:** `Accept-Encoding` header now includes `zstd` (`gzip, deflate, br, zstd`) to match Chrome 124+ behaviour

## [1.0.1] — 2026-04-11

### Sources
- **Added 9 new sources:** `proxynova_comb` (COMB breach search, free), `shodan_internetdb` (IP intel, free), `circl_hashlookup` (NSRL hash lookup, free), `ipapi_is` (IP geolocation, free), `threatfox` (abuse.ch IOC database), `urlhaus` (abuse.ch malware URLs), `malwarebazaar` (abuse.ch hash lookup), `fullhunt_subdomains` (attack surface), `netlas_search` (internet scanner)
- **Removed 7 dead sources:** `threatcrowd_email`, `threatcrowd_domain` (DNS dead), `spyse_domain`, `spyse_ip` (API shut down), `hashes_org` (DNS dead), `leakstats_pw` (DNS dead), `checkleaked` (endpoint gone)
- **Fixed:** `dehashed` endpoint migrated from `/search` to `/v2/search`
- **Fixed:** `hudsonrock_osint` endpoint corrected to `search-by-email` with `input_type: email` (was `search-by-login` with `input_type: username`)
- **Fixed:** `scylla_sh_search` migrated from dead `scylla.sh` to active `scylla.so`
- **Fixed:** `emailrep_io` now requires API key (`EMAILREP_API_KEY`) — free unauthenticated tier removed by provider
- **Fixed:** `duckduckgo_api` repurposed from dead DDG Instant Answer API to SearXNG JSON search

### Engine
- **Fixed:** POST 429 `Retry-After` cap was 4s (should be 30s, matching GET path)
- **Fixed:** Linear retry backoff replaced with exponential backoff + jitter in all 4 retry paths (`_get`, `_post`, `Session.get`, `Session.post`)
- **Fixed:** `--reset-sources` now removes orphaned plugins from `~/.nox/sources/` in addition to copying new ones
- **Fixed:** DDG HTML scraper replaced with SearXNG JSON API across all call sites — DDG HTML endpoint bot-blocked since 2025
- **Fixed:** SearXNG instance pool updated: `searx.be` (403), `search.bus-hit.me` (DNS dead), `searxng.site` (SSL error) replaced with 6 active instances; pool extracted to module-level `_SEARX_INSTANCES` constant
- **Fixed:** All 11 dead paste site APIs removed from `ScrapeEngine.PASTE_SITES`; paste intelligence now routed through SearXNG dorks and IntelX

### WAF Resilience
- **Updated:** User-Agent pool updated to Chrome/135, Firefox/136, Edge/135 (was Chrome/131, Firefox/133)
- **Added:** `Sec-CH-UA`, `Sec-CH-UA-Mobile`, `Sec-CH-UA-Platform` Client Hints headers for Chromium-based UAs
- **Fixed:** `_CH_UA_MAP` ordering — Edge UA now correctly gets `"Microsoft Edge"` brand (was getting `"Google Chrome"` due to dict iteration order)
- **Fixed:** `_search()` sync method no longer passes `use_cloudscraper=True` to SearXNG JSON API calls

### Hash Cracking
- **Removed 6 dead/paywalled cracker APIs:** nitrxgen (DNS dead), hash.help (DNS dead), hashkiller (403), hashes.com free path (404), md5decrypt (403), cmd5 (paywalled — returns `CMD5-ERROR:-1` for all hashes)
- **Added:** Local rockyou wordlist as primary crack path (no external calls, no rate limits, no data leakage)
- **Added:** `hashes.com` keyed API as external fallback (`HASHES_COM_API_KEY`)

### Dependencies
- **Added:** `brotli>=1.1.0` — required for aiohttp to decompress `br`-encoded responses

### Config
- **Added 7 new API key slots:** `EMAILREP_API_KEY`, `HASHES_COM_API_KEY`, `THREATFOX_API_KEY`, `URLHAUS_API_KEY`, `MALWAREBAZAAR_API_KEY`, `FULLHUNT_API_KEY`, `NETLAS_API_KEY`

## [1.0.0] — 2026-04-02

### Initial Release

- 124 Pydantic v2-validated JSON source plugins across breach, network, OSINT, and threat-intel categories
- Fully async execution engine (`asyncio` + `aiohttp`) with JA3 TLS fingerprinting and per-request jitter
- `--autoscan` pipeline: breach scan → recursive identity pivot (depth 2) → Google/DDG dorking → paste/Telegram scraping
- `--fullscan`: breach scan + pivot only
- `--scan` / REPL `scan`: breach sources only
- Guardian Proxy Engine: automatic proxy rotation with fail-safe kill-switch
- Risk scoring engine (0–100) with time-decay, source confidence weighting, persistence multipliers, and HVT detection
- Recursive Avalanche Engine: every discovered asset re-injected as a new scan seed across breach, dork, and scrape concurrently
- Union-Find identity clustering across all breach records
- Forensic PDF/HTML/JSON/CSV/Markdown reporting with Executive Summary dashboard
- Hash identification and multi-engine cracking (dictionary + mutations + online rainbow tables)
- Deep password strength analysis with entropy, leet-speak detection, and crack-time estimates
- Interactive REPL with full feature parity with the CLI
- Full audit logging: all scan events mirrored to `~/.nox/logs/nox.log`
- Isolated `.deb` packaging for Kali Linux (PEP 668 compliant — zero system pollution)
- `~/.config/nox-cli/apikeys.json` credential store (chmod 0600)