mirror of
https://github.com/nox-project/nox-framework.git
synced 2026-06-08 16:07:17 +00:00
release: v1.0.2
- 124 sources (+1 xposedornot, bgpview replaced with ripestat) - Fix gravatar MD5 transform, fofa base64 query encoding - Fix misp_search URL resolution, threatconnect HMAC placeholder - Fix spycloud, duckduckgo, mailboxlayer/numverify/ipstack/ipinfodb endpoints - Fix DeHashEngine v1→v2, DorkEngine engine label, backup_endpoints consumed - Fix Retry-After HTTP-date parsing, Hashmob API schema, FIPS hashlib crash - Fix DB.close() event loop leak, _random_headers CH-UA override - Add query_transform mechanism (md5_lower, fofa_domain) - Lower scores: spyonweb, pipl_search, twitter_v2, hudsonrock rate_limit - Clean all internal tracking comments, fix Italian docstring
This commit is contained in:
nox-project
@@ -2,6 +2,45 @@
|
|||||||
|
|
||||||
All notable changes to NOX are documented here.
|
All notable changes to NOX are documented here.
|
||||||
|
|
||||||
|
## [1.0.2] — 2026-04-14
|
||||||
|
|
||||||
|
### Sources
|
||||||
|
- **Fixed:** `misp_search` — `MISP_URL` added to `api_key_slots` so the instance base URL is resolved at runtime; `health_check_url` corrected from unresolvable placeholder to `https://misp.local`
|
||||||
|
- **Fixed:** `threatconnect_search` — removed unresolvable `{TC_SIGNATURE}` HMAC placeholder from the `Authorization` header; `reliability_score` lowered to `2`, `is_volatile` set to `true`
|
||||||
|
- **Fixed:** `spycloud_breach` — endpoint corrected from `breach/data/emails` to `breach/catalog/emails` (standard breach lookup tier)
|
||||||
|
- **Fixed:** `duckduckgo_api` — primary instance updated to `search.sapti.me`; 5 backup SearXNG instances added to `backup_endpoints` (now consumed by the engine)
|
||||||
|
- **Fixed:** `gravatar` — endpoint now MD5-hashes the email before URL substitution via new `query_transform: md5_lower` field; raw email was returning 404 on every query
|
||||||
|
- **Replaced:** `bgpview_ip` → `ripestat_ip` (RIPE Stat prefix-overview API) — BGPView free API decommissioned January 2025; RIPE Stat is free, keyless, and stable (`reliability_score: 5`)
|
||||||
|
- **Fixed:** `twitter_v2` — marked `is_volatile=true`, `confidence` lowered to `0.1`; free-tier bearer tokens receive HTTP 403 since February 2024
|
||||||
|
- **Fixed:** `fofa_info` — `qbase64` parameter now receives `base64(domain="<target>")` via `query_transform: fofa_domain`; raw domain was producing malformed queries
|
||||||
|
- **Fixed:** `pipl_search` — Pipl shut down public REST API in Q3 2024; `reliability_score` lowered to `2`, `confidence` to `0.3`, `is_volatile=true`
|
||||||
|
- **Fixed:** `spyonweb` — API confirmed unreachable; `reliability_score` lowered to `1`, `confidence` to `0.1`, `is_volatile=true`
|
||||||
|
- **Fixed:** `hudsonrock_osint` — `is_volatile=true`; `rate_limit` raised from `5.0` to `30.0` to respect Cavalier API throttling (~10 req/hour free tier)
|
||||||
|
- **Fixed:** `mailboxlayer`, `numverify`, `ipstack`, `ipinfodb` — endpoints and `health_check_url` migrated from `http://` to `https://`; API keys were being transmitted in cleartext before the server-side redirect
|
||||||
|
- **Added:** `xposedornot` plugin (free, public breach analytics)
|
||||||
|
- **Added:** `MISP_URL` to service registry and `apikeys.json` — back-filled automatically on first run after upgrade
|
||||||
|
- Source count: 123 → 124
|
||||||
|
|
||||||
|
### Config
|
||||||
|
- **Fixed:** Duplicate `xposedornot` entry removed from `SERVICE_REGISTRY` in `config_handler.py`
|
||||||
|
|
||||||
|
### Engine
|
||||||
|
- **Fixed:** `_parse_retry_after` helper added — `int()` on an HTTP-date `Retry-After` header raised `ValueError`, causing the retry loop to abort as a hard failure; all 5 call sites in `_get`, `_post`, `Session.get`, and `Session.post` updated
|
||||||
|
- **Fixed:** `_random_headers` — `Sec-CH-UA` Client Hints were emitted even when a Firefox UA was passed via the `extra` override; guard now evaluates the final `User-Agent` after overrides are applied
|
||||||
|
- **Fixed:** `HashEngine._hashmob` — Hashmob API v2 changed request field from `"hash"` to `"hashes"` (array) and response schema from `{found, result}` to `{data: [{plaintext}]}`
|
||||||
|
- **Fixed:** `DeHashEngine` — both `_lookup` and the sync fallback were calling the deprecated `/search` (v1) endpoint; updated to `/v2/search`
|
||||||
|
- **Fixed:** `DorkEngine.run` — results were labelled with the requested engine name (`google`/`bing`/`ddg`) instead of `SearXNG` which is the actual backend; the 3× request multiplication (one pass per engine name, all hitting the same SearXNG pool) is eliminated
|
||||||
|
- **Fixed:** `DB.close()` — background event loop was stopped but never closed, leaving the loop object open on process exit
|
||||||
|
- **Fixed:** `NoxSourceProvider._fetch` — `backup_endpoints` defined in source plugins were parsed but never consumed; primary endpoint failure now falls through to backups in order
|
||||||
|
- **Fixed:** `_local_crack_sync_blocking` — `hashlib.md5/sha1` now called with `usedforsecurity=False` to prevent hard crash on FIPS-enabled systems (RHEL 9, hardened Kali); Python 3.8 compat guard included
|
||||||
|
|
||||||
|
### Codebase
|
||||||
|
- All internal tracking comments replaced with clean prose throughout `nox.py`, `build_sources.py`, and all helper modules
|
||||||
|
|
||||||
|
### Build
|
||||||
|
- `BUILD_DATE` updated to `2026-04-14`
|
||||||
|
- `pyproject.toml` version bumped to `1.0.2`; `requests` minimum pin aligned to `>=2.32.3`
|
||||||
|
|
||||||
## [1.0.1] — 2026-04-13
|
## [1.0.1] — 2026-04-13
|
||||||
|
|
||||||
### Sources
|
### Sources
|
||||||
|
|||||||
@@ -11,13 +11,13 @@
|
|||||||
|
|
||||||
**Cyber Threat Intelligence Framework**
|
**Cyber Threat Intelligence Framework**
|
||||||
|
|
||||||
[](https://github.com/nox-project/nox-framework/releases/tag/v1.0.1)
|
[](https://github.com/nox-project/nox-framework/releases/tag/v1.0.2)
|
||||||
[](https://www.python.org/)
|
[](https://www.python.org/)
|
||||||
[](LICENSE.txt)
|
[](LICENSE.txt)
|
||||||
[](https://www.kali.org/)
|
[](https://www.kali.org/)
|
||||||
[](https://blackarch.org/)
|
[](https://blackarch.org/)
|
||||||
[](https://github.com/nox-project/nox-framework)
|
[](https://github.com/nox-project/nox-framework)
|
||||||
[](https://github.com/nox-project/nox-framework)
|
[](https://github.com/nox-project/nox-framework)
|
||||||
|
|
||||||
*OSINT framework for red teaming, digital forensics, and corporate exposure analysis.*
|
*OSINT framework for red teaming, digital forensics, and corporate exposure analysis.*
|
||||||
|
|
||||||
@@ -31,7 +31,7 @@ NOX is a purpose-built cyber threat intelligence engine designed for operators w
|
|||||||
|
|
||||||
| Capability | Detail |
|
| Capability | Detail |
|
||||||
|-|-|
|
|-|-|
|
||||||
| ⚡ **Async Execution Engine** | Massively parallel scanning across 123 intelligence feeds with no sequential bottlenecks and no blocking I/O. |
|
| ⚡ **Async Execution Engine** | Massively parallel scanning across 124 intelligence feeds with no sequential bottlenecks and no blocking I/O. |
|
||||||
| 🛡️ **Guardian Engine** | Integrated OPSEC layer with automatic proxy rotation and SOCKS5 support. Fail-safe kill-switch halts all traffic if the transport circuit is unavailable. |
|
| 🛡️ **Guardian Engine** | Integrated OPSEC layer with automatic proxy rotation and SOCKS5 support. Fail-safe kill-switch halts all traffic if the transport circuit is unavailable. |
|
||||||
| 🧠 **Risk Scoring** | Dynamic 0–100 scoring with time-decay, source confidence weighting, password complexity analysis, persistence multipliers, and HVT detection. |
|
| 🧠 **Risk Scoring** | Dynamic 0–100 scoring with time-decay, source confidence weighting, password complexity analysis, persistence multipliers, and HVT detection. |
|
||||||
| 🔗 **Recursive Avalanche Engine** | Every discovered asset — username, email, cracked password, phone — is automatically re-injected as a new scan seed. Per-asset pipeline runs sequentially (breach → crack → dork → scrape); child assets run concurrently. Identifiers from all four phases feed the pivot queue. Global deduplication and configurable depth cap prevent runaway recursion. |
|
| 🔗 **Recursive Avalanche Engine** | Every discovered asset — username, email, cracked password, phone — is automatically re-injected as a new scan seed. Per-asset pipeline runs sequentially (breach → crack → dork → scrape); child assets run concurrently. Identifiers from all four phases feed the pivot queue. Global deduplication and configurable depth cap prevent runaway recursion. |
|
||||||
@@ -43,7 +43,7 @@ NOX is a purpose-built cyber threat intelligence engine designed for operators w
|
|||||||
|
|
||||||
| Feature | Description |
|
| Feature | Description |
|
||||||
|-|-|
|
|-|-|
|
||||||
| **123 JSON Plugin Sources** | Every intelligence source is a JSON plugin. The execution engine contains zero hardcoded source logic. |
|
| **124 JSON Plugin Sources** | Every intelligence source is a JSON plugin. The execution engine contains zero hardcoded source logic. |
|
||||||
| **Async Core** | Full `asyncio` event loop with JA3 fingerprinting, SSL session management, per-request jitter, and configurable concurrency. |
|
| **Async Core** | Full `asyncio` event loop with JA3 fingerprinting, SSL session management, per-request jitter, and configurable concurrency. |
|
||||||
| **Autoscan Pipeline** | `--autoscan` triggers: breach scan → recursive pivot → Google/Bing/SearXNG dorking → paste/Telegram scraping — all in one command. |
|
| **Autoscan Pipeline** | `--autoscan` triggers: breach scan → recursive pivot → Google/Bing/SearXNG dorking → paste/Telegram scraping — all in one command. |
|
||||||
| **Recursive Avalanche Engine** | Every identifier discovered — from breach records, dork hits, or scraped paste/Telegram content — is re-injected as a new seed. Per-asset pipeline is sequential (breach → crack → dork → scrape); child assets run concurrently via `asyncio.gather`. A global `seen_assets` set prevents infinite loops. Concurrency and depth are fully configurable at runtime via `--threads` and `--depth`. |
|
| **Recursive Avalanche Engine** | Every identifier discovered — from breach records, dork hits, or scraped paste/Telegram content — is re-injected as a new seed. Per-asset pipeline is sequential (breach → crack → dork → scrape); child assets run concurrently via `asyncio.gather`. A global `seen_assets` set prevents infinite loops. Concurrency and depth are fully configurable at runtime via `--threads` and `--depth`. |
|
||||||
@@ -108,7 +108,7 @@ Supported fields: `name`, `endpoint`, `method`, `headers`, `regex_pattern` (or `
|
|||||||
```
|
```
|
||||||
For each asset (seed + every discovered identifier):
|
For each asset (seed + every discovered identifier):
|
||||||
├─ Phase 1 — Breach Scan
|
├─ Phase 1 — Breach Scan
|
||||||
│ 123 sources queried in parallel (async)
|
│ 124 sources queried in parallel (async)
|
||||||
│
|
│
|
||||||
├─ Phase 2 — Hash Crack (non-blocking, concurrent)
|
├─ Phase 2 — Hash Crack (non-blocking, concurrent)
|
||||||
│ Hashes found in breach data → rainbow-table APIs → cracked plaintext
|
│ Hashes found in breach data → rainbow-table APIs → cracked plaintext
|
||||||
@@ -258,7 +258,7 @@ nox-cli --help
|
|||||||
The post-install script automatically:
|
The post-install script automatically:
|
||||||
1. Creates an isolated virtual environment at `/opt/nox-cli/.venv`
|
1. Creates an isolated virtual environment at `/opt/nox-cli/.venv`
|
||||||
2. Installs all Python dependencies inside the venv (PEP 668 compliant — zero system pollution)
|
2. Installs all Python dependencies inside the venv (PEP 668 compliant — zero system pollution)
|
||||||
3. Builds the 123 source plugins
|
3. Builds the 124 source plugins
|
||||||
4. Links `/usr/bin/nox-cli` → `/opt/nox-cli/nox-wrapper.sh`
|
4. Links `/usr/bin/nox-cli` → `/opt/nox-cli/nox-wrapper.sh`
|
||||||
|
|
||||||
### Option 2: From Source
|
### Option 2: From Source
|
||||||
|
|||||||
+59
-38
@@ -69,8 +69,8 @@ class SourceConfig(BaseModel):
|
|||||||
bypass_required: Optional[List[str]] = None # omitted when empty
|
bypass_required: Optional[List[str]] = None # omitted when empty
|
||||||
user_agent_type: Optional[str] = None # omitted when absent
|
user_agent_type: Optional[str] = None # omitted when absent
|
||||||
backup_endpoints: List[str] = Field(default_factory=list)
|
backup_endpoints: List[str] = Field(default_factory=list)
|
||||||
# H2: optional confidence override — when set, takes precedence over formula
|
|
||||||
confidence: Optional[float] = None
|
confidence: Optional[float] = None
|
||||||
|
query_transform: Optional[str] = None # e.g. "md5_lower"
|
||||||
# Two-phase poll support (e.g. IntelX: POST → job_id → GET results)
|
# Two-phase poll support (e.g. IntelX: POST → job_id → GET results)
|
||||||
poll_endpoint: Optional[str] = None
|
poll_endpoint: Optional[str] = None
|
||||||
poll_id_field: Optional[str] = None
|
poll_id_field: Optional[str] = None
|
||||||
@@ -86,12 +86,10 @@ class SourceConfig(BaseModel):
|
|||||||
|
|
||||||
@model_validator(mode="after")
|
@model_validator(mode="after")
|
||||||
def _validate_source(self) -> "SourceConfig":
|
def _validate_source(self) -> "SourceConfig":
|
||||||
# H1: GET endpoints must contain {target} placeholder
|
|
||||||
if self.method.upper() == "GET" and "{target}" not in self.endpoint:
|
if self.method.upper() == "GET" and "{target}" not in self.endpoint:
|
||||||
raise ValueError(
|
raise ValueError(
|
||||||
f"'{self.name}': GET endpoint must contain {{target}} placeholder: {self.endpoint!r}"
|
f"'{self.name}': GET endpoint must contain {{target}} placeholder: {self.endpoint!r}"
|
||||||
)
|
)
|
||||||
# L3: volatile sources must have reliability_score ≤ 4 (was > 3, now > 4)
|
|
||||||
if self.is_volatile and self.reliability_score > 4:
|
if self.is_volatile and self.reliability_score > 4:
|
||||||
raise ValueError(
|
raise ValueError(
|
||||||
f"'{self.name}': is_volatile sources must have reliability_score ≤ 4"
|
f"'{self.name}': is_volatile sources must have reliability_score ≤ 4"
|
||||||
@@ -100,11 +98,10 @@ class SourceConfig(BaseModel):
|
|||||||
|
|
||||||
def to_json(self) -> str:
|
def to_json(self) -> str:
|
||||||
data = self.model_dump(exclude_none=True)
|
data = self.model_dump(exclude_none=True)
|
||||||
# Drop is_volatile / bypass_required / user_agent_type when falsy
|
# Drop falsy optional fields
|
||||||
for key in ("is_volatile", "bypass_required", "user_agent_type"):
|
for key in ("is_volatile", "bypass_required", "user_agent_type", "query_transform"):
|
||||||
if not data.get(key):
|
if not data.get(key):
|
||||||
data.pop(key, None)
|
data.pop(key, None)
|
||||||
# H2: use explicit confidence if set, otherwise derive from reliability_score
|
|
||||||
data["confidence"] = (
|
data["confidence"] = (
|
||||||
round(self.confidence, 2)
|
round(self.confidence, 2)
|
||||||
if self.confidence is not None
|
if self.confidence is not None
|
||||||
@@ -140,6 +137,8 @@ def _mk(
|
|||||||
poll_id_field: Optional[str] = None,
|
poll_id_field: Optional[str] = None,
|
||||||
poll_id_param: Optional[str] = None,
|
poll_id_param: Optional[str] = None,
|
||||||
poll_json_root: Optional[str] = None,
|
poll_json_root: Optional[str] = None,
|
||||||
|
confidence: Optional[float] = None,
|
||||||
|
query_transform: Optional[str] = None,
|
||||||
) -> SourceConfig:
|
) -> SourceConfig:
|
||||||
return SourceConfig(
|
return SourceConfig(
|
||||||
name=name, category=category, endpoint=endpoint, method=method,
|
name=name, category=category, endpoint=endpoint, method=method,
|
||||||
@@ -163,6 +162,8 @@ def _mk(
|
|||||||
poll_id_field=poll_id_field,
|
poll_id_field=poll_id_field,
|
||||||
poll_id_param=poll_id_param,
|
poll_id_param=poll_id_param,
|
||||||
poll_json_root=poll_json_root,
|
poll_json_root=poll_json_root,
|
||||||
|
confidence=confidence,
|
||||||
|
query_transform=query_transform or None,
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
@@ -261,15 +262,27 @@ FREE_PUBLIC_SOURCES: List[SourceConfig] = [
|
|||||||
tags=["passive", "threat"],
|
tags=["passive", "threat"],
|
||||||
health_check_url="https://pulsedive.com", reliability_score=4),
|
health_check_url="https://pulsedive.com", reliability_score=4),
|
||||||
|
|
||||||
|
_base("xposedornot", "breach_data",
|
||||||
|
"https://api.xposedornot.com/v1/breach-analytics?email={target}", "GET",
|
||||||
|
{"breaches": "$.ExposedBreaches.breaches_details"},
|
||||||
|
rate_limit=2.0,
|
||||||
|
headers={"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/135.0.0.0 Safari/537.36"},
|
||||||
|
input_type="email", output_type=["email", "domain"],
|
||||||
|
normalization_map={"breach": "breach_name", "domain": "domain",
|
||||||
|
"xposed_date": "breach_date", "xposed_data": "data_types",
|
||||||
|
"password_risk": "password_risk"},
|
||||||
|
tags=["passive", "stealth"],
|
||||||
|
health_check_url="https://api.xposedornot.com", reliability_score=4, confidence=0.75),
|
||||||
|
|
||||||
_base("hudsonrock_osint", "breach_data",
|
_base("hudsonrock_osint", "breach_data",
|
||||||
"https://cavalier.hudsonrock.com/api/json/v2/osint-tools/search-by-email?email={target}", "GET",
|
"https://cavalier.hudsonrock.com/api/json/v2/osint-tools/search-by-email?email={target}", "GET",
|
||||||
{"stealers": "$.stealers"},
|
{"stealers": "$.stealers"},
|
||||||
rate_limit=5.0,
|
rate_limit=30.0,
|
||||||
headers={"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/135.0.0.0 Safari/537.36"},
|
headers={"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/135.0.0.0 Safari/537.36"},
|
||||||
input_type="email", output_type=["email", "domain", "username"],
|
input_type="email", output_type=["email", "domain", "username"],
|
||||||
normalization_map={"stealers": "breach_record"},
|
normalization_map={"stealers": "breach_record"},
|
||||||
tags=["passive", "stealth"],
|
tags=["passive", "stealth"],
|
||||||
health_check_url="https://cavalier.hudsonrock.com", reliability_score=3),
|
health_check_url="https://cavalier.hudsonrock.com", reliability_score=3, is_volatile=True),
|
||||||
|
|
||||||
_base("ipinfo_io", "geolocation",
|
_base("ipinfo_io", "geolocation",
|
||||||
"https://ipinfo.io/{target}/json", "GET",
|
"https://ipinfo.io/{target}/json", "GET",
|
||||||
@@ -288,12 +301,13 @@ FREE_PUBLIC_SOURCES: List[SourceConfig] = [
|
|||||||
tags=["passive", "fast"],
|
tags=["passive", "fast"],
|
||||||
health_check_url="https://ipapi.co", reliability_score=4),
|
health_check_url="https://ipapi.co", reliability_score=4),
|
||||||
|
|
||||||
_base("bgpview_ip", "network",
|
_base("ripestat_ip", "network",
|
||||||
"https://api.bgpview.io/ip/{target}", "GET",
|
"https://stat.ripe.net/data/prefix-overview/data.json?resource={target}", "GET",
|
||||||
{"prefixes": "$.data.prefixes[*].prefix"},
|
{"asns": "$.data.asns[*].asn", "holder": "$.data.asns[0].holder"},
|
||||||
input_type="ip", output_type=["ip"],
|
input_type="ip", output_type=["ip"],
|
||||||
|
normalization_map={"asn": "asn_number", "holder": "asn_org"},
|
||||||
tags=["passive", "infrastructure"],
|
tags=["passive", "infrastructure"],
|
||||||
health_check_url="https://api.bgpview.io", reliability_score=2, is_volatile=True),
|
health_check_url="https://stat.ripe.net", reliability_score=5),
|
||||||
|
|
||||||
_auth("emailrep_io", "email_rep",
|
_auth("emailrep_io", "email_rep",
|
||||||
"https://emailrep.io/{target}", "GET",
|
"https://emailrep.io/{target}", "GET",
|
||||||
@@ -327,6 +341,7 @@ FREE_PUBLIC_SOURCES: List[SourceConfig] = [
|
|||||||
{"name": "$.entry[0].displayName"},
|
{"name": "$.entry[0].displayName"},
|
||||||
rate_limit=2.0,
|
rate_limit=2.0,
|
||||||
input_type="email", output_type=["username"],
|
input_type="email", output_type=["username"],
|
||||||
|
query_transform="md5_lower",
|
||||||
tags=["passive"],
|
tags=["passive"],
|
||||||
health_check_url="https://www.gravatar.com", reliability_score=4),
|
health_check_url="https://www.gravatar.com", reliability_score=4),
|
||||||
|
|
||||||
@@ -445,12 +460,19 @@ FREE_PUBLIC_SOURCES: List[SourceConfig] = [
|
|||||||
health_check_url="https://checkurl.phishtank.com", reliability_score=4),
|
health_check_url="https://checkurl.phishtank.com", reliability_score=4),
|
||||||
|
|
||||||
_base("duckduckgo_api", "search",
|
_base("duckduckgo_api", "search",
|
||||||
"https://searx.tiekoetter.com/search?q={target}&format=json&categories=general", "GET",
|
"https://search.sapti.me/search?q={target}&format=json&categories=general", "GET",
|
||||||
{"results": "$.results"},
|
{"results": "$.results"},
|
||||||
input_type="any", output_type=["url"],
|
input_type="any", output_type=["url"],
|
||||||
normalization_map={"url": "url", "title": "title"},
|
normalization_map={"url": "url", "title": "title"},
|
||||||
tags=["passive", "fast"],
|
tags=["passive", "fast"],
|
||||||
health_check_url="https://searx.tiekoetter.com", reliability_score=3, is_volatile=True),
|
backup_endpoints=[
|
||||||
|
"https://searx.tiekoetter.com/search?q={target}&format=json&categories=general",
|
||||||
|
"https://searx.perennialte.ch/search?q={target}&format=json&categories=general",
|
||||||
|
"https://search.mdosch.de/search?q={target}&format=json&categories=general",
|
||||||
|
"https://paulgo.io/search?q={target}&format=json&categories=general",
|
||||||
|
"https://priv.au/search?q={target}&format=json&categories=general",
|
||||||
|
],
|
||||||
|
health_check_url="https://search.sapti.me", reliability_score=3, is_volatile=True),
|
||||||
|
|
||||||
_base("cve_search", "vulns",
|
_base("cve_search", "vulns",
|
||||||
"https://cve.circl.lu/api/cve/{target}", "GET",
|
"https://cve.circl.lu/api/cve/{target}", "GET",
|
||||||
@@ -482,7 +504,6 @@ FREE_PUBLIC_SOURCES: List[SourceConfig] = [
|
|||||||
tags=["passive", "stealth"],
|
tags=["passive", "stealth"],
|
||||||
health_check_url="https://api.proxynova.com", reliability_score=3, is_volatile=True),
|
health_check_url="https://api.proxynova.com", reliability_score=3, is_volatile=True),
|
||||||
|
|
||||||
# ── New free sources (v1.0.1) ─────────────────────────────────────────────
|
|
||||||
|
|
||||||
_base("shodan_internetdb", "scanners",
|
_base("shodan_internetdb", "scanners",
|
||||||
"https://internetdb.shodan.io/{target}", "GET",
|
"https://internetdb.shodan.io/{target}", "GET",
|
||||||
@@ -597,6 +618,7 @@ AUTHENTICATED_PREMIUM_SOURCES: List[SourceConfig] = [
|
|||||||
{"results": "$.results"},
|
{"results": "$.results"},
|
||||||
api_key_slots=["{FOFA_API_KEY}", "{FOFA_EMAIL}"],
|
api_key_slots=["{FOFA_API_KEY}", "{FOFA_EMAIL}"],
|
||||||
input_type="domain", output_type=["ip", "domain"],
|
input_type="domain", output_type=["ip", "domain"],
|
||||||
|
query_transform="fofa_domain",
|
||||||
tags=["passive", "infrastructure"],
|
tags=["passive", "infrastructure"],
|
||||||
health_check_url="https://fofa.info", reliability_score=4),
|
health_check_url="https://fofa.info", reliability_score=4),
|
||||||
|
|
||||||
@@ -732,11 +754,11 @@ AUTHENTICATED_PREMIUM_SOURCES: List[SourceConfig] = [
|
|||||||
_auth("threatconnect_search", "threat_intel",
|
_auth("threatconnect_search", "threat_intel",
|
||||||
"https://api.threatconnect.com/v2/indicators/{target}", "GET",
|
"https://api.threatconnect.com/v2/indicators/{target}", "GET",
|
||||||
{"data": "$.data"},
|
{"data": "$.data"},
|
||||||
headers={"Authorization": "TC {TC_API_KEY}:{TC_SIGNATURE}"},
|
headers={"Authorization": "TC {TC_API_KEY}"},
|
||||||
api_key_slots=["{TC_API_KEY}"],
|
api_key_slots=["{TC_API_KEY}"],
|
||||||
input_type="any", output_type=["ip", "domain"],
|
input_type="any", output_type=["ip", "domain"],
|
||||||
tags=["passive", "threat"],
|
tags=["passive", "threat"],
|
||||||
health_check_url="https://api.threatconnect.com", reliability_score=4),
|
health_check_url="https://api.threatconnect.com", reliability_score=2, is_volatile=True),
|
||||||
|
|
||||||
_auth("threatportal", "threat_intel",
|
_auth("threatportal", "threat_intel",
|
||||||
"https://threatportal.io/api/v1/search?q={target}", "GET",
|
"https://threatportal.io/api/v1/search?q={target}", "GET",
|
||||||
@@ -797,11 +819,11 @@ AUTHENTICATED_PREMIUM_SOURCES: List[SourceConfig] = [
|
|||||||
"{MISP_URL}/attributes/restSearch", "POST",
|
"{MISP_URL}/attributes/restSearch", "POST",
|
||||||
{"attributes": "$.Attribute[*].value"},
|
{"attributes": "$.Attribute[*].value"},
|
||||||
headers={"Authorization": "{MISP_API_KEY}", "Content-Type": "application/json"},
|
headers={"Authorization": "{MISP_API_KEY}", "Content-Type": "application/json"},
|
||||||
api_key_slots=["{MISP_API_KEY}"],
|
api_key_slots=["{MISP_API_KEY}", "{MISP_URL}"],
|
||||||
input_type="any", output_type=["ip", "domain", "hash"],
|
input_type="any", output_type=["ip", "domain", "hash"],
|
||||||
payload_template={"returnFormat": "json", "value": "{target}"},
|
payload_template={"returnFormat": "json", "value": "{target}"},
|
||||||
tags=["passive", "threat"],
|
tags=["passive", "threat"],
|
||||||
health_check_url="{MISP_URL}", reliability_score=4),
|
health_check_url="https://misp.local", reliability_score=4),
|
||||||
]
|
]
|
||||||
|
|
||||||
AUTHENTICATED_PREMIUM_SOURCES += [
|
AUTHENTICATED_PREMIUM_SOURCES += [
|
||||||
@@ -879,7 +901,7 @@ AUTHENTICATED_PREMIUM_SOURCES += [
|
|||||||
health_check_url="https://leakcheck.io", reliability_score=4),
|
health_check_url="https://leakcheck.io", reliability_score=4),
|
||||||
|
|
||||||
_auth("spycloud_breach", "breaches",
|
_auth("spycloud_breach", "breaches",
|
||||||
"https://api.spycloud.io/enterprise-v2/breach/data/emails/{target}", "GET",
|
"https://api.spycloud.io/enterprise-v2/breach/catalog/emails/{target}", "GET",
|
||||||
{"results": "$.results"},
|
{"results": "$.results"},
|
||||||
headers={"X-API-Key": "{SPYCLOUD_API_KEY}"},
|
headers={"X-API-Key": "{SPYCLOUD_API_KEY}"},
|
||||||
api_key_slots=["{SPYCLOUD_API_KEY}"],
|
api_key_slots=["{SPYCLOUD_API_KEY}"],
|
||||||
@@ -990,7 +1012,8 @@ AUTHENTICATED_PREMIUM_SOURCES += [
|
|||||||
api_key_slots=["{SPYONWEB_API_KEY}"],
|
api_key_slots=["{SPYONWEB_API_KEY}"],
|
||||||
input_type="domain", output_type=["domain"],
|
input_type="domain", output_type=["domain"],
|
||||||
tags=["passive"],
|
tags=["passive"],
|
||||||
health_check_url="https://api.spyonweb.com", reliability_score=3),
|
health_check_url="https://api.spyonweb.com", reliability_score=1,
|
||||||
|
is_volatile=True, confidence=0.1),
|
||||||
|
|
||||||
# ── WHOIS ─────────────────────────────────────────────────────────────────
|
# ── WHOIS ─────────────────────────────────────────────────────────────────
|
||||||
|
|
||||||
@@ -1077,7 +1100,8 @@ AUTHENTICATED_PREMIUM_SOURCES += [
|
|||||||
api_key_slots=["{PIPL_API_KEY}"],
|
api_key_slots=["{PIPL_API_KEY}"],
|
||||||
input_type="email", output_type=["username", "domain", "phone"],
|
input_type="email", output_type=["username", "domain", "phone"],
|
||||||
tags=["passive"],
|
tags=["passive"],
|
||||||
health_check_url="https://api.pipl.com", reliability_score=4),
|
health_check_url="https://api.pipl.com", reliability_score=2,
|
||||||
|
is_volatile=True, confidence=0.3),
|
||||||
|
|
||||||
# ── Email Reputation ──────────────────────────────────────────────────────
|
# ── Email Reputation ──────────────────────────────────────────────────────
|
||||||
|
|
||||||
@@ -1116,12 +1140,12 @@ AUTHENTICATED_PREMIUM_SOURCES += [
|
|||||||
health_check_url="https://api.hunter.io", reliability_score=4),
|
health_check_url="https://api.hunter.io", reliability_score=4),
|
||||||
|
|
||||||
_auth("mailboxlayer", "email_rep",
|
_auth("mailboxlayer", "email_rep",
|
||||||
"http://apilayer.net/api/check?access_key={MAILBOX_API_KEY}&email={target}", "GET",
|
"https://apilayer.net/api/check?access_key={MAILBOX_API_KEY}&email={target}", "GET",
|
||||||
{"score": "$.score"},
|
{"score": "$.score"},
|
||||||
api_key_slots=["{MAILBOX_API_KEY}"],
|
api_key_slots=["{MAILBOX_API_KEY}"],
|
||||||
input_type="email", output_type=["email"],
|
input_type="email", output_type=["email"],
|
||||||
tags=["passive"],
|
tags=["passive"],
|
||||||
health_check_url="http://apilayer.net", reliability_score=3),
|
health_check_url="https://apilayer.net", reliability_score=3),
|
||||||
|
|
||||||
_auth("abstract_email", "email_rep",
|
_auth("abstract_email", "email_rep",
|
||||||
"https://emailvalidation.abstractapi.com/v1/?api_key={ABSTRACT_API_KEY}&email={target}", "GET",
|
"https://emailvalidation.abstractapi.com/v1/?api_key={ABSTRACT_API_KEY}&email={target}", "GET",
|
||||||
@@ -1149,7 +1173,8 @@ AUTHENTICATED_PREMIUM_SOURCES += [
|
|||||||
api_key_slots=["{TWITTER_BEARER_TOKEN}"],
|
api_key_slots=["{TWITTER_BEARER_TOKEN}"],
|
||||||
input_type="username", output_type=["username"],
|
input_type="username", output_type=["username"],
|
||||||
tags=["passive"],
|
tags=["passive"],
|
||||||
health_check_url="https://api.twitter.com", reliability_score=1),
|
health_check_url="https://api.twitter.com", reliability_score=1,
|
||||||
|
is_volatile=True, confidence=0.1),
|
||||||
|
|
||||||
_auth("github_code_search", "code",
|
_auth("github_code_search", "code",
|
||||||
"https://api.github.com/search/code?q={target}", "GET",
|
"https://api.github.com/search/code?q={target}", "GET",
|
||||||
@@ -1172,13 +1197,13 @@ AUTHENTICATED_PREMIUM_SOURCES += [
|
|||||||
# ── Geolocation ───────────────────────────────────────────────────────────
|
# ── Geolocation ───────────────────────────────────────────────────────────
|
||||||
|
|
||||||
_auth("ipstack", "geolocation",
|
_auth("ipstack", "geolocation",
|
||||||
"http://api.ipstack.com/{target}?access_key={IPSTACK_API_KEY}", "GET",
|
"https://api.ipstack.com/{target}?access_key={IPSTACK_API_KEY}", "GET",
|
||||||
{"country": "$.country_name"},
|
{"country": "$.country_name"},
|
||||||
api_key_slots=["{IPSTACK_API_KEY}"],
|
api_key_slots=["{IPSTACK_API_KEY}"],
|
||||||
input_type="ip", output_type=["ip"],
|
input_type="ip", output_type=["ip"],
|
||||||
normalization_map={"country_name": "geo_country"},
|
normalization_map={"country_name": "geo_country"},
|
||||||
tags=["passive", "fast"],
|
tags=["passive", "fast"],
|
||||||
health_check_url="http://api.ipstack.com", reliability_score=4),
|
health_check_url="https://api.ipstack.com", reliability_score=4),
|
||||||
|
|
||||||
_auth("ipgeolocation_io", "geolocation",
|
_auth("ipgeolocation_io", "geolocation",
|
||||||
"https://api.ipgeolocation.io/ipgeo?apiKey={IPGEO_API_KEY}&ip={target}", "GET",
|
"https://api.ipgeolocation.io/ipgeo?apiKey={IPGEO_API_KEY}&ip={target}", "GET",
|
||||||
@@ -1207,27 +1232,24 @@ AUTHENTICATED_PREMIUM_SOURCES += [
|
|||||||
health_check_url="https://extreme-ip-lookup.com", reliability_score=3),
|
health_check_url="https://extreme-ip-lookup.com", reliability_score=3),
|
||||||
|
|
||||||
_auth("ipinfodb", "geolocation",
|
_auth("ipinfodb", "geolocation",
|
||||||
"http://api.ipinfodb.com/v3/ip-city/?key={IPINFODB_API_KEY}&ip={target}&format=json", "GET",
|
"https://api.ipinfodb.com/v3/ip-city/?key={IPINFODB_API_KEY}&ip={target}&format=json", "GET",
|
||||||
{"city": "$.cityName"},
|
{"city": "$.cityName"},
|
||||||
api_key_slots=["{IPINFODB_API_KEY}"],
|
api_key_slots=["{IPINFODB_API_KEY}"],
|
||||||
input_type="ip", output_type=["ip"],
|
input_type="ip", output_type=["ip"],
|
||||||
normalization_map={"cityName": "geo_city"},
|
normalization_map={"cityName": "geo_city"},
|
||||||
tags=["passive"],
|
tags=["passive"],
|
||||||
health_check_url="http://api.ipinfodb.com", reliability_score=3),
|
health_check_url="https://api.ipinfodb.com", reliability_score=3),
|
||||||
|
|
||||||
# ── Phone ─────────────────────────────────────────────────────────────────
|
# ── Phone ─────────────────────────────────────────────────────────────────
|
||||||
|
|
||||||
_auth("numverify", "phone",
|
_auth("numverify", "phone",
|
||||||
"http://apilayer.net/api/validate?access_key={NUMVERIFY_API_KEY}&number={target}", "GET",
|
"https://apilayer.net/api/validate?access_key={NUMVERIFY_API_KEY}&number={target}", "GET",
|
||||||
{"valid": "$.valid", "carrier": "$.carrier"},
|
{"valid": "$.valid", "carrier": "$.carrier"},
|
||||||
api_key_slots=["{NUMVERIFY_API_KEY}"],
|
api_key_slots=["{NUMVERIFY_API_KEY}"],
|
||||||
input_type="phone", output_type=["phone"],
|
input_type="phone", output_type=["phone"],
|
||||||
normalization_map={"valid": "phone_valid", "carrier": "phone_carrier"},
|
normalization_map={"valid": "phone_valid", "carrier": "phone_carrier"},
|
||||||
tags=["passive"],
|
tags=["passive"],
|
||||||
health_check_url="http://apilayer.net", reliability_score=4),
|
health_check_url="https://apilayer.net", reliability_score=4),
|
||||||
|
|
||||||
# ── Hashes ────────────────────────────────────────────────────────────────
|
|
||||||
# hashes_org removed — service unavailable
|
|
||||||
|
|
||||||
# ── Search ────────────────────────────────────────────────────────────────
|
# ── Search ────────────────────────────────────────────────────────────────
|
||||||
|
|
||||||
@@ -1248,7 +1270,6 @@ AUTHENTICATED_PREMIUM_SOURCES += [
|
|||||||
tags=["passive"],
|
tags=["passive"],
|
||||||
health_check_url="https://api.bing.microsoft.com", reliability_score=5),
|
health_check_url="https://api.bing.microsoft.com", reliability_score=5),
|
||||||
|
|
||||||
# ── New authenticated sources (v1.0.1) ───────────────────────────────────
|
|
||||||
|
|
||||||
_auth("threatfox", "threat_intel",
|
_auth("threatfox", "threat_intel",
|
||||||
"https://threatfox-api.abuse.ch/api/v1/", "POST",
|
"https://threatfox-api.abuse.ch/api/v1/", "POST",
|
||||||
@@ -1315,9 +1336,9 @@ AUTHENTICATED_PREMIUM_SOURCES += [
|
|||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
def build_nox_sources(output_dir: str = None) -> None:
|
def build_nox_sources(output_dir: str = None) -> None:
|
||||||
# H3: resolve output_dir relative to this script's location, not CWD.
|
# Resolve output_dir relative to this script's location so the command
|
||||||
# This ensures `python /opt/nox-cli/build_sources.py` from any directory
|
# `python /opt/nox-cli/build_sources.py` always writes to the correct
|
||||||
# always writes to /opt/nox-cli/sources/ instead of ./sources/.
|
# package sources/ directory regardless of the working directory.
|
||||||
if output_dir is None:
|
if output_dir is None:
|
||||||
output_dir = str(Path(__file__).resolve().parent / "sources")
|
output_dir = str(Path(__file__).resolve().parent / "sources")
|
||||||
os.makedirs(output_dir, exist_ok=True)
|
os.makedirs(output_dir, exist_ok=True)
|
||||||
|
|||||||
@@ -65,7 +65,6 @@ import re
|
|||||||
import sys
|
import sys
|
||||||
import time
|
import time
|
||||||
import threading
|
import threading
|
||||||
# Module-level lock for thread-safe proxy env var assignment (Bug 9 fix)
|
|
||||||
_PROXY_ENV_LOCK = threading.Lock()
|
_PROXY_ENV_LOCK = threading.Lock()
|
||||||
import argparse
|
import argparse
|
||||||
import csv
|
import csv
|
||||||
@@ -151,7 +150,7 @@ except Exception:
|
|||||||
VERSION = _sp2.check_output(["dpkg-query", "-W", "-f=${Version}", "nox-cli"], stderr=_sp2.DEVNULL).decode().strip() or VERSION
|
VERSION = _sp2.check_output(["dpkg-query", "-W", "-f=${Version}", "nox-cli"], stderr=_sp2.DEVNULL).decode().strip() or VERSION
|
||||||
except Exception:
|
except Exception:
|
||||||
pass
|
pass
|
||||||
BUILD_DATE = "2026-04-02"
|
BUILD_DATE = "2026-04-14"
|
||||||
|
|
||||||
# ── Smart Path Layout ──────────────────────────────────────────────────
|
# ── Smart Path Layout ──────────────────────────────────────────────────
|
||||||
HOME_NOX = Path.home() / ".nox"
|
HOME_NOX = Path.home() / ".nox"
|
||||||
@@ -205,8 +204,8 @@ def initialize_environment() -> None:
|
|||||||
cfg.write(fh)
|
cfg.write(fh)
|
||||||
|
|
||||||
# Smart source discovery: seed ~/.nox/sources/ from package sources/
|
# Smart source discovery: seed ~/.nox/sources/ from package sources/
|
||||||
# B6: only copy if destination is absent — never silently overwrite
|
# Only copies files that are absent — never overwrites user-customised sources.
|
||||||
# user-customised sources. Use --reset-sources to force a full resync.
|
# Use --reset-sources to force a full resync.
|
||||||
candidate = _PKG_ROOT / "sources"
|
candidate = _PKG_ROOT / "sources"
|
||||||
if not candidate.is_dir():
|
if not candidate.is_dir():
|
||||||
candidate = Path("/usr/share/nox-cli/sources")
|
candidate = Path("/usr/share/nox-cli/sources")
|
||||||
@@ -279,7 +278,6 @@ class NoxConfig:
|
|||||||
self.allow_leak = False
|
self.allow_leak = False
|
||||||
self.no_online_crack = False
|
self.no_online_crack = False
|
||||||
self.max_threads = Cfg.CONCURRENCY
|
self.max_threads = Cfg.CONCURRENCY
|
||||||
# A9/I3: pivot control — readable by AvalancheScanner
|
|
||||||
self.no_pivot = False
|
self.no_pivot = False
|
||||||
self.pivot_depth = Cfg.PIVOT_DEPTH
|
self.pivot_depth = Cfg.PIVOT_DEPTH
|
||||||
|
|
||||||
@@ -626,7 +624,7 @@ class Record:
|
|||||||
return hashlib.sha256(f"{em}:{pw}".encode()).hexdigest()
|
return hashlib.sha256(f"{em}:{pw}".encode()).hexdigest()
|
||||||
|
|
||||||
def get_fingerprint(self) -> str:
|
def get_fingerprint(self) -> str:
|
||||||
"""Genera un hash univoco per evitare duplicati nel database."""
|
"""Return a SHA-256 fingerprint for cross-source deduplication."""
|
||||||
data_str = f"{self.source}|{self.email}|{self.password}|{self.phone}|{self.address}"
|
data_str = f"{self.source}|{self.email}|{self.password}|{self.phone}|{self.address}"
|
||||||
return hashlib.sha256(data_str.encode()).hexdigest()
|
return hashlib.sha256(data_str.encode()).hexdigest()
|
||||||
|
|
||||||
@@ -663,8 +661,8 @@ class RiskEngine:
|
|||||||
pts = 0.0
|
pts = 0.0
|
||||||
if record.password:
|
if record.password:
|
||||||
pts += 60
|
pts += 60
|
||||||
# I5: adjust base points by password complexity
|
# Adjust base points by password complexity.
|
||||||
# Weak passwords (trivially guessable) score lower; strong ones score higher.
|
# Weak passwords score lower; strong ones score higher.
|
||||||
try:
|
try:
|
||||||
_pa_score = PassAnalyzer().analyze(record.password).get("score", 50)
|
_pa_score = PassAnalyzer().analyze(record.password).get("score", 50)
|
||||||
if _pa_score < 30:
|
if _pa_score < 30:
|
||||||
@@ -1069,7 +1067,6 @@ class DatabaseManager:
|
|||||||
iid = row["id"]
|
iid = row["id"]
|
||||||
for pivot_val, count in profile.pivot_count.items():
|
for pivot_val, count in profile.pivot_count.items():
|
||||||
if count > 1:
|
if count > 1:
|
||||||
# I6: use Detect.qtype instead of length heuristic
|
|
||||||
_ptype = Detect.qtype(pivot_val)
|
_ptype = Detect.qtype(pivot_val)
|
||||||
if _ptype not in ("email", "username", "phone", "domain", "ip"):
|
if _ptype not in ("email", "username", "phone", "domain", "ip"):
|
||||||
_ptype = "username"
|
_ptype = "username"
|
||||||
@@ -1190,7 +1187,6 @@ class DatabaseManager:
|
|||||||
iid = row["id"]
|
iid = row["id"]
|
||||||
for pivot_val, count in profile.pivot_count.items():
|
for pivot_val, count in profile.pivot_count.items():
|
||||||
if count > 1:
|
if count > 1:
|
||||||
# I6: use Detect.qtype instead of length heuristic
|
|
||||||
_ptype = Detect.qtype(pivot_val)
|
_ptype = Detect.qtype(pivot_val)
|
||||||
if _ptype not in ("email", "username", "phone", "domain", "ip"):
|
if _ptype not in ("email", "username", "phone", "domain", "ip"):
|
||||||
_ptype = "username"
|
_ptype = "username"
|
||||||
@@ -1514,10 +1510,14 @@ class DB:
|
|||||||
|
|
||||||
def close(self) -> None:
|
def close(self) -> None:
|
||||||
"""Stop the background event loop thread and release resources."""
|
"""Stop the background event loop thread and release resources."""
|
||||||
if self._use_async and hasattr(self, "_loop") and self._loop.is_running():
|
if not (self._use_async and hasattr(self, "_loop")):
|
||||||
|
return
|
||||||
|
if self._loop.is_running():
|
||||||
self._loop.call_soon_threadsafe(self._loop.stop)
|
self._loop.call_soon_threadsafe(self._loop.stop)
|
||||||
if hasattr(self, "_loop_thread"):
|
if hasattr(self, "_loop_thread"):
|
||||||
self._loop_thread.join(timeout=5)
|
self._loop_thread.join(timeout=5)
|
||||||
|
if not self._loop.is_closed():
|
||||||
|
self._loop.close()
|
||||||
|
|
||||||
def __del__(self) -> None:
|
def __del__(self) -> None:
|
||||||
try:
|
try:
|
||||||
@@ -1607,19 +1607,21 @@ def _random_headers(extra: Optional[Dict] = None) -> Dict[str, str]:
|
|||||||
"Sec-Fetch-Site": random.choice(_SEC_FETCH_SITE_POOL),
|
"Sec-Fetch-Site": random.choice(_SEC_FETCH_SITE_POOL),
|
||||||
"Cache-Control": "max-age=0",
|
"Cache-Control": "max-age=0",
|
||||||
}
|
}
|
||||||
# Add Sec-CH-UA Client Hints for Chromium-based UAs (Firefox omits these)
|
if extra:
|
||||||
if "Firefox" not in ua:
|
h.update(extra)
|
||||||
ch_ua = next((v for k, v in _CH_UA_MAP if k in ua), None)
|
# Derive the final UA after applying overrides so that a Firefox UA passed
|
||||||
|
# via `extra` correctly suppresses Chromium-only Sec-CH-UA headers.
|
||||||
|
final_ua = h["User-Agent"]
|
||||||
|
if "Firefox" not in final_ua:
|
||||||
|
ch_ua = next((v for k, v in _CH_UA_MAP if k in final_ua), None)
|
||||||
if ch_ua:
|
if ch_ua:
|
||||||
h["Sec-CH-UA"] = ch_ua
|
h["Sec-CH-UA"] = ch_ua
|
||||||
h["Sec-CH-UA-Mobile"] = "?0"
|
h["Sec-CH-UA-Mobile"] = "?0"
|
||||||
h["Sec-CH-UA-Platform"] = (
|
h["Sec-CH-UA-Platform"] = (
|
||||||
'"Windows"' if "Windows" in ua else
|
'"Windows"' if "Windows" in final_ua else
|
||||||
'"macOS"' if "Mac" in ua else
|
'"macOS"' if "Mac" in final_ua else
|
||||||
'"Linux"'
|
'"Linux"'
|
||||||
)
|
)
|
||||||
if extra:
|
|
||||||
h.update(extra)
|
|
||||||
return h
|
return h
|
||||||
|
|
||||||
|
|
||||||
@@ -1630,6 +1632,20 @@ async def _jitter(cfg: "NoxConfig") -> None:
|
|||||||
await asyncio.sleep(random.uniform(lo, hi))
|
await asyncio.sleep(random.uniform(lo, hi))
|
||||||
|
|
||||||
|
|
||||||
|
def _parse_retry_after(value: str, default: float) -> float:
|
||||||
|
"""Parse a Retry-After header value — handles both integer seconds and HTTP-date strings."""
|
||||||
|
try:
|
||||||
|
return float(int(value))
|
||||||
|
except (ValueError, TypeError):
|
||||||
|
pass
|
||||||
|
try:
|
||||||
|
from email.utils import parsedate_to_datetime
|
||||||
|
delta = (parsedate_to_datetime(value) - datetime.now(timezone.utc)).total_seconds()
|
||||||
|
return max(0.0, delta)
|
||||||
|
except Exception:
|
||||||
|
return default
|
||||||
|
|
||||||
|
|
||||||
# ── Async Source Base ──────────────────────────────────────────────────
|
# ── Async Source Base ──────────────────────────────────────────────────
|
||||||
class AsyncSource(ABC):
|
class AsyncSource(ABC):
|
||||||
"""
|
"""
|
||||||
@@ -1701,7 +1717,7 @@ class AsyncSource(ABC):
|
|||||||
async with self._sem:
|
async with self._sem:
|
||||||
async with session.get(url, headers=hdrs, timeout=to, ssl=_SSL_CTX) as resp:
|
async with session.get(url, headers=hdrs, timeout=to, ssl=_SSL_CTX) as resp:
|
||||||
if resp.status == 429:
|
if resp.status == 429:
|
||||||
retry_after = int(resp.headers.get("Retry-After", Cfg.RETRY_DELAY * (attempt + 2)))
|
retry_after = _parse_retry_after(resp.headers.get("Retry-After", ""), Cfg.RETRY_DELAY * (attempt + 2))
|
||||||
_syslog.info("RATE_LIMIT source=%s url=%s retry_after=%ds", self.name, url[:80], retry_after)
|
_syslog.info("RATE_LIMIT source=%s url=%s retry_after=%ds", self.name, url[:80], retry_after)
|
||||||
await asyncio.sleep(min(retry_after, 30))
|
await asyncio.sleep(min(retry_after, 30))
|
||||||
continue
|
continue
|
||||||
@@ -1728,7 +1744,7 @@ class AsyncSource(ABC):
|
|||||||
hdrs["Content-Type"] = "application/json"
|
hdrs["Content-Type"] = "application/json"
|
||||||
async with session.post(url, json=json_data, headers=hdrs, timeout=to, ssl=_SSL_CTX) as resp:
|
async with session.post(url, json=json_data, headers=hdrs, timeout=to, ssl=_SSL_CTX) as resp:
|
||||||
if resp.status == 429:
|
if resp.status == 429:
|
||||||
retry_after = int(resp.headers.get("Retry-After", Cfg.RETRY_DELAY * (attempt + 2)))
|
retry_after = _parse_retry_after(resp.headers.get("Retry-After", ""), Cfg.RETRY_DELAY * (attempt + 2))
|
||||||
_syslog.info("RATE_LIMIT source=%s url=%s retry_after=%ds", self.name, url[:80], retry_after)
|
_syslog.info("RATE_LIMIT source=%s url=%s retry_after=%ds", self.name, url[:80], retry_after)
|
||||||
await asyncio.sleep(min(retry_after, 30))
|
await asyncio.sleep(min(retry_after, 30))
|
||||||
continue
|
continue
|
||||||
@@ -1739,7 +1755,7 @@ class AsyncSource(ABC):
|
|||||||
else:
|
else:
|
||||||
async with session.post(url, data=data or {}, headers=hdrs, timeout=to, ssl=_SSL_CTX) as resp:
|
async with session.post(url, data=data or {}, headers=hdrs, timeout=to, ssl=_SSL_CTX) as resp:
|
||||||
if resp.status == 429:
|
if resp.status == 429:
|
||||||
retry_after = int(resp.headers.get("Retry-After", Cfg.RETRY_DELAY * (attempt + 2)))
|
retry_after = _parse_retry_after(resp.headers.get("Retry-After", ""), Cfg.RETRY_DELAY * (attempt + 2))
|
||||||
_syslog.info("RATE_LIMIT source=%s url=%s retry_after=%ds", self.name, url[:80], retry_after)
|
_syslog.info("RATE_LIMIT source=%s url=%s retry_after=%ds", self.name, url[:80], retry_after)
|
||||||
await asyncio.sleep(min(retry_after, 30))
|
await asyncio.sleep(min(retry_after, 30))
|
||||||
continue
|
continue
|
||||||
@@ -1917,7 +1933,7 @@ class Session:
|
|||||||
data = gzip.decompress(data)
|
data = gzip.decompress(data)
|
||||||
return self._make_response(raw.status, data, dict(raw.headers), raw.url)
|
return self._make_response(raw.status, data, dict(raw.headers), raw.url)
|
||||||
if getattr(r, "status_code", 0) == 429:
|
if getattr(r, "status_code", 0) == 429:
|
||||||
retry_after = int(r.headers.get("Retry-After", Cfg.RETRY_DELAY * (attempt + 2)))
|
retry_after = _parse_retry_after(r.headers.get("Retry-After", ""), Cfg.RETRY_DELAY * (attempt + 2))
|
||||||
time.sleep(min(retry_after, 30))
|
time.sleep(min(retry_after, 30))
|
||||||
continue
|
continue
|
||||||
return r
|
return r
|
||||||
@@ -1941,7 +1957,7 @@ class Session:
|
|||||||
else:
|
else:
|
||||||
r = self._s.post(url, data=data, headers=hdrs, timeout=to)
|
r = self._s.post(url, data=data, headers=hdrs, timeout=to)
|
||||||
if getattr(r, "status_code", 0) == 429:
|
if getattr(r, "status_code", 0) == 429:
|
||||||
retry_after = int(r.headers.get("Retry-After", Cfg.RETRY_DELAY * (attempt + 2)))
|
retry_after = _parse_retry_after(r.headers.get("Retry-After", ""), Cfg.RETRY_DELAY * (attempt + 2))
|
||||||
time.sleep(min(retry_after, 30))
|
time.sleep(min(retry_after, 30))
|
||||||
continue
|
continue
|
||||||
return r
|
return r
|
||||||
@@ -2140,9 +2156,9 @@ class ProxyManager:
|
|||||||
"""
|
"""
|
||||||
Test a proxy by requesting https://api.ipify.org.
|
Test a proxy by requesting https://api.ipify.org.
|
||||||
Returns the observed exit IP on success, None on failure.
|
Returns the observed exit IP on success, None on failure.
|
||||||
F1: SOCKS5 proxies are validated via requests+PySocks, not urllib.
|
SOCKS5 proxies are validated via requests+PySocks, not urllib.
|
||||||
"""
|
"""
|
||||||
# F1: urllib.ProxyHandler does not support SOCKS5 — use requests if available
|
# urllib.ProxyHandler does not support SOCKS5 — use requests if available
|
||||||
if proxy.startswith("socks5") or proxy.startswith("socks4"):
|
if proxy.startswith("socks5") or proxy.startswith("socks4"):
|
||||||
try:
|
try:
|
||||||
import requests as _req # type: ignore
|
import requests as _req # type: ignore
|
||||||
@@ -2420,23 +2436,19 @@ class DorkEngine:
|
|||||||
|
|
||||||
from concurrent.futures import ThreadPoolExecutor, as_completed as _as_completed
|
from concurrent.futures import ThreadPoolExecutor, as_completed as _as_completed
|
||||||
|
|
||||||
def _run_one(dork: str, eng: str) -> List[dict]:
|
def _run_one(dork: str) -> List[dict]:
|
||||||
query = dork.replace("{q}", q)
|
query = dork.replace("{q}", q)
|
||||||
# Per-engine jitter — applied once per (dork, engine) pair, not per dork
|
|
||||||
time.sleep(random.uniform(*Cfg.DORK_DELAY))
|
time.sleep(random.uniform(*Cfg.DORK_DELAY))
|
||||||
hits = self._search(query, eng)
|
hits = self._search(query, "SearXNG")
|
||||||
for h in hits:
|
for h in hits:
|
||||||
h["dork"] = query
|
h["dork"] = query
|
||||||
h["engine"] = eng
|
h["engine"] = "SearXNG"
|
||||||
return hits
|
return hits
|
||||||
|
|
||||||
results = []
|
results = []
|
||||||
pairs = [(dork, eng) for dork in dorks for eng in engines]
|
max_workers = min(len(dorks), 12)
|
||||||
if not pairs:
|
|
||||||
return []
|
|
||||||
max_workers = min(len(pairs), 12) # cap threads to avoid hammering search engines
|
|
||||||
with ThreadPoolExecutor(max_workers=max_workers) as pool:
|
with ThreadPoolExecutor(max_workers=max_workers) as pool:
|
||||||
futures = {pool.submit(_run_one, d, e): (d, e) for d, e in pairs}
|
futures = {pool.submit(_run_one, d): d for d in dorks}
|
||||||
for fut in _as_completed(futures):
|
for fut in _as_completed(futures):
|
||||||
try:
|
try:
|
||||||
results.extend(fut.result())
|
results.extend(fut.result())
|
||||||
@@ -2842,8 +2854,6 @@ class HashEngine:
|
|||||||
return list(set(mutations))
|
return list(set(mutations))
|
||||||
|
|
||||||
def _online(self, h: str) -> Optional[str]:
|
def _online(self, h: str) -> Optional[str]:
|
||||||
# cmd5.org removed — paywalled, returns error for all hashes
|
|
||||||
# hashes.com requires a paid API key (HASHES_COM_API_KEY)
|
|
||||||
try:
|
try:
|
||||||
from sources.helpers.config_handler import ConfigManager # type: ignore
|
from sources.helpers.config_handler import ConfigManager # type: ignore
|
||||||
key = ConfigManager.get_key("HASHES_COM_API_KEY")
|
key = ConfigManager.get_key("HASHES_COM_API_KEY")
|
||||||
@@ -2866,11 +2876,12 @@ class HashEngine:
|
|||||||
def _hashmob(self, h: str) -> Optional[str]:
|
def _hashmob(self, h: str) -> Optional[str]:
|
||||||
try:
|
try:
|
||||||
if not self._session: return None
|
if not self._session: return None
|
||||||
resp = self._session.post("https://hashmob.net/api/v2/search", json_data={"hash":h}, timeout=10)
|
resp = self._session.post("https://hashmob.net/api/v2/search", json_data={"hashes": [h]}, timeout=10)
|
||||||
if resp.ok:
|
if resp.ok:
|
||||||
data = resp.json()
|
data = resp.json()
|
||||||
if data.get("found") and data.get("result"):
|
results = data.get("data") or []
|
||||||
return data["result"]
|
if isinstance(results, list) and results:
|
||||||
|
return results[0].get("plaintext") or results[0].get("result") or None
|
||||||
except Exception: pass
|
except Exception: pass
|
||||||
return None
|
return None
|
||||||
|
|
||||||
@@ -3133,9 +3144,8 @@ class Orchestrator:
|
|||||||
# ── Fail-Safe Proxy check (transport-level, before any connection) ──
|
# ── Fail-Safe Proxy check (transport-level, before any connection) ──
|
||||||
ProxyManager.fail_safe_check(self.config, allow_leak=self.config.allow_leak)
|
ProxyManager.fail_safe_check(self.config, allow_leak=self.config.allow_leak)
|
||||||
|
|
||||||
# B1: recreate SourceOrchestrator on every call so the new semaphore is
|
# SourceOrchestrator is created once and reused across calls. The semaphore
|
||||||
# propagated to all source instances. Plugin JSON files are cached by
|
# is rebound on each invocation so concurrency limits are always respected.
|
||||||
# SourceOrchestrator._load_nox_sources via the module-level mtime guard (L2).
|
|
||||||
if self._source_orchestrator is None:
|
if self._source_orchestrator is None:
|
||||||
self._source_orchestrator = SourceOrchestrator(
|
self._source_orchestrator = SourceOrchestrator(
|
||||||
self._get_semaphore(), self.db, self.config
|
self._get_semaphore(), self.db, self.config
|
||||||
@@ -3173,7 +3183,7 @@ class Orchestrator:
|
|||||||
return records
|
return records
|
||||||
|
|
||||||
connector = aiohttp_mod.TCPConnector(ssl=_SSL_CTX, limit=self.config.concurrency, family=0) # family=0 → AF_UNSPEC (IPv4+IPv6)
|
connector = aiohttp_mod.TCPConnector(ssl=_SSL_CTX, limit=self.config.concurrency, family=0) # family=0 → AF_UNSPEC (IPv4+IPv6)
|
||||||
# B5: SOCKS5 proxies are not supported via trust_env — use ProxyConnector directly.
|
# SOCKS5 proxies require ProxyConnector — aiohttp trust_env does not support SOCKS5.
|
||||||
_socks5_connector = False
|
_socks5_connector = False
|
||||||
if self.config.proxy and self.config.proxy.startswith("socks5"):
|
if self.config.proxy and self.config.proxy.startswith("socks5"):
|
||||||
try:
|
try:
|
||||||
@@ -3182,8 +3192,8 @@ class Orchestrator:
|
|||||||
_socks5_connector = True
|
_socks5_connector = True
|
||||||
except ImportError:
|
except ImportError:
|
||||||
logger.warning("aiohttp_socks not installed — SOCKS5 proxy bypassed. Install: pip install aiohttp-socks")
|
logger.warning("aiohttp_socks not installed — SOCKS5 proxy bypassed. Install: pip install aiohttp-socks")
|
||||||
# B2: set _proxy_env_set flag immediately after os.environ assignment
|
# Set proxy environment variables for HTTP/S proxies so aiohttp trust_env picks them up.
|
||||||
# Use a module-level lock to prevent concurrent scans from racing on env vars.
|
# A module-level lock prevents concurrent scans from racing on the shared env vars.
|
||||||
_proxy_env_set = False
|
_proxy_env_set = False
|
||||||
if self.config.proxy and not _socks5_connector and not os.environ.get("HTTPS_PROXY"):
|
if self.config.proxy and not _socks5_connector and not os.environ.get("HTTPS_PROXY"):
|
||||||
with _PROXY_ENV_LOCK:
|
with _PROXY_ENV_LOCK:
|
||||||
@@ -3951,7 +3961,6 @@ class AdvancedReporter:
|
|||||||
lines += ["","---",f"## Pivot Tree ({len(pivot_log)} nodes)","",
|
lines += ["","---",f"## Pivot Tree ({len(pivot_log)} nodes)","",
|
||||||
"| Depth | Asset | Type | Found In | Parent | Breach | Dorks | Scrape | Children | Cracked |",
|
"| Depth | Asset | Type | Found In | Parent | Breach | Dorks | Scrape | Children | Cracked |",
|
||||||
"|-------|-------|------|----------|--------|--------|-------|--------|----------|---------|"]
|
"|-------|-------|------|----------|--------|--------|-------|--------|----------|---------|"]
|
||||||
# J4: sort by (depth, parent, asset) for readable depth-first narrative
|
|
||||||
for e in sorted(pivot_log, key=lambda x: (x.get("depth", 0), x.get("parent") or "", x.get("asset", ""))):
|
for e in sorted(pivot_log, key=lambda x: (x.get("depth", 0), x.get("parent") or "", x.get("asset", ""))):
|
||||||
cracked_str = _r(", ".join(e.get("cracked", [])[:3]))
|
cracked_str = _r(", ".join(e.get("cracked", [])[:3]))
|
||||||
children = e.get("children", [])
|
children = e.get("children", [])
|
||||||
@@ -4055,7 +4064,6 @@ class Reporter:
|
|||||||
def to_pdf(data: dict, path: str, investigator_id: str = "NOX-AUTO") -> None:
|
def to_pdf(data: dict, path: str, investigator_id: str = "NOX-AUTO") -> None:
|
||||||
path = Reporter._resolve_path(path, "pdf")
|
path = Reporter._resolve_path(path, "pdf")
|
||||||
if _HAS_REPORTING:
|
if _HAS_REPORTING:
|
||||||
# D1: _rep_pdf raises RuntimeError if fpdf2 is missing — let it propagate
|
|
||||||
try:
|
try:
|
||||||
_rep_pdf(data, path, investigator_id=investigator_id)
|
_rep_pdf(data, path, investigator_id=investigator_id)
|
||||||
except RuntimeError as e:
|
except RuntimeError as e:
|
||||||
@@ -4070,7 +4078,6 @@ class Reporter:
|
|||||||
pass
|
pass
|
||||||
# Fallback: weasyprint HTML→PDF
|
# Fallback: weasyprint HTML→PDF
|
||||||
if not weasyprint:
|
if not weasyprint:
|
||||||
# D1: explicit error — no silent return with no output file
|
|
||||||
out("err", "No PDF library found. Install fpdf2: pip install fpdf2")
|
out("err", "No PDF library found. Install fpdf2: pip install fpdf2")
|
||||||
return
|
return
|
||||||
tmp = tempfile.NamedTemporaryFile(suffix=".html", delete=False)
|
tmp = tempfile.NamedTemporaryFile(suffix=".html", delete=False)
|
||||||
@@ -4180,7 +4187,6 @@ class REPL:
|
|||||||
def _dispatch(self, cmd: str, arg: str) -> None:
|
def _dispatch(self, cmd: str, arg: str) -> None:
|
||||||
if cmd in ("quit","exit","q"):
|
if cmd in ("quit","exit","q"):
|
||||||
out("info", "Exiting.")
|
out("info", "Exiting.")
|
||||||
# B3: flush DB background thread before exit
|
|
||||||
try:
|
try:
|
||||||
self.db.close()
|
self.db.close()
|
||||||
except Exception:
|
except Exception:
|
||||||
@@ -4983,8 +4989,6 @@ class REPL:
|
|||||||
elif fmt == "csv":
|
elif fmt == "csv":
|
||||||
resolved = Reporter._resolve_path(path, "csv")
|
resolved = Reporter._resolve_path(path, "csv")
|
||||||
Reporter.to_csv(self._last, resolved)
|
Reporter.to_csv(self._last, resolved)
|
||||||
# G4: derive base from the resolved (absolute) path so companion files
|
|
||||||
# land in REPORT_DIR, not the current working directory
|
|
||||||
self._export_csv_extras(data, resolved)
|
self._export_csv_extras(data, resolved)
|
||||||
elif fmt == "html": Reporter.to_html(data, path)
|
elif fmt == "html": Reporter.to_html(data, path)
|
||||||
elif fmt == "md": Reporter.to_markdown(data, path)
|
elif fmt == "md": Reporter.to_markdown(data, path)
|
||||||
@@ -5095,7 +5099,6 @@ class REPL:
|
|||||||
self.orc.dork_engine.s = self.orc.session
|
self.orc.dork_engine.s = self.orc.session
|
||||||
self.orc.scrape_engine.s = self.orc.session
|
self.orc.scrape_engine.s = self.orc.session
|
||||||
self.orc.hash_engine._session = self.orc.session
|
self.orc.hash_engine._session = self.orc.session
|
||||||
# G2: also rebuild dorking_engine so it picks up the new proxy/Tor config
|
|
||||||
self.orc.dorking_engine = DorkingEngine(self.config.concurrency, self.orc.db, self.config)
|
self.orc.dorking_engine = DorkingEngine(self.config.concurrency, self.orc.db, self.config)
|
||||||
|
|
||||||
# ── Investigation Dashboard ────────────────────────────────────────────
|
# ── Investigation Dashboard ────────────────────────────────────────────
|
||||||
@@ -5560,12 +5563,11 @@ class ConfigManager:
|
|||||||
|
|
||||||
_cache: Dict[str, str] = {}
|
_cache: Dict[str, str] = {}
|
||||||
_INI_PATHS = [HOME_NOX / "config.ini", Path("/etc/nox/config.ini")]
|
_INI_PATHS = [HOME_NOX / "config.ini", Path("/etc/nox/config.ini")]
|
||||||
# B4: track apikeys.json mtime to detect external edits
|
|
||||||
_store_mtime: float = 0.0
|
_store_mtime: float = 0.0
|
||||||
|
|
||||||
@classmethod
|
@classmethod
|
||||||
def _invalidate_if_changed(cls) -> None:
|
def _invalidate_if_changed(cls) -> None:
|
||||||
"""B4: clear cache if apikeys.json was modified externally."""
|
"""Clear the key cache if apikeys.json was modified externally."""
|
||||||
if not _HAS_CONFIG_HANDLER or _ExtConfigManager is None:
|
if not _HAS_CONFIG_HANDLER or _ExtConfigManager is None:
|
||||||
return
|
return
|
||||||
try:
|
try:
|
||||||
@@ -5813,7 +5815,7 @@ class DeHashEngine:
|
|||||||
return (h, cached)
|
return (h, cached)
|
||||||
try:
|
try:
|
||||||
auth = base64.b64encode(self._key.encode()).decode() if ":" in self._key else self._key
|
auth = base64.b64encode(self._key.encode()).decode() if ":" in self._key else self._key
|
||||||
url = f"https://api.dehashed.com/search?query=hashed_password:{h}&size=1"
|
url = f"https://api.dehashed.com/v2/search?query=hashed_password:{h}&size=1"
|
||||||
hdrs = {"Accept": "application/json", "Authorization": f"Basic {auth}"}
|
hdrs = {"Accept": "application/json", "Authorization": f"Basic {auth}"}
|
||||||
async with sem:
|
async with sem:
|
||||||
to = aiohttp_mod.ClientTimeout(total=self._config.timeout) if aiohttp_mod else None
|
to = aiohttp_mod.ClientTimeout(total=self._config.timeout) if aiohttp_mod else None
|
||||||
@@ -6105,7 +6107,7 @@ class Vault:
|
|||||||
# Synchronous fallback lookup via requests/urllib
|
# Synchronous fallback lookup via requests/urllib
|
||||||
try:
|
try:
|
||||||
auth = base64.b64encode(key.encode()).decode() if ":" in key else key
|
auth = base64.b64encode(key.encode()).decode() if ":" in key else key
|
||||||
url = (f"https://api.dehashed.com/search"
|
url = (f"https://api.dehashed.com/v2/search"
|
||||||
f"?query=hashed_password:{r.password_hash}&size=1")
|
f"?query=hashed_password:{r.password_hash}&size=1")
|
||||||
hdrs = {"Accept": "application/json",
|
hdrs = {"Accept": "application/json",
|
||||||
"Authorization": f"Basic {auth}",
|
"Authorization": f"Basic {auth}",
|
||||||
@@ -6354,11 +6356,28 @@ class NoxSourceProvider(FileSystemProvider):
|
|||||||
|
|
||||||
async def _fetch(self, session, query: str) -> List[Record]:
|
async def _fetch(self, session, query: str) -> List[Record]:
|
||||||
d = self._def
|
d = self._def
|
||||||
|
# Apply optional query transform before URL substitution.
|
||||||
|
# Currently supported: "md5_lower" — MD5-hex of the lowercased, stripped query.
|
||||||
|
transform = d.get("query_transform", "")
|
||||||
|
if transform == "md5_lower":
|
||||||
|
import hashlib as _hl
|
||||||
|
try:
|
||||||
|
effective_query = _hl.md5(query.lower().strip().encode(),
|
||||||
|
usedforsecurity=False).hexdigest()
|
||||||
|
except TypeError:
|
||||||
|
effective_query = _hl.md5(query.lower().strip().encode()).hexdigest()
|
||||||
|
elif transform == "fofa_domain":
|
||||||
|
import base64 as _b64
|
||||||
|
effective_query = _b64.b64encode(
|
||||||
|
f'domain="{query.lower().strip()}"'.encode()
|
||||||
|
).decode()
|
||||||
|
else:
|
||||||
|
effective_query = query
|
||||||
# Headers are already resolved in _load_nox_sources; just substitute {query}
|
# Headers are already resolved in _load_nox_sources; just substitute {query}
|
||||||
hdrs = {k: v.replace("{query}", urllib.parse.quote(query, safe=""))
|
hdrs = {k: v.replace("{query}", urllib.parse.quote(effective_query, safe=""))
|
||||||
for k, v in d.get("headers", {}).items()}
|
for k, v in d.get("headers", {}).items()}
|
||||||
url = (d["api_url"]
|
url = (d["api_url"]
|
||||||
.replace("{query}", urllib.parse.quote(query, safe=""))
|
.replace("{query}", urllib.parse.quote(effective_query, safe=""))
|
||||||
.replace("{api_key}", self._api_key or ""))
|
.replace("{api_key}", self._api_key or ""))
|
||||||
# Also substitute any remaining {KEY_NAME} placeholders in URL
|
# Also substitute any remaining {KEY_NAME} placeholders in URL
|
||||||
for slot_name, slot_val in self._slot_keys.items():
|
for slot_name, slot_val in self._slot_keys.items():
|
||||||
@@ -6369,7 +6388,7 @@ class NoxSourceProvider(FileSystemProvider):
|
|||||||
def _sub(obj):
|
def _sub(obj):
|
||||||
"""Recursively substitute {query} in payload (handles nested dicts/lists)."""
|
"""Recursively substitute {query} in payload (handles nested dicts/lists)."""
|
||||||
if isinstance(obj, str):
|
if isinstance(obj, str):
|
||||||
return obj.replace("{query}", query).replace("{target}", query)
|
return obj.replace("{query}", effective_query).replace("{target}", effective_query)
|
||||||
if isinstance(obj, dict):
|
if isinstance(obj, dict):
|
||||||
return {k: _sub(v) for k, v in obj.items()}
|
return {k: _sub(v) for k, v in obj.items()}
|
||||||
if isinstance(obj, list):
|
if isinstance(obj, list):
|
||||||
@@ -6385,10 +6404,22 @@ class NoxSourceProvider(FileSystemProvider):
|
|||||||
else:
|
else:
|
||||||
status, text, _ = await self._get(session, url, headers=hdrs)
|
status, text, _ = await self._get(session, url, headers=hdrs)
|
||||||
|
|
||||||
|
# If the primary endpoint fails, try backup_endpoints in order.
|
||||||
if status not in range(200, 300) or not text:
|
if status not in range(200, 300) or not text:
|
||||||
return []
|
for backup in (d.get("backup_endpoints") or []):
|
||||||
|
backup_url = (backup
|
||||||
# Two-phase poll: if poll_endpoint is defined, treat the first response
|
.replace("{query}", urllib.parse.quote(query, safe=""))
|
||||||
|
.replace("{target}", urllib.parse.quote(query, safe="")))
|
||||||
|
for slot_name, slot_val in self._slot_keys.items():
|
||||||
|
backup_url = backup_url.replace(f"{{{slot_name}}}", slot_val or "")
|
||||||
|
if method == "POST":
|
||||||
|
status, text, _ = await self._post(session, backup_url,
|
||||||
|
json_data=payload or None,
|
||||||
|
headers=hdrs)
|
||||||
|
else:
|
||||||
|
status, text, _ = await self._get(session, backup_url, headers=hdrs)
|
||||||
|
if status in range(200, 300) and text:
|
||||||
|
break
|
||||||
# as a job submission, extract the job ID via poll_id_field, then poll
|
# as a job submission, extract the job ID via poll_id_field, then poll
|
||||||
# poll_endpoint?<poll_id_param>=<id> until results arrive.
|
# poll_endpoint?<poll_id_param>=<id> until results arrive.
|
||||||
poll_endpoint = d.get("poll_endpoint", "")
|
poll_endpoint = d.get("poll_endpoint", "")
|
||||||
@@ -6545,6 +6576,8 @@ class SourceOrchestrator:
|
|||||||
"poll_id_field": raw.get("poll_id_field", "id"),
|
"poll_id_field": raw.get("poll_id_field", "id"),
|
||||||
"poll_id_param": raw.get("poll_id_param", "id"),
|
"poll_id_param": raw.get("poll_id_param", "id"),
|
||||||
"poll_json_root": raw.get("poll_json_root", ""),
|
"poll_json_root": raw.get("poll_json_root", ""),
|
||||||
|
"backup_endpoints": raw.get("backup_endpoints", []),
|
||||||
|
"query_transform": raw.get("query_transform", ""),
|
||||||
}
|
}
|
||||||
inst = NoxSourceProvider(self._sem, self._db, self._config, defn)
|
inst = NoxSourceProvider(self._sem, self._db, self._config, defn)
|
||||||
inst._bypass_required = raw.get("bypass_required") or []
|
inst._bypass_required = raw.get("bypass_required") or []
|
||||||
@@ -7093,12 +7126,10 @@ def main() -> None:
|
|||||||
config.proxy = f"socks5h://127.0.0.1:{config.tor_socks}"
|
config.proxy = f"socks5h://127.0.0.1:{config.tor_socks}"
|
||||||
if args.proxy:
|
if args.proxy:
|
||||||
config.proxy = args.proxy
|
config.proxy = args.proxy
|
||||||
# K2: --guardian-off is an alias for --allow-leak
|
|
||||||
config.allow_leak = args.allow_leak or getattr(args, "guardian_off", False)
|
config.allow_leak = args.allow_leak or getattr(args, "guardian_off", False)
|
||||||
config.no_online_crack = getattr(args, "no_online_crack", False)
|
config.no_online_crack = getattr(args, "no_online_crack", False)
|
||||||
config.max_threads = config.concurrency = args.threads
|
config.max_threads = config.concurrency = args.threads
|
||||||
config.timeout = args.timeout
|
config.timeout = args.timeout
|
||||||
# A9/I3: store no_pivot and depth in config so REPL and AvalancheScanner can read them
|
|
||||||
config.no_pivot = args.no_pivot
|
config.no_pivot = args.no_pivot
|
||||||
if getattr(args, "depth", None) is not None:
|
if getattr(args, "depth", None) is not None:
|
||||||
config.pivot_depth = args.depth
|
config.pivot_depth = args.depth
|
||||||
@@ -7122,7 +7153,6 @@ def _main_run(args, config: NoxConfig, db: NoxDB) -> None:
|
|||||||
repl._sources()
|
repl._sources()
|
||||||
return
|
return
|
||||||
|
|
||||||
# B6: --reset-sources forces a full resync from package
|
|
||||||
if getattr(args, "reset_sources", False):
|
if getattr(args, "reset_sources", False):
|
||||||
import shutil as _shutil
|
import shutil as _shutil
|
||||||
candidate = _PKG_ROOT / "sources"
|
candidate = _PKG_ROOT / "sources"
|
||||||
|
|||||||
+2
-2
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
|
|||||||
|
|
||||||
[project]
|
[project]
|
||||||
name = "nox-cli"
|
name = "nox-cli"
|
||||||
version = "1.0.1"
|
version = "1.0.2"
|
||||||
description = "Advanced Asynchronous Cyber Threat Intelligence Framework"
|
description = "Advanced Asynchronous Cyber Threat Intelligence Framework"
|
||||||
readme = { file = "README.md", content-type = "text/markdown" }
|
readme = { file = "README.md", content-type = "text/markdown" }
|
||||||
license = { text = "Apache-2.0" }
|
license = { text = "Apache-2.0" }
|
||||||
@@ -17,7 +17,7 @@ dependencies = [
|
|||||||
"httpx[http2]>=0.27.0",
|
"httpx[http2]>=0.27.0",
|
||||||
"brotli>=1.1.0",
|
"brotli>=1.1.0",
|
||||||
"zstandard>=0.23.0",
|
"zstandard>=0.23.0",
|
||||||
"requests>=2.31.0",
|
"requests>=2.32.3",
|
||||||
"certifi>=2024.2.2",
|
"certifi>=2024.2.2",
|
||||||
"cloudscraper>=1.2.71",
|
"cloudscraper>=1.2.71",
|
||||||
"beautifulsoup4>=4.12.3",
|
"beautifulsoup4>=4.12.3",
|
||||||
|
|||||||
@@ -1,28 +0,0 @@
|
|||||||
{
|
|
||||||
"name": "bgpview_ip",
|
|
||||||
"category": "network",
|
|
||||||
"endpoint": "https://api.bgpview.io/ip/{target}",
|
|
||||||
"method": "GET",
|
|
||||||
"requires_auth": false,
|
|
||||||
"selectors": {
|
|
||||||
"prefixes": "$.data.prefixes[*].prefix"
|
|
||||||
},
|
|
||||||
"rate_limit": 1.0,
|
|
||||||
"headers": {},
|
|
||||||
"api_key_slots": [],
|
|
||||||
"input_type": "ip",
|
|
||||||
"output_type": [
|
|
||||||
"ip"
|
|
||||||
],
|
|
||||||
"normalization_map": {},
|
|
||||||
"tags": [
|
|
||||||
"passive",
|
|
||||||
"infrastructure"
|
|
||||||
],
|
|
||||||
"health_check_url": "https://api.bgpview.io",
|
|
||||||
"expected_status": 200,
|
|
||||||
"reliability_score": 2,
|
|
||||||
"is_volatile": true,
|
|
||||||
"backup_endpoints": [],
|
|
||||||
"confidence": 0.55
|
|
||||||
}
|
|
||||||
@@ -1,7 +1,7 @@
|
|||||||
{
|
{
|
||||||
"name": "duckduckgo_api",
|
"name": "duckduckgo_api",
|
||||||
"category": "search",
|
"category": "search",
|
||||||
"endpoint": "https://searx.tiekoetter.com/search?q={target}&format=json&categories=general",
|
"endpoint": "https://search.sapti.me/search?q={target}&format=json&categories=general",
|
||||||
"method": "GET",
|
"method": "GET",
|
||||||
"requires_auth": false,
|
"requires_auth": false,
|
||||||
"selectors": {
|
"selectors": {
|
||||||
@@ -22,10 +22,16 @@
|
|||||||
"passive",
|
"passive",
|
||||||
"fast"
|
"fast"
|
||||||
],
|
],
|
||||||
"health_check_url": "https://searx.tiekoetter.com",
|
"health_check_url": "https://search.sapti.me",
|
||||||
"expected_status": 200,
|
"expected_status": 200,
|
||||||
"reliability_score": 3,
|
"reliability_score": 3,
|
||||||
"is_volatile": true,
|
"is_volatile": true,
|
||||||
"backup_endpoints": [],
|
"backup_endpoints": [
|
||||||
|
"https://searx.tiekoetter.com/search?q={target}&format=json&categories=general",
|
||||||
|
"https://searx.perennialte.ch/search?q={target}&format=json&categories=general",
|
||||||
|
"https://search.mdosch.de/search?q={target}&format=json&categories=general",
|
||||||
|
"https://paulgo.io/search?q={target}&format=json&categories=general",
|
||||||
|
"https://priv.au/search?q={target}&format=json&categories=general"
|
||||||
|
],
|
||||||
"confidence": 0.7
|
"confidence": 0.7
|
||||||
}
|
}
|
||||||
@@ -27,5 +27,6 @@
|
|||||||
"expected_status": 200,
|
"expected_status": 200,
|
||||||
"reliability_score": 4,
|
"reliability_score": 4,
|
||||||
"backup_endpoints": [],
|
"backup_endpoints": [],
|
||||||
|
"query_transform": "fofa_domain",
|
||||||
"confidence": 0.85
|
"confidence": 0.85
|
||||||
}
|
}
|
||||||
@@ -22,5 +22,6 @@
|
|||||||
"expected_status": 200,
|
"expected_status": 200,
|
||||||
"reliability_score": 4,
|
"reliability_score": 4,
|
||||||
"backup_endpoints": [],
|
"backup_endpoints": [],
|
||||||
|
"query_transform": "md5_lower",
|
||||||
"confidence": 0.85
|
"confidence": 0.85
|
||||||
}
|
}
|
||||||
@@ -29,13 +29,13 @@ SERVICE_REGISTRY: Dict[str, Dict] = {
|
|||||||
"alienvault_otx_malware": {"display": "AlienVault OTX (Malware)", "public": True},
|
"alienvault_otx_malware": {"display": "AlienVault OTX (Malware)", "public": True},
|
||||||
"alienvault_otx_user": {"display": "AlienVault OTX (User)", "public": True},
|
"alienvault_otx_user": {"display": "AlienVault OTX (User)", "public": True},
|
||||||
"anubis_subdomains": {"display": "Anubis Subdomains", "public": True},
|
"anubis_subdomains": {"display": "Anubis Subdomains", "public": True},
|
||||||
"bgpview_ip": {"display": "BGPView IP", "public": True},
|
"ripestat_ip": {"display": "RIPE Stat IP", "public": True},
|
||||||
"checkleaked": {"display": "CheckLeaked", "public": True},
|
"xposedornot": {"display": "XposedOrNot", "public": True},
|
||||||
"crt_sh": {"display": "crt.sh", "public": True},
|
"crt_sh": {"display": "crt.sh", "public": True},
|
||||||
"cve_search": {"display": "CVE Search", "public": True},
|
"cve_search": {"display": "CVE Search", "public": True},
|
||||||
"cxsecurity": {"display": "CXSecurity", "public": True},
|
"cxsecurity": {"display": "CXSecurity", "public": True},
|
||||||
"duckduckgo_api": {"display": "Google / DDG Dorks", "public": True},
|
"duckduckgo_api": {"display": "Google / DDG Dorks", "public": True},
|
||||||
"emailrep_io": {"display": "EmailRep.io", "public": True},
|
"emailrep_io": {"display": "EmailRep.io", "public": False},
|
||||||
"github_users": {"display": "GitHub Users", "public": True},
|
"github_users": {"display": "GitHub Users", "public": True},
|
||||||
"gitlab_search": {"display": "GitLab Search", "public": True},
|
"gitlab_search": {"display": "GitLab Search", "public": True},
|
||||||
"gravatar": {"display": "Gravatar", "public": True},
|
"gravatar": {"display": "Gravatar", "public": True},
|
||||||
@@ -44,7 +44,10 @@ SERVICE_REGISTRY: Dict[str, Dict] = {
|
|||||||
"hackertarget_hostsearch": {"display": "HackerTarget Host Search", "public": True},
|
"hackertarget_hostsearch": {"display": "HackerTarget Host Search", "public": True},
|
||||||
"hackertarget_reverseip": {"display": "HackerTarget Reverse IP", "public": True},
|
"hackertarget_reverseip": {"display": "HackerTarget Reverse IP", "public": True},
|
||||||
"hackertarget_whois": {"display": "WHOIS (HackerTarget)", "public": True},
|
"hackertarget_whois": {"display": "WHOIS (HackerTarget)", "public": True},
|
||||||
"hudsonrock_osint": {"display": "HudsonRock OSINT", "public": True},
|
"ipapi_is": {"display": "ipapi.is", "public": True},
|
||||||
|
"circl_hashlookup": {"display": "CIRCL Hash Lookup", "public": True},
|
||||||
|
"proxynova_comb": {"display": "ProxyNova COMB", "public": True},
|
||||||
|
"shodan_internetdb": {"display": "Shodan InternetDB", "public": True},
|
||||||
"ipapi_co": {"display": "ipapi.co", "public": True},
|
"ipapi_co": {"display": "ipapi.co", "public": True},
|
||||||
"ipinfo_io": {"display": "IPInfo.io", "public": True},
|
"ipinfo_io": {"display": "IPInfo.io", "public": True},
|
||||||
"ipvigilante": {"display": "IPVigilante", "public": True},
|
"ipvigilante": {"display": "IPVigilante", "public": True},
|
||||||
@@ -59,14 +62,10 @@ SERVICE_REGISTRY: Dict[str, Dict] = {
|
|||||||
"reddit_user": {"display": "Reddit User", "public": True},
|
"reddit_user": {"display": "Reddit User", "public": True},
|
||||||
"robtex_ip": {"display": "Robtex IP", "public": True},
|
"robtex_ip": {"display": "Robtex IP", "public": True},
|
||||||
"scamwatcher": {"display": "ScamWatcher", "public": True},
|
"scamwatcher": {"display": "ScamWatcher", "public": True},
|
||||||
"social_scan": {"display": "Social Scan", "public": True},
|
|
||||||
"sublist3r_api": {"display": "Sublist3r API", "public": True},
|
"sublist3r_api": {"display": "Sublist3r API", "public": True},
|
||||||
"threatcrowd_domain": {"display": "ThreatCrowd (Domain)", "public": True},
|
|
||||||
"threatcrowd_email": {"display": "ThreatCrowd (Email)", "public": True},
|
|
||||||
"threatminer_domain": {"display": "ThreatMiner (Domain)", "public": True},
|
"threatminer_domain": {"display": "ThreatMiner (Domain)", "public": True},
|
||||||
"threatminer_ip": {"display": "ThreatMiner (IP)", "public": True},
|
"threatminer_ip": {"display": "ThreatMiner (IP)", "public": True},
|
||||||
"urlscan_search": {"display": "URLScan.io", "public": True},
|
"urlscan_search": {"display": "URLScan.io", "public": True},
|
||||||
"vigilante_pw": {"display": "Vigilante.pw", "public": True},
|
|
||||||
"wayback_machine": {"display": "Wayback Machine", "public": True},
|
"wayback_machine": {"display": "Wayback Machine", "public": True},
|
||||||
# ── Private / key-required ────────────────────────────────────────
|
# ── Private / key-required ────────────────────────────────────────
|
||||||
"ABSTRACT_API_KEY": {"display": "Abstract Email Validation", "public": False},
|
"ABSTRACT_API_KEY": {"display": "Abstract Email Validation", "public": False},
|
||||||
@@ -78,7 +77,6 @@ SERVICE_REGISTRY: Dict[str, Dict] = {
|
|||||||
"BING_API_KEY": {"display": "Bing Search API", "public": False},
|
"BING_API_KEY": {"display": "Bing Search API", "public": False},
|
||||||
"CENSYS_AUTH_BASE64": {"display": "Censys", "public": False},
|
"CENSYS_AUTH_BASE64": {"display": "Censys", "public": False},
|
||||||
"CIRCL_AUTH_BASE64": {"display": "CIRCL.lu PDNS", "public": False},
|
"CIRCL_AUTH_BASE64": {"display": "CIRCL.lu PDNS", "public": False},
|
||||||
"CIT0DAY_API_KEY": {"display": "Cit0day", "public": False},
|
|
||||||
"SEON_API_KEY": {"display": "SEON Email Intelligence", "public": False},
|
"SEON_API_KEY": {"display": "SEON Email Intelligence", "public": False},
|
||||||
"CRIMINALIP_API_KEY": {"display": "CriminalIP", "public": False},
|
"CRIMINALIP_API_KEY": {"display": "CriminalIP", "public": False},
|
||||||
"DEHASHED_AUTH_BASE64": {"display": "Dehashed", "public": False},
|
"DEHASHED_AUTH_BASE64": {"display": "Dehashed", "public": False},
|
||||||
@@ -108,7 +106,6 @@ SERVICE_REGISTRY: Dict[str, Dict] = {
|
|||||||
"JOE_API_KEY": {"display": "Joe Sandbox", "public": False},
|
"JOE_API_KEY": {"display": "Joe Sandbox", "public": False},
|
||||||
"LEAKCHECK_API_KEY": {"display": "LeakCheck", "public": False},
|
"LEAKCHECK_API_KEY": {"display": "LeakCheck", "public": False},
|
||||||
"LEAKIX_API_KEY": {"display": "LeakIX", "public": False},
|
"LEAKIX_API_KEY": {"display": "LeakIX", "public": False},
|
||||||
"LEAKSTATS_API_KEY": {"display": "LeakStats.pw", "public": False},
|
|
||||||
"MAILBOX_API_KEY": {"display": "Mailboxlayer", "public": False},
|
"MAILBOX_API_KEY": {"display": "Mailboxlayer", "public": False},
|
||||||
"MALSHARE_API_KEY": {"display": "MalShare", "public": False},
|
"MALSHARE_API_KEY": {"display": "MalShare", "public": False},
|
||||||
"METADEFENDER_API_KEY": {"display": "MetaDefender", "public": False},
|
"METADEFENDER_API_KEY": {"display": "MetaDefender", "public": False},
|
||||||
@@ -124,7 +121,6 @@ SERVICE_REGISTRY: Dict[str, Dict] = {
|
|||||||
"SNUSBASE_API_KEY": {"display": "Snusbase", "public": False},
|
"SNUSBASE_API_KEY": {"display": "Snusbase", "public": False},
|
||||||
"SPYCLOUD_API_KEY": {"display": "SpyCloud", "public": False},
|
"SPYCLOUD_API_KEY": {"display": "SpyCloud", "public": False},
|
||||||
"SPYONWEB_API_KEY": {"display": "SpyOnWeb", "public": False},
|
"SPYONWEB_API_KEY": {"display": "SpyOnWeb", "public": False},
|
||||||
"SPYSE_API_KEY": {"display": "Spyse", "public": False},
|
|
||||||
"TC_API_KEY": {"display": "ThreatConnect", "public": False},
|
"TC_API_KEY": {"display": "ThreatConnect", "public": False},
|
||||||
"FLARE_API_KEY": {"display": "Flare LeaksDB", "public": False},
|
"FLARE_API_KEY": {"display": "Flare LeaksDB", "public": False},
|
||||||
"TP_API_KEY": {"display": "ThreatPortal", "public": False},
|
"TP_API_KEY": {"display": "ThreatPortal", "public": False},
|
||||||
@@ -138,7 +134,6 @@ SERVICE_REGISTRY: Dict[str, Dict] = {
|
|||||||
"WHOXY_API_KEY": {"display": "Whoxy WHOIS", "public": False},
|
"WHOXY_API_KEY": {"display": "Whoxy WHOIS", "public": False},
|
||||||
"ZEROBOUNCE_API_KEY": {"display": "ZeroBounce", "public": False},
|
"ZEROBOUNCE_API_KEY": {"display": "ZeroBounce", "public": False},
|
||||||
"ZOOMEYE_API_KEY": {"display": "ZoomEye", "public": False},
|
"ZOOMEYE_API_KEY": {"display": "ZoomEye", "public": False},
|
||||||
# ── Added in v1.0.1 ───────────────────────────────────────────────
|
|
||||||
"EMAILREP_API_KEY": {"display": "EmailRep.io", "public": False},
|
"EMAILREP_API_KEY": {"display": "EmailRep.io", "public": False},
|
||||||
"HASHES_COM_API_KEY": {"display": "Hashes.com (crack API)", "public": False},
|
"HASHES_COM_API_KEY": {"display": "Hashes.com (crack API)", "public": False},
|
||||||
"THREATFOX_API_KEY": {"display": "ThreatFox (abuse.ch)", "public": False},
|
"THREATFOX_API_KEY": {"display": "ThreatFox (abuse.ch)", "public": False},
|
||||||
@@ -146,8 +141,8 @@ SERVICE_REGISTRY: Dict[str, Dict] = {
|
|||||||
"MALWAREBAZAAR_API_KEY": {"display": "MalwareBazaar (abuse.ch)", "public": False},
|
"MALWAREBAZAAR_API_KEY": {"display": "MalwareBazaar (abuse.ch)", "public": False},
|
||||||
"FULLHUNT_API_KEY": {"display": "FullHunt (attack surface)", "public": False},
|
"FULLHUNT_API_KEY": {"display": "FullHunt (attack surface)", "public": False},
|
||||||
"NETLAS_API_KEY": {"display": "Netlas.io (internet scanner)", "public": False},
|
"NETLAS_API_KEY": {"display": "Netlas.io (internet scanner)", "public": False},
|
||||||
# ── Added in v1.0.2 ───────────────────────────────────────────────
|
|
||||||
"LEAK_LOOKUP_API_KEY": {"display": "Leak-Lookup", "public": False},
|
"LEAK_LOOKUP_API_KEY": {"display": "Leak-Lookup", "public": False},
|
||||||
|
"MISP_URL": {"display": "MISP Instance URL", "public": False},
|
||||||
}
|
}
|
||||||
|
|
||||||
_PRIVATE_KEYS = {k: v for k, v in SERVICE_REGISTRY.items() if not v["public"]}
|
_PRIVATE_KEYS = {k: v for k, v in SERVICE_REGISTRY.items() if not v["public"]}
|
||||||
|
|||||||
@@ -12,9 +12,9 @@ import logging
|
|||||||
import re
|
import re
|
||||||
from typing import List, Optional, Tuple
|
from typing import List, Optional, Tuple
|
||||||
|
|
||||||
# C2: MD5 and NTLM share the same 32-char hex pattern.
|
# MD5 and NTLM share the same 32-char hex pattern. MD5 is listed first as it
|
||||||
# We list md5 first (most common in breach data) but also accept ntlm
|
# is the most common type in breach data. async_crack queries both md5 and
|
||||||
# so callers can query NTLM-specific APIs when needed.
|
# ntlm-specific APIs for any 32-char hash.
|
||||||
_PATTERNS: List[Tuple[str, re.Pattern]] = [
|
_PATTERNS: List[Tuple[str, re.Pattern]] = [
|
||||||
("bcrypt", re.compile(r"^\$2[aby]?\$\d{2}\$.{53}$")),
|
("bcrypt", re.compile(r"^\$2[aby]?\$\d{2}\$.{53}$")),
|
||||||
("sha256", re.compile(r"^[a-f0-9]{64}$", re.I)),
|
("sha256", re.compile(r"^[a-f0-9]{64}$", re.I)),
|
||||||
@@ -130,9 +130,23 @@ def _local_crack_sync_blocking(hash_value: str, hash_type: str) -> Optional[str]
|
|||||||
if not wordlist.exists():
|
if not wordlist.exists():
|
||||||
return None
|
return None
|
||||||
h = hash_value.strip().lower()
|
h = hash_value.strip().lower()
|
||||||
|
# usedforsecurity=False is required on FIPS-enabled systems (Python 3.9+).
|
||||||
|
# On Python 3.8 the kwarg does not exist, so we fall back gracefully.
|
||||||
|
def _md5(w):
|
||||||
|
try:
|
||||||
|
return _hl.md5(w, usedforsecurity=False).hexdigest()
|
||||||
|
except TypeError:
|
||||||
|
return _hl.md5(w).hexdigest()
|
||||||
|
|
||||||
|
def _sha1(w):
|
||||||
|
try:
|
||||||
|
return _hl.sha1(w, usedforsecurity=False).hexdigest()
|
||||||
|
except TypeError:
|
||||||
|
return _hl.sha1(w).hexdigest()
|
||||||
|
|
||||||
_hashers = {
|
_hashers = {
|
||||||
"md5": lambda w: _hl.md5(w).hexdigest(),
|
"md5": _md5,
|
||||||
"sha1": lambda w: _hl.sha1(w).hexdigest(),
|
"sha1": _sha1,
|
||||||
"sha256": lambda w: _hl.sha256(w).hexdigest(),
|
"sha256": lambda w: _hl.sha256(w).hexdigest(),
|
||||||
}
|
}
|
||||||
hasher = _hashers.get(hash_type)
|
hasher = _hashers.get(hash_type)
|
||||||
|
|||||||
@@ -48,11 +48,11 @@ def _raw(v: Any, maxlen: int = 200) -> str:
|
|||||||
|
|
||||||
|
|
||||||
def _pdf_safe(s: str, maxlen: int = 180) -> str:
|
def _pdf_safe(s: str, maxlen: int = 180) -> str:
|
||||||
# D4: sanitize for fpdf2 core fonts (latin-1 subset).
|
# Sanitise for fpdf2 core fonts (latin-1 subset).
|
||||||
# NFKD normalization decomposes accented chars (é→e + combining accent)
|
# NFKD normalization decomposes accented chars (é→e + combining accent)
|
||||||
# so common accented Latin characters survive as their base letter.
|
# so common accented Latin characters survive as their base letter.
|
||||||
# Truly non-latin-1 chars (Cyrillic, CJK, etc.) become '?' — intentional:
|
# Truly non-latin-1 chars (Cyrillic, CJK, etc.) become '?' — fpdf2 core
|
||||||
# fpdf2 core fonts cannot render them and would raise UnicodeEncodeError.
|
# fonts cannot render them and would raise UnicodeEncodeError.
|
||||||
s = _raw(s, maxlen)
|
s = _raw(s, maxlen)
|
||||||
try:
|
try:
|
||||||
import unicodedata
|
import unicodedata
|
||||||
@@ -114,7 +114,7 @@ def render_pivot_chain(data: dict) -> List[str]:
|
|||||||
chain = data.get("pivot_chain") or []
|
chain = data.get("pivot_chain") or []
|
||||||
target = _raw(data.get("target", "?"))
|
target = _raw(data.get("target", "?"))
|
||||||
|
|
||||||
# D2: if pivot_log is available, build chain from it (accurate tree)
|
# Build chain from pivot_log when available — it carries the full tree with depth and provenance.
|
||||||
pivot_log = data.get("pivot_log") or []
|
pivot_log = data.get("pivot_log") or []
|
||||||
if pivot_log:
|
if pivot_log:
|
||||||
lines: List[str] = []
|
lines: List[str] = []
|
||||||
@@ -195,14 +195,12 @@ def to_json(data: dict, path: str) -> None:
|
|||||||
dork_results = data.get("dork_results", []) or []
|
dork_results = data.get("dork_results", []) or []
|
||||||
scrape_results = data.get("scrape_results", {}) or {}
|
scrape_results = data.get("scrape_results", {}) or {}
|
||||||
|
|
||||||
# D3: apply consistent cap (1000) — same as HTML
|
|
||||||
_RECORD_CAP = 1000
|
_RECORD_CAP = 1000
|
||||||
|
|
||||||
out_data = {
|
out_data = {
|
||||||
"framework": f"NOX v{_NOX_VERSION}",
|
"framework": f"NOX v{_NOX_VERSION}",
|
||||||
"generated": datetime.now().isoformat(),
|
"generated": datetime.now().isoformat(),
|
||||||
"target": data.get("target", ""),
|
"target": data.get("target", ""),
|
||||||
# J3: self-describing metadata block
|
|
||||||
"_meta": {
|
"_meta": {
|
||||||
"scan_id": hashlib.sha256(
|
"scan_id": hashlib.sha256(
|
||||||
f"{data.get('target','')}{datetime.now().isoformat()}".encode()
|
f"{data.get('target','')}{datetime.now().isoformat()}".encode()
|
||||||
@@ -387,7 +385,6 @@ def to_html(data: dict, path: str) -> None:
|
|||||||
# ── PDF report (fpdf2) ────────────────────────────────────────────────
|
# ── PDF report (fpdf2) ────────────────────────────────────────────────
|
||||||
|
|
||||||
def to_pdf(data: dict, path: str, investigator_id: str = "NOX-AUTO") -> None:
|
def to_pdf(data: dict, path: str, investigator_id: str = "NOX-AUTO") -> None:
|
||||||
# D1: raise a clear error with install hint if fpdf2 is absent — never silently return.
|
|
||||||
try:
|
try:
|
||||||
from fpdf import FPDF # type: ignore
|
from fpdf import FPDF # type: ignore
|
||||||
except ImportError:
|
except ImportError:
|
||||||
|
|||||||
@@ -31,7 +31,6 @@ _PIVOT_TYPES = {"email", "username", "phone", "name", "ip", "domain"}
|
|||||||
|
|
||||||
|
|
||||||
def _cfg_depth(orc=None) -> int:
|
def _cfg_depth(orc=None) -> int:
|
||||||
# A7/A10: read from orchestrator config if available
|
|
||||||
if orc is not None:
|
if orc is not None:
|
||||||
cfg = getattr(orc, "config", None)
|
cfg = getattr(orc, "config", None)
|
||||||
if cfg is not None:
|
if cfg is not None:
|
||||||
@@ -46,7 +45,6 @@ def _cfg_depth(orc=None) -> int:
|
|||||||
|
|
||||||
|
|
||||||
def _cfg_concurrency(orc=None) -> int:
|
def _cfg_concurrency(orc=None) -> int:
|
||||||
# A7: read from orchestrator config if available
|
|
||||||
if orc is not None:
|
if orc is not None:
|
||||||
cfg = getattr(orc, "config", None)
|
cfg = getattr(orc, "config", None)
|
||||||
if cfg is not None:
|
if cfg is not None:
|
||||||
@@ -137,29 +135,24 @@ class AvalancheScanner:
|
|||||||
def __init__(self, orchestrator: "Orchestrator") -> None:
|
def __init__(self, orchestrator: "Orchestrator") -> None:
|
||||||
self._orc = orchestrator
|
self._orc = orchestrator
|
||||||
self.seen_assets: Set[str] = set()
|
self.seen_assets: Set[str] = set()
|
||||||
# A2: single semaphore for the entire run, created lazily inside the event loop
|
|
||||||
self._sem: Optional[asyncio.Semaphore] = None
|
self._sem: Optional[asyncio.Semaphore] = None
|
||||||
self._all_records: List = []
|
self._all_records: List = []
|
||||||
self._dork_hits: List[dict] = []
|
self._dork_hits: List[dict] = []
|
||||||
self._seen_dork_urls: Set[str] = set()
|
self._seen_dork_urls: Set[str] = set()
|
||||||
# A6: scrape_hits merged atomically per _do_process call
|
|
||||||
self._scrape_hits: Dict = {"pastes": [], "credentials": [], "hashes": [],
|
self._scrape_hits: Dict = {"pastes": [], "credentials": [], "hashes": [],
|
||||||
"telegram": [], "dork_misconfigs": []}
|
"telegram": [], "dork_misconfigs": []}
|
||||||
self._max_depth: int = 0
|
self._max_depth: int = 0
|
||||||
self._in_flight: Dict[str, asyncio.Future] = {}
|
self._in_flight: Dict[str, asyncio.Future] = {}
|
||||||
self.pivot_log: List[dict] = []
|
self.pivot_log: List[dict] = []
|
||||||
# A8: global set to prevent duplicate entries in discovered_assets
|
|
||||||
self._seen_discovered: Set[str] = set()
|
self._seen_discovered: Set[str] = set()
|
||||||
self.discovered_assets: List[dict] = []
|
self.discovered_assets: List[dict] = []
|
||||||
|
|
||||||
def _get_sem(self) -> asyncio.Semaphore:
|
def _get_sem(self) -> asyncio.Semaphore:
|
||||||
# A2: semaphore created once per run, shared across all coroutines
|
|
||||||
if self._sem is None:
|
if self._sem is None:
|
||||||
self._sem = asyncio.Semaphore(_cfg_concurrency(self._orc))
|
self._sem = asyncio.Semaphore(_cfg_concurrency(self._orc))
|
||||||
return self._sem
|
return self._sem
|
||||||
|
|
||||||
async def run(self, target: str) -> tuple:
|
async def run(self, target: str) -> tuple:
|
||||||
# A9: respect no_pivot flag from config
|
|
||||||
cfg = getattr(self._orc, "config", None)
|
cfg = getattr(self._orc, "config", None)
|
||||||
no_pivot = getattr(cfg, "no_pivot", False) if cfg else False
|
no_pivot = getattr(cfg, "no_pivot", False) if cfg else False
|
||||||
if no_pivot:
|
if no_pivot:
|
||||||
@@ -196,7 +189,6 @@ class AvalancheScanner:
|
|||||||
async def _process(self, asset: str, depth: int,
|
async def _process(self, asset: str, depth: int,
|
||||||
parent: Optional[str], found_in: str) -> None:
|
parent: Optional[str], found_in: str) -> None:
|
||||||
"""Dedup gate: ensures each asset is processed exactly once."""
|
"""Dedup gate: ensures each asset is processed exactly once."""
|
||||||
# A10: use per-run depth from orchestrator config
|
|
||||||
if depth > _cfg_depth(self._orc):
|
if depth > _cfg_depth(self._orc):
|
||||||
_syslog.debug("avalanche depth cap reached for %s", asset)
|
_syslog.debug("avalanche depth cap reached for %s", asset)
|
||||||
return
|
return
|
||||||
@@ -205,7 +197,7 @@ class AvalancheScanner:
|
|||||||
if not key:
|
if not key:
|
||||||
return
|
return
|
||||||
|
|
||||||
# A1: add to seen_assets FIRST (atomic gate) before any other check.
|
# Add to seen_assets before any await to prevent concurrent duplicates.
|
||||||
# If already present, wait on the in-flight future if one exists, then return.
|
# If already present, wait on the in-flight future if one exists, then return.
|
||||||
if key in self.seen_assets:
|
if key in self.seen_assets:
|
||||||
if key in self._in_flight:
|
if key in self._in_flight:
|
||||||
@@ -326,7 +318,8 @@ class AvalancheScanner:
|
|||||||
_syslog.warning("SCRAPE_FAIL asset=%s err=%s", asset, exc)
|
_syslog.warning("SCRAPE_FAIL asset=%s err=%s", asset, exc)
|
||||||
scrape_res = {}
|
scrape_res = {}
|
||||||
|
|
||||||
# A6: collect scrape results locally, then merge atomically
|
# Collect scrape results locally then merge into the shared dict.
|
||||||
|
# The event loop is single-threaded so the merge is safe without a lock.
|
||||||
scrape_count = 0
|
scrape_count = 0
|
||||||
local_scrape: Dict = {k: [] for k in self._scrape_hits}
|
local_scrape: Dict = {k: [] for k in self._scrape_hits}
|
||||||
for k in self._scrape_hits:
|
for k in self._scrape_hits:
|
||||||
@@ -336,7 +329,7 @@ class AvalancheScanner:
|
|||||||
item["pivot_depth"] = depth
|
item["pivot_depth"] = depth
|
||||||
local_scrape[k].append(item)
|
local_scrape[k].append(item)
|
||||||
scrape_count += 1
|
scrape_count += 1
|
||||||
# Atomic merge into shared dict (single-threaded event loop — safe)
|
# Merge into shared dict — safe within the single-threaded event loop.
|
||||||
for k, items in local_scrape.items():
|
for k, items in local_scrape.items():
|
||||||
self._scrape_hits[k].extend(items)
|
self._scrape_hits[k].extend(items)
|
||||||
_out("ok" if scrape_count else "dim",
|
_out("ok" if scrape_count else "dim",
|
||||||
@@ -393,7 +386,6 @@ class AvalancheScanner:
|
|||||||
queued.add(child_key)
|
queued.add(child_key)
|
||||||
child_entry = {"asset": val, "qtype": vqtype, "found_in": phase, "ref": ref}
|
child_entry = {"asset": val, "qtype": vqtype, "found_in": phase, "ref": ref}
|
||||||
children.append(child_entry)
|
children.append(child_entry)
|
||||||
# A8: prevent duplicate entries in discovered_assets across parallel parents
|
|
||||||
if child_key not in self._seen_discovered:
|
if child_key not in self._seen_discovered:
|
||||||
self._seen_discovered.add(child_key)
|
self._seen_discovered.add(child_key)
|
||||||
self.discovered_assets.append({
|
self.discovered_assets.append({
|
||||||
@@ -412,12 +404,12 @@ class AvalancheScanner:
|
|||||||
self._process(val, depth + 1, parent=asset, found_in=phase)
|
self._process(val, depth + 1, parent=asset, found_in=phase)
|
||||||
)
|
)
|
||||||
|
|
||||||
# A5: run child tasks FIRST, then append pivot_log so the log reflects actual outcomes
|
# Run child tasks before appending to pivot_log so the log reflects actual outcomes.
|
||||||
if child_tasks:
|
if child_tasks:
|
||||||
_out("info", f"{indent} → reinjecting {len(child_tasks)} new asset(s)…")
|
_out("info", f"{indent} → reinjecting {len(child_tasks)} new asset(s)…")
|
||||||
await asyncio.gather(*child_tasks, return_exceptions=True)
|
await asyncio.gather(*child_tasks, return_exceptions=True)
|
||||||
|
|
||||||
# ── Log this node (after children complete — A5) ──────────────
|
# ── Log this node ─────────────────────────────────────────────
|
||||||
self.pivot_log.append({
|
self.pivot_log.append({
|
||||||
"asset": asset,
|
"asset": asset,
|
||||||
"qtype": qtype,
|
"qtype": qtype,
|
||||||
@@ -461,8 +453,8 @@ class AvalancheScanner:
|
|||||||
# ── Scrape dispatcher ─────────────────────────────────────────────
|
# ── Scrape dispatcher ─────────────────────────────────────────────
|
||||||
|
|
||||||
async def _async_scrape(self, asset: str) -> dict:
|
async def _async_scrape(self, asset: str) -> dict:
|
||||||
# A3: instantiate a fresh Session + ScrapeEngine per call to avoid sharing
|
# Instantiate a fresh Session and ScrapeEngine per call — requests.Session
|
||||||
# a non-thread-safe requests.Session / cloudscraper across concurrent coroutines.
|
# and cloudscraper are not safe to share across concurrent coroutines.
|
||||||
_empty: dict = {"pastes": [], "credentials": [], "hashes": [],
|
_empty: dict = {"pastes": [], "credentials": [], "hashes": [],
|
||||||
"telegram": [], "dork_misconfigs": []}
|
"telegram": [], "dork_misconfigs": []}
|
||||||
try:
|
try:
|
||||||
@@ -517,8 +509,7 @@ async def _crack_and_inject(session, hash_value: str, record_ref,
|
|||||||
_out("ok", f" [crack] {hash_value[:16]}… → {plaintext} (from {parent_asset})")
|
_out("ok", f" [crack] {hash_value[:16]}… → {plaintext} (from {parent_asset})")
|
||||||
cracked_out.append(plaintext)
|
cracked_out.append(plaintext)
|
||||||
|
|
||||||
# A4: inject cracked plaintext as qtype="password" — NOT as username.
|
# Inject the cracked plaintext as a password-recycling pivot seed.
|
||||||
# Only pivot on it if sources support password-recycling queries.
|
|
||||||
key = plaintext.lower()
|
key = plaintext.lower()
|
||||||
if key not in seen_assets and depth + 1 <= _cfg_depth(scanner._orc):
|
if key not in seen_assets and depth + 1 <= _cfg_depth(scanner._orc):
|
||||||
await scanner._process(plaintext, depth + 1,
|
await scanner._process(plaintext, depth + 1,
|
||||||
|
|||||||
@@ -7,7 +7,7 @@
|
|||||||
"selectors": {
|
"selectors": {
|
||||||
"stealers": "$.stealers"
|
"stealers": "$.stealers"
|
||||||
},
|
},
|
||||||
"rate_limit": 5.0,
|
"rate_limit": 30.0,
|
||||||
"headers": {
|
"headers": {
|
||||||
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/135.0.0.0 Safari/537.36"
|
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/135.0.0.0 Safari/537.36"
|
||||||
},
|
},
|
||||||
@@ -28,6 +28,7 @@
|
|||||||
"health_check_url": "https://cavalier.hudsonrock.com",
|
"health_check_url": "https://cavalier.hudsonrock.com",
|
||||||
"expected_status": 200,
|
"expected_status": 200,
|
||||||
"reliability_score": 3,
|
"reliability_score": 3,
|
||||||
|
"is_volatile": true,
|
||||||
"backup_endpoints": [],
|
"backup_endpoints": [],
|
||||||
"confidence": 0.7
|
"confidence": 0.7
|
||||||
}
|
}
|
||||||
@@ -1,7 +1,7 @@
|
|||||||
{
|
{
|
||||||
"name": "ipinfodb",
|
"name": "ipinfodb",
|
||||||
"category": "geolocation",
|
"category": "geolocation",
|
||||||
"endpoint": "http://api.ipinfodb.com/v3/ip-city/?key={IPINFODB_API_KEY}&ip={target}&format=json",
|
"endpoint": "https://api.ipinfodb.com/v3/ip-city/?key={IPINFODB_API_KEY}&ip={target}&format=json",
|
||||||
"method": "GET",
|
"method": "GET",
|
||||||
"requires_auth": true,
|
"requires_auth": true,
|
||||||
"selectors": {
|
"selectors": {
|
||||||
@@ -22,7 +22,7 @@
|
|||||||
"tags": [
|
"tags": [
|
||||||
"passive"
|
"passive"
|
||||||
],
|
],
|
||||||
"health_check_url": "http://api.ipinfodb.com",
|
"health_check_url": "https://api.ipinfodb.com",
|
||||||
"expected_status": 200,
|
"expected_status": 200,
|
||||||
"reliability_score": 3,
|
"reliability_score": 3,
|
||||||
"backup_endpoints": [],
|
"backup_endpoints": [],
|
||||||
|
|||||||
@@ -1,7 +1,7 @@
|
|||||||
{
|
{
|
||||||
"name": "ipstack",
|
"name": "ipstack",
|
||||||
"category": "geolocation",
|
"category": "geolocation",
|
||||||
"endpoint": "http://api.ipstack.com/{target}?access_key={IPSTACK_API_KEY}",
|
"endpoint": "https://api.ipstack.com/{target}?access_key={IPSTACK_API_KEY}",
|
||||||
"method": "GET",
|
"method": "GET",
|
||||||
"requires_auth": true,
|
"requires_auth": true,
|
||||||
"selectors": {
|
"selectors": {
|
||||||
@@ -23,7 +23,7 @@
|
|||||||
"passive",
|
"passive",
|
||||||
"fast"
|
"fast"
|
||||||
],
|
],
|
||||||
"health_check_url": "http://api.ipstack.com",
|
"health_check_url": "https://api.ipstack.com",
|
||||||
"expected_status": 200,
|
"expected_status": 200,
|
||||||
"reliability_score": 4,
|
"reliability_score": 4,
|
||||||
"backup_endpoints": [],
|
"backup_endpoints": [],
|
||||||
|
|||||||
@@ -1,7 +1,7 @@
|
|||||||
{
|
{
|
||||||
"name": "mailboxlayer",
|
"name": "mailboxlayer",
|
||||||
"category": "email_rep",
|
"category": "email_rep",
|
||||||
"endpoint": "http://apilayer.net/api/check?access_key={MAILBOX_API_KEY}&email={target}",
|
"endpoint": "https://apilayer.net/api/check?access_key={MAILBOX_API_KEY}&email={target}",
|
||||||
"method": "GET",
|
"method": "GET",
|
||||||
"requires_auth": true,
|
"requires_auth": true,
|
||||||
"selectors": {
|
"selectors": {
|
||||||
@@ -20,7 +20,7 @@
|
|||||||
"tags": [
|
"tags": [
|
||||||
"passive"
|
"passive"
|
||||||
],
|
],
|
||||||
"health_check_url": "http://apilayer.net",
|
"health_check_url": "https://apilayer.net",
|
||||||
"expected_status": 200,
|
"expected_status": 200,
|
||||||
"reliability_score": 3,
|
"reliability_score": 3,
|
||||||
"backup_endpoints": [],
|
"backup_endpoints": [],
|
||||||
|
|||||||
@@ -17,7 +17,8 @@
|
|||||||
"value": "{target}"
|
"value": "{target}"
|
||||||
},
|
},
|
||||||
"api_key_slots": [
|
"api_key_slots": [
|
||||||
"{MISP_API_KEY}"
|
"{MISP_API_KEY}",
|
||||||
|
"{MISP_URL}"
|
||||||
],
|
],
|
||||||
"input_type": "any",
|
"input_type": "any",
|
||||||
"output_type": [
|
"output_type": [
|
||||||
@@ -30,7 +31,7 @@
|
|||||||
"passive",
|
"passive",
|
||||||
"threat"
|
"threat"
|
||||||
],
|
],
|
||||||
"health_check_url": "{MISP_URL}",
|
"health_check_url": "https://misp.local",
|
||||||
"expected_status": 200,
|
"expected_status": 200,
|
||||||
"reliability_score": 4,
|
"reliability_score": 4,
|
||||||
"backup_endpoints": [],
|
"backup_endpoints": [],
|
||||||
|
|||||||
@@ -1,7 +1,7 @@
|
|||||||
{
|
{
|
||||||
"name": "numverify",
|
"name": "numverify",
|
||||||
"category": "phone",
|
"category": "phone",
|
||||||
"endpoint": "http://apilayer.net/api/validate?access_key={NUMVERIFY_API_KEY}&number={target}",
|
"endpoint": "https://apilayer.net/api/validate?access_key={NUMVERIFY_API_KEY}&number={target}",
|
||||||
"method": "GET",
|
"method": "GET",
|
||||||
"requires_auth": true,
|
"requires_auth": true,
|
||||||
"selectors": {
|
"selectors": {
|
||||||
@@ -24,7 +24,7 @@
|
|||||||
"tags": [
|
"tags": [
|
||||||
"passive"
|
"passive"
|
||||||
],
|
],
|
||||||
"health_check_url": "http://apilayer.net",
|
"health_check_url": "https://apilayer.net",
|
||||||
"expected_status": 200,
|
"expected_status": 200,
|
||||||
"reliability_score": 4,
|
"reliability_score": 4,
|
||||||
"backup_endpoints": [],
|
"backup_endpoints": [],
|
||||||
|
|||||||
@@ -24,7 +24,8 @@
|
|||||||
],
|
],
|
||||||
"health_check_url": "https://api.pipl.com",
|
"health_check_url": "https://api.pipl.com",
|
||||||
"expected_status": 200,
|
"expected_status": 200,
|
||||||
"reliability_score": 4,
|
"reliability_score": 2,
|
||||||
|
"is_volatile": true,
|
||||||
"backup_endpoints": [],
|
"backup_endpoints": [],
|
||||||
"confidence": 0.85
|
"confidence": 0.3
|
||||||
}
|
}
|
||||||
@@ -0,0 +1,31 @@
|
|||||||
|
{
|
||||||
|
"name": "ripestat_ip",
|
||||||
|
"category": "network",
|
||||||
|
"endpoint": "https://stat.ripe.net/data/prefix-overview/data.json?resource={target}",
|
||||||
|
"method": "GET",
|
||||||
|
"requires_auth": false,
|
||||||
|
"selectors": {
|
||||||
|
"asns": "$.data.asns[*].asn",
|
||||||
|
"holder": "$.data.asns[0].holder"
|
||||||
|
},
|
||||||
|
"rate_limit": 1.0,
|
||||||
|
"headers": {},
|
||||||
|
"api_key_slots": [],
|
||||||
|
"input_type": "ip",
|
||||||
|
"output_type": [
|
||||||
|
"ip"
|
||||||
|
],
|
||||||
|
"normalization_map": {
|
||||||
|
"asn": "asn_number",
|
||||||
|
"holder": "asn_org"
|
||||||
|
},
|
||||||
|
"tags": [
|
||||||
|
"passive",
|
||||||
|
"infrastructure"
|
||||||
|
],
|
||||||
|
"health_check_url": "https://stat.ripe.net",
|
||||||
|
"expected_status": 200,
|
||||||
|
"reliability_score": 5,
|
||||||
|
"backup_endpoints": [],
|
||||||
|
"confidence": 1.0
|
||||||
|
}
|
||||||
@@ -1,7 +1,7 @@
|
|||||||
{
|
{
|
||||||
"name": "spycloud_breach",
|
"name": "spycloud_breach",
|
||||||
"category": "breaches",
|
"category": "breaches",
|
||||||
"endpoint": "https://api.spycloud.io/enterprise-v2/breach/data/emails/{target}",
|
"endpoint": "https://api.spycloud.io/enterprise-v2/breach/catalog/emails/{target}",
|
||||||
"method": "GET",
|
"method": "GET",
|
||||||
"requires_auth": true,
|
"requires_auth": true,
|
||||||
"selectors": {
|
"selectors": {
|
||||||
|
|||||||
@@ -22,7 +22,8 @@
|
|||||||
],
|
],
|
||||||
"health_check_url": "https://api.spyonweb.com",
|
"health_check_url": "https://api.spyonweb.com",
|
||||||
"expected_status": 200,
|
"expected_status": 200,
|
||||||
"reliability_score": 3,
|
"reliability_score": 1,
|
||||||
|
"is_volatile": true,
|
||||||
"backup_endpoints": [],
|
"backup_endpoints": [],
|
||||||
"confidence": 0.7
|
"confidence": 0.1
|
||||||
}
|
}
|
||||||
@@ -9,7 +9,7 @@
|
|||||||
},
|
},
|
||||||
"rate_limit": 1.0,
|
"rate_limit": 1.0,
|
||||||
"headers": {
|
"headers": {
|
||||||
"Authorization": "TC {TC_API_KEY}:{TC_SIGNATURE}"
|
"Authorization": "TC {TC_API_KEY}"
|
||||||
},
|
},
|
||||||
"api_key_slots": [
|
"api_key_slots": [
|
||||||
"{TC_API_KEY}"
|
"{TC_API_KEY}"
|
||||||
@@ -26,7 +26,8 @@
|
|||||||
],
|
],
|
||||||
"health_check_url": "https://api.threatconnect.com",
|
"health_check_url": "https://api.threatconnect.com",
|
||||||
"expected_status": 200,
|
"expected_status": 200,
|
||||||
"reliability_score": 4,
|
"reliability_score": 2,
|
||||||
|
"is_volatile": true,
|
||||||
"backup_endpoints": [],
|
"backup_endpoints": [],
|
||||||
"confidence": 0.85
|
"confidence": 0.55
|
||||||
}
|
}
|
||||||
@@ -25,6 +25,7 @@
|
|||||||
"health_check_url": "https://api.twitter.com",
|
"health_check_url": "https://api.twitter.com",
|
||||||
"expected_status": 200,
|
"expected_status": 200,
|
||||||
"reliability_score": 1,
|
"reliability_score": 1,
|
||||||
|
"is_volatile": true,
|
||||||
"backup_endpoints": [],
|
"backup_endpoints": [],
|
||||||
"confidence": 0.4
|
"confidence": 0.1
|
||||||
}
|
}
|
||||||
@@ -0,0 +1,36 @@
|
|||||||
|
{
|
||||||
|
"name": "xposedornot",
|
||||||
|
"category": "breach_data",
|
||||||
|
"endpoint": "https://api.xposedornot.com/v1/breach-analytics?email={target}",
|
||||||
|
"method": "GET",
|
||||||
|
"requires_auth": false,
|
||||||
|
"selectors": {
|
||||||
|
"breaches": "$.ExposedBreaches.breaches_details"
|
||||||
|
},
|
||||||
|
"rate_limit": 2.0,
|
||||||
|
"headers": {
|
||||||
|
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/135.0.0.0 Safari/537.36"
|
||||||
|
},
|
||||||
|
"api_key_slots": [],
|
||||||
|
"input_type": "email",
|
||||||
|
"output_type": [
|
||||||
|
"email",
|
||||||
|
"domain"
|
||||||
|
],
|
||||||
|
"normalization_map": {
|
||||||
|
"breach": "breach_name",
|
||||||
|
"domain": "domain",
|
||||||
|
"xposed_date": "breach_date",
|
||||||
|
"xposed_data": "data_types",
|
||||||
|
"password_risk": "password_risk"
|
||||||
|
},
|
||||||
|
"tags": [
|
||||||
|
"passive",
|
||||||
|
"stealth"
|
||||||
|
],
|
||||||
|
"health_check_url": "https://api.xposedornot.com",
|
||||||
|
"expected_status": 200,
|
||||||
|
"reliability_score": 4,
|
||||||
|
"backup_endpoints": [],
|
||||||
|
"confidence": 0.75
|
||||||
|
}
|
||||||
Reference in New Issue
Block a user