release: v1.0.5

CHANGELOG.md

@@ -2,6 +2,34 @@
 
 All notable changes to NOX are documented here.
 
+## [1.0.5] — 2026-05-06
+
+### Sources
+- **Fixed:** `pypi_user` — endpoint was querying the PyPI package API (`/pypi/{target}/json`) instead of user profiles, returning package metadata for any username that matched a package name and 404 for all others. Source now uses the PyPI XML-RPC `user_packages` method, returning the actual list of packages owned by the target username.
+
+### Engine
+- **Fixed:** `Detect.qtype` — dotted-quad strings with out-of-range octets (e.g. `255.255.255.256`) were classified as `phone` instead of `username`. Invalid IPs now fall through to `username`.
+- **Fixed:** `Record.dedup_key` — records with no identity fields (email, username, password) from different sources all produced the same SHA-256 hash, causing all but the first to be silently dropped from the DB. Source name is now included in the key when identity fields are absent.
+- **Added:** `raw_payload` support in `NoxSourceProvider` — POST sources can now send raw string bodies (e.g. XML-RPC) in addition to JSON and form-encoded payloads.
+- **Fixed:** `FileSystemProvider._by_regex` — regex-extracted values that are not email addresses (e.g. package names, usernames) are now correctly placed in the `username` field instead of `email`.
+
+## [1.0.4] — 2026-04-22
+
+### Engine
+- **Fixed:** `_build_ssl_context` — custom TLS context had zero CA certificates loaded, causing `SSLCertVerificationError` on all HTTPS connections. CA bundle now loaded via `certifi` with `load_default_certs()` fallback.
+- **Fixed:** `_NOISE_RE` in reporting — `ssl.`, `aiohttp.`, `asyncio.` were substring-matched, silently zeroing legitimate emails and domains (e.g. `user@ssl.example.com`). Patterns now anchored to start-of-string or whitespace.
+- **Fixed:** `COMBO_RE` in `ScrapeEngine` — multiline greedy match produced credentials with embedded newlines in email/password fields. Pattern now excludes newlines from both capture groups.
+- **Fixed:** `_in_flight` dict in `AvalancheScanner` — entries were never removed after processing, causing unbounded memory growth on deep scans. Entry is now popped in the `finally` block after the future resolves.
+- **Fixed:** `DorkingEngine.__init__` — `ProxyManager.get_proxies()` was called eagerly on every `Orchestrator` instantiation, triggering a proxy fetch and OPSEC warning even for local-only commands (`--crack`, `--list-sources`, `--analyze`). Proxy fetch is now lazy.
+- **Fixed:** `ScrapeEngine._fetch_content` — IntelX paste content fetch used `DB.get_key("intelx")` which does not read `apikeys.json`. Replaced with `Vault.get("INTELX_API_KEY")` for consistent key resolution.
+- **Fixed:** `AsyncSource._rec` and `RiskEngine.score` — plugin `confidence` values were stored in `NoxSourceProvider._confidence` but never transferred to `Record.source_confidence`. All 124 plugin records received a flat `0.5` confidence regardless of their declared value. `_rec()` now injects `self._confidence` into the record; `RiskEngine.score()` uses `record.source_confidence` as fallback when the source is not in `_SRC_CONFIDENCE`.
+- **Fixed:** `ConfigManager.get_key` — `None` results were cached, preventing env vars set after the first lookup from being detected in the same session. Only positive values are now cached.
+- **Fixed:** Recursive Avalanche Engine — identifiers extracted from paste content (`paste["patterns"]`) were not being harvested as pivot seeds. All `scrape_res["pastes"]` pattern matches are now fed into `_extract_ids_from_text`.
+
+### Sources
+- **Fixed:** `circl_hashlookup` — endpoint hardcoded to `/lookup/md5/{target}` but `input_type=hash` accepts SHA1/SHA256. SHA1 and SHA256 backup endpoints added; engine now routes each hash type to the correct path.
+- **Updated:** `crt_sh` — `reliability_score` lowered from 5 to 3, `is_volatile=true` added to reflect documented intermittent availability.
+
 ## [1.0.3] — 2026-04-15
 
 ### Engine

README.md

@@ -11,7 +11,7 @@
 
 **Cyber Threat Intelligence Framework**
 
-[![Status](https://img.shields.io/badge/Status-v1.0.3-success)](https://github.com/nox-project/nox-framework/releases/tag/v1.0.3)
+[![Status](https://img.shields.io/badge/Status-v1.0.5-success)](https://github.com/nox-project/nox-framework/releases/tag/v1.0.5)
 [![Python](https://img.shields.io/badge/Python-3.8%2B-blue?logo=python&logoColor=white)](https://www.python.org/)
 [![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](LICENSE.txt)
 [![Kali Linux](https://img.shields.io/badge/Kali%20Linux-Ready-557C94?logo=kalilinux&logoColor=white)](https://www.kali.org/)

build_deb.sh

@@ -1,10 +1,10 @@
 #!/usr/bin/env bash
 set -e
 
-# NOX v1.0.3 — .deb build script (FPM)
+# NOX v1.0.5 — .deb build script (FPM)
 # Requires: fpm  →  gem install fpm
 
-VERSION="1.0.3"
+VERSION="1.0.5"
 PKG_NAME="nox-cli"
 ARCH="all"
 OUT_DIR="dist"

build_sources.py

@@ -53,6 +53,8 @@ class SourceConfig(BaseModel):
     rate_limit:       float                    = 1.0
     headers:          Dict[str, str]           = Field(default_factory=dict)
     payload_template: Optional[Dict[str, Any]] = None
+    raw_payload:      Optional[str]            = None
+    regex_pattern:    Optional[str]            = None
     api_key_slots:    List[str]                = Field(default_factory=list)
 
     # ── Typing & pivoting ───────────────────────────────────────────────────
@@ -121,6 +123,8 @@ def _mk(
     rate_limit: float = 1.0,
     headers: Optional[Dict[str, str]] = None,
     payload_template: Optional[Dict[str, Any]] = None,
+    raw_payload: Optional[str] = None,
+    regex_pattern: Optional[str] = None,
     api_key_slots: Optional[List[str]] = None,
     input_type: InputType = "any",
     output_type: Optional[List[str]] = None,
@@ -146,6 +150,8 @@ def _mk(
         rate_limit=rate_limit,
         headers=headers or {},
         payload_template=payload_template,
+        raw_payload=raw_payload,
+        regex_pattern=regex_pattern,
         api_key_slots=api_key_slots or [],
         input_type=input_type,
         output_type=output_type or [],
@@ -188,7 +194,7 @@ FREE_PUBLIC_SOURCES: List[SourceConfig] = [
           input_type="domain", output_type=["domain"],
           normalization_map={"name_value": "domain"},
           tags=["passive", "fast"],
-          health_check_url="https://crt.sh", reliability_score=5),
+          health_check_url="https://crt.sh", reliability_score=3, is_volatile=True),
 
     _base("hackertarget_hostsearch", "dns_recon",
           "https://api.hackertarget.com/hostsearch/?q={target}", "GET",
@@ -416,11 +422,15 @@ FREE_PUBLIC_SOURCES: List[SourceConfig] = [
           health_check_url="https://ipvigilante.com", reliability_score=3, is_volatile=True),
 
     _base("pypi_user", "social",
-          "https://pypi.org/pypi/{target}/json", "GET",
-          {"info": "$.info"},
+          "https://pypi.org/pypi", "POST",
+          {},
+          headers={"Content-Type": "text/xml"},
           input_type="username", output_type=["username"],
+          normalization_map={},
+          regex_pattern=r"<string>Owner</string></value>\s*<value><string>([^<]+)</string>",
           tags=["passive"],
-          health_check_url="https://pypi.org", reliability_score=5),
+          health_check_url="https://pypi.org", reliability_score=4,
+          raw_payload="<?xml version='1.0'?><methodCall><methodName>user_packages</methodName><params><param><value>{target}</value></param></params></methodCall>"),
 
     _base("npm_user", "social",
           "https://registry.npmjs.org/-/v1/search?text=maintainer:{target}", "GET",
@@ -521,7 +531,11 @@ FREE_PUBLIC_SOURCES: List[SourceConfig] = [
           normalization_map={"FileName": "filename", "MD5": "hash_md5"},
           tags=["passive", "fast"],
           health_check_url="https://hashlookup.circl.lu",
-          reliability_score=5),
+          reliability_score=5,
+          backup_endpoints=[
+              "https://hashlookup.circl.lu/lookup/sha1/{target}",
+              "https://hashlookup.circl.lu/lookup/sha256/{target}",
+          ]),
 
     _base("ipapi_is", "geolocation",
           "https://api.ipapi.is/?q={target}", "GET",

docs/nox-cli.1

@@ -1,4 +1,4 @@
-.TH NOX\-CLI 1 "2026-03-30" "1.0.0" "NOX Framework"
+.TH NOX\-CLI 1 "2026-05-06" "1.0.5" "NOX Framework"
 .SH NAME
 nox-cli \- Advanced Asynchronous Cyber Threat Intelligence Framework
 .SH SYNOPSIS

nox.py

@@ -150,7 +150,7 @@ except Exception:
             VERSION = _sp2.check_output(["dpkg-query", "-W", "-f=${Version}", "nox-cli"], stderr=_sp2.DEVNULL).decode().strip() or VERSION
         except Exception:
             pass
-BUILD_DATE = "2026-04-15"
+BUILD_DATE = "2026-05-06"
 
 # ── Smart Path Layout ──────────────────────────────────────────────────
 HOME_NOX    = Path.home() / ".nox"
@@ -618,9 +618,13 @@ class Record:
         return d
 
     def dedup_key(self) -> str:
-        """SHA-256 of normalised email:password for cross-source deduplication."""
+        """SHA-256 of normalised email:password for cross-source deduplication.
+        When identity fields are absent, source is included to prevent enrichment
+        records from different sources colliding into a single DB entry."""
         em = (self.email or self.username or "").lower().strip()
         pw = (self.password or self.password_hash or "").strip()
+        if not em and not pw:
+            return hashlib.sha256(f"{self.source}:{em}:{pw}".encode()).hexdigest()
         return hashlib.sha256(f"{em}:{pw}".encode()).hexdigest()
 
     def get_fingerprint(self) -> str:
@@ -646,7 +650,7 @@ class RiskEngine:
 
     @staticmethod
     def score(record: "Record") -> "Record":
-        conf = _SRC_CONFIDENCE.get(record.source, 0.5)
+        conf = _SRC_CONFIDENCE.get(record.source, record.source_confidence)
         record.source_confidence = conf
 
         dtypes_str = " ".join(record.data_types).lower() if record.data_types else ""
@@ -1540,6 +1544,11 @@ def _build_ssl_context() -> ssl.SSLContext:
     ctx.set_ciphers(Cfg.TLS_CIPHERS)
     ctx.check_hostname = True
     ctx.verify_mode    = ssl.CERT_REQUIRED
+    try:
+        import certifi
+        ctx.load_verify_locations(certifi.where())
+    except Exception:
+        ctx.load_default_certs()
     return ctx
 
 
@@ -1702,6 +1711,7 @@ class AsyncSource(ABC):
 
     def _rec(self, **kw) -> Record:
         kw.setdefault("source", self.name)
+        kw.setdefault("source_confidence", getattr(self, "_confidence", 0.5))
         sev = kw.pop("severity", Severity.MEDIUM)
         r   = Record(**{k: v for k, v in kw.items() if k in Record.__dataclass_fields__})
         r.severity = sev
@@ -1732,7 +1742,7 @@ class AsyncSource(ABC):
                 _syslog.debug("API_FAIL source=%s url=%s error=%s", self.name, url[:80], exc)
         return 0, "", b""
 
-    async def _post(self, session: "aiohttp.ClientSession", url: str, json_data: Dict = None, data: Dict = None, headers: Dict = None, timeout: int = None) -> Tuple[int, str, bytes]:
+    async def _post(self, session: "aiohttp.ClientSession", url: str, json_data: Dict = None, data: Dict = None, headers: Dict = None, timeout: int = None, raw_body: str = None) -> Tuple[int, str, bytes]:
         """Perform a POST with jitter and retry logic."""
         await _jitter(self._config)
         to   = aiohttp_mod.ClientTimeout(total=timeout or self._config.timeout) if aiohttp_mod else None
@@ -1740,7 +1750,18 @@ class AsyncSource(ABC):
         for attempt in range(Cfg.RETRIES):
             try:
                 async with self._sem:
-                    if json_data is not None:
+                    if raw_body is not None:
+                        async with session.post(url, data=raw_body.encode(), headers=hdrs, timeout=to, ssl=_SSL_CTX) as resp:
+                            if resp.status == 429:
+                                retry_after = _parse_retry_after(resp.headers.get("Retry-After", ""), Cfg.RETRY_DELAY * (attempt + 2))
+                                _syslog.info("RATE_LIMIT source=%s url=%s retry_after=%ds", self.name, url[:80], retry_after)
+                                await asyncio.sleep(min(retry_after, 30))
+                                continue
+                            body = await resp.read()
+                            if resp.status >= 400:
+                                _syslog.warning("API_ERROR source=%s status=%d url=%s", self.name, resp.status, url[:80])
+                            return resp.status, await resp.text(errors="replace"), body
+                    elif json_data is not None:
                         hdrs["Content-Type"] = "application/json"
                         async with session.post(url, json=json_data, headers=hdrs, timeout=to, ssl=_SSL_CTX) as resp:
                             if resp.status == 429:
@@ -1834,6 +1855,7 @@ class Detect:
         q = q.strip()
         if re.match(r"^[\w.+-]+@[\w-]+\.[\w.]+$", q):                                                    return "email"
         if re.match(r"^\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}$", q) and all(0 <= int(o) <= 255 for o in q.split(".")): return "ip"
+        if re.match(r"^\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}$", q):                                                    return "username"
         if re.match(r"^(\+?\d{1,3}[\s.-]?)?\(?\d{2,4}\)?[\s.-]?\d{3,4}[\s.-]?\d{3,4}$", q):             return "phone"
         if re.match(r"^[a-fA-F0-9]{32,128}$", q):                                                        return "hash"
         if re.match(r"^\$2[aby]?\$", q) or re.match(r"^\$argon2", q) or re.match(r"^\$[156]\$", q):      return "hash"
@@ -2236,10 +2258,12 @@ class DorkingEngine(Src):
         super().__init__(*args, **kwargs)
         self._dead_proxies: set = set()
         self._proxy_index: int = 0
-        self.proxies = ProxyManager.get_proxies()
+        self.proxies: list = []
         self._dead_instances: set = set()
 
     def _get_next_proxy(self) -> Optional[str]:
+        if not self.proxies:
+            self.proxies = ProxyManager.get_proxies()
         live = [p for p in self.proxies if p not in self._dead_proxies]
         if not live:
             return None
@@ -2498,7 +2522,7 @@ class ScrapeEngine:
     CRED_RE  = re.compile(r"[\w.+-]+@[\w-]+\.[\w.-]+\s*[:;|]\s*\S+", re.IGNORECASE)
     EMAIL_RE = re.compile(r"[\w.+-]+@[\w-]+\.[\w.]+")
     HASH_RE  = re.compile(r"\b[a-f0-9]{32,128}\b", re.IGNORECASE)
-    COMBO_RE = re.compile(r"^[^:]+:[^:]+$", re.MULTILINE)
+    COMBO_RE = re.compile(r"^[^\n:]+:[^\n:]+$", re.MULTILINE)
 
     PATTERNS = [
         (re.compile(r"(?:password|passwd|pass|pwd)\s*[:=]\s*\S+", re.I),                                                    "Password"),
@@ -2722,7 +2746,7 @@ class ScrapeEngine:
                 return ""
             raw_urls: dict = {}  # paste fetch URLs — resolved per site name
             if site == "IntelX":
-                key = self.db.get_key("intelx")
+                key = Vault.get("INTELX_API_KEY")
                 if key:
                     resp = self.s.get(f"https://2.intelx.io/file/read?type=1&systemid={pid}&k={key}", timeout=15)
                     if resp.ok:
@@ -4003,7 +4027,7 @@ class AdvancedReporter:
         else:
             lines.append("_No pivot assets discovered._")
 
-        with open(path, "w", encoding="utf-8") as fh:
+        with open(path, "w", encoding="utf-8", errors="replace") as fh:
             fh.write("\n".join(lines) + "\n")
         out("ok", f"Markdown saved: {path}")
 
@@ -6266,8 +6290,12 @@ class FileSystemProvider(AsyncSource):
         records = []
         for m in re.finditer(pattern, text):
             groups = m.groups()
+            val = groups[0] if len(groups) > 0 else ""
+            # Route to username if the value is not an email address
+            is_email = "@" in val
             records.append(self._rec(
-                email    = groups[0] if len(groups) > 0 else "",
+                email    = val if is_email else "",
+                username = val if not is_email else "",
                 password = groups[1] if len(groups) > 1 else "",
                 breach_name = self.name,
                 data_types  = [self.name, "Credentials"],
@@ -6398,9 +6426,14 @@ class NoxSourceProvider(FileSystemProvider):
         payload = _sub(d.get("payload") or {})
 
         if method == "POST":
-            status, text, _ = await self._post(session, url,
-                                                json_data=payload or None,
-                                                headers=hdrs)
+            if isinstance(payload, str):
+                status, text, _ = await self._post(session, url,
+                                                    raw_body=payload,
+                                                    headers=hdrs)
+            else:
+                status, text, _ = await self._post(session, url,
+                                                    json_data=payload or None,
+                                                    headers=hdrs)
         else:
             status, text, _ = await self._get(session, url, headers=hdrs)
 
@@ -6567,8 +6600,8 @@ class SourceOrchestrator:
                     "output_type":           raw.get("output_type", []),
                     "pivot_types":           raw.get("pivot_types", []),
                     "confidence":            raw.get("confidence", 0.5),
-                    # payload_template → payload for POST sources
-                    "payload":               raw.get("payload_template") or raw.get("payload") or {},
+                    # payload_template → payload for POST sources; raw_payload takes precedence
+                    "payload":               raw.get("raw_payload") or raw.get("payload_template") or raw.get("payload") or {},
                     # Pass resolved slot keys so FileSystemProvider can use them
                     "_slot_keys":            slot_keys,
                     # Two-phase poll support

pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "nox-cli"
-version = "1.0.3"
+version = "1.0.5"
 description = "Advanced Asynchronous Cyber Threat Intelligence Framework"
 readme = { file = "README.md", content-type = "text/markdown" }
 license = { text = "Apache-2.0" }

setup.py

@@ -9,7 +9,7 @@ requirements = [
 
 setup(
     name="nox-cli",
-    version="1.0.0",
+    version="1.0.5",
     author="nox-project",
     description="Advanced Asynchronous Cyber Threat Intelligence Framework",
     long_description=Path("README.md").read_text(),

sources/circl_hashlookup.json

@@ -26,6 +26,9 @@
     "health_check_url": "https://hashlookup.circl.lu",
     "expected_status": 200,
     "reliability_score": 5,
-    "backup_endpoints": [],
+    "backup_endpoints": [
+        "https://hashlookup.circl.lu/lookup/sha1/{target}",
+        "https://hashlookup.circl.lu/lookup/sha256/{target}"
+    ],
     "confidence": 1.0
 }

sources/crt_sh.json

@@ -25,7 +25,8 @@
     ],
     "health_check_url": "https://crt.sh",
     "expected_status": 200,
-    "reliability_score": 5,
+    "reliability_score": 3,
+    "is_volatile": true,
     "backup_endpoints": [],
-    "confidence": 1.0
+    "confidence": 0.7
 }

sources/helpers/config_handler.py

@@ -228,7 +228,8 @@ class ConfigManager:
             return cls._cache[key_name]
         val = os.environ.get(key_name, "") or cls._get_store().get(key_name, "")
         result = None if (not val or val == UNIVERSAL_PLACEHOLDER) else val
-        cls._cache[key_name] = result
+        if result is not None:
+            cls._cache[key_name] = result
         return result
 
     # Backward-compatible alias used by nox.py internals

sources/helpers/reporting.py

@@ -14,9 +14,9 @@ from typing import Any, Dict, List
 # ── Noise patterns stripped from all report output ────────────────────
 _NOISE_RE = re.compile(
     r"(Traceback \(most recent|File \".*\.py\"|TimeoutError|ProxyError"
-    r"|ConnectionError|aiohttp\.|ClientConnector|ssl\.|asyncio\."
-    r"|Task exception|NoneType|Object of type)",
-    re.I,
+    r"|ConnectionError|ClientConnector|Task exception|NoneType|Object of type"
+    r"|(?:^|[\s(])aiohttp\.|(?:^|[\s(])asyncio\.|(?:^|[\s(])ssl\.)",
+    re.I | re.MULTILINE,
 )
 _CTRL_RE = re.compile(r"[\x00-\x08\x0b\x0c\x0e-\x1f\x7f-\x9f]")
 

sources/helpers/scanner.py

@@ -227,6 +227,7 @@ class AvalancheScanner:
         finally:
             if not fut.done():
                 fut.set_result(None)
+            self._in_flight.pop(key, None)
 
     # ── Core pipeline ─────────────────────────────────────────────────
 

sources/pypi_user.json

@@ -1,14 +1,16 @@
 {
     "name": "pypi_user",
     "category": "social",
-    "endpoint": "https://pypi.org/pypi/{target}/json",
-    "method": "GET",
+    "endpoint": "https://pypi.org/pypi",
+    "method": "POST",
     "requires_auth": false,
-    "selectors": {
-        "info": "$.info"
-    },
+    "selectors": {},
     "rate_limit": 1.0,
-    "headers": {},
+    "headers": {
+        "Content-Type": "text/xml"
+    },
+    "raw_payload": "<?xml version='1.0'?><methodCall><methodName>user_packages</methodName><params><param><value>{target}</value></param></params></methodCall>",
+    "regex_pattern": "<string>Owner</string></value>\\s*<value><string>([^<]+)</string>",
     "api_key_slots": [],
     "input_type": "username",
     "output_type": [
@@ -20,7 +22,7 @@
     ],
     "health_check_url": "https://pypi.org",
     "expected_status": 200,
-    "reliability_score": 5,
+    "reliability_score": 4,
     "backup_endpoints": [],
-    "confidence": 1.0
+    "confidence": 0.85
 }