mirror of
https://github.com/nox-project/nox-framework.git
synced 2026-06-13 10:21:21 +00:00
NOX Framework v1.0.0
This commit is contained in:
nox-project
@@ -0,0 +1,28 @@
|
||||
name: CI
|
||||
|
||||
on:
|
||||
push:
|
||||
branches: [main]
|
||||
pull_request:
|
||||
branches: [main]
|
||||
|
||||
jobs:
|
||||
test:
|
||||
runs-on: ubuntu-latest
|
||||
strategy:
|
||||
matrix:
|
||||
python-version: ["3.11", "3.12"]
|
||||
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- name: Set up Python ${{ matrix.python-version }}
|
||||
uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: ${{ matrix.python-version }}
|
||||
|
||||
- name: Install dependencies
|
||||
run: pip install pytest pytest-asyncio aiohttp pydantic colorama rich fpdf2 dnspython phonenumbers stem cloudscraper
|
||||
|
||||
- name: Run tests
|
||||
run: python -m pytest tests/ -v
|
||||
@@ -0,0 +1,40 @@
|
||||
name: Release
|
||||
|
||||
on:
|
||||
push:
|
||||
tags:
|
||||
- "v*"
|
||||
|
||||
jobs:
|
||||
release:
|
||||
runs-on: ubuntu-latest
|
||||
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- name: Set up Python
|
||||
uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: "3.11"
|
||||
|
||||
- name: Install dependencies
|
||||
run: pip install pytest pytest-asyncio aiohttp pydantic colorama rich fpdf2 dnspython phonenumbers stem cloudscraper
|
||||
|
||||
- name: Run tests
|
||||
run: python -m pytest tests/ -v
|
||||
|
||||
- name: Install fpm
|
||||
run: |
|
||||
sudo apt-get install -y ruby ruby-dev build-essential
|
||||
sudo gem install fpm
|
||||
|
||||
- name: Build .deb
|
||||
run: bash build_deb.sh
|
||||
|
||||
- name: Create GitHub Release
|
||||
uses: softprops/action-gh-release@v2
|
||||
with:
|
||||
files: dist/*.deb
|
||||
generate_release_notes: true
|
||||
env:
|
||||
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
+44
@@ -0,0 +1,44 @@
|
||||
# Python
|
||||
__pycache__/
|
||||
*.pyc
|
||||
*.pyo
|
||||
*.pyd
|
||||
.pytest_cache/
|
||||
*.egg-info/
|
||||
dist/
|
||||
build/
|
||||
MANIFEST
|
||||
|
||||
# Virtual environments
|
||||
.venv/
|
||||
venv/
|
||||
env/
|
||||
ENV/
|
||||
|
||||
# NOX runtime artifacts — never commit
|
||||
nox_cache.db
|
||||
*.log
|
||||
proxies.txt
|
||||
*.deb
|
||||
reports/
|
||||
|
||||
# Credentials — never commit
|
||||
apikeys.json
|
||||
credentials.ini
|
||||
*.key
|
||||
*.pem
|
||||
|
||||
# OS
|
||||
.DS_Store
|
||||
Thumbs.db
|
||||
.idea/
|
||||
.vscode/
|
||||
*.swp
|
||||
*.swo
|
||||
|
||||
# Env files
|
||||
.env
|
||||
.env.*
|
||||
|
||||
# Internal development notes — not for distribution
|
||||
IMPROVEMENTS.md
|
||||
@@ -0,0 +1,24 @@
|
||||
# Changelog
|
||||
|
||||
All notable changes to NOX are documented here.
|
||||
|
||||
## [1.0.0] — 2026-04-02
|
||||
|
||||
### Initial Release
|
||||
|
||||
- 124 Pydantic v2-validated JSON source plugins across breach, network, OSINT, and threat-intel categories
|
||||
- Fully async execution engine (`asyncio` + `aiohttp`) with JA3 TLS fingerprinting and per-request jitter
|
||||
- `--autoscan` pipeline: breach scan → recursive identity pivot (depth 2) → Google/DDG dorking → paste/Telegram scraping
|
||||
- `--fullscan`: breach scan + pivot only
|
||||
- `--scan` / REPL `scan`: breach sources only
|
||||
- Guardian Proxy Engine: automatic proxy rotation with fail-safe kill-switch
|
||||
- Risk scoring engine (0–100) with time-decay, source confidence weighting, persistence multipliers, and HVT detection
|
||||
- Recursive Avalanche Engine: every discovered asset re-injected as a new scan seed across breach, dork, and scrape concurrently
|
||||
- Union-Find identity clustering across all breach records
|
||||
- Forensic PDF/HTML/JSON/CSV/Markdown reporting with Executive Summary dashboard
|
||||
- Hash identification and multi-engine cracking (dictionary + mutations + online rainbow tables)
|
||||
- Deep password strength analysis with entropy, leet-speak detection, and crack-time estimates
|
||||
- Interactive REPL with full feature parity with the CLI
|
||||
- Full audit logging: all scan events mirrored to `~/.nox/logs/nox.log`
|
||||
- Isolated `.deb` packaging for Kali Linux (PEP 668 compliant — zero system pollution)
|
||||
- `~/.config/nox-cli/apikeys.json` credential store (chmod 0600)
|
||||
@@ -0,0 +1,35 @@
|
||||
# Contributing to NOX
|
||||
|
||||
## Before You Start
|
||||
|
||||
NOX is a security tool. All contributions must comply with the [Legal Disclaimer](README.md#legal-disclaimer) and the [Apache 2.0 License](LICENSE.txt).
|
||||
|
||||
## Adding an Intelligence Source
|
||||
|
||||
All sources are defined exclusively in `build_sources.py`. Never edit `sources/*.json` directly — they are auto-generated artifacts.
|
||||
|
||||
1. Add a `_base()` (public) or `_auth()` (API key required) call in `build_sources.py`
|
||||
2. Run `python build_sources.py` to regenerate and validate all plugins
|
||||
3. Verify with `nox-cli --sources`
|
||||
|
||||
## Code Style
|
||||
|
||||
- Python 3.8+ compatible
|
||||
- No new runtime dependencies without justification in the PR
|
||||
- All async I/O through `aiohttp` — no `requests` in hot paths
|
||||
- Error handling: log at `DEBUG`, never crash the scan loop
|
||||
|
||||
## Pull Request Checklist
|
||||
|
||||
- [ ] `python3 -m py_compile nox.py` passes
|
||||
- [ ] `python build_sources.py` completes without errors
|
||||
- [ ] No credentials, API keys, or personal data in the diff
|
||||
- [ ] `sources/*.json` regenerated if `build_sources.py` was modified
|
||||
|
||||
## Reporting Bugs
|
||||
|
||||
Open a GitHub issue with:
|
||||
- NOX version (`nox-cli --version`)
|
||||
- Python version
|
||||
- Minimal reproduction steps
|
||||
- Expected vs actual behaviour
|
||||
@@ -0,0 +1,38 @@
|
||||
# Legal Disclaimer
|
||||
|
||||
**NOX (the “Tool”) is provided for educational and authorised security research purposes only.**
|
||||
|
||||
By using this Tool, you agree to the following terms:
|
||||
|
||||
1. **Authorisation**
|
||||
You must have explicit permission from the owner of any target system or data before scanning it. Unauthorised access to computer systems, networks, or data is illegal in most jurisdictions.
|
||||
|
||||
2. **No Warranty**
|
||||
The Tool is provided “AS IS”, without any warranty of any kind, express or implied. The authors and contributors assume no liability for any damage or legal consequences arising from the use of this Tool.
|
||||
|
||||
3. **Compliance with Laws**
|
||||
You are solely responsible for ensuring that your use of this Tool complies with all applicable local, national, and international laws and regulations.
|
||||
|
||||
4. **Third‑Party Services**
|
||||
The Tool interacts with public websites and APIs. You must respect the terms of service of those services. The authors are not responsible for any violation of those terms by users of the Tool.
|
||||
|
||||
5. **Ethical Use**
|
||||
You shall use the Tool only for legitimate security testing, personal data protection, or academic research. Any use that violates privacy rights, intellectual property rights, or otherwise causes harm is strictly prohibited.
|
||||
|
||||
6. **Indemnification**
|
||||
You agree to indemnify and hold harmless the authors and contributors from any claims, damages, or expenses arising from your misuse of the Tool.
|
||||
|
||||
7. **Export Control**
|
||||
This Tool may be subject to export control laws. You agree to comply with all applicable export and import restrictions.
|
||||
|
||||
8. **No Guarantee of Accuracy**
|
||||
The Tool aggregates data from third‑party sources and may contain inaccuracies. The authors do not guarantee the correctness, completeness, or timeliness of any information obtained through the Tool.
|
||||
|
||||
9. **Termination**
|
||||
The authors reserve the right to revoke permission to use this Tool at any time for any reason.
|
||||
|
||||
**By downloading, installing, or using NOX, you acknowledge that you have read, understood, and agree to be bound by this disclaimer.**
|
||||
|
||||
---
|
||||
|
||||
If you do not agree with these terms, do not use the Tool.
|
||||
+207
@@ -0,0 +1,207 @@
|
||||
|
||||
---
|
||||
|
||||
### `LICENSE`
|
||||
|
||||
```txt
|
||||
Apache License
|
||||
Version 2.0, January 2004
|
||||
http://www.apache.org/licenses/
|
||||
|
||||
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
|
||||
|
||||
1. Definitions.
|
||||
|
||||
"License" shall mean the terms and conditions for use, reproduction,
|
||||
and distribution as defined by Sections 1 through 9 of this document.
|
||||
|
||||
"Licensor" shall mean the copyright owner or entity authorized by
|
||||
the copyright owner that is granting the License.
|
||||
|
||||
"Legal Entity" shall mean the union of the acting entity and all
|
||||
other entities that control, are controlled by, or are under common
|
||||
control with that entity. For the purposes of this definition,
|
||||
"control" means (i) the power, direct or indirect, to cause the
|
||||
direction or management of such entity, whether by contract or
|
||||
otherwise, or (ii) ownership of fifty percent (50%) or more of the
|
||||
outstanding shares, or (iii) beneficial ownership of such entity.
|
||||
|
||||
"You" (or "Your") shall mean an individual or Legal Entity
|
||||
exercising permissions granted by this License.
|
||||
|
||||
"Source" form shall mean the preferred form for making modifications,
|
||||
including but not limited to software source code, documentation
|
||||
source, and configuration files.
|
||||
|
||||
"Object" form shall mean any form resulting from mechanical
|
||||
transformation or translation of a Source form, including but
|
||||
not limited to compiled object code, generated documentation,
|
||||
and conversions to other media types.
|
||||
|
||||
"Work" shall mean the work of authorship, whether in Source or
|
||||
Object form, made available under the License, as indicated by a
|
||||
copyright notice that is included in or attached to the work
|
||||
(an example is provided in the Appendix below).
|
||||
|
||||
"Derivative Works" shall mean any work, whether in Source or Object
|
||||
form, that is based on (or derived from) the Work and for which the
|
||||
editorial revisions, annotations, elaborations, or other modifications
|
||||
represent, as a whole, an original work of authorship. For the purposes
|
||||
of this License, Derivative Works shall not include works that remain
|
||||
separable from, or merely link (or bind by name) to the interfaces of,
|
||||
the Work and Derivative Works thereof.
|
||||
|
||||
"Contribution" shall mean any work of authorship, including
|
||||
the original version of the Work and any modifications or additions
|
||||
to that Work or Derivative Works thereof, that is intentionally
|
||||
submitted to Licensor for inclusion in the Work by the copyright owner
|
||||
or by an individual or Legal Entity authorized to submit on behalf of
|
||||
the copyright owner. For the purposes of this definition, "submitted"
|
||||
means any form of electronic, verbal, or written communication sent
|
||||
to the Licensor or its representatives, including but not limited to
|
||||
communication on electronic mailing lists, source code control systems,
|
||||
and issue tracking systems that are managed by, or on behalf of, the
|
||||
Licensor for the purpose of discussing and improving the Work, but
|
||||
excluding communication that is conspicuously marked or otherwise
|
||||
designated in writing by the copyright owner as "Not a Contribution."
|
||||
|
||||
"Contributor" shall mean Licensor and any individual or Legal Entity
|
||||
on behalf of whom a Contribution has been received by Licensor and
|
||||
subsequently incorporated within the Work.
|
||||
|
||||
2. Grant of Copyright License. Subject to the terms and conditions of
|
||||
this License, each Contributor hereby grants to You a perpetual,
|
||||
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
||||
copyright license to reproduce, prepare Derivative Works of,
|
||||
publicly display, publicly perform, sublicense, and distribute the
|
||||
Work and such Derivative Works in Source or Object form.
|
||||
|
||||
3. Grant of Patent License. Subject to the terms and conditions of
|
||||
this License, each Contributor hereby grants to You a perpetual,
|
||||
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
||||
(except as stated in this section) patent license to make, have made,
|
||||
use, offer to sell, sell, import, and otherwise transfer the Work,
|
||||
where such license applies only to those patent claims licensable
|
||||
by such Contributor that are necessarily infringed by their
|
||||
Contribution(s) alone or by combination of their Contribution(s)
|
||||
with the Work to which such Contribution(s) was submitted. If You
|
||||
institute patent litigation against any entity (including a
|
||||
cross-claim or counterclaim in a lawsuit) alleging that the Work
|
||||
or a Contribution incorporated within the Work constitutes direct
|
||||
or contributory patent infringement, then any patent licenses
|
||||
granted to You under this License for that Work shall terminate
|
||||
as of the date such litigation is filed.
|
||||
|
||||
4. Redistribution. You may reproduce and distribute copies of the
|
||||
Work or Derivative Works thereof in any medium, with or without
|
||||
modifications, and in Source or Object form, provided that You
|
||||
meet the following conditions:
|
||||
|
||||
(a) You must give any other recipients of the Work or
|
||||
Derivative Works a copy of this License; and
|
||||
|
||||
(b) You must cause any modified files to carry prominent notices
|
||||
stating that You changed the files; and
|
||||
|
||||
(c) You must retain, in the Source form of any Derivative Works
|
||||
that You distribute, all copyright, patent, trademark, and
|
||||
attribution notices from the Source form of the Work,
|
||||
excluding those notices that do not pertain to any part of
|
||||
the Derivative Works; and
|
||||
|
||||
(d) If the Work includes a "NOTICE" text file as part of its
|
||||
distribution, then any Derivative Works that You distribute must
|
||||
include a readable copy of the attribution notices contained
|
||||
within such NOTICE file, excluding those notices that do not
|
||||
pertain to any part of the Derivative Works, in at least one
|
||||
of the following places: within a NOTICE text file distributed
|
||||
as part of the Derivative Works; within the Source form or
|
||||
documentation, if provided along with the Derivative Works; or,
|
||||
within a display generated by the Derivative Works, if and
|
||||
wherever such third-party notices normally appear. The contents
|
||||
of the NOTICE file are for informational purposes only and
|
||||
do not modify the License. You may add Your own attribution
|
||||
notices within Derivative Works that You distribute, alongside
|
||||
or as an addendum to the NOTICE text from the Work, provided
|
||||
that such additional attribution notices cannot be construed
|
||||
as modifying the License.
|
||||
|
||||
You may add Your own copyright statement to Your modifications and
|
||||
may provide additional or different license terms and conditions
|
||||
for use, reproduction, or distribution of Your modifications, or
|
||||
for any such Derivative Works as a whole, provided Your use,
|
||||
reproduction, and distribution of the Work otherwise complies with
|
||||
the conditions stated in this License.
|
||||
|
||||
5. Submission of Contributions. Unless You explicitly state otherwise,
|
||||
any Contribution intentionally submitted for inclusion in the Work
|
||||
by You to the Licensor shall be under the terms and conditions of
|
||||
this License, without any additional terms or conditions.
|
||||
Notwithstanding the above, nothing herein shall supersede or modify
|
||||
the terms of any separate license agreement you may have executed
|
||||
with Licensor regarding such Contributions.
|
||||
|
||||
6. Trademarks. This License does not grant permission to use the trade
|
||||
names, trademarks, service marks, or product names of the Licensor,
|
||||
except as required for reasonable and customary use in describing the
|
||||
origin of the Work and reproducing the content of the NOTICE file.
|
||||
|
||||
7. Disclaimer of Warranty. Unless required by applicable law or
|
||||
agreed to in writing, Licensor provides the Work (and each
|
||||
Contributor provides its Contributions) on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
|
||||
implied, including, without limitation, any warranties or conditions
|
||||
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
|
||||
PARTICULAR PURPOSE. You are solely responsible for determining the
|
||||
appropriateness of using or redistributing the Work and assume any
|
||||
risks associated with Your exercise of permissions under this License.
|
||||
|
||||
8. Limitation of Liability. In no event and under no legal theory,
|
||||
whether in tort (including negligence), contract, or otherwise,
|
||||
unless required by applicable law (such as deliberate and grossly
|
||||
negligent acts) or agreed to in writing, shall any Contributor be
|
||||
liable to You for damages, including any direct, indirect, special,
|
||||
incidental, or consequential damages of any character arising as a
|
||||
result of this License or out of the use or inability to use the
|
||||
Work (including but not limited to damages for loss of goodwill,
|
||||
work stoppage, computer failure or malfunction, or any and all
|
||||
other commercial damages or losses), even if such Contributor
|
||||
has been advised of the possibility of such damages.
|
||||
|
||||
9. Accepting Warranty or Additional Liability. While redistributing
|
||||
the Work or Derivative Works thereof, You may choose to offer,
|
||||
and charge a fee for, acceptance of support, warranty, indemnity,
|
||||
or other liability obligations and/or rights consistent with this
|
||||
License. However, in accepting such obligations, You may act only
|
||||
on Your own behalf and on Your sole responsibility, not on behalf
|
||||
of any other Contributor, and only if You agree to indemnify,
|
||||
defend, and hold each Contributor harmless for any liability
|
||||
incurred by, or claims asserted against, such Contributor by reason
|
||||
of your accepting any such warranty or additional liability.
|
||||
|
||||
END OF TERMS AND CONDITIONS
|
||||
|
||||
APPENDIX: How to apply the Apache License to your work.
|
||||
|
||||
To apply the Apache License to your work, attach the following
|
||||
boilerplate notice, with the fields enclosed by brackets "[]"
|
||||
replaced with your own identifying information. (Don't include
|
||||
the brackets!) The text should be enclosed in the appropriate
|
||||
comment syntax for the file format. We also recommend that a
|
||||
file or class name and description of purpose be included on the
|
||||
same "printed page" as the copyright notice for easier
|
||||
identification within third-party archives.
|
||||
|
||||
Copyright 2025 NOX Project
|
||||
|
||||
Licensed under the Apache License, Version 2.0 (the "License");
|
||||
you may not use this file except in compliance with the License.
|
||||
You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing, software
|
||||
distributed under the License is distributed on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
See the License for the specific language governing permissions and
|
||||
limitations under the License.
|
||||
@@ -0,0 +1,502 @@
|
||||
<div align="center">
|
||||
|
||||
```
|
||||
███╗ ██╗ ██████╗ ██╗ ██╗
|
||||
████╗ ██║██╔═══██╗╚██╗██╔╝
|
||||
██╔██╗ ██║██║ ██║ ╚███╔╝
|
||||
██║╚██╗██║██║ ██║ ██╔██╗
|
||||
██║ ╚████║╚██████╔╝██╔╝ ██╗
|
||||
╚═╝ ╚═══╝ ╚═════╝ ╚═╝ ╚═╝
|
||||
```
|
||||
|
||||
**Cyber Threat Intelligence Framework**
|
||||
|
||||
[](https://github.com/nox-project/nox-framework/releases/tag/v1.0.0)
|
||||
[](https://www.python.org/)
|
||||
[](LICENSE.txt)
|
||||
[](https://www.kali.org/)
|
||||
[](https://github.com/nox-project/nox-framework)
|
||||
[](https://github.com/nox-project/nox-framework)
|
||||
|
||||
*OSINT framework for red teaming, digital forensics, and corporate exposure analysis.*
|
||||
|
||||
</div>
|
||||
|
||||
---
|
||||
|
||||
## Introduction
|
||||
|
||||
NOX is a purpose-built cyber threat intelligence engine designed for operators who require speed, operational security, and depth in a single cohesive framework. It is not a wrapper around existing tools — it is a fully async, plugin-driven intelligence platform with a strict separation between execution logic and source definitions.
|
||||
|
||||
| Capability | Detail |
|
||||
|-|-|
|
||||
| ⚡ **Async Execution Engine** | Massively parallel scanning across 124 intelligence feeds with no sequential bottlenecks and no blocking I/O. |
|
||||
| 🛡️ **Guardian Engine** | Integrated OPSEC layer with automatic proxy rotation and SOCKS5 support. Fail-safe kill-switch halts all traffic if the transport circuit is unavailable. |
|
||||
| 🧠 **Risk Scoring** | Dynamic 0–100 scoring with time-decay, source confidence weighting, password complexity analysis, persistence multipliers, and HVT detection. |
|
||||
| 🔗 **Recursive Avalanche Engine** | Every discovered asset — username, email, cracked password, phone — is automatically re-injected as a new scan seed. Per-asset pipeline runs sequentially (breach → crack → dork → scrape); child assets run concurrently. Identifiers from all four phases feed the pivot queue. Global deduplication and configurable depth cap prevent runaway recursion. |
|
||||
| 🔍 **Autoscan** | Single command triggers breach scan + recursive pivot + dorking + paste scraping — fully automated, no manual chaining. |
|
||||
|
||||
---
|
||||
|
||||
## Features
|
||||
|
||||
| Feature | Description |
|
||||
|-|-|
|
||||
| **124 JSON Plugin Sources** | Every intelligence source is a JSON plugin. The execution engine contains zero hardcoded source logic. |
|
||||
| **Async Core** | Full `asyncio` event loop with JA3 fingerprinting, SSL session management, per-request jitter, and configurable concurrency. |
|
||||
| **Autoscan Pipeline** | `--autoscan` triggers: breach scan → recursive pivot → Google/Bing/DDG dorking → paste/Telegram scraping — all in one command. |
|
||||
| **Recursive Avalanche Engine** | Every identifier discovered — from breach records, dork hits, or scraped paste/Telegram content — is re-injected as a new seed. Per-asset pipeline is sequential (breach → crack → dork → scrape); child assets run concurrently via `asyncio.gather`. A global `seen_assets` set prevents infinite loops. Concurrency and depth are fully configurable at runtime via `--threads` and `--depth`. |
|
||||
| **Hash Pivoting** | Hashes found in breach data are automatically identified (MD5/SHA1/SHA256/NTLM/bcrypt) and cracked via concurrent background API queries. Cracked plaintexts are injected into the pivot queue as password-recycling seeds. Failures are logged silently — the scan never stops. |
|
||||
| **Guardian Proxy Engine** | Zero-config OPSEC layer: reads `proxies.txt` if present; otherwise auto-fetches and validates a high-anonymity proxy pool in-memory. Full SOCKS5/HTTP/S and Tor support. |
|
||||
| **API Key Rotation** | `api_key_slots` per source — NOX round-robins across multiple keys to bypass per-key rate limits. |
|
||||
| **Identity Graphing** | Union-Find correlation engine unifies breach records into identity clusters across all sources, using type-aware pivot classification. |
|
||||
| **Enterprise Forensic Reports** | Professional PDF/HTML/JSON/CSV/Markdown reports with Executive Summary dashboard (Total Time, Nodes Discovered, Cleartext Passwords, Pivot Depth), interactive Pivot Chain Visualization, and strict data sanitization — no technical noise in output. JSON exports are self-describing with a full metadata block. |
|
||||
| **HVT Detection** | Auto-flags C-level, Admin, DevOps, and government domain accounts as High-Value Targets. |
|
||||
| **Dorking Engine** | Passive document discovery via Google/Bing/DDG dorks with PDF/Office metadata extraction. |
|
||||
| **Scraping Engine** | Paste site indexing, Telegram CTI channel monitoring, credential extraction, and misconfiguration discovery. Each autoscan asset gets a dedicated scrape session — no shared state. |
|
||||
| **Proxy / Tor** | SOCKS5, HTTP/S proxy, full Tor routing via `stem`, and automatic Guardian fallback. SOCKS5 proxies are validated and routed correctly via `aiohttp-socks`. |
|
||||
| **Secure Key Store** | API keys managed via `~/.config/nox-cli/apikeys.json` (chmod 0600). Unconfigured keys are silently skipped. Keys set via environment variable are picked up automatically without restarting. |
|
||||
| **System Logging** | All scan events, phase completions, pivot discoveries, API events, rate-limits, and crack attempts are written to `~/.nox/logs/nox.log`. Only actionable intelligence reaches the terminal. |
|
||||
| **Plugin Debug** | `--list-sources` prints a full operator debug table: plugin name, input type, confidence score, key status (configured / not configured / public), and any JSON parse errors. |
|
||||
|
||||
---
|
||||
|
||||
## Architecture
|
||||
|
||||
### Plugin-Driven Design
|
||||
|
||||
NOX operates on a strict separation of concerns: `nox.py` is a **pure, agnostic execution engine** — it handles async I/O, JA3 fingerprinting, SSL session management, recursive pivoting, and result correlation. It contains no hardcoded intelligence logic.
|
||||
|
||||
All intelligence is defined as **JSON plugins** in `sources/`. These plugins are the sole source of truth for what NOX queries, how it authenticates, and what it extracts. The build tool `build_sources.py` is the only authorised way to create or modify them.
|
||||
|
||||
```
|
||||
build_sources.py ──► sources/*.json ──► nox.py (runtime loader)
|
||||
[Builder] [Plugins] [Execution Engine]
|
||||
```
|
||||
|
||||
> [!IMPORTANT]
|
||||
> **`sources/*.json` files are auto-generated artifacts. Never edit them directly.**
|
||||
> All source additions and modifications must be made in `build_sources.py` and applied by running `python build_sources.py`. Manual edits will be overwritten on the next build.
|
||||
|
||||
#### Source Schema
|
||||
|
||||
```json
|
||||
{
|
||||
"name": "MyPrivateDB",
|
||||
"endpoint": "https://api.myprivatedb.com/search?q={target}",
|
||||
"method": "GET",
|
||||
"headers": { "Authorization": "Bearer {MY_API_KEY}" },
|
||||
"regex_pattern": "([\\w.+-]+@[\\w-]+\\.[\\w.]+):([\\S]+)",
|
||||
"required_api_key_name": "MY_API_KEY",
|
||||
"api_key_slots": ["{MY_API_KEY}"],
|
||||
"input_type": "email",
|
||||
"output_type": ["username", "ip"],
|
||||
"pivot_types": ["email", "username"],
|
||||
"confidence": 0.9
|
||||
}
|
||||
```
|
||||
|
||||
Supported fields: `name`, `endpoint`, `method`, `headers`, `regex_pattern` (or `json_root` + `normalization_map`), `required_api_key_name`, `api_key_slots`, `input_type`, `output_type`, `pivot_types`, `confidence`.
|
||||
|
||||
---
|
||||
|
||||
### Autoscan Pipeline
|
||||
|
||||
`--autoscan` (CLI) / `autoscan` (REPL) executes the full intelligence pipeline in a single command:
|
||||
|
||||
```
|
||||
For each asset (seed + every discovered identifier):
|
||||
├─ Phase 1 — Breach Scan
|
||||
│ 124 sources queried in parallel (async)
|
||||
│
|
||||
├─ Phase 2 — Hash Crack (non-blocking, concurrent)
|
||||
│ Hashes found in breach data → rainbow-table APIs → cracked plaintext
|
||||
│ → password-recycling breach scan
|
||||
│
|
||||
├─ Phase 3 — Dorking
|
||||
│ Google/Bing/DDG dorks → leaked docs, .env files, SQL dumps
|
||||
│ → new identifiers extracted and re-injected
|
||||
│
|
||||
└─ Phase 4 — Scraping
|
||||
Pastebin, IntelX, Telegram CTI channels → credential extraction
|
||||
→ new identifiers extracted and re-injected
|
||||
|
||||
All identifiers discovered in phases 1–4 are re-injected as new seeds.
|
||||
Child assets are processed concurrently via asyncio.gather.
|
||||
```
|
||||
|
||||
`scan` (without `--autoscan`) runs Phase 1 only — breach sources, no pivot/dork/scrape.
|
||||
|
||||
---
|
||||
|
||||
### Recursive Avalanche Engine
|
||||
|
||||
Every identifier discovered during a scan — from breach records, dork hits, or scraped paste/Telegram content — is treated as a new intelligence seed. For each asset, the engine runs four phases sequentially: breach scan → hash crack → dork → scrape. Identifiers extracted from **all four phases** are harvested and re-injected as new seeds. Child assets are then processed concurrently via `asyncio.gather`.
|
||||
|
||||
```
|
||||
target@company.com
|
||||
└─► [Breach] username: j.doe ──► [Breach + Crack + Dork + Scrape]
|
||||
│ └─► github.com/jdoe ──► [Breach + Crack + Dork + Scrape]
|
||||
└─► [Breach] hash: 5f4dcc... ──► [AutoCrack] → "password123"
|
||||
│ └─► [Breach] password-recycling scan across all sources
|
||||
└─► [Dork] new@email.com ──► [Breach + Crack + Dork + Scrape]
|
||||
└─► [Scrape/paste] admin@corp.com ──► [Breach + Crack + Dork + Scrape]
|
||||
```
|
||||
|
||||
- **`seen_assets` set** — global deduplication; no identifier is ever processed twice, regardless of which phase discovered it
|
||||
- **Global semaphore** — single shared concurrency cap across the entire discovery tree, respecting `--threads`
|
||||
- **`--depth N`** — configurable pivot depth (default: 2); hard backstop prevents runaway recursion
|
||||
- **`--no-pivot`** — disable recursive enrichment for a fast breach-only scan
|
||||
|
||||
---
|
||||
|
||||
### Hash Pivoting
|
||||
|
||||
When a hash is found in breach data during `--autoscan`:
|
||||
|
||||
1. Hash type is identified (MD5/NTLM, SHA1, SHA256, bcrypt)
|
||||
2. Multiple rainbow-table APIs are queried **concurrently** in a background task
|
||||
3. **If cracked** — plaintext is logged, the record is updated, and the password is injected into the pivot queue for password-recycling analysis across all breach sources
|
||||
4. **If not cracked** — failure is logged to `nox_system.log`, the hash is preserved in the report, and pivoting on all other assets continues immediately
|
||||
|
||||
The crack process is fully non-blocking. A timeout or API failure never pauses the scan. Use `--no-online-crack` to restrict cracking to the local wordlist only (no data sent to third-party APIs).
|
||||
|
||||
---
|
||||
|
||||
### Guardian Proxy Engine
|
||||
|
||||
The Guardian Engine is NOX's zero-config OPSEC layer. It activates automatically when no `--proxy` or `--tor` flag is supplied.
|
||||
|
||||
**Resolution order:**
|
||||
|
||||
1. **`proxies.txt`** — if present in the working directory, NOX loads and rotates through the listed proxies.
|
||||
2. **Dynamic fetch** — if `proxies.txt` is absent, the Guardian Engine fetches a fresh list of high-anonymity public proxies, validates each one, and holds the validated pool in-memory for the session. Nothing is written to disk.
|
||||
3. **Direct connection** — if no valid proxies are found, NOX falls back to a direct connection and emits a warning.
|
||||
|
||||
> [!WARNING]
|
||||
> Public proxy pools are inherently untrusted infrastructure. For sensitive engagements, always supply a controlled proxy via `--proxy` or route through Tor via `--tor`.
|
||||
|
||||
| Flag | Behaviour |
|
||||
|-|-|
|
||||
| `--proxy <url>` | Route all traffic through the specified HTTP/S or SOCKS5 proxy. Disables Guardian. |
|
||||
| `--tor` | Route all traffic through Tor (requires `tor` service on port 9050). Disables Guardian. |
|
||||
| `--guardian-off` | Bypass the OPSEC kill-switch and connect directly. |
|
||||
| *(no flag)* | Guardian Engine activates automatically. |
|
||||
|
||||
---
|
||||
|
||||
### Reporting
|
||||
|
||||
All report formats include an **Executive Summary dashboard**:
|
||||
|
||||
| Metric | Description |
|
||||
|-|-|
|
||||
| Total Time | Wall-clock duration of the full scan |
|
||||
| Nodes Discovered | Unique identities surfaced across all sources |
|
||||
| Cleartext Passwords | Plaintext credentials found or cracked |
|
||||
| Pivot Depth | Depth reached by the recursive avalanche engine |
|
||||
|
||||
Reports also include a **Pivot Chain Visualization** showing the full relational path from initial seed to final discovery:
|
||||
|
||||
```
|
||||
[seed@corp.com] -> [LeakA / username:jdoe] -> [Dork: leaked .env] -> [new@email.com]
|
||||
```
|
||||
|
||||
JSON exports include a `_meta` block with `scan_id`, `target`, `timestamp`, `nox_version`, and `pivot_depth_reached` — making every export self-describing for ingestion into case management platforms.
|
||||
|
||||
All output is sanitized — proxy errors, timeouts, and tracebacks are stripped. Only actionable intelligence is included.
|
||||
|
||||
---
|
||||
|
||||
## Filesystem Layout
|
||||
|
||||
```
|
||||
~/.nox/
|
||||
├── sources/ # Auto-generated JSON source plugins
|
||||
├── reports/ # Generated forensic reports
|
||||
├── logs/ # Runtime log (nox.log)
|
||||
├── wordlists/ # Hash cracking wordlists
|
||||
├── vault/ # Secure storage
|
||||
└── nox_cache.db # Forensic persistence database (SQLite)
|
||||
|
||||
~/.config/nox-cli/
|
||||
├── apikeys.json # API keys — chmod 0600, never committed to VCS
|
||||
└── logs/
|
||||
└── nox_system.log # Silent system log: API events, rate-limits, crack attempts
|
||||
|
||||
# .deb install (isolated venv)
|
||||
/opt/nox-cli/
|
||||
├── nox.py
|
||||
├── build_sources.py
|
||||
├── requirements.txt
|
||||
├── sources/
|
||||
└── .venv/ # Isolated Python environment (PEP 668 compliant)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- **Python 3.8+**
|
||||
- **pip** (`python3-pip` on Debian/Kali)
|
||||
- **Tor** *(optional)* — required only for `--tor`. On Kali: `sudo apt install tor -y`. The `tor` service must be running on port `9050`.
|
||||
|
||||
---
|
||||
|
||||
## Installation
|
||||
|
||||
### Option 1: Debian / Kali Linux — Isolated .deb (Recommended)
|
||||
|
||||
Download the `.deb` package from the [Releases page](https://github.com/nox-project/nox-framework/releases), then run:
|
||||
|
||||
```bash
|
||||
sudo dpkg -i nox-cli_*_all.deb
|
||||
nox-cli --help
|
||||
```
|
||||
|
||||
The post-install script automatically:
|
||||
1. Creates an isolated virtual environment at `/opt/nox-cli/.venv`
|
||||
2. Installs all Python dependencies inside the venv (PEP 668 compliant — zero system pollution)
|
||||
3. Builds the 124 source plugins
|
||||
4. Links `/usr/bin/nox-cli` → `/opt/nox-cli/nox-wrapper.sh`
|
||||
|
||||
### Option 2: From Source
|
||||
|
||||
```bash
|
||||
git clone https://github.com/nox-project/nox-framework.git
|
||||
cd nox-framework
|
||||
pip install -r requirements.txt
|
||||
python build_sources.py
|
||||
python3 nox.py
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Quick Start
|
||||
|
||||
**Step 1 — Build source plugins** *(from source only — .deb does this automatically)*
|
||||
|
||||
```bash
|
||||
python build_sources.py
|
||||
```
|
||||
|
||||
**Step 2 — Configure API keys**
|
||||
|
||||
`build_sources.py` creates `~/.config/nox-cli/apikeys.json` on first run, pre-populated with every supported service. The file is `chmod 0600` and is never committed to VCS.
|
||||
|
||||
This is the **single canonical key store** — all sources read from it at runtime.
|
||||
|
||||
```bash
|
||||
# Edit the file directly
|
||||
nano ~/.config/nox-cli/apikeys.json
|
||||
|
||||
# Or inspect plugin status and key configuration
|
||||
nox-cli --list-sources
|
||||
```
|
||||
|
||||
> [!NOTE]
|
||||
> Any key set to `INSERT_API_KEY_HERE` or `""` is treated as unconfigured — that source is silently skipped. Sources without a key requirement are always active.
|
||||
>
|
||||
> **Load priority:** environment variable (e.g. `export HIBP_API_KEY=xxx`) → `~/.config/nox-cli/apikeys.json`
|
||||
|
||||
**Step 3 — Execute**
|
||||
|
||||
> [!NOTE]
|
||||
> **OPSEC Kill-Switch:** By default, NOX activates the Guardian Engine (auto proxy rotation). Use `--guardian-off` to connect directly.
|
||||
|
||||
```bash
|
||||
# Breach scan — input type auto-detected (email / domain / ip / username / hash / phone)
|
||||
nox-cli -t target@company.com
|
||||
|
||||
# Full autoscan: breach + recursive pivot + dork + scrape
|
||||
nox-cli -t target@company.com --autoscan
|
||||
|
||||
# Autoscan with Tor routing
|
||||
nox-cli -t target@company.com --autoscan --tor
|
||||
|
||||
# Autoscan with SOCKS5 proxy + PDF report
|
||||
nox-cli -t target@company.com --autoscan --proxy socks5://127.0.0.1:1080 -o report.pdf --format pdf
|
||||
|
||||
# Autoscan with custom pivot depth
|
||||
nox-cli -t target@company.com --autoscan --depth 3
|
||||
|
||||
# Breach scan only — no pivot, no dork, no scrape
|
||||
nox-cli -t target@company.com --no-pivot
|
||||
|
||||
# Domain scan
|
||||
nox-cli -t company.com
|
||||
|
||||
# Hash identification and cracking
|
||||
nox-cli --crack 5f4dcc3b5aa765d61d8327deb882cf99
|
||||
|
||||
# Hash cracking — local wordlist only, no third-party API calls
|
||||
nox-cli --crack 5f4dcc3b5aa765d61d8327deb882cf99 --no-online-crack
|
||||
|
||||
# Password strength analysis
|
||||
nox-cli --analyze "P@ssw0rd123"
|
||||
|
||||
# Google dorking
|
||||
nox-cli --dork target@company.com
|
||||
|
||||
# Paste / Telegram scraping
|
||||
nox-cli --scrape target@company.com
|
||||
|
||||
# Compare scan against last cached result — show only new findings
|
||||
nox-cli -t target@company.com --diff
|
||||
|
||||
# Plugin debug: loaded sources, input types, confidence, key status
|
||||
nox-cli --list-sources
|
||||
|
||||
# Force resync of source plugins from package
|
||||
nox-cli --reset-sources
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## CLI Reference
|
||||
|
||||
```
|
||||
usage: nox-cli [-h] [-t TARGET] [-i] [--version]
|
||||
[--autoscan] [--fullscan] [--no-pivot] [--depth N]
|
||||
[--dork TARGET] [--scrape TARGET]
|
||||
[--crack HASH] [--no-online-crack]
|
||||
[--analyze PASS] [--list-sources] [--reset-sources]
|
||||
[--tor] [--proxy URL] [--guardian-off] [--allow-leak]
|
||||
[--threads N] [--timeout N]
|
||||
[-o FILE] [--format {json,csv,html,md,pdf}]
|
||||
[--diff]
|
||||
|
||||
-t, --target TARGET Target to scan (auto-detected type)
|
||||
-i, --interactive Launch interactive REPL
|
||||
--version Show version and exit
|
||||
--autoscan Full pipeline: breach + pivot + dork + scrape
|
||||
--fullscan Breach + pivot only (no dork/scrape)
|
||||
--no-pivot Disable recursive pivot enrichment
|
||||
--depth N Avalanche pivot depth (default: 2)
|
||||
--dork TARGET Google/Bing/DDG dorking for leaked documents
|
||||
--scrape TARGET Paste site + Telegram scraping
|
||||
--crack HASH Identify and crack a hash
|
||||
--no-online-crack Local wordlist only — no data sent to third-party APIs
|
||||
--analyze PASS Deep password strength analysis
|
||||
--list-sources Plugin debug: input type, confidence, key status
|
||||
--reset-sources Force resync of source plugins from package
|
||||
--tor Route all traffic through Tor (port 9050)
|
||||
--proxy URL HTTP/S or SOCKS5 proxy URL
|
||||
--guardian-off Bypass OPSEC kill-switch (direct connection)
|
||||
--allow-leak Allow direct connection if proxy/Tor is unavailable
|
||||
--threads N Concurrency limit (default: 20)
|
||||
--timeout N Request timeout in seconds (default: 15)
|
||||
-o, --output FILE Output file path
|
||||
--format FORMAT Output format: json, csv, html, md, pdf
|
||||
--diff Show only new findings vs last cached scan
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## REPL
|
||||
|
||||
Launch the interactive REPL with no arguments:
|
||||
|
||||
```bash
|
||||
nox-cli
|
||||
```
|
||||
|
||||
```
|
||||
Command Description
|
||||
----------- ---------------------------------------------------------------
|
||||
autoscan Full pipeline: breach + pivot + dork + scrape
|
||||
scan Breach intelligence scan only
|
||||
dork Google/Bing/DDG dorking for leaked documents
|
||||
scrape Paste site + Telegram scraping
|
||||
crack Identify and crack a hash
|
||||
analyze Deep password strength analysis
|
||||
graph ASCII identity graph of last scan
|
||||
visualize ASCII relationship map (Target → Data → Pivots)
|
||||
pivot <n> Re-scan using result #n as new pivot seed
|
||||
search <q> Filter in-memory records by keyword
|
||||
sources Plugin debug: input type, confidence, key status
|
||||
export Export results (json / csv / html / md / pdf)
|
||||
tor Toggle Tor routing on/off
|
||||
proxy Set or clear proxy URL
|
||||
config Configure threads / timeout / depth
|
||||
help Show this menu
|
||||
quit Exit NOX
|
||||
```
|
||||
|
||||
**Examples:**
|
||||
|
||||
```
|
||||
nox> autoscan target@company.com
|
||||
nox> graph
|
||||
nox> visualize
|
||||
nox> pivot 3
|
||||
nox> search admin
|
||||
nox> export pdf investigation.pdf
|
||||
nox> sources
|
||||
nox> config threads 30
|
||||
nox> config depth 3
|
||||
nox> proxy socks5://127.0.0.1:1080
|
||||
nox> tor
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Source Management
|
||||
|
||||
### Adding a Source
|
||||
|
||||
**1. Define in `build_sources.py`:**
|
||||
|
||||
```python
|
||||
_auth("NewIntelDB", "breaches",
|
||||
"https://api.newinteldb.com/v1/search?q={target}", "GET",
|
||||
{"results": "$.results"},
|
||||
headers={"X-API-Key": "{NEWINTELDB_API_KEY}"},
|
||||
api_key_slots=["{NEWINTELDB_API_KEY}"],
|
||||
normalization_map={"email": "email", "password": "password"},
|
||||
input_type="email",
|
||||
output_type=["username", "ip"],
|
||||
confidence=0.85)
|
||||
```
|
||||
|
||||
**2. Rebuild:**
|
||||
|
||||
```bash
|
||||
python build_sources.py
|
||||
```
|
||||
|
||||
> [!NOTE]
|
||||
> The builder validates every source at build time: GET endpoints must contain `{target}`, volatile sources must have `reliability_score ≤ 4`, and the `confidence` field can be set explicitly to override the formula-derived value.
|
||||
|
||||
---
|
||||
|
||||
## Building the .deb Package
|
||||
|
||||
```bash
|
||||
gem install fpm
|
||||
bash build_deb.sh
|
||||
sudo dpkg -i dist/nox-cli_*_all.deb
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Legal Disclaimer
|
||||
|
||||
> [!WARNING]
|
||||
> **NOX is intended exclusively for:**
|
||||
> - Authorised penetration testing and red team engagements with explicit written consent
|
||||
> - Corporate exposure analysis on assets you own or are contracted to assess
|
||||
> - Digital forensics and incident response
|
||||
> - Academic and security research in controlled, isolated environments
|
||||
>
|
||||
> **Unauthorised use of this tool against systems, networks, or individuals without explicit written permission is a criminal offence** under the Computer Fraud and Abuse Act (CFAA, 18 U.S.C. § 1030), the Computer Misuse Act 1990 (CMA), and equivalent legislation in all major jurisdictions worldwide.
|
||||
>
|
||||
> The authors and contributors of NOX accept **no liability** for any direct, indirect, incidental, or consequential damages arising from misuse of this software. By downloading, installing, or executing NOX, you unconditionally agree to comply with all applicable local, national, and international laws, and to only target systems and data for which you hold explicit, documented authorisation.
|
||||
>
|
||||
> **If you do not agree to these terms, do not use this software.**
|
||||
|
||||
---
|
||||
|
||||
## License
|
||||
|
||||
[Apache License 2.0](LICENSE.txt)
|
||||
+34
@@ -0,0 +1,34 @@
|
||||
# Security Policy
|
||||
|
||||
## Supported Versions
|
||||
|
||||
| Version | Supported |
|
||||
|---------|-----------|
|
||||
| 1.0.x | ✅ Active |
|
||||
|
||||
## Reporting a Vulnerability
|
||||
|
||||
Report security vulnerabilities **privately** — do not open a public issue.
|
||||
|
||||
**Contact:** open a [GitHub Security Advisory](https://github.com/nox-project/nox-framework/security/advisories/new)
|
||||
|
||||
Include:
|
||||
- A clear description of the vulnerability
|
||||
- Steps to reproduce
|
||||
- Potential impact
|
||||
- Suggested fix (if any)
|
||||
|
||||
You will receive an acknowledgement within 48 hours. Critical vulnerabilities are patched within 7 days.
|
||||
|
||||
## Scope
|
||||
|
||||
In-scope:
|
||||
- Remote code execution via crafted source plugin or API response
|
||||
- Credential leakage from the vault or apikeys.json
|
||||
- OPSEC bypass (real IP exposure when proxy/Tor is configured)
|
||||
- Dependency vulnerabilities with a direct exploit path
|
||||
|
||||
Out of scope:
|
||||
- Issues requiring physical access to the machine
|
||||
- Social engineering
|
||||
- Vulnerabilities in third-party APIs queried by NOX
|
||||
Executable
+41
@@ -0,0 +1,41 @@
|
||||
#!/usr/bin/env bash
|
||||
set -e
|
||||
|
||||
# NOX v1.0.0 — .deb build script (FPM)
|
||||
# Requires: fpm → gem install fpm
|
||||
|
||||
VERSION="1.0.0"
|
||||
PKG_NAME="nox-cli"
|
||||
ARCH="all"
|
||||
OUT_DIR="dist"
|
||||
|
||||
command -v fpm &>/dev/null || { echo "[!] fpm not found: gem install fpm" >&2; exit 1; }
|
||||
|
||||
mkdir -p "$OUT_DIR"
|
||||
echo "[*] Building ${PKG_NAME}_${VERSION}_${ARCH}.deb ..."
|
||||
|
||||
fpm \
|
||||
--input-type dir \
|
||||
--output-type deb \
|
||||
--name "$PKG_NAME" \
|
||||
--version "$VERSION" \
|
||||
--architecture "$ARCH" \
|
||||
--maintainer "nox-project <nox-project@users.noreply.github.com>" \
|
||||
--description "NOX — Cyber Threat Intelligence Framework — 120+ async breach sources, pivot engine, HVT detection" \
|
||||
--url "https://github.com/nox-project/nox-framework" \
|
||||
--license "Apache-2.0" \
|
||||
--depends "python3" \
|
||||
--depends "python3-venv" \
|
||||
--depends "python3-pip" \
|
||||
--after-install postinst.sh \
|
||||
--package "${OUT_DIR}/${PKG_NAME}_${VERSION}_${ARCH}.deb" \
|
||||
--force \
|
||||
nox.py=/opt/nox-cli/nox.py \
|
||||
build_sources.py=/opt/nox-cli/build_sources.py \
|
||||
requirements.txt=/opt/nox-cli/requirements.txt \
|
||||
sources/=/opt/nox-cli/sources/ \
|
||||
sources/helpers/=/opt/nox-cli/sources/helpers/ \
|
||||
nox-wrapper.sh=/opt/nox-cli/nox-wrapper.sh \
|
||||
docs/nox-cli.1=/usr/share/man/man1/nox-cli.1
|
||||
|
||||
echo "[+] Built: ${OUT_DIR}/${PKG_NAME}_${VERSION}_${ARCH}.deb"
|
||||
+1341
File diff suppressed because it is too large
Load Diff
Vendored
+5
@@ -0,0 +1,5 @@
|
||||
nox-cli (1.0.0-1) kali-dev; urgency=low
|
||||
|
||||
* Initial release to Kali Linux.
|
||||
|
||||
-- nox-project <nox-project@users.noreply.github.com> Thu, 02 Apr 2026 20:00:00 +0200
|
||||
Vendored
+35
@@ -0,0 +1,35 @@
|
||||
Source: nox-cli
|
||||
Section: net
|
||||
Priority: optional
|
||||
Maintainer: nox-project <nox-project@users.noreply.github.com>
|
||||
Build-Depends: debhelper-compat (= 13), dh-python, python3-all, python3-setuptools
|
||||
Standards-Version: 4.6.2
|
||||
Rules-Requires-Root: no
|
||||
Homepage: https://github.com/nox-project/nox-framework
|
||||
|
||||
Package: nox-cli
|
||||
Architecture: all
|
||||
Depends: ${python3:Depends}, ${misc:Depends},
|
||||
python3-requests,
|
||||
python3-aiohttp,
|
||||
python3-pydantic,
|
||||
python3-colorama,
|
||||
python3-rich,
|
||||
python3-bs4,
|
||||
python3-lxml,
|
||||
python3-dnspython,
|
||||
python3-phonenumbers,
|
||||
python3-aiosqlite | python3-pip
|
||||
Recommends:
|
||||
python3-stem,
|
||||
tor
|
||||
Description: Advanced Asynchronous Cyber Threat Intelligence Framework.
|
||||
nox-cli is an open-source OSINT and breach intelligence framework
|
||||
supporting 120+ JSON-plugin data sources. It performs asynchronous
|
||||
multi-source lookups against email addresses, domains, IP addresses,
|
||||
usernames, phone numbers, and hashes. Features include recursive
|
||||
identity pivoting, risk scoring, HVT detection, dorking, scraping,
|
||||
hash cracking, and forensic PDF reporting.
|
||||
.
|
||||
All Python dependencies are installed inside an isolated virtual
|
||||
environment at /opt/nox-cli/.venv (PEP 668 compliant).
|
||||
Vendored
+24
@@ -0,0 +1,24 @@
|
||||
Format: https://www.debian.org/doc/packaging-manuals/copyright-format/1.0/
|
||||
Upstream-Name: nox-cli
|
||||
Upstream-Contact: nox-project <nox-project@users.noreply.github.com>
|
||||
Source: https://github.com/nox-project/nox-framework
|
||||
|
||||
Files: *
|
||||
Copyright: 2024-2026 nox-project
|
||||
License: Apache-2.0
|
||||
|
||||
License: Apache-2.0
|
||||
Licensed under the Apache License, Version 2.0 (the "License");
|
||||
you may not use this file except in compliance with the License.
|
||||
You may obtain a copy of the License at
|
||||
.
|
||||
https://www.apache.org/licenses/LICENSE-2.0
|
||||
.
|
||||
Unless required by applicable law or agreed to in writing, software
|
||||
distributed under the License is distributed on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
See the License for the specific language governing permissions and
|
||||
limitations under the License.
|
||||
.
|
||||
On Debian systems, the full text of the Apache License, Version 2.0
|
||||
can be found in the file /usr/share/common-licenses/Apache-2.0.
|
||||
Vendored
+1
@@ -0,0 +1 @@
|
||||
sources/*.json usr/share/nox-cli/sources
|
||||
+11
@@ -0,0 +1,11 @@
|
||||
#!/bin/sh
|
||||
set -e
|
||||
if [ "$1" = "purge" ]; then
|
||||
rm -rf /opt/nox-cli
|
||||
rm -rf /root/.nox /root/.config/nox-cli
|
||||
# Also clean for the invoking user if SUDO_USER is set
|
||||
if [ -n "$SUDO_USER" ]; then
|
||||
UHOME=$(getent passwd "$SUDO_USER" | cut -d: -f6)
|
||||
rm -rf "${UHOME}/.nox" "${UHOME}/.config/nox-cli"
|
||||
fi
|
||||
fi
|
||||
+3
@@ -0,0 +1,3 @@
|
||||
#!/bin/sh
|
||||
set -e
|
||||
rm -f /usr/bin/nox-cli
|
||||
+3
@@ -0,0 +1,3 @@
|
||||
#!/usr/bin/make -f
|
||||
%:
|
||||
dh $@ --with python3 --buildsystem=pybuild
|
||||
Vendored
+3
@@ -0,0 +1,3 @@
|
||||
Tests: smoke
|
||||
Depends: @
|
||||
Restrictions: allow-stderr
|
||||
+5
@@ -0,0 +1,5 @@
|
||||
#!/bin/sh
|
||||
set -e
|
||||
nox-cli --help > /dev/null
|
||||
nox-cli --version > /dev/null
|
||||
echo "smoke: OK"
|
||||
+109
@@ -0,0 +1,109 @@
|
||||
.TH NOX\-CLI 1 "2026-03-30" "1.0.0" "NOX Framework"
|
||||
.SH NAME
|
||||
nox-cli \- Advanced Asynchronous Cyber Threat Intelligence Framework
|
||||
.SH SYNOPSIS
|
||||
.B nox-cli
|
||||
[\fIOPTIONS\fR]
|
||||
.SH DESCRIPTION
|
||||
.B nox-cli
|
||||
is an open-source OSINT and breach intelligence framework supporting 120+
|
||||
JSON-plugin data sources. It performs asynchronous multi-source lookups
|
||||
against email addresses, domains, IP addresses, usernames, phone numbers,
|
||||
and hashes. Results can be exported in JSON, CSV, HTML, Markdown, or PDF.
|
||||
.PP
|
||||
On first run, the application creates \fI~/.nox/\fR with a default
|
||||
\fIconfig.ini\fR and seeds the sources directory from the package data.
|
||||
.SH OPTIONS
|
||||
.TP
|
||||
.BR \-t ", " \-\-target " " \fITARGET\fR
|
||||
Target to scan (email, domain, IP, username, phone, or hash).
|
||||
.TP
|
||||
.BR \-i ", " \-\-interactive
|
||||
Launch the interactive REPL shell.
|
||||
.TP
|
||||
.BR \-\-version
|
||||
Print version number and exit.
|
||||
.TP
|
||||
.BR \-\-autoscan
|
||||
Full pipeline: breach scan + recursive identity pivot + dorking + paste/Telegram scraping.
|
||||
Equivalent to running all phases in sequence on the target and every discovered asset.
|
||||
.TP
|
||||
.BR \-\-fullscan
|
||||
Full scan including pivot enrichment, dorking, and scraping (alias for \-\-autoscan).
|
||||
.TP
|
||||
.BR \-\-no\-pivot
|
||||
Disable recursive pivot enrichment during a full scan.
|
||||
.TP
|
||||
.BR \-\-dork " " \fITARGET\fR
|
||||
Run Google dorking against the specified target.
|
||||
.TP
|
||||
.BR \-\-scrape " " \fITARGET\fR
|
||||
Run web scraping and Telegram indexing against the specified target.
|
||||
.TP
|
||||
.BR \-\-crack " " \fIHASH\fR
|
||||
Attempt to crack the given hash using online rainbow-table APIs and local wordlists.
|
||||
.B WARNING:
|
||||
submitting hashes to online APIs leaks them to third-party services.
|
||||
Use \fB\-\-no\-online\-crack\fR to restrict cracking to local wordlists only.
|
||||
.TP
|
||||
.BR \-\-no\-online\-crack
|
||||
Disable all online rainbow-table API queries during hash cracking.
|
||||
Only local wordlist-based cracking is performed. No hash data is sent to
|
||||
external services. Recommended for sensitive engagements.
|
||||
.TP
|
||||
.BR \-\-analyze " " \fIPASSWORD\fR
|
||||
Analyze a password for strength and breach exposure.
|
||||
.TP
|
||||
.BR \-\-apikeys
|
||||
Show the API key configuration dashboard. Displays configured and unconfigured
|
||||
keys for all supported services.
|
||||
.TP
|
||||
.BR \-\-allow\-leak
|
||||
Bypass the fail-safe OPSEC kill-switch and allow direct connections even when
|
||||
a proxy or Tor circuit is unavailable. Use only in controlled environments.
|
||||
.TP
|
||||
.BR \-\-tor
|
||||
Route all requests through the local Tor SOCKS proxy (port 9050).
|
||||
.TP
|
||||
.BR \-\-proxy " " \fIURL\fR
|
||||
Use the specified proxy URL for all requests.
|
||||
.TP
|
||||
.BR \-\-threads " " \fIN\fR
|
||||
Maximum concurrency level (default: 20).
|
||||
.TP
|
||||
.BR \-\-timeout " " \fISECONDS\fR
|
||||
Per-request timeout in seconds (default: 15).
|
||||
.TP
|
||||
.BR \-o ", " \-\-output " " \fIFILE\fR
|
||||
Write results to the specified output file.
|
||||
.TP
|
||||
.BR \-\-format " " \fI{json,csv,html,md,pdf}\fR
|
||||
Output format (default: json).
|
||||
.TP
|
||||
.BR \-\-diff
|
||||
Compare the current scan results against the last cached scan for the same
|
||||
target and display only new findings. Records already present in the local
|
||||
SQLite cache are suppressed. Useful for recurring exposure monitoring.
|
||||
.SH FILES
|
||||
.TP
|
||||
.I ~/.nox/config.ini
|
||||
Per-user configuration file. Created automatically on first run.
|
||||
Contains \fB[settings]\fR (concurrency, timeout, stealth, rate limits)
|
||||
and \fB[api_keys]\fR sections.
|
||||
.TP
|
||||
.I /etc/nox/config.ini
|
||||
System-wide configuration file. Used as fallback when the per-user file
|
||||
does not exist.
|
||||
.TP
|
||||
.I ~/.nox/sources/
|
||||
Directory containing JSON source definition files.
|
||||
.TP
|
||||
.I ~/.nox/reports/
|
||||
Default output directory for generated reports.
|
||||
.TP
|
||||
.I ~/.nox/logs/nox.log
|
||||
Application log file.
|
||||
.SH BUGS
|
||||
Report bugs at https://github.com/nox-project/nox-framework/issues
|
||||
.SH AUTHOR
|
||||
nox-project
|
||||
Executable
+14
@@ -0,0 +1,14 @@
|
||||
#!/usr/bin/env bash
|
||||
set -e
|
||||
|
||||
VENV="/opt/nox-cli/.venv"
|
||||
NOX="/opt/nox-cli/nox.py"
|
||||
|
||||
if [[ ! -f "$VENV/bin/python" ]]; then
|
||||
echo "[!] NOX Framework venv missing at $VENV — reinstall: sudo dpkg -i nox-cli_*.deb" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
export PYTHONPATH="/opt/nox-cli:${PYTHONPATH:-}"
|
||||
export NOX_PROG_NAME="nox-cli"
|
||||
exec "$VENV/bin/python" "$NOX" "$@"
|
||||
Executable
+51
@@ -0,0 +1,51 @@
|
||||
#!/bin/sh
|
||||
set -e
|
||||
|
||||
INSTALL_DIR="/opt/nox-cli"
|
||||
VENV="$INSTALL_DIR/.venv"
|
||||
WRAPPER="$INSTALL_DIR/nox-wrapper.sh"
|
||||
BIN="/usr/bin/nox-cli"
|
||||
NOX_VERSION=$(grep '^VERSION=' "$INSTALL_DIR/build_deb.sh" 2>/dev/null | cut -d'"' -f2 || echo "1.0.0")
|
||||
|
||||
case "$1" in
|
||||
configure)
|
||||
echo "[*] NOX Framework: Setting up isolated virtual environment..."
|
||||
|
||||
# 1. Create venv if absent
|
||||
if [ ! -f "$VENV/bin/python" ]; then
|
||||
python3 -m venv "$VENV"
|
||||
echo "[+] Virtual environment created at $VENV"
|
||||
else
|
||||
echo "[*] Virtual environment already exists — skipping creation."
|
||||
fi
|
||||
|
||||
# 2. Upgrade pip inside venv
|
||||
"$VENV/bin/pip" install --quiet --upgrade pip
|
||||
|
||||
# 3. Install dependencies strictly inside venv
|
||||
"$VENV/bin/pip" install --quiet -r "$INSTALL_DIR/requirements.txt"
|
||||
echo "[+] Dependencies installed."
|
||||
|
||||
# 4. Build source plugins
|
||||
"$VENV/bin/python" "$INSTALL_DIR/build_sources.py" > /dev/null 2>&1 || true
|
||||
chmod -R 644 "$INSTALL_DIR/sources/"*.json 2>/dev/null || true
|
||||
echo "[+] Source plugins built."
|
||||
|
||||
# 5. Link wrapper to /usr/bin/nox-cli
|
||||
chmod +x "$WRAPPER"
|
||||
ln -sf "$WRAPPER" "$BIN"
|
||||
echo "[+] Executable linked: $BIN"
|
||||
|
||||
echo "[+] NOX v${NOX_VERSION} installed. Run: nox-cli --help"
|
||||
;;
|
||||
|
||||
abort-upgrade|abort-remove|abort-deconfigure)
|
||||
;;
|
||||
|
||||
*)
|
||||
echo "postinst called with unknown argument: $1" >&2
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
|
||||
exit 0
|
||||
@@ -0,0 +1,40 @@
|
||||
[build-system]
|
||||
requires = ["setuptools>=68.0", "wheel"]
|
||||
build-backend = "setuptools.build_meta"
|
||||
|
||||
[project]
|
||||
name = "nox-cli"
|
||||
version = "1.0.0"
|
||||
description = "Advanced Asynchronous Cyber Threat Intelligence Framework"
|
||||
readme = { file = "README.md", content-type = "text/markdown" }
|
||||
license = { text = "Apache-2.0" }
|
||||
authors = [{ name = "nox-project" }]
|
||||
requires-python = ">=3.8"
|
||||
dependencies = [
|
||||
"aiohttp>=3.9.0",
|
||||
"aiohttp-socks>=0.8.4",
|
||||
"aiosqlite>=0.20.0",
|
||||
"httpx[http2]>=0.27.0",
|
||||
"requests>=2.31.0",
|
||||
"certifi>=2024.2.2",
|
||||
"cloudscraper>=1.2.71",
|
||||
"beautifulsoup4>=4.12.3",
|
||||
"lxml>=5.1.0",
|
||||
"dnspython>=2.6.0",
|
||||
"phonenumbers>=8.13.0",
|
||||
"pydantic>=2.0.0",
|
||||
"pydantic-core>=2.0.0",
|
||||
"colorama>=0.4.6",
|
||||
"rich>=13.7.0",
|
||||
"stem>=1.8.2",
|
||||
"fpdf2>=2.7.9",
|
||||
]
|
||||
|
||||
[project.scripts]
|
||||
nox-cli = "nox:main"
|
||||
|
||||
[tool.setuptools.packages.find]
|
||||
where = ["."]
|
||||
|
||||
[tool.setuptools.package-data]
|
||||
"*" = ["sources/*.json"]
|
||||
@@ -0,0 +1,34 @@
|
||||
# NOX — Cyber Threat Intelligence Framework
|
||||
# Python 3.8+ | pip install -r requirements.txt
|
||||
|
||||
# ── Core (Async) ───────────────────────────────────────────────────────
|
||||
aiohttp>=3.9.0
|
||||
aiohttp-socks>=0.8.4 # SOCKS4/5 proxy support for aiohttp
|
||||
aiosqlite>=0.20.0 # async SQLite (forensic persistence DB)
|
||||
httpx[http2]>=0.27.0 # Guardian Engine: dynamic proxy fetch + HTTP/2
|
||||
|
||||
# ── Intelligence & Scraping ────────────────────────────────────────────
|
||||
requests>=2.31.0
|
||||
certifi>=2024.2.2 # up-to-date CA bundle for SSL verification
|
||||
cloudscraper>=1.2.71 # Cloudflare-protected endpoint bypass
|
||||
beautifulsoup4>=4.12.3
|
||||
lxml>=5.1.0 # fast BS4 parser; required for HTML/XML scraping
|
||||
dnspython>=2.6.0 # DNS resolution (MX, A, TXT lookups)
|
||||
phonenumbers>=8.13.0 # phone number parsing and validation
|
||||
|
||||
# ── Validation ─────────────────────────────────────────────────────────
|
||||
pydantic>=2.0.0 # source schema validation and build engine
|
||||
pydantic-core>=2.0.0 # Rust-backed core for Pydantic v2
|
||||
|
||||
# ── CLI / UI ───────────────────────────────────────────────────────────
|
||||
colorama>=0.4.6
|
||||
rich>=13.7.0
|
||||
|
||||
# ── Tor Circuit Control ────────────────────────────────────────────────
|
||||
# Requires the system `tor` package (sudo apt install tor on Kali).
|
||||
# Used by --tor flag and the `tor` REPL command.
|
||||
stem>=1.8.2
|
||||
|
||||
# ── Reporting ──────────────────────────────────────────────────────────
|
||||
# Required for `--format pdf` and `export --format pdf`.
|
||||
fpdf2>=2.7.9
|
||||
@@ -0,0 +1,30 @@
|
||||
from setuptools import setup
|
||||
from pathlib import Path
|
||||
|
||||
requirements = [
|
||||
line.strip()
|
||||
for line in Path("requirements.txt").read_text().splitlines()
|
||||
if line.strip() and not line.startswith("#")
|
||||
]
|
||||
|
||||
setup(
|
||||
name="nox-cli",
|
||||
version="1.0.0",
|
||||
author="nox-project",
|
||||
description="Advanced Asynchronous Cyber Threat Intelligence Framework",
|
||||
long_description=Path("README.md").read_text(),
|
||||
long_description_content_type="text/markdown",
|
||||
license="Apache-2.0",
|
||||
python_requires=">=3.8",
|
||||
py_modules=["nox"],
|
||||
install_requires=requirements,
|
||||
entry_points={
|
||||
"console_scripts": [
|
||||
"nox-cli=nox:main",
|
||||
],
|
||||
},
|
||||
data_files=[
|
||||
("share/nox-cli/sources", [str(p) for p in Path("sources").glob("*.json")]),
|
||||
("share/man/man1", ["docs/nox-cli.1"]),
|
||||
],
|
||||
)
|
||||
@@ -0,0 +1,29 @@
|
||||
{
|
||||
"name": "abstract_email",
|
||||
"category": "email_rep",
|
||||
"endpoint": "https://emailvalidation.abstractapi.com/v1/?api_key={ABSTRACT_API_KEY}&email={target}",
|
||||
"method": "GET",
|
||||
"requires_auth": true,
|
||||
"selectors": {
|
||||
"quality": "$.quality_score"
|
||||
},
|
||||
"rate_limit": 1.0,
|
||||
"headers": {},
|
||||
"api_key_slots": [
|
||||
"{ABSTRACT_API_KEY}"
|
||||
],
|
||||
"input_type": "email",
|
||||
"output_type": [
|
||||
"email"
|
||||
],
|
||||
"normalization_map": {},
|
||||
"tags": [
|
||||
"passive",
|
||||
"fast"
|
||||
],
|
||||
"health_check_url": "https://emailvalidation.abstractapi.com",
|
||||
"expected_status": 200,
|
||||
"reliability_score": 4,
|
||||
"backup_endpoints": [],
|
||||
"confidence": 0.85
|
||||
}
|
||||
@@ -0,0 +1,33 @@
|
||||
{
|
||||
"name": "abuseipdb",
|
||||
"category": "threat_intel",
|
||||
"endpoint": "https://api.abuseipdb.com/api/v2/check?ipAddress={target}",
|
||||
"method": "GET",
|
||||
"requires_auth": true,
|
||||
"selectors": {
|
||||
"score": "$.data.abuseConfidenceScore"
|
||||
},
|
||||
"rate_limit": 1.0,
|
||||
"headers": {
|
||||
"Key": "{ABUSEIPDB_API_KEY}"
|
||||
},
|
||||
"api_key_slots": [
|
||||
"{ABUSEIPDB_API_KEY}"
|
||||
],
|
||||
"input_type": "ip",
|
||||
"output_type": [
|
||||
"ip"
|
||||
],
|
||||
"normalization_map": {
|
||||
"abuseConfidenceScore": "abuse_score"
|
||||
},
|
||||
"tags": [
|
||||
"passive",
|
||||
"threat"
|
||||
],
|
||||
"health_check_url": "https://api.abuseipdb.com",
|
||||
"expected_status": 200,
|
||||
"reliability_score": 5,
|
||||
"backup_endpoints": [],
|
||||
"confidence": 1.0
|
||||
}
|
||||
@@ -0,0 +1,29 @@
|
||||
{
|
||||
"name": "alienvault_otx_domain",
|
||||
"category": "threat_intel",
|
||||
"endpoint": "https://otx.alienvault.com/api/v1/indicators/domain/{target}/general",
|
||||
"method": "GET",
|
||||
"requires_auth": false,
|
||||
"selectors": {
|
||||
"pulses": "$.pulse_info.count",
|
||||
"tags": "$.tags"
|
||||
},
|
||||
"rate_limit": 1.0,
|
||||
"headers": {},
|
||||
"api_key_slots": [],
|
||||
"input_type": "domain",
|
||||
"output_type": [
|
||||
"domain",
|
||||
"ip"
|
||||
],
|
||||
"normalization_map": {},
|
||||
"tags": [
|
||||
"passive",
|
||||
"threat"
|
||||
],
|
||||
"health_check_url": "https://otx.alienvault.com",
|
||||
"expected_status": 200,
|
||||
"reliability_score": 5,
|
||||
"backup_endpoints": [],
|
||||
"confidence": 1.0
|
||||
}
|
||||
@@ -0,0 +1,28 @@
|
||||
{
|
||||
"name": "alienvault_otx_ip",
|
||||
"category": "threat_intel",
|
||||
"endpoint": "https://otx.alienvault.com/api/v1/indicators/IPv4/{target}/general",
|
||||
"method": "GET",
|
||||
"requires_auth": false,
|
||||
"selectors": {
|
||||
"asn": "$.asn",
|
||||
"country": "$.country_name"
|
||||
},
|
||||
"rate_limit": 1.0,
|
||||
"headers": {},
|
||||
"api_key_slots": [],
|
||||
"input_type": "ip",
|
||||
"output_type": [
|
||||
"domain"
|
||||
],
|
||||
"normalization_map": {},
|
||||
"tags": [
|
||||
"passive",
|
||||
"threat"
|
||||
],
|
||||
"health_check_url": "https://otx.alienvault.com",
|
||||
"expected_status": 200,
|
||||
"reliability_score": 5,
|
||||
"backup_endpoints": [],
|
||||
"confidence": 1.0
|
||||
}
|
||||
@@ -0,0 +1,27 @@
|
||||
{
|
||||
"name": "alienvault_otx_malware",
|
||||
"category": "threat_intel",
|
||||
"endpoint": "https://otx.alienvault.com/api/v1/indicators/file/{target}/analysis",
|
||||
"method": "GET",
|
||||
"requires_auth": false,
|
||||
"selectors": {
|
||||
"malware": "$.analysis.malware"
|
||||
},
|
||||
"rate_limit": 1.0,
|
||||
"headers": {},
|
||||
"api_key_slots": [],
|
||||
"input_type": "hash",
|
||||
"output_type": [
|
||||
"hash"
|
||||
],
|
||||
"normalization_map": {},
|
||||
"tags": [
|
||||
"passive",
|
||||
"threat"
|
||||
],
|
||||
"health_check_url": "https://otx.alienvault.com",
|
||||
"expected_status": 200,
|
||||
"reliability_score": 5,
|
||||
"backup_endpoints": [],
|
||||
"confidence": 1.0
|
||||
}
|
||||
@@ -0,0 +1,26 @@
|
||||
{
|
||||
"name": "alienvault_otx_user",
|
||||
"category": "social",
|
||||
"endpoint": "https://otx.alienvault.com/api/v1/users/{target}/general",
|
||||
"method": "GET",
|
||||
"requires_auth": false,
|
||||
"selectors": {
|
||||
"pulses": "$.pulse_count"
|
||||
},
|
||||
"rate_limit": 1.0,
|
||||
"headers": {},
|
||||
"api_key_slots": [],
|
||||
"input_type": "username",
|
||||
"output_type": [
|
||||
"username"
|
||||
],
|
||||
"normalization_map": {},
|
||||
"tags": [
|
||||
"passive"
|
||||
],
|
||||
"health_check_url": "https://otx.alienvault.com",
|
||||
"expected_status": 200,
|
||||
"reliability_score": 5,
|
||||
"backup_endpoints": [],
|
||||
"confidence": 1.0
|
||||
}
|
||||
@@ -0,0 +1,27 @@
|
||||
{
|
||||
"name": "anubis_subdomains",
|
||||
"category": "dns_recon",
|
||||
"endpoint": "https://jldc.me/anubis/subdomains/{target}",
|
||||
"method": "GET",
|
||||
"requires_auth": false,
|
||||
"selectors": {
|
||||
"subdomains": "$.*"
|
||||
},
|
||||
"rate_limit": 1.0,
|
||||
"headers": {},
|
||||
"api_key_slots": [],
|
||||
"input_type": "domain",
|
||||
"output_type": [
|
||||
"domain"
|
||||
],
|
||||
"normalization_map": {},
|
||||
"tags": [
|
||||
"passive"
|
||||
],
|
||||
"health_check_url": "https://jldc.me",
|
||||
"expected_status": 200,
|
||||
"reliability_score": 3,
|
||||
"is_volatile": true,
|
||||
"backup_endpoints": [],
|
||||
"confidence": 0.7
|
||||
}
|
||||
@@ -0,0 +1,32 @@
|
||||
{
|
||||
"name": "anyrun",
|
||||
"category": "threat_intel",
|
||||
"endpoint": "https://api.any.run/v1/analysis?hash={target}",
|
||||
"method": "GET",
|
||||
"requires_auth": true,
|
||||
"selectors": {
|
||||
"tasks": "$.tasks"
|
||||
},
|
||||
"rate_limit": 1.0,
|
||||
"headers": {
|
||||
"Authorization": "API-Key {ANYRUN_API_KEY}"
|
||||
},
|
||||
"api_key_slots": [
|
||||
"{ANYRUN_API_KEY}"
|
||||
],
|
||||
"input_type": "hash",
|
||||
"output_type": [
|
||||
"hash"
|
||||
],
|
||||
"normalization_map": {},
|
||||
"tags": [
|
||||
"passive",
|
||||
"threat",
|
||||
"heavy"
|
||||
],
|
||||
"health_check_url": "https://api.any.run",
|
||||
"expected_status": 200,
|
||||
"reliability_score": 4,
|
||||
"backup_endpoints": [],
|
||||
"confidence": 0.85
|
||||
}
|
||||
@@ -0,0 +1,27 @@
|
||||
{
|
||||
"name": "bgpview_ip",
|
||||
"category": "network",
|
||||
"endpoint": "https://api.bgpview.io/ip/{target}",
|
||||
"method": "GET",
|
||||
"requires_auth": false,
|
||||
"selectors": {
|
||||
"prefixes": "$.data.prefixes[*].prefix"
|
||||
},
|
||||
"rate_limit": 1.0,
|
||||
"headers": {},
|
||||
"api_key_slots": [],
|
||||
"input_type": "ip",
|
||||
"output_type": [
|
||||
"ip"
|
||||
],
|
||||
"normalization_map": {},
|
||||
"tags": [
|
||||
"passive",
|
||||
"infrastructure"
|
||||
],
|
||||
"health_check_url": "https://api.bgpview.io",
|
||||
"expected_status": 200,
|
||||
"reliability_score": 4,
|
||||
"backup_endpoints": [],
|
||||
"confidence": 0.85
|
||||
}
|
||||
@@ -0,0 +1,30 @@
|
||||
{
|
||||
"name": "binaryedge_dns",
|
||||
"category": "dns_recon",
|
||||
"endpoint": "https://api.binaryedge.io/v2/query/domains/subdomain/{target}",
|
||||
"method": "GET",
|
||||
"requires_auth": true,
|
||||
"selectors": {
|
||||
"subs": "$.subs"
|
||||
},
|
||||
"rate_limit": 1.0,
|
||||
"headers": {
|
||||
"X-Key": "{BINARYEDGE_API_KEY}"
|
||||
},
|
||||
"api_key_slots": [
|
||||
"{BINARYEDGE_API_KEY}"
|
||||
],
|
||||
"input_type": "domain",
|
||||
"output_type": [
|
||||
"domain"
|
||||
],
|
||||
"normalization_map": {},
|
||||
"tags": [
|
||||
"passive"
|
||||
],
|
||||
"health_check_url": "https://api.binaryedge.io",
|
||||
"expected_status": 200,
|
||||
"reliability_score": 4,
|
||||
"backup_endpoints": [],
|
||||
"confidence": 0.85
|
||||
}
|
||||
@@ -0,0 +1,33 @@
|
||||
{
|
||||
"name": "binaryedge_exposed",
|
||||
"category": "scanners",
|
||||
"endpoint": "https://api.binaryedge.io/v2/query/ip/{target}",
|
||||
"method": "GET",
|
||||
"requires_auth": true,
|
||||
"selectors": {
|
||||
"ports": "$.events[*].port"
|
||||
},
|
||||
"rate_limit": 1.0,
|
||||
"headers": {
|
||||
"X-Key": "{BINARYEDGE_API_KEY}"
|
||||
},
|
||||
"api_key_slots": [
|
||||
"{BINARYEDGE_API_KEY}"
|
||||
],
|
||||
"input_type": "ip",
|
||||
"output_type": [
|
||||
"ip"
|
||||
],
|
||||
"normalization_map": {
|
||||
"port": "open_port"
|
||||
},
|
||||
"tags": [
|
||||
"passive",
|
||||
"infrastructure"
|
||||
],
|
||||
"health_check_url": "https://api.binaryedge.io",
|
||||
"expected_status": 200,
|
||||
"reliability_score": 4,
|
||||
"backup_endpoints": [],
|
||||
"confidence": 0.85
|
||||
}
|
||||
@@ -0,0 +1,30 @@
|
||||
{
|
||||
"name": "bing_search_api",
|
||||
"category": "search",
|
||||
"endpoint": "https://api.bing.microsoft.com/v7.0/search?q={target}",
|
||||
"method": "GET",
|
||||
"requires_auth": true,
|
||||
"selectors": {
|
||||
"urls": "$.webPages.value[*].url"
|
||||
},
|
||||
"rate_limit": 1.0,
|
||||
"headers": {
|
||||
"Ocp-Apim-Subscription-Key": "{BING_API_KEY}"
|
||||
},
|
||||
"api_key_slots": [
|
||||
"{BING_API_KEY}"
|
||||
],
|
||||
"input_type": "any",
|
||||
"output_type": [
|
||||
"url"
|
||||
],
|
||||
"normalization_map": {},
|
||||
"tags": [
|
||||
"passive"
|
||||
],
|
||||
"health_check_url": "https://api.bing.microsoft.com",
|
||||
"expected_status": 200,
|
||||
"reliability_score": 5,
|
||||
"backup_endpoints": [],
|
||||
"confidence": 1.0
|
||||
}
|
||||
@@ -0,0 +1,32 @@
|
||||
{
|
||||
"name": "breachaware",
|
||||
"category": "breaches",
|
||||
"endpoint": "https://api.breachaware.com/v1/search?query={target}",
|
||||
"method": "GET",
|
||||
"requires_auth": true,
|
||||
"selectors": {
|
||||
"breaches": "$.breaches"
|
||||
},
|
||||
"rate_limit": 1.0,
|
||||
"headers": {
|
||||
"X-API-KEY": "{BA_API_KEY}"
|
||||
},
|
||||
"api_key_slots": [
|
||||
"{BA_API_KEY}"
|
||||
],
|
||||
"input_type": "email",
|
||||
"output_type": [
|
||||
"email"
|
||||
],
|
||||
"normalization_map": {},
|
||||
"tags": [
|
||||
"passive",
|
||||
"stealth"
|
||||
],
|
||||
"health_check_url": "https://api.breachaware.com",
|
||||
"expected_status": 200,
|
||||
"reliability_score": 3,
|
||||
"is_volatile": true,
|
||||
"backup_endpoints": [],
|
||||
"confidence": 0.7
|
||||
}
|
||||
@@ -0,0 +1,29 @@
|
||||
{
|
||||
"name": "breachdirectory",
|
||||
"category": "breaches",
|
||||
"endpoint": "https://breachdirectory.com/api/search?key={BD_API_KEY}&email={target}",
|
||||
"method": "GET",
|
||||
"requires_auth": true,
|
||||
"selectors": {
|
||||
"found": "$.found"
|
||||
},
|
||||
"rate_limit": 1.0,
|
||||
"headers": {},
|
||||
"api_key_slots": [
|
||||
"{BD_API_KEY}"
|
||||
],
|
||||
"input_type": "email",
|
||||
"output_type": [
|
||||
"email"
|
||||
],
|
||||
"normalization_map": {},
|
||||
"tags": [
|
||||
"passive",
|
||||
"stealth"
|
||||
],
|
||||
"health_check_url": "https://breachdirectory.com",
|
||||
"expected_status": 200,
|
||||
"reliability_score": 4,
|
||||
"backup_endpoints": [],
|
||||
"confidence": 0.85
|
||||
}
|
||||
@@ -0,0 +1,33 @@
|
||||
{
|
||||
"name": "censys_hosts",
|
||||
"category": "scanners",
|
||||
"endpoint": "https://search.censys.io/api/v2/hosts/search?q={target}",
|
||||
"method": "GET",
|
||||
"requires_auth": true,
|
||||
"selectors": {
|
||||
"results": "$.result.hits[*].ip"
|
||||
},
|
||||
"rate_limit": 1.0,
|
||||
"headers": {
|
||||
"Authorization": "Basic {CENSYS_AUTH_BASE64}"
|
||||
},
|
||||
"api_key_slots": [
|
||||
"{CENSYS_AUTH_BASE64}"
|
||||
],
|
||||
"input_type": "domain",
|
||||
"output_type": [
|
||||
"ip"
|
||||
],
|
||||
"normalization_map": {
|
||||
"ip": "ip_address"
|
||||
},
|
||||
"tags": [
|
||||
"passive",
|
||||
"infrastructure"
|
||||
],
|
||||
"health_check_url": "https://search.censys.io",
|
||||
"expected_status": 200,
|
||||
"reliability_score": 5,
|
||||
"backup_endpoints": [],
|
||||
"confidence": 1.0
|
||||
}
|
||||
@@ -0,0 +1,30 @@
|
||||
{
|
||||
"name": "checkleaked",
|
||||
"category": "breaches",
|
||||
"endpoint": "https://api.checkleaked.cc/check/{target}",
|
||||
"method": "GET",
|
||||
"requires_auth": false,
|
||||
"selectors": {
|
||||
"found": "$.found"
|
||||
},
|
||||
"rate_limit": 1.0,
|
||||
"headers": {},
|
||||
"api_key_slots": [],
|
||||
"input_type": "email",
|
||||
"output_type": [
|
||||
"email"
|
||||
],
|
||||
"normalization_map": {},
|
||||
"tags": [
|
||||
"passive",
|
||||
"stealth"
|
||||
],
|
||||
"health_check_url": "https://api.checkleaked.cc",
|
||||
"expected_status": 200,
|
||||
"reliability_score": 2,
|
||||
"is_volatile": true,
|
||||
"backup_endpoints": [
|
||||
"https://checkleaked.cc/api/check/{target}"
|
||||
],
|
||||
"confidence": 0.55
|
||||
}
|
||||
@@ -0,0 +1,30 @@
|
||||
{
|
||||
"name": "circl_lu_pdns",
|
||||
"category": "dns_recon",
|
||||
"endpoint": "https://www.circl.lu/pdns/query/{target}",
|
||||
"method": "GET",
|
||||
"requires_auth": true,
|
||||
"selectors": {
|
||||
"resolutions": "$.[*].rdata"
|
||||
},
|
||||
"rate_limit": 1.0,
|
||||
"headers": {
|
||||
"Authorization": "Basic {CIRCL_AUTH_BASE64}"
|
||||
},
|
||||
"api_key_slots": [
|
||||
"{CIRCL_AUTH_BASE64}"
|
||||
],
|
||||
"input_type": "domain",
|
||||
"output_type": [
|
||||
"ip"
|
||||
],
|
||||
"normalization_map": {},
|
||||
"tags": [
|
||||
"passive"
|
||||
],
|
||||
"health_check_url": "https://www.circl.lu",
|
||||
"expected_status": 200,
|
||||
"reliability_score": 4,
|
||||
"backup_endpoints": [],
|
||||
"confidence": 0.85
|
||||
}
|
||||
@@ -0,0 +1,32 @@
|
||||
{
|
||||
"name": "cit0day",
|
||||
"category": "breaches",
|
||||
"endpoint": "https://cit0day.in/api/v1/search?query={target}",
|
||||
"method": "GET",
|
||||
"requires_auth": true,
|
||||
"selectors": {
|
||||
"results": "$.results"
|
||||
},
|
||||
"rate_limit": 1.0,
|
||||
"headers": {
|
||||
"Authorization": "Bearer {CIT0DAY_API_KEY}"
|
||||
},
|
||||
"api_key_slots": [
|
||||
"{CIT0DAY_API_KEY}"
|
||||
],
|
||||
"input_type": "email",
|
||||
"output_type": [
|
||||
"email"
|
||||
],
|
||||
"normalization_map": {},
|
||||
"tags": [
|
||||
"passive",
|
||||
"stealth"
|
||||
],
|
||||
"health_check_url": "https://cit0day.in",
|
||||
"expected_status": 200,
|
||||
"reliability_score": 2,
|
||||
"is_volatile": true,
|
||||
"backup_endpoints": [],
|
||||
"confidence": 0.55
|
||||
}
|
||||
@@ -0,0 +1,33 @@
|
||||
{
|
||||
"name": "clearbit_enrich",
|
||||
"category": "enrichment",
|
||||
"endpoint": "https://person.clearbit.com/v2/people/find?email={target}",
|
||||
"method": "GET",
|
||||
"requires_auth": true,
|
||||
"selectors": {
|
||||
"full_name": "$.name.fullName"
|
||||
},
|
||||
"rate_limit": 1.0,
|
||||
"headers": {
|
||||
"Authorization": "Bearer {CLEARBIT_API_KEY}"
|
||||
},
|
||||
"api_key_slots": [
|
||||
"{CLEARBIT_API_KEY}"
|
||||
],
|
||||
"input_type": "email",
|
||||
"output_type": [
|
||||
"username",
|
||||
"domain"
|
||||
],
|
||||
"normalization_map": {
|
||||
"fullName": "full_name"
|
||||
},
|
||||
"tags": [
|
||||
"passive"
|
||||
],
|
||||
"health_check_url": "https://person.clearbit.com",
|
||||
"expected_status": 200,
|
||||
"reliability_score": 4,
|
||||
"backup_endpoints": [],
|
||||
"confidence": 0.85
|
||||
}
|
||||
@@ -0,0 +1,33 @@
|
||||
{
|
||||
"name": "criminalip_asset",
|
||||
"category": "scanners",
|
||||
"endpoint": "https://api.criminalip.io/v1/asset/ip/report?ip={target}",
|
||||
"method": "GET",
|
||||
"requires_auth": true,
|
||||
"selectors": {
|
||||
"score": "$.score"
|
||||
},
|
||||
"rate_limit": 1.0,
|
||||
"headers": {
|
||||
"x-api-key": "{CRIMINALIP_API_KEY}"
|
||||
},
|
||||
"api_key_slots": [
|
||||
"{CRIMINALIP_API_KEY}"
|
||||
],
|
||||
"input_type": "ip",
|
||||
"output_type": [
|
||||
"ip"
|
||||
],
|
||||
"normalization_map": {
|
||||
"score": "risk_score"
|
||||
},
|
||||
"tags": [
|
||||
"passive",
|
||||
"threat"
|
||||
],
|
||||
"health_check_url": "https://api.criminalip.io",
|
||||
"expected_status": 200,
|
||||
"reliability_score": 4,
|
||||
"backup_endpoints": [],
|
||||
"confidence": 0.85
|
||||
}
|
||||
@@ -0,0 +1,31 @@
|
||||
{
|
||||
"name": "crt_sh",
|
||||
"category": "certificate_transparency",
|
||||
"endpoint": "https://crt.sh/?q={target}&output=json",
|
||||
"method": "GET",
|
||||
"requires_auth": false,
|
||||
"selectors": {
|
||||
"domains": "$.*.name_value"
|
||||
},
|
||||
"rate_limit": 1.0,
|
||||
"headers": {
|
||||
"Accept": "application/json"
|
||||
},
|
||||
"api_key_slots": [],
|
||||
"input_type": "domain",
|
||||
"output_type": [
|
||||
"domain"
|
||||
],
|
||||
"normalization_map": {
|
||||
"name_value": "domain"
|
||||
},
|
||||
"tags": [
|
||||
"passive",
|
||||
"fast"
|
||||
],
|
||||
"health_check_url": "https://crt.sh",
|
||||
"expected_status": 200,
|
||||
"reliability_score": 5,
|
||||
"backup_endpoints": [],
|
||||
"confidence": 1.0
|
||||
}
|
||||
@@ -0,0 +1,28 @@
|
||||
{
|
||||
"name": "cve_search",
|
||||
"category": "vulns",
|
||||
"endpoint": "https://cve.circl.lu/api/cve/{target}",
|
||||
"method": "GET",
|
||||
"requires_auth": false,
|
||||
"selectors": {
|
||||
"summary": "$.summary"
|
||||
},
|
||||
"rate_limit": 1.0,
|
||||
"headers": {},
|
||||
"api_key_slots": [],
|
||||
"input_type": "cve",
|
||||
"output_type": [
|
||||
"cve"
|
||||
],
|
||||
"normalization_map": {
|
||||
"summary": "vuln_description"
|
||||
},
|
||||
"tags": [
|
||||
"passive"
|
||||
],
|
||||
"health_check_url": "https://cve.circl.lu",
|
||||
"expected_status": 200,
|
||||
"reliability_score": 4,
|
||||
"backup_endpoints": [],
|
||||
"confidence": 0.85
|
||||
}
|
||||
@@ -0,0 +1,27 @@
|
||||
{
|
||||
"name": "cxsecurity",
|
||||
"category": "vulns",
|
||||
"endpoint": "https://cxsecurity.com/cvejson.php?cve={target}",
|
||||
"method": "GET",
|
||||
"requires_auth": false,
|
||||
"selectors": {
|
||||
"title": "$.title"
|
||||
},
|
||||
"rate_limit": 1.0,
|
||||
"headers": {},
|
||||
"api_key_slots": [],
|
||||
"input_type": "cve",
|
||||
"output_type": [
|
||||
"cve"
|
||||
],
|
||||
"normalization_map": {},
|
||||
"tags": [
|
||||
"passive"
|
||||
],
|
||||
"health_check_url": "https://cxsecurity.com",
|
||||
"expected_status": 200,
|
||||
"reliability_score": 3,
|
||||
"is_volatile": true,
|
||||
"backup_endpoints": [],
|
||||
"confidence": 0.7
|
||||
}
|
||||
@@ -0,0 +1,41 @@
|
||||
{
|
||||
"name": "dehashed",
|
||||
"category": "breaches",
|
||||
"endpoint": "https://api.dehashed.com/search?query={target}",
|
||||
"method": "GET",
|
||||
"requires_auth": true,
|
||||
"selectors": {
|
||||
"entries": "$.entries"
|
||||
},
|
||||
"rate_limit": 1.0,
|
||||
"headers": {
|
||||
"Authorization": "Basic {DEHASHED_AUTH_BASE64}",
|
||||
"Accept": "application/json"
|
||||
},
|
||||
"api_key_slots": [
|
||||
"{DEHASHED_AUTH_BASE64}"
|
||||
],
|
||||
"input_type": "email",
|
||||
"output_type": [
|
||||
"email",
|
||||
"username",
|
||||
"ip"
|
||||
],
|
||||
"normalization_map": {
|
||||
"email": "email_address",
|
||||
"username": "username",
|
||||
"password": "plaintext_password",
|
||||
"hashed_password": "password_hash",
|
||||
"ip_address": "ip_address",
|
||||
"name": "full_name"
|
||||
},
|
||||
"tags": [
|
||||
"passive",
|
||||
"stealth"
|
||||
],
|
||||
"health_check_url": "https://api.dehashed.com",
|
||||
"expected_status": 200,
|
||||
"reliability_score": 5,
|
||||
"backup_endpoints": [],
|
||||
"confidence": 1.0
|
||||
}
|
||||
@@ -0,0 +1,30 @@
|
||||
{
|
||||
"name": "dnsdb_pdns",
|
||||
"category": "dns_recon",
|
||||
"endpoint": "https://api.dnsdb.info/lookup/rrset/name/{target}",
|
||||
"method": "GET",
|
||||
"requires_auth": true,
|
||||
"selectors": {
|
||||
"rdata": "$.[*].rdata"
|
||||
},
|
||||
"rate_limit": 1.0,
|
||||
"headers": {
|
||||
"X-API-Key": "{DNSDB_API_KEY}"
|
||||
},
|
||||
"api_key_slots": [
|
||||
"{DNSDB_API_KEY}"
|
||||
],
|
||||
"input_type": "domain",
|
||||
"output_type": [
|
||||
"ip"
|
||||
],
|
||||
"normalization_map": {},
|
||||
"tags": [
|
||||
"passive"
|
||||
],
|
||||
"health_check_url": "https://api.dnsdb.info",
|
||||
"expected_status": 200,
|
||||
"reliability_score": 5,
|
||||
"backup_endpoints": [],
|
||||
"confidence": 1.0
|
||||
}
|
||||
@@ -0,0 +1,31 @@
|
||||
{
|
||||
"name": "domaintools_whois",
|
||||
"category": "whois",
|
||||
"endpoint": "https://api.domaintools.com/v1/{target}/whois/",
|
||||
"method": "GET",
|
||||
"requires_auth": true,
|
||||
"selectors": {
|
||||
"whois": "$.response.whois.record"
|
||||
},
|
||||
"rate_limit": 1.0,
|
||||
"headers": {
|
||||
"Authorization": "Basic {DT_AUTH_BASE64}"
|
||||
},
|
||||
"api_key_slots": [
|
||||
"{DT_AUTH_BASE64}"
|
||||
],
|
||||
"input_type": "domain",
|
||||
"output_type": [
|
||||
"email",
|
||||
"domain"
|
||||
],
|
||||
"normalization_map": {},
|
||||
"tags": [
|
||||
"passive"
|
||||
],
|
||||
"health_check_url": "https://api.domaintools.com",
|
||||
"expected_status": 200,
|
||||
"reliability_score": 5,
|
||||
"backup_endpoints": [],
|
||||
"confidence": 1.0
|
||||
}
|
||||
@@ -0,0 +1,27 @@
|
||||
{
|
||||
"name": "duckduckgo_api",
|
||||
"category": "search",
|
||||
"endpoint": "https://api.duckduckgo.com/?q={target}&format=json",
|
||||
"method": "GET",
|
||||
"requires_auth": false,
|
||||
"selectors": {
|
||||
"abstract": "$.Abstract"
|
||||
},
|
||||
"rate_limit": 1.0,
|
||||
"headers": {},
|
||||
"api_key_slots": [],
|
||||
"input_type": "any",
|
||||
"output_type": [
|
||||
"url"
|
||||
],
|
||||
"normalization_map": {},
|
||||
"tags": [
|
||||
"passive",
|
||||
"fast"
|
||||
],
|
||||
"health_check_url": "https://api.duckduckgo.com",
|
||||
"expected_status": 200,
|
||||
"reliability_score": 5,
|
||||
"backup_endpoints": [],
|
||||
"confidence": 1.0
|
||||
}
|
||||
@@ -0,0 +1,29 @@
|
||||
{
|
||||
"name": "emailhippo",
|
||||
"category": "email_rep",
|
||||
"endpoint": "https://api.emailhippo.com/v3/verify?apiKey={HIPPO_API_KEY}&email={target}",
|
||||
"method": "GET",
|
||||
"requires_auth": true,
|
||||
"selectors": {
|
||||
"status": "$.meta.status"
|
||||
},
|
||||
"rate_limit": 1.0,
|
||||
"headers": {},
|
||||
"api_key_slots": [
|
||||
"{HIPPO_API_KEY}"
|
||||
],
|
||||
"input_type": "email",
|
||||
"output_type": [
|
||||
"email"
|
||||
],
|
||||
"normalization_map": {},
|
||||
"tags": [
|
||||
"passive",
|
||||
"fast"
|
||||
],
|
||||
"health_check_url": "https://api.emailhippo.com",
|
||||
"expected_status": 200,
|
||||
"reliability_score": 4,
|
||||
"backup_endpoints": [],
|
||||
"confidence": 0.85
|
||||
}
|
||||
@@ -0,0 +1,29 @@
|
||||
{
|
||||
"name": "emailrep_io",
|
||||
"category": "email_rep",
|
||||
"endpoint": "https://emailrep.io/{target}",
|
||||
"method": "GET",
|
||||
"requires_auth": false,
|
||||
"selectors": {
|
||||
"reputation": "$.reputation"
|
||||
},
|
||||
"rate_limit": 2.0,
|
||||
"headers": {},
|
||||
"api_key_slots": [],
|
||||
"input_type": "email",
|
||||
"output_type": [
|
||||
"email"
|
||||
],
|
||||
"normalization_map": {
|
||||
"reputation": "email_reputation"
|
||||
},
|
||||
"tags": [
|
||||
"passive",
|
||||
"fast"
|
||||
],
|
||||
"health_check_url": "https://emailrep.io",
|
||||
"expected_status": 200,
|
||||
"reliability_score": 4,
|
||||
"backup_endpoints": [],
|
||||
"confidence": 0.85
|
||||
}
|
||||
@@ -0,0 +1,28 @@
|
||||
{
|
||||
"name": "extreme_ip_lookup",
|
||||
"category": "geolocation",
|
||||
"endpoint": "https://extreme-ip-lookup.com/json/{target}?key={EXTREME_API_KEY}",
|
||||
"method": "GET",
|
||||
"requires_auth": true,
|
||||
"selectors": {
|
||||
"org": "$.org"
|
||||
},
|
||||
"rate_limit": 1.0,
|
||||
"headers": {},
|
||||
"api_key_slots": [
|
||||
"{EXTREME_API_KEY}"
|
||||
],
|
||||
"input_type": "ip",
|
||||
"output_type": [
|
||||
"ip"
|
||||
],
|
||||
"normalization_map": {},
|
||||
"tags": [
|
||||
"passive"
|
||||
],
|
||||
"health_check_url": "https://extreme-ip-lookup.com",
|
||||
"expected_status": 200,
|
||||
"reliability_score": 3,
|
||||
"backup_endpoints": [],
|
||||
"confidence": 0.7
|
||||
}
|
||||
@@ -0,0 +1,31 @@
|
||||
{
|
||||
"name": "fofa_info",
|
||||
"category": "scanners",
|
||||
"endpoint": "https://fofa.info/api/v1/search/all?email={FOFA_EMAIL}&key={FOFA_API_KEY}&qbase64={target}",
|
||||
"method": "GET",
|
||||
"requires_auth": true,
|
||||
"selectors": {
|
||||
"results": "$.results"
|
||||
},
|
||||
"rate_limit": 1.0,
|
||||
"headers": {},
|
||||
"api_key_slots": [
|
||||
"{FOFA_API_KEY}",
|
||||
"{FOFA_EMAIL}"
|
||||
],
|
||||
"input_type": "domain",
|
||||
"output_type": [
|
||||
"ip",
|
||||
"domain"
|
||||
],
|
||||
"normalization_map": {},
|
||||
"tags": [
|
||||
"passive",
|
||||
"infrastructure"
|
||||
],
|
||||
"health_check_url": "https://fofa.info",
|
||||
"expected_status": 200,
|
||||
"reliability_score": 4,
|
||||
"backup_endpoints": [],
|
||||
"confidence": 0.85
|
||||
}
|
||||
@@ -0,0 +1,31 @@
|
||||
{
|
||||
"name": "fraudlabspro",
|
||||
"category": "threat_intel",
|
||||
"endpoint": "https://api.fraudlabspro.com/v1/ip/check?key={FLP_API_KEY}&ip={target}",
|
||||
"method": "GET",
|
||||
"requires_auth": true,
|
||||
"selectors": {
|
||||
"fraud": "$.fraudlabspro_score"
|
||||
},
|
||||
"rate_limit": 1.0,
|
||||
"headers": {},
|
||||
"api_key_slots": [
|
||||
"{FLP_API_KEY}"
|
||||
],
|
||||
"input_type": "ip",
|
||||
"output_type": [
|
||||
"ip"
|
||||
],
|
||||
"normalization_map": {
|
||||
"fraudlabspro_score": "fraud_score"
|
||||
},
|
||||
"tags": [
|
||||
"passive",
|
||||
"threat"
|
||||
],
|
||||
"health_check_url": "https://api.fraudlabspro.com",
|
||||
"expected_status": 200,
|
||||
"reliability_score": 4,
|
||||
"backup_endpoints": [],
|
||||
"confidence": 0.85
|
||||
}
|
||||
@@ -0,0 +1,34 @@
|
||||
{
|
||||
"name": "fullcontact",
|
||||
"category": "enrichment",
|
||||
"endpoint": "https://api.fullcontact.com/v3/person.enrich",
|
||||
"method": "POST",
|
||||
"requires_auth": true,
|
||||
"selectors": {
|
||||
"social": "$.socialProfiles"
|
||||
},
|
||||
"rate_limit": 1.0,
|
||||
"headers": {
|
||||
"Authorization": "Bearer {FULLCONTACT_API_KEY}"
|
||||
},
|
||||
"payload_template": {
|
||||
"email": "{target}"
|
||||
},
|
||||
"api_key_slots": [
|
||||
"{FULLCONTACT_API_KEY}"
|
||||
],
|
||||
"input_type": "email",
|
||||
"output_type": [
|
||||
"username",
|
||||
"domain"
|
||||
],
|
||||
"normalization_map": {},
|
||||
"tags": [
|
||||
"passive"
|
||||
],
|
||||
"health_check_url": "https://api.fullcontact.com",
|
||||
"expected_status": 200,
|
||||
"reliability_score": 4,
|
||||
"backup_endpoints": [],
|
||||
"confidence": 0.85
|
||||
}
|
||||
@@ -0,0 +1,30 @@
|
||||
{
|
||||
"name": "github_code_search",
|
||||
"category": "code",
|
||||
"endpoint": "https://api.github.com/search/code?q={target}",
|
||||
"method": "GET",
|
||||
"requires_auth": true,
|
||||
"selectors": {
|
||||
"urls": "$.items[*].html_url"
|
||||
},
|
||||
"rate_limit": 1.0,
|
||||
"headers": {
|
||||
"Authorization": "token {GITHUB_TOKEN}"
|
||||
},
|
||||
"api_key_slots": [
|
||||
"{GITHUB_TOKEN}"
|
||||
],
|
||||
"input_type": "any",
|
||||
"output_type": [
|
||||
"url"
|
||||
],
|
||||
"normalization_map": {},
|
||||
"tags": [
|
||||
"passive"
|
||||
],
|
||||
"health_check_url": "https://api.github.com",
|
||||
"expected_status": 200,
|
||||
"reliability_score": 5,
|
||||
"backup_endpoints": [],
|
||||
"confidence": 1.0
|
||||
}
|
||||
@@ -0,0 +1,30 @@
|
||||
{
|
||||
"name": "github_search_repos",
|
||||
"category": "social",
|
||||
"endpoint": "https://api.github.com/search/repositories?q={target}",
|
||||
"method": "GET",
|
||||
"requires_auth": true,
|
||||
"selectors": {
|
||||
"total": "$.total_count"
|
||||
},
|
||||
"rate_limit": 1.0,
|
||||
"headers": {
|
||||
"Authorization": "token {GITHUB_TOKEN}"
|
||||
},
|
||||
"api_key_slots": [
|
||||
"{GITHUB_TOKEN}"
|
||||
],
|
||||
"input_type": "username",
|
||||
"output_type": [
|
||||
"username"
|
||||
],
|
||||
"normalization_map": {},
|
||||
"tags": [
|
||||
"passive"
|
||||
],
|
||||
"health_check_url": "https://api.github.com",
|
||||
"expected_status": 200,
|
||||
"reliability_score": 5,
|
||||
"backup_endpoints": [],
|
||||
"confidence": 1.0
|
||||
}
|
||||
@@ -0,0 +1,31 @@
|
||||
{
|
||||
"name": "github_users",
|
||||
"category": "social",
|
||||
"endpoint": "https://api.github.com/users/{target}",
|
||||
"method": "GET",
|
||||
"requires_auth": false,
|
||||
"selectors": {
|
||||
"bio": "$.bio",
|
||||
"blog": "$.blog"
|
||||
},
|
||||
"rate_limit": 2.0,
|
||||
"headers": {
|
||||
"User-Agent": "NOX"
|
||||
},
|
||||
"api_key_slots": [],
|
||||
"input_type": "username",
|
||||
"output_type": [
|
||||
"username",
|
||||
"domain"
|
||||
],
|
||||
"normalization_map": {},
|
||||
"tags": [
|
||||
"passive",
|
||||
"fast"
|
||||
],
|
||||
"health_check_url": "https://api.github.com",
|
||||
"expected_status": 200,
|
||||
"reliability_score": 5,
|
||||
"backup_endpoints": [],
|
||||
"confidence": 1.0
|
||||
}
|
||||
@@ -0,0 +1,26 @@
|
||||
{
|
||||
"name": "gitlab_search",
|
||||
"category": "social",
|
||||
"endpoint": "https://gitlab.com/api/v4/users?username={target}",
|
||||
"method": "GET",
|
||||
"requires_auth": false,
|
||||
"selectors": {
|
||||
"id": "$.[*].id"
|
||||
},
|
||||
"rate_limit": 1.0,
|
||||
"headers": {},
|
||||
"api_key_slots": [],
|
||||
"input_type": "username",
|
||||
"output_type": [
|
||||
"username"
|
||||
],
|
||||
"normalization_map": {},
|
||||
"tags": [
|
||||
"passive"
|
||||
],
|
||||
"health_check_url": "https://gitlab.com",
|
||||
"expected_status": 200,
|
||||
"reliability_score": 5,
|
||||
"backup_endpoints": [],
|
||||
"confidence": 1.0
|
||||
}
|
||||
@@ -0,0 +1,52 @@
|
||||
{
|
||||
"name": "google_safebrowsing",
|
||||
"category": "threat_intel",
|
||||
"endpoint": "https://safebrowsing.googleapis.com/v4/threatMatches:find?key={GOOGLE_API_KEY}",
|
||||
"method": "POST",
|
||||
"requires_auth": true,
|
||||
"selectors": {
|
||||
"matches": "$.matches"
|
||||
},
|
||||
"rate_limit": 1.0,
|
||||
"headers": {},
|
||||
"payload_template": {
|
||||
"client": {
|
||||
"clientId": "nox",
|
||||
"clientVersion": "1.0"
|
||||
},
|
||||
"threatInfo": {
|
||||
"threatTypes": [
|
||||
"MALWARE",
|
||||
"SOCIAL_ENGINEERING"
|
||||
],
|
||||
"platformTypes": [
|
||||
"ANY_PLATFORM"
|
||||
],
|
||||
"threatEntryTypes": [
|
||||
"URL"
|
||||
],
|
||||
"threatEntries": [
|
||||
{
|
||||
"url": "{target}"
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"api_key_slots": [
|
||||
"{GOOGLE_API_KEY}"
|
||||
],
|
||||
"input_type": "url",
|
||||
"output_type": [
|
||||
"url"
|
||||
],
|
||||
"normalization_map": {},
|
||||
"tags": [
|
||||
"passive",
|
||||
"threat"
|
||||
],
|
||||
"health_check_url": "https://safebrowsing.googleapis.com",
|
||||
"expected_status": 200,
|
||||
"reliability_score": 5,
|
||||
"backup_endpoints": [],
|
||||
"confidence": 1.0
|
||||
}
|
||||
@@ -0,0 +1,29 @@
|
||||
{
|
||||
"name": "google_search_custom",
|
||||
"category": "search",
|
||||
"endpoint": "https://www.googleapis.com/customsearch/v1?key={GOOGLE_CX_KEY}&cx={GOOGLE_CX_ID}&q={target}",
|
||||
"method": "GET",
|
||||
"requires_auth": true,
|
||||
"selectors": {
|
||||
"items": "$.items[*].link"
|
||||
},
|
||||
"rate_limit": 1.0,
|
||||
"headers": {},
|
||||
"api_key_slots": [
|
||||
"{GOOGLE_CX_KEY}",
|
||||
"{GOOGLE_CX_ID}"
|
||||
],
|
||||
"input_type": "any",
|
||||
"output_type": [
|
||||
"url"
|
||||
],
|
||||
"normalization_map": {},
|
||||
"tags": [
|
||||
"passive"
|
||||
],
|
||||
"health_check_url": "https://www.googleapis.com",
|
||||
"expected_status": 200,
|
||||
"reliability_score": 5,
|
||||
"backup_endpoints": [],
|
||||
"confidence": 1.0
|
||||
}
|
||||
@@ -0,0 +1,26 @@
|
||||
{
|
||||
"name": "gravatar",
|
||||
"category": "social",
|
||||
"endpoint": "https://www.gravatar.com/{target}.json",
|
||||
"method": "GET",
|
||||
"requires_auth": false,
|
||||
"selectors": {
|
||||
"name": "$.entry[0].displayName"
|
||||
},
|
||||
"rate_limit": 2.0,
|
||||
"headers": {},
|
||||
"api_key_slots": [],
|
||||
"input_type": "email",
|
||||
"output_type": [
|
||||
"username"
|
||||
],
|
||||
"normalization_map": {},
|
||||
"tags": [
|
||||
"passive"
|
||||
],
|
||||
"health_check_url": "https://www.gravatar.com",
|
||||
"expected_status": 200,
|
||||
"reliability_score": 4,
|
||||
"backup_endpoints": [],
|
||||
"confidence": 0.85
|
||||
}
|
||||
@@ -0,0 +1,35 @@
|
||||
{
|
||||
"name": "greynoise_community",
|
||||
"category": "threat_intel",
|
||||
"endpoint": "https://api.greynoise.io/v3/community/{target}",
|
||||
"method": "GET",
|
||||
"requires_auth": true,
|
||||
"selectors": {
|
||||
"noise": "$.noise",
|
||||
"classification": "$.classification"
|
||||
},
|
||||
"rate_limit": 1.0,
|
||||
"headers": {
|
||||
"key": "{GREYNOISE_API_KEY}"
|
||||
},
|
||||
"api_key_slots": [
|
||||
"{GREYNOISE_API_KEY}"
|
||||
],
|
||||
"input_type": "ip",
|
||||
"output_type": [
|
||||
"ip"
|
||||
],
|
||||
"normalization_map": {
|
||||
"noise": "is_noise",
|
||||
"classification": "threat_class"
|
||||
},
|
||||
"tags": [
|
||||
"passive",
|
||||
"threat"
|
||||
],
|
||||
"health_check_url": "https://api.greynoise.io",
|
||||
"expected_status": 200,
|
||||
"reliability_score": 5,
|
||||
"backup_endpoints": [],
|
||||
"confidence": 1.0
|
||||
}
|
||||
@@ -0,0 +1,27 @@
|
||||
{
|
||||
"name": "hackernews_user",
|
||||
"category": "social",
|
||||
"endpoint": "https://hacker-news.firebaseio.com/v0/user/{target}.json",
|
||||
"method": "GET",
|
||||
"requires_auth": false,
|
||||
"selectors": {
|
||||
"karma": "$.karma"
|
||||
},
|
||||
"rate_limit": 1.0,
|
||||
"headers": {},
|
||||
"api_key_slots": [],
|
||||
"input_type": "username",
|
||||
"output_type": [
|
||||
"username"
|
||||
],
|
||||
"normalization_map": {},
|
||||
"tags": [
|
||||
"passive",
|
||||
"fast"
|
||||
],
|
||||
"health_check_url": "https://hacker-news.firebaseio.com",
|
||||
"expected_status": 200,
|
||||
"reliability_score": 5,
|
||||
"backup_endpoints": [],
|
||||
"confidence": 1.0
|
||||
}
|
||||
@@ -0,0 +1,27 @@
|
||||
{
|
||||
"name": "hackertarget_dnslookup",
|
||||
"category": "dns_recon",
|
||||
"endpoint": "https://api.hackertarget.com/dnslookup/?q={target}",
|
||||
"method": "GET",
|
||||
"requires_auth": false,
|
||||
"selectors": {
|
||||
"records": "text_lines"
|
||||
},
|
||||
"rate_limit": 1.0,
|
||||
"headers": {},
|
||||
"api_key_slots": [],
|
||||
"input_type": "domain",
|
||||
"output_type": [
|
||||
"ip"
|
||||
],
|
||||
"normalization_map": {},
|
||||
"tags": [
|
||||
"passive",
|
||||
"fast"
|
||||
],
|
||||
"health_check_url": "https://api.hackertarget.com",
|
||||
"expected_status": 200,
|
||||
"reliability_score": 4,
|
||||
"backup_endpoints": [],
|
||||
"confidence": 0.85
|
||||
}
|
||||
@@ -0,0 +1,28 @@
|
||||
{
|
||||
"name": "hackertarget_hostsearch",
|
||||
"category": "dns_recon",
|
||||
"endpoint": "https://api.hackertarget.com/hostsearch/?q={target}",
|
||||
"method": "GET",
|
||||
"requires_auth": false,
|
||||
"selectors": {
|
||||
"hosts": "text_lines"
|
||||
},
|
||||
"rate_limit": 1.0,
|
||||
"headers": {},
|
||||
"api_key_slots": [],
|
||||
"input_type": "domain",
|
||||
"output_type": [
|
||||
"ip",
|
||||
"domain"
|
||||
],
|
||||
"normalization_map": {},
|
||||
"tags": [
|
||||
"passive",
|
||||
"fast"
|
||||
],
|
||||
"health_check_url": "https://api.hackertarget.com",
|
||||
"expected_status": 200,
|
||||
"reliability_score": 4,
|
||||
"backup_endpoints": [],
|
||||
"confidence": 0.85
|
||||
}
|
||||
@@ -0,0 +1,26 @@
|
||||
{
|
||||
"name": "hackertarget_reverseip",
|
||||
"category": "dns_recon",
|
||||
"endpoint": "https://api.hackertarget.com/reverseiplookup/?q={target}",
|
||||
"method": "GET",
|
||||
"requires_auth": false,
|
||||
"selectors": {
|
||||
"domains": "text_lines"
|
||||
},
|
||||
"rate_limit": 1.0,
|
||||
"headers": {},
|
||||
"api_key_slots": [],
|
||||
"input_type": "ip",
|
||||
"output_type": [
|
||||
"domain"
|
||||
],
|
||||
"normalization_map": {},
|
||||
"tags": [
|
||||
"passive"
|
||||
],
|
||||
"health_check_url": "https://api.hackertarget.com",
|
||||
"expected_status": 200,
|
||||
"reliability_score": 4,
|
||||
"backup_endpoints": [],
|
||||
"confidence": 0.85
|
||||
}
|
||||
@@ -0,0 +1,27 @@
|
||||
{
|
||||
"name": "hackertarget_whois",
|
||||
"category": "whois",
|
||||
"endpoint": "https://api.hackertarget.com/whois/?q={target}",
|
||||
"method": "GET",
|
||||
"requires_auth": false,
|
||||
"selectors": {
|
||||
"raw": "text_lines"
|
||||
},
|
||||
"rate_limit": 1.0,
|
||||
"headers": {},
|
||||
"api_key_slots": [],
|
||||
"input_type": "domain",
|
||||
"output_type": [
|
||||
"email",
|
||||
"domain"
|
||||
],
|
||||
"normalization_map": {},
|
||||
"tags": [
|
||||
"passive"
|
||||
],
|
||||
"health_check_url": "https://api.hackertarget.com",
|
||||
"expected_status": 200,
|
||||
"reliability_score": 4,
|
||||
"backup_endpoints": [],
|
||||
"confidence": 0.85
|
||||
}
|
||||
@@ -0,0 +1,28 @@
|
||||
{
|
||||
"name": "hashes_org",
|
||||
"category": "hashes",
|
||||
"endpoint": "https://hashes.org/api.php?key={HASHES_API_KEY}&query={target}",
|
||||
"method": "GET",
|
||||
"requires_auth": true,
|
||||
"selectors": {
|
||||
"found": "$.results"
|
||||
},
|
||||
"rate_limit": 1.0,
|
||||
"headers": {},
|
||||
"api_key_slots": [
|
||||
"{HASHES_API_KEY}"
|
||||
],
|
||||
"input_type": "hash",
|
||||
"output_type": [
|
||||
"hash"
|
||||
],
|
||||
"normalization_map": {},
|
||||
"tags": [
|
||||
"passive"
|
||||
],
|
||||
"health_check_url": "https://hashes.org",
|
||||
"expected_status": 200,
|
||||
"reliability_score": 3,
|
||||
"backup_endpoints": [],
|
||||
"confidence": 0.7
|
||||
}
|
||||
@@ -0,0 +1,243 @@
|
||||
"""
|
||||
sources/helpers/config_handler.py — NOX Framework
|
||||
Unified credential management via ~/.config/nox-cli/apikeys.json (XDG).
|
||||
|
||||
Priority: environment variable → apikeys.json → None
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
from typing import Dict, Optional
|
||||
|
||||
# ── Shared constant — import this everywhere instead of a raw string ───
|
||||
UNIVERSAL_PLACEHOLDER = "INSERT_API_KEY_HERE"
|
||||
|
||||
# ── XDG config path ────────────────────────────────────────────────────
|
||||
_CONFIG_DIR = Path(os.environ.get("XDG_CONFIG_HOME", Path.home() / ".config")) / "nox-cli"
|
||||
_APIKEYS_FILE = _CONFIG_DIR / "apikeys.json"
|
||||
|
||||
# ── Complete service registry ──────────────────────────────────────────
|
||||
# Format: key_name → {"display": str, "public": bool}
|
||||
# public=True → no key needed, always active
|
||||
# public=False → requires a real API key (goes into apikeys.json)
|
||||
SERVICE_REGISTRY: Dict[str, Dict] = {
|
||||
# ── Public / keyless ──────────────────────────────────────────────
|
||||
"alienvault_otx_domain": {"display": "AlienVault OTX (Domain)", "public": True},
|
||||
"alienvault_otx_ip": {"display": "AlienVault OTX (IP)", "public": True},
|
||||
"alienvault_otx_malware": {"display": "AlienVault OTX (Malware)", "public": True},
|
||||
"alienvault_otx_user": {"display": "AlienVault OTX (User)", "public": True},
|
||||
"anubis_subdomains": {"display": "Anubis Subdomains", "public": True},
|
||||
"bgpview_ip": {"display": "BGPView IP", "public": True},
|
||||
"checkleaked": {"display": "CheckLeaked", "public": True},
|
||||
"crt_sh": {"display": "crt.sh", "public": True},
|
||||
"cve_search": {"display": "CVE Search", "public": True},
|
||||
"cxsecurity": {"display": "CXSecurity", "public": True},
|
||||
"duckduckgo_api": {"display": "Google / DDG Dorks", "public": True},
|
||||
"emailrep_io": {"display": "EmailRep.io", "public": True},
|
||||
"github_users": {"display": "GitHub Users", "public": True},
|
||||
"gitlab_search": {"display": "GitLab Search", "public": True},
|
||||
"gravatar": {"display": "Gravatar", "public": True},
|
||||
"hackernews_user": {"display": "HackerNews User", "public": True},
|
||||
"hackertarget_dnslookup": {"display": "HackerTarget DNS Lookup", "public": True},
|
||||
"hackertarget_hostsearch": {"display": "HackerTarget Host Search", "public": True},
|
||||
"hackertarget_reverseip": {"display": "HackerTarget Reverse IP", "public": True},
|
||||
"hackertarget_whois": {"display": "WHOIS (HackerTarget)", "public": True},
|
||||
"hudsonrock_osint": {"display": "HudsonRock OSINT", "public": True},
|
||||
"ipapi_co": {"display": "ipapi.co", "public": True},
|
||||
"ipinfo_io": {"display": "IPInfo.io", "public": True},
|
||||
"ipvigilante": {"display": "IPVigilante", "public": True},
|
||||
"keybase_lookup": {"display": "Keybase Lookup", "public": True},
|
||||
"keybase_proofs": {"display": "Keybase Proofs", "public": True},
|
||||
"maltiverse_ip": {"display": "Maltiverse IP", "public": True},
|
||||
"npm_user": {"display": "NPM User", "public": True},
|
||||
"packetstorm": {"display": "PacketStorm", "public": True},
|
||||
"phishtank_check": {"display": "PhishTank", "public": True},
|
||||
"pulsedive": {"display": "Pulsedive (Free)", "public": True},
|
||||
"pypi_user": {"display": "PyPI User", "public": True},
|
||||
"reddit_user": {"display": "Reddit User", "public": True},
|
||||
"robtex_ip": {"display": "Robtex IP", "public": True},
|
||||
"scamwatcher": {"display": "ScamWatcher", "public": True},
|
||||
"social_scan": {"display": "Social Scan", "public": True},
|
||||
"sublist3r_api": {"display": "Sublist3r API", "public": True},
|
||||
"threatcrowd_domain": {"display": "ThreatCrowd (Domain)", "public": True},
|
||||
"threatcrowd_email": {"display": "ThreatCrowd (Email)", "public": True},
|
||||
"threatminer_domain": {"display": "ThreatMiner (Domain)", "public": True},
|
||||
"threatminer_ip": {"display": "ThreatMiner (IP)", "public": True},
|
||||
"urlscan_search": {"display": "URLScan.io", "public": True},
|
||||
"vigilante_pw": {"display": "Vigilante.pw", "public": True},
|
||||
"wayback_machine": {"display": "Wayback Machine", "public": True},
|
||||
# ── Private / key-required ────────────────────────────────────────
|
||||
"ABSTRACT_API_KEY": {"display": "Abstract Email Validation", "public": False},
|
||||
"ABUSEIPDB_API_KEY": {"display": "AbuseIPDB", "public": False},
|
||||
"ANYRUN_API_KEY": {"display": "Any.run", "public": False},
|
||||
"BA_API_KEY": {"display": "BreachAware", "public": False},
|
||||
"BD_API_KEY": {"display": "BreachDirectory", "public": False},
|
||||
"BINARYEDGE_API_KEY": {"display": "BinaryEdge", "public": False},
|
||||
"BING_API_KEY": {"display": "Bing Search API", "public": False},
|
||||
"CENSYS_AUTH_BASE64": {"display": "Censys", "public": False},
|
||||
"CIRCL_AUTH_BASE64": {"display": "CIRCL.lu PDNS", "public": False},
|
||||
"CIT0DAY_API_KEY": {"display": "Cit0day", "public": False},
|
||||
"CLEARBIT_API_KEY": {"display": "Clearbit Enrich", "public": False},
|
||||
"CRIMINALIP_API_KEY": {"display": "CriminalIP", "public": False},
|
||||
"DEHASHED_AUTH_BASE64": {"display": "Dehashed", "public": False},
|
||||
"DNSDB_API_KEY": {"display": "DNSDB Passive DNS", "public": False},
|
||||
"DT_AUTH_BASE64": {"display": "DomainTools WHOIS", "public": False},
|
||||
"EXTREME_API_KEY": {"display": "Extreme IP Lookup", "public": False},
|
||||
"FLP_API_KEY": {"display": "FraudLabsPro", "public": False},
|
||||
"FOFA_API_KEY": {"display": "FOFA", "public": False},
|
||||
"FOFA_EMAIL": {"display": "FOFA (account email)", "public": False},
|
||||
"FULLCONTACT_API_KEY": {"display": "FullContact", "public": False},
|
||||
"GITHUB_TOKEN": {"display": "GitHub (Code/Repo Search)", "public": False},
|
||||
"GOOGLE_API_KEY": {"display": "Google Safe Browsing", "public": False},
|
||||
"GOOGLE_CX_KEY": {"display": "Google Custom Search (API key)", "public": False},
|
||||
"GOOGLE_CX_ID": {"display": "Google Custom Search (CX ID)", "public": False},
|
||||
"GREYNOISE_API_KEY": {"display": "GreyNoise", "public": False},
|
||||
"HASHES_API_KEY": {"display": "Hashes.org", "public": False},
|
||||
"HIBP_API_KEY": {"display": "HaveIBeenPwned", "public": False},
|
||||
"HIPPO_API_KEY": {"display": "EmailHippo", "public": False},
|
||||
"HUNTER_API_KEY": {"display": "Hunter.io", "public": False},
|
||||
"HYBRID_API_KEY": {"display": "Hybrid Analysis", "public": False},
|
||||
"INTELX_API_KEY": {"display": "IntelX", "public": False},
|
||||
"INTEZER_API_KEY": {"display": "Intezer", "public": False},
|
||||
"IPDATA_API_KEY": {"display": "IPData.co", "public": False},
|
||||
"IPGEO_API_KEY": {"display": "IPGeolocation.io", "public": False},
|
||||
"IPINFODB_API_KEY": {"display": "IPInfoDB", "public": False},
|
||||
"IPQS_API_KEY": {"display": "IPQualityScore", "public": False},
|
||||
"IPSTACK_API_KEY": {"display": "IPStack", "public": False},
|
||||
"JOE_API_KEY": {"display": "Joe Sandbox", "public": False},
|
||||
"LEAKCHECK_API_KEY": {"display": "LeakCheck", "public": False},
|
||||
"LEAKIX_API_KEY": {"display": "LeakIX", "public": False},
|
||||
"LEAKSTATS_API_KEY": {"display": "LeakStats.pw", "public": False},
|
||||
"MAILBOX_API_KEY": {"display": "Mailboxlayer", "public": False},
|
||||
"MALSHARE_API_KEY": {"display": "MalShare", "public": False},
|
||||
"METADEFENDER_API_KEY": {"display": "MetaDefender", "public": False},
|
||||
"MISP_API_KEY": {"display": "MISP", "public": False},
|
||||
"NUMVERIFY_API_KEY": {"display": "Numverify", "public": False},
|
||||
"ONYPHE_API_KEY": {"display": "Onyphe", "public": False},
|
||||
"PASSIVETOTAL_AUTH_BASE64": {"display": "PassiveTotal / RiskIQ", "public": False},
|
||||
"PIPL_API_KEY": {"display": "Pipl", "public": False},
|
||||
"PULSEDIVE_API_KEY": {"display": "Pulsedive (Premium)", "public": False},
|
||||
"RF_TOKEN": {"display": "Recorded Future", "public": False},
|
||||
"SECURITYTRAILS_API_KEY": {"display": "SecurityTrails", "public": False},
|
||||
"SHODAN_API_KEY": {"display": "Shodan", "public": False},
|
||||
"SNUSBASE_API_KEY": {"display": "Snusbase", "public": False},
|
||||
"SPYCLOUD_API_KEY": {"display": "SpyCloud", "public": False},
|
||||
"SPYONWEB_API_KEY": {"display": "SpyOnWeb", "public": False},
|
||||
"SPYSE_API_KEY": {"display": "Spyse", "public": False},
|
||||
"TC_API_KEY": {"display": "ThreatConnect", "public": False},
|
||||
"TINES_API_KEY": {"display": "Tines Breach", "public": False},
|
||||
"TP_API_KEY": {"display": "ThreatPortal", "public": False},
|
||||
"TWITTER_BEARER_TOKEN": {"display": "Twitter / X API v2", "public": False},
|
||||
"URLVOID_API_KEY": {"display": "URLVoid", "public": False},
|
||||
"VIEWDNS_API_KEY": {"display": "ViewDNS", "public": False},
|
||||
"VIRUSTOTAL_API_KEY": {"display": "VirusTotal", "public": False},
|
||||
"VULNERS_API_KEY": {"display": "Vulners", "public": False},
|
||||
"WF_API_KEY": {"display": "WhoisFreaks", "public": False},
|
||||
"WHOISXML_API_KEY": {"display": "WhoisXML API", "public": False},
|
||||
"WHOXY_API_KEY": {"display": "Whoxy WHOIS", "public": False},
|
||||
"ZEROBOUNCE_API_KEY": {"display": "ZeroBounce", "public": False},
|
||||
"ZOOMEYE_API_KEY": {"display": "ZoomEye", "public": False},
|
||||
}
|
||||
|
||||
_PRIVATE_KEYS = {k: v for k, v in SERVICE_REGISTRY.items() if not v["public"]}
|
||||
|
||||
|
||||
# ── Store helpers ──────────────────────────────────────────────────────
|
||||
|
||||
def _default_store() -> Dict[str, str]:
|
||||
"""Return a dict of all private service keys set to UNIVERSAL_PLACEHOLDER."""
|
||||
return {k: UNIVERSAL_PLACEHOLDER for k in _PRIVATE_KEYS}
|
||||
|
||||
|
||||
def _write_store(data: Dict[str, str]) -> None:
|
||||
"""Atomically write data to apikeys.json with chmod 0600."""
|
||||
try:
|
||||
_CONFIG_DIR.mkdir(mode=0o700, parents=True, exist_ok=True)
|
||||
_CONFIG_DIR.chmod(0o700)
|
||||
tmp = _APIKEYS_FILE.with_suffix(".tmp")
|
||||
tmp.write_text(json.dumps(data, indent=4, sort_keys=True), encoding="utf-8")
|
||||
tmp.replace(_APIKEYS_FILE)
|
||||
_APIKEYS_FILE.chmod(0o600)
|
||||
except PermissionError as exc:
|
||||
raise RuntimeError(f"[config_handler] Cannot write {_APIKEYS_FILE}: {exc}") from exc
|
||||
|
||||
|
||||
def _load_store() -> Dict[str, str]:
|
||||
"""Load apikeys.json, creating it with defaults if absent. Self-heals on corrupt files."""
|
||||
_CONFIG_DIR.mkdir(mode=0o700, parents=True, exist_ok=True)
|
||||
_CONFIG_DIR.chmod(0o700)
|
||||
if not _APIKEYS_FILE.exists():
|
||||
print(" \033[92m[+]\033[0m Initializing NOX Environment in ~/.config/nox-cli/")
|
||||
_write_store(_default_store())
|
||||
return _default_store()
|
||||
try:
|
||||
text = _APIKEYS_FILE.read_text(encoding="utf-8").strip()
|
||||
if not text:
|
||||
raise json.JSONDecodeError("Empty file", "", 0)
|
||||
data = json.loads(text)
|
||||
if not isinstance(data, dict):
|
||||
raise json.JSONDecodeError("Root is not a JSON object", text, 0)
|
||||
# Back-fill keys added in newer versions
|
||||
new_keys = {k: UNIVERSAL_PLACEHOLDER for k in _PRIVATE_KEYS if k not in data}
|
||||
if new_keys:
|
||||
data.update(new_keys)
|
||||
_write_store(data)
|
||||
return data
|
||||
except json.JSONDecodeError:
|
||||
bak = _APIKEYS_FILE.with_suffix(".json.bak")
|
||||
_APIKEYS_FILE.rename(bak)
|
||||
print(f"[!] Malformed apikeys.json detected — backed up to {bak.name} and reset to defaults.")
|
||||
defaults = _default_store()
|
||||
_write_store(defaults)
|
||||
return defaults
|
||||
except PermissionError as exc:
|
||||
raise RuntimeError(f"[config_handler] Cannot read {_APIKEYS_FILE}: {exc}") from exc
|
||||
|
||||
|
||||
# ── ConfigManager ──────────────────────────────────────────────────────
|
||||
|
||||
class ConfigManager:
|
||||
"""
|
||||
Unified API key manager.
|
||||
|
||||
Resolution order per key:
|
||||
1. Environment variable (exact key name)
|
||||
2. ~/.config/nox-cli/apikeys.json
|
||||
3. Returns None if value equals UNIVERSAL_PLACEHOLDER or is absent
|
||||
"""
|
||||
|
||||
_cache: Dict[str, Optional[str]] = {}
|
||||
_store: Optional[Dict[str, str]] = None
|
||||
|
||||
@classmethod
|
||||
def _get_store(cls) -> Dict[str, str]:
|
||||
if cls._store is None:
|
||||
cls._store = _load_store()
|
||||
return cls._store
|
||||
|
||||
@classmethod
|
||||
def get_key(cls, key_name: str) -> Optional[str]:
|
||||
"""Return the configured value, or None if missing/placeholder."""
|
||||
if key_name in cls._cache:
|
||||
return cls._cache[key_name]
|
||||
val = os.environ.get(key_name, "") or cls._get_store().get(key_name, "")
|
||||
result = None if (not val or val == UNIVERSAL_PLACEHOLDER) else val
|
||||
cls._cache[key_name] = result
|
||||
return result
|
||||
|
||||
# Backward-compatible alias used by nox.py internals
|
||||
get = get_key
|
||||
|
||||
@classmethod
|
||||
def set(cls, key_name: str, value: str) -> None:
|
||||
"""Persist a key to apikeys.json and update the in-memory cache."""
|
||||
store = cls._get_store()
|
||||
store[key_name] = value
|
||||
_write_store(store)
|
||||
cls._cache[key_name] = None if value == UNIVERSAL_PLACEHOLDER else value
|
||||
|
||||
@classmethod
|
||||
def config_path(cls) -> Path:
|
||||
return _APIKEYS_FILE
|
||||
@@ -0,0 +1,119 @@
|
||||
"""
|
||||
sources/helpers/cracker.py
|
||||
Resilient async hash cracker for NOX autoscan.
|
||||
|
||||
Detects MD5 / SHA1 / SHA256 / bcrypt hashes inside breach records,
|
||||
fires background crack attempts against available APIs, and returns
|
||||
results without ever blocking the main pivot pipeline.
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
import re
|
||||
from typing import List, Optional, Tuple
|
||||
|
||||
# C2: MD5 and NTLM share the same 32-char hex pattern.
|
||||
# We list md5 first (most common in breach data) but also accept ntlm
|
||||
# so callers can query NTLM-specific APIs when needed.
|
||||
_PATTERNS: List[Tuple[str, re.Pattern]] = [
|
||||
("bcrypt", re.compile(r"^\$2[aby]?\$\d{2}\$.{53}$")),
|
||||
("sha256", re.compile(r"^[a-f0-9]{64}$", re.I)),
|
||||
("sha1", re.compile(r"^[a-f0-9]{40}$", re.I)),
|
||||
("md5", re.compile(r"^[a-f0-9]{32}$", re.I)),
|
||||
# ntlm shares the 32-char hex pattern — detected as md5 first,
|
||||
# but async_crack queries both md5 and ntlm APIs for 32-char hashes.
|
||||
]
|
||||
|
||||
# Writes to ~/.config/nox-cli/logs/nox_system.log — never to terminal
|
||||
_syslog = logging.getLogger("nox.system")
|
||||
|
||||
# Per-API timeout — each individual rainbow-table query budget
|
||||
_API_TIMEOUT = 8
|
||||
# Global crack budget — hard cap regardless of API count or response order
|
||||
CRACK_TIMEOUT = 20
|
||||
|
||||
|
||||
def detect_hash(value: str) -> Optional[str]:
|
||||
"""Return hash type string if value matches a known hash pattern, else None."""
|
||||
v = value.strip()
|
||||
for htype, pat in _PATTERNS:
|
||||
if pat.match(v):
|
||||
return htype
|
||||
return None
|
||||
|
||||
|
||||
async def _query_api(session, url: str, fmt: str) -> Optional[str]:
|
||||
"""Single API query — returns plaintext or None. Never raises."""
|
||||
try:
|
||||
import aiohttp
|
||||
to = aiohttp.ClientTimeout(total=_API_TIMEOUT)
|
||||
async with session.get(url, timeout=to) as resp:
|
||||
if resp.status != 200:
|
||||
return None
|
||||
if fmt == "text":
|
||||
text = (await resp.text()).strip()
|
||||
# Reject empty, too-long, or obvious error responses
|
||||
if not text or len(text) > 128:
|
||||
return None
|
||||
tl = text.lower()
|
||||
if any(tl.startswith(p) for p in ("not found", "error", "invalid", "no result", "not in", "cmd5-error", "not exist", "code erreur", "erreur", "unknown")):
|
||||
return None
|
||||
return text
|
||||
data = await resp.json(content_type=None)
|
||||
return data.get("result") or data.get("plaintext") or data.get("plain") or None
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
|
||||
async def async_crack(session, hash_value: str, hash_type: str) -> Optional[str]:
|
||||
"""
|
||||
Query multiple rainbow-table APIs concurrently.
|
||||
Returns first plaintext found, or None. bcrypt is skipped.
|
||||
|
||||
C1: create tasks upfront for cancellation, but await each via asyncio.shield
|
||||
inside as_completed — no double wait_for wrapping.
|
||||
C2: for 32-char hex (md5/ntlm ambiguity), also query NTLM-specific APIs.
|
||||
|
||||
Per-API timeout: 8s. Global budget: 20s (CRACK_TIMEOUT).
|
||||
All tasks are cancelled as soon as the first result is found.
|
||||
"""
|
||||
if hash_type == "bcrypt":
|
||||
return None
|
||||
|
||||
h = hash_value.strip().lower()
|
||||
apis = [
|
||||
(f"https://www.nitrxgen.net/md5db/{h}", "text"),
|
||||
(f"https://hashes.com/en/api/hash?hash={h}", "json"),
|
||||
(f"https://hash.help/api/lookup/{h}", "json"),
|
||||
(f"https://hashkiller.io/api/search.php?hash={h}", "json"),
|
||||
(f"https://md5decrypt.net/Api/api.php?hash={h}&hash_type={hash_type}&email=&code=", "text"),
|
||||
(f"https://www.cmd5.org/api.ashx?hash={h}", "text"),
|
||||
]
|
||||
# C2: for 32-char hashes (md5/ntlm ambiguous), add NTLM-specific endpoint
|
||||
if hash_type == "md5" and len(h) == 32:
|
||||
apis.append((f"https://hashes.com/en/api/hash?hash={h}&type=ntlm", "json"))
|
||||
|
||||
# C1: create tasks so we can cancel them; shield each before passing to wait_for
|
||||
# so cancellation of the shield future does not cancel the underlying task prematurely.
|
||||
tasks = [asyncio.create_task(_query_api(session, url, fmt)) for url, fmt in apis]
|
||||
result: Optional[str] = None
|
||||
try:
|
||||
for fut in asyncio.as_completed(tasks):
|
||||
try:
|
||||
res = await asyncio.wait_for(asyncio.shield(fut), timeout=_API_TIMEOUT)
|
||||
except (asyncio.TimeoutError, asyncio.CancelledError):
|
||||
continue
|
||||
except Exception:
|
||||
continue
|
||||
if res:
|
||||
result = res
|
||||
break
|
||||
except Exception:
|
||||
pass
|
||||
finally:
|
||||
# Cancel all remaining tasks and await to suppress pending-task warnings
|
||||
for t in tasks:
|
||||
if not t.done():
|
||||
t.cancel()
|
||||
await asyncio.gather(*[t for t in tasks if not t.done()], return_exceptions=True)
|
||||
return result
|
||||
@@ -0,0 +1,658 @@
|
||||
"""
|
||||
sources/helpers/reporting.py
|
||||
NOX Enterprise Reporting — Executive Summary, Pivot Chain, Data Sanitization.
|
||||
"""
|
||||
|
||||
import hashlib
|
||||
import html as _html
|
||||
import json
|
||||
import re
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict, List
|
||||
|
||||
# ── Noise patterns stripped from all report output ────────────────────
|
||||
_NOISE_RE = re.compile(
|
||||
r"(Traceback \(most recent|File \".*\.py\"|TimeoutError|ProxyError"
|
||||
r"|ConnectionError|aiohttp\.|ClientConnector|ssl\.|asyncio\."
|
||||
r"|Task exception|NoneType|Object of type)",
|
||||
re.I,
|
||||
)
|
||||
_CTRL_RE = re.compile(r"[\x00-\x08\x0b\x0c\x0e-\x1f\x7f-\x9f]")
|
||||
|
||||
|
||||
def _nox_ver() -> str:
|
||||
try:
|
||||
from nox import VERSION # type: ignore
|
||||
return VERSION
|
||||
except ImportError:
|
||||
return "1.0.0"
|
||||
|
||||
|
||||
def _clean(v: Any, maxlen: int = 200) -> str:
|
||||
"""Strip control chars, technical noise, HTML-escape, truncate."""
|
||||
s = str(v) if v is not None else ""
|
||||
s = _CTRL_RE.sub("", s)
|
||||
if _NOISE_RE.search(s):
|
||||
return ""
|
||||
return _html.escape(s[:maxlen])
|
||||
|
||||
|
||||
def _raw(v: Any, maxlen: int = 200) -> str:
|
||||
"""Strip control chars only — no HTML escaping (PDF / plain-text paths)."""
|
||||
s = str(v) if v is not None else ""
|
||||
s = _CTRL_RE.sub("", s)
|
||||
if _NOISE_RE.search(s):
|
||||
return ""
|
||||
return s[:maxlen]
|
||||
|
||||
|
||||
def _pdf_safe(s: str, maxlen: int = 180) -> str:
|
||||
# D4: sanitize for fpdf2 core fonts (latin-1 subset).
|
||||
# NFKD normalization decomposes accented chars (é→e + combining accent)
|
||||
# so common accented Latin characters survive as their base letter.
|
||||
# Truly non-latin-1 chars (Cyrillic, CJK, etc.) become '?' — intentional:
|
||||
# fpdf2 core fonts cannot render them and would raise UnicodeEncodeError.
|
||||
s = _raw(s, maxlen)
|
||||
try:
|
||||
import unicodedata
|
||||
normalized = unicodedata.normalize("NFKD", s)
|
||||
return normalized.encode("ascii", errors="replace").decode("ascii")
|
||||
except Exception:
|
||||
return s.encode("latin-1", errors="replace").decode("latin-1")
|
||||
|
||||
|
||||
def _rget(r: Any, k: str) -> str:
|
||||
if isinstance(r, dict):
|
||||
return str(r.get(k, "") or "")
|
||||
return str(getattr(r, k, "") or "")
|
||||
|
||||
|
||||
# ── Executive summary builder ─────────────────────────────────────────
|
||||
|
||||
def build_exec_summary(data: dict) -> dict:
|
||||
"""
|
||||
Returns a dict with all dashboard KPIs needed by every format.
|
||||
Expects data keys: records, analysis, scan_meta (optional).
|
||||
"""
|
||||
records = data.get("records", [])
|
||||
meta = data.get("scan_meta", {}) or {}
|
||||
analysis = data.get("analysis", {}) or {}
|
||||
|
||||
cleartext = sum(1 for r in records if _rget(r, "password"))
|
||||
nodes = len({_rget(r, "email") or _rget(r, "username") for r in records} - {""})
|
||||
elapsed = meta.get("elapsed_seconds")
|
||||
depth = meta.get("pivot_depth", len(data.get("pivot_chain", [])))
|
||||
|
||||
buckets: Dict[str, int] = {"Critical": 0, "High": 0, "Medium": 0, "Low": 0, "Info": 0}
|
||||
for r in records:
|
||||
rs = float(_rget(r, "risk_score") or 0)
|
||||
if rs >= 90: buckets["Critical"] += 1
|
||||
elif rs >= 70: buckets["High"] += 1
|
||||
elif rs >= 40: buckets["Medium"] += 1
|
||||
elif rs >= 10: buckets["Low"] += 1
|
||||
else: buckets["Info"] += 1
|
||||
|
||||
return {
|
||||
"total_records": len(records),
|
||||
"nodes_discovered": nodes,
|
||||
"cleartext_passwords": cleartext,
|
||||
"pivot_depth": depth,
|
||||
"elapsed": f"{elapsed:.1f}s" if elapsed is not None else "N/A",
|
||||
"buckets": buckets,
|
||||
"hvt_count": analysis.get("hvt_count", sum(1 for r in records if getattr(r, "is_hvt", False))),
|
||||
}
|
||||
|
||||
|
||||
# ── Pivot chain renderer ──────────────────────────────────────────────
|
||||
|
||||
def render_pivot_chain(data: dict) -> List[str]:
|
||||
"""
|
||||
Build a human-readable pivot chain.
|
||||
D2: check pivot_log first before falling back to record-based reconstruction.
|
||||
"""
|
||||
chain = data.get("pivot_chain") or []
|
||||
target = _raw(data.get("target", "?"))
|
||||
|
||||
# D2: if pivot_log is available, build chain from it (accurate tree)
|
||||
pivot_log = data.get("pivot_log") or []
|
||||
if pivot_log:
|
||||
lines: List[str] = []
|
||||
for e in pivot_log:
|
||||
depth = e.get("depth", 0)
|
||||
asset = _raw(e.get("asset", ""))
|
||||
phase = _raw(e.get("found_in", e.get("source", "?")))
|
||||
parent = _raw(e.get("parent") or "")
|
||||
prefix = " " * depth
|
||||
if depth == 0:
|
||||
lines.append(f"[SEED] {asset}")
|
||||
else:
|
||||
lines.append(f"{prefix}└─ [{phase}] {asset} ← {parent}")
|
||||
return lines if lines else [f"[SEED] {target} (no pivot data)"]
|
||||
|
||||
if len(chain) <= 1:
|
||||
# No pivot data — reconstruct best-effort from records
|
||||
records = data.get("records", [])
|
||||
lines = [f"[SEED] {target}"]
|
||||
seen: set = {target.lower()}
|
||||
for r in records[:40]:
|
||||
src = _raw(_rget(r, "source"))
|
||||
em = _raw(_rget(r, "email"))
|
||||
usr = _raw(_rget(r, "username"))
|
||||
ident = em or usr
|
||||
if not ident or ident.lower() in seen:
|
||||
continue
|
||||
seen.add(ident.lower())
|
||||
lines.append(f" └─ [{src}] → {ident}")
|
||||
dork_results = data.get("dork_results") or []
|
||||
for d in dork_results[:5]:
|
||||
url = _raw(d.get("url", ""))
|
||||
if url and url.lower() not in seen:
|
||||
seen.add(url.lower())
|
||||
lines.append(f" └─ [Dork] → {url[:80]}")
|
||||
return lines if len(lines) > 1 else [f"[SEED] {target} (no pivot data)"]
|
||||
|
||||
# Ordered pivot chain from AvalancheScanner
|
||||
lines = [f"[SEED] {_raw(chain[0])}"]
|
||||
for node in chain[1:]:
|
||||
lines.append(f" └─ [Pivot] → {_raw(node)}")
|
||||
return lines
|
||||
|
||||
|
||||
# ── JSON report ───────────────────────────────────────────────────────
|
||||
|
||||
def to_json(data: dict, path: str) -> None:
|
||||
summary = build_exec_summary(data)
|
||||
chain = render_pivot_chain(data)
|
||||
records = data.get("records", [])
|
||||
|
||||
def _ser(o):
|
||||
try:
|
||||
from enum import Enum
|
||||
if isinstance(o, Enum):
|
||||
return o.name
|
||||
except ImportError:
|
||||
pass
|
||||
if hasattr(o, "to_dict"):
|
||||
return o.to_dict()
|
||||
return str(o)
|
||||
|
||||
clean_records = []
|
||||
for r in records:
|
||||
d = r.to_dict() if hasattr(r, "to_dict") else (r if isinstance(r, dict) else {})
|
||||
# drop noise fields
|
||||
clean_records.append({
|
||||
k: v for k, v in d.items()
|
||||
if k not in ("raw_data", "metadata") and not _NOISE_RE.search(str(v or ""))
|
||||
})
|
||||
|
||||
try:
|
||||
from nox import VERSION as _NOX_VERSION # type: ignore
|
||||
except ImportError:
|
||||
_NOX_VERSION = "1.0.0"
|
||||
|
||||
# Include dork and scrape results in JSON output
|
||||
dork_results = data.get("dork_results", []) or []
|
||||
scrape_results = data.get("scrape_results", {}) or {}
|
||||
|
||||
# D3: apply consistent cap (1000) — same as HTML
|
||||
_RECORD_CAP = 1000
|
||||
|
||||
out_data = {
|
||||
"framework": f"NOX v{_NOX_VERSION}",
|
||||
"generated": datetime.now().isoformat(),
|
||||
"target": data.get("target", ""),
|
||||
# J3: self-describing metadata block
|
||||
"_meta": {
|
||||
"scan_id": hashlib.sha256(
|
||||
f"{data.get('target','')}{datetime.now().isoformat()}".encode()
|
||||
).hexdigest()[:16],
|
||||
"target": data.get("target", ""),
|
||||
"timestamp": datetime.now().isoformat(),
|
||||
"nox_version": _NOX_VERSION,
|
||||
"sources_queried": summary.get("total_records", 0),
|
||||
"pivot_depth_reached": summary.get("pivot_depth", 0),
|
||||
"record_cap": _RECORD_CAP,
|
||||
"truncated": len(clean_records) > _RECORD_CAP,
|
||||
},
|
||||
"executive_summary": summary,
|
||||
"pivot_chain": chain,
|
||||
"records": clean_records[:_RECORD_CAP],
|
||||
"dork_results": dork_results,
|
||||
"scrape_results": scrape_results,
|
||||
}
|
||||
Path(path).write_text(json.dumps(out_data, indent=2, default=_ser), encoding="utf-8")
|
||||
print(f"[+] JSON report saved: {path}")
|
||||
|
||||
|
||||
# ── HTML report ───────────────────────────────────────────────────────
|
||||
|
||||
_CSS = (
|
||||
"*{margin:0;padding:0;box-sizing:border-box}"
|
||||
"body{font-family:'Courier New',monospace;background:#0a0a0a;color:#e0e0e0;padding:20px}"
|
||||
".hdr{text-align:center;padding:28px;border:1px solid #333;margin-bottom:18px;background:#111}"
|
||||
".hdr h1{color:#00ff41;font-size:26px;letter-spacing:4px}"
|
||||
".hdr p{color:#888;margin-top:5px;font-size:12px}"
|
||||
".kpis{display:grid;grid-template-columns:repeat(auto-fit,minmax(160px,1fr));gap:10px;margin:14px 0}"
|
||||
".kpi{background:#111;border:1px solid #333;padding:16px;text-align:center}"
|
||||
".kpi .n{font-size:30px;font-weight:bold;color:#00ff41}"
|
||||
".kpi .l{color:#888;font-size:10px;margin-top:3px}"
|
||||
".kpi.warn .n{color:#ff6600} .kpi.crit .n{color:#ff0040}"
|
||||
".sec{margin:18px 0} .sec h2{color:#00ff41;border-bottom:1px solid #333;padding-bottom:5px;margin-bottom:10px}"
|
||||
".chain{background:#0d1a0d;border:1px solid #1a3a1a;padding:12px;font-size:11px;color:#00cc33;word-break:break-all;margin:8px 0}"
|
||||
"table{width:100%;border-collapse:collapse} th,td{padding:7px;border:1px solid #222;font-size:11px;word-break:break-all}"
|
||||
"th{background:#1a1a1a;color:#00ff41;text-transform:uppercase;font-size:10px} td{background:#0d0d0d}"
|
||||
"tr.c td{background:#1a0005} tr.h td{background:#1a0a00} tr.m td{background:#1a1500}"
|
||||
".pw{color:#ff0040;font-weight:bold}"
|
||||
)
|
||||
|
||||
|
||||
def to_html(data: dict, path: str) -> None:
|
||||
summary = build_exec_summary(data)
|
||||
chain = render_pivot_chain(data)
|
||||
target = _clean(data.get("target", "Unknown"))
|
||||
records = data.get("records", [])
|
||||
ts = datetime.now().strftime("%Y-%m-%d %H:%M:%S UTC")
|
||||
|
||||
# KPI dashboard
|
||||
kpis = (
|
||||
f'<div class="kpi"><div class="n">{summary["total_records"]}</div><div class="l">TOTAL RECORDS</div></div>'
|
||||
f'<div class="kpi"><div class="n">{summary["nodes_discovered"]}</div><div class="l">NODES DISCOVERED</div></div>'
|
||||
f'<div class="kpi crit"><div class="n">{summary["cleartext_passwords"]}</div><div class="l">CLEARTEXT PASSWORDS</div></div>'
|
||||
f'<div class="kpi warn"><div class="n">{summary["hvt_count"]}</div><div class="l">HIGH-VALUE TARGETS</div></div>'
|
||||
f'<div class="kpi"><div class="n">{summary["pivot_depth"]}</div><div class="l">PIVOT DEPTH</div></div>'
|
||||
f'<div class="kpi"><div class="n">{summary["elapsed"]}</div><div class="l">TOTAL TIME</div></div>'
|
||||
)
|
||||
|
||||
# Severity table
|
||||
sev_rows = "".join(
|
||||
f"<tr><td>{lvl}</td><td>{cnt}</td></tr>"
|
||||
for lvl, cnt in summary["buckets"].items() if cnt
|
||||
)
|
||||
|
||||
# Pivot chain
|
||||
chain_html = "".join(f'<div class="chain">{_clean(c)}</div>' for c in chain)
|
||||
|
||||
# Credential rows (top 500, noise-free)
|
||||
cred_rows = ""
|
||||
for r in records[:500]:
|
||||
rs = float(_rget(r, "risk_score") or 0)
|
||||
cls = "c" if rs >= 90 else "h" if rs >= 70 else "m" if rs >= 40 else ""
|
||||
em = _clean(_rget(r, "email") or _rget(r, "username"))
|
||||
pw = _clean(_rget(r, "password"))
|
||||
src = _clean(_rget(r, "source"))
|
||||
bd = _clean(_rget(r, "breach_date"))
|
||||
hvt = " ⚑" if getattr(r, "is_hvt", False) or (isinstance(r, dict) and r.get("is_hvt")) else ""
|
||||
cred_rows += (
|
||||
f"<tr class='{cls}'><td>{em}{hvt}</td>"
|
||||
f"<td class='pw'>{pw}</td><td>{src}</td><td>{bd}</td><td>{rs:.0f}</td></tr>"
|
||||
)
|
||||
|
||||
# Dork results section
|
||||
dork_results = data.get("dork_results", []) or []
|
||||
dork_rows = ""
|
||||
for h in dork_results:
|
||||
url = h.get("url", "")
|
||||
title = h.get("title", "") or h.get("dork", "")
|
||||
snippet = h.get("snippet", "")
|
||||
engine = h.get("engine", "")
|
||||
link = (f'<a href="{_clean(url)}" style="color:#00ff41" target="_blank">{_clean(url[:80])}</a>'
|
||||
if url else _clean(title[:80]))
|
||||
dork_rows += (
|
||||
f"<tr><td>{link}</td><td>{_clean(snippet[:120])}</td>"
|
||||
f"<td>{_clean(h.get('dork','')[:80])}</td><td>{_clean(engine)}</td></tr>"
|
||||
)
|
||||
dork_section = (
|
||||
f'<div class="sec"><h2>Dork Results ({len(dork_results)} hits)</h2>'
|
||||
f'<table><thead><tr><th>URL / Title</th><th>Snippet</th><th>Dork Query</th><th>Engine</th></tr></thead>'
|
||||
f'<tbody>{dork_rows if dork_rows else "<tr><td colspan=4 style=text-align:center>No dork hits</td></tr>"}</tbody></table></div>'
|
||||
)
|
||||
|
||||
# Scrape results section
|
||||
scrape_results = data.get("scrape_results", {}) or {}
|
||||
pastes = scrape_results.get("pastes", [])
|
||||
creds_sc = scrape_results.get("credentials", [])
|
||||
tg_hits = scrape_results.get("telegram", [])
|
||||
mc_hits = scrape_results.get("dork_misconfigs", [])
|
||||
|
||||
paste_rows = ""
|
||||
for p in pastes:
|
||||
site = _clean(p.get("site", ""))
|
||||
pid = p.get("id", "")
|
||||
pats = _clean(", ".join(f"{k}({len(v)})" for k, v in (p.get("patterns") or {}).items()))
|
||||
paste_rows += f"<tr><td>{site}</td><td>{_clean(pid)}</td><td>{pats}</td></tr>"
|
||||
|
||||
cred_sc_rows = ""
|
||||
for c in creds_sc:
|
||||
cred_sc_rows += (
|
||||
f"<tr><td class='pw'>{_clean(c.get('raw','')[:120])}</td>"
|
||||
f"<td>{_clean(c.get('source',''))}</td><td>{_clean(c.get('paste_id',''))}</td></tr>"
|
||||
)
|
||||
|
||||
tg_rows = ""
|
||||
for t in tg_hits:
|
||||
ch = _clean(t.get("channel", ""))
|
||||
text = _clean(t.get("text", "")[:200])
|
||||
pats = _clean(", ".join(f"{k}({len(v)})" for k, v in (t.get("patterns") or {}).items()))
|
||||
link = f'<a href="https://t.me/s/{ch}" style="color:#00ff41" target="_blank">t.me/s/{ch}</a>'
|
||||
tg_rows += f"<tr><td>{link}</td><td>{text}</td><td>{pats}</td></tr>"
|
||||
|
||||
mc_rows = ""
|
||||
for m in mc_hits:
|
||||
url_m = m.get("url", "")
|
||||
title_m = _clean(m.get("title", "")[:80])
|
||||
dork_m = _clean(m.get("dork", "")[:80])
|
||||
link_m = (f'<a href="{_clean(url_m)}" style="color:#ff0040" target="_blank">{_clean(url_m[:80])}</a>'
|
||||
if url_m else title_m)
|
||||
mc_rows += f"<tr><td>{link_m}</td><td>{title_m}</td><td>{dork_m}</td></tr>"
|
||||
|
||||
scrape_section = (
|
||||
f'<div class="sec"><h2>Scrape Results</h2>'
|
||||
f'<h3 style="color:#aaa;margin:10px 0 5px">Pastes ({len(pastes)})</h3>'
|
||||
f'<table><thead><tr><th>Site</th><th>Paste ID</th><th>Patterns</th></tr></thead>'
|
||||
f'<tbody>{paste_rows or "<tr><td colspan=3 style=text-align:center>None</td></tr>"}</tbody></table>'
|
||||
f'<h3 style="color:#aaa;margin:10px 0 5px">Extracted Credentials ({len(creds_sc)})</h3>'
|
||||
f'<table><thead><tr><th>Raw Credential</th><th>Source</th><th>Paste ID</th></tr></thead>'
|
||||
f'<tbody>{cred_sc_rows or "<tr><td colspan=3 style=text-align:center>None</td></tr>"}</tbody></table>'
|
||||
f'<h3 style="color:#aaa;margin:10px 0 5px">Telegram CTI ({len(tg_hits)})</h3>'
|
||||
f'<table><thead><tr><th>Channel</th><th>Message</th><th>Patterns</th></tr></thead>'
|
||||
f'<tbody>{tg_rows or "<tr><td colspan=3 style=text-align:center>None</td></tr>"}</tbody></table>'
|
||||
f'<h3 style="color:#aaa;margin:10px 0 5px">Misconfigurations ({len(mc_hits)})</h3>'
|
||||
f'<table><thead><tr><th>URL</th><th>Title</th><th>Dork</th></tr></thead>'
|
||||
f'<tbody>{mc_rows or "<tr><td colspan=3 style=text-align:center>None</td></tr>"}</tbody></table>'
|
||||
f'</div>'
|
||||
)
|
||||
|
||||
page = (
|
||||
f'<!DOCTYPE html><html><head><meta charset="utf-8">'
|
||||
f'<title>NOX — {target}</title><style>{_CSS}</style></head><body>'
|
||||
f'<div class="hdr"><h1>[ NOX ]</h1>'
|
||||
f'<p>Target: {target} | {ts} | NOX v{_nox_ver()}</p></div>'
|
||||
f'<div class="sec"><h2>Executive Summary</h2>'
|
||||
f'<div class="kpis">{kpis}</div>'
|
||||
f'<table><thead><tr><th>Severity</th><th>Count</th></tr></thead>'
|
||||
f'<tbody>{sev_rows}</tbody></table></div>'
|
||||
f'<div class="sec"><h2>Pivot Chain</h2>{chain_html}</div>'
|
||||
f'{dork_section}'
|
||||
f'{scrape_section}'
|
||||
f'<div class="sec"><h2>Credential Records (top 500)</h2>'
|
||||
f'<table><thead><tr><th>Identity</th><th>Password</th><th>Source</th>'
|
||||
f'<th>Date</th><th>Risk</th></tr></thead><tbody>{cred_rows}</tbody></table></div>'
|
||||
f'</body></html>'
|
||||
)
|
||||
Path(path).write_text(page, encoding="utf-8")
|
||||
print(f"[+] HTML report saved: {path}")
|
||||
|
||||
|
||||
# ── PDF report (fpdf2) ────────────────────────────────────────────────
|
||||
|
||||
def to_pdf(data: dict, path: str, investigator_id: str = "NOX-AUTO") -> None:
|
||||
# D1: raise a clear error with install hint if fpdf2 is absent — never silently return.
|
||||
try:
|
||||
from fpdf import FPDF # type: ignore
|
||||
except ImportError:
|
||||
msg = "[!] fpdf2 not installed — PDF report cannot be generated. Run: pip install fpdf2"
|
||||
print(msg)
|
||||
raise RuntimeError(msg)
|
||||
|
||||
summary = build_exec_summary(data)
|
||||
chain = render_pivot_chain(data)
|
||||
target = _raw(data.get("target", "Unknown"))
|
||||
records = data.get("records", [])
|
||||
ts = datetime.now().strftime("%Y-%m-%d %H:%M:%S UTC")
|
||||
|
||||
class _PDF(FPDF):
|
||||
def header(self):
|
||||
self.set_font("Helvetica", "B", 8)
|
||||
self.set_text_color(120, 120, 120)
|
||||
self.cell(0, 5, "NOX - FORENSIC INTELLIGENCE REPORT - CONFIDENTIAL", align="R")
|
||||
self.ln(3)
|
||||
|
||||
def footer(self):
|
||||
self.set_y(-12)
|
||||
self.set_font("Helvetica", "", 8)
|
||||
self.set_text_color(150, 150, 150)
|
||||
self.cell(0, 5, _pdf_safe(f"Page {self.page_no()} | {target[:50]}"), align="C")
|
||||
|
||||
pdf = _PDF(orientation="P", unit="mm", format="A4")
|
||||
pdf.set_auto_page_break(auto=True, margin=15)
|
||||
pdf.set_margins(15, 15, 15)
|
||||
|
||||
# ── Cover page ────────────────────────────────────────────────────
|
||||
pdf.add_page()
|
||||
pdf.set_fill_color(15, 15, 15)
|
||||
pdf.rect(0, 0, 210, 297, "F")
|
||||
pdf.set_y(65)
|
||||
pdf.set_font("Helvetica", "B", 26)
|
||||
pdf.set_text_color(0, 220, 60)
|
||||
pdf.cell(0, 12, "FORENSIC INTELLIGENCE REPORT", align="C")
|
||||
pdf.ln(8)
|
||||
pdf.set_font("Helvetica", "B", 13)
|
||||
pdf.set_text_color(200, 200, 200)
|
||||
pdf.cell(0, 8, _pdf_safe(f"Target: {target}"), align="C")
|
||||
pdf.ln(6)
|
||||
pdf.set_font("Helvetica", "", 10)
|
||||
pdf.set_text_color(140, 140, 140)
|
||||
for line in [f"Generated: {ts}", f"Investigator: {investigator_id}",
|
||||
f"Framework: NOX v{_nox_ver()}", "Classification: RESTRICTED"]:
|
||||
pdf.cell(0, 6, _pdf_safe(line), align="C")
|
||||
pdf.ln(5)
|
||||
|
||||
# ── Executive Summary ─────────────────────────────────────────────
|
||||
pdf.add_page()
|
||||
pdf.set_fill_color(255, 255, 255)
|
||||
pdf.set_text_color(0, 0, 0)
|
||||
pdf.set_font("Helvetica", "B", 15)
|
||||
pdf.cell(0, 10, "Executive Summary", ln=True)
|
||||
pdf.set_draw_color(0, 180, 50)
|
||||
pdf.set_line_width(0.4)
|
||||
pdf.line(15, pdf.get_y(), 195, pdf.get_y())
|
||||
pdf.ln(4)
|
||||
|
||||
kpis = [
|
||||
("Total Time", summary["elapsed"]),
|
||||
("Nodes Discovered", str(summary["nodes_discovered"])),
|
||||
("Cleartext Passwords Found", str(summary["cleartext_passwords"])),
|
||||
("Pivot Depth", str(summary["pivot_depth"])),
|
||||
("Total Records", str(summary["total_records"])),
|
||||
("High-Value Targets", str(summary["hvt_count"])),
|
||||
]
|
||||
pdf.set_font("Helvetica", "B", 10)
|
||||
for label, value in kpis:
|
||||
pdf.set_fill_color(245, 245, 245)
|
||||
pdf.cell(95, 7, _pdf_safe(label), border=1, fill=True)
|
||||
pdf.set_font("Helvetica", "", 10)
|
||||
pdf.cell(80, 7, _pdf_safe(value), border=1, ln=True)
|
||||
pdf.set_font("Helvetica", "B", 10)
|
||||
pdf.ln(4)
|
||||
|
||||
# Severity breakdown
|
||||
pdf.set_font("Helvetica", "B", 11)
|
||||
pdf.cell(0, 7, "Severity Breakdown", ln=True)
|
||||
_sev_c = {"Critical": (220,0,30), "High": (220,100,0),
|
||||
"Medium": (200,180,0), "Low": (0,150,50), "Info": (100,100,100)}
|
||||
total_b = max(sum(summary["buckets"].values()), 1)
|
||||
for level, count in summary["buckets"].items():
|
||||
pdf.set_font("Helvetica", "", 9)
|
||||
pdf.cell(35, 6, _pdf_safe(level), border=1)
|
||||
pdf.cell(20, 6, str(count), border=1)
|
||||
bar_w = int(count / total_b * 120)
|
||||
x, y = pdf.get_x(), pdf.get_y()
|
||||
pdf.cell(125, 6, "", border=1)
|
||||
if bar_w:
|
||||
rc, gc, bc = _sev_c.get(level, (100, 100, 100))
|
||||
pdf.set_fill_color(rc, gc, bc)
|
||||
pdf.rect(x + 1, y + 1, bar_w, 4, "F")
|
||||
pdf.ln()
|
||||
|
||||
# ── Pivot Chain ───────────────────────────────────────────────────
|
||||
pdf.ln(5)
|
||||
pdf.set_font("Helvetica", "B", 11)
|
||||
pdf.cell(0, 7, "Pivot Chain Visualization", ln=True)
|
||||
pdf.line(15, pdf.get_y(), 195, pdf.get_y())
|
||||
pdf.ln(3)
|
||||
pdf.set_font("Courier", "", 8)
|
||||
pdf.set_fill_color(240, 255, 240)
|
||||
for c_line in chain:
|
||||
# Word-wrap long chains at 100 chars
|
||||
for chunk in [c_line[i:i+100] for i in range(0, max(len(c_line), 1), 100)]:
|
||||
pdf.set_x(15)
|
||||
pdf.cell(180, 5, _pdf_safe(chunk), border=0, ln=True, fill=True)
|
||||
pdf.ln(3)
|
||||
|
||||
# ── Credential Findings ───────────────────────────────────────────
|
||||
pdf.add_page()
|
||||
pdf.set_font("Helvetica", "B", 13)
|
||||
pdf.set_text_color(0, 0, 0)
|
||||
pdf.cell(0, 9, "Credential Findings", ln=True)
|
||||
pdf.line(15, pdf.get_y(), 195, pdf.get_y())
|
||||
pdf.ln(3)
|
||||
|
||||
cols = [("Identity", 60), ("Password", 45), ("Source", 35), ("Date", 25), ("Risk", 15)]
|
||||
|
||||
def _write_col_headers():
|
||||
pdf.set_font("Helvetica", "B", 8)
|
||||
pdf.set_fill_color(30, 30, 30)
|
||||
pdf.set_text_color(255, 255, 255)
|
||||
for col_name, col_w in cols:
|
||||
pdf.cell(col_w, 6, col_name, border=1, fill=True)
|
||||
pdf.ln()
|
||||
pdf.set_text_color(0, 0, 0)
|
||||
|
||||
_write_col_headers()
|
||||
|
||||
for r in records[:500]:
|
||||
pw = _rget(r, "password")
|
||||
if not pw and not _rget(r, "email") and not _rget(r, "username"):
|
||||
continue # skip noise rows with no actionable data
|
||||
rs = float(_rget(r, "risk_score") or 0)
|
||||
if rs >= 90: pdf.set_fill_color(255, 220, 220)
|
||||
elif rs >= 70: pdf.set_fill_color(255, 240, 220)
|
||||
else: pdf.set_fill_color(255, 255, 255)
|
||||
pdf.set_font("Helvetica", "", 7)
|
||||
# Auto page-break with repeated column headers (§5.1)
|
||||
if pdf.get_y() > pdf.h - 25:
|
||||
pdf.add_page()
|
||||
_write_col_headers()
|
||||
vals = [
|
||||
_pdf_safe(_rget(r, "email") or _rget(r, "username"), 38),
|
||||
_pdf_safe(pw, 28),
|
||||
_pdf_safe(_rget(r, "source"), 22),
|
||||
_pdf_safe(_rget(r, "breach_date"), 14),
|
||||
f"{rs:.0f}",
|
||||
]
|
||||
for val, (_, w) in zip(vals, cols):
|
||||
pdf.cell(w, 5, val, border=1, fill=True)
|
||||
pdf.ln()
|
||||
|
||||
# ── Dork Results ─────────────────────────────────────────────────
|
||||
dork_results = data.get("dork_results", []) or []
|
||||
if dork_results:
|
||||
pdf.add_page()
|
||||
pdf.set_font("Helvetica", "B", 13)
|
||||
pdf.set_text_color(0, 0, 0)
|
||||
pdf.cell(0, 9, _pdf_safe(f"Dork Results ({len(dork_results)} hits)"), ln=True)
|
||||
pdf.line(15, pdf.get_y(), 195, pdf.get_y())
|
||||
pdf.ln(3)
|
||||
pdf.set_font("Helvetica", "B", 8)
|
||||
pdf.set_fill_color(30, 30, 30); pdf.set_text_color(255, 255, 255)
|
||||
for col_name, col_w in [("URL / Title", 95), ("Snippet", 55), ("Engine", 30)]:
|
||||
pdf.cell(col_w, 6, col_name, border=1, fill=True)
|
||||
pdf.ln(); pdf.set_text_color(0, 0, 0)
|
||||
for h in dork_results[:200]:
|
||||
pdf.set_fill_color(245, 245, 255); pdf.set_font("Helvetica", "", 7)
|
||||
url = _pdf_safe(h.get("url", h.get("title", "")), 65)
|
||||
snippet = _pdf_safe(h.get("snippet", ""), 38)
|
||||
engine = _pdf_safe(h.get("engine", ""), 20)
|
||||
for val, w in zip([url, snippet, engine], [95, 55, 30]):
|
||||
pdf.cell(w, 5, val, border=1, fill=True)
|
||||
pdf.ln()
|
||||
|
||||
# ── Scrape Results ────────────────────────────────────────────────
|
||||
scrape_results = data.get("scrape_results", {}) or {}
|
||||
pastes = scrape_results.get("pastes", [])
|
||||
creds_sc = scrape_results.get("credentials", [])
|
||||
tg_hits = scrape_results.get("telegram", [])
|
||||
mc_hits = scrape_results.get("dork_misconfigs", [])
|
||||
|
||||
if pastes or creds_sc or tg_hits or mc_hits:
|
||||
pdf.add_page()
|
||||
pdf.set_font("Helvetica", "B", 13)
|
||||
pdf.set_text_color(0, 0, 0)
|
||||
pdf.cell(0, 9, "Scrape Results", ln=True)
|
||||
pdf.line(15, pdf.get_y(), 195, pdf.get_y())
|
||||
pdf.ln(3)
|
||||
|
||||
if pastes:
|
||||
pdf.set_font("Helvetica", "B", 10)
|
||||
pdf.cell(0, 7, _pdf_safe(f"Pastes ({len(pastes)})"), ln=True)
|
||||
pdf.set_font("Helvetica", "B", 8)
|
||||
pdf.set_fill_color(30, 30, 30); pdf.set_text_color(255, 255, 255)
|
||||
for col_name, col_w in [("Site", 25), ("Paste ID", 80), ("Patterns", 75)]:
|
||||
pdf.cell(col_w, 6, col_name, border=1, fill=True)
|
||||
pdf.ln(); pdf.set_text_color(0, 0, 0)
|
||||
for p in pastes[:100]:
|
||||
pdf.set_fill_color(245, 245, 245); pdf.set_font("Helvetica", "", 7)
|
||||
site = _pdf_safe(p.get("site", ""), 15)
|
||||
pid = _pdf_safe(p.get("id", ""), 55)
|
||||
pats = _pdf_safe(", ".join(f"{k}({len(v)})" for k, v in (p.get("patterns") or {}).items()), 50)
|
||||
for val, w in zip([site, pid, pats], [25, 80, 75]):
|
||||
pdf.cell(w, 5, val, border=1, fill=True)
|
||||
pdf.ln()
|
||||
pdf.ln(3)
|
||||
|
||||
if creds_sc:
|
||||
pdf.set_font("Helvetica", "B", 10)
|
||||
pdf.cell(0, 7, _pdf_safe(f"Extracted Credentials ({len(creds_sc)})"), ln=True)
|
||||
pdf.set_font("Helvetica", "B", 8)
|
||||
pdf.set_fill_color(30, 30, 30); pdf.set_text_color(255, 255, 255)
|
||||
for col_name, col_w in [("Raw Credential", 120), ("Source", 30), ("Paste ID", 30)]:
|
||||
pdf.cell(col_w, 6, col_name, border=1, fill=True)
|
||||
pdf.ln(); pdf.set_text_color(0, 0, 0)
|
||||
for c in creds_sc[:150]:
|
||||
pdf.set_fill_color(255, 240, 240); pdf.set_font("Helvetica", "", 7)
|
||||
raw = _pdf_safe(c.get("raw", ""), 80)
|
||||
src = _pdf_safe(c.get("source", ""), 20)
|
||||
pid = _pdf_safe(c.get("paste_id", ""), 20)
|
||||
for val, w in zip([raw, src, pid], [120, 30, 30]):
|
||||
pdf.cell(w, 5, val, border=1, fill=True)
|
||||
pdf.ln()
|
||||
pdf.ln(3)
|
||||
|
||||
if tg_hits:
|
||||
pdf.set_font("Helvetica", "B", 10)
|
||||
pdf.cell(0, 7, _pdf_safe(f"Telegram CTI ({len(tg_hits)})"), ln=True)
|
||||
pdf.set_font("Helvetica", "B", 8)
|
||||
pdf.set_fill_color(30, 30, 30); pdf.set_text_color(255, 255, 255)
|
||||
for col_name, col_w in [("Channel", 50), ("Message Excerpt", 100), ("Patterns", 30)]:
|
||||
pdf.cell(col_w, 6, col_name, border=1, fill=True)
|
||||
pdf.ln(); pdf.set_text_color(0, 0, 0)
|
||||
for t in tg_hits[:80]:
|
||||
pdf.set_fill_color(245, 245, 255); pdf.set_font("Helvetica", "", 7)
|
||||
link = _pdf_safe(f"t.me/s/{t.get('channel','')}", 35)
|
||||
text = _pdf_safe(t.get("text", ""), 70)
|
||||
pats = _pdf_safe(", ".join(f"{k}({len(v)})" for k, v in (t.get("patterns") or {}).items()), 25)
|
||||
for val, w in zip([link, text, pats], [50, 100, 30]):
|
||||
pdf.cell(w, 5, val, border=1, fill=True)
|
||||
pdf.ln()
|
||||
pdf.ln(3)
|
||||
|
||||
if mc_hits:
|
||||
pdf.set_font("Helvetica", "B", 10)
|
||||
pdf.cell(0, 7, _pdf_safe(f"Misconfigurations ({len(mc_hits)})"), ln=True)
|
||||
pdf.set_font("Helvetica", "B", 8)
|
||||
pdf.set_fill_color(30, 30, 30); pdf.set_text_color(255, 255, 255)
|
||||
for col_name, col_w in [("URL", 90), ("Title", 60), ("Dork", 30)]:
|
||||
pdf.cell(col_w, 6, col_name, border=1, fill=True)
|
||||
pdf.ln(); pdf.set_text_color(0, 0, 0)
|
||||
for m in mc_hits[:80]:
|
||||
pdf.set_fill_color(255, 245, 230); pdf.set_font("Helvetica", "", 7)
|
||||
url_m = _pdf_safe(m.get("url", ""), 60)
|
||||
title_m = _pdf_safe(m.get("title", ""), 40)
|
||||
dork_m = _pdf_safe(m.get("dork", ""), 25)
|
||||
for val, w in zip([url_m, title_m, dork_m], [90, 60, 30]):
|
||||
pdf.cell(w, 5, val, border=1, fill=True)
|
||||
pdf.ln()
|
||||
|
||||
pdf.output(path)
|
||||
print(f"[+] PDF report saved: {path}")
|
||||
@@ -0,0 +1,525 @@
|
||||
"""
|
||||
sources/helpers/scanner.py
|
||||
Recursive Avalanche Engine for NOX autoscan.
|
||||
|
||||
Pipeline per asset (sequential phases):
|
||||
Phase 1 — Breach scan
|
||||
Phase 2 — Hash crack (non-blocking, on breach results)
|
||||
Phase 3 — Dork
|
||||
Phase 4 — Scrape
|
||||
→ Harvest new identifiers from all phases
|
||||
→ Reinject every new unique identifier (not seen before) recursively
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
import re
|
||||
from typing import TYPE_CHECKING, Dict, List, Optional, Set, Tuple
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from nox import Orchestrator
|
||||
|
||||
_syslog = logging.getLogger("nox.system")
|
||||
|
||||
_EMAIL_RE = re.compile(r"[\w.+-]+@[\w-]+\.[\w.]+")
|
||||
_USERNAME_RE = re.compile(r"(?:github\.com|twitter\.com|linkedin\.com/in|reddit\.com/u)/([A-Za-z0-9_.-]{3,39})", re.I)
|
||||
_PHONE_RE = re.compile(r"\+\d[\d\s.\-()]{7,14}\d|\b\d{3}[\s.\-]\d{3}[\s.\-]\d{4}\b")
|
||||
_NAME_RE = re.compile(r"\b([A-Z][a-z]{1,20}(?:\s+[A-Z][a-z]{1,20}){1,3})\b")
|
||||
|
||||
_DORK_LIMIT = 20
|
||||
_PIVOT_TYPES = {"email", "username", "phone", "name", "ip", "domain"}
|
||||
|
||||
|
||||
def _cfg_depth(orc=None) -> int:
|
||||
# A7/A10: read from orchestrator config if available
|
||||
if orc is not None:
|
||||
cfg = getattr(orc, "config", None)
|
||||
if cfg is not None:
|
||||
v = getattr(cfg, "pivot_depth", None)
|
||||
if v is not None:
|
||||
return int(v)
|
||||
try:
|
||||
from nox import Cfg # type: ignore
|
||||
return Cfg.PIVOT_DEPTH
|
||||
except ImportError:
|
||||
return 2
|
||||
|
||||
|
||||
def _cfg_concurrency(orc=None) -> int:
|
||||
# A7: read from orchestrator config if available
|
||||
if orc is not None:
|
||||
cfg = getattr(orc, "config", None)
|
||||
if cfg is not None:
|
||||
v = getattr(cfg, "concurrency", None)
|
||||
if v is not None:
|
||||
return int(v)
|
||||
try:
|
||||
from nox import Cfg # type: ignore
|
||||
return Cfg.CONCURRENCY
|
||||
except ImportError:
|
||||
return 15
|
||||
|
||||
|
||||
def _out(level: str, msg: str) -> None:
|
||||
try:
|
||||
from nox import out as _nox_out # type: ignore
|
||||
_nox_out(level, msg)
|
||||
except Exception:
|
||||
import sys
|
||||
print(f"[{level}] {msg}", file=sys.stderr)
|
||||
|
||||
|
||||
def _extract_ids_from_text(text: str, exclude: str = "") -> List[Tuple[str, str]]:
|
||||
"""Extract pivotable identifiers from free text, excluding the current asset."""
|
||||
found: List[Tuple[str, str]] = []
|
||||
excl = exclude.lower()
|
||||
for m in _EMAIL_RE.findall(text):
|
||||
v = m.lower()
|
||||
if v != excl:
|
||||
found.append((v, "email"))
|
||||
for m in _USERNAME_RE.findall(text):
|
||||
v = m.lower()
|
||||
if v != excl:
|
||||
found.append((v, "username"))
|
||||
for m in _PHONE_RE.findall(text):
|
||||
clean = re.sub(r"[\s.\-()]", "", m)
|
||||
if 8 <= len(clean) <= 15 and clean != excl:
|
||||
found.append((clean, "phone"))
|
||||
for m in _NAME_RE.findall(text):
|
||||
if len(m.split()) >= 2 and m.lower() != excl:
|
||||
found.append((m, "name"))
|
||||
return found
|
||||
|
||||
|
||||
def _ids_from_records(records: list, exclude: str = "") -> List[Tuple[str, str, str]]:
|
||||
"""
|
||||
Extract pivotable identifiers from breach records.
|
||||
Returns (value, qtype, ref) where ref is the source/breach name for logging.
|
||||
"""
|
||||
found: List[Tuple[str, str, str]] = []
|
||||
excl = exclude.lower()
|
||||
for r in records:
|
||||
src = getattr(r, "source", "") or ""
|
||||
breach = getattr(r, "breach_name", "") or src
|
||||
for val, qtype in [
|
||||
(getattr(r, "email", ""), "email"),
|
||||
(getattr(r, "username", ""), "username"),
|
||||
(getattr(r, "phone", ""), "phone"),
|
||||
(getattr(r, "full_name", ""), "name"),
|
||||
(getattr(r, "ip_address", ""), "ip"),
|
||||
(getattr(r, "domain", ""), "domain"),
|
||||
]:
|
||||
if val and len(val) > 2 and val.lower() != excl:
|
||||
found.append((val.strip(), qtype, breach))
|
||||
meta = getattr(r, "metadata", {}) or {}
|
||||
for em in meta.get("emails", []):
|
||||
if em and em.lower() != excl:
|
||||
found.append((em.lower(), "email", breach))
|
||||
return found
|
||||
|
||||
|
||||
# ── Pivot log entry schema ─────────────────────────────────────────────────
|
||||
# {
|
||||
# "asset": str, # identifier scanned
|
||||
# "qtype": str, # email/username/phone/name/domain/ip
|
||||
# "depth": int, # 0=seed, 1=first pivot, …
|
||||
# "parent": str|None, # asset that discovered this one
|
||||
# "found_in": str, # phase that found this asset: seed/breach/dork/scrape/hash_crack
|
||||
# "records": int, # breach records found for this asset
|
||||
# "dorks": int, # dork hits found for this asset
|
||||
# "scrape": int, # scrape items found for this asset
|
||||
# "children": List[dict], # [{asset, qtype, found_in, ref}] — new assets discovered
|
||||
# "cracked": List[str], # plaintexts cracked from hashes in breach results
|
||||
# }
|
||||
|
||||
|
||||
class AvalancheScanner:
|
||||
def __init__(self, orchestrator: "Orchestrator") -> None:
|
||||
self._orc = orchestrator
|
||||
self.seen_assets: Set[str] = set()
|
||||
# A2: single semaphore for the entire run, created lazily inside the event loop
|
||||
self._sem: Optional[asyncio.Semaphore] = None
|
||||
self._all_records: List = []
|
||||
self._dork_hits: List[dict] = []
|
||||
self._seen_dork_urls: Set[str] = set()
|
||||
# A6: scrape_hits merged atomically per _do_process call
|
||||
self._scrape_hits: Dict = {"pastes": [], "credentials": [], "hashes": [],
|
||||
"telegram": [], "dork_misconfigs": []}
|
||||
self._max_depth: int = 0
|
||||
self._in_flight: Dict[str, asyncio.Future] = {}
|
||||
self.pivot_log: List[dict] = []
|
||||
# A8: global set to prevent duplicate entries in discovered_assets
|
||||
self._seen_discovered: Set[str] = set()
|
||||
self.discovered_assets: List[dict] = []
|
||||
|
||||
def _get_sem(self) -> asyncio.Semaphore:
|
||||
# A2: semaphore created once per run, shared across all coroutines
|
||||
if self._sem is None:
|
||||
self._sem = asyncio.Semaphore(_cfg_concurrency(self._orc))
|
||||
return self._sem
|
||||
|
||||
async def run(self, target: str) -> tuple:
|
||||
# A9: respect no_pivot flag from config
|
||||
cfg = getattr(self._orc, "config", None)
|
||||
no_pivot = getattr(cfg, "no_pivot", False) if cfg else False
|
||||
if no_pivot:
|
||||
try:
|
||||
from nox import Detect # type: ignore
|
||||
qtype = Detect.qtype(target)
|
||||
except ImportError:
|
||||
qtype = "email"
|
||||
async with self._get_sem():
|
||||
try:
|
||||
records = await self._orc._full_async_scan(target, qtype)
|
||||
except Exception:
|
||||
records = []
|
||||
self._all_records.extend(records)
|
||||
self.seen_assets.add(target.lower().strip())
|
||||
self.pivot_log.append({
|
||||
"asset": target, "qtype": qtype, "depth": 0, "parent": None,
|
||||
"found_in": "seed", "records": len(records), "dorks": 0,
|
||||
"scrape": 0, "children": [], "cracked": [],
|
||||
})
|
||||
return self._all_records, self._dork_hits, self._scrape_hits
|
||||
await self._process(target, depth=0, parent=None, found_in="seed")
|
||||
return self._all_records, self._dork_hits, self._scrape_hits
|
||||
|
||||
def get_discovered_assets(self) -> List[dict]:
|
||||
"""Return flat list of all discovered assets with full provenance."""
|
||||
return self.discovered_assets
|
||||
|
||||
def get_max_depth(self) -> int:
|
||||
return self._max_depth
|
||||
|
||||
# ── Dedup gate ────────────────────────────────────────────────────
|
||||
|
||||
async def _process(self, asset: str, depth: int,
|
||||
parent: Optional[str], found_in: str) -> None:
|
||||
"""Dedup gate: ensures each asset is processed exactly once."""
|
||||
# A10: use per-run depth from orchestrator config
|
||||
if depth > _cfg_depth(self._orc):
|
||||
_syslog.debug("avalanche depth cap reached for %s", asset)
|
||||
return
|
||||
|
||||
key = asset.lower().strip()
|
||||
if not key:
|
||||
return
|
||||
|
||||
# A1: add to seen_assets FIRST (atomic gate) before any other check.
|
||||
# If already present, wait on the in-flight future if one exists, then return.
|
||||
if key in self.seen_assets:
|
||||
if key in self._in_flight:
|
||||
try:
|
||||
await self._in_flight[key]
|
||||
except Exception:
|
||||
pass
|
||||
return
|
||||
|
||||
self.seen_assets.add(key)
|
||||
|
||||
# If already in-flight (shouldn't happen after the seen_assets check above,
|
||||
# but guard defensively), wait and return.
|
||||
if key in self._in_flight:
|
||||
try:
|
||||
await self._in_flight[key]
|
||||
except Exception:
|
||||
pass
|
||||
return
|
||||
|
||||
loop = asyncio.get_running_loop()
|
||||
fut: asyncio.Future = loop.create_future()
|
||||
self._in_flight[key] = fut
|
||||
|
||||
try:
|
||||
await self._do_process(asset, depth, parent, found_in)
|
||||
finally:
|
||||
if not fut.done():
|
||||
fut.set_result(None)
|
||||
|
||||
# ── Core pipeline ─────────────────────────────────────────────────
|
||||
|
||||
async def _do_process(self, asset: str, depth: int,
|
||||
parent: Optional[str], found_in: str) -> None:
|
||||
"""
|
||||
Sequential pipeline:
|
||||
Phase 1 — Breach scan
|
||||
Phase 2 — Hash crack (concurrent, non-blocking)
|
||||
Phase 3 — Dork
|
||||
Phase 4 — Scrape
|
||||
→ Harvest all new identifiers with phase+ref annotation
|
||||
→ Reinject every unseen identifier
|
||||
"""
|
||||
if depth > self._max_depth:
|
||||
self._max_depth = depth
|
||||
|
||||
try:
|
||||
from nox import Detect # type: ignore
|
||||
qtype = Detect.qtype(asset)
|
||||
except ImportError:
|
||||
qtype = "email"
|
||||
|
||||
indent = " " * depth
|
||||
_out("pivot" if depth > 0 else "info",
|
||||
f"{indent}[depth={depth}] {'↳' if depth > 0 else '◉'} {asset} ({qtype})"
|
||||
+ (f" ← {found_in} via {parent}" if parent else " [SEED]"))
|
||||
_syslog.info("AVALANCHE asset=%s depth=%d parent=%s found_in=%s",
|
||||
asset, depth, parent or "—", found_in)
|
||||
|
||||
# ── Phase 1: Breach scan ──────────────────────────────────────
|
||||
async with self._get_sem():
|
||||
try:
|
||||
records: List = await self._orc._full_async_scan(asset, qtype)
|
||||
except Exception as exc:
|
||||
_syslog.warning("BREACH_FAIL asset=%s err=%s", asset, exc)
|
||||
records = []
|
||||
|
||||
_out("ok" if records else "dim",
|
||||
f"{indent} [breach] {len(records)} records")
|
||||
_syslog.info("BREACH_DONE asset=%s records=%d", asset, len(records))
|
||||
self._all_records.extend(records)
|
||||
|
||||
# ── Phase 2: Hash crack (non-blocking) ────────────────────────
|
||||
cracked_plaintexts: List[str] = []
|
||||
try:
|
||||
from sources.helpers.cracker import detect_hash # type: ignore
|
||||
import aiohttp as _aio # type: ignore
|
||||
async with _aio.ClientSession(connector=_aio.TCPConnector(limit=5)) as _cs:
|
||||
crack_tasks = [
|
||||
_crack_and_inject(_cs, getattr(r, "password_hash", ""), r,
|
||||
self.seen_assets, self._all_records,
|
||||
self, depth, asset, cracked_plaintexts)
|
||||
for r in records
|
||||
if getattr(r, "password_hash", "") and not getattr(r, "password", "")
|
||||
and detect_hash(getattr(r, "password_hash", ""))
|
||||
]
|
||||
if crack_tasks:
|
||||
await asyncio.gather(*crack_tasks, return_exceptions=True)
|
||||
except ImportError:
|
||||
pass
|
||||
|
||||
# ── Phase 3: Dork ─────────────────────────────────────────────
|
||||
_out("info", f"{indent} [dork] querying for {asset}…")
|
||||
try:
|
||||
dork_res = await self._async_dork(asset, qtype)
|
||||
except Exception as exc:
|
||||
_syslog.warning("DORK_FAIL asset=%s err=%s", asset, exc)
|
||||
dork_res = []
|
||||
|
||||
dork_count = 0
|
||||
for hit in (dork_res or [])[:_DORK_LIMIT]:
|
||||
url = hit.get("url", "") or hit.get("title", "")
|
||||
if url and url not in self._seen_dork_urls:
|
||||
self._seen_dork_urls.add(url)
|
||||
hit["pivot_asset"] = asset
|
||||
hit["pivot_depth"] = depth
|
||||
self._dork_hits.append(hit)
|
||||
dork_count += 1
|
||||
_out("ok" if dork_count else "dim",
|
||||
f"{indent} [dork] {dork_count} hits")
|
||||
_syslog.info("DORK_DONE asset=%s hits=%d", asset, dork_count)
|
||||
|
||||
# ── Phase 4: Scrape ───────────────────────────────────────────
|
||||
_out("info", f"{indent} [scrape] querying for {asset}…")
|
||||
try:
|
||||
scrape_res = await self._async_scrape(asset)
|
||||
except Exception as exc:
|
||||
_syslog.warning("SCRAPE_FAIL asset=%s err=%s", asset, exc)
|
||||
scrape_res = {}
|
||||
|
||||
# A6: collect scrape results locally, then merge atomically
|
||||
scrape_count = 0
|
||||
local_scrape: Dict = {k: [] for k in self._scrape_hits}
|
||||
for k in self._scrape_hits:
|
||||
for item in (scrape_res or {}).get(k, []):
|
||||
if isinstance(item, dict):
|
||||
item["pivot_asset"] = asset
|
||||
item["pivot_depth"] = depth
|
||||
local_scrape[k].append(item)
|
||||
scrape_count += 1
|
||||
# Atomic merge into shared dict (single-threaded event loop — safe)
|
||||
for k, items in local_scrape.items():
|
||||
self._scrape_hits[k].extend(items)
|
||||
_out("ok" if scrape_count else "dim",
|
||||
f"{indent} [scrape] {scrape_count} items")
|
||||
_syslog.info("SCRAPE_DONE asset=%s items=%d", asset, scrape_count)
|
||||
|
||||
# ── Harvest new identifiers with phase+ref annotation ─────────
|
||||
# Each entry: (value, qtype, found_in_phase, ref)
|
||||
new_ids: List[Tuple[str, str, str, str]] = []
|
||||
|
||||
# From breach records
|
||||
for val, vqtype, ref in _ids_from_records(records, exclude=asset):
|
||||
if vqtype in _PIVOT_TYPES:
|
||||
new_ids.append((val, vqtype, "breach", ref))
|
||||
|
||||
# From dork hits
|
||||
for hit in (dork_res or [])[:_DORK_LIMIT]:
|
||||
url = hit.get("url", "")
|
||||
dork = hit.get("dork", "")
|
||||
ref = url or dork
|
||||
text = f"{hit.get('title','')} {hit.get('snippet','')} {url} {dork}"
|
||||
for val, vqtype in _extract_ids_from_text(text, exclude=asset):
|
||||
if vqtype in _PIVOT_TYPES:
|
||||
new_ids.append((val, vqtype, "dork", ref[:120]))
|
||||
|
||||
# From scrape results
|
||||
for cred in (scrape_res or {}).get("credentials", []):
|
||||
raw = cred.get("raw", "")
|
||||
ref = f"paste:{cred.get('paste_id','')}" or cred.get("source", "scrape")
|
||||
for val, vqtype in _extract_ids_from_text(raw, exclude=asset):
|
||||
if vqtype in _PIVOT_TYPES:
|
||||
new_ids.append((val, vqtype, "scrape", ref))
|
||||
for tg in (scrape_res or {}).get("telegram", []):
|
||||
ref = f"t.me/{tg.get('channel','')}"
|
||||
for val, vqtype in _extract_ids_from_text(tg.get("text", ""), exclude=asset):
|
||||
if vqtype in _PIVOT_TYPES:
|
||||
new_ids.append((val, vqtype, "scrape", ref))
|
||||
for mc in (scrape_res or {}).get("dork_misconfigs", []):
|
||||
ref = mc.get("url", mc.get("title", "misconfig"))
|
||||
for val, vqtype in _extract_ids_from_text(
|
||||
f"{mc.get('title','')} {mc.get('snippet','')}", exclude=asset):
|
||||
if vqtype in _PIVOT_TYPES:
|
||||
new_ids.append((val, vqtype, "scrape", ref[:120]))
|
||||
|
||||
# ── Deduplicate and queue children ────────────────────────────
|
||||
children: List[dict] = []
|
||||
child_tasks = []
|
||||
queued: Set[str] = set()
|
||||
|
||||
for val, vqtype, phase, ref in new_ids:
|
||||
child_key = val.lower().strip()
|
||||
if not child_key or child_key in self.seen_assets or child_key in queued:
|
||||
continue
|
||||
queued.add(child_key)
|
||||
child_entry = {"asset": val, "qtype": vqtype, "found_in": phase, "ref": ref}
|
||||
children.append(child_entry)
|
||||
# A8: prevent duplicate entries in discovered_assets across parallel parents
|
||||
if child_key not in self._seen_discovered:
|
||||
self._seen_discovered.add(child_key)
|
||||
self.discovered_assets.append({
|
||||
"asset": val,
|
||||
"qtype": vqtype,
|
||||
"phase": phase,
|
||||
"ref": ref,
|
||||
"parent": asset,
|
||||
"depth": depth + 1,
|
||||
})
|
||||
_out("pivot",
|
||||
f"{indent} ↳ new asset [{phase}]: {val} ({vqtype}) ref: {ref[:60]}")
|
||||
_syslog.info("PIVOT_QUEUE asset=%s qtype=%s phase=%s ref=%s parent=%s depth=%d",
|
||||
val, vqtype, phase, ref[:80], asset, depth + 1)
|
||||
child_tasks.append(
|
||||
self._process(val, depth + 1, parent=asset, found_in=phase)
|
||||
)
|
||||
|
||||
# A5: run child tasks FIRST, then append pivot_log so the log reflects actual outcomes
|
||||
if child_tasks:
|
||||
_out("info", f"{indent} → reinjecting {len(child_tasks)} new asset(s)…")
|
||||
await asyncio.gather(*child_tasks, return_exceptions=True)
|
||||
|
||||
# ── Log this node (after children complete — A5) ──────────────
|
||||
self.pivot_log.append({
|
||||
"asset": asset,
|
||||
"qtype": qtype,
|
||||
"depth": depth,
|
||||
"parent": parent,
|
||||
"found_in": found_in,
|
||||
"records": len(records),
|
||||
"dorks": dork_count,
|
||||
"scrape": scrape_count,
|
||||
"children": children,
|
||||
"cracked": cracked_plaintexts or [],
|
||||
})
|
||||
|
||||
# ── Dork dispatcher ───────────────────────────────────────────────
|
||||
|
||||
async def _async_dork(self, asset: str, qtype: str = "email") -> list:
|
||||
try:
|
||||
import aiohttp as _aio # type: ignore
|
||||
import ssl as _ssl
|
||||
connector = _aio.TCPConnector(limit=10, ssl=_ssl.create_default_context(), family=0)
|
||||
async with _aio.ClientSession(connector=connector) as session:
|
||||
recs = await self._orc.dorking_engine.async_search(session, asset, qtype)
|
||||
return [
|
||||
{
|
||||
"url": r.raw_data.get("url", "") if hasattr(r, "raw_data") else "",
|
||||
"title": r.raw_data.get("url", r.raw_data.get("dork", "")) if hasattr(r, "raw_data") else "",
|
||||
"snippet": "",
|
||||
"dork": r.raw_data.get("dork", "") if hasattr(r, "raw_data") else "",
|
||||
"engine": "DDG",
|
||||
}
|
||||
for r in recs
|
||||
]
|
||||
except ImportError:
|
||||
loop = asyncio.get_running_loop()
|
||||
result = await loop.run_in_executor(None, self._orc.dork, asset)
|
||||
return result if isinstance(result, list) else []
|
||||
except Exception as exc:
|
||||
_syslog.debug("DORK_ERR asset=%s err=%s", asset, exc)
|
||||
return []
|
||||
|
||||
# ── Scrape dispatcher ─────────────────────────────────────────────
|
||||
|
||||
async def _async_scrape(self, asset: str) -> dict:
|
||||
# A3: instantiate a fresh Session + ScrapeEngine per call to avoid sharing
|
||||
# a non-thread-safe requests.Session / cloudscraper across concurrent coroutines.
|
||||
_empty: dict = {"pastes": [], "credentials": [], "hashes": [],
|
||||
"telegram": [], "dork_misconfigs": []}
|
||||
try:
|
||||
loop = asyncio.get_running_loop()
|
||||
try:
|
||||
from nox import Session, NoxConfig, ScrapeEngine # type: ignore
|
||||
_cfg = getattr(self._orc, "config", None) or NoxConfig()
|
||||
_session = Session(_cfg)
|
||||
_engine = ScrapeEngine(_session, self._orc.db)
|
||||
qtype = "email"
|
||||
try:
|
||||
from nox import Detect # type: ignore
|
||||
qtype = Detect.qtype(asset)
|
||||
except Exception:
|
||||
pass
|
||||
result = await loop.run_in_executor(None, _engine.run, asset, qtype)
|
||||
except Exception:
|
||||
result = await loop.run_in_executor(None, self._orc.scrape, asset)
|
||||
return result if isinstance(result, dict) else _empty
|
||||
except Exception as exc:
|
||||
_syslog.debug("SCRAPE_ERR asset=%s err=%s", asset, exc)
|
||||
return _empty
|
||||
|
||||
|
||||
# ── Hash crack helper ──────────────────────────────────────────────────────
|
||||
|
||||
async def _crack_and_inject(session, hash_value: str, record_ref,
|
||||
seen_assets: Set[str], all_records: list,
|
||||
scanner: "AvalancheScanner",
|
||||
depth: int, parent_asset: str,
|
||||
cracked_out: List[str]) -> None:
|
||||
from sources.helpers.cracker import detect_hash, async_crack, CRACK_TIMEOUT # type: ignore
|
||||
hash_type = detect_hash(hash_value)
|
||||
if not hash_type:
|
||||
return
|
||||
try:
|
||||
plaintext = await asyncio.wait_for(
|
||||
async_crack(session, hash_value, hash_type), timeout=CRACK_TIMEOUT)
|
||||
except (asyncio.TimeoutError, Exception) as exc:
|
||||
_syslog.debug("CRACK_FAIL hash=%s reason=%s", hash_value[:16], exc)
|
||||
return
|
||||
|
||||
if not plaintext:
|
||||
_syslog.debug("CRACK_FAIL hash=%s reason=no_result", hash_value[:16])
|
||||
return
|
||||
|
||||
record_ref.password = plaintext
|
||||
record_ref.hash_type = hash_type
|
||||
if "Cracked" not in (record_ref.data_types or []):
|
||||
record_ref.data_types = list(record_ref.data_types) + ["Cracked"]
|
||||
_syslog.info("CRACK_OK hash=%s plain=%s parent=%s", hash_value[:16], plaintext, parent_asset)
|
||||
_out("ok", f" [crack] {hash_value[:16]}… → {plaintext} (from {parent_asset})")
|
||||
cracked_out.append(plaintext)
|
||||
|
||||
# A4: inject cracked plaintext as qtype="password" — NOT as username.
|
||||
# Only pivot on it if sources support password-recycling queries.
|
||||
key = plaintext.lower()
|
||||
if key not in seen_assets and depth + 1 <= _cfg_depth(scanner._orc):
|
||||
await scanner._process(plaintext, depth + 1,
|
||||
parent=parent_asset, found_in="hash_crack")
|
||||
@@ -0,0 +1,35 @@
|
||||
{
|
||||
"name": "hibp_breached",
|
||||
"category": "breaches",
|
||||
"endpoint": "https://haveibeenpwned.com/api/v3/breachedaccount/{target}",
|
||||
"method": "GET",
|
||||
"requires_auth": true,
|
||||
"selectors": {
|
||||
"breaches": "$.*.Name"
|
||||
},
|
||||
"rate_limit": 1.5,
|
||||
"headers": {
|
||||
"hibp-api-key": "{HIBP_API_KEY}",
|
||||
"User-Agent": "NOX-Framework"
|
||||
},
|
||||
"api_key_slots": [
|
||||
"{HIBP_API_KEY}"
|
||||
],
|
||||
"input_type": "email",
|
||||
"output_type": [
|
||||
"email",
|
||||
"domain"
|
||||
],
|
||||
"normalization_map": {
|
||||
"Name": "breach_name"
|
||||
},
|
||||
"tags": [
|
||||
"passive",
|
||||
"stealth"
|
||||
],
|
||||
"health_check_url": "https://haveibeenpwned.com",
|
||||
"expected_status": 200,
|
||||
"reliability_score": 5,
|
||||
"backup_endpoints": [],
|
||||
"confidence": 1.0
|
||||
}
|
||||
@@ -0,0 +1,30 @@
|
||||
{
|
||||
"name": "hudsonrock_osint",
|
||||
"category": "breach_data",
|
||||
"endpoint": "https://cavalier.hudsonrock.com/api/json/v2/osint-tools/search-by-login?username={target}",
|
||||
"method": "GET",
|
||||
"requires_auth": false,
|
||||
"selectors": {
|
||||
"stealers": "$.stealers"
|
||||
},
|
||||
"rate_limit": 1.0,
|
||||
"headers": {},
|
||||
"api_key_slots": [],
|
||||
"input_type": "username",
|
||||
"output_type": [
|
||||
"email",
|
||||
"domain"
|
||||
],
|
||||
"normalization_map": {
|
||||
"stealers": "breach_record"
|
||||
},
|
||||
"tags": [
|
||||
"passive",
|
||||
"stealth"
|
||||
],
|
||||
"health_check_url": "https://cavalier.hudsonrock.com",
|
||||
"expected_status": 200,
|
||||
"reliability_score": 4,
|
||||
"backup_endpoints": [],
|
||||
"confidence": 0.85
|
||||
}
|
||||
@@ -0,0 +1,30 @@
|
||||
{
|
||||
"name": "hunter_io",
|
||||
"category": "discovery",
|
||||
"endpoint": "https://api.hunter.io/v2/domain-search?domain={target}&api_key={HUNTER_API_KEY}",
|
||||
"method": "GET",
|
||||
"requires_auth": true,
|
||||
"selectors": {
|
||||
"emails": "$.data.emails[*].value"
|
||||
},
|
||||
"rate_limit": 1.0,
|
||||
"headers": {},
|
||||
"api_key_slots": [
|
||||
"{HUNTER_API_KEY}"
|
||||
],
|
||||
"input_type": "domain",
|
||||
"output_type": [
|
||||
"email"
|
||||
],
|
||||
"normalization_map": {
|
||||
"value": "email_address"
|
||||
},
|
||||
"tags": [
|
||||
"passive"
|
||||
],
|
||||
"health_check_url": "https://api.hunter.io",
|
||||
"expected_status": 200,
|
||||
"reliability_score": 5,
|
||||
"backup_endpoints": [],
|
||||
"confidence": 1.0
|
||||
}
|
||||
@@ -0,0 +1,29 @@
|
||||
{
|
||||
"name": "hunter_verify",
|
||||
"category": "email_rep",
|
||||
"endpoint": "https://api.hunter.io/v2/email-verifier?email={target}&api_key={HUNTER_API_KEY}",
|
||||
"method": "GET",
|
||||
"requires_auth": true,
|
||||
"selectors": {
|
||||
"result": "$.data.result"
|
||||
},
|
||||
"rate_limit": 1.0,
|
||||
"headers": {},
|
||||
"api_key_slots": [
|
||||
"{HUNTER_API_KEY}"
|
||||
],
|
||||
"input_type": "email",
|
||||
"output_type": [
|
||||
"email"
|
||||
],
|
||||
"normalization_map": {},
|
||||
"tags": [
|
||||
"passive",
|
||||
"fast"
|
||||
],
|
||||
"health_check_url": "https://api.hunter.io",
|
||||
"expected_status": 200,
|
||||
"reliability_score": 4,
|
||||
"backup_endpoints": [],
|
||||
"confidence": 0.85
|
||||
}
|
||||
@@ -0,0 +1,37 @@
|
||||
{
|
||||
"name": "hybrid_analysis",
|
||||
"category": "threat_intel",
|
||||
"endpoint": "https://www.hybrid-analysis.com/api/v2/search/hash",
|
||||
"method": "POST",
|
||||
"requires_auth": true,
|
||||
"selectors": {
|
||||
"verdict": "$.verdict"
|
||||
},
|
||||
"rate_limit": 1.0,
|
||||
"headers": {
|
||||
"api-key": "{HYBRID_API_KEY}"
|
||||
},
|
||||
"payload_template": {
|
||||
"hash": "{target}"
|
||||
},
|
||||
"api_key_slots": [
|
||||
"{HYBRID_API_KEY}"
|
||||
],
|
||||
"input_type": "hash",
|
||||
"output_type": [
|
||||
"hash"
|
||||
],
|
||||
"normalization_map": {
|
||||
"verdict": "malware_verdict"
|
||||
},
|
||||
"tags": [
|
||||
"passive",
|
||||
"threat",
|
||||
"heavy"
|
||||
],
|
||||
"health_check_url": "https://www.hybrid-analysis.com",
|
||||
"expected_status": 200,
|
||||
"reliability_score": 4,
|
||||
"backup_endpoints": [],
|
||||
"confidence": 0.85
|
||||
}
|
||||
@@ -0,0 +1,30 @@
|
||||
{
|
||||
"name": "intelx_phone",
|
||||
"category": "breaches",
|
||||
"endpoint": "https://2.intelx.io/phone/search?phone={target}",
|
||||
"method": "GET",
|
||||
"requires_auth": true,
|
||||
"selectors": {
|
||||
"results": "$.results"
|
||||
},
|
||||
"rate_limit": 1.0,
|
||||
"headers": {
|
||||
"x-key": "{INTELX_API_KEY}"
|
||||
},
|
||||
"api_key_slots": [
|
||||
"{INTELX_API_KEY}"
|
||||
],
|
||||
"input_type": "phone",
|
||||
"output_type": [
|
||||
"phone"
|
||||
],
|
||||
"normalization_map": {},
|
||||
"tags": [
|
||||
"passive"
|
||||
],
|
||||
"health_check_url": "https://2.intelx.io",
|
||||
"expected_status": 200,
|
||||
"reliability_score": 5,
|
||||
"backup_endpoints": [],
|
||||
"confidence": 1.0
|
||||
}
|
||||
@@ -0,0 +1,44 @@
|
||||
{
|
||||
"name": "intelx_search",
|
||||
"category": "breaches",
|
||||
"endpoint": "https://2.intelx.io/intelligent/search",
|
||||
"method": "POST",
|
||||
"requires_auth": true,
|
||||
"selectors": {
|
||||
"id": "$.id"
|
||||
},
|
||||
"rate_limit": 1.0,
|
||||
"headers": {
|
||||
"x-key": "{INTELX_API_KEY}"
|
||||
},
|
||||
"payload_template": {
|
||||
"term": "{target}",
|
||||
"buckets": [],
|
||||
"lookuplevel": 0,
|
||||
"maxresults": 100,
|
||||
"timeout": 0,
|
||||
"datefrom": "",
|
||||
"dateto": "",
|
||||
"sort": 4,
|
||||
"media": 0,
|
||||
"terminate": []
|
||||
},
|
||||
"api_key_slots": [
|
||||
"{INTELX_API_KEY}"
|
||||
],
|
||||
"input_type": "email",
|
||||
"output_type": [
|
||||
"email",
|
||||
"domain"
|
||||
],
|
||||
"normalization_map": {},
|
||||
"tags": [
|
||||
"passive",
|
||||
"stealth"
|
||||
],
|
||||
"health_check_url": "https://2.intelx.io",
|
||||
"expected_status": 200,
|
||||
"reliability_score": 5,
|
||||
"backup_endpoints": [],
|
||||
"confidence": 1.0
|
||||
}
|
||||
@@ -0,0 +1,31 @@
|
||||
{
|
||||
"name": "intezer",
|
||||
"category": "threat_intel",
|
||||
"endpoint": "https://analyze.intezer.com/api/v2-0/get-analysis-by-hash/{target}",
|
||||
"method": "GET",
|
||||
"requires_auth": true,
|
||||
"selectors": {
|
||||
"result": "$.result"
|
||||
},
|
||||
"rate_limit": 1.0,
|
||||
"headers": {
|
||||
"Authorization": "Bearer {INTEZER_API_KEY}"
|
||||
},
|
||||
"api_key_slots": [
|
||||
"{INTEZER_API_KEY}"
|
||||
],
|
||||
"input_type": "hash",
|
||||
"output_type": [
|
||||
"hash"
|
||||
],
|
||||
"normalization_map": {},
|
||||
"tags": [
|
||||
"passive",
|
||||
"threat"
|
||||
],
|
||||
"health_check_url": "https://analyze.intezer.com",
|
||||
"expected_status": 200,
|
||||
"reliability_score": 4,
|
||||
"backup_endpoints": [],
|
||||
"confidence": 0.85
|
||||
}
|
||||
@@ -0,0 +1,33 @@
|
||||
{
|
||||
"name": "ipapi_co",
|
||||
"category": "geolocation",
|
||||
"endpoint": "https://ipapi.co/{target}/json/",
|
||||
"method": "GET",
|
||||
"requires_auth": false,
|
||||
"selectors": {
|
||||
"asn": "$.asn",
|
||||
"org": "$.org"
|
||||
},
|
||||
"rate_limit": 1.0,
|
||||
"headers": {
|
||||
"User-Agent": "Mozilla/5.0"
|
||||
},
|
||||
"api_key_slots": [],
|
||||
"input_type": "ip",
|
||||
"output_type": [
|
||||
"domain"
|
||||
],
|
||||
"normalization_map": {
|
||||
"asn": "asn_number",
|
||||
"org": "asn_org"
|
||||
},
|
||||
"tags": [
|
||||
"passive",
|
||||
"fast"
|
||||
],
|
||||
"health_check_url": "https://ipapi.co",
|
||||
"expected_status": 200,
|
||||
"reliability_score": 4,
|
||||
"backup_endpoints": [],
|
||||
"confidence": 0.85
|
||||
}
|
||||
@@ -0,0 +1,31 @@
|
||||
{
|
||||
"name": "ipdata_co",
|
||||
"category": "geolocation",
|
||||
"endpoint": "https://api.ipdata.co/{target}?api-key={IPDATA_API_KEY}",
|
||||
"method": "GET",
|
||||
"requires_auth": true,
|
||||
"selectors": {
|
||||
"threat": "$.threat"
|
||||
},
|
||||
"rate_limit": 1.0,
|
||||
"headers": {},
|
||||
"api_key_slots": [
|
||||
"{IPDATA_API_KEY}"
|
||||
],
|
||||
"input_type": "ip",
|
||||
"output_type": [
|
||||
"ip"
|
||||
],
|
||||
"normalization_map": {
|
||||
"threat": "threat_info"
|
||||
},
|
||||
"tags": [
|
||||
"passive",
|
||||
"fast"
|
||||
],
|
||||
"health_check_url": "https://api.ipdata.co",
|
||||
"expected_status": 200,
|
||||
"reliability_score": 4,
|
||||
"backup_endpoints": [],
|
||||
"confidence": 0.85
|
||||
}
|
||||
@@ -0,0 +1,31 @@
|
||||
{
|
||||
"name": "ipgeolocation_io",
|
||||
"category": "geolocation",
|
||||
"endpoint": "https://api.ipgeolocation.io/ipgeo?apiKey={IPGEO_API_KEY}&ip={target}",
|
||||
"method": "GET",
|
||||
"requires_auth": true,
|
||||
"selectors": {
|
||||
"isp": "$.isp"
|
||||
},
|
||||
"rate_limit": 1.0,
|
||||
"headers": {},
|
||||
"api_key_slots": [
|
||||
"{IPGEO_API_KEY}"
|
||||
],
|
||||
"input_type": "ip",
|
||||
"output_type": [
|
||||
"ip"
|
||||
],
|
||||
"normalization_map": {
|
||||
"isp": "asn_org"
|
||||
},
|
||||
"tags": [
|
||||
"passive",
|
||||
"fast"
|
||||
],
|
||||
"health_check_url": "https://api.ipgeolocation.io",
|
||||
"expected_status": 200,
|
||||
"reliability_score": 4,
|
||||
"backup_endpoints": [],
|
||||
"confidence": 0.85
|
||||
}
|
||||
@@ -0,0 +1,31 @@
|
||||
{
|
||||
"name": "ipinfo_io",
|
||||
"category": "geolocation",
|
||||
"endpoint": "https://ipinfo.io/{target}/json",
|
||||
"method": "GET",
|
||||
"requires_auth": false,
|
||||
"selectors": {
|
||||
"org": "$.org",
|
||||
"city": "$.city"
|
||||
},
|
||||
"rate_limit": 1.0,
|
||||
"headers": {},
|
||||
"api_key_slots": [],
|
||||
"input_type": "ip",
|
||||
"output_type": [
|
||||
"domain"
|
||||
],
|
||||
"normalization_map": {
|
||||
"org": "asn_org",
|
||||
"city": "geo_city"
|
||||
},
|
||||
"tags": [
|
||||
"passive",
|
||||
"fast"
|
||||
],
|
||||
"health_check_url": "https://ipinfo.io",
|
||||
"expected_status": 200,
|
||||
"reliability_score": 5,
|
||||
"backup_endpoints": [],
|
||||
"confidence": 1.0
|
||||
}
|
||||
@@ -0,0 +1,30 @@
|
||||
{
|
||||
"name": "ipinfodb",
|
||||
"category": "geolocation",
|
||||
"endpoint": "http://api.ipinfodb.com/v3/ip-city/?key={IPINFODB_API_KEY}&ip={target}&format=json",
|
||||
"method": "GET",
|
||||
"requires_auth": true,
|
||||
"selectors": {
|
||||
"city": "$.cityName"
|
||||
},
|
||||
"rate_limit": 1.0,
|
||||
"headers": {},
|
||||
"api_key_slots": [
|
||||
"{IPINFODB_API_KEY}"
|
||||
],
|
||||
"input_type": "ip",
|
||||
"output_type": [
|
||||
"ip"
|
||||
],
|
||||
"normalization_map": {
|
||||
"cityName": "geo_city"
|
||||
},
|
||||
"tags": [
|
||||
"passive"
|
||||
],
|
||||
"health_check_url": "http://api.ipinfodb.com",
|
||||
"expected_status": 200,
|
||||
"reliability_score": 3,
|
||||
"backup_endpoints": [],
|
||||
"confidence": 0.7
|
||||
}
|
||||
@@ -0,0 +1,31 @@
|
||||
{
|
||||
"name": "ipqualityscore_email",
|
||||
"category": "email_rep",
|
||||
"endpoint": "https://ipqualityscore.com/api/json/email/{IPQS_API_KEY}/{target}",
|
||||
"method": "GET",
|
||||
"requires_auth": true,
|
||||
"selectors": {
|
||||
"fraud_score": "$.fraud_score"
|
||||
},
|
||||
"rate_limit": 1.0,
|
||||
"headers": {},
|
||||
"api_key_slots": [
|
||||
"{IPQS_API_KEY}"
|
||||
],
|
||||
"input_type": "email",
|
||||
"output_type": [
|
||||
"email"
|
||||
],
|
||||
"normalization_map": {
|
||||
"fraud_score": "email_fraud_score"
|
||||
},
|
||||
"tags": [
|
||||
"passive",
|
||||
"fast"
|
||||
],
|
||||
"health_check_url": "https://ipqualityscore.com",
|
||||
"expected_status": 200,
|
||||
"reliability_score": 4,
|
||||
"backup_endpoints": [],
|
||||
"confidence": 0.85
|
||||
}
|
||||
@@ -0,0 +1,31 @@
|
||||
{
|
||||
"name": "ipstack",
|
||||
"category": "geolocation",
|
||||
"endpoint": "http://api.ipstack.com/{target}?access_key={IPSTACK_API_KEY}",
|
||||
"method": "GET",
|
||||
"requires_auth": true,
|
||||
"selectors": {
|
||||
"country": "$.country_name"
|
||||
},
|
||||
"rate_limit": 1.0,
|
||||
"headers": {},
|
||||
"api_key_slots": [
|
||||
"{IPSTACK_API_KEY}"
|
||||
],
|
||||
"input_type": "ip",
|
||||
"output_type": [
|
||||
"ip"
|
||||
],
|
||||
"normalization_map": {
|
||||
"country_name": "geo_country"
|
||||
},
|
||||
"tags": [
|
||||
"passive",
|
||||
"fast"
|
||||
],
|
||||
"health_check_url": "http://api.ipstack.com",
|
||||
"expected_status": 200,
|
||||
"reliability_score": 4,
|
||||
"backup_endpoints": [],
|
||||
"confidence": 0.85
|
||||
}
|
||||
@@ -0,0 +1,27 @@
|
||||
{
|
||||
"name": "ipvigilante",
|
||||
"category": "geolocation",
|
||||
"endpoint": "https://ipvigilante.com/json/{target}",
|
||||
"method": "GET",
|
||||
"requires_auth": false,
|
||||
"selectors": {
|
||||
"city": "$.data.city_name"
|
||||
},
|
||||
"rate_limit": 1.0,
|
||||
"headers": {},
|
||||
"api_key_slots": [],
|
||||
"input_type": "ip",
|
||||
"output_type": [
|
||||
"ip"
|
||||
],
|
||||
"normalization_map": {},
|
||||
"tags": [
|
||||
"passive"
|
||||
],
|
||||
"health_check_url": "https://ipvigilante.com",
|
||||
"expected_status": 200,
|
||||
"reliability_score": 3,
|
||||
"is_volatile": true,
|
||||
"backup_endpoints": [],
|
||||
"confidence": 0.7
|
||||
}
|
||||
@@ -0,0 +1,32 @@
|
||||
{
|
||||
"name": "joesandbox",
|
||||
"category": "threat_intel",
|
||||
"endpoint": "https://www.joesandbox.com/api/v2/analysis/search?q={target}",
|
||||
"method": "GET",
|
||||
"requires_auth": true,
|
||||
"selectors": {
|
||||
"id": "$.[*].id"
|
||||
},
|
||||
"rate_limit": 1.0,
|
||||
"headers": {
|
||||
"X-JoeSandbox-Api-Key": "{JOE_API_KEY}"
|
||||
},
|
||||
"api_key_slots": [
|
||||
"{JOE_API_KEY}"
|
||||
],
|
||||
"input_type": "hash",
|
||||
"output_type": [
|
||||
"hash"
|
||||
],
|
||||
"normalization_map": {},
|
||||
"tags": [
|
||||
"passive",
|
||||
"threat",
|
||||
"heavy"
|
||||
],
|
||||
"health_check_url": "https://www.joesandbox.com",
|
||||
"expected_status": 200,
|
||||
"reliability_score": 4,
|
||||
"backup_endpoints": [],
|
||||
"confidence": 0.85
|
||||
}
|
||||
@@ -0,0 +1,26 @@
|
||||
{
|
||||
"name": "keybase_lookup",
|
||||
"category": "social",
|
||||
"endpoint": "https://keybase.io/_/api/1.0/user/lookup.json?username={target}",
|
||||
"method": "GET",
|
||||
"requires_auth": false,
|
||||
"selectors": {
|
||||
"id": "$.them[0].id"
|
||||
},
|
||||
"rate_limit": 1.0,
|
||||
"headers": {},
|
||||
"api_key_slots": [],
|
||||
"input_type": "username",
|
||||
"output_type": [
|
||||
"username"
|
||||
],
|
||||
"normalization_map": {},
|
||||
"tags": [
|
||||
"passive"
|
||||
],
|
||||
"health_check_url": "https://keybase.io",
|
||||
"expected_status": 200,
|
||||
"reliability_score": 4,
|
||||
"backup_endpoints": [],
|
||||
"confidence": 0.85
|
||||
}
|
||||
@@ -0,0 +1,26 @@
|
||||
{
|
||||
"name": "keybase_proofs",
|
||||
"category": "social",
|
||||
"endpoint": "https://keybase.io/_/api/1.0/user/lookup.json?usernames={target}",
|
||||
"method": "GET",
|
||||
"requires_auth": false,
|
||||
"selectors": {
|
||||
"proofs": "$.them[0].proofs_summary.all[*].namestr"
|
||||
},
|
||||
"rate_limit": 1.0,
|
||||
"headers": {},
|
||||
"api_key_slots": [],
|
||||
"input_type": "username",
|
||||
"output_type": [
|
||||
"username"
|
||||
],
|
||||
"normalization_map": {},
|
||||
"tags": [
|
||||
"passive"
|
||||
],
|
||||
"health_check_url": "https://keybase.io",
|
||||
"expected_status": 200,
|
||||
"reliability_score": 4,
|
||||
"backup_endpoints": [],
|
||||
"confidence": 0.85
|
||||
}
|
||||
@@ -0,0 +1,32 @@
|
||||
{
|
||||
"name": "leak_lookup",
|
||||
"category": "breaches",
|
||||
"endpoint": "https://leak-lookup.com/api/search",
|
||||
"method": "POST",
|
||||
"requires_auth": false,
|
||||
"selectors": {
|
||||
"results": "$.message"
|
||||
},
|
||||
"rate_limit": 1.0,
|
||||
"headers": {},
|
||||
"payload_template": {
|
||||
"query": "{target}",
|
||||
"type": "email_address"
|
||||
},
|
||||
"api_key_slots": [],
|
||||
"input_type": "email",
|
||||
"output_type": [
|
||||
"email"
|
||||
],
|
||||
"normalization_map": {},
|
||||
"tags": [
|
||||
"passive",
|
||||
"stealth"
|
||||
],
|
||||
"health_check_url": "https://leak-lookup.com",
|
||||
"expected_status": 200,
|
||||
"reliability_score": 3,
|
||||
"is_volatile": true,
|
||||
"backup_endpoints": [],
|
||||
"confidence": 0.7
|
||||
}
|
||||
@@ -0,0 +1,33 @@
|
||||
{
|
||||
"name": "leakcheck",
|
||||
"category": "breaches",
|
||||
"endpoint": "https://leakcheck.io/api/v2/query/{target}",
|
||||
"method": "GET",
|
||||
"requires_auth": true,
|
||||
"selectors": {
|
||||
"sources": "$.sources"
|
||||
},
|
||||
"rate_limit": 1.0,
|
||||
"headers": {
|
||||
"X-API-Key": "{LEAKCHECK_API_KEY}"
|
||||
},
|
||||
"api_key_slots": [
|
||||
"{LEAKCHECK_API_KEY}"
|
||||
],
|
||||
"input_type": "email",
|
||||
"output_type": [
|
||||
"email"
|
||||
],
|
||||
"normalization_map": {
|
||||
"sources": "breach_sources"
|
||||
},
|
||||
"tags": [
|
||||
"passive",
|
||||
"stealth"
|
||||
],
|
||||
"health_check_url": "https://leakcheck.io",
|
||||
"expected_status": 200,
|
||||
"reliability_score": 4,
|
||||
"backup_endpoints": [],
|
||||
"confidence": 0.85
|
||||
}
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user