corp-osint
Maps a company to subsidiaries, brands, websites, urlscan/Playwright/DNSLytics GTM discoveries, IdefixTools Entra tenant→domains, RDAP/WHOIS, ASN/IP ownership, Azure tenant IDs, and dangling parked domains still verified in Entra. Use for corporate OSINT, GTM/tenant reverse domains, or dangling DNS/Entra findings.
- License 许可
- Apache-2.0
- 所属仓库
- Salto7/Peon
- 最近更新
- 2026-11-09
- Stars
- ★ 0
- Forks
- 0
01 前言 Front Matter
- name
- corp-osint
- license
- Apache-2.0
- metadata
- { "author": "corp-OSINT", "version": "3.0.0", "category": "osint" }
- description
- Maps a company to subsidiaries, brands, websites, urlscan/Playwright/DNSLytics GTM discoveries, IdefixTools Entra tenant→domains, RDAP/WHOIS, ASN/IP ownership, Azure tenant IDs, and dangling parked domains still verified in Entra. Use for corporate OSINT, GTM/tenant reverse domains, or dangling DNS/Entra findings.
- compatibility
- Python 3.10+, network, aiohttp, tenacity, Playwright chromium; optional whois, URLSCAN_API_KEY, DNSLYTICS_API_KEY. Works with any agent that supports skills and/or MCP.
02 资源文件 3
- 参考文档mcp.md3.4 KB
- 参考文档report-schema.md5.2 KB
- 参考文档sources.md4.4 KB
03 SKILL.md 正文
Corporate OSINT — Entity Expansion (v3)
Passive public-source recon only. Plugin architecture:
- One entrypoint:
uv run python main.py - One abstract base:
corp_osint.plugin.Plugin - All stages live in
corp_osint/plugins/and inheritPlugin - Orchestrator:
CorpOsintPipeline(Plugin)runs registered plugins
Install
From the repo root (corp-OSINT/), one-time:
cd /path/to/corp-OSINT
# https://docs.astral.sh/uv/getting-started/installation/
uv sync
uv run playwright install chromium
# optional (RDAP fallback / richer rate limits)
sudo apt install whois # or: brew install whois
export SEC_USER_AGENT='CorpOSINT/3.0 (research; you@example.com)'
export URLSCAN_API_KEY=... # optional — higher urlscan limits
export DNSLYTICS_API_KEY=... # optional — paid GTM reverse API
# optional proxy (HTTP or SOCKS) for aiohttp + Playwright:
export CORP_OSINT_PROXY='socks5://127.0.0.1:1080'
# export HTTP_PROXY=... HTTPS_PROXY=... ALL_PROXY=... # also honored
# export NO_PROXY=localhost,127.0.0.1,.local
Requirements: uv, Python 3.10+, network access.
MCP hosts launch corp-osint-mcp via uv run (see .mcp.json, docs/mcp.md).
Verify install:
uv run python main.py --list-plugins
uv run python main.py --plugin azure_tenant --domain microsoft.com
Agents: run the Install block if uv is missing or Playwright/Chromium is not installed.
Agent rules (quality)
- Pipeline JSON is raw evidence, not truth. Regex/EDGAR parsers can emit
header junk (
Name of the Subsidiary) or invent slug domains. Critically filter before presenting findings. - Present the compact
summaryfirst (alsosummary_text). Each domain row:domain,sources,evidence,confidence,verified. - Do not invent subsidiary
.coms. Subsidiaries are names until urlscan, Entra tenant domains, DNSLytics, or a verified homepage links them. If--domain/ MCPdomainis omitted, Playwright runs first to seed the parent site; inventing parent.com/.netis only a last-resort fallback. If--domain/ MCPdomainis omitted, Playwright runs first to seed the parent site; inventing parent.com/.netis only a last-resort fallback. If--domain/ MCPdomainis omitted, Playwright runs first to seed the parent site; inventing parent.com/.netis only a last-resort fallback. - Prefer AI judgment over brittle string matches when classifying parking pages, brand vs legal-shell names, and whether a domain belongs to the company.
Quick run
uv run python main.py "Example Corp" --domain example.com --out report.json
uv run python main.py -v "Example Corp" --domain example.com --summary-only
uv run python main.py --list-plugins
uv run python main.py --plugin dnslytics_gtm --domain example.com
Logging (stderr via logger only — no print): default INFO stage progress;
-v / -vv DEBUG; -q warnings for progress (result JSON still emitted).
Reports are compact by default (no WHOIS dumps); summary + summary_text
are always first keys. Use --summary-only for summary JSON alone.
Flags: --skip STAGES, --only STAGES (comma-separated:
edgar,urlscan,playwright,asn,dnslytics,tenant_domains,entra_verify,dangling,resolve_sites),
--summary-only, --proxy URL, --direct, --no-proxy HOSTS, --plugin NAME,
--tenant-id, -v/-vv, -q. Tuning caps live in PipelineConfig.limits
(defaults); not exposed on the CLI.
Proxy (HTTP + SOCKS)
All aiohttp traffic and Playwright Chromium share one proxy setting:
# SOCKS5
uv run python main.py "Example Corp" --domain example.com --proxy socks5://127.0.0.1:1080
# HTTP proxy + bypass list
uv run python main.py --plugin azure_tenant --domain microsoft.com \
--proxy http://127.0.0.1:8080 --no-proxy localhost,127.0.0.1
# Force direct (ignore env proxies)
uv run python main.py --plugin azure_tenant --domain microsoft.com --direct
Also: CORP_OSINT_PROXY, ALL_PROXY, HTTPS_PROXY, HTTP_PROXY, and
NO_PROXY / CORP_OSINT_NO_PROXY. Requires aiohttp-socks (in requirements.txt).
Architecture
| Piece | Role |
|---|---|
uv run python main.py |
Single entrypoint |
corp_osint.plugin.Plugin |
Abstract base — every module inherits this |
corp_osint.plugins.* |
Concrete plugins (edgar, urlscan, …) |
corp_osint.plugins.pipeline.CorpOsintPipeline |
Orchestrator plugin (name="pipeline") |
corp_osint.http.AsyncHttpClient |
aiohttp + throttle + 429 retries |
To add a stage: subclass Plugin, implement async def run(**kwargs), register in
plugins/__init__.py → PLUGIN_REGISTRY.
Checklist
- [ ] 1 Parent (EDGAR CIK / legal name / domain)
- [ ] 2 Subsidiaries / brands (names; filter Exhibit 21 headers)
- [ ] 3 Resolve parent domain only (no mass subsidiary .com guessing)
- [ ] 4 urlscan + Playwright domain expansion
- [ ] 5 DNSLytics: GTM tags → reverse domains
- [ ] 6 RDAP/WHOIS + ASN/IP + Azure tenants (parallel)
- [ ] 7 IdefixTools tenant lookup → more Entra-verified domains
- [ ] 8 Flag parked domains whose Entra tenant matches company (dangling)
- [ ] 9 Emit JSON + compact summary (company / domains / sources / confidence / verified)
- [ ] 10 Agent critically filters pipeline output before final answer
Tenant domains (IdefixTools)
Same public flow as https://idefixtools.com/tenant-lookup.html:
uv run python main.py --plugin tenant_domains --domain example.com
uv run python main.py --plugin tenant_domains --tenant-id <guid>
Uses func-idefix-tenantlookup.azurewebsites.net/api/lookup|domains plus
Autodiscover GetFederationInformation fallback.
Entra OIDC verify (shared)
Once a company Entra tenant ID is known, every found domain is checked via shared
corp_osint.plugins.entra.fetch_oidc against:
https://login.microsoftonline.com/{domain}/v2.0/.well-known/openid-configuration
Matches (issuer tenant == company tenant) are marked matches_company_tenant /
relationship_verified. Reuses the same helper as azure_tenant and
tenant_domains (no duplicated OIDC parsing). Cap: LimitsConfig.entra_verify_max
(default 200). Skip with --skip entra_verify.
Dangling / parked / unregistered (Entra)
After tenant→domains + OIDC verify, dangling_check probes Entra-verified apex
domains only (skips subdomains like app.example.com — RDAP/DNS risk is on the
registrable name). Flagged as dangling when Entra-verified and any of:
| kind | Meaning |
|---|---|
parked |
Parking / for-sale page |
nxdomain |
DNS does not resolve |
unregistered |
RDAP not found / available |
unreachable |
DNS may work but HTTP dead |
Results: dangling_domains[] with kind, plus dangling_check.counts.
Cap: LimitsConfig.dangling_max_domains (default 200). Skip with --skip dangling.
DNSLytics / GTM
uv run python main.py --plugin dnslytics_gtm --domain example.com
# optional paid API:
export DNSLYTICS_API_KEY=...
Flow: Playwright homepage GTM extract → DNSLytics UI via Playwright → reverse
html.tag:GTM-XXXX. On HTTP 429: honor Retry-After / exponential backoff, recycle
browser page (up to 5 attempts), HTML scrape fallback. Spacing capped at 3s
(+200ms incremental backoff on 429). Official reverse API used when
DNSLYTICS_API_KEY set.
Retries / 429 (incremental sleeps)
AsyncHttpClient: up to 6 attempts; parsesRetry-After; per-host cooldown- On 429 / 5xx / transient errors: incremental sleep in +200ms chunks
(200ms → 400ms → 600ms …), hard-capped at 3s total
(
rate-limit … chunk=N sleep=… remaining=…on stderr) - Host pacing uses the same 200ms / 3s policy when a cooldown is active
- urlscan queries run sequentially; inter-query gap grows by +200ms (max 3s)
- DNSLytics / Playwright: same incremental backoff on 429 / empty / bot signals
- Playwright: Bing retries + DuckDuckGo fallback; homepage link discovery;
default
LimitsConfig.playwright_max_queries= 6
Progress logging
Every plugin emits stderr progress around run():
INFO progress [edgar] starting
INFO progress [edgar] resolve company=…
INFO progress [edgar] done (12.3s)
Looping modules also log step progress (query 2/6, probing example.com, …).
Use -v for DEBUG detail; -q for quieter progress (result still logged).
Speed notes
- Domain enrichments (RDAP/Azure/ASN) run concurrently
- Playwright preferred for discovery; urlscan still sequential where safe
- ASN defaults to observed IPs only (
PipelineConfig.asn_deepfor prefixes) - Caps:
LimitsConfig.max_entities/max_enrich_domains(defaults 12 / 6) - Host throttles in
corp_osint.config.ThrottleConfig
Compact summary shape
Report includes summary and summary_text:
{
"company": "Example Corporation",
"primary_domain": "example.com",
"domains": [
{
"domain": "example.com",
"sources": ["https://…"],
"evidence": "Primary / query domain",
"confidence": "high",
"verified": true
}
],
"entities": [
{
"name": "Example Brand LLC",
"domains": [],
"sources": ["https://www.sec.gov/…"],
"evidence": "Exhibit 21 …",
"confidence": "high",
"verified": true
}
]
}
Output rules
Cite sources; never invent CIKs/tenants/ASNs/GTM IDs or subsidiary domains; separate confirmed vs candidate; no port scans.
Using with an agent
- MCP: register
corp-osint-mcpwith your host (see mcp.md); any MCP client works. - Skill file: this playbook is plain Markdown with front matter. Copy or link
docs/into your agent's skills/instructions location (for example acorp-osint/skills folder) — the path is host-specific. - CLI fallback: agents without MCP can run
uv run python main.py ….
See sources.md and report-schema.md.
04 同仓库更多 Skills
bbot-passive-osint
Use BBOT for passive, authorized OSINT and attack-surface discovery to identify related domains, subdomains, URLs, IPs, organizations, and other candidate targets without launching active scans.
allowed-tools-parens
Fixture for Bash(git diff *) tokenization