collecting-open-source-intelligence

Collecting Open-Source Intelligence

When to Use

Use this skill when:

• Investigating external infrastructure associated with a phishing campaign targeting your organization
• Enriching threat actor profiles with publicly observable indicators (WHOIS, ASN data, SSL certificates)
• Conducting authorized attack surface discovery to understand your organization's external exposure

Do not use this skill for active scanning against targets without explicit written authorization — OSINT collection must remain passive (no packets sent to target systems) unless scope permits active recon.

Prerequisites

• Maltego CE or commercial license for graph-based link analysis
• Shodan API key (https://shodan.io) for internet-wide device/service discovery
• OSINT Framework familiarity (https://osintframework.com) for tool selection
• SpiderFoot HX or open-source SpiderFoot for automated OSINT correlation

Workflow

Step 1: Define Collection Requirements

Establish the intelligence requirement (IR) before collecting. Document:

• Target: threat actor group, malicious domain, IP range, or organization
• Priority Intelligence Requirements (PIRs): What specific questions need answering?
• Legal authority: Passive OSINT is legal; active probing requires authorization
• Data handling: TLP classification for collected intelligence

Step 2: Passive DNS and WHOIS Investigation

# Passive DNS via SecurityTrails API
curl "https://api.securitytrails.com/v1/domain/evil-domain.com/dns/a" \
  -H "apikey: YOUR_KEY"

# WHOIS history via ARIN / RIPE
whois -h whois.arin.net evil-domain.com

# Certificate transparency logs (no API key required)
curl "https://crt.sh/?q=%.evil-domain.com&output=json" | jq '.[].name_value'

Certificate transparency logs reveal all subdomains for a target domain, often exposing staging, VPN, or internal infrastructure inadvertently made public.

Step 3: Shodan Infrastructure Mapping

import shodan

api = shodan.Shodan("YOUR_SHODAN_API_KEY")

# Search for specific C2 framework signatures (Cobalt Strike beacon)
results = api.search('product:"Cobalt Strike" port:443')
for r in results['matches']:
    print(r['ip_str'], r['port'], r['org'], r.get('ssl', {}).get('cert', {}).get('subject', ''))

# Find infrastructure associated with a known threat actor's ASN
results = api.search('asn:AS12345 http.title:"Redirector"')

Correlate Shodan results with passive DNS to build infrastructure clusters.

Step 4: Maltego Graph Analysis

In Maltego, use these built-in transforms for threat actor infrastructure mapping:

1. Start with a known malicious domain (Entity: Domain)
2. Run "To IP Address [DNS]" → identifies hosting IPs
3. Run "To Shared Hosting" → identifies co-hosted domains (potentially same threat actor)
4. Run "To DNS Name [Reverse DNS]" → identifies PTR records
5. Run "To Whois" → identifies registrant email/organization
6. Pivot on registrant email → "To Domains [Registrant Email]" → expands to all domains registered with same email

Maltego Maltego Cyber threat intelligence transforms (VirusTotal, Shodan, PassiveTotal, URLScan) extend graph coverage.

Step 5: Dark Web and Paste Site Monitoring

Use SpiderFoot HX or manual searches for:

• Paste sites (Pastebin, Ghostbin): search for leaked credentials, IOCs, malware configs
• Dark web forums: via Tor browser with appropriate operational security
• GitHub/GitLab: search for exposed credentials or organization-specific strings

# SpiderFoot CLI for automated OSINT
python sf.py -s evil-domain.com -m sfp_shodan,sfp_virustotal,sfp_passivetotal \
  -o TF -R result.json

Key Concepts

Term	Definition
Passive OSINT	Intelligence collection that does not send any packets to target systems — uses public databases, search engines, cached data
PIR	Priority Intelligence Requirement — specific question the intelligence collection must answer, preventing unfocused data gathering
Certificate Transparency	Public log of all SSL/TLS certificates issued by CAs, enabling discovery of subdomains via crt.sh
Pivoting	Using one data point (IP, email, registrant name) to discover related infrastructure or accounts
ASN	Autonomous System Number — block of IP addresses under a single routing policy; useful for clustering threat actor infrastructure
Co-hosted Domains	Multiple domains resolving to the same IP, potentially indicating shared attacker infrastructure

Tools & Systems

• Maltego: Graph-based link analysis platform with 50+ data source transforms for IP, domain, email, and social media analysis
• Shodan: Internet-wide scanner database with 1B+ indexed devices; supports banner, port, SSL certificate, and vulnerability searches
• SpiderFoot: Automated OSINT tool with 200+ modules covering DNS, WHOIS, dark web, breach data, and social media
• Recon-ng: Python-based OSINT framework with modular design for domain, email, and social media reconnaissance
• crt.sh: Free certificate transparency search engine for subdomain and certificate discovery
• OSINT Framework (osintframework.com): Curated directory of OSINT tools organized by intelligence category

Common Pitfalls

• Leaving digital footprints: Visiting a threat actor's website or Shodan-queried IP can alert the adversary. Use Tor or VPN with a dedicated OSINT VM.
• Confirmation bias in graph analysis: Maltego graphs can create false connections. Verify each pivot independently before treating as confirmed.
• Outdated data: WHOIS privacy services and bulletproof hosting rotate frequently. Always check data timestamps — 6-month-old passive DNS may no longer be valid.
• Attribution overconfidence: Infrastructure overlap does not guarantee same threat actor. False flag operations deliberately share indicators across groups.
• Legal boundaries: Some OSINT tools perform active scans (port scanning, banner grabbing). Confirm tool behavior before use against external targets without authorization.