ti-ioc-extraction
Automated IOC extraction from threat reports, logs, and unstructured text — parse hashes, IPs, domains, URLs, email addresses, and CVEs. Covers regex-based extraction, defanging/refanging, bulk hash lookup, IOC deduplication, YARA rule generation from IOCs, and STIX/TAXII formatting for sharing.
- License 许可
- Apache-2.0
- 最近更新
- 2026-11-09
- Stars
- ★ 0
- Forks
- 0
01 前言 Front Matter
- name
- ti-ioc-extraction
- metadata
- { "tags": "ioc, extraction, threat-intelligence, defang, refang, stix, taxii, misp, hashes, domains, ips", "subdomain": "analyst", "when_to_use": "ioc extraction, indicator of compromise, parse hashes, extract ips, extract domains, defang, refang, ioc-finder, ioc_fanger, cyberchef, misp, stix, taxii, yara from iocs, bulk hash lookup, threat report parsing, ioc deduplication", "mitre_attack": "T1590, T1592" }
- description
- Automated IOC extraction from threat reports, logs, and unstructured text — parse hashes, IPs, domains, URLs, email addresses, and CVEs. Covers regex-based extraction, defanging/refanging, bulk hash lookup, IOC deduplication, YARA rule generation from IOCs, and STIX/TAXII formatting for sharing.
- allowed-tools
- Bash Read Write
03 SKILL.md 正文
IOC Extraction from Unstructured Text
Extract Indicators of Compromise from threat intelligence reports, incident logs, paste dumps, and raw text. Normalize, deduplicate, and format for downstream consumption (MISP, STIX/TAXII feeds, YARA rules, blocklists).
Quick Reference
# Install tools
pip install ioc-finder ioc_fanger stix2 pymisp
# One-shot extraction from a report file
python3 -c "
from ioc_finder import find_iocs
text = open('/workspace/report.txt').read()
iocs = find_iocs(text)
for k,v in iocs.items():
if v: print(f'{k}: {v}')
"
# Refang a defanged IOC list
python3 -c "
from ioc_fanger import fang
text = open('/workspace/defanged_iocs.txt').read()
print(fang(text))
" > /workspace/refanged_iocs.txt
# Quick regex extraction (no deps)
grep -oP '\b[0-9a-fA-F]{32}\b' /workspace/report.txt | sort -u > /workspace/md5_hashes.txt
grep -oP '\b[0-9a-fA-F]{64}\b' /workspace/report.txt | sort -u > /workspace/sha256_hashes.txt
grep -oP '\b(?:[0-9]{1,3}\.){3}[0-9]{1,3}\b' /workspace/report.txt | sort -u > /workspace/ips.txt
MITRE ATT&CK Mapping
| Technique | ID | Relevance |
|---|---|---|
| Gather Victim Network Information | T1590 | Extracted IPs, domains, and URLs map victim-facing infrastructure |
| Gather Victim Host Information | T1592 | Extracted file hashes, registry keys, and file paths indicate host-level artifacts |
| Phishing | T1566 | Extracted email addresses and sender domains from phishing report IOCs |
| Indicator Removal | T1070 | Awareness of IOC types attackers attempt to obscure or rotate |
1. Regex-Based IOC Extraction
When ioc-finder is unavailable or you need precise control, use these regex patterns:
# MD5 hashes (32 hex chars, standalone word boundary)
grep -oP '\b[0-9a-fA-F]{32}\b' /workspace/<REPORT> | sort -u > /workspace/iocs/md5.txt
# SHA1 hashes (40 hex chars)
grep -oP '\b[0-9a-fA-F]{40}\b' /workspace/<REPORT> | sort -u > /workspace/iocs/sha1.txt
# SHA256 hashes (64 hex chars)
grep -oP '\b[0-9a-fA-F]{64}\b' /workspace/<REPORT> | sort -u > /workspace/iocs/sha256.txt
# IPv4 addresses
grep -oP '\b(?:(?:25[0-5]|2[0-4]\d|[01]?\d\d?)\.){3}(?:25[0-5]|2[0-4]\d|[01]?\d\d?)\b' \
/workspace/<REPORT> | sort -u > /workspace/iocs/ipv4.txt
# IPv6 addresses (simplified — catches common forms)
grep -oiP '(?:[0-9a-f]{1,4}:){7}[0-9a-f]{1,4}|(?:[0-9a-f]{1,4}:){1,7}:|::(?:[0-9a-f]{1,4}:){0,5}[0-9a-f]{1,4}' \
/workspace/<REPORT> | sort -u > /workspace/iocs/ipv6.txt
# Domain names (basic, excludes common FPs)
grep -oP '(?:[a-zA-Z0-9](?:[a-zA-Z0-9-]{0,61}[a-zA-Z0-9])?\.)+(?:com|net|org|io|ru|cn|xyz|top|info|biz|cc|tk|ml|ga|cf|pw|buzz|su|onion)\b' \
/workspace/<REPORT> | sort -u > /workspace/iocs/domains.txt
# URLs (http/https/ftp)
grep -oP 'https?://[^\s"<>\])+' /workspace/<REPORT> | sort -u > /workspace/iocs/urls.txt
# Email addresses
grep -oP '[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}' \
/workspace/<REPORT> | sort -u > /workspace/iocs/emails.txt
# CVE identifiers
grep -oP 'CVE-\d{4}-\d{4,7}' /workspace/<REPORT> | sort -u > /workspace/iocs/cves.txt
# MITRE ATT&CK technique IDs
grep -oP 'T\d{4}(?:\.\d{3})?' /workspace/<REPORT> | sort -u > /workspace/iocs/mitre_ids.txt
False Positive Reduction
# Filter out private/reserved IPs from extracted list
grep -vP '^(10\.|172\.(1[6-9]|2\d|3[01])\.|192\.168\.|127\.|0\.|255\.)' \
/workspace/iocs/ipv4.txt > /workspace/iocs/ipv4_public.txt
# Filter out known benign domains (Google, Microsoft, etc.)
grep -viP '(google\.com|microsoft\.com|windowsupdate\.com|apple\.com|amazonaws\.com|cloudflare\.com)$' \
/workspace/iocs/domains.txt > /workspace/iocs/domains_filtered.txt
# Remove hash-like strings that are too short or all-zero
grep -vP '^0{32,}$' /workspace/iocs/md5.txt | grep -vP '^(.)\1{31,}$' > /workspace/iocs/md5_clean.txt
2. Defanging and Refanging
Threat reports often defang IOCs to prevent accidental clicks. You must refang before lookups.
# Refang with ioc_fanger (Python)
python3 -c "
from ioc_fanger import fang
import sys
text = sys.stdin.read()
print(fang(text))
" < /workspace/defanged_report.txt > /workspace/refanged_report.txt
# Manual refang with sed (no deps)
sed -e 's/\[.\]/./g' \
-e 's/hxxp/http/g' \
-e 's/hxxps/https/g' \
-e 's/ dot /./g' \
-e 's/\[at\]/@/g' \
-e 's/{dot}/./g' \
/workspace/defanged_report.txt > /workspace/refanged_report.txt
# Defang IOCs for safe sharing
python3 -c "
from ioc_fanger import defang
import sys
text = sys.stdin.read()
print(defang(text))
" < /workspace/iocs/urls.txt > /workspace/iocs/urls_defanged.txt
# Manual defang with sed
sed -e 's/\./[.]/g' \
-e 's|http://|hxxp://|g' \
-e 's|https://|hxxps://|g' \
-e 's/@/[at]/g' \
/workspace/iocs/urls.txt > /workspace/iocs/urls_defanged.txt
Common Defang Patterns to Recognize
| Original | Defanged Variants |
|---|---|
192.168.1.1 |
192[.]168[.]1[.]1, 192.168.1 .1, 192(.)168(.)1(.)1 |
http://evil.com |
hxxp://evil[.]com, hxxp://evil(.)com, http[:]//evil.com |
user@evil.com |
user[at]evil[.]com, user[@]evil.com |
3. Programmatic Extraction with ioc-finder
# Full extraction pipeline
python3 << 'PYEOF'
from ioc_finder import find_iocs
import json, sys
text = open('/workspace/<REPORT>').read()
iocs = find_iocs(text)
# Filter to populated categories only
results = {k: list(set(v)) for k, v in iocs.items() if v}
with open('/workspace/iocs/extracted.json', 'w') as f:
json.dump(results, f, indent=2, default=str)
# Summary
for category, items in sorted(results.items()):
print(f" {category}: {len(items)} unique")
print(f"\nTotal categories with hits: {len(results)}")
PYEOF
# ioc-finder extracts these categories automatically:
# - md5s, sha1s, sha256s, sha512s, ssdeeps
# - ipv4s, ipv6s, urls, domains, email_addresses
# - asns, cves, registry_key_paths, file_paths
# - mac_addresses, bitcoin_addresses, xmpp_addresses
# - google_adsense_ids, google_analytics_ids
4. Bulk Hash Lookup
# VirusTotal bulk lookup (requires API key)
VT_API_KEY="<API_KEY>"
while IFS= read -r hash; do
result=$(curl -s "https://www.virustotal.com/api/v3/files/$hash" \
-H "x-apikey: $VT_API_KEY")
detections=$(echo "$result" | python3 -c "
import sys, json
d = json.load(sys.stdin)
stats = d.get('data',{}).get('attributes',{}).get('last_analysis_stats',{})
print(f\"{stats.get('malicious',0)}/{sum(stats.values())}\")" 2>/dev/null)
echo "$hash $detections" >> /workspace/iocs/vt_results.txt
sleep 15 # Free API: 4 requests/min
done < /workspace/iocs/sha256.txt
# MalwareBazaar lookup (free, no rate limit issues)
while IFS= read -r hash; do
curl -s -X POST "https://mb-api.abuse.ch/api/v1/" \
-d "query=get_info&hash=$hash" \
-o "/workspace/iocs/mb_${hash}.json"
done < /workspace/iocs/sha256.txt
# AbuseIPDB bulk check
ABUSE_KEY="<API_KEY>"
while IFS= read -r ip; do
result=$(curl -s "https://api.abuseipdb.com/api/v2/check?ipAddress=$ip" \
-H "Key: $ABUSE_KEY" -H "Accept: application/json")
score=$(echo "$result" | python3 -c "import sys,json; print(json.load(sys.stdin)['data']['abuseConfidenceScore'])" 2>/dev/null)
echo "$ip abuse_score=$score" >> /workspace/iocs/abuse_results.txt
done < /workspace/iocs/ipv4_public.txt
5. IOC Deduplication and Normalization
python3 << 'PYEOF'
import json, hashlib, re
from collections import defaultdict
with open('/workspace/iocs/extracted.json') as f:
iocs = json.load(f)
deduped = {}
stats = {}
for category, items in iocs.items():
# Normalize: lowercase domains/URLs/emails, uppercase hashes
if category in ('md5s', 'sha1s', 'sha256s', 'sha512s'):
normalized = sorted(set(h.lower() for h in items))
elif category in ('domains', 'urls', 'email_addresses'):
normalized = sorted(set(i.lower().rstrip('.') for i in items))
elif category in ('ipv4s', 'ipv6s'):
normalized = sorted(set(items))
else:
normalized = sorted(set(items))
if normalized:
deduped[category] = normalized
stats[category] = {'original': len(items), 'deduped': len(normalized)}
with open('/workspace/iocs/deduped.json', 'w') as f:
json.dump(deduped, f, indent=2)
print("Deduplication stats:")
for cat, s in sorted(stats.items()):
removed = s['original'] - s['deduped']
print(f" {cat}: {s['original']} -> {s['deduped']} (removed {removed} dupes)")
PYEOF
6. STIX/TAXII Formatting
# Convert extracted IOCs to STIX 2.1 bundle
python3 << 'PYEOF'
from stix2 import (Bundle, Indicator, Identity, MarkingDefinition,
TLP_WHITE, TLP_GREEN, TLP_AMBER)
import json, datetime
identity = Identity(
name="Decepticon TI Team",
identity_class="organization"
)
with open('/workspace/iocs/deduped.json') as f:
iocs = json.load(f)
indicators = []
# Map IOC types to STIX patterns
pattern_map = {
'ipv4s': lambda x: f"[ipv4-addr:value = '{x}']",
'ipv6s': lambda x: f"[ipv6-addr:value = '{x}']",
'domains': lambda x: f"[domain-name:value = '{x}']",
'urls': lambda x: f"[url:value = '{x}']",
'md5s': lambda x: f"[file:hashes.MD5 = '{x}']",
'sha1s': lambda x: f"[file:hashes.'SHA-1' = '{x}']",
'sha256s': lambda x: f"[file:hashes.'SHA-256' = '{x}']",
'email_addresses': lambda x: f"[email-addr:value = '{x}']",
}
for ioc_type, items in iocs.items():
if ioc_type not in pattern_map:
continue
for item in items:
indicator = Indicator(
name=f"{ioc_type.rstrip('s').upper()}: {item}",
pattern=pattern_map[ioc_type](item),
pattern_type="stix",
valid_from=datetime.datetime.now(datetime.timezone.utc),
created_by_ref=identity.id,
object_marking_refs=[TLP_AMBER],
labels=["malicious-activity"]
)
indicators.append(indicator)
bundle = Bundle(objects=[identity] + indicators)
with open('/workspace/iocs/stix_bundle.json', 'w') as f:
f.write(bundle.serialize(pretty=True))
print(f"STIX bundle: {len(indicators)} indicators in {len(iocs)} categories")
PYEOF
Push to MISP
# Upload IOCs to MISP instance
python3 << 'PYEOF'
from pymisp import PyMISP, MISPEvent, MISPAttribute
import json
misp = PyMISP('<MISP_URL>', '<MISP_API_KEY>', ssl=False)
event = MISPEvent()
event.info = "IOCs extracted from <REPORT_NAME>"
event.distribution = 0 # Organization only
event.threat_level_id = 2 # Medium
event.analysis = 1 # Ongoing
with open('/workspace/iocs/deduped.json') as f:
iocs = json.load(f)
type_map = {
'ipv4s': 'ip-dst',
'domains': 'domain',
'urls': 'url',
'md5s': 'md5',
'sha1s': 'sha1',
'sha256s': 'sha256',
'email_addresses': 'email-src',
}
for ioc_type, items in iocs.items():
misp_type = type_map.get(ioc_type)
if not misp_type:
continue
for item in items:
attr = MISPAttribute()
attr.type = misp_type
attr.value = item
attr.to_ids = True
event.add_attribute(**attr)
result = misp.add_event(event)
print(f"MISP event created: {result.get('Event', {}).get('id', 'ERROR')}")
PYEOF
7. CyberChef Recipes for IOC Processing
CyberChef provides browser-based IOC transforms. Key recipes for common tasks:
# Extract IPs — CyberChef recipe (paste into recipe URL)
Extract_IP_addresses()&Sort('Line feed',false,'Alpha')&Unique('Line feed')
# Extract URLs
Extract_URLs(false)&Sort('Line feed',false,'Alpha')&Unique('Line feed')
# Extract email addresses
Extract_email_addresses(false)&Unique('Line feed')
# Defang URLs
Defang_URL(true,true,true,'Valid domains and full URLs')
# Decode Base64-encoded IOCs (common in obfuscated reports)
From_Base64('A-Za-z0-9+/=',true,false)&Extract_IP_addresses()
# Hex-encoded IOC decode
From_Hex('Auto')&Extract_URLs(false)
# Full pipeline: decode + extract + defang + dedup
From_Base64('A-Za-z0-9+/=',true,false)&Extract_URLs(false)&Unique('Line feed')&Defang_URL()
Tools & Resources
| Tool | Purpose | Install |
|---|---|---|
| ioc-finder | Python IOC extractor — 20+ IOC types | pip install ioc-finder |
| ioc_fanger | Defang/refang IOC strings | pip install ioc_fanger |
| CyberChef | Browser-based IOC transforms | https://gchq.github.io/CyberChef/ |
| PyMISP | Python MISP API client | pip install pymisp |
| stix2 | STIX 2.1 Python library | pip install stix2 |
| MalwareBazaar | Free hash lookup (abuse.ch) | API: mb-api.abuse.ch |
| VirusTotal | Hash/IP/domain lookup | API key required |
| AbuseIPDB | IP reputation lookup | API key required |
| ThreatFox | IOC sharing (abuse.ch) | API: threatfox-api.abuse.ch |
Detection Signatures
| Pattern | Indicates | Example Match |
|---|---|---|
[0-9a-fA-F]{32} |
MD5 hash | d41d8cd98f00b204e9800998ecf8427e |
[0-9a-fA-F]{40} |
SHA1 hash | da39a3ee5e6b4b0d3255bfef95601890afd80709 |
[0-9a-fA-F]{64} |
SHA256 hash | e3b0c44298fc1c149afbf4c8996fb924... |
hxxp[s]?:// |
Defanged URL | hxxps://evil[.]com/payload |
[.] in hostname |
Defanged domain/IP | 192[.]168[.]1[.]1 |
CVE-\d{4}-\d{4,} |
CVE identifier | CVE-2024-12345 |
T\d{4}(\.\d{3})? |
MITRE technique ID | T1566.001 |
Error Handling & Edge Cases
| Problem | Cause | Solution |
|---|---|---|
| Hash regex matches CSS colors | 6-char hex without word boundary | Use \b anchors + length check (32/40/64) |
| IP regex matches version strings | 1.2.3.4 in software versions |
Cross-reference with CIDR ranges; discard private IPs if hunting external C2 |
| ioc-finder returns empty | Input encoding issue (UTF-16, BOM) | Convert to UTF-8 first: iconv -f UTF-16 -t UTF-8 |
| VT rate limit hit | Free API = 4 req/min | Add sleep 15 between requests; use premium key for bulk |
| STIX bundle too large | Thousands of IOCs | Split into per-type bundles or paginate |
| MISP event creation fails | SSL cert / auth error | Verify MISP_URL, API key, and set ssl=False for self-signed |
| Defanged IOCs not caught | Unusual defang format ({dot}, [dot]) |
Extend sed/regex to cover vendor-specific defang styles |
| Duplicate IOCs across reports | Multiple sources, same campaign | Merge with dedup script before MISP import |
Decision Gate
IF source is a PDF/DOCX threat report
→ Extract text first (pdftotext / python-docx), then run ioc-finder
IF source is a raw paste / email dump
→ Refang first (ioc_fanger), then extract
IF > 100 hashes extracted
→ Use bulk lookup (MalwareBazaar batch API) instead of per-hash VT queries
IF IOCs need sharing with external partners
→ Format as STIX 2.1 bundle with TLP marking
IF IOCs are for internal blocklist only
→ Export as flat CSV: type,value,first_seen,source
IF ioc-finder unavailable
→ Fall back to grep-based regex extraction (Section 1)
04 同仓库更多 Skills
wps-pixie-dust
WPS Pixie-Dust offline nonce attack (reaver -K / pixiewps) and fallback online PIN brute (bully) to recover the AP's WPA PSK without capturing a handshake.
wpa3-sae
WPA3-SAE transition-mode downgrade (DragonShift), SSID Confusion CVE-2023-52424, Dragonblood side-channels, and SAE captive-portal credential recovery against WPA3-Personal networks.
wpa2-psk
WPA/WPA2-PSK handshake capture via targeted deauth + PMKID (no deauth required) + offline hashcat cracking. The most common consumer/SMB encryption mode in 2026.
wpa-enterprise-eap
WPA/WPA2/WPA3-Enterprise (802.1X/EAP) rogue-RADIUS evil-twin for MSCHAPv2 capture, GTC downgrade, and PEAP relay. MSCHAPv2 capture equals a NetNTLM hash — the primary wireless on-ramp to Active Directory.
krack-fragattacks
KRACK key-reinstallation (CVE-2017-13077..13082) and FragAttacks fragmentation/aggregation flaws (CVE-2020-24586..24588, CVE-2020-26139..26147) against legacy or embedded 802.11 supplicants with poor patch cadence.
evil-twin-karma
Evil-twin rogue AP with KARMA/Mana PNL-probe response, captive-portal credential capture, and post-association MITM for PSK/open networks. Distinct from wpa-enterprise-eap which targets 802.1X.