performing-log-source-onboarding-in-siem
Perform structured log source onboarding into SIEM platforms (Splunk, Elastic, Sentinel, QRadar, or similar) by prioritizing sources with a tiered value framework, configuring collectors, building parsers, normalizing fields to a common schema (e.g. CIM), and validating data quality. Use when planning or executing the integration of new log sources — such as Active Directory, firewalls, EDR, VPN, DNS, or cloud audit logs — into a SIEM for security monitoring and detection.
- License 许可
- Apache-2.0
- 最近更新
- 2026-09-11
- Stars
- ★ 0
- Forks
- 0
01 前言 Front Matter
- name
- performing-log-source-onboarding-in-siem
- tags
- [ "siem", "log-onboarding", "log-management", "data-ingestion", "parsing", "normalization", "soc" ]
- author
- mahipal
- domain
- cybersecurity
- license
- Apache-2.0
- version
- 1.0
- nist_csf
- [ "DE.CM-01", "DE.AE-02", "RS.MA-01", "DE.AE-06" ]
- subdomain
- soc-operations
- description
- Perform structured log source onboarding into SIEM platforms (Splunk, Elastic, Sentinel, QRadar, or similar) by prioritizing sources with a tiered value framework, configuring collectors, building parsers, normalizing fields to a common schema (e.g. CIM), and validating data quality. Use when planning or executing the integration of new log sources — such as Active Directory, firewalls, EDR, VPN, DNS, or cloud audit logs — into a SIEM for security monitoring and detection.
- mitre_attack
- [ "T1078", "T1685.002", "T1685.005", "T1566" ]
02 资源文件 7
- 资产文件assets/template.md1.1 KB
- 许可证LICENSE11.0 KB
- 参考文档references/api-reference.md1.6 KB
- 参考文档references/standards.md1.2 KB
- 参考文档references/workflows.md1.3 KB
- 脚本scripts/agent.py7.6 KB
- 脚本scripts/process.py7.9 KB
03 SKILL.md 正文
Performing Log Source Onboarding in SIEM
Overview
Log source onboarding is the systematic process of integrating new data sources into a SIEM platform to enable security monitoring and detection. Proper onboarding requires planning data sources, configuring collection agents, building parsers, normalizing fields to a common schema, and validating data quality. According to the UK NCSC, onboarding should prioritize log sources that provide the highest security value relative to their ingestion cost.
When to Use
- When conducting security assessments that involve performing log source onboarding in siem
- When following incident response procedures for related security events
- When performing scheduled security testing or auditing activities
- When validating security controls through hands-on testing
Prerequisites
- SIEM platform deployed (Splunk, Elastic, Sentinel, QRadar, or similar)
- Network access from source systems to SIEM collectors
- Administrative access on source systems for agent installation
- Common Information Model (CIM) or equivalent schema documentation
- Change management approval for production system modifications
Log Source Priority Framework
Tier 1 - Critical (Onboard First)
| Source | Log Type | Security Value |
|---|---|---|
| Active Directory | Security Event Logs | Authentication, privilege escalation |
| Firewalls | Traffic logs | Network access, C2 detection |
| EDR/AV | Endpoint alerts | Malware, process execution |
| VPN/Remote Access | Connection logs | Unauthorized access |
| DNS Servers | Query logs | C2 beaconing, data exfiltration |
| Email Gateway | Email security logs | Phishing, BEC |
Tier 2 - High Priority
| Source | Log Type | Security Value |
|---|---|---|
| Web Proxy | HTTP/HTTPS logs | Web-based attacks, data exfiltration |
| Cloud platforms (AWS/Azure/GCP) | Audit logs | Cloud security posture |
| Database servers | Audit/query logs | Data access, SQL injection |
| DHCP/IPAM | Address allocation | Asset tracking |
| File servers | Access logs | Data access monitoring |
Tier 3 - Standard
| Source | Log Type | Security Value |
|---|---|---|
| Application servers | App logs | Application-level attacks |
| Print servers | Print logs | Data loss prevention |
| Badge/physical access | Access logs | Physical security correlation |
| Network devices (switches/routers) | Syslog | Network anomalies |
Onboarding Process
Step 1: Discovery and Assessment
1. Identify the log source:
- System type and version
- Log format (syslog, CEF, JSON, Windows Events, etc.)
- Log volume estimate (EPS - events per second)
- Network location and firewall requirements
2. Assess security value:
- What threats can this source help detect?
- Which MITRE ATT&CK techniques does it cover?
- Is there an existing SIEM parser?
3. Estimate ingestion cost:
- Daily volume in GB
- License impact (per-GB or per-EPS pricing)
- Storage retention requirements
Step 2: Configure Log Collection
Syslog-Based Collection (Firewalls, Network Devices)
# rsyslog configuration for receiving syslog
# /etc/rsyslog.d/10-siem-collection.conf
# UDP reception
module(load="imudp")
input(type="imudp" port="514" ruleset="siem_forwarding")
# TCP reception
module(load="imtcp")
input(type="imtcp" port="514" ruleset="siem_forwarding")
# TLS reception
module(load="imtcp" StreamDriver.AuthMode="x509/name"
StreamDriver.Mode="1" StreamDriver.Name="gtls")
input(type="imtcp" port="6514" ruleset="siem_forwarding")
ruleset(name="siem_forwarding") {
# Forward to SIEM
action(type="omfwd" target="siem.company.com" port="9514"
protocol="tcp" queue.type="LinkedList"
queue.filename="siem_fwd" queue.maxdiskspace="1g"
queue.saveonshutdown="on" action.resumeRetryCount="-1")
}
Windows Event Log Collection (Splunk Universal Forwarder)
# inputs.conf on Splunk Universal Forwarder
[WinEventLog://Security]
disabled = 0
index = wineventlog
sourcetype = WinEventLog:Security
evt_resolve_ad_obj = 1
checkpointInterval = 5
[WinEventLog://System]
disabled = 0
index = wineventlog
sourcetype = WinEventLog:System
[WinEventLog://Microsoft-Windows-Sysmon/Operational]
disabled = 0
index = wineventlog
sourcetype = XmlWinEventLog:Microsoft-Windows-Sysmon/Operational
renderXml = true
[WinEventLog://Microsoft-Windows-PowerShell/Operational]
disabled = 0
index = wineventlog
sourcetype = XmlWinEventLog:Microsoft-Windows-PowerShell/Operational
Cloud Log Collection (AWS CloudTrail)
{
"AWSTemplateFormatVersion": "2010-09-09",
"Resources": {
"CloudTrailToSIEM": {
"Type": "AWS::CloudTrail::Trail",
"Properties": {
"TrailName": "siem-cloudtrail",
"S3BucketName": "company-cloudtrail-logs",
"IsLogging": true,
"IsMultiRegionTrail": true,
"IncludeGlobalServiceEvents": true,
"EnableLogFileValidation": true,
"EventSelectors": [
{
"ReadWriteType": "All",
"IncludeManagementEvents": true,
"DataResources": [
{
"Type": "AWS::S3::Object",
"Values": ["arn:aws:s3"]
}
]
}
]
}
}
}
}
Step 3: Parse and Normalize
Custom Parser Example (Splunk props.conf/transforms.conf)
# props.conf
[custom:firewall:logs]
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)
TIME_PREFIX = ^
TIME_FORMAT = %Y-%m-%dT%H:%M:%S%z
MAX_TIMESTAMP_LOOKAHEAD = 30
TRANSFORMS-firewall = firewall_extract_fields
FIELDALIAS-src = src_addr AS src_ip
FIELDALIAS-dst = dst_addr AS dest_ip
EVAL-action = case(fw_action=="allow", "allowed", fw_action=="deny", "blocked", true(), "unknown")
EVAL-vendor_product = "Custom Firewall"
LOOKUP-geo = geo_ip_lookup ip AS dest_ip OUTPUT country, city, latitude, longitude
# transforms.conf
[firewall_extract_fields]
REGEX = ^(\S+)\s+(\S+)\s+action=(\w+)\s+src=(\S+):(\d+)\s+dst=(\S+):(\d+)\s+proto=(\w+)\s+bytes=(\d+)
FORMAT = timestamp::$1 hostname::$2 fw_action::$3 src_addr::$4 src_port::$5 dst_addr::$6 dst_port::$7 protocol::$8 bytes::$9
CIM Field Mapping
| Raw Field | CIM Field | Data Model |
|---|---|---|
| src_addr | src_ip | Network_Traffic |
| dst_addr | dest_ip | Network_Traffic |
| dst_port | dest_port | Network_Traffic |
| fw_action | action | Network_Traffic |
| bytes_sent + bytes_recv | bytes | Network_Traffic |
| user_name | user | Authentication |
| login_result | action | Authentication |
| process_path | process | Endpoint |
Step 4: Validate Data Quality
# Verify events are arriving
index=new_source earliest=-1h
| stats count by sourcetype, host, source
# Check field extraction quality
index=new_source earliest=-1h
| stats count(src_ip) as has_src count(dest_ip) as has_dest count(action) as has_action count by sourcetype
| eval src_coverage=round(has_src/count*100,1)
| eval dest_coverage=round(has_dest/count*100,1)
| eval action_coverage=round(has_action/count*100,1)
# Verify CIM compliance
| datamodel Network_Traffic search
| search sourcetype=new_sourcetype
| stats count by source, sourcetype
# Check for timestamp parsing issues
index=new_source earliest=-1h
| eval time_diff=abs(_time - _indextime)
| stats avg(time_diff) as avg_lag max(time_diff) as max_lag by host
| where avg_lag > 300
Step 5: Enable Detection Coverage
# Verify existing correlation searches work with new source
index=new_source sourcetype=new_sourcetype
| tstats count from datamodel=Authentication by _time span=1h
| timechart span=1h count
# Create source-specific detection rule
[New Source - Authentication Anomaly]
search = index=new_source sourcetype=new_sourcetype action=failure \
| stats count by src_ip, user \
| where count > 10
Onboarding Checklist
- [ ] Log source assessed and approved
- [ ] Network connectivity verified
- [ ] Collection agent/method configured
- [ ] Log forwarding confirmed
- [ ] Parser/field extraction configured
- [ ] CIM compliance validated
- [ ] Data model acceleration enabled
- [ ] Volume within license budget
- [ ] Retention policy configured
- [ ] Detection rules enabled/created
- [ ] Dashboard updated
- [ ] Documentation completed
- [ ] SOC team notified
References
04 同仓库更多 Skills
runpodctl
Runpod CLI for managing GPU/CPU workloads from the terminal — pods, serverless endpoints, templates, network volumes, Hub deploys, models, SSH, and file transfer (send/receive). Use for terminal/CI/scripting, Hub browse/deploy, SSH setup, `doctor`, or when the Runpod MCP tools are not connected. For structured tool calls in an MCP-enabled session, prefer runpod-mcp.
runpod
Start here for any Runpod task — running GPU/CPU pods, deploying serverless endpoints, templates, network volumes, building images, or understanding how Runpod works. Routes to the right skill (runpod-mcp, runpodctl, flash, companion-clis, runpod-usage, runpod-templates, runpod-migrate) and indexes two dozen live-verified end-to-end examples (golden paths) — use it when the lane is unclear, and for any multi-step or provisioning task even when it is not.
runpod-usage
How Runpod works and how to work it — pods vs serverless, GPU/VRAM selection, storage, building a container, networking, plus the agentic pod development loop (provision → ssh-exec → set up → poll readiness) and on-pod install hygiene (uv/apt). Use to answer "how does X work", "which GPU", "how do I build a container", or "how do I stand up a workload on a pod". Guidance, not a tool — execute with runpodctl, runpod-mcp, or flash.
runpod-templates
Runpod's official prebuilt pod templates (ComfyUI, PyTorch, and others): what each image ships, which ports and paths it uses, how to pick one, how to pin a version, and what is missing on first boot. Use when a task says "run <X> on a pod" and an official template already covers it, instead of building an image — and when a user needs help with or wants to fix something about a Runpod template they are already running (won't boot, can't reach the UI, missing models, wrong CUDA line, broken ComfyUI workflow metadata on imported workflows): start here and route onward. Deploy with runpodctl or runpod-mcp; end-to-end walkthroughs live in the runpod router's golden paths.
runpod-migrate
Migrate a codebase from the Runpod GraphQL API or REST v1 to REST v2 — inventory which parts use which API version, rewrite the call sites, flag breaking changes, and verify. Use when someone asks to move to v2, asks what v2 would change, or asks which Runpod API their code is on. For managing infrastructure rather than migrating code, use runpod-mcp or runpodctl.
runpod-mcp
Manage Runpod infrastructure — pods, serverless endpoints, jobs, templates, network volumes, container-registry auth, GPU/CPU catalog, and billing — via the Runpod MCP server's structured tool calls. Use when the Runpod MCP tools (create-pod, list-endpoints, …) are connected in this session, or to connect them (hosted OAuth or local npx). Prefer this over runpodctl for plain infra CRUD when MCP is available; use runpodctl for the terminal, file transfer, or SSH setup.