firehose-delivery-stream-deployer
Provisions Kinesis Data Firehose delivery streams with production defaults: source selection (Direct PUT, Kinesis Data Stream, Amazon MSK, CloudWatch Logs), Lambda transformation for ETL, format conversion (Parquet/ORC/JSON via dynamic schema), S3 destination (prefix patterns, error prefix, buffering hints), OpenSearch destination, HTTP endpoint destination (Splunk/custom), Redshift destination (COPY via S3 staging), backup configuration, retry and duration settings, CloudWatch logging, and server-side encryption (KMS). Emits a READY_TO_DEPLOY checklist with verification commands. Use when creating a Firehose delivery stream, configuring S3/OpenSearch/Redshift/HTTP destinations, setting up Lambda transformation, or enabling Parquet format conversion. Triggers: create firehose delivery stream, firehose s3 destination, firehose lambda transformation, firehose parquet conversion, firehose opensearch destination, firehose redshift destination, firehose splunk http endpoint, firehose buffering hints.
- License 许可
- Apache-2.0
- 最近更新
- 2026-09-24
- Stars
- ★ 4
- Forks
- 0
01 前言 Front Matter
- name
- firehose-delivery-stream-deployer
- license
- Apache-2.0
- metadata
- { "tags": "aws, kinesis, firehose, delivery-stream, cloudops, deploy, analytics, provisioning, s3-destination, lambda-transformation, parquet, kms-encryption", "phase": "1", "author": "Jacky Chan — AWS Community Builder", "domain": "aws-cloudops", "family": "Analytics", "version": "0.1.0", "keywords": "aws, kinesis, firehose, delivery stream, cloudops, deploy, provisioning, s3 destination, opensearch, redshift, lambda transformation, parquet, format conversion, buffering hints, kms encryption", "task_type": "deploy", "complexity": "high", "entry_point": "false", "skill_class": "capability", "when_to_use": "Invoke when the user wants to create a Kinesis Data Firehose delivery stream, configure S3/OpenSearch/Redshift/HTTP endpoint destinations, set up Lambda transformation for ETL, enable Parquet or ORC format conversion, configure buffering hints, or set up KMS encryption. Do NOT invoke for Kinesis Data Streams (use kinesis-stream-deployer), Kinesis Data Analytics (use kinesis-analytics-deployer), or Firehose auditing/troubleshooting.", "dependencies": "aws-orchestrator", "requires_llm": "true", "verdict_shape": "READY_TO_DEPLOY | PREREQUISITES_MISSING", "lifecycle_status": "active", "supports_pipeline": "true" }
- description
- Provisions Kinesis Data Firehose delivery streams with production defaults: source selection (Direct PUT, Kinesis Data Stream, Amazon MSK, CloudWatch Logs), Lambda transformation for ETL, format conversion (Parquet/ORC/JSON via dynamic schema), S3 destination (prefix patterns, error prefix, buffering hints), OpenSearch destination, HTTP endpoint destination (Splunk/custom), Redshift destination (COPY via S3 staging), backup configuration, retry and duration settings, CloudWatch logging, and server-side encryption (KMS). Emits a READY_TO_DEPLOY checklist with verification commands. Use when creating a Firehose delivery stream, configuring S3/OpenSearch/Redshift/HTTP destinations, setting up Lambda transformation, or enabling Parquet format conversion. Triggers: create firehose delivery stream, firehose s3 destination, firehose lambda transformation, firehose parquet conversion, firehose opensearch destination, firehose redshift destination, firehose splunk http endpoint, firehose buffering hints.
- compatibility
- Agent runtime that reads SKILL.md (Claude Code, Cursor, Windsurf, Codex, Gemini). For live deployment: AWS CLI v2 with firehose, s3, lambda, iam, and kms access. Works with Terraform aws_kinesis_firehose_delivery_stream resources and CloudFormation AWS::KinesisFirehose::DeliveryStream templates.
02 资源文件 19
- 参考文档eval/test-cases.yaml6.4 KB
- 参考文档evals/baselines/buffering-hints-tradeoff.md736 B
- 参考文档evals/baselines/missing-s3-bucket.md663 B
- 参考文档evals/baselines/opensearch-destination-retry-backup.md649 B
- 参考文档evals/baselines/s3-parquet-lambda-transform.md691 B
- 参考文档evals/baselines/splunk-http-endpoint.md637 B
- 参考文档evals/eval_config.yaml1.8 KB
- 参考文档evals/evals.json9.8 KB
- 参考文档evals/prompts/buffering-hints-tradeoff.md688 B
- 参考文档evals/prompts/missing-s3-bucket.md414 B
- 参考文档evals/prompts/opensearch-destination-retry-backup.md745 B
- 参考文档evals/prompts/s3-parquet-lambda-transform.md987 B
- 参考文档evals/prompts/splunk-http-endpoint.md565 B
- 参考文档examples/README.md7.8 KB
- 参考文档references/advanced-patterns.md5.0 KB
- 参考文档references/destinations-and-buffering.md11.0 KB
- 参考文档references/error-handling.md1.1 KB
- 参考文档references/transformation-and-encryption.md10.2 KB
- 参考文档skill.usf.yaml4.3 KB
03 SKILL.md 正文
Firehose Delivery Stream Deployer
An AWS CloudOps agent skill that provisions Kinesis Data Firehose delivery streams with correct defaults. The skill walks the operator through source selection (Direct PUT, Kinesis Data Stream, MSK, CloudWatch Logs), processing/transformation (Lambda function for ETL), format conversion (Parquet/ORC/JSON via dynamic schema), destination selection (S3, OpenSearch, HTTP endpoint, Redshift), backup configuration, buffering hints, retry/duration settings, CloudWatch logging, and server-side encryption (KMS), captures data pipeline decisions, explains why each default matters, and emits a READY_TO_DEPLOY checklist with copy-pasteable verification commands.
Activation keywords
create Firehose delivery stream, Firehose S3 destination, Firehose Lambda transformation, Firehose Parquet conversion, Firehose OpenSearch destination, Firehose Redshift destination, Firehose Splunk HTTP endpoint, Firehose buffering hints.
STRICT output contract
When this skill is invoked with a Firehose-provisioning request
(create a delivery stream, configure a destination, set up Lambda
transformation, enable format conversion, or a partial configuration),
the agent MUST respond with the READY_TO_DEPLOY checklist defined in
the "Output format" section using the literal all-caps labels
FIREHOSE:, VERDICT:, CHECKLIST:, and VERIFICATION_COMMANDS:.
Do NOT preface the checklist with prose, headings, or disclaimers —
emit the block as the first lines of the response. This contract is
what assertion-based evals and downstream provisioning pipelines rely
on; deviating from the literal labels breaks automation silently.
If any prerequisite is missing, the verdict is PREREQUISITES_MISSING
with a specific gap citation in the checklist (marked [✗]), and
READY_TO_DEPLOY MUST NOT also appear.
Quick navigation
| Section | When to read |
|---|---|
| Prerequisites | Always — verify before provisioning |
| Step 1 — Source selection | Direct PUT vs Kinesis Stream vs MSK vs CW Logs |
| Step 2 — Lambda transformation | ETL processing |
| Step 3 — Format conversion | Parquet/ORC/JSON via dynamic schema |
| Step 4 — S3 destination | Prefix patterns, error prefix, buffering |
| Step 5 — OpenSearch destination | Index management, retry |
| Step 6 — HTTP endpoint destination | Splunk/custom endpoints |
| Step 7 — Redshift destination | COPY via S3 staging |
| Step 8 — Backup configuration | Source record backup |
| Step 9 — Buffering hints | Latency vs cost trade-off |
| Step 10 — Retry and duration | Error handling |
| Step 11 — CloudWatch logging | Error logging |
| Step 12 — Server-side encryption (KMS) | Encryption at rest |
| Step 13 — Recent features | Latest |
| NEVER do these things | Review before signing off |
| Output format | The literal checklist template |
| references/destinations-and-buffering.md | Destination + buffering detail |
| references/transformation-and-encryption.md | Lambda + KMS detail |
Mindset
One-line takeaway: A Firehose delivery stream is a fully managed ETL pipeline that ingests data from a source, optionally transforms it via Lambda, optionally converts the format (Parquet/ORC), and delivers it to a destination (S3, OpenSearch, HTTP endpoint, Redshift). Buffering hints control the latency-vs-cost trade-off. The source, transformation, and destination must all be configured correctly for data to flow end-to-end.
Three misconceptions dominate Firehose misdesign at provisioning time:
-
"Buffering hints don't matter." They do. Buffering hints (SizeInMBs and IntervalInSeconds) control when Firehose flushes buffered data to the destination. Too low = frequent small writes (high cost from many small S3 objects, high OpenSearch indexing overhead). Too high = high latency. A baseline model often leaves defaults without considering the trade-off.
-
"Lambda transformation and format conversion are interchangeable." They are NOT. Lambda transformation runs custom code on each batch of records (any transformation logic). Format conversion is a built-in feature that converts JSON to Parquet or ORC using a Glue table schema — no code needed. They CAN be used together (Lambda transforms, then format conversion serializes to Parquet), but the data flow order matters.
-
"S3 prefix patterns are cosmetic." They are not. S3 prefix patterns (Hive-style
year=!{timestamp:yyyy}/month=!{timestamp:MM}/) determine how data is partitioned in S3. Partitioning affects Athena query performance (partition pruning), Glue crawler efficiency, and cost (scanning less data). A flat prefix means every query scans all data.
Configuration dependency graph (novel heuristic)
Firehose configurations are NOT independent. The source must be configured before the stream can receive data. Lambda transformation must reference an existing Lambda function. Format conversion requires a Glue table schema. The destination type determines backup and retry behavior. Use this graph to sequence provisioning.
| Configuration | Hard dependencies (API error without) | Silent failure / immutability | Enables downstream |
|---|---|---|---|
| Delivery stream | source config valid; destination config valid | stream is CREATING then ACTIVE; cannot deliver until ACTIVE | the stream ARN for producers |
| Source (Direct PUT) | none | producers use PutRecord/PutRecordBatch API | data ingestion |
| Source (Kinesis Stream) | Kinesis Data Stream exists | stream must be in same region; Firehose reads via enhanced fan-out or polling | stream-to-firehose pipe |
| Source (MSK) | MSK cluster exists; topic identified | Firehose reads from MSK topic directly | MSK-to-firehose pipe |
| Source (CloudWatch Logs) | CW Logs subscription filter | subscription filter sends to Firehose | log-to-firehose pipe |
| Lambda transformation | Lambda function exists; IAM role permits invoke | Lambda receives records, returns transformed + failed records | custom ETL processing |
| Format conversion (Parquet/ORC) | Glue database + table with schema; IAM role permits Glue | conversion uses the Glue table schema; wrong schema = malformed output | columnar format for analytics |
| S3 destination | S3 bucket exists; IAM role permits S3 write | prefix pattern determines partitioning; error prefix catches failed records | S3 data lake delivery |
| OpenSearch destination | OpenSearch domain exists; IAM role permits write | index name must be valid; retry buffer holds failed writes | OpenSearch indexing |
| HTTP endpoint destination | endpoint URL accessible; API key/auth configured | Firehose POSTs batches; endpoint must ACK within timeout | Splunk / custom delivery |
| Redshift destination | Redshift cluster exists; S3 staging bucket; COPY command | Firehose stages to S3 then issues COPY; username/password or IAM auth | Redshift loading |
| Backup configuration | S3 bucket for backup | source record backup (pre-transformation); failed record backup | data recovery |
| KMS encryption | KMS key exists; IAM role permits kms:Decrypt | encryption at rest for buffered data and S3 delivery | encryption compliance |
| CloudWatch logging | CW log group exists; IAM role permits logs | error logging enabled per destination type | operational visibility |
The buffering-hints + format-conversion row is the one a baseline model misses. Buffering hints interact with format conversion: if buffering is too small, the Parquet writer produces many tiny files (row group overhead dominates). If buffering is too large, latency increases. The correct tuning depends on the format and downstream query patterns.
Cross-dependency gotchas:
- Lambda transformation runs BEFORE format conversion. If Lambda returns malformed JSON, format conversion to Parquet fails silently (records go to the error prefix).
- The IAM role for the delivery stream must trust firehose.amazonaws.com and have permissions for ALL configured destinations, Lambda, Glue, and KMS as applicable.
- S3 prefix patterns use Firehose-specific expression syntax
(
!{timestamp:yyyy},!{partitionKeyFromQuery:...}). Using the wrong syntax produces literal folder names instead of dynamic partitions.
Expert heuristic: buffering hints trade-off (latency vs cost)
Buffering-hints latency-vs-cost model and per-destination recommendations moved to references. → references/advanced-patterns.md § Expert heuristic: buffering hints trade-off
Expert heuristic: Lambda transformation blueprints
Transformation flow, blueprint catalog, and Parquet JSON-validity rules moved to references. → references/advanced-patterns.md § Expert heuristic: Lambda transformation blueprints
Expert heuristic: S3 partition projection via Hive-style prefix
Flat vs Hive-style prefix comparison and error-prefix patterns moved to references. → references/advanced-patterns.md § Expert heuristic: S3 partition projection
Prerequisites (verify before provisioning)
| Prerequisite | Why it matters | How to verify |
|---|---|---|
| S3 bucket exists (S3/Redshift destination) | Firehose delivers to S3; Redshift stages via S3 | aws s3api head-bucket --bucket <name> |
| OpenSearch domain exists (OpenSearch destination) | Firehose writes to an OpenSearch index | aws opensearch describe-domain --domain-name <name> |
| Redshift cluster exists (Redshift destination) | Firehose issues COPY from S3 staging | aws redshift describe-clusters |
| Lambda function exists (if transformation) | Firehose invokes Lambda for ETL | aws lambda get-function --function-name <name> |
| Glue database + table (if format conversion) | Parquet/ORC conversion uses Glue schema | aws glue get-table --database-name <db> --name <table> |
| KMS key exists (if SSE enabled) | Encryption at rest for buffered data | aws kms describe-key --key-id <key-id> |
| IAM role with firehose trust policy | Firehose assumes the role for all service access | Verify trust policy and permissions |
| CloudWatch log group (if logging enabled) | Error logging destination | aws logs describe-log-groups |
If any prerequisite is missing, output VERDICT: PREREQUISITES_MISSING
and cite the specific gap.
Step 1 — Source selection
| Source type | How it works | When to use |
|---|---|---|
| Direct PUT | Producers call PutRecord/PutRecordBatch API | Custom producers, API Gateway, CloudWatch Logs subscription, IoT |
| Kinesis Data Stream | Firehose reads from a KDS stream | Existing KDS pipeline needing delivery |
| Amazon MSK | Firehose reads from MSK topic directly | Kafka-based streaming, no consumer management |
| CloudWatch Logs | Subscription filter sends logs to Firehose | Log processing, SIEM ingestion |
Direct PUT (most common): no source configuration needed beyond the type. Producers use the Firehose API.
Kinesis Data Stream: specify the stream ARN. Firehose can use enhanced fan-out (recommended for high throughput) or polling.
Step 2 — Lambda transformation
Lambda transformation runs custom ETL on each batch. Configure the Lambda function ARN and a buffer size (1-128 MB) that controls batch size.
# In the Firehose config:
ProcessingConfiguration:
Enabled: true
Processors:
- Type: Lambda
Parameters:
- ParameterName: LambdaArn
ParameterValue: arn:aws:lambda:us-east-1:123456789012:function:firehose-transform
- ParameterName: BufferSizeInMBs
ParameterValue: "3"
- ParameterName: RoleArn
ParameterValue: arn:aws:iam::123456789012:role/FirehoseStreamRole
Lambda output contract: the function must return a JSON object with
records (each with recordId, result: Ok/Dropped, and data as
base64). Records marked Dropped are silently discarded.
Key implication: if using format conversion to Parquet, the Lambda output MUST be valid JSON matching the Glue table schema. Malformed JSON causes conversion failure.
Step 3 — Format conversion (Parquet/ORC/JSON)
Format conversion uses a Glue table schema to serialize records to columnar formats (Parquet, ORC) — no code needed.
# In the S3 destination config:
DataFormatConversionConfiguration:
Enabled: true
InputFormatConfiguration:
Deserializer:
OpenXJsonSerDe: {}
OutputFormatConfiguration:
Serializer:
ParquetSerDe: {} # or OrcSerDe
SchemaConfiguration:
DatabaseName: my_glue_db
TableName: my_glue_table
RoleArn: arn:aws:iam::123456789012:role/FirehoseStreamRole
Region: us-east-1
Requirements:
- Glue database and table must exist with the correct schema.
- The IAM role must have
glue:GetTablepermissions. - Input records must be valid JSON (if using Lambda transformation, ensure the output is valid JSON).
- Buffering hints should be larger for Parquet (64-128 MB) to produce optimal row group sizes.
Step 4 — S3 destination
The most common destination. Configure bucket ARN, prefix, error prefix, buffering hints, compression, and encryption.
S3DestinationConfiguration:
RoleARN: arn:aws:iam::123456789012:role/FirehoseStreamRole
BucketARN: arn:aws:s3:::my-data-lake
Prefix: "data/year=!{timestamp:yyyy}/month=!{timestamp:MM}/day=!{timestamp:dd}/"
ErrorOutputPrefix: "errors/!{firehose:error-output-type}/year=!{timestamp:yyyy}/month=!{timestamp:MM}/day=!{timestamp:dd}/"
BufferingHints:
SizeInMBs: 64
IntervalInSeconds: 300
CompressionFormat: UNCOMPRESSED # SNAPPY/GZIP/ZIP for non-Parquet
EncryptionConfiguration:
KMSEncryptionConfig:
AWSKMSKeyARN: arn:aws:kms:us-east-1:123456789012:key/abc123
Prefix patterns: use Hive-style partitioning for Athena
compatibility. Use !{timestamp:yyyy}, !{timestamp:MM},
!{timestamp:dd}, !{timestamp:HH} for time-based partitioning.
Compression: for Parquet format conversion, leave compression as UNCOMPRESSED (Parquet has built-in compression). For JSON, use GZIP or SNAPPY.
Step 5 — OpenSearch destination
Full OpenSearch destination configuration (domain ARN, index rotation, retry, S3 backup) moved to references. → references/destinations-and-buffering.md § OpenSearch destination
Step 6 — HTTP endpoint destination (Splunk/custom)
Full HTTP endpoint configuration (Splunk HEC URL, access key, buffering, retry) moved to references. → references/destinations-and-buffering.md § HTTP endpoint destination
Step 7 — Redshift destination
Full Redshift destination configuration (JDBC URL, COPY command, S3 staging) moved to references. → references/destinations-and-buffering.md § Redshift destination
Step 8 — Backup configuration
Firehose can back up source records (pre-transformation) or failed records (post-transformation delivery failure).
| Backup mode | What it backs up | Destination |
|---|---|---|
SourceRecord (S3 dest) |
All source records before transformation | S3 backup bucket |
FailedDocumentsOnly (OpenSearch/HTTP dest) |
Records that failed delivery | S3 backup bucket |
FailedDataOnly (Redshift dest) |
Records that failed COPY | S3 backup bucket |
Recommendation: always enable backup for production streams. The backup bucket should be separate from the main delivery bucket.
Step 9 — Buffering hints (latency vs cost trade-off)
| Destination | Recommended SizeInMBs | Recommended IntervalInSeconds | Rationale |
|---|---|---|---|
| S3 (JSON) | 5-15 MB | 300s | Balance object size vs latency |
| S3 (Parquet) | 64-128 MB | 300-900s | Larger row groups = better compression + Athena performance |
| OpenSearch | 5 MB | 60-300s | Smaller batches reduce indexing overhead |
| HTTP endpoint | 5 MB | 60s | Low latency for downstream SIEM |
| Redshift | 64-128 MB | 300-900s | Fewer, larger COPY operations |
Too-small buffering for Parquet = many tiny files. Each tiny Parquet file has row group overhead. Athena scanning many tiny files is slow and expensive. Aim for 64+ MB per Parquet file.
Step 10 — Retry and duration settings
| Setting | Default | Range | Effect |
|---|---|---|---|
RetryOptions.DurationInSeconds |
300 | 0-7200 | How long Firehose retries failed writes before giving up |
| S3 backpressure | automatic | N/A | Firehose handles S3 throttling internally |
After the retry duration expires, failed records go to the backup/error S3 prefix.
Step 11 — CloudWatch logging
Firehose logs delivery errors to CloudWatch Logs. Each destination type has its own log stream prefix.
# Enable logging in the delivery stream config
--extended-config (via CloudWatch logging)
Enable CloudWatch logging to capture format conversion errors, Lambda transformation errors, and destination delivery failures.
Step 12 — Server-side encryption (KMS)
Firehose supports KMS encryption for data at rest (in the buffer before delivery) and for S3 delivery.
# In the S3 destination config:
EncryptionConfiguration:
KMSEncryptionConfig:
AWSKMSKeyARN: arn:aws:kms:us-east-1:123456789012:key/abc123
# Or use NoEncryptionConfig for unencrypted (not recommended)
The IAM role must have kms:Decrypt and kms:GenerateDataKey permissions for the specified KMS key.
Step 13 — Recent features
2023-2025 Firehose feature notes (MSK source, CW Logs subscription, partition projection, enhanced fan-out) moved to references. → references/advanced-patterns.md § Step 13 — Recent features
NEVER do these things
-
NEVER use default buffering hints for Parquet destinations. The default 5 MB produces many tiny Parquet files with row group overhead. Use 64-128 MB for Parquet to produce optimal file sizes for Athena.
-
NEVER use a flat S3 prefix for analytics destinations. A flat prefix (
data/) means Athena scans all files. Use Hive-style partitioned prefixes (data/year=!{timestamp:yyyy}/month=.../) for partition pruning. -
NEVER assume Lambda transformation and format conversion are interchangeable. Lambda runs custom code per batch; format conversion uses a Glue schema. They can be used together (Lambda transforms, then conversion serializes to Parquet), but the Lambda output must be valid JSON matching the Glue schema.
-
NEVER deploy without a backup configuration in production. Without backup, failed records are lost. Configure
FailedDocumentsOnlybackup at minimum. -
NEVER use the same S3 bucket for main delivery and backup. Failed records should go to a separate bucket or prefix for easier reprocessing. Mixing them complicates downstream ETL.
-
NEVER leave encryption disabled for sensitive data. Configure KMS encryption for both the Firehose buffer and S3 delivery. The IAM role must have kms:Decrypt and kms:GenerateDataKey permissions.
-
NEVER forget the IAM role trust policy. The delivery stream IAM role must trust
firehose.amazonaws.comand have permissions for ALL configured destinations, Lambda (if transformation), Glue (if format conversion), and KMS (if encryption). -
NEVER use GZIP compression with Parquet format conversion. Parquet has built-in compression. Set CompressionFormat to UNCOMPRESSED when using format conversion. GZIP is for JSON/CSV.
-
NEVER assume CloudWatch Logs subscription works without the correct IAM role. The subscription filter needs a role that permits
firehose:PutRecordBatch. Verify the trust policy and permissions. -
NEVER set retry duration too low for flaky destinations. The default 300s may be too short for OpenSearch under load. Increase
DurationInSeconds(up to 7200s) for destinations with transient failures.
Output format
FIREHOSE: <stream-name> (<source> → <destination>)
VERDICT: READY_TO_DEPLOY | PREREQUISITES_MISSING
CHECKLIST:
[✓|✗] Source: Direct PUT | Kinesis Stream (<stream-name>) | MSK (<cluster>/<topic>) | CloudWatch Logs
[✓|✗] Lambda transformation: enabled (<function-arn>, buffer <N>MB) | disabled
[✓|✗] Format conversion: Parquet (Glue <db>/<table>) | ORC | JSON (no conversion)
[✓|✗] Destination: S3 (<bucket>, prefix <prefix>) | OpenSearch (<domain>, index <index>) | HTTP (<url>) | Redshift (<cluster>)
[✓|✗] Buffering hints: <size>MB / <interval>s (tuned for <format/destination>)
[✓|✗] S3 prefix: Hive-style partitioned (<pattern>) | flat
[✓|✗] Error prefix: <error-prefix>
[✓|✗] Compression: <format> (UNCOMPRESSED for Parquet)
[✓|✗] Backup: <mode> → <backup-bucket> | disabled
[✓|✗] Retry: <duration>s
[✓|✗] CloudWatch logging: enabled | disabled
[✓|✗] KMS encryption: enabled (<key-arn>) | disabled
[✓|✗] IAM role: <role-arn> (trust: firehose.amazonaws.com)
[✓|✗] Tags: <key=value list>
VERIFICATION_COMMANDS:
aws firehose describe-delivery-stream --delivery-stream-name <name> --region <region>
aws firehose describe-delivery-stream --delivery-stream-name <name> --query 'DeliveryStreamDescription.DeliveryStreamStatus' --region <region>
Worked example — S3 Parquet with Lambda transformation
FIREHOSE: events-to-s3-parquet (Direct PUT → S3)
VERDICT: READY_TO_DEPLOY
CHECKLIST:
[✓] Source: Direct PUT
[✓] Lambda transformation: enabled (arn:aws:lambda:us-east-1:123456789012:function:firehose-transform, buffer 3MB)
[✓] Format conversion: Parquet (Glue analytics/events_table)
[✓] Destination: S3 (arn:aws:s3:::my-data-lake, prefix data/year=!{timestamp:yyyy}/month=!{timestamp:MM}/day=!{timestamp:dd}/)
[✓] Buffering hints: 128MB / 300s (tuned for Parquet — optimal row group size)
[✓] S3 prefix: Hive-style partitioned (year/month/day)
[✓] Error prefix: errors/!{firehose:error-output-type}/year=!{timestamp:yyyy}/month=!{timestamp:MM}/day=!{timestamp:dd}/
[✓] Compression: UNCOMPRESSED (Parquet has built-in compression)
[✓] Backup: FailedDocumentsOnly → arn:aws:s3:::firehose-backup
[✓] Retry: 300s
[✓] CloudWatch logging: enabled
[✓] KMS encryption: enabled (arn:aws:kms:us-east-1:123456789012:key/abc123)
[✓] IAM role: arn:aws:iam::123456789012:role/FirehoseStreamRole (trust: firehose.amazonaws.com)
[✓] Tags: Environment=production, Pipeline=events
VERIFICATION_COMMANDS:
aws firehose describe-delivery-stream --delivery-stream-name events-to-s3-parquet --region us-east-1
aws firehose describe-delivery-stream --delivery-stream-name events-to-s3-parquet --query 'DeliveryStreamDescription.DeliveryStreamStatus' --region us-east-1
Error handling
Failure-mode deep dives (error prefix, tiny S3 files, OpenSearch lag, stuck CREATING, CW Logs subscription) moved to references. → references/error-handling.md § Error handling
References (load on demand)
Consult these only when the corresponding topic comes up:
- references/destinations-and-buffering.md — OpenSearch / HTTP endpoint / Redshift destination configs (moved from Steps 5-7), buffering deep dive
- references/transformation-and-encryption.md — Lambda blueprints, format conversion, KMS encryption, IAM roles
- references/error-handling.md — failure modes moved from § Error handling
- references/advanced-patterns.md — expert heuristics (buffering trade-off, Lambda blueprints, partition projection) and recent AWS features
Domain
AWS CloudOps / Kinesis Data Firehose Delivery Stream Provisioning & Data Pipeline Automation.
AWS documentation
- Firehose Guide — https://docs.aws.amazon.com/firehose/latest/dev/what-is-this-service.html
- Create delivery stream — https://docs.aws.amazon.com/firehose/latest/dev/basic-create.html
- S3 destination — https://docs.aws.amazon.com/firehose/latest/dev/create-destination.html#create-destination-s3
- OpenSearch destination — https://docs.aws.amazon.com/firehose/latest/dev/create-destination.html#create-destination-opensearch
- Lambda transformation — https://docs.aws.amazon.com/firehose/latest/dev/data-transformation.html
- Format conversion — https://docs.aws.amazon.com/firehose/latest/dev/data-conversion.html
- Buffering hints — https://docs.aws.amazon.com/firehose/latest/dev/basic-deliver.html#buffering-hints
- S3 prefix patterns — https://docs.aws.amazon.com/firehose/latest/dev/s3-prefixes.html
- KMS encryption — https://docs.aws.amazon.com/firehose/latest/dev/encryption.html
- HTTP endpoint destination — https://docs.aws.amazon.com/firehose/latest/dev/httpdeliveryrequestresponse.html
04 同仓库更多 Skills
xray-tracing-deployer
Deploys AWS X-Ray distributed tracing with production-grade configuration: X-Ray daemon deployment (EC2 systemd, ECS sidecar, EKS DaemonSet, Lambda built-in), SDK instrumentation (Java, Python, Node.js, Go, .NET) with annotations and metadata, sampling rules (default reservoir + rate vs custom rules with URL / method / service-name predicates), service map generation, trace query and retrieval, groups and insights, CloudWatch ServiceLens integration, IAM permissions (PutTraceSegments, GetSamplingRules), and latest features (Lambda Powertools tracing, OpenTelemetry / ADOT Collector support, W3C trace context). Emits a READY_TO_DEPLOY checklist with every configuration item verified. Use when enabling X-Ray on a new application, adding tracing to ECS / EKS / Lambda, validating a sampling configuration, or generating daemon deployment CLI and IaC templates. Triggers: X-Ray, distributed tracing, X-Ray daemon, sampling rules, service map, X-Ray SDK, OpenTelemetry, ADOT, Lambda Powertools, CloudWatch ServiceLens.
wellarchitected-workload-auditor
Audits AWS Well-Architected Tool workloads for review staleness, per-pillar high-risk issue counts, milestone tracking gaps, and remediation plan completeness. Emits a deterministic verdict (STALE_REVIEW | HIGH_RISK | CONFIG_GAP | OK) per workload with per-pillar risk breakdown, milestone trend analysis, and specific remediation. Use when reviewing Well-Architected workloads, checking WA review freshness, auditing high-risk issues per pillar, validating milestone coverage, or tracking remediation plans.
wellarchitected-review-operator
Operates AWS Well-Architected Tool reviews end-to-end — creates workloads, runs pillar reviews (operational excellence, security, reliability, performance, cost optimization, sustainability, Prosperity), answers questions with risk-tier rationale, generates reports and improvement plans, creates milestones, and integrates with Trusted Advisor and Well-Architected Labs. Runs deterministic pre-checks (workload existence, lens availability, pillar coverage), emits the exact update-answer / get-consolidated-report CLI behind a CONFIRM gate, and verifies risk-tier transitions. Emits READY | BLOCKED | COMPLETED. Use when creating a workload, answering pillar questions, generating an improvement plan, creating a milestone, or integrating with Trusted Advisor. Triggers: well-architected tool, well architected review, workload review, pillar, operational excellence, security, reliability, performance, cost optimization, sustainability, prosperity, improvement plan, milestone, trusted advisor.
wafv2-web-acl-deployer
Provisions AWS WAFv2 Web ACLs with correct production defaults: scope selection (CLOUDFRONT vs REGIONAL), managed rule groups (AWSManagedRulesCommonRuleSet, SQLiRuleSet, LinuxRuleSet, WindowsRuleSet, IPReputationList, BotControlRuleSet, AmazonIpReputationList, ATP), custom rules (byte match, IP set, regex pattern, geo match, size constraint, rate-based), rule priority ordering, logging destinations (Kinesis Firehose, CloudWatch Logs, S3), resource association (ALB, API Gateway, CloudFront), rate-based rules with aggregate keys (IP, forwarded IP), Challenge and CAPTCHA actions, and account takeover prevention. Emits a READY_TO_DEPLOY checklist. Use when creating a Web ACL, attaching managed rules, configuring rate limiting, enabling WAF logging, associating a Web ACL with an ALB/API Gateway/CloudFront, or configuring CAPTCHA/Challenge actions. Triggers: create WAF Web ACL, WAFv2, managed rule groups, WAF rate limiting, WAF logging, WAF CAPTCHA, WAF Challenge action, bot control, ATP managed rules.
wafv2-web-acl-auditor
Audits AWS WAFv2 Web ACL configurations to determine whether the ACL provides effective protection: default-action posture (Allow vs Block), managed-rule-group coverage gaps, rule effectiveness (BLOCK vs COUNT, shadow/bypass rules, stale exclusions), rate-based rule correctness, logging and visibility configuration, and text-transformation bypass vectors. Emits a deterministic verdict per Web ACL. Use when reviewing a WAFv2 Web ACL for security posture, checking managed-rule coverage, validating rate-limit configuration, auditing WAF logging, or hardening a WAF before production deployment. Triggers: WAF, WAFv2, Web ACL, managed rule group, AWSManagedRules, rate-based rule, CAPTCHA, Challenge, OverrideAction, Count mode, forwarded IP, Bot Control, ATP, text transformation, WAF logging, WAF visibility, OWASP, SQLi, XSS, RFI.
waf-rule-deployer
Provisions AWS WAFv2 rule sets with production defaults: Web ACL creation (CloudFront scope in us-east-1 vs Regional scope), rule groups (managed vs custom) with capacity planning in WCU (Web ACL Capacity Units, 1500 default per ACL), managed rule groups (AWSManagedRulesCommonRuleSet, AWSManagedRulesSQLiRuleSet, AWSManagedRulesAmazonIpReputationList, AWSManagedRulesLinuxRuleSet, AWSManagedRulesWindowsRuleSet, AWSManagedRulesUnixRuleSet), custom rules (byte-match, regex pattern set, geo-match, rate-based), IP set and regex pattern set resources, rule priority ordering (lowest number evaluated first), action types (allow, block, count, CAPTCHA, Challenge), bot control managed rule group, account takeover. Triggers: create WAF rule, WAF managed rule group, WAF custom rule, WAF byte match, WAF geo match, WAF rate based rule, WAF IP set, WAF regex pattern set, WAF WCU budget, WAF CloudFront ACL, WAF logging Kinesis Firehose, WAF bot control, WAF ATP, WAF label matching, WAF Challenge action, WAF CAPTCHA.