[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M7A8FYYG3CCAKF14CYSNSC6M":3,"skill-same:01M7A8FYYG3CCAKF14CYSNSC6M":142},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":19,"linkOnly":20,"markdownContent":21,"markdownBody":22,"frontMatter":23,"refName":42,"commitSha":43,"skillDirectory":44,"forksCount":45,"repositoryHtmlUrl":46,"repositoryHomepageUrl":6,"ownerHtmlUrl":47,"ownerBlogUrl":6,"ownerDescription":48,"updatedAt":49,"install":50,"resources":63},"01M7A8FYYG3CCAKF14CYSNSC6M","firehose-delivery-stream-deployer",null,"Provisions Kinesis Data Firehose delivery streams with production defaults: source selection (Direct PUT, Kinesis Data Stream, Amazon MSK, CloudWatch Logs), Lambda transformation for ETL, format conversion (Parquet\u002FORC\u002FJSON via dynamic schema), S3 destination (prefix patterns, error prefix, buffering hints), OpenSearch destination, HTTP endpoint destination (Splunk\u002Fcustom), Redshift destination (COPY via S3 staging), backup configuration, retry and duration settings, CloudWatch logging, and server-side encryption (KMS). Emits a READY_TO_DEPLOY checklist with verification commands. Use when creating a Firehose delivery stream, configuring S3\u002FOpenSearch\u002FRedshift\u002FHTTP destinations, setting up Lambda transformation, or enabling Parquet format conversion. Triggers: create firehose delivery stream, firehose s3 destination, firehose lambda transformation, firehose parquet conversion, firehose opensearch destination, firehose redshift destination, firehose splunk http endpoint, firehose buffering hints.","jhkchan\u002Faws-agent-skills","jhkchan","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F1083902?v=4","01M7A2SVQYP9GQTPTDXP5JKYQ4","Jacky Chan","01M7A2SVS117YWCTZABB75HZR1",4,"Apache-2.0","https:\u002F\u002Fgithub.com\u002Fjhkchan\u002Faws-agent-skills\u002Fblob\u002Fmain\u002Fskills\u002Ffirehose-delivery-stream-deployer\u002FSKILL.md",[],[],"2026-11-15 14:19:17",false,"---\nname: firehose-delivery-stream-deployer\ndescription: 'Provisions Kinesis Data Firehose delivery streams with production defaults: source selection (Direct PUT, Kinesis Data Stream, Amazon MSK, CloudWatch Logs), Lambda transformation for ETL, format conversion (Parquet\u002FORC\u002FJSON via dynamic schema), S3 destination (prefix patterns, error prefix, buffering hints), OpenSearch destination, HTTP endpoint destination (Splunk\u002Fcustom), Redshift destination (COPY via S3 staging), backup configuration, retry and duration settings, CloudWatch logging, and server-side encryption (KMS). Emits a READY_TO_DEPLOY checklist with verification commands. Use when creating a Firehose delivery stream, configuring S3\u002FOpenSearch\u002FRedshift\u002FHTTP destinations, setting up Lambda transformation, or enabling Parquet format conversion. Triggers: create firehose delivery stream, firehose s3 destination, firehose lambda transformation, firehose parquet conversion, firehose opensearch destination, firehose redshift destination, firehose splunk http endpoint, firehose buffering hints.'\nlicense: Apache-2.0\ncompatibility: 'Agent runtime that reads SKILL.md (Claude Code, Cursor, Windsurf, Codex, Gemini). For live deployment: AWS CLI v2 with firehose, s3, lambda, iam, and kms access. Works with Terraform aws_kinesis_firehose_delivery_stream resources and CloudFormation AWS::KinesisFirehose::DeliveryStream templates.'\nmetadata:\n  domain: aws-cloudops\n  complexity: high\n  requires_llm: 'true'\n  phase: '1'\n  supports_pipeline: 'true'\n  entry_point: 'false'\n  family: Analytics\n  task_type: deploy\n  skill_class: capability\n  lifecycle_status: active\n  verdict_shape: READY_TO_DEPLOY | PREREQUISITES_MISSING\n  version: 0.1.0\n  author: Jacky Chan — AWS Community Builder\n  tags: aws, kinesis, firehose, delivery-stream, cloudops, deploy, analytics, provisioning, s3-destination, lambda-transformation, parquet, kms-encryption\n  dependencies: aws-orchestrator\n  keywords: aws, kinesis, firehose, delivery stream, cloudops, deploy, provisioning, s3 destination, opensearch, redshift, lambda transformation, parquet, format conversion, buffering hints, kms encryption\n  when_to_use: Invoke when the user wants to create a Kinesis Data Firehose delivery stream, configure S3\u002FOpenSearch\u002FRedshift\u002FHTTP endpoint destinations, set up Lambda transformation for ETL, enable Parquet or ORC format conversion, configure buffering hints, or set up KMS encryption. Do NOT invoke for Kinesis Data Streams (use kinesis-stream-deployer), Kinesis Data Analytics (use kinesis-analytics-deployer), or Firehose auditing\u002Ftroubleshooting.\n---\n\n# Firehose Delivery Stream Deployer\n\nAn AWS CloudOps agent skill that provisions Kinesis Data Firehose\ndelivery streams with correct defaults. The skill walks the operator\nthrough source selection (Direct PUT, Kinesis Data Stream, MSK,\nCloudWatch Logs), processing\u002Ftransformation (Lambda function for ETL),\nformat conversion (Parquet\u002FORC\u002FJSON via dynamic schema), destination\nselection (S3, OpenSearch, HTTP endpoint, Redshift), backup\nconfiguration, buffering hints, retry\u002Fduration settings, CloudWatch\nlogging, and server-side encryption (KMS), captures data pipeline\ndecisions, explains why each default matters, and emits a\nREADY_TO_DEPLOY checklist with copy-pasteable verification commands.\n\n## Activation keywords\n\ncreate Firehose delivery stream, Firehose S3 destination, Firehose\nLambda transformation, Firehose Parquet conversion, Firehose\nOpenSearch destination, Firehose Redshift destination, Firehose Splunk\nHTTP endpoint, Firehose buffering hints.\n\n## STRICT output contract\n\nWhen this skill is invoked with a Firehose-provisioning request\n(create a delivery stream, configure a destination, set up Lambda\ntransformation, enable format conversion, or a partial configuration),\nthe agent MUST respond with the READY_TO_DEPLOY checklist defined in\nthe \"Output format\" section using the literal all-caps labels\n`FIREHOSE:`, `VERDICT:`, `CHECKLIST:`, and `VERIFICATION_COMMANDS:`.\nDo NOT preface the checklist with prose, headings, or disclaimers —\nemit the block as the first lines of the response. This contract is\nwhat assertion-based evals and downstream provisioning pipelines rely\non; deviating from the literal labels breaks automation silently.\n\nIf any prerequisite is missing, the verdict is `PREREQUISITES_MISSING`\nwith a specific gap citation in the checklist (marked `[✗]`), and\n`READY_TO_DEPLOY` MUST NOT also appear.\n\n## Quick navigation\n\n| Section | When to read |\n|---|---|\n| Prerequisites | Always — verify before provisioning |\n| Step 1 — Source selection | Direct PUT vs Kinesis Stream vs MSK vs CW Logs |\n| Step 2 — Lambda transformation | ETL processing |\n| Step 3 — Format conversion | Parquet\u002FORC\u002FJSON via dynamic schema |\n| Step 4 — S3 destination | Prefix patterns, error prefix, buffering |\n| Step 5 — OpenSearch destination | Index management, retry |\n| Step 6 — HTTP endpoint destination | Splunk\u002Fcustom endpoints |\n| Step 7 — Redshift destination | COPY via S3 staging |\n| Step 8 — Backup configuration | Source record backup |\n| Step 9 — Buffering hints | Latency vs cost trade-off |\n| Step 10 — Retry and duration | Error handling |\n| Step 11 — CloudWatch logging | Error logging |\n| Step 12 — Server-side encryption (KMS) | Encryption at rest |\n| Step 13 — Recent features | Latest |\n| NEVER do these things | Review before signing off |\n| Output format | The literal checklist template |\n| references\u002Fdestinations-and-buffering.md | Destination + buffering detail |\n| references\u002Ftransformation-and-encryption.md | Lambda + KMS detail |\n\n## Mindset\n\n**One-line takeaway:** A Firehose delivery stream is a fully managed\nETL pipeline that ingests data from a source, optionally transforms it\nvia Lambda, optionally converts the format (Parquet\u002FORC), and delivers\nit to a destination (S3, OpenSearch, HTTP endpoint, Redshift).\nBuffering hints control the latency-vs-cost trade-off. The source,\ntransformation, and destination must all be configured correctly for\ndata to flow end-to-end.\n\nThree misconceptions dominate Firehose misdesign at provisioning time:\n\n- **\"Buffering hints don't matter.\"** They do. Buffering hints\n  (SizeInMBs and IntervalInSeconds) control when Firehose flushes\n  buffered data to the destination. Too low = frequent small writes\n  (high cost from many small S3 objects, high OpenSearch indexing\n  overhead). Too high = high latency. A baseline model often leaves\n  defaults without considering the trade-off.\n\n- **\"Lambda transformation and format conversion are\n  interchangeable.\"** They are NOT. Lambda transformation runs custom\n  code on each batch of records (any transformation logic). Format\n  conversion is a built-in feature that converts JSON to Parquet or\n  ORC using a Glue table schema — no code needed. They CAN be used\n  together (Lambda transforms, then format conversion serializes to\n  Parquet), but the data flow order matters.\n\n- **\"S3 prefix patterns are cosmetic.\"** They are not. S3 prefix\n  patterns (Hive-style `year=!{timestamp:yyyy}\u002Fmonth=!{timestamp:MM}\u002F`)\n  determine how data is partitioned in S3. Partitioning affects Athena\n  query performance (partition pruning), Glue crawler efficiency, and\n  cost (scanning less data). A flat prefix means every query scans all\n  data.\n\n## Configuration dependency graph (novel heuristic)\n\nFirehose configurations are NOT independent. The source must be\nconfigured before the stream can receive data. Lambda transformation\nmust reference an existing Lambda function. Format conversion requires\na Glue table schema. The destination type determines backup and retry\nbehavior. Use this graph to sequence provisioning.\n\n| Configuration | Hard dependencies (API error without) | Silent failure \u002F immutability | Enables downstream |\n|---|---|---|---|\n| Delivery stream | source config valid; destination config valid | stream is CREATING then ACTIVE; cannot deliver until ACTIVE | the stream ARN for producers |\n| Source (Direct PUT) | none | producers use PutRecord\u002FPutRecordBatch API | data ingestion |\n| Source (Kinesis Stream) | Kinesis Data Stream exists | stream must be in same region; Firehose reads via enhanced fan-out or polling | stream-to-firehose pipe |\n| Source (MSK) | MSK cluster exists; topic identified | Firehose reads from MSK topic directly | MSK-to-firehose pipe |\n| Source (CloudWatch Logs) | CW Logs subscription filter | subscription filter sends to Firehose | log-to-firehose pipe |\n| Lambda transformation | Lambda function exists; IAM role permits invoke | Lambda receives records, returns transformed + failed records | custom ETL processing |\n| Format conversion (Parquet\u002FORC) | Glue database + table with schema; IAM role permits Glue | conversion uses the Glue table schema; wrong schema = malformed output | columnar format for analytics |\n| S3 destination | S3 bucket exists; IAM role permits S3 write | prefix pattern determines partitioning; error prefix catches failed records | S3 data lake delivery |\n| OpenSearch destination | OpenSearch domain exists; IAM role permits write | index name must be valid; retry buffer holds failed writes | OpenSearch indexing |\n| HTTP endpoint destination | endpoint URL accessible; API key\u002Fauth configured | Firehose POSTs batches; endpoint must ACK within timeout | Splunk \u002F custom delivery |\n| Redshift destination | Redshift cluster exists; S3 staging bucket; COPY command | Firehose stages to S3 then issues COPY; username\u002Fpassword or IAM auth | Redshift loading |\n| Backup configuration | S3 bucket for backup | source record backup (pre-transformation); failed record backup | data recovery |\n| KMS encryption | KMS key exists; IAM role permits kms:Decrypt | encryption at rest for buffered data and S3 delivery | encryption compliance |\n| CloudWatch logging | CW log group exists; IAM role permits logs | error logging enabled per destination type | operational visibility |\n\n**The buffering-hints + format-conversion row is the one a baseline\nmodel misses.** Buffering hints interact with format conversion: if\nbuffering is too small, the Parquet writer produces many tiny files\n(row group overhead dominates). If buffering is too large, latency\nincreases. The correct tuning depends on the format and downstream\nquery patterns.\n\n**Cross-dependency gotchas:**\n- Lambda transformation runs BEFORE format conversion. If Lambda\n  returns malformed JSON, format conversion to Parquet fails silently\n  (records go to the error prefix).\n- The IAM role for the delivery stream must trust firehose.amazonaws.com\n  and have permissions for ALL configured destinations, Lambda, Glue,\n  and KMS as applicable.\n- S3 prefix patterns use Firehose-specific expression syntax\n  (`!{timestamp:yyyy}`, `!{partitionKeyFromQuery:...}`). Using the\n  wrong syntax produces literal folder names instead of dynamic\n  partitions.\n\n## Expert heuristic: buffering hints trade-off (latency vs cost)\n\nBuffering-hints latency-vs-cost model and per-destination recommendations moved to references.\n→ [references\u002Fadvanced-patterns.md](references\u002Fadvanced-patterns.md) § Expert heuristic: buffering hints trade-off\n\n## Expert heuristic: Lambda transformation blueprints\n\nTransformation flow, blueprint catalog, and Parquet JSON-validity rules moved to references.\n→ [references\u002Fadvanced-patterns.md](references\u002Fadvanced-patterns.md) § Expert heuristic: Lambda transformation blueprints\n\n## Expert heuristic: S3 partition projection via Hive-style prefix\n\nFlat vs Hive-style prefix comparison and error-prefix patterns moved to references.\n→ [references\u002Fadvanced-patterns.md](references\u002Fadvanced-patterns.md) § Expert heuristic: S3 partition projection\n\n## Prerequisites (verify before provisioning)\n\n| Prerequisite | Why it matters | How to verify |\n|---|---|---|\n| S3 bucket exists (S3\u002FRedshift destination) | Firehose delivers to S3; Redshift stages via S3 | `aws s3api head-bucket --bucket \u003Cname>` |\n| OpenSearch domain exists (OpenSearch destination) | Firehose writes to an OpenSearch index | `aws opensearch describe-domain --domain-name \u003Cname>` |\n| Redshift cluster exists (Redshift destination) | Firehose issues COPY from S3 staging | `aws redshift describe-clusters` |\n| Lambda function exists (if transformation) | Firehose invokes Lambda for ETL | `aws lambda get-function --function-name \u003Cname>` |\n| Glue database + table (if format conversion) | Parquet\u002FORC conversion uses Glue schema | `aws glue get-table --database-name \u003Cdb> --name \u003Ctable>` |\n| KMS key exists (if SSE enabled) | Encryption at rest for buffered data | `aws kms describe-key --key-id \u003Ckey-id>` |\n| IAM role with firehose trust policy | Firehose assumes the role for all service access | Verify trust policy and permissions |\n| CloudWatch log group (if logging enabled) | Error logging destination | `aws logs describe-log-groups` |\n\nIf any prerequisite is missing, output `VERDICT: PREREQUISITES_MISSING`\nand cite the specific gap.\n\n## Step 1 — Source selection\n\n| Source type | How it works | When to use |\n|---|---|---|\n| Direct PUT | Producers call PutRecord\u002FPutRecordBatch API | Custom producers, API Gateway, CloudWatch Logs subscription, IoT |\n| Kinesis Data Stream | Firehose reads from a KDS stream | Existing KDS pipeline needing delivery |\n| Amazon MSK | Firehose reads from MSK topic directly | Kafka-based streaming, no consumer management |\n| CloudWatch Logs | Subscription filter sends logs to Firehose | Log processing, SIEM ingestion |\n\n**Direct PUT (most common):** no source configuration needed beyond\nthe type. Producers use the Firehose API.\n\n**Kinesis Data Stream:** specify the stream ARN. Firehose can use\nenhanced fan-out (recommended for high throughput) or polling.\n\n## Step 2 — Lambda transformation\n\nLambda transformation runs custom ETL on each batch. Configure the\nLambda function ARN and a buffer size (1-128 MB) that controls batch\nsize.\n\n```bash\n# In the Firehose config:\nProcessingConfiguration:\n  Enabled: true\n  Processors:\n    - Type: Lambda\n      Parameters:\n        - ParameterName: LambdaArn\n          ParameterValue: arn:aws:lambda:us-east-1:123456789012:function:firehose-transform\n        - ParameterName: BufferSizeInMBs\n          ParameterValue: \"3\"\n        - ParameterName: RoleArn\n          ParameterValue: arn:aws:iam::123456789012:role\u002FFirehoseStreamRole\n```\n\n**Lambda output contract:** the function must return a JSON object with\n`records` (each with `recordId`, `result: Ok\u002FDropped`, and `data` as\nbase64). Records marked `Dropped` are silently discarded.\n\n**Key implication:** if using format conversion to Parquet, the Lambda\noutput MUST be valid JSON matching the Glue table schema. Malformed\nJSON causes conversion failure.\n\n## Step 3 — Format conversion (Parquet\u002FORC\u002FJSON)\n\nFormat conversion uses a Glue table schema to serialize records to\ncolumnar formats (Parquet, ORC) — no code needed.\n\n```bash\n# In the S3 destination config:\nDataFormatConversionConfiguration:\n  Enabled: true\n  InputFormatConfiguration:\n    Deserializer:\n      OpenXJsonSerDe: {}\n  OutputFormatConfiguration:\n    Serializer:\n      ParquetSerDe: {}  # or OrcSerDe\n  SchemaConfiguration:\n    DatabaseName: my_glue_db\n    TableName: my_glue_table\n    RoleArn: arn:aws:iam::123456789012:role\u002FFirehoseStreamRole\n    Region: us-east-1\n```\n\n**Requirements:**\n- Glue database and table must exist with the correct schema.\n- The IAM role must have `glue:GetTable` permissions.\n- Input records must be valid JSON (if using Lambda transformation,\n  ensure the output is valid JSON).\n- Buffering hints should be larger for Parquet (64-128 MB) to produce\n  optimal row group sizes.\n\n## Step 4 — S3 destination\n\nThe most common destination. Configure bucket ARN, prefix, error\nprefix, buffering hints, compression, and encryption.\n\n```bash\nS3DestinationConfiguration:\n  RoleARN: arn:aws:iam::123456789012:role\u002FFirehoseStreamRole\n  BucketARN: arn:aws:s3:::my-data-lake\n  Prefix: \"data\u002Fyear=!{timestamp:yyyy}\u002Fmonth=!{timestamp:MM}\u002Fday=!{timestamp:dd}\u002F\"\n  ErrorOutputPrefix: \"errors\u002F!{firehose:error-output-type}\u002Fyear=!{timestamp:yyyy}\u002Fmonth=!{timestamp:MM}\u002Fday=!{timestamp:dd}\u002F\"\n  BufferingHints:\n    SizeInMBs: 64\n    IntervalInSeconds: 300\n  CompressionFormat: UNCOMPRESSED  # SNAPPY\u002FGZIP\u002FZIP for non-Parquet\n  EncryptionConfiguration:\n    KMSEncryptionConfig:\n      AWSKMSKeyARN: arn:aws:kms:us-east-1:123456789012:key\u002Fabc123\n```\n\n**Prefix patterns:** use Hive-style partitioning for Athena\ncompatibility. Use `!{timestamp:yyyy}`, `!{timestamp:MM}`,\n`!{timestamp:dd}`, `!{timestamp:HH}` for time-based partitioning.\n\n**Compression:** for Parquet format conversion, leave compression as\nUNCOMPRESSED (Parquet has built-in compression). For JSON, use GZIP or\nSNAPPY.\n\n## Step 5 — OpenSearch destination\n\nFull OpenSearch destination configuration (domain ARN, index rotation, retry, S3 backup) moved to references.\n→ [references\u002Fdestinations-and-buffering.md](references\u002Fdestinations-and-buffering.md) § OpenSearch destination\n\n## Step 6 — HTTP endpoint destination (Splunk\u002Fcustom)\n\nFull HTTP endpoint configuration (Splunk HEC URL, access key, buffering, retry) moved to references.\n→ [references\u002Fdestinations-and-buffering.md](references\u002Fdestinations-and-buffering.md) § HTTP endpoint destination\n\n## Step 7 — Redshift destination\n\nFull Redshift destination configuration (JDBC URL, COPY command, S3 staging) moved to references.\n→ [references\u002Fdestinations-and-buffering.md](references\u002Fdestinations-and-buffering.md) § Redshift destination\n\n## Step 8 — Backup configuration\n\nFirehose can back up source records (pre-transformation) or failed\nrecords (post-transformation delivery failure).\n\n| Backup mode | What it backs up | Destination |\n|---|---|---|\n| `SourceRecord` (S3 dest) | All source records before transformation | S3 backup bucket |\n| `FailedDocumentsOnly` (OpenSearch\u002FHTTP dest) | Records that failed delivery | S3 backup bucket |\n| `FailedDataOnly` (Redshift dest) | Records that failed COPY | S3 backup bucket |\n\n**Recommendation:** always enable backup for production streams. The\nbackup bucket should be separate from the main delivery bucket.\n\n## Step 9 — Buffering hints (latency vs cost trade-off)\n\n| Destination | Recommended SizeInMBs | Recommended IntervalInSeconds | Rationale |\n|---|---|---|---|\n| S3 (JSON) | 5-15 MB | 300s | Balance object size vs latency |\n| S3 (Parquet) | 64-128 MB | 300-900s | Larger row groups = better compression + Athena performance |\n| OpenSearch | 5 MB | 60-300s | Smaller batches reduce indexing overhead |\n| HTTP endpoint | 5 MB | 60s | Low latency for downstream SIEM |\n| Redshift | 64-128 MB | 300-900s | Fewer, larger COPY operations |\n\n**Too-small buffering for Parquet = many tiny files.** Each tiny\nParquet file has row group overhead. Athena scanning many tiny files is\nslow and expensive. Aim for 64+ MB per Parquet file.\n\n## Step 10 — Retry and duration settings\n\n| Setting | Default | Range | Effect |\n|---|---|---|---|\n| `RetryOptions.DurationInSeconds` | 300 | 0-7200 | How long Firehose retries failed writes before giving up |\n| S3 backpressure | automatic | N\u002FA | Firehose handles S3 throttling internally |\n\nAfter the retry duration expires, failed records go to the backup\u002Ferror\nS3 prefix.\n\n## Step 11 — CloudWatch logging\n\nFirehose logs delivery errors to CloudWatch Logs. Each destination type\nhas its own log stream prefix.\n\n```bash\n# Enable logging in the delivery stream config\n--extended-config (via CloudWatch logging)\n```\n\nEnable CloudWatch logging to capture format conversion errors,\nLambda transformation errors, and destination delivery failures.\n\n## Step 12 — Server-side encryption (KMS)\n\nFirehose supports KMS encryption for data at rest (in the buffer before\ndelivery) and for S3 delivery.\n\n```bash\n# In the S3 destination config:\nEncryptionConfiguration:\n  KMSEncryptionConfig:\n    AWSKMSKeyARN: arn:aws:kms:us-east-1:123456789012:key\u002Fabc123\n\n# Or use NoEncryptionConfig for unencrypted (not recommended)\n```\n\n**The IAM role must have kms:Decrypt and kms:GenerateDataKey\npermissions** for the specified KMS key.\n\n## Step 13 — Recent features\n\n2023-2025 Firehose feature notes (MSK source, CW Logs subscription, partition projection, enhanced fan-out) moved to references.\n→ [references\u002Fadvanced-patterns.md](references\u002Fadvanced-patterns.md) § Step 13 — Recent features\n\n## NEVER do these things\n\n1. **NEVER use default buffering hints for Parquet destinations.** The\n   default 5 MB produces many tiny Parquet files with row group\n   overhead. Use 64-128 MB for Parquet to produce optimal file sizes\n   for Athena.\n\n2. **NEVER use a flat S3 prefix for analytics destinations.** A flat\n   prefix (`data\u002F`) means Athena scans all files. Use Hive-style\n   partitioned prefixes (`data\u002Fyear=!{timestamp:yyyy}\u002Fmonth=...\u002F`)\n   for partition pruning.\n\n3. **NEVER assume Lambda transformation and format conversion are\n   interchangeable.** Lambda runs custom code per batch; format\n   conversion uses a Glue schema. They can be used together (Lambda\n   transforms, then conversion serializes to Parquet), but the Lambda\n   output must be valid JSON matching the Glue schema.\n\n4. **NEVER deploy without a backup configuration in production.**\n   Without backup, failed records are lost. Configure\n   `FailedDocumentsOnly` backup at minimum.\n\n5. **NEVER use the same S3 bucket for main delivery and backup.**\n   Failed records should go to a separate bucket or prefix for easier\n   reprocessing. Mixing them complicates downstream ETL.\n\n6. **NEVER leave encryption disabled for sensitive data.** Configure\n   KMS encryption for both the Firehose buffer and S3 delivery. The\n   IAM role must have kms:Decrypt and kms:GenerateDataKey permissions.\n\n7. **NEVER forget the IAM role trust policy.** The delivery stream IAM\n   role must trust `firehose.amazonaws.com` and have permissions for\n   ALL configured destinations, Lambda (if transformation), Glue (if\n   format conversion), and KMS (if encryption).\n\n8. **NEVER use GZIP compression with Parquet format conversion.**\n   Parquet has built-in compression. Set CompressionFormat to\n   UNCOMPRESSED when using format conversion. GZIP is for JSON\u002FCSV.\n\n9. **NEVER assume CloudWatch Logs subscription works without the\n   correct IAM role.** The subscription filter needs a role that\n   permits `firehose:PutRecordBatch`. Verify the trust policy and\n   permissions.\n\n10. **NEVER set retry duration too low for flaky destinations.** The\n    default 300s may be too short for OpenSearch under load. Increase\n    `DurationInSeconds` (up to 7200s) for destinations with transient\n    failures.\n\n## Output format\n\n```text\nFIREHOSE: \u003Cstream-name> (\u003Csource> → \u003Cdestination>)\nVERDICT: READY_TO_DEPLOY | PREREQUISITES_MISSING\nCHECKLIST:\n  [✓|✗] Source: Direct PUT | Kinesis Stream (\u003Cstream-name>) | MSK (\u003Ccluster>\u002F\u003Ctopic>) | CloudWatch Logs\n  [✓|✗] Lambda transformation: enabled (\u003Cfunction-arn>, buffer \u003CN>MB) | disabled\n  [✓|✗] Format conversion: Parquet (Glue \u003Cdb>\u002F\u003Ctable>) | ORC | JSON (no conversion)\n  [✓|✗] Destination: S3 (\u003Cbucket>, prefix \u003Cprefix>) | OpenSearch (\u003Cdomain>, index \u003Cindex>) | HTTP (\u003Curl>) | Redshift (\u003Ccluster>)\n  [✓|✗] Buffering hints: \u003Csize>MB \u002F \u003Cinterval>s (tuned for \u003Cformat\u002Fdestination>)\n  [✓|✗] S3 prefix: Hive-style partitioned (\u003Cpattern>) | flat\n  [✓|✗] Error prefix: \u003Cerror-prefix>\n  [✓|✗] Compression: \u003Cformat> (UNCOMPRESSED for Parquet)\n  [✓|✗] Backup: \u003Cmode> → \u003Cbackup-bucket> | disabled\n  [✓|✗] Retry: \u003Cduration>s\n  [✓|✗] CloudWatch logging: enabled | disabled\n  [✓|✗] KMS encryption: enabled (\u003Ckey-arn>) | disabled\n  [✓|✗] IAM role: \u003Crole-arn> (trust: firehose.amazonaws.com)\n  [✓|✗] Tags: \u003Ckey=value list>\nVERIFICATION_COMMANDS:\n  aws firehose describe-delivery-stream --delivery-stream-name \u003Cname> --region \u003Cregion>\n  aws firehose describe-delivery-stream --delivery-stream-name \u003Cname> --query 'DeliveryStreamDescription.DeliveryStreamStatus' --region \u003Cregion>\n```\n\n### Worked example — S3 Parquet with Lambda transformation\n\n```text\nFIREHOSE: events-to-s3-parquet (Direct PUT → S3)\nVERDICT: READY_TO_DEPLOY\nCHECKLIST:\n  [✓] Source: Direct PUT\n  [✓] Lambda transformation: enabled (arn:aws:lambda:us-east-1:123456789012:function:firehose-transform, buffer 3MB)\n  [✓] Format conversion: Parquet (Glue analytics\u002Fevents_table)\n  [✓] Destination: S3 (arn:aws:s3:::my-data-lake, prefix data\u002Fyear=!{timestamp:yyyy}\u002Fmonth=!{timestamp:MM}\u002Fday=!{timestamp:dd}\u002F)\n  [✓] Buffering hints: 128MB \u002F 300s (tuned for Parquet — optimal row group size)\n  [✓] S3 prefix: Hive-style partitioned (year\u002Fmonth\u002Fday)\n  [✓] Error prefix: errors\u002F!{firehose:error-output-type}\u002Fyear=!{timestamp:yyyy}\u002Fmonth=!{timestamp:MM}\u002Fday=!{timestamp:dd}\u002F\n  [✓] Compression: UNCOMPRESSED (Parquet has built-in compression)\n  [✓] Backup: FailedDocumentsOnly → arn:aws:s3:::firehose-backup\n  [✓] Retry: 300s\n  [✓] CloudWatch logging: enabled\n  [✓] KMS encryption: enabled (arn:aws:kms:us-east-1:123456789012:key\u002Fabc123)\n  [✓] IAM role: arn:aws:iam::123456789012:role\u002FFirehoseStreamRole (trust: firehose.amazonaws.com)\n  [✓] Tags: Environment=production, Pipeline=events\nVERIFICATION_COMMANDS:\n  aws firehose describe-delivery-stream --delivery-stream-name events-to-s3-parquet --region us-east-1\n  aws firehose describe-delivery-stream --delivery-stream-name events-to-s3-parquet --query 'DeliveryStreamDescription.DeliveryStreamStatus' --region us-east-1\n```\n\n## Error handling\n\nFailure-mode deep dives (error prefix, tiny S3 files, OpenSearch lag, stuck CREATING, CW Logs subscription) moved to references.\n→ [references\u002Ferror-handling.md](references\u002Ferror-handling.md) § Error handling\n\n## References (load on demand)\n\nConsult these only when the corresponding topic comes up:\n\n- [references\u002Fdestinations-and-buffering.md](references\u002Fdestinations-and-buffering.md) — OpenSearch \u002F HTTP endpoint \u002F Redshift destination configs (moved from Steps 5-7), buffering deep dive\n- [references\u002Ftransformation-and-encryption.md](references\u002Ftransformation-and-encryption.md) — Lambda blueprints, format conversion, KMS encryption, IAM roles\n- [references\u002Ferror-handling.md](references\u002Ferror-handling.md) — failure modes moved from § Error handling\n- [references\u002Fadvanced-patterns.md](references\u002Fadvanced-patterns.md) — expert heuristics (buffering trade-off, Lambda blueprints, partition projection) and recent AWS features\n\n## Domain\n\nAWS CloudOps \u002F Kinesis Data Firehose Delivery Stream Provisioning &\nData Pipeline Automation.\n\n## AWS documentation\n\n- **Firehose Guide** — https:\u002F\u002Fdocs.aws.amazon.com\u002Ffirehose\u002Flatest\u002Fdev\u002Fwhat-is-this-service.html\n- **Create delivery stream** — https:\u002F\u002Fdocs.aws.amazon.com\u002Ffirehose\u002Flatest\u002Fdev\u002Fbasic-create.html\n- **S3 destination** — https:\u002F\u002Fdocs.aws.amazon.com\u002Ffirehose\u002Flatest\u002Fdev\u002Fcreate-destination.html#create-destination-s3\n- **OpenSearch destination** — https:\u002F\u002Fdocs.aws.amazon.com\u002Ffirehose\u002Flatest\u002Fdev\u002Fcreate-destination.html#create-destination-opensearch\n- **Lambda transformation** — https:\u002F\u002Fdocs.aws.amazon.com\u002Ffirehose\u002Flatest\u002Fdev\u002Fdata-transformation.html\n- **Format conversion** — https:\u002F\u002Fdocs.aws.amazon.com\u002Ffirehose\u002Flatest\u002Fdev\u002Fdata-conversion.html\n- **Buffering hints** — https:\u002F\u002Fdocs.aws.amazon.com\u002Ffirehose\u002Flatest\u002Fdev\u002Fbasic-deliver.html#buffering-hints\n- **S3 prefix patterns** — https:\u002F\u002Fdocs.aws.amazon.com\u002Ffirehose\u002Flatest\u002Fdev\u002Fs3-prefixes.html\n- **KMS encryption** — https:\u002F\u002Fdocs.aws.amazon.com\u002Ffirehose\u002Flatest\u002Fdev\u002Fencryption.html\n- **HTTP endpoint destination** — https:\u002F\u002Fdocs.aws.amazon.com\u002Ffirehose\u002Flatest\u002Fdev\u002Fhttpdeliveryrequestresponse.html\n","# Firehose Delivery Stream Deployer\n\nAn AWS CloudOps agent skill that provisions Kinesis Data Firehose\ndelivery streams with correct defaults. The skill walks the operator\nthrough source selection (Direct PUT, Kinesis Data Stream, MSK,\nCloudWatch Logs), processing\u002Ftransformation (Lambda function for ETL),\nformat conversion (Parquet\u002FORC\u002FJSON via dynamic schema), destination\nselection (S3, OpenSearch, HTTP endpoint, Redshift), backup\nconfiguration, buffering hints, retry\u002Fduration settings, CloudWatch\nlogging, and server-side encryption (KMS), captures data pipeline\ndecisions, explains why each default matters, and emits a\nREADY_TO_DEPLOY checklist with copy-pasteable verification commands.\n\n## Activation keywords\n\ncreate Firehose delivery stream, Firehose S3 destination, Firehose\nLambda transformation, Firehose Parquet conversion, Firehose\nOpenSearch destination, Firehose Redshift destination, Firehose Splunk\nHTTP endpoint, Firehose buffering hints.\n\n## STRICT output contract\n\nWhen this skill is invoked with a Firehose-provisioning request\n(create a delivery stream, configure a destination, set up Lambda\ntransformation, enable format conversion, or a partial configuration),\nthe agent MUST respond with the READY_TO_DEPLOY checklist defined in\nthe \"Output format\" section using the literal all-caps labels\n`FIREHOSE:`, `VERDICT:`, `CHECKLIST:`, and `VERIFICATION_COMMANDS:`.\nDo NOT preface the checklist with prose, headings, or disclaimers —\nemit the block as the first lines of the response. This contract is\nwhat assertion-based evals and downstream provisioning pipelines rely\non; deviating from the literal labels breaks automation silently.\n\nIf any prerequisite is missing, the verdict is `PREREQUISITES_MISSING`\nwith a specific gap citation in the checklist (marked `[✗]`), and\n`READY_TO_DEPLOY` MUST NOT also appear.\n\n## Quick navigation\n\n| Section | When to read |\n|---|---|\n| Prerequisites | Always — verify before provisioning |\n| Step 1 — Source selection | Direct PUT vs Kinesis Stream vs MSK vs CW Logs |\n| Step 2 — Lambda transformation | ETL processing |\n| Step 3 — Format conversion | Parquet\u002FORC\u002FJSON via dynamic schema |\n| Step 4 — S3 destination | Prefix patterns, error prefix, buffering |\n| Step 5 — OpenSearch destination | Index management, retry |\n| Step 6 — HTTP endpoint destination | Splunk\u002Fcustom endpoints |\n| Step 7 — Redshift destination | COPY via S3 staging |\n| Step 8 — Backup configuration | Source record backup |\n| Step 9 — Buffering hints | Latency vs cost trade-off |\n| Step 10 — Retry and duration | Error handling |\n| Step 11 — CloudWatch logging | Error logging |\n| Step 12 — Server-side encryption (KMS) | Encryption at rest |\n| Step 13 — Recent features | Latest |\n| NEVER do these things | Review before signing off |\n| Output format | The literal checklist template |\n| references\u002Fdestinations-and-buffering.md | Destination + buffering detail |\n| references\u002Ftransformation-and-encryption.md | Lambda + KMS detail |\n\n## Mindset\n\n**One-line takeaway:** A Firehose delivery stream is a fully managed\nETL pipeline that ingests data from a source, optionally transforms it\nvia Lambda, optionally converts the format (Parquet\u002FORC), and delivers\nit to a destination (S3, OpenSearch, HTTP endpoint, Redshift).\nBuffering hints control the latency-vs-cost trade-off. The source,\ntransformation, and destination must all be configured correctly for\ndata to flow end-to-end.\n\nThree misconceptions dominate Firehose misdesign at provisioning time:\n\n- **\"Buffering hints don't matter.\"** They do. Buffering hints\n  (SizeInMBs and IntervalInSeconds) control when Firehose flushes\n  buffered data to the destination. Too low = frequent small writes\n  (high cost from many small S3 objects, high OpenSearch indexing\n  overhead). Too high = high latency. A baseline model often leaves\n  defaults without considering the trade-off.\n\n- **\"Lambda transformation and format conversion are\n  interchangeable.\"** They are NOT. Lambda transformation runs custom\n  code on each batch of records (any transformation logic). Format\n  conversion is a built-in feature that converts JSON to Parquet or\n  ORC using a Glue table schema — no code needed. They CAN be used\n  together (Lambda transforms, then format conversion serializes to\n  Parquet), but the data flow order matters.\n\n- **\"S3 prefix patterns are cosmetic.\"** They are not. S3 prefix\n  patterns (Hive-style `year=!{timestamp:yyyy}\u002Fmonth=!{timestamp:MM}\u002F`)\n  determine how data is partitioned in S3. Partitioning affects Athena\n  query performance (partition pruning), Glue crawler efficiency, and\n  cost (scanning less data). A flat prefix means every query scans all\n  data.\n\n## Configuration dependency graph (novel heuristic)\n\nFirehose configurations are NOT independent. The source must be\nconfigured before the stream can receive data. Lambda transformation\nmust reference an existing Lambda function. Format conversion requires\na Glue table schema. The destination type determines backup and retry\nbehavior. Use this graph to sequence provisioning.\n\n| Configuration | Hard dependencies (API error without) | Silent failure \u002F immutability | Enables downstream |\n|---|---|---|---|\n| Delivery stream | source config valid; destination config valid | stream is CREATING then ACTIVE; cannot deliver until ACTIVE | the stream ARN for producers |\n| Source (Direct PUT) | none | producers use PutRecord\u002FPutRecordBatch API | data ingestion |\n| Source (Kinesis Stream) | Kinesis Data Stream exists | stream must be in same region; Firehose reads via enhanced fan-out or polling | stream-to-firehose pipe |\n| Source (MSK) | MSK cluster exists; topic identified | Firehose reads from MSK topic directly | MSK-to-firehose pipe |\n| Source (CloudWatch Logs) | CW Logs subscription filter | subscription filter sends to Firehose | log-to-firehose pipe |\n| Lambda transformation | Lambda function exists; IAM role permits invoke | Lambda receives records, returns transformed + failed records | custom ETL processing |\n| Format conversion (Parquet\u002FORC) | Glue database + table with schema; IAM role permits Glue | conversion uses the Glue table schema; wrong schema = malformed output | columnar format for analytics |\n| S3 destination | S3 bucket exists; IAM role permits S3 write | prefix pattern determines partitioning; error prefix catches failed records | S3 data lake delivery |\n| OpenSearch destination | OpenSearch domain exists; IAM role permits write | index name must be valid; retry buffer holds failed writes | OpenSearch indexing |\n| HTTP endpoint destination | endpoint URL accessible; API key\u002Fauth configured | Firehose POSTs batches; endpoint must ACK within timeout | Splunk \u002F custom delivery |\n| Redshift destination | Redshift cluster exists; S3 staging bucket; COPY command | Firehose stages to S3 then issues COPY; username\u002Fpassword or IAM auth | Redshift loading |\n| Backup configuration | S3 bucket for backup | source record backup (pre-transformation); failed record backup | data recovery |\n| KMS encryption | KMS key exists; IAM role permits kms:Decrypt | encryption at rest for buffered data and S3 delivery | encryption compliance |\n| CloudWatch logging | CW log group exists; IAM role permits logs | error logging enabled per destination type | operational visibility |\n\n**The buffering-hints + format-conversion row is the one a baseline\nmodel misses.** Buffering hints interact with format conversion: if\nbuffering is too small, the Parquet writer produces many tiny files\n(row group overhead dominates). If buffering is too large, latency\nincreases. The correct tuning depends on the format and downstream\nquery patterns.\n\n**Cross-dependency gotchas:**\n- Lambda transformation runs BEFORE format conversion. If Lambda\n  returns malformed JSON, format conversion to Parquet fails silently\n  (records go to the error prefix).\n- The IAM role for the delivery stream must trust firehose.amazonaws.com\n  and have permissions for ALL configured destinations, Lambda, Glue,\n  and KMS as applicable.\n- S3 prefix patterns use Firehose-specific expression syntax\n  (`!{timestamp:yyyy}`, `!{partitionKeyFromQuery:...}`). Using the\n  wrong syntax produces literal folder names instead of dynamic\n  partitions.\n\n## Expert heuristic: buffering hints trade-off (latency vs cost)\n\nBuffering-hints latency-vs-cost model and per-destination recommendations moved to references.\n→ [references\u002Fadvanced-patterns.md](references\u002Fadvanced-patterns.md) § Expert heuristic: buffering hints trade-off\n\n## Expert heuristic: Lambda transformation blueprints\n\nTransformation flow, blueprint catalog, and Parquet JSON-validity rules moved to references.\n→ [references\u002Fadvanced-patterns.md](references\u002Fadvanced-patterns.md) § Expert heuristic: Lambda transformation blueprints\n\n## Expert heuristic: S3 partition projection via Hive-style prefix\n\nFlat vs Hive-style prefix comparison and error-prefix patterns moved to references.\n→ [references\u002Fadvanced-patterns.md](references\u002Fadvanced-patterns.md) § Expert heuristic: S3 partition projection\n\n## Prerequisites (verify before provisioning)\n\n| Prerequisite | Why it matters | How to verify |\n|---|---|---|\n| S3 bucket exists (S3\u002FRedshift destination) | Firehose delivers to S3; Redshift stages via S3 | `aws s3api head-bucket --bucket \u003Cname>` |\n| OpenSearch domain exists (OpenSearch destination) | Firehose writes to an OpenSearch index | `aws opensearch describe-domain --domain-name \u003Cname>` |\n| Redshift cluster exists (Redshift destination) | Firehose issues COPY from S3 staging | `aws redshift describe-clusters` |\n| Lambda function exists (if transformation) | Firehose invokes Lambda for ETL | `aws lambda get-function --function-name \u003Cname>` |\n| Glue database + table (if format conversion) | Parquet\u002FORC conversion uses Glue schema | `aws glue get-table --database-name \u003Cdb> --name \u003Ctable>` |\n| KMS key exists (if SSE enabled) | Encryption at rest for buffered data | `aws kms describe-key --key-id \u003Ckey-id>` |\n| IAM role with firehose trust policy | Firehose assumes the role for all service access | Verify trust policy and permissions |\n| CloudWatch log group (if logging enabled) | Error logging destination | `aws logs describe-log-groups` |\n\nIf any prerequisite is missing, output `VERDICT: PREREQUISITES_MISSING`\nand cite the specific gap.\n\n## Step 1 — Source selection\n\n| Source type | How it works | When to use |\n|---|---|---|\n| Direct PUT | Producers call PutRecord\u002FPutRecordBatch API | Custom producers, API Gateway, CloudWatch Logs subscription, IoT |\n| Kinesis Data Stream | Firehose reads from a KDS stream | Existing KDS pipeline needing delivery |\n| Amazon MSK | Firehose reads from MSK topic directly | Kafka-based streaming, no consumer management |\n| CloudWatch Logs | Subscription filter sends logs to Firehose | Log processing, SIEM ingestion |\n\n**Direct PUT (most common):** no source configuration needed beyond\nthe type. Producers use the Firehose API.\n\n**Kinesis Data Stream:** specify the stream ARN. Firehose can use\nenhanced fan-out (recommended for high throughput) or polling.\n\n## Step 2 — Lambda transformation\n\nLambda transformation runs custom ETL on each batch. Configure the\nLambda function ARN and a buffer size (1-128 MB) that controls batch\nsize.\n\n```bash\n# In the Firehose config:\nProcessingConfiguration:\n  Enabled: true\n  Processors:\n    - Type: Lambda\n      Parameters:\n        - ParameterName: LambdaArn\n          ParameterValue: arn:aws:lambda:us-east-1:123456789012:function:firehose-transform\n        - ParameterName: BufferSizeInMBs\n          ParameterValue: \"3\"\n        - ParameterName: RoleArn\n          ParameterValue: arn:aws:iam::123456789012:role\u002FFirehoseStreamRole\n```\n\n**Lambda output contract:** the function must return a JSON object with\n`records` (each with `recordId`, `result: Ok\u002FDropped`, and `data` as\nbase64). Records marked `Dropped` are silently discarded.\n\n**Key implication:** if using format conversion to Parquet, the Lambda\noutput MUST be valid JSON matching the Glue table schema. Malformed\nJSON causes conversion failure.\n\n## Step 3 — Format conversion (Parquet\u002FORC\u002FJSON)\n\nFormat conversion uses a Glue table schema to serialize records to\ncolumnar formats (Parquet, ORC) — no code needed.\n\n```bash\n# In the S3 destination config:\nDataFormatConversionConfiguration:\n  Enabled: true\n  InputFormatConfiguration:\n    Deserializer:\n      OpenXJsonSerDe: {}\n  OutputFormatConfiguration:\n    Serializer:\n      ParquetSerDe: {}  # or OrcSerDe\n  SchemaConfiguration:\n    DatabaseName: my_glue_db\n    TableName: my_glue_table\n    RoleArn: arn:aws:iam::123456789012:role\u002FFirehoseStreamRole\n    Region: us-east-1\n```\n\n**Requirements:**\n- Glue database and table must exist with the correct schema.\n- The IAM role must have `glue:GetTable` permissions.\n- Input records must be valid JSON (if using Lambda transformation,\n  ensure the output is valid JSON).\n- Buffering hints should be larger for Parquet (64-128 MB) to produce\n  optimal row group sizes.\n\n## Step 4 — S3 destination\n\nThe most common destination. Configure bucket ARN, prefix, error\nprefix, buffering hints, compression, and encryption.\n\n```bash\nS3DestinationConfiguration:\n  RoleARN: arn:aws:iam::123456789012:role\u002FFirehoseStreamRole\n  BucketARN: arn:aws:s3:::my-data-lake\n  Prefix: \"data\u002Fyear=!{timestamp:yyyy}\u002Fmonth=!{timestamp:MM}\u002Fday=!{timestamp:dd}\u002F\"\n  ErrorOutputPrefix: \"errors\u002F!{firehose:error-output-type}\u002Fyear=!{timestamp:yyyy}\u002Fmonth=!{timestamp:MM}\u002Fday=!{timestamp:dd}\u002F\"\n  BufferingHints:\n    SizeInMBs: 64\n    IntervalInSeconds: 300\n  CompressionFormat: UNCOMPRESSED  # SNAPPY\u002FGZIP\u002FZIP for non-Parquet\n  EncryptionConfiguration:\n    KMSEncryptionConfig:\n      AWSKMSKeyARN: arn:aws:kms:us-east-1:123456789012:key\u002Fabc123\n```\n\n**Prefix patterns:** use Hive-style partitioning for Athena\ncompatibility. Use `!{timestamp:yyyy}`, `!{timestamp:MM}`,\n`!{timestamp:dd}`, `!{timestamp:HH}` for time-based partitioning.\n\n**Compression:** for Parquet format conversion, leave compression as\nUNCOMPRESSED (Parquet has built-in compression). For JSON, use GZIP or\nSNAPPY.\n\n## Step 5 — OpenSearch destination\n\nFull OpenSearch destination configuration (domain ARN, index rotation, retry, S3 backup) moved to references.\n→ [references\u002Fdestinations-and-buffering.md](references\u002Fdestinations-and-buffering.md) § OpenSearch destination\n\n## Step 6 — HTTP endpoint destination (Splunk\u002Fcustom)\n\nFull HTTP endpoint configuration (Splunk HEC URL, access key, buffering, retry) moved to references.\n→ [references\u002Fdestinations-and-buffering.md](references\u002Fdestinations-and-buffering.md) § HTTP endpoint destination\n\n## Step 7 — Redshift destination\n\nFull Redshift destination configuration (JDBC URL, COPY command, S3 staging) moved to references.\n→ [references\u002Fdestinations-and-buffering.md](references\u002Fdestinations-and-buffering.md) § Redshift destination\n\n## Step 8 — Backup configuration\n\nFirehose can back up source records (pre-transformation) or failed\nrecords (post-transformation delivery failure).\n\n| Backup mode | What it backs up | Destination |\n|---|---|---|\n| `SourceRecord` (S3 dest) | All source records before transformation | S3 backup bucket |\n| `FailedDocumentsOnly` (OpenSearch\u002FHTTP dest) | Records that failed delivery | S3 backup bucket |\n| `FailedDataOnly` (Redshift dest) | Records that failed COPY | S3 backup bucket |\n\n**Recommendation:** always enable backup for production streams. The\nbackup bucket should be separate from the main delivery bucket.\n\n## Step 9 — Buffering hints (latency vs cost trade-off)\n\n| Destination | Recommended SizeInMBs | Recommended IntervalInSeconds | Rationale |\n|---|---|---|---|\n| S3 (JSON) | 5-15 MB | 300s | Balance object size vs latency |\n| S3 (Parquet) | 64-128 MB | 300-900s | Larger row groups = better compression + Athena performance |\n| OpenSearch | 5 MB | 60-300s | Smaller batches reduce indexing overhead |\n| HTTP endpoint | 5 MB | 60s | Low latency for downstream SIEM |\n| Redshift | 64-128 MB | 300-900s | Fewer, larger COPY operations |\n\n**Too-small buffering for Parquet = many tiny files.** Each tiny\nParquet file has row group overhead. Athena scanning many tiny files is\nslow and expensive. Aim for 64+ MB per Parquet file.\n\n## Step 10 — Retry and duration settings\n\n| Setting | Default | Range | Effect |\n|---|---|---|---|\n| `RetryOptions.DurationInSeconds` | 300 | 0-7200 | How long Firehose retries failed writes before giving up |\n| S3 backpressure | automatic | N\u002FA | Firehose handles S3 throttling internally |\n\nAfter the retry duration expires, failed records go to the backup\u002Ferror\nS3 prefix.\n\n## Step 11 — CloudWatch logging\n\nFirehose logs delivery errors to CloudWatch Logs. Each destination type\nhas its own log stream prefix.\n\n```bash\n# Enable logging in the delivery stream config\n--extended-config (via CloudWatch logging)\n```\n\nEnable CloudWatch logging to capture format conversion errors,\nLambda transformation errors, and destination delivery failures.\n\n## Step 12 — Server-side encryption (KMS)\n\nFirehose supports KMS encryption for data at rest (in the buffer before\ndelivery) and for S3 delivery.\n\n```bash\n# In the S3 destination config:\nEncryptionConfiguration:\n  KMSEncryptionConfig:\n    AWSKMSKeyARN: arn:aws:kms:us-east-1:123456789012:key\u002Fabc123\n\n# Or use NoEncryptionConfig for unencrypted (not recommended)\n```\n\n**The IAM role must have kms:Decrypt and kms:GenerateDataKey\npermissions** for the specified KMS key.\n\n## Step 13 — Recent features\n\n2023-2025 Firehose feature notes (MSK source, CW Logs subscription, partition projection, enhanced fan-out) moved to references.\n→ [references\u002Fadvanced-patterns.md](references\u002Fadvanced-patterns.md) § Step 13 — Recent features\n\n## NEVER do these things\n\n1. **NEVER use default buffering hints for Parquet destinations.** The\n   default 5 MB produces many tiny Parquet files with row group\n   overhead. Use 64-128 MB for Parquet to produce optimal file sizes\n   for Athena.\n\n2. **NEVER use a flat S3 prefix for analytics destinations.** A flat\n   prefix (`data\u002F`) means Athena scans all files. Use Hive-style\n   partitioned prefixes (`data\u002Fyear=!{timestamp:yyyy}\u002Fmonth=...\u002F`)\n   for partition pruning.\n\n3. **NEVER assume Lambda transformation and format conversion are\n   interchangeable.** Lambda runs custom code per batch; format\n   conversion uses a Glue schema. They can be used together (Lambda\n   transforms, then conversion serializes to Parquet), but the Lambda\n   output must be valid JSON matching the Glue schema.\n\n4. **NEVER deploy without a backup configuration in production.**\n   Without backup, failed records are lost. Configure\n   `FailedDocumentsOnly` backup at minimum.\n\n5. **NEVER use the same S3 bucket for main delivery and backup.**\n   Failed records should go to a separate bucket or prefix for easier\n   reprocessing. Mixing them complicates downstream ETL.\n\n6. **NEVER leave encryption disabled for sensitive data.** Configure\n   KMS encryption for both the Firehose buffer and S3 delivery. The\n   IAM role must have kms:Decrypt and kms:GenerateDataKey permissions.\n\n7. **NEVER forget the IAM role trust policy.** The delivery stream IAM\n   role must trust `firehose.amazonaws.com` and have permissions for\n   ALL configured destinations, Lambda (if transformation), Glue (if\n   format conversion), and KMS (if encryption).\n\n8. **NEVER use GZIP compression with Parquet format conversion.**\n   Parquet has built-in compression. Set CompressionFormat to\n   UNCOMPRESSED when using format conversion. GZIP is for JSON\u002FCSV.\n\n9. **NEVER assume CloudWatch Logs subscription works without the\n   correct IAM role.** The subscription filter needs a role that\n   permits `firehose:PutRecordBatch`. Verify the trust policy and\n   permissions.\n\n10. **NEVER set retry duration too low for flaky destinations.** The\n    default 300s may be too short for OpenSearch under load. Increase\n    `DurationInSeconds` (up to 7200s) for destinations with transient\n    failures.\n\n## Output format\n\n```text\nFIREHOSE: \u003Cstream-name> (\u003Csource> → \u003Cdestination>)\nVERDICT: READY_TO_DEPLOY | PREREQUISITES_MISSING\nCHECKLIST:\n  [✓|✗] Source: Direct PUT | Kinesis Stream (\u003Cstream-name>) | MSK (\u003Ccluster>\u002F\u003Ctopic>) | CloudWatch Logs\n  [✓|✗] Lambda transformation: enabled (\u003Cfunction-arn>, buffer \u003CN>MB) | disabled\n  [✓|✗] Format conversion: Parquet (Glue \u003Cdb>\u002F\u003Ctable>) | ORC | JSON (no conversion)\n  [✓|✗] Destination: S3 (\u003Cbucket>, prefix \u003Cprefix>) | OpenSearch (\u003Cdomain>, index \u003Cindex>) | HTTP (\u003Curl>) | Redshift (\u003Ccluster>)\n  [✓|✗] Buffering hints: \u003Csize>MB \u002F \u003Cinterval>s (tuned for \u003Cformat\u002Fdestination>)\n  [✓|✗] S3 prefix: Hive-style partitioned (\u003Cpattern>) | flat\n  [✓|✗] Error prefix: \u003Cerror-prefix>\n  [✓|✗] Compression: \u003Cformat> (UNCOMPRESSED for Parquet)\n  [✓|✗] Backup: \u003Cmode> → \u003Cbackup-bucket> | disabled\n  [✓|✗] Retry: \u003Cduration>s\n  [✓|✗] CloudWatch logging: enabled | disabled\n  [✓|✗] KMS encryption: enabled (\u003Ckey-arn>) | disabled\n  [✓|✗] IAM role: \u003Crole-arn> (trust: firehose.amazonaws.com)\n  [✓|✗] Tags: \u003Ckey=value list>\nVERIFICATION_COMMANDS:\n  aws firehose describe-delivery-stream --delivery-stream-name \u003Cname> --region \u003Cregion>\n  aws firehose describe-delivery-stream --delivery-stream-name \u003Cname> --query 'DeliveryStreamDescription.DeliveryStreamStatus' --region \u003Cregion>\n```\n\n### Worked example — S3 Parquet with Lambda transformation\n\n```text\nFIREHOSE: events-to-s3-parquet (Direct PUT → S3)\nVERDICT: READY_TO_DEPLOY\nCHECKLIST:\n  [✓] Source: Direct PUT\n  [✓] Lambda transformation: enabled (arn:aws:lambda:us-east-1:123456789012:function:firehose-transform, buffer 3MB)\n  [✓] Format conversion: Parquet (Glue analytics\u002Fevents_table)\n  [✓] Destination: S3 (arn:aws:s3:::my-data-lake, prefix data\u002Fyear=!{timestamp:yyyy}\u002Fmonth=!{timestamp:MM}\u002Fday=!{timestamp:dd}\u002F)\n  [✓] Buffering hints: 128MB \u002F 300s (tuned for Parquet — optimal row group size)\n  [✓] S3 prefix: Hive-style partitioned (year\u002Fmonth\u002Fday)\n  [✓] Error prefix: errors\u002F!{firehose:error-output-type}\u002Fyear=!{timestamp:yyyy}\u002Fmonth=!{timestamp:MM}\u002Fday=!{timestamp:dd}\u002F\n  [✓] Compression: UNCOMPRESSED (Parquet has built-in compression)\n  [✓] Backup: FailedDocumentsOnly → arn:aws:s3:::firehose-backup\n  [✓] Retry: 300s\n  [✓] CloudWatch logging: enabled\n  [✓] KMS encryption: enabled (arn:aws:kms:us-east-1:123456789012:key\u002Fabc123)\n  [✓] IAM role: arn:aws:iam::123456789012:role\u002FFirehoseStreamRole (trust: firehose.amazonaws.com)\n  [✓] Tags: Environment=production, Pipeline=events\nVERIFICATION_COMMANDS:\n  aws firehose describe-delivery-stream --delivery-stream-name events-to-s3-parquet --region us-east-1\n  aws firehose describe-delivery-stream --delivery-stream-name events-to-s3-parquet --query 'DeliveryStreamDescription.DeliveryStreamStatus' --region us-east-1\n```\n\n## Error handling\n\nFailure-mode deep dives (error prefix, tiny S3 files, OpenSearch lag, stuck CREATING, CW Logs subscription) moved to references.\n→ [references\u002Ferror-handling.md](references\u002Ferror-handling.md) § Error handling\n\n## References (load on demand)\n\nConsult these only when the corresponding topic comes up:\n\n- [references\u002Fdestinations-and-buffering.md](references\u002Fdestinations-and-buffering.md) — OpenSearch \u002F HTTP endpoint \u002F Redshift destination configs (moved from Steps 5-7), buffering deep dive\n- [references\u002Ftransformation-and-encryption.md](references\u002Ftransformation-and-encryption.md) — Lambda blueprints, format conversion, KMS encryption, IAM roles\n- [references\u002Ferror-handling.md](references\u002Ferror-handling.md) — failure modes moved from § Error handling\n- [references\u002Fadvanced-patterns.md](references\u002Fadvanced-patterns.md) — expert heuristics (buffering trade-off, Lambda blueprints, partition projection) and recent AWS features\n\n## Domain\n\nAWS CloudOps \u002F Kinesis Data Firehose Delivery Stream Provisioning &\nData Pipeline Automation.\n\n## AWS documentation\n\n- **Firehose Guide** — https:\u002F\u002Fdocs.aws.amazon.com\u002Ffirehose\u002Flatest\u002Fdev\u002Fwhat-is-this-service.html\n- **Create delivery stream** — https:\u002F\u002Fdocs.aws.amazon.com\u002Ffirehose\u002Flatest\u002Fdev\u002Fbasic-create.html\n- **S3 destination** — https:\u002F\u002Fdocs.aws.amazon.com\u002Ffirehose\u002Flatest\u002Fdev\u002Fcreate-destination.html#create-destination-s3\n- **OpenSearch destination** — https:\u002F\u002Fdocs.aws.amazon.com\u002Ffirehose\u002Flatest\u002Fdev\u002Fcreate-destination.html#create-destination-opensearch\n- **Lambda transformation** — https:\u002F\u002Fdocs.aws.amazon.com\u002Ffirehose\u002Flatest\u002Fdev\u002Fdata-transformation.html\n- **Format conversion** — https:\u002F\u002Fdocs.aws.amazon.com\u002Ffirehose\u002Flatest\u002Fdev\u002Fdata-conversion.html\n- **Buffering hints** — https:\u002F\u002Fdocs.aws.amazon.com\u002Ffirehose\u002Flatest\u002Fdev\u002Fbasic-deliver.html#buffering-hints\n- **S3 prefix patterns** — https:\u002F\u002Fdocs.aws.amazon.com\u002Ffirehose\u002Flatest\u002Fdev\u002Fs3-prefixes.html\n- **KMS encryption** — https:\u002F\u002Fdocs.aws.amazon.com\u002Ffirehose\u002Flatest\u002Fdev\u002Fencryption.html\n- **HTTP endpoint destination** — https:\u002F\u002Fdocs.aws.amazon.com\u002Ffirehose\u002Flatest\u002Fdev\u002Fhttpdeliveryrequestresponse.html\n",{"name":5,"license":15,"metadata":24,"description":7,"compatibility":41},{"tags":25,"phase":26,"author":27,"domain":28,"family":29,"version":30,"keywords":31,"task_type":32,"complexity":33,"entry_point":34,"skill_class":35,"when_to_use":36,"dependencies":37,"requires_llm":38,"verdict_shape":39,"lifecycle_status":40,"supports_pipeline":38},"aws, kinesis, firehose, delivery-stream, cloudops, deploy, analytics, provisioning, s3-destination, lambda-transformation, parquet, kms-encryption","1","Jacky Chan — AWS Community Builder","aws-cloudops","Analytics","0.1.0","aws, kinesis, firehose, delivery stream, cloudops, deploy, provisioning, s3 destination, opensearch, redshift, lambda transformation, parquet, format conversion, buffering hints, kms encryption","deploy","high","false","capability","Invoke when the user wants to create a Kinesis Data Firehose delivery stream, configure S3\u002FOpenSearch\u002FRedshift\u002FHTTP endpoint destinations, set up Lambda transformation for ETL, enable Parquet or ORC format conversion, configure buffering hints, or set up KMS encryption. Do NOT invoke for Kinesis Data Streams (use kinesis-stream-deployer), Kinesis Data Analytics (use kinesis-analytics-deployer), or Firehose auditing\u002Ftroubleshooting.","aws-orchestrator","true","READY_TO_DEPLOY | PREREQUISITES_MISSING","active","Agent runtime that reads SKILL.md (Claude Code, Cursor, Windsurf, Codex, Gemini). For live deployment: AWS CLI v2 with firehose, s3, lambda, iam, and kms access. Works with Terraform aws_kinesis_firehose_delivery_stream resources and CloudFormation AWS::KinesisFirehose::DeliveryStream templates.","main","eeace51bd5dd1db2030729d83eeab988861553aa","skills\u002Ffirehose-delivery-stream-deployer",0,"https:\u002F\u002Fgithub.com\u002Fjhkchan\u002Faws-agent-skills","https:\u002F\u002Fgithub.com\u002Fjhkchan","CTO & Co-Founder @Beever-AI  & @votee. CAISP. AWS Community Builder. Organizer@GDGCloudHK. MongoDB Champion. Technopreneur + Infornographer.","2026-09-24 08:54:10",{"source":8,"skillName":5,"commands":51,"prompt":61,"zipAvailable":62},[52,55,58],{"tool":53,"command":54},"npx","npx skills add jhkchan\u002Faws-agent-skills --skill firehose-delivery-stream-deployer",{"tool":56,"command":57},"bunx","bunx skills add jhkchan\u002Faws-agent-skills --skill firehose-delivery-stream-deployer",{"tool":59,"command":60},"pnpm","pnpm dlx skills add jhkchan\u002Faws-agent-skills --skill firehose-delivery-stream-deployer","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fjhkchan\u002Faws-agent-skills\u002Fblob\u002Fmain\u002Fskills\u002Ffirehose-delivery-stream-deployer\u002FSKILL.md",true,[64,70,74,78,82,86,90,94,98,102,106,110,114,118,122,126,130,134,138],{"path":65,"resourceType":66,"resourceTypeDesc":67,"sizeBytes":68,"sourceHtmlUrl":69},"eval\u002Ftest-cases.yaml","REFERENCE","参考文档",6532,"https:\u002F\u002Fgithub.com\u002Fjhkchan\u002Faws-agent-skills\u002Fblob\u002Fmain\u002Fskills\u002Ffirehose-delivery-stream-deployer\u002Feval\u002Ftest-cases.yaml",{"path":71,"resourceType":66,"resourceTypeDesc":67,"sizeBytes":72,"sourceHtmlUrl":73},"evals\u002Fbaselines\u002Fbuffering-hints-tradeoff.md",736,"https:\u002F\u002Fgithub.com\u002Fjhkchan\u002Faws-agent-skills\u002Fblob\u002Fmain\u002Fskills\u002Ffirehose-delivery-stream-deployer\u002Fevals\u002Fbaselines\u002Fbuffering-hints-tradeoff.md",{"path":75,"resourceType":66,"resourceTypeDesc":67,"sizeBytes":76,"sourceHtmlUrl":77},"evals\u002Fbaselines\u002Fmissing-s3-bucket.md",663,"https:\u002F\u002Fgithub.com\u002Fjhkchan\u002Faws-agent-skills\u002Fblob\u002Fmain\u002Fskills\u002Ffirehose-delivery-stream-deployer\u002Fevals\u002Fbaselines\u002Fmissing-s3-bucket.md",{"path":79,"resourceType":66,"resourceTypeDesc":67,"sizeBytes":80,"sourceHtmlUrl":81},"evals\u002Fbaselines\u002Fopensearch-destination-retry-backup.md",649,"https:\u002F\u002Fgithub.com\u002Fjhkchan\u002Faws-agent-skills\u002Fblob\u002Fmain\u002Fskills\u002Ffirehose-delivery-stream-deployer\u002Fevals\u002Fbaselines\u002Fopensearch-destination-retry-backup.md",{"path":83,"resourceType":66,"resourceTypeDesc":67,"sizeBytes":84,"sourceHtmlUrl":85},"evals\u002Fbaselines\u002Fs3-parquet-lambda-transform.md",691,"https:\u002F\u002Fgithub.com\u002Fjhkchan\u002Faws-agent-skills\u002Fblob\u002Fmain\u002Fskills\u002Ffirehose-delivery-stream-deployer\u002Fevals\u002Fbaselines\u002Fs3-parquet-lambda-transform.md",{"path":87,"resourceType":66,"resourceTypeDesc":67,"sizeBytes":88,"sourceHtmlUrl":89},"evals\u002Fbaselines\u002Fsplunk-http-endpoint.md",637,"https:\u002F\u002Fgithub.com\u002Fjhkchan\u002Faws-agent-skills\u002Fblob\u002Fmain\u002Fskills\u002Ffirehose-delivery-stream-deployer\u002Fevals\u002Fbaselines\u002Fsplunk-http-endpoint.md",{"path":91,"resourceType":66,"resourceTypeDesc":67,"sizeBytes":92,"sourceHtmlUrl":93},"evals\u002Feval_config.yaml",1808,"https:\u002F\u002Fgithub.com\u002Fjhkchan\u002Faws-agent-skills\u002Fblob\u002Fmain\u002Fskills\u002Ffirehose-delivery-stream-deployer\u002Fevals\u002Feval_config.yaml",{"path":95,"resourceType":66,"resourceTypeDesc":67,"sizeBytes":96,"sourceHtmlUrl":97},"evals\u002Fevals.json",10049,"https:\u002F\u002Fgithub.com\u002Fjhkchan\u002Faws-agent-skills\u002Fblob\u002Fmain\u002Fskills\u002Ffirehose-delivery-stream-deployer\u002Fevals\u002Fevals.json",{"path":99,"resourceType":66,"resourceTypeDesc":67,"sizeBytes":100,"sourceHtmlUrl":101},"evals\u002Fprompts\u002Fbuffering-hints-tradeoff.md",688,"https:\u002F\u002Fgithub.com\u002Fjhkchan\u002Faws-agent-skills\u002Fblob\u002Fmain\u002Fskills\u002Ffirehose-delivery-stream-deployer\u002Fevals\u002Fprompts\u002Fbuffering-hints-tradeoff.md",{"path":103,"resourceType":66,"resourceTypeDesc":67,"sizeBytes":104,"sourceHtmlUrl":105},"evals\u002Fprompts\u002Fmissing-s3-bucket.md",414,"https:\u002F\u002Fgithub.com\u002Fjhkchan\u002Faws-agent-skills\u002Fblob\u002Fmain\u002Fskills\u002Ffirehose-delivery-stream-deployer\u002Fevals\u002Fprompts\u002Fmissing-s3-bucket.md",{"path":107,"resourceType":66,"resourceTypeDesc":67,"sizeBytes":108,"sourceHtmlUrl":109},"evals\u002Fprompts\u002Fopensearch-destination-retry-backup.md",745,"https:\u002F\u002Fgithub.com\u002Fjhkchan\u002Faws-agent-skills\u002Fblob\u002Fmain\u002Fskills\u002Ffirehose-delivery-stream-deployer\u002Fevals\u002Fprompts\u002Fopensearch-destination-retry-backup.md",{"path":111,"resourceType":66,"resourceTypeDesc":67,"sizeBytes":112,"sourceHtmlUrl":113},"evals\u002Fprompts\u002Fs3-parquet-lambda-transform.md",987,"https:\u002F\u002Fgithub.com\u002Fjhkchan\u002Faws-agent-skills\u002Fblob\u002Fmain\u002Fskills\u002Ffirehose-delivery-stream-deployer\u002Fevals\u002Fprompts\u002Fs3-parquet-lambda-transform.md",{"path":115,"resourceType":66,"resourceTypeDesc":67,"sizeBytes":116,"sourceHtmlUrl":117},"evals\u002Fprompts\u002Fsplunk-http-endpoint.md",565,"https:\u002F\u002Fgithub.com\u002Fjhkchan\u002Faws-agent-skills\u002Fblob\u002Fmain\u002Fskills\u002Ffirehose-delivery-stream-deployer\u002Fevals\u002Fprompts\u002Fsplunk-http-endpoint.md",{"path":119,"resourceType":66,"resourceTypeDesc":67,"sizeBytes":120,"sourceHtmlUrl":121},"examples\u002FREADME.md",7980,"https:\u002F\u002Fgithub.com\u002Fjhkchan\u002Faws-agent-skills\u002Fblob\u002Fmain\u002Fskills\u002Ffirehose-delivery-stream-deployer\u002Fexamples\u002FREADME.md",{"path":123,"resourceType":66,"resourceTypeDesc":67,"sizeBytes":124,"sourceHtmlUrl":125},"references\u002Fadvanced-patterns.md",5159,"https:\u002F\u002Fgithub.com\u002Fjhkchan\u002Faws-agent-skills\u002Fblob\u002Fmain\u002Fskills\u002Ffirehose-delivery-stream-deployer\u002Freferences\u002Fadvanced-patterns.md",{"path":127,"resourceType":66,"resourceTypeDesc":67,"sizeBytes":128,"sourceHtmlUrl":129},"references\u002Fdestinations-and-buffering.md",11262,"https:\u002F\u002Fgithub.com\u002Fjhkchan\u002Faws-agent-skills\u002Fblob\u002Fmain\u002Fskills\u002Ffirehose-delivery-stream-deployer\u002Freferences\u002Fdestinations-and-buffering.md",{"path":131,"resourceType":66,"resourceTypeDesc":67,"sizeBytes":132,"sourceHtmlUrl":133},"references\u002Ferror-handling.md",1096,"https:\u002F\u002Fgithub.com\u002Fjhkchan\u002Faws-agent-skills\u002Fblob\u002Fmain\u002Fskills\u002Ffirehose-delivery-stream-deployer\u002Freferences\u002Ferror-handling.md",{"path":135,"resourceType":66,"resourceTypeDesc":67,"sizeBytes":136,"sourceHtmlUrl":137},"references\u002Ftransformation-and-encryption.md",10473,"https:\u002F\u002Fgithub.com\u002Fjhkchan\u002Faws-agent-skills\u002Fblob\u002Fmain\u002Fskills\u002Ffirehose-delivery-stream-deployer\u002Freferences\u002Ftransformation-and-encryption.md",{"path":139,"resourceType":66,"resourceTypeDesc":67,"sizeBytes":140,"sourceHtmlUrl":141},"skill.usf.yaml",4430,"https:\u002F\u002Fgithub.com\u002Fjhkchan\u002Faws-agent-skills\u002Fblob\u002Fmain\u002Fskills\u002Ffirehose-delivery-stream-deployer\u002Fskill.usf.yaml",[143,150,157,164,171,178],{"skillNo":144,"name":145,"nameZh":6,"description":146,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":147,"categories":148,"keywords":149,"publishedAt":19},"01M7ADVD60X4PCAPKSE58P9NDC","xray-tracing-deployer","Deploys AWS X-Ray distributed tracing with production-grade configuration: X-Ray daemon deployment (EC2 systemd, ECS sidecar, EKS DaemonSet, Lambda built-in), SDK instrumentation (Java, Python, Node.js, Go, .NET) with annotations and metadata, sampling rules (default reservoir + rate vs custom rules with URL \u002F method \u002F service-name predicates), service map generation, trace query and retrieval, groups and insights, CloudWatch ServiceLens integration, IAM permissions (PutTraceSegments, GetSamplingRules), and latest features (Lambda Powertools tracing, OpenTelemetry \u002F ADOT Collector support, W3C trace context). Emits a READY_TO_DEPLOY checklist with every configuration item verified. Use when enabling X-Ray on a new application, adding tracing to ECS \u002F EKS \u002F Lambda, validating a sampling configuration, or generating daemon deployment CLI and IaC templates. Triggers: X-Ray, distributed tracing, X-Ray daemon, sampling rules, service map, X-Ray SDK, OpenTelemetry, ADOT, Lambda Powertools, CloudWatch ServiceLens.","https:\u002F\u002Fgithub.com\u002Fjhkchan\u002Faws-agent-skills\u002Fblob\u002Fmain\u002Fskills\u002Fxray-tracing-deployer\u002FSKILL.md",[],[],{"skillNo":151,"name":152,"nameZh":6,"description":153,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":154,"categories":155,"keywords":156,"publishedAt":19},"01M7ADTVJ7DCBM9CB1FY51GH7J","wellarchitected-workload-auditor","Audits AWS Well-Architected Tool workloads for review staleness, per-pillar high-risk issue counts, milestone tracking gaps, and remediation plan completeness. Emits a deterministic verdict (STALE_REVIEW | HIGH_RISK | CONFIG_GAP | OK) per workload with per-pillar risk breakdown, milestone trend analysis, and specific remediation. Use when reviewing Well-Architected workloads, checking WA review freshness, auditing high-risk issues per pillar, validating milestone coverage, or tracking remediation plans.","https:\u002F\u002Fgithub.com\u002Fjhkchan\u002Faws-agent-skills\u002Fblob\u002Fmain\u002Fskills\u002Fwellarchitected-workload-auditor\u002FSKILL.md",[],[],{"skillNo":158,"name":159,"nameZh":6,"description":160,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":161,"categories":162,"keywords":163,"publishedAt":19},"01M7ADT59G6FTTNZYHT2QRPNNP","wellarchitected-review-operator","Operates AWS Well-Architected Tool reviews end-to-end — creates workloads, runs pillar reviews (operational excellence, security, reliability, performance, cost optimization, sustainability, Prosperity), answers questions with risk-tier rationale, generates reports and improvement plans, creates milestones, and integrates with Trusted Advisor and Well-Architected Labs. Runs deterministic pre-checks (workload existence, lens availability, pillar coverage), emits the exact update-answer \u002F get-consolidated-report CLI behind a CONFIRM gate, and verifies risk-tier transitions. Emits READY | BLOCKED | COMPLETED. Use when creating a workload, answering pillar questions, generating an improvement plan, creating a milestone, or integrating with Trusted Advisor. Triggers: well-architected tool, well architected review, workload review, pillar, operational excellence, security, reliability, performance, cost optimization, sustainability, prosperity, improvement plan, milestone, trusted advisor.","https:\u002F\u002Fgithub.com\u002Fjhkchan\u002Faws-agent-skills\u002Fblob\u002Fmain\u002Fskills\u002Fwellarchitected-review-operator\u002FSKILL.md",[],[],{"skillNo":165,"name":166,"nameZh":6,"description":167,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":168,"categories":169,"keywords":170,"publishedAt":19},"01M7ADSH5JA9QHNPPHQCD7EVA7","wafv2-web-acl-deployer","Provisions AWS WAFv2 Web ACLs with correct production defaults: scope selection (CLOUDFRONT vs REGIONAL), managed rule groups (AWSManagedRulesCommonRuleSet, SQLiRuleSet, LinuxRuleSet, WindowsRuleSet, IPReputationList, BotControlRuleSet, AmazonIpReputationList, ATP), custom rules (byte match, IP set, regex pattern, geo match, size constraint, rate-based), rule priority ordering, logging destinations (Kinesis Firehose, CloudWatch Logs, S3), resource association (ALB, API Gateway, CloudFront), rate-based rules with aggregate keys (IP, forwarded IP), Challenge and CAPTCHA actions, and account takeover prevention. Emits a READY_TO_DEPLOY checklist. Use when creating a Web ACL, attaching managed rules, configuring rate limiting, enabling WAF logging, associating a Web ACL with an ALB\u002FAPI Gateway\u002FCloudFront, or configuring CAPTCHA\u002FChallenge actions. Triggers: create WAF Web ACL, WAFv2, managed rule groups, WAF rate limiting, WAF logging, WAF CAPTCHA, WAF Challenge action, bot control, ATP managed rules.","https:\u002F\u002Fgithub.com\u002Fjhkchan\u002Faws-agent-skills\u002Fblob\u002Fmain\u002Fskills\u002Fwafv2-web-acl-deployer\u002FSKILL.md",[],[],{"skillNo":172,"name":173,"nameZh":6,"description":174,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":175,"categories":176,"keywords":177,"publishedAt":19},"01M7ADRYFAT8R73FBNCHRC573X","wafv2-web-acl-auditor","Audits AWS WAFv2 Web ACL configurations to determine whether the ACL provides effective protection: default-action posture (Allow vs Block), managed-rule-group coverage gaps, rule effectiveness (BLOCK vs COUNT, shadow\u002Fbypass rules, stale exclusions), rate-based rule correctness, logging and visibility configuration, and text-transformation bypass vectors. Emits a deterministic verdict per Web ACL. Use when reviewing a WAFv2 Web ACL for security posture, checking managed-rule coverage, validating rate-limit configuration, auditing WAF logging, or hardening a WAF before production deployment. Triggers: WAF, WAFv2, Web ACL, managed rule group, AWSManagedRules, rate-based rule, CAPTCHA, Challenge, OverrideAction, Count mode, forwarded IP, Bot Control, ATP, text transformation, WAF logging, WAF visibility, OWASP, SQLi, XSS, RFI.","https:\u002F\u002Fgithub.com\u002Fjhkchan\u002Faws-agent-skills\u002Fblob\u002Fmain\u002Fskills\u002Fwafv2-web-acl-auditor\u002FSKILL.md",[],[],{"skillNo":179,"name":180,"nameZh":6,"description":181,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":182,"categories":183,"keywords":184,"publishedAt":19},"01M7ADR8W7G72JX4KFGN03QK8Q","waf-rule-deployer","Provisions AWS WAFv2 rule sets with production defaults: Web ACL creation (CloudFront scope in us-east-1 vs Regional scope), rule groups (managed vs custom) with capacity planning in WCU (Web ACL Capacity Units, 1500 default per ACL), managed rule groups (AWSManagedRulesCommonRuleSet, AWSManagedRulesSQLiRuleSet, AWSManagedRulesAmazonIpReputationList, AWSManagedRulesLinuxRuleSet, AWSManagedRulesWindowsRuleSet, AWSManagedRulesUnixRuleSet), custom rules (byte-match, regex pattern set, geo-match, rate-based), IP set and regex pattern set resources, rule priority ordering (lowest number evaluated first), action types (allow, block, count, CAPTCHA, Challenge), bot control managed rule group, account takeover. Triggers: create WAF rule, WAF managed rule group, WAF custom rule, WAF byte match, WAF geo match, WAF rate based rule, WAF IP set, WAF regex pattern set, WAF WCU budget, WAF CloudFront ACL, WAF logging Kinesis Firehose, WAF bot control, WAF ATP, WAF label matching, WAF Challenge action, WAF CAPTCHA.","https:\u002F\u002Fgithub.com\u002Fjhkchan\u002Faws-agent-skills\u002Fblob\u002Fmain\u002Fskills\u002Fwaf-rule-deployer\u002FSKILL.md",[],[]]