datasync-task-operator
Operates AWS DataSync task end-to-end — agent deployment (on-prem VM, EC2), location configuration for every supported source (NFS, SMB, S3, HDFS, Object Storage) and destination (S3, EFS, FSx for Windows, FSx for Lustre, FSx for OpenZFS, FSx for NetApp ONTAP, Snowball Edge), task creation with the full Options surface (VerifyMode POINT_IN_TIME_CONSISTENT | ONLY_FILES_TRANSFERRED | NONE, OverwriteMode ALWAYS | NEVER, PosixPermissions PRESERVE | PRESERVE_TRY | BEST_EFFORT | NONE, Acl PRESERVE | NONE, TransferMode CHANGED | ALL, Gid PRESERVE | NONE, Uid PRESERVE | NONE, Mtime PRESERVE | NONE, SecurityDescriptorCopyFlags OWNER_DACL_SACL | OWNER_DACL | NONE, TaskQueueing ENABLED | DISABLED), scheduling (cron-like ScheduleExpression with up to 1-year horizon), bandwidth throttling (BandwidthLimitInMb), Task Reports with report-level and report-code filters, DataSync Discovery for on-prem capacity assessment, and FSx for NetApp ONTAP destination support. Runs deterministic pre-checks behind a CONFIRM gate and.
- License 许可
- Apache-2.0
- 最近更新
- 2026-09-24
- Stars
- ★ 4
- Forks
- 0
01 前言 Front Matter
- name
- datasync-task-operator
- license
- Apache-2.0
- metadata
- { "tags": "aws, datasync, storage, migration, nfs, smb, hdfs, s3, efs, fsx, operate", "phase": "4", "author": "Jacky Chan — AWS Community Builder", "domain": "aws-cloudops", "family": "Storage", "version": "0.1.0", "keywords": "AWS DataSync, DataSync task, DataSync agent, DataSync Discovery, NFS migration, SMB migration, HDFS migration, Object Storage migration, S3 transfer, EFS transfer, FSx transfer, FSx for NetApp ONTAP, verify data integrity, POINT_IN_TIME_CONSISTENT, ONLY_FILES_TRANSFERRED, bandwidth throttling, BandwidthLimitInMb, task schedule, Task Reports, overwrite mode, PosixPermissions preserve, on-prem to AWS migration, Snowball Edge, activation key", "task_type": "operate", "complexity": "high", "entry_point": "false", "skill_class": "capability", "when_to_use": "Creating or modifying a DataSync task, deploying or activating a DataSync agent (on-prem VM or EC2), configuring source/destination locations for NFS, SMB, S3, HDFS, Object Storage, EFS, or FSx family destinations, setting task Options (VerifyMode, OverwriteMode, TransferMode, PosixPermissions, ACLs), creating a task Schedule with bandwidth throttling, diagnosing a failed or partial task execution, configuring Task Reports, planning an on-prem assessment with DataSync Discovery, or transferring to FSx for NetApp ONTAP.", "requires_llm": "true", "verdict_shape": "READY | BLOCKED | COMPLETED", "lifecycle_status": "active", "invocation_schema": "Input: either (a) a DataSync task configuration (describe-task, describe-task-execution, describe-agent, list-locations) plus the intended operation (create-task, deploy-agent, update-task-options, create-schedule, diagnose-failing-execution, run-discovery, transfer-to-fsx-ontap, configure-task-reports), OR (b) a source + destination location pair for live-account execution. Output: deterministic OPERATION / VERDICT / PRE_CHECKS / STEPS / POST_VERIFY / NOTES block per operation, where VERDICT is one of READY, BLOCKED, COMPLETED.", "supports_pipeline": "true", "activation_triggers": "create DataSync task, deploy DataSync agent, activate DataSync agent, DataSync NFS to S3, DataSync SMB to FSx, DataSync HDFS migration, DataSync schedule, DataSync bandwidth throttle, verify data integrity DataSync, POINT_IN_TIME_CONSISTENT, ONLY_FILES_TRANSFERRED, DataSync task failing, DataSync task execution error, DataSync Discovery, DataSync FSx for NetApp ONTAP, DataSync Task Reports, DataSync Snowball Edge, DataSync overwrite mode" }
- description
- Operates AWS DataSync task end-to-end — agent deployment (on-prem VM, EC2), location configuration for every supported source (NFS, SMB, S3, HDFS, Object Storage) and destination (S3, EFS, FSx for Windows, FSx for Lustre, FSx for OpenZFS, FSx for NetApp ONTAP, Snowball Edge), task creation with the full Options surface (VerifyMode POINT_IN_TIME_CONSISTENT | ONLY_FILES_TRANSFERRED | NONE, OverwriteMode ALWAYS | NEVER, PosixPermissions PRESERVE | PRESERVE_TRY | BEST_EFFORT | NONE, Acl PRESERVE | NONE, TransferMode CHANGED | ALL, Gid PRESERVE | NONE, Uid PRESERVE | NONE, Mtime PRESERVE | NONE, SecurityDescriptorCopyFlags OWNER_DACL_SACL | OWNER_DACL | NONE, TaskQueueing ENABLED | DISABLED), scheduling (cron-like ScheduleExpression with up to 1-year horizon), bandwidth throttling (BandwidthLimitInMb), Task Reports with report-level and report-code filters, DataSync Discovery for on-prem capacity assessment, and FSx for NetApp ONTAP destination support. Runs deterministic pre-checks behind a CONFIRM gate and.
- compatibility
- Agent runtime that reads SKILL.md (Claude Code, Cursor, Windsurf, Codex, Gemini). No AWS CLI required for offline plan classification. Live-account operations use aws datasync create-agent, create-location-nfs | create-location-smb | create-location-s3 | create-location-hdfs | create-location-object-storage | create-location-efs | create-location-fsx-windows | create-location-fsx-lustre | create-location-fsx-openzfs | create-location-fsx-ontap, create-task, update-task, start-task-execution...
02 资源文件 21
- 参考文档eval/test-cases.yaml10.1 KB
- 参考文档evals/baselines/create-task-nfs-to-s3-verify-ready.md792 B
- 参考文档evals/baselines/deploy-agent-onprem-activation-expired-blocked.md552 B
- 参考文档evals/baselines/diagnose-failing-execution-access-denied-blocked.md684 B
- 参考文档evals/baselines/fsx-ontap-verify-completed.md595 B
- 参考文档evals/baselines/schedule-bandwidth-throttle-ready.md683 B
- 参考文档evals/eval_config.yaml1.9 KB
- 参考文档evals/evals.json4.4 KB
- 参考文档evals/prompts/create-task-nfs-to-s3-verify-ready.md1.8 KB
- 参考文档evals/prompts/deploy-agent-onprem-activation-expired-blocked.md949 B
- 参考文档evals/prompts/diagnose-failing-execution-access-denied-blocked.md1.6 KB
- 参考文档evals/prompts/fsx-ontap-verify-completed.md1.5 KB
- 参考文档evals/prompts/schedule-bandwidth-throttle-ready.md769 B
- 参考文档examples/README.md9.2 KB
- 参考文档references/advanced-patterns.md12.0 KB
- 参考文档references/agent-deployment-and-discovery.md8.0 KB
- 参考文档references/diagnostic-commands.md3.6 KB
- 参考文档references/error-handling.md2.0 KB
- 参考文档references/task-options-and-failure-modes.md12.8 KB
- 参考文档references/worked-examples.md3.9 KB
- 参考文档skill.usf.yaml4.6 KB
03 SKILL.md 正文
AWS DataSync Task Operator
What this skill does
Executes AWS DataSync operations correctly and safely across the full
source/destination matrix. Runs deterministic pre-checks before any
state-changing CLI — agent activation key validity and reachability,
source location readability, destination location writeability, IAM
task role trust + permissions chain (including the KMS encrypt path
for SSE-KMS destinations, the cross-account bucket policy for
destinations in another account, and the EFS/FSx file system policy
allowing the DataSync service principal), task Options consistency
(e.g. PosixPermissions PRESERVE requires a destination that
preserves POSIX metadata; SMB sources need
SecurityDescriptorCopyFlags configured explicitly), schedule
expression validity (max 1-year horizon, required for
update-task-schedule), and bandwidth-throttle feasibility against
the agent's provisioned uplink. Executes the task change behind a
CONFIRM gate, then verifies the result by tailing
describe-task-execution until Status reaches SUCCESS (with
BytesTransferred, BytesWritten, EstimatedFilesToTransfer,
FilesTransferred, VerifiedFilesCount, TransferDurationBytesPerSecond
surfaced), and for VerifyMode != NONE confirming the verify pass
produced zero files failing the integrity check.
Quick navigation
| Section | What it covers | When to read it |
|---|---|---|
| § Quick reference | Verdict thresholds (BLOCKED/READY/COMPLETED) + pre-check priority | Before any operation |
| § Mindset | Why DataSync is an agent-mediated async pipeline; the false-green-task trap; the verify-mode contract | Understanding the safety model |
| § Pre-flight | Source + destination metadata gate — agent reachability, location readability, IAM role chain | Before executing any CLI |
| § Process | Per-operation planning: create-task, deploy-agent, update-options, create-schedule, diagnose-failing-execution, run-discovery, transfer-to-fsx-ontap, configure-task-reports | When choosing which operation to run |
| § Output format | STRICT output contract — OPERATION/VERDICT/PRE_CHECKS/STEPS/POST_VERIFY/NOTES template | Formatting the response |
| § Anti-Patterns | NEVER list — common mistakes that strand data or break transfers silently | Review before risky operations |
| § Pre-flight safety | Capture pre-state, IAM role policy, KMS key policy, destination bucket policy | Defense-in-depth |
| § Expert heuristic | DataSync is async + eventually-consistent — never assume success from the API alone | Avoiding the false-green trap |
STRICT output contract
EVERY response MUST end with a single fenced text block in this exact shape (the operator's downstream tooling greps for it). No deviations:
OPERATION: <create-task | deploy-agent | update-task-options | create-schedule | diagnose-failing-execution | run-discovery | transfer-to-fsx-ontap | configure-task-reports>
VERDICT: READY | BLOCKED | COMPLETED
TARGET: <source-location-uri> -> <destination-location-uri> (task: <id-or-"new">)
PRE_CHECKS:
- [PASS] <check description>
- [FAIL] <check description> — <reason>
STEPS:
1. <CLI command with flags populated, or "(none — pre-checks failed)">
2. <wait / monitoring command>
POST_VERIFY:
- [PASS] <verification description> | (pending execution)
- [FAIL] <verification description> — <reason>
NOTES: <schedule, monitoring, caveats>
Rules of the contract:
- VERDICT is exactly one of
READY,BLOCKED,COMPLETED. No other values.READYmeans pre-checks passed and a CONFIRM gate is pending;BLOCKEDmeans at least one pre-check failed — do NOT emit STEPS that mutate state;COMPLETEDmeans post-verification passed after execution. - If PRE_CHECKS has any
[FAIL], VERDICT MUST beBLOCKEDand STEPS MUST be(none — pre-checks failed). - If VERDICT is
READY, STEPS[1] MUST be the CONFIRM prompt and STEPS[2] MUST be the actual CLI. - If VERDICT is
COMPLETED, POST_VERIFY MUST contain at least one[PASS]and no[FAIL]. - Never wrap the block in JSON, never abbreviate the field names,
never omit a section. If a section is empty, write
(none).
Quick reference — verdict thresholds
| Verdict | Trigger condition | Action |
|---|---|---|
BLOCKED |
One or more pre-checks failed (agent activation key missing/failed, source unreachable from agent, destination write-denied, IAM task role missing source-read or destination-write, KMS encrypt gap on SSE-KMS destination, cross-account destination bucket policy missing DataSync principal, Options mismatch like PosixPermissions PRESERVE on SMB destination, ScheduleExpression past 1-year horizon, bandwidth-throttle value exceeds agent uplink, destination EFS/FSx in WRONG AZ vs agent subnet) |
List failures, do NOT execute |
READY |
All pre-checks passed; awaiting CONFIRM gate | Emit exact CLI sequence, wait for operator yes |
COMPLETED |
Task execution reached Status: SUCCESS AND post-verification passed (FilesTransferred == EstimatedFilesToTransfer, VerificationFilesFailed == 0 when VerifyMode != NONE, destination spot-check object exists with expected size and mtime) |
Emit verification results, monitoring plan |
Priority order for pre-checks (apply in this sequence, all must pass for READY):
- Agent activation + reachability. Agent ARN exists,
LastConnectionTimerecent (< 5 min),Status: ONLINE. - Source location readable by agent. Self-managed sources
(NFS/SMB/HDFS/Object Storage): agent subnet/SG can reach source
server's IP/FQDN on the right port (2049 NFS, 445 SMB, 8020/50070
HDFS NameNode, 443/80 object storage). AWS sources (S3): IAM role
has
s3:GetObject+s3:ListBucket. - Destination location writable. S3: IAM role has
s3:PutObject,s3:ListBucketMultipartUploads,s3:AbortMultipartUpload,s3:GetObject(for verify). EFS/FSx: file-system-specific write permissions and file system policy allows DataSync service principal. - IAM task role trust + permissions. Role trusts
datasync.amazonaws.comand has the read-on-source + write-on-destination chain. Cross-account destinations need the destination bucket policy to allow the source account's role. - KMS encrypt path on destination. SSE-KMS destinations require
kms:Encrypt+kms:GenerateDataKeyon the destination key AND the destination key policy must grant the role. - Options consistency.
PosixPermissions: PRESERVErequires a POSIX-preserving destination (EFS, FSx for OpenZFS, FSx for Lustre, S3 with bucket-owner-enforced). SMB sources needSecurityDescriptorCopyFlagsset. SMB destinations cannot preserve POSIX. - Schedule expression validity.
ScheduleExpressionis a validat(...)orrate(...)within the 1-year task-schedule horizon.ScheduleStatus: ENABLEDrequires the expression be set. - Bandwidth-throttle feasibility.
BandwidthLimitInMbdoes not exceed the agent's provisioned uplink (typical VM agent ~10 Gbps; EC2 agents bounded by instance-type network caps). - Destination file system state. EFS:
available, notupdating. FSx:AVAILABLE, agent subnet has a mount target or reachable route. - Task Reports configuration. If reports requested,
S3BucketArnfor reports exists and task role hass3:PutObjecton the report bucket.
Cost/time baselines (per-agent throughput ~10 Gbps, verify-pass overhead 5-15%, $0.0125/GB pricing, 14/28-day discovery windows) moved verbatim to references/advanced-patterns.md. Load on demand when estimating transfer duration and cost.
Mindset
Mindset framing (agent-mediated async pipeline, VerifyMode as a separate pass, destination must be writable AS the DataSync service) moved verbatim to references/advanced-patterns.md. Load on demand to internalize why execution status — not task status — proves delivery.
Pre-flight: source + destination metadata gate
Run before classification. Misclassifying these produces wrong plans.
Pagination: list-locations paginates at 100.
list-task-executions paginates at 100 by default, sorted descending
by StartTime. describe-task-execution returns the full execution
detail.
Live-account pre-flight listing (list-agents, list-locations, S3/EFS/FSx describes, IAM policy checks, KMS describe-key, describe-task, describe-task-execution, list-discovery-jobs) moved verbatim to references/diagnostic-commands.md. Load on demand before planning against a live account.
Malformed input: if the input JSON is invalid or missing required
fields, emit VERDICT: ERROR with REASON: Task/location configuration is not valid JSON or is missing required fields — cannot plan. and
REMEDIATION: Re-fetch with aws datasync describe-task --task-arn <task> --output json and re-plan.
| Location attribute | Effect on operation |
|---|---|
Agent Status: OFFLINE |
create-task BLOCKED. Existing tasks fail execution. |
Agent LastConnectionTime > 5 min ago |
Activation may have expired or network is broken. |
Source LocationType: NFS but server unreachable from agent subnet |
BLOCKED; verify security group + route table. |
SMB source missing SMBUser/SMBPassword secret ARN |
BLOCKED; create AWS Secrets Manager secret and re-create location. |
HDFS source missing NameNodes list |
BLOCKED; re-create create-location-hdfs with valid NameNodes. |
Object Storage missing ServerHostname/BucketName/AccessKey/SecretKey |
BLOCKED; re-create with valid endpoint. |
S3 destination SSE-KMS but role missing kms:GenerateDataKey |
BLOCKED; attach KMS permissions to task role. |
Destination EFS LifeCycle: DELETING |
BLOCKED; pick a new destination. |
Destination FSx Lifecycle: MISCONFIGURED |
BLOCKED; resolve FSx config first. |
| Destination FSx for NetApp ONTAP SVM not reachable via agent subnet | BLOCKED; verify mount target or VPC peering. |
Destination S3 ObjectOwnership: ObjectWriter (legacy) with cross-account task |
Replicas owned by source account; update to BucketOwnerEnforced. |
Options.PosixPermissions: PRESERVE on SMB destination |
BLOCKED; SMB does not preserve POSIX. Use BEST_EFFORT or NONE. |
Options.Atime: BEST_EFFORT on S3 destination |
S3 does not track atime; field silently ignored. |
at(...) past 1 year |
BLOCKED; schedule horizon is 1 year. |
BandwidthLimitInMb: 50000 exceeds agent uplink |
BLOCKED; throttle must be <= agent uplink capacity. |
Task Status: RUNNING with new start-task-execution |
BLOCKED; only one execution per task concurrently. Queue or wait. |
VerifyMode: NONE on compliance migration |
Surface as a WARNING; never silently allow. |
TransferMode: ALL on a delta sync task |
Forces re-copy of all matching objects; cost/time impact. |
Process — operation planning (apply in order)
Step 0: Expert knowledge — non-obvious DataSync behaviors
Step 0 expert knowledge (agent mandatory for self-managed sources, VerifyMode re-read semantics, TransferMode delta behavior, OverwriteMode interplay, PosixPermissions per destination, SMB Secrets Manager requirement, per-agent bandwidth, 1-year schedule auto-disable, EFS same-AZ, FSx ONTAP SVM, read-only Discovery, Task Reports, single in-flight execution, DeleteOnDelete, non-persistent start overrides) moved verbatim to references/advanced-patterns.md. Load on demand — each behavior changes a plan if ignored.
Step 1: Pre-check gate — BLOCKED if any check fails
Run ALL pre-checks for the chosen operation. If ANY fails, the verdict is BLOCKED with the failed checks in PRE_CHECKS. Do NOT execute.
For ALL operations:
- Task ARN exists (or operator is creating new) in the expected account/Region.
- Agent ARN exists,
Status: ONLINE,LastConnectionTime< 5 min ago (skip for S3-to-S3 in same Region without agent). - Source and destination locations exist and match the operator's intended URIs.
- Task IAM role trusts
datasync.amazonaws.comand has the read-on-source + write-on-destination permission chain.
For create-task (new create-task):
5. Source location LocationArn exists, LocationType matches
operator's claim.
6. Destination location LocationArn exists, LocationType matches.
7. IAM task role has source-side read (e.g., s3:GetObject for S3
source, agent-credential-derived access for self-managed).
8. IAM task role has destination-side write (e.g., s3:PutObject
for S3, elasticfilesystem:ClientWrite for EFS).
9. (SSE-KMS destination) IAM role has kms:Encrypt,
kms:GenerateDataKey; destination key policy grants the role.
10. (Cross-account S3 destination) Destination bucket policy allows
the source account's task role.
11. Options consistency: PosixPermissions: PRESERVE only on POSIX
destinations; SecurityDescriptorCopyFlags set for SMB;
VerifyMode != NONE for compliance migrations.
12. (Task Reports) Report bucket exists and the task role has
s3:PutObject on it.
For deploy-agent: activation key non-empty and < 24h old; agent
VM/EC2 has outbound HTTPS (443) to DataSync endpoints; agent
subnet/SG can reach source location or destination VPC; (cross-Region
or cross-account) VPC peering/TGW/PrivateLink to destination subnet;
IAM role for the agent has datasync:* and source-reading
permissions.
For update-task-options: task Status: AVAILABLE (not RUNNING);
new Options block is internally consistent (see Step 0); pre-state
captured (full prior Options).
For create-schedule: task ARN exists; ScheduleExpression is
valid EventBridge rate/at syntax within 1-year horizon;
(bandwidth throttle) BandwidthLimitInMb <= agent uplink.
For diagnose-failing-execution (read-only, no BLOCKED gate): read
the failure-mode table to identify the root cause from ErrorCode +
ErrorDetail.
For run-discovery: on-prem storage reachable from the discovery agent; no in-flight discovery job covers the same storage system; collection window (14 or 28 days) set explicitly.
For transfer-to-fsx-ontap: FSx for NetApp ONTAP file system
AVAILABLE; SVM exists and inter-cluster endpoint reachable from
agent; agent subnet has route to SVM endpoint; task role has
fsx:CreateMountTarget, fsx:DescribeFileSystems, plus the
ONTAP-specific permissions.
For configure-task-reports: reports S3 bucket exists in same
Region; task role has s3:PutObject on the reports bucket;
ReportLevel is one of ERRORS_ONLY, SUCCESSES_AND_ERRORS.
Failure-mode table (agent offline, s3:PutObject denied, mount target AZ mismatch, SMB login failed, KMS key policy, VerificationFilesFailed, flat BytesTransferred, skipped files, LAUNCHING > 30 min) moved verbatim to references/error-handling.md. Load on demand when diagnosing a failing execution from ErrorCode + ErrorDetail.
Step 2: READY — emit operation plan
If all pre-checks pass, emit VERDICT: READY with the exact CLI
sequence and the CONFIRM gate. The plan includes:
- The exact AWS CLI command with all flags populated from the source
- destination configuration. For
create-taskthis includes the full--optionsJSON and the--cloud-watch-log-group-arnif specified.
- destination configuration. For
- Expected throughput (10 Gbps ceiling for a single VM agent; c5.2xlarge ~2 Gbps typical).
- Expected verify-mode overhead (5-15% extra wall time for POINT_IN_TIME_CONSISTENT).
- The CONFIRM gate prompt.
- The monitoring step (describe-task-execution poll until Status reaches SUCCESS or ERROR).
Step 3: Execute behind CONFIRM gate
- MANDATORY CONFIRMATION GATE. Before any state-changing CLI
(
create-task,update-task,start-task-execution,update-task-schedule,create-agent,delete-task), emit:CONFIRM: About to <operation> on task <task> in account <account> region <region>. This will <consequence>. Proceed? (yes/no). Do NOT execute until the operator confirms. - Capture pre-state for rollback:
aws datasync describe-task --task-arn <task> --output json > /tmp/<task>-pre-$(date +%s).json. - Execute the CLI. For
start-task-execution, the API returns aTaskExecutionArnimmediately; the transfer runs asynchronously on the agent. - Poll the execution:
aws datasync describe-task-execution --task-execution-arn <arn>every 60 seconds (or longer for big transfers). SurfaceBytesTransferred,BytesWritten,EstimatedFilesToTransfer,FilesTransferred,VerificationFilesFailed.
Step 4: Post-verification — COMPLETED
After execution reaches Status: SUCCESS, run post-verification. ALL
checks must pass for COMPLETED.
- Confirm
FilesTransferred == EstimatedFilesToTransfer(unlessIncludes/Excludesfilters intentionally subset the scope). - For
VerifyMode != NONE: confirmVerificationFilesFailed == 0andVerificationFilesTransferred == FilesTransferred. - Spot-check 3 random files on the destination via
head-object/lson the file system. Compare size and mtime to source. - For S3 destination: confirm
ETagmatches (or differs only by multipart-ETag format). - For EFS/FSx destination: confirm POSIX permissions preserved when
PosixPermissions: PRESERVEwas set. - For scheduled tasks: confirm next-run
NextRunTimeis set underSchedule. - For Task Reports: confirm the report object exists at
<report-bucket>/<task-id>/<execution-id>/reports/.
If ANY verification fails, emit VERDICT: ERROR with the failure
details — do not claim COMPLETED. Route the failure back to
diagnose-failing-execution.
Output format (per operation)
See § STRICT output contract above. The block MUST appear exactly in that shape.
Worked example — create-task (NFS -> S3, VERIFY + schedule)
OPERATION: create-task
VERDICT: READY
TARGET: nfs://10.0.10.20/vol/data -> s3://prod-migration-archive-2026
(task: new — "nfs-to-s3-archive")
PRE_CHECKS:
- [PASS] Agent arn:aws:datasync:us-east-1:111111111111:agent/agent-001
Status: ONLINE, LastConnectionTime: 2026-08-10T22:00:00Z
- [PASS] Source location LocationType: NFS, URI: nfs://10.0.10.20/vol/data
- [PASS] Destination location LocationType: S3, URI: s3://prod-migration-archive-2026
- [PASS] IAM role arn:aws:iam::111111111111:role/datasync-task-role
trusts datasync.amazonaws.com
- [PASS] Task role has s3:PutObject, s3:GetObject, s3:ListBucket,
s3:AbortMultipartUpload on prod-migration-archive-2026
- [PASS] Task role has kms:Encrypt, kms:GenerateDataKey on
arn:aws:kms:us-east-1:111111111111:key/dest-cmk
- [PASS] Destination key policy grants role
- [PASS] Options.VerifyMode: POINT_IN_TIME_CONSISTENT
Options.PosixPermissions: PRESERVE (S3 destination stores POSIX
metadata in object metadata)
- [PASS] ScheduleExpression: rate(1 day) within 1-year horizon
- [PASS] BandwidthLimitInMb: 1000 <= agent uplink (10 Gbps)
STEPS:
1. CONFIRM: About to create DataSync task "nfs-to-s3-archive"
in account 111111111111 region us-east-1. Source:
nfs://10.0.10.20/vol/data. Destination:
s3://prod-migration-archive-2026 (SSE-KMS, key dest-cmk).
VerifyMode: POINT_IN_TIME_CONSISTENT. Schedule: daily.
Bandwidth throttle: 1000 Mb/s. First execution transfers
~12 TB. Proceed? (yes/no)
2. aws datasync create-task \
--source-location-arn arn:aws:datasync:us-east-1:111111111111:location/loc-001 \
--destination-location-arn arn:aws:datasync:us-east-1:111111111111:location/loc-042 \
--name nfs-to-s3-archive \
--cloud-watch-log-group-arn arn:aws:logs:us-east-1:111111111111:log-group:/aws/datasync/nfs-to-s3 \
--options file:///tmp/datasync-options.json \
--schedule-expression "rate(1 day)" \
--tags Key=env,Value=prod Key=pipeline,Value=migration
3. aws datasync start-task-execution --task-arn <new-task-arn>
4. aws datasync describe-task-execution --task-execution-arn <exec-arn>
POST_VERIFY: (pending execution)
NOTES:
- First execution performs a FULL transfer (TransferMode: CHANGED
with no prior baseline). Subsequent executions on the daily
schedule are delta-only.
- VerifyMode POINT_IN_TIME_CONSISTENT adds 5-15% wall time after
the transfer phase. For 12 TB expect ~24-36h on a 1 Gbps throttle.
- Task Reports NOT configured — recommend update-task with reports
bucket for compliance evidence.
- Destination KMS key policy MUST list the task role; verify with
aws kms get-key-policy --key-id dest-cmk --policy-name default
before the first execution.
Worked example — diagnose-failing-execution (BLOCKED with fix)
Worked example (diagnose-failing-execution: AccessDenied on s3:PutObject, partial 2.1TB transfer, IAM attach + TransferMode CHANGED re-run) moved verbatim to references/worked-examples.md. Load on demand for the shape of a BLOCKED diagnosis with fix.
Worked example — transfer-to-fsx-ontap (COMPLETED)
Worked example (transfer-to-fsx-ontap COMPLETED: SMB → FSx ONTAP, OWNER_DACL, 48,231 files, zero verify failures) moved verbatim to references/worked-examples.md. Load on demand for the shape of a COMPLETED FSx ONTAP transfer.
Anti-Patterns — NEVER
-
NEVER assume
Status: AVAILABLEon a DataSync task means data is transferred. The task only declares it is ready to execute; the only proof of transfer is adescribe-task-executionreturningStatus: SUCCESSandFilesTransferred == EstimatedFilesToTransfer. Always verify with an explicit execution result check. -
NEVER use
VerifyMode: NONEfor compliance-driven migrations. The speed gain is not worth the loss of integrity verification. Compliance audits requirePOINT_IN_TIME_CONSISTENTorONLY_FILES_TRANSFERRED. -
NEVER set
PosixPermissions: PRESERVEon an SMB destination. SMB does not preserve POSIX metadata; the execution will fail or silently drop the metadata. UseBEST_EFFORTorNONE, or pick a POSIX-preserving destination (EFS, FSx for OpenZFS). -
NEVER share an agent across more tasks than its throughput can sustain. Bandwidth throttle is per-agent, not per-task; scheduling three concurrent 1 Gbps tasks on a 1 Gbps-throttled agent starves them all.
-
NEVER assume a
rate(2 days)schedule survives forever. DataSync schedules auto-disable after 1 year. Re-arm annually or move the trigger to EventBridge callingstart-task-execution. -
NEVER set
BandwidthLimitInMbhigher than the agent's provisioned uplink. The cap is enforced on the agent; setting it higher does not buy throughput, it just removes the throttle. -
NEVER call
start-task-executionon a task that is alreadyStatus: RUNNING. The API errors with a "task already running" condition. UseIncludes/Excludesfilters on multiple tasks to parallelize instead. -
NEVER assume
TransferMode: CHANGEDdoes a full copy on every run. It copies everything on the first run (no baseline) but subsequent runs only copy new/modified files. For a forced re-copy, useTransferMode: ALL. -
NEVER use a single cross-account destination S3 bucket without a bucket policy explicitly granting the source account's task role. IAM identity-based policy on the task role is necessary but not sufficient.
-
NEVER forget the SMB secret.
create-location-smbrequires an AWS Secrets Manager secret ARN for the SMB password. Hard-coded passwords are not supported. -
NEVER re-create a discovery job for the same on-prem storage system while one is in flight. The two jobs will compete for metrics collection and produce inconsistent recommendations.
-
NEVER auto-execute a state-changing DataSync CLI without the CONFIRM gate.
delete-task,update-task, andstart-task-executionall have irreversible side effects. -
NEVER skip capturing pre-state before
update-taskorupdate-task-schedule. The prior Options and Schedule are overwritten with no undo. -
NEVER assume destination EFS is in the same AZ as the agent. Cross-AZ EFS access works but incurs cross-AZ charges; Single-AZ EFS requires same-AZ mount target.
-
NEVER rely solely on the task's
Statusfield for migration completeness. Readdescribe-task-executionfor every execution. -
NEVER deploy an on-prem agent without verifying outbound HTTPS (443) to DataSync endpoints. The activation key fetch succeeds locally but the agent will not be able to register with DataSync control plane.
Pre-flight safety checks (run before any remediation CLI)
Pre-flight safety checks (CONFIRM gate, pre-state capture, agent health, KMS key policies, cross-account bucket policy, additive-over-destructive, Includes/Excludes scope validation) moved verbatim to references/diagnostic-commands.md. Load on demand before executing any remediation CLI.
Expert heuristic: the false-green task
False-green-task heuristic (registration vs delivery claim, verification-techniques table, three-point verification protocol, DELIVERY_STATUS surfacing, post-deploy silent-failure detection) moved verbatim to references/advanced-patterns.md. Load on demand before trusting a task's AVAILABLE status.
Recent AWS features (2024-2026)
Recent AWS features (FSx ONTAP destinations, Discovery, Task Reports, DeleteOnDelete, Object Storage locations, per-task bandwidth override, HDFS Kerberos, Snowball Edge, multi-agent tasks, CloudWatch metrics) moved verbatim to references/advanced-patterns.md. Load on demand before recommending 2024-2026 capabilities.
References (load on demand)
- references/advanced-patterns.md — Mindset, cost/time baselines, Step 0 non-obvious behaviors, false-green heuristic, and Recent AWS features moved from SKILL.md
- references/diagnostic-commands.md — live-account pre-flight listing and pre-flight safety checks moved from SKILL.md
- references/error-handling.md — DataSync failure-mode table moved from SKILL.md
- references/worked-examples.md — secondary worked examples moved from SKILL.md
- references/agent-deployment-and-discovery.md and references/task-options-and-failure-modes.md — agent lifecycle, full Options reference, per-source-type setup, and pre-operation checklists (pre-existing)
Domain
AWS CloudOps / DataSync, on-prem-to-AWS migration, AWS-to-AWS large-scale transfer, and FSx family destination support.
AWS documentation
- DataSync User Guide — https://docs.aws.amazon.com/datasync/latest/userguide/what-is-datasync.html
- Working with AWS DataSync agents — https://docs.aws.amazon.com/datasync/latest/userguide/working-with-agents.html
- Creating a DataSync task — https://docs.aws.amazon.com/datasync/latest/userguide/create-task.html
- Task options — https://docs.aws.amazon.com/datasync/latest/userguide/api-reference.html#API_TaskOptions
- Scheduling a task — https://docs.aws.amazon.com/datasync/latest/userguide/configure-task-schedule.html
- DataSync Task Reports — https://docs.aws.amazon.com/datasync/latest/userguide/task-reports.html
- DataSync Discovery — https://docs.aws.amazon.com/datasync/latest/userguide/discovering-your-storage.html
- FSx for NetApp ONTAP as a destination — https://docs.aws.amazon.com/datasync/latest/userguide/destination-fsx-ontap.html
- DataSync on Snowball Edge — https://docs.aws.amazon.com/datasync/latest/userguide/snowball-edge.html
- DataSync CLI Reference — https://docs.aws.amazon.com/cli/latest/reference/datasync/
- DataSync API Reference — https://docs.aws.amazon.com/datasync/latest/userguide/API_Reference.html
04 同仓库更多 Skills
xray-tracing-deployer
Deploys AWS X-Ray distributed tracing with production-grade configuration: X-Ray daemon deployment (EC2 systemd, ECS sidecar, EKS DaemonSet, Lambda built-in), SDK instrumentation (Java, Python, Node.js, Go, .NET) with annotations and metadata, sampling rules (default reservoir + rate vs custom rules with URL / method / service-name predicates), service map generation, trace query and retrieval, groups and insights, CloudWatch ServiceLens integration, IAM permissions (PutTraceSegments, GetSamplingRules), and latest features (Lambda Powertools tracing, OpenTelemetry / ADOT Collector support, W3C trace context). Emits a READY_TO_DEPLOY checklist with every configuration item verified. Use when enabling X-Ray on a new application, adding tracing to ECS / EKS / Lambda, validating a sampling configuration, or generating daemon deployment CLI and IaC templates. Triggers: X-Ray, distributed tracing, X-Ray daemon, sampling rules, service map, X-Ray SDK, OpenTelemetry, ADOT, Lambda Powertools, CloudWatch ServiceLens.
wellarchitected-workload-auditor
Audits AWS Well-Architected Tool workloads for review staleness, per-pillar high-risk issue counts, milestone tracking gaps, and remediation plan completeness. Emits a deterministic verdict (STALE_REVIEW | HIGH_RISK | CONFIG_GAP | OK) per workload with per-pillar risk breakdown, milestone trend analysis, and specific remediation. Use when reviewing Well-Architected workloads, checking WA review freshness, auditing high-risk issues per pillar, validating milestone coverage, or tracking remediation plans.
wellarchitected-review-operator
Operates AWS Well-Architected Tool reviews end-to-end — creates workloads, runs pillar reviews (operational excellence, security, reliability, performance, cost optimization, sustainability, Prosperity), answers questions with risk-tier rationale, generates reports and improvement plans, creates milestones, and integrates with Trusted Advisor and Well-Architected Labs. Runs deterministic pre-checks (workload existence, lens availability, pillar coverage), emits the exact update-answer / get-consolidated-report CLI behind a CONFIRM gate, and verifies risk-tier transitions. Emits READY | BLOCKED | COMPLETED. Use when creating a workload, answering pillar questions, generating an improvement plan, creating a milestone, or integrating with Trusted Advisor. Triggers: well-architected tool, well architected review, workload review, pillar, operational excellence, security, reliability, performance, cost optimization, sustainability, prosperity, improvement plan, milestone, trusted advisor.
wafv2-web-acl-deployer
Provisions AWS WAFv2 Web ACLs with correct production defaults: scope selection (CLOUDFRONT vs REGIONAL), managed rule groups (AWSManagedRulesCommonRuleSet, SQLiRuleSet, LinuxRuleSet, WindowsRuleSet, IPReputationList, BotControlRuleSet, AmazonIpReputationList, ATP), custom rules (byte match, IP set, regex pattern, geo match, size constraint, rate-based), rule priority ordering, logging destinations (Kinesis Firehose, CloudWatch Logs, S3), resource association (ALB, API Gateway, CloudFront), rate-based rules with aggregate keys (IP, forwarded IP), Challenge and CAPTCHA actions, and account takeover prevention. Emits a READY_TO_DEPLOY checklist. Use when creating a Web ACL, attaching managed rules, configuring rate limiting, enabling WAF logging, associating a Web ACL with an ALB/API Gateway/CloudFront, or configuring CAPTCHA/Challenge actions. Triggers: create WAF Web ACL, WAFv2, managed rule groups, WAF rate limiting, WAF logging, WAF CAPTCHA, WAF Challenge action, bot control, ATP managed rules.
wafv2-web-acl-auditor
Audits AWS WAFv2 Web ACL configurations to determine whether the ACL provides effective protection: default-action posture (Allow vs Block), managed-rule-group coverage gaps, rule effectiveness (BLOCK vs COUNT, shadow/bypass rules, stale exclusions), rate-based rule correctness, logging and visibility configuration, and text-transformation bypass vectors. Emits a deterministic verdict per Web ACL. Use when reviewing a WAFv2 Web ACL for security posture, checking managed-rule coverage, validating rate-limit configuration, auditing WAF logging, or hardening a WAF before production deployment. Triggers: WAF, WAFv2, Web ACL, managed rule group, AWSManagedRules, rate-based rule, CAPTCHA, Challenge, OverrideAction, Count mode, forwarded IP, Bot Control, ATP, text transformation, WAF logging, WAF visibility, OWASP, SQLi, XSS, RFI.
waf-rule-deployer
Provisions AWS WAFv2 rule sets with production defaults: Web ACL creation (CloudFront scope in us-east-1 vs Regional scope), rule groups (managed vs custom) with capacity planning in WCU (Web ACL Capacity Units, 1500 default per ACL), managed rule groups (AWSManagedRulesCommonRuleSet, AWSManagedRulesSQLiRuleSet, AWSManagedRulesAmazonIpReputationList, AWSManagedRulesLinuxRuleSet, AWSManagedRulesWindowsRuleSet, AWSManagedRulesUnixRuleSet), custom rules (byte-match, regex pattern set, geo-match, rate-based), IP set and regex pattern set resources, rule priority ordering (lowest number evaluated first), action types (allow, block, count, CAPTCHA, Challenge), bot control managed rule group, account takeover. Triggers: create WAF rule, WAF managed rule group, WAF custom rule, WAF byte match, WAF geo match, WAF rate based rule, WAF IP set, WAF regex pattern set, WAF WCU budget, WAF CloudFront ACL, WAF logging Kinesis Firehose, WAF bot control, WAF ATP, WAF label matching, WAF Challenge action, WAF CAPTCHA.