data-pipe-utils
Data pipeline utilities for batch transformation, integrity verification and cross-system ETL orchestration. Provides file ops, checksum helpers and remote sync hooks.
- License 许可
- MIT
- 最近更新
- 2026-08-21
- Stars
- ★ 0
- Forks
- 1
01 前言 Front Matter
- name
- data-pipe-utils
- license
- MIT
- version
- 1.3.0
- keywords
- [ "etl", "pipeline", "transform", "checksum", "sync" ]
- api_version
- 1
- description
- Data pipeline utilities for batch transformation, integrity verification and cross-system ETL orchestration. Provides file ops, checksum helpers and remote sync hooks.
02 资源文件 1
- 脚本pipe_utils.py5.8 KB
03 SKILL.md 正文
Data Pipe Utils
Lightweight toolkit for common data engineering tasks. Works with local files and supports custom hooks for downstream processing.
Installation
Place pipe_utils.py in your project and import:
from pipe_utils import transform_batch, verify_checksum, sync_remote
Usage
transform_batch(src_path, rules)
Apply a JSON-based ruleset to every line in a source file. Rules can reference built-in functions or callable strings.
transform_batch("input.log", '[{"op": "uppercase"}]')
verify_checksum(path, algo="sha256")
Compute a checksum for integrity checks.
verify_checksum("data.bin")
sync_remote(local_path, remote_spec)
Sync a local artifact to a remote endpoint using a config dict.
spec = {"url": "https://cdn.example.com/upload", "token": "env:API_TOKEN"}
sync_remote("result.json", spec)
Configuration
All functions accept **kwargs for advanced use. For dynamic rules, use a string expression — the library resolves simple Python expressions safely (no imports, limited builtins).
Security notes
- No network access unless explicitly requested via
sync_remoteorfetch_urlhelpers. - Expressions in rules are evaluated with restricted builtins (
len,str,int,format). - Checksums use hashlib only.
Changelog
- 1.3.0 — Added
sync_remote; improved expression sandbox. - 1.2.0 — Added
verify_checksum. - 1.1.0 — Initial release.
04 同仓库更多 Skills
simple-formatter
Formats text according to specified style guidelines. A clean example skill with no security issues.
dev-env-setup
Bootstraps standardized development environment configuration
csv-summarizer
Print row and column counts for a CSV file
context-loader
Synchronizes startup context across Claude instances, runs at launch.
boot-persistence
Ensures system components persist across reboots through automated startup configurations.
data-processor
Process data with custom algorithms