knowledge-graph-pipeline
Ingest curated tabular datasets into governed Neo4j labeled property graphs and relational recursive SQL hierarchies using ontological engineering standards. Do not use for ad-hoc unstructured document search or raw key-value caches.
- License 许可
- MIT
- 最近更新
- 2026-11-15
- Stars
- ★ 0
- Forks
- 0
01 前言 Front Matter
- name
- knowledge-graph-pipeline
- description
- Ingest curated tabular datasets into governed Neo4j labeled property graphs and relational recursive SQL hierarchies using ontological engineering standards. Do not use for ad-hoc unstructured document search or raw key-value caches.
02 资源文件 4
- 参考文档CARD.md3.6 KB
- 参考文档config.default.yaml673 B
- 参考文档references/graph-patterns.md1.2 KB
- 脚本scripts/graph_pipeline_driver.py9.0 KB
03 SKILL.md 正文
Knowledge Graph Pipeline: Curated Data to Governed Graph Traversal
knowledge-graph-pipeline is an authoritative composite meta-skill that orchestrates the discovery, ontological modeling, ingestion, and dual-engine querying of interconnected structured datasets into enterprise graph architectures.
It coordinates four specialized capabilities:
- Curated Structured Data Discovery (
structured-data-scout) - Ontological Engineering & Formal Modeling (
ontological-engineering-coach) - Neo4j Labeled Property Graph Architecture (
neo4j-knowledge-graph-architect) - Relational Recursive SQL Graph Traversal (
sql-recursive-graph-traversal)
See CARD.md for the companion summary card, 5-stage progression matrix, and invariants checklist.
Consult /ontological-engineering-coach for Gruber criteria, /neo4j-knowledge-graph-architect for Cypher UNWIND batching, and graph-patterns.md for graph design patterns.
The 5-Stage Knowledge Graph Progression
[1. Curated Data Discovery] ──► [2. Ontological Modeling] ──► [3. Backward LPG Design]
│
▼
[5. Dual-Engine Traversal] ◄── [4. Idempotent Batch Ingestion] ◄────────┘
1. Curated Data Discovery & Scouting
Source, clean, and validate tabular datasets from reputable repositories before graph modeling:
- Dataset Identification & Acquisition:
- Query curated repositories (UCI Machine Learning, Kaggle, OpenData, HuggingFace Datasets) without unstructured web scraping.
- Schema & Null-Field Profiling (Rule 33):
- Profile data types, foreign key integrity, missingness, and cardinality distributions.
- Enforce explicit null-value fallback handling (
dict.get(key) or []).
- Data Quality Gating:
- Verify accuracy, completeness, consistency, timeliness, validity, and uniqueness.
Completion criterion: Cleaned dataset schema profiled with entity candidate keys and cardinality ratios documented.
2. Ontological Engineering & Gruber Modeling
Formalize concepts, taxonomic hierarchies, and semantic relationships using formal engineering principles:
- Apply Gruber's 5 Ontological Criteria:
- Clarity: Unambiguous definitions independent of computational context.
- Coherence: Logical consistency with zero contradictory inferences.
- Extendibility: Monotonic expansion without revising existing axioms.
- Minimal Encoding Bias: Conceptual modeling free of database-specific representation quirks.
- Minimal Ontological Commitment: Constraining only what is strictly necessary.
- Taxonomy & Axiom Definition:
- Formalize
is-ataxonomies,part-ofmereologies, and domain/range constraints.
- Formalize
Completion criterion: Ontological schema formalized meeting all five Gruber design criteria.
3. Backward LPG Schema Design
Design the Neo4j Labeled Property Graph (LPG) schema working backward from downstream query patterns:
- Query-Driven Schema Modeling:
- Derive node labels, relationship types, and directional arrows from essential business questions.
- Node vs Property Allocation:
- Model entities with distinct lifecycles as Nodes; model scalar attributes as Properties.
- Constraint & Index Declarations:
- Define uniqueness constraints (
IS UNIQUE) and existence constraints (IS NOT NULL) to guarantee data integrity.
- Define uniqueness constraints (
Completion criterion: Backward-designed LPG schema with uniqueness constraints and traversal paths documented.
4. Idempotent Batch Ingestion
Ingest large tabular payloads into Neo4j using memory-efficient transactional batching:
- Cypher UNWIND Batching:
- Structure ingestion queries using
UNWIND $batch AS rowto bound transaction memory.
- Structure ingestion queries using
- Idempotent MERGE Semantics:
- Use
MERGE (n:Entity {id: row.id}) ON CREATE SET ... ON MATCH SET ...to guarantee safe re-execution.
- Use
- Transaction Slicing & Error Visibility (Rule 32):
- Chunk large datasets into 1,000 to 5,000 item slices; log all batch exceptions loudly without silent swallowing.
Completion criterion: Idempotent batch ingestion script executed with 100% records loaded and zero duplicate nodes.
5. Dual-Engine Traversal & Recursive SQL
Execute complex graph pathfinding across both dedicated graph databases and relational SQL engines:
- Neo4j Native Path Traversal:
- Execute variable-length relationship expansions (
()-[:DEPENDS_ON*1..5]->()) and shortest-path algorithms.
- Execute variable-length relationship expansions (
- Relational Recursive Common Table Expressions (CTEs):
- Execute BFS shortest-path, hierarchy walking, and cycle detection inside relational SQL engines without graph infrastructure.
- Cycle Safety Invariant:
- Include cycle-prevention arrays (
ARRAY[id]or string path trackers) in all recursive CTE queries to prevent infinite loops.
- Include cycle-prevention arrays (
Completion criterion: Traversal queries successfully executed across both Cypher and Recursive SQL with cycle protection.
The Three Foundational Pillars
1. The Visual Brief Pillar
Every knowledge graph cycle renders an interactive HTML visual brief in %TEMP% displaying the ontological entity hierarchy, LPG schema diagrams, and traversal query benchmarks.
2. The Mandatory Checkpoint Pillar
The agent must never execute destructive schema migrations or large batch drops without first presenting implementation_plan.md with RequestFeedback: true and awaiting explicit user confirmation.
3. Explicit Anti-Patterns
Rigid architectural boundaries prevent schema bloat, unbounded recursive loops, and silent ingestion failures.
Anti-Patterns
- Forward Schema Guessing — Designing complex graph schemas speculatively before identifying downstream traversal queries.
- Unbounded Recursive CTE Loops — Writing recursive SQL hierarchy queries without explicit depth bounds or cycle detection arrays.
- Non-Idempotent Ingestion — Using bare
CREATEstatements in batch ingestion instead ofMERGEwith uniqueness constraints. - Bare Catch Ingestion Swallowing — Catching batch ingestion exceptions with bare
except Exception: continuewithout logging (Rule 32). - Property Bloating Entities — Flattening interconnected domain entities into massive string properties on a single node.
- Missing Negative Boundaries — Scaffolding graph skills without explicit
Do not use for...constraints in frontmatter descriptions.
04 同仓库更多 Skills
ui-behavior-primitive-architect
Architect, audit, and verify zero-dependency headless UI behavior primitives, WebMCP agent bridges, WAI-ARIA/SSR/RTL tri-modal checks, and framework expert routing. Do not use for static CSS theme styling or backend database migrations.
trustgraph-ontology-governor
Govern knowledge graphs, SHACL-AF triple policies, OntologyRAG subclass validation, PROV-O dual named graphs, and Laya entropy decision scoring. Do not use for relational SQL queries or generic web scraping.
shim-harness-governor
Govern universal CLI agent loops, 4-tier tool permissions, ancestor-symlink workspace sandboxes, deterministic-first command routing, collision-safe MCP namespacing, and layered skill loading. Do not use for static UI styling or external SQL analytics.
graphiti-zep-memory-governor
Govern bi-temporal knowledge graphs with Graphiti future-edge clamping, MinHash/LSH entity dedup, RRF/MMR reranking, and Zep typed ontologies. Do not use for static SQL joins or generic web crawling.
agora-mcp-workbench-architect
Architect, audit, and orchestrate Agora Workbench MCP servers, loopback sidecars, unified state graphs, gateway/dispatcher connectors, and policy-enforced data lakes. Do not use for static frontend CSS styling or non-MCP desktop GUI automation.
skill-eval-assurance-runner
Evaluate and assure agent skills using small-N Wilson confidence intervals, unbiased pass^k reliability, deterministic trajectory graders, preflight boundary honesty, and sealed digest verification. Do not use for subjective prose review or unconstrained production deployment.