[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6AE9WK5FSTZH3AG7MRAJT2Z":3,"skill-same:01M6AE9WK5FSTZH3AG7MRAJT2Z":97},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":23,"publishedAt":35,"linkOnly":36,"markdownContent":37,"markdownBody":38,"frontMatter":39,"refName":44,"commitSha":45,"skillDirectory":46,"forksCount":47,"repositoryHtmlUrl":48,"repositoryHomepageUrl":6,"ownerHtmlUrl":49,"ownerBlogUrl":50,"ownerDescription":51,"updatedAt":52,"install":53,"resources":66},"01M6AE9WK5FSTZH3AG7MRAJT2Z","pii-detection-pipeline",null,"Build automated PII detection and redaction pipelines using spaCy NER, Microsoft Presidio, and AWS Macie integration. Includes confidence scoring, custom entity type definitions, batch processing workflows, and multi-format document scanning for structured and unstructured data sources.","mukul975\u002FPrivacy-Data-Protection-Skills","mukul975","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F42860185?v=4","01M6ADEQR8T5M4EEG6C67BQCWF","Mahipal","01M6ADNKWX1HF4Q5DKEXT5M774",336,"Apache-2.0","https:\u002F\u002Fgithub.com\u002Fmukul975\u002FPrivacy-Data-Protection-Skills\u002Fblob\u002Fmain\u002Fplugins\u002Fprivacy-engineering-skills\u002Fskills\u002Fpii-detection-pipeline\u002FSKILL.md",[18],{"categoryNo":19,"categoryName":20,"taxonomyType":21,"taxonomyTypeDesc":22},"01JBSEEDCAT000000000000033","数据分析师","PROFESSION","适用职业",[24,25,26,27,28,29,30,31,32,33,34],"pipeline","detection","pii detection pipeline","microsoft","workflows","batch","integration","engineering","privacy","data","aws","2026-11-10 10:12:30",false,"---\nname: pii-detection-pipeline\ndescription: >-\n  Build automated PII detection and redaction pipelines using spaCy NER,\n  Microsoft Presidio, and AWS Macie integration. Includes confidence scoring,\n  custom entity type definitions, batch processing workflows, and multi-format\n  document scanning for structured and unstructured data sources.\nlicense: Apache-2.0\nmetadata:\n  author: mukul975\n  version: \"1.0\"\n  domain: privacy\n  subdomain: privacy-engineering\n  tags: \"pii-detection, presidio, spacy-ner, data-redaction, aws-macie\"\n---\n\n# Automated PII Detection and Redaction Pipeline\n\n## Overview\n\nAutomated PII detection is a foundational capability for privacy engineering, enabling organizations to discover, classify, and protect personal data at scale. This skill covers building production-grade PII detection pipelines that combine rule-based pattern matching, machine learning-based Named Entity Recognition (NER), and cloud-native discovery services.\n\n## PII Entity Types Catalog\n\n### Direct Identifiers\n\n| Entity Type | Examples | Detection Method | Risk Level |\n|------------|---------|-----------------|------------|\n| PERSON_NAME | \"John Smith\", \"Maria Garcia\" | NER model | High |\n| EMAIL_ADDRESS | \"j.smith@cipherengineeringlabs.com\" | Regex pattern | High |\n| PHONE_NUMBER | \"+1-555-0123\", \"(555) 012-3456\" | Regex + validation | High |\n| SSN | \"123-45-6789\" | Regex + checksum | Critical |\n| PASSPORT_NUMBER | \"AB1234567\" | Regex per country format | Critical |\n| DRIVER_LICENSE | \"D123-4567-8901\" | Regex per state\u002Fcountry | Critical |\n| CREDIT_CARD | \"4111-1111-1111-1111\" | Regex + Luhn checksum | Critical |\n| IBAN | \"GB82 WEST 1234 5698 7654 32\" | Regex + modulo-97 check | High |\n| IP_ADDRESS | \"192.168.1.1\", \"2001:db8::1\" | Regex (IPv4\u002FIPv6) | Medium |\n| MAC_ADDRESS | \"00:1A:2B:3C:4D:5E\" | Regex pattern | Medium |\n\n### Quasi-Identifiers\n\n| Entity Type | Examples | Detection Method | Risk Level |\n|------------|---------|-----------------|------------|\n| DATE_OF_BIRTH | \"1990-01-15\", \"January 15, 1990\" | NER + date parsing | Medium |\n| POSTAL_CODE | \"10001\", \"SW1A 1AA\" | Regex per country | Medium |\n| AGE | \"35 years old\", \"age: 42\" | NER + context | Low-Medium |\n| GENDER | \"male\", \"female\", \"non-binary\" | Dictionary + context | Low |\n| NATIONALITY | \"British\", \"Japanese\" | Dictionary + NER | Low-Medium |\n| LOCATION | \"123 Main St\", \"New York\" | NER model | Medium |\n\n### Sensitive Categories\n\n| Entity Type | Examples | Detection Method | Risk Level |\n|------------|---------|-----------------|------------|\n| MEDICAL_RECORD | \"MRN: 12345678\" | Regex + context | Critical |\n| HEALTH_CONDITION | \"diabetes\", \"HIV positive\" | Medical NER + dictionary | Critical |\n| RELIGIOUS_BELIEF | \"Muslim\", \"Catholic\" | Dictionary + context | High |\n| POLITICAL_OPINION | \"Democratic Party member\" | Dictionary + context | High |\n| SEXUAL_ORIENTATION | \"gay\", \"bisexual\" | Dictionary + context | High |\n| BIOMETRIC_DATA | \"fingerprint hash: ...\" | Context + pattern | Critical |\n| GENETIC_DATA | \"BRCA1 positive\" | Medical dictionary | Critical |\n\n## Microsoft Presidio Implementation\n\n### Pipeline Architecture\n\n```\nInput Data --> Presidio Analyzer --> Detected Entities --> Presidio Anonymizer --> Redacted Output\n                    |                      |                      |\n                    v                      v                      v\n             +------------+        +-------------+        +---------------+\n             | Recognizers|        | Score Filter |        | Operators     |\n             | - Pattern  |        | (threshold)  |        | - Replace     |\n             | - NER      |        |              |        | - Redact      |\n             | - Custom   |        |              |        | - Hash        |\n             +------------+        +-------------+        | - Mask        |\n                                                          | - Encrypt     |\n                                                          +---------------+\n```\n\n### Core Implementation\n\n```python\n\"\"\"\nPII detection and redaction pipeline using Microsoft Presidio.\nSupports structured and unstructured text with configurable\nentity types, confidence thresholds, and redaction strategies.\n\"\"\"\n\nfrom presidio_analyzer import AnalyzerEngine, PatternRecognizer, Pattern\nfrom presidio_analyzer.nlp_engine import NlpEngineProvider\nfrom presidio_anonymizer import AnonymizerEngine\nfrom presidio_anonymizer.entities import OperatorConfig\nimport json\nfrom dataclasses import dataclass, field\n\n\n@dataclass\nclass DetectionResult:\n    entity_type: str\n    text: str\n    start: int\n    end: int\n    score: float\n    source: str\n\n\n@dataclass\nclass PipelineConfig:\n    language: str = \"en\"\n    score_threshold: float = 0.5\n    entities_to_detect: list[str] = field(default_factory=lambda: [\n        \"PERSON\", \"EMAIL_ADDRESS\", \"PHONE_NUMBER\", \"CREDIT_CARD\",\n        \"US_SSN\", \"US_DRIVER_LICENSE\", \"IBAN_CODE\", \"IP_ADDRESS\",\n        \"LOCATION\", \"DATE_TIME\", \"NRP\", \"MEDICAL_LICENSE\",\n        \"US_PASSPORT\", \"US_BANK_NUMBER\", \"UK_NHS\"\n    ])\n    redaction_strategy: str = \"replace\"  # replace, redact, hash, mask, encrypt\n\n\nclass PIIDetectionPipeline:\n    \"\"\"\n    Production PII detection pipeline built on Microsoft Presidio.\n    Supports custom entity types and configurable redaction strategies.\n    \"\"\"\n\n    def __init__(self, config: PipelineConfig):\n        self.config = config\n\n        # Initialize NLP engine with spaCy\n        nlp_config = {\n            \"nlp_engine_name\": \"spacy\",\n            \"models\": [{\"lang_code\": config.language, \"model_name\": \"en_core_web_lg\"}]\n        }\n        nlp_engine = NlpEngineProvider(nlp_configuration=nlp_config).create_engine()\n\n        # Initialize analyzer\n        self.analyzer = AnalyzerEngine(nlp_engine=nlp_engine)\n\n        # Register custom recognizers\n        self._register_custom_recognizers()\n\n        # Initialize anonymizer\n        self.anonymizer = AnonymizerEngine()\n\n    def _register_custom_recognizers(self):\n        \"\"\"Register custom PII recognizers beyond built-in types.\"\"\"\n\n        # UK National Insurance Number\n        nino_pattern = Pattern(\n            name=\"uk_nino\",\n            regex=r\"\\b[A-CEGHJ-PR-TW-Z]{2}\\s?\\d{2}\\s?\\d{2}\\s?\\d{2}\\s?[A-D]\\b\",\n            score=0.85\n        )\n        nino_recognizer = PatternRecognizer(\n            supported_entity=\"UK_NINO\",\n            patterns=[nino_pattern],\n            supported_language=\"en\"\n        )\n        self.analyzer.registry.add_recognizer(nino_recognizer)\n\n        # Vehicle Registration Number (UK format)\n        vrn_pattern = Pattern(\n            name=\"uk_vrn\",\n            regex=r\"\\b[A-Z]{2}\\d{2}\\s?[A-Z]{3}\\b\",\n            score=0.6\n        )\n        vrn_recognizer = PatternRecognizer(\n            supported_entity=\"UK_VRN\",\n            patterns=[vrn_pattern],\n            supported_language=\"en\"\n        )\n        self.analyzer.registry.add_recognizer(vrn_recognizer)\n\n        # Employee ID (Cipher Engineering Labs format: CEL-XXXXX)\n        emp_id_pattern = Pattern(\n            name=\"employee_id\",\n            regex=r\"\\bCEL-\\d{5}\\b\",\n            score=0.95\n        )\n        emp_id_recognizer = PatternRecognizer(\n            supported_entity=\"EMPLOYEE_ID\",\n            patterns=[emp_id_pattern],\n            supported_language=\"en\"\n        )\n        self.analyzer.registry.add_recognizer(emp_id_recognizer)\n\n    def detect(self, text: str) -> list[DetectionResult]:\n        \"\"\"\n        Detect PII entities in text.\n\n        Args:\n            text: Input text to scan\n\n        Returns:\n            List of detected PII entities with confidence scores\n        \"\"\"\n        results = self.analyzer.analyze(\n            text=text,\n            entities=self.config.entities_to_detect,\n            language=self.config.language,\n            score_threshold=self.config.score_threshold\n        )\n\n        return [\n            DetectionResult(\n                entity_type=r.entity_type,\n                text=text[r.start:r.end],\n                start=r.start,\n                end=r.end,\n                score=r.score,\n                source=r.analysis_explanation.recognizer if r.analysis_explanation else \"unknown\"\n            )\n            for r in results\n        ]\n\n    def redact(self, text: str) -> tuple[str, list[DetectionResult]]:\n        \"\"\"\n        Detect and redact PII from text.\n\n        Returns:\n            Tuple of (redacted_text, list_of_detections)\n        \"\"\"\n        # Detect entities\n        analyzer_results = self.analyzer.analyze(\n            text=text,\n            entities=self.config.entities_to_detect,\n            language=self.config.language,\n            score_threshold=self.config.score_threshold\n        )\n\n        # Configure redaction operator\n        operators = self._get_operators()\n\n        # Apply redaction\n        anonymized = self.anonymizer.anonymize(\n            text=text,\n            analyzer_results=analyzer_results,\n            operators=operators\n        )\n\n        detections = [\n            DetectionResult(\n                entity_type=r.entity_type,\n                text=text[r.start:r.end],\n                start=r.start,\n                end=r.end,\n                score=r.score,\n                source=\"presidio\"\n            )\n            for r in analyzer_results\n        ]\n\n        return anonymized.text, detections\n\n    def _get_operators(self) -> dict:\n        \"\"\"Configure anonymization operators based on strategy.\"\"\"\n        if self.config.redaction_strategy == \"replace\":\n            return {\"DEFAULT\": OperatorConfig(\"replace\", {\"new_value\": \"\u003CREDACTED>\"})}\n        elif self.config.redaction_strategy == \"hash\":\n            return {\"DEFAULT\": OperatorConfig(\"hash\", {\"hash_type\": \"sha256\"})}\n        elif self.config.redaction_strategy == \"mask\":\n            return {\"DEFAULT\": OperatorConfig(\"mask\", {\n                \"type\": \"mask\",\n                \"masking_char\": \"*\",\n                \"chars_to_mask\": 100,\n                \"from_end\": False\n            })}\n        elif self.config.redaction_strategy == \"redact\":\n            return {\"DEFAULT\": OperatorConfig(\"redact\", {})}\n        else:\n            return {\"DEFAULT\": OperatorConfig(\"replace\", {\"new_value\": \"\u003CREDACTED>\"})}\n\n\nclass BatchPIIProcessor:\n    \"\"\"\n    Process large volumes of documents for PII detection.\n    Supports structured (CSV\u002FJSON) and unstructured (text) formats.\n    \"\"\"\n\n    def __init__(self, pipeline: PIIDetectionPipeline):\n        self.pipeline = pipeline\n\n    def process_csv(\n        self,\n        input_path: str,\n        output_path: str,\n        columns_to_scan: list[str] = None\n    ) -> dict:\n        \"\"\"\n        Scan and redact PII in a CSV file.\n\n        Args:\n            input_path: Path to input CSV\n            output_path: Path to write redacted CSV\n            columns_to_scan: Columns to scan (all if None)\n\n        Returns:\n            Summary statistics of detections\n        \"\"\"\n        import pandas as pd\n\n        df = pd.read_csv(input_path)\n        stats = {\"total_rows\": len(df), \"detections\": {}, \"columns_scanned\": []}\n\n        scan_columns = columns_to_scan or df.columns.tolist()\n\n        for col in scan_columns:\n            if col not in df.columns:\n                continue\n\n            stats[\"columns_scanned\"].append(col)\n            col_detections = []\n\n            for idx, value in df[col].items():\n                if pd.isna(value):\n                    continue\n\n                text = str(value)\n                redacted, detections = self.pipeline.redact(text)\n                df.at[idx, col] = redacted\n\n                for d in detections:\n                    col_detections.append(d.entity_type)\n\n            # Count detection types for this column\n            for entity_type in set(col_detections):\n                count = col_detections.count(entity_type)\n                key = f\"{col}:{entity_type}\"\n                stats[\"detections\"][key] = count\n\n        df.to_csv(output_path, index=False)\n        stats[\"total_detections\"] = sum(stats[\"detections\"].values())\n\n        return stats\n\n    def process_text_files(\n        self,\n        file_paths: list[str],\n        output_dir: str\n    ) -> dict:\n        \"\"\"\n        Batch process text files for PII detection and redaction.\n\n        Returns summary statistics.\n        \"\"\"\n        import os\n\n        stats = {\n            \"files_processed\": 0,\n            \"total_detections\": 0,\n            \"entity_counts\": {},\n            \"high_risk_files\": []\n        }\n\n        for file_path in file_paths:\n            with open(file_path, \"r\", encoding=\"utf-8\") as f:\n                text = f.read()\n\n            redacted, detections = self.pipeline.redact(text)\n\n            # Write redacted output\n            output_path = os.path.join(output_dir, os.path.basename(file_path))\n            with open(output_path, \"w\", encoding=\"utf-8\") as f:\n                f.write(redacted)\n\n            # Update statistics\n            stats[\"files_processed\"] += 1\n            stats[\"total_detections\"] += len(detections)\n\n            for d in detections:\n                stats[\"entity_counts\"][d.entity_type] = (\n                    stats[\"entity_counts\"].get(d.entity_type, 0) + 1\n                )\n\n            # Flag high-risk files (contain critical PII)\n            critical_types = {\"US_SSN\", \"CREDIT_CARD\", \"MEDICAL_LICENSE\", \"UK_NHS\"}\n            if any(d.entity_type in critical_types for d in detections):\n                stats[\"high_risk_files\"].append(file_path)\n\n        return stats\n```\n\n## spaCy NER Custom Training\n\n### Training Custom Entity Types\n\n```python\n\"\"\"\nTrain custom spaCy NER model for domain-specific PII entities.\n\"\"\"\n\nimport spacy\nfrom spacy.training import Example\nimport random\n\n\ndef create_training_data() -> list[tuple[str, dict]]:\n    \"\"\"\n    Create training examples for custom PII entity types.\n\n    Returns list of (text, annotations) tuples.\n    \"\"\"\n    training_data = [\n        (\n            \"Patient MRN 12345678 was admitted on 2024-01-15\",\n            {\"entities\": [(12, 20, \"MEDICAL_RECORD_NUMBER\")]}\n        ),\n        (\n            \"Employee CEL-00142 reported the incident\",\n            {\"entities\": [(9, 18, \"EMPLOYEE_ID\")]}\n        ),\n        (\n            \"Policy holder number PLH-2024-99887 filed a claim\",\n            {\"entities\": [(22, 35, \"POLICY_NUMBER\")]}\n        ),\n        (\n            \"The customer with loyalty ID LYL-A1B2C3 requested data export\",\n            {\"entities\": [(29, 39, \"LOYALTY_ID\")]}\n        ),\n    ]\n    return training_data\n\n\ndef train_custom_ner(\n    base_model: str = \"en_core_web_lg\",\n    training_data: list = None,\n    n_iter: int = 30,\n    output_dir: str = \".\u002Fcustom_ner_model\"\n):\n    \"\"\"\n    Fine-tune spaCy NER model with custom PII entity types.\n\n    Args:\n        base_model: Base spaCy model to fine-tune\n        training_data: List of (text, annotations) tuples\n        n_iter: Number of training iterations\n        output_dir: Directory to save trained model\n    \"\"\"\n    if training_data is None:\n        training_data = create_training_data()\n\n    nlp = spacy.load(base_model)\n\n    # Get or create NER pipe\n    if \"ner\" not in nlp.pipe_names:\n        ner = nlp.add_pipe(\"ner\", last=True)\n    else:\n        ner = nlp.get_pipe(\"ner\")\n\n    # Add custom entity labels\n    custom_labels = set()\n    for _, annotations in training_data:\n        for ent in annotations.get(\"entities\", []):\n            custom_labels.add(ent[2])\n\n    for label in custom_labels:\n        ner.add_label(label)\n\n    # Train\n    optimizer = nlp.resume_training()\n    other_pipes = [pipe for pipe in nlp.pipe_names if pipe != \"ner\"]\n\n    with nlp.disable_pipes(*other_pipes):\n        for iteration in range(n_iter):\n            random.shuffle(training_data)\n            losses = {}\n\n            for text, annotations in training_data:\n                doc = nlp.make_doc(text)\n                example = Example.from_dict(doc, annotations)\n                nlp.update([example], drop=0.35, sgd=optimizer, losses=losses)\n\n            if iteration % 10 == 0:\n                print(f\"Iteration {iteration}, Losses: {losses}\")\n\n    nlp.to_disk(output_dir)\n    return nlp\n```\n\n## AWS Macie Integration\n\n### Architecture\n\n```\nS3 Buckets --> Macie Classification Jobs --> Findings --> EventBridge --> Lambda\n                                                |                          |\n                                                v                          v\n                                        Security Hub              Remediation\n                                        (centralized)             - Tag sensitive\n                                                                  - Encrypt\n                                                                  - Notify owner\n                                                                  - Quarantine\n```\n\n### Macie Job Configuration\n\n```python\n\"\"\"\nAWS Macie integration for cloud-native PII detection in S3 buckets.\n\"\"\"\n\nimport boto3\nfrom datetime import datetime\n\n\nclass MacieIntegration:\n    \"\"\"\n    Configure and manage AWS Macie classification jobs for\n    automated PII detection across S3 data stores.\n    \"\"\"\n\n    def __init__(self, region: str = \"us-east-1\"):\n        self.macie_client = boto3.client(\"macie2\", region_name=region)\n\n    def create_classification_job(\n        self,\n        bucket_name: str,\n        job_name: str,\n        custom_data_identifiers: list[str] = None,\n        schedule: str = \"ONE_TIME\"\n    ) -> str:\n        \"\"\"\n        Create a Macie classification job for an S3 bucket.\n\n        Args:\n            bucket_name: Target S3 bucket\n            job_name: Descriptive job name\n            custom_data_identifiers: IDs of custom data identifier resources\n            schedule: ONE_TIME or SCHEDULED\n\n        Returns:\n            Job ID\n        \"\"\"\n        job_config = {\n            \"name\": job_name,\n            \"description\": f\"PII detection scan for {bucket_name}\",\n            \"jobType\": schedule,\n            \"s3JobDefinition\": {\n                \"bucketDefinitions\": [\n                    {\n                        \"accountId\": self._get_account_id(),\n                        \"buckets\": [bucket_name]\n                    }\n                ],\n                \"scoping\": {\n                    \"includes\": {\n                        \"and\": [\n                            {\n                                \"simpleScopeTerm\": {\n                                    \"comparator\": \"STARTS_WITH\",\n                                    \"key\": \"OBJECT_EXTENSION\",\n                                    \"values\": [\"csv\", \"json\", \"txt\", \"pdf\", \"docx\", \"xlsx\", \"parquet\"]\n                                }\n                            }\n                        ]\n                    }\n                }\n            },\n            \"managedDataIdentifierSelector\": \"ALL\",\n            \"tags\": {\n                \"Team\": \"Privacy Engineering\",\n                \"Purpose\": \"PII Detection\"\n            }\n        }\n\n        if custom_data_identifiers:\n            job_config[\"customDataIdentifierIds\"] = custom_data_identifiers\n\n        response = self.macie_client.create_classification_job(**job_config)\n        return response[\"jobId\"]\n\n    def create_custom_data_identifier(\n        self,\n        name: str,\n        description: str,\n        regex: str,\n        keywords: list[str] = None,\n        maximum_match_distance: int = 50\n    ) -> str:\n        \"\"\"\n        Create a custom data identifier for organization-specific PII patterns.\n\n        Returns:\n            Custom data identifier ID\n        \"\"\"\n        params = {\n            \"name\": name,\n            \"description\": description,\n            \"regex\": regex,\n            \"maximumMatchDistance\": maximum_match_distance\n        }\n\n        if keywords:\n            params[\"keywords\"] = keywords\n\n        response = self.macie_client.create_custom_data_identifier(**params)\n        return response[\"customDataIdentifierId\"]\n\n    def _get_account_id(self) -> str:\n        \"\"\"Get the current AWS account ID.\"\"\"\n        sts = boto3.client(\"sts\")\n        return sts.get_caller_identity()[\"Account\"]\n\n    def get_findings_summary(self, job_id: str) -> dict:\n        \"\"\"Get summary of findings from a classification job.\"\"\"\n        response = self.macie_client.list_findings(\n            findingCriteria={\n                \"criterion\": {\n                    \"classificationDetails.jobId\": {\n                        \"eq\": [job_id]\n                    }\n                }\n            }\n        )\n\n        findings = []\n        if response[\"findingIds\"]:\n            details = self.macie_client.get_findings(findingIds=response[\"findingIds\"])\n            findings = details[\"findings\"]\n\n        summary = {\n            \"total_findings\": len(findings),\n            \"severity_counts\": {},\n            \"entity_type_counts\": {},\n            \"affected_objects\": []\n        }\n\n        for finding in findings:\n            severity = finding.get(\"severity\", {}).get(\"description\", \"unknown\")\n            summary[\"severity_counts\"][severity] = (\n                summary[\"severity_counts\"].get(severity, 0) + 1\n            )\n\n            sensitive_data = finding.get(\"classificationDetails\", {}).get(\n                \"result\", {}\n            ).get(\"sensitiveData\", [])\n\n            for sd in sensitive_data:\n                category = sd.get(\"category\", \"unknown\")\n                summary[\"entity_type_counts\"][category] = (\n                    summary[\"entity_type_counts\"].get(category, 0)\n                    + sd.get(\"totalCount\", 0)\n                )\n\n            resource = finding.get(\"resourcesAffected\", {}).get(\"s3Object\", {})\n            if resource:\n                summary[\"affected_objects\"].append(resource.get(\"key\", \"unknown\"))\n\n        return summary\n```\n\n## Confidence Scoring Framework\n\n| Score Range | Confidence Level | Recommended Action |\n|-------------|-----------------|-------------------|\n| 0.95 - 1.00 | Very High | Auto-redact |\n| 0.80 - 0.94 | High | Auto-redact with logging |\n| 0.60 - 0.79 | Medium | Flag for human review |\n| 0.40 - 0.59 | Low | Log only, no action |\n| 0.00 - 0.39 | Very Low | Ignore |\n\n## References\n\n- Microsoft Presidio Documentation: microsoft.github.io\u002Fpresidio\n- spaCy NER Documentation: spacy.io\u002Fusage\u002Flinguistic-features#named-entities\n- AWS Macie Documentation: docs.aws.amazon.com\u002Fmacie\n- Google Cloud DLP API Documentation\n- NIST SP 800-188 — De-Identifying Government Datasets\n- Article 29 WP Opinion 05\u002F2014 on Anonymisation Techniques\n","# Automated PII Detection and Redaction Pipeline\n\n## Overview\n\nAutomated PII detection is a foundational capability for privacy engineering, enabling organizations to discover, classify, and protect personal data at scale. This skill covers building production-grade PII detection pipelines that combine rule-based pattern matching, machine learning-based Named Entity Recognition (NER), and cloud-native discovery services.\n\n## PII Entity Types Catalog\n\n### Direct Identifiers\n\n| Entity Type | Examples | Detection Method | Risk Level |\n|------------|---------|-----------------|------------|\n| PERSON_NAME | \"John Smith\", \"Maria Garcia\" | NER model | High |\n| EMAIL_ADDRESS | \"j.smith@cipherengineeringlabs.com\" | Regex pattern | High |\n| PHONE_NUMBER | \"+1-555-0123\", \"(555) 012-3456\" | Regex + validation | High |\n| SSN | \"123-45-6789\" | Regex + checksum | Critical |\n| PASSPORT_NUMBER | \"AB1234567\" | Regex per country format | Critical |\n| DRIVER_LICENSE | \"D123-4567-8901\" | Regex per state\u002Fcountry | Critical |\n| CREDIT_CARD | \"4111-1111-1111-1111\" | Regex + Luhn checksum | Critical |\n| IBAN | \"GB82 WEST 1234 5698 7654 32\" | Regex + modulo-97 check | High |\n| IP_ADDRESS | \"192.168.1.1\", \"2001:db8::1\" | Regex (IPv4\u002FIPv6) | Medium |\n| MAC_ADDRESS | \"00:1A:2B:3C:4D:5E\" | Regex pattern | Medium |\n\n### Quasi-Identifiers\n\n| Entity Type | Examples | Detection Method | Risk Level |\n|------------|---------|-----------------|------------|\n| DATE_OF_BIRTH | \"1990-01-15\", \"January 15, 1990\" | NER + date parsing | Medium |\n| POSTAL_CODE | \"10001\", \"SW1A 1AA\" | Regex per country | Medium |\n| AGE | \"35 years old\", \"age: 42\" | NER + context | Low-Medium |\n| GENDER | \"male\", \"female\", \"non-binary\" | Dictionary + context | Low |\n| NATIONALITY | \"British\", \"Japanese\" | Dictionary + NER | Low-Medium |\n| LOCATION | \"123 Main St\", \"New York\" | NER model | Medium |\n\n### Sensitive Categories\n\n| Entity Type | Examples | Detection Method | Risk Level |\n|------------|---------|-----------------|------------|\n| MEDICAL_RECORD | \"MRN: 12345678\" | Regex + context | Critical |\n| HEALTH_CONDITION | \"diabetes\", \"HIV positive\" | Medical NER + dictionary | Critical |\n| RELIGIOUS_BELIEF | \"Muslim\", \"Catholic\" | Dictionary + context | High |\n| POLITICAL_OPINION | \"Democratic Party member\" | Dictionary + context | High |\n| SEXUAL_ORIENTATION | \"gay\", \"bisexual\" | Dictionary + context | High |\n| BIOMETRIC_DATA | \"fingerprint hash: ...\" | Context + pattern | Critical |\n| GENETIC_DATA | \"BRCA1 positive\" | Medical dictionary | Critical |\n\n## Microsoft Presidio Implementation\n\n### Pipeline Architecture\n\n```\nInput Data --> Presidio Analyzer --> Detected Entities --> Presidio Anonymizer --> Redacted Output\n                    |                      |                      |\n                    v                      v                      v\n             +------------+        +-------------+        +---------------+\n             | Recognizers|        | Score Filter |        | Operators     |\n             | - Pattern  |        | (threshold)  |        | - Replace     |\n             | - NER      |        |              |        | - Redact      |\n             | - Custom   |        |              |        | - Hash        |\n             +------------+        +-------------+        | - Mask        |\n                                                          | - Encrypt     |\n                                                          +---------------+\n```\n\n### Core Implementation\n\n```python\n\"\"\"\nPII detection and redaction pipeline using Microsoft Presidio.\nSupports structured and unstructured text with configurable\nentity types, confidence thresholds, and redaction strategies.\n\"\"\"\n\nfrom presidio_analyzer import AnalyzerEngine, PatternRecognizer, Pattern\nfrom presidio_analyzer.nlp_engine import NlpEngineProvider\nfrom presidio_anonymizer import AnonymizerEngine\nfrom presidio_anonymizer.entities import OperatorConfig\nimport json\nfrom dataclasses import dataclass, field\n\n\n@dataclass\nclass DetectionResult:\n    entity_type: str\n    text: str\n    start: int\n    end: int\n    score: float\n    source: str\n\n\n@dataclass\nclass PipelineConfig:\n    language: str = \"en\"\n    score_threshold: float = 0.5\n    entities_to_detect: list[str] = field(default_factory=lambda: [\n        \"PERSON\", \"EMAIL_ADDRESS\", \"PHONE_NUMBER\", \"CREDIT_CARD\",\n        \"US_SSN\", \"US_DRIVER_LICENSE\", \"IBAN_CODE\", \"IP_ADDRESS\",\n        \"LOCATION\", \"DATE_TIME\", \"NRP\", \"MEDICAL_LICENSE\",\n        \"US_PASSPORT\", \"US_BANK_NUMBER\", \"UK_NHS\"\n    ])\n    redaction_strategy: str = \"replace\"  # replace, redact, hash, mask, encrypt\n\n\nclass PIIDetectionPipeline:\n    \"\"\"\n    Production PII detection pipeline built on Microsoft Presidio.\n    Supports custom entity types and configurable redaction strategies.\n    \"\"\"\n\n    def __init__(self, config: PipelineConfig):\n        self.config = config\n\n        # Initialize NLP engine with spaCy\n        nlp_config = {\n            \"nlp_engine_name\": \"spacy\",\n            \"models\": [{\"lang_code\": config.language, \"model_name\": \"en_core_web_lg\"}]\n        }\n        nlp_engine = NlpEngineProvider(nlp_configuration=nlp_config).create_engine()\n\n        # Initialize analyzer\n        self.analyzer = AnalyzerEngine(nlp_engine=nlp_engine)\n\n        # Register custom recognizers\n        self._register_custom_recognizers()\n\n        # Initialize anonymizer\n        self.anonymizer = AnonymizerEngine()\n\n    def _register_custom_recognizers(self):\n        \"\"\"Register custom PII recognizers beyond built-in types.\"\"\"\n\n        # UK National Insurance Number\n        nino_pattern = Pattern(\n            name=\"uk_nino\",\n            regex=r\"\\b[A-CEGHJ-PR-TW-Z]{2}\\s?\\d{2}\\s?\\d{2}\\s?\\d{2}\\s?[A-D]\\b\",\n            score=0.85\n        )\n        nino_recognizer = PatternRecognizer(\n            supported_entity=\"UK_NINO\",\n            patterns=[nino_pattern],\n            supported_language=\"en\"\n        )\n        self.analyzer.registry.add_recognizer(nino_recognizer)\n\n        # Vehicle Registration Number (UK format)\n        vrn_pattern = Pattern(\n            name=\"uk_vrn\",\n            regex=r\"\\b[A-Z]{2}\\d{2}\\s?[A-Z]{3}\\b\",\n            score=0.6\n        )\n        vrn_recognizer = PatternRecognizer(\n            supported_entity=\"UK_VRN\",\n            patterns=[vrn_pattern],\n            supported_language=\"en\"\n        )\n        self.analyzer.registry.add_recognizer(vrn_recognizer)\n\n        # Employee ID (Cipher Engineering Labs format: CEL-XXXXX)\n        emp_id_pattern = Pattern(\n            name=\"employee_id\",\n            regex=r\"\\bCEL-\\d{5}\\b\",\n            score=0.95\n        )\n        emp_id_recognizer = PatternRecognizer(\n            supported_entity=\"EMPLOYEE_ID\",\n            patterns=[emp_id_pattern],\n            supported_language=\"en\"\n        )\n        self.analyzer.registry.add_recognizer(emp_id_recognizer)\n\n    def detect(self, text: str) -> list[DetectionResult]:\n        \"\"\"\n        Detect PII entities in text.\n\n        Args:\n            text: Input text to scan\n\n        Returns:\n            List of detected PII entities with confidence scores\n        \"\"\"\n        results = self.analyzer.analyze(\n            text=text,\n            entities=self.config.entities_to_detect,\n            language=self.config.language,\n            score_threshold=self.config.score_threshold\n        )\n\n        return [\n            DetectionResult(\n                entity_type=r.entity_type,\n                text=text[r.start:r.end],\n                start=r.start,\n                end=r.end,\n                score=r.score,\n                source=r.analysis_explanation.recognizer if r.analysis_explanation else \"unknown\"\n            )\n            for r in results\n        ]\n\n    def redact(self, text: str) -> tuple[str, list[DetectionResult]]:\n        \"\"\"\n        Detect and redact PII from text.\n\n        Returns:\n            Tuple of (redacted_text, list_of_detections)\n        \"\"\"\n        # Detect entities\n        analyzer_results = self.analyzer.analyze(\n            text=text,\n            entities=self.config.entities_to_detect,\n            language=self.config.language,\n            score_threshold=self.config.score_threshold\n        )\n\n        # Configure redaction operator\n        operators = self._get_operators()\n\n        # Apply redaction\n        anonymized = self.anonymizer.anonymize(\n            text=text,\n            analyzer_results=analyzer_results,\n            operators=operators\n        )\n\n        detections = [\n            DetectionResult(\n                entity_type=r.entity_type,\n                text=text[r.start:r.end],\n                start=r.start,\n                end=r.end,\n                score=r.score,\n                source=\"presidio\"\n            )\n            for r in analyzer_results\n        ]\n\n        return anonymized.text, detections\n\n    def _get_operators(self) -> dict:\n        \"\"\"Configure anonymization operators based on strategy.\"\"\"\n        if self.config.redaction_strategy == \"replace\":\n            return {\"DEFAULT\": OperatorConfig(\"replace\", {\"new_value\": \"\u003CREDACTED>\"})}\n        elif self.config.redaction_strategy == \"hash\":\n            return {\"DEFAULT\": OperatorConfig(\"hash\", {\"hash_type\": \"sha256\"})}\n        elif self.config.redaction_strategy == \"mask\":\n            return {\"DEFAULT\": OperatorConfig(\"mask\", {\n                \"type\": \"mask\",\n                \"masking_char\": \"*\",\n                \"chars_to_mask\": 100,\n                \"from_end\": False\n            })}\n        elif self.config.redaction_strategy == \"redact\":\n            return {\"DEFAULT\": OperatorConfig(\"redact\", {})}\n        else:\n            return {\"DEFAULT\": OperatorConfig(\"replace\", {\"new_value\": \"\u003CREDACTED>\"})}\n\n\nclass BatchPIIProcessor:\n    \"\"\"\n    Process large volumes of documents for PII detection.\n    Supports structured (CSV\u002FJSON) and unstructured (text) formats.\n    \"\"\"\n\n    def __init__(self, pipeline: PIIDetectionPipeline):\n        self.pipeline = pipeline\n\n    def process_csv(\n        self,\n        input_path: str,\n        output_path: str,\n        columns_to_scan: list[str] = None\n    ) -> dict:\n        \"\"\"\n        Scan and redact PII in a CSV file.\n\n        Args:\n            input_path: Path to input CSV\n            output_path: Path to write redacted CSV\n            columns_to_scan: Columns to scan (all if None)\n\n        Returns:\n            Summary statistics of detections\n        \"\"\"\n        import pandas as pd\n\n        df = pd.read_csv(input_path)\n        stats = {\"total_rows\": len(df), \"detections\": {}, \"columns_scanned\": []}\n\n        scan_columns = columns_to_scan or df.columns.tolist()\n\n        for col in scan_columns:\n            if col not in df.columns:\n                continue\n\n            stats[\"columns_scanned\"].append(col)\n            col_detections = []\n\n            for idx, value in df[col].items():\n                if pd.isna(value):\n                    continue\n\n                text = str(value)\n                redacted, detections = self.pipeline.redact(text)\n                df.at[idx, col] = redacted\n\n                for d in detections:\n                    col_detections.append(d.entity_type)\n\n            # Count detection types for this column\n            for entity_type in set(col_detections):\n                count = col_detections.count(entity_type)\n                key = f\"{col}:{entity_type}\"\n                stats[\"detections\"][key] = count\n\n        df.to_csv(output_path, index=False)\n        stats[\"total_detections\"] = sum(stats[\"detections\"].values())\n\n        return stats\n\n    def process_text_files(\n        self,\n        file_paths: list[str],\n        output_dir: str\n    ) -> dict:\n        \"\"\"\n        Batch process text files for PII detection and redaction.\n\n        Returns summary statistics.\n        \"\"\"\n        import os\n\n        stats = {\n            \"files_processed\": 0,\n            \"total_detections\": 0,\n            \"entity_counts\": {},\n            \"high_risk_files\": []\n        }\n\n        for file_path in file_paths:\n            with open(file_path, \"r\", encoding=\"utf-8\") as f:\n                text = f.read()\n\n            redacted, detections = self.pipeline.redact(text)\n\n            # Write redacted output\n            output_path = os.path.join(output_dir, os.path.basename(file_path))\n            with open(output_path, \"w\", encoding=\"utf-8\") as f:\n                f.write(redacted)\n\n            # Update statistics\n            stats[\"files_processed\"] += 1\n            stats[\"total_detections\"] += len(detections)\n\n            for d in detections:\n                stats[\"entity_counts\"][d.entity_type] = (\n                    stats[\"entity_counts\"].get(d.entity_type, 0) + 1\n                )\n\n            # Flag high-risk files (contain critical PII)\n            critical_types = {\"US_SSN\", \"CREDIT_CARD\", \"MEDICAL_LICENSE\", \"UK_NHS\"}\n            if any(d.entity_type in critical_types for d in detections):\n                stats[\"high_risk_files\"].append(file_path)\n\n        return stats\n```\n\n## spaCy NER Custom Training\n\n### Training Custom Entity Types\n\n```python\n\"\"\"\nTrain custom spaCy NER model for domain-specific PII entities.\n\"\"\"\n\nimport spacy\nfrom spacy.training import Example\nimport random\n\n\ndef create_training_data() -> list[tuple[str, dict]]:\n    \"\"\"\n    Create training examples for custom PII entity types.\n\n    Returns list of (text, annotations) tuples.\n    \"\"\"\n    training_data = [\n        (\n            \"Patient MRN 12345678 was admitted on 2024-01-15\",\n            {\"entities\": [(12, 20, \"MEDICAL_RECORD_NUMBER\")]}\n        ),\n        (\n            \"Employee CEL-00142 reported the incident\",\n            {\"entities\": [(9, 18, \"EMPLOYEE_ID\")]}\n        ),\n        (\n            \"Policy holder number PLH-2024-99887 filed a claim\",\n            {\"entities\": [(22, 35, \"POLICY_NUMBER\")]}\n        ),\n        (\n            \"The customer with loyalty ID LYL-A1B2C3 requested data export\",\n            {\"entities\": [(29, 39, \"LOYALTY_ID\")]}\n        ),\n    ]\n    return training_data\n\n\ndef train_custom_ner(\n    base_model: str = \"en_core_web_lg\",\n    training_data: list = None,\n    n_iter: int = 30,\n    output_dir: str = \".\u002Fcustom_ner_model\"\n):\n    \"\"\"\n    Fine-tune spaCy NER model with custom PII entity types.\n\n    Args:\n        base_model: Base spaCy model to fine-tune\n        training_data: List of (text, annotations) tuples\n        n_iter: Number of training iterations\n        output_dir: Directory to save trained model\n    \"\"\"\n    if training_data is None:\n        training_data = create_training_data()\n\n    nlp = spacy.load(base_model)\n\n    # Get or create NER pipe\n    if \"ner\" not in nlp.pipe_names:\n        ner = nlp.add_pipe(\"ner\", last=True)\n    else:\n        ner = nlp.get_pipe(\"ner\")\n\n    # Add custom entity labels\n    custom_labels = set()\n    for _, annotations in training_data:\n        for ent in annotations.get(\"entities\", []):\n            custom_labels.add(ent[2])\n\n    for label in custom_labels:\n        ner.add_label(label)\n\n    # Train\n    optimizer = nlp.resume_training()\n    other_pipes = [pipe for pipe in nlp.pipe_names if pipe != \"ner\"]\n\n    with nlp.disable_pipes(*other_pipes):\n        for iteration in range(n_iter):\n            random.shuffle(training_data)\n            losses = {}\n\n            for text, annotations in training_data:\n                doc = nlp.make_doc(text)\n                example = Example.from_dict(doc, annotations)\n                nlp.update([example], drop=0.35, sgd=optimizer, losses=losses)\n\n            if iteration % 10 == 0:\n                print(f\"Iteration {iteration}, Losses: {losses}\")\n\n    nlp.to_disk(output_dir)\n    return nlp\n```\n\n## AWS Macie Integration\n\n### Architecture\n\n```\nS3 Buckets --> Macie Classification Jobs --> Findings --> EventBridge --> Lambda\n                                                |                          |\n                                                v                          v\n                                        Security Hub              Remediation\n                                        (centralized)             - Tag sensitive\n                                                                  - Encrypt\n                                                                  - Notify owner\n                                                                  - Quarantine\n```\n\n### Macie Job Configuration\n\n```python\n\"\"\"\nAWS Macie integration for cloud-native PII detection in S3 buckets.\n\"\"\"\n\nimport boto3\nfrom datetime import datetime\n\n\nclass MacieIntegration:\n    \"\"\"\n    Configure and manage AWS Macie classification jobs for\n    automated PII detection across S3 data stores.\n    \"\"\"\n\n    def __init__(self, region: str = \"us-east-1\"):\n        self.macie_client = boto3.client(\"macie2\", region_name=region)\n\n    def create_classification_job(\n        self,\n        bucket_name: str,\n        job_name: str,\n        custom_data_identifiers: list[str] = None,\n        schedule: str = \"ONE_TIME\"\n    ) -> str:\n        \"\"\"\n        Create a Macie classification job for an S3 bucket.\n\n        Args:\n            bucket_name: Target S3 bucket\n            job_name: Descriptive job name\n            custom_data_identifiers: IDs of custom data identifier resources\n            schedule: ONE_TIME or SCHEDULED\n\n        Returns:\n            Job ID\n        \"\"\"\n        job_config = {\n            \"name\": job_name,\n            \"description\": f\"PII detection scan for {bucket_name}\",\n            \"jobType\": schedule,\n            \"s3JobDefinition\": {\n                \"bucketDefinitions\": [\n                    {\n                        \"accountId\": self._get_account_id(),\n                        \"buckets\": [bucket_name]\n                    }\n                ],\n                \"scoping\": {\n                    \"includes\": {\n                        \"and\": [\n                            {\n                                \"simpleScopeTerm\": {\n                                    \"comparator\": \"STARTS_WITH\",\n                                    \"key\": \"OBJECT_EXTENSION\",\n                                    \"values\": [\"csv\", \"json\", \"txt\", \"pdf\", \"docx\", \"xlsx\", \"parquet\"]\n                                }\n                            }\n                        ]\n                    }\n                }\n            },\n            \"managedDataIdentifierSelector\": \"ALL\",\n            \"tags\": {\n                \"Team\": \"Privacy Engineering\",\n                \"Purpose\": \"PII Detection\"\n            }\n        }\n\n        if custom_data_identifiers:\n            job_config[\"customDataIdentifierIds\"] = custom_data_identifiers\n\n        response = self.macie_client.create_classification_job(**job_config)\n        return response[\"jobId\"]\n\n    def create_custom_data_identifier(\n        self,\n        name: str,\n        description: str,\n        regex: str,\n        keywords: list[str] = None,\n        maximum_match_distance: int = 50\n    ) -> str:\n        \"\"\"\n        Create a custom data identifier for organization-specific PII patterns.\n\n        Returns:\n            Custom data identifier ID\n        \"\"\"\n        params = {\n            \"name\": name,\n            \"description\": description,\n            \"regex\": regex,\n            \"maximumMatchDistance\": maximum_match_distance\n        }\n\n        if keywords:\n            params[\"keywords\"] = keywords\n\n        response = self.macie_client.create_custom_data_identifier(**params)\n        return response[\"customDataIdentifierId\"]\n\n    def _get_account_id(self) -> str:\n        \"\"\"Get the current AWS account ID.\"\"\"\n        sts = boto3.client(\"sts\")\n        return sts.get_caller_identity()[\"Account\"]\n\n    def get_findings_summary(self, job_id: str) -> dict:\n        \"\"\"Get summary of findings from a classification job.\"\"\"\n        response = self.macie_client.list_findings(\n            findingCriteria={\n                \"criterion\": {\n                    \"classificationDetails.jobId\": {\n                        \"eq\": [job_id]\n                    }\n                }\n            }\n        )\n\n        findings = []\n        if response[\"findingIds\"]:\n            details = self.macie_client.get_findings(findingIds=response[\"findingIds\"])\n            findings = details[\"findings\"]\n\n        summary = {\n            \"total_findings\": len(findings),\n            \"severity_counts\": {},\n            \"entity_type_counts\": {},\n            \"affected_objects\": []\n        }\n\n        for finding in findings:\n            severity = finding.get(\"severity\", {}).get(\"description\", \"unknown\")\n            summary[\"severity_counts\"][severity] = (\n                summary[\"severity_counts\"].get(severity, 0) + 1\n            )\n\n            sensitive_data = finding.get(\"classificationDetails\", {}).get(\n                \"result\", {}\n            ).get(\"sensitiveData\", [])\n\n            for sd in sensitive_data:\n                category = sd.get(\"category\", \"unknown\")\n                summary[\"entity_type_counts\"][category] = (\n                    summary[\"entity_type_counts\"].get(category, 0)\n                    + sd.get(\"totalCount\", 0)\n                )\n\n            resource = finding.get(\"resourcesAffected\", {}).get(\"s3Object\", {})\n            if resource:\n                summary[\"affected_objects\"].append(resource.get(\"key\", \"unknown\"))\n\n        return summary\n```\n\n## Confidence Scoring Framework\n\n| Score Range | Confidence Level | Recommended Action |\n|-------------|-----------------|-------------------|\n| 0.95 - 1.00 | Very High | Auto-redact |\n| 0.80 - 0.94 | High | Auto-redact with logging |\n| 0.60 - 0.79 | Medium | Flag for human review |\n| 0.40 - 0.59 | Low | Log only, no action |\n| 0.00 - 0.39 | Very Low | Ignore |\n\n## References\n\n- Microsoft Presidio Documentation: microsoft.github.io\u002Fpresidio\n- spaCy NER Documentation: spacy.io\u002Fusage\u002Flinguistic-features#named-entities\n- AWS Macie Documentation: docs.aws.amazon.com\u002Fmacie\n- Google Cloud DLP API Documentation\n- NIST SP 800-188 — De-Identifying Government Datasets\n- Article 29 WP Opinion 05\u002F2014 on Anonymisation Techniques\n",{"name":5,"license":15,"metadata":40,"description":7},{"tags":41,"author":9,"domain":32,"version":42,"subdomain":43},"pii-detection, presidio, spacy-ner, data-redaction, aws-macie","1.0","privacy-engineering","main","9b2ef9eae161c00a17241d42a388571321b33e9f","plugins\u002Fprivacy-engineering-skills\u002Fskills\u002Fpii-detection-pipeline",67,"https:\u002F\u002Fgithub.com\u002Fmukul975\u002FPrivacy-Data-Protection-Skills","https:\u002F\u002Fgithub.com\u002Fmukul975","Mahipal.engineer","🔐 Cybersecurity | 💻 Dev | 📷 Street Photographer\r\n🎓 MSc | 📚 Research & AI Security","2026-03-17 04:06:52",{"source":8,"skillName":5,"commands":54,"prompt":64,"zipAvailable":65},[55,58,61],{"tool":56,"command":57},"npx","npx skills add mukul975\u002FPrivacy-Data-Protection-Skills --skill pii-detection-pipeline",{"tool":59,"command":60},"bunx","bunx skills add mukul975\u002FPrivacy-Data-Protection-Skills --skill pii-detection-pipeline",{"tool":62,"command":63},"pnpm","pnpm dlx skills add mukul975\u002FPrivacy-Data-Protection-Skills --skill pii-detection-pipeline","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fmukul975\u002FPrivacy-Data-Protection-Skills\u002Fblob\u002Fmain\u002Fplugins\u002Fprivacy-engineering-skills\u002Fskills\u002Fpii-detection-pipeline\u002FSKILL.md",true,[67,73,79,83,87,93],{"path":68,"resourceType":69,"resourceTypeDesc":70,"sizeBytes":71,"sourceHtmlUrl":72},"agents\u002Fopenai.yaml","REFERENCE","参考文档",102,"https:\u002F\u002Fgithub.com\u002Fhareki\u002Fdotfiles\u002Fblob\u002Fmain\u002Fagents\u002F.agents\u002Fskills\u002Fwriting-for-agents\u002Fagents\u002Fopenai.yaml",{"path":74,"resourceType":75,"resourceTypeDesc":76,"sizeBytes":77,"sourceHtmlUrl":78},"assets\u002Ftemplate.md","ASSET","资产文件",1055,"https:\u002F\u002Fgithub.com\u002Fmukul975\u002FPrivacy-Data-Protection-Skills\u002Fblob\u002Fmain\u002Fplugins\u002Fprivacy-engineering-skills\u002Fskills\u002Fpii-detection-pipeline\u002Fassets\u002Ftemplate.md",{"path":80,"resourceType":69,"resourceTypeDesc":70,"sizeBytes":81,"sourceHtmlUrl":82},"references\u002Fstandards.md",942,"https:\u002F\u002Fgithub.com\u002Fmukul975\u002FPrivacy-Data-Protection-Skills\u002Fblob\u002Fmain\u002Fplugins\u002Fprivacy-engineering-skills\u002Fskills\u002Fpii-detection-pipeline\u002Freferences\u002Fstandards.md",{"path":84,"resourceType":69,"resourceTypeDesc":70,"sizeBytes":85,"sourceHtmlUrl":86},"references\u002Fworkflows.md",1527,"https:\u002F\u002Fgithub.com\u002Fmukul975\u002FPrivacy-Data-Protection-Skills\u002Fblob\u002Fmain\u002Fplugins\u002Fprivacy-engineering-skills\u002Fskills\u002Fpii-detection-pipeline\u002Freferences\u002Fworkflows.md",{"path":88,"resourceType":89,"resourceTypeDesc":90,"sizeBytes":91,"sourceHtmlUrl":92},"scripts\u002Fprocess.py","SCRIPT","脚本",6128,"https:\u002F\u002Fgithub.com\u002Fmukul975\u002FPrivacy-Data-Protection-Skills\u002Fblob\u002Fmain\u002Fplugins\u002Fprivacy-engineering-skills\u002Fskills\u002Fpii-detection-pipeline\u002Fscripts\u002Fprocess.py",{"path":94,"resourceType":69,"resourceTypeDesc":70,"sizeBytes":95,"sourceHtmlUrl":96},"SKILL-MECHANICS.md",2629,"https:\u002F\u002Fgithub.com\u002Fhareki\u002Fdotfiles\u002Fblob\u002Fmain\u002Fagents\u002F.agents\u002Fskills\u002Fwriting-for-agents\u002FSKILL-MECHANICS.md",[98,124,139,152,169,182],{"skillNo":99,"name":100,"nameZh":6,"description":101,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":102,"categories":103,"keywords":115,"publishedAt":35},"01M6AG6K8R3XW10ENWH2FH18RH","workplace-email-privacy","Implements email and internet monitoring compliance in the workplace per Barbulescu v Romania (ECHR Grand Chamber), EDPB guidance, and national labour law. Covers acceptable use policies, legitimate expectation of privacy, proportionality testing, and content vs metadata monitoring. Keywords: email monitoring, Barbulescu, workplace privacy, internet monitoring, acceptable use policy, ECHR, proportionality.","https:\u002F\u002Fgithub.com\u002Fmukul975\u002FPrivacy-Data-Protection-Skills\u002Fblob\u002Fmain\u002Fskills\u002Fprivacy\u002Fworkplace-email-privacy\u002FSKILL.md",[104,109,112],{"categoryNo":105,"categoryName":106,"taxonomyType":107,"taxonomyTypeDesc":108},"01JBSEEDCAT000000000000005","邮件与沟通","SCENARIO","使用场景",{"categoryNo":110,"categoryName":111,"taxonomyType":107,"taxonomyTypeDesc":108},"01JBSEEDCAT000000000000008","测试与质量",{"categoryNo":113,"categoryName":114,"taxonomyType":21,"taxonomyTypeDesc":22},"01JBSEEDCAT000000000000037","律师与法务",[116,117,32,118,119,33,120,121,122,123],"email","workplace email privacy","compliance","content","policy","testing","monitoring","metadata",{"skillNo":125,"name":126,"nameZh":6,"description":127,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":128,"categories":129,"keywords":132,"publishedAt":35},"01M6AG6FRCBJY7D061BJZMNV49","whistleblower-data","Implements data protection compliance for whistleblowing systems under EU Directive 2019\u002F1937 and GDPR. Covers anonymous reporting channels, identity protection for whistleblowers and accused persons, retention limits, access restrictions, and retaliation prevention. Addresses national transpositions and DPA guidance. Keywords: whistleblower, Directive 2019\u002F1937, anonymous reporting, identity protection, retaliation, retention, reporting channel.","https:\u002F\u002Fgithub.com\u002Fmukul975\u002FPrivacy-Data-Protection-Skills\u002Fblob\u002Fmain\u002Fskills\u002Fprivacy\u002Fwhistleblower-data\u002FSKILL.md",[130,131],{"categoryNo":19,"categoryName":20,"taxonomyType":21,"taxonomyTypeDesc":22},{"categoryNo":113,"categoryName":114,"taxonomyType":21,"taxonomyTypeDesc":22},[33,133,134,135,118,136,137,32,138],"whistleblower data","identity","reporting","data protection","gdpr","prevention",{"skillNo":140,"name":141,"nameZh":6,"description":142,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":143,"categories":144,"keywords":146,"publishedAt":35},"01M6AG6CJP0PW3X65HC7VP9SR9","vendor-termination-data","Vendor termination data return and deletion procedures per GDPR Article 28(3)(g). Covers data extraction formats, deletion certification requirements, transition planning, residual data handling, and post-termination verification.","https:\u002F\u002Fgithub.com\u002Fmukul975\u002FPrivacy-Data-Protection-Skills\u002Fblob\u002Fmain\u002Fskills\u002Fprivacy\u002Fvendor-termination-data\u002FSKILL.md",[145],{"categoryNo":19,"categoryName":20,"taxonomyType":21,"taxonomyTypeDesc":22},[33,147,148,149,150,137,32,151],"vendor termination data","planning","verification","extraction","certification",{"skillNo":153,"name":154,"nameZh":6,"description":155,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":156,"categories":157,"keywords":159,"publishedAt":35},"01M6AG69E277DXZB2NF6KV17ZA","vendor-risk-scoring","Vendor privacy risk tiering methodology for processor management. Covers scoring factors including data volume, sensitivity, transfer locations, certifications, breach history, and control maturity with weighted risk calculation and tier assignment.","https:\u002F\u002Fgithub.com\u002Fmukul975\u002FPrivacy-Data-Protection-Skills\u002Fblob\u002Fmain\u002Fskills\u002Fprivacy\u002Fvendor-risk-scoring\u002FSKILL.md",[158],{"categoryNo":19,"categoryName":20,"taxonomyType":21,"taxonomyTypeDesc":22},[160,161,162,163,164,165,166,167,32,33,168],"vendor risk scoring","risk","risk scoring","methodology","risk-scoring","assessment","transfer","certifications","processor",{"skillNo":170,"name":171,"nameZh":6,"description":172,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":173,"categories":174,"keywords":176,"publishedAt":35},"01M6AG6655ABFKT2BBWHCP4PAT","vendor-privacy-due-diligence","Pre-contract vendor privacy due diligence per GDPR Article 28(1). Covers risk questionnaires, technical controls assessment, certification review, data flow analysis, and documented sufficiency decisions for processor engagement.","https:\u002F\u002Fgithub.com\u002Fmukul975\u002FPrivacy-Data-Protection-Skills\u002Fblob\u002Fmain\u002Fskills\u002Fprivacy\u002Fvendor-privacy-due-diligence\u002FSKILL.md",[175],{"categoryNo":113,"categoryName":114,"taxonomyType":21,"taxonomyTypeDesc":22},[32,177,137,178,165,179,180,33,168,161,151,181],"vendor privacy due diligence","review","onboarding","flow","analysis",{"skillNo":183,"name":184,"nameZh":6,"description":185,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":186,"categories":187,"keywords":195,"publishedAt":35},"01M6AG62C3VD8SYNGH4N7RHZTE","vendor-privacy-audit","On-site and remote vendor audit procedures per GDPR Article 28(3)(h). Covers audit planning, evidence collection methodologies, finding classification, remediation tracking, and audit report generation for processor compliance verification.","https:\u002F\u002Fgithub.com\u002Fmukul975\u002FPrivacy-Data-Protection-Skills\u002Fblob\u002Fmain\u002Fskills\u002Fprivacy\u002Fvendor-privacy-audit\u002FSKILL.md",[188,191,194],{"categoryNo":189,"categoryName":190,"taxonomyType":107,"taxonomyTypeDesc":108},"01JBSEEDCAT000000000000018","写作与文案",{"categoryNo":192,"categoryName":193,"taxonomyType":21,"taxonomyTypeDesc":22},"01JBSEEDCAT000000000000036","内容创作者",{"categoryNo":113,"categoryName":114,"taxonomyType":21,"taxonomyTypeDesc":22},[196,197,32,198,149,199,118,137,200,168,201,202,203,148],"vendor privacy audit","audit","report","audit report","remediation","collection","remediation-tracking","classification"]