[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M79E6X09MEEX2VH8RGQ845HF":3,"skill-same:01M79E6X09MEEX2VH8RGQ845HF":48},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":19,"linkOnly":20,"markdownContent":21,"markdownBody":22,"frontMatter":23,"refName":24,"commitSha":25,"skillDirectory":26,"forksCount":27,"repositoryHtmlUrl":28,"repositoryHomepageUrl":29,"ownerHtmlUrl":30,"ownerBlogUrl":31,"ownerDescription":32,"updatedAt":33,"install":34,"resources":47},"01M79E6X09MEEX2VH8RGQ845HF","pyspark-and-apache-spark-analytics",null,"Master production PySpark and Apache Spark analytics, distributed data pipelines, Delta Lake\u002FIceberg optimization, memory management, and skew resolution.","hamzabellouch\u002Fagent-skills","hamzabellouch","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F237856042?v=4","01M79BRYW7XAMK3YDWQEMTWGWS","Hamza Bellouch","01M79BRYWXQV3JK2TGVTX8GN2A",8,"MIT","https:\u002F\u002Fgithub.com\u002Fhamzabellouch\u002Fagent-skills\u002Fblob\u002Fmain\u002FProgramming\u002FData_Engineering\u002FPipelines_and_ETL\u002Fpyspark-and-apache-spark-analytics\u002FSKILL.md",[],[],"2026-11-15 14:19:17",false,"---\nname: pyspark-and-apache-spark-analytics\ndescription: Master production PySpark and Apache Spark analytics, distributed data pipelines, Delta Lake\u002FIceberg optimization, memory management, and skew resolution.\n---\n\n# PySpark & Apache Spark Analytics\n\nComprehensive engineering guide for architecting, optimizing, and operating large-scale distributed data processing pipelines using PySpark and Apache Spark 3.x\u002F4.x.\n\n---\n\n## 1. Core Principles & Architecture\n\n### 1.1 Execution Model\n- **Driver**: Plans execution, constructs DAGs, manages SparkSession, coordinates tasks, and collects metrics.\n- **Executors**: Worker processes executing task threads, maintaining in-memory storage (RAM), and handling spill to disk.\n- **Transformations vs Actions**:\n  - **Narrow Transformations**: Operations where each input partition maps to at most one output partition (`map`, `filter`, `flatmap`). No network shuffle.\n  - **Wide Transformations**: Operations requiring data exchange across cluster partitions (`groupByKey`, `reduceByKey`, `join`, `distinct`, `repartition`). Triggers a **Shuffle Stage**.\n  - **Actions**: Trigger execution of physical execution plan (`count`, `collect`, `write`, `first`, `take`).\n\n### 1.2 Data Lake storage Formats: Delta Lake & Apache Iceberg\n- **Acid Transactions**: Transaction log (`_delta_log` or Iceberg metadata trees) providing ACID semantics, time travel, and rollback.\n- **Medallion Architecture**:\n  - **Bronze (Raw)**: Append-only ingestion layer preserving original schema (JSON, CSV, Parquet).\n  - **Silver (Cleaned\u002FEnriched)**: Deduplicated, typed, normalized, and joined domain entities.\n  - **Gold (Aggregated\u002FBusiness)**: Star-schema dimensions\u002Ffacts, aggregated metrics, wide tables optimized for BI\u002FML queries.\n\n---\n\n## 2. Idempotency & Partitioning Strategies\n\n### 2.1 Dynamic Partition Overwrite & Upserts\nIdempotence guarantees that re-running a pipeline execution produces identical state without duplicate records or partial writes.\n\n#### PySpark Dynamic Partition Overwrite\n```python\nfrom pyspark.sql import SparkSession\n\nspark = SparkSession.builder \\\n    .appName(\"IdempotentPartitionWriter\") \\\n    .config(\"spark.sql.sources.partitionOverwriteMode\", \"dynamic\") \\\n    .getOrCreate()\n\n# Ensures only affected partitions are replaced upon re-run\ndf_processed.write \\\n    .format(\"parquet\") \\\n    .mode(\"overwrite\") \\\n    .partitionBy(\"event_date\") \\\n    .save(\"s3a:\u002F\u002Fdata-warehouse\u002Fsilver\u002Fuser_events\u002F\")\n```\n\n#### Delta Lake MERGE (Upsert) Pattern\n```python\nfrom delta.tables import DeltaTable\n\ndef upsert_to_delta(micro_batch_df, batch_id):\n    delta_table = DeltaTable.forPath(spark, \"s3a:\u002F\u002Fdata-warehouse\u002Fsilver\u002Fuser_profiles\")\n    \n    delta_table.alias(\"target\").merge(\n        micro_batch_df.alias(\"updates\"),\n        \"target.user_id = updates.user_id\"\n    ).whenMatchedUpdate(set={\n        \"email\": \"updates.email\",\n        \"updated_at\": \"updates.updated_at\",\n        \"status\": \"updates.status\"\n    }).whenNotMatchedInsert(values={\n        \"user_id\": \"updates.user_id\",\n        \"email\": \"updates.email\",\n        \"created_at\": \"updates.created_at\",\n        \"updated_at\": \"updates.updated_at\",\n        \"status\": \"updates.status\"\n    }).execute()\n```\n\n---\n\n## 3. Performance Optimization & Tuning\n\n### 3.1 Skew Mitigation Techniques\nData skew occurs when one partition receives disproportionately more data, stalling stage completion (99% task completion syndrome).\n\n#### Salting Technique for Skewed Joins\n```python\nfrom pyspark.sql import functions as F\n\nSALT_FACTOR = 8\n\n# Add random salt to skewed left table\ndf_skewed = df_large.withColumn(\"salt\", (F.rand() * SALT_FACTOR).cast(\"integer\"))\n\n# Explode right table to match all salt keys\ndf_small_salted = df_small.withColumn(\"salt_array\", F.array([F.lit(i) for i in range(SALT_FACTOR)])) \\\n    .withColumn(\"salt\", F.explode(\"salt_array\")) \\\n    .drop(\"salt_array\")\n\n# Perform join on join_key AND salt key\ndf_joined = df_skewed.join(\n    df_small_salted,\n    on=[\"join_key\", \"salt\"],\n    how=\"inner\"\n).drop(\"salt\")\n```\n\n### 3.2 Adaptive Query Execution (AQE)\nEnable AQE to dynamically re-optimize execution plans based on runtime statistics:\n```python\nspark.conf.set(\"spark.sql.adaptive.enabled\", \"true\")\nspark.conf.set(\"spark.sql.adaptive.coalescePartitions.enabled\", \"true\")\nspark.conf.set(\"spark.sql.adaptive.skewJoin.enabled\", \"true\")\nspark.conf.set(\"spark.sql.adaptive.skewJoin.skewedPartitionFactor\", \"5\")\nspark.conf.set(\"spark.sql.adaptive.skewJoin.skewedPartitionThresholdInBytes\", \"256MB\")\n```\n\n### 3.3 Broadcast Join Tuning\nAvoid network shuffles for small lookup tables (\u003C10MB default, up to 1GB with tuning):\n```python\nfrom pyspark.sql.functions import broadcast\n\n# Explicitly hint Spark to broadcast small dimension\ndf_fact.join(broadcast(df_dim_small), \"dim_id\", \"inner\")\n```\n\n---\n\n## 4. Production Anti-Patterns\n\n| Anti-Pattern | Operational Risk | Production Remedy |\n| :--- | :--- | :--- |\n| Calling `.collect()` on large DataFrames | Driver Out-Of-Memory (OOM) crash | Use `.take(n)`, write directly to storage, or sample data |\n| Using `DataFrame.toPandas()` in distributed loop | Serialization bottleneck & Driver OOM | Use PySpark Pandas API or PyArrow UDFs (`pandas_udf`) |\n| Unpartitioned `.repartition(1)` before write | Single executor bottleneck & disk spill | Use `.coalesce(n)` or dynamic partition overwrites |\n| Row-by-row `.iterrows()` in Python UDFs | Context switching overhead (JVM to Python) | Native Spark SQL functions or vectorized `pandas_udf` |\n| Over-partitioning (tiny files problem) | NameNode \u002F S3 metadata strain, slow listing | Target ~128MB - 512MB file sizes, run `OPTIMIZE` \u002F `VORDER` |\n\n---\n\n## 5. Production Implementation Blueprint\n\n### Scalable Pipeline with Vectorized UDFs & Structural Monitoring\n\n```python\nimport sys\nimport logging\nfrom typing import Iterator\nimport pandas as pd\nfrom pyspark.sql import SparkSession, DataFrame\nfrom pyspark.sql import functions as F\nfrom pyspark.sql.types import DoubleType, StructType, StructField, StringType, TimestampType\n\nlogging.basicConfig(level=logging.INFO)\nlogger = logging.getLogger(\"SparkAnalyticsEngine\")\n\ndef build_spark_session(app_name: str) -> SparkSession:\n    \"\"\"Configures high-throughput Spark session optimized for S3\u002FDelta processing.\"\"\"\n    return SparkSession.builder \\\n        .appName(app_name) \\\n        .config(\"spark.sql.adaptive.enabled\", \"true\") \\\n        .config(\"spark.sql.adaptive.coalescePartitions.enabled\", \"true\") \\\n        .config(\"spark.serializer\", \"org.apache.spark.serializer.KryoSerializer\") \\\n        .config(\"spark.sql.extensions\", \"io.delta.sql.DeltaSparkSessionExtension\") \\\n        .config(\"spark.sql.catalog.spark_catalog\", \"org.apache.spark.sql.delta.catalog.DeltaCatalog\") \\\n        .getOrCreate()\n\n# Vectorized Pandas UDF for scalable ML\u002FAnalytical scoring\n@F.pandas_udf(DoubleType())\ndef calculate_risk_score_udf(amount_series: pd.Series, velocity_series: pd.Series) -> pd.Series:\n    \"\"\"Vectorized calculation executed in Arrow batches across worker nodes.\"\"\"\n    return (amount_series * 0.45) + (velocity_series * 1.85)\n\ndef process_transaction_bronze_to_silver(\n    spark: SparkSession,\n    bronze_path: str,\n    silver_path: str,\n    partition_date: str\n) -> None:\n    \"\"\"Idempotent Bronze-to-Silver transformation pipeline.\"\"\"\n    logger.info(f\"Ingesting Bronze payload from {bronze_path} for date: {partition_date}\")\n\n    raw_df = spark.read.format(\"delta\").load(bronze_path) \\\n        .filter(F.col(\"event_date\") == partition_date)\n\n    # Data Quality Cleanse & Deduplication\n    cleaned_df = raw_df \\\n        .filter(F.col(\"transaction_id\").isNotNull()) \\\n        .dropDuplicates([\"transaction_id\"]) \\\n        .withColumn(\"amount\", F.col(\"amount\").cast(DoubleType())) \\\n        .withColumn(\"velocity\", F.col(\"velocity\").cast(DoubleType())) \\\n        .withColumn(\"risk_score\", calculate_risk_score_udf(F.col(\"amount\"), F.col(\"velocity\"))) \\\n        .withColumn(\"processed_at\", F.current_timestamp())\n\n    logger.info(\"Writing clean records to Silver Delta table with partition isolation...\")\n    \n    cleaned_df.write \\\n        .format(\"delta\") \\\n        .mode(\"overwrite\") \\\n        .option(\"replaceWhere\", f\"event_date = '{partition_date}'\") \\\n        .save(silver_path)\n\n    logger.info(f\"Successfully processed Silver partition: {partition_date}\")\n\nif __name__ == \"__main__\":\n    spark = build_spark_session(\"ProductionSparkPipeline\")\n    try:\n        process_transaction_bronze_to_silver(\n            spark=spark,\n            bronze_path=\"s3a:\u002F\u002Fenterprise-datalake\u002Fbronze\u002Ftransactions\",\n            silver_path=\"s3a:\u002F\u002Fenterprise-datalake\u002Fsilver\u002Ftransactions\",\n            partition_date=\"2026-07-18\"\n        )\n    except Exception as e:\n        logger.error(f\"Pipeline execution failure: {str(e)}\")\n        sys.exit(1)\n    finally:\n        spark.stop()\n```\n","# PySpark & Apache Spark Analytics\n\nComprehensive engineering guide for architecting, optimizing, and operating large-scale distributed data processing pipelines using PySpark and Apache Spark 3.x\u002F4.x.\n\n---\n\n## 1. Core Principles & Architecture\n\n### 1.1 Execution Model\n- **Driver**: Plans execution, constructs DAGs, manages SparkSession, coordinates tasks, and collects metrics.\n- **Executors**: Worker processes executing task threads, maintaining in-memory storage (RAM), and handling spill to disk.\n- **Transformations vs Actions**:\n  - **Narrow Transformations**: Operations where each input partition maps to at most one output partition (`map`, `filter`, `flatmap`). No network shuffle.\n  - **Wide Transformations**: Operations requiring data exchange across cluster partitions (`groupByKey`, `reduceByKey`, `join`, `distinct`, `repartition`). Triggers a **Shuffle Stage**.\n  - **Actions**: Trigger execution of physical execution plan (`count`, `collect`, `write`, `first`, `take`).\n\n### 1.2 Data Lake storage Formats: Delta Lake & Apache Iceberg\n- **Acid Transactions**: Transaction log (`_delta_log` or Iceberg metadata trees) providing ACID semantics, time travel, and rollback.\n- **Medallion Architecture**:\n  - **Bronze (Raw)**: Append-only ingestion layer preserving original schema (JSON, CSV, Parquet).\n  - **Silver (Cleaned\u002FEnriched)**: Deduplicated, typed, normalized, and joined domain entities.\n  - **Gold (Aggregated\u002FBusiness)**: Star-schema dimensions\u002Ffacts, aggregated metrics, wide tables optimized for BI\u002FML queries.\n\n---\n\n## 2. Idempotency & Partitioning Strategies\n\n### 2.1 Dynamic Partition Overwrite & Upserts\nIdempotence guarantees that re-running a pipeline execution produces identical state without duplicate records or partial writes.\n\n#### PySpark Dynamic Partition Overwrite\n```python\nfrom pyspark.sql import SparkSession\n\nspark = SparkSession.builder \\\n    .appName(\"IdempotentPartitionWriter\") \\\n    .config(\"spark.sql.sources.partitionOverwriteMode\", \"dynamic\") \\\n    .getOrCreate()\n\n# Ensures only affected partitions are replaced upon re-run\ndf_processed.write \\\n    .format(\"parquet\") \\\n    .mode(\"overwrite\") \\\n    .partitionBy(\"event_date\") \\\n    .save(\"s3a:\u002F\u002Fdata-warehouse\u002Fsilver\u002Fuser_events\u002F\")\n```\n\n#### Delta Lake MERGE (Upsert) Pattern\n```python\nfrom delta.tables import DeltaTable\n\ndef upsert_to_delta(micro_batch_df, batch_id):\n    delta_table = DeltaTable.forPath(spark, \"s3a:\u002F\u002Fdata-warehouse\u002Fsilver\u002Fuser_profiles\")\n    \n    delta_table.alias(\"target\").merge(\n        micro_batch_df.alias(\"updates\"),\n        \"target.user_id = updates.user_id\"\n    ).whenMatchedUpdate(set={\n        \"email\": \"updates.email\",\n        \"updated_at\": \"updates.updated_at\",\n        \"status\": \"updates.status\"\n    }).whenNotMatchedInsert(values={\n        \"user_id\": \"updates.user_id\",\n        \"email\": \"updates.email\",\n        \"created_at\": \"updates.created_at\",\n        \"updated_at\": \"updates.updated_at\",\n        \"status\": \"updates.status\"\n    }).execute()\n```\n\n---\n\n## 3. Performance Optimization & Tuning\n\n### 3.1 Skew Mitigation Techniques\nData skew occurs when one partition receives disproportionately more data, stalling stage completion (99% task completion syndrome).\n\n#### Salting Technique for Skewed Joins\n```python\nfrom pyspark.sql import functions as F\n\nSALT_FACTOR = 8\n\n# Add random salt to skewed left table\ndf_skewed = df_large.withColumn(\"salt\", (F.rand() * SALT_FACTOR).cast(\"integer\"))\n\n# Explode right table to match all salt keys\ndf_small_salted = df_small.withColumn(\"salt_array\", F.array([F.lit(i) for i in range(SALT_FACTOR)])) \\\n    .withColumn(\"salt\", F.explode(\"salt_array\")) \\\n    .drop(\"salt_array\")\n\n# Perform join on join_key AND salt key\ndf_joined = df_skewed.join(\n    df_small_salted,\n    on=[\"join_key\", \"salt\"],\n    how=\"inner\"\n).drop(\"salt\")\n```\n\n### 3.2 Adaptive Query Execution (AQE)\nEnable AQE to dynamically re-optimize execution plans based on runtime statistics:\n```python\nspark.conf.set(\"spark.sql.adaptive.enabled\", \"true\")\nspark.conf.set(\"spark.sql.adaptive.coalescePartitions.enabled\", \"true\")\nspark.conf.set(\"spark.sql.adaptive.skewJoin.enabled\", \"true\")\nspark.conf.set(\"spark.sql.adaptive.skewJoin.skewedPartitionFactor\", \"5\")\nspark.conf.set(\"spark.sql.adaptive.skewJoin.skewedPartitionThresholdInBytes\", \"256MB\")\n```\n\n### 3.3 Broadcast Join Tuning\nAvoid network shuffles for small lookup tables (\u003C10MB default, up to 1GB with tuning):\n```python\nfrom pyspark.sql.functions import broadcast\n\n# Explicitly hint Spark to broadcast small dimension\ndf_fact.join(broadcast(df_dim_small), \"dim_id\", \"inner\")\n```\n\n---\n\n## 4. Production Anti-Patterns\n\n| Anti-Pattern | Operational Risk | Production Remedy |\n| :--- | :--- | :--- |\n| Calling `.collect()` on large DataFrames | Driver Out-Of-Memory (OOM) crash | Use `.take(n)`, write directly to storage, or sample data |\n| Using `DataFrame.toPandas()` in distributed loop | Serialization bottleneck & Driver OOM | Use PySpark Pandas API or PyArrow UDFs (`pandas_udf`) |\n| Unpartitioned `.repartition(1)` before write | Single executor bottleneck & disk spill | Use `.coalesce(n)` or dynamic partition overwrites |\n| Row-by-row `.iterrows()` in Python UDFs | Context switching overhead (JVM to Python) | Native Spark SQL functions or vectorized `pandas_udf` |\n| Over-partitioning (tiny files problem) | NameNode \u002F S3 metadata strain, slow listing | Target ~128MB - 512MB file sizes, run `OPTIMIZE` \u002F `VORDER` |\n\n---\n\n## 5. Production Implementation Blueprint\n\n### Scalable Pipeline with Vectorized UDFs & Structural Monitoring\n\n```python\nimport sys\nimport logging\nfrom typing import Iterator\nimport pandas as pd\nfrom pyspark.sql import SparkSession, DataFrame\nfrom pyspark.sql import functions as F\nfrom pyspark.sql.types import DoubleType, StructType, StructField, StringType, TimestampType\n\nlogging.basicConfig(level=logging.INFO)\nlogger = logging.getLogger(\"SparkAnalyticsEngine\")\n\ndef build_spark_session(app_name: str) -> SparkSession:\n    \"\"\"Configures high-throughput Spark session optimized for S3\u002FDelta processing.\"\"\"\n    return SparkSession.builder \\\n        .appName(app_name) \\\n        .config(\"spark.sql.adaptive.enabled\", \"true\") \\\n        .config(\"spark.sql.adaptive.coalescePartitions.enabled\", \"true\") \\\n        .config(\"spark.serializer\", \"org.apache.spark.serializer.KryoSerializer\") \\\n        .config(\"spark.sql.extensions\", \"io.delta.sql.DeltaSparkSessionExtension\") \\\n        .config(\"spark.sql.catalog.spark_catalog\", \"org.apache.spark.sql.delta.catalog.DeltaCatalog\") \\\n        .getOrCreate()\n\n# Vectorized Pandas UDF for scalable ML\u002FAnalytical scoring\n@F.pandas_udf(DoubleType())\ndef calculate_risk_score_udf(amount_series: pd.Series, velocity_series: pd.Series) -> pd.Series:\n    \"\"\"Vectorized calculation executed in Arrow batches across worker nodes.\"\"\"\n    return (amount_series * 0.45) + (velocity_series * 1.85)\n\ndef process_transaction_bronze_to_silver(\n    spark: SparkSession,\n    bronze_path: str,\n    silver_path: str,\n    partition_date: str\n) -> None:\n    \"\"\"Idempotent Bronze-to-Silver transformation pipeline.\"\"\"\n    logger.info(f\"Ingesting Bronze payload from {bronze_path} for date: {partition_date}\")\n\n    raw_df = spark.read.format(\"delta\").load(bronze_path) \\\n        .filter(F.col(\"event_date\") == partition_date)\n\n    # Data Quality Cleanse & Deduplication\n    cleaned_df = raw_df \\\n        .filter(F.col(\"transaction_id\").isNotNull()) \\\n        .dropDuplicates([\"transaction_id\"]) \\\n        .withColumn(\"amount\", F.col(\"amount\").cast(DoubleType())) \\\n        .withColumn(\"velocity\", F.col(\"velocity\").cast(DoubleType())) \\\n        .withColumn(\"risk_score\", calculate_risk_score_udf(F.col(\"amount\"), F.col(\"velocity\"))) \\\n        .withColumn(\"processed_at\", F.current_timestamp())\n\n    logger.info(\"Writing clean records to Silver Delta table with partition isolation...\")\n    \n    cleaned_df.write \\\n        .format(\"delta\") \\\n        .mode(\"overwrite\") \\\n        .option(\"replaceWhere\", f\"event_date = '{partition_date}'\") \\\n        .save(silver_path)\n\n    logger.info(f\"Successfully processed Silver partition: {partition_date}\")\n\nif __name__ == \"__main__\":\n    spark = build_spark_session(\"ProductionSparkPipeline\")\n    try:\n        process_transaction_bronze_to_silver(\n            spark=spark,\n            bronze_path=\"s3a:\u002F\u002Fenterprise-datalake\u002Fbronze\u002Ftransactions\",\n            silver_path=\"s3a:\u002F\u002Fenterprise-datalake\u002Fsilver\u002Ftransactions\",\n            partition_date=\"2026-07-18\"\n        )\n    except Exception as e:\n        logger.error(f\"Pipeline execution failure: {str(e)}\")\n        sys.exit(1)\n    finally:\n        spark.stop()\n```\n",{"name":5,"description":7},"main","e905af21e5f1e1c3ca605e9656c3a7930360a84e","Programming\u002FData_Engineering\u002FPipelines_and_ETL\u002Fpyspark-and-apache-spark-analytics",1,"https:\u002F\u002Fgithub.com\u002Fhamzabellouch\u002Fagent-skills","https:\u002F\u002Fhamzabellouch.github.io\u002F","https:\u002F\u002Fgithub.com\u002Fhamzabellouch","hamzabellouch.github.io","CEO & Founder of @tknoglobal | Researcher in Technology & Applied Sciences.","2026-10-17 03:20:19",{"source":8,"skillName":5,"commands":35,"prompt":45,"zipAvailable":46},[36,39,42],{"tool":37,"command":38},"npx","npx skills add hamzabellouch\u002Fagent-skills --skill pyspark-and-apache-spark-analytics",{"tool":40,"command":41},"bunx","bunx skills add hamzabellouch\u002Fagent-skills --skill pyspark-and-apache-spark-analytics",{"tool":43,"command":44},"pnpm","pnpm dlx skills add hamzabellouch\u002Fagent-skills --skill pyspark-and-apache-spark-analytics","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fhamzabellouch\u002Fagent-skills\u002Fblob\u002Fmain\u002FProgramming\u002FData_Engineering\u002FPipelines_and_ETL\u002Fpyspark-and-apache-spark-analytics\u002FSKILL.md",true,[],[49,56,63,70,77,84],{"skillNo":50,"name":51,"nameZh":6,"description":52,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":53,"categories":54,"keywords":55,"publishedAt":19},"01M79FHH2H815A9TAP6WNKH4Q8","security-audit","Comprehensive security guidance, source code audit, vulnerability assessment, and penetration test review for codebases, APIs, microservices, CLI tools, libraries, and LLM-backed applications. Triggers on requests to audit a codebase, conduct security reviews, find vulnerabilities or flaws, pen-test source code, or when the user asks in Arabic (فحص أمني، تدقيق الكود أمنياً، مراجعة الثغرات، فحص أمان المشروع). Supports Guidance mode (focused review) and Full Audit mode (deterministic 6-phase harness).","https:\u002F\u002Fgithub.com\u002Fhamzabellouch\u002Fagent-skills\u002Fblob\u002Fmain\u002FSecurity\u002FSecurity_Audit_and_Code_Review\u002Fsecurity-audit\u002FSKILL.md",[],[],{"skillNo":57,"name":58,"nameZh":6,"description":59,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":60,"categories":61,"keywords":62,"publishedAt":19},"01M79FHBGXVBY6VQ1X7ETAZMF1","malware-analysis-reversing","Production malware analysis workflows, reverse engineering methodologies, isolated sandbox triage, static\u002Fdynamic binary analysis, memory forensics, and YARA detection rule authoring.","https:\u002F\u002Fgithub.com\u002Fhamzabellouch\u002Fagent-skills\u002Fblob\u002Fmain\u002FSecurity\u002FOffensive_Security_and_Pentesting\u002Fmalware-analysis-reversing\u002FSKILL.md",[],[],{"skillNo":64,"name":65,"nameZh":6,"description":66,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":67,"categories":68,"keywords":69,"publishedAt":19},"01M79FH6EACDR801TZCE4R8SAC","application-security-pentesting","Production Application Security (AppSec) standards, OWASP Top 10 vulnerability remediation, SAST\u002FDAST pipeline integration, secure code review, threat modeling, and defensive security posture.","https:\u002F\u002Fgithub.com\u002Fhamzabellouch\u002Fagent-skills\u002Fblob\u002Fmain\u002FSecurity\u002FOffensive_Security_and_Pentesting\u002Fapplication-security-pentesting\u002FSKILL.md",[],[],{"skillNo":71,"name":72,"nameZh":6,"description":73,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":74,"categories":75,"keywords":76,"publishedAt":19},"01M79FH0VKVT44FK923K520ZQA","oauth2-oidc-security-flows","Implement enterprise OAuth 2.1 and OpenID Connect (OIDC) authorization code flow with Proof Key for Code Exchange (PKCE), state and nonce verification, refresh token rotation, token revocation, and centralized IdP federation (Keycloak, Auth0, Okta). Trigger when building secure authentication, single sign-on (SSO), or third-party delegated authorization.","https:\u002F\u002Fgithub.com\u002Fhamzabellouch\u002Fagent-skills\u002Fblob\u002Fmain\u002FSecurity\u002FIdentity_and_Access_Management_IAM\u002Foauth2-oidc-security-flows\u002FSKILL.md",[],[],{"skillNo":78,"name":79,"nameZh":6,"description":80,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":81,"categories":82,"keywords":83,"publishedAt":19},"01M79FGVVB3R4T88XXD8FRSV5B","jwt-session-management-hardening","Harden JWT-based authentication and session lifecycles using asymmetric signing (RS256\u002FEdDSA), JWKS rotation, strict claims validation, Redis-backed sliding window sessions, revocation lists, and secure HttpOnly cookie storage. Trigger when implementing token validation, session revocation, or hardening API authentication.","https:\u002F\u002Fgithub.com\u002Fhamzabellouch\u002Fagent-skills\u002Fblob\u002Fmain\u002FSecurity\u002FIdentity_and_Access_Management_IAM\u002Fjwt-session-management-hardening\u002FSKILL.md",[],[],{"skillNo":85,"name":86,"nameZh":6,"description":87,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":88,"categories":89,"keywords":90,"publishedAt":19},"01M79FGNCPZ85CVEY0B3NE4H3B","soc2-compliance-automation","Best practices for implementing SOC 2 Type II compliance controls, continuous evidence collection, infrastructure policy-as-code, and automated security monitoring. Use when automating Trust Services Criteria (Security, Availability, Confidentiality, Processing Integrity, Privacy), audit logging, IAM least privilege, or CI\u002FCD security gates.","https:\u002F\u002Fgithub.com\u002Fhamzabellouch\u002Fagent-skills\u002Fblob\u002Fmain\u002FSecurity\u002FGovernance_Compliance_and_Legal_Tech\u002Fsoc2-compliance-automation\u002FSKILL.md",[],[]]