[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6XETN040VGWR60XX42SS9VR":3,"skill-same:01M6XETN040VGWR60XX42SS9VR":64},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":9,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":15,"categories":16,"keywords":17,"publishedAt":32,"linkOnly":33,"markdownContent":34,"markdownBody":35,"frontMatter":36,"refName":37,"commitSha":38,"skillDirectory":39,"forksCount":13,"repositoryHtmlUrl":40,"repositoryHomepageUrl":6,"ownerHtmlUrl":41,"ownerBlogUrl":6,"ownerDescription":42,"updatedAt":43,"install":44,"resources":57},"01M6XETN040VGWR60XX42SS9VR","feature_engineering",null,"Engineer dataset features before ML or Causal Inference. Methods include encoding categorical variables, scaling numerics, creating interactions, and selecting relevant features.","Waaaalnut\u002Fskillsbench","Waaaalnut","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F103995656?v=4","01M6XCS9K1V51EY3JMB7DGMX24","01M6XCS9Q4WPW88W62C55G1H44",0,"Apache-2.0","https:\u002F\u002Fgithub.com\u002FWaaaalnut\u002Fskillsbench\u002Fblob\u002Fmain\u002Ftasks\u002Ftrend-anomaly-causal-inference\u002Fenvironment\u002Fskills\u002Ffeature_engineering\u002FSKILL.md",[],[18,19,20,21,22,23,24,25,26,27,28,29,30,31],"feature engineering","engineering","feature","inference","causal inference","ml","encoding","engineer","methods","scaling","dataset","relevant","Variables","features","2026-11-10 10:12:30",false,"---\nname: feature_engineering\ndescription: Engineer dataset features before ML or Causal Inference. Methods include encoding categorical variables, scaling numerics, creating interactions, and selecting relevant features.\n---\n\n# Feature Engineering Framework\n\nComprehensive, modular feature engineering framework general tabular datasets. Provides strategy-based operations including numerical scaling, categorical encoding, polynomial features, and feature selection through a configurable pipeline.\n\n## Core Components\n\n### FeatureEngineeringStrategies\nCollection of static methods for feature engineering operations:\n\n#### Numerical Features (if intepretability is not a concern)\n- `scale_numerical(df, columns, method)` - Scale using 'standard', 'minmax', or 'robust'\n- `create_bins(df, columns, n_bins, strategy)` - Discretize using 'uniform', 'quantile', or 'kmeans'\n- `create_polynomial_features(df, columns, degree)` - Generate polynomial and interaction terms\n- `create_interaction_features(df, column_pairs)` - Create multiplication interactions\n- `create_log_features(df, columns)` - Log-transform for skewed distributions\n\n#### Categorical Features\n- `encode_categorical(df, columns, method)` - Encode using 'onehot', 'label', 'frequency', or 'hash'\n- `create_category_aggregations(df, categorical_col, numerical_cols, agg_funcs)` - Group-level statistics\n\n#### Binary Features\n- `convert_to_binary(df, columns)` - Convert Yes\u002FNo, True\u002FFalse to 0\u002F1 (data type to int)\n\n#### Data Quality Validation\n- `validate_numeric_features(df, exclude_cols)` - Verify all features are numeric (except ID columns)\n- `validate_no_constants(df, exclude_cols)` - Remove constant columns with no variance\n\n#### Feature Selection\n- `select_features_variance(df, columns, threshold)` - Remove low-variance features (default: 0.01). For some columns that consist of almost the same values, we might consider to drop due to the low variance it brings in order to reduce dimensionality.\n- `select_features_correlation(df, columns, threshold)` - Remove highly correlated features\n\n### FeatureEngineeringPipeline\nOrchestrates multiple feature engineering steps with logging.\n\n**CRITICAL REQUIREMENTS:**\n1. **ALL output features MUST be numeric (int or float)** - DID analysis cannot use string\u002Fobject columns\n2. **Preview data types BEFORE processing**: `df.dtypes` and `df.head()` to check actual values\n3. **Encode ALL categorical variables** - strings like \"degree\", \"age_range\" must be converted to numbers\n4. **Verify output**: Final dataframe should have `df.select_dtypes(include='number').shape[1] == df.shape[1] - 1` (excluding ID column)\n\n## Usage Example\n\n```python\nfrom feature_engineering import FeatureEngineeringStrategies, FeatureEngineeringPipeline\n\n# Create pipeline\npipeline = FeatureEngineeringPipeline(name=\"Demographics\")\n\n# Add feature engineering steps\npipeline.add_step(\n    FeatureEngineeringStrategies.convert_to_binary,\n    columns=['\u003Ccolumn5>', '\u003Ccolumn2>'],\n    description=\"Convert binary survey responses to 0\u002F1\"\n).add_step(\n    FeatureEngineeringStrategies.encode_categorical,\n    columns=['\u003Ccolumn3>', '\u003Ccolumn7>'],\n    method='onehot',\n    description=\"One-hot encode categorical features\"\n).add_step(\n    FeatureEngineeringStrategies.scale_numerical,\n    columns=['\u003Ccolumn10>', '\u003Ccolumn1>'],\n    method='standard',\n    description=\"Standardize numerical features\"\n).add_step(\n    FeatureEngineeringStrategies.validate_numeric_features,\n    exclude_cols=['\u003CID Column>'],\n    description=\"Verify all features are numeric before modeling\"\n).add_step(\n    FeatureEngineeringStrategies.validate_no_constants,\n    exclude_cols=['\u003CID Column>'],\n    description=\"Remove constant columns with no predictive value\"\n).add_step(\n    FeatureEngineeringStrategies.select_features_variance,\n    columns=[],  # Empty = auto-select all numerical\n    threshold=0.01,\n    description=\"Remove low-variance features\"\n)\n\n# Execute pipeline\n# df_complete: complete returns original columns and the engineered features\ndf_complete = pipeline.execute(your_cleaned_df, verbose=True)\n\n# Shortcut: Get the ID Column with the all needed enigneered features\nengineered_features = pipeline.get_engineered_features()\ndf_id_pure_features = df_complete[['\u003CID Column>']+engineered_features]\n\n# Get execution log\nlog_df = pipeline.get_log()\n```\n\n## Input\n- A valid dataFrame that would be sent to feature engineering after any data processing, imputation, or drop (A MUST)\n\n## Output\n- DataFrame with both original and engineered columns\n- Engineered feature names accessible via `pipeline.get_engineered_features()`\n- Execution log available via `pipeline.get_log()`\n\n## Key Features\n- Multiple encoding methods for categorical variables\n- Automatic handling of high-cardinality categoricals\n- Polynomial and interaction feature generation\n- Built-in feature selection for dimensionality reduction\n- Pipeline pattern for reproducible transformations\n\n## Best Practices\n- **Always validate data types** before downstream analysis: Use `validate_numeric_features()` after encoding\n- **Check for constant columns** that provide no information: Use `validate_no_constants()` before modeling\n- Convert binary features before other transformations\n- Use one-hot encoding for low-cardinality categoricals\n- Use KNN imputation if missing value could be inferred from other relevant columns\n- Use hash encoding for high-cardinality features (IDs, etc.)\n- Apply variance threshold to remove constant features\n- Check correlation matrix before modeling to avoid multicollinearity\n- MAKE SURE ALL ENGINEERED FEATURES ARE NUMERICAL\n","# Feature Engineering Framework\n\nComprehensive, modular feature engineering framework general tabular datasets. Provides strategy-based operations including numerical scaling, categorical encoding, polynomial features, and feature selection through a configurable pipeline.\n\n## Core Components\n\n### FeatureEngineeringStrategies\nCollection of static methods for feature engineering operations:\n\n#### Numerical Features (if intepretability is not a concern)\n- `scale_numerical(df, columns, method)` - Scale using 'standard', 'minmax', or 'robust'\n- `create_bins(df, columns, n_bins, strategy)` - Discretize using 'uniform', 'quantile', or 'kmeans'\n- `create_polynomial_features(df, columns, degree)` - Generate polynomial and interaction terms\n- `create_interaction_features(df, column_pairs)` - Create multiplication interactions\n- `create_log_features(df, columns)` - Log-transform for skewed distributions\n\n#### Categorical Features\n- `encode_categorical(df, columns, method)` - Encode using 'onehot', 'label', 'frequency', or 'hash'\n- `create_category_aggregations(df, categorical_col, numerical_cols, agg_funcs)` - Group-level statistics\n\n#### Binary Features\n- `convert_to_binary(df, columns)` - Convert Yes\u002FNo, True\u002FFalse to 0\u002F1 (data type to int)\n\n#### Data Quality Validation\n- `validate_numeric_features(df, exclude_cols)` - Verify all features are numeric (except ID columns)\n- `validate_no_constants(df, exclude_cols)` - Remove constant columns with no variance\n\n#### Feature Selection\n- `select_features_variance(df, columns, threshold)` - Remove low-variance features (default: 0.01). For some columns that consist of almost the same values, we might consider to drop due to the low variance it brings in order to reduce dimensionality.\n- `select_features_correlation(df, columns, threshold)` - Remove highly correlated features\n\n### FeatureEngineeringPipeline\nOrchestrates multiple feature engineering steps with logging.\n\n**CRITICAL REQUIREMENTS:**\n1. **ALL output features MUST be numeric (int or float)** - DID analysis cannot use string\u002Fobject columns\n2. **Preview data types BEFORE processing**: `df.dtypes` and `df.head()` to check actual values\n3. **Encode ALL categorical variables** - strings like \"degree\", \"age_range\" must be converted to numbers\n4. **Verify output**: Final dataframe should have `df.select_dtypes(include='number').shape[1] == df.shape[1] - 1` (excluding ID column)\n\n## Usage Example\n\n```python\nfrom feature_engineering import FeatureEngineeringStrategies, FeatureEngineeringPipeline\n\n# Create pipeline\npipeline = FeatureEngineeringPipeline(name=\"Demographics\")\n\n# Add feature engineering steps\npipeline.add_step(\n    FeatureEngineeringStrategies.convert_to_binary,\n    columns=['\u003Ccolumn5>', '\u003Ccolumn2>'],\n    description=\"Convert binary survey responses to 0\u002F1\"\n).add_step(\n    FeatureEngineeringStrategies.encode_categorical,\n    columns=['\u003Ccolumn3>', '\u003Ccolumn7>'],\n    method='onehot',\n    description=\"One-hot encode categorical features\"\n).add_step(\n    FeatureEngineeringStrategies.scale_numerical,\n    columns=['\u003Ccolumn10>', '\u003Ccolumn1>'],\n    method='standard',\n    description=\"Standardize numerical features\"\n).add_step(\n    FeatureEngineeringStrategies.validate_numeric_features,\n    exclude_cols=['\u003CID Column>'],\n    description=\"Verify all features are numeric before modeling\"\n).add_step(\n    FeatureEngineeringStrategies.validate_no_constants,\n    exclude_cols=['\u003CID Column>'],\n    description=\"Remove constant columns with no predictive value\"\n).add_step(\n    FeatureEngineeringStrategies.select_features_variance,\n    columns=[],  # Empty = auto-select all numerical\n    threshold=0.01,\n    description=\"Remove low-variance features\"\n)\n\n# Execute pipeline\n# df_complete: complete returns original columns and the engineered features\ndf_complete = pipeline.execute(your_cleaned_df, verbose=True)\n\n# Shortcut: Get the ID Column with the all needed enigneered features\nengineered_features = pipeline.get_engineered_features()\ndf_id_pure_features = df_complete[['\u003CID Column>']+engineered_features]\n\n# Get execution log\nlog_df = pipeline.get_log()\n```\n\n## Input\n- A valid dataFrame that would be sent to feature engineering after any data processing, imputation, or drop (A MUST)\n\n## Output\n- DataFrame with both original and engineered columns\n- Engineered feature names accessible via `pipeline.get_engineered_features()`\n- Execution log available via `pipeline.get_log()`\n\n## Key Features\n- Multiple encoding methods for categorical variables\n- Automatic handling of high-cardinality categoricals\n- Polynomial and interaction feature generation\n- Built-in feature selection for dimensionality reduction\n- Pipeline pattern for reproducible transformations\n\n## Best Practices\n- **Always validate data types** before downstream analysis: Use `validate_numeric_features()` after encoding\n- **Check for constant columns** that provide no information: Use `validate_no_constants()` before modeling\n- Convert binary features before other transformations\n- Use one-hot encoding for low-cardinality categoricals\n- Use KNN imputation if missing value could be inferred from other relevant columns\n- Use hash encoding for high-cardinality features (IDs, etc.)\n- Apply variance threshold to remove constant features\n- Check correlation matrix before modeling to avoid multicollinearity\n- MAKE SURE ALL ENGINEERED FEATURES ARE NUMERICAL\n",{"name":5,"description":7},"main","7eeb258ada72b7f58119b0e7a2c00ea89d9beedd","tasks\u002Ftrend-anomaly-causal-inference\u002Fenvironment\u002Fskills\u002Ffeature_engineering","https:\u002F\u002Fgithub.com\u002FWaaaalnut\u002Fskillsbench","https:\u002F\u002Fgithub.com\u002FWaaaalnut","student of Zhejiang University","2026-04-21 23:16:23",{"source":8,"skillName":5,"commands":45,"prompt":55,"zipAvailable":56},[46,49,52],{"tool":47,"command":48},"npx","npx skills add Waaaalnut\u002Fskillsbench --skill feature_engineering",{"tool":50,"command":51},"bunx","bunx skills add Waaaalnut\u002Fskillsbench --skill feature_engineering",{"tool":53,"command":54},"pnpm","pnpm dlx skills add Waaaalnut\u002Fskillsbench --skill feature_engineering","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002FWaaaalnut\u002Fskillsbench\u002Fblob\u002Fmain\u002Ftasks\u002Ftrend-anomaly-causal-inference\u002Fenvironment\u002Fskills\u002Ffeature_engineering\u002FSKILL.md",true,[58],{"path":59,"resourceType":60,"resourceTypeDesc":61,"sizeBytes":62,"sourceHtmlUrl":63},"scripts\u002Ffeature_engineering.py","SCRIPT","脚本",24124,"https:\u002F\u002Fgithub.com\u002FWaaaalnut\u002Fskillsbench\u002Fblob\u002Fmain\u002Ftasks\u002Ftrend-anomaly-causal-inference\u002Fenvironment\u002Fskills\u002Ffeature_engineering\u002Fscripts\u002Ffeature_engineering.py",[65,79,94,122,144,163],{"skillNo":66,"name":67,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":9,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":68,"categories":69,"keywords":75,"publishedAt":32},"01M6XEWMJ6VY3ZC53MPDCPG3GH","data-reconciliation","https:\u002F\u002Fgithub.com\u002FWaaaalnut\u002Fskillsbench\u002Fblob\u002Fmain\u002Ftasks\u002Fxlsx-recover-data\u002Fenvironment\u002Fskills\u002Fdata-reconciliation\u002FSKILL.md",[70],{"categoryNo":71,"categoryName":72,"taxonomyType":73,"taxonomyTypeDesc":74},"01JBSEEDCAT000000000000033","数据分析师","PROFESSION","适用职业",[76,77,78],"data reconciliation","reconciliation","data",{"skillNo":80,"name":81,"nameZh":6,"description":82,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":9,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":83,"categories":84,"keywords":85,"publishedAt":32},"01M6XEW8MNPMF1KRXV6ATGFRQF","speech-to-text","Transcribe video to timestamped text using Whisper tiny model (pre-installed).","https:\u002F\u002Fgithub.com\u002FWaaaalnut\u002Fskillsbench\u002Fblob\u002Fmain\u002Ftasks\u002Fvideo-tutorial-indexer\u002Fenvironment\u002Fskills\u002Fspeech-to-text\u002FSKILL.md",[],[86,87,88,89,90,91,92,93],"speech","text","speech to text","transcribe video","whisper","model","Video","transcribe",{"skillNo":95,"name":96,"nameZh":6,"description":97,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":9,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":98,"categories":99,"keywords":105,"publishedAt":32},"01M6XEW3FQZP0T1RKQQJFXVY9T","video-processor","Process videos by removing segments and concatenating remaining parts. Use when you need to remove detected pauses\u002Fopenings from videos, create highlight reels, or batch process segment removals using ffmpeg filter_complex.","https:\u002F\u002Fgithub.com\u002FWaaaalnut\u002Fskillsbench\u002Fblob\u002Fmain\u002Ftasks\u002Fvideo-silence-remover\u002Fenvironment\u002Fskills\u002Fvideo-processor\u002FSKILL.md",[100],{"categoryNo":101,"categoryName":102,"taxonomyType":103,"taxonomyTypeDesc":104},"01JBSEEDCAT000000000000016","图像与多媒体","SCENARIO","使用场景",[92,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121],"processor","video processor","batch","create","reels","when","parts","segments","process","videos","you","segment","filter","ffmpeg","use","remove",{"skillNo":123,"name":124,"nameZh":6,"description":125,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":9,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":126,"categories":127,"keywords":129,"publishedAt":32},"01M6XEVXJ5KZ0NTY3XZKRY4AS1","silence-detector","Detect initial silence segments in audio\u002Fvideo using energy-based analysis. Use when you need to find low-energy periods at the start of recordings (title slides, setup time, pre-roll silence).","https:\u002F\u002Fgithub.com\u002FWaaaalnut\u002Fskillsbench\u002Fblob\u002Fmain\u002Ftasks\u002Fvideo-silence-remover\u002Fenvironment\u002Fskills\u002Fsilence-detector\u002FSKILL.md",[128],{"categoryNo":101,"categoryName":102,"taxonomyType":103,"taxonomyTypeDesc":104},[130,131,132,133,111,116,134,135,136,137,138,113,139,140,92,141,142,143,120],"detector","silence detector","title","slides","time","energy","at","recordings","detect","find","Audio","analysis","start","setup",{"skillNo":145,"name":146,"nameZh":6,"description":147,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":9,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":148,"categories":149,"keywords":150,"publishedAt":32},"01M6XEVT3NGRN1FT7JPMAPXS6M","segment-combiner","Combine multiple segment detection results into a unified list. Use when you need to merge segments from different detectors, prepare removal lists for video processing, or consolidate detection outputs.","https:\u002F\u002Fgithub.com\u002FWaaaalnut\u002Fskillsbench\u002Fblob\u002Fmain\u002Ftasks\u002Fvideo-silence-remover\u002Fenvironment\u002Fskills\u002Fsegment-combiner\u002FSKILL.md",[],[117,151,92,152,153,154,155,156,111,157,158,113,116,159,160,120,161,162],"segment combiner","detection","prepare","merge","video processing","for","outputs","combine","processing","consolidate","results","list",{"skillNo":164,"name":165,"nameZh":6,"description":166,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":9,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":167,"categories":168,"keywords":172,"publishedAt":32},"01M6XEVMPVASGS2KX86QK4FMT7","report-generator","Generate compression reports for video processing. Use when you need to create structured JSON reports with duration statistics, compression ratios, and segment details after video processing.","https:\u002F\u002Fgithub.com\u002FWaaaalnut\u002Fskillsbench\u002Fblob\u002Fmain\u002Ftasks\u002Fvideo-silence-remover\u002Fenvironment\u002Fskills\u002Freport-generator\u002FSKILL.md",[169],{"categoryNo":170,"categoryName":171,"taxonomyType":73,"taxonomyTypeDesc":74},"01JBSEEDCAT000000000000034","行政与办公人员",[173,174,175,117,111,156,116,176,159,177,178,179,120,155,180,92,181,182,109],"generator","report generator","report","with","statistics","structured","reports","json","compression","generate"]