bigquery-bigframes
Generates Python code using BigQuery DataFrames (BigFrames), the pandas/scikit-learn-style API over BigQuery. Use when writing BigFrames code or doing pandas-style dataframe/ML work against BigQuery (e.g. in a notebook). Don't use for SQL-first workflows or the google-cloud-bigquery client library — use bigquery-basics.
- License 许可
- Apache-2.0
- 最近更新
- 2026-08-23
- Stars
- ★ 0
- Forks
- 0
01 前言 Front Matter
- name
- bigquery-bigframes
- license
- Apache-2.0
- metadata
- { "version": "v1", "publisher": "google" }
- description
- Generates Python code using BigQuery DataFrames (BigFrames), the pandas/scikit-learn-style API over BigQuery. Use when writing BigFrames code or doing pandas-style dataframe/ML work against BigQuery (e.g. in a notebook). Don't use for SQL-first workflows or the google-cloud-bigquery client library — use bigquery-basics.
02 资源文件 4
- 其他OWNERS69 B
- 参考文档references/BIGFRAMES.md2.6 KB
- 参考文档references/linear_regression.md1.4 KB
- 参考文档references/logistic_regression.md1.1 KB
03 SKILL.md 正文
BigFrames (BigQuery DataFrame) basics
BigFrames is a Python library that lets you take advantage of BigQuery data processing by using familiar Python APIs.
Dataframe API best practices
- Stay in the Cloud: Perform data cleaning, transformation, and analysis via BigFrames methods to leverage BigQuery's scale rather than downloading data.
- Prefer partial ordering mode: Enable partial ordering mode right after
importing BigFrames. This speeds up data processing significantly by relaxing
row-sequence constraints.
import bigframes.pandas as bpd bpd.options.bigquery.ordering_mode = 'partial' - Use
peek()for data preview: Usepeek(n)to preview data instead ofhead(n).peek(n)randomly samplesnrows and is significantly faster.head(n)returns rows in strict order and fails inpartialordering mode unless the DataFrame has been explicitly sorted. - Avoid materializing data locally: Methods like
to_pandas()download all data to client memory, bypassing BigQuery’s distributed computation and risking Out of Memory (OOM) errors. Do not materialize data locally unless:- The dataset is small enough to fit safely in memory.
- An error message explicitly requires local materialization.
- Prefer Dataframe API over SQL queries: Do not write raw SQL queries via
read_gbq()if a DataFrame/Series method achieves the same result, as it breaks the Pandas abstraction and prevents lazy query execution. - Accessors over UDFs/Lambdas:
- Use built-in accessors (e.g.,
df.col.str.*,df.col.dt.*) instead of remote User Defined Functions (UDFs). UDFs require extra resources and time to deploy. - Do not use lambdas with
Series.map()orDataFrame.apply(). These methods do not accept functions withoutudforremote_functiondecorators.
# Avoid: df["upper"] = df["name"].map(lambda x: x.upper()) # Prefer: df["upper"] = df["name"].str.upper() - Use built-in accessors (e.g.,
- Schema Verification: Do not assume the schema of intermediate outputs.
Proactively verify schemas using
.dtypesand inspect sample records usingdisplay()with.peek(). - Visualization: Plot directly from the BigFrames DataFrame/Series when
possible. BigFrames is compatible with Matplotlib and Seaborn. If direct
plotting fails, use the
.plotaccessor. If the dataset is too large to plot, aggregate or sample the data before calling.to_pandas()to plot locally.
Machine Learning
- Use
bigframes.bigquery.mlpackage: Do not use Scikit-learn or other ML libraries with BigQuery DataFrames. Standard Scikit-learn models require bringing data into local client memory, whereasbigframes.bigquery.mldelegates training directly to BigQuery's scalable ML engine. Import functions frombigframes.bigquery.ml.
Reference Directory
- Linear Regression: Train a linear regression model to predict numerical values.
- Logistic Regression: Train a logistic regression model to predict boolean values.
BigFrames ML (Legacy)
The BigFrames ML package (bigframes.ml) is a legacy package that mimics the
scikit-learn API but is no longer recommended for new projects. Only use this
package if the user explicitly requests BigFrames ML.
- Legacy Imports: When legacy BigFrames ML is requested, import tools and
classes from
bigframes.mlinstead ofbigframes.bigquery.ml. - DataFrame Return on Prediction: Unlike Scikit-learn, BigFrames'
predict()method always returns a DataFrame containing both predictions and features, rather than a single series of predictions. - No
random_state: Do not pass arandom_stateargument when instantiating BigFrames ML models, as this parameter is not supported in the BigFrames ML package. - Automatic Scaling: Do not use
OneHotEncoderorStandardScalerunless explicitly requested, as scaling is handled automatically. - Hyperparameter Tuning: Write custom loops for hyperparameter tuning, as
BigFrames lacks
GridSearchCVorRandomizedSearchCV. - ARIMA Plus (Forecasting):
- Import from
bigframes.ml.forecasting. - Sort data chronologically and split around a timepoint before training.
- Ensure the prediction horizon is less than or equal to the training horizon.
- Import from
- PCA: BigFrames' PCA class lacks a
transform()method. Usepredict()instead. - Model Persistence: To persist a model, use
model.to_gbq(). To load a persisted model, usebpd.read_gbq_model().
04 同仓库更多 Skills
skill-repair
Use this to fix and re-install agent skills that have failed installation. This skill provides the necessary context and permissions to surgically update the `manifest.json` after a fix has been applied.
notebook-guidance
This skill guides the use of Jupyter notebooks for data analysis, exploration, and visualization, particularly with BigQuery. It outlines best practices for notebook execution and validation (supporting both cell-by-cell execution and full notebook generation depending on tool availability), library installation, and structuring notebooks for clarity. It also covers specific rules for data cleaning, plotting, and integrating with BigQuery SQL and machine learning workflows. Relevant when any of the following conditions are true: 1. The user request involves a data analysis, data exploration, data visualization, or data insights task that requires multiple steps, queries, or visualizations to answer. 2. The user explicitly requests a notebook (.ipynb). 3. You are creating, editing, or executing cells in a Jupyter notebook. 4. You need to query BigQuery from within a notebook. DO NOT use the Python BigQuery client library; instead, you MUST use the `%%bqsql` magics explained in this skill.
ml-best-practices
CRITICAL RULE: You MUST use this skill whenever the task involves any machine learning tasks or data analysis. Use this skill if the user's prompt or requirements mention any of the following: * Clustering * Classification * Regression * Time series forecasting * Statistical testing * Model comparison * ML * Data analysis SQL/BigQuery ML HANDOFF: If the user requires a SQL solution, use this skill to dictate the ANALYSIS STEPS (e.g., markdown analysis cells, visualization logic), but defer to `bigquery` for all SQL syntax.
managing-python-dependencies
Ensures proper Python dependency management, avoiding global `pip install` and adhering to project-specific tooling. Use this skill if any of the following are true: 1. Attempting to run `pip install {package_name}`. 2. Python packages or dependencies need to be added or modified. 3. Initiating a new Python project. 4. Creating a new notebook, even if just using BigQuery cells. 5. Generating Python code that includes `import` statements for third-party libraries. 6. Before executing Python scripts via the terminal to ensure the correct virtual environment is active.
google-cloud-storage-basics
Stores, retrieves, and manages data as objects in Cloud Storage (Google Cloud Storage, or GCS) buckets. Use when you need to interact with Cloud Storage — create or configure buckets, upload, download, stream, or transfer data, organize objects with folders, generate signed URLs, control access (IAM, ACLs, public access prevention), set storage classes and tiering (Standard, Nearline, Coldline, Archive), manage cost and lifecycle, protect data (versioning, encryption/CMEK, retention and Bucket Lock, object holds, soft delete), host static websites, trigger Pub/Sub notifications on object changes, mount buckets as a file system (gcsfuse), or optimize storage performance at any scale. Covers the gcloud storage / gsutil CLI, JSON and XML APIs, client libraries, Terraform, and Cloud Storage MCP servers. Don't use for block storage (Persistent Disk), data warehousing/analytics (BigQuery), or databases (Cloud SQL, Spanner, Bigtable, Firestore).
gcs-security-assessment
Assesses security posture, evaluates risks, and checks SAIF compliance for Google Cloud Storage buckets or projects. Use when the user requests security scans, vulnerability checks, or SAIF assessments. Don't use when: The user is asking about non-GCS resources (Compute Engine, GKE, etc.), investigating a live production outage, or asking general security questions not tied to a specific project or bucket.