modeling-and-evaluation
Use for H2O LLM Studio model wrappers, losses, metrics, evaluation outputs, inference routing, plots, and AI-judge metric behavior.
- License 许可
- Apache-2.0
- 最近更新
- 2026-10-08
- Stars
- ★ 11
- Forks
- 3
01 前言 Front Matter
- name
- modeling-and-evaluation
- license
- Apache 2.0
- metadata
- { "disco-role": "operating" }
- description
- Use for H2O LLM Studio model wrappers, losses, metrics, evaluation outputs, inference routing, plots, and AI-judge metric behavior.
- disable-model-invocation
- true
02 资源文件 3
- 参考文档references/evaluation-workflows.md8.2 KB
- 参考文档references/model-and-metric-reference.md9.7 KB
- 参考文档references/troubleshooting.md7.7 KB
03 SKILL.md 正文
H2O LLM Studio modeling and evaluation
Use this sub-skill when the task is about how H2O LLM Studio maps a problem type to model wrappers, losses, metrics, inference behavior, generated prediction outputs, validation plots, or AI-judge metrics.
Do not use this sub-skill for:
- creating or repairing experiment YAML/data schemas: use
configuration-and-data; - launching, scheduling, or debugging training runs: use
training-and-experiments; - interactive chat, model-card export, or Hugging Face publishing: use
export-and-prompt.
Operating workflow
- Identify the
problem_typefirst. Valid modeling/evaluation problem types are:text_causal_language_modeling,text_sequence_to_sequence_modeling,text_dpo_modeling,text_causal_classification_modeling, andtext_causal_regression_modeling. - Load references/model-and-metric-reference.md for class mappings, forward/generate contracts, losses, metric direction, and expected result keys.
- Load references/evaluation-workflows.md for
run_inference,run_eval, prediction files, plot files, generation-vs-forward routing, and GPT/MT-Bench judge behavior. - For unsupported metrics, NaNs, empty outputs, OpenAI/Azure endpoint issues, missing
prompts/, or classification/regression shape errors, load references/troubleshooting.md. - Use scripts/inspect_problem_type.py for a safe local inspection that lists the configured model/loss/metric/plot classes without instantiating or downloading a model.
Quick commands
python scripts/inspect_problem_type.py --problem-type text_causal_classification_modeling --list-metrics
python scripts/inspect_problem_type.py --problem-type all --json
python scripts/inspect_problem_type.py --problem-type text_dpo_modeling --verify-imports
The inspection script is safe by default: it prints static and import-level metadata only. It does not start training, call an AI judge, load Hugging Face weights, or create model instances.
Decision rules
- For generative validation metrics other than
Perplexity, causal LM, sequence-to-sequence, and DPO use generation;Perplexityuses a forward pass. - Classification and regression are non-generation tasks. They always use a forward pass and postprocess logits or regression head outputs into predictions before metrics are computed.
GPTmetrics call an OpenAI-compatible Chat Completions endpoint and can incur network cost. Require endpoint, credential, and budget confirmation before running; use static reasoning or mocked tests when possible.- Validation artifacts are written by the training/evaluation workflow, not by the model classes themselves. Expect raw prediction pickle, prediction CSV, and parquet plot data when evaluation reaches rank 0 successfully.
04 同仓库更多 Skills
obsidian-clipper-template-creator
Guide for creating templates for the Obsidian Web Clipper. Use when you want to create a new clipping template, understand available variables, or format clipped content.
obsidian-clipper-template-creator
Guide for creating templates for the Obsidian Web Clipper. Use when you want to create a new clipping template, understand available variables, or format clipped content.
python-api
Programmatic Object-Detection-Metrics API guidance for BoundingBox, BoundingBoxes, Evaluator, enums, coordinate conversion, AP/mAP semantics, and source-style import diagnostics.
file-evaluation
Folder-based PASCAL VOC AP/mAP evaluation from ground-truth and detection text files.
object-detection-metrics
Guides legacy Object-Detection-Metrics PASCAL VOC AP/mAP evaluation from detection text folders and direct Python API classes.
nx-workspace-patterns
Configure and optimize Nx monorepo workspaces. Use when setting up Nx, configuring project boundaries, optimizing build caching, or implementing affected commands.