evaluation-and-optimization

AdalFlow evaluation and optimization

Use this sub-skill when the user wants to score task outputs, select or load AdalFlow benchmark datasets, make Parameter objects trainable, wrap a task in AdalComponent, run Trainer.diagnose or Trainer.fit, configure few-shot/text-gradient optimizers, or use optimize_anything for bounded text-artifact search.

Route boundaries

Stay here for:

• AnswerMatchAcc, RetrieverEvaluator, and cautious LLMasJudge evaluation.
• Dataset loaders for GSM8K, TREC, HotpotQA, and BBH-style tasks, including small-size and skip-expensive guidance.
• Parameter, ParameterType, GradComponent, EvalFnToTextLoss, LLMAsTextLoss, BootstrapFewShot, TGDOptimizer, AdalComponent, Trainer, and optimize_anything workflows.

Route away instead of duplicating guidance:

• RAG construction, document splitting, local/vector retrievers, and index pipelines: use retrieval-rag-and-data-pipelines.
• Provider model-client configuration, credentials, Generator, and fake provider clients: use model-client-and-generator-workflows.
• Tracing/logging artifacts and callback storage: use tracing-observability-and-configuration.

Reference map

• Evaluation metrics: metric APIs, input shapes, output objects, and no-network smoke examples.
• Optimization and training: Parameter/gradient concepts, AdalComponent contracts, Trainer.fit/diagnose, few-shot/text-gradient optimizer setup, and optimize_anything.
• Datasets and benchmarks: loader classes, fields, optional dependencies, downloads, and benchmark constraints.
• Troubleshooting: evaluator shape errors, metric extras, dataset misses, checkpoint/resume, validation-set requirements, provider cost, runtime limits, and multiprocessing issues.
• Metric smoke script: service-free exact/fuzzy/F1/retriever metric check.

Safe operating defaults

1. Score before optimizing. Build a deterministic metric with AnswerMatchAcc or RetrieverEvaluator and run a tiny local smoke case before any model-backed judge or optimizer.
2. Treat live judging and text gradients as expensive. LLMasJudge, LLMAsTextLoss, backward engines, teacher generators, and TGDOptimizer can invoke model providers. Require explicit provider setup, credentials, budget, sample limits, and cache/checkpoint choices.
3. Use bounded data. Dataset loaders may download external data and optional packages. Prefer size=... or pre-existing local data for first tests.
4. Make AdalComponent contracts explicit. prepare_task returns (callable, kwargs) for one sample; prepare_eval returns (eval_fn, kwargs) for inference scoring; prepare_loss returns (loss_fn, kwargs) with Parameter inputs and correct eval_input values.
5. Prefer diagnosis before training. Trainer.diagnose is useful for identifying failing samples and prompt/output issues. Use num_workers=1 for minimal debugging, then raise concurrency only after stable.
6. Do not imply full benchmark training is safe. Benchmark scripts and full prompt optimization runs are optional, network/provider/dataset dependent, and should be reduced to a tiny user-approved run before scaling.

Minimal decision checklist

• What is being scored: final answer strings, structured labels, retrieved contexts/IDs, or model-judged quality?
• Is the metric service-free, optional-package-backed, or provider-backed?
• Are predictions raw values, GeneratorOutput.data, or Parameter.full_response.data?
• Are trainable values marked with the right ParameterType (PROMPT for instructions, DEMOS for examples)?
• Does the training data provide stable id fields and a real validation set?
• What are the maximum samples, optimizer steps, worker count, and checkpoint/resume plan?