使用 pydantic_evals.generate_dataset 用 LLM 自动生成评测数据集【免费下载链接】pydantic-aiHow Python does AI. Agents, realtime voice, image generation, embeddings. Every model, every interface, typed end to end.项目地址: https://gitcode.com/GitHub_Trending/py/pydantic-ai导读本篇文章聚焦 pydantic-ai 仓库中pydantic_evals.generation模块提供的generate_dataset函数讲解如何用 LLM 自动生成符合自定义输入、输出、元数据 schema 的评测数据集Dataset。你将掌握该函数的完整参数、底层实现原理如何构造 Agent、如何解析与回退、如何落盘并通过仓库内examples/pydantic_ai_examples/evals的真实示例与docs/evals/how-to/dataset-management.md的完整代码学会在评测工作流中用一行异步调用产出可直接用于Dataset.evaluate()的 YAML/JSON 数据集。一、pydantic_evals.generation模块定位pydantic_evals是 pydantic-ai 仓库中的评测子系统其Dataset用于组织一批测试用例每个用例包含inputs、可选的expected_output、metadata与专属evaluators并可对任务函数进行批量评估见 pydantic_evals/pydantic_evals/dataset.py。手工编写评测数据集在用例量大、输入复杂时成本很高因此该仓库提供了 pydantic_evals/pydantic_evals/generation.pyUtilities for generating example datasets for pydantic_evals. This module provides functions for generating sample datasets for testing and examples, using LLMs to create realistic test data with proper structure.该模块只导出一个公共 API——generate_dataset__all__ (generate_dataset,)其目标是用 LLM 生成既有真实感、又严格符合类型 schema的测试用例集同时把评估器evaluator的引用一并纳入生成的 JSON Schema保证生成结果从第一天起就能被Dataset.from_file直接反序列化使用。二、generate_dataset完整签名与参数说明函数签名位于 pydantic_evals/pydantic_evals/generation.pyasync def generate_dataset( *, dataset_type: type[Dataset[InputsT, OutputT, MetadataT]], path: Path | str | None None, custom_evaluator_types: Sequence[type[Evaluator[InputsT, OutputT, MetadataT]]] (), model: models.Model | models.KnownModelName openai:gpt-5.2, n_examples: int 3, extra_instructions: str | None None, ) - Dataset[InputsT, OutputT, MetadataT]:所有参数均为关键字参数*逐个说明参数类型默认值作用dataset_typetype[Dataset[InputsT, OutputT, MetadataT]]必填目标数据集类型携带你定义的输入、输出、元数据类型三个泛型参数用于生成 JSON SchemapathPath \| str \| NoneNone可选落盘路径。传入后生成的 Dataset 会写入该文件且数据集默认名取Path(path).stem文件主名不传则返回内存对象custom_evaluator_typesSequence[type[Evaluator[...]]]()自定义评估器类序列会并入生成的 JSON Schema 与序列化结果modelmodels.Model \| models.KnownModelNameopenai:gpt-5.2用于生成的 Pydantic AI 模型字符串形式是 KnownModelName也可传models.Model实例如带自定义 base_url 的 OpenAI 客户端n_examplesint3要生成的测试用例数量extra_instructionsstr \| NoneNone额外指令会作为用户消息发给 LLM用于描述要生成的业务场景返回一个结构完整的Dataset对象。若 LLM 的响应无法解析为合法数据集会抛出ValidationError此时函数会先打印模型的原始响应Raw response from model: ...便于调试见 generation.py。三、底层实现从 Schema 到 Agent 的完整调用链generate_dataset的实现可以拆成四个阶段理解它有助于你写出更可靠的生成指令。1. 先由类型生成 JSON Schemaoutput_schema dataset_type.model_json_schema_with_evaluators(custom_evaluator_types)model_json_schema_with_evaluators定义在 pydantic_evals/pydantic_evals/dataset.py它从Dataset的三个泛型参数解析出in_type / out_type / meta_type再结合默认评估器注册表DEFAULT_EVALUATORS与自定义评估器类型动态构造内部的Case/DatasetPydantic 模型并生成 JSON Schema。因此生成结果的字段结构、evaluators里可用的评估器名称都受到该 Schema 的约束。2. 构造一个输出原始 JSON 文本的 Agentagent Agent( model, system_prompt( fGenerate an object that is in compliance with this JSON schema:\n{output_schema}\n\n fInclude {n_examples} example cases. You must not include any characters in your response before the opening { of the JSON object, or after the closing }. ), output_typestr, )值得注意的实现细节Agent 的output_type是str原始文本而非结构化输出。源码注释generation.py说明这是为了规避StructuredDict对内联 JSON Schema 转换的限制TODO 引用了一个 upstream pydantic issue。也就是说结构校验并不依赖模型端的结构化输出而是在拿到文本后由Dataset.from_text(..., fmtjson)通过 Pydantic 完成强类型校验——这一设计也意味着「模型必须输出一个干净、前后无多余字符的 JSON 对象」这正是 system prompt 最后一句约束的来源。3. 运行 Agent 并剥离 Markdown 围栏result await agent.run(extra_instructions or Please generate the object.) output strip_markdown_fences(result.output)当extra_instructions为None时发送给模型的用户消息就是默认的Please generate the object.。strip_markdown_fences来自pydantic_ai._utils负责移除模型常见的json ...包裹兼容模型把 JSON 放在 Markdown 代码块里的情况。4. 反序列化、落盘并返回result dataset_type.from_text(output, fmtjson, default_namedefault_name, custom_evaluator_typescustom_evaluator_types) if path is not None: result.to_file(path, custom_evaluator_typescustom_evaluator_types) return resultfrom_text/to_file均在 dataset.py 中实现from_text(fmtjson)走model_validate_json并把default_name作为数据集名兜底to_file会根据扩展名推断yaml或json格式_infer_fmt并顺带写出一个{stem}_schema.jsonJSON Schema 文件。对于 YAML 输出还会在文件头写入# yaml-language-server: $schema...注释让 VS Code、PyCharm 等编辑器在编辑数据集时获得类型检查和自动补全。四、真实示例仓库内的 time_range 数据集生成仓库在 examples/pydantic_ai_examples/evals/example_01_generate_dataset.py 中给出了端到端用法。其目标是为一个时间范围推断 Agent生成 10 条评测用例import asyncio from pathlib import Path from types import NoneType from pydantic_ai_examples.evals.models import TimeRangeInputs, TimeRangeResponse from pydantic_evals import Dataset from pydantic_evals.generation import generate_dataset async def main(): dataset await generate_dataset( dataset_typeDataset[TimeRangeInputs, TimeRangeResponse, NoneType], modelopenai:gpt-5.2, # Use a smarter model since this is a more complex task that is only run once n_examples10, extra_instructions Generate a dataset of test cases for the time range inference agent. Include a variety of inputs that might be given to the agent, including some where the only reasonable response is a TimeRangeBuilderError, and some where a TimeRangeBuilderSuccess is expected. Make use of the IsInstance evaluator to ensure that the inputs and outputs are of the appropriate type. When appropriate, use the LLMJudge evaluator to provide a more precise description of the time range the agent should have inferred. ... Leave the model and include_input arguments to LLMJudge as their default values (null). Also add a dataset-wide LLMJudge evaluator to ensure that the explanation or error_message fields are appropriate to be displayed to the user (e.g., written in second person, etc.). , ) dataset.to_file( Path(__file__).parent / datasets / time_range_v1.yaml, fmtyaml, ) if __name__ __main__: asyncio.run(main())这个示例展示了extra_instructions的几种典型用法非常值得借鉴描述业务目标与多样性要求要求输入覆盖成功推断与只能返回错误两类情况指定使用哪些评估器要求使用IsInstance做类型断言、用LLMJudge为歧义输入提供更精确的判定准则甚至要求数据集级添加一个 LLMJudge 检查面向用户的文案风格给出约束细节如 LLMJudge 的model/include_input保持默认值nullrubric 只包含用户 prompt 中未呈现的信息。输入/输出模型定义在 examples/pydantic_ai_examples/evals/models.pyTimeRangeInputs是一个TypedDict含prompt与now输出为TimeRangeBuilderSuccess | TimeRangeBuilderError的联合类型。生成结果落在 examples/pydantic_ai_examples/evals/datasets/time_range_v1.yaml其中既有Single day mention这类直白用例也有Ambiguous mentionConfusing relative references这类刻意制造的模糊/冲突输入并附带IsInstance、LLMJudge评估器配置与数据集级LLMJudge# yaml-language-server: $schematime_range_v1_schema.json cases: - name: Ambiguous mention inputs: prompt: Check logs from last week or so, around early May now: 2023-10-28T09:30:00Z expected_output: min_timestamp_with_offset: 2023-10-21T09:30:00Z max_timestamp_with_offset: 2023-10-28T09:30:00Z evaluators: - IsInstance: TimeRangeBuilderSuccess - LLMJudge: We want to interpret conflicting references by default to the more recent timeframe; confirm the explanation addresses ignoring early May. evaluators: - LLMJudge: Ensure the explanation or error_message fields are truly appropriate for user display, in a second-person or friendly style.五、最小可运行示例完整可复制来自 docs/evals/how-to/dataset-management.md 的官方示例定义了 Question/Answer 场景的三个模型生成 2 条用例并保存为 YAMLfrom __future__ import annotations from pathlib import Path from pydantic import BaseModel, Field from pydantic_evals import Dataset from pydantic_evals.generation import generate_dataset class QuestionInputs(BaseModel, use_attribute_docstringsTrue): # (1)! Model for question inputs. question: str A question to answer context: str | None None Optional context for the question class AnswerOutput(BaseModel, use_attribute_docstringsTrue): # (2)! Model for expected answer outputs. answer: str The answer to the question confidence: float Field(ge0, le1) Confidence level (0-1) class MetadataType(BaseModel, use_attribute_docstringsTrue): # (3)! Metadata model for test cases. difficulty: str Difficulty level (easy, medium, hard) category: str Question category async def main(): dataset await generate_dataset( # (4)! dataset_typeDataset[QuestionInputs, AnswerOutput, MetadataType], n_examples2, extra_instructions Generate question-answer pairs about world capitals and landmarks. Make sure to include both easy and challenging questions. , ) output_file Path(questions_cases.yaml) dataset.to_file(output_file) # (5)!要点说明用BaseModel定义任务输入 schema字段 docstring 在use_attribute_docstringsTrue下会成为 LLM 看到的重要语义提示应写得准确定义期望输出 schema可借助Field(ge0, le1)等约束让生成结果符合业务边界定义元数据 schema调用generate_dataset生成包含 2 个用例的Datasetto_file同时写出questions_cases_schema.jsonYAML 文件头附yaml-language-server注释。运行前需补上asyncio.run(main())。生成得到的 YAML 大致如下官方示例输出# yaml-language-server: $schemaquestions_cases_schema.json name: generated cases: - name: Easy Capital Question inputs: question: What is the capital of France? context: null metadata: difficulty: easy category: Geography expected_output: answer: Paris confidence: 0.95 evaluators: - EqualsExpected - name: Challenging Landmark Question inputs: question: Which world-famous landmark is located on the banks of the Seine River? context: null metadata: difficulty: hard category: Landmarks expected_output: answer: Eiffel Tower confidence: 0.9 evaluators: - EqualsExpected evaluators: [] report_evaluators: []若希望生成 JSON 文件只需把输出文件名改为questions_cases.jsonto_file会根据.json后缀写出 JSON并在文件内放置$schema: questions_cases_schema.json键$schema虽非正式规范但 VS Code 与 PyCharm 均支持据此做编辑期校验。六、让生成结果更可控的实践建议结合源码实现与仓库示例以下几点能显著提升生成质量与可用性精心设计类型与字段 docstringgenerate_dataset把整个 JSON Schema 塞进 system prompt字段注释越明确模型生成的示例越贴近真实业务use_attribute_docstringsTrue会让 docstring 进入 schema 的description直接影响模型理解。在extra_instructions里写用例多样性要求仓库 time_range 示例明确要求同时包含成功与失败路径、歧义输入、边界情况这是让评测集覆盖真实分布的关键同时可以要求模型为用例挑选合适的评估器如IsInstance、LLMJudge。内置评估器还包括Equals、EqualsExpected、Contains、MaxDuration、GEval、HasMatchingSpan等见 pydantic_evals/pydantic_evals/evaluators/common.py。自定义评估器要在调用时登记如果你定义了dataclass装饰的Evaluator子类应通过custom_evaluator_types传入使其进入 JSON Schema 和后续from_file的反序列化注册表仓库示例中ValidateTimeRange、UserMessageIsConcise、AgentCalledTool即通过CUSTOM_EVALUATOR_TYPES传递见 examples/pydantic_ai_examples/evals/custom_evaluators.py 与 examples/pydantic_ai_examples/evals/example_02_add_custom_evaluators.py。为复杂任务选择更强的模型默认模型为openai:gpt-5.2时间范围示例也在注释中强调Use a smarter model since this is a more complex task——生成任务对 schema 遵循能力要求较高简单任务可用默认值复杂任务建议显式指定。失败时先看原始输出一旦抛ValidationError函数会先打印Raw response from model:及模型原始文本据此可判断是模型没遵守 JSON 格式、字段缺失还是类型不符再针对性补充extra_instructions。生成后可继续加工generate_dataset返回的是标准Dataset对象之后可以继续add_case、add_evaluator可指定specific_case只给某个用例加评估器再to_file保存为 v2 版本仓库的example_02正是这一工作流。七、与评测工作流衔接生成好的数据集可以直接接入评估流程Dataset.evaluate(task, ...)会对每个用例并发执行任务函数、运行数据集级与用例级评估器并产出报告repeat参数可让每个用例多次运行以聚合统计见 pydantic_evals/pydantic_evals/dataset.py。因此典型的完整流程是定义输入/输出/元数据类型调用generate_dataset生成并保存数据集人工审查/增补用例与评估器可选参考 dataset-management.md 的Creating Datasets / Adding Cases Dynamicallydataset.evaluate(task)得到EvaluationReport或dataset.evaluate_sync(task)用于同步环境。关于数据集序列化细节JSON Schema 生成、自定义评估器注册、$schema引用可继续阅读 dataset-serialization.md关于各内置评估器的完整说明参见 docs/evals/evaluators/overview.md。八、注意事项与限制generate_dataset是异步函数必须在事件循环中调用示例统一使用asyncio.run(main())。生成的用例由 LLM 产生不能保证完全符合业务语义ValidationError之外的结构性正确、语义性错误需要人工抽查或用LLMJudge等评估器兜底。path参数的默认数据集名取自文件主名Path(path).stem不传path时名为generated见 generation.py。模型必须输出干净的 JSON 对象前后无多余字符否则解析失败代码已通过strip_markdown_fences兼容 Markdown 代码块包裹的情况。仓库对generate_dataset的测试目前仅验证其可导入tests/evals/test_dataset.py中test_import_generate_dataset注释指出该函数tough to test in an interesting way outside an example因此实际生成行为建议以仓库中的 example 脚本为参照运行验证。【免费下载链接】pydantic-aiHow Python does AI. Agents, realtime voice, image generation, embeddings. Every model, every interface, typed end to end.项目地址: https://gitcode.com/GitHub_Trending/py/pydantic-ai创作声明：本文部分内容由AI辅助生成（AIGC），仅供参考
