AdCampaignAgent-SFT

An end-to-end SFT pipeline for mobile game UA tool-calling agents — rule-based synthetic data generation, Qwen3 LoRA fine-tuning, and an 11-metric benchmark suite across 15 ad-domain tools.

| 🤗 HuggingFace Dataset | Report Bug | Request Feature |

About

AdCampaignAgent-SFT is an open-source end-to-end pipeline for building, training, and evaluating tool-calling agents for mobile game advertising.

The repository covers six capabilities:

1. Generate structured Ad Agent seed records across 7 ad-operation workflows
2. Convert seeds into tool-call conversations in OpenAI Messages format and multiturn prefixes
3. Fine-tune local models (Qwen3) with LoRA across multiple training configurations
4. Provide 15 ad-domain tools for local runtime testing
5. Inspect or interact with local models for tool-calling behavior via REPL
6. Evaluate models with an 11-metric benchmark suite (format, routing, content, system-level)

The domain is mobile game UA (User Acquisition), with workflows grounded in:

• campaign performance analysis
• creative search
• creative upload
• anomaly diagnosis
• benchmark and policy lookup
• refusal handling for off-topic or unauthorized requests

Current Scope

Included in the main flow

• Rule-based seed generation
• Train/test seed splitting with stratification by workflow / scene / clarify flag
• Conversation conversion to SFT-ready OpenAI Messages records
• Message-format multiturn prefix expansion
• Qwen3 LoRA fine-tuning with 4 experimental configurations under src/train
• 15 Ad Campaign Agent tools under src/tools
• Local model tool-call REPL and inspector under src/inference
• Datapipeline validation and regression tests under tests/datapipeline
• 11-metric benchmark suite under src/benchmark

Not in the current main flow

• rag-system is a legacy travel-guide RAG subsystem kept in the repo as historical material. It is not part of the current Ad Agent training or inference pipeline.

Dataset Snapshot

Metric	Value
Theoretical seed total	2,750
Current primary training format	OpenAI Messages
Clarify samples (static estimate)	≈965.64
Unique tools	15
Core workflows	7
Platforms covered	Google · Meta · TikTok · AppLovin · Unity
Game genres	Casual · Puzzle · Hyper-casual · RPG · Strategy
Tool-call heavy records	Majority of direct business samples
Refusal coverage	Off-topic · Unauthorized internal · Unauthorized external

Repository Layout

AdCampaignAgent-SFT/
├── data/
│   ├── raw/                 # seed records, e.g. ad_agent_seeds_*.json
│   ├── processed/           # train/test split outputs
│   └── ready2train/         # final message and multiturn datasets
├── docs/                    # repo documentation
├── images/                  # project images
├── models/                  # local checkpoints and LoRA adapters
├── prompts/                 # reusable prompt assets
├── src/
│   ├── common/              # shared path and utility helpers
│   ├── datapipeline/        # seed generation / split / conversion / multiturn expansion
│   ├── benchmark/           # benchmark schema, metrics, and runner
│   ├── inference/           # local REPL and single-shot inspector
│   ├── tools/               # 15 ad-domain runtime tools + tool schema
│   └── train/               # LoRA fine-tuning, merging, and dataset inspection
├── tests/
│   ├── datapipeline/        # datapipeline regression tests
│   ├── inference/           # online tool-call smoke runner
│   └── tools/               # schema and tool-level tests
├── scripts/                 # shell helpers for training, benchmarking, and REPL
└── rag-system/              # legacy travel RAG assets, not in main flow

Requirements

• Python 3.13+
• uv

uv --version

Setup

git clone https://github.com/ChaoyuWang04/AdCampaignAgent-SFT.git
cd AdCampaignAgent-SFT
uv venv
uv sync

Main Workflow

1. Generate seed records

This creates raw Ad Agent seed data under data/raw/.

uv run python src/datapipeline/0_generate_base_dataset.py

Output example:

data/raw/ad_agent_seeds_20260403_153000_zh.json

2. Split seeds into train / test

uv run python src/datapipeline/1_split_dataset.py
# Then enter the seed file name, for example:
# ad_agent_seeds_20260403_153000_zh.json

The split script currently stratifies by:

• workflow_name
• scene_tag
• needs_clarification

Typical outputs:

• data/processed/ad_agent_seeds_20260403_153000_zh_train.json
• data/processed/ad_agent_seeds_20260403_153000_zh_test.json

3. Convert seeds into tool-call conversations

uv run python src/datapipeline/2_convert_dataset.py

Typical inputs:

• data/processed/ad_agent_seeds_20260403_153000_zh_train.json
• data/processed/ad_agent_seeds_20260403_153000_zh_test.json

Typical outputs:

• data/ready2train/ad_agent_sft_*_zh_train.json
• data/ready2train/ad_agent_sft_*_zh_test.json

4. Expand message-format data into multiturn samples

uv run python src/datapipeline/3_conversation_splitter.py

Typical output:

• data/ready2train/ad_agent_sft_*_multiturn.json

Tools

The runtime tool schema lives in:

• src/tools/all_tools.json

The tool implementations live in:

• src/tools

Current tool families:

• creative search
• creative upload
• campaign / creative analytics
• anomaly diagnosis
• benchmark / policy / knowledge retrieval

See:

• src/tools/README.md

Local Inference

Local tool-call inspector

Single-shot local model inspection for tool-calling behavior.

uv run python src/inference/local_toolcall_inspector.py \
  --scenario campaign_metrics \
  --local_files_only

Useful for:

• checking chat template rendering
• checking whether a local model emits tool calls
• checking parsed tool-call arguments

Local tool-call REPL

Interactive local REPL that can:

• send user turns to a local model
• parse tool calls
• execute local tools
• feed tool results back to the model

uv run python src/inference/local_toolcall_repl.py --local_files_only

You can also use the fixed-config shell wrapper:

bash scripts/local_repl.sh

The REPL defaults to:

• tool schema: src/tools/all_tools.json
• system prompt: prompts/ad_agent_system_prompt.txt

You can override the system prompt with:

uv run python src/inference/local_toolcall_repl.py \
  --system-file prompts/ad_agent_system_prompt.txt

or:

uv run python src/inference/local_toolcall_repl.py \
  --system-text "You are a mobile game UA assistant..."

External Model Runner

For OpenAI-compatible external model tool-call debugging, use:

uv run python tests/inference/online_toolcall_runner.py

This is intentionally kept under tests/inference/ because it is an external-model smoke runner, not part of the local inference core.

Prompt Assets

Prompt files are stored under:

• prompts

Current default runtime prompt:

• prompts/ad_agent_system_prompt.txt

Training

Training scripts live under src/train and shell helpers under scripts.

Typical training flow:

1. Generate seeds
2. Split train/test
3. Convert to message-format data
4. Optionally expand multiturn message data
5. Inspect dataset formatting (scripts/inspect_datasets.sh)
6. Run LoRA fine-tuning:

bash scripts/train_model.sh

Or directly:

uv run python src/train/train_qwen.py \
  --train_file data/ready2train/ad_agent_sft_*_zh_train.json \
  --eval_file  data/ready2train/ad_agent_sft_*_zh_test.json \
  --output_dir models/my_lora_output

7. Merge LoRA adapter into base model (scripts/merge_lora_into_base.sh)

For detailed paths and script descriptions, see:

• docs/Project_UsageGuide.md

Roadmap

• Rule-based Ad Agent seed generation
• 15-tool ad-domain runtime schema
• OpenAI Messages / multiturn conversion
• Qwen3 LoRA fine-tuning (4 experimental configurations)
• Local tool-call inspector
• Local tool-call REPL
• Online tool-call runner for external models
• 11-metric benchmark suite (format / routing / content / system)
• Replace legacy travel RAG with ad-domain knowledge RAG
• Expand benchmark and policy retrieval quality
• Publish fine-tuned checkpoints to HuggingFace

Contributing

Issues and pull requests are welcome. If you want to extend the repo, the highest-leverage areas are:

• better ad-domain RAG replacement
• stronger tool-call eval coverage
• cleaner training / reporting automation

Links

• Project: https://github.com/ChaoyuWang04/AdCampaignAgent-SFT
• Dataset: https://huggingface.co/datasets/SamWang0405/AdCampaignAgent-SFT
• Author: Chaoyu Wang

License

Distributed under the MIT License. See LICENSE for more information.