Skip to content

DEV.md

Latest commit

 

History

History
180 lines (132 loc) · 6.97 KB

DEV.md

File metadata and controls

180 lines (132 loc) · 6.97 KB

Development

Guide for working on Clear Your Tools from a source checkout or integrating the Python package.

Prerequisites

Install on your machine before the setup steps below:

Tool Used for
uv Python deps, prek, uv run
Python 3.13+ App and SDK (see pyproject.toml)
Rust (stable) Editable cyt-indexer-sdk (maturin), cargo prek hooks, TypeScript native build
Node.js ≥20 TypeScript SDK build and prek hooks under sdk/typescript/
ast-grep CLI ast-grep / ast-scan prek hooks (install)

Registry E2E (published crates/PyPI/npm only) needs cargo, npm, and network access — see sdk/e2e/README.md.

Setup

Dependencies are managed with uv:

uv sync --all-extras

TypeScript SDK (for prek run -a)

native.cjs, native.d.ts, *.node, and dist/ under sdk/typescript/ are gitignored — they are built from Rust, not copied from another checkout. After clone:

cd sdk/typescript
npm ci
npm run build   # needs Node ≥20 and a Rust toolchain

Run this before the first full prek run -a (typecheck runs before the typescript-build hook). Python-only work needs only uv sync.

From PyPI (proxy + pruners):

uv pip install 'clear-your-tools[all]'

Copy API keys (or use ~/.config/cyt/.env):

cp .env.example .env
# Edit .env — at minimum DEEPINFRA_API_KEY (reranker) and OPENROUTER_API_KEY (upstream + optional LLM stage)

Checks

After setup, run hooks locally before pushing:

uv run prek run -a          # all pre-commit hooks (build TS SDK first; see above)
task ci                     # Python checks mirroring CI (sync, ruff, mypy, pytest, build)

TypeScript-only hooks: task -d sdk prek or cd sdk/typescript && npm test.

Skip one hook: SKIP=<hook-id> git commit … (for example SKIP=pytest). Registry E2E against live packages: sdk/e2e/scripts/run-local.sh.

Run from a checkout

uv run cyt proxy --port 8834

Other CLI entry points work the same way with uv run (for example uv run cyt stats totals).

Repository layout

.
├── README.md
├── DEV.md
├── LIMITATIONS.md
├── pyproject.toml
├── count_request_tokens.py      # estimate savings on a captured request JSON
└── src/
    └── cyt/                       # installable package (Clear Your Tools)
        ├── config/                # load_config, defaults.yaml
        ├── common/                # path_constants, runtime_constants, token_usage, pricing
        ├── indexer/               # build, retrieve, catalog_io, tokens (tiktoken)
        ├── pruners/               # llm, rerank, policies
        └── proxy/                 # transport, reverse, anthropic, stats, cli

Library usage

from cyt.indexer import CatalogIndex, build_catalog_index, load_catalog, retrieve_tools
from cyt.pruners import rerank_catalog_dict, llm_catalog_dict
from cyt.pruners.policies import configure_policies_from_config
from cyt.proxy.reverse import create_app  # requires clear-your-tools[proxy]

Advanced (submodules — not re-exported from cyt.indexer):

from cyt.indexer.catalog_io import CatalogBuilder, write_catalog_index
from cyt.indexer.tokens import count_tokens, count_json_tokens, compact_json
from cyt.indexer.build import collect_enums, prepare_tool_entry, prepare_system_tool_entry
from cyt.common.path_constants import DECOMPOSED_PREFIX

Configuration reference

Main config file: config.yaml in the working directory, or ~/.config/cyt/config.yaml (created on first run). Run cyt setup for an interactive wizard that writes the user config and optional ~/.config/cyt/.env. Bundled defaults ship in the package as cyt.config.defaults.yaml.

User-facing guides (pricing overrides, rerank → llm pipeline, OpenRouter vs OpenAI pruning models): README.md — Configuration.

Section Purpose
pruning.tools.policy.system_tool / mcp_tool Default pruning behavior for system vs MCP tools
pruning.tools.policy.minimum_tools Tool-count threshold for rerank/llm stages
pruning.tools.pipelines.rerank.model_nick Reranker catalog nick for the rerank stage
pruning.tools.pipelines.llm.model_nick LLM pruner catalog nick for the llm stage
pruning.tools.pipelines.bm25.index_dir BM25 index directory
pruning.tools.sequence Ordered list of stages: rerank, llm, bm25
pruning.tools.policy.per_tool Per-tool policy overrides
models.providers[] + model provider_nick Provider credentials (legacy inline fields still work)
models.rerankers / models.llm Remote model definitions and API keys
network.proxy.reverse Listen port, upstream URLs, HTTP/2, TLS
stats Stats DB path, optional full tool JSON storage

Legacy paths (pruning.pipeline, pruning.policy, pruning.<stage>, …) resolve via src/cyt/config/legacy.py.

Environment variables (see .env.example):

  • DEEPINFRA_API_KEY — reranker stage
  • OPENROUTER_API_KEY — upstream forwarding and optional LLM stage

Estimate savings on a captured request

Use count_request_tokens.py on a JSON snapshot from debug dry-run (see debug/):

uv run count_request_tokens.py \
  --tool-savings-percent 85 \
  --requestfile temp_example_claude_call.json

HTTP/2 and TLS

Some clients prefer HTTP/2. Generate a local certificate (gitignored under src/crt/):

mkdir -p src/crt
openssl req -x509 -nodes -days 365 -newkey rsa:4096 \
  -keyout src/crt/key.pem \
  -out src/crt/cert.pem \
  -subj "/CN=localhost" \
  -addext "subjectAltName=DNS:localhost,IP:127.0.0.1"

Trust the cert on macOS: Keychain Access → System → import cert.pem → Trust → "Always Trust".

Run with HTTP/2:

uv pip install h2 'hypercorn[h2]'
cyt proxy --http2-serve \
  --ssl-keyfile src/crt/key.pem \
  --ssl-certfile src/crt/cert.pem \
  --port 8834

TLS settings can also live in config.yaml under network.proxy.reverse.http2.ssl.