README.md

convtools — write transformations as expressions, run them as Python

convtools lets you declare data transformations in plain Python, then compiles them into tiny, optimized Python functions at runtime. You keep your data in native iterables (lists, dicts, generators, CSV streams)—no heavy container required.

Why pick convtools?

• Stay in Python. Compose transformations as expressions: pipes, filters, joins, group‑bys, reducers, window functions, and more. Then call .gen_converter() to get a real Python function.
• Stream‑friendly. Works directly on iterators and files; the Table helper processes CSV‑like data without loading everything into memory.
• Powerful aggregations. Rich reducers (Sum, CountDistinct, MaxRow, ArraySorted, Dict*, TopK…) with per‑reducer where filters and defaults. Nested aggregations are first‑class.
• Debuggable & inspectable. Print the generated code with debug=True or set global options via c.OptionsCtx. Works with pdb/pydevd.
• Plays nicely with Pandas/Polars. It’s not a DataFrame; it’s a code‑generation layer. Use it when you want lean, composable transforms over native Python data.

---

Installation

pip install convtools

Documentation

• convtools.readthedocs.io

• 🤖 LLM-friendly docs: llms.txt — a concise, machine-readable overview of convtools for AI assistants and code generators

Group by example

from convtools import conversion as c

input_data = [
    {"a": 5, "b": "foo"},
    {"a": 10, "b": "foo"},
    {"a": 10, "b": "bar"},
    {"a": 10, "b": "bar"},
    {"a": 20, "b": "bar"},
]

conv = (
    c.group_by(c.item("b"))
    .aggregate(
        {
            "b": c.item("b"),
            "a_first": c.ReduceFuncs.First(c.item("a")),
            "a_max": c.ReduceFuncs.Max(c.item("a")),
        }
    )
    .pipe(
        c.aggregate({
            "b_values": c.ReduceFuncs.Array(c.item("b")),
            "mode_a_first": c.ReduceFuncs.Mode(c.item("a_first")),
            "median_a_max": c.ReduceFuncs.Median(c.item("a_max")),
        })
    )
    .gen_converter()
)

assert conv(input_data) == {
    'b_values': ['foo', 'bar'],
    'mode_a_first': 10,
    'median_a_max': 15.0
}

Built-in reducers like c.ReduceFuncs.First

Built-in reducers are exposed as c.ReduceFuncs.*:

• Value reducers: Array, ArrayDistinct, ArraySorted, Average, Correlation, Count, CountDistinct, Covariance, First, FirstN, Last, LastN, Max, MaxRow, Median, Min, MinRow, Mode, Percentile, PopulationStdDev, PopulationVariance, StdDev, Sum, SumOrNone, TopK, Variance
• Dict reducers: Dict, DictArray, DictArrayDistinct, DictCount, DictCountDistinct, DictFirst, DictFirstN, DictLast, DictLastN, DictMax, DictMin, DictSum, DictSumOrNone

Dict reducers aggregate into dictionaries whose values are reduced per key. For reducer arguments, defaults, None handling, and examples, see Aggregations.

You can also define custom reducers with c.reduce by passing any two-argument reduce function.

---

When should I reach for convtools?

• You need composable transforms over native Python data (lists/dicts/generators/CSV), not a DataFrame.
• You want to express business rules declaratively and generate fast, readable Python functions.
• You need aggregations/joins/pipes that you can reuse across scripts and services.

---

Contributing

• Star the repo and share use‑cases in Discussions -- it really helps.

• To report a bug or suggest enhancements, please open an issue and/or submit a pull request.

• Reporting a Security Vulnerability: see the security policy.