RFC 002: Parametrized Tests¶

Status: Draft Created: 2024-12-08

Summary¶

Add pytest-style parametrized tests via the @parametrize decorator.

Motivation¶

Parametrized tests allow:

DRY testing - One test definition, many inputs
Clear failures - Each parameter set is a separate test case
Readable test data - Parameters are explicit, not hidden in loops

Design¶

Basic Syntax¶

from testing import parametrize

@parametrize("a, b, expected", [
    (1, 2, 3),
    (0, 0, 0),
    (-1, 1, 0),
    (100, 200, 300),
])
def test_add(a: int, b: int, expected: int) -> None:
    assert_eq(add(a, b), expected)

Output:

test_math.incn::test_add[1-2-3] PASSED
test_math.incn::test_add[0-0-0] PASSED
test_math.incn::test_add[-1-1-0] PASSED
test_math.incn::test_add[100-200-300] PASSED

Named Parameters¶

@parametrize("input, expected", [
    ("hello", "HELLO"),
    ("World", "WORLD"),
    ("", ""),
], ids=["lowercase", "mixed", "empty"])
def test_upper(input: str, expected: str) -> None:
    assert_eq(input.upper(), expected)

Output:

test_string.incn::test_upper[lowercase] PASSED
test_string.incn::test_upper[mixed] PASSED
test_string.incn::test_upper[empty] PASSED

Multiple Parametrize Decorators¶

@parametrize("x", [1, 2, 3])
@parametrize("y", [10, 20])
def test_multiply(x: int, y: int) -> None:
    result = x * y
    assert_true(result > 0)

Generates 6 tests (cartesian product):

test_multiply[1-10], test_multiply[1-20],
test_multiply[2-10], test_multiply[2-20],
test_multiply[3-10], test_multiply[3-20]

Complex Types¶

model TestCase:
    input: str
    expected: int
    description: str

@parametrize("case", [
    TestCase(input="hello", expected=5, description="simple word"),
    TestCase(input="", expected=0, description="empty string"),
    TestCase(input="hello world", expected=11, description="with space"),
])
def test_length(case: TestCase) -> None:
    assert_eq(len(case.input), case.expected)

Combining with Fixtures¶

@fixture
def database() -> Database:
    db = Database.connect("test.db")
    yield db
    db.close()

@parametrize("key, value", [
    ("name", "Alice"),
    ("age", "30"),
])
def test_insert(database: Database, key: str, value: str) -> None:
    database.insert(key, value)
    assert_eq(database.get(key), value)

Implementation¶

Phase 1: Parser Changes¶

Parse @parametrize decorator:

DecoratorArg::Parametrize {
    argnames: String,           # "a, b, expected"
    argvalues: Vec<Tuple>,      # [(1,2,3), (0,0,0), ...]
    ids: Option<Vec<String>>,   # Optional test IDs
}

Phase 2: Test Discovery¶

When discovering tests:

Detect @parametrize decorator
Parse argnames and argvalues
Generate N TestInfo entries, one per parameter set

struct TestInfo {
    file_path: PathBuf,
    function_name: String,
    markers: Vec<TestMarker>,
    parameters: Option<ParametrizeInfo>,  // NEW
}

struct ParametrizeInfo {
    argnames: Vec<String>,
    values: Vec<Value>,
    id: String,  // e.g., "1-2-3" or custom ID
}

Phase 3: Codegen¶

Generate a separate Rust test function for each parameter set:

// From: test_add with params [(1,2,3), (0,0,0)]

#[test]
fn test_add_1_2_3() {
    let a: i64 = 1;
    let b: i64 = 2;
    let expected: i64 = 3;
    assert_eq!(add(a, b), expected);
}

#[test]
fn test_add_0_0_0() {
    let a: i64 = 0;
    let b: i64 = 0;
    let expected: i64 = 0;
    assert_eq!(add(a, b), expected);
}

Phase 4: Output Formatting¶

Display parameterized test names:

test_math.incn::test_add[1-2-3] PASSED
test_math.incn::test_add[0-0-0] PASSED

On failure, show which parameters failed:

FAILED test_math.incn::test_add[1-2-3]

    Parameters: a=1, b=2, expected=3

    assert_eq(add(a, b), expected)
    AssertionError: 2 != 3

Edge Cases¶

Empty Parameter List¶

@parametrize("x", [])
def test_nothing(x: int) -> None:
    pass

Result: No tests generated, warning emitted.

Type Mismatches¶

@parametrize("x", [1, "two", 3.0])
def test_typed(x: int) -> None:
    pass

Result: Compile error - parameter types must match function signature.

Nested Tuples¶

@parametrize("point", [
    ((0, 0), "origin"),
    ((1, 1), "diagonal"),
])
def test_points(point: Tuple[Tuple[int, int], str]) -> None:
    coords, name = point
    ...

Alternatives Considered¶

1. Loop in Test¶

def test_add_all() -> None:
    cases = [(1,2,3), (0,0,0)]
    for a, b, expected in cases:
        assert_eq(add(a, b), expected)

Rejected:

Stops at first failure
No individual test reporting
Harder to identify which case failed

2. Test Tables (Go-style)¶

def test_add() -> None:
    table = [
        {"a": 1, "b": 2, "expected": 3},
        {"a": 0, "b": 0, "expected": 0},
    ]
    for case in table:
        with subtest(case):
            assert_eq(add(case["a"], case["b"]), case["expected"])

Rejected: More verbose, requires subtest infrastructure.

Open Questions¶

Lazy evaluation: Should parameter values be evaluated at collection time or test time?
Indirect parametrization: Should parameters be able to reference fixtures?
Skip individual params: @parametrize(..., marks=[skip, None, xfail])?

Checklist¶

[ ] Parser: @parametrize decorator args (argnames, argvalues, ids)
[ ] Discovery: expand parameter sets into TestInfo entries
[ ] Codegen: generate per-parameter Rust test functions
[ ] Reporting: parameterized test names and failure context
[ ] Validation: type-check param values vs function signature
[ ] Edge cases: empty parameter list warnings
[ ] Fixture interaction: combine with fixtures safely