Testing Architecture#

This document explains geomstats’ sophisticated testing infrastructure, designed for comprehensive testing of geometric operations across multiple backends with automatic vectorization validation.

Overview #

Geomstats uses a three-layer testing architecture that separates:

What to test (test data)
How to test it (test case logic)
Where to test it (concrete test instances)

This architecture enables:

Testing one implementation across many geometric spaces
Automatic vectorization testing for batch operations
Backend-agnostic test definitions (NumPy, PyTorch, Autograd)
Parametrized testing without code duplication
Shared test logic across 41+ geometry types

The Three Layers #

Layer 1: Test Data (What to Test)#

Location: tests/tests_geomstats/*/data/*.py

Test data classes define:

Input values and expected outputs
Number of test trials
Tolerances for numerical comparisons
Which tests to skip or mark as expected failures
Random data generation parameters

Example: tests/tests_geomstats/test_geometry/data/special_orthogonal.py

from geomstats.test.data import TestData

class SpecialOrthogonalMatrices2TestData(TestData):
    """Test data for SO(2) matrices."""

    def belongs_test_data(self):
        """Provide test cases for the belongs() method."""
        theta = gs.pi / 3
        data = [
            dict(point=rotation_matrix(theta), expected=True),
            dict(point=gs.zeros((2, 2)), expected=False),
            dict(
                point=gs.stack([rotation_matrix(theta), gs.zeros((2, 2))]),
                expected=gs.array([True, False])
            ),
        ]
        return self.generate_tests(data)

    def identity_test_data(self):
        """Test cases for identity element."""
        data = [dict(expected=gs.eye(2))]
        return self.generate_tests(data)

Key features:

Method names match pattern: <method_name>_test_data()
Returns list of dicts with test parameters
generate_tests() applies pytest markers
Can set class-level tolerances, skips, trials

Layer 2: Test Cases (How to Test)#

Location: geomstats/test_cases/geometry/*.py

Test case classes define the actual test logic, reusable across different spaces:

Example: geomstats/test_cases/geometry/special_orthogonal.py

from geomstats.test_cases.geometry.base import LevelSetTestCase

class SpecialOrthogonalMatricesTestCase(LevelSetTestCase):
    """Reusable tests for special orthogonal matrices."""

    def test_belongs(self, point, expected, atol):
        """Test the belongs method."""
        result = self.space.belongs(point)
        self.assertAllClose(result, expected, atol=atol)

    @pytest.mark.random
    def test_rotation_vector_from_matrix_after_matrix_from_rotation_vector(
        self, n_points, atol
    ):
        """Test that conversions are inverse operations."""
        vec = self._get_random_rotation_vector(n_points)
        mat = self.space.matrix_from_rotation_vector(vec)
        vec_reconstructed = self.space.rotation_vector_from_matrix(mat)
        self.assertAllClose(vec, vec_reconstructed, atol=atol)

Key features:

Test method parameters match data method return values
Uses self.space to access the space being tested
Inherits from base test cases (TestCase, LieGroupTestCase, etc.)
Can use helper methods like assertAllClose

Layer 3: Concrete Tests (Where to Test)#

Location: tests/tests_geomstats/test_geometry/*.py

Concrete test classes instantiate the tests for specific spaces:

Example: tests/tests_geomstats/test_geometry/test_special_orthogonal.py

from geomstats.geometry.special_orthogonal import _SpecialOrthogonalMatrices
from geomstats.test.parametrizers import DataBasedParametrizer
from geomstats.test_cases.geometry.special_orthogonal import (
    SpecialOrthogonalMatricesTestCase,
)
from .data.special_orthogonal import SpecialOrthogonalMatrices2TestData

@pytest.mark.smoke
class TestSpecialOrthogonalMatrices2(
    SpecialOrthogonalMatricesTestCase,  # Layer 2: test logic
    metaclass=DataBasedParametrizer     # Magic: automatic parametrization
):
    space = _SpecialOrthogonalMatrices(n=2, equip=False)  # What to test
    testing_data = SpecialOrthogonalMatrices2TestData()   # Layer 1: test data

The DataBasedParametrizer metaclass automatically:

Discovers test methods (e.g., test_belongs)
Finds matching data methods (e.g., belongs_test_data)
Applies @pytest.mark.parametrize with the test data
Generates vectorization tests automatically

How It Works: The Metaclass Magic #

The Parametrizer Metaclass #

File: geomstats/test/parametrizers.py

The DataBasedParametrizer metaclass automatically connects test methods to their data:

class DataBasedParametrizer(type):
    """Metaclass that auto-generates parametrized tests.

    For each test method test_foo(), looks for a foo_test_data() method
    in the testing_data class and automatically parametrizes the test.
    """

How it works:

When a test class is defined, the metaclass inspects it
For each test_* method, it looks for *_test_data() in testing_data
If found, applies @pytest.mark.parametrize automatically
Generates vectorization tests based on configuration

Example transformation:

# What you write:
class TestMySpace(MyTestCase, metaclass=DataBasedParametrizer):
    space = MySpace()
    testing_data = MyTestData()

# What the metaclass generates (conceptually):
class TestMySpace(MyTestCase):
    space = MySpace()

    @pytest.mark.parametrize("point,expected,atol", [
        (data1["point"], data1["expected"], data1.get("atol", 1e-6)),
        (data2["point"], data2["expected"], data2.get("atol", 1e-6)),
    ])
    def test_belongs(self, point, expected, atol):
        result = self.space.belongs(point)
        self.assertAllClose(result, expected, atol=atol)

Test Data Generation #

File: geomstats/test/data.py

The TestData base class provides helpers for generating test data:

class TestData:
    """Base class for test data."""

    trials = 3  # Number of random test repetitions
    N_RANDOM_POINTS = [1, 2, 3]  # Points for random tests
    N_VEC_REPS = [1, 2, 3]  # Repetitions for vectorization tests

    def generate_tests(self, data, marks=()):
        """Add pytest markers to test data."""
        # Adds markers, tolerances, returns formatted data

    def generate_random_data(self, marks=()):
        """Generate random test data."""
        data = []
        for n_points in self.N_RANDOM_POINTS:
            for _ in range(self.trials):
                data.append(dict(n_points=n_points))
        return self.generate_tests(data, marks=marks)

    def generate_vec_data(self, vec_type="sym"):
        """Generate vectorization test data."""
        data = []
        for n_reps in self.N_VEC_REPS:
            data.append(dict(n_reps=n_reps))
        return self.generate_tests(data, marks=(pytest.mark.vec,))

Vectorization Testing #

Why Vectorization Matters #

Geomstats is a numerical library where operations must work on:

Single points: shape = (dim,)
Batches of points: shape = (batch_size, dim)
Multiple batches: shape = (batch1, batch2, dim)

Example:

# Single distance
d = metric.dist(point_a, point_b)  # scalar

# Batch distances
d = metric.dist(batch_of_points_a, batch_of_points_b)  # array of distances

Vectorization testing ensures functions handle all these cases correctly.

Automatic Vectorization Tests #

File: geomstats/test/vectorization.py

The framework automatically generates vectorization tests:

@pytest.mark.vec
def test_belongs_vec(self, n_reps, atol):
    """Auto-generated vectorization test for belongs()."""
    # Generate single point
    point = self.space.random_point()
    expected = self.space.belongs(point)

    # Generate vectorized data
    vec_data = generate_vectorization_data(
        data=[dict(point=point, expected=expected, atol=atol)],
        arg_names=["point"],
        expected_name="expected",
        n_reps=n_reps,
    )
    self._test_vectorization(vec_data)

What it tests:

n_reps=1: belongs(point) vs belongs([point])
n_reps=2: belongs([point1]) vs belongs([point1, point1])
n_reps=3: belongs([point1, point2]) vs belongs([point1, point1, point2, point2])

This ensures proper broadcasting behavior.

Backend Compatibility #

Testing Across Backends #

Geomstats supports three backends:

NumPy (default)
PyTorch
Autograd

Tests must work across all backends. The test infrastructure handles this through:

Backend-agnostic assertions

# Good: Works with all backends
self.assertAllClose(result, expected, atol=1e-6)

# Bad: NumPy-specific
np.testing.assert_allclose(result, expected)

Backend markers for conditional tests

from geomstats.test.test_case import np_only, torch_only

@np_only
def test_numpy_specific_feature(self):
    """Only runs with NumPy backend."""
    pass

@torch_only
def test_pytorch_specific_feature(self):
    """Only runs with PyTorch backend."""
    pass

Backend detection

if self.np_backend():
    # NumPy-specific logic
    pass
elif self.pytorch_backend():
    # PyTorch-specific logic
    pass

Test Markers #

Geomstats uses pytest markers to categorize tests:

# In pyproject.toml
markers = [
    "smoke: simple and basic numerical tests.",
    "random: tests that use randomized data.",
    "validation: not smoke, neither random.",
    "vec: vectorization tests.",
    "shape: array shape tests.",
    "type: checks output types.",
    "mathprop: mathematical properties tests.",
    "slow: for slow tests.",
    "redundant: redundant test.",
    "ignore: deselect tests.",
]

Usage:

# Run only smoke tests (fast)
$ pytest -m smoke

# Run all except slow tests
$ pytest -m "not slow"

# Run smoke and random tests
$ pytest -m "smoke or random"

Advanced Features #

Tolerance Configuration #

Different tests may need different numerical tolerances:

class MyTestData(TestData):
    tolerances = {
        "projection_belongs": {"atol": 1e-5},
        "exp_log_inverse": {"atol": 1e-6, "rtol": 1e-5},
    }

These tolerances are automatically passed to test methods.

Skip Configuration #

Skip tests that aren’t applicable:

class MyTestData(TestData):
    skips = (
        "projection_belongs",  # No projection method
        "distance_is_symmetric",  # Not a metric space
    )

Expected Failures #

Mark known issues as expected failures:

class MyTestData(TestData):
    xfails = (
        "exp_belongs",  # Known issue #123
    )

Random Data Generators #

File: geomstats/test/random.py

Specialized random data generators for different spaces:

from geomstats.test.random import RandomDataGenerator

class MyRandomDataGenerator(RandomDataGenerator):
    def point(self, n_points=1):
        """Generate random points on the space."""
        return self.space.random_point(n_points)

    def tangent_vec(self, point=None, n_points=1):
        """Generate random tangent vectors."""
        if point is None:
            point = self.point(n_points)
        return self.space.to_tangent(
            gs.random.normal(size=(n_points,) + self.space.shape),
            point
        )

Inheritance Hierarchies #

Test Case Inheritance #

Test cases follow the inheritance hierarchy of the spaces they test:

TestCase (base)
├── OpenSetTestCase
│   └── VectorSpaceTestCase
│       └── EuclideanTestCase
├── LevelSetTestCase
│   └── LieGroupTestCase
│       ├── MatrixLieGroupTestCase
│       │   └── SpecialOrthogonalMatricesTestCase
│       └── SpecialOrthogonalVectorsTestCase
└── QuotientMetricTestCase

This enables automatic testing of inherited properties.

Test Data Inheritance #

Test data classes also inherit:

class LieGroupTestData(TestData):
    """Common tests for all Lie groups."""

    def identity_test_data(self):
        pass

    def compose_test_data(self):
        pass

class SpecialOrthogonalMatricesTestData(LieGroupTestData):
    """Adds SO(n)-specific tests."""

    def are_antipodals_test_data(self):
        pass

Working with the Architecture #

When to Use the Architecture #

Use the three-layer architecture when:

Adding a new geometric space with standard operations
Testing operations that need vectorization validation
Writing tests that apply to multiple similar spaces
Contributing core geometry functionality

Don’t use it when:

Writing a simple bug fix test
Testing a one-off edge case
Learning the codebase for the first time

Adding Tests for a New Space #

Step 1: Create test data class

# tests/tests_geomstats/test_geometry/data/my_space.py

from geomstats.test.data import TestData

class MySpaceTestData(TestData):
    def belongs_test_data(self):
        data = [
            dict(point=valid_point, expected=True),
            dict(point=invalid_point, expected=False),
        ]
        return self.generate_tests(data)

Step 2: Create or reuse test case class

# geomstats/test_cases/geometry/my_space.py

from geomstats.test_cases.geometry.base import OpenSetTestCase

class MySpaceTestCase(OpenSetTestCase):
    def test_my_specific_method(self, input_val, expected, atol):
        result = self.space.my_specific_method(input_val)
        self.assertAllClose(result, expected, atol=atol)

Step 3: Create concrete test

# tests/tests_geomstats/test_geometry/test_my_space.py

from geomstats.geometry.my_space import MySpace
from geomstats.test.parametrizers import DataBasedParametrizer
from geomstats.test_cases.geometry.my_space import MySpaceTestCase
from .data.my_space import MySpaceTestData

@pytest.mark.smoke
class TestMySpace(MySpaceTestCase, metaclass=DataBasedParametrizer):
    space = MySpace(dim=3)
    testing_data = MySpaceTestData()

Debugging Parametrized Tests #

When a parametrized test fails, pytest shows which parameter set failed:

FAILED tests/test_geometry/test_my_space.py::TestMySpace::test_belongs[point0] - AssertionError
FAILED tests/test_geometry/test_my_space.py::TestMySpace::test_belongs[point1] - AssertionError

To run a specific parameter set:

$ pytest tests/test_geometry/test_my_space.py::TestMySpace::test_belongs[point0]

To see the actual parameter values, run with -vv:

$ pytest -vv tests/test_geometry/test_my_space.py::TestMySpace::test_belongs

Common Pitfalls #

Pitfall 1: Method Name Mismatch #

The test method and data method must match:

# Wrong: Names don't match
def test_belongs(...):  # test method
    pass

def belong_test_data(self):  # data method (typo!)
    pass

# Correct:
def test_belongs(...):  # test method
    pass

def belongs_test_data(self):  # data method
    pass

Pitfall 2: Parameter Name Mismatch #

Test method parameters must match data dict keys:

# Wrong:
def test_belongs(self, point, expected_result, atol):  # expected_result
    pass

def belongs_test_data(self):
    return [dict(point=p, expected=e)]  # expected (mismatch!)

# Correct:
def test_belongs(self, point, expected, atol):
    pass

def belongs_test_data(self):
    return [dict(point=p, expected=e)]

Pitfall 3: Forgetting generate_tests()#

Always wrap data with generate_tests():

# Wrong:
def belongs_test_data(self):
    return [dict(point=p, expected=e)]  # No markers applied!

# Correct:
def belongs_test_data(self):
    data = [dict(point=p, expected=e)]
    return self.generate_tests(data)

Pitfall 4: Not Testing Vectorization #

Add vectorization test data:

class MyTestData(TestData):
    def my_method_vec_test_data(self):
        """Vectorization test for my_method."""
        return self.generate_vec_data()

Summary #

The three-layer architecture may seem complex initially, but it provides:

Massive code reuse - One test case, many instantiations
Automatic vectorization testing - Ensures batch operations work
Clear separation of concerns - What vs how vs where
Maintainability - Changes propagate automatically
Consistency - Enforces testing patterns across the codebase

For simple contributions, you don’t need to use this architecture. But when adding comprehensive tests for geometric spaces, it provides powerful automation and consistency.