Skip to content

Testing and Verifying Examples

One of the great thing about using programmatically defined examples is that it enables testing, type verification, and interaction.

Discovering and Interacting with Examples

By default, all examples are added to the docstring of any function that includes them. These examples are grouped under an "Examples:" section at the bottom of the __doc__ string. If you want to use or interact with one of the examples, you can easily do so via the examples libraries get_examples function:

from examples import get_examples

import module_with_examples


get_examples(module_with_examples.function_with_examples)[0].use()

The function returns a list of all examples for a passed-in function or module. Any of these examples can be introspected, interacted with, and directly used. For a full definition of the actions you can perform against a single example, see the API reference documentation for the CallableExample class.

Verifying Examples

The most basic mechanism eXamples provides for ensuring examples don't fall out of sync with the function they call is signature_verification. Signature verification ensures the parameters presented in the example match up with the parameters of the associated function. By default, it then takes the additional step of verifying that the types provided and returned by the example match those specified by the functions type annotations.

Signature verification can be performed over a single example, a function, a module, or all examples defined. In general, for most projects, a module is the right level of specificity, to ensure that signatures are verified across your project.

from examples import verify_signatures

import module_with_examples


def test_example_signatures():
    verify_signatures(module_with_examples)

Testing Examples

Beyond signature verification, examples can also operate as complete test cases for your project. When using examples as test cases, success happens when:

  • Your example doesn't specify a return value AND calling the function with your example's provided parameters doesn't raise an exception.
  • Your example does specify a return value AND calling the function with your example's provided parameters returns that exact return value.
  • Your example doesn't specify a return value, but it does specify an _example_raises exception type or instance, AND a matching exception is raised when calling the function.

To test examples, you can use eXample's test_examples function. This function follows the same general guidelines as signature verification, and can also run over a function or module.

from examples import test_examples

import module_with_examples


def test_examples():
    test_examples(module_with_examples)

Testing and Verifying Examples in One Step

For many projects, it makes sense to both verify the signature of and test all examples. eXamples provides a verify_and_test_examples convenience function to do them both in one step:

from examples import verify_and_test_examples

import module_with_examples


def test_examples_including_their_signature():
    verify_and_test_examples(module_with_examples)