Plugins and Configuration¶

SMQTK provides generic plugin and introspective configuration mixin classes to support interface implementation discovery and their translation form/to JSON as a plain-text configuration format.

While these two mixins function independently and can be utilized on their own, their combination is symbiotic and allows for users of SMQTK algorithms and representations to create tools in terms of interfaces and leave the specific selection of implementations for configuration time.

Later, we will introduce the two categories of configurable and (usually) pluggable class classes found within SMQTK.

Plugins¶

Motivation: We want to be able to define interfaces to generic concepts and structures around which higher order tools can be defined without strictly catering themselves to any particular implementation, while additionally allowing freedom in implementation variety without overly restricting implementation location.

In SMQTK, this is addressed via the get_plugins() function and the Pluggable abstract mixin class.

Interfaces and Implementations¶

Classes that inherit from the Pluggable mixin are considered either pluggable interfaces or plugin implementations depending on whether they fully implement abstract methods.

Interface implementations bundled within SMQTK are generally defined alongside their parent interfaces. However, other sources, e.g. other python packages, may expose their own plugin implementations via setting a system environment variable or via python extensions.

The `Configurable` Mixin¶

Motivation: We want generic helpers to enable serializable configuration for classes while minimally impacting standard class development.

SMQTK provides the Configurable mixin class as well as other helper utility functions in smqtk.utils.configuration for generating, and producing class instances from, configurations. These use python’s introspect module to determine default configurations.

Currently this module deals in JSON for input and output configuration. Python dictionaries are used as a medium in between serialization and class input/output.

Classes that inherit from Configurable do need to at a minimum implement the get_config() instance method. This does detract from the “minimal impact” intent of this mixin, but other methods of allowing introspection of internal parameters require additional structural components in the parent/implementing class.

Algorithms and Representations - The Combination¶

Interfaces found in SMQTK are generally binned into two categories: representations and algorithms.

Algorithms are interfaces to some function or operation, specifically parameterized through their constructor and generally parameterized via the algorithm’s interface. The SmqtkAlgorithm base class inherits from both Pluggable and Configurable mixins so that all descendents gain access to the synergy they provide. These are located under the smqtk.algorithms sub-module.

Representations are interfaces to structures that are intended to specifically store some sort of data structure. Currently, the SmqtkRepresentation only inherits directly from Configurable, as there are some representational structures which desire configurability but to which variable implementations do not make sense (like DescriptorElementFactory). However most sub-classes do additionally inherit from Pluggable (like DescriptorElement). These are located under the smqtk.representation sub-module.

Implementing a Pluggable Interface¶

The following are examples of how to add and expose new plugin implementations for existing algorithm and representation interfaces.

SMQTK’s plugin discovery allows for exposure of plugin implementations in 3 ways:

Parallel in location to parent interface.

Python module path of implementation model included in the SMQTK_PLUGIN_PATH environment variable (see reference for formatting).

An entrypoint in a python package’s setup.py.

Within SMQTK¶

A new interface implementation within the SMQTK source-tree is generally implemented or exposed parallel to where the parent interface is defined.

As an example, we will show how to create a new implementation for the Classifier algorithm interface. This interface is defined within SMQTK at, from the root of the source tree, python/smqtk/algorithms/classifier/_interface_classifier.py. We will create a new file, some_impl.py, that will be placed in the same directory with the intention that our new plugin will be picked up based on parallel locality to the parent interface class.

We’ll define our new class, lets call it SomeImpl, in a file some_impl.py:

python/
└── smqtk/
    └── algorithms/
        └── classifier/
            ├── ...
            ├── _interface_classifier.py
            ├── some_impl.py     # new

In this file we will need to define the SomeImpl class and all parent class abstract methods in order for the class to satisfy the definition of an “implementation”:

from smqtk.algorithms import Classifier

class SomeImpl (Classifier):
    """
    Some documentation for this specific implementation.
    """

    # Abstract methods from Pluggable.
    # (Classifier -> SmqtkAlgorithm -> Pluggable)
    @classmethod
    def is_usable(cls):
        ...

    # Our implementation-specific constructor.
    def __init__(self, paramA=1, paramB=2):
        ...

    # Abstract methods from Configurable.
    # (Classifier -> SmqtkAlgorithm -> Configurable)
    def get_config(self):
        return {
            "paramA": ...,
            "paramB": ...,
        }

    # Classifier's abstract methods.
    def get_labels(self):
        ...

    def _classify_arrays(self, array_iter):
        ...

With all abstract methods defined, this implementation should now be included in the returned set of implementation types for the parent Classifier interface:

>>> from smqtk.algorithms import Classifier
>>> Classifier.get_impls()
set([..., SomeImpl, ...])

SomeImpl above should also be all set for configuration because of it defining get_config() and because it’s constructor is only anticipating JSON-conpliant types. If more complicated types are desired by the constructor the additional methods would need to be overriden/extended as defined in the smqtk.utils.configuration module.

More Complicated Implementations¶

Interface-parallel implementation discovery also allows for nested sub-modules. This is useful when an implementation requires specific or extensive support utilities. The __init__.py file of the sub-module should at least expose concrete implementation classes that should be exported as attributes for the plugin discovery to find. For example, such a nested sub-module implementation might look like the following on the filesystem:

python/
└── smqtk/
    └── algorithms/
        └── classifier/
            ├── ...
            ├── some_impl.py     # from above
            └── other_impl/      # new
                └── __init__.py  # new

Within another python package¶

When implementing a pluggable interface in another python package, the proper method of export is via a package’s entrypoint specifications using the smqtk_plugins key.

For example, let’s assume that a separate python package, OtherPackage we’ll call it, defines a Classifier-implementing sub-class OtherClassifier in the module OtherPackage.other_classifier. This module location can be exposed via the package’s setup.py entrypoints metadata, using the smqtk_plugins key, like the following:

from setuptools import setup

...

setup(
    ...
    entry_points={
        'smqtk_plugins': 'my_plugins = OtherPackage.other_classifier'
    }
)

If the other module had multiple sub-modules in which SMQTK plugins were defined the entry_points['smqtk_plugins'] entry may instead be a list:

setup(
    ...
    entry_points={
        'smqtk_plugins': [
            'classifier_plugins = OtherPackage.other_classifier',
            'other_plugins = OtherPackage.other_plugins',
        ]
    }
)

Reference¶

`smqtk.utils.configuration`¶

Helper interface and functions for generalized object configuration, to and from JSON-compliant dictionaries.

While this interface and utility methods should be general enough to add JSON-compliant dictionary-based configuration to any object, this was created in mind with the SMQTK plugin module.

Standard configuration dictionaries should be JSON compliant take the following general format:

{
    "type": "one-of-the-keys-below",
    "ClassName1": {
        "param1": "val1",
        "param2": "val2"
    },
    "ClassName2": {
        "p1": 4.5,
        "p2": null
    }
}

The “type” key is considered a special key that should always be present and it specifies one of the other keys within the same dictionary. Each other key in the dictionary should be the name of a Configurable inheriting class type. Usually, the classes named within a block inherit from a common interface and the “type” value denotes a selection of a specific sub-class for use, though this is not required property of these constructs.

class smqtk.utils.configuration.Configurable[source]¶

Interface for objects that should be configurable via a configuration dictionary consisting of JSON types.

classmethod from_config(config_dict, merge_default=True)[source]¶

Instantiate a new instance of this class given the configuration JSON-compliant dictionary encapsulating initialization arguments.

This base method is adequate without modification when a class’s constructor argument types are JSON-compliant. If one or more are not, however, this method then needs to be overridden in order to convert from a JSON-compliant stand-in into the more complex object the constructor requires. It is recommended that when complex types are used they also inherit from the Configurable in order to hopefully make easier the conversion to and from JSON-compliant stand-ins.

When this method does need to be overridden, this usually looks like the following pattern:

class MyClass (Configurable):

    @classmethod
    def from_config(cls, config_dict, merge_default=True):
        # Optionally guarantee default values are present in the
        # configuration dictionary.  This statement pairs with the
        # ``merge_default=False`` parameter in the super call.
        # This also in effect shallow copies the given non-dictionary
        # entries of ``config_dict`` due to the merger with the
        # default config.
        if merge_default:
            config_dict = merge_dict(cls.get_default_config(),
                                     config_dict)

        #
        # Perform any overriding here.
        #

        # Create and return an instance using the super method.
        return super(MyClass, cls).from_config(config_dict,
                                               merge_default=False)

This method should not be called via super unless an instance of the class is desired.

Parameters:	config_dict (dict) – JSON compliant dictionary encapsulating a configuration. merge_default (bool) – Merge the given configuration on top of the default provided by `get_default_config`.
Returns:	Constructed instance from the provided config.
Return type:	Configurable

get_config()[source]¶

Return a JSON-compliant dictionary that could be passed to this class’s from_config method to produce an instance with identical configuration.

In the most cases, this involves naming the keys of the dictionary based on the initialization argument names as if it were to be passed to the constructor via dictionary expansion. In some cases, where it doesn’t make sense to store some object constructor parameters are expected to be supplied at as configuration values (i.e. must be supplied at runtime), this method’s returned dictionary may leave those parameters out. In such cases, the object’s from_config class-method would also take additional positional arguments to fill in for the parameters that this returned configuration lacks.

Returns:	JSON type compliant configuration dictionary.
Return type:	dict

classmethod get_default_config()[source]¶

Generate and return a default configuration dictionary for this class. This will be primarily used for generating what the configuration dictionary would look like for this class without instantiating it.

By default, we observe what this class’s constructor takes as arguments, turning those argument names into configuration dictionary keys. If any of those arguments have defaults, we will add those values into the configuration dictionary appropriately. The dictionary returned should only contain JSON compliant value types.

It is not be guaranteed that the configuration dictionary returned from this method is valid for construction of an instance of this class.

Returns:	Default configuration dictionary for the class.
Return type:	dict

>>> # noinspection PyUnresolvedReferences
>>> class SimpleConfig(Configurable):
...     def __init__(self, a=1, b='foo'):
...         self.a = a
...         self.b = b
...     def get_config(self):
...         return {'a': self.a, 'b': self.b}
>>> self = SimpleConfig()
>>> config = self.get_default_config()
>>> assert config == {'a': 1, 'b': 'foo'}

smqtk.utils.configuration.cls_conf_from_config_dict(config, type_iter)[source]¶

Helper function for getting the appropriate type and configuration sub-dictionary based on the provided “standard” SMQTK configuration dictionary format (see above module documentation).

Parameters:	config (dict) – Configuration dictionary to draw from. type_iter (collections.Iterable[type]) – An iterable of class types to select from.
Raises:	ValueError – This may be raised if: type field not present in `config`. type field set to `None` type field did not match any available configuration in the given config. Type field did not specify any implementation key.
Returns:	Appropriate class type from `type_iter` that matches the configured type as well as the sub-dictionary from the configuration. From this return, `type.from_config(config)` should be callable.
Return type:	(type, dict)

smqtk.utils.configuration.cls_conf_to_config_dict(cls, conf)[source]¶

Helper function for creating the appropriate “standard” smqtk configuration dictionary given a Configurable-implementing class and a configuration for that class.

This very simple function simply arranges a class, using its __name__ property, and an associated dictionary into a normal pattern used for configuration in SMQTK:

>>> class SomeClass (object):

… pass >>> cls_conf_to_config_dict(SomeClass, {0: 0, ‘a’: ‘b’}) == { … ‘type’: ‘SomeClass’, … ‘SomeClass’: {0: 0, ‘a’: ‘b’} … } True

Parameters:	cls (type[Configurable]) – A class type implementing the Configurable interface. conf (dict) – SMQTK standard type-optioned configuration dictionary for the given class and dictionary pair.
Returns:	“Standard” SMQTK JSON-compliant configuration dictionary
Return type:	dict

smqtk.utils.configuration.configuration_test_helper(inst, config_ignored_params=frozenset(), from_config_args=())[source]¶

Helper function for testing the get_default_config/from_config/get_config methods for class types that in part implement the Configurable mixin class. This function also tests that inst’s parent class type’s get_default_config returns a dictionary whose keys’ match the constructor’s inspected parameters (except “self” of course).

This constructs 3 additional instances based on the given instance following the pattern:

inst-1  ->  inst-2  ->  inst-3
        ->  inst-4

This refers to inst-2 and inst-4 being constructed from the config from inst, and inst-3 being constructed from the config of inst-2. The equivalence of each instance’s config is cross-checked with the other instances. This is intended to check that a configuration yields the same class configurations and that the config does not get mutated by nested instance construction.

This function uses assert calls to check for consistency.

We return all instances constructed in case the caller wants to make additional instance integrity checks.

Parameters:	inst (Configurable) – Configurable-mixin inheriting class to test. config_ignored_params (set[str]) – Set of parameter names in the instance type’s constructor that are ignored by `get_default_config` and `from_config`. This is empty by default. from_config_args (tuple) – Optional additional positional arguments to the input `inst.from_config` method after the configuration dictionary.
Returns:	Instance 2, 3, and 4 as described above.
Return type:	(Configurable,Configurable,Configurable)

smqtk.utils.configuration.from_config_dict(config, type_iter, *args)[source]¶

Helper function for instantiating an instance of a class given the configuration dictionary config from available types provided by type_iter via the Configurable interface’s from_config class-method.

args are additionally positional arguments to be passed to the type’s from_config method on return.

Example: >>> from smqtk.representation import DescriptorElement >>> example_config = { … ‘type’: ‘DescriptorMemoryElement’, … ‘DescriptorMemoryElement’: {}, … } >>> inst = from_config_dict(example_config, DescriptorElement.get_impls(), … ‘type-str’, ‘some-uuid’) >>> from smqtk.representation.descriptor_element.local_elements import DescriptorMemoryElement >>> isinstance(inst, DescriptorMemoryElement) True

Raises:	ValueError – This may be raised if: type field not present in `config`. type field set to `None` type field did not match any available configuration in the given config. Type field did not specify any implementation key. AssertionError – This may be raised if the class specified as the configuration type, is present in the given `type_iter` but is not a subclass of the `Configurable` interface. TypeError – Insufficient/incorrect initialization parameters were specified for the specified `type`’s constructor.
Parameters:	config (dict) – Configuration dictionary to draw from. type_iter (collections.Iterable[type]) – An iterable of class types to select from. args (object) – Other positional arguments to pass to the configured class’ `from_config` class method.
Returns:	Instance of the configured class type as specified in `config` and as available in `type_iter`.
Return type:	smqtk.utils.configuration.Configurable

smqtk.utils.configuration.make_default_config(configurable_iter)[source]¶

Generated default configuration dictionary for the given iterable of Configurable-inheriting types.

For example, assuming the following simple class that descends from Configurable, we would expect the following behavior:

>>> class ExampleConfigurableType (Configurable):
...     def __init__(self, a, b):
...        ''' Dummy constructor '''
>>> make_default_config([ExampleConfigurableType]) == {
...     'type': None,
...     'ExampleConfigurableType': {
...         'a': None,
...         'b': None,
...     }
... }
True

Note that technically ExampleConfigurableType is still abstract as it does not implement get_config. The above call to make_default_config still functions because we only use the get_default_config class method and do not instantiate any types given to this function. While functionally acceptable, it is generally not recommended to draw configurations from abstract classes.

Parameters:	configurable_iter (collections.Iterable[type]) – An iterable of class types class types that sub-class `Configurable`.
Returns:	Base configuration dictionary with an empty `type` field, and containing the types and initialization parameter specification for all implementation types available from the provided getter method.
Return type:	dict[str, object]

smqtk.utils.configuration.to_config_dict(c_inst)[source]¶

Helper function that transforms the configuration dictionary retrieved from configurable_inst into the “standard” SMQTK configuration dictionary format (see above module documentation).

For example, with a simple DataFileElement: >>> from smqtk.representation.data_element.file_element import DataFileElement >>> e = DataFileElement(filepath=’/path/to/file.txt’, … readonly=True) >>> to_config_dict(e) == { … “type”: “DataFileElement”, … “DataFileElement”: { … “filepath”: “/path/to/file.txt”, … “readonly”: True, … “explicit_mimetype”: None, … } … } True

Parameters:	c_inst (Configurable) – Instance of a class type that subclasses the `Configurable` interface.
Returns:	Standard format configuration dictionary.
Return type:	dict

`smqtk.utils.plugin`¶

Helper interface and function for implementing class discovery.

Plugins may be accessed by one of the following ways:

Be defined within SMQTK next to their interface.
The environment variable SMQTK_PLUGIN_PATH may be set to a number of :-separated (; on Windows) python module paths to where plugin classes are defined.
Other installed python packages may define one or more extensions for the namespace “smqtk_plugins”. This should be a single or list of extensions that specify modules within the installed package where plugins for export are implemented. Note that we desire modules, not objects, for our extensions.

For example:
```
...
entry_points={
    "smqtk_plugins": "my_package = my_package.plugins"
]
...
```
Or:
```
...
entry_points = {
    "smqtk_plugins": [
        "my_package_mode_1 = my_package.mode_1.plugins",
        "my_package_mode_2 = my_package.mode_2.plugins",
    ]
}
...
```

exception smqtk.utils.plugin.NotUsableError[source]¶: Exception thrown when a pluggable class is constructed but does not report as usable.

class smqtk.utils.plugin.Pluggable[source]¶

Interface for classes that have plugin implementations

classmethod get_impls(warn=True, reload_modules=False)[source]¶

Discover and return a set of classes that implement the calling class.

See the get_plugins function for more details on the logic of how implementing classes (aka “plugins”) are discovered.

The class-level variables PLUGIN_ENV_VAR and PLUGIN_HELPER_VAR may be overridden to change what environment and helper variable are looked for, respectively.

Parameters:	warn (bool) –

If we should warn about module import failures.

Parameters:	reload_modules (bool) – Explicitly reload discovered modules from source.
Returns:	Set of discovered class types descending from type `interface_type` and `smqtk.utils.plugin.Pluggable` whose keys are the string names of the class types.
Return type:	set[type[Pluggable]]

classmethod is_usable()[source]¶

Check whether this class is available for use.

Since certain plugin implementations may require additional dependencies that may not yet be available on the system, this method should check for those dependencies and return a boolean saying if the implementation is usable.

NOTES:

This should be a class method
When an implementation is deemed not usable, this should emit a

warning detailing why the implementation is not available for use.

Returns:	Boolean determination of whether this implementation is usable.
Return type:	bool

smqtk.utils.plugin.get_plugins(interface_type, env_var, helper_var, warn=True, reload_modules=False)[source]¶

Discover and return classes implementing the given interface_class.

Discoverable implementations may either be located in sub-modules parallel to the definition of the interface class or be located in modules specified in the environment variable env_var.

In order to specify additional out-of-scope python modules containing interface-class implementations, additions to the given environment variable must be made. Entries must be separated by the standard PATH separating character based on the operating OS standard (e.g. ‘;’ (for windows) or ‘:’ for most everything else). Entries should be importable python module paths.

When looking at module attributes, we only acknowledge those that start with an alphanumeric character. ‘_’ prefixed attributes are effectively hidden from discovery by this function when merely scanning a module’s attributes.

We required that the base class that we are checking for also descends from the Pluggable interface defined above. This allows us to check if a loaded class is_usable.

Within a module we first look for a helper variable by the name provided, which can either be a single class object or an iterable of class objects, to be specifically exported. If the variable is set to None, we skip that module and do not import anything. If the variable is not present, we look at attributes defined in that module for classes that descend from the given base class type. If none of the above are found, or if an exception occurs, the module is skipped.

Parameters:	interface_type (type) – Interface class type of which we want to discover implementations of (the plugins). env_var (str) – String name of an environment variable defining additional python module paths, whose child modules are searched for implementing sub-classes of the specified base type. helper_var (str) – Name of the expected module helper attribute. warn (bool) – If we should warn about module import failures. reload_modules (bool) – Explicitly reload discovered modules from source instead of taking a potentially cached version of the module.
Returns:	Set of discovered class types descending from type `interface_type` and `smqtk.utils.plugin.Pluggable` whose keys are the string names of the class types.
Return type:	set[type[Pluggable]]

Reload Use Warning¶

While the smqtk.utils.plugin.get_plugins() function allows for reloading discovered modules for potentially new content, this is not recommended under normal conditions. When reloading a plugin module after pickle serializing an instance of an implementation, deserialization causes an error because the original class type that was pickled is no longer valid as the reloaded module overwrote the previous plugin class type.