SMQTK Architecture Overview

SMQTK is mainly comprised of 4 high level components, with additional sub-modules for tests, utilities and other control structures.

• Data Abstraction
  • Data Representation Structures
  • Data Support Structures
• Algorithms
  • Algorithm Interfaces
  • Algorithm Models and Generation
• Web Service and Demonstration Applications
  • runApplication
  • SmqtkWebApp
  • Sample Web Applications
• Utilities and Applications
  • Utility Modules
  • Utility Scripts

Plugin Architecture

Each of these main components are housed within distinct sub-modules under smqtk and adhere to a plugin pattern for the dynamic discovery of implementations.

In SMQTK, data structures and algorithms are first defined by an abstract interface class that lays out what that services the data structure, or methods that the algorithm, should provide. This allows users to treat instances of structures and algorithms in a generic way, based on their defined high level functionality, without needing to knowing what specific implementation is running underneath. It lies, of course, to the implementations of these interfaces to provide the concrete functionality.

When creating a new data structure or algorithm interface, the pattern is that each interface is defined inside its own sub-module in the __init__.py file. This file also defines a function get_..._impls() (replacing the ... with the name of the interface) that returns a mapping of implementation class names to the implementation class type, by calling the general helper method smqtk.utils.plugin.get_plugins(). This helper method looks for modules defined parallel to the __init__.py file as well as classes defined in modules listed in an environment variable (defined by the specific call to get_plugins()). The function then extracts classes that extend from the specified interface class as denoted by a helper variable in the discovered module or by searching attributes exposed by the module. See the doc-string of smqtk.utils.plugin.get_plugins() for more information on how plugin modules are discovered.

Adding a new Interface and Internal Implementation

For example, lets say we’re creating a new data representation interface called FooBar. We would create a directory and __init__.py file (python module) to house the interface as follows:

python/
└── smqtk/
    └── representation/
        └── foo_bar/          # new
            └── __init__.py   # new

Since we are making a new data representation interface, our new interface should descend from the smqtk.representation.SmqtkRepresentation interface (algorithm interfaces would descend from smqtk.algorithms.SmqtkAlgorithm). The SmqtkRepresentation base-class descends from the Configurable interface (interface class sets __metaclass__ = abc.ABCMeta, thus it is not set in the example below).

The __init__.py file for our new sub-module might look something like the following, defining a new abstract class:

import abc

from smqtk.representation import SmqtkRepresentation
from smqtk.utils.plugin import Pluggable, get_plugins


class FooBar (SmqtkRepresentation, Pluggable):
    """
    Some documentation on what this does.
    """
    # Interface methods and/or abstract functionality here.
    # -> See the abc module on how to decorate abstract methods.

    @abc.abstractmethod
    def do_something(self):
        """ Does Something """


def get_foo_bar_impls(reload_modules=False):
    import os.path as osp
    from smqtk.utils.plugin import get_plugins
    this_dir = osp.abspath(osp.dirname(__file__))
    env_var = 'FOO_BAR_PATH'
    helper_var = 'FOO_BAR_CLASS'
    return get_plugins(__name__, this_dir, env_var, helper_var, FooBar,
                       reload_modules)

When adding a an implementation class, if it is sufficient to be contained in a single file, a new module can be added like:

python/
└── smqtk/
    └── representation/
        └── foo_bar/
            ├── __init__.py
            └── some_impl.py  # new

Where some_impl.py might look like:

from smqtk.representation.foo_bar import FooBar

class SomeImpl (FooBar):
    """
    Some documentation
    """
    # Implementation of abstract methods here

Implementation classes can also live inside of a nested sub-module. This is useful when an implementation class requires specific or extensive support utilities (for example, see the DescriptorGenerator implementation ColorDescriptor).:

python/
└── smqtk/
    └── representation/
        └── foo_bar/
            ├── __init__.py
            ├── some_impl.py
            └── other_impl/      # new
                └── __init__.py  # new

Where the __init__.py file should at least expose concrete implementation classes that should be exported as attributes for the plugin getter to discover.

Both Pluggable and Configurable

It is important to note that our new interface, as defined above, descends from both the Configurable interface (transitive through the SmqtkRepresentation base-class) and the Pluggable interface.

The Configurable interface allows classes to be instantiated via a dictionary with JSON-compliant data types. In conjunction with the plugin getter function (get_foo_bar_impls in our example above), we are able to select and construct specific implementations of an interface via a configuration or during runtime (e.g. via a transcoded JSON object). With this flexibility, an application can set up a pipeline using the high-level interfaces as reference, allowing specific implementations to be swapped in an out via configuration.

Reload Use Warning

While the smqtk.utils.plugin.get_plugins() function allows for reloading discovered modules for potentially new content, this is not recommended under normal conditions. When reloading a plugin module after pickle serializing an instance of an implementation, deserialization causes an error because the original class type that was pickled is no longer valid as the reloaded module overwrote the previous plugin class type.

Function and Interface Reference

smqtk.utils.plugin.get_plugins(base_module_str, internal_dir, dir_env_var, helper_var, baseclass_type, warn=True, reload_modules=False)

Discover and return classes found in the SMQTK internal plugin directory and any additional directories specified via an environment variable.

In order to specify additional out-of-SMQTK python modules containing base-class implementations, additions to the given environment variable must be made. Entries must be separated by either a ‘;’ (for windows) or ‘:’ (for everything else). This is the same as for the PATH environment variable on your platform. Entries should be paths to importable modules containing attributes for potential import.

When looking at module attributes, we acknowledge those that start with an alphanumeric character (‘_’ prefixed attributes are hidden from import by this function).

We required that the base class that we are checking for also descends from the Pluggable interface defined above. This allows us to check if a loaded class is_usable.

Within a module we first look for a helper variable by the name provided, which can either be a single class object or an iterable of class objects, to be specifically exported. If the variable is set to None, we skip that module and do not import anything. If the variable is not present, we look at attributes defined in that module for classes that descend from the given base class type. If none of the above are found, or if an exception occurs, the module is skipped.

Parameters:

• base_module_str (str) – SMQTK internal string module path in which internal plugin modules are located.
• internal_dir (str) – Directory path to where SMQTK internal plugin modules are located.
• dir_env_var (str) – String name of an environment variable to look for that may optionally define additional directory paths to search for modules that may implement additional child classes of the base type.
• helper_var (str) – Name of the expected module helper attribute.
• baseclass_type (type) – Class type that discovered classes should descend from (inherit from).
• warn (bool) – If we should warn about module import failures.
• reload_modules (bool) – Explicitly reload discovered modules from source instead of taking a potentially cached version of the module.

Returns:

Map of discovered class objects descending from type baseclass_type and smqtk.utils.plugin.Pluggable whose keys are the string names of the class types.

Return type:

dict[str, type]

class smqtk.utils.plugin.Pluggable

Interface for classes that have plugin implementations

classmethod is_usable()

Check whether this class is available for use.

Since certain plugin implementations may require additional dependencies that may not yet be available on the system, this method should check for those dependencies and return a boolean saying if the implementation is usable.

NOTES:

• This should be a class method

• When an implementation is deemed not usable, this should emit a

  warning detailing why the implementation is not available for use.

Returns:Boolean determination of whether this implementation is usable. Return type:bool

class smqtk.utils.configurable_interface.Configurable

Interface for objects that should be configurable via a configuration dictionary consisting of JSON types.

classmethod from_config(config_dict, merge_default=True)

Instantiate a new instance of this class given the configuration JSON-compliant dictionary encapsulating initialization arguments.

This method should not be called via super unless an instance of the class is desired.

Parameters:

• config_dict (dict) – JSON compliant dictionary encapsulating a configuration.
• merge_default (bool) – Merge the given configuration on top of the default provided by get_default_config.

Returns:

Constructed instance from the provided config.

Return type:

Configurable

get_config()

Return a JSON-compliant dictionary that could be passed to this class’s from_config method to produce an instance with identical configuration.

In the common case, this involves naming the keys of the dictionary based on the initialization argument names as if it were to be passed to the constructor via dictionary expansion.

Returns:JSON type compliant configuration dictionary. Return type:dict

classmethod get_default_config()

Generate and return a default configuration dictionary for this class. This will be primarily used for generating what the configuration dictionary would look like for this class without instantiating it.

By default, we observe what this class’s constructor takes as arguments, turning those argument names into configuration dictionary keys. If any of those arguments have defaults, we will add those values into the configuration dictionary appropriately. The dictionary returned should only contain JSON compliant value types.

It is not be guaranteed that the configuration dictionary returned from this method is valid for construction of an instance of this class.

Returns:Default configuration dictionary for the class. Return type:dict

>>> class SimpleConfig(Configurable):
...     def __init__(self, a=1, b='foo'):
...         self.a = a
...         self.b = b
...     def get_config(self):
...         return {'a': self.a, 'b': self.b}
>>> self = SimpleConfig()
>>> config = self.get_default_config()
>>> assert config == {'a': 1, 'b': 'foo'}