SMQTK v0.2 Release Notes

This is a minor release if SMQTK that provides both new functionality and fixes over the previous version v0.1.

The highlights of this release are new and updated interface classes, an updated plugin system, new HBase and PostgreSQL DataElement implementations, and a new wrapper for Caffe CNN descriptor extraction.

Additional one-off scripts were added for reference as well as a more generally usable utility for listing out available plugins for the running system and environment.

Additional notes about the release are provided below.

Updates / New Features since v0.1

General

  • Added SmqtkObject, SmqtkAlgorithm and SmqtkRepresentation interfaces for high level classification of sub-classes and encapsulation if high level general functionality (like logging).

  • Removed GENIE and MASIR archive directories. There is a tag for the last hash where they were present in this repository. Not removed from history so cloning the SMQTK repo is still large.

  • Removed geospace web application sub-module (moved elsewhere).

Documentation

  • Update documentaiton to reStructured text files and added support for building Sphinx documentation pages.

Plugins

  • Added Pluggable interface, intended for abstract classes whose implementations are expected to be provided via dynamic plugins, and propagated its use within the code base.

  • get_plugins function now ensures that loaded classes descend from Pluggable and check that they are currently usable.

Data Elements

  • Added HBase backend.

  • Added PostgreSQL backend.

  • Added asynchronous conversion of an iterable of DataElement instances into a numpy matrix. Supports multiprocessing and threading approaches.

Data Sets

  • Added default implementation of contains method to abstract interface.

  • Separated out original DataFileSet into separate file-based and in-memory implementations.

  • Added file caching of memory-based data sets.

Descriptor Generators

  • Expanded construction parameters for ColorDescriptor implementations so as to remove most class-level variables.

  • Added CaffeDefaultImageNet implementation and support files. This is intended to be used with the cnn_feature_extractor binary optionally built with SMQTK.

Nearest Neighbors

  • Removed model FLANN implementation model filepath defaults, allowing purely in-memory use without model persistance.

Web Tools

  • Added static file hosting flask blueprint in the IQR demo for serving arbitrary directories as a source of static files. Removed need to write generated files into source tree in order to host them.

  • Fixed base flask app interface to be Pluggable.

Python Utilities

  • Shifted some functions around into locations where it makes more sense for them to live

    • smqtk.utils.safe_create_dir -> smqtk.utils.file_utils.safe_create_dir

    • smqtk.utils.touch -> smqtk.utils.file_utils.touch

Tools / Scripts

  • Added plugin summarization script for listing names and description of currently available plugins for the various SMQTK interfaces.

  • Changed IQR model generation example script to use the same configuration file that would be passed to the IQR web app (simplification).

  • Added machine specific ITQ code generation scripts

Fixes since v0.1

IQR web application demo

  • Fixed preview cache to clean up after itself.

Code Index

  • Fixed the way MemoryCodeIndex updated descriptor count so as not to count descriptor overwrites as new descriptors.

Descriptor Generators

  • Fixed ColorDescriptor implementation use of pyflann.FLANN.nn_index when the distance method is “hik” (inverted results order and distance values).

  • Fixed ColorDescriptor is_usable check to catch stdout/stderr output.

Nearest Neighbors

  • Fixed issue with FLANN implementation where containing directories for output files were not being created first.

Relevancy Index

  • Fixed bug in LibSvmHikRelevancyIndex where negative distance values would cause an error.

IQR Utils

  • Fixed incorrect default RelevancyIndex configuration.

Tests

  • Fixed tests due to DataSet implementation split

Tools / Scripts

  • Fixed various bugs in compute scripts

Miscellaneous

  • Removed various unnecessary print statements originally for debugging.

  • Removed redundant uses of metaclass declarations.