SMQTK v0.13.0 Release Notes¶
This release incorporates updates and fixes performed on the VIGILANT project and approved for public release (case number 88ABW-2019-5287). Some of the major updates and fixes in this include:
- Object detection algorithm interface and supporting DetectionElement interface and implementations.
- Revised plugin implementation accessor via the mixin class instead what used
to be manually implemented side-car functions for every interface. Also moved
some configuration specific functions out of the plugin utility module and
into a configuration utility submodule, where the
Configurable
mixin class has also moved to. - Moves unit tests out of the installed SMQTK package and into a dedicated sub-directory in the repository.
Updates / New Features¶
Algorithms
- Added
ImageReader
algorithm interface- Added matrix reading short-cut if DataElement instance provided has a
matrix
attribute/property. - Added PIL (pillow) implementation with tests.
- Added GDAL implementation with tests.
- Added matrix reading short-cut if DataElement instance provided has a
- Descriptor Generators
- Change
CaffeDescriptorGenerator
constructor to takeDataElement
instances rather than URIs.
- Change
- HashIndex
- SkLearnBallTreeHashIndex
- Fixed numpy load call to explicitly allow loading pickled components due to a parameter default change in numpy version 1.16.3.
- SkLearnBallTreeHashIndex
- Object Detection
- Added initial abstract interface.
- Added “ImageMatrixObjectDetector” interface for object detectors that specifically operate on image data and standardizes the use of an “ImageReader” algorithm to provide the pixel matrix as input.
- Nearest Neighbors
- FAISS
- Gracefully handle addition of duplicated descriptors to avoid making index unusable due to an unexpected external failure.
- Make use of new
get_many
method of key-value stores to improve runtime performance. - Make use of new
get_many_vectors
classmethod of DescriptorElement to improve runtime performance.
- LSH Hash Functor
- Use
ProgressReporter
in itq to avoid bugs from deprecatedreport_progress
function
- Use
- FAISS
Compute Functions
- Add
compute_transformed_descriptors
function tocompute_functions.py
for conducting searches with augmented copies of an image
Misc.
- Updated numpy version in requirements.txt to current versions. Also split versioning between python 2 and 3 due to split availability.
- Resolve python static analysis warnings and errors.
Representation
- Added
AxisAlignedBoundingBox
class for describing N-dimensional euclidean spatial regions. - Added
DetectionElement
interface, and in-memory implementation, with associated unit tests. - Added
DetectionElementFactory
class for factory construction ofDetectionElement
instances. - Add use of
smqtk.utils.configuration.cls_conf_from_config_dict
andsmqtk.utils.configuration.cls_conf_to_config_dict
to appropriate methods in factory classes. - Add
get_many
method toKeyValueStore
interface class and provide an optimized implementation of it for thePostgresKeyValueStore
implementation class. - Add
get_many_vectors
classmethod for efficiently retrieving vectors from several descriptor elements at once - Add efficient implementation of
_get_many_vectors
for Postgres descriptor elements. - Updated
MemoryKeyValueStore.add_many
to usedict.update
method instead of manually updating keys. - Removed unnecessary method override in
DataFileElement
. - Added
MatrixDataElement
representation that stores anumpy.ndarray
instance internally, generating bytes on-the-fly when requested. DataMemoryElement
now raises a TypeError if a non-bytes-line object is passed during construction or setting of bytes. Configuration mixin hooks have been updated to convert to and from strings for JSON-compliant dictionary input and output. Fixed various usages of DataMemoryElement to actually pass bytes.
Tests
- Moved tests out of main package tree.
- Added use of
pytest-runner
insetup.py
, removingrun_tests.sh
script. New method of running tests ispython setup.py test
.
Utilities
- Added to
Pluggable
interface theget_impls
method, replacing the separateget_*_impls
functions defined for each interface type. Removed previousget_*_impls
functions from algorithm and representation interfaces, adjusting tests and utilities as appropriate. - Renamed
smqtk.utils.configurable
tosmqtk.utils.configuration
. Ramifications fixed throughout the codebase. Added documentation to doc-strings. - Added
cls_conf_from_config_dict
andcls_conf_to_config_dict
intermediate helper functions tosmqtk.utils.configuration
for thefrom_config_dict
andto_config_dict
sub-problems, respectively. This was motivated by duplicated functionality in element factory classfrom_config
andget_config
methods. - Moved some helper functions from
smqtk.utils.plugin``to ``smqtk.utils.configuration
as those functions more specifically had to do with configuration dictionary construction and manipulation. Ramifications fixed throughout the codebase. - Updated
smqtk.utils.plugin.get_plugins
signature and return. Now more simply takes the interface class (previously referred to as the base-class) instead of the original first two positional, string arguments as they could be easily introspected from the interface class object. Ramifications fixed throughout the codebase. - Added
ContentTypeValidator
interface for algorithms that operate on rawDataElement
instances, providing methods for validating reported content types against a sub-class defined set of “valid” types. Applied toDescriptorGenerator
interface. - Replace usage of
smqtk.utils.bin_utils.report_progress
with theProgressReporter
class throughout package. - Removed bundled “jsmin” in favor of using pip installed package.
- Moved
merge_dict
out ofsmqtk/utils/__init__.py
and into its own module. - Created
combinatorics
utils module, movedncr
function to here. - Renamed various utility modules that included
_utils
in their name to not include_utils
for the sake of reducing redundancy. - Removed
FileModificationMonitor
utility class due to having no current use anywhere as well as its tests non-deterministically failing (issues with timing and probably lack of sufficient use of mock, time to fix not worth its lack of use). Thewatchdog
python package should be used instead. - Added entry-point extension method of plugin discovery.
- Added warning to
smqtk.utils.file.safe_file_write
when used on Windows platforms.
Fixes¶
Algorithms
- Nearest Neighbors
- FAISS
- Fix issue with storing and retrieving index IDs as numpy types by casting to python native integers due to an incompatibility with some KeyValueStore implementations (specificially an issue with the PostgreSQL implementation).
- FAISS
Misc.
- Removed some unused imports.
Representation
- Fixed bug with
ClassificationElement.max_label
where an exception would be raised if there was no label with associated confidence greater than 0. - Fix some postgres test comparisons due to not being able to
byte
case Binary instances in python 3. Instead using thegetquoted
conversion for the sake of actual/expected comparisons.
Tests
- Moved
--cov
options from pytest.ini file into the runner script. This fixes debugger break-pointing in some IDEs (e.g. PyCharm). - Fix various minor testing errors.
Utilities
- Fix
ZeroDivisionError
insmqtk.utils.bin_utils.report_progress
. Also added deprecation warning to this function.