Sunday, 4 October 2015

Feature Generation

The background

The next step in the process is to examine the image data and extract features which can be tracked across multiple images.

OpenMVG offers a selection of feature extraction algorithms. Basically the two primary feature extraction methods are:

Both offer some tuning of parameters for feature recognition, and the selection of the feature extraction method has some impact on the later processing stages.

Generate the initial features

To generate the list of features in an image we use the base class openMVG::features::Image_describer.

This provides the base class for a number of implementations which are used to extract features from an image, specifically AKAZE, AKAZE_OCV (The OpenCV AKAZE) and SIFT.
The actual points in the image are stored in a regions object openMVG::features::Regions, which provides a simple mechanism to extract the features into a region, and then serialise it to a file. Each region object represents a set of regions as “2D position” and “attributes”. The region attributes are referred to as Descriptors.

It's actually very simple to use, without getting involved in the details. The basic workflow is (using AKAZE as an example):
  • Create a unique point to an openMVG::features::Image_describer
  • Create a new openMVG::features::AKAZE_Image_describer and assign it to the generic image descriptor pointer
  • Call the image describer Allocate method, passing a reference to a openMVG::features::Regions pointer
  • Then, For each view
    • Create a openMVG::features::Regions to represent a set of regions
    • Read the image data as a greyscale, using openMVG::image::ReadImage()
    • Call the image describer Describe method on the image
    • Store the results to a file using the image describer Save method

This results in a set of feature and description files (“.desc” and “.feat”) for each image. The openMVG::features::Regions pointer reference is used later to pass to functions which need to know how to interpret the feature data.
Note that we must use the .feat and .desc name convention. This is because when we use the library functions to reload this information for later processing it will use this convention to associate the image files with feature and description information.

void ImageList::generateFeatures(void)
std::unique_ptr image_describer;
openMVG::image::Image imageGray;

  image_describer.reset(new openMVG::features::AKAZE_Image_describer(
        openMVG::features::AKAZEParams(openMVG::features::AKAZEConfig(), openMVG::features::AKAZE_MSURF),

  for(openMVG::sfm::Views::const_iterator iterViews = _sfm_data.views.begin();
                                 iterViews != _sfm_data.views.end(); 
  const openMVG::sfm::View * view = iterViews->second.get();
  const std::string sFeat = stlplus::create_filespec(getDirectory(), 
                                                     stlplus::basename_part(view->s_Img_path), "feat");
  const std::string sDesc = stlplus::create_filespec(getDirectory(), 
                                                     stlplus::basename_part(view->s_Img_path), "desc");

    if (!stlplus::file_exists(sFeat))
      openMVG::image::ReadImage(view->s_Img_path.c_str(), &imageGray);
      std::unique_ptr regions;
      image_describer->Describe(imageGray, regions);
      image_describer->Save(regions.get(), sFeat, sDesc);

In working code it's generally worth reloading the feature and descriptor files, rather than going through the lengthy process of generating each time, so the processing should check for the existence of the file in question before recalculating.

(also note the C++ 11 use of “.get()” “.reset()”, etc to reference the smart pointer directly rather than the referenced object)

One wrinkle is the use of openMVG::features::Regions: This type is a list of Region descriptors, used to save, load and manipulate the feature/descriptor lists. This basically falls into one of two types – either Scalar or Binary regions based on the type of the descriptor.

Depending on the type of descriptor chosen then there are some limits on which filters we can apply to the features. So we have:
  • SIFT, AKAZE Float & AKAZE LIOP: Scalar Regions
  • AKAZE Binary (MLDB): Binary Regions

And when we run the matcher (at the next step) we can choose
  • Binary Regions: Brute Force Hamming
  • Scalar Regions: Brute Force & ANN L2 matchers

So when we put together this section of code we must initially select the image description algorithm we use and the configuration, and this may give use choices on the filter implementation we can choose.

More on this later...