Customizing the CN Tracker

Goal

In this tutorial you will learn how to

• Set custom parameters for CN tracker.
• Use your own feature-extractor function for the CN tracker.

This document contains tutorial for the cv::TrackerKCF.

Source Code

#include <opencv2/core/utility.hpp>
#include <opencv2/tracking.hpp>
#include <opencv2/videoio.hpp>
#include <opencv2/highgui.hpp>
#include <iostream>
#include <cstring>
#include "samples_utility.hpp"
 
using namespace std;
using namespace cv;
 
// prototype of the functino for feature extractor
void sobelExtractor(const Mat img, const Rect roi, Mat& feat);
 
int main( int argc, char** argv ){
  // show help
  if(argc<2){
    cout<<
      " Usage: tracker <video_name>\n"
      " examples:\n"
      " example_tracking_kcf Bolt/img/%04d.jpg\n"
      " example_tracking_kcf faceocc2.webm\n"
      << endl;
    return 0;
  }
 
  // declares all required variables
  Rect roi;
  Mat frame;
 
  TrackerKCF::Params param;
  param.desc_pca = TrackerKCF::GRAY | TrackerKCF::CN;
  param.desc_npca = 0;
  param.compress_feature = true;
  param.compressed_size = 2;
 
  // create a tracker object
  Ptr<TrackerKCF> tracker = TrackerKCF::create(param);
 
  tracker->setFeatureExtractor(sobelExtractor);
 
  // set input video
  std::string video = argv[1];
  VideoCapture cap(video);
 
  // get bounding box
  cap >> frame;
  roi=selectROI("tracker",frame);
 
  //quit if ROI was not selected
  if(roi.width==0 || roi.height==0)
    return 0;
 
  // initialize the tracker
  tracker->init(frame,roi);
 
  // perform the tracking process
  printf("Start the tracking process, press ESC to quit.\n");
  for ( ;; ){
    // get frame from the video
    cap >> frame;
 
    // stop the program if no more images
    if(frame.rows==0 || frame.cols==0)
      break;
 
    // update the tracking result
    tracker->update(frame,roi);
 
    // draw the tracked object
    rectangle( frame, roi, Scalar( 255, 0, 0 ), 2, 1 );
 
    // show image with the tracked object
    imshow("tracker",frame);
 
    //quit on ESC button
    if(waitKey(1)==27)break;
  }
 
  return 0;
}
 
void sobelExtractor(const Mat img, const Rect roi, Mat& feat){
    Mat sobel[2];
    Mat patch;
    Rect region=roi;
 
    // extract patch inside the image
    if(roi.x<0){region.x=0;region.width+=roi.x;}
    if(roi.y<0){region.y=0;region.height+=roi.y;}
    if(roi.x+roi.width>img.cols)region.width=img.cols-roi.x;
    if(roi.y+roi.height>img.rows)region.height=img.rows-roi.y;
    if(region.width>img.cols)region.width=img.cols;
    if(region.height>img.rows)region.height=img.rows;
 
    patch=img(region).clone();
    cvtColor(patch,patch, COLOR_BGR2GRAY);
 
    // add some padding to compensate when the patch is outside image border
    int addTop,addBottom, addLeft, addRight;
    addTop=region.y-roi.y;
    addBottom=(roi.height+roi.y>img.rows?roi.height+roi.y-img.rows:0);
    addLeft=region.x-roi.x;
    addRight=(roi.width+roi.x>img.cols?roi.width+roi.x-img.cols:0);
 
    copyMakeBorder(patch,patch,addTop,addBottom,addLeft,addRight,BORDER_REPLICATE);
 
    Sobel(patch, sobel[0], CV_32F,1,0,1);
    Sobel(patch, sobel[1], CV_32F,0,1,1);
 
    merge(sobel,2,feat);
 
    feat=feat/255.0-0.5; // normalize to range -0.5 .. 0.5
}

Explanation

This part explains how to set custom parameters and use your own feature-extractor function for the CN tracker. If you need a more detailed information to use cv::Tracker, please refer to Introduction to OpenCV Tracker.

1. Set Custom Parameters

     TrackerKCF::Params param;
     param.desc_pca = TrackerKCF::GRAY | TrackerKCF::CN;
     param.desc_npca = 0;
     param.compress_feature = true;
     param.compressed_size = 2;

   To set custom paramters, an object should be created. Each tracker algorithm has their own parameter format. So, in this case we should use parameter from cv::TrackerKCF since we are interested in modifying the parameter of this tracker algorithm.

   There are several parameters that can be configured as explained in cv::TrackerKCF::Params. For this tutorial, we focussed on the feature extractor functions.

   Several feature types can be used in cv::TrackerKCF. In this case, the grayscale value (1 dimension) and color-names features (10 dimension), will be merged as 11 dimension feature and then compressed into 2 dimension as specified in the code.

   If you want to use another type of pre-defined feature-extractor function, you can check in cv::TrackerKCF::MODE. We will leave the non-compressed feature as 0 since we want to use a customized function.

2. Using a custom function

   You can define your own feature-extractor function for the CN tracker. However, you need to take care about several things:

   • The extracted feature should have the same size as the size of the given bounding box (width and height). For the number of channels you can check the limitation in cv::Mat.
   • You can only use features that can be compared using Euclidean distance. Features like local binary pattern (LBP) may not be suitable since it should be compared using Hamming distance.

   Since the size of the extracted feature should be in the same size with the given bounding box, we need to take care whenever the given bounding box is partially out of range. In this case, we can copy part of image contained in the bounding box as shown in the snippet below.

       // extract patch inside the image
       if(roi.x<0){region.x=0;region.width+=roi.x;}
       if(roi.y<0){region.y=0;region.height+=roi.y;}
       if(roi.x+roi.width>img.cols)region.width=img.cols-roi.x;
       if(roi.y+roi.height>img.rows)region.height=img.rows-roi.y;
       if(region.width>img.cols)region.width=img.cols;
       if(region.height>img.rows)region.height=img.rows;

   Whenever the copied image is smaller than the given bounding box, padding should be given to the sides where the bounding box is partially out of frame.

       // add some padding to compensate when the patch is outside image border
       int addTop,addBottom, addLeft, addRight;
       addTop=region.y-roi.y;
       addBottom=(roi.height+roi.y>img.rows?roi.height+roi.y-img.rows:0);
       addLeft=region.x-roi.x;
       addRight=(roi.width+roi.x>img.cols?roi.width+roi.x-img.cols:0);
    
       copyMakeBorder(patch,patch,addTop,addBottom,addLeft,addRight,BORDER_REPLICATE);

3. Defining the feature

   In this tutorial, the extracted feature is response of the Sobel filter in x and y direction. Those Sobel filter responses are concatenated, resulting a feature with 2 channels.

       Sobel(patch, sobel[0], CV_32F,1,0,1);
       Sobel(patch, sobel[1], CV_32F,0,1,1);
    
       merge(sobel,2,feat);

4. Post processing

   Make sure to normalize the feature with range -0.5 to 0.5

       feat=feat/255.0-0.5; // normalize to range -0.5 .. 0.5