Public Member Functions | Public Attributes

PatchList2 Class Reference
[Auxilliary Tools]

Auxilliary Tool: A class that does the necessary book keeping for filtering an image quickly at many scales. More...

#include <PatchList2.h>

Inheritance diagram for PatchList2:
Inheritance graph
[legend]

List of all members.

Public Member Functions

 PatchList2 (cv::Size basePatchSize, cv::Size minSize=cv::Size(0, 0), cv::Size maxSize=cv::Size(0, 0), double scaleInc=1.2, double stepWidth=1, int dontCopyImage=0)
 Constructor.
 PatchList2 ()
 Default Constructor.
 PatchList2 (const PatchList2 &copy)
 Copy Constructor.
PatchList2 & operator= (const PatchList2 &rhs)
 Assignment operator.
virtual ~PatchList2 ()
 Destructor.
cv::Size getBasePatchSize () const
 Get the base patch size, which is size of the object detector that you plan to apply. It is also the default search minSize, and other scale sizes are computed relative to it.
void setBasePatchSize (cv::Size baseSize)
 Set the base patch size, which is size of the object detector that you plan to apply. It is also the default search minSize, and other scale sizes are computed relative to it.
void setImage (const cv::Mat &newImage)
 Set the image to filter or process for object detection.
void resetListToScale (int scale)
 Prepare the data structure to search for objects at a certain scale.
void accumulateAndRemovePatchesBelowThreshold (double threshold)
 Adds the values in destInds to an accumulator. Any accumulator values that are below threshold are removed from the list.
void keepOnlyLocalMaxima (int radius)
 Remove image patch candidates that have a nearby patch (within radius horizontal or vertical pixels) with a higher value.
void getRemainingPatches (std::vector< SearchResult > &searchResults) const
 Get info about the patches that have not yet been removed from the PatchList since calling resetListToScale(). Each SearchResult contains info about the location of the patch in the original image, and its current accumulator value.
void getProbImage (cv::Mat &dest, double faceAreaPrior=.5, double faceInvisibleProb=.1) const
 Compute a pixel-by-pixel estimate for this scale that there is an object located at that pixel.
cv::Size getPatchSizeAtScale (int scale=-1) const
 Get the effective object size if searching at a given (or the current) scale.
cv::Size getFilterSizeAtScale (int scale=-1) const
 Get the filter size that should be applied at this scale. For PatchList, this is always getBasePatchSize(). In PatchList, this always returns getBasePatchSize(). In FastPatchList, this always returns getPatchSizeAtScale(scale).
cv::Size getImageSizeAtScale (int scale=-1) const
 Get the size of the valid convolution image, for viewing filtering and accumulator results.
int getIntegralWidthStepAtScale (int scale=-1) const
 Get the width step of the integral image, i.e. how many integral_type elements are there in srcInds[i][j] before a pixel that is directly below [j].
int getCurrentListLength () const
 Get the number of remaining patches at this scale. Initially, this has all patches in the scale, but may be reduced by repeated calls to accumulateAndRemovePatchesBelowThreshold().
int getTotalPatches () const
 Get the total number of patches available to this PatchList at all scales.
int getNumScales () const
 Get the total number of scales.
std::vector< ImagePatch2 > getNearbyPatches (cv::Rect roi, int spatialRadius=0, int scaleRadius=0) const
 Get patches near a region of interest. This is useful for augmenting the number of positive examples in a dataset by grabbing nearby patches, or for finding the image patch that is closest to a labelled location.
void fillImageWithPixelsOfSearchPatch (cv::Mat &dest, const SearchResult &r) const
 Get the exact same pixels from the image patch that the filtering process sees. This is useful for pulling out hard background patches for training.
void setDontCopyImageData (int flag=0)
 Improve efficiency by only copying integral data (not image data). This makes fillImageWithPixelsOfSearchPatch fail.
void setDontScaleDownPatches (int flag=0)
 Improve detail of patches when getting pixels of patch -- at the cost of extra memory. Only affects FastPatchList.
void getFilterImage (cv::Mat &image) const
 Output of image filtering process as an image which can be visualized.
void getAccumImage (cv::Mat &image) const
 Output of filtering accumulation process as an image which can be visualized.

Public Attributes

std::vector< const int * > srcInds
 A list of pointers to the top-left of image patches still considered as candidate locations of the object. This pointer is exposed to allow BoxFeature to evaluate the image with maximal efficiency.
std::vector< double * > destInds
 A list of pointers to corresponding to the filtering outputs of the remaining candidate patches in srcInds.

Detailed Description

Auxilliary Tool: A class that does the necessary book keeping for filtering an image quickly at many scales.

Specifically, a PatchList gives BoxFeature, FeatureRegressor, and GentleBoostCascadedClassifier a way to apply a single weak learner to all remaining candidate object image patches. GentleBoost requires that each weak learner output be accumulated with the output of other weak learners. PatchList maintains this accumulator.

They key parameters controlling a PatchList are the size of the image (using setImage), the base patch size, and the scale increment factor. Here as an example with an image of size 320x240, a base patch size of 30, and a scale increment of 1.2.

At scale 0, the image is simply copied, and an integral image is computed. It can be filtered in a region up to 29 pixels from the border by a filter of size 30x30 (which is the base patch size). ImageSize represents the size of this valid filtering region, which is 291x211 pixels and is useful for displaying an image representation of the fitlering output. Each of these pixels contains the output of the filtering of one patch, so there are 61401 patches at scale 0.

At scale 1, the size of patches to search increases by a factor of 1.2, giving a patchWidth of 36. Internally, the image is downscaled by a factor of 1.2, giving a 266x200 image, with a 237x171 valid filtering region, and 40527 total patches.

At scale 12, the patch size would be 30*(1.2^12)=267x267. This is larger than the height of the image, and so the largest scale is 11. Since the scales start at 0, there are 12 total scales:

The PatchList has 12 scales containing 170925 total patches.

Other parameters that can be set are maxSize, minSize, and stepWidth. MaxSize changes the maximum searched patch size. If (0,0), then the smaller image dimension is used. MinSize changes the minimum searched patch size. If (0,0), the base patch size is used. Otherwise, the image is scaled in size by a factor of basePatchSize/minSize before searching. StepWidth reduces the number of patches searched by skipping rows and columns. If StepWidth is 1, each patch location is searched. If it is 2, every other one is searched. If it is 2/3, then two consecutive patches are searched and one is skipped.

PatchList is effecicient about not doing more memory allocation than it needs. A rule of thumb is that giving a PatchList images of different sizes will incur a slowdown, but if all images are the same size, then subsequent calls to setImage will be faster.

For example, a typical time for the first call to setImage with a 320x240 image, a base patch size of 30x30, and a scale increment of 1.2 is 15 ms, while subsequent calls with the same image size and parameters takes 1.5 ms.

Author:
Nicholas Butko
Date:
2010 version 0.4

Constructor & Destructor Documentation

PatchList2::PatchList2 ( cv::Size  basePatchSize,
cv::Size  minSize = cv::Size(0, 0),
cv::Size  maxSize = cv::Size(0, 0),
double  scaleInc = 1.2,
double  stepWidth = 1,
int  dontCopyImage = 0 
)

Constructor.

Parameters:
basePatchSizeThe size of the patches that will be evaluated for containing the object. This should match the size of whatever filter will be applied to the patches. It may be set later, using setBasePatchSize(). At subsequent scales, the image is scaled up or down relative to this size.
minSizeDon't search for objects smaller than minSize. If minSize is (0,0) then baseObjectSize is used.
maxSizeDon't search for objects larger than maxSize. If maxSize is (0,0), then the upper bound on the object size is the smallest dimension of the image size.
scaleIncRelative size of the object in the next scale, compared to the current one. scaleInc must be greater than 1.
stepWidthSearch interval between subsequent patches. A stepWidth of 1 means slide the object evaluation window over one pixel. 2 means only evaluate patches at every other pixel. 1.5 means search two and skip the third, etc.
dontCopyImageSet to non-zero if you want efficiency, but not to look at patches.

Member Function Documentation

void PatchList2::accumulateAndRemovePatchesBelowThreshold ( double  threshold )

Adds the values in destInds to an accumulator. Any accumulator values that are below threshold are removed from the list.

Parameters:
thresholdAccumulator values below threshold will be removed from the PatchList. -INFINITY will insure all patches are kept.
void PatchList2::fillImageWithPixelsOfSearchPatch ( cv::Mat &  dest,
const SearchResult &  r 
) const

Get the exact same pixels from the image patch that the filtering process sees. This is useful for pulling out hard background patches for training.

Parameters:
imDestination image for pixels.
rA Patch SearchResult from getRemainingPatches() to query.
void PatchList2::getAccumImage ( cv::Mat &  image ) const

Output of filtering accumulation process as an image which can be visualized.

This image contains that data that was accumulated by successive calls to accumulateAndRemovePatchesBelowThreshold().

Size PatchList2::getBasePatchSize (  ) const

Get the base patch size, which is size of the object detector that you plan to apply. It is also the default search minSize, and other scale sizes are computed relative to it.

Returns:
The base patch size.
int PatchList2::getCurrentListLength (  ) const

Get the number of remaining patches at this scale. Initially, this has all patches in the scale, but may be reduced by repeated calls to accumulateAndRemovePatchesBelowThreshold().

Returns:
Number of patches that have not been removed.
void PatchList2::getFilterImage ( cv::Mat &  image ) const

Output of image filtering process as an image which can be visualized.

This image contains that data that is written in destInds by BoxFeature and FeatureRegressor and GentleBoostCascadedClassifier.

Size PatchList2::getFilterSizeAtScale ( int  scale = -1 ) const

Get the filter size that should be applied at this scale. For PatchList, this is always getBasePatchSize(). In PatchList, this always returns getBasePatchSize(). In FastPatchList, this always returns getPatchSizeAtScale(scale).

You must call setImage() before. When querying scales for information, the maximum valid value is getNumScales()-1, and the minimum is 0.

Parameters:
scaleThe scale to query. If -1, use the current. Otherwise, any valid scale can be queried.
Size PatchList2::getImageSizeAtScale ( int  scale = -1 ) const

Get the size of the valid convolution image, for viewing filtering and accumulator results.

You must call setImage() before because it is impossible to determine the filtered image size without knowing the image size. When querying scales for information, the maximum valid value is getNumScales()-1, and the minimum is 0.

Parameters:
scaleThe scale to query. If -1, use the current. Otherwise, any valid scale can be queried.
int PatchList2::getIntegralWidthStepAtScale ( int  scale = -1 ) const

Get the width step of the integral image, i.e. how many integral_type elements are there in srcInds[i][j] before a pixel that is directly below [j].

You must call setImage() before because it is impossible to determine the size of the integral image without knowing the image size. When querying scales for information, the maximum valid value is getNumScales()-1, and the minimum is 0.

Parameters:
scaleThe scale to query. If -1, use the current. Otherwise, any valid scale can be queried.
vector< ImagePatch2 > PatchList2::getNearbyPatches ( cv::Rect  roi,
int  spatialRadius = 0,
int  scaleRadius = 0 
) const

Get patches near a region of interest. This is useful for augmenting the number of positive examples in a dataset by grabbing nearby patches, or for finding the image patch that is closest to a labelled location.

Parameters:
roiA region of interest in the image.
spatialRadiusTake patches that are slightly left/right/up/down from roi.Note that a radius of 1 gives 9 patches, and a radius of 2 gives 25.
scaleRadiusTake patches that are slightly larger or smaller than the roi. This size is radius is in terms of search scale, not pixels. Note that a radius of 1 gives 3 patches, and a radius of 2 gives 5.

!!!! TODO : Check this out. -- Decide if we want this to be size of base patch or size of patch in image

int PatchList2::getNumScales (  ) const

Get the total number of scales.

You must call setImage() before because it is impossible to determine the number of scales without knowing the image size. When querying scales for information, the maximum valid value is getNumScales()-1, and the minimum is 0.

Returns:
Total number of patches available to this PatchList.
Size PatchList2::getPatchSizeAtScale ( int  scale = -1 ) const

Get the effective object size if searching at a given (or the current) scale.

You must call setImage() before because it is impossible to determine the number of patch sizes without knowing the image size. When querying scales for information, the maximum valid value is getNumScales()-1, and the minimum is 0.

Parameters:
scaleThe scale to query. If -1, use the current. Otherwise, any valid scale can be queried.
void PatchList2::getRemainingPatches ( std::vector< SearchResult > &  searchResults ) const

Get info about the patches that have not yet been removed from the PatchList since calling resetListToScale(). Each SearchResult contains info about the location of the patch in the original image, and its current accumulator value.

Parameters:
searchResultsThe vector that will be filled with the remaining patches. The contents of the vector passed in are entirely overwritten.
int PatchList2::getTotalPatches (  ) const

Get the total number of patches available to this PatchList at all scales.

Returns:
Total number of patches available to this PatchList.
void PatchList2::keepOnlyLocalMaxima ( int  radius )

Remove image patch candidates that have a nearby patch (within radius horizontal or vertical pixels) with a higher value.

Parameters:
radiusRadius (in pixels) to do non-maximal suppression. Pixels are scale-adjusted, so the suppression region grows with the size of the object.
void PatchList2::resetListToScale ( int  scale )

Prepare the data structure to search for objects at a certain scale.

You must call setImage() before because it is impossible to determine various scale info without knowing the image size. When querying scales for information, the maximum valid value is getNumScales()-1, and the minimum is 0.

Scales number from 0 to getNumScales()-1. 0 searches for smallest objects, getNumScales()-1 searches for the largest.

void PatchList2::setBasePatchSize ( cv::Size  baseSize )

Set the base patch size, which is size of the object detector that you plan to apply. It is also the default search minSize, and other scale sizes are computed relative to it.

This can be changed somewhat freely, but setImage() must be called afterward.

Parameters:
baseSizeThe base patch size.
void PatchList2::setDontCopyImageData ( int  flag = 0 )

Improve efficiency by only copying integral data (not image data). This makes fillImageWithPixelsOfSearchPatch fail.

Parameters:
flagSet to non-zero if you want efficiency, but not to look at patches.
void PatchList2::setDontScaleDownPatches ( int  flag = 0 )

Improve detail of patches when getting pixels of patch -- at the cost of extra memory. Only affects FastPatchList.

Parameters:
flagSet to non-zero if you want big patches for training with FastPatchList.
void PatchList2::setImage ( const cv::Mat &  newImage )

Set the image to filter or process for object detection.

Parameters:
newImageThe image to process must be a single channel image of CV_8U. It can be of any size, but passing images of the same size repeatedly will boost performance.

Member Data Documentation

std::vector<double*> PatchList2::destInds

A list of pointers to corresponding to the filtering outputs of the remaining candidate patches in srcInds.

This is a place for BoxFeature, FeatureRegressor, GentleBoostCascadedClassifier etc. to store their outputs.

The documentation for this class was generated from the following files:
Generated on Thu Dec 9 2010 16:05:49 for Nick's Machine Perception Toolbox by  doxygen 1.7.2