The evaluation consists of testing the feature performance within an image classification system that was deployed at PASCAL 2008 Challenge, namely SurreyUvA_SRKDA.
A brief description of the system is given below and the details can be found in:
J. Zhang and M. Marszalek and S. Lazebnik and C. Schmid,
Local Features and Kernels for Classification of Texture and Object Categories: A Comprehensive Study, International Journal of Computer Vision, 73(2):213-238, 2007.
[paper pdf]
[techreport pdf]
K.E.A. van de Sande and T. Gevers and C.G.M. Snoek,
Evaluation of Color Descriptors for Object and Scene Recognition, IEEE International Conference on Computer Vision and Pattern Recognition, 2008.
[pdf]
1. Training
The training is performed on the image data from PASCAL VOC 2007.
The descriptors are clustered using kmeans algorithm into 4000 clusters.
Each image is represented by a histogram of cluster occurrences. This is a 4000
bin histogram where the bins correspond to the clusters. Each bin contains the number of descriptors in the image that fall into the cluster corresponding to that bin.
Euclidean similarity measure is used to compare descriptors but other metrics can also be used if required.
Chi-square distance and generalized RBF kernel is used for kernel construction:
K(x_i, x_j) = exp(-rho*d(x_i, x_j))
where d(x_i, x_j) is the Chi-square distance between two histograms x_i and x_j.
2. Testing
The test features are matched to the clusters and the occurrence histogram is produced.
Classification is performed using SRKDA and SVM kernel function with Chi-squared distance measure.
3. Feature Computation
Features should be provided in the format specified below:
#comments: x y scale orientation ...descriptor
k
n
m
p1_1 ... p1_k d1_1 d1_2 d1_3 ... d1_n
:
:
pm_1 ... pm_k dm_1 dm_2 dm_3 ... dm_n
The file contains:
- a comment line (starting with the character #) in which the parameters can be named i.e. x y scale ...
- k,n,m indicate the number of features (lines), parameters, and dimensions
- k — number of feature parameters [max 10]
- n — number of descriptor dimensions [max 500]
- m — number of feature vectors in the file [max 5000]
- p and d can be float point values
The parameters are optional since only descriptors will be used for evaluation.
An example output file example.surrey is provided for further clarification.
The features should be computed for all images from PASCAL 2007 VOC dataset:
- 5011 Training/validation images [tar]
- 4952 test images [tar]
Please adopt naming convention: image_name.id (id as an abbreviation of your institution).
Example steps to prepare the data:
- Download the data
- Compute features for every image and save it in image_name.id. This should result in 5011+4952 plain text files.
- The number of parameters per feature (k) must not exceed 10
- The number of dimensions per descriptor (n) must not exceed 500
- The number of descriptors per image (m) must not exceed 5000
If you wish to evaluate interest point detector only, please use code from here
to compute descriptors (preferably sift) for your interest points and then submit.
If you wish to evaluate descriptor only, please use code from here
to detect interest points (preferably hessian laplace) and then submit.
-
Compress with tar.gz or zip all files into one (the size of this file must not exceed 5GB)
-
Contact Mark Barnard (
) for details how to upload the data.