Always backup your data

Visual Studio just crashed and gave me the blue screen of death..

My code has become corrupted and restore points won't work. I've looked at the dump report, and it probably has to do with some faulty drivers. I'm installing a service pack for Visual Studio right now, and SVN for future backups. I'll likely spend the rest of the day re-writing the code from scratch.

Edit: Finished re-writing the code and got SVN backup repository running. It took about 4 or 5 hours total.

Experimenting with Edge Detection and Region Identification

After using k=2 for the k-Nearest Neighbor classifier, accuracy improved to nearly 70%.
Here is the confusion matrix:

[10, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0;
  0, 22, 2, 4, 0, 0, 0, 0, 2, 0, 0;
  0, 0, 23, 0, 0, 0, 0, 0, 1, 0, 0;
  0, 7, 0, 15, 0, 0, 0, 0, 0, 0, 0;
  0, 4, 3, 0, 2, 0, 0, 0, 0, 0, 0;
  0, 1, 4, 0, 2, 3, 0, 0, 1, 0, 1;
  0, 5, 0, 1, 0, 0, 4, 0, 0, 1, 0;
  0, 0, 0, 0, 0, 0, 0, 17, 0, 0, 0;
  0, 1, 4, 0, 0, 0, 0, 0, 13, 0, 0;
  0, 6, 0, 3, 0, 0, 1, 0, 0, 13, 0;
  1, 0, 1, 0, 2, 0, 0, 0, 1, 0, 14]
accuracy: 0.697436 (136/195)


So a ~70% classification rate is better than I had anticipated, but we would like our model to be more accurate and robust. One way to do this is to add more features. Today I've been looking at how to use edge and contour detection tools to extract these kinds of features.

Using Canny Edge detection in OpenCV, I was able to extract some decent edge images.
Parameters:
Lower threshold: 50
Upper threshold: 200
I chose these thresholds based on a sample OpenCV program (houghlines.cpp).
They work well right now, but perhaps later I can look into some form of automatic thresholding.

Here are the original images and the edge extracted images:

 

 

 

 

 


As one will notice, the edge detection identifies small details such as hair, light reflection, or wing folds. In order to filter out this "noise," I ran a median blur filter on the original images before performing the edge extraction:
Here is an example of the blurred image in grayscale mode alongside the original.

From here, I used the OpenCV Contours structure to identify polygons or enclosed shapes formed by the edges. Separate contours are labeled with different colors:

Contour image using 3x3 kernel for median blurring

Contour image using 5x5 kernel for median blurring

I believe the ultimate goal of using the contour detection would be:
To be able to identify the different regions of the wasp (head, thorax, abdomen, antennae, wings) and perhaps even the patterns specific to that species of wasp (such as the 4-window pattern on the back of this "Apodynerus f.formosensis"). This could be a simplified to a feature such as Number_of_Regions. One could measure the length-width ratio and the general shape of said regions, and those would be examples of morphological features that are highly descriptive.

Problems with this approach:
As one will notice that even after median blurring, there are still unwanted details picked up by the edge detector. When the blurring kernel is chosen to be too large, we lose the finer details such as the back pattern or legs. When the blurring kernel is chosen to be too small, some regions are closed off, leading to multiple contours where there should only be one (such as the wing).

In addition, I don't know an easy way to remove small, undesirable contours. I thought of thresholding the contours based on pixel count, but that would remove small, desirable details such as the back pattern. I will look into it.

Nearest Neighbor Classification using Color Histograms as features

Was able to get the K-Nearest Neighbor classifier running today and the results are better than I expected.
Skip down to view results.

Took about half a day debugging and trying to understand OpenCV's new structures as most of the example code online uses deprecated OpenCV structures.

Problems encountered:
The machine learning model in OpenCV takes a very specific format for the feature vectors, so it took some time concatenating everything together into one big matrix. Basically it was a pain using the cv::Mat structure because documentation is spread out across their older deprecated structures like cvMat and sample code online uses mostly the older structs.
Set elements:
Mat A;
A.at<type>(i,j) = ...

Set rows:
Mat A;
Mat Ai = A.row(i);
otherRow.copyTo(Ai);

In the future, I would recommend using MATLAB for ANY Matrix handling as it is a hassle in OpenCV, but I don't think I want to spend my time stitching MATLAB and C++ together right now.

Results:
First I split the data set in half into training and testing sets.

Then, I extracted the 2-D histogram features (Hue & Saturation) as documented in my last post, and pass them and the truth labels into the cv::kNearest classifier.

Here are the pictures of a single candidate from each species for reference:

I decided to test the classifier with only the first two classes at first, and it received 100% accuracy!
I used the following parameters:
number of hue bins: 30
number of saturation bins: 32
k for k-nearest: 1 (nearest neighbor)

After adding one class at a time and re-running the classifier, accuracy was still quite high for the first four classes, about ~90%.

Then the accuracy started to drop to about 75% after a few more classes were added, and after adding in all 11 classes it is now at 67% (138/204).
I tried varying the parameters, but it didn't make much difference.

Here is the confusion matrix representing the final classifier results for today:
A confusion matrix shows the number of samples that were predicted to be X, but were actually Y.
Ideally, we want the numbers to be as high across the diagonal of the matrix as possible.
x-axis represents predicted label, and y-axis represents actual label.

       0  1  2  3  4  5  6  7  8  9  10        

0 |  [8, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1;
1 |   0, 26, 0, 1, 0, 0, 2, 0, 1, 1, 0;
2 |   0, 2, 19, 0, 0, 0, 0, 0, 4, 0, 2;
3 |   0, 4, 0, 16, 0, 0, 1, 0, 0, 1, 0;
4 |   0, 4, 4, 0, 0, 0, 1, 0, 0, 0, 0;
5 |   0, 1, 5, 0, 1, 4, 0, 0, 0, 0, 2;
6 |   0, 4, 0, 0, 0, 0, 2, 0, 4, 1, 0;
7 |   0, 0, 0, 0, 0, 0, 0, 17, 0, 0, 0;
8 |   0, 0, 5, 0, 0, 0, 0, 0, 13, 0, 0;
9 |   0, 5, 0, 3, 0, 0, 0, 0, 0, 16, 0;
10|  0, 0, 5, 0, 0, 0, 0, 0, 0, 0, 17]

Take note that classes 4, 5, and 6 had the most number of incorrectly identified samples. Class 4 had no correctly identified samples. At a quick glance, it looks to me as though all their colors are dull, and perhaps the reason why the classifier does a poor job of distinguishing them.

Notes:
Things to try out later:

• vary number of bins for hue and saturation
• vary k value for kNearest classifier
• remove images that contain knife blade such as:
  Since I'm not sure how to segment it out yet.
• Produce graphs of data clusters
• Produce graphs of performance when varying parameters

On the Computer Vision side of things... I think I will need to begin looking into re-orienting the insects so they all face the same direction, edge detection/contour detection, and maybe using hough transforms to detect lines/circles.

Color Histograms as a means for comparison

So I've decided on using the OpenCV 2.2 Library in C++ for the project.

For today's goal I experimented with using color histograms for comparing the images. Color histograms are a good starting point because it is independent of orientation and size of the insects, making it an easy feature to pull out.

Images were first compressed using pyrDown so the image processing could be done more quickly.
Image sizes went from about 2-4 Megabytes down to a few hundred Kilobytes.

First converted the image to HSV format (for richer color representation, as opposed to RGB histograms), and used a rough binary image as the mask to disregard the background when using calcHist.
HSV image on left, binary image on right.

For compareHist, I chose Chi Squared distance (somewhat arbitrarily for now) and then I ran a quick dummy test on 1 train image, and 4 test images:

http://opencv.willowgarage.com/documentation/cpp/imgproc_histograms.html?highlight=calchist#calcHist

The true positive image produced a distance of about 30000, while the other images were higher than 80000.

train.jpg

Test images:

distance: 109642

distance: 30000.1

distance: 86790.7

distance: 181502

• Of course, when putting in all the other images, this approach might not be as successful, considering I only included species that seemed significantly distinct in color. 
• I spent a lot of time today trying to understand/use the k-nearest neighbor classifier in openCV. The openCV model allows you to easily train a set of images, then predict the class of a test image:
  http://opencv.willowgarage.com/documentation/cpp/ml_k_nearest_neighbors.html
  K-nearest neighbor is both effective and simple, which is why I have chosen it as a starting point: http://en.wikipedia.org/wiki/K-nearest_neighbor_algorithm
• I spent the rest of my time figuring out how to load all the image data and class labels into OpenCV Mat structures. I wasn't able to find an easy way navigate directories in C++ without having to download some fancy library, so I wrote a python script to write the absolute paths of the image files, along with their ground truth class label (parent directory) to an output file for easy redirection to C++. 
• The image reading takes a long time, so it might be better to compress the images as opposed to compressing them during runtime.
• The Knn classifier expects row vectors for features, but the histograms are N-dimensional matrices, so I will try reshaping the matrices into vectors before feeding them to the classifier.

Defining the Project

Project Description:

I have about ten weeks to build a model that will recognize different species of wasps based on overhead images of the wasps. The idea is to train the model on a set of images, and then test the model on another set of images and rate its performance (ie. correctly identified / total number of examples). Here are two examples of the kinds of images I will be dealing with.





Project Location:
I will be working at the Taiwan Forestry Research Institute (TFRI) in Taipei, Taiwan for the summer.

Background:
4th year, Computer Engineering Major at UCSD expected B.S. in June 2012.
Relevant Coursework:
CSE 150: Introduction to Artificial Intelligence - received an A
CSE 152: Introduction to Computer vision - received an A
Independent study on Machine Learning under Dr. Tony Fountain

Goals:
I have a decent grasp on the theoretical concepts of machine learning and computer vision, but I have had little hands-on practice and do not understand how to apply the what I learned to real-world problems, so I believe that this will be both a difficult and rewarding project.
My goal for the summer is to:
- have a better understanding of machine learning and computer vision
- understand how to apply the methodologies to real-world applications
- understand the relevance and importance of these kinds of applications
- decide my focus for graduate school
- eat a lot and have fun!

Project Mentors:
UCSD:
Dr. Tony Fountain

TFRI:
Dr. Chau Chin Lin
Dr. Sheng-Shan Lu
Dr. Yu-Huang Wang

Also, here is a link to my travel blog while I am in Taiwan.
http://adrian-prime2011.blogspot.com/