For my problem, the parameter I was trying to pick was the cost-factor -j. In general, the principled way of doing this is as follows: Perform 10-fold (or n-fold) cross validation for each value of the cost-factor, on the training dataset. Get the average accuracy, precision, and recall for each set of models (where the set of models is the n-folds). Pick the value of the cost that gives the best cross validation results. Now the best results are determined by your requirements, and whether you are willing to trade off some recall for some additional precision etc. With the selected value of the parameter (cost-factor in this case), train a new model on the entire training dataset.

Now the basic idea behind n-fold cross validation is that the training dataset is divided into n-folds. In case of stratified cross validation (which is what NetSVMLight does), each fold contains approximately the same proportion of positive and negative class labels as the entire dataset. Also, each feature vector is randomly assigned to one of the n folds. This ensures a fair distribution. Now using a pre-determined value for all parameters, a model is constructed using n-1 of the n folds. This model is then tested on the remaining unseen fold, which yields a value for precision, recall and accuracy. This process is then repeated n-1 times. Each fold is used once for testing. The results of each fold are averaged and reported as the results for the n-fold cross validation.

Hence, in order to pick a parameter, such an n-fold cross validation is performed for each value of the parameter to be chosen. The value of the parameter that gives the most desired cross validation results is picked.

In the code above, the ConstructNFolds method first constructs each of the n-folds on the disk. In the first for-loop, ten different values for the parameter are tried (you can try as many as you wish). The results of each cross validation set are stored in a dictionary, along with the corresponding value of the parameter being tested. The foreach loop simply goes over the dictionary, and saves the results to a file and prints them to the console.

For my dataset, I tried 10 different values of the cost-parameter and recorded the corresponding precision, recall and accuracy from each run of the 10-fold cross validation.

At cost = 0.75, I think there is a reasonable tradeoff between precision and recall. Hence I pick this value and train my SVM over the entire training dataset.

The latest source and binaries can be downloaded here or checked out using SVN.

There are a bunch of extensions and additions listed on the website that provide an interface to SVMlight. Most of these (atleast the ones in python) simply provide an interface to set parameters and run the executables from within the code. Atleast one of the Python extensions did not have the ability to support different kernels. I was finding it hard to understand some of them due to insufficient documentation. Then I thought of writing one in C# simply because I did not find one featured on the SVMlight page. Here is a list of features:

• Well documented
• Strongly typed parameters for svm_learn and svm_classify
• Supports all kernels provided by svmlight
• Supports most commonly used parameters (including kernel params and cross validation)
• Can provide support for others parameters upon request
• Constructs training and test sets from a given dataset and percentage split
• Creates a list of all misclassified instances for further analysis

Here is an example usage:

You can download the source, binaries and documentation from:

http://code.google.com/p/net-svmlight/

It was my birthday yesterday. I was overwhelmed by the number of friends who wished me on Facebook, and I really wanted to thank each one of them by commenting on each “Happy Birthday” message. But when the number of such wishes runs into an unmanageable number, then you know that manually sitting and typing “thank you” (leave alone a more personalized thank you) seems to be quite daunting. Now, I declare a disclaimer that I really do thank all my friends for their good wishes, but I could not resist the idea of automating the process of writing thank you messages in response to each birthday wish.

So I set out with the assumption, that for each birthday message on my wall, I will create a comment carrying a personalized thank you, with my friend’s first name. This seemed like a worthwhile thing to do, since I would be able to individually thank each of my friends; and considering that the only other alternative was to be lazy and do nothing at all (since I was not going to be able to do this task manually).

So I set out to hack a python script using the Facebook Graph API. A few things to be considered:

1. When you retrieve items from your wall, facebook usually returns multiple pages of data. After playing around with this for a while, I did not want to play with it, simply because it was increasing the complexity of what I wanted to do, and I was plainly finding it difficult to understand the meaning of their paging parameters.
2. So I determined that starting from the newest post on my wall, there were not more than a hundred posts that were birthday wishes (some of them were not, and would need to be filtered away).
3. I decided to identify the birthday wishes using the criterion that such a message would contain the word “happy” in it (simply because there were different ways in which people wrote birthday e.g. bday, budday and various others). This may not be a very smart filtering mechanism, but it seemed to have pretty good precision for me, and I was willing to let go of the minor loss of recall (I would deal with these exceptional cases manually).
4. So here is the basic algorithm:
   • Get the list of the 100 most recent posts on my wall
   • If the post does not contain the word “happy”, or already has a comment on it, or doesn’t contain a “message” section in the JSON response, then discard it.
   • Else, extract the messageID, and make an additional HTTP request using the ID of the message sender to determine the sender’s first name
   • Make a new HTTP post to https://graph.facebook.com/messageID/comments using the appropriate access token to write the comment.
5. One final note about access tokens: The list of wall posts could be obtained using the access token described in my previous post about the Facebook Graph API, but in order to be able to post on my wall, I needed another access_token that would contain write permissions. So, I had to implement the OAuth authentication workflow and request the publish_stream permission. This procedure is not documented very well on http://developers.facebook.com/docs/authentication/.
   • Essentially, I made a request to

     https://www.facebook.com/dialog/oauth?client_id=YOUR_APP_ID&redirect_uri=http://www.facebook.com/connect/login_success.html&scope=publish_stream

     This made my application request me for write permission to my wall. Since, I am the app creator, I authorized it.

   • This returned a code in the redirect URL, which I used to make another HTTP request to

     https://graph.facebook.com/oauth/access_token?client_id=YOUR_APP_ID&redirect_uri=http://www.facebook.com/connect/login_success.html&client_secret=YOUR_CLIENT_SECRET&code=CODE_FROM_ABOVE

   • The response contains a new access token that can be used for write operations.

Armed with this new access token, and the old one I already have, I could now read posts on my wall, and write comments to them. Here is the code in Python:

One final thing: there were at-least two posts on my wall, that were completely missed by the Graph API, and no matter what I did, they were just not retrieved. I had to respond to these manually.

Today, I tried using the RestFB library to extract information about my friends from facebook. Now, there is a different kind of a hack that I had to use in order to get an access token: Login to facebook and go to this page: http://developers.facebook.com/docs/reference/api/user/. Now, click on the example link on that page. In your browser address bar, you will see an access token. Save this in your configuration file (this access token expires after a certain duration).

There are a few further to jump through to simply access data. The RestFB library gives you a list of your friend names and ids. Even though it has methods to access other information such as location, gender etc. of your friends, all of these method calls return null UNLESS each of your friends gives your “application” specific permissions to access these fields. Crazy ain’t it? Look at this thread on stackoverflow. So much distress to simply programmatically access information of my friends that I already can see in my facebook account.

So I had to take a different approach: Calling the graph API directly using direct HTTP GET requests as myself, rather than go through the application that needs to be specifically approved by my 350+ friends on FB.

Since I already have my friends’ user IDs (from the previous call using restFB), I made individual HTTP requests to the following URL:

https://graph.facebook.com/id?access_token=abcd (where abcd is the access token I received using the method described above). In the HTTP response body, you receive a JSON string which represents your friend’s information.

One final thing: making a GET request to an https link is painful in java. I used a crude hack in order to do this (essentially install an all trusting trust manager…never do this). Here is the complete code:

I am writing a simple console application that needs to collect some data and do some social network analysis. The complicated browser based authentication workflow supported by both FB and Twitter are a huge hindrance to this purpose, since they assume that you are making a web application with several users. Essentially the process is that your application exchanges your API key and some other secret key for an auth token (specific to each user), and this is done using the browser.

I discovered after a lot of digging in that, Twitter decided to be a little considerate by providing a single user auth_token that can be used for single-user scenarios like mine. So, I can simply start using my secret auth_token and completely bypass the OAuth workflow. To obtain your access token, login at dev.twitter.com, select your app, and click on “My Access Token”.

Given below is an example. I hope this is useful for people who have requirements similar to mine. I will follow up with a post about how to get this working for Facebook.

Intuition and Design

Here is definition of the classifier functionality (from wikipedia):

Now this means, that for each possible class label, multiply together the conditional probability of each feature, given the class label. This means, for us to implement the classifier, all we need to do, is compute these individual conditional probabilities for each label, for each feature, p(Fi | Cj), and multiply them together with the prior probability for that label p(Cj). The label for which we get the largest product, is the label returned by the classifier.

In order to compute these individual conditional probabilities, we use the Maximum Likelihood Estimation method. In a very short sentence, we approximate these probabilities using the counts from the input/training vectors.

Hence we have: p(Fi | Cj) = count( Fi ^ Cj) / count(Cj)

That is, we count from the training corpus, the ratio of the number of occurrences of the feature Fi and the label Cj together to the total number of occurrences of the label Cj.

Zero Probability Problem

What if we have never seen a particular feature Fa and a particular label Cb together in the training dataset? Whenever they occur in the test data, p(Fa | Cb) will be zero. Hence the overall product will also be zero. This is a problem with maximum likelihood estimates. Just because a particular observation was not made during training does not mean that it will never occur in the test data. In order to remedy this issue, we use what is known as smoothing. The simplest kind of smoothing that we use in this code, is called “add one smoothing”. Essentially, the probability for an unseen event should be greater than one. We achieve this by adding one to each zero count. The net effect should be that we redistribute some of the probability mass from the non-zero count observations to the zero-count observations. Hence, we also need to increase the total count for each label by the number of possible observations, in order to maintain the total probability mass at 1.

e.g. if we have two classes C = 0 and C = 1, then after smoothing, the smoothed MLE probabilities can be written as:

p-smoothed(Fi | Cj) = [count(Fi ^ Cj) + 1]/[count(Cj) + N] where N is the total number of observations across all features in the training corpus.

Code

For simplicity, we will use Weka’s ARFF file format as input. We have a single class called Model which has a few dictionaries and lists to store the counts and feature vector details. In this implementation, we only deal with discrete valued features.

The dictionary ‘features’ saves all possible values for a feature. ‘featureNameList‘ is simply a list that contains the names of the features in the same order that it appears in the ARFF file. This is because our features dictionary does not have any intrinsic order, and we need to maintain feature order explicitly. ‘featureCounts‘ contains the actual counts for co-occurrence of each feature value with each label value. The keys for this dictionary are tuples of the form (class_label, feature_name, feature_value). Hence, if we have observed the feature F1 with the value ‘x’ for the label ‘yes’, fifteen times, then we will have the entry {(‘yes’, ‘F1′, 15)} in the dictionary. Note how the default values for counts in this dictionary is ’1′ instead of ’0′. This is because we are smoothing the counts. The ‘featureVectors‘ list actually contains all the input feature vectors from the ARFF file. The last feature in this vector is the class label itself, as is the convention with weka ARFF files. Finally, ‘labelCounts‘ stores the counts of the class labels themselves, i.e. now many times did we see the label Ci during training.
We also have the following member functions in the Model class:

The above method simply reads the feature names (including class labels), their possible values, and the feature vectors themselves; and populate the appropriate data structures defined above.

The TrainClassifier method simply counts the number of co-occurrences of each feature value with each class label, and stores them in the form of 3-tuples. These counts are automatically smoothed by using add-one smoothing as the default value of count for this dictionary is ’1′. The counts of the labels is also adjusted by incrementing these counts by the total number of observations.

Finally, we have the Classify method, that accepts as argument, a single feature vector (as a list), and computes the product of individual conditional probabilities (smoothed MLE) for each label. The final computed probabilities for each label are stored in the ‘probabilityPerLabel‘ dictionary. In the last line, we return the entry from probabilityPerLabel which has the highest probability. Note that the multiplication is actually done as addition in the log domain as the numbers involved are extremely small. Also, one of the factors used in this multiplication, is the prior probability of having this class label.
Here is the complete code, including a test method:

Download the sample ARFF file to try it out.

Update: I found a bug in the last but one(th) line of the GetValues() function. This line gets the possible attribute values from the arff file and stores them in self.featureNameList. This method did not deal with whitespaces correctly. Update this line to:<code>self.features[self.featureNameList[len(self.featureNameList) - 1]] = [featureName.strip() for featureName in line[line.find('{')+1: line.find('}')].strip().split(‘,’)]</code>

<code>sudo apt-get install php5 mysql-server apache2 phpmyadmin</code>

It worked fine, but then, my python script that runs on another machine began to complain that it could not connect to my MySQL server:

 Can’t connect to MySQL server on ‘server’

Now that was just ridiculous because this has never happened before. So I trolled and trolled till I found what I was looking for:

http://www.webmasterworld.com/forum10/6141.htm

So, turns out that there is this tiny piece of configuration information in your /etc/mysql/my.cnf file that says:

bind-address = 127.0.0.1

which essentially means that all connections coming in from anywhere other than the local machine will not be entertained. Remove or comment that line and restart your server. Things start working!

The interesting thing about this problem is how some naive approaches, even though easy to code and efficient enough, do not achieve the desired randomness.

Lets define the problem: You have an input array which has elements in positions 1 through n. The objective is to produce a random permutation of the array. By a random permutation, we mean to say that each permutation of the array is equally likely. So a truly randomized algorithm will generate each permutation with a probability of 1/n!

The first approach that comes to mind is the naive approach of generating a random number between 1 and n for each element in the array and placing the element at the position indicated by the random number. This could be accomplished by using another auxillary array and placing elements into the new positions generated. The pseudocode could look something like the following:

NaiveShuffle(A[1...n], B[1....n])            //randomly permutes the elements in array A
for i from 1 to n
random <- RandomNumber(1,n)
B[random] <- A[i]
for i from 1 to n
A[i] <- B[i]


The above approach is very crude in the sense it uses an auxillary array and also parses through the array twice instead of just once. Still, the worst case running time of the above algorithm is O(n).

Random(1, n) generates a random number 1 and n (both inclusive). We assume that the random number generator generates truly random numbers in the interval specified. Also, we assume some kind of collision resolution mechanism. We are assuming that this method returns a random number in O(1) time.

All that said, a O(n) algorithm is not bad at all for this purpose. There is only one problem, this algorithm is WRONG! (hah…fat chance, didn’t we name it a naive algorithm?). Why is that? This is because in every iteration, we generate n possible choices. Since there are n such iterations, the total number of permutations generated is n.n.n.n…….n times = n^n

The  total number of permutations possible while shuffling an array is n!. Since n^n is not exactly divisible by n!, there have to be some permutations which appear more frequently than the others (basic pigeonhole principle). Thus this naive algorithm does not generate truly random permutations.

Lets try something else. We generate a random priority between 1 and n^3 the Random(1, n^3)  routine, and assign a it to each element in the array. Then sort the array based on these weights.

e.g. if the original array is A<1,2,3,4> and we generate priorities randomly as P<34,56,8,77>, then when we sort array A based on the increasing order of the priorities assigned using array P, then we get the shuffled array as <3, 1, 2, 4>.

We used the interval [1, n^3] to generate priorities so as to reduce collisions. I will not delve into the correctness of this algorithm. The running time of this shuffling by sorting algorithm depends on which sorting algorithm we use. Typically it is Big-Theta(n lg n).

Another approach to solve this problem, is to shuffle by swapping:
Shuffle[1...n]
for i from 2 to n
temp = Generate(x from 1 to i)
swap(i,temp)

Notice that the interval for generating random numbers keeps decreasing. For the first iteration, there are n choices. For the 2nd iteration, there are (n-1) choices and so on.

Hence the total number of choices generated by this algorithm is: n(n-1)(n-2)…..3.2.1 = n!

which is exactly equal to the total number of possible permutations. Moreover, since we iterate over the array only once, this algorithm runs in O(n) time.

Hello {0}
Mail body

Where {0} is a placeholder that will be replaced by the receiver name. mail1.txt and mail2.txt have the same structure except for the mail body depending on the requirement.
Next, I needed to create a list of names and the corresponding email IDs to which the mails are to be sent. Also, I needed a flag that will indicate the type of message that is to be sent, i.e. mail1 or mail2. I created a comma separated file with the following format:

<Receiver’s name>, <email id>, <mail body to be sent, i.e. 1 or 2>

I wrote a console application in C# that uses the Microsoft Office 2007 Primary Interop assemblies to automate sending emails to all these contacts specified. The emails get sent using the default account configured in your Outlook. The code looks something like the following. Please note that this is a quick hack which actually works and that I have not really done a lot of error handling or exception management on this because I know the conditions under which this will be used.

Microsoft.Office.Interop.Outlook.Application app = null;
Microsoft.Office.Interop.Outlook._NameSpace ns = null;
Microsoft.Office.Interop.Outlook.PostItem item = null;
Microsoft.Office.Interop.Outlook.MAPIFolder inboxFolder = null;
Microsoft.Office.Interop.Outlook.MAPIFolder subFolder = null;
Microsoft.Office.Interop.Outlook.MailItem memo = null;
Microsoft.Office.Interop.Outlook.MAPIFolder sentFolder = null;
StreamReader addressReader = null;
StreamReader contentsReader = null;
StreamWriter logWriter = null; 

try
{
addressReader = new StreamReader(ConfigurationManager.AppSettings["addresses"]);
String currentLine = String.Empty;
String[] currentReceiver = null;
String messageBodyFile = String.Empty;
logWriter = new StreamWriter(Path.Combine(Environment.CurrentDirectory, "Log.txt"), false);
while (!addressReader.EndOfStream)
{
currentLine = addressReader.ReadLine();
currentReceiver = currentLine.Split(',');
switch (currentReceiver[2])
{
case "1":
messageBodyFile = ConfigurationManager.AppSettings["contentsFile1"];
break; 

case "2":
messageBodyFile = ConfigurationManager.AppSettings["contentsFile2"];
break; 

default:
Console.WriteLine("Could not send email to ", currentReceiver[0]);
logWriter.WriteLine("Could not send email to ", currentReceiver[0]);
currentReceiver[1] = String.Empty;
break;
} 

#region EmailInit 

app = new Microsoft.Office.Interop.Outlook.Application();
ns = app.GetNamespace("MAPI");
ns.Logon(null, null, false, false);
sentFolder = ns.GetDefaultFolder(OlDefaultFolders.olFolderSentMail);
memo = (Microsoft.Office.Interop.Outlook.MailItem)app.CreateItem(OlItemType.olMailItem); 

#endregion 

contentsReader = new StreamReader(messageBodyFile);
memo.To = currentReceiver[1].Trim();
memo.Subject = ConfigurationManager.AppSettings["mailSubject"].Trim();
memo.Body = String.Format(contentsReader.ReadToEnd(), currentReceiver[0]);
memo.BodyFormat = OlBodyFormat.olFormatHTML;
memo.Send();
Console.WriteLine("{0}: Sent email with body {1} to {2}:{3}", DateTime.Now, currentReceiver[2], currentReceiver[0], currentReceiver[1]);
logWriter.WriteLine("{0}: Sent email with body {1} to {2}:{3}", DateTime.Now, currentReceiver[2], currentReceiver[0], currentReceiver[1]);
contentsReader.Close();
contentsReader.Dispose();
}
} 

catch (System.Exception ex)
{
Console.WriteLine(ex.ToString());
EventLog.WriteEntry("Email Automation", ex.Message, EventLogEntryType.Error);
} 

finally
{
ns = null;
app = null;
inboxFolder = null;
addressReader.Close();
addressReader.Dispose();
logWriter.Close();
logWriter.Dispose();
}

So I got this funny little compilation error which left me stumped:

/tmp/cckI2FzP.o:(.eh_frame+0×11): undefined reference to `__gxx_personality_v0′
collect2: ld returned 1 exit status

I googled and found that this error is normally related to C++, but I was writing code in plain old C. So what was wrong? I found later that I had named my code file as List.C instead of List.c. After renaming it to List.c, all was well.

Turns out that filename extensions in linux are case sensitive (wonder why I did not run into that problem all these years),  and that C is a commonly used extension for C++