Handwriting recognition using Tensorflow and Keras

Handwriting recognition aka classifying each handwritten document by its writer is a challenging problem due to huge variation in individual writing styles. The traditional approach to solving this would be to extract language dependent features like curvature of different letters, spacing b/w letters etc. and then use a classifier like SVM to distinguish between writers. In the blog, I want to demonstrate a deep learning based approach to identifying these features. We will pass small patches of handwritten images to a CNN and train with a softmax classification loss.

To demonstrate the effectiveness of this technique, lets use it to classify English Handwritten text.

You can find the full code on my Github repo

Getting our data

The IAM Handwriting database is the biggest database of English handwriting images. It has 1539 pages of scanned text written by 600+ writers. For the purpose of this demo we will take the top 50 writers with the most amount of data. For each writer, the database is a collection of individual sentences written by them. See samples below for one writer:

Neural networks don’t require much preprocessing of raw data. So we will not make any modifications to these images however instead of passing the full image to the neural network, we will pass small patches of text.

Generating patches of data

We want the neural network to understand the writing style of individual writer and we would prefer that this neural network be text independent (can work on any language). So instead of passing individual sentences or words, we will pass it random patches of text. This is done by randomly cropping 113x113 sized patches from every sentence. The image below is a collage of 8 such patches.

We can write a generator function to move over each sentence and generate random patches of images from it. For every image we will limit the no. of patches to 30% of total patches that can be generated. For more information on how to write this generator function, please check out my Github repo.

Model and Results

For this task we build a convolution neural network (CNN) in Keras using Tensorflow backend. We will use a standard CNN with multiple convolution and maxpool layers, a few dense layers and a final output layer with softmax activation. RELU activation was used between the convolution and dense layers and model was optimized using Adam optimizer.

The size of the model needs to be proportional to the size of the data. Three blocks of convolution -maxpool layers and couple of dense layers was sufficient for this problem. See model summary below:

After a bit of hyper parameter tuning we got to a loss of 94% on the test dataset which the model was never exposed to.

See two patches below that the model classified as the same writer. The shape of “t” seems very similar so would also make intuitive sense that they belong to the same writer.