The original images which will be preprocessed and passed to a convolutional neural network are scanned A4 letters with a resolution of about 2550x3500 pixels at 300dpi.

At first some letters will be manually converted into text for labeling, which is required for the training process. A self written tool is used to assign a position to each character in the transcription. This process can be seen in the screenshot above. The tool was written with OpenCV and allows a preview of the next two words, highlights the current letter which is to be labeled and shows some bounding box with hints for the later cropping process. (If you want the source just write me some message but i also plan to release the project after some results)

After each letter has some assigned position, the image still needs to be preprocessed/enhanced as the color of the font is quite close to the background color. This step can be done automatically in the future. For each labeled character a 32x32px sub image will be cropped and additionally for each of those images small pixel translations, scaling and sinus wave transformations will be applied to generate a lot more training data for the learning process.

For the CNN i planned to use Tensorflow as it has been proven to be quite flexible for this task.

I will present the network configuration and further results in the next blog posts :)

Current letter counts after labeling two letters:

     16 ,
      2 !
     24 .
      2 0
      4 1
      3 2
      1 3
      2 4
      1 9
     71 a
      3 ä
     22 b
     54 c
     33 d
    162 e
     17 f
     26 g
     76 h
     81 i
      5 j
     13 k
     41 l
     22 m
    100 n
     26 o
      2 ö
      5 p
     65 r
     48 s
      4 ß
     57 t
     22 u
      4 ü
      1 Ü
      7 v
     31 w
     13 z

You can follow the project on the blog, github or via rss with the following urls:

• Blog: https://harpoon.0x17.de/blog/
• RSS: https://harpoon.0x17.de/blog/index.xml
• GitHub Harpoon: https://github.com/HarpoonOrg/Harpoon
• GitHub HarpoonClient: https://github.com/HarpoonOrg/HarpoonClient

Have fun reading! :)

Here are some screenshots of the current state:

Those screenshots need a little description, so here you go:

Harpoon consists of a server program and three different clients: Web/Desktop/Mobile.

In the first screenshot you can see the browser interface (topleft), whereas in the second screenshot you can see the desktop client.

The project is still work in progress as some essential parts are missing. Those are showing the backlog, user login and yet server configuration. In the next few updates those topics will be adressed and i will keep you up to date.

If you have questions you can find me on IRC. Just click the following link: freenode.net channel #harpoon.

Follow me on twitter, watch the project on github, read my RSS or join #harpoon on freenode.net to receieve status updates. :)