# Forums

General questions

## Picture of handwritten notes

I want to recognize images of handwritten notes. Is it possible with your technology?

2 people have this question

MyScript cannot be used to open image files such as JPG, PDF, because it is not based on Optical Character Recognition technology.
MyScript uses an online signal (digital ink) which is richer than the mere optical signal from an image (made of pixels). It provides a clean digital signal which includes dynamic information about the X Y coordinates capture sequence and allows reaching much accurate handwriting recognition results. This digital ink can be captured using digital pens or any touch sensitive device such as touch screen, tablets and smartphones.

Unfortunately, we are not aware of a way to convert an image to digital ink. We are focused on handwriting recognition of ink captured by using digital pens or touch sensitive devices as explained above.
Best regards

Would there be any possible way, i can convert the Image/PDF into some sort of input for MyScript, I would'nt mind if this would somehow compromise the end-results a little bit. I have a strong need to extract text from an image

Unfortunately, we have no solution for this use case today. A previous post on this forum mentioned a similar project for Math with bezier curve, maybe it could help you:
http://www.professorcloud.com/svg-to-canvas/
Best regards

Hello,

we also have a strong need for the recognition (in our case Math) based on image input.
Therefore, we are also willing to create a work-around solution.

In order to make this work, it would be really helpful if you could give us some more information/documentation what exactly you mean by digital ink? Which events are you exactly listening for? And which events do we have to fake in which way in order to simulate human touch input?
Is it maybe something like this:
http://stackoverflow.com/questions/16650419/draw-in-canvas-by-finger-android

Our current idea for the simulator algorithm is as follows:
extract individual characters (classical OCR step)
decide which character to analyze first/second/.. (small Machine Learning problem)
for each character:
+ derive approximate vector drawing path of character
+ decide which path would be drawn first/second/.. by a human and where the human would start (small Machine Learning problem)
+ input the touch events based on the path order

Since we have some rough idea how to implement all steps except for the last one - we would really appreciate your help.

We are also willing to work together with you on this solution if it might be important/promising to advancing your business.

Florian

Update:
after having tried some experiments with the ATK Math widget, there are some new findings:

the widget is able to detect the mentioned Math rules (fraction, subscript, etc) based on Symbols when those are at the right spot.
http://doc.myscript.com/MyScriptATK/Android/math/maw.html

Thus, those symbols can be generated artificially and don't have to be inserted using digital ink.

This reduces the problem of hand-written formula recognition off paper:
We only have to detect all numbers and formula signs based on the handwriting.
This is a classical OCR pipeline which can be solved using openCV.

Most likely, we will solve this problem or search for some existing components and stack them together.

Best regards,
Florian

Dear Florian,

thank you for contacting us.

In order to make this work, it would be really helpful if you could give us some more information/documentation what exactly you mean by “digital ink”?

>>Digital ink basically consists in strokes. A Stroke is a serie of X and Y coordinates that are gotten in between a pen down and a pen up. From this, you understand characters (or words) can be made of one or several strokes.

Which events are you exactly listening for? And which events do we have to fake in which way in order to “simulate” human touch input?
>>In the mathWidget, there is not API to easily add strokes. The workaround would consist in adding a view in which you would have to draw the strokes. As you guess, this is likely to be demanding. Or, another solution may consist in using the math SDK, but this would demand more work on your side, as you need to manage the ink capture, the computation, the display...

Our current idea for the simulator algorithm is as follows:
extract individual characters (classical OCR step)

>>The problem with OCR is to get a clean ink. Also, you do not have information such as the starting.end point of strokes. Also, in case a character is made of several strokes, it may be difficult to extract these.

Thus, those symbols can be generated artificially and don’t have to be inserted using “digital ink”.

This reduces the problem of hand-written formula recognition off paper:
We “only” have to detect all numbers and formula signs based on the handwriting.
This is a “classical” OCR pipeline which can be solved using openCV.

Most likely, we will solve this problem or search for some existing components and stack them together.