Interactive Ink

Ink SDK + offline support

Hello!

We'd like to backup the strokes that users draw during the short periods when their devices loose internet connection.

For this we could use the 'changed' event and store the strokes in browsers localstorage, and after the connection is restored - we would grab the strokes from localstorage and send to Myscript for recognition. We'd like to backup the strokes in localstorage, so that we could restore them in case browser window would be reloaded.

There are two questions:

1. What is the best way to save the strokes while being offline (the export is not available)?

2. How can we send the stored strokes to Myscript after users get back online?

Thanks!


Dear Nadi,

Indeed, the export function is not available while offline.

What can be done is use the "import" function to import the strokes that have not been sent. See import ink: https://developer.myscript.com/docs/interactive-ink/1.0/web/websockets/messages/

{
  type: 'importInk',
  strokes: [{
    gesture: false,
    pointerType: 'PEN',
    pointerId: 1,
    "x": [273, 278, 281],
    "y": [121, 128, 133],
    "t": [3185.7900000000004, 3213.8150000000005, 3222.5350000000003],
    "p": [1.0, 1.0, 1.0]
  },{
    gesture: false,
    pointerType: 'PEN',
    pointerId: 1,
    "x": [173, 178, 181],
    "y": [221, 228, 233],
    "t": [6185.7900000000004, 6213.8150000000005,6222.5350000000003],
    "p": [1.0, 1.0, 1.0]
  }]
}

You should then store by yourself the stroke information (w, y, t...), then fill in the structure and import it.

Best regards,

Olivier

Thanks Oliver, I get the following error from MyScript while trying to import your example strokes: 

"unable to parse message com.fasterxml.jackson.databind.JsonMappingException: Can not deserialize instance of java.util.ArrayList out of START_OBJECT token
 at [Source: {"type":"importInk","strokes":{"type":"importInk","strokes":[{"gesture":false,"pointerType":"PEN","pointerId":1,"x":[273,278,281],"y":[121,128,133],"t":[3185.7900000000004,3213.8150000000005,3222.5350000000003],"p":[1,1,1]},{"gesture":false,"pointerType":"PEN","pointerId":1,"x":[173,178,181],"y":[221,228,233],"t":[6185.79,6213.8150000000005,6222.535],"p":[1,1,1]}]}}; line: 1, column: 31] (through reference chain: com.myscript.cloud.iink.message.receive.ImportInkMessage["strokes"]) {"type":"importInk","strokes":{"type":"importInk","strokes":[{"gesture":false,"pointerType":"PEN","pointerId":1,"x":[273,278,281],"y":[121,128,133],"t":[3185.7900000000004,3213.8150000000005,3222.5350000000003],"p":[1,1,1]},{"gesture":false,"pointerType":"PEN","pointerId":1,"x":[173,178,181],"y":[221,228,233],"t":[6185.79,6213.8150000000005,6222.535],"p":[1,1,1]}]}}"

 Please advise.

Dear Nadi,


you seem to use "type":"importInk" twice:

{"type":"importInk","strokes":{"type":"importInk","strokes":

Can you please check your requests, and ensure these are formatted as follows: {"type":"importInk","strokes":[{"gesture":false,...


Best regards,


Olivier

Hi Oliver, it looks like the editor.importInk() method wraps the passed object. It seems that ImportInk method expects the strokes array: https://myscript.github.io/MyScriptJS/docs/class/src/Editor.js~Editor.html#instance-method-importInk

Passing only the strokes array worked fine. Thank you!

Hi Oliver, 

One more related question - what would be the best way to save the strokes offline?

Right now I'm saving the rawStrokes from the model in the "changed" event:

 

evt.srcElement.editor.model.rawStrokes

The issue with this approach is that when I import these strokes, they contain all strokes that were written, including those that were erased with gestures. 

Here is an example of the result of the importInk():

image


User writes "a b c", then he strokes out the "c" and is left with "a b" (which is correct). After the "rawStrokes" import, I get the above result, while I'm expecting to get only the strokes for "a b", because "c" was previously deleted with gestures.

I tried importing with both "gestures: true" and "gestures: false", but it doesn't affect the behaviour. 

 

{ gestures: false }

Please advise. 


1 person likes this

To clarify - I'm using "gesture:true/false", not "gestures" as I wrote above.

Dear Nadi,

currently, the gesture detection only works when time infomation for each point is provided. If not, setting gesture parameter to true or false will not change anything.

Can you please provide with an example of request?

Best regards,

Olivier

Hi Oliver,

Here is an example of strokes array that I get from "changed" events (evt.srcElement.editor.model.rawStrokes):

  

[{"type":"stroke","x":[318,323,326,330,332,333,335,336,338,340,340,339,337,336,336,335,335,335,335,335,335],"y":[80,77,74,69,66,63,59,55,50,46,51,56,65,78,91,105,112,117,121,124,127],"t":[1513939059308,1513939059468,1513939059485,1513939059501,1513939059518,1513939059535,1513939059568,1513939059585,1513939059618,1513939059651,1513939059851,1513939059868,1513939059885,1513939059901,1513939059918,1513939059936,1513939059951,1513939059968,1513939059985,1513939060001,1513939060035],"p":[0.5,0.7585263597233582,0.7222966378316396,0.774350629289587,0.7007103955941203,0.6831899108492184,0.7292125754002905,0.7185264889882718,0.7531128489300326,0.7292125754002905,0.7436746598382117,0.7461846839143089,0.8133650328603405,0.6389126863152722,0.639444872453601,0.6253579195069219,0.78472787299413,0.7436746598382117,0.7145158745241171,0.6761535882018319,0.6761535882018319],"l":[0,5.830951894845301,10.073592581964586,16.476716819397435,20.082268094861423,23.2445457550298,27.71668171002938,31.839787335647042,37.22495214278155,41.697088097781126,46.697088097781126,51.79610761137391,61.015652068666796,74.05405687907209,87.05405687907209,101.08972572669029,108.08972572669029,113.08972572669029,117.08972572669029,120.08972572669029,123.08972572669029],"width":3.7795275590551185,"pointerType":"mouse","pointerId":1,"color":"green","-myscript-pen-width":2,"-myscript-pen-fill-style":"none","-myscript-pen-fill-color":"#FFFFFF00","id":"pendingStroke-0"}]

If there are other options to get strokes with timestamps - please let me know.

Thanks!

Nadi

 

Dear nadi,

we checked again on our side, and the gestures normally work fine.

For this, you should endure the writing of the gesture is "fast enough", so that our system sees it as an erasure and not dash.

Best regards,

Olivier

Hi Oliver,

The "importInk" method doesn't accept the "timestamp" property - it returns this error message: "unable to parse message com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException: Unrecognized field "timestamp" (class com.myscript.cloud.iink.message.receive.batch.BatchStroke), not marked as ignorable (7 known properties: "p", "t", "pointerType", "pointerId", "gesture", "x", "y"])

You said that in order to process gestures, I need to provide "time information for each point". How does MyScript server expect to receive the time information? Please provide more details or point to the docs.

Thanks,

Nadi

Hi Oliver!

I figured out that the "t" param contains all the timestamps (please correct me if I'm wrong).

Still, I can't import any strokes so that gestures would be recognized. Please check the following example:

 

domEl.editor.importInk(
              [{"x":[243,242,241,239,237,235,232,230,228,225,223,222,221,220,219,218,217,217,216,216,216,216,216,216,216,217,217,219,220,222,224,227,229,231,233,235,236,238,239,240,241,241,241,242,242,243,243,243,243,243,243,243,243,243,243,243,243,243,244,244,244,245,245,245,246,247,247,248,250,253,256,259,261,262,263,264,265,265,265,265],"y":[86,83,82,81,80,80,79,79,79,79,79,80,81,83,84,87,88,90,91,92,93,94,95,96,97,98,99,100,100,100,100,100,100,100,100,99,97,96,94,92,90,88,86,84,83,81,80,79,79,79,80,81,82,84,85,86,88,92,94,95,96,97,97,98,98,99,99,99,99,99,100,100,100,100,100,100,100,100,100,99],"gesture":true,"t":[1514945368722,1514945368947,1514945368963,1514945368980,1514945368997,1514945369013,1514945369030,1514945369046,1514945369063,1514945369080,1514945369098,1514945369114,1514945369130,1514945369147,1514945369164,1514945369180,1514945369197,1514945369214,1514945369231,1514945369247,1514945369264,1514945369281,1514945369298,1514945369314,1514945369330,1514945369347,1514945369364,1514945369381,1514945369397,1514945369414,1514945369430,1514945369447,1514945369463,1514945369480,1514945369497,1514945369513,1514945369530,1514945369548,1514945369564,1514945369584,1514945369597,1514945369614,1514945369630,1514945369647,1514945369664,1514945369680,1514945369697,1514945369714,1514945369732,1514945369864,1514945369881,1514945369897,1514945369914,1514945369933,1514945369950,1514945369967,1514945369984,1514945370015,1514945370031,1514945370047,1514945370064,1514945370080,1514945370097,1514945370114,1514945370131,1514945370147,1514945370163,1514945370180,1514945370198,1514945370214,1514945370230,1514945370247,1514945370264,1514945370280,1514945370298,1514945370314,1514945370330,1514945370348,1514945370364,1514945370466]},{"x":[252,252,248,243,234,227,219,215,212,209,209,208,208],"y":[52,53,58,68,85,100,115,123,130,135,136,137,137],"gesture":true,"t":[1514945371371,1514945371598,1514945371614,1514945371631,1514945371648,1514945371664,1514945371681,1514945371698,1514945371714,1514945371731,1514945371748,1514945371764,1514945371780]}]
            );

 It's being consistently imported as:

image


This "a" is not being erased after importing. (However, it's being successfully erased when written the same way in the live editor.) 

I tried making the erasing gestures with different speeds, it doesn't help. I tried using different erasing gestures - still not being recognized after importing. I tried "erasing" both letters and numbers - still no luck.

Could you please provide a working example of the strokes array, which contain erasing gestures and these gestures being recognized by using the importInk() method?

Thanks,

Nadi


Hi Nadi,I am leading the team that is developing cloud and web technology ate MyScript.The typical way to store ink data at MyScript is using the JIIX format”. If you are willing to store strokes without the ones detected with gesture your option is to use the jiix export (based on what is recognize)In your very valid use case it will not help you much as your strokes could be collected offline. 

Sadly gestures detection (strike-through in particular) is not so accurate when importing multiple strokes the same time.

If writing offline is really a strong requirement maybe considering "On device" iink SDK is a better option.

Regards

Pierre-Alban

Login or Signup to post a comment