iink SDK on Device

Answered

IINK becomes unresponsive

IInk becomes unresponsive when we do the following operations on Android:

We write Japanese text using complex characters and frequently use Undo instead of the Eraser to delete text, write more text, delete some of it again with undo, and repeat this for a while.

At some point, Iink becomes unresponsive and it is impossible to write strokes (no strokes are drawn when writing with the stylus) or save the document. The part type is a Text Document.

Did you ever see such a problem, or have any idea about what might be causing it and how it could be avoided?

Thank you!


Best Answer

Dear Nicolas,


This is likely a platform issue, not an iink issue ; indeed, as we get the events as-is from the Android, it appears Android is not returning the proper pointerId.


At present, we do not plan to patch our code for such behavior. Nevertheless, it is likely your above code should prevent this from occuring.


Best regards,


Olivier


Dear Nicolas,


after exchanging with our QA team, this is not a behavior we have faced.


Can you confirm you are using the latest 1.4.1 release of the iink SDK (the one available on github) ?


How long does it take to reproduce? Are you simply writing? Or other actions such as zoomin/zoomout are done? Do you see any error message in the logcat?


Are you able to reproduce with our getStarted out of the box? Ideally, would you have a small scenario that allows to reproduce?


Best regards,


Olivier

Thank you for looking into this.

Unfortunately, the problem was experienced by our client and we were unable to reproduce it ourselves, so we don't have logs. They seem to be able to reproduce it quite quickly (often in less than 1 minuntes) but only when using an older device (Galaxy Tab S3). Regarding other actions, they do some document scrolling but it is not clear from the video they sent us

if they zoom in and out.

Dear Nicolas,


Thank you for the update.


I will track the behavior internally, but without any scenario or log, we cannot commit any correction will be brought to it.


Best regards,


Olivier

Hello Olivier,

We gathered more details about this issue where iink stops being responsive while we are writing.

- The problem can happen even when we don't perform Undo/Redo.

- The problem is more frequent with a low spec tablet. We tested with Galaxy Tab 3 and Galaxy Tab S6 Light and the problem was more frequent with Galaxy Tab 3.

- When we become unable to write, all iink functionality (saving for example) appears to stop working. Our app's buttons and other functionality unrelated to iink are still responding, so it seems like only iink is stuck.

- Even though iink is stuck, there is no exception or other error being output.

- We use iink's IEditorListener.contentChanged() to Export the contents of TextBlocks that were modified. They are exported using Editor.export_() using both the Text and the JIIX formats.

- We also have a background task saving the current document using  ContentPackage.saveToTemp() every 10 seconds.

The differences in the handwriting part between the GetStarted demo app and our app are that we export changes in contentChanged() and save regularly to temp.

Is it possible that if we save to temp or export a text block while at the same time inputting strokes, it can cause iink to become stuck? If that's the case, is it necessary to wait or block to avoid performing such operations at the same time?

Thank you for your help!

Dear Nicolas,


as we already explained in January, the contentChanged is triggered very often (several times per second), and exporting as JIIX in the latter can really slow down the application: https://developer-support.myscript.com/support/discussions/topics/16000030242


Based on this, it seems you are facign this behavior, in particular on low-tier devices.


The only solution is that you proceed as explained in the above topic.


Best regards,


Olivier

Thank you Olivier.

We understand that exporting JIIX data with a high frequency can slow down the application, but after implementing it, we find that the speed is bearable.

Our problem is that iink suddenly hangs. 1 second we are writing normally with no noticeable lag, and the next, strokes completely stop being drawn on the screen in response to stylus input. Even after waiting for more than 1 minute, there is no stroke output on the screen.

Is the hanging problem caused by the high load of exporting a lot of JIIX data?

Dear Nicolas,


yes, this corresponds to the behavior we noticed, and also our customers.


Our recommendation is to not export in the contentChanged.


Best regards,


Olivier

Hi Olivier,

We investigated more about this problem where iink become unresponsive and understood a few things.

- If we try to zoom while iink is unresponsive, it throws an exception and the app crashes. This seems to be the same problem as described here:  https://developer-support.myscript.com/support/discussions/topics/16000024603

- There are cases where editor.pointerDown() is performed in InputController.handleOnTouchForPointer() but editor.pointerUp() or editor.pointerCancel() is not.

To fix the 2 problems above, we made the following changes to the reference implementation's InputController.java.

- We added a list to manage pointerIds.

- After editor.pointerDown() is called, we add the pointerId to the list.

- After editor.pointerUp() or editor.pointerCancel() is called, we delete the pointerId for the list.

- Before calling editor.pointerDown(), we check the list and if it contains the same pointerId we call editor.pointerCancel().

- When the event MotionEvent.ACTION_CANCEL comes, we call editor.pointerCancel() for every pointerId in the list.

The following shows the changes we made to the code:


 

--- InputController.java      2021-08-30 18:23:16.504268700 +0900
+++ InputController.java      2021-09-06 16:52:06.536356500 +0900
@@ -17,6 +17,7 @@ import com.myscript.iink.PointerType;
 import com.myscript.iink.graphics.Point;

 import java.util.EnumSet;
+import java.util.HashMap;

 public class InputController implements View.OnTouchListener, GestureDetector.OnGestureListener
 {
@@ -33,6 +34,7 @@ public class InputController implements
   private final GestureDetectorCompat gestureDetector;
   private IInputControllerListener _listener;
   private final long eventTimeOffset;
+  private HashMap<Integer, PointerType> pointerMap = new HashMap<>();

   public InputController(Context context, IRenderTarget renderTarget, Editor editor)
   {
@@ -107,10 +109,16 @@ public class InputController implements

     int historySize = event.getHistorySize();

+    boolean isPointerDown = pointerMap.containsKey(pointerId);
+
     switch (actionMask)
     {
       case MotionEvent.ACTION_POINTER_DOWN:
       case MotionEvent.ACTION_DOWN:
+        if(isPointerDown) {
+          editor.pointerCancel(pointerId);
+        }
+        pointerMap.put(pointerId, iinkPointerType);
         editor.pointerDown(event.getX(pointerIndex), event.getY(pointerIndex), eventTimeOffset + event.getEventTime(), event.getPressure(), iinkPointerType, pointerId);
         return true;

@@ -138,11 +146,17 @@ public class InputController implements
             pointerEvents[i] = new PointerEvent(PointerEventType.MOVE, event.getHistoricalX(pointerIndex, i), event.getHistoricalY(pointerIndex, i), eventTimeOffset + event.getHistoricalEventTime(i), event.getHistoricalPressure(pointerIndex, i), iinkPointerType, pointerId);
           editor.pointerEvents(pointerEvents, true);
         }
-        editor.pointerUp(event.getX(pointerIndex), event.getY(pointerIndex), eventTimeOffset + event.getEventTime(), event.getPressure(), iinkPointerType, pointerId);
+        if(isPointerDown) {
+          editor.pointerUp(event.getX(pointerIndex), event.getY(pointerIndex), eventTimeOffset + event.getEventTime(), event.getPressure(), iinkPointerType, pointerId);
+          pointerMap.remove(pointerId);
+        }
         return true;

       case MotionEvent.ACTION_CANCEL:
-        editor.pointerCancel(pointerId);
+        for(Integer key : pointerMap.keySet()) {
+          editor.pointerCancel(key);
+        }
+        pointerMap.clear();
         return true;

       default:

 With the changes above, we tested our app for 2 hours and did not experience the problem where iink becomes unresponsive.

However, we would like to check with you to know if the above changes could have a bad effect on the iink library.

Thank you!

We found a way to reproduce the problem with some reliability using the reference application "iink GetStarted" with a Samsung Galaxy Tab S6 Lite (Android 10). It might also be able to use this way with other devices that can detect when the stylus is close to the screen without touching it. 

1. Start the sample app "iink GetStarted"

2. Set the Mode to AUTO

3. Scroll the screen up and down using all 5 fingers.

4. Bring the stylus close to the screen while performing step 3.

5. Confirm that scrolling stops regardless of the movement of the fingers and then remove the fingers from the screen.

6. Write some text with the stylus.

7. Repeat steps 3-6 and after a few repetitions, it becomes impossible to write with the pen.

8. Continue to repeat steps 3-6 and eventually an exception will be thrown (see the attached image).

image

It also looks like if ACTION_CANCEL is called during a multi-touch editor.pointerCancel() is only called for 1 pointerId in the processing of ACTION_CANCEL in InputController.handleOnTouchForPointer(), but after that, the ACTION_UP / ACTION_CANCEL events are not called for the other pointerIds.

Would it be necessary to call editor.pointerCancel() for every pointerId in ACTION_CANCEL?

Dear Nicolas,

Indeed, if pointerUp or pointerCancel events are missing, this can have an impact on the performances. Your above fix is then a proper solution.


Also, as we get the events "as-is" from the Android onTouch, such events are likely to occur with your above scenario. Developers can then patch the uireferenceimplementation accordingly.


Best regards,

Olivier

Thank you Olivier.

How about the problem with "iink GetStarted" we added above just before you replied?

Answer

Dear Nicolas,


This is likely a platform issue, not an iink issue ; indeed, as we get the events as-is from the Android, it appears Android is not returning the proper pointerId.


At present, we do not plan to patch our code for such behavior. Nevertheless, it is likely your above code should prevent this from occuring.


Best regards,


Olivier