Interactive Ink

unable to convert pointerEvents to text in android or accurately at MyScriptJS

I have been trying to convert x-y coordinates obtained from the wacom clipboard my handwritten text-

image

i have converted it to PointerEvents as-

image

follows as how i obtained pointer events array from clipboard strokes

pointerEvents2[k] = new PointerEvent(PointerEventType.CANCEL, saved_strokes.get(i)[j].x, saved_strokes.get(i)[j].y, saved_strokes.get(i)[j].timestampInMillis, 1.0f, PointerType.PEN, 1);

editor.pointerEvents(pointerEvents2,false);
editor.waitForIdle();
try {
editor.export_(null, MimeType.TEXT);
} catch (IOException e) {
e.printStackTrace();
}

*Editor have been initialised and all but i still get error at editor.pointerEvents(pointerEvents2,false);

Please help me out where it got wrong.

I also tried same text points conversion on MyScriptJs but it isn't accurate with words its doing fine with the characters-

image

at-

https://myscript.github.io/MyScriptJS/examples/v4/pointer_events.html

txt

Dear Jai Hada,


thank you for contacting us and your question.


Currently, we are a bit puzzled you are using the "Cancel" PointerEventType, as this latter cancels an ongoing pointer trace. You will then not get any recognition.


Currently, when adding coordinates in an "off-screen mode", each stroke should be added starting by a "DOWN" PointerEventType, and finish by a "UP" PointerEventType. All points in between should be added using a "MOVE" PointerEventType.


A stroke should then be added as explained in the "Series of events" part of our documentation: https://developer.myscript.com/docs/interactive-ink/1.2/android/fundamentals/editing/#series-of-events


Can you proceed as explained, and let us know if it doesn't work?


Best regards,


Olivier

Dear Olivier

Thanks for your time, I did converted the pointer events as you suggested but it results the same let me post the error too if it might help.

image

above is pointerEvents with down, move and up actions

File file = new File(getFilesDir(), packageName);

                ContentPackage pkg = engine.createPackage(file);

                ContentPart part = pkg.createPart("Text");

                editor.setPart(part);

                editor.pointerEvents(pointerEvents2, false);

                editor.waitForIdle();

                try {

                    editor.export_(null, MimeType.TEXT);

                } catch (IOException e) {

                    e.printStackTrace();

                } 

but i still get the error at NativeFunctions.pointerEvents

here is error screen-

image



Also at the MyScriptJS

https://myscript.github.io/MyScriptJS/examples/v4/pointer_events.html

if we feed  pointer events stroke by stroke it converts right but all strokes are fed via pointerEvents it converts with inaccuracy as i mentioned earlier so i need to know if there is a way to feed it with complete pointerEvents array and it provide accurate corresponding text

Thanks with regards

Jai Singh

Dear Jai,


we are facing difficulties understanding what's not fine on your side.  Indeed, you ink should normally be properly recognized.


Please attach your code, so that we can understand what is wrong in this latter one.


Best regards,


Olivier

Hello Olivier,

I am attaching the class and few snapshots of data in arrays .

image

below are the savedstroke data for reference

image

image

 

// Copyright MyScript. All rights reserved.

package com.hcflab.mbroadcast.factory.myscript.iink.getstarted;

import android.media.Image;
import android.os.Bundle;
import android.os.Environment;
import android.support.annotation.Nullable;
import android.support.v7.app.AppCompatActivity;
import android.util.Log;
import android.view.Menu;
import android.view.MenuItem;
import android.view.View;
import android.widget.ImageButton;
import android.widget.ImageView;

import com.hcflab.mbroadcast.factory.AppGlobals;
import com.hcflab.mbroadcast.factory.R;
import com.myscript.iink.Configuration;
import com.myscript.iink.ContentPackage;
import com.myscript.iink.ContentPart;
import com.myscript.iink.ConversionState;
import com.myscript.iink.Editor;
import com.myscript.iink.Engine;
import com.myscript.iink.IEditorListener;
import com.myscript.iink.MimeType;
import com.myscript.iink.PointerEvent;
import com.myscript.iink.PointerEventType;
import com.myscript.iink.PointerType;
import com.myscript.iink.uireferenceimplementation.EditorView;
import com.myscript.iink.uireferenceimplementation.InputController;
import com.wacom.cdlcore.RawPoint;

import java.io.File;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.OutputStreamWriter;
import java.util.ArrayList;

public class MainActivity extends AppCompatActivity implements View.OnClickListener
{
  private static final String TAG = "MainActivity";

  private Engine engine;
  private ContentPackage contentPackage;
  private ContentPart contentPart;
  private EditorView editorView;
  ImageView convert, convert2;
  DocumentController documentController;
    ArrayList<RawPoint[]> saved_strokes = new ArrayList<RawPoint[]>();
  @Override
  protected void onCreate(@Nullable Bundle savedInstanceState)
  {
    super.onCreate(savedInstanceState);

    ErrorActivity.installHandler(this);

    engine = IInkApplication.getEngine();

    // configure recognition
    Configuration conf = engine.getConfiguration();
    String confDir = "zip://" + getPackageCodePath() + "!/assets/conf";
    conf.setStringArray("configuration-manager.search-path", new String[]{confDir});
    String tempDir = getFilesDir().getPath() + File.separator + "tmp";
    conf.setString("content-package.temp-folder", tempDir);

    setContentView(R.layout.myscript_main);

    editorView = findViewById(R.id.editor_view);
    convert = findViewById(R.id.convert);
    convert2 = findViewById(R.id.convert2);
    editorView.setEngine(engine);

    /*
      o implement, the preview, you should get the text result from the jiix file, as follows:
  -first, ensure you have an iink Editor properly initialized, such as in the code samples
          -Create a content package: ContentPackage pkg = engine.createPackage(« myfile.iink »)
          -Create a text content part: ContentPart part = pkg.createPart(« Text »)
          -Set this part to the editor: Editor.setPart(part)
          -Feed the editor with your ink using editor.pointerEvents(eventArray, false), where eventArray is your ink converted into pointer events (pointerDown, pointerMove+, pointerUp) and false means the gestures where deactivated
          -wait for the recognition to end: editor.waitForIdle()
          -export in JIIX format: editor.export_(MimeType.JIIX)

          => You can then get words with corresponding input range. This way, to a give ink, you can match the corresponding words.
    */



    convert2.setOnClickListener(new View.OnClickListener() {
        @Override
        public void onClick(View view) {
            Editor editor = editorView.getEditor();
            final File path =
                    Environment.getExternalStoragePublicDirectory
                            (
                                    File.separator + getResources().getString(R.string.app_name) + File.separator + "myscript" + File.separator
                            );

            // Make sure the path directory exists.
            if(!path.exists())
            {
                // Make it, if it doesn't exit
                path.mkdirs();
            }
            final File file2 = new File(path, "conversion.txt");

            saved_strokes.clear();
            int k = 0;
            saved_strokes =AppGlobals.saved_strokes2;
            int size=0, size2=0;
            for (int z=0;z<saved_strokes.size();z++){
                size+=saved_strokes.get(z).length;
            }

            String json ="";
            PointerEvent[] pointerEvents2 = new PointerEvent[size + 1];
            for (int i=0; i<saved_strokes.size(); i++){
                for (int j=0; j<saved_strokes.get(i).length; j++){

                    if (j==0) {
                        pointerEvents2[k] = new PointerEvent(PointerEventType.DOWN, saved_strokes.get(i)[j].x, saved_strokes.get(i)[j].y, saved_strokes.get(i)[j].timestampInMillis, 1.0f, PointerType.PEN, 1);
                    }else if (j==(saved_strokes.get(i).length-1)) {
                        pointerEvents2[k] = new PointerEvent(PointerEventType.UP, saved_strokes.get(i)[j].x, saved_strokes.get(i)[j].y, saved_strokes.get(i)[j].timestampInMillis, 1.0f, PointerType.PEN, 1);
                    }else {
                        pointerEvents2[k] = new PointerEvent(PointerEventType.MOVE, saved_strokes.get(i)[j].x, saved_strokes.get(i)[j].y, saved_strokes.get(i)[j].timestampInMillis, 1.0f, PointerType.PEN, 1);
                    }
                    k+=1;
                }
            }

            try {

                ContentPackage pkg = engine.createPackage(file2);
                ContentPart part = pkg.createPart("Text");
                editor.setPart(part);
                editor.pointerEvents(pointerEvents2, false);
                editor.waitForIdle();
                try {
                    editor.export_(null, MimeType.TEXT);
                } catch (IOException e) {
                    e.printStackTrace();
                }
            } catch (IOException e) {
                e.printStackTrace();
            }
        }
    });

    final Editor editor = editorView.getEditor();
    editor.addListener(new IEditorListener()
    {
      @Override
      public void partChanging(Editor editor, ContentPart oldPart, ContentPart newPart)
      {
        // no-op
      }

      @Override
      public void partChanged(Editor editor)
      {
        invalidateOptionsMenu();
        invalidateIconButtons();
      }

      @Override
      public void contentChanged(Editor editor, String[] blockIds)
      {
        invalidateOptionsMenu();
        invalidateIconButtons();
      }

      @Override
      public void onError(Editor editor, String blockId, String message)
      {
        Log.e(TAG, "Failed to edit block \"" + blockId + "\"" + message);
      }
    });

    setInputMode(InputController.INPUT_MODE_FORCE_PEN); // If using an active pen, put INPUT_MODE_AUTO here

    String packageName = "File2.iink";
    File file = new File(getFilesDir(), packageName);
    try
    {
      contentPackage = engine.createPackage(file);
      contentPart = contentPackage.createPart("Diagram"); // Choose type of content (possible values are: "Text Document", "Text", "Diagram", "Math", and "Drawing")
    }
    catch (IOException e)
    {
      Log.e(TAG, "Failed to open package \"" + packageName + "\"", e);
    }
    catch (IllegalArgumentException e)
    {
      Log.e(TAG, "Failed to open package \"" + packageName + "\"", e);
    }

    setTitle("Type: " + contentPart.getType());

    // wait for view size initialization before setting part
    editorView.post(new Runnable()
    {
      @Override
      public void run()
      {
        editorView.getRenderer().setViewOffset(0, 0);
        editorView.getRenderer().setViewScale(1);
        editorView.setVisibility(View.VISIBLE);
        editor.setPart(contentPart);
      }
    });

    findViewById(R.id.button_input_mode_forcePen).setOnClickListener(this);
    findViewById(R.id.button_input_mode_forceTouch).setOnClickListener(this);
    findViewById(R.id.button_input_mode_auto).setOnClickListener(this);
    findViewById(R.id.button_undo).setOnClickListener(this);
    findViewById(R.id.button_redo).setOnClickListener(this);
    findViewById(R.id.button_clear).setOnClickListener(this);

    invalidateIconButtons();
  }

    public void writeToFile2(String data)
    {
        // Get the directory for the user's public pictures directory.
        final File path =
                Environment.getExternalStoragePublicDirectory
                        (
                                File.separator + getResources().getString(R.string.app_name) + File.separator + "myscript" + File.separator
                        );

        // Make sure the path directory exists.
        if(!path.exists())
        {
            // Make it, if it doesn't exit
            path.mkdirs();
        }
        final File file2 = new File(path, "points.txt");
        if (file2.exists()){
            file2.delete();
        }
        final File file = new File(path, "points.txt");


        // Save your stream, don't forget to flush() it before closing it.

        try
        {
            file.createNewFile();
            FileOutputStream fOut = new FileOutputStream(file,true);
            OutputStreamWriter myOutWriter = new OutputStreamWriter(fOut);
            myOutWriter.append(data);

            myOutWriter.close();

            fOut.flush();
            fOut.close();
        }
        catch (IOException e)
        {
            Log.e("Exception", "File write failed: " + e.toString());
        }
    }

  public String to_string(int [] z){
      if (z.length > 0) {
          StringBuilder nameBuilder = new StringBuilder();

          for (int n : z) {
              nameBuilder.append("'").append(n).append("',");
//              nameBuilder.append("'").append(n.replace("'", "\\'")).append("',");
              // can also do the following
              // nameBuilder.append("'").append(n.replace("'", "''")).append("',");
          }

          nameBuilder.deleteCharAt(nameBuilder.length() - 1);

          return nameBuilder.toString();
      }else{
          return "";
      }
  }


  public String to_stringl(long [] x){
      if (x.length > 0) {
          StringBuilder nameBuilder = new StringBuilder();

          for (long n : x) {
              nameBuilder.append("'").append(n).append("',");
//              nameBuilder.append("'").append(n.replace("'", "\\'")).append("',");
              // can also do the following
              // nameBuilder.append("'").append(n.replace("'", "''")).append("',");
          }

          nameBuilder.deleteCharAt(nameBuilder.length() - 1);

          return nameBuilder.toString();
      }else{
          return "";
      }
  }

  @Override
  protected void onDestroy()
  {
    editorView.setOnTouchListener(null);
    editorView.close();

    if (contentPart != null)
    {
      contentPart.close();
      contentPart = null;
    }
    if (contentPackage != null)
    {
      contentPackage.close();
      contentPackage = null;
    }


    // IInkApplication has the ownership, do not close here
    engine = null;

    super.onDestroy();
  }

  @Override
  public boolean onCreateOptionsMenu(Menu menu)
  {
    getMenuInflater().inflate(R.menu.myscript_menu, menu);

    MenuItem convertMenuItem = menu.findItem(R.id.menu_convert);
    convertMenuItem.setEnabled(true);

    return super.onCreateOptionsMenu(menu);
  }

  @Override
  public boolean onOptionsItemSelected(MenuItem item)
  {
    switch (item.getItemId())
    {
      case R.id.menu_convert:
      {
        Editor editor = editorView.getEditor();
        ConversionState[] supportedStates = editor.getSupportedTargetConversionStates(null);
        if (supportedStates.length > 0)
          editor.convert(null, supportedStates[0]);
        return true;
      }
      default:
      {
        return super.onOptionsItemSelected(item);
      }
    }
  }

  @Override
  public void onClick(View v)
  {
    switch (v.getId())
    {
      case R.id.button_input_mode_forcePen:
        setInputMode(InputController.INPUT_MODE_FORCE_PEN);
        break;
      case R.id.button_input_mode_forceTouch:
        setInputMode(InputController.INPUT_MODE_FORCE_TOUCH);
        break;
      case R.id.button_input_mode_auto:
        setInputMode(InputController.INPUT_MODE_AUTO);
        break;
      case R.id.button_undo:
        editorView.getEditor().undo();
        break;
      case R.id.button_redo:
        editorView.getEditor().redo();
        break;
      case R.id.button_clear:
        editorView.getEditor().clear();
        break;
      default:
        Log.e(TAG, "Failed to handle click event");
        break;
    }
  }

  private void setInputMode(int inputMode)
  {
    editorView.setInputMode(inputMode);
    findViewById(R.id.button_input_mode_forcePen).setEnabled(inputMode != InputController.INPUT_MODE_FORCE_PEN);
    findViewById(R.id.button_input_mode_forceTouch).setEnabled(inputMode != InputController.INPUT_MODE_FORCE_TOUCH);
    findViewById(R.id.button_input_mode_auto).setEnabled(inputMode != InputController.INPUT_MODE_AUTO);
  }

  private void invalidateIconButtons()
  {
    Editor editor = editorView.getEditor();
    final boolean canUndo = editor.canUndo();
    final boolean canRedo = editor.canRedo();
    runOnUiThread(new Runnable()
    {
      @Override
      public void run()
      {
        ImageButton imageButtonUndo = (ImageButton) findViewById(R.id.button_undo);
        imageButtonUndo.setEnabled(canUndo);
        ImageButton imageButtonRedo = (ImageButton) findViewById(R.id.button_redo);
        imageButtonRedo.setEnabled(canRedo);
        ImageButton imageButtonClear = (ImageButton) findViewById(R.id.button_clear);
        imageButtonClear.setEnabled(contentPart != null);
      }
    });
  }
}

 

Dear Jai,


First, regarding the MyScriptJS not accurate, this is likely because the guidelines are activated by default, while they should not be in off-screen usage. This can be done setting the guidelines option to false ("text.guides.enable" option to "false"). You should also ensure to set it to false in your JAVA project.


Regarding your JAVA code, after looking at it, it looks correct. I am nevertheless puzzled by your "saved_strokes" variable. What does it look like? When doing "size+=saved_strokes.get(z).length;" what is the value of the size variable? When filling your pointerEvents2 array, is it properly filled? Indeed, I am thinking it may not be properly filled. Can you please check it carefully?


Best regards,


Olivier

iam not able to download this app pls help it says wrong email address when i click on certificate where as email id is correct 

Dear Nadeem,


I am not sure to understand your message.


Is it related to the current topic? If not, please let us know, as we want to avoid topics to be mixed.


Otherwise, I checked your account, and I confirm it was properly created with the email address you used for this post.


Also, what are you say you are "not able to download this app", which application are you referring to? Which error message do you exactly have? Feel free to attach a screenshot if it helps understand.


Best regards,


Olivier

Dear Olivier,

thanks for your solution on MyScriptJS text.guides.enable to false did solved my problem, now conversion is quite accurate.

>I am nevertheless puzzled by your "saved_strokes" variable. What does it look like? 

and on java saved_strokes is the array_list of strokes array

image

>When doing "size+=saved_strokes.get(z).length;" what is the value of the size variable?

here each item represent a stroke and each stroke contains a collection of x-y coordinates which you can see here 98, 112, 46.. so so size+=saved_strokes.get(z).length it give sum of all x-y points of all stroke just to initialise the pointerEvents array so size for above case will be 540 as instance.

>When filling your pointerEvents2 array, is it properly filled? Indeed, I am thinking it may not be properly filled. Can you please check it carefully?

pointsEvents2 array is accurate according to me as the size also equals the 540 equal to above points and same data now fed on MyScriptJS it does return accurate conversion, so i am not sure what is wrong here.

I do appreciate your time and effort please help this crash solution

Regards 

Jai Singh Hada

Hello Olivier,

I also want  to ask how to get multi line text as recognition result as for now it returns single line text regardless input strokes were multi line.

Is there any way to make inputTextType to multi_line_text kind of something as i couldn't find for MyScriptJS.

image

is textInputMode i found related can it work?

Thanks and regards

Jai Singh Hada


Dear Jai,


after reviewing your code, the issue occurs because yur array has an empty element.


When declaring the pointerEvent2 arrays, you have one element over: PointerEvent[] pointerEvents2 = new PointerEvent[size + 1];

Instead, you should declare it as follows: PointerEvent[] pointerEvents2 = new PointerEvent[size + 1];


Regarding your second question, you do not have the possibility to force to multi-line mode. This is managed internally.


Best regards,


Olivier



1 person likes this

Dear Olivier,

Thanks for all your prompt support it does solved all my issues.


Thanks with regards,

Jai Singh Hada

Login or Signup to post a comment