iink SDK on Device

Answered

unable to convert pointerEvents to text in android or accurately at MyScriptJS

I have been trying to convert x-y coordinates obtained from the wacom clipboard my handwritten text-


image

i have converted it to PointerEvents as-


image

follows as how i obtained pointer events array from clipboard strokes

pointerEvents2[k] = new PointerEvent(PointerEventType.CANCEL, saved_strokes.get(i)[j].x, saved_strokes.get(i)[j].y, saved_strokes.get(i)[j].timestampInMillis, 1.0f, PointerType.PEN, 1);


editor.pointerEvents(pointerEvents2,false);
editor.waitForIdle();
try {
    editor.export_(null, MimeType.TEXT);
} catch (IOException e) {
    e.printStackTrace();
}


*Editor have been initialised and all but i still get error at editor.pointerEvents(pointerEvents2,false);

Please help me out where it got wrong.

I also tried same text points conversion on MyScriptJs but it isn't accurate with words its doing fine with the characters-


image

at-


https://myscript.github.io/MyScriptJS/examples/v4/pointer_events.html

txt

Best Answer

Dear Jai,


after reviewing your code, the issue occurs because yur array has an empty element.


When declaring the pointerEvent2 arrays, you have one element over: PointerEvent[] pointerEvents2 = new PointerEvent[size + 1];

Instead, you should declare it as follows: PointerEvent[] pointerEvents2 = new PointerEvent[size + 1];


Regarding your second question, you do not have the possibility to force to multi-line mode. This is managed internally.


Best regards,


Olivier



1 person has this question

Hello Olivier,

I am attaching the class and few snapshots of data in arrays .

image

below are the savedstroke data for reference

image

image

 

// Copyright MyScript. All rights reserved.

package com.hcflab.mbroadcast.factory.myscript.iink.getstarted;

import android.media.Image;
import android.os.Bundle;
import android.os.Environment;
import android.support.annotation.Nullable;
import android.support.v7.app.AppCompatActivity;
import android.util.Log;
import android.view.Menu;
import android.view.MenuItem;
import android.view.View;
import android.widget.ImageButton;
import android.widget.ImageView;

import com.hcflab.mbroadcast.factory.AppGlobals;
import com.hcflab.mbroadcast.factory.R;
import com.myscript.iink.Configuration;
import com.myscript.iink.ContentPackage;
import com.myscript.iink.ContentPart;
import com.myscript.iink.ConversionState;
import com.myscript.iink.Editor;
import com.myscript.iink.Engine;
import com.myscript.iink.IEditorListener;
import com.myscript.iink.MimeType;
import com.myscript.iink.PointerEvent;
import com.myscript.iink.PointerEventType;
import com.myscript.iink.PointerType;
import com.myscript.iink.uireferenceimplementation.EditorView;
import com.myscript.iink.uireferenceimplementation.InputController;
import com.wacom.cdlcore.RawPoint;

import java.io.File;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.OutputStreamWriter;
import java.util.ArrayList;

public class MainActivity extends AppCompatActivity implements View.OnClickListener
{
  private static final String TAG = "MainActivity";

  private Engine engine;
  private ContentPackage contentPackage;
  private ContentPart contentPart;
  private EditorView editorView;
  ImageView convert, convert2;
  DocumentController documentController;
    ArrayList<RawPoint[]> saved_strokes = new ArrayList<RawPoint[]>();
  @Override
  protected void onCreate(@Nullable Bundle savedInstanceState)
  {
    super.onCreate(savedInstanceState);

    ErrorActivity.installHandler(this);

    engine = IInkApplication.getEngine();

    // configure recognition
    Configuration conf = engine.getConfiguration();
    String confDir = "zip://" + getPackageCodePath() + "!/assets/conf";
    conf.setStringArray("configuration-manager.search-path", new String[]{confDir});
    String tempDir = getFilesDir().getPath() + File.separator + "tmp";
    conf.setString("content-package.temp-folder", tempDir);

    setContentView(R.layout.myscript_main);

    editorView = findViewById(R.id.editor_view);
    convert = findViewById(R.id.convert);
    convert2 = findViewById(R.id.convert2);
    editorView.setEngine(engine);

    /*
      o implement, the preview, you should get the text result from the jiix file, as follows:
  -first, ensure you have an iink Editor properly initialized, such as in the code samples
          -Create a content package: ContentPackage pkg = engine.createPackage(« myfile.iink »)
          -Create a text content part: ContentPart part = pkg.createPart(« Text »)
          -Set this part to the editor: Editor.setPart(part)
          -Feed the editor with your ink using editor.pointerEvents(eventArray, false), where eventArray is your ink converted into pointer events (pointerDown, pointerMove+, pointerUp) and false means the gestures where deactivated
          -wait for the recognition to end: editor.waitForIdle()
          -export in JIIX format: editor.export_(MimeType.JIIX)

          => You can then get words with corresponding input range. This way, to a give ink, you can match the corresponding words.
    */



    convert2.setOnClickListener(new View.OnClickListener() {
        @Override
        public void onClick(View view) {
            Editor editor = editorView.getEditor();
            final File path =
                    Environment.getExternalStoragePublicDirectory
                            (
                                    File.separator + getResources().getString(R.string.app_name) + File.separator + "myscript" + File.separator
                            );

            // Make sure the path directory exists.
            if(!path.exists())
            {
                // Make it, if it doesn't exit
                path.mkdirs();
            }
            final File file2 = new File(path, "conversion.txt");

            saved_strokes.clear();
            int k = 0;
            saved_strokes =AppGlobals.saved_strokes2;
            int size=0, size2=0;
            for (int z=0;z<saved_strokes.size();z++){
                size+=saved_strokes.get(z).length;
            }

            String json ="";
            PointerEvent[] pointerEvents2 = new PointerEvent[size + 1];
            for (int i=0; i<saved_strokes.size(); i++){
                for (int j=0; j<saved_strokes.get(i).length; j++){

                    if (j==0) {
                        pointerEvents2[k] = new PointerEvent(PointerEventType.DOWN, saved_strokes.get(i)[j].x, saved_strokes.get(i)[j].y, saved_strokes.get(i)[j].timestampInMillis, 1.0f, PointerType.PEN, 1);
                    }else if (j==(saved_strokes.get(i).length-1)) {
                        pointerEvents2[k] = new PointerEvent(PointerEventType.UP, saved_strokes.get(i)[j].x, saved_strokes.get(i)[j].y, saved_strokes.get(i)[j].timestampInMillis, 1.0f, PointerType.PEN, 1);
                    }else {
                        pointerEvents2[k] = new PointerEvent(PointerEventType.MOVE, saved_strokes.get(i)[j].x, saved_strokes.get(i)[j].y, saved_strokes.get(i)[j].timestampInMillis, 1.0f, PointerType.PEN, 1);
                    }
                    k+=1;
                }
            }

            try {

                ContentPackage pkg = engine.createPackage(file2);
                ContentPart part = pkg.createPart("Text");
                editor.setPart(part);
                editor.pointerEvents(pointerEvents2, false);
                editor.waitForIdle();
                try {
                    editor.export_(null, MimeType.TEXT);
                } catch (IOException e) {
                    e.printStackTrace();
                }
            } catch (IOException e) {
                e.printStackTrace();
            }
        }
    });

    final Editor editor = editorView.getEditor();
    editor.addListener(new IEditorListener()
    {
      @Override
      public void partChanging(Editor editor, ContentPart oldPart, ContentPart newPart)
      {
        // no-op
      }

      @Override
      public void partChanged(Editor editor)
      {
        invalidateOptionsMenu();
        invalidateIconButtons();
      }

      @Override
      public void contentChanged(Editor editor, String[] blockIds)
      {
        invalidateOptionsMenu();
        invalidateIconButtons();
      }

      @Override
      public void onError(Editor editor, String blockId, String message)
      {
        Log.e(TAG, "Failed to edit block \"" + blockId + "\"" + message);
      }
    });

    setInputMode(InputController.INPUT_MODE_FORCE_PEN); // If using an active pen, put INPUT_MODE_AUTO here

    String packageName = "File2.iink";
    File file = new File(getFilesDir(), packageName);
    try
    {
      contentPackage = engine.createPackage(file);
      contentPart = contentPackage.createPart("Diagram"); // Choose type of content (possible values are: "Text Document", "Text", "Diagram", "Math", and "Drawing")
    }
    catch (IOException e)
    {
      Log.e(TAG, "Failed to open package \"" + packageName + "\"", e);
    }
    catch (IllegalArgumentException e)
    {
      Log.e(TAG, "Failed to open package \"" + packageName + "\"", e);
    }

    setTitle("Type: " + contentPart.getType());

    // wait for view size initialization before setting part
    editorView.post(new Runnable()
    {
      @Override
      public void run()
      {
        editorView.getRenderer().setViewOffset(0, 0);
        editorView.getRenderer().setViewScale(1);
        editorView.setVisibility(View.VISIBLE);
        editor.setPart(contentPart);
      }
    });

    findViewById(R.id.button_input_mode_forcePen).setOnClickListener(this);
    findViewById(R.id.button_input_mode_forceTouch).setOnClickListener(this);
    findViewById(R.id.button_input_mode_auto).setOnClickListener(this);
    findViewById(R.id.button_undo).setOnClickListener(this);
    findViewById(R.id.button_redo).setOnClickListener(this);
    findViewById(R.id.button_clear).setOnClickListener(this);

    invalidateIconButtons();
  }

    public void writeToFile2(String data)
    {
        // Get the directory for the user's public pictures directory.
        final File path =
                Environment.getExternalStoragePublicDirectory
                        (
                                File.separator + getResources().getString(R.string.app_name) + File.separator + "myscript" + File.separator
                        );

        // Make sure the path directory exists.
        if(!path.exists())
        {
            // Make it, if it doesn't exit
            path.mkdirs();
        }
        final File file2 = new File(path, "points.txt");
        if (file2.exists()){
            file2.delete();
        }
        final File file = new File(path, "points.txt");


        // Save your stream, don't forget to flush() it before closing it.

        try
        {
            file.createNewFile();
            FileOutputStream fOut = new FileOutputStream(file,true);
            OutputStreamWriter myOutWriter = new OutputStreamWriter(fOut);
            myOutWriter.append(data);

            myOutWriter.close();

            fOut.flush();
            fOut.close();
        }
        catch (IOException e)
        {
            Log.e("Exception", "File write failed: " + e.toString());
        }
    }

  public String to_string(int [] z){
      if (z.length > 0) {
          StringBuilder nameBuilder = new StringBuilder();

          for (int n : z) {
              nameBuilder.append("'").append(n).append("',");
//              nameBuilder.append("'").append(n.replace("'", "\\'")).append("',");
              // can also do the following
              // nameBuilder.append("'").append(n.replace("'", "''")).append("',");
          }

          nameBuilder.deleteCharAt(nameBuilder.length() - 1);

          return nameBuilder.toString();
      }else{
          return "";
      }
  }


  public String to_stringl(long [] x){
      if (x.length > 0) {
          StringBuilder nameBuilder = new StringBuilder();

          for (long n : x) {
              nameBuilder.append("'").append(n).append("',");
//              nameBuilder.append("'").append(n.replace("'", "\\'")).append("',");
              // can also do the following
              // nameBuilder.append("'").append(n.replace("'", "''")).append("',");
          }

          nameBuilder.deleteCharAt(nameBuilder.length() - 1);

          return nameBuilder.toString();
      }else{
          return "";
      }
  }

  @Override
  protected void onDestroy()
  {
    editorView.setOnTouchListener(null);
    editorView.close();

    if (contentPart != null)
    {
      contentPart.close();
      contentPart = null;
    }
    if (contentPackage != null)
    {
      contentPackage.close();
      contentPackage = null;
    }


    // IInkApplication has the ownership, do not close here
    engine = null;

    super.onDestroy();
  }

  @Override
  public boolean onCreateOptionsMenu(Menu menu)
  {
    getMenuInflater().inflate(R.menu.myscript_menu, menu);

    MenuItem convertMenuItem = menu.findItem(R.id.menu_convert);
    convertMenuItem.setEnabled(true);

    return super.onCreateOptionsMenu(menu);
  }

  @Override
  public boolean onOptionsItemSelected(MenuItem item)
  {
    switch (item.getItemId())
    {
      case R.id.menu_convert:
      {
        Editor editor = editorView.getEditor();
        ConversionState[] supportedStates = editor.getSupportedTargetConversionStates(null);
        if (supportedStates.length > 0)
          editor.convert(null, supportedStates[0]);
        return true;
      }
      default:
      {
        return super.onOptionsItemSelected(item);
      }
    }
  }

  @Override
  public void onClick(View v)
  {
    switch (v.getId())
    {
      case R.id.button_input_mode_forcePen:
        setInputMode(InputController.INPUT_MODE_FORCE_PEN);
        break;
      case R.id.button_input_mode_forceTouch:
        setInputMode(InputController.INPUT_MODE_FORCE_TOUCH);
        break;
      case R.id.button_input_mode_auto:
        setInputMode(InputController.INPUT_MODE_AUTO);
        break;
      case R.id.button_undo:
        editorView.getEditor().undo();
        break;
      case R.id.button_redo:
        editorView.getEditor().redo();
        break;
      case R.id.button_clear:
        editorView.getEditor().clear();
        break;
      default:
        Log.e(TAG, "Failed to handle click event");
        break;
    }
  }

  private void setInputMode(int inputMode)
  {
    editorView.setInputMode(inputMode);
    findViewById(R.id.button_input_mode_forcePen).setEnabled(inputMode != InputController.INPUT_MODE_FORCE_PEN);
    findViewById(R.id.button_input_mode_forceTouch).setEnabled(inputMode != InputController.INPUT_MODE_FORCE_TOUCH);
    findViewById(R.id.button_input_mode_auto).setEnabled(inputMode != InputController.INPUT_MODE_AUTO);
  }

  private void invalidateIconButtons()
  {
    Editor editor = editorView.getEditor();
    final boolean canUndo = editor.canUndo();
    final boolean canRedo = editor.canRedo();
    runOnUiThread(new Runnable()
    {
      @Override
      public void run()
      {
        ImageButton imageButtonUndo = (ImageButton) findViewById(R.id.button_undo);
        imageButtonUndo.setEnabled(canUndo);
        ImageButton imageButtonRedo = (ImageButton) findViewById(R.id.button_redo);
        imageButtonRedo.setEnabled(canRedo);
        ImageButton imageButtonClear = (ImageButton) findViewById(R.id.button_clear);
        imageButtonClear.setEnabled(contentPart != null);
      }
    });
  }
}

 

Dear Jai,


we are facing difficulties understanding what's not fine on your side.  Indeed, you ink should normally be properly recognized.


Please attach your code, so that we can understand what is wrong in this latter one.


Best regards,


Olivier

Also at the MyScriptJS

https://myscript.github.io/MyScriptJS/examples/v4/pointer_events.html

if we feed  pointer events stroke by stroke it converts right but all strokes are fed via pointerEvents it converts with inaccuracy as i mentioned earlier so i need to know if there is a way to feed it with complete pointerEvents array and it provide accurate corresponding text

Thanks with regards

Jai Singh

Dear Olivier

Thanks for your time, I did converted the pointer events as you suggested but it results the same let me post the error too if it might help.

image

above is pointerEvents with down, move and up actions

File file = new File(getFilesDir(), packageName);

                ContentPackage pkg = engine.createPackage(file);

                ContentPart part = pkg.createPart("Text");

                editor.setPart(part);

                editor.pointerEvents(pointerEvents2, false);

                editor.waitForIdle();

                try {

                    editor.export_(null, MimeType.TEXT);

                } catch (IOException e) {

                    e.printStackTrace();

                } 

but i still get the error at NativeFunctions.pointerEvents

here is error screen-

image



Dear Jai Hada,


thank you for contacting us and your question.


Currently, we are a bit puzzled you are using the "Cancel" PointerEventType, as this latter cancels an ongoing pointer trace. You will then not get any recognition.


Currently, when adding coordinates in an "off-screen mode", each stroke should be added starting by a "DOWN" PointerEventType, and finish by a "UP" PointerEventType. All points in between should be added using a "MOVE" PointerEventType.


A stroke should then be added as explained in the "Series of events" part of our documentation: https://developer.myscript.com/docs/interactive-ink/1.2/android/fundamentals/editing/#series-of-events


Can you proceed as explained, and let us know if it doesn't work?


Best regards,


Olivier