Interactive Ink

iink on iOS without rendering?

Hi folks!

Any way to get rid of the rendering? While playing with the "GetStartedSwift" project, I figured out how to initiate the engine and how to create the IINKEditor. But it seems the editor needs an IINKRenderer as well as an FontMetricsProvider (both seems very rendering specific). My app uses an own approach on rendering. So I like to provide only the data points. Any ideas?

Best regards

Chris


Hi Chris,


The Interactive Ink SDK API requires an IINKRenderer and a FontMetricsProvider to be provided to the editor. The dpi information provided by the renderer, for instance, is important to get a sense of the scale of what was written and will impact the quality of the recognition; the font metrics provider is required in case typeset data needs to be layouted at some point (certain cases of import or export). However, you do not need to display anything: the IINKRenderer asks for a render target: you can provide your own object implementing IINKIRenderTarget with methods that do nothing.


Best regards,

Thomas

Hi Thomas,

I'm having the same use-case - I just want to put some strokes in and get a word out. My current implementation looks like this follows but I always get "" (empty string) in return:

FontMetricsProvider, minimal implementation, I don't want to render anything: 

public class FontMetricsProvider: NSObject, IINKIFontMetricsProvider {

    public func getCharacterBoundingBoxes(_ text: IINKText!, spans: [IINKTextSpan]!) -> [NSValue]! {
        return []
    }

    public func getFontSizePx(_ style: IINKStyle!) -> Float {
        return style.fontSize
    }

}

My custom engine initializer (I wrapped the static lib in a dynamic framework because I need it in multiple other frameworks):

extension IINKEngine {

    public convenience init?(certificateData: Data = Data(bytes: myCertificate.bytes, count: myCertificate.length), configurationPath: String = "/recognition-assets/conf") {
        self.init(certificate: certificateData)

        // Configure the iink runtime environment
        let configurationPath = Bundle(for: IINKEngine.self).bundlePath.appending(configurationPath)
        do {
            try configuration.setStringArray("configuration-manager.search-path", value: [configurationPath]) // Tells the engine where to load the recognition assets from.
        } catch {
            print("Should not happen, please check your resources assets : " + error.localizedDescription)
            return nil
        }

        // Set the temporary directory
        do {
            try configuration.setString("content-package.temp-folder", value: NSTemporaryDirectory().appending("/HandwritingRecognitionAssets"))
        } catch {
            print("Failed to set temporary folder: " + error.localizedDescription)
            return nil
        }
    }

}

This is the meat:  

public class HandwritingRecognizer {

    private let engine: IINKEngine? = IINKEngine()
    private lazy var editor: IINKEditor? = {
        let scaledDPI = Float(264 / UIScreen.main.scale)
        guard let renderer = self.engine?.createRenderer(withDpiX: scaledDPI, dpiY: scaledDPI, target: nil) else { return nil }
        let editor = self.engine?.createEditor(renderer)
        editor?.setFontMetricsProvider(FontMetricsProvider())
        return editor
    }()

    private lazy var package = self.createPackage(packageName: UUID().uuidString)

    private var currentPart: IINKContentPart?

    init() {}

    public func processStrokes() {
        guard let editor = self.editor else {
            return
        }

        editor.setViewSize(CGSize(width: 5, height: 10))

        addH()
        do {
            editor.part = try package?.getPart(currentPart!.identifier)
        } catch {
            print("Error while creating package : " + error.localizedDescription)
        }
        let text = try? editor.export_(nil, mimeType: .text)
        print(text)
    }

    private func addH() {
//        // left
//        let eventDown1 = IINKPointerEvent(eventType: .down, x: 0, y: 0,  t: 1, f: 1, pointerType: .pen, pointerId: 0)
//        let eventMove1 = IINKPointerEvent(eventType: .move, x: 0, y: 10, t: 2, f: 1, pointerType: .pen, pointerId: 0)
//        let eventUp1   = IINKPointerEvent(eventType: .up, x: 0,   y: 10, t: 3, f: 1, pointerType: .pen, pointerId: 0)
//
//        // right
//        let eventDown2 = IINKPointerEvent(eventType: .down, x: 5, y: 0,  t: 4, f: 1, pointerType: .pen, pointerId: 0)
//        let eventMove2 = IINKPointerEvent(eventType: .move, x: 5, y: 10, t: 5, f: 1, pointerType: .pen, pointerId: 0)
//        let eventUp2   = IINKPointerEvent(eventType: .up,   x: 5, y: 10, t: 6, f: 1, pointerType: .pen, pointerId: 0)
//
//        // middle
//        let eventDown3 = IINKPointerEvent(eventType: .down, x: 0, y: 5, t: 7, f: 1, pointerType: .pen, pointerId: 0)
//        let eventMove3 = IINKPointerEvent(eventType: .move, x: 5, y: 5, t: 8, f: 1, pointerType: .pen, pointerId: 0)
//        let eventUp3   = IINKPointerEvent(eventType: .up,   x: 5, y: 5, t: 9, f: 1, pointerType: .pen, pointerId: 0)
//
//        var hEvents = [eventDown1, eventMove1, eventUp1,
//                       eventDown2, eventMove2, eventUp2,
//                       eventDown3, eventMove3, eventUp3]
//        try! editor?.pointerEvents(&hEvents, count: hEvents.count, doProcessGestures: false)

        try! editor?.pointerDown(CGPoint(x: 0, y: 0),  at: 1, force: 1, type: .pen, pointerId: 0, error: nil)
        try! editor?.pointerMove(CGPoint(x: 0, y: 10), at: 2, force: 1, type: .pen, pointerId: 0)
        try! editor?.pointerUp(CGPoint(x: 0, y: 10), at: 3, force: 1, type: .pen, pointerId: 0)

        // right
        try! editor?.pointerDown(CGPoint(x: 5, y: 0), at: 4, force: 1, type: .pen, pointerId: 0, error: nil)
        try! editor?.pointerMove(CGPoint(x: 5, y: 10), at: 5, force: 1, type: .pen, pointerId: 0)
        try! editor?.pointerUp(CGPoint(x: 5, y: 10), at: 6, force: 1, type: .pen, pointerId: 0)

        // middle
        try! editor?.pointerDown(CGPoint(x: 0, y: 5), at: 7, force: 1, type: .pen, pointerId: 0, error: nil)
        try! editor?.pointerMove(CGPoint(x: 5, y: 5), at: 8, force: 1, type: .pen, pointerId: 0)
        try! editor?.pointerUp(CGPoint(x: 5, y: 5), at: 9, force: 1, type: .pen, pointerId: 0)
    }

    func createPackage(packageName: String) -> IINKContentPackage? {
        // Create a new content package with name
        let urlWithExtension = FileManager.default.temporaryDirectory
            .appendingPathComponent("com.company.product.handwriting", isDirectory: true)
            .appendingPathComponent(packageName, isDirectory: false)
            .appendingPathExtension("iink")

        try! FileManager.default.createDirectory(at: urlWithExtension.deletingLastPathComponent(), withIntermediateDirectories: true, attributes: nil)
        let fullPath = urlWithExtension.absoluteString.replacingOccurrences(of: "file://", with: "").decomposedStringWithCanonicalMapping
        guard let engine = engine else { return nil }
        var resultPackage: IINKContentPackage?
        do {
            resultPackage = try engine.createPackage(fullPath)
        } catch {
            do {
                resultPackage = try engine.openPackage(fullPath)
            } catch {
                resultPackage = nil
            }
        }
        do {
            currentPart = try resultPackage?.createPart("Text")
        } catch {
            currentPart = nil
        }
        return resultPackage
    }

}

  However, when processStrokes() gets called, it always prints "" (the empty string). I think I'm missing something but I don't know what.

Is my current understanding correct?

A "package" is a sorted collection of parts (acc. to the docs), i.e. assuming one uses documents it would be the whole document.

A "part" is whatever the user writes in one piece, e.g. a word or a complete sentence.

A "block" is some sub-thingy of a part, maybe a letter, maybe a number, maybe a part of a word (or a letter?), I usually don't care about this.

===

In the example code above a create a package for everything the engine should recognize (because I don't want the engine to have the context of the anything else), just feed some strokes and get a word (or list of candidates) out. I don't get any errors logged so I assume my setup is correct (it would log a message if it didn't have any recognition-assets (I use analyzer, en_US, math and shape), right?). I find it really hard to debug this because I don't see how I can get the points of an editor (to see if I fed them correctly into it), etc.

Any help is very much appreciated, thank you! 

Hi,

If you keep your proposed analogy "package" = "document", then you have "part" = "page", a page being of a given type (some text, a diagram, some math, etc.). A block would be a semantic subdivision of a page, for instance a paragraph.

I see two potential issues with the code:

  1. I am under the impression that you send the events to the editor before you attach the part to it. You should do the other way round. The part "gathers" the data iink will work on.
  2. As recognition may take some time to complete, you need to call waitForIdle on the editor object before attempting to export.

To get configuration error, you should implement the onError method of IINKEditorDelegate. It will raise an error whether you attempt to use an invalid configuration.

I hope this will help,

Best regards,
Thomas

2 people like this

Hi Thomas,

Thank you very much for your explanation and your pointers! After I adjusted the code according to your remarks it works perfectly fine! :)

Login or Signup to post a comment