Capacitor Plugin for Text Detection Part 3 : Web Implementation of the Plugin

Vennela Kodali April 8, 2020

This is part 3/6 of the "Capacitor Plugin for Text Detection" series.

In the previous post, we created an iOS plugin. In this post, while we aren't quite implementing a web plugin, we'll modify the web plugin to reflect the fact, and also add a shim for our native plugins.

Web implementation of the plugin

We aren't implementing a web plugin as part of this tutorial, but the out-of-the-box plugin came with a web implementation as well. If we leave it as it is, when the application is run on the web, it'll error out because it can't find detectText function in the web plugin. To avoid that, we'll add a detectText method but not quite implement it. We'll throw an error mentioning that the web plugin hasn't been implemented.

(Optional) In addition, we're also going to add a shim between our native plugins and client code here.

Step 1

Web implementation of the plugin is located at cap-ml/src/

web.ts contains the code for the web plugin
index.ts exports the plugins, equivalent to Plugin.h we saw in the ios plugin
definitions.ts defines the interfaces for the plugin, equivalent to PLugin.m in the ios Plugin

Open web.ts and change the echo function to detectText, because that's what we're doing. We're rejecting the call right away because we aren't implementing a web plugin here.

async detectText(filepath: string, orientation?: ImageOrientation): Promise<TextDetection[]> {
    return Promise.reject("Web Plugin Not implemented")
  }

Step 2

Accordingly, we'd have the following modifications in definitions.ts as well. Since we aren't implementing anything on the web side,

update the PluginRegistry interface to

interface PluginRegistry {
  CapML: {};
}

remove the export CapMLPlugin ... that we aren't using
add export default {};

Step 3

Open index.ts and remove export * from './web.'

Step 4: (Optional) Add a shim to our 'text-detection' native plugins

In the src directory, create a folder text-detector
cd into text-detector and create a index.ts file
In index.ts -

Import our CapML plugin like

  import { Plugins } from '@capacitor/core';
  const { CapML } = Plugins;

Add a TextDetector class. This class acts as a shim between our native plugins and client side code. So client code can call TextDetector.detectText which in turn would call CapML.detectText

export class TextDetector implements TextDetectorInterface{
  async detectText(filename: string, orientation?: ImageOrientation): Promise<TextDetection[]> {
    const response = await CapML.detectText({filename, orientation})
    return response.textDetections
  }
}

Define an interface for our return type TextDetection. From the previous post where we implemented the ios plugin, we know that we're receiving an array of text detections with 4 bounding coordinates of a rectangle(or a skewed rectangle) and a string of text. So TextDetection will look like

export interface TextDetection {
  bottomLeft: [number, number]; // [x-coordinate, y-coordinate]
  bottomRight: [number, number]; // [x-coordinate, y-coordinate]
  topLeft: [number, number]; // [x-coordinate, y-coordinate]
  topRight: [number, number]; // [x-coordinate, y-coordinate]
  text: string;
}

and an enum for ImageOrientations. ImageOrientation provides us with orientation options to pass into the plugin. CoreML and MLKit are pretty good at detecting text at a certain angle, so handling 90deg rotations covers most scenarios.

export enum ImageOrientation {
  Up = "UP",
  Down = "DOWN",
  Left = "LEFT",
  Right = "RIGHT",
}

In src/index.ts, export the text-detector we just created

export * from './text-detector';

That wraps up our web implementation on the plugin side.

Having a shim like this, adds an additional step, but will help us enforce return types and certain parameter types like TextDetection and ImageOrientation. Also if we eventually add more functionality, for example like barcode detection, we can keep it separate for the user by creating another class like

export class BarcodeDetector implements BarcodeDetectorInterface {
  async detectBarcodes(...): Promise<BarcodeDetection[]> {
    const response = await CapML.detectBarcodes({...})
    return response.barcodeDetections
  }
}

User can then import which ever implementation they'd like to use into their client code, instead of having to import the entire plugin.

import { TextDetector, TextDetection, ImageOrientation } from 'cap-ml'; // for text detections
// or
import { BarcodeDetector, BarcodeDetection } from 'cap-ml'; // for barcode detections