Shape Detection API: Detecting barcodes, faces and text inside an image

Did you know that your device has the amazing ability to detect different shapes in images? Right now, whether you are on a mobile, tablet or desktop, it is capable of detecting faces, barcodes and even text in an image. Pretty amazing, huh?!

According to the HTTP archive, images make up around 60% of all file types on the web, which is huge when you consider the scale of the web. Being able to detect shapes inside of images is quite computationally expensive and often needs to be completed on the server. There are some great online services such as Amazon’s Rekognition, Google’s Cloud Vision API and Microsoft’s Computer Vision API that provide this functionality, but it can be costly depending on your usage. While hardware manufacturers have been supporting these features for a long time, Web Apps do not yet have access to these hardware capabilities, which makes the use of computationally demanding libraries or third party APIs necessary.

This is where the Shape Detection API comes in. It provides developers with a standard interface on top of some underlying hardware features on the user’s device that can be accessed directly via the web. This opens up a world of possibilities for web developers and allows you to truly tap into the power of the hardware that exists on mobile devices.

Using this API, you could build a barcode scanner, stock checker, or even produce fun effects like Snapchat Lenses. There are so many options!

With just a few lines of code, the Shape Detection API gives you the ability to detect barcodes, faces and even text inside of an image. In this article, I am going to run through two examples; one that will detect faces inside of an image, and another that will detect the text inside an image.

Face Detection

By adding the following code to your web page, the Shape Detection API will allow you to determine any faces inside of an image. The API will return the number of faces detected and their location in the image as a bounding box.

The code above might seem a little scary, but it’s easier than you think when we break it down. Firstly, we get a reference to the image on the page - this is the image that we are going to use to pass to the API to determine any faces inside of it. Next, we invoke the detect function on the FaceDetector API and pass through the image. The API then returns us a list of faces that it found inside the array and their details. With the results from the API, I am then looping through the faces and simply print out the count and their heights and widths.

You could even take this a step further and draw a rectangle around the images if you wanted to display them on the image. If you’d like to see a working example of this in action, I recommend checking out the example on the WICG documentation to see more.

Text Detection

Let’s try another one of the features of the Shape Detection API; this time using text detection.

In the code above, we start off by getting a reference to an image in the HTML. Next, we invoke the API using the detect function on the TextDetector API that will return a JavaScript Promise with a list of the text found inside the image. From there, I am simply logging the results to the console.

If you’d like to experiment with this using your own device, I recommend checking out the WICG example to see this in action.

Browser Support

While the code above shows basic examples, it does demonstrate the power of the Shape Detection API and how easily you can get started experimenting with this functionality. This code is built upon the Shape Detection API which is currently in the WICG which means it is still in incubation and in an experimentation phase. At the time of writing this article, both Opera and Chrome support this feature, although it is hidden behind a feature flag. Firefox have shown public support and it will be great to see this spread to many other browsers.

If you’d like to start experimenting with the different features available, I recommend using the Codepen snippets on the WICG document for Face Detection, Barcode Detection and Text Detection. Before the detection will work, you will need to visit chrome://flags/#enable-experimental-web-platform-features and enable the feature.

At this years Google I/O, the Shape Detection API was talked about as one of the many great features that allow us as web developers to tap into the hardware capabilities of mobile devices. I recommend watching this video where Alex Russell explains it in action.

Summary

There is so many great features coming to the web and I’m keen to see how developers will use it to build amazing web apps. If you’d like to learn more about the Shape detection API, I thoroughly recommend reading the WICG documentation. Paul Kinlan also wrote a great article about using the Shape Detection API to determine barcodes inside of an images.