Skip to main content

Frame Processors

What are frame processors?#

Frame processors are functions that are written in JavaScript (or TypeScript) which can be used to process frames the camera "sees". Inside those functions you can call Frame Processor Plugins, which are high performance native functions specifically designed for certain use-cases.

For example, you might want to create a Hotdog/Not Hotdog detector app without writing any native code, while still achieving native performance:

function App() {  const frameProcessor = useFrameProcessor((frame) => {    'worklet'    const isHotdog = detectIsHotdog(frame)    console.log(isHotdog ? "Hotdog!" : "Not Hotdog.")  }, [])
  return (    <Camera      {...cameraProps}      frameProcessor={frameProcessor}    />  )}

Frame processors are by far not limited to Hotdog detection, other examples include:

  • AI for facial recognition
  • AI for object detection
  • Using Tensorflow, MLKit Vision, Apple Vision or other libraries
  • Creating realtime video-chats using WebRTC to directly send the camera frames over the network
  • Creating scanners for QR codes, Barcodes or even custom codes such as Snapchat's SnapCodes or Apple's AppClips
  • Creating snapchat-like filters, e.g. draw a dog-mask filter over the user's face
  • Creating color filters with depth-detection

Because they are written in JS, Frame Processors are simple, powerful, extensible and easy to create while still running at native performance. (Frame Processors can run up to 1000 times a second!) Also, you can use fast-refresh to quickly see changes while developing or publish over-the-air updates to tweak the Hotdog detector's sensitivity in live apps without pushing a native update.

note

Frame Processors require react-native-reanimated 2.2.0 or higher. Also make sure to add

import 'react-native-reanimated'

to the top of the file when using useFrameProcessor.

Interacting with Frame Processors#

Since Frame Processors run in Worklets, you can directly use JS values such as React state:

// can be controlled with a Sliderconst [sensitivity, setSensitivity] = useState(0.4)
const frameProcessor = useFrameProcessor((frame) => {  'worklet'  const isHotdog = detectIsHotdog(frame, sensitivity)  console.log(isHotdog ? "Hotdog!" : "Not Hotdog.")}, [sensitivity])

You can also easily read from, and assign to Shared Values to create smooth, 60 FPS animations.

In this example, we detect a cat in the frame - if a cat was found, we assign the catBounds Shared Value to the coordinates of the cat (relative to the screen) which we can then use in a useAnimatedStyle hook to position the red rectangle surrounding the cat. This updates in realtime on the UI Thread, and can also be smoothly animated with withSpring or withTiming.

// represents position of the cat on the screen ๐Ÿˆconst catBounds = useSharedValue({ top: 0, left: 0, right: 0, bottom: 0 })
// continously sets 'catBounds' to the cat's coordinates on screenconst frameProcessor = useFrameProcessor((frame) => {  'worklet'  catBounds.value = scanFrameForCat(frame)}, [catBounds])
// uses 'catBounds' to position the red rectangle on screen.// smoothly updates on UI thread whenever 'catBounds' is changedconst boxOverlayStyle = useAnimatedStyle(() => ({  position: 'absolute',  borderWidth: 1,  borderColor: 'red',  ...catBounds.value}), [catBounds])
return (  <View>    <Camera {...cameraProps} frameProcessor={frameProcessor} />    // draws a red rectangle on the screen which surrounds the cat    <Reanimated.View style={boxOverlayStyle} />  </View>)

And you can also call back to the React-JS thread by using runOnJS:

const onQRCodeDetected = useCallback((qrCode: string) => {  navigation.push("ProductPage", { productId: qrCode })}, [])
const frameProcessor = useFrameProcessor((frame) => {  'worklet'  const qrCodes = scanQRCodes(frame)  if (qrCodes.length > 0) {    runOnJS(onQRCodeDetected)(qrCodes[0])  }}, [onQRCodeDetected])

Using Frame Processor Plugins#

Frame Processor Plugins are distributed through npm. To install the vision-camera-image-labeler plugin, run:

npm i vision-camera-image-labelercd ios && pod install

Then add it to your babel.config.js. For the Image Labeler, this will be __labelImage:

module.exports = {  plugins: [    [      'react-native-reanimated/plugin',      {        globals: ['__labelImage'],      },    ],
note

You have to restart metro-bundler for changes in the babel.config.js file to take effect.

That's it! ๐ŸŽ‰ Now you can use it:

const frameProcessor = useFrameProcessor((frame) => {  'worklet'  const labels = labelImage(frame)  // ...}, [])

Check out Frame Processor community plugins to discover plugins, or start creating a plugin yourself!

Benchmarks#

Frame Processors are really fast. I have used MLKit Vision Image Labeling to label 4k Camera frames in realtime, and measured the following results:

  • Fully natively (written in pure Objective-C, no React interaction at all), I have measured an average of 68ms per call.
  • As a Frame Processor Plugin (written in Objective-C, called through a JS Frame Processor function), I have measured an average of 69ms per call.

This means that the Frame Processor API only takes ~1ms longer than a fully native implementation, making it the fastest and easiest way to run any sort of Frame Processing in React Native.

All measurements are recorded on an iPhone 11 Pro, benchmarked total execution time of the captureOutput function by using CFAbsoluteTimeGetCurrent. Running smaller images (lower than 4k resolution) is much quicker and many algorithms can even run at 60 FPS.

Avoiding Frame-drops#

Frame Processors will be synchronously called for each frame the Camera sees and have to finish executing before the next frame arrives, otherwise the next frame(s) will be dropped. For a frame rate of 30 FPS, you have about 33ms to finish processing frames. Use frameProcessorFps to throttle the frame processor's FPS. For a QR Code Scanner, 5 FPS (200ms) might suffice, while a object tracking AI might run at the same frame rate as the Camera itself (e.g. 60 FPS (16ms)).

ESLint react-hooks plugin#

If you are using the react-hooks ESLint plugin, make sure to add useFrameProcessor to additionalHooks inside your ESLint config. (See "advanced configuration")

Technical#

Frame Processors#

Frame Processors are JS functions that will be workletized using react-native-reanimated. They are created on a parallel camera thread using a separate JavaScript Runtime ("VisionCamera JS-Runtime") and are invoked synchronously (using JSI) without ever going over the bridge. In a Frame Processor you can write normal JS code, call back to the React-JS Thread (e.g. setState), use Shared Values and call Frame Processor Plugins.

See the example Frame Processor

Frame Processor Plugins#

Frame Processor Plugins are native functions (written in Objective-C, Swift, C++, Java or Kotlin) that are injected into the VisionCamera JS-Runtime. They can be synchronously called from your JS Frame Processors (using JSI) without ever going over the bridge. Because VisionCamera provides an easy-to-use plugin API, you can easily create a Frame Processor Plugin yourself. Some examples include Barcode Scanning, Face Detection, Image Labeling, Text Recognition and more.

Learn how to create Frame Processor Plugins, or check out the example Frame Processor Plugin for iOS or Android.

The Frame object#

The Frame Processor gets called with a Frame object, which is a JSI HostObject. It holds a reference to the native (C++) Frame Image Buffer (~10 MB in size) and exposes properties such as width, height, bytesPerRow and more to JavaScript so you can synchronously access them. The Frame object can be passed around in JS, as well as returned from- and passed to a native Frame Processor Plugin.

See this tweet for more information.

Disabling Frame Processors#

The Frame Processor API spawns a secondary JavaScript Runtime which consumes a small amount of extra CPU and RAM. If you're not using Frame Processors at all, you can disable them by setting the VISION_CAMERA_DISABLE_FRAME_PROCESSORS flag. Inside your project.pbxproj, find the GCC_PREPROCESSOR_DEFINITIONS parameter and add the flag:

GCC_PREPROCESSOR_DEFINITIONS = (  "DEBUG=1",  "VISION_CAMERA_DISABLE_FRAME_PROCESSORS=1",  "$(inherited)",);

Make sure to add this to your Debug-, as well as your Release-configuration.


๐Ÿš€ Next section: Zooming (or creating a Frame Processor Plugin)#