Before we start using MediaPipe in TouchDesigner, we need to be familiar with the use of the Script TOP and Script CHOP first. For the Script TOP, we can generate the image (TOP) directly from Python code. In the following example, we are going to pass through the incoming image from Video Device In TOP to the output window with minimal manipulation in Python inside the Script TOP. The OpenCV in TouchDesigner reference page in the Derivative website is a good starting point.

We create a very simple TouchDesigner project, connecting the Video Device In to the Script TOP and then to the Output window. Note that the Script TOP comes with an associated Script Text DAT. We are going to modify the default Python code inside this text area with the name script1_callbacks.

We can directly edit the Python code inside the Text DAT by turning on the Viewer Active button in the bottom right corner. Alternately, we can click the Edit button in the parameter window to open the code in your default code editor, XCode in my case.

# me - this DAT
# scriptOp - the OP which is cooking
import numpy as np
# press 'Setup Parameters' in the OP to call this function to re-create the parameters.
def onSetupParameters(scriptOp):
     page = scriptOp.appendCustomPage('Custom')
     p = page.appendFloat('Valuea', label='Value A')
     p = page.appendFloat('Valueb', label='Value B')
     return

# called whenever custom pulse parameter is pushed
def onPulse(par):
    return

def onCook(scriptOp):
    image = scriptOp.inputs[0].numpyArray(delayed=True, writable=True)
    image *= 255
    image = image.astype('uint8')
    scriptOp.copyNumpyArray(image)
    return

The code has 3 functions, onSetupParameters, onPulse and onCook. We only use the onCook for this example. Cooking is the update of a node when necessary for very frame. The detailed explanation can be found from the TouchDesigner Cook page. Essentially, we can consider it as frame by frame update of the node we are working on. The first function, onSetupParameters is triggered by a button in the parameter window under the Setup tab. We can consider it the initialisation of the process. The second function, onPulse, will not be used here since we do not have any Pulse button or Pulse parameters defined here. We are going to walk through the simple onCook function.

In the first line, scriptOp (the current node), will retrieve its first input, 0, (the Video Device In) and convert the current video frame in a NumPy array. The format of the array is Height x Width x RGBA. Each colour pixel is a 32 bit floating point number within the range of 0 to 1. In our case, the video size is 1280 x 720. The 2 optional parameters, delayed=True and writable=True will be explained in the TOP class reference. In this example, we aim to convert the 32 bit floating point colour format to 8 bit unsigned integer for output.

In the second line, each colour pixel will multiply 255 by itself to convert the colour range between 0 to 255.

The third line, the NumPy array is modified into 8 bit unsigned integer format, uint8.

The last line will copy back the NumPy array, with the function copyNumpyArray, into the Script TOP texture for output.

The final TouchDesigner project can be downloaded from this GitHub repository.