Author Archives: bryan

Segmentation Mask with MediaPipe in TouchDesigner

The tutorial is an updated version of the MediaPipe Pose using the new segmentation mask function to identify the human body tracked. It used the Script TOP to generate the mask image. Users can further enhance the image with the Threshold TOP for display purpose. Similar to the previous tutorials, it assumes the installation of Python 3.7 and the MediaPipe library through Pip.

The source TouchDesigner project file is available in my TouchDesigner GitHub repository. The Python code is relatively straighforward. The pose tracking results will include an array (segmentation_mask) of the size of the tracked image. Each pixel will have a value between 0.0 to 1.0. Darker value will be the background while brighter value will likely be the tracked body. Here is the full listing.

# me - this DAT
# scriptOp - the OP which is cooking

import numpy as np
import cv2
import mediapipe as mp

mp_drawing =
mp_pose =

pose = mp_pose.Pose(

def onSetupParameters(scriptOp):

# called whenever custom pulse parameter is pushed
def onPulse(par):

def onCook(scriptOp):
	input = scriptOp.inputs[0].numpyArray(delayed=True)
	if input is not None:
		image = cv2.cvtColor(input, cv2.COLOR_RGBA2RGB)
		image *= 255
		image = image.astype('uint8')
		results = pose.process(image)
		if results.segmentation_mask is not None:
			rgb = cv2.cvtColor(results.segmentation_mask, cv2.COLOR_GRAY2RGB)
			rgb = rgb * 255
			rgb = rgb.astype(np.uint8)
			black = np.zeros(image.shape, dtype=np.uint8)
Segmentation mask in MediaPipe

Face Mirror (2021)

It is a site specific interactive installation created for the 2nd UNSCHEDULED Art Fair, Karin Weber Gallery. The artwork made use of the mirror of the former changing room of the Top Shop, Central. It performs a face detection to convert the visitor’s face into an animated figure, resembling the yesterday cyberspace creature trapped inside a mirror box. The exhibition period is from 02 to 06 September 2021 in the G/F, 1/F, Asia Standard Tower, 59-65 Queen’s Road, Central.

1st documentation video
Final version of documentation video

Face detection with Dlib in TouchDesigner

The example will continue to use a Script CHOP, Python and TouchDesigner for a face detection function. Instead of using the MediaPipe library, it will use the Dlib Python binding. It refers to the face detector example program from the Dlib distribution. Dlib is a popular C++ based programming toolkit for various applications. Its image processing library contains a number of face detection functions. Python binding is also available.

The main face detection capability is defined in the following statements.

import dlib

detector = dlib.get_frontal_face_detector()

rects = detector(image, 0)

The Script CHOP will generate the following channels

  • cx (centre of the rectangle – horizontal)
  • cy (centre of the rectangle – vertical)
  • width
  • height

for the largest face it detected from the live image.

The complete project is available in the FaceDetectionDlib1 GitHub folder.

MediaPipe in TouchDesigner 10

This is the last part of the series, using MediaPipe in TouchDesigner. The following example is a continuation of the last post of pose tracking. This version will use a Script CHOP to output the position information of the torso tracked in the film sequence. The output window will display four numbers (11, 12, 23, 24) on the four corners of the torso. The four numbers are the indices of the pose landmarks corresponding to the torso of the body.

The Script CHOP will output 3 channels

  • pose:x
  • pose:y
  • pose:visibility

Each channel has 33 samples, corresponding to the 33 pose landmarks. The visibility channel will indicate how likely the landmark is visible in the image. The following code segment describes how it is done.

xpos = []
ypos = []
visb = []

if results.pose_landmarks:
    for p in results.pose_landmarks.landmark:

    tx = scriptOp.appendChan('pose:x')         
    ty = scriptOp.appendChan('pose:y')         
    tv = scriptOp.appendChan('pose:visibility')
    tx.vals = xpos         
    ty.vals = ypos         
    tv.vals = visb
    scriptOp.rate = me.time.rate         
    scriptOp.numSamples = len(xpos)

The final TouchDesigner project folder MediaPipePoseCHOP is now available in the GitHub repository.

MediaPipe in TouchDesigner 9

The following example illustrates the Pose Tracking solution in the Google MediaPipe, using TouchDesigner. It will display the tracking result in a Script TOP. Instead of using the live Video Device In TOP, it uses the Movie File In to track the dancing movement from 2 film clips. The project also makes use of a Keyboard In CHOP to switch between the 2 film clips.

The project does not resize the original film clip with a Resolution TOP. It performs the resize function within the Python code in the Script TOP with the OpenCV function cv2.resize(). Each pose detected will generate 33 pose landmarks. The details can be found from the following diagram.

Image from the Google MediaPipe

Together with the original video image, the drawing utility will generate the pose skeleton with the following code segment.

    image, results.pose_landmarks, mp_pose.POSE_CONNECTIONS)

The final TouchDesigner project is available in the GitHub folder as MediaPipePoseTOP. Owing to file size and copyright concerns, the two film clips are not included in GitHub.

MediaPipe in TouchDesigner 8

The following example presents a more general approach to obtain the hand tracking details in a Script CHOP. We can then use other TouchDesigner CHOPs to extract the data for visualisation.

For simplicity, it also detects one single hand. For each hand tracked, it will generate 21 landmarks as shown in the diagram from the last post. The Script CHOP will produce 2 channels, hand:x and hand:y. Each of the channel will have 21 samples, corresponding to the 21 hand landmarks from MediaPipe. The following code segment describes how it is done.

detail_x = []
detail_y = []
if results.multi_hand_landmarks:
    for hand in results.multi_hand_landmarks:
        for pt in hand.landmark:

    tx = scriptOp.appendChan('hand:x')   
    ty = scriptOp.appendChan('hand:y')  
    tx.vals = detail_x   
    ty.vals = detail_y  
    scriptOp.numSamples = len(detail_x)

scriptOp.rate = me.time.rate

The TouchDesigner project also uses Shuffle CHOP to swap the 21 samples into 21 channels. We can then select the 5 channels corresponding to the 5 finger tips (4, 8, 12, 16, 20) for visualisation. The final project is available for download in the MediaPipeHandCHOP2 folder of the GitHub repository.

MediaPipe in TouchDesigner 7

This example is the continuation of the last post using hand tracking in MediaPipe with TouchDesigner. This version will use a Script CHOP, instead of a Script TOP. The CHOP will produce channels related to the x and y positions of the Wrist and the Index Finger Tip. We can make use of the numbers to create interactive animation accordingly.

The MediaPipe hand tracking solution will generate 21 landmarks including all positions of the 5 fingers and the wrist. Details of the 21 landmarks are in the following diagram.

Image from the Google MediaPipe

For simplicity, the example only detects one hand. The indices 0 and 8 correspond to the WRIST and the INDEX_FINGER_TIP respectively. The following code segment illustrates how it generates the channels for the Script CHOP.

wrist = []
index_tip = []
num_hands = 0
if results.multi_hand_landmarks:
    num_hands += 1
    for hand in results.multi_hand_landmarks:

tf = scriptOp.appendChan('hands')
tf.vals = [num_hands]

if len(wrist) > 0:
    twx = scriptOp.appendChan('wrist:x')
    twy = scriptOp.appendChan('wrist:y')

    twx.vals = [wrist[0].x]
    twy.vals = [wrist[0].y]

if len(index_tip) > 0:
    tix = scriptOp.appendChan('index_tip:x') 
    tiy = scriptOp.appendChan('index_tip:y')

    tix.vals = [index_tip[0].x]  
    tiy.vals = [index_tip[0].y]

scriptOp.rate = me.time.rate

MediaPipe in TouchDesigner 6

This tutorial introduces the use of hand tracking in the Google MediaPipe with TouchDesigner. Similar to the previous posts, part 1 of hand tracking will just be a visualisation of the hand details from a Script TOP. It will use the MediaPipe drawing utility to display the hand details directly onto the Video Device In image for output.

The TouchDesigner project can now be downloaded from the MediaPipeHandTOP GitHub directory.