MediaPipe in TouchDesigner 7

This example is the continuation of the last post using hand tracking in MediaPipe with TouchDesigner. This version will use a Script CHOP, instead of a Script TOP. The CHOP will produce channels related to the x and y positions of the Wrist and the Index Finger Tip. We can make use of the numbers to create interactive animation accordingly.

The MediaPipe hand tracking solution will generate 21 landmarks including all positions of the 5 fingers and the wrist. Details of the 21 landmarks are in the following diagram.

For simplicity, the example only detects one hand. The indices 0 and 8 correspond to the WRIST and the INDEX_FINGER_TIP respectively. The following code segment illustrates how it generates the channels for the Script CHOP.

wrist = []
index_tip = []
num_hands = 0
if results.multi_hand_landmarks:
    num_hands += 1
    for hand in results.multi_hand_landmarks:
        wrist.append(hand.landmark[0])
        index_tip.append(hand.landmark[8])


tf = scriptOp.appendChan('hands')
tf.vals = [num_hands]

if len(wrist) > 0:
    twx = scriptOp.appendChan('wrist:x')
    twy = scriptOp.appendChan('wrist:y')

    twx.vals = [wrist[0].x]
    twy.vals = [wrist[0].y]

if len(index_tip) > 0:
    tix = scriptOp.appendChan('index_tip:x') 
    tiy = scriptOp.appendChan('index_tip:y')

    tix.vals = [index_tip[0].x]  
    tiy.vals = [index_tip[0].y]

scriptOp.rate = me.time.rate