MediaPipe in TouchDesigner 4

The following example is a simple demonstration of the Face Mesh function from MediaPipe in TouchDesigner. It is very similar to the previous face detection example. Again, we are going to use the Script TOP to integrate with MediaPipe and display the face mesh information together with the live webcam image.

Instead of flipping the image vertically in the Python code, this version will perform the flipping in the TouchDesigner Flip TOP, both vertically and horizontally (mirror image). We also reduce the resolution from the original 1280 x 720 to 640 x 360 for better performance. The Face Mesh information is drawn directly to the output image in the Script TOP.

Here is also the Python code in the Script TOP

# me - this DAT
# scriptOp - the OP which is cooking
import numpy
import cv2
import mediapipe as mp

mp_drawing = mp.solutions.drawing_utils
mp_face_mesh = mp.solutions.face_mesh

point_spec = mp_drawing.DrawingSpec(
    color=(0, 100, 255),
    thickness=1,
    circle_radius=1
)
line_spec = mp_drawing.DrawingSpec(
    color=(255, 200, 0),
    thickness=2,
    circle_radius=1
)
face_mesh = mp_face_mesh.FaceMesh(
    min_detection_confidence=0.5,
    min_tracking_confidence=0.5
)

# press 'Setup Parameters' in the OP to call this function to re-create the parameters.
def onSetupParameters(scriptOp):
    page = scriptOp.appendCustomPage('Custom')
    p = page.appendFloat('Valuea', label='Value A')
    p = page.appendFloat('Valueb', label='Value B')
    return

# called whenever custom pulse parameter is pushed
def onPulse(par):
    return

def onCook(scriptOp):
    input = scriptOp.inputs[0].numpyArray(delayed=True)
    if input is not None:
        frame = input * 255
        frame = frame.astype('uint8')
        frame = cv2.cvtColor(frame, cv2.COLOR_RGBA2RGB)
        results = face_mesh.process(frame)
        if results.multi_face_landmarks:
            for face_landmarks in results.multi_face_landmarks:
                mp_drawing.draw_landmarks(
                    image=frame,
                    landmark_list=face_landmarks,
                    connections=mp_face_mesh.FACE_CONNECTIONS,
                    landmark_drawing_spec=point_spec,
                    connection_drawing_spec=line_spec)

       frame = cv2.cvtColor(frame, cv2.COLOR_RGB2RGBA)
       scriptOp.copyNumpyArray(frame)
    return

Similar to previous examples, the important code is in the onCook function. The face_mesh will process each frame and draw the results in the frame instance for final display.

The TouchDesigner project is now available in the MediaPipeFaceMeshTOP folder of the GitHub repository.