{"id":3244,"date":"2021-05-24T13:23:23","date_gmt":"2021-05-24T05:23:23","guid":{"rendered":"http:\/\/www.magicandlove.com\/blog\/?p=3244"},"modified":"2021-05-24T18:47:20","modified_gmt":"2021-05-24T10:47:20","slug":"mediapipe-in-touchdesigner-2","status":"publish","type":"post","link":"http:\/\/www.magicandlove.com\/blog\/2021\/05\/24\/mediapipe-in-touchdesigner-2\/","title":{"rendered":"MediaPipe in TouchDesigner 2"},"content":{"rendered":"\n

Now we are ready to integrate the MediaPipe functions in TouchDesigner after we learnt the basic of the Script TOP. The first one we are going to do is the Face Detection<\/a>. We just use the Script TOP to display the bounding boxes of the detected faces without sending the face details elsewhere for processing. In the next example after this, we shall send the bounding box details to a Script CHOP<\/a>. <\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

In order to have the mirror image effect, we use the Flip TOP<\/a> with a horizontal flip. We also add a Resolution TOP<\/a> to reduce the original 1280 x 720 to half, i.e. 640 x 360 for better performance. Of course, we can achieve the same result by changing the Output Resolution of the Flip TOP from its Common tab.<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n
# me - this DAT\n# scriptOp - the OP which is cooking\nimport numpy\nimport cv2\nimport mediapipe as mp\n\nmp_face = mp.solutions.face_detection\nmp_drawing = mp.solutions.drawing_utils\n\nface = mp_face.FaceDetection(\n     min_detection_confidence=0.7\n )\n\n# press 'Setup Parameters' in the OP to call this function to re-create the parameters.\ndef onSetupParameters(scriptOp):\n    return\n# called whenever custom pulse parameter is pushed\ndef onPulse(par):\n    return\n\ndef onCook(scriptOp):\n    input = scriptOp.inputs[0].numpyArray(delayed=True)\n    if input is not None:\n        frame = cv2.cvtColor(input, cv2.COLOR_RGBA2RGB)\n        frame = cv2.flip(frame, 0)\n        frame *= 255\n        frame = frame.astype('uint8')\n        results = face.process(frame)\n        if results.detections:\n            for detection in results.detections:\n                mp_drawing.draw_detection(frame, detection)\n\n        frame = cv2.flip(frame, 0)\n        scriptOp.copyNumpyArray(frame)\n    return\n<\/pre>\n\n\n\n
In the first place, we need to import MediaPipe into the Python code. The next step is to define a few variables to work with the face detection, mp_face<\/strong> and visualisation of the detected face, mp_drawing<\/strong>, and finally the face detection class instance, face<\/strong>, with the detection confidence value.<\/p>\n\n\n\n
To process the video, we also convert the RGBA frame into RGB only. It is found that the image format MediaPipe face detection expected is vertically flipped as compared with the TouchDesigner TOP. In the Python code, we first flip the image vertically before sending it to the face detection with face.process(frame)<\/strong>. After the mp_drawing<\/strong> utility draws the detection results onto the frame<\/strong>, we also flip the image vertically again for output to the Script TOP. The object, results.detections<\/strong> contains all the details of the detected faces. Each face will be visualised with a bounding box and 6 dots indicating the two ears, eyes, nose tip and the mouth centre.<\/p>\n\n\n\n
The TouchDesigner project file is in this GitHub repository<\/a>.<\/p>\n","protected":false},"excerpt":{"rendered":"
Now we are ready to integrate the MediaPipe functions in TouchDesigner after we learnt the basic of the Script TOP. The first one we are going to do is the Face Detection. We just use the Script TOP to display the bounding boxes of the detected faces without sending the face details elsewhere for processing. […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[179],"tags":[180,182,84],"class_list":["post-3244","post","type-post","status-publish","format-standard","hentry","category-tutorials","tag-mediapipe","tag-python","tag-touchdesigner"],"_links":{"self":[{"href":"http:\/\/www.magicandlove.com\/blog\/wp-json\/wp\/v2\/posts\/3244","targetHints":{"allow":["GET"]}}],"collection":[{"href":"http:\/\/www.magicandlove.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/www.magicandlove.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/www.magicandlove.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/www.magicandlove.com\/blog\/wp-json\/wp\/v2\/comments?post=3244"}],"version-history":[{"count":16,"href":"http:\/\/www.magicandlove.com\/blog\/wp-json\/wp\/v2\/posts\/3244\/revisions"}],"predecessor-version":[{"id":3264,"href":"http:\/\/www.magicandlove.com\/blog\/wp-json\/wp\/v2\/posts\/3244\/revisions\/3264"}],"wp:attachment":[{"href":"http:\/\/www.magicandlove.com\/blog\/wp-json\/wp\/v2\/media?parent=3244"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/www.magicandlove.com\/blog\/wp-json\/wp\/v2\/categories?post=3244"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/www.magicandlove.com\/blog\/wp-json\/wp\/v2\/tags?post=3244"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}