{"id":3458,"date":"2022-04-15T22:00:38","date_gmt":"2022-04-15T14:00:38","guid":{"rendered":"http:\/\/www.magicandlove.com\/blog\/?p=3458"},"modified":"2022-04-15T22:25:05","modified_gmt":"2022-04-15T14:25:05","slug":"segmentation-mask-with-mediapipe-in-touchdesigner","status":"publish","type":"post","link":"http:\/\/www.magicandlove.com\/blog\/2022\/04\/15\/segmentation-mask-with-mediapipe-in-touchdesigner\/","title":{"rendered":"Segmentation Mask with MediaPipe in TouchDesigner"},"content":{"rendered":"\n<p>The tutorial is an updated version of the <a rel=\"noreferrer noopener\" href=\"https:\/\/google.github.io\/mediapipe\/solutions\/pose.html\" target=\"_blank\">MediaPipe Pose <\/a>using the new segmentation mask function to identify the human body tracked. It used the <a rel=\"noreferrer noopener\" href=\"https:\/\/docs.derivative.ca\/Script_TOP\" target=\"_blank\">Script TOP<\/a> to generate the mask image. Users can further enhance the image with the <a rel=\"noreferrer noopener\" href=\"https:\/\/docs.derivative.ca\/Threshold_TOP\" target=\"_blank\">Threshold TOP<\/a> for display purpose. Similar to the previous tutorials, it assumes the installation of Python 3.7 and the MediaPipe library through Pip. <\/p>\n\n\n\n<p>The source TouchDesigner project file is available in my <a rel=\"noreferrer noopener\" href=\"https:\/\/github.com\/chungbwc\/TouchDesigner\/tree\/main\/SegmentationMask\" target=\"_blank\">TouchDesigner GitHub repository<\/a>. The Python code is relatively straighforward. The pose tracking results will include an array (segmentation_mask) of the size of the tracked image. Each pixel will have a value between 0.0 to 1.0. Darker value will be the background while brighter value will likely be the tracked body. Here is the full listing.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code># me - this DAT\n# scriptOp - the OP which is cooking\n\nimport numpy as np\nimport cv2\nimport mediapipe as mp\n\nmp_drawing = mp.solutions.drawing_utils\nmp_pose = mp.solutions.pose\n\npose = mp_pose.Pose(\n\tmin_detection_confidence=0.5,\n\tmin_tracking_confidence=0.5,\n\tenable_segmentation=True\n)\n\ndef onSetupParameters(scriptOp):\n\treturn\n\n# called whenever custom pulse parameter is pushed\ndef onPulse(par):\n\treturn\n\ndef onCook(scriptOp):\n\tinput = scriptOp.inputs&#91;0].numpyArray(delayed=True)\n\tif input is not None:\n\t\timage = cv2.cvtColor(input, cv2.COLOR_RGBA2RGB)\n\t\timage *= 255\n\t\timage = image.astype('uint8')\n\t\tresults = pose.process(image)\n\t\t\n\t\tif results.segmentation_mask is not None:\n\t\t\trgb = cv2.cvtColor(results.segmentation_mask, cv2.COLOR_GRAY2RGB)\n\t\t\trgb = rgb * 255\n\t\t\trgb = rgb.astype(np.uint8)\n\t\t\tscriptOp.copyNumpyArray(rgb)\n\t\telse:\n\t\t\tblack = np.zeros(image.shape, dtype=np.uint8)\n\t\t\tscriptOp.copyNumpyArray(black)\n\treturn<\/code><\/pre>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"576\" src=\"http:\/\/www.magicandlove.com\/blog\/wp-content\/uploads\/2022\/04\/SegmentationMask-1024x576.jpg\" alt=\"Segmentation mask in MediaPipe\" class=\"wp-image-3465\" srcset=\"http:\/\/www.magicandlove.com\/blog\/wp-content\/uploads\/2022\/04\/SegmentationMask-1024x576.jpg 1024w, http:\/\/www.magicandlove.com\/blog\/wp-content\/uploads\/2022\/04\/SegmentationMask-300x169.jpg 300w, http:\/\/www.magicandlove.com\/blog\/wp-content\/uploads\/2022\/04\/SegmentationMask-768x432.jpg 768w, http:\/\/www.magicandlove.com\/blog\/wp-content\/uploads\/2022\/04\/SegmentationMask-690x388.jpg 690w, http:\/\/www.magicandlove.com\/blog\/wp-content\/uploads\/2022\/04\/SegmentationMask-980x551.jpg 980w, http:\/\/www.magicandlove.com\/blog\/wp-content\/uploads\/2022\/04\/SegmentationMask.jpg 1280w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n","protected":false},"excerpt":{"rendered":"<p>The tutorial is an updated version of the MediaPipe Pose using the new segmentation mask function to identify the human body tracked. It used the Script TOP to generate the mask image. Users can further enhance the image with the Threshold TOP for display purpose. Similar to the previous tutorials, it assumes the installation of [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[179],"tags":[180,189,84],"class_list":["post-3458","post","type-post","status-publish","format-standard","hentry","category-tutorials","tag-mediapipe","tag-segmentation-mask","tag-touchdesigner"],"_links":{"self":[{"href":"http:\/\/www.magicandlove.com\/blog\/wp-json\/wp\/v2\/posts\/3458","targetHints":{"allow":["GET"]}}],"collection":[{"href":"http:\/\/www.magicandlove.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/www.magicandlove.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/www.magicandlove.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/www.magicandlove.com\/blog\/wp-json\/wp\/v2\/comments?post=3458"}],"version-history":[{"count":5,"href":"http:\/\/www.magicandlove.com\/blog\/wp-json\/wp\/v2\/posts\/3458\/revisions"}],"predecessor-version":[{"id":3467,"href":"http:\/\/www.magicandlove.com\/blog\/wp-json\/wp\/v2\/posts\/3458\/revisions\/3467"}],"wp:attachment":[{"href":"http:\/\/www.magicandlove.com\/blog\/wp-json\/wp\/v2\/media?parent=3458"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/www.magicandlove.com\/blog\/wp-json\/wp\/v2\/categories?post=3458"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/www.magicandlove.com\/blog\/wp-json\/wp\/v2\/tags?post=3458"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}