{"id":3357,"date":"2021-06-03T21:27:29","date_gmt":"2021-06-03T13:27:29","guid":{"rendered":"http:\/\/www.magicandlove.com\/blog\/?p=3357"},"modified":"2021-06-03T21:42:56","modified_gmt":"2021-06-03T13:42:56","slug":"mediapipe-in-touchdesigner-9","status":"publish","type":"post","link":"http:\/\/www.magicandlove.com\/blog\/2021\/06\/03\/mediapipe-in-touchdesigner-9\/","title":{"rendered":"MediaPipe in TouchDesigner 9"},"content":{"rendered":"\n<p>The following example illustrates the <a rel=\"noreferrer noopener\" href=\"https:\/\/google.github.io\/mediapipe\/solutions\/pose.html\" target=\"_blank\">Pose Tracking<\/a> solution in the Google <a rel=\"noreferrer noopener\" href=\"https:\/\/google.github.io\/mediapipe\/\" target=\"_blank\">MediaPipe<\/a>, using <a rel=\"noreferrer noopener\" href=\"https:\/\/derivative.ca\/\" target=\"_blank\">TouchDesigner<\/a>. It will display the tracking result in a <a rel=\"noreferrer noopener\" href=\"https:\/\/docs.derivative.ca\/Script_TOP\" target=\"_blank\">Script TOP<\/a>. Instead of using the live <a rel=\"noreferrer noopener\" href=\"https:\/\/docs.derivative.ca\/Video_Device_In_TOP\" target=\"_blank\">Video Device In TOP<\/a>, it uses the <a rel=\"noreferrer noopener\" href=\"https:\/\/docs.derivative.ca\/Movie_File_In_TOP\" target=\"_blank\">Movie File In<\/a> to track the dancing movement from 2 film clips. The project also makes use of a <a rel=\"noreferrer noopener\" href=\"https:\/\/docs.derivative.ca\/Keyboard_In_CHOP\" target=\"_blank\">Keyboard In CHOP<\/a> to switch between the 2 film clips.<\/p>\n\n\n\n<figure class=\"wp-block-gallery columns-2 is-cropped wp-block-gallery-1 is-layout-flex wp-block-gallery-is-layout-flex\"><ul class=\"blocks-gallery-grid\"><li class=\"blocks-gallery-item\"><figure><a href=\"http:\/\/www.magicandlove.com\/blog\/wp-content\/uploads\/2021\/06\/PoseTOP1.png\"><img loading=\"lazy\" decoding=\"async\" width=\"300\" height=\"169\" src=\"http:\/\/www.magicandlove.com\/blog\/wp-content\/uploads\/2021\/06\/PoseTOP1-300x169.png\" alt=\"\" data-id=\"3360\" data-full-url=\"http:\/\/www.magicandlove.com\/blog\/wp-content\/uploads\/2021\/06\/PoseTOP1.png\" data-link=\"http:\/\/www.magicandlove.com\/blog\/2021\/06\/03\/mediapipe-in-touchdesigner-9\/posetop1\/\" class=\"wp-image-3360\" srcset=\"http:\/\/www.magicandlove.com\/blog\/wp-content\/uploads\/2021\/06\/PoseTOP1-300x169.png 300w, http:\/\/www.magicandlove.com\/blog\/wp-content\/uploads\/2021\/06\/PoseTOP1-1024x576.png 1024w, http:\/\/www.magicandlove.com\/blog\/wp-content\/uploads\/2021\/06\/PoseTOP1-768x432.png 768w, http:\/\/www.magicandlove.com\/blog\/wp-content\/uploads\/2021\/06\/PoseTOP1-1536x864.png 1536w, http:\/\/www.magicandlove.com\/blog\/wp-content\/uploads\/2021\/06\/PoseTOP1-2048x1152.png 2048w, http:\/\/www.magicandlove.com\/blog\/wp-content\/uploads\/2021\/06\/PoseTOP1-690x388.png 690w, http:\/\/www.magicandlove.com\/blog\/wp-content\/uploads\/2021\/06\/PoseTOP1-980x551.png 980w\" sizes=\"auto, (max-width: 300px) 100vw, 300px\" \/><\/a><\/figure><\/li><li class=\"blocks-gallery-item\"><figure><a href=\"http:\/\/www.magicandlove.com\/blog\/wp-content\/uploads\/2021\/06\/PoseTOP2.png\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"564\" src=\"http:\/\/www.magicandlove.com\/blog\/wp-content\/uploads\/2021\/06\/PoseTOP2-1024x564.png\" alt=\"\" data-id=\"3361\" data-full-url=\"http:\/\/www.magicandlove.com\/blog\/wp-content\/uploads\/2021\/06\/PoseTOP2.png\" data-link=\"http:\/\/www.magicandlove.com\/blog\/2021\/06\/03\/mediapipe-in-touchdesigner-9\/posetop2\/\" class=\"wp-image-3361\" srcset=\"http:\/\/www.magicandlove.com\/blog\/wp-content\/uploads\/2021\/06\/PoseTOP2-1024x564.png 1024w, http:\/\/www.magicandlove.com\/blog\/wp-content\/uploads\/2021\/06\/PoseTOP2-300x165.png 300w, http:\/\/www.magicandlove.com\/blog\/wp-content\/uploads\/2021\/06\/PoseTOP2-768x423.png 768w, http:\/\/www.magicandlove.com\/blog\/wp-content\/uploads\/2021\/06\/PoseTOP2-1536x846.png 1536w, http:\/\/www.magicandlove.com\/blog\/wp-content\/uploads\/2021\/06\/PoseTOP2-2048x1129.png 2048w, http:\/\/www.magicandlove.com\/blog\/wp-content\/uploads\/2021\/06\/PoseTOP2-690x380.png 690w, http:\/\/www.magicandlove.com\/blog\/wp-content\/uploads\/2021\/06\/PoseTOP2-980x540.png 980w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/a><\/figure><\/li><\/ul><\/figure>\n\n\n\n<p>The project does not resize the original film clip with a <a rel=\"noreferrer noopener\" href=\"https:\/\/docs.derivative.ca\/Resolution_TOP\" target=\"_blank\">Resolution TOP<\/a>. It performs the resize function within the Python code in the Script TOP with the <a rel=\"noreferrer noopener\" href=\"https:\/\/opencv.org\/\" target=\"_blank\">OpenCV<\/a> function <strong>cv2.resize()<\/strong>.  Each pose detected will generate 33 pose landmarks. The details can be found from the following diagram.<\/p>\n\n\n\n<figure class=\"wp-block-image size-large is-resized\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/google.github.io\/mediapipe\/images\/mobile\/pose_tracking_full_body_landmarks.png\" alt=\"\" width=\"772\" height=\"438\"\/><figcaption>Image from the Google MediaPipe<\/figcaption><\/figure>\n\n\n\n<p>Together with the original video image, the drawing utility will generate the pose skeleton with the following code segment.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">mp_drawing.draw_landmarks(\n    image, results.pose_landmarks, mp_pose.POSE_CONNECTIONS)<\/pre>\n\n\n\n<p>The final TouchDesigner project is available in the GitHub folder as <a href=\"https:\/\/github.com\/chungbwc\/TouchDesigner\/tree\/main\/MediaPipePoseTOP\" target=\"_blank\" rel=\"noreferrer noopener\">MediaPipePoseTOP<\/a>. Owing to file size and copyright concerns, the two film clips are not included in GitHub.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>The following example illustrates the Pose Tracking solution in the Google MediaPipe, using TouchDesigner. It will display the tracking result in a Script TOP. Instead of using the live Video Device In TOP, it uses the Movie File In to track the dancing movement from 2 film clips. The project also makes use of a [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[179],"tags":[180,185,182,84],"class_list":["post-3357","post","type-post","status-publish","format-standard","hentry","category-tutorials","tag-mediapipe","tag-pose-tracking","tag-python","tag-touchdesigner"],"_links":{"self":[{"href":"http:\/\/www.magicandlove.com\/blog\/wp-json\/wp\/v2\/posts\/3357","targetHints":{"allow":["GET"]}}],"collection":[{"href":"http:\/\/www.magicandlove.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/www.magicandlove.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/www.magicandlove.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/www.magicandlove.com\/blog\/wp-json\/wp\/v2\/comments?post=3357"}],"version-history":[{"count":7,"href":"http:\/\/www.magicandlove.com\/blog\/wp-json\/wp\/v2\/posts\/3357\/revisions"}],"predecessor-version":[{"id":3367,"href":"http:\/\/www.magicandlove.com\/blog\/wp-json\/wp\/v2\/posts\/3357\/revisions\/3367"}],"wp:attachment":[{"href":"http:\/\/www.magicandlove.com\/blog\/wp-json\/wp\/v2\/media?parent=3357"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/www.magicandlove.com\/blog\/wp-json\/wp\/v2\/categories?post=3357"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/www.magicandlove.com\/blog\/wp-json\/wp\/v2\/tags?post=3357"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}