{"id":3305,"date":"2021-05-31T20:57:26","date_gmt":"2021-05-31T12:57:26","guid":{"rendered":"http:\/\/www.magicandlove.com\/blog\/?p=3305"},"modified":"2021-06-02T20:22:50","modified_gmt":"2021-06-02T12:22:50","slug":"mediapipe-in-touchdesigner-5","status":"publish","type":"post","link":"http:\/\/www.magicandlove.com\/blog\/2021\/05\/31\/mediapipe-in-touchdesigner-5\/","title":{"rendered":"MediaPipe in TouchDesigner 5"},"content":{"rendered":"\n<p>This is the continuation of <a href=\"http:\/\/www.magicandlove.com\/blog\/2021\/05\/31\/mediapipe-in-touchdesigner-4\/\">the last post<\/a> with slight modifications. Instead of just displaying the face mesh details in a Script TOP, it tries to visualise all the face mesh points in a 3D space. As the facial landmarks returned from the MediaPipe contain three dimensional information, it is possible to enumerate all the points and display them in a <a rel=\"noreferrer noopener\" href=\"https:\/\/docs.derivative.ca\/Script_SOP\" target=\"_blank\">Script SOP<\/a>. We are going to use the <a href=\"https:\/\/docs.derivative.ca\/ScriptSOP_Class\" target=\"_blank\" rel=\"noreferrer noopener\">appendPoint()<\/a> function to generate the point cloud and the <a href=\"https:\/\/docs.derivative.ca\/ScriptSOP_Class\" target=\"_blank\" rel=\"noreferrer noopener\">appendPoly()<\/a> function to create the face mesh.<\/p>\n\n\n\n<p>The data returned from the MediaPipe contains the 468 facial landmarks, based on the <a rel=\"noreferrer noopener\" href=\"https:\/\/google.github.io\/mediapipe\/solutions\/face_mesh.html\" target=\"_blank\">Canonical Face Model<\/a>. The face mesh information (triangles), however, is not available from the results obtained from the MediaPipe solutions. Nevertheless, we can obtain such information from the <a rel=\"noreferrer noopener\" href=\"https:\/\/github.com\/google\/mediapipe\/blob\/master\/mediapipe\/modules\/face_geometry\/data\/geometry_pipeline_metadata_landmarks.pbtxt\" target=\"_blank\">meta data of the facial landmarks<\/a> from its GitHub. To simplify the process, I have edited the data into <a rel=\"noreferrer noopener\" href=\"https:\/\/github.com\/chungbwc\/TouchDesigner\/blob\/main\/MediaPipeFaceMeshSOP\/mesh.csv\" target=\"_blank\">this CSV mesh file<\/a>. It is expected that the mesh.csv file is located in the TouchDesigner project folder, together with the TOE project file. Here are the first few lines of the mesh.csv file,<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">173,155,133\n246,33,7\n382,398,362\n263,466,249\n308,415,324<\/pre>\n\n\n\n<p>Each line is the data for a triangular mesh of the face. The 3 numbers are the indices of the vertices defined in the 468 facial landmarks. The visualisation of the landmarks is also available in the <a href=\"https:\/\/github.com\/google\/mediapipe\/tree\/master\/mediapipe\/modules\/face_geometry\/data\" target=\"_blank\" rel=\"noreferrer noopener\">MediaPipe GitHub<\/a>.<\/p>\n\n\n\n<figure class=\"wp-block-image size-large is-resized\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/raw.githubusercontent.com\/google\/mediapipe\/master\/mediapipe\/modules\/face_geometry\/data\/canonical_face_model_uv_visualization.png\" alt=\"Canonical face model\" width=\"600\" height=\"600\"\/><figcaption>Image from the Google MediaPipe GitHub<\/figcaption><\/figure>\n\n\n\n<p>The TouchDesigner project will render the Script SOP with the standard Geometry, Camera, Light and the <a rel=\"noreferrer noopener\" href=\"https:\/\/docs.derivative.ca\/Render_TOP\" target=\"_blank\">Render TOP.<\/a> <\/p>\n\n\n\n<figure class=\"wp-block-gallery columns-2 is-cropped wp-block-gallery-1 is-layout-flex wp-block-gallery-is-layout-flex\"><ul class=\"blocks-gallery-grid\"><li class=\"blocks-gallery-item\"><figure><a href=\"http:\/\/www.magicandlove.com\/blog\/wp-content\/uploads\/2021\/05\/FaceMeshSOP1-1.png\"><img loading=\"lazy\" decoding=\"async\" width=\"300\" height=\"188\" src=\"http:\/\/www.magicandlove.com\/blog\/wp-content\/uploads\/2021\/05\/FaceMeshSOP1-1-300x188.png\" alt=\"\" data-id=\"3317\" data-full-url=\"http:\/\/www.magicandlove.com\/blog\/wp-content\/uploads\/2021\/05\/FaceMeshSOP1-1.png\" data-link=\"http:\/\/www.magicandlove.com\/blog\/2021\/05\/31\/mediapipe-in-touchdesigner-5\/facemeshsop1-1\/\" class=\"wp-image-3317\" srcset=\"http:\/\/www.magicandlove.com\/blog\/wp-content\/uploads\/2021\/05\/FaceMeshSOP1-1-300x188.png 300w, http:\/\/www.magicandlove.com\/blog\/wp-content\/uploads\/2021\/05\/FaceMeshSOP1-1-1024x640.png 1024w, http:\/\/www.magicandlove.com\/blog\/wp-content\/uploads\/2021\/05\/FaceMeshSOP1-1-768x480.png 768w, http:\/\/www.magicandlove.com\/blog\/wp-content\/uploads\/2021\/05\/FaceMeshSOP1-1-1536x960.png 1536w, http:\/\/www.magicandlove.com\/blog\/wp-content\/uploads\/2021\/05\/FaceMeshSOP1-1-690x431.png 690w, http:\/\/www.magicandlove.com\/blog\/wp-content\/uploads\/2021\/05\/FaceMeshSOP1-1-980x613.png 980w, http:\/\/www.magicandlove.com\/blog\/wp-content\/uploads\/2021\/05\/FaceMeshSOP1-1.png 1920w\" sizes=\"auto, (max-width: 300px) 100vw, 300px\" \/><\/a><\/figure><\/li><li class=\"blocks-gallery-item\"><figure><a href=\"http:\/\/www.magicandlove.com\/blog\/wp-content\/uploads\/2021\/05\/FaceMeshSOP2.png\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"579\" src=\"http:\/\/www.magicandlove.com\/blog\/wp-content\/uploads\/2021\/05\/FaceMeshSOP2-1024x579.png\" alt=\"\" data-id=\"3318\" data-full-url=\"http:\/\/www.magicandlove.com\/blog\/wp-content\/uploads\/2021\/05\/FaceMeshSOP2.png\" data-link=\"http:\/\/www.magicandlove.com\/blog\/2021\/05\/31\/mediapipe-in-touchdesigner-5\/facemeshsop2\/\" class=\"wp-image-3318\" srcset=\"http:\/\/www.magicandlove.com\/blog\/wp-content\/uploads\/2021\/05\/FaceMeshSOP2-1024x579.png 1024w, http:\/\/www.magicandlove.com\/blog\/wp-content\/uploads\/2021\/05\/FaceMeshSOP2-300x170.png 300w, http:\/\/www.magicandlove.com\/blog\/wp-content\/uploads\/2021\/05\/FaceMeshSOP2-768x434.png 768w, http:\/\/www.magicandlove.com\/blog\/wp-content\/uploads\/2021\/05\/FaceMeshSOP2-1536x869.png 1536w, http:\/\/www.magicandlove.com\/blog\/wp-content\/uploads\/2021\/05\/FaceMeshSOP2-2048x1158.png 2048w, http:\/\/www.magicandlove.com\/blog\/wp-content\/uploads\/2021\/05\/FaceMeshSOP2-690x390.png 690w, http:\/\/www.magicandlove.com\/blog\/wp-content\/uploads\/2021\/05\/FaceMeshSOP2-980x554.png 980w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/a><\/figure><\/li><\/ul><\/figure>\n\n\n\n<p>I&#8217;ll not go through all the code here. The following paragraphs cover some of the essential elements in the Python code. The first one is the initialisation of the face mesh information from the <strong>mesh.csv<\/strong> file.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">triangles = []\nmesh_file = project.folder + \"\/mesh.csv\"\nmf = open(mesh_file, \"r\")\nmesh_list = mf.read().split('\\n')\nfor m in mesh_list:\n    temp = m.split(',')\n    x = temp[0]\n    y = temp[1]\n    z = temp[2]\n    triangles.append([x, y, z])<\/pre>\n\n\n\n<p>The variable <strong>triangles<\/strong> is the list of all triangles from the canonical face model. Each entry is a list of 3 indices to the entries of the corresponding points in the 468 facial landmarks. The second one is the code to generate the face point cloud and the mesh.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">for pt in landmarks:\n    p = scriptOp.appendPoint()\n    p.x = pt.x\n    p.y = pt.y\n    p.z = pt.z\n\nfor poly in triangles:         \n    pp = scriptOp.appendPoly(3, closed=True, addPoints=False)  \n    pp[0].point = scriptOp.points[poly[0]]        \n    pp[1].point = scriptOp.points[poly[1]]       \n    pp[2].point = scriptOp.points[poly[2]]\n<\/pre>\n\n\n\n<p>The first <strong>for<\/strong> loop creates all the points from the facial landmarks using the <strong>appendPoint()<\/strong> function. The second <strong>for<\/strong> loop creates all the triangular meshes from information stored in the variable <strong>triangles<\/strong> using the <strong>appendPoly()<\/strong> function.<\/p>\n\n\n\n<p>After we draw the 3D face model, we also compute the normals of the model by using another <a rel=\"noreferrer noopener\" href=\"https:\/\/docs.derivative.ca\/Attribute_Create_SOP\" target=\"_blank\">Attribute Create SOP<\/a>.<\/p>\n\n\n\n<p>The final TouchDesigner project is available in the <a href=\"https:\/\/github.com\/chungbwc\/TouchDesigner\/tree\/main\/MediaPipeFaceMeshSOP\" target=\"_blank\" rel=\"noreferrer noopener\">MediaPipeFaceMeshSOP<\/a> repository. <\/p>\n","protected":false},"excerpt":{"rendered":"<p>This is the continuation of the last post with slight modifications. Instead of just displaying the face mesh details in a Script TOP, it tries to visualise all the face mesh points in a 3D space. As the facial landmarks returned from the MediaPipe contain three dimensional information, it is possible to enumerate all the [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[179],"tags":[183,180,182,84],"class_list":["post-3305","post","type-post","status-publish","format-standard","hentry","category-tutorials","tag-face-mesh","tag-mediapipe","tag-python","tag-touchdesigner"],"_links":{"self":[{"href":"http:\/\/www.magicandlove.com\/blog\/wp-json\/wp\/v2\/posts\/3305","targetHints":{"allow":["GET"]}}],"collection":[{"href":"http:\/\/www.magicandlove.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/www.magicandlove.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/www.magicandlove.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/www.magicandlove.com\/blog\/wp-json\/wp\/v2\/comments?post=3305"}],"version-history":[{"count":20,"href":"http:\/\/www.magicandlove.com\/blog\/wp-json\/wp\/v2\/posts\/3305\/revisions"}],"predecessor-version":[{"id":3330,"href":"http:\/\/www.magicandlove.com\/blog\/wp-json\/wp\/v2\/posts\/3305\/revisions\/3330"}],"wp:attachment":[{"href":"http:\/\/www.magicandlove.com\/blog\/wp-json\/wp\/v2\/media?parent=3305"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/www.magicandlove.com\/blog\/wp-json\/wp\/v2\/categories?post=3305"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/www.magicandlove.com\/blog\/wp-json\/wp\/v2\/tags?post=3305"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}