This is the presentation I made for the Hong Kong Dorkbot event – Design in Code artist talk, together with the book launch, organized by Videotage and K11.
Domain Mall Interactive Wall
This is an interactive display for the Domain Mall public art project, with Otto Li. The mall is in Yau Tong, next to the MTR station. Two Kinect cameras capture the participant images to react with the animation. The season and weather information trigger further variations of the visual imageries.
Build OpenCV on Mac OSX
When I posted the use of OpenCV in Processing, there were a number of queries about installing and building OpenCV in OSX. In this post, I try to summary the process. It assumes basic understandings of the Unix environment of the OSX system with the use of the Terminal command. Here are the steps.
Multimedia Programming with Pure Data
The book is finally published. It is the first English book on Pure Data that covers both visual (the GEM library) and audio elements. Check it out from the Packt Publishing.
The new face detection and Processing
Here is the slightly modified version of the OpenCV Java tutorial example with face detection, ported to Processing.
import processing.video.*; import org.opencv.core.Core; import org.opencv.core.Mat; import org.opencv.core.MatOfRect; import org.opencv.core.Point; import org.opencv.core.Rect; import org.opencv.core.Scalar; import org.opencv.core.CvType; import org.opencv.core.Size; import org.opencv.objdetect.CascadeClassifier; import java.awt.image.BufferedImage; import java.awt.image.WritableRaster; import java.awt.image.Raster; Capture cap; int pixCnt; BufferedImage bm; CascadeClassifier faceDetector; MatOfRect faceDetections; void setup() { size(640, 480); System.loadLibrary(Core.NATIVE_LIBRARY_NAME); println(Core.VERSION); cap = new Capture(this, width, height); cap.start(); bm = new BufferedImage(width, height, BufferedImage.TYPE_4BYTE_ABGR); pixCnt = width*height*4; faceDetector = new CascadeClassifier(dataPath("haarcascade_frontalface_default.xml")); faceDetections = new MatOfRect(); } void convert(PImage _i) { bm.setRGB(0, 0, _i.width, _i.height, _i.pixels, 0, _i.width); Raster rr = bm.getRaster(); byte [] b1 = new byte[pixCnt]; rr.getDataElements(0, 0, _i.width, _i.height, b1); Mat m1 = new Mat(_i.height, _i.width, CvType.CV_8UC4); m1.put(0, 0, b1); Mat m2 = new Mat(_i.height, _i.width, CvType.CV_8UC1); Imgproc.cvtColor(m1, m2, Imgproc.COLOR_BGRA2GRAY); faceDetector.detectMultiScale(m2, faceDetections, 3, 1, Objdetect.CASCADE_DO_CANNY_PRUNING, new Size(40, 40), new Size(240, 240)); bm.flush(); m2.release(); m1.release(); } void draw() { if (!cap.available()) return; background(0); cap.read(); convert(cap); image(cap, 0, 0); for (Rect rect: faceDetections.toArray()) { noFill(); stroke(255, 0, 0); rect(rect.x, rect.y, rect.width, rect.height); } } |
OpenCV 2.4.4 and Processing
The latest OpenCV 2.4.4 supports the desktop version of Java. We can use the OpenCV functions in the Processing environment. Before we write a library for it, we can temporarily test it by putting the opencv-244.jar and libopencv_java244.dylib into the code folder of your sketch. For Windows, the dll file should go there as well.
And we also need to perform conversion between the PImage and the OpenCV Mat. I do not want to loop through the pixels array to use bitwise operations. In this case, I use the Java BufferedImage and WritableRaster classes with the 8 bits and 4 channels standard format.
Here are the example codes that perform the GaussianBlur and Canny functions.
import processing.video.*; import org.opencv.core.Core; import org.opencv.core.Mat; import org.opencv.core.CvType; import org.opencv.core.Size; import java.awt.image.BufferedImage; import java.awt.image.WritableRaster; import java.awt.image.Raster; Capture cap; int pixCnt; BufferedImage bm; PImage img; void setup() { size(640, 480); System.loadLibrary(Core.NATIVE_LIBRARY_NAME); println(Core.VERSION); cap = new Capture(this, width, height); cap.start(); bm = new BufferedImage(width, height, BufferedImage.TYPE_4BYTE_ABGR); img = createImage(width, height, ARGB); pixCnt = width*height*4; } void convert(PImage _i) { bm.setRGB(0, 0, _i.width, _i.height, _i.pixels, 0, _i.width); Raster rr = bm.getRaster(); byte [] b1 = new byte[pixCnt]; rr.getDataElements(0, 0, _i.width, _i.height, b1); Mat m1 = new Mat(_i.height, _i.width, CvType.CV_8UC4); m1.put(0, 0, b1); Mat m2 = new Mat(_i.height, _i.width, CvType.CV_8UC1); Imgproc.cvtColor(m1, m2, Imgproc.COLOR_BGRA2GRAY); Imgproc.GaussianBlur(m2, m2, new Size(7, 7), 1.5, 1.5); Imgproc.Canny(m2, m2, 0, 30, 3, false); Imgproc.cvtColor(m2, m1, Imgproc.COLOR_GRAY2BGRA); m1.get(0, 0, b1); WritableRaster wr = bm.getRaster(); wr.setDataElements(0, 0, _i.width, _i.height, b1); bm.getRGB(0, 0, _i.width, _i.height, img.pixels, 0, _i.width); img.updatePixels(); bm.flush(); m2.release(); m1.release(); } void draw() { if (cap.available()) { background(0); cap.read(); convert(cap); image(img, 0, 0); } } |
Drawing Machines
In the recent preparation for classes, I come into this topic of drawing machines. Here are some examples from YouTube.
And the Machine Art exhibition
Art Machines Machine Art
Kinect for Processing Library Page
The Kinect for Processing library page is now ready with documentation, example, download and source codes. Happy coding.
Kinect in Windows for Processing 5
This is a more or less finished version of the Kinect for Processing library. It includes the basic skeleton tracking, the RGB image, depth image and a mask image. I shall move the related posts to the research page with more documentation on the data structure and method description. Stay tune.
The sample Processing code:
import pKinect.PKinect; import pKinect.SkeletonData; PKinect kinect; PFont font; ArrayList<SkeletonData> bodies; PImage img; void setup() { size(640, 480); background(0); kinect = new PKinect(this); bodies = new ArrayList<SkeletonData>(); smooth(); font = loadFont("LucidaSans-18.vlw"); textFont(font, 18); textAlign(CENTER); img = loadImage("background.png"); } void draw() { background(0); image(kinect.GetImage(), 320, 0, 320, 240); image(kinect.GetDepth(), 320, 240, 320, 240); image(img, 0, 240, 320, 240); image(kinect.GetMask(), 0, 240, 320, 240); for (int i=0; i<bodies.size(); i++) { drawSkeleton(bodies.get(i)); drawPosition(bodies.get(i)); } } void mousePressed() { println(frameRate); } void drawPosition(SkeletonData _s) { noStroke(); fill(0, 100, 255); String s1 = str(_s.dwTrackingID); text(s1, _s.position.x*width/2, _s.position.y*height/2); } void drawSkeleton(SkeletonData _s) { // Body DrawBone(_s, PKinect.NUI_SKELETON_POSITION_HEAD, PKinect.NUI_SKELETON_POSITION_SHOULDER_CENTER); DrawBone(_s, PKinect.NUI_SKELETON_POSITION_SHOULDER_CENTER, PKinect.NUI_SKELETON_POSITION_SHOULDER_LEFT); DrawBone(_s, PKinect.NUI_SKELETON_POSITION_SHOULDER_CENTER, PKinect.NUI_SKELETON_POSITION_SHOULDER_RIGHT); DrawBone(_s, PKinect.NUI_SKELETON_POSITION_SHOULDER_CENTER, PKinect.NUI_SKELETON_POSITION_SPINE); DrawBone(_s, PKinect.NUI_SKELETON_POSITION_SHOULDER_LEFT, PKinect.NUI_SKELETON_POSITION_SPINE); DrawBone(_s, PKinect.NUI_SKELETON_POSITION_SHOULDER_RIGHT, PKinect.NUI_SKELETON_POSITION_SPINE); DrawBone(_s, PKinect.NUI_SKELETON_POSITION_SPINE, PKinect.NUI_SKELETON_POSITION_HIP_CENTER); DrawBone(_s, PKinect.NUI_SKELETON_POSITION_HIP_CENTER, PKinect.NUI_SKELETON_POSITION_HIP_LEFT); DrawBone(_s, PKinect.NUI_SKELETON_POSITION_HIP_CENTER, PKinect.NUI_SKELETON_POSITION_HIP_RIGHT); DrawBone(_s, PKinect.NUI_SKELETON_POSITION_HIP_LEFT, PKinect.NUI_SKELETON_POSITION_HIP_RIGHT); // Left Arm DrawBone(_s, PKinect.NUI_SKELETON_POSITION_SHOULDER_LEFT, PKinect.NUI_SKELETON_POSITION_ELBOW_LEFT); DrawBone(_s, PKinect.NUI_SKELETON_POSITION_ELBOW_LEFT, PKinect.NUI_SKELETON_POSITION_WRIST_LEFT); DrawBone(_s, PKinect.NUI_SKELETON_POSITION_WRIST_LEFT, PKinect.NUI_SKELETON_POSITION_HAND_LEFT); // Right Arm DrawBone(_s, PKinect.NUI_SKELETON_POSITION_SHOULDER_RIGHT, PKinect.NUI_SKELETON_POSITION_ELBOW_RIGHT); DrawBone(_s, PKinect.NUI_SKELETON_POSITION_ELBOW_RIGHT, PKinect.NUI_SKELETON_POSITION_WRIST_RIGHT); DrawBone(_s, PKinect.NUI_SKELETON_POSITION_WRIST_RIGHT, PKinect.NUI_SKELETON_POSITION_HAND_RIGHT); // Left Leg DrawBone(_s, PKinect.NUI_SKELETON_POSITION_HIP_LEFT, PKinect.NUI_SKELETON_POSITION_KNEE_LEFT); DrawBone(_s, PKinect.NUI_SKELETON_POSITION_KNEE_LEFT, PKinect.NUI_SKELETON_POSITION_ANKLE_LEFT); DrawBone(_s, PKinect.NUI_SKELETON_POSITION_ANKLE_LEFT, PKinect.NUI_SKELETON_POSITION_FOOT_LEFT); // Right Leg DrawBone(_s, PKinect.NUI_SKELETON_POSITION_HIP_RIGHT, PKinect.NUI_SKELETON_POSITION_KNEE_RIGHT); DrawBone(_s, PKinect.NUI_SKELETON_POSITION_KNEE_RIGHT, PKinect.NUI_SKELETON_POSITION_ANKLE_RIGHT); DrawBone(_s, PKinect.NUI_SKELETON_POSITION_ANKLE_RIGHT, PKinect.NUI_SKELETON_POSITION_FOOT_RIGHT); } void DrawBone(SkeletonData _s, int _j1, int _j2) { noFill(); stroke(255, 255, 0); if (_s.skeletonPositionTrackingState[_j1] != PKinect.NUI_SKELETON_POSITION_NOT_TRACKED && _s.skeletonPositionTrackingState[_j2] != PKinect.NUI_SKELETON_POSITION_NOT_TRACKED) { line(_s.skeletonPositions[_j1].x*width/2, _s.skeletonPositions[_j1].y*height/2, _s.skeletonPositions[_j2].x*width/2, _s.skeletonPositions[_j2].y*height/2); } } void appearEvent(SkeletonData _s) { if (_s.trackingState == PKinect.NUI_SKELETON_NOT_TRACKED) { return; } synchronized(bodies) { bodies.add(_s); } } void disappearEvent(SkeletonData _s) { synchronized(bodies) { for (int i=bodies.size()-1; i>=0; i--) { if (_s.dwTrackingID == bodies.get(i).dwTrackingID) { bodies.remove(i); } } } } void moveEvent(SkeletonData _b, SkeletonData _a) { if (_a.trackingState == PKinect.NUI_SKELETON_NOT_TRACKED) { return; } synchronized(bodies) { for (int i=bodies.size()-1; i>=0; i--) { if (_b.dwTrackingID == bodies.get(i).dwTrackingID) { bodies.get(i).copy(_a); break; } } } } |
You can download the example and the library here.