CVImage and PixelFlow in Processing

This is a quick demonstration of using the CVImage library, from the book, Pro Processing for Images and Computer Vision with OpenCV, and the PixelFlow library from Thomas Diewald.
 
Here is the video documentation.

The full source code is below with one additional class.

Main Processing sketch

import cvimage.*;
import processing.video.*;
import com.thomasdiewald.pixelflow.java.DwPixelFlow;
import com.thomasdiewald.pixelflow.java.fluid.DwFluid2D;
import org.opencv.core.*;
import org.opencv.objdetect.CascadeClassifier;
import org.opencv.objdetect.Objdetect;
 
// Face detection size
final int W = 320, H = 180;
Capture cap;
CVImage img;
CascadeClassifier face;
float ratio;
DwFluid2D fluid;
PGraphics2D pg_fluid;
MyFluidData fluidFunc;
 
void settings() {
  size(1280, 720, P2D);
}
 
void setup() {
  background(0);
  System.loadLibrary(Core.NATIVE_LIBRARY_NAME);
  println(Core.VERSION);
  cap = new Capture(this, width, height);
  cap.start();
  img = new CVImage(W, H);
  face = new CascadeClassifier(dataPath("haarcascade_frontalface_default.xml"));
  ratio = float(width)/W;
 
  DwPixelFlow context = new DwPixelFlow(this);
  context.print();
  context.printGL();
  fluid = new DwFluid2D(context, width, height, 1);
  fluid.param.dissipation_velocity = 0.60f;
  fluid.param.dissipation_density = 0.99f;
  fluid.param.dissipation_temperature = 1.0f;
  fluid.param.vorticity = 0.001f;
 
  fluidFunc = new MyFluidData();
  fluid.addCallback_FluiData(fluidFunc);
  pg_fluid = (PGraphics2D) createGraphics(width, height, P2D);
  pg_fluid.smooth(4);
}
 
void draw() {
  if (!cap.available()) 
    return;
  background(0);
  cap.read();
  cap.updatePixels();
 
  img.copy(cap, 0, 0, cap.width, cap.height, 
    0, 0, img.width, img.height);
  img.copyTo();
 
  Mat grey = img.getGrey();
  MatOfRect faces = new MatOfRect();
 
  face.detectMultiScale(grey, faces, 1.15, 3, 
    Objdetect.CASCADE_SCALE_IMAGE, 
    new Size(60, 60), new Size(200, 200));
  Rect [] facesArr = faces.toArray();
  if (facesArr.length > 0) {
    fluidFunc.findFace(true);
  } else {
    fluidFunc.findFace(false);
  }
  for (Rect r : facesArr) {
    float cx = r.x + r.width/2.0;
    float cy = r.y + r.height/2.0;
    fluidFunc.setPos(new PVector(cx*ratio, cy*ratio));
  }
  fluid.update();
  pg_fluid.beginDraw();
  pg_fluid.background(0);
  pg_fluid.image(cap, 0, 0);
  pg_fluid.endDraw();
  fluid.renderFluidTextures(pg_fluid, 0);
  image(pg_fluid, 0, 0);
  pushStyle();
  noStroke();
  fill(0);
  text(nf(round(frameRate), 2, 0), 10, 20);
  popStyle();
  grey.release();
  faces.release();
}

The class definition of MyFluidData

private class MyFluidData implements DwFluid2D.FluidData {
  float intensity;
  float radius;
  float temperature;
  color c;
  boolean first;
  boolean face;
  PVector pos;
  PVector last;
 
  public MyFluidData() {
    super();
    intensity = 1.0f;
    radius = 25.0f;
    temperature = 5.0f;
    c = color(255, 255, 255);
    first = true;
    pos = new PVector(0, 0);
    last = new PVector(0, 0);
    face = false;
  }
 
  public void findFace(boolean f) {
    face = f;
  }
 
  public void setPos(PVector p) {
    if (first) {
      pos.x = p.x;
      pos.y = p.y;
      last.x = pos.x;
      last.y = pos.y;
      first = false;
    } else {
      last.x = pos.x;
      last.y = pos.y;
      pos.x = p.x;
      pos.y = p.y;
    }
  }
 
  @Override
    public void update(DwFluid2D f) {
 
    if (face) {
      float px = pos.x;
      float py = height - pos.y;
      float vx = (pos.x - last.x) * 10.0f;
      float vy = (pos.y - last.y) * -10.0f;
      c = color(random(100, 255), random(100, 255), random(50, 100));
      f.addVelocity(px, py, radius, vx, vy);
      f.addDensity (px, py, radius, 
        red(c)/255, green(c)/255, blue(c)/255, 
        intensity);
      f.addTemperature(px, py, radius, temperature);
    }
  }
}

OpenCV 3.3 Java Build

The new release of OpenCV 3.3 is out now. I again prepare the Java build for the CVImage Processing library use. It also includes the optflow extra module for motion history applications. Here is the list of the 3 OpenCV releases.

The book Pro Processing for Images and Computer Vision with OpenCV will be released soon. It will include the detailed build instructions in multiple platforms.

OpenCV 3.2 Java Build

In preparing for the forthcoming book in Processing and OpenCV, I have tried to build the Java binding in OpenCV 3.2. It worked easily for the basic components. Nevertheless, when I included the contribution moduleoptflow, it failed. After a number of attempts in various platforms, I found it was due to the gen_java.py script in folder opencv-3.2.0/modules/java/generator. I tried to add back the import details for the class DenseOpticalFlow. It worked again. Here is what I patch in the gen_java.py script.

For those who do not want to build it yourselves, you can download a pre-built version of the OpenCV 3.2 Java library. You can use it with Processing immediately. I have tested it with the current Processing at 3.3. It contains the following files for various platforms in 64 bit:

  • libopencv_java320.dylib
  • libopencv_java320.so
  • opencv_java320.dll
  • opencv-320.jar

Enjoy and happy coding.

 

Artificial Neural Network in OpenCV with Processing

This is the first trial of the Machine Learning module, artificial neural network in OpenCV with Processing. I used the same OpenCV 3.1.0 Java built files. The program took the live stream video (PImage) from webcam and down-sampled to a grid of just 8 x 6 pixels of greyscale. It started by default in the training mode such that I could click on the left hand side of the screen for an image without a hat and on the right hand side for an image of myself wearing a hat. By pressing the SPACE key, it switched to the predict mode where by clicking the video would send the image to the neural network to see if I was wearing a hat or not. I used around 20 images for positive response and 20 images for negative response.

Here are the source codes.
 
The main program

import processing.video.*;
 
Capture cap;
boolean training;
ANN ann;
int w, h;
 
void setup() {
  size(640, 480);
  System.loadLibrary(Core.NATIVE_LIBRARY_NAME);
  println(Core.VERSION);
  cap = new Capture(this, width, height);
  cap.start();
  background(0);
  training = true;
  w = 8;
  h = 6;
  ann = new ANN(w*h);
}
 
void draw() {
  image(cap, 0, 0);
}
 
void captureEvent(Capture c) {
  c.read();
}
 
void mousePressed() {
  PImage img = new PImage(w, h, ARGB);
  img.copy(cap, 0, 0, width, height, 0, 0, w, h);
  img.updatePixels();
  img.filter(GRAY);
  String fName = "";
  float [] grey = getGrey(img);
  if (training) {
    float label = 0.0;
    if (mouseX < width/2) {
      label = 0.0;
    } else {
      label = 1.0;
    }
    ann.addData(grey, label);
    fName = (label == 0.0) ? "Negative" : "Positive";
    fName += nf(ann.getCount(), 4) + ".png";
    img.save(dataPath("") + "/" + fName);
  } else {
    float val = ann.predict(grey);
    float [] res = ann.getResult();
    val = res[0];
    float diff0 = abs(val);
    float diff1 = abs(val - 1);
    if (diff0 < diff1) {
      println("Without hat");
    } else {
      println("With hat");
    }
  }
}
 
float [] getGrey(PImage m) {
  float [] g = new float[w*h];
  if (m.width != w || m.height != h) 
    return g;
  for (int i=0; i<m.pixels.length; i++) {
    color c = m.pixels[i];
    g[i] = red(c) / 256.0;
  }
  return g;
}
 
void keyPressed() {
  if (keyCode == 32) {
    training = !training;
    if (!training) 
      ann.train();
  }
  println("Training status is " + training);
}

The Artificial Neural Network class

import org.opencv.core.Core;
import org.opencv.core.CvType;
import org.opencv.core.MatOfInt;
import org.opencv.core.MatOfFloat;
import org.opencv.ml.ANN_MLP;
 
public class ANN {
  final int MAX_DATA = 1000;
  ANN_MLP mlp;
  int input;
  int output;
  ArrayList<float []>train;
  ArrayList<Float>label;
  MatOfFloat result;
  String model;
 
  public ANN(int i) {
    input = i;
    output = 1;
    mlp = ANN_MLP.create();
    MatOfInt m1 = new MatOfInt(input, input/2, output);
    mlp.setLayerSizes(m1);
    mlp.setActivationFunction(ANN_MLP.SIGMOID_SYM);
    mlp.setTrainMethod(ANN_MLP.RPROP);
    result = new MatOfFloat();
    train = new ArrayList<float[]>();
    label = new ArrayList<Float>();
    model = dataPath("trainModel.xml");
  }
 
  void addData(float [] t, float l) {
    if (t.length != input) 
      return;
    if (train.size() >= MAX_DATA) 
      return;
    train.add(t);
    label.add(l);
  }
 
  int getCount() {
    return train.size();
  }
 
  void train() {
    float [][] tr = new float[train.size()][input];
    for (int i=0; i<train.size(); i++) {
      for (int j=0; j<train.get(i).length; j++) {
        tr[i][j] = train.get(i)[j];
      }
    }
    MatOfFloat response = new MatOfFloat();
    response.fromList(label);
    float [] trf = flatten(tr);
    Mat trainData = new Mat(train.size(), input, CvType.CV_32FC1);
    trainData.put(0, 0, trf);
    mlp.train(trainData, Ml.ROW_SAMPLE, response);
    trainData.release();
    response.release();
    train.clear();
    label.clear();
  }
 
  float predict(float [] i) {
    if (i.length != input) 
      return -1;
    Mat test = new Mat(1, input, CvType.CV_32FC1);
    test.put(0, 0, i);
    float val = mlp.predict(test, result, 0);
    return val;
  }
 
  float [] getResult() {
    float [] r = result.toArray();
    return r;
  }
 
  float [] flatten(float [][] a) {
    if (a.length == 0) 
      return new float[]{};
    int rCnt = a.length;
    int cCnt = a[0].length;
    float [] res = new float[rCnt*cCnt];
    int idx = 0;
    for (int r=0; r<rCnt; r++) {
      for (int c=0; c<cCnt; c++) {
        res[idx] = a[r][c];
        idx++;
      }
    }
    return res;
  }
}

OpenCV 3.1 release Java build

It is the Java build of the new OpenCV 3.1 release. The zipped file contains the Java build for the 64 bit versions of Linux, Mac OSX and Windows. It is available for download at

OpenCV 3.1.0 Java build (64 bit)

Again, you can copy the individual files to your Processing code folder to play with the experiments in this website.

OpenCV and Processing 20

It is a little side tracked from the previous posts. The example demonstrated the use of Kalman Filter in OpenCV with Processing.

OpenCV and Processing 16

This example continues from the last post to compute the optical flow between 2 greyscale images by using the calcOpticalFlowPyrLK() function in the Video module. The new position of the pixels tracked will be delivered in a MatOfPoint2f object. By using the last and current position of the feature points, we can plot the path of the pixel movements. Furthermore, we can use such information for interactive or generative drawings, found in my artwork, Movement in Time.


Continue reading “OpenCV and Processing 16”