The artwork is the second part of the Movement in Time series. Part 1 used 100 popular Hollywood film sequences to generate animated action paintings. Part 2 of the project analysed the fighting sequences in traditional Chinese martial art films. The results will match against the brush stroke data from the famous Cursive style Chinese calligraphy text – the One Thousand Characters Classics 千字文. In the end, the fight sequences will automatically generate a piece of unique text from the character database. Part of the project is funded by the Faculty Research Grant from the Hong Kong Baptist University.
The Cursive style Chinese calligraphy
In Tang Dynasty, there was a legendary story about the famous Chinese calligrapher, Zhang Xu 張旭. When he was drunk as usual, he saw a sword dance performed by the madam Gong Sun 公孫大娘. Since then, he was inspired to create the Wild Cursive style in Chinese calligraphy. It was the first motivation that I tried to use digital media to connect these two different types of traditional Chinese art forms, calligraphy and martial art.
Calligraphic work of Zhang Xu
Image from Wikipedia
With the help from a former student, Ms. Lisa LAM, I digitised all the 1,000 Chinese characters using a drawing tablet with a customer program developed in Processing. Each character will have a separate XML file to keep the brush stroke information. Here is a sample of an XML file.
<character> <stroke time="0"> <point> <x>0.21666667</x> <y>0.395</y> <w>0.032258064</w> <t>5</t> </point> <point> <x>0.21666667</x> <y>0.39666668</y> <w>0.121212125</w> <t>33</t> </point>
Given the XML file, I can recreate each Chinese character either in still image or animation.
Animated writing of the first eight characters
Given the touch sensitivity of the drawing tablet, I can simulate the depth information in 3D. It introduces an additional stylistic rendering of the Chinese characters in three dimensional space. Here are some experiments with 3D rendering of the Chinese characters.
The Chinese martial art films
The second component of the project is a collection of the famous Chinese martial art films. I studied and digitised a number of Chinese martial art films from director ranging from the traditional King Hu to the more contemporary Ang Lee.
I developed a custom software, in Processing and OpenCV, to use different methods of motion analysis to extract motion data from the fighting sequences. By experimenting with the motion data as virtual forces, I managed to animate a piece of string (thread) dancing across the screen.
Here are some examples of using the OpenCV functions, such as motion history, dense optical flow to analyse the fighting sequences from the films: Hero (2002), Seven Swords (2005) and Crouching Tiger, Hidden Dragon (2000).
Representation of the 1000 characters
Given the 1000 XML files of the one thousand Chinese characters and some mechanisms to extract fighting data from the martial art film sequences, I have to find a way to enable matching between them. In this case, I go back to my experience of learning Chinese calligraphy. In my childhood, like other kids, I learnt Chinese calligraphy by copying the grand masters’ works with the aids of a grid chart. Normally, we use a 3 x 3 grid chart to layout the brush strokes of each character. In the testings, I have used different grid sizes such as 3 x 3, 5 x 5, and 9 x 9.
With the 1000 Cursive style Chinese calligraphic characters, I also explore different ways to represent them in terms of point density, stroke direction, etc. The results will lead to different ways to enhance matching from the encoded fighting sequence to a unique Chinese characters. Here are some testing results to represent each of the Cursive style characters in a 9 x 9 grid, using both point density and stroke direction, respectively.
Point density model
Stroke direction model
The two images below summarize how I model the 1000 characters as point density and stroke direction.
Matching the fighting sequence and characters
In this phase, I explored the machine learning library from OpenCV and the Weka Machine Learning library to test run the matching against a fighting sequence and a Cursive style character from the database. Before I started matching with the fighting sequences, I developed another testing software to cross match a live character writing exercise with the 1000 characters in the database, using both the OpenCV machine learning library and the Weka library, with the K-nearest Neighbour matching.
Matching results for the two films, Hero and Crouching Tiger, Hidden Dragon
In November 2016, I was invited to participate in the Japan Media Arts Festival, Hong Kong Special Exhibition in The Annex, Hong Kong. The artwork includes four martial art film sequences, Crouching Tiger, Hidden Dragon, Hero, House of Flying Daggers (2004).
It is a computational animation where a custom software written in Processing executes live to extract motion data from the film sequences and match in real time with the 1000 characters in the database. The closest match one will be drawn on the fly in the screen. The display screen consists of 4 parts. The top left part is the original film sequence rendered in dense optical flow vectors. The bottom left part is the summary of all the optical flow data and reduced to a 5 x 5 grid indicating the most prominent movement on screen. In each frame of the playback, the motion data summarized in the 5 x 5 grid will match with the 1000 characters database. The closest matched character will be shown on the bottom right part of the screen animating the brush strokes of the characters. The top right part indicates where the current brush stroke position along with the path it travels in previous frames.
Recently, I enhance the work to accumulate all the characters matched successfully in the fighting sequence to form a piece of poetic text, on the top right corner of the screen. The order of the characters is determined purely by the fighting sequences. Quite obviously, they do not intend to make any senses. Nevertheless, when we continue to read along the characters stream, it appears like a poem and occasionally seems to carry meanings beyond our imaginations.
The artwork is released as open source material. The source codes can be found in the repository Movement in Time – Part 2