Face detection indicated by spinning 3D model rendered with OpenGL and blended into the reality.

This is the last blog post. After a few weeks of the project, things changed and I didn’t focus that much on this blog as planned. My apologies.

Tomorrow I’ll present this project and it’s over for now. Overall it’s amazing what can be done. With enough time I totally could build a killer app. Technology is here but not many solutions already. I must admit, I found some topics quite advanced (OpenCV on this device, using private API, rendering OpenGL on top of it, …). It’s different from building yet another webshop with the latest Java web framework thingy and drawing nice UML diagrams.

As stated in this blog with the new soft- and hardware a lot more can be achieved. I also found a surprisingly enthusiastic post on the same blog about an app in the app store detecting some faces and drawing funny pictures over it. People aren’t expecting this technology yet. With a few weeks or months of engineering one could build a really cutting edge app.

I still have a lot to learn before building a product with such technologies involved. But we have time. It’s always day one.

A grafic for the Durchblick App.

Analyzing with Activity Monitor 90% of the main CPU, according to AnandTech a Samsung S5PC100 ARM Cortex-A8 with 833 MHz underclocked to 600 MHz, is not enough for face detection. I also found out by CPU Sampler that only 20% of the CPU time is spent on cvRunHaarClassifierCascade—the actual face detection. Everything else is consumed to run everything else.

Still, face detection took about 3 secs while the rest of the processing loop only took a few ms. Also, the there were a lot of false positives.

Optimizing the parameters gave better results: for face detection 2 secs even in complex environment with less false positives.

		CvSeq *faces = cvHaarDetectObjects(iplImage,
										   cascade,
										   storage,
										   1.1, // double scale_factor CV_DEFAULT(1.1)
										   2, // int min_neighbors CV_DEFAULT(3)
										   CV_HAAR_DO_CANNY_PRUNING, // int flags CV_DEFAULT(0)
										   cvSize(30, 30)); // CvSize min_size CV_DEFAULT(cvSize(0,0))

I could get a big improvement by scaling down the image by factor 2 and smooth it with a 5×5 gaussian kernel: 0.3 secs, downside: face must be approximately the size of a thumb on the screen.

If you are like me you like to draw moustaches on people’s faces—no need for yet another frame around a detected face.

Lena with live face detection and display of a moustache

The moustache flickered away because he avoid face detection in the previous frame.

But here the problem with the way I access the camera image and occlusion manifested itself. The moustache got displayed, then removed because no face was recognized and then with the uncluttered face detected and displayed again.

The final tests I did with a dashed rounded rectagle with light and dark grey in it for good contrast against any object. I place it around the face so no occlusion of the face takes place.

Detected face with a bounding rectangle.

I can detect multiple faces in one frame. But it’s difficult to smoothly match the rectangles from the previous frame to the new ones. For now I constrain the displaying to one face an do a nice fade in and move animation.

It’s at the very beginning but quite impressive.

A C example of “Face Detection using OpenCV” in the Wiki helped me using this OpenCV function.

Useful links about building OpenCV for other platforms and linking to the library:

• OpenCVWiki explains ways how to build for Mac OS X: Mac OS X OpenCV Port
• Basics of cross-compiling: Cross-compiling for iPhone
• Building for different platforms: Creating an universal static OpenCV library in (Snow) Leopard
• Makefile, patches, good build tutorial and a example project: Using OpenCV on iPhone
• Linking tips: OpenCV vs. Apple iPhone

It seems that dynamic linking with a private .framework doesn’t work and OpenCV only works linked as a static library.

With all the given work what had to be done:

1. Build OpenCV-2.0.0 for iPhone OS 3.1 (simulator and device) according to this good tutorial.
2. Under Target > Build > Linking > Other Linker Flags (These flags are passed with all linker invocations. [OTHER_LDFLAGS]) add Build Setting Conditions for Any iPhone Device/Any Architecture and Any iPhone OS Simulator/Any Architecture. Add the flags -lstdc++, -lz, $(SRCROOT)/OpenCV/opencv_{device,simulator}/lib/libcv.a and libcxcore.a
3. Under Target > Build > Search Paths > Header Search Paths (This is a list of paths to folders to be searched by the compiler for included or imported header files when compiling C, Objective-C, C++, or Objective-C++. Paths are delimited by whitespace, so any paths with spaces in them need to be properly quoted. [HEADER_SEARCH_PATHS, -I]) add Build Setting Conditions for Any iPhone Device/Any Architecture and Any Iphone OS Simulator/Any Architecture. Add the path $(SRCROOT)/OpenCV/opencv_{device,simulator}/include without recursive flag.
4. #import <opencv/cv.h>

Other option is adding headers to project and #import "cv.h".

The compiler threw some linking warnings like ld: warning: can't add line info to anonymous symbol __ZN2cv9ExceptionD1Ev.lsda from /Users/raff/Development/Durchblick/OpenCV/opencv_simulator/lib/libcxcore.a(lib_cxcore_la-cxmathfuncs.o) when building for the simulator which I couldn’t solve. I guess I simply could’ve ignore them. When using the precompiled lib there were no warnings.