It took a lot of pain, but I got color based object tracking working for Wade Brainerd‘s Colors! Activity, which he ported from Colors! for Nintendo DS by Jens Andersson. The tracking for what I have right now is calibrated to use the XO’s AC adapter. It’s a bright, uniform, unique color, and it’s something that every XO owner has. It’s pretty nifty watching the brush follow the camera around, but the real magic happens with you enable “Pressure Sensitivity” in the palette menu of Colors! The brush will actually change size and opacity based on how far you hold the charger from the camera. It works better than I could have ever expected, and this is just the start. The next step I’m planning is to let the user pick any real life object they want and use it as a brush, with the color of the object being the color of the brush on screen.
To use it, make sure you’re in a reasonably lit room. It won’t work in pitch black, but you don’t have to be sitting in a tanning bed either. Start Colors! and navigate to the second tab, Paint. Click on the leftmost icon, Palette, and enable both “Pressure Sensitive” checkboxes. Click on the Palette button again to close that window. Click on the rightmost button, the magnifying glass called Video Paint. Hold the charger out a foot or two from the camera, and hold the left mouse button to paint. Become the next van Gogh, minus the insanity and plus an ear.
How this works is the tricky part. Most of the work involved figuring out what color format the v4l2src plugin for Gstreamer was outputting. Gstreamer itself lied and said yuv 565 or rgb 565 depending on where you looked. In reality, its YUYV. That is, there are 4 bytes per 2 pixels. The first Y represents the luma (brightness) of the first pixel, the second Y is the brightness of the second, U is the blue chrominance of both pixels, and V is the red chrominance of both. It’s pretty strange, but it works well, since your eyes see luminosity much more accurately than they see color. To threshold out the specific hue of green that the AC adapter is, I converted the YUVY to the HSV colorspace: hue, saturation, and value. Hue filtering is ideal for color tracking, in that you can find a very narrow band of color accurately almost regardless of lighting conditions. After thresholding, its just a matter of finding the centroid of the filtered pixels. This also means that if there is something of a very similar color in view, it’s going to throw off the detection. The next step will be to calculate connected components and filter those. It’ll be more reliable and more accurate, but I’m not sure the XO has the processing power to run it at a usable speed. We’ll see…
More information about colorspaces: