Paint with Color Based Object Tracking on the OLPC XO

AC Adapter Art

 It took a lot of pain, but I got color based object tracking working for Wade Brainerd‘s Colors! Activity, which he ported from Colors! for Nintendo DS by Jens Andersson.  The tracking for what I have right now is calibrated to use the XO’s AC adapter.  It’s a bright, uniform, unique color, and it’s something that every XO owner has.  It’s pretty nifty watching the brush follow the camera around, but the real magic happens with you enable “Pressure Sensitivity” in the palette menu of Colors!  The brush will actually change size and opacity based on how far you hold the charger from the camera.  It works better than I could have ever expected, and this is just the start.  The next step I’m planning is to let the user pick any real life object they want and use it as a brush, with the color of the object being the color of the brush on screen.

Download my modified Colors! XO Activity

Colors! Screenshot

To use it, make sure you’re in a reasonably lit room.  It won’t work in pitch black, but you don’t have to be sitting in a tanning bed either.  Start Colors! and navigate to the second tab, Paint.  Click on the leftmost icon, Palette, and enable both “Pressure Sensitive” checkboxes.  Click on the Palette button again to close that window.  Click on the rightmost button, the magnifying glass called Video Paint.  Hold the charger out a foot or two from the camera, and hold the left mouse button to paint.  Become the next van Gogh, minus the insanity and plus an ear.

 How this works is the tricky part. Most of the work involved figuring out what color format the v4l2src plugin for Gstreamer was outputting.  Gstreamer itself lied and said yuv 565 or rgb 565 depending on where you looked.  In reality, its YUYV.  That is, there are 4 bytes per 2 pixels.  The first Y represents the luma (brightness) of the first pixel, the second Y is the brightness of the second, U is the blue chrominance of both pixels, and V is the red chrominance of both.  It’s pretty strange, but it works well, since your eyes see luminosity much more accurately than they see color.  To threshold out the specific hue of green that the AC adapter is, I converted the YUVY to the HSV colorspace: hue, saturation, and value.  Hue filtering is ideal for color tracking, in that you can find a very narrow band of color accurately almost regardless of lighting conditions.  After thresholding, its just a matter of finding the centroid of the filtered pixels.  This also means that if there is something of a very similar color in view, it’s going to throw off the detection.  The next step will be to calculate connected components and filter those.  It’ll be more reliable and more accurate, but I’m not sure the XO has the processing power to run it at a usable speed.  We’ll see…

More information about colorspaces:

Related Posts

13 Replies to “Paint with Color Based Object Tracking on the OLPC XO”

  1. Nice work!

    “Gstreamer itself lied and said yuv 565 or rgb 565 depending on where you looked.” – is there a filed bug for that?

  2. Thanks all!

    @Jan Schmidt: I’m not sure. I think it’s just a quirk of v4l2src, and not necessarily a bug.

    @Sameer Verma: Yea, the Wii VR stuff is really amazing. The guy who did that actually goes to my university.

  3. I tried to install your version of Colors! but I’m having problems. The original version from! works fine. It downloads and installs with an icon. Your version downloads, but when I look for an icon, there’s nothing. I also don’t see anything in /home/olpc/Activities so it looks like something is falling apart during installation. Any suggestions?

  4. So, this is what has worked for me thus far. I’ve downloaded your version of the activity, but instead of clicking on “OK” in Browse, I click on “Open”. This opens the file in EToys. From there, I extract all files to /home/olpc/Activities

    A reboot shows the Colors icon in the list, and the activity works from there on. Referencing the toolbar in the screenshot above, the second “zoomout” icon is the one that starts the Video painting mode.

    Kudos on making this activity. Its very cool.

  5. The .xo itself was fine. There were/are just absurd issues between Apache, Browse, and Sugar regarding mime-types. It may or may not be fixed now.

  6. I made a little project for Etoys that track colors with the video camera. Its done in unoptimized Smalltalk so resolution and speed is not so great but I plan to make tracking colored objects a general input for Etoys so kids can make simple games with it. Here is a Etoys project with camera tracking:


  7. This is a very cool hack. I like using it to show friends what you can do with an XO.

    Will you be adding this to the current version of Colors!? (11?)

  8. Ben, my semester just ended, and adding an improved version of it to the next version of Colors! is near the top of my to do list. I may have something by the end of the week.

Leave a Reply to nrp Cancel reply

Your email address will not be published.