monitoring gaia travis build status using webmail LED notifiers

usb LED webmail notifiers showing build status

For Firefox OS the Gaia UI currently uses Travis CI to run a series of test jobs in parallel for each pull request.  While Travis has a neat ember.js-based live-updating web UI, I usually find myself either staring at my build watching it go nowhere or forgetting about it entirely.  The latter is usually what ends up happening since we have a finite number of builders available, we have tons of developers, each build takes 5 jobs, and some of those jobs can take up to 35 minutes to run when they finally get a turn to run.

I recently noticed ThinkGeek had a bunch of Dream Cheeky USB LED notifiers on sale.  They’re each a USB-controlled tri-color LED in a plastic case that acts as a nice diffuser.  Linux’s “usbled” driver exposes separate red/green/blue files via sysfs that you can echo numbers into to control them.  While the driver and USB protocol inherently support a range of 0-255, it seems like 0-63 or 0-64 is all they give.  The color gamut isn’t amazing but is quite respectable and they are bright enough that they are useful in daylight.  I made a node.js library at https://github.com/asutherland/gaudy-leds that can do some basic tricks and is available on npm as “gaudy-leds”.  You can tell it to do things by doing “gaudy-leds set red green blue purple”, etc.  I added a bunch of commander sub-commands, so “gaudy-leds –help” should give a lot more details than the currently spartan readme.

I couldn’t find any existing tools/libraries to easily watch a Travis CI build and invoke commands like that (though I feel like they must exist) so I wrote https://github.com/asutherland/travis-build-watcher.  While the eventual goal is to not have to manually activate it at all, right now I can point it at a Travis build or a github pull request and it will poll appropriately so it ends up at the latest build and updates the state of the LEDs each time it polls.

Relevant notes / context:

  • There is a storied history of people hooking build/tree status up to LED lights and real traffic lights and stuff like that.  I think if you use Jenkins you’re particularly in luck.  This isn’t anything particularly new or novel, but the webmail notifiers are a great off-the-shelf solution.  The last time I did something like this I used a phidgets LED64 in a rice paper lamp and the soldering was much more annoying than dealing with a mess of USB cables.  Also, it could really only display one status at a time.
  • There are obviously USB port scalability issues, but you can get a 24-port USB hub for ~$40 from Amazon/monoprice/etc.  (They all seem to be made by the same manufacturer.)  I coincidentally bought 24 of the notifiers after my initial success with 6, so I am really prepared for an explosion in test jobs!
  • While I’m currently trying to keep things UNIXy with a bunch of small command-line tools operating together, I think I would like to have some kind of simple message-bus mechanism so that:
    • mozilla-central mach builds can report status as they go
    • webhooks / other async mechanisms can be used to improve efficiency and require less manual triggering/interaction on my end.  So if I re-spin a build from the web UI I won’t need to re-trigger the script locally and such.  Please let me know if you’re aware of existing solutions in this space, I didn’t find much and am planning to just use redis as glue for a bunch of small/independent pieces plus a more daemonish node process for polling / interacting with the web/AMQP.
  • There are efforts underway to overhaul the continuous integration mechanism used for Gaia.  This should address delays in starting tests by being able to throw more resources at them as well as allow notification by whatever Mozilla Pulse’s successor is.

6 thoughts on “monitoring gaia travis build status using webmail LED notifiers

  1. If you want to tinker more, take a Raspberry Pi (they are incredibly easy software-wise), and connect a motor to its GPIO.

    Then you have a hammer and you rule :). Imagine a loud “dong” on a huge bell when something fails. 🙂

  2. That sounds terrifying! Now, maybe if the servo dispensed candy or other forms of reward pellets… 🙂

  3. Nifty idea, but a bit overkill for me. On the software side, however, you probably could have used BuildNotify (which is mentioned in the Travis-CI docs).

    https://bitbucket.org/Anay/buildnotify/

    It’ll add an “only green if all are green” tray icon and supports notification of a user-selectable set of conditions by a custom command like

    notify-send -u normal -a ‘Travis-CI’ -i travis-ci #status# #projects#

    Wouldn’t be too difficult to twiddle sysfs files via that.

  4. Hah, yeah, I tend to lean towards overkill. I was not aware of buildnotify, very neat! Thanks very much for taking the time to make me aware of it!

    I see now that it’s mentioned in passing on http://docs.travis-ci.com/user/cc-menu/ (visible in the docs sidebar under “CCMenu / CCTray Feeds”). It looks like it can do a lot of neat tricks too, although it would need to learn a new trick or two to understand the github pull request use-case I have. (Specifically, I really only care about pull requests I make or failures in the apps I work on rather than the whole tree.)

  5. We’ve looked at doing something like this for our Jenkins CI environments at work. Only problem is, there’s just too many of them… with one for each supported version of each module plus end-to-end testbeds and platform builds, there’d be almost a hundred lights on the board, with at least a few of them red at any point in time.

  6. Yeah, then you’re into mozilla-central TBPL (https://tbpl.mozilla.org/) issues where a bunch of lights probably isn’t useful other than an as ambient but not particularly directly actionable display.

    One option would be to use something like one or more Novation Launchpad devices which is an 8-by-8 red-green LED-backed button matrix USB-MIDI controller. I bought one of these long ago with the goal of having it show build status and having button presses bring up the log/build information on my display but never ended up hooking it up because my tests ran fast enough and the LEDs weren’t bright enough. I see now on http://global.novationmusic.com/launchpad-s-development that there’s a new, if more expensive, Launchpad S that looks to be much brighter. There’s also a new smaller Launchpad Mini that’s the same effective price as the original. It’s not immediately clear if it shares the brightness as the S, but skimming random message boards it sounds like it may be.

    I suspect I’ll be hooking up my launchpad in the near future since I have ample desk-space and I’ve gotten the tooling ball rolling on this front. I’ll be sure to blog about it if I do. (And would love to hear about other people making progress on that front too!)

Comments are closed.