Monthly Archives: March 2011

unified JS/C++ backtraces, colorized backtraces, colorized source listing for gdb 7.3/archer-trunk updated

Posted on by

type "mbt", see this.

gdb has had integrated python support for some time and it is truly awesome.  The innards have changed here and there, so my previous efforts to show colorized/fancy backtraces, show colorized source listings, and provide unified JS/C++ mozilla backtraces (with colors still) have experienced some bit-rot over time.

type "cbt", see this

I have de-bitrotted the changes, hard.  Hard in this case means that the changes depend on gdb 7.3 which only exists in the future or in the archer repo you can find on this wiki page, or maybe in gdb upstream but I already had the archer repo checked out…

type "sl", see this.

In any event, if you like gdb and work with mozilla, you will not only want these things, but also Jim Blandy’s mozilla-archer repo which has magic SpiderMonkey JS helpers (one of which mozbt depends on; those JSStrings are crazy, yo!).  Coincidentally, he also depends on the future, and his README tells you how to check out the archer branch in greater detail.

You can get your own copy of these very exciting gdb python things from my github pythongdb-gaudy repo.

The important notes for the unified backtrace are:

  • It does’t work with the tracing JIT; the trace JIT doesn’t produce the required frames during its normal operation.
  • It might work with the method JIT if gdb could unwind method JIT frames, but it doesn’t seem to do that yet.  (Or I didn’t build with the right flag to tell it to use a frame pointer/etc., or I am using the wrong gdb branch/etc.)  Once the Method JIT starts telling gdb about its generated code with debug symbols and the performance issues in gdb’s JIT interface are dealt with, you can just use the colorizing backtrace because no special logic is required to interleave frames at that point and Jim Blandy’s magic JS pretty printers should ‘just work’.
  • Don’t forget that if you just want a pure JS backtrace (and with all kinds of fanciness) and have a debug build, you can just do “call DumpJSStack()” in gdb and it will call back into XPConnect and invoke the C++ code that authoritatively knows how to walk this stuff.
  • If you’re cool and using generators, then you’re stuck with the interpreter, so it will totally work.

arbitrarypushlog now more realtimey; likes wide screens, humans

Posted on by

Arbitrary Pushlog (ArbPL) recap:

  • It shows revision control pushes and the tinderbox build results for specific build trees like tinderboxpushlog (TBPL).
  • It is backed by a server which has a database so your browser is not going to launch a DOS attack on the mozilla infrastructure.  On the other hand, if that server falls over…
  • It figures out what failed by parsing the failed test logs.  It passes the time savings of this onto you in the form of listing the failures.  It also secretly knows things about failures that it does not tell you.
  • For Thunderbird mozmill test runs that experience failures it scrapes the fancy logging goodness out of the logs and exposes them in a pretty UI.  See the teaser post for screenshots.  You can also check to see if we’ve fixed all the Thunderbird mozmill oranges or not and maybe see a fresh new log!
  • It’s not capable of serving as a (full) replacement for TBPL at this time.  It does not show tree status (although that is easily fixed and should be fixed sometime soon), and it currently cannot star builds.

What’s new:

  • Widescreen display friendly.  It previously assumed you were me and ran your screen in portrait mode because that is the way you can see the most code; now it spills to two columns if your window is at least 1400px wide.  (See top screenshot.)
  • TraceMonkey’s mobile build matrix is split out from its desktop build matrix because the column sets did not meaningfully overlap.  No attempt has been made to deal with the other unique aspects of the TraceMonkey tree, although at least all known TraceMonkey builds should be categorized.  (If the builds that were triggered off another repo were split out into their own tinderbox tree so they could use the same mechanism Thunderbird and other comm-central complex repos use, no special work would be required.  It sounds like the tupled repo is going away soon, so it likely should become a moot issue.)

  • It now does fancy hierarchical grouping of the failures.  The builder names’ backgrounds are in gray because the failures have been starred.  The builder names would be in orange if the failing builds had not been starred.

  • When people merge things or otherwise push more than 8 changesets, we don’t show you everything they pushed, just the top 3 and provide a link to the full log elsewhere.

  • I made an effort to help explain what is going on with builds in non-secret code.  You click on a build, you get this lovely box.  Unfortunately I apparently also forgot to expose the secret code in the popup like I was planning.

And last, but not least…!

  • ArbPL now uses socket.io magic to provide realtime-ish updates.  Your browser creates a websocket/weird flash object bridge/XHR long-poll/etc. in order to be able to get the build information that’s hot off the press.  The “ish” part of realtime-ish comes from the bit where the presses are a cron job that runs every 3 minutes.  I think I have appropriate abort/retry logic in place now to contend with the half-written JSON and hanging HTTP requests, but I’m still sticking with the safety of a cron job with a built-in death clock until the logs agree with me.

Important caveats:

  • If the server restarts, the client is smart enough to reconnect, but not smart enough to re-establish its subscriptions with the server.  Refreshing the page or clicking the ArbPL logo in the upper left to go back to the list of trees and then picking a tree will resolve this issue.  The server only restarts when I push to production, as it were, so there is some other problem if you find yourself constantly needing to refresh.
  • Builders in the failure list look like they want to do something when you click on them, but this is a lie for all non-Thunderbird mozmill builders at the current time; those are the secrets ArbPL is keeping from you.

You can use ArbPL for yourself at arbpl.visophyte.org.  You can make your own using the source on github.

As noted by mcote in his post on autolog, the automation team is working on something in this domain, autolog, and I’m hoping to work with them to collaborate / pass-the-torch / or port Thunderbird’s mozmill logging to their solution so Thunderbird can have its fancy logging cake but not have to bake it too.

If you would like to discuss ArbPL/autolog/etc., I think the mozilla.tools group is supposed to be the right place for that.  One caveat is that I can’t seem to find the mailing list proper to sign up, just the newsgroup and google group…?  (Neither https://lists.mozilla.org/listinfo or https://mail.mozilla.org/listinfo/ seem to know about it.)

build matrices for arbitrarypushlog

Posted on by

A TraceMonkey Push

As I mentioned in my teaser on rich thunderbird mozmill logs, in order to get the build logs and provide failure clustering you already have to do most of the stuff tinderboxpushlog does.  One of the key things is summarizing the builds in a way that is useful, where the definition of “useful” probably varies a lot between users.

While Thunderbird has an extremely boring set of build types (build, xpcshell, mozmill), my first take on summarizing them was no good.  While fixing it, I decided to feature creep (out of my hobby time allocation) and see if I could create a presentation that could handle the prolific Firefox and TraceMonkey trees.

A Firefox Push

While I am not going to claim it’s perfect, I like it.  It’s probably also important to caveat that not all Tracemonkey builds are categorized.  The mobile talos runs identify themselves by a completely different set of names from the desktop ones, and there’s not really room for columns for that.  Additionally, some builds cite a revision for “mobile-browser”, but we ignore that extra meta-data.  Although the design was intended to handle Thunderbird’s repository where each build is a tuple of “comm-central” and “mozilla-central” revision used, we really need to have that tuple for every build in the tinderbox tree, and TraceMonkey is not providing it.  (We could kick builds without the info to the “outer” push as an enhancement.)

A Thunderbird (well, comm-central) push.

As a gesture of friendship to non-Thunderbird trees, we now also process mochitest and reftest logs, although I’m failing to surface some of the details retrieved in the UI.

Anywho, you can see arbpl in action for yourself at arbpl.visophyte.org.  It cron scrapes every 5 minutes.  The error recovery logic is not production-grade yet; the scraper can fall victim to partially written tinderbox JSON files on the tinderbox server, which means that some trees might not see updates for longer than that.  And various other things may go wrong too.  The client does not auto-refresh or use Socket.IO or anything.  If you want to run your own, hit github.  If you want to read the source, you should probably hit github too, because the production serving mode is reasonably optimized and crams all the JS into a single (gzipped) file.

teaser: Rich contextual information for Thunderbird mozmill failures

Posted on by

Sometimes Thunderbird mozmill unit tests fail.  When they do, it’s frequently a mystery.  My logsploder work helped reduce the mystery for my local mozmill test runs, but did nothing for tinderbox runs or developers without tool fever.  Thanks to recent renewed activity on the Thunderbird front-end, this has become more of a problem and so it was time for another round of amortized tool building towards a platonic ideal.

The exciting pipeline that leads to screenshots like you see in this post:

  • Thunderbird Mozmill tests run with the testing framework set to log semi-rich object representations to in-memory per-test buckets.
  • In the event of a test failure, the in-Thunderbird test framework:
    • Gathers information about the state of the system now that we know there is a failure and emit it.  This takes the form of canvas-based screenshots (using chrome privileges) of all open windows in the application, plus the currently focused element in the window.
    • Emits (up to) the last 10 seconds of log events from the previous test.
    • Emits all of the log events from the current test.
  • The python test driver receives the emitted data and dumps it to stdout in a series of JSON blobs that are surrounded by magical annotations.
  • A node.js daemon doing the database-based tinderboxpushlog thing (like my previous Jetpack/CouchDB work that found CouchDB to not be a great thing to directly expose to users and died, but now with node and hbase) processes the tinderbox logs for the delicious JSON blobs.
    • It also processes xpcshell runs and creates an MD5 hash that attempts to characterize the specific fingerprint of the run.  Namely, xpcshell emits lines prefixed with “TEST-” that have a regular form to describe when the pending test count changes or when specific comparison operations or failures occur.  It is assumed that tests will not comparison check values that are effectively random values, that the comparisons will tend to be deterministic (even if there are random orderings in the test) or notable when not deterministic, and thus that the trace derived from this filtering and hashing will be sufficiently consistent that similar failures should result in identical hashes.
    • Nothing is done with mochitests because: Thunderbird does not have them, they don’t appear to emit any context about their failures, and as far as I can tell, frequently the source of a mochitest failure is the fault of an earlier test that claimed it had completed but still had some kind of ripples happening that broke a later test.
  • A wmsy-based UI does the UI thing.

    The particular failure shown here is an interesting one where the exception is telling us that a popup we expected to open never opened.  But when we look at the events from the log, we can see that what happened is the popup opened and then immediately closed itself.  Given that this happened (locally) on linux, this made me suspect that the window manager didn’t let the popup acquire focus and perform a capture.  It turns out that I forgot to install the xfwm4 window manager on my new machine which my xvnc session’s xstartup script was trying to run in order to get a window manager that plays nicely with mozmill and our focus needs.  (Many window managers have configurable focus protection that converts a window trying to grab focus into an attention-requested notification.)

    This is a teaser because it will probably be a few more days before the required patch lands on comm-central, I use RequireJS‘ fancy new optimizer to make the client load process more efficient, and I am happy that the server daemons can keep going for long stretches of time without messing up.  The server and client code is available on github, and the comm-central patch from hg.