A primitive visualization augments the gloda “other messages by author” listing by showing the messages sent by the author over time.  Messages are stacked by day.  The currently selected message is in darkest blue and also very wide.  Other messages from the same thread/conversation are in lighter blue and less wide.  Messages not in the conversation are light grey and rather narrow.

It’s not clickable, it lacks any form of scale or any feedback at all, and there are scaling issues.  (If anyone wants to save me the effort of figuring out how to get the canvas to maintain a 1:1 pixel mapping to the actual display and still ‘flex’ by adding/losing pixels, please do drop me a message or leave a comment.)  These will all change, but not yet.

I’ve pushed the changes to the mercurial repos and updated the stable tag, but I’m not publishing updated xpi’s, so you’ll need to roll your own if you care.  (The DB schema has not changed and so does not need to be blown away.)

I am declaring milestone 1 of gloda (the global database extension for Thunderbird 3.x) / expmess (the experimental message view extension for Thunderbird 3.x) reached.  Milestone 1 basically consists of:

• It statically indexes all of your folders.  It does not track changes made to your mailboxes.  It will become confused and angry as time goes on and your message stores change but it stays the same.  Thankfully, it is also passive aggressive and will merely stop doing useful things rather than trying to eat your data.  It also refuses to change its ways; if you try and trick it into indexing a message it has already processed but you moved, it will not update the message’s index.  You can, however, trick it into indexing new messages.
• The indexing sorta happens in the background and has pretty, if dubious from an UX perspective, progress bars (see screenshot).  This was stolen from M3, making M1 wildly more usable than originally planned.  At least on my computer, I didn’t notice much performance impact from the indexing, but my system is arguably fairly beefy.  This can all be tweaked though, especially once we hook the nsIIdleService in.
• It adds a “data mine” pane to the right side of the message window.  It has a splitter so you can hide it if you want, but you can never be rid of it.  The data mine shows you the other messages in the current thread and other messages sent by the author… globally!
• If you double-click on a message in the “data mine” added by expmess, it will take you there!  This is stolen from M2.
• It will print out a lot of debug on standard out.  It used to print more.

Having said all that, you can get the XPI’s here if you are using Thunderbird/Shredder 3.0a2pre or later, and your build is from July 5th 2008 or later.  You need to install both of them if you want anything interesting to happen.  The easiest way to do this is go to “Tools”…”Add-ons” in Shredder, and drag the links into the add-ons pane, at which point it will prompt you and such.  These extensions will not auto-update.

• gloda m1.xpi
• expmess m1.xpi

And the code (in mercurial) is here:

• gloda
• expmess

Because of the static indexing, you will probably want to install this extension, mention something about needing to wear sunglasses because of the brightness of the future, and then uninstall it.

Un-installation consists of:

1. Disable / remove the gloda and expmess extensions.
2. Delete the global-messages-db.sqlite file from your profile directory.  Or don’t.  It’s up to you.

I’ll be following this post up with a newsgroup post on mozilla.dev.apps.thunderbird on Monday with more details about planning out the rest of the milestones, as well as the arbitrary changes I had made to my (always tenative) milestone 1 plan.  Discussion about the global database is probably best directed to the newsgroup, but feel free to post comments here if you want too.

The screenshot scenario is that we have 2 months of apache-dev archives in two separate folders, July and August.  We have a message selected in the month of July, yet the ‘data mine’ to the right is able to show the messages in the thread spanning both months (and therefore folders), as well as showing all the other messages sent by the author spanning both months (and therefore folders).

I’ll post again with some links and what not once I hit the actual milestone 1.  Be aware that every-day usability is targeted for milestone 3, but that should still happen pre-summit.

As an example starting point, take a look at messageWindow.js’s global reads or global writes.  From either of these you can click on other files in the sidebar to go to them.  There is no execution trace, so the overview diagram won’t be any help.  Be sure you have some free memory available and won’t try and hurt me if Firefox does something crazy.  Note: for long documents, some delay is expected as it attempts to apply various fix-ups.  Also note: clicking on things in the global reads/global writes tab won’t get you anywhere for now.

A quick example of a global read from that file (and who writes to that global):

gDBView

• Top Level: commandglue.js
• Function: ClearThreadPane (in commandglue.js)
• Function: CreateBareDBView (in commandglue.js)
• Function: RerootFolder (in commandglue.js)
• Function: SwitchView (in commandglue.js)
• Function: openFolderTab (in mailWindowOverlay.js)
• Function: setMailTabState (in mailWindowOverlay.js)
• Function: restorePreSearchView (in searchBar.js)
• Function: MsgGroupBySort (in threadPane.js)

A quick example of a global write from that file (and who reads from that global):

SelectFolder

• Function: OpenMessageByHeader (in mailContextMenus.js)
• Function: OpenInboxForServer (in mailWindow.js)
• Function: selectFolder (in mailWindow.js)
• Function: openFolderTab (in mailWindowOverlay.js)
• Function: LoadNavigatedToMessage (in messageWindow.js)
• Function: LoadNavigatedToMessage (in msgMail3PaneWindow.js)
• Function: OnLocationTreeSelect (in msgMail3PaneWindow.js)
• Function: SelectServer (in msgMail3PaneWindow.js)
• Function: loadStartFolder (in msgMail3PaneWindow.js)
• Function: RenameFolder (in widgetglue.js)
• Function: loadInboxForNewAccount (in accountUtils.js)
• Function: DropOnFolderTree (in messengerdnd.js)
• Function: CrossFolderNavigation (in msgViewNavigation.js)

The code, as always, is available at http://hg.mozilla.org/users/bugmail_asutherland.org/pecobro/

The title isn’t true yet, but it’s disturbingly almost-true.  Pecobro’s hackish underpinnings have been “accidentally” replaced with surprisingly forward-looking code capable of supporting a much fancier feature set.  (I didn’t mean to, but I got tricked because the incremental cost to doing the ‘right thing’ was always so low.)

The trace that you can see here, by clicking on any of this text what has coloring (and using firefox 3 in a session that you don’t mind if it crashes), is of running Mark Banner (Standard8)’s Thunderbird bloatTest on OS X with DTrace probes enabled, but without actually doing the bloat testing.  So Thunderbird starts up, opens the address book, closes it, opens the compose window, closes it, and then quits.

Here is a preliminary processed trace from a run triggering bug 296453.  Be forewarned that there is some missing information from the underlying trace and so it’s not all that it could be.  I think the XPConnect probes need to be fleshed out slightly more (and processed).

The code (both pecobro and mozilla codebase patches) is at: http://hg.mozilla.org/users/bugmail_asutherland.org/

What crazy overkill things can pecobro do now?

• Parse all of the javascript code used in Thunderbird, including nearly all (maybe all?) of the 1.x enhancements.  The parser is still a bit hackish, especially when it comes to support of regexes, but it actually parses things.  Thanks to Chris Lambrou for posting his initial JavaScript.g antlr3 grammar (and BSD licensing it) that I used as a basis for my hacky version.  This was certainly a learning experience; I now know that javascript can never replace python in my heart…
• Find all of the places functions can be defines in javascript code and do a pretty good job at figuring out reasonable names for them.  This even includes “(new)? Function()”.  This allows us to use the function-info DTrace probes to use line number information to identify functions.  We do this so that we can identify anonymous functions.
• Handle XBL as well as we handle pure-javascript files.  (Parsing/AST fix-ups/syntax highlighting with AST traversal/etc.)
• Parse/interpret disturbingly large portions of makefiles.  Sure, it ignores the rules, and I’ve only implemented the functions I require, but it does all the conditional and include stuff with variable expansion, including some key functions like foreach/filter/etc.
• Parse jar manifests sufficiently well to figure out what file ends up at what chrome path.
• Cache its javascript parse results at both the AST and (sorta-semantic) processing level, among other things.  Hooray for cerealizer which dares persist/de-persist that which pickle dare not.  (Nested classes.)  (Note: We don’t cache XBL stuff yet, and the ASTs are large enough that de-persisting them can be quite perceptible.)

What is not crazy, but still nice:

• The traces are now much more reliable thanks to modifications to the Mozilla USDT probes to provide sufficient information for us to distinguish between JavaScript execution contexts (namely, JSContext).  Modifications were also made to avoid the ‘guessing’/ghost value problems that happen when native functions were called.
• A foolish bug in the sparkbar calculation has been fixed.  aka “Now With Accuracy”!
• If a function in a trace called another function or was called by another function, this will be displayed inline in the source display.
• Infer ‘native’ functions (with the above additional modifications to the Mozilla USDT probes), assigning them to an object.  This ends up being Object, ChromeWindow, or other native Class type.  Some of this may be mooted by getting the XPConnect probes reliably in the mix.

What doesn’t it do?

• Add the XPConnect js_Invoke probes; I punted on that because that was turning out to be difficult to get right.
• It ignores .xul files for now.  Although xul files primarily appears in a caller context (as told by function-info probes), they can also be a callee when someone pulls a fast one and inlines some simple code in a script tag.  We brutally mis-attribute the call to the previous function when this happens.  This will eventually get resolved because we will need to understand .xul files for namespace reasons.  Also, people sound like they’re interested in who overlays what and the like, and that’s sorta right up our alley thanks to all the overkill stuff we do.
• Exhaustively determine reads from/contributions to the ‘global’ (window or what not) namespace/scope.  The groundwork is there in terms of understanding the contribution of top-level statements to the namespace or its reads from it, but we don’t traverse into functions.
• Associate functions with an object/type (ignoring the native function case).  This requires more semantic understanding.
• Clicking on functions still doesn’t do anything.  I disabled that before my first post on pecobro due to firefox-crashing issues, and still haven’t gotten around to resolving it.
• A usable overview visualization.  The overview diagram has become cluttered by the presence of so many ‘relevant’ (touched) files.

At the beginning of last week, I had gotten dtrace working on a Mac Mini using the Mozilla javascript-provider probes.  Very cool stuff, but it left me with several questions about what would really be best to do next:

• How do I best understand what I’m seeing?  (Most of the codebase is brand new to me…)
• How do I share the data with others in a way that is both comprehensible and allows them to draw their own conclusions from the data?
• What can I do to reduce the effort required to work on performance problems?

I was tempted to just try and just dig into the system so I could have something to show immediately, but I knew it would still take a while to see the big picture just using an editor/ctags/lxr/opengrok, even informed by dtrace.  And even then, that big picture doesn’t scale well; whatever picture I managed to formulate would be stuck inside my head…

So my solution was to try and build a tool that could help me accomplish my short-term goals soon, and have the potential to grow into a usable solution to all of the above… eventually.  The goal, in a nutshell, is to provide a code browser for javascript that is able to integrate performance information (retrieved from traces) alongside the code.  Seeing that lxr/mxr and opengrok didn’t understand javascript or XBL all that well, it also seemed feasible to try and improve on their browsing capabilities for javascript.  A far-down-the-road goal is also to be able to pull in information from the underlying C++ code as well, potentially leveraging dehydra, etc.  (This would primarily be for understanding what happens when we leave the javascript layer, not trying to be the same solution for C++ space.)

So what can it do so far?  You can go try it for yourself if you like as long as you keep your expectations very low and realize the current state does not reflect all of the bullet points below.  Also, you probably want firefox 3.0.  Or you can read my bullet points:

• Parse custom DTrace script output!  The Mozilla DTrace probe points could probably use a little love to improve what we are able to get out.  Also, I think it’s betraying us somewhere.
• Parse JavaScript! Sorta!  (I hacked in support for the regular expression syntax, but I haven’t corrected the ambiguity with division, so things with division break.  Also, there’s at least one or two other glitches that cause early termination.) [Yay antlr!]
• Parse XBL!  Even with entity inlining!  Even when people put #ifdefs in the XML document! Sorta!  We don’t actually do anything intelligent with the XBL right now or with its JavaScript, but it won’t take much to get that much improved. [Yay elementtree!]
• Visualize some stuff!  Inter-file relationship graph in the overview.  In the code and ‘Funcs’ sidebar tab you get a sparkbar where each bar represents a time interval.  The height of the par is the percentage of possible time we could have spent in that time interval.  Red means we belive that time was spent in the function itself, green means we think we spent that time in calls to other functions. [Yay visophyte!]
• Navigate with history!  Click on the overview graph and you go to things.  Click on the file names in the ‘Files’ list and you go to the files.  I tried to make it so you could click on function names in the side bars to go to them, but jquery.scrollTo and/or firefox 3.0b5 had serious crashing issues.  [Yay jquery, jquery.history!]
• See syntax-highlighted code with random headings intertwined (shows the parser worked) and potentially a visualization.  [Yay pygments!]

My hope in the near-term is to fix the outright bugs (parsing issues), get XBL going, and then augment the function information with more trace-derived data including more traditional call-stacks, etc.  Then the tool should be sufficiently usable that my immediate focus can change to creating automated tests to actually gather performance/execution traces so we can use the tool for what I started it for.  This may also get shelved for a while if it turns out that we need action (patches) immediately.

This visualization is the result of an adapted visichron.py script (from my chronicle-recorder ‘chroniquery’ bindings) run against the output of a custom DTrace script (using the Mozilla JS providers) on OS X.  It’s a proof-of-possibility rather than anything immediately useful.

The differences from the last visichron post (using chronicle-recorder as a back-end) are:

• Node hues are distinct based on the file the javascript was executed from.  (Saturation still varies with amount of time spent in the function.)
• Graph layout is done using graphviz’s neato’s “ipsep” (experimental) mode.  This works fantastically well at reducing/eliminating overlap.  Having said that, a hierarchical layout may make more sense.
• Ring colors are based on call depth (so redundantly encoded with the ring radius) rather than any knowledge about the control-flow.  Full control-flow information is not readily available and would be extremely expensive, but we could provide at least some degree of approximation using the calls made by the function as indicators.  Of course, the ring visualization at this point and for this purpose is probably better represented as non-nested (side-by-side rings of different radii; not containing each other) faux-continuous ring slices with transparency varying by amount of time spent in the function at that call-depth.  This would simplify the object model, allowing for non-insane use of the SVG backend and interactivity enhancements.

Hobby-wise, this will translate into a focus on visualizing e-mail from within Thunderbird. My nascent Python library, visophyte, will likely have a JavaScript sibling, but will not be abandoned.

Job-wise, I expect to focus on improving Thunderbird for both users and extension developers. Although my interest in creating a useful visualization extension will inform my efforts, it will not be my focus. Which is to say, do not worry that I will be engaging in flights of fancy and neglecting the core of Thunderbird. However, do be happy that good visualizations will depend on non-trivial data analysis and snappy, interactive behaviour and that this should translate into good things for Thunderbird, even if you don’t like shiny things.

It wasn’t an easy decision to leave my current employer (I have only great things to say about The PTR Group; check them out if you’re in the greater DC metro area), but Mozilla Messaging is an exceedingly rare opportunity that I could not pass up.

My personal laptop rolls amd64-style (rather than i686), and chronicle-recorder’s valgrind component was not working on it (”illegal instruction”). I have done some vendor-branch dancing to get chronicle-recorder’s valgrind sub-directory to use valgrind 3.3.0. A bzr branch of just the valgrind subdirectory (drop-in or build separately and make sure you invoke valgrind from that location) is available here: http://www.visophyte.org/rev_control/bzr/valgrind-chronicle/valgrind-chronicle/

A probably faster location to bzr branch from is here: http://clicky.visophyte.org/rev_control/bzr/valgrind-chronicle/valgrind-chronicle/

and a tarball of the tip is here: http://www.visophyte.org/rev_control/tarballs/valgrind-chronicle-r6.tar.bz2

I have also updated chroniquery (my python binding for chronicle-query’s JSON interface, and its own set of tools that build on it) to work with amd64. Its bzr branch is here: http://www.visophyte.org/rev_control/bzr/chroniquery/trunk/

The goal of all of this was to be able to run chronicle against Thunderbird, which I was able to do. Unfortunately, visichron (the visualizing part of chroniquery using visophyte) is not quite ready for such an undertaking at this time.  (Not to mention a few C++ symbols issues…)

However, it was powerful enough to handle visualizing the trace resulting from invoking python on a python file with just “print ‘hello world’” in it. So that’s what you see here, albeit limited to only 3 call levels deep. Click on the upper picture to see the whole thing at 2000×2000, or just look on the lower picture to get an idea of what’s going on. Just like my first post using visichron, the rings are colored based on a (naive) control-flow-taken basis. The ring colors are per-function, however. Also, the node colors are ‘hot’ colored based on how many ‘ticks’ were spent inside the functions, multiple counting for recursion.

Other interesting changes include some primitive watch functionality for chronisole’s ‘trace’ mode. Also, the previously unmentioned readchron.py now understands and prints BunchedEffect and RegEffect info. (readchron aspires to be along the lines of readelf for chronicle databases, but with more colors.)