Tag Archives: jetpack

test-case-mode support for jetpack unit tests in emacs

Posted on by

Use Jetpack?  Occasionally write unit tests so you won’t be a complete hypocrite when criticizing other people’s code?  Think that picture up above looks more useful than this?:

error: fail: list contents ("4,5,6,7" != "3,4,5,6,7")
info: Traceback (most recent call last):
  File "resource://wmsy-jetpack-core-lib/timer.js", line 57, in notify
    this._callback.apply(null, []);
  File "resource://wmsy-jetpack-core-lib/unit-test.js", line 257, in anonymous
    timer.setTimeout(function() { onDone(self); }, 0);
  File "resource://wmsy-jetpack-core-lib/unit-test.js", line 282, in runNextTest
    self.start({test: test, onDone: runNextTest});
  File "resource://wmsy-jetpack-core-lib/unit-test.js", line 300, in start
    this.test.testFunction(this);
  File "resource://wmsy-jetpack-core-lib/unit-test.js", line 57, in runTest
    test(runner);
  File "resource://wmsy-wmsy-tests/test-vs-static.js", line 37, in anonymous
    slice.seek(6, 2, 2);
  File "resource://wmsy-wmsy-lib/wmsy/viewslice-static.js", line 51, in anonymous
    this._list.slice(this.bufLow, this.bufHigh));
  File "resource://wmsy-wmsy-tests/test-vs-static.js", line 31, in anonymous
    "list contents");
  File "resource://wmsy-jetpack-core-lib/unit-test.js", line 229, in assertEqual
    this.fail(message);
  File "resource://wmsy-jetpack-core-lib/unit-test.js", line 147, in fail
    console.trace();

Then do I have an .el for you!  This lives here.  You mileage may vary and may involve things catching on fire which can, in turn, affect your mileage.

ediosk: an emacs buffer switcher for the rest of us

Posted on by

Emacs users and would-be emacs users, are you tired of those emacs developers in their ivory towers not supporting buffer switching via touch-screen on a computer that’s not running emacs and using modern web browser technology instead of disproven parentheses-based technology?  Be tired no more!*

Thanks to Christopher Wellons and Chunye Wang’s work on emacs-httpd it is a simple matter to expose a JSON representation of the current set of frames/windows/buffers in your emacs session and provide non-REST manipulation mechanisms via a webserver implemented in elisp.

Once you have exposed an API, it is a subsequent simple matter to implement some JavaScript that understands these things and presents a nice UI.  In this case, we have used the Jetpack SDK, wmsy (the Widget Manifesting SYstem, an widgeting framework I am developing), and protovis.

The screenshot basically captures the entire feature-set:

  • A protovis-based visualization that shows the location of all of the emacs “windows” (the things that show buffers).  Emacs reports to us the pixel-space coordinates/sizes of the “frames” (GUI windows) and “windows”, so this all comes magically for free.  The downside is your emacs windows need to be in the same coordinate space, so use of multiple X displays without use of DMX will likely lead to weird overlap.
    • The selected “window” in each “frame” gets a diamond.  The active frame’s diamond gets to be black instead of gray.
    • Clicking on a “window” focuses/raises the containing “frame” and selects the “window”.
  • Buffers grouped by the directory they live in (if they have a backing file).
    • Buffers visible in windows have their background composed of the colors for all the windows they are in.
    • Buffers that are modified have their text colored red.
    • Buffers that have not been freshly displayed in a window recently have their text colored grey.
    • Clicking on a buffer displays it in what the UI believes to be the currently selected frame’s currently selected window.

* This entire paragraph is a joke**; no flaming necessary.

** ediosk is not a joke though.  I seriously have a touch-screen monitor hooked up to my windows build machine to the right of my two monitors hooked up to my linux/primary development machine.  While c-x b (icicle mode) will still be my dominant buffer switching mechanism, I expect ediosk to prove useful/amusing for cases where the number of buffers greatly exceeds my mental stack, when I am switching contexts, or when I am working in multiple branches simultaneously.

understanding where layout goes wrong with gecko reflow debug logs (Part 1)

Posted on by

HTML and CSS provide awesome layout potential, but harnessing that potential to do your bidding can sometimes be problematic.  I suspect all people who have written HTML documents have been in a situation where they have randomly permuted the document structure and CSS of something that should work in the hopes of evolving it into something that actually does work.  Matching wits with a black box that is invulnerable to even pirate-grade profanity is generally not a pleasant experience.

It turns out gecko has a way to let you see inside that black box.  Well, actually, multiple ways.  There’s a layout debugger (that seems broken on trunk?) that can display visual overlays for box sizes/events/reflow counts, dump info about paints/invalidations/events as they happen as well as dumping the current content tree and frames.  Even better, gecko’s frame reflow debugging mechanism will dump most of the inputs and outputs of each stage of reflow calculations as they happen.  With some comparatively minor patches[1] we can augment this information so that we can isolate reflow decisions to their origin presentation shell/associated URL and so that we know the tag name, element id, and class information on HTML nodes subjected to reflow calculations.  A reasonably sane person would want to do this if they were planning to be doing a lot of potentially complicated HTML layout work and would a) benefit from better understanding how layout actually works, b) not want to propagate layout cargo culting or its ilk from the code being replaced, c) not want to waste days of their lives later the next time this happens, d) help locate and fix layout bugs if bugs they be so that all might benefit.

Of course, with logs that effectively amount to execution traces, examining them by hand is frequently intractable unless you really know what you’re looking for or are dealing with a toy example.  My non-reduced problem case resulted in 58,107 lines, for one.  So writing a tool is a good idea, and writing it in JS using Jetpack doubly so.

In any event, the problem is I am trying to use the flexible box model to create an area of the screen that takes up as much space as possible.  In this space I want to be able to house a virtual scrolling widget so I use “overflow: hidden”.  Regrettably, when my logic goes to populate the div, the box ends up resizing itself and now the whole page wants to scroll.  Very sad.  (Things work out okay with an explicitly sized box which is why my unit tests for the virtual scrolling widget pass…)

Let’s formulate a query on the div of interest (which I will conceal) and then see what the first little bit of output is:

*** Box 24 tag:div id: classes:default-bugzilla-ui-bug-page-runBox
*** scroll 25 tag:div id: classes:default-bugzilla-ui-bug-page-runs
scroll 25 variant 1  (parent 24) first line 406
  why: GetPrefSize
  inputs: boxAvail: 0,UC
          boxLast: 0,0
          reflowAvailable: 0,UC
          reflowComputed: 0,UC
          reflowExtra: dirty v-resize
  output: prefWidth: 0
          minWidth: 0
          reflowDims: 0,0
          prefSize: 0,0
          minSize: 0,0
          maxSize: UC,UC
  parent concluded: minSize: 0,0
                    maxSize: UC,UC
                    prefSize: 2,0
scroll 25 variant 2  (parent 24) first line 406
  why: Layout
  inputs: boxAvail: 771,1684
          boxLast: 0,0
          reflowAvailable: 771,UC
          reflowComputed: 771,1684
          reflowExtra: dirty dirty-children h-resize v-resize
  output: prefSize: 0,0
          minSize: 0,0
          maxSize: UC,UC
          reflowDims: 771,1684
          layout: 2,0,771,1684
  parent concluded: minSize: 0,0
                    maxSize: UC,UC
                    prefSize: 0,0
                    layout: 0,0,773,1684

This is the general pattern we will see to the reflows.  The parent will ask it what size it wants to be and it will usually  respond with “ridiculously wide but not so tall”.  (Not in this first base case, but the next one responds with a prefsize of “1960,449”, and that’s pixels.) The parent will then perform layout and say “no, you need to be taller than you want to be”, at least until I start cramming stuff in there.

So we skim down the output to find out where things first went off the rails…

scroll 25 variant 16  (parent 24) first line 20548
  why: GetPrefSize
  inputs: boxAvail: 1960,UC
          boxLast: 771,1686
          reflowAvailable: 1960,UC
          reflowComputed: 1960,UC
          reflowExtra: dirty-children h-resize v-resize
  output: prefWidth: 1960
          minWidth: 352
          reflowDims: 1960,1755
          prefSize: 1960,1755
          minSize: 352,1755
          maxSize: UC,UC
  parent concluded: minSize: 0,0
                    maxSize: UC,UC
                    prefSize: 1962,1755
scroll 25 variant 17  (parent 24) first line 20548
  why: Layout
  inputs: boxAvail: 771,1755
          boxLast: 1960,1755
          reflowAvailable: 771,UC
          reflowComputed: 771,1755
          reflowExtra: dirty-children h-resize
  output: prefSize: 0,0
          minSize: 352,1755
          maxSize: UC,UC
          reflowDims: 771,1755
          layout: 2,0,771,1755
  parent concluded: minSize: 0,0
                    maxSize: UC,UC
                    prefSize: 0,0
                    layout: 0,0,773,1755

Okay, that looks pretty likely to be the area of concern.  The parent asked it for its ideal size, so it told it, but then the parent apparently decided to enlarge itself too.  That is not what we wanted.  We would have been cool if just the scroll #25 enlarged itself (or its block child #26 that corresponds to the same content node but which I have elided because it always says the same thing as its parent #25) since some frame needs to end up holding the overflow coordinate space.

Thus concludes part 1 of our exciting saga.  In part 2, we hopefully figure out what the problem is and how to fix it.  Lest anyone suggest the root problem is that I am completely off base and am not remotely reasonably sane for choosing this as a strategy to solve the problem… it works in chrome.  Which is not to say that my html/css is correct and firefox’s layout is wrong; it’s quite possible for layout engines to err or deal with unspecified behaviour cases in my favor, after all.  But it does make me want to understand what the layout engine is thinking and be able to do so with a minimum of effort in the future, since I doubt this is the last time I will not immediately understand the problem or that layout engines will differ in their behaviour.

For those who want to play along at home: the raw gzip’d /tmp/framedebug file (gunzip to /tmp/framedebug) that is the entirety of the trunk firefox log with debug output, the spliced output just for the one window derived from an invocation of “cfx run splice” (it will show up under the URL in /tmp/framedumps), and the output of the output of “cfx run summarize serial-12-0 summarize unique 22,24,25,26.  Those unique identifiers are deterministic but arbitrary values for the given file.  We discovered them by using the query on the CSS class using “cfx run summarize serial-12-0 summarize class wlib-wlib-virt-wlib-virt-container-root”.  The hg repo for the processing tool is here, the mozilla-central patches are: first and second.  A minor jetpack patch is also required for the command line stuff to work.

1: I was initially trying to avoid patching anything.  This didn’t work out, but it did cause the initial log file splicing logic to leverage the arena allocation scheme of presentation shells to allow us to to map frames back to their URLs.  Sadly, it turned out the arena allocation blocks requested from the upstream allocators are really small (4k or 1k) and all from the same source and so I had to instrument the allocation as well as adding debug output of the window/docshell/presshell linkages.  The output adds an unacceptable additional level of noise to the default DEBUG case; the right thing to do is likely to cause the reflow log debugging to emit the document URL before each logging outburst if it is different from the last outburst.

Thunderbird Quick Filter Bar extensions, they’re a thing!

Posted on by

The previously discussed Quick Filter Bar interface landed in time for Thunderbird (Lanikai) 3.1 beta 2 (whose release is real-soon-now).  Although the Quick Filter Bar already contains dangerously high levels of awesome, we made sure to make it extensible so you can cram even more awesome in.

As an example, I have created an extension that enables you to ‘pivot’ based on the currently selected message(s).

In its most straightforward manner of operation, you can click on an e-mail address in the message reader header and pivot by that e-mail address.  Out of the box, this will show you all the messages in the current folder sent by that user.  You can also (or only) show messages where that user is on the to/cc lines by toggling the buttons on the expando bar appropriately.

You can also constrain the display to only show messages within some time interval of the message(s) in question.

The more confusing way to access the pivot functionality is to simply toggle the facet on the quick filter bar.  When you toggle the filter on, we walk all of the selected messages and build up a list of the observed e-mail addresses for all of the senders and all of the recipients.  One trick is that we filter out any e-mail addresses associated with your own account in order to avoid the filter becoming too inclusive.  We save those e-mail addresses into our state.  We also walk all the messages and find the time range that the messages cover and save that.  These are used as the basis for the specific constraints you can enable from the expando bar.  Because the values are saved, the filter does not change when you change your selected messages.  You need to toggle the pivot filter off and on again (or use the ‘pivot’ option on the email address popup) to change the data we are pivoting on.

The extension can be found on AMO and in source form.  In a forward-looking move that is probably complicating things for now, I used the Jetpack SDK and its XUL extension support to implement the extension.  In practice, all the example uses Jetpack for is its CommonJS module system and the standard mozilla-style JS modules would have been sufficient.  All the UI stuff is done purely in XUL overlays bound to the backing code using addEventListener.

The broad strokes for implementing a Quick Filter Bar extension are:

  1. Overlay your toolbarbutton into the quick filter bar, preferably the #quick-filter-bar-collapsible-buttons box.  If you have an expando bar you want to contribute, you probably want to put that into the #quick-filter-bar-expando.  As things currently stand, multiple expando bars being uncollapsed at the same time will probably turn out poorly, so you may need to be pragmatic and put your bar outside that box or do something clever.
  2. Import resource:///modules/quickFilterManager.js, implement your filter, and register it with QuickFilterManager.defineFilter() (whose documentation explains the interface you need to conform to).

We will likely not be doing up full documentation for this extension point beyond the in-code documentation and in-tree/extension examples.  This is an advanced extension point by virtue of it touching nsIMsgDBHdrs, nsIMsgDBViews, nsIMsgSearchSessions, and friends, all of which have sharp edges.  The way forward with soft, human-friendly documented abstractions is gloda and that’s where (my) effort will be concentrated.  The awesome in this extension point is making it practical for people who already have the scars to utilize the full power of the message view/traditional search subsystem without losing a limb and do so in a previously impossibly short period of time.

PS: I made the icon!

Tinderbox results in bugzilla, jetpack times 2, CouchDB, review board

Posted on by

mstange‘s Tinderboxpushlog is awesome.  You know it, I know it, the many people whose sanity has been saved by it know it.  It is a fantastic improvement on checking the tinderbox; it lets you know the current state of the tree, the recent history of the tree, and how these things are correlated with recent commits.  What it is not good at (nor intended for) is to be a historical record.  Tinderboxpushlog ‘scrapes’ the tinderbox on-demand using tinderbox time windows and does not have the ability to key off anything but the time.

So if one refactored the scraper into a CommonJS module suitable for use with the newly rebooted jetpack, hooked it up to a cron job, and crammed its outputinto a CouchDB database, what would you get?

Exactly, something you can hook up to johnath and ehsan‘s magic bugzilla jetpack (last mentioned on the blog here).  You can install it from here.  (johnath/ehsan, please feel free to pull the changes into the upstream repo; I’m still wary of randomly pushing things into other people’s user repos…).  I know the presentation is ugly, not the least because the text labels are inconsistent with tinderboxpushlog; feel free to push into my user repo with improvements.  Oh, and for this to work you need to put an honest-to-goodness URL in your bugzilla comment or attachment description; there is no all-powerful regex if you type things out by hand.

You can find the thing that pushes thing into the couch using refactored tinderboxpushlog logic here.  Some cuddlefish runner tweaks (not all of which are likely advisable) can be found in my user jetpack-sdk repo.  (The new jetpack reboot is wicked awesome, by the way.)

Right now the cron job is running against the MozillaTry, Firefox, and Thunderbird3.1 trees on the tinderbox every 5 minutes.  While it should be pretty easy for me to keep the cron job and couch server online, I make no guarantees about the schema used in the couch, just that I will keep the jetpack in sync with it.  And if the service starts exceeding the resources of my (personal) linode, I may have to tweak polling rates or what not (‘what not’ meaning ‘up to and including turning it off’).

There is other work happening in this area and I am excited about that.  For example, I think brasstacks has an encompassing vision that should help provide historical data about which tests failed, etc.  With any luck, my efforts here will be mooted by buildbot and magic awesome dust.

The mention of reviewboard in the title is just that if you are using my review board stuff and put a link to your try server commit in the attachment description, we will use that to pull the official hg changeset as the basis for the diff.  The main benefit of this is that if your patch depends on other patches in your queue that are not yet in the trunk, the diff will still work out.  Specifically, if your queue had A, B, and C applied (where C is the tip) and you link to C, then we will provide a diff of C relative to B.  Please be aware that hg.mozilla.org is rather slow about providing the hg changeset diffs on demand so this will be at least an order of magnitude slower than the fetch of an already-available patch from bugzilla.  Repo with changes is here.

Review Board and Bugzilla reviews, take 3

Posted on by

I’ve updated my review board setup once more (part 2, part 1).  The low barrier to entry is now even lower.  “How low?”, you might ask.  “On the ground!”, I might say.  “What other low low price features with big big value are on offer? With more facts and less spiel?”, you might then also ask…

  • It now works with patches that have a header.  Patches that were the result of an mq import and then directly uploaded will tend to have headers.  This was why sometimes patches would fail to import with unlikely errors about empty patches.
  • There is now a magic URL scheme that automatically pulls the patch, creates a review and bounces you to the review.  If the review already exists, it just directly bounces you.  That URL scheme is http://reviews.visophyte.org/r/bzpatch/bug###/attach###/.  There are no authentication requirements on using this URL nor viewing the diff or associated reviews.  However, if you want to actually use the review mechanism to make comments then you will need to login via OpenID.  See part 2 for more on that.
  • I modified johnath and Ehsan Akhgari’s magic bugzilla jetpack so that it also adds a “Review” link to patches.  My modified repo is here.  You can install it from here.  You can see what the word “Review” looks like up in the first screenshot.
  • I upgraded to the reviewboard git trunk.  This adds some improvements to the diff display such as showing you function context information, even if the patch did not include it.  (And even if it did, too.  A lot of patches involving C++ truncate the function signature, whereas as you can see in the screenshot, you get the full text!)  I believe it is also supposed to be clever about recognizing moved blocks.

Limitations and other notes:

  • Patch fetching is synchronous and can take a while because we also parse it all up before we return.  Do not sit there hitting reload.  We’ll give you an error message if it doesn’t work out.  Not a great one, but an error message nonetheless.
  • Patches are still assumed to be against mozilla-central or comm-central (depending on the bugzilla product) trunk as of the moment we fetch the patch.  This means bit-rotted patches that you are only looking at now may fail to apply.  At the same time, patches where you clicked on the link back when it was timely and go back to look at them now that they are going out of style are still going to be applied against the same revision they were in the first place.
  • The repo with my modified changes (on the bzreview-master branch) was nuked and re-created because of the svn -> git transition by the reviewboard people.  So if you previously pulled, you should probably blow your old repo away rather than end up with a weird hybrid mixture.  (btw, hg-git works very nicely, with the caveat that its bookmark-based representation of the git branching idiom confuses pbranch really quite badly.)
  • Ping me on IRC or drop me an e-mail if you are experiencing reliable problems after having determined that the patch in question is not just full of gibberish.

Thunderbird Jetpack Teasers: Words per Minute in Compose

Posted on by 
jetpack.future.import("thunderbird.compose");
jetpack.thunderbird.compose.appendComposePanel({
  onReady: function (panel, composeContext) {
    let doc = panel.contentDocument;
    let msgNode = $("<span />", doc.body).appendTo(doc.body);
 
    let started = Date.now();
    setInterval(function() {
      let words = composeContext.getPlaintextContents().split(/\s+/);
      let secs = Math.ceil((Date.now() - started) / 1000);
      let wordsPerMinute = Math.floor((words.length * 60) / secs);
      msgNode.text(wordsPerMinute + " words per minute.");
    }, 1000);
 
    panel.show();
  },
  html: <><body style="overflow: hidden"></body></>
});

thunderbird-jetpack-words-per-minute-example

Thunderbird Jetpack messageDisplay.overrideMessageDisplay fun.

Posted on by

jetpack-twitter-follow-notification

As part of our goal to make it easy to write extensions for Thunderbird 3, we’ve been working on getting Jetpack running under Thunderbird and exposing Thunderbird-specific points. This is all experimental, but it’s having good results.

The first example replaces the message you get from twitter when someone follows you and instead shows you that person’s twitter page so you can see what they’ve written. Unfortunately, if you try and click on links on the page you will become sad because they all try and trigger your web browser. But Standard8 is hard at work resolving the content display issues. Besides demonstrating registration via a regex over the sender’s e-mail address, it also shows us extracting message headers from the message. Also, we introduce a small HTML snippet that precedes the nested web browser so it’s not just an embedded web browser.

jetpack.future.import("thunderbird.messageDisplay");
jetpack.thunderbird.messageDisplay.overrideMessageDisplay({
  match: {
    fromAddress: /twitter-follow-[^@]+@postmaster.twitter.com/
  },
  onDisplay: function(aGlodaMsg, aMimeMsg) {
    let desc = aMimeMsg.get("X-Twittersendername", "some anonymous jerk") +
      " has followed you on Twitter.  Check out their twitter page below.";
    return {
      beforeHtml:
        <>
          <div style="background-color: black; color: white; padding: 3px; margin: 3px; -moz-border-radius: 3px;">
            {desc}
          </div>
        </>
      url: "http://twitter.com/" + aMimeMsg.get("X-Twittersenderscreenname")
    };
  }
});

jetpack-amazon-big-total

Our second example of the extension point replaces e-mails from Amazon about an order (order confirmation and shipment confirmation) with the amount of money you spent on the order in BIG LETTERS (or rather BIG NUMBERS). It uses a regular expression run against the message body to find the total order cost. Then it generates a simple web page to present the information to you.

jetpack.future.import("thunderbird.messageDisplay");
jetpack.thunderbird.messageDisplay.overrideMessageDisplay({
  match: {
    fromAddress: /(?:auto-confirm|ship-confirm)@amazon.(?:com|ca)/
  },
  _totalRe: /Total(?: for this Order)?:[^$]+\$\s*(\d+\.\d{2})/,
  onDisplay: function(aGlodaMsg, aMimeMsg, aMsgHdr) {
    let bodyText = aMimeMsg.coerceBodyToPlaintext(aMsgHdr.folder);
    let match = this._totalRe.exec(bodyText);
    let total = match ? match[1] : "hard to say";
    return {
      html:
      <>
        <style><![CDATA[
          body { background-color: #ffffff; }
          .amount { font-size: 800%; }
        ]]></style>
        <body>
          you spent... <span class="amount">${total}</span>
        </body>
      </>
    };
  }
});

The modified version of Jetpack can be found here on the “thunderbird” branch. “about:jetpack” can be triggered from the “Tools” menu. Besides the development jetpack, you can also add jetpacks from the about:jetpack “Installed Features” tab by providing a URL directly to the javascript file. Unfortunately, I just tried installed more than one Feature at the same time and that fell down. I’m unclear if that’s a Thunderbird content issue, a problem with my changes, or a problem in Jetpack/Ubiquity that may go away when I update the branch.