{"id":916,"date":"2016-03-21T05:28:09","date_gmt":"2016-03-21T09:28:09","guid":{"rendered":"https:\/\/www.visophyte.org\/blog\/?p=916"},"modified":"2016-03-21T05:28:09","modified_gmt":"2016-03-21T09:28:09","slug":"web-worker-assisted-email-visualizations-using-vega","status":"publish","type":"post","link":"https:\/\/www.visophyte.org\/blog\/2016\/03\/21\/web-worker-assisted-email-visualizations-using-vega\/","title":{"rendered":"Web Worker-assisted Email Visualizations using Vega"},"content":{"rendered":"

\"Faceted<\/a><\/p>\n

tl;dr: glodastrophe<\/a>, the experimental entirely-client-side JS desktop-ish email app now supports Vega<\/a>-based visualizations in addition to new support infrastructure for extension-y things and creating derived views based on the search\/filter infrastructure.<\/p>\n

Two of the dreams of Mozilla Messaging<\/a> were:<\/p>\n

    \n
  1. Shareable email workflows (credit to :davida<\/a>).\u00a0 If you could figure out how to set up your email client in a way that worked for you, you should be able to share that with others in a way that doesn’t require them to manually duplicate your efforts and ideally without you having to write code.\u00a0 (And ideally without anyone having to review code\/anything in order to ensure there are no privacy or security problems in the workflow.)<\/li>\n
  2. Useful email visualizations.\u00a0 While in the end, the only visualization ever shipped with Thunderbird was the simple timeline view of the faceted global search, various experiments happened along the way, some abandoned.\u00a0 For example, the following screenshot shows one of the earlier stages of faceted search development where each facet attempted to visualize the relative proportion of messages sharing that facet.<\/li>\n<\/ol>\n

    \"faceted<\/a><\/p>\n

    At the time, the protovis<\/a> JS visualization library was the state of the art.\u00a0 Its successor the amazing, continually evolving d3<\/a> has eclipsed it.\u00a0 d3, being a JS library, requires someone to write JS code.\u00a0 A visualization written directly in JS runs into the whole code review issue.\u00a0 What would be ideal is a means of specifying visualizations that is substantially more inert and easy to sandbox.<\/p>\n

    Enter, Vega<\/a>, a visualization grammar that can be expressed in JSON that can not only define “simple” static visualizations<\/a>, but also mind-blowing gapminder-style interactive visualizations<\/a>.\u00a0 Also, it has some very clever dataflow stuff<\/a> under the hood and builds on d3 and its well-proven magic.\u00a0 I performed a fairly extensive survey of the current visualization, faceting, and data processing options to help bring visualizations and faceted filtered search to glodastrophe and other potential gaia mail consumers like the Firefox OS Gaia Mail App.<\/p>\n

    Digression: Two relevant significant changes in how the gaia mail backend was designed compared to its predecessor Thunderbird (and its global database) are:<\/p>\n

      \n
    1. As much as can possibly be done in a DOM\/Web Worker(s) is done so.\u00a0 This greatly assists in UI responsiveness.\u00a0 Thunderbird has to do most things on the main thread because of hard-to-unwind implementation choices that permeate the codebase.<\/li>\n
    2. It’s assumed that the local mail client may only have a subset of the messages known to the server, that the server may be smart, and that it’s possible to convince servers to support new functionality.\u00a0 In many ways, this is still aspirational (the backend has not yet implemented search on server), but the architecture has always kept this in mind.<\/li>\n<\/ol>\n

      In terms of visualizations, what this means is that we pre-chew as much of the data in the worker as we can, drastically reducing both the amount of computation that needs to happen on the main (page) thread and the amount of data we have to send to it.\u00a0 It also means that we could potentially farm all of this out to the server if its search capabilities are sufficiently advanced.\u00a0 And\/or the backend could cache previous results.<\/p>\n

      For example, in the faceted visualizations on the sidebar (placed side-by-side here):<\/p>\n

      \"faceted-histograms\"<\/p>\n

      In the “Prolific Authors” visualization definition<\/a>, the backend in the worker constructs a Vega dataflow (only!).\u00a0 The search\/filter mechanism is spun up and the visualization’s data gathering needs specify that we will load the messages that belong to each conversation in consideration.\u00a0 Then for each message we extract the author and age of the message and feed that to the dataflow graph.\u00a0 The data transforms bin the messages by date, facet the messages by author, and aggregate the message bins within each author.\u00a0 We then sort the authors by the number of messages they authored, and limit it to the top 5 authors which we then alphabetically sort.\u00a0 If we were doing this on the front-end, we’d have to send all N messages from the back-end.\u00a0 Instead, we send over just 5 histograms with a maximum of 60 data-points in each histogram, one per bin.<\/p>\n

      Same deal with “Prolific domains”, but we extract the author’s mail domain and aggregate based on that.<\/p>\n

      \"Authored<\/p>\n

      Similarly, the overview Authored content size over time heatmap visualization<\/a> sends only the aggregated heatmap bins over the wire, not all the messages.\u00a0 Elaborating, for each message body part, we (now) compute an estimate of the number of actual “fresh” content bytes in the message.\u00a0 Anything we can detect as a quote or a mailing list footer or multiple paragraphs of legal disclaimers doesn’t count.\u00a0 The x-axis bins by time; now is on the right, the oldest considered message is on the left.\u00a0 The y-axis bins by the log of the authored content size.\u00a0 Messages with zero new bytes are at the bottom, massive essays are at the top.\u00a0 The current visualization is useless, but I think the ingredients can and will be used to create something more informative.<\/p>\n

      Other notable glodastrophe changes since the last blog post:<\/p>\n