visophyte: shiny? shiny.
type: channel
description: Andrew Sutherland writes things but (almost) always includes pictures to look at.
date: 2024-04-24T02:04:19Z
update period: hourly
update frequency: 1
update base: 2000-01-01T12:00+00:00
generator agent: https://wordpress.org/
items: Andrew’s Searchfox Roadmap 2022
items: Andrew’s Searchfox Vision 2022
items: Web Worker-assisted Email Visualizations using Vega
items: An email conversation summary visualization
items: Talk Script: Firefox OS Email Performance Strategies
items: webpd: a Polymer-based web UI for the beets music library manager
items: monitoring gaia travis build status using webmail LED notifiers
items: about:nosy can now show CPU usage using jsprobes
items: about:nosy is about:memory with charts, helps you lay blame more easily
items: The joy of integrated logging and log viewing with fancy logs

Andrew’s Searchfox Roadmap 2022
type: item
creator: Andrew Sutherland
date: 2022-11-10T05:49:25Z
subject: Uncategorized
description: Searchfox (source, config source) is Mozilla’s primary code searching tool for Firefox introduced by Bill McCloskey in 2016 which built upon prior work on DXR. This roadmap post is the second of two posts attempting to lay out where my … Continue reading →
encoded: Searchfox (source, config source) is Mozilla’s primary code searching tool for Firefox introduced by Bill McCloskey in 2016 which built upon prior work on DXR. This roadmap post is the second of two posts attempting to lay out where my personal efforts to enhance searchfox are headed and the decision making framework that guides them. The first post was a more abstract product vision document and can be found here. Discoverable, Extensible, Powerful Queries Searchfox has a new “query” endpoint introduced in bug 1762817 which is intended to enable more powerful queries. Queries are parsed using :kats‘ query-parser crate which allows us to support our existing (secret) key:value syntax in a more rigorous way (and with automatic parse correction for the inevitable typos). In order to have a sane execution model, these key/value pairs are mapped through an extensible configuration file into a pipeline / graph execution model whose clap-based commands form the basis of our testing mechanism and can also be manually built and run from the command-line via searchfox-tool. <img fetchpriority="high" decoding="async" width="1024" height="390" src="https://www.visophyte.org/blog/wp-content/uploads/2022/11/searchfox-query-example-1024x390.png" alt="A graphviz diagram of the execution graph for the "foo" query. The diagram captures that the execution graph consists of 2 phases, with the first phase consisting of 3 parallel pipelines: "file-search", "semantic-search", and "text-search". Those 3 pipelines feed into "compile-results" which then passes its output to the 2nd phase which contains the "display" job. If you're interested in more details, see below for the "check output for the query" link which links the backing JSON which is the basis for the graph." class="wp-image-985" srcset="https://www.visophyte.org/blog/wp-content/uploads/2022/11/searchfox-query-example-1024x390.png 1024w, https://www.visophyte.org/blog/wp-content/uploads/2022/11/searchfox-query-example-300x114.p…

Andrew’s Searchfox Vision 2022
type: item
creator: Andrew Sutherland
date: 2022-10-05T14:12:49Z
subject: Mozilla
subject: Searchfox
description: Searchfox (source, config source) is Mozilla’s primary code searching tool for Firefox introduced by Bill McCloskey in 2016 which built upon prior work on DXR. This product vision post describes my personal vision for searchfox and the rationale that underpins … Continue reading →
encoded: Searchfox (source, config source) is Mozilla’s primary code searching tool for Firefox introduced by Bill McCloskey in 2016 which built upon prior work on DXR. This product vision post describes my personal vision for searchfox and the rationale that underpins it. I’m also writing an accompanying road map that describes specific potential enhancements in support of this vision which I will publish soon and goes into the concrete potential features that would be implemented in the spirit of this vision. Note that the process of developing searchfox is iterative and done in consultation with its users and other contributors, primarily in the searchfox channel on chat.mozilla.org and in its bugzilla component. Accordingly, these documents should be viewed as a basis for discussion rather than a strict project plan. The Whys Of Searchfox Searchfox is a Tool For System Understanding Searchfox enables exploration and understanding of the Firefox codebase as it exists now, as it existed in the past, and to support understanding of the ramifications of potential changes. Searchfox Is A Tool For Shared System Understanding Firefox is a complex piece of software which has more going on than any one person can understand at a time. Searchfox enables Firefox’s contributors to leverage the documentation artifacts of other teams and contributors when exploring in isolation, and to communicate more effectively when interacting. Searchfox Is Not The Only Tool Searchfox integrates relevant data from automation and other tools in the Firefox development ecosystem where they make sense and provides deep links into those tools or first steps to help you get started without having to start from nothing. The Hows Of Searchfox Searchfox is Immediate: Low Latency and Complete Searchfox’s results should be available in their entirety when the page load completes, ideally in much less than a second. There should be no asynchronous lazy loading or spinners. Practically speaking, if you could …

Web Worker-assisted Email Visualizations using Vega
type: item
creator: Andrew Sutherland
date: 2016-03-21T09:28:09Z
subject: Email
subject: Firefox OS
subject: Mozilla
subject: Shiny
subject: Visualizing
subject: glodastrophe
description: tl;dr: glodastrophe, the experimental entirely-client-side JS desktop-ish email app now supports Vega-based visualizations in addition to new support infrastructure for extension-y things and creating derived views based on the search/filter infrastructure. Two of the dreams of Mozilla Messaging were: Shareable … Continue reading →
encoded: <img loading="lazy" decoding="async" class="alignnone size-thumbnail wp-image-917" src="https://www.visophyte.org/blog/wp-content/uploads/2016/03/20160321-034849-482x600.png" alt="Faceted and overview visualizations" width="482" height="600" srcset="https://www.visophyte.org/blog/wp-content/uploads/2016/03/20160321-034849-482x600.png 482w, https://www.visophyte.org/blog/wp-content/uploads/2016/03/20160321-034849-241x300.png 241w, https://www.visophyte.org/blog/wp-content/uploads/2016/03/20160321-034849.png 669w" sizes="auto, (max-width: 482px) 100vw, 482px" /> tl;dr: glodastrophe, the experimental entirely-client-side JS desktop-ish email app now supports Vega-based visualizations in addition to new support infrastructure for extension-y things and creating derived views based on the search/filter infrastructure. Two of the dreams of Mozilla Messaging were: Shareable email workflows (credit to :davida). If you could figure out how to set up your email client in a way that worked for you, you should be able to share that with others in a way that doesn’t require them to manually duplicate your efforts and ideally without you having to write code. (And ideally without anyone having to review code/anything in order to ensure there are no privacy or security problems in the workflow.) Useful email visualizations. While in the end, the only visualization ever shipped with Thunderbird was the simple timeline view of the faceted global search, various experiments happened along the way, some abandoned. For example, the following screenshot shows one of the earlier stages of faceted search development where each facet attempted to visualize the relative proportion of messages sharing that facet. <img loading="lazy" decoding="async" class="alignnone size-thumbnail wp-image-918" src="https://www.visophyte.org/blog/wp-content/uploads/2016/03/faceted-top-and-clicky-with-gpbugzilla-faceting-600x193.png" alt="faceted search UI prototype" width="600" height="193" srcset="https://w…

An email conversation summary visualization
type: item
creator: Andrew Sutherland
date: 2016-02-01T11:29:33Z
subject: Email
subject: Firefox OS
subject: Mozilla
subject: Shiny
subject: Visualizing
subject: glodastrophe
description: We’ve been overhauling the Firefox OS Gaia Email app and its back-end to understand email conversations. I also created a react.js-based desktop-ish development UI, glodastrophe, that consumes the same back-end. My first attempt at summaries for glodastrophe was the following: … Continue reading →
encoded: We’ve been overhauling the Firefox OS Gaia Email app and its back-end to understand email conversations. I also created a react.js-based desktop-ish development UI, glodastrophe, that consumes the same back-end. My first attempt at summaries for glodastrophe was the following: <img loading="lazy" decoding="async" class="alignnone size-full wp-image-910" src="https://www.visophyte.org/blog/wp-content/uploads/2016/02/old-summaries.png" alt="old summaries; 3 message tidbits" width="601" height="161" srcset="https://www.visophyte.org/blog/wp-content/uploads/2016/02/old-summaries.png 601w, https://www.visophyte.org/blog/wp-content/uploads/2016/02/old-summaries-600x161.png 600w, https://www.visophyte.org/blog/wp-content/uploads/2016/02/old-summaries-300x80.png 300w" sizes="auto, (max-width: 601px) 100vw, 601px" /> The back-end derives a conversation summary object from all of the messages that make up the conversation whenever any message in the conversation changes. While there are some things that are always computed (the number of messages in the conversation, whether there are any unread messages, any starred/flagged messages, etc.), the back-end also provides hooks for the front-end to provide application logic to do its own processing to meet its UI needs. In the case of this conversation summary, the application logic finds the first 3 unread messages in the conversation and stashes their date, author, and extracted snippet (if any) in a list of “tidbits”. This also is used to determine the height of the conversation summary in the conversation list. (The virtual list is aware of a quantized coordinate space where each conversation summary object is between 1 and 4 units high in this case.) While this is interesting because it’s something Thunderbird’s thread pane could not do, it’s not clear that the tidbits are an efficient use of screen real-estate. At least not when the number of unread messages in the conversation exceeds the 3 we cap the tidbits at. <img load…

Talk Script: Firefox OS Email Performance Strategies
type: item
creator: Andrew Sutherland
date: 2015-04-30T20:11:58Z
subject: Firefox OS
subject: Mozilla
subject: talks
description: Last week I gave a talk at the Philly Tech Week 2015 Dev Day organized by the delightful people at technical.ly on some of the tricks/strategies we use in the Firefox OS Gaia Email app. Note that the credit for … Continue reading →
encoded: Last week I gave a talk at the Philly Tech Week 2015 Dev Day organized by the delightful people at technical.ly on some of the tricks/strategies we use in the Firefox OS Gaia Email app. Note that the credit for implementing most of these techniques goes to the owner of the Email app’s front-end, James Burke. Also, a special shout-out to Vivien for the initial DOM Worker patches for the email app. I tried to avoid having slides that both I would be reading aloud as the audience read silently, so instead of slides to share, I have the talk script. Well, I also have the slides here, but there’s not much to them. The headings below are the content of the slides, except for the one time I inline some code. Note that the live presentation must have differed slightly, because I’m sure I’m much more witty and clever in person than this script would make it seem… Cover Slide: Who! Hi, my name is Andrew Sutherland. I work at Mozilla on the Firefox OS Email Application. I’m here to share some strategies we used to make our HTML5 app Seem faster and sometimes actually Be faster. What’s A Firefox OS (Screenshot Slide) But first: What is a Firefox OS? It’s a multiprocess Firefox gecko engine on an android linux kernel where all the apps including the system UI are implemented using HTML5, CSS, and JavaScript. All the apps use some combination of standard web APIs and APIs that we hope to standardize in some form. <img loading="lazy" decoding="async" class="alignnone size-medium wp-image-901" src="http://www.visophyte.org/blog/wp-content/uploads/2015/04/screenshot-1-169x300.png" alt="Firefox OS homescreen screenshot" width="169" height="300" srcset="https://www.visophyte.org/blog/wp-content/uploads/2015/04/screenshot-1-169x300.png 169w, https://www.visophyte.org/blog/wp-content/uploads/2015/04/screenshot-1.png 320w" sizes="auto, (max-width: 169px) 100vw, 169px" /> <img loading="lazy" decoding="async" class="alignnone size-medium wp-image-902" src="http://www.visophyte.org/blog/wp-…

webpd: a Polymer-based web UI for the beets music library manager
type: item
creator: Andrew Sutherland
date: 2014-04-06T16:56:01Z
subject: Music
subject: Shiny
subject: beets
subject: d3.js
subject: echonest
subject: polymer
subject: webpd
description: beets is the extensible music database tool every programmer with a music collection has dreamed of writing. At its simplest it’s a clever tagger that can normalize your music against the MusicBrainz database and then store the results in a … Continue reading →
encoded: <img loading="lazy" decoding="async" class="alignnone size-full wp-image-878" alt="beets webpd filtered artists list" src="http://www.visophyte.org/blog/wp-content/uploads/2014/04/beets-webpd-filtered-artists.png" width="863" height="713" srcset="https://www.visophyte.org/blog/wp-content/uploads/2014/04/beets-webpd-filtered-artists.png 863w, https://www.visophyte.org/blog/wp-content/uploads/2014/04/beets-webpd-filtered-artists-600x495.png 600w, https://www.visophyte.org/blog/wp-content/uploads/2014/04/beets-webpd-filtered-artists-300x247.png 300w" sizes="auto, (max-width: 863px) 100vw, 863px" /> beets is the extensible music database tool every programmer with a music collection has dreamed of writing. At its simplest it’s a clever tagger that can normalize your music against the MusicBrainz database and then store the results in a searchable SQLite database. But with plugins it can fetch album art, use the Discogs music database for tagging too, calculate ReplayGain values for all your music, integrate meta-data from The Echo Nest, etc. It even has a Music Player Daemon server-mode (bpd) and a simple HTML interface (web) that lets you search for tracks and play them in your browse using the HTML5 audio tag. I’ve tried a lot of music players through the years (alphabetically: amarok, banshee, exaile, quodlibet, rhythmbox). They all are great music players and (at least!) satisfy the traditional Artist/Album/Track hierarchy use-case, but when you exceed 20,000 tracks and you have a lot of compilation cd’s, that frequently ends up not being enough. Extending them usually turned out to be too hard / not fun enough, although sometimes it was just a question of time and seeking greener pastures. But enough context; if you’re reading my blog you probably are on board with the web platform being the greatest platform ever. The notable bits of the implementation are: Server-wise, it’s a mash-up of beets’ MPD-alike plugin bpd and its web plugin. Rather than needing to speak …

monitoring gaia travis build status using webmail LED notifiers
type: item
creator: Andrew Sutherland
date: 2014-04-03T13:58:54Z
subject: Firefox OS
subject: Mozilla
subject: gaudy
subject: LEDs
subject: Travis CI
description: For Firefox OS the Gaia UI currently uses Travis CI to run a series of test jobs in parallel for each pull request. While Travis has a neat ember.js-based live-updating web UI, I usually find myself either staring at my … Continue reading →
encoded: <img loading="lazy" decoding="async" class="alignnone size-full wp-image-862" alt="usb LED webmail notifiers showing build status" src="http://www.visophyte.org/blog/wp-content/uploads/2014/04/gaudy-leds-build-colors.jpg" width="600" height="209" srcset="https://www.visophyte.org/blog/wp-content/uploads/2014/04/gaudy-leds-build-colors.jpg 600w, https://www.visophyte.org/blog/wp-content/uploads/2014/04/gaudy-leds-build-colors-300x104.jpg 300w" sizes="auto, (max-width: 600px) 100vw, 600px" /> For Firefox OS the Gaia UI currently uses Travis CI to run a series of test jobs in parallel for each pull request. While Travis has a neat ember.js-based live-updating web UI, I usually find myself either staring at my build watching it go nowhere or forgetting about it entirely. The latter is usually what ends up happening since we have a finite number of builders available, we have tons of developers, each build takes 5 jobs, and some of those jobs can take up to 35 minutes to run when they finally get a turn to run. I recently noticed ThinkGeek had a bunch of Dream Cheeky USB LED notifiers on sale. They’re each a USB-controlled tri-color LED in a plastic case that acts as a nice diffuser. Linux’s “usbled” driver exposes separate red/green/blue files via sysfs that you can echo numbers into to control them. While the driver and USB protocol inherently support a range of 0-255, it seems like 0-63 or 0-64 is all they give. The color gamut isn’t amazing but is quite respectable and they are bright enough that they are useful in daylight. I made a node.js library at https://github.com/asutherland/gaudy-leds that can do some basic tricks and is available on npm as “gaudy-leds”. You can tell it to do things by doing “gaudy-leds set red green blue purple”, etc. I added a bunch of commander sub-commands, so “gaudy-leds –help” should give a lot more details than the currently spartan readme. I couldn’t find any existing tools/libraries to easily watch a Travis CI build and invoke comma…

about:nosy can now show CPU usage using jsprobes
type: item
creator: Andrew Sutherland
date: 2012-02-12T07:01:20Z
subject: Mozilla
subject: Program Execution
subject: Visualizing
subject: about:gc
subject: about:jsprobes
subject: about:nosy
subject: jsprobes
description: Refresher: about:nosy was introduced one week ago as a way to see the memory usage of your tabs and extensions (that live in compartments). It sorta looked like this: Except those (green) bars on the right are new. The ones … Continue reading →
encoded: Refresher: about:nosy was introduced one week ago as a way to see the memory usage of your tabs and extensions (that live in compartments). It sorta looked like this: <img loading="lazy" decoding="async" class="alignnone size-full wp-image-845" title="about:nosy with nothing expanded" src="http://www.visophyte.org/blog/wp-content/uploads/2012/02/about-nosy-overview-scaled.png" alt="" width="596" height="432" srcset="https://www.visophyte.org/blog/wp-content/uploads/2012/02/about-nosy-overview-scaled.png 596w, https://www.visophyte.org/blog/wp-content/uploads/2012/02/about-nosy-overview-scaled-300x217.png 300w" sizes="auto, (max-width: 596px) 100vw, 596px" /> Except those (green) bars on the right are new. The ones on the left, they show memory. The ones on the right, they show CPU usage. New data points enter on the left, surf across to the right, then fall into oblivion. I have it on good authority that researchers are looking into some way of labeling these things so that you don’t have to read a blog post to understand them. But until they do, I have a blogging gig, so let’s hope it takes them a bit longer. <img loading="lazy" decoding="async" class="alignnone size-full wp-image-846" title="about:nosy memory and cpu usage for devmo weather demo" src="http://www.visophyte.org/blog/wp-content/uploads/2012/02/about-nosy-devmo-weather-cpu.png" alt="" width="552" height="46" srcset="https://www.visophyte.org/blog/wp-content/uploads/2012/02/about-nosy-devmo-weather-cpu.png 552w, https://www.visophyte.org/blog/wp-content/uploads/2012/02/about-nosy-devmo-weather-cpu-300x25.png 300w" sizes="auto, (max-width: 552px) 100vw, 552px" /> The above is a picture of the bars belonging to a tab viewing a Mozilla Demo Studio demo by ybochatay https://developer.mozilla.org/en-US/demos/detail/animation-of-weather-fronts/launch. What’s really neat about this example is that you can see it was going to town on the CPU in the past (right; past), then usage fell off a cliff. Why? Because …

about:nosy is about:memory with charts, helps you lay blame more easily
type: item
creator: Andrew Sutherland
date: 2012-02-05T05:25:52Z
subject: clicky
subject: Mozilla
subject: Program Execution
subject: Shiny
subject: Visualizing
subject: about:memory
subject: about:nosy
subject: memory reporters
description: about:memory and the memory reporter infrastructure that powers it are amazing. They provide an explicit hierarchy that breaks down the memory use in the system to the subsystems and increasingly the causes of allocation. about:memory looks like this (if you … Continue reading →
encoded: about:memory and the memory reporter infrastructure that powers it are amazing. They provide an explicit hierarchy that breaks down the memory use in the system to the subsystems and increasingly the causes of allocation. about:memory looks like this (if you stand a few feet back from your monitor and take off your glasses): <img loading="lazy" decoding="async" class="alignnone size-medium wp-image-829" title="Example (non-verbose) about:memory session" src="http://www.visophyte.org/blog/wp-content/uploads/2012/02/about-memory-example-225x300.png" alt="" width="225" height="300" srcset="https://www.visophyte.org/blog/wp-content/uploads/2012/02/about-memory-example-225x300.png 225w, https://www.visophyte.org/blog/wp-content/uploads/2012/02/about-memory-example-450x600.png 450w, https://www.visophyte.org/blog/wp-content/uploads/2012/02/about-memory-example.png 695w" sizes="auto, (max-width: 225px) 100vw, 225px" /> If you are going to look at about:memory, it is probably for one of two reasons: You are Nicholas Nethercote or one of his merry band of MemShrink hackers kicking ass and taking names (of inefficient uses of memory). In this case, about:memory is exactly what you need. You suspect some tab in Firefox has gone crazy and you want to figure out which one it is and take your vengeance upon it. Vengeance can take the form of thinking mean thoughts, closing the tab, or writing a snarky tweet. about:memory will let you do this, but you have to look at a lot of text and you may already be too late to find the culprit! If only there was an easier way… Enter about:nosy: <img loading="lazy" decoding="async" class="alignnone size-thumbnail wp-image-832" title="about:nosy showing three tabs open from cnn.com" src="http://www.visophyte.org/blog/wp-content/uploads/2012/02/about-nosy-tabs-cnn-600x104.png" alt="" width="600" height="104" srcset="https://www.visophyte.org/blog/wp-content/uploads/2012/02/about-nosy-tabs-cnn-600x104.png 600w, https://www.visophyte.org/blog/wp-c…

The joy of integrated logging and log viewing with fancy logs
type: item
creator: Andrew Sutherland
date: 2011-09-29T02:16:31Z
subject: Debugging
subject: Mozilla
subject: Program Execution
subject: arbpl
subject: deuxdrop
subject: loggest
description: The deuxdrop messaging experiment‘s current incarnation exists as an (under development) Jetpack that runs in Firefox. I am still trying to shake out functionality to be driven by the UI rather than headless unit tests. While firebug has been a … Continue reading →
encoded: The deuxdrop messaging experiment‘s current incarnation exists as an (under development) Jetpack that runs in Firefox. I am still trying to shake out functionality to be driven by the UI rather than headless unit tests. While firebug has been a great help, another boon has been the logging framework and log viewing framework developed for the unit tests. (Previous posts here and here). Since the log is made up of structured JSON data, all the log processing logic is written in JS, and the log viewing UI is HTML/CSS/JS, it is trivial to embed the log viewer into the Jetpack itself. If you type about:loggest in the URL bar (or better yet, create a bookmark on the bookmark bar and click that), the log viewer is displayed. Deuxdrop’s client daemon logic (which runs in a hidden frame), uses a log reaper that runs at 1-second intervals. If anything happens log-wise during that second, it is packaged and added to a circular-buffer style list of the last 60 seconds where anything happened. When the log viewer starts up, it asks for and receives the data. The result looks like the above small screenshot. If no errors were logged during the time interval, it is automatically collapsed. Let us experience the joy of integrated logging by looking at a real problem I recently encountered. In the development UI (accessible via about:dddev), I brought up a list of my contacts after starting a conversation. It looks like this right now: <img loading="lazy" decoding="async" class="alignnone size-full wp-image-811" title="peeps-tab-user-erroneously-included" src="http://www.visophyte.org/blog/wp-content/uploads/2011/09/peeps-tab-user-erroneously-included.png" alt="" width="505" height="91" srcset="https://www.visophyte.org/blog/wp-content/uploads/2011/09/peeps-tab-user-erroneously-included.png 505w, https://www.visophyte.org/blog/wp-content/uploads/2011/09/peeps-tab-user-erroneously-included-300x54.png 300w" sizes="auto, (max-width: 505px) 100vw, 505px" /> The problem is that I, the u…