{"id":562,"date":"2010-09-07T16:36:22","date_gmt":"2010-09-08T00:36:22","guid":{"rendered":"http:\/\/www.visophyte.org\/blog\/?p=562"},"modified":"2011-01-10T14:27:51","modified_gmt":"2011-01-10T22:27:51","slug":"documentation-for-complex-things-you-dont-basically-already-understand","status":"publish","type":"post","link":"https:\/\/www.visophyte.org\/blog\/2010\/09\/07\/documentation-for-complex-things-you-dont-basically-already-understand\/","title":{"rendered":"Documentation for complex things (you don\u2019t basically already understand)"},"content":{"rendered":"<p style=\"padding-left: 30px;\"><a href=\"http:\/\/clicky.visophyte.org\/examples\/cjs\/narscribblus\/web\/docloader.html?doc=wmsy\/refdocs\/viewslice-interposing.skwbl\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-566\" title=\"editable-and-output\" src=\"http:\/\/www.visophyte.org\/blog\/wp-content\/uploads\/2010\/09\/editable-and-output.png\" alt=\"\" width=\"538\" height=\"337\" srcset=\"https:\/\/www.visophyte.org\/blog\/wp-content\/uploads\/2010\/09\/editable-and-output.png 538w, https:\/\/www.visophyte.org\/blog\/wp-content\/uploads\/2010\/09\/editable-and-output-300x187.png 300w\" sizes=\"auto, (max-width: 538px) 100vw, 538px\" \/><\/a><\/p>\n<h3>The Problem<\/h3>\n<p>One of my focuses at <a href=\"http:\/\/www.mozillamessaging.com\/en-US\/\">MoMo<\/a> is to improve the plight of <a href=\"http:\/\/www.mozillamessaging.com\/en-US\/thunderbird\/\">Thunderbird<\/a> extension developers.\u00a0 An important aspect of this is improving the platform they are exposed to.\u00a0 Any platform usually entails a fair amount of complexity.\u00a0 The trick is that you only pay for the things that are new-to-you, learning-wise.<\/p>\n<p>The &#8216;web&#8217; as a platform is not particularly simple; it&#8217;s got a lot of pieces, some of which are fantastically complex (ex: layout engines).\u00a0 But those bits are frequently orthogonal, can be learned incrementally, have reams of available documentation, extensive tools that can aid in understanding, and, most importantly, are already reasonably well known to a comparatively large population.\u00a0 The further you get from the web-become-platform, the more new things you need to learn and the more hand-holding you need if you&#8217;re not going to just read the source or trial-and-error your way through.\u00a0 (Not that those are bad ways to roll; but not a lot of people make it all the way through those gauntlets.)<\/p>\n<p>I am working to make Thunderbird more extensible in more than a replace-some-function\/widget-and-hope-no-other-extensions-had-similar-ideas sort of way.\u00a0 I am also working to make Thunderbird and its extensions more scalable and performant without requiring a lot of engineering work on the part of every extension.\u00a0 This entails new mini-platforms and non-trivial new things to learn.<\/p>\n<p>There is, of course, no point in building a spaceship if no one is going to fly it into space and fight space pirates.\u00a0 Which is to say, the training program for astronauts with all its sword-fighting lessons is just as important as the spaceship, and just buying them each a copy of &#8220;sword-fighting for dummies who live in the future&#8221; won&#8217;t cut it.<\/p>\n<p>Translating this into modern-day pre-space-pirate terminology, it would be dumb to make a super fancy extension API if no one uses it.\u00a0 And given that the platform is far enough from pure-web and universally familiar subject domains, a lot of hand-holding is in order.\u00a0 Since there is no pre-existing development community familiar with the framework, they can&#8217;t practically be human hands either.<\/p>\n<p><a href=\"http:\/\/www.visophyte.org\/blog\/wp-content\/uploads\/2010\/09\/definewidget-popup.png\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-thumbnail wp-image-567\" title=\"definewidget-popup\" src=\"http:\/\/www.visophyte.org\/blog\/wp-content\/uploads\/2010\/09\/definewidget-popup-600x310.png\" alt=\"\" width=\"600\" height=\"310\" srcset=\"https:\/\/www.visophyte.org\/blog\/wp-content\/uploads\/2010\/09\/definewidget-popup-600x310.png 600w, https:\/\/www.visophyte.org\/blog\/wp-content\/uploads\/2010\/09\/definewidget-popup-300x155.png 300w, https:\/\/www.visophyte.org\/blog\/wp-content\/uploads\/2010\/09\/definewidget-popup.png 932w\" sizes=\"auto, (max-width: 600px) 100vw, 600px\" \/><\/a><\/p>\n<h3>The Requirements<\/h3>\n<p>I assert the following things are therefore important for the documentation to be able to do:<\/p>\n<ul>\n<li>Start with an explained, working example.<\/li>\n<li>Let the student modify the example with as little work on their part as possible so that they can improve their mental model of how things actually work.<\/li>\n<li>Link to other relevant documentation that explains what is going on, especially reference\/API docs, without the user having to open a browser window and manually go search\/cross-reference things for themselves.<\/li>\n<li>Let the student convert the modified example into something they can then use as the basis for an extension.<\/li>\n<\/ul>\n<p><a href=\"http:\/\/clicky.visophyte.org\/examples\/cjs\/narscribblus\/web\/docloader.html?doc=wmsy\/refdocs\/viewslice-interposing.skwbl\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-568\" title=\"trace-output\" src=\"http:\/\/www.visophyte.org\/blog\/wp-content\/uploads\/2010\/09\/trace-output.png\" alt=\"\" width=\"539\" height=\"194\" srcset=\"https:\/\/www.visophyte.org\/blog\/wp-content\/uploads\/2010\/09\/trace-output.png 539w, https:\/\/www.visophyte.org\/blog\/wp-content\/uploads\/2010\/09\/trace-output-300x107.png 300w\" sizes=\"auto, (max-width: 539px) 100vw, 539px\" \/><\/a><\/p>\n<h3>The In-Process Solution: Narscribblus<\/h3>\n<p>So, I honestly was quite willing to settle for an existing solution that was anywhere close to what I needed.\u00a0 Specifically, the ability to automatically deep-link source code to the most appropriate documentation for the bits on hand.\u00a0 It has become quite common to have JS APIs that take an object where you can have a non-trivial number of keys with very specific semantics, and my new developer friendly(-ish) APIs are no exception.<\/p>\n<p>Unfortunately, most existing JavaScript documentation tools are based on the doxygen\/JavaDoc model of markup that:<\/p>\n<ul>\n<li>Was built for static languages where your types need to be defined.\u00a0 You can then document each component of the type by hanging doc-blocks off them.\u00a0 In contrast, in JS if you have a complex Object\/dictionary argument that you want to hang stuff of of, your best bet may be to just create a dummy object\/type for documentation purposes.\u00a0 JSDoc and friends do support a somewhat enriched syntax\u00a0 like &#8220;@param arg.attr&#8221;, but run into the fact that the syntax&#8230;<\/li>\n<li>Is basically ad-hoc with limited extensibility.\u00a0 I&#8217;m not talking about the ability to add additional doctags or declare specific regions of markup that should be passed through a plugin, which is pretty common.\u00a0 In this case, I mean that it is very easy to hit a wall in the markup language that you can&#8217;t resolve without making an end-run around the existing markup language entirely.\u00a0 As per the previous bullet point, if you want to nest rich type definitions, you can quickly run out of road.<\/li>\n<\/ul>\n<p>The net result is that it&#8217;s hard to even describe the data types you are working with, let alone have tools that are able to infer links into their nested structure.<\/p>\n<p><a href=\"http:\/\/clicky.visophyte.org\/examples\/cjs\/narscribblus\/web\/docloader.html?doc=wmsy\/typedocs\/WmsyObjDef.skwbl&amp;forcelang=narscribblus\/raw\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-569\" title=\"skwbl-highlighted-widgetdef\" src=\"http:\/\/www.visophyte.org\/blog\/wp-content\/uploads\/2010\/09\/skwbl-highlighted-widgetdef.png\" alt=\"\" width=\"625\" height=\"336\" srcset=\"https:\/\/www.visophyte.org\/blog\/wp-content\/uploads\/2010\/09\/skwbl-highlighted-widgetdef.png 625w, https:\/\/www.visophyte.org\/blog\/wp-content\/uploads\/2010\/09\/skwbl-highlighted-widgetdef-600x322.png 600w, https:\/\/www.visophyte.org\/blog\/wp-content\/uploads\/2010\/09\/skwbl-highlighted-widgetdef-300x161.png 300w\" sizes=\"auto, (max-width: 625px) 100vw, 625px\" \/><\/a><\/p>\n<p>So what is my solution?<\/p>\n<ul>\n<li>Steal as much as possible from <a href=\"http:\/\/docs.racket-lang.org\/index.html\">Racket<\/a> (formerly PLT Scheme)&#8217;s documentation tool, <a href=\"http:\/\/docs.racket-lang.org\/scribble\/index.html\">Scribble<\/a>.\u00a0 To get a quick understanding of the brilliance of Racket and Scribble, check out the <a href=\"http:\/\/docs.racket-lang.org\/quick\/index.html\">quick introduction to racket<\/a>.\u00a0 For those of you who don&#8217;t click through, you are missing out on examples that automatically hyperlink to the documentation for invoked methods, plus pictures capturing the results of the output in the document.\n<ul>\n<li>We steal the <a href=\"http:\/\/docs.racket-lang.org\/scribble\/reader.html\">syntax<\/a> insofar as it is possible without implementing a scheme interpreter.\u00a0 The syntax amounts to @command[s-expression stuff where whitespace does not matter]{text stuff which can have more @command stuff in it and whitespace generally does matter}.\u00a0 The brilliance is that everything is executed and there are no heuristics you need to guess at and that fall down.<\/li>\n<li>Our limitation is that while Racket is a prefix language and can use reader macros and have the entire documents be <em>processed in the same fashion as source code and totally understood by the text editor<\/em>, such purity is somewhat beyond us.\u00a0 But we do what we can.<\/li>\n<\/ul>\n<\/li>\n<li>Use narcissus, <a href=\"http:\/\/brendaneich.com\/\">Brendan Eich<\/a>\/mozilla&#8217;s JS meta-circular interpreter thing, to handle parsing JavaScript.\u00a0 Although we don&#8217;t have reader macros, we play at having them.\u00a0 If you&#8217;ve ever tried to parse JavaScript, you know it&#8217;s a nightmare that requires the lexer to be aware of the parsing state thanks to the regexp syntax.\u00a0 So in order for us to be able to parse JavaScript inline without depending on weird escaping syntaxes, when parsing our documents we use narcissus to make sure that we parse JavaScript as JavaScript; we just break out when we hit our closing squiggly brace.\u00a0 No getting tricked by regular expressions, comments, etc.<\/li>\n<li>Use the abstract interpreter logic from <a href=\"http:\/\/pcwalton.blogspot.com\/\">Patrick Walton<\/a>&#8216;s <a href=\"http:\/\/pcwalton.blogspot.com\/2010\/05\/introducing-jsctags.html\">jsctags<\/a> (and actually stole its CommonJS-ified narcissus as the basis for our hacked up one too) as the basis for abstract interpretation to facilitate being able to linkify all our JavaScript code.\u00a0 The full narcissus stack is basically:\n<ul>\n<li>The narcissus lexer has been modified to optionally generate a log of all tokens it generates for the lowest level of syntax highlighting.<\/li>\n<li>The narcissus parser has been modified to, when generating a token log, link syntax tokens to their AST parse nodes.<\/li>\n<li>The abstract interpreter logic has been modified to annotate parse nodes with semantic links so that we can traverse the tokens to be able to say &#8220;hey, this is attribute &#8216;foo&#8217; in an object that is argument index 1 of an invocation of function &#8216;bar'&#8221; where we were able to resolve bar to a documented node somewhere.\u00a0 (We also can infer some object\/class organization as a result of the limited abstract interpretation.)<\/li>\n<li>We do not use any of the fancy static analysis stuff that is going on as of late with the DoctorJS stuff.\u00a0 Automated stuff is sweet and would be nice to hook in, but the goal here is friendly documentation.<\/li>\n<li>The abstract interpreter has been given an implementation of CommonJS require that causes it to load other source documents and recursively process them (including annotating documentation blocks onto them.)<\/li>\n<\/ul>\n<\/li>\n<li>We use bespin as the text editor to let you interactively edit code and then see the changes.\u00a0 Unfortunately, I did not hook bespin up to the syntaxy magic we get when we are not using bespin.\u00a0 I punted because of CommonJS loader snafus.\u00a0 I did, however, make the &#8216;apply changes&#8217; button use narcissus to syntax check things (with surprisingly useful error messages in some cases).<\/li>\n<\/ul>\n<p>Extra brief nutshell facts:<\/p>\n<ul>\n<li>It&#8217;s all CommonJS code.\u00a0 The web enabled version which I link to above and below runs using a slightly modified version of <a href=\"http:\/\/github.com\/Gozala\">Gozala<\/a>&#8216;s <a href=\"http:\/\/github.com\/Gozala\/teleport\">teleport<\/a> loader.\u00a0 It can also run privileged under Jetpack, but there are a few unimplemented sharp edges relating to Jetpack loader semantics.\u00a0 (Teleport has been modified mainly to play more like jetpack, especially in cases where its aggressive regexes go looking for jetpack libs that aren&#8217;t needed on the web.)\u00a0 For mindshare reasons, I will probably migrate off of teleport for web loading and may consider adding some degree of node.js support.\u00a0 The interactive functionality currently reaches directly into the DOM, so some minor changes would be required for the server model, but that was all anticipated.\u00a0 (And the non-interactive &#8220;manual&#8221; language already outputs plain HTML documents.)<\/li>\n<li>The web version uses a loader to generate the page which gets displayed in an iframe inside the page.\u00a0 The jetpack version generates a page and then does horrible things using <a href=\"http:\/\/www.toolness.com\/wp\/\">Atul<\/a>&#8216;s custom-protocol mechanism to get the page displayed but defying normal browser navigation; it should either move to an encapsulated loader or implement a proper custom protocol.<\/li>\n<\/ul>\n<p>Anywho, there is still a lot of work that can and will be done (more &#8216;can&#8217; than &#8216;will&#8217;), but I think I&#8217;ve got all the big rocks taken care of and things aren&#8217;t just blue sky dreams, so I figure I&#8217;d give a brief intro for those who are interested.<\/p>\n<p>Feel free to check out the live <a href=\"http:\/\/clicky.visophyte.org\/examples\/cjs\/narscribblus\/web\/docloader.html?doc=wmsy\/refdocs\/viewslice-interposing.skwbl\">example interactive tutorialish thing<\/a> linked to in some of the images, and its <a href=\"http:\/\/clicky.visophyte.org\/examples\/cjs\/narscribblus\/web\/docloader.html?doc=wmsy%2Frefdocs%2Fviewslice-interposing.skwbl&amp;forcelang=narscribblus%2Fraw\">syntax highlighted source<\/a>.\u00a0 Keep in mind that lots of inefficient XHRs currently happen, so it could take a few seconds for things to happen.\u00a0 The type hierarchy emission and styling still likely has a number of issues including potential failures to popup on clicks.\u00a0 (Oh, and you need to click on the source of the popup to toggle it&#8230;)<\/p>\n<p>Here&#8217;s a <a href=\"http:\/\/clicky.visophyte.org\/examples\/cjs\/narscribblus\/web\/docloader.html?doc=wmsy\/refdocs\/emitreceive.skwbl\">bonus example<\/a> to look at too, keeping in mind that the first few blocks using the elided js magic have not yet been wrapped in something that provides them with the semantic context to do magic linking.\u00a0 And the <a href=\"http:\/\/hg.mozilla.org\/users\/bugmail_asutherland.org\/narscribblus\/\">narscribblus repo<\/a> is here.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>The Problem One of my focuses at MoMo is to improve the plight of Thunderbird extension developers.\u00a0 An important aspect of this is improving the platform they are exposed to.\u00a0 Any platform usually entails a fair amount of complexity.\u00a0 The &hellip; <a href=\"https:\/\/www.visophyte.org\/blog\/2010\/09\/07\/documentation-for-complex-things-you-dont-basically-already-understand\/\">Continue reading <span class=\"meta-nav\">&rarr;<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"inline_featured_image":false,"footnotes":""},"categories":[6],"tags":[64,65,90,99,89],"class_list":["post-562","post","type-post","status-publish","format-standard","hentry","category-thunderbird","tag-doccelerator","tag-docs","tag-documentation","tag-jstut","tag-narscribblus"],"_links":{"self":[{"href":"https:\/\/www.visophyte.org\/blog\/wp-json\/wp\/v2\/posts\/562","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.visophyte.org\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.visophyte.org\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.visophyte.org\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.visophyte.org\/blog\/wp-json\/wp\/v2\/comments?post=562"}],"version-history":[{"count":9,"href":"https:\/\/www.visophyte.org\/blog\/wp-json\/wp\/v2\/posts\/562\/revisions"}],"predecessor-version":[{"id":575,"href":"https:\/\/www.visophyte.org\/blog\/wp-json\/wp\/v2\/posts\/562\/revisions\/575"}],"wp:attachment":[{"href":"https:\/\/www.visophyte.org\/blog\/wp-json\/wp\/v2\/media?parent=562"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.visophyte.org\/blog\/wp-json\/wp\/v2\/categories?post=562"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.visophyte.org\/blog\/wp-json\/wp\/v2\/tags?post=562"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}