{"id":363,"date":"2009-08-03T20:54:59","date_gmt":"2009-08-04T04:54:59","guid":{"rendered":"http:\/\/www.visophyte.org\/blog\/?p=363"},"modified":"2009-08-03T20:54:59","modified_gmt":"2009-08-04T04:54:59","slug":"using-vmware-recordreplay-and-vprobes-for-low-time-distortion-performance-profiling","status":"publish","type":"post","link":"https:\/\/www.visophyte.org\/blog\/2009\/08\/03\/using-vmware-recordreplay-and-vprobes-for-low-time-distortion-performance-profiling\/","title":{"rendered":"Using VMWare Record\/Replay and VProbes for low time-distortion performance profiling"},"content":{"rendered":"

\"profile-performance-graph-enumerateProps\"<\/a><\/p>\n

The greatest problem with performance profiling is getting as much information as possible while affecting the results as little as possible.\u00a0 For my work on pecobro<\/a> I used mozilla’s JavaScript DTrace probes.\u00a0 Because the probes are limited to notifications of all function invocations\/returns with no discretion and there is no support for JS backtraces, the impact on performance was heavy.\u00a0 Although I have never seriously entertained using chronicle-recorder<\/a> (via chroniquery<\/a>) for performance investigations, it is a phenomenal tool and it would be fantastic if it were usable for this purpose.<\/p>\n

VMware introduced with Workstation 6<\/a>\/6.5<\/a> the ability to efficiently record VM execution by recording the non-deterministic parts of VM execution.\u00a0 When you hit the record button it takes a snapshot and then does its thing.\u00a0 For a 2 minute execution trace where Thunderbird is started up and gloda starts indexing and adaptively targets for 80% cpu usage, I have a 1G memory snapshot (the amount of memory allocated to the VM), a 57M vmlog file, and a 28M vmsn file.\u00a0 There is also and a 40M disk delta file (against the disk snapshot), but I presume that’s a side effect of the execution rather than a component of it.<\/p>\n

The record\/replay functionality is the key to being able to analyze performance while minimizing the distortion of the data-gathering mechanisms.\u00a0 There are apparently a lot of other solutions in the pipeline, many of them open source.\u00a0 VMware peeps apparently also created a record\/replay-ish mechanism<\/a> for valgrind<\/a>, valgrind-rr, which roc<\/a> has thought about leveraging for chronicle-recorder<\/a>.\u00a0 I have also heard of Xen solutions to the problem, but am not currently aware of any usable solutions today.\u00a0 And of course, there are many precursors to VMware’s work, but this blog post is not a literature survey.<\/p>\n

There are 3 ways to get data out of a VM under replay, only 2 of which are usable for my purposes.<\/p>\n

    \n
  1. Use gdb\/the gdb remote target protocol.\u00a0 The VMware server opens up a port that you can attach to.\u00a0 The server has some built-in support to understand linux processes if you spoon feed it some critical offsets.\u00a0 Once you do that, “info threads” lists every process in the image as a thread which you can attach to.\u00a0 If you do the dance right, gdb provides perfect back-traces and you can set breakpoints and generally do your thing.\u00a0 You can even rewind execution if you want, but since that means restoring state at the last checkpoint and running execution forward until it reaches the right spot, it’s not cheap.\u00a0 In contrast, chronicle-recorder can run (process) time backwards, albeit at a steep initial cost.<\/li>\n
  2. Use VProbes<\/a>.\u00a0 Using a common analogy, dtrace is like a domesticated assassin black bear that comes from the factory understanding English and knowing how to get you a beer from the fridge as well as off your enemies.\u00a0 VProbes, in contrast, is a grizzly bear that speaks no English.\u00a0 Assuming you can convince it to go after your enemies, it will completely demolish them.\u00a0 And you can probably teach it to get you a beer too, it just takes a lot more effort.<\/li>\n
  3. Use VAssert<\/a>.\u00a0 Just like asserts only happen in debug builds, VAsserts only happen during replay (but not during recording).\u00a0 Except for the requirement that you think ahead to VAssert-enable your code, it’s awesome because, like static dtrace probes, you can use your code that already understands your code rather than trying to wail on things from outside using gdb or the like.\u00a0 This one was not an option because it is Windows only as of WS 6.5.\u00a0 (And Windows was not an option because building mozilla in a VM is ever so slow, and, let’s face it, I’m a linux kind of guy.\u00a0 At least until someone buys me a solid gold house and a rocket car.)<\/li>\n<\/ol>\n

    \"profile-performance-graph-callbackDriver-doubleClicked\"<\/a><\/p>\n

    My first step in this direction has been using a combination of #1 and #2 to get javascript backtraces using a timer-interval probe.\u00a0 The probe roughly does the following:<\/p>\n