Good morning. Welcome to session 318, Using Xray and DTrace for In-Depth Analysis. My name is Steve Lewallen, and I'm the Performance Tools Manager within developer technologies at Apple. And this is our last installment of a three part series really this week on Tracing with Xray and DTrace. So in today's agenda, we're going to do a quick overview of Xray and DTrace in case any of you missed any of the other sessions. Then we're going to go over building custom DTrace instruments in Xray.

We'll take a slight turn and then we'll discuss alternate ways to use Xray other than the normal double click it in the finder, etc. Those ways include using quick start keys, which is a point and shoot mechanism, a command line Xray and remote tracing with Xray and DTrace. Finally, we're going to talk about inserting DTrace static probes into your code.

So let's quickly do an Xray review. As you may recall, when you start Xray you're presented with a template chooser. And the template chooser has templates that are subsets of instruments geared towards solving a particular problem. If you don't want to apply instruments in that manner, you can go to the Xray Instrument Library and choose instruments from there and apply them to your document. In other case, we then set up your instruments if they have any other settings that you need to configure, we try our best to have good defaults, but you may want to fiddle with those.

And finally, you pick a target, what do you want to trace and gather data on? Record it, you gather the date and you use the different facilities in Xray to mine the data. Go in to a call tree view, inverting it, filtering out different libraries, etcetera. That's a very quick overview of Xray. So now let's move on to an overview of DTrace.

So DTrace is a dynamic tracing framework that Sun developed and open sourced on Solaris. Over the past year we have take it, imported it to our Intel and power PC platforms. It's a zero disabled cost tracing mechanism. It's always there, but until you use it, you're not really paying a cost for it. After you're done using it, again, you're not paying a cost for it again. Its system wide, so it's available in the kernel on up and it's on all of our installs, whether that's a user install, a developer tools install, or the server.

And it's scriptable. It's scriptable using the D language. This language, when you write it, create a script with it, is compiled down to byte code and interpreted in the kernel. Therefore, it's designed to be very tight and compact, safe and performant. So it actually doesn't have loop structures, for example, or branching. You can't define your own user functions or call out to user space, but it is still very powerful.

Even variable declarations are optional. It's very dynamic. You just kind of go with the flow as you do with a lot of scripting languages. So it all centers around the concept of a probe. This defines the events that you're interested in finding information about. And you can describe product kits or conditions upon which the probe fires, the actions which should be taken when that happens.

And to get a list of available probes on the system, you can use the DTrace command line option, -l. So how does DTrace work? Well, whether you have a DTrace script or you have created a DTrace instrument in Xray, that probe is sent to Lib DTrace and compiled down to byte code and sent on to the kernel to the DTrace virtual machine. The DTrace Virtual Machine analyzes that and attaches it to the various providers that you've specified interest in.

If those providers have information to send back, they do so back through the virtual machine, back through Lib DTrace and out to either the console, if you ran from the command line, or through the DTrace instrument and on to the Xray UI. So that's a quick overview of both Xray and DTrace. Now let's move on to building custom instruments in Xray with DTrace.

Why would you want to do this? Well, you know your own code and you can create domain specific probe points for your own code. Maybe you have a database and you want to track begin and end transaction or something. You can also tailor this for different tasks. You could be bug hunting, so you're just varying your scripts, looking for different bugs, following your nose.

Perhaps your creating an aggression test suite such that you would run a set of DTrace instruments in Xray to gather general performance data of your app before each release to see how memory use has grown or shrunk and performance, etcetera. Or perhaps you're a network administrator who is creating scripts to monitor system on the network.

Now what you see before you now is the DTrace Instrument Builder in Xray. To use it, the first thing you need to do is actually describe your instrument. You give it a name, a category, you can use one of your own or ours, and a good description. Now you should make sure that you use good key words in these fields because this represents your instrument in the library, the Xray Library we just saw during the Xray review. And if someone searches in that library, they'll hit your keywords if you've chosen appropriately.

So the next thing you want to do is define the body of your probe. You get a chance up front to declare any global variables, structures, etcetera, you may want to use, and you also get a chance in the begin section to do any initialization or any preflighting.

And then you come to the body of your instrument, your probe. So the first thing you want to do is describe the conditions upon which your probe should fire. That could be oh, I want it to fire every second at some frequency. Or it could be I want it to fire whenever the foo function is entered and a first argument is between values A and values B, and you can extend that predicate mechanism further.

Next you get a chance to take some action. Now an action that we use is called recording data. And when you record data that means you're sending it from kernel and out of DTrace to Xray for display to the user. A DTrace probe actually has all sorts of data that it could send, it has the arguments to the functions that you're probing in return values, it has DTrace variables that DTrace provides, such as the process ID, the user ID, current working directory of what's being traced, etcetera, and then you have global data in the kernel and your processes that you could access.

Now before you return data back to Xray, if you've chosen to do so, you can do some marshalling. In this extremely trivial example, let's say that when the probe fired, the field my integer was of value zero. It would be incremented by the value of one and then one would be returned from DTrace out to Xray. And finally you get a chance to do any post flighting, any clean up that you wish to do.

So let's go to a demo and see how would use the DTrace Builder in Xray to build a custom instrument. So what I want to do, is I want to build a small instrument that will measure the CPU load on my system and graph it appropriately in the track view.

So I'm going to open Xray now and I'll select a blank template and I'm going to go to the Instrument Menu and select Build New Instrument. So the first thing that I need to do is describe my instrument So I'll say CPU load and I'll put it in the system category.

That's where I put sampler and all of those other types of instruments. And I'll say measures average CPU load. All right. Now I said that I wanted to have this CPU Load Graph. What I want to do is I want to emit CPU load every second so I get a nice, even graph.

So I'll go to my provider types here and I'll select the profile provider. This is a frequency provider, so it can do things at various frequencies. I want to do it every one second. Now I need to actually gather the data that I'm going to use to graph the CPU load.

Now don't worry if you can't see this, or it's not very clear, we're going to describe it in slides later, in detail. But basically, I'm accessing a kernel variable and stuffing it in the local data variable load one. So the next thing I want to do is actually emit that and record it so it's sent back to Xray. So I'll go and I'll say record data. It's a custom type.

And I will use the load one variable I have there in my script, and I'll give it a name. This name is going to represent it in the detail view and the graph as well. I'll just say CPU load average and I'll give it the type of integer.

So now I've described my instrument, I've specified a provider to use, that one second frequency profile provider probe, and I've gathered my data and emitted it, sent it back to Xray. Now I hit OK. Now I want to actually configure my instrument so I get the kind of graph style that I want in the Track View.

I'm going to go to the inspector in the instrument and click on it and I'll select type, sorry, I'll select style of block graph. That'll give me a nice, even, incremental, thick block graph that you can see back in the back of the room. And now I'm going to select the statistic I want to graph. So by default, whether it is implicit or explicit, Xray will list all numeric values it sees coming from that script.

In this case, we were emitting the CPU load average, so I have it here. So that's what I'm going to use to do my graph. I'll hit Done and now I'll go and I'll choose a target. Now this is the CPU load of the whole machine, so there really isn't a target, so we'll say all processes. So I'll go back to the target chooser and I'll say all processes and now I'll use the record button in the upper left to gather some data and I'll go to the terminal and just exercise the machine a bit.

Do a few things. Hit control C and then I'll go back to Xray. We can zoom in on the data, I hit stop, and there's an example of building a custom DTrace instrument in Xray. So now let's look at the details of how we did that. Let's quit the demo there and we'll go back to the slides.

So, what I wanted to do was to emit the CPU load every second, so I used the profile provider, which is a frequency based provider, to do that. I fetched the data from a global variable in the kernel called ABE and Run. This is actually a variable that is known in various flavors of UNIX. You can actually learn more about it on the web, on various flavors, and discussions about how best to compute it, but the kernel essentially puts the one minute, five minute and fifteen minute load average into this variable.

We access that variable with a little scoping operator called backquote. You put that just before the global variable. That tells DTrace hey, I have a variable here, but it's not in my script. Look outside for it. And finally we stuff the computation of the load into the load one variable and emitted it back to Xray and we had that nice, pretty graph.

So that was a demo of building a custom instrument. Now remember, the scope operators. There are two of them actually in that demo. One was the back slash to access data outside of the script itself, in the kernel in this case. And the second was this scope operator. Now using the this scope operator is a clause local scoping and it's just the best practice in case that variable would be declared somewhere else in the script, you can make sure that it's only known within that probe.

So now let's take a bit of a turn and let's talk about alternate ways to invoke Xray outside of the common way of double clicking the finder. Various of these ways may fit certain circumstances you may find yourself in and just may be darn convenient for you. So the first alternate way to invoke Xray is called using quick start keys. This is a way to instantly invoke a template, its instruments, to start tracing a given process, even when Xray isn't running.

So why is this so great? Well, it's convenient. You don't need to go all the way to the finder and then developer tools, applications, launch a performance tool like Xray, set it up and then trace something. But it also can be a necessity in certain circumstances, for example, you may have an app that begins to malfunction.

And it does so but because perhaps it's stomping on memory, and if you deactivate it, you go to the finder and therefore it goes to the background, then it may sort of put, that action may so perturb the target app that it actually crashes on you and you don't get any opportunity to see what was going on whatsoever.

Or it could that it's simply gone completely nuts and it's using so many CPU cycles that it makes it very painful to go back to the finder and wait a bit and load up your application and set it up and trace it, etcetera. So it's always convenient to use Quick Start Keys, sometimes it's a necessity. So let's take a demo look at that. All right. I'll close the terminal here. All right. The first thing I want to do is set up my templates.

So I will go to Xray preferences and I will select the Quick Start panel and enter a global hot key. I can assign a global hot key to any template, whether it's templates that are built into Xray or custom templates. In this case, I'm going to say control option command S should launch the CPU sampler template. And I will close the preferences and I will quit the application.

And now if you've attended WWDC in the last few years and if you continue to do so and go to my sessions, you're going to learn that I don't miss an opportunity to show off my daughter, who I'm so proud of, and this isn't an exception. So I have this little app, it's called Baby Viewer. And these are pictures of my daughter. A cheap attempt for, yeah, so this app deactivates, stops working when I make it go to the background because there's no point in eating up GPU, CPU cycles if no one's looking at it.

So how would I trace that? Well, I'd use that Quick Start Key. So I'll activate the app again, and now I'm going to hit my global hot key. Control Option Command S. Now Xray wasn't running, but now it is. It started in the background and is actually tracing this application, which is incrementing these files about once a second.

Now to stop this, I can just point at the application again and stop the trace. And if I quite the app, we see that we have sampled the app at the even increment that we expect to. So that's just a convenient, easy way to use Xray. So let's go back to the slides.

And let's review, actually, how we did that, actually move the demo up here. So you use the Xray Preferences Panel to assign a global hot key to any template, whether it's your own or custom templates or built in templates or custom templates. The only caveat here is that all the instruments in the template have to have the ability to attach to a running process. Now that actually happens to be just about all the instruments that we have.

Any instrument you build with DTrace, object alloc and links, for example, need to trace only launched processes. So you can choose to run with the UI or headless. So let's say you like to take traces but you don't want the UI to pop up, you just want to look at the data later on.

You can do so by going to the first preferences panel in Xray and changing the default location for where documents are saved. And then you can set a time out, say five seconds, and then you can trace something and you'll hear a beep, it will trace it for five seconds, you'll hear another beep and then the document will be placed in a location you specified.

So again, all you do is you move your pointer to a window that is owned by the process you wish to trace, you hit your global hot key you've assigned, that starts the trace, and if you want to stop the trace, you repeat the whole process. You point the pointer, your mouse pointer, at a window owned by the process you want to trace and you hit it again and that stops the trace.

So we've seen the demo, so let's move on. So, and another alternate way to use Xray is view the Xray command line. Now you can find out information about that from the Xray man page. And you can use this with either a trace template, one that we created or one that you create for yourself, or a trace document. It can already have lots of runs in it and you can just add a new run to it.

And you can use this to launch or attach to an existing process. If you're launching a process, you can provide built in command line arguments and environment variables, etcetera, to change the behavior of the app you're interesting in launching. To terminate it you can specify a time out. Say you wanted to sample it for five seconds or look at file activity for five seconds. You can do that with the -l Option.

You can also just hit control C in the terminal and that will just stop the trace and the document will be written out. And finally you could also just wait for the process to end. In any event, you'll be told when the trace ends where the document was saved to.

Moving on, this is an often requested feature and our first step towards this is the ability to trace something remotely with Xray. So the first step towards this is to allow you to export DTrace from Xray and run it on another machine or locally on the command line and then import the data that results back into Xray and use the GUI to look up that data.

So why would you want to do this? Well, Xray isn't available on all systems. It's only a developer install. But DTrace is available on all machines, a user install, a developer install, Leopard server. You can also use this to take advantage of a feature in DTrace called anonymous tracing.

So you could construct a script in DTrace, in DTrace instruments in Xray, export that, install the resulting script in the kernel, reboot, and then get that primordial trace data well before you even get to the log in window. And then check that data out and then import it back into Xray. So his would be useful, obviously for us back at Apple, but if you were a device driver writer, this would also be very useful.

And finally you can just use this feature to learn about how we build DTrace scripts in our instruments, when you manipulate the GUI, what happens. You can take a look at the existing instruments that we ship with Xray that are in DTrace. And just learn about scripting and probe construction. So to do this, you start out with a trace document populated only by DTrace instruments. Obviously we can't create a DTrace script for say the object alloc instrument or UR Recorder instrument.

Next, you use the export menu from the file menu bar menu and so long as all the instruments in your document are DTrace based, this menu item will be enabled. So you select it and you provide a name for your script. You also have to tell us what you're going to trace, what type of target you're going to trace.

Whether it's all processes, the system, like we say with the CPU demo or it's a target you will launch from DTrace using the DTrace -c option, or it's an existing process. And we need to know this because you slightly modify DTrace script with different macros, etcetera, based on how you're going to use it.

So once you do that and you've run DTrace with that script, you can import the data back into Xray. Now, one caveat there is you have to remember to import it back to a trace document in Xray that is of the same form that you use to export the script to begin with so that it has the exact same DTrace instruments.

Now otherwise, we wouldn't know how to match up the incoming data with the actual instruments you specified. Now we do our best to determine if there's incompatibility between the trace document you have and what you're trying to import will alert you to that and we will stop the trace, but if you get adventurous and you start tinkering with that script and then you generate data and try to import it back in, results will be unpredictable. So let's go to a demo of that now.

So what I'm going to do is launch Xray again. And I'll select a blank template once again. Since we're familiar with it, I'm going to use the CPU load instrument. So I will type in CPU, there's my new CPU Load instrument that we did strictly with DTrace. And now I'll go to the file menu and say DTrace script export.

And let's just show where that will go. I'll put it in the home user account. I'm going to give it a target of all processes since it was the CPU load measuring instrument. I'll say cpu.d. And I'll save that and now I'm going to open up terminal. And let's resize this a bit. And first, let's just take a look at that script so this is basically what you get on this very, very simple script. It has a header. It emits a UID, for example, that we use to try to match up instruments with the data.

And then we see the probe that we used. We used a profile probe firing every one second. And that probe is gathering kernel level data and emitting it. Now back to Xray. Now, this script is a bit different than the script that we used when we controlled the whole widget. When we have defined it in Xray and when we run it in Xray.

When we run it from Xray, we have the opportunity to get back to data and binary format, we mark up the script. There's certain pieces of information. In case you made an error, we can tell you more about the context of that error, etcetera. When you run something from DTrace itself, it's all text based, so the script is just a little bit different. But essentially, it's functioning in the same way, probing the same things.

So, let's go and run that. I need to run DTrace's root, so I'll say pseudo dtrace -S, my script and I want to output this data to a file so I can import that file back. So I'll say cpu.out. I need to use a .o there to do that.

All right. And now I'll hit enter and, whoa, it's warning me. Hang on. Okay. Now I have authenticated. And let's give the CPU something to do. Run everything in slash again. And that's enough. Control C that. Close the terminal window. And now to terminate the trace, I'll just hit control C. And we can briefly look at this data.

Again, it's data that was, excuse me, taken once a second, so it's not very voluminous. It's all text based. We see lots of stack traces, etcetera. I could have turned off stack tracing if I wanted to when I built the instrument. And now let's import that data back into the instrument. I have my trace document again and it has the instrument that was used to export it.

I'll go under the file menu to DTrace data import and I'll select cpu.out this time and it imports the data and there we have our load. And I can expand that out and change the style to block graph and there we have an example of running DTrace outside of Xray based on Xray instruments and then importing it again. So if we can go back to the slides.

So this can be very useful, again, if you need to look at something on another machine. We've had various system administrators ask for this feature. If you need to look at something on a user install that you don't have Xray on, you don't have the developer tools installed on, and it's just very useful. So now I'd like to invite up Dave Payne to talk to you about DTrace static probes.

Thank you, Steve. So to give you an idea of DTrace static probes, I'd first like to paint the context of the categories of different DTrace providers out there so you can see how static probes compare to the other probes on the system. Then we'll take a look at adding static probes to a project and then look at some examples of using static probes.

So I like to split the DTrace providers on the system into three primary categories. The system wider providers, the per process providers, and the user LAN static providers that we'll be talking about here. So let's take a look at each one of these. By the system wide providers, I'm referring to those where we've put the probe points into the kernel.

In these cases, the provider name when you're using these probes does not include a process ID. So many of these you'll have seen in examples. For example, so first off, the column on the left here is the name that the DTrace system uses. The column in the middle is a more human readable name that we use in the Xray instrument builder, if you want to use these.

So a few primary examples, the SysCall provider is the interface between your applications and the kernel. So you can watch all of the entries and returns from System Calls and get an idea of their arguments and return values. But the probe point for that is in the kernel.

FBT is the function boundary tracing inside the kernel. So you can actually see the activity that's going on within the kernel itself, which could be helpful if you're a device driver writer, for example. The Proc Provider gives the kernel's view of the creation of, the creation of termination of processes and threads and it also informs you of signal delivery and signal handling activity.

And the profile and tick providers, as you saw Steve exercise, you do periodic activities in a DTrace script. So there's a couple of others that we've recently implemented that we don't yet have in the Xray instrument builder, but you can go to custom and just specify the name lock stats for VM info.

So I mentioned that these are system wide, but a lot of times you're just interesting in one process and you don't want to put the overhead of having these hit by every process, so to do that you can put the process ID in a predicate. You can compare the executable name or the process ID against the variables that DTrace provides for you to use in your scripts.

The process providers let you put probe points into your application code itself, into your applications, your libraries, your plugin modules. These are, let you probe functions or Objective-C Methods. You can specify entry or return or offsets within a function, or even an absolute address. To use these, you must specify a process ID.

The primary provider that Sun implemented in OpenSolaris was the PID Provider, but we've also implemented another one, the Objective-C Provider. And I'll show you the difference here. So with the PID provider, you have a probe name of the form provider name module, which would be your library or executable name, the function name and then an actual name, which would be entry or return. So for example, if I want to trace all the NSLog function calls, now that's implemented in foundations, which is a Cocoa Library, but it's a C Function essentially. So I could say pid273:foundation is the module, NSLog:entry.

Whereas for Objective-C provider, it's the provider name followed by the class name, the method name, which can be a class or an instance method, and then again the name entry or return. So if we want to trace all allocations of NS Strings, I can use OBJC Process ID 1017, colon NSString colon plus alloc star. I'm using the wildcard of star here because the method that's actually called is alloc with zone. And then again I want the entries there.

With this, if you wanted to, if you had a method name that you wanted to examine that was colon separated, you would need to use a question mark or an asterisk wild card to be able to specify that in that third method field there. So how does user LAN static providers compare? Well these are probes that you can put into your own code that have meaning within your code and are no necessarily tied to the names of any particular functions or methods.

So these can be semantically meaningful points The advantage of that is that if you're changing the implementation of your application over time so that you function names or method names change, these are nice and stable. You're DTrace scripts and your Xray instruments can continue to work with this.

So this is a great way to have these probe points always be there so you can dynamically turn on tracing or logging or let your customers do so. As with all DTrace probes, these are zero cost when you're not using them. It's just actually a no op in your code.

A primary distinction between this and the Objective-C provider is that this provides you a way, because your app, you've coded into your application the knowledge that this is a probe point. Now with the PID in Objective-C Providers, DTrace let's you record the arguments but your code doesn't know that the probe is going to be put there so the kernel says okay, I'll fetch the arguments and there's no Objective-C runtime in the kernel, there's no objectives. We can't extract the descriptions of Objective-C Objects there. Whereas with this, you can code in the knowledge of an Objective-C Object, get it's description, and be able to record that as an argument.

Because these are coded into your application, or let's say it's a framework, if you have multiple processes using that framework, these probe points are in all those processes. So you can write your scripts to focus on just one of those processes or focus on all of them. So for example, core data has static probes in it as we ship it with Leopard today. I can watch all of the begin fetch method calls or begin fetch operations within core data by saying CoreData asterisk for the process ID colon, colon, colon BeginFetch.

So some things to think about as your designing static probes, you want to think about where the interesting points in your application that you might want to probe to see what's happening. So, for example, you might have a long operation that's composed of multiple steps, let's say it's three steps, you might want to record the start and stop of each one of those steps so you can see the relative amount of time that each step takes. If you've got an activity that has particular points of interest along the way, you can have a probe point to record progress for that.

Then you want to think about what arguments to record, so here you're providing the raw data for people to be able to consume later. Think about what data you would want to be able to report in log messages generated by a DTrace script that's monitoring your program, or statistics for graphing in an Xray instrument. Then in order for your customers, say your internal developers, to be able to use these probes, you want to tell people what they actually are and what the arguments of them are.

So the steps for adding static probes to a project, we'll go through these. Number one is to write a provider description file. Number two would be to import that file, the header that corresponds to that into the sources files where you're going to put the probes. Now the header doesn't actually exist yet. So the third step is to generate the header file, and we'll talk about why I'm saying this in this order. Then the fourth step is to go ahead and add those probe invocations to your probe.

To take a look at this, I'm going to be using the grand perspective project as an example. Some of you may have seen this used earlier in the show. Grand Perspective let's me point to the directory on my disk and scan recursively down through all the sub directories, gather the size of each one of them and roll that information back up to be able to report the sizes of any directory in that hierarchy.

So my first step in putting in static probes to monitor this, I've decided for each directory I want to know the, when I start the directory and when I end it. So I put in a probe point for start directory scan and I want to record the directory path and let's say the depth, or how deeply nested in the directory hierarchy I am. When the directory scan finishes, I should know the size for that directory. And I'm going to record that in kilobytes. So I call the provider description file GrandPerspectiveProbes.d. .d is a standard DTrace suffix, and in this case .d is a suffix that Xcode knows about.

So a couple of important items here. When we're recording the probe arguments, these must actually be C because the header file that gets generated for this is going to reflect those arguments and is going to get compiled into your program. This is the DTrace program itself that's doing this generation of the header file so we can record numeric data types like int, short or long. It actually has a 64 bit space to record each argument. We can record a char star that's a pointer to a string out in the user space of the application or a void star that would be an arbitrary pointer.

So I had mentioned that we provide a way to look at Cocoa object descriptions. So I can do that by taking my object, myobj, calling the description method, that of course I've written on that, and then making a UTF8String method call for that so that I can get out a char star that I can actually pass down into the kernel. So again, I'm doing the interpretation of my Cocoa objects up in user space here. Step two, simply go into the header source code file where I want to put probe points and import my header file.

Step three, then, is to generate that header file. Now the reason I imported the header file first is because this tells Xcode when it does it's dependency scanning, it says oh, there's a header file here that I don't actually have, do I have a way to gather this, to generate this? Oh, okay, I know that for .d files I can generate those into a header file using this dtrace -h -s command. So the build system will automatically generate the header file for you and then you can use code completion then to enter in the probe macro names.

If you have a make based project, you simply add a rule to your Makefile to put that dtrace -h -s in explicitly. And that's all you have to do. The implementation of the header file has macros in it that have somewhat complex implementation that give the linker directives of what information to put in with your binary so that DTrace will know that these probes exists in your binary. So these header files are not really meant to be human readable. The primary thing that we're interested in is the probe macro definitions that they put in there. So for example, I had written start directory scan with a char star of the path and the depth.

So the header file contains macros corresponding to these, the form of the macro names is always going to be the provider name, which in this case was GRANDPERSPECTIVE, all in uppercase, and then _START, _DIRECTORY, _SCAN. And you'll notice the second one here, also, which is GRANDPERSPECTIVE_START_DIRECTORY_SCAN_ENABLED. So why is that? Well, we'll talk about that in a couple of minutes.

Okay. Great. So the fourth step is we go ahead and add the probe implications to our code. So in this case, at the top of my method where I'll be recursively descending down through my directory hierarchy, this is called for each directory, I want to put in a probe point to say okay, I'm starting this directory and here's the depth.

So I'm just going to be kind of extensive about this and calculate the depth by doing, taking my string, getting an NS Array of the path components and getting a count of that. I could have passed it recursively down through, but hey, let's take more time. Then I get my UTF8 path and I make my macro call. Then at the end of my function, where I've got the directory size, I make a corresponding call and record that.

The audience says, well Dave, you had said that these were zero cost probes. That sure doesn't look zero cost to me. Well that's what the is enabled probe is all about. This lets you avoid the expensive cost of any potential argument marshalling that you want to do by having a quick test and branch that if the probe is not actually enabled, it'll just skip right over this code.

When you do enable the probes, then this test will success and it will go through the argument marshalling and it will make the corresponding macro call. So we can see here that it's the same code as before just using the is enabled to wrap each of the probe points here, because gee, I want to avoid the cost of that divide there on the second one there also. Their size divided by 1024.

Okay. So let's see if the demo gods are with me to actually show this in action. Go ahead and hide terminal. I'll bring up my Grand Perspective project. So my first step is I'll go down here to other sources as a place where I want to create my new file. So I'm going to create an empty file in the project for the probe descriptions. So we'll call that GrandPerspectiveProbes.d. So again, the standard .d suffix. Finish with that.

Now I magically type in the same thing that you saw in the slides before, so start directory scan and end directory scan macros. All right. Probe definitions. Now we'll go ahead and save that. Make sure we're built. Okay. Now I want to go up to the source file where I'm going to want to put these two probe points. I'll just add in the #import GrandPerspectiveProbes.h. Save again.

And let's actually bring up the build panel here so we can see this in action. And, whoop, okay. For some reason it's failing the first time around. But that was actually where it executed the dtrace -h -s. So anyway, trust me, it did that on the first build through. So now the header file exists and I can start using these probe points in my program. So I'll go down into the source file, into the routine that gets called for each directory.

And I'll go down through here to the point where at the top I've got the full directory path for the directory. And we'll go ahead and magically type in our probe definition. So here again, I'm using the is enabled probe to test whether it is enabled. Doing all the expensive argument marshalling and then calling the macro. At the bottom of my routine, where I've finished going through the full directory hierarchy scan for this directory, I can put in my end directory probe.

So same thing here as you saw before. Let's go ahead and save and build. Now if I run this, we can see hey, very nice. Very nice. But I'd like to be able to see what's going on there. So let's actually bring up Xray, I'll choose the blank template and bring up the library.

Now in the interest of time, I've already created an Xray instrument that uses these two probe points. So I've got two probe clauses in my Grand Perspective Xray instrument. The first one I named startDirectoryScan. I call it type custom, because Xray didn't know about it. The provider name is GrandPerspective, and I use the star wildcard to indicate that I want to gather this for all processes that happen to hit this probe.

And then for the first one I say start directory scan and I record arg0 as the path, arg1 as the depth and in this case, I'll record a third argument of the directory size of zero because I'm just starting the directory. I don't know what it is yet.

We close that down. We can take a look at the second one. Very similar. I call it endDirectoryScan. Again, it's custom. GrandPerspective star, end_Directory_Scan as the probe name. So again, this matches what I put in the provider description files. This is the lower case version of this. So I'm recording arg0, the path, arg1, the depth, and now I do have argument two on this probe, which is the directory size.

Whoop. So I'm going to go ahead and zoom this up a little bit so we're seeing that I'd like to graph both the depth and the directory size as we go through. I should be able to attach to the Grand Perspective project, or application that's already running, where you can see that the window's still there. Go ahead and scan the folder again. The applications folder. Whoop. It would help to start recording, as Steve points out.

Scan folder, applications. All right. Now we see data coming out and in this case, the Grand Perspective black screen. Oh well. We're here to demo the static probes. So we can see lots of data being gathered as we go through. At the top we've got, we can see that we're descending through the directory hierarchy with a depth of two and start, start, start, finally an end, where we get the directory size.

We can see the path name on each one that I've recorded from static probes. I can go to the end and see that again we descend down through and finish up slash applications, reports, the directory size of almost a gigabyte in slash applications. I can sort by these various columns and there we go. Okay. Let's go back to slides please.

So we've seen how you can add static probes to your project and I mentioned that we're starting to put them into various places in the system as well. So we've got static probes in Libsystem, the plockstat provider is actually one that's documented in the openSolaris documentation for DTrace, so this tells us information about what's going on with locking on our P Threads.

I had mentioned that we've got static probes in core data. And in Leopard Beta we're shipping the scripting languages Ruby, Python and Perl with static probes already embedded in them, so each of these defines their own provider that you can use. You can see these using the DTrace -l command to list the probes. You can grep for a specific provider name or for the process ID to filter down because there's thousands of probe points on the system.

This is the provider definition file that I've actually copied out of the core data source code. We can see that with this we can track the begin and end activity of fetching, of faulting, of relationship faulting and of saving in core data. So you'll notice that that's a .d file because that's using the d language to define it.

And then here I give an example of a DTrace script that I could write that could monitor all core data activity using those probes. Again, this is also using the standard .d suffix we interpret which, whether it's a provider definition or a script to use these, we interpret that based on the context you're using it in and the contents, of course.

So in the case of Ruby, this is actually the most extensive of the scripting languages. We're taking advantage of some work that was done in the open source community by a company called Joyer. We can see function, entry and returns, garbage collection begin and end, and the create, the start and finish and freeing of object creation.

So actually, on your systems in developer examples, Ruby, DTrace, print_calls.d, we have scripts, two scripts there. This one that shows the entry and return of all Ruby functions, printing the file name, the line number, the class and function name. And this will work for your Cocoa Ruby projects as well. And then the second one to observe memory usage so we can track garbage collection begin and end and object creation and freeing.

So let's go ahead and see a demo of some of the other static probe activity in the system. First, I'd like to go up and go ahead and give you an idea of if I do the pseudo dtrace -l, in case you missed it in the Steve Peters' DTrace session yesterday, I'll just execute it.

19,722 probes. If I grep for just the core data probes, we can see that there's a lot of processes that are using core data, so process 1326, 1321, 1237, 183, 176, so all of these probes are already there in the system. I could monitor them all at once with one script.

But what I'm going to do is bring up the group viewer project. Make sure that that's built. And this is a simple core databased application. I'll go into Xray, again, I'll use a blank template and bring in a core data instrument that I had previously created. And in this case, I would also like to monitor the reads and writes activity.

So if we look at the core data probe instrument here in DTrace, in Xray, we can see that I've got two probe points on this, one to watch the begins of fault activity and one to watch the begins of fetch activities. In each case I record the string that is the URI of the object ID of the object in question.

So I can go ahead and launch the group viewer executable from Xray and bring up a sample document in it. And we can see that we had a fair amount of read and write activity at the start of it and there was also a delay when we open that window up. Now, as I scroll through my list of members, I can start to see core data activity as we begin faulting operations to bring in the member data as I was scrolling through the list. So that's kind of interesting. Lots of read write activity at the start.

Fetching activity or faulting activity later on. So if you saw my demo on Monday, you might recognize that we had in the interests of prematurely optimizing to give the user good user experience, we had actually prefetched all this data when we loaded the document so that it would be there when you needed it. But as our customers started using larger files, that started to be a performance hit and maybe we can do something about that.

So if I go back into the project, and in this case, my window has the lists in it, the lists are backed by array controllers that have SQL Light activity. I can take a new advantage of a new feature in Leopard that we've added to the core data array, all the controllers in core data, and this is use lazy fetching. So I'll go ahead and turn on lazy fetching and build. Now let's go through this again to see if we get any different behavior. So make sure that GroupViewer is not running. Okay. Go ahead and run a second time through.

Load up our document. Much less read write activity up front. I didn't show it before, but we probably had over 10,000. Over 10,000 reads and writes up front here. We had 206. I go back to GroupViewer, scroll through the list again. Now we can see that as I'm doing my faulting in core data, it's reading the data in just when I need it. So you can see the correlation of static probes that we can define with semantic meaning and some of these system probes that are in the system. So you can do that for your apps as well. Okay? So let's go back to slides.

So to recap, for adding static probes to your project, you would write the provider description file, you would import the corresponding header file in the source code where you need that header, number three would generate the provider header file, which XCode will do automatically for you, and number four you would add the probe invocation macros to your code. So on the provider description file you do need to make sure that that is a file that is in the source target for your project.

So in this session we've taken a look, another look, at the power of Xray in DTrace and the power of them combined. We've looked at building customer DTrace instruments. We've seen a variety of kind of advanced ways to invoke Xray for specialized circumstances. And we've seen how you can extend the power of this system by instrumenting your own code with static probes. So for more information you can contact our developer tools evangelist, Matt Formica, or consult the documentation websites. You can look at the DTrace documentation up on Sun's BigAdmin website. Just Google for that.