Okay. Welcome, everybody. My name is Mikey McDougall. I'm from the Java team at Apple, and I'm really happy to be on stage to talk about debugging Java. So, uh, I see a lot of you have your portables, so that's really good. Let's, uh--let's make sure you're in the right room, first of all.

So this is, in many ways, a tips and tricks session. These are, um, things that we've gathered over the last two to three years that we've found to be really handy in terms of debugging problems with Java. And also, I'm gonna talk a little bit about, um, sort of what goes on inside Apple, you know, give you a little bit of the inside scoop.

And also, you know, we're called the--the Black Belt Java Debugging Session, but I don't know any karate. Um, but I'm gonna deliver a little bit of action and adventure. All the stuff that I know about karate, I learned from watching action movies. So we're gonna--we're gonna bring a little of that in.

So this is an overview of what we're gonna do for the next, uh, hour to hour and 15 minutes. I'm gonna talk some about Developer Previews, and our Developer Preview process, and how to work safely with Developer Previews. Go really quickly over some of the basic tools for Java debugging.

Um, I'm gonna talk about some common problems people bump into in Java. Hopefully, if you sit through this session, I'll--I'll cue you in to some things that can go wrong, and we'll save you a bunch of time. Then we're gonna go really in-depth. We're gonna analyze one crash all the way through.

Um, we're gonna go into a section on how to take things apart. I learn best not by building things up, but by picking things apart. So we're gonna pick things apart for a little bit. And then we're gonna have a hands-on session, a brief one with Shark, which is a really great performance analysis tool that can be used for Java. And then I'm gonna go and tell a little bit more about the sort of inside story at Apple.

So my first section is about the Developer Previews, and it's actually just not the Developer Previews. It's about working with buggy software in general. So here we go. One of the things that you have to be prepared for when you're working with software that's sort of a developer preview or new is that things might not go quite exactly how you expect.

But I still think that everyone in this room should be installing and running our developer previews. We work really hard so that you're the James Bond and you get away from the villain and, you know, not--not actually end up with a big mess on your computer. So this is where you get the developer previews. They're available--actually, we released the developer preview today.

So Java SE6, Developer Preview 5 is the, you know, official name for the whole thing. We've been--as long as I've been at Apple, I've been fighting to get developer previews out to people like you because I think that it's a really, really good way for everybody to win.

The thing about working with the developer previews is you take a little bit of risk. But you get great stuff. You get all of the, you know, latest bug fixes. So you're really, really up to date. You get to actually work with Java 6 before sort of the thousands of people that have Tiger installed would normally get to. So you really get to use the cool new stuff.

And the other thing that's really important for you as developers is you actually get our attention. If you're working with our developer previews and you file a bug and you say, "Hey, guys, this bug is a bug and it's a bug in your latest developer preview," the amount of attention that you can expect from me or from one of my colleagues goes up a little bit because we know that you're working where we're working. So we're sort of in the same space.

At the same time, you have to be prepared for a little bit of trouble. We work really, really hard to make sure that the developer previews are really solid. But like in 2003, I think it was early 2003, late 2003, we shipped a developer preview that like totally hosed everyone who used IntelliJ. So for like three weeks, the IntelliJ people were like, "Ah, what did you guys do?" And then, you know, three weeks later, we released another developer preview.

We fixed it. So--but you have to be prepared for a certain amount of trouble. And the other thing is there is no uninstaller. So, you know, once you're on board, you're sort of all the way on board. So that's two of the things that you need to be thinking about.

So there's ways to work with developer previews that we've found to be very, very successful. One is to have a sandbox and always leave yourself a safe place to go to. And so we're going to do a demo now of how being prepared helps. I'm going to bring up Barry Langdon-Lassane, one of my colleagues at Apple Computer.

And we're going to do something way worse than installing a developer preview that may or may not be flaky. We're going to do terrible, terrible things to Barry's machine. There you go. BARRY LANGDON- Thank you, Mikey. Yes, we're going to do dangerous things to my machine. Now, there's lots of dangerous things you can do to your machine. I'll leave it to your imagination.

This example is a little bit contrived in that we wrote the installer ourselves. But let's pretend I was on the internet, I was surfing the internet, and I found this really cool application. And I downloaded it, and I wanted to install it. So here's the application. Let's see what happens when I install this on my machine.

Yeah, yeah, yeah, warnings, warnings. It's fine. So let's install it. Now, this is going to be important later. My machine has multiple partitions, and I'm going to be destroying one of them. So I have to pick which one I want to destroy. I've got a precious data partition. Actually, that doesn't have an OS on it, so it's pretty safe.

An admin partition, and my development partition. And the development partition is where I generally do risky things. And let's see what happens when I do something risky. And I'm going to stop using the mouse, because it keeps going into-- That mode. OK. So I'm going to install this risky software that I downloaded. Oh, it asks for my admin password.

Sure. So it's installing and I'm feeling good because, you know, it's my development partition and I want to try something new and see what's going on and uh-oh. Uh-oh, this looks really bad. I don't see a dock. This close button's kind of flashing and mmm. Okay, my machine, let's see, the volume control I think is even worse. Yeah, my machine is completely hosed. So hopefully, before Mikey comes back to me later in the talk, I'll have my machine back up and running, and then I'll show you what I did. So back to you, Mikey.

Back to slides. So we're real time demoing how being prepared helps. So I want to recognize that everyone has their own technique for being safe. And you guys are all really smart people. And there's lots of different techniques. So this one that we have is tuned for working with our developer previews. I reinstall my system software a lot because I work at Apple.

And one of the things that I want to just drive home is, if you're prepared, you can avoid a whole lot of problems. So this is Disk Utility. Many of you folks are familiar with this. And it's probably hard to read. So I'm going to zoom in. This is actually Disk Utility looking at my machine at home.

And this is how I have-- how I'm set up. I dedicate four-- I actually have four boot partitions. I have one like Barry for my development, testing, you know, crazy risky stuff. Actually, I take it and make a copy of the Tiger, you know, tent, you know, original Tiger install DVD. And I've put that image right on my hard disk.

It turns out that if you do that and you boot off that hard disk and you install a minimal install of your system, it takes you like a minute to get yourself back to, like, a bootable system on a new partition. I also, like Barry, have an admin partition where I keep mail and all the sort of applications that I use all the time. And finally, I've got-- I have Panther on one of my things because I do regression testing. I want to know what's going on.

You know, sometimes I need to step back a few years and find out what's going-- what's going down. I make two partitions, one for my wife and kids and myself to sort of live on, and then another one for my precious data. So this is just one way that you can set yourself up to be prepared for working with software that you can't uninstall.

So there's some considerations here, like, how many gigabytes should I set aside for a partition? Will I really need to install my system software all that frequently? So one thing that's true is, I mean, you're here today. We've given you Leopard. You sort of need to reinstall your system software. And it's kind of nice if you have a separate partition that's set aside already, sort of prepared for a place for that to land.

And it's also nice if you have a partition that's set aside for developer previews. Where if something goes badly wrong, you can go back to another place and be happy. So how much space to set aside? If you go back to that last slide and you looked really quickly, you saw that I set aside 12 gigabytes for my partitions. And you can get by in 12 gigabytes, especially with Tiger. There's enough room there. But these days, I would suggest that you set aside about 20 gigabytes.

When you start getting Leopard on there and you get something like GarageBand or some big productivity app, all of a sudden, what I thought back in the day was going to be plenty of room starts seeming a little tight. So 20 gigabytes is a good number these days.

And, you know, I realized I was going to be telling people, "Set aside 20 of your gigabytes for system software." And so I wanted to figure out exactly how much is that going to cost in terms of, like, dollars and cents. So I went out and I looked on the web for, like, prices for hard disks. It's like $1 to $1.25, $1.50 a gigabyte these days. So you get a pretty good insurance policy for about 30 bucks.

So I think that's a good investment, so... There are alternatives. And some of these alternatives are very important if you're doing things like dual booting, one of the new Intel Macs to Windows and to the Mac, because there are certain difficulties if you try to do a lot of different partitions and have one of them be a Windows partition.

So an external bootable drive is a really, really good idea. And another way to do sort of risky stuff, you can have your development system be on an external FireWire drive. There's some really great backup software. I strongly suggest that everybody back up their really, really precious data. And the other thing is I've talked to some developers, and they're like, you know, I'd just stay two developer previews back or three developer previews back. I do not recommend that. But with care and luck and research, you can be OK.

So that was how to set up to use developer previews and my sort of pitch for everyone, please do install our developer previews. There's something else, and I actually bumped into this the same day that I was putting these slides together for the first time. A bit of preparation that you can do on your machine for when you get hung or stuck in some really important way.

What I had happen is my user interface froze up completely. My mouse wasn't working. Everything was really sort of frozen. But I'd encountered this situation before. This is my system preferences at home. And there's two things that you can select so that when you get in this situation, you have some chance of recovering. One is make sure that you can remote log in.

And the other thing is I think that Apple Remote Desktop is a really, really great tool. So you can do a whole lot of things on a machine where your monitor is just-- your monitor might get kicked in by some unhappy person. You can still do a lot with your machine if you turn on Apple Remote Desktop.

So you ping the machine, make sure, you know, sure enough it's still running. You can SSH in. Once you SSH in, you have all your old friends. You can run top to see what's taking up all the processes. This is one of my favorites. You can find out which Java processes are running, just in case it's Java that's hung for you. And then you can attach to anything that you might want to. You do a full investigation and write up your bug.

So, yeah, so, So there's two things here, Developer Previews, and just two little safety tips. Make sure that you have at least one more boot partition somewhere that you can boot off of. And I really think that everyone should enable SSH, just so you can always get back in somehow. So that's Developer Previews. Let's talk about some of the really basic tools for Java debugging.

Okay. So, these aren't quite hand grenades but--but they're handy. First of all, who--who in this room has--has done this sort of plankton level debugging? You put a printf in your code, right? Okay. Now, who has run into somebody who is also a developer who's like, I put a printf in my code and I don't know where it's going to.

So everyone, I--I felt like I had to say at the root level, you can find standard error or standard out by going to applications utilities console.app. Like that's where--that's where everything sort of spews to. The other thing is try to make sure that anything that you actually leave in your code like that is important.

Because when people are trying to figure out, you know, some--something bad has happened to their machine, they look there if they're sophisticated and they see like--if they see a bunch of logging stuff and it's coming from your app, they're going to be like, "Oh, I know whose problem it is." And it might not be if you just left logging turned on or something like that. So, the other thing is, I think that everyone should actually use the right tools. I'm going to fix this slide.

These are not white belt tools. These are white box tools. And this is actually where everyone should be doing their debugging. I think that if you can work with code-- I usually can't work with code-- that's great. If you can work with an IDE, with an integrated debugger, that's where you should be doing your work.

That's not where I live. And so I wanted to give you some of the tips from where I live. And these are more like, not so much black belt tools, but black box tools. So I spend a lot of my time in the terminal, and there's a couple of things that you should know are available in the terminal. First of all, if Java is running, you can get a bunch of really good information about what's happening in the current Java application by sending it a sig 3, a kill dash quit.

And this will cause standard error, which I mentioned in the slide, like, one or two back, to dump out all the thread information. Any threads that are deadlocked will come up there. So you should really know about kill dash quits. Really handy for finding out what's going on with your application.

Also, for the times when you're hung, sample and GDB-- actually, GDB is slightly more powerful, but it's sort of what I think of as an expert-friendly tool. It's not really user-friendly. But you can do a lot of work with GDB. If you attach to the processing question, you can do thread apply all backtrace, and you get a bunch of information there. And also, Shark has come to the command line. So you can actually-- if you've got a hung application, you can use Shark to profile it and get a bunch of information about it.

The other thing is when things go badly wrong and you're not hung, but you go so far as to crash, you need to know where the log files are. Everyone should know this information who's in this room and cares about Java debugging. I sort of like when I get access to somebody's machine for the first time. I kind of go here and find out how many times Java has crashed on them and look around. It's kind of cool. We're going to go in depth on these two locations later on in the session. But everyone needs to hear about them.

There's some really good command line options that you can pass to Java to get a bunch of good information that can help you debug your problems. This one is probably the most useful generically speaking. If you tell Java, dash verbose class, as every class gets loaded in, it tells you about what's getting loaded in. And if it's often the case that the last thing that gets loaded in is the thing that has the problem. So this can give you a lot of really good clues as to what's going wrong with your application if you have a reproducible problem. So verbose class.

This one is not so much for debugging in general, but for understanding the performance characteristics of your application. This is actually really important to do at some point in your career. Watch what happens with your application. Watch how fast or slow garbage collection is running and some of the overall characteristics of it.

For folks who are doing user interface stuff and see something that might be a little bit off, and you're wondering, "Is that Apple, or is that me?" This is a good option to know about. You can set the default look and feel, in particular for Swing, with an option on the command line. And this is how you set it to be not Apple's Aqua look and feel, but to be the metal look and feel.

So this is very handy. Now, we've got two different graphics pipelines now in Java 5. One with Quartz on, uses all the sort of underlying Quartz support, and one with Quartz off, which uses Sun's 2D renderer. And so if you're seeing graphics anomalies, it's sometimes very interesting to go from one pipeline to the other pipeline to figure out just whether-- I mean, this can be just good information. The other thing is these two pipelines have different performance characteristics, so it's really a good idea to look at your application in both situations if you are sort of performance sensitive, like many games might be.

And we're going to talk about Shark later on. Java needs a little bit of a clue if you're going to use Shark's Java-specific profiling and Mark So, times when you need to know how to pass this command line in if you're going to use Shark, and Shark is a great tool.

Um, no talk about debugging would be complete if you didn't talk about leaks. So there's sort of the-- the quick and dirty way of finding leaks is to use jmap -histo for histogram. And what--if you use this command and you point it at the process ID of a running Java process, it will go through the Java heap, and it will collect up a list of every single object that you have, and it'll put, like, a number next to each one of those. You know, like, I have 10 million integers or something like that.

And the way to do leak detection with this is really, really coarse leak detection is to wait five minutes while you have some load on your application and run it again, see what changes. Do it a third time, see what changes. And you can actually get, at a sort of coarse level, some really good information about what sort of leaks your application may or may not have.

We've managed to use jprofiler to find, like, really smaller-- so, you know, there are good tools out there. I think jprofiler's a good one. I wanted to give them a plug here. It's really good for finding leaks, as-- as anyone uses--no. So I also polled different people at Apple to say, "What's your favorite tool for finding bugs?" So you may or may not know that the Apple Music Store sits on top of Java, and they run many, many, many machines with many, many, many Java processes. And so when I talked to these folks, they said that the MXBeans was-- I've never used MXBeans myself, so the caveat there. But they swore up and down that these are the greatest tools for doing debugging of real-time, large deployments. So, MXBeans.

And the other thing is, there are other good tools that are coming out. So the future looks bright. I have this--most of this session is like the last three years or two years. But there are some interesting things that are coming out in the future. X-Ray, we've got--we had a demonstration earlier.

Dtrace is coming in Java 6, and that will allow you to hook in at a very sort of fundamental level, see what's going on. And finally, there's a whole lot of JVMTI agents that are sort of coming online now that JVMTI is available in a lot of places.

So those are common tools. Let's talk for a little bit about three common problems that people bump into sort of at different levels of Java. These are sort of the usual suspects. And you should know about them. So the first is if you use Swing. The thing that you need to know about Swing is Swing is not thread-safe. We have a lot of people who, in their second month of Java, they sort of bump into threads. And then they combine that with what they learned in the first month of Java, which is Swing is cool. And all of a sudden they have deadlocks.

So you need to know, if you're working in Swing, that you need to start to understand how threading and Java all work together. You need to learn about the event queue. You need to learn about the event dispatch thread, which is the event queue. And you make sure that any time that you're manipulating the user interface of a Swing application, you're doing it from the right place. And one of the ways to do that is to look up and read about these two utilities. InvokeLater is the asynchronous version, and InvokeInWait is the synchronous version of invoking a method that's going to call something like that. on using swing.

Now this is a really fun one, and everyone should sort of like, if you're reading your mail, you should stop reading your mail and look up at the screen, because this is kind of important. Volatile variables are subtle. You're getting into the realm of data synchronization here. So look up at the screen here. How many people have written a worker thread like this? Wow, something is running, do something again. Like, really, really straightforward stuff.

And there's action listeners too. So who has an action listener where they're like, push the button, turn it off? Right? Now, this actually can cause huge problems when you get to a machine that has multiple cores, when you have a really aggressive just-in-time compiler, and you have the Java memory model, which allows... Let me pull out what's really important here. And that's that this variable here, which is a Boolean, is shared across two threads.

Now, we'll just fade out everything that's not important. If one of those threads... gets compiled and sort of aggressively compiled, the Java memory model allows for that to be put into a register on, like, one processor for one thread, and the same sort of named variable to be put into another processor, another register on another processor in another thread. So unless you declare this variable to be volatile, it's possible that you could just push this button until you're blue in the face, and, you know, the run loop will--I mean, the working thread will just keep on working.

And this is a very hard problem to debug, because if you, like, put it into a debugger, all of a sudden, you're not compiled anymore. You're running interpreted. And everything sort of magically works again. It's one of those vanishing bugs. So, again, you need to learn about the threading characteristics that underlie Java.

Sort of the good news about being a Java programmer is you get exposed to threads, like, two years, three years earlier than the rest of the world does. Sort of the bad news is, you get exposed to threads, like, two years, three years earlier than the rest of the world does. Sort of the bad news is, you get exposed to threads, like, two years, three years earlier than the rest of the world does.

And the good news is, you really have to think about them. The last problem that I wanted to talk about is sort of up at the user interface level, not sort of down at the memory model level. And that's that dots per inch are not the same on the Mac as they are on Windows. And this can bite you.

So the real root problem here is that when you draw text, when you draw string, you're drawing in points. And points is a fraction of an inch. And when you draw graphics, usually you're drawing in pixels. So if you draw 100 pixel by a 40 pixel box, and then you pick some font that you think looks great, and you just bang it in there at a fixed offset, that will look great on one of your platforms that you might want to support. But it might not look so great on another platform that you want to support.

So be aware of this. Do a little bit of math and use standard layout. So just be prepared to handle this sort of a situation. So those are the three common suspects that I wanted to draw people's attention to. Swing's not thread safe. Volatile is important. And cross-platform layout can be complicated.

So that was our usual suspects. Now we're going to go a little bit more in depth. We're going to take one crash. We're going to take it all the way through. So people have run into this problem before, like they have a crash and you know you come to some person like, "Oh you know, sometimes it just crashes." Well, no, people do not just We're going to go into an analysis of a Java crash. We're going to take it all the way through.

The Java information is available in that first log location that I mentioned before, twitto/library/logs/java. And it'll be in a file that has prepended to it the process ID of the crashing thread. And the native information is in a similar location. You guys can all read. One of the things that you should know is that analyzing a crash and analyzing a hang are very, very, very similar.

So everything that I'm about to go over for analyzing a crash can sort of be used in situ for analyzing a crash if you just sort of gather the appropriate information. So all the Java information corresponds to what you would get out of a kill-quit, one of the things that I mentioned earlier.

And all of the native information you could get out in GDB or by using sample. So let's look at the really big picture. So on one side you've got native code, on the other side you've got Java, and then you've got threads that sort of span across them with JNI sort of making the bridge between them.

We will take a look at the Java side first. So we're in Java. All of this is in library logs java. So what I actually did is I went to one of our machines in one of our labs at Apple, and we run a test harness. We run it every night. Actually we run it several times a day. And I pulled up one crash that had happened sort of more than once.

It happened a lot. One of the things that I've done here is, if you look at my ls command, I passed -l for long, -r for reverse, and t for time so that the very last crash is sort of at the very bottom of my listing. You can end up with a lot of different crashing threads if you're really putting a lot of stress on Java. So it's important to know that sort of-- it's important to line things up correctly. So we're going to look at process ID 16009.

So this is the view of java.native.crash from sort of high up. You guys, I've got this written in a one-point font, so I don't expect you to be able to read this. The first thing that you have at the top is the crashing thread, if Java can find it. And you get a list of all the threads that are running.

You get a list of all the libraries that are active, and then you get a list of the VM arguments. So all these are sort of handy things to know about Java when it crashes. And I think that the most important, most interesting section is the thread section. Now you guys, I'm going to take out A lot of this. And just make it more readable for you.

This is a lot like what you would see if you had an exception in Java and you just did a print stack trace on an exception. The bottom bits are going to be more close to your code, things that you've written. The top bits are going to be sort of what Java uses to fulfill the requests that your code has made. So if you look closely here at the bottom is a test case that we've got written in JUnit.

It's called Close Window While Drawing. And you can see sort of further on up the stack that calls Fill Rect, which is something that pretty much everyone's familiar with. And what, you know, sort of at the top of the stack is java2d.loops.maskblit.maskblit. And this is a JNI call and this is one of your sort of signposts for things. Now you're in JNI and you're starting to leave Java. And so it's good to sort of fix this in your mind as, okay, that's the last thing that I did in Java before the crash happened.

So now we're going to sort of switch sides. We're going to go from the Java side over to the native side. And we're going to look a little bit about and follow the crash through on the native side. So all this information is in the crash reporter section of your logs folder.

So again, in a half-point font that no one can read. One of the things to know about a crash log is that in Tiger, it's all prepended together. So the first crash is at the front, and the second crash is right after the first crash. And the crash that you're usually interested in is 125 pages down, if you're me, and you've crashed a lot. So one of the important things to know is you need to look right at the bottom.

So let's take this apart. There's header information. Again, when you're running in Java, you have a lot of threads running, so there's a list of all the threads. And then again, there's a list of all the libraries. So let's dig into the header. One of the things that you want to do is you want to make sure that you're looking at the right bits. So I looked at this. OK, this is about the time that the crash happened, so I'm cool here.

The other thing that you want to look at is you want to look at the process ID, and make sure that that lines up with the crash log that you were looking at before. So sure enough, we've got the right process ID. So we're in 16.0.0.9. We're good.

The last thing you need to look at-- I've got this in red here-- is which thread crashed, because that's where the real meat of the problem is going to be. That's thread 14. So three important bits-- the date, the process ID, and the crashing thread. There's other important bits there, but those are ones that I find to be particularly useful. So there was a bunch of threads. We want the 14th thread. So let's zoom in on that. So the crash thread, again, is always interesting.

Now I'm gonna color this for you to make it easier. Again, you sort of see that signpost that I told you about before. So this was the last thing that we saw leaving Java, and it's sort of halfway through the native stack. So why is that? Well, every, you know, I've sort of color-coded everything for you. Everything below, in blue, is, uh, client.dialib.

That's what the native side of Java-- running Java looks like, and you can pretty much ignore that. That's just Java doing its job running Java code. Everything above your JNI call is sort of where things went wrong on the native side, servicing whatever JNI request you have. And you might actually see, if you have some really complicated situation, where you go in and out of, um, of Java and native code. So look for things like whether you're in client.dialib or whether you're in, say, AWT like we are here.

So let's again take out and show a few bits to sort of give a few more clues about what's going on with this particular crash. You can see that after we got our request to blit a bunch of stuff, we ended up in some code saying, OK, we're in a drawable context. Let's finish it up.

OK. Now we're going to call a free handler. So now we're starting to get all the clues about what's going on. So this is really characteristic of what happens when you're over on the side of the world that's not garbage collected, when you're doing things like managing memory yourself. So what's going on here is we've over-freed something. We've gotten rid of it and disposed of it before we were quite done using it.

So while I've got you in the native crash, I want to show you a few more things. Thread 0 is the AppKit thread. And the AppKit thread is where a lot of cool stuff happens. It's where Cocoa sort of does all of its grinding. I wanted to show you what sort of a happy AppKit thread looks like.

So take a sort of closer look at the screen here. There's a couple of things. You'll know that you're on the AppKit thread. You'll see something like AWT starter, start AWT, if that's sort of a sign that you're looking at the correct thread. The other thing is, this is sort of a happy thread because it's sitting in Mach message trap plus 7. So it's just waiting for something interesting to happen. So nothing is going wrong. But I wanted you guys to see, it's good to know what waiting threads look like so you know what you can ignore.

So semaphore.wait, that's a waiting thread. Mach message trap, that's a waiting thread. These are things that you can sort of safely ignore. So now that I've told you what you can ignore, let's go back to our big picture. There's the native side you need to know about, the Java side, which is where you usually start your analysis. There's many threads that cross across them. And these are threads that you need to sort of learn about to do sort of black belt Java debugging. The AppKit thread is where, as I said earlier, Cocoa does a lot of its work.

So this is a thread that you'll look at quite frequently, a thread that you need to know about. The event dispatch thread is where Java does all of its work. So, not all, but much of its work. And so that's another thread that you need to sort of learn about a bit. We actually, in this example, were in some other thread when we crashed. We're thread 14, just a worker thread that I'd spawned off.

So let's just all tie it together, so we can have some closure on this one bug all the way through. We were in my test, closed window while drawing. We called fill rec somewhere partway through. You could see further down on the Java side that we made a JNI call. Took us across the JNI boundary to the native side of things. We did some more native stuff. And then we crashed. OK, great.

So that's one bug all the way through. Except, well, whose bug is it? Who are you going to send Bruce Lee after? Well, in this particular case, it's Apple. If you have a crash, and it's pure Java code, and you know it's pure Java code, it may not be our bug.

Wow. It may not be our bug, but it's our problem. So send it to us, bugreporter.apple.com. I've got this wrong. Bugreporter.apple.com. If you see some other odd behavior, or you know that you have a JNI library that you've put in place yourself, you need to sort of apply a little bit more brain power, think about what's going on. So there are other considerations.

One of the things that's really good in debugging is to know that configuration matters a lot. So it's good to have various versions of Java around. And of course, we make that easy for you by deploying various versions of Java. But also, if you can get access to a Linux box or a Windows box and see if it happens there, see if the bug happens in Swing and in both looks and feels, Metal and Aqua.

The other thing that you should know when you're doing general analysis is that Java Dev is a really, really good resource. How many folks here are on Java Dev? I probably know some of you from not your faces, but your emails. Java Dev is really technical, and it's really friendly. So I encourage everyone to sign up and become part of this mailing list.

So that's one bug all the way through. We saw the Java side, we saw the native side, we saw basically that you need to send it to Apple or think about it. Let's talk a little bit about how I learned about things, because I think it's a good way to learn about things. Let's take things apart.

Um, hopefully you'll--won't be the villain. You'll be the good guy, but we'll be deft about taking things apart. So for me, that--I mean, that's how I learned. So, the other thing is, sometimes you use other people's bits, and you sort of need to know how to get at all the bits of those bits.

And I have one--one sort of plug here, you know, use your powers for good and not evil. Everything I'm about to tell you can be used to steal other people's code. Don't do that. Um, but let's get right into it. There are four ways to deploy Java on the Macintosh, and we're gonna take apart all four of those ways.

We're gonna take apart--we're gonna start with jar files. Jar files are sort of the--the, um, base case of taking things apart. Uh, so let's talk about how you pull apart a jar file. You're pretty much already done once you have the jar file. Um, there are--there's a command up here on the screen that you should sort of commit to memory, jar-xvf, and then whatever your jar file's name is, .jar, will completely unpackage, uh, a jar file. So I'm gonna take a jar file for you.

And the reason that's important to know about this is many, many people, including Apple, we distribute sample code, uh, as jar files. So if somebody gives you a sample code, you know, often the class files, the executable stuff, the manifest, and actually the source will be right there.

Um, so it's good to know how to pull this apart. And then if you want to run it on the command line, the main class is tucked away in the manifest, which I've got up on the screen here how to get into. So if you just cat that, you can find the main class, and then you can say, "Java, maybe xrun shark," an important command to be able to do, and then the main class, and then you're done.

So applets are a little bit trickier to take apart, especially in these Web 2.0 days, when more and more people are deploying not straight HTML. The first time I had to take an applet apart was actually, I was doing a tutorial. I was walking through a tutorial, and the person had his applet up animating-- I forget, it was spinning a square or something like that-- and he compiled this code.

Run it, and I compiled the code. It's a class not found. And what happened is the person who was doing the three-page tutorial had given me his source code for what he thought of was the important class, and there was two other files tucked away, like a quit handler or something like that.

So it's important to be able to get at these bits. Curl is a good command to know about. So take a look up at the screen here. Curl is CURL, is another way of saying it. So you can look at URLs. Dash O is where it's going to stick the bits when it's done.

It took me a while to find something that wasn't actually a website. So xyzcba.org doesn't actually exist, one of the few websites that's not taken yet. And this is how you would pull down the HTML that was displayed at this location so you could work with it in an editor or something like that.

And then you sort of have to apply your brain a little bit. You'll have to look for the applet tag. The applet tag tells you pretty much everything that you need to know to get back to the thing that we talked about a slide ago, how to get to the jar file.

So in this case, you'd say, oh, OK, there's where the applet is, there's the archive, all the important information. And then you go and use curl again to get the jar. And then once you've done that, you've sort of got all the bits in your pocket. So that's how to pick apart an applet. Now, most of you folks are, at least in some sense, Mac developers. So the best way to deploy a Java application on the Mac is to make a bundled application.

And I didn't really understand application bundles until I started pulling them apart, trying to find the Java bits inside. And you have to dig for a little bit to get there. Every application on the Mac is really just a folder that the finder knows to treat as a single entity. So you can actually go in there and just look at it with LS or CD.

There's a contents folder. And there's two important things in the contents folder that you need to know about. One is the Info.plist. And we talk about the Info.plist in our release notes and things like that. This is a place where lots of different Java options happen. And I just wanted to draw your attention to it while I had it on the screen. Everyone should know that the Info.plist is down inside the contents area. And the other thing is if you keep on digging resources in Java, that's where you actually find the jar file that you have deployed.

So if you're like me and you like to work on the command line and you want to run your thing from the command line, that's where you find the bits. So bundling applications is a big deal. And so there's a lot of documentation available so you you can go dig into that.

Java Web Start is the best way to deploy your applications cross-platform. And one of the reasons that I talked about applets before was because applets and Java Web Start have a lot in common. So when you want to pick apart Java Web Start, what you're looking for is the JNLP file, which is just basically an XML file, really, that specifies sort of all the bits that you might need to run Java executable.

So the three sort of important tags that you want to look for is the home page, like where all the bits live, so that you could use, say, curl to grab them, the jar file that you're looking for, and finally, you're going to want to find the main class.

So look for the word "main." And those are sort of the three bits that you need to pull apart a Web Start application. And sort of the reason to know this is they're also the three bits that you need to know to put in there to deploy a Web Start application, so sort of turning around, deploying on the Mac. So, whew.

OK, Mikey. So my machine, it's back up and running again. And it's back to the state it was before the terrible things happened. So let me tell you a little bit about what happened off camera while Mikey was talking. So while you were still showing your disk partitioning scheme, I had rebooted into my admin partition and restored my system, which I'll talk about in a sec. And I believe I had finished restoring about the time you had hand grenades up on the screen and were talking about the black box tools.

So I had restored it. And then I rebooted into the development partition and made sure everything was the way I wanted it to be. And you were talking about threads and volatility in Java at that point. And then I got tired and I had to go sit down.

So that's what happened in a nutshell. The key point that I wanted to show you is sort of my recovery strategy, which is slightly different from Mikey's multiple partitions. I don't have as much disk space on my portable to have a ton of partitions. So what I do is I freeze-dry partitions the way I want them to be.

So on this machine, on my data partition, I have an image that I call Tiger Demo 4. This is a freeze-dried image of my demo machine, my development partition, the way I wanted it to be. And I use Disk Utility to restore it. And I'm just going to go through the motions here. I'm not going to re-restore it. You drag your image into Disk Utility.

Click on it, you choose Restore. You tell it which image you want to restore. You tell it which partition you want to restore to. And then you click Restore. And that's what took a long time, which is why we did it off camera. After that, I now have a development partition that is back exactly the way I wanted it.

The great thing about disk images is you can, as opposed to using the install DVD, is with disk images you can set up your users the way you want, all your system preferences the way you want, networking, everything, and then just freeze it. Then you can do risky things, and then you can bring it back to life, which is what I did. And the details on how do you create the disk image, they're just in Help for Disk Utility.

If you go to Disk Utility Help, type Restore Image, you'll find the page that tells you all the steps to do to create a restore image. It's very simple. But don't forget step eight here, because it's non-intuitive. The step eight says choose images, scan image for restore. That's right here. Images, scan image for restore. There we go.

That was the one key thing that I thought was not obvious. Everything else is pretty straightforward. You can create images of any partition, store them on FireWire drives, store them on servers, and get to them. And restore your machine, and then your machine will be back and ready to do dangerous things again. Things like handing it off to a summer intern to do WWDC, hands-on labs. Oh, wait, one more thing. Let me throw this thing away. It's gone. All right. Thank you.

So this is Brett Groshong. He's an intern with us at Apple Computer. Before we go off into the hands-on section, I wanted to actually tell you some of the really, really cruel things that we did to Barry's computer. We actually went into his systems folder and we removed all the frameworks. We took them all away. And I think we started killing processes after that. So it was pretty severe. So let's talk about Shark, because Shark is like a really, really great tool. Dr. Evil would be proud. Shark is super powerful, and you can do amazing things.

So I wanted to throw up one more time the required hands-on material for this section. I really think that everyone in this room who's got a portable should do the Tai Chi thing and do what I do and do what Brett does. So I'm going to give you a few seconds to get your portals out. Wanted to give you a fair warning. In the run-throughs for this, we actually had one or two people over the course of the several run-throughs crash.

So now is a good time to, if you're chatting, say goodbye. I'm going to pay attention to Mikey for a while. If you've been composing an email, save that and quit mail. And I really do want people to take the time to do this hands-on section, or at least pay really close attention as we go through the hands-on section. So just to get a sense, how many people are going to work on Shark with me? Come on, one or two more. All right. That's good enough for me. This is one of my goals. Let's go to B.

Okay, so you should all have some version of the Chud tools and you should have a sort of deeply flawed application that Brett and I have put together for sort of showing performance. So we're going to do a few steps. We're going to launch Shark which will sort of satisfy one of my burning desires which is to get more people to run Shark, and then we're going to go into the block animation folder and we're going to do this old school.

We're going to bring up the terminal and compile it there and run it there, and then we're going to look at it in Shark unsuccessfully, and then we're going to look at it successfully, and then we're going to sort of carry on from there. So let's get started. So the first thing that I want you to do is go into slash developer or so go to your root volume and Go into /developer, applications. Don't do this as fast as I'm saying. Applications, performance tools, Shark.

So, applications, performance tools. And everyone go ahead and launch Shark. Okay, who's got Shark up and running? I'm going to wait for a few more hands here. I really want everyone to run Shark because it's such a great tool. Okay, so, Shark is running. So now--forget about Shark, because there's other not so great things to do--we're gonna do the block animation demo.

So if you guys have downloaded this in, like, DMG form, you can't edit a mounted DMG. So make sure that you drag that into a folder or something like that so that you can actually edit it. So--and now, I want everyone to go to the terminal. So if you're Black Belt, you can do this in, like, four keystrokes, but we're gonna do it the long way. We're gonna actually go /applications/utilities/ terminal for T.

Launch the terminal. OK, how many folks are with me in the terminal now? Cool. This is great. I haven't lost anyone. OK, so now cd, change directory. So we're going to make the working directory be the block animation demo working directory. So cd, and then you can either drag block animation into the terminal, or Brett's a good typist, so he's going to type the whole thing. No, he's not. He's going to do that. Anyway, so I want you to change directory to block animation. Go ahead and hit return, Brett. Now you don't have to do this ls thing, but I'm going to make sure that he's the right place. I'll type ls, Brett.

And sure enough, there's my Java file. Now I don't know which version of Java you guys are running, so I want to recompile this, and I want everyone else to recompile it too. So now the way you compile in the terminal is to say javac. So javac space capital block capital animation dot java.

and Richard Okay, so how many people are now compiling my interesting but deeply flawed animation demo? Okay, so not everyone yet, so I'm going to wait a little bit longer here. Compiling, compiling, compiling. Okay, good enough. So let's run it. Java, space, capital block, capital animation, no Java now, because--and it should run.

Okay, how many of you see something similar to what's up on the screen? I'm going to wait a little bit longer for people to catch up, because it's important to be able to see this. Cool. So I'm going at the right speed. So let me talk about what's going on here.

We have a big J frame, and then we have lots and lots and lots of panels inside that. And every panel has a timer task associated with it. And every second, plus a little bit, the timer task fires and says, repaint me. And repaint me with a different color. And so it kind of grinds away.

So let's take a look at this with Shark, and we'll bump into the first problem that everyone who tries to do a Java profile bumps into. So let's bring Shark forward, everybody. So go down into your dock and click on Shark. And what we want to do is a Java profile of this thing. So let's go down to the ninth item. He's going to hold that pop-up for a long time, but you guys can just go down to Java Time Profile and select that. So go ahead.

So Java Time Profile. Now, how many people see nothing available? So this is the first problem that everybody who bumps into a problem bumps into, and I wanted to actually do this in the hands-on to drive home what you have to do to fix this problem. So what you have to do to fix this problem is type x run shark. It's a command that you need to give Java to give it a few hints.

Actually, it turns on some JVM PI bits that Shark can use to find out what's going on. So let's go back to the application, click on it, and just quit. Okay, so how many have had the thing go by? Great. Now, I want you to run it again. javaspace-X-run-shark. I'll let you look at what Brett's got up on stage here so that you can get it typed in correctly.

And what you will see is Shark for Java enabled. So this is your cue that you've done everything correctly. So how many people have now got Shark for Java enabled? OK. Anybody who's actually not doing this on the portable, sort of scoot over to somebody else, because this is so cool.

Okay, good. So this is the other thing, which is I think that the, so let's, let's, Well, I talk about crashing things. You try to put this in a bigger font in the terminal so that they can see it. So bugreporter.apple.com. Like I said, sharks have teeth. You know, sometimes we have definitely seen people crashing this, but for the person who, like, shouted out, um, "Take the time. "Take the time to put the crash log "and what steps you did in Bug Reporter," because we really--we do have a staff team on the other end of this way cool tool.

Um, and they need to know that there are Java developers out here who desperately want to profile their applications with Shark. Um, I don't know exactly what you did. We definitely are looking for, like, exactly the steps to reproduce this problem. We know it exists, so anyway. And now you can see... and David Koehn. So let's go back to Shark.

Those of you who haven't crashed, And now you should see that Java is available for being profiled now. So how many folks actually now have Java as something that they can profile? Cool. Okay, so the way that the Java profile looks is it takes time slices. Boom, boom, boom.

It looks at what's going on. So I don't want you to start doing that yet, because I want everyone who hasn't crashed to have a really successful experience with Shark. So go to the Config menu first, in case your machine is not as fast as Brett's fast machine, and go to the Mini Config Editor.

Ring it up. And it's set by default to take really, really quick samples. And that can overwhelm things, especially if you have a really badly written block animation demo running. So change that to be 50. So only take 20 samples a second. Don't take quite so many samples every second. And now I'd like everyone to press start, and then give yourself about a two count. One, two, three, and press stop.

Okay, everyone who's done this has now successfully profiled a Java application. You should see, everyone's going to see something a little bit different because, you know, we're all on different machines with different timings and stuff like that. But now you can see where my code is spending all of its time. So how many folks see something sort of like what we've got here? Now I don't think we can fix the font here, but scooch over to somebody else's machine and look what's going on, or look up close.

So, So this is really cool. This tells you where you're spending all of your time, while your application is going. And it's as easy as this to profile a Java application. Now, I've been using performance tools for a really long time, because I've been programming for a really long time.

And one of the things that's great about Shark is they've taken, iteratively, really advice from people like me, and smarter people than me. And they've incorporated into Shark. And one of things that you'll probably see is that you're not spending a lot of time actually in code that has the word block animation anywhere there. So one of the cool tools that Shark has is called a data mining tool.

So what we're going to do is I want everyone to select the first line and go to the data mining tools. And what we're going to do is we're going to say, any time that you spend, we're going to take the most expensive function, and we're going to charge it upwards.

So go to the first line and say, charge library to callers. Now, he's in Java. He's in Lang. But you might be in like Apple A, but he-- so go ahead and do this, Brett. Charge it up to callers. And what that's done is that's taken all the time that you spent in that library, and it said, I don't care about that library. I want you to pretend that you're in the library that was sitting right above it.

And actually, I want everyone in the audience, and Brett here, to do that a couple more times until we're actually up into code that has the word block animation in it. It might take a while. If you've taken a long sample, it might take three or four, maybe, hours.

But it's a good idea to do that. And then you can see that the code charges upwards. And everyone should keep on charging it up until you get to a blank line in library there. And the word block animation should be one of the symbols that you see in the symbol section.

How many of you folks see a, are already charged all the way up? How many of you folks are still charging up? Okay, keep on charging up, because, um, and Brett's going to do the same thing here. The reason that I'm doing this is not just to show you that you can charge things upwards, because I want to get to what I think is the most compelling view of-- I mean, this is the thing that sold me on Shark. So who's--I mean, how many people are charged all the way up here? Should I wait for anybody else? Okay, let's go on. So what I want you to do now is go to that blank field in the library section, and I want you to double-click right there.

Okay, this is a super cool view. Like, this is showing you in your code on a line-per-line basis where you are spending your time. This, like, totally sold me on Shark. When I saw this, I'm like, okay, I have to have Shark. And this is just, like, one of the many things that Shark can do.

So what you're seeing now, hopefully, if you're having a successful demo or hands-on experience, is that you're spending pretty much all of your time calling FillRect and a little bit of your time doing some other things like setting the color. So how many people are still with me here? Aside from Brett, who's, of course, still with me. Great.

Okay, so now you have profiled a Java application, and you have sort of seen where it's spending all your time. And you should do this to your applications, because you can learn a lot of stuff about the real-time performance of your application this way. Okay, so I'm gonna pick up the pace a little bit here so you guys can stop, like, trying to follow along and sort of watch on screen.

We're gonna close this profile, and we're gonna sort of dig into some other aspects of the runtime characteristics of this program with another profile. So, Brett, let's go to the second profile that we like to do. We're gonna do System Trace. So we'll just start it and stop it briefly.

And the great view--this is a really good view for getting sort of a global picture of what's going on, not just in your system, but-- not just in your application, but in the whole system. And this is really powerful, 'cause it's watching every single kernel call on every single thread, um, and providing you that information in real time, uh, sort of, uh, on a really fine-grained basis. And this will actually, um, let us see what's--what's wrong with my--my sort of poorly-written block animation demo. The other--I mean, the other view was sort of interesting. It told us we were spending a lot of time filling rectangles, but that wasn't too surprising.

Um, I mean, of course you're filling rectangles. You're doing, like, lots of little flashy animations. So this is sort of a big summary view. We're gonna go to the timeline trace view, 'cause this is--this is the other sort of selling people on Shark portion. So every single thread has got every single kernel call and a stack associated with that kernel call, like, sort of-- right there. So you really can see at a really close level what's going on with your application. But let's go into sort of the course level of what's going on with this application.

Um, you can see that we have a lot of threads. I mean, we have a lot of threads, a really, really lot of threads. Um, and they're all very, very occasionally, about once a second, making a kernel call. Now, this is the wrong way to do animation, so-- that's how we're doing for time here. Good. Let's actually fix this, um, problem, Brett.

So we actually--this is--this is-- one of the reasons that I wanted to do this particular demo is as we were coming up with the demo to show you guys and to have you walk through, we actually sort of inadvertently inserted this real-life bug into the problem, into the program.

So we're going to fix it. So let's go into the-- we'll launch it in our favorite editor, which is Xcode, or Emacs, whichever, or VI. So what we've got is we've got really three classes here. We've got the J frame, which holds everything. We've got J panels, which do the little animation.

And it turns out that what we did incorrectly is timers-- you only really need one timer thread associated with any given set-- any given application. And what we did is we instantiated a timer-- not just a timer task to do the animation, but a timer thread, too, in every single one of those J panels. So all we needed to do to fix this bug, and what Brett's doing now, is we pulled that timer out of the J panel, just sort of hoisted it up into the J frame, and recompiled.

So all you need to do is take it out of the inner class, make it a private timer in the J frame, and recompile and rerun. And all of a sudden, you'll have-- The thing to take home here is that there are different aspects of performance that you need to think about.

If you're doing animation correctly, We here? Pardon me? I also crashed the show. Okay, good. So you should file a bug. bugreporter.apple.com. Shark is a great tool, but it does have teeth. You just have to be aware of that. I think that one of the reasons that crashes too is, I think many systems don't like to have 600 animation threads running at a time while you're doing really, really low-level probing of the system. John, did it crash? Okay.

Well, again, 600 threads doing animation all at the same time can... Okay, bugreporter.apple.com. We take all of them seriously, and we'll actually go into that. And so, so here we are, running again. It actually performs essentially the same, but now it's doing it with, like, one animation thread. Now, since we're using timers, the frequency is going to be the same.

But if you look at it in the other really course level performance tool, Activity Monitor, everyone should know that if you can't use Shark, you can always use Activity Monitor to get a really course idea of what's going on. And now you can see that in the old version we had 591 threads, and in the new version we have 14. So much happier. Hopefully, you guys have all followed along with me, seen that sharks have teeth, but also seen that you can profile a job application. Let's go back to slides, please.

So there's two other modes of Shark that you should sort of know exist. There's one for watching allocations, and there's one for watching on a method-by-method level. And there's two other sessions that I want to point you at. So take a quick look at the slide, because they're going away. One's tomorrow morning, bright and early, performance--not tomorrow morning, Thursday morning, bright and early. And the other one is just generally performance tools in the Mac, which will include Shark.

So there's two other modes of Shark that you should sort of know exist. There's one for watching allocations, and there's one for watching on a method-by-method level. And there's two other sessions that I want to point you at. So take a quick look at the slide, because they're going away.

One's tomorrow morning, bright and early, performance--not tomorrow morning, Thursday morning, bright and early. And the other one is just generally performance tools in the Mac, which will include Shark. Inside Apple. It's not quite like being inside an alien vessel, but there are similarities because it is a bug hunt.

So Apple is--so this is the history according to Mikey. Once upon a time, a really smart person at Apple wrote a really cutting edge tool for tracking bugs. We call it--it's called Radar. It's internal only unfortunately. But for its time, it was just, like, the best way for handling bugs that existed. And this was like 10 or 12 years ago, 14 years ago.

And consequently, because it was such a good tool, Apple became sort of bug-centric in its process so that, you know, 10 years later, we're still very, very bug-centric. If I wanted to go and change Java code just to the point of saying this, you know, cwindow.m needs more comments, I would write a bug saying, "Bug title, cwindow.m needs more comments," and put it in state analyze.

And so the development process inside of Apple has it be the case that anytime we touch code, we have a bug that's associated with that code talking about what the problem is. So this is a powerful thing for you folks to realize and it's why I've been hammering home bugreporter.apple.com. Like, for things to change, there--we speak bug.

Everything that we--that we do internally sort of has a bug attached to it. So if you know sort of the language that we're talking, which is bug, then you have more power over us and you can get us to do what you want. So, yeah, so bugs. Bugs are really important and Apple is bug-driven. Java at Apple is very, very bug-driven.

The other thing is give us some clue about how important the bug is in sort of the grand scheme of things to you. So if this is keeping you from switching from 1.4 to 1.5, we need to know that because we want everyone to go to 1.5. If this is going to prevent you from going from 1.5 to 1.6, or if this is, you know, I can't deploy to 20,000 customers, tell us because we need to know. And we sort of factor that in.

Now this one should be obvious and it's sort of annoying that it's true. But if you sort of keep coming back--and again, this goes back to developer previews--be persistent. Be a champion of the things that you want to get fixed. And that works really well. The way that this ties into developer previews is if you get a developer preview from us and you happen to know that there's one particular bug that you would really like for us to address, and you actually say, you know, take the time to say, "Does the bug still happen?" and you put that information in the bug, we know that you still care.

We know that there's at least one person out there who like still today, honest to God, wants this bug to be fixed. So be persistent and keep coming back to your bugs. Don't let us let them die. We have great bug tracking tools, but the more that you do to sort of keep the bugs that you care about the most hot, the more that we can do.

The other thing is that, you know, every now and then we get, like, a bunch of bugs that we're like, you know, Java does the wrong thing and you get really, really angry. Like, try to keep that out of the bugs. It won't actually make your bug go faster to be nice to me and my team, but what will actually happen is we will last longer. So you know, we won't burn out.

So let me just run through the last, like, hour and 15 minutes. Talked about Developer Previews. I think they're really important. As long as I've been at Apple, and probably as long as I will be at Apple, I'll be pushing for Developer Previews. So use what I'm giving you. Talked a bit about the basic tools. Kill-Quit is one that everyone should know. We did a sort of really deep crash analysis, walked it all the way through all the logs.

Talked about how to pull things apart. We went hands-on with Shark. Thank you all for profiling a Java application with me. And then I talked a bit about what goes on inside Apple. So we've got about five minutes left. Here are some other sessions that you might want to attend.