So good afternoon. Welcome to session 622. I'm Bill Bumgarner. I'm a manager of Enterprise Objects Framework and the Core Data Framework. I've also done WebObjects development forever, since 2.0. I'll also be joined by Max Muller, who's lead engineer in the iTunes Music Store, which is probably the highest throughput WebObjects application ever built. If you know of another example, I'd love to hear it. I can think of a few that were pretty high throughput, but nothing like that. Between us, we've done some optimization and would like to share some wisdom with you. We hope it's wisdom anyway.

As an introduction here, scalability and low latency is critical when you're doing web-based applications or when you're doing any kind of server-based application. Building one successfully is a challenge. It requires a lot of analysis, a lot of tuning, planning. And also one of the sort of critical areas is security. Security always complicates these things. Security is kind of the antithesis of efficiency and convenience. So, you know, that will challenge things. And it's not just about the code, it's about the entire development environment and deployment environment, the whole process. and we're gonna cover on a bunch of those points. This is gonna be a very information rich session. You're not gonna see a lot of pretty pictures. We'd also like to get through the content fairly quickly because we've always found that talking with you, the developers who are living this stuff day to day and answering questions about your specific problems always brings up new things and is always interesting to everyone. So what you'll learn is a little bit about, and these are all guidelines. Unfortunately, I don't have the perfect formula for everyone. Is how to architect apps to be fast and scalable. Some of the patterns and pitfalls for success and failure to achieve a fast application that works. How you can analyze existing applications to determine where the bottlenecks are and then address them. And finally, it's not just about the code, it's about getting it out there and into production. Once you get it into production, there's a lot more services available to you the WebObjects application itself. There's other tools and resources you can use depending on your budget and your traffic rates to be able to keep an app up and running.

And we're also going to cover things like when good apps go bad. There will be a Fox special this fall. Um... Like if you'll notice if you deploy on, say, your eight processor CPU, or eight CPU Solaris box, you can see your CPU usage go to 800%. That's always fun.

You can even see it go to 900%, and then it takes down the machine next to it. There's also problems with like when your working set gets larger than physical memory and then disk is obviously a lot slower than RAM. And when you start hitting the disk for memory, that's bad. Also, there's the situations that a lot of people seem to miss, which is like network saturation when packets or connections are being dropped. I know of one example of a very large company who will remain unnamed who called the FBI because they thought they were being hacked and it turned out that they had two Windows boxes and their traffic rates were so high that the TCP/IP stack was falling over and dropping connections. So, you know, that gets back to the analysis thing and calling the FBI to analyze the performance problems of your apps is probably not a good idea.

There's also just the basic situation of like when your responses require too much computation. If you're going off and calculating pi to 100,000 digits to bring up your welcome page, that's going to be a problem. Your database can be obviously overwhelmed. You're hitting it way too often or you're just pulling too much data from it or pushing too much data into it. That can be bad. And a great one is external services because no external service ever wants to admit they're wrong. So instrumenting your apps such that when they're wrong, you can prove it will make your life a lot easier.

So how can we fix these problems? Well, unfortunately, dollars are always involved. The basic problem is you can just kind of throw money at it and get more CPUs, get more memory, get more network, get a faster database, that kind of thing. But that won't always work, and that's what we're going to focus on is when that doesn't work and you need to throw engineering at it.

And what you're really looking to do in that case is do less work to generate the response or to make more efficient use of the database or ideally don't hit the database at all. Or optimize your external service integration and that becomes especially critical as your site grows in size and you need to start relying on those services more and more.

So good rules to code by. These are somewhat obvious, but they bear repeating because when you get into the thick of things, it's easy to forget. Test-driven development. Now this is something that's really become popular in the recent years, and I can't emphasize how valuable this is. Put the unit tests together. Do it before you write the code. That tests the capabilities and the requirements of the code, and then run them every single time you build. And when you push to deployment, run them there and run many of them in parallel if you can. They will uncover so many obvious problems and save you a huge amount of time.

You know, another obvious one that everyone misses, including myself, is make it work, make it right, make it fast. As developers, we often like to make problems a lot harder than they really are because then we're, you know, I don't know, our egos are boosted when we solve a hard problem. It's like, look, I've sorted something 20 times faster than the next guy. Never mind it's only 10 elements. Don't optimize without analysis. And I can't say this enough times. I've walked into so many situations where someone's got the world's most optimized means of writing out the HTML page that's only used once when the user signs up the first time. It's like, come on, you know, go fix the problems that are really there. And also, optimize in small tests and test your results after each of those steps. And this gets back to unit testing. If you've got the unit tests in place, then this makes optimization a lot easier because you can go off, you can do the optimization and know immediately if you've broken things. And the last point is just, it's an obvious one, but everyone does it. If it ain't broke, don't fix it, which as object-oriented developers, that means if it works now, don't generalize it, 'cause that's one that we commonly do.

Also, there's a lot of optimization that you can do at design time. if you understand your problem well. And any optimizations you can do before the first lines of code are written are always good, keeping in mind that you shouldn't optimize things too prematurely. So you really want to understand the usage patterns of your applications, know how users are going to use it, know how it's going to be administrated. Now clearly, you can't be omniscient, but you can do a lot of upfront work there. And of course, make your entry page fast. I can't tell you the number of times I walked into a client and I hit the entry page and it did 600 SQL queries. It's like, no, no, no, no. Make it static. Even if you have to have an entry page that just comes up that's just long enough to get them into the site, that's fine. Make that fast. Your business logic in your application should be designed around the response generation. It should be designed around what the customers are going to be doing with the application, even if that makes the administrative tools less convenient.

You know, the administrative tools, they need to be powerful. They need to be intuitive. But if they take a little bit of time, hey, that's okay, because it's all about administrating the content for the purposes of the customer. And if your business is like any business I've ever heard of, "It's the customer that pays the bills." And I've seen a lot of people that lose sight of that. You also wanna make sure you're retaining and reusing data.

and know when it's out of date. This will come up again, but it's something that bears repeating. You know, if you go into the database to pull out the list of states in the country, it's more likely not going to change in the next five minutes. So keep it around. That introduces some complexity in working with enterprise objects, because of course you can't create relationships between entities that are in different editing contexts. So you have to be a little bit careful. And you wanna manage that cache data carefully, 'Cause if you end up caching everything all the time and just leaving it in memory, then you're gonna get back to the part where you run out of memory, your machine starts swapping, and then your performance goes to hell. So you also want to let the database server share in the work. Databases have been around a long time, and they do things like sorting stuff really fast. They also do indexes and all this other stuff, and you really want to leverage those wherever possible.

So, one of the optimizations is to minimize your memory footprint. If you can minimize the footprint of the application, then you can have more instances running. You can balance the load more effectively. And this means sharing data across sessions, which is not something WebObjects does naturally out of the box. You're going to have to play some games there to get that kind of thing to work. It's not hard. It is detail-oriented. You also want to clean up thoroughly. It annoys me to see a developer make the statement that because of garbage collection, they don't have to care about cleaning up their code. Not every object that's going to be collected by the garbage collector is just using memory. It may be using scarce resources like connections to quote servers, or it may be keeping a file descriptor open, or something like that. You want to let the code know that it's done and over with. Also, any nulled out reference, the garbage collector doesn't have to traverse to deal with collecting. So that's another good way to ensure cleaning up happens quickly.

You want to also clear your transient instance variables when they're no longer in use. And this is not just an optimization issue. This is also a debugging issue. If you have stale state in your object graph and you don't clean up when you're done and then you come along at some later point in time through some code path that you didn't really think about and you run across that transient data and now it's out of date and you didn't know it, you're hosed. Speaking from experience, traders on trading desks get really irritated when they start seeing the wrong prices.

some scars. You also want to do things like set your right session timeout value. Sessions will stick around and then they'll automatically go away. Unfortunately on the web there's no quit button, so it's hard to know when the session should go away. So you need, this gets back to looking at the usage patterns. Look at the use patterns of your app, understand how they are, and understand when it's safe to make the session go away. Then there's also instrument. I can't say this enough either, is to instrument your applications. We've got some wonderful tools built in for instrumenting them, which Max is going to demonstrate shortly. They do a great job of allowing you to detect when resources are being used or when things are getting out of control. You want to review those results often.

Ideally, you want the instrumentation to be something you can turn on dynamically in production so that if any customer calls up and goes, "Your app's not working right," you can turn this stuff on and figure out what's going on. And unfortunately, because we're building applications where it's guaranteed that our development environment is about as different from our deployment environment as is possible, there's gonna be a whole series of problems that will only come up in production, which makes life an adventure. So instrument and collect and then analyze.

Also, you want to really plan the data access, plan when your query is, when your cache is, when your cache updating is going to happen, and understand the data latency issues. Now, data latency is all about looking at your application and understanding when the data becomes stale and how often do you really need to know to let app A know that app B's state has updated. And a great example of this is I ran into a client and a site, and they needed some optimization, And their site would just grind to a halt when it had a bunch of users. And what was happening is they had all this user-specific state, a shopping cart. And every time the shopping cart got updated for user one, all other 30 app instances got notified that that user's shopping cart was updated, even though that user's shopping cart was only on one session in one app instance. And there was a case where they used a generic notification method. And by simply removing that, all of a sudden their app was stable, even under higher loads. and they could have more shopping carts. That's always good. You also wanna do things in memory wherever possible and to try to get zero queries per response. I mean, the fewer times you go to the database, the fewer times you go to disk, the better off you are. And especially when you get into the high throughput sites like the music store, any database hit is gonna be orders of magnitude more expensive than being able to just blast back some static thing from memory. You want to manage your faulting and manage your caching so that you can explicitly update your stale data. You generally want to avoid situations where WebObjects is either deciding to populate relationships on its own or populate caches on its own because it will choose a very general purpose solution that is probably not optimized to your actual use patterns.

And you also want to use a shared read-only, well, I say read-only shared, it should just be really called a read-only editing context for reference data. It's a little bit tricky because again, you get into the situation where you've got to be careful about how you make relationships to and from objects that were fetched into that editing context. But that way you can have that single shared editing context that's read-only, so it never pays the penalty of doing updates or inserts or anything like that.

And then another one that's really something that's more of a modern optimization, this is something that's become much easier with recent releases of WebObjects, is to partition your functionality across multiple applications. And what that means is that if you've got, say, a site where it has an expensive search operation plus shopping cart management plus, say, a library of information plus a couple of other different things, an administrative tool, then you partition those different features into different applications. and then you can control the number of application instances individually and control the configuration of those applications individually to optimize those particular applications for the use they encounter.

That's very important. And with direct-to-actions and with putting the session ID in cookies and then being able to reconnect across different apps, you can achieve a lot of efficiency. What you can also do is use optimized object models per application. So you can go into Enterprise Objects Modeler, and you can bring up your object model, and you can have your full object model for your administrative tool. It's got read, it's got write, it's got everything.

But then all those entities where you don't need to, say, edit particular fields in your application your customers see, you can create a second EO model that has just the fields you need for that particular application. This will reduce the memory footprint size, it'll reduce the amount of data going to and from the database. It just overall will make the app faster. It can be a little tricky because of course then you've got to keep the two things in sync. That's a pain, but you know, if you're faced with this problem it can really help a lot.

That course obviously maximizes reuse through frameworks. Again, got to point it out because some people forget about that. I've run into that a number of times. And you also want to partition between sessionful and sessionless and threaded and non-threaded because threading is always a very complex issue. There will be certain apps where threading is an obvious optimization and certain other apps where it's not so obvious. In particular, things like where you have multiple writers to the same database, you probably don't wanna thread that 'cause bad things can happen when you cross commits or do partial transactions. Sessionful versus sessionless, there may be a number of apps like search apps are often things that don't need to have per user state. And so if you can get rid of the session in a search app and then make it such that it's only caching, has one big cache for all the search data, You can make things extremely fast, and Max can talk about some of the optimizations they did in the music store that go even beyond that.

Okay, so you've done all the right things in development time. It's been the world's most perfect development schedule, and you even delivered early. And you got the app in production, and now it's too slow, or it's using too much memory and you're thrashing the disks, or the CPUs are just like big space heaters, or it just occasionally crawls to its knees just really, really slow. So now what do you do?

Well, the first thing is don't be silly. And this comes from years of experience. I've been very silly myself and have seen many developers do really silly things. Turn on or off the obvious flags. Woe caching enabled, yeah, that's a good one to have on. Woe debugging enabled is a really good one to turn off. It'll bring an app to its knees, go into production. all of a sudden 100,000 users hit it and it's trying to log every SQL query? Yeah, not a good idea.

There's the built-in NS log facility, plus there's, of course, log4j, which is wonderful because they can both be dynamically configured such that you can hit a production server and pay the penalty on logging on only the areas that you know are problematic. And that gets back to instrumenting and making sure that you have dynamic instrumentation so you can actually catch problems in production. Put indices on your database tables. Sounds obvious, but that's one that people often miss. And then they'll go into production, their data sizes grow to a couple orders of magnitude bigger than they are in development, and all of a sudden they're left scratching their heads as to why the heck it's so slow just to bring up a simple page. And this is another one that's funny. Minimize the size of the generated content. We ran into a number of sites where the initial page load, which is a nice big beautiful page with a couple hundred images on it and a bunch of text and all this other stuff, it was about 160K of HTML, which is way too much, of which 40K of that was comments. And another 30K of that was because the image URLs were all slash images slash clients slash sites slash codenames slash foobar.jpeg or gif. And by simply erasing all of that and making it /i/ for the images, we were able to reduce the page from 160K down to like 50, 60, 70K. Still too much. And of course, again, analyze, analyze, analyze.

One of the challenges with building a WebObjects-based app or other dynamic applications is that as a user goes through the site, unless you specifically do the engineering work, you're not going to get a good record of what the heck they did. So what you need to do is leverage the tools available to you. There's WoEvent. There's the WoEstatistics store, both of which Max can show parts of. You can also capture the direct action activity.

Direct actions are the one exception to the rule. Direct actions, because the URL has the name of the action that was fired in it, the direct actions leave a mark in the log as to what the user's been doing. If you also pass the session ID in the URL, which is an option, then that means that you can differentiate between different users in your logs. Obviously, you want to tune the most used areas first. If your copyright page is slow, probably nobody cares. So go for the high traffic areas. Also, just as a recommendation, I would ensure that your checkout process or other payment gathering process actually runs fast. I had a client that was wondering why no one was paying for anything on their site. It was because when you clicked the button to go to the thing where you filled out your credit card information, it took a minute and a half to load.

And everyone went away and bought their stuff elsewhere. So, you know, optimize those kinds of things. It's not even always the most used pages that are the ones that need the optimization. It's the user experience that needs optimization. And there's also just a wealth of third-party tools available, which a lot of people aren't necessarily aware of. Both Optimize and J-Probe work really well. Web server log analysis tools, there is a slew of free ones available.

Plus, there's also commercial products like Urchin that work very, very well. And there's also every major database server out there has SQL query analysis tools built in. use them, show plan, that kind of thing. The database server will actually come back and tell you why your queries took so long. And enterprise objects, of course, will provide you with tools for optimizing those queries or even running SQL by hand, something to be avoided, but sometimes it can't be. Now I'd like to bring Max up, who's going to demonstrate some of this stuff.

Thank you, Bill. So what I did here is built two very simple little applications. One is a Cocoa web services app that just makes a simple query to a WebObjects app I have also running on this machine. And it just brings back just a bunch of SOAP objects and lists them here and then you can click through them and either choose to, then if it's two Rs, you can choose to update them.

And that basically makes the SOAP callback. So it's just a very simple little Cocoa app. And on the server side, we have a very simple WebObjects app that threw in a bit of direct web. So you can basically see the current users. This is something that you can do out of the box very, very quickly. So the question then becomes, if things are slow, what do you do? So the first thing you can look at is the stats page, which will kind of just look at your overall statistics of your pages and say, what am I doing wrong? And it also gives you a good idea of Whoa, stats. Also can give you a really good idea of just where the high traffic sites that you're hitting, or the pages that are coming up. So you can see that this has had 66 pages rendered so far. And what you can see here is that this will give me the number that has been served and the averages and the outliers.

So obviously, given this app, I should optimize the Whoa display page, because this is what's getting hit the most. But it also can give you a good idea that you can see that the first query is taking 1.2 seconds to come up, and the first inspect page relatively quickly, and the list page, you can see that it's rendered eight times, but the first time it took 3.66 seconds, whereas the average is.6 seconds. So that can sometimes tell you that you have something that's coming up that is rather slow. And so to then drill down and figure out what exactly am I doing wrong with that, you can go over to something called the events. So the stats basically just gives you kind of an overall look. And with the music store, we'll log into various different apps and just kind of check out to see what the current apps are looking like in terms of what their averages are, what the outliers are, because sometimes we'll get some very long ones that we'll need to start looking into. So for the events, so this is, it's actually already, but let's just go to the setup. So this is all just stuff that's just, you know, comes right with WebObjects, very simple stuff. As long as you specify the password, nothing worse than trying a million passwords and realizing, oh, it's not quite in the properties file, and those are, it's right there, so. So we're turning all of the events on, so we're setting it to everything, and then we're going back to our app and we're hitting the list all users. So that we can go right back to this and show the event log.

Now, this has many different options, and it can somewhat be rather non-intuitive exactly how these are organized. The one that I always like is I always just like to look at the events grouped by the page and by the component. As this can show, that off of the main page here, The main is obviously the one that's hurting the most. And the query page is coming up slow as well, relative to everything else. So pretty much when you look at this, you can say, well, geez, the main page, that's the pig.

And so when you turn event logging on, it pretty much just covers all your application, and so anything that's going on is getting logged with events. So then you can say, "Well, geez, okay, it's not the new list page. It must be something else." So it just kind of gives you an ability to drill down. So you can see that this -- and voila, list users. What's going on here?

Ah, objects with fetch specification. I think I've seen one of those before. So you can see it on list users. On the pull, that's the action. That's what I had bound up to the action link. So when you're clicking on the link, it says list my users. So you can see that what's hurting here is the fetch. Right here is the fetch spec of pulling up the users. So, Later on we'll come back and basically show some more fine-grained tools after we go through some of the database optimization to then try to discover what's going on. So if we can go back to the slides.

So as Bill mentioned, my name is Max Muller. I am one of the lead engineers in the iTunes Music Store, and I've been on this now since the very beginning. So we worked very hard and came across all the issues that Bill, that Pitt went over and made some of the mistakes, even though we've all been, you know, pretty much most of us on the team have been doing this for, you know, since the WebObjects three days. So this is kind of just more stuff that we've stumbled across, that we've had to optimize.

Being one of the lead engineers, I pretty much get to eat, breathe, and live optimizing these applications. When we launched in Europe, the average from when we had launched to Europe, we were selling on average 5.86 songs per second. That's how many we'd, since we launched. So if you're doing many things per second over an average, you know, obviously the peak's much higher and the trough's lower, small little problems can very quickly turn into very large problems. And that's one thing that we really found. You know, we had to spend a lot of time tuning the database because WebObjects itself is very fast. If you've got just a pure WoW app, it's not doing any database work and you've somehow made it slow, you've really done something wrong.

'Cause yeah, 'cause out of the box, it's very fast. I mean, you're able to generate responses very quickly and the request responses are very quickly. So, So in terms of getting down to the database work, a lot of the stuff that you can have happening in kind of your administration apps can also very quickly affect things that are taking place in your production applications. So we had a content management application that our content team is constantly in there working on, building the new storefronts. And we would notice that sometimes during the day, the store would get slow, and it turns out that they're in there just doing all these queries that are bringing out very large sets of results. And it's actually causing the database a lot of pain.

And so the store itself is starting to get slow because the database is having to service all the content requests rather than actually the requests of people wanting to buy music, which is not a good thing. So putting in place fetch limits, putting in place requiring certain queries can significantly reduce the amount of database work or that your database is doing, which your users actually might not be seeing.

So there's also a number of tools from database vendors taking queries and handing them off to your DBAs. It can also be very handy. One bit that's out there that we did was when we opened a new connection to our database, there's actually a stored procedure that you can call in Oracle to put in information about the connection. Because by default, for all the JDBC stuff, when it connects in, all that the DBA is going to be able to see is that it's a Java process.

Yeah, well, that doesn't really help you if you've got a whole heterogeneous mix of, you know, back-end processing apps, you know, store apps, and a very large environment of Java applications. So you can put in information into the connection so that if they see some query that's running amok, they can actually look at the connection that's causing that query. So we put in, you know, the application name, the host it's on, even when it started up. 'Cause sometimes we found that we forget to stop an instance and it would be off and la la land when we would have rolled a new version of software and be like, "Wow, what's going on? "I thought we fixed this problem." And lo and behold, it's like, "Well, that guy's actually "been running for a week," kind of thing. So binary data in the database is definitely a no-no, especially if you accidentally check the locking column to where EOF thinks it needs to lock on that attribute. And so it'll issue the where clause.

If you do, moving down to a 2-1 relationship, also this goes back to what Bill was saying about having certain models or certain attributes only for what you're working on, for your administration work, and different ones for maybe the consumers. So maybe you have a large clob field that people enter in a bunch of notes about, about an album that shows up, you know, this came from this, blah, blah, blah, blah, blah, blah. Well, that clob, which could get very large, we don't want the store app pulling that thing up. So you can either take the approach of creating a model or at runtime you can just turn off that attribute. Say, really, you don't need to worry about that one. Leave that in the database when we're in this kind of read-only mode, 'cause nobody's gonna be, you don't need the clob. So, yeah.

So we use the shared editing context for pretty much just reference data, kind of complete types, in the sense that it's kind of like a, just a type safe enum that where it says, you know, key one is this key, and key two is this key, instead of kind of having a lookup, that's pretty much what we use it for. So when your app starts up, all the shared editing context information is loaded, and then it doesn't have to be refetched. And likewise, when you trip the relationships to the shared, to EOs in the shared context, it's not, it doesn't require a database trip.

So there is inter-app messaging. If you need to synchronize states between applications for critical pieces of snapshots, a lot of the times just telling it that it's no longer a valid snapshot is good enough. So it's not that you really wanna move all of the snapshots over and say, here's the new snapshot. It's more just along the lines of saying, hey, you know, this snapshot for this guy, it got updated by this one, so the next time you need it, you better go get it from the database.

Raw rows is a useful technique for pulling back large content where you don't need all the snapshots, stuff that you're not going to be editing. Within the store, we have all these popularity caches that are getting rebuilt. It's like if you like this, you like that. As the number of items that you can buy expands, we'll pull that stuff in with a raw fetch actually in a separate thread. So this thread can just kind of sit in the background every so often and determines the needs to go out to the database and pull in a new set of, the, sorry, recommendations between, so when we're rendering some of the pages, it can sync it up. So it'll basically refresh this cache, you know, in the background, in the background thread, using raw, using the raw fetches.

Caching and memory, it's good. A lot of the times if you have kind of a read-only application you can look at instead of using one kind of shared editing context that you'll then pull stuff into and then hold onto and so the application will hold the reference to this. It's not a shared editing context but it's a shared editing context. So, I'm gonna go back to the slides. Adapter debugging enabled, obviously. That's when you just can turn it on and say, what SQL's going on here? This one, it's a godsend. Java length throwables. Being able to generate backtraces anywhere is very handy.

I'll show you how we can change the logging pattern at runtime to start throwing in backtraces anywhere we want, which can really help. A lot of the times, if you're just looking at the SQL, You'd be like, whoa, hold on, where's that coming from? One of the very common mistakes that a lot of people will make is if they fetch an object and then they're like, oh, they pre-fetch everything, and then on the next request, they'll say, well, we need to go ahead and validate the editing context here so we make sure we have fresh data.

But the problem is you've got the object graph there, and so you've got an object, and you've spent the time to pre-fetch out all the stuff that you got it in two or three or four fetches, but then you've invalidated all the data underneath you, and so all of a sudden you start to trip over these things again, and it's like, oh, that's actually been turned back into a fault, I need to go out to the database again. And you're like, well, jeez, I prefetched everything, and now I've got SQL going out the yin-yang. So turning on adapter debugging can help you see it, but being able to actually see the backtrace of where the faults are firing. So I'll show you in the demo what we, a trick that we use in the music store quite a bit, where we can turn on, there's actually a delegate hook and we can throw back traces whenever a fault is fired. So a lot of times we'll turn that on and then go to render a page that has somehow started to get slow for some reason. And a lot of the times it isn't because that page itself has gotten slow, it's because something else is triggering something that's causing the snapshots to get old or wiped out. So yeah, excess faulting, that's a hot one. Also another trick that you can do with the Java length throwables are in a constructor of a Java object, if you have debugging turned on, some debugging flag, you can actually create a throwable object in the constructor and stash that away in an Ivar. At which point then, at any point later on, you can always ask the object, what's the stack trace where you were created? Which can be very handy in, say, Woe Sessions, where sometimes we'll, we have several apps that are completely sessionless, and all of a sudden we'll start to see sessions popping up, and we'll be like, well, what's going on? And so we'll set this value, and then we can, at a later point, get the Woe Session store, just basically it will say, dump out all your sessions and give me the backtrace, 'cause there's somebody who's doing something bad. And more often than not, it's a Woe Active Image. Somebody put a Woe Active Image in, and if you don't bind it up correctly, it'll go ahead and create a session and create a component action for you. Very handy, but not what you want when somebody just accidentally forgot to do a binding and then all of a sudden you've got these pages that are generating lots of sessions 'cause a lot of the times they won't be referenced and so you'll get a lot of them created. So you can get one request and you can somehow get multiple sessions created which just causes all sorts of nightmares. Fetching it from pop-ups, yes.

Yeah, yeah. The local instance of object you have to also be aware of is if your fetch timestamp lag is set, the fetch timestamp lag is saying, how new snapshot do you care about? So oftentimes what people do is they'll create a new editing context, they'll set the fetch timestamp lag to right now. Say, I want everything fresh. And then they'll start doing the local instance of object, and of course, then when they touch the object, it goes to the database.

So you could have fetched all this stuff, and then you're like, oh, I need to do a local instance now. Let me create a fresh editing context. And so that usage pattern, all of a sudden you'd be like, well, I thought I was doing something good, but it turns out that local instance of object can actually cause a lot of trips to the database. Um, simplified object model, if you can. We're at three or 400 entities right now at the music store and it's growing more each week. The... The deep inheritance, the vertical inheritance is the only efficient form of inheritance in EOF. I've used it for many years now, it works rather well.

It allows you to have a user and then a person user and all these kind of things mapped onto the same table and you can still have relationships to the top abstract entity. So when you trip a relationship, you could be getting all the different sub entities, but it all, it all, it all, it, but EOF handles that gracefully for you underneath the covers. You know, the other form of inheritance is across multiple hierarchy, across multiple tables. And that one is rather inefficient because anytime you trip a fault, it's gonna be like, is it in this table? Is it in this table? Is it in this table? So if you do, if you do have to use that type of inheritance, the best way is to always trip, to always model the relationships down to all the sub non-abstract entities.

The views for the database queries, if you can get an efficient one that has the bind, that a lot of the times if you don't need, if you don't need all the bind variables coming through in a view, it can be efficient. The XS back pointers is a really hot ticket one. 'Cause a lot of the times you'll have a situation where You have a user, from a music store example, so you have a user that's got many purchases. So when somebody clicks buy, you're gonna be creating a purchase for them. And so the tendency might be just to say, create a purchase, add object to both sides relationship, to get the user on and save it. Well, if you recall Steve's keynote a few while back, the number one person in the music store, 27,115 songs at that point. That's a whole lot of purchases. And so what happens is when you add an object to both sides of relationship, if that relationship hasn't been faulted, you're gonna trip it, which means you're gonna be pulling in all those things. So all of a sudden you're like, well, geez, why is my app getting slow on random intervals? This buy took three minutes in production. What the heck happened there? Now, not only that, but the memory footprint went through the roof. Well, we just pulled in 27,000 things just because it's trying to purchase one more thing. So if you don't have to trip the relationships, whereas if you just create the purchase, set the user, save it to the database, that's fine. So the back relationships, or you can just not model it. Just remove the modeling in the model completely. So I mean, hope this isn't too advanced. Just trying to cover a bunch of stuff that we've found. Um, That's right. That's a little known technique. Little known database technique.

So, databases are, they're built for these kind of things. And they're, that's what you pay the big bucks for the big tools is that they provide all the tools and if you've got a good DBA you can get in there or you can get in there yourself and look, there's, if you can identify your top queries in your database then you can then start to go back and look to see where they're coming from in your applications. about once a month, our DBAs will send us a spreadsheet and be like, all right, here's the top 10, go for it, kind of thing. So, then we start hunting around, like, okay, where's this one coming from? And, okay, who did this, kind of thing. So, it's very useful.

Yes. So generated SQL is obviously one. We basically focus on optimizing the parts that are getting hit the most. We don't optimize the copyright page if the copyright page is slow. There's... And stored procedures, they're... They're useful for some things, in some places where they're very useful, but other places where you are, if you're using a stored procedure to update rows that you're also modeling, you can really wind up in a state very quickly where you've just executed a stored procedure call, it's updated something underneath, and now your snapshot's out of date. And there are techniques that you can use to keep your snapshots up to date, but it's a pain. So if we can go back over to demo four.

So I'll just show a few bits here. We're doing okay on time. So coming back here, we can see that the list users is inefficient. So then it's, well, what do we do? So the first thing is you check your logs. Okay, nothing in the logs. So, I'll bring up this and I'll show you just a few tricks that we use. And all this stuff is in Project Wonder that we contributed back. So nothing I'm doing here is proprietary or something like that. So the first thing we can start to look at is we can be like, well, geez, let's look at our -- so this is without restarting the app, by the way. So I just -- let's start looking at our database traffic. Let's see what's going on for our little app.

So, list users. All right. Whoa, whole lot of SQL there, huh? So, we can see that we're fetching stuff from the user table, but then all of a sudden we've got user infos all over here. Whole lot of user info columns. So you're like, what's going on here? So has user infos. There's six there. Well, there's more than six queries. Actually, there is six queries. Sorry. So this is-- So let's do one more thing. SQL's not really that useful here. So let's look at fault firing. Let's see when we're actually firing faults. So all that this thing, so let's go here. Yee-haw. So then we can look down here, and so here's main. Here's my list users method. So the setDataSource, so let me just show you the code.

Very, very simple. So I'm just creating a database data source of users, setting the data source on a list page, handing it off. That's all the code that's going on here. And we have one, and services that the other app is using to talk to it. And this is just the plain vanilla.

out of the box, WebObjects handling all the services. So I wrote a bit of code here, I did it myself so that I could This is a user service. And so I exposed the find users method. I used the, if you saw Bob's talk on the first part of web, on the introduction to web objects about, you know, there's this WS make stubs command line app that you can run. So all I did was I wrote a find users and an update user. Takes the user ID, the first name, the last name. And the find user takes a first name and last name. And then in my application, I said, whoa, web service register, register this guy.

And I ran the make stubs on this. It generated for the Cocoa side all these stubs, the WS generated object. And then I just wrapped it in a little bit of user services. So I mean, it took all of a few hours to do. Nothing complex here at all.

Actually, not even a few hours, half an hour. So going back here now, we can see We can see basically the backtrace, and so that's happening on a set data source. This is our fetch specification. So we can see that we're fetching users, no qualifiers, no prefetching keys. Then, next we're fetching user info, So here's main, here's setDataSource, here's fetch. Whoa, what's going on here? Awake from fetch. So you can see I'm just walking up the tree, user line 33. So I say, well, what's going on in user line 33? User line 33, "Oh, if first name is Max and I've got this test user info, then I'm fetching it 10 times." Not so good. So, let me set this to false.

But left it on in production too. So I can clear out the console, go back to the web app now. And list users go back to this guy. And lo and behold, a little bit better. So these are just a few techniques that you can use to kind of quickly get your head around kind of what's going on.

I'll show you one more, which is, So we use Log4j for pretty much everything we do. And one of the nice features that it has is, you can see it, this current one right here, I'm saying my conversion pattern I want to use, I'm putting, I'm saying a date, I wanna have my memory stats. Yeah, so this is used versus free memory.

what category is logging, the line number that it's calling at, the priority level, which, you know, this is all log4j stuff of, you know, priority of, you know, debug or fatal or, I forget what x is, m is message and then a new line. So when we go back and look at, when one of these gets called, let's see, So, sorry, I deleted all my stuff. So we're back here.

So let's go back and look at the first part of this line. Yeah, we have the date. So far this app is using 11 megabytes and it's got 22.95 free. This is being called from the method, from the class, ERX database context delegate, line 149, this is a debug, and then this is the message. So it's printing the stack trace itself. If I turn, trying to think, So if I go back to this, if I turn the fault firing off, because that guy was going ahead and putting its own stack trace in there. So save this, let's go back here.

One more time back to the home page. So list users. So here we have the exact same, all the same information coming in. This is the log4j bridge that's just capturing NSLog events and routing to log4j. And so again, we're getting messages coming through here. Now what happens is then, lo and behold, in production, now for some reason there's something going wrong. You've got some random SQL coming out from this application. So you connect in, you change the pattern to this pattern. So this gives you the web object, so it'll give you the name. It'll give you the number of sessions. It'll give you the WOPORT it's bound on. It'll give you the PID of the process. Date format, give you your VM stats.

Oh, control Z. priority, but then I also put %at at the end, which says, you know, go ahead and dump that backtrace. I want to see where this is coming from. So now when we turn the fire hose on, Get the firehose on, one more time. And lo and behold, lots and lots, oh, update log. Okay, I didn't, I turned it off. So you can see that this does have all the information here. So it has the name of the application, that's the PID, that's the port. So far I've created 10 active sessions, date, memory used, all this kind of stuff, as well as stack trace of where that line message is coming. So these are just a few of the techniques that we use to hunt down performance problems where looking at where the faults are firing, where the database traffic is. And then you can also look at the woe events if you wanna get more fine grained and look at where your components are potentially causing problems. So let's see, back to slides. - Thank you. Thank you, Max. So as you can see, There's a tremendous wealth of debugging opportunities, both offered by WebObjects and also in third-party tools. Everything Max demonstrated is, like you said, available. Project Wonder has a tremendous wealth of stuff in it. Even if you don't want to take on the full project, going there and reading some of the code and learning from there is a wonderful way to learn about some of the optimization opportunities available, as well as a number of other things. So now moving on. Okay, so you've got your database fast. Now we need to start making the actual application fast. There's some tricks to optimizing components. Woe components are a wonderful thing. I mean, they're readily reusable, and you can plug and play and all that, except for plugging and playing woe components has a price. And if you have pages that are getting hit a lot, if you have a deep nesting hierarchy of woe components, you'll find that there's a lot of overhead there, and a lot of times you can reduce the overhead by simplifying the component nesting. kind of unfortunate because it moves away from reuse a little bit, but it will yield a lot of efficiency at times. The next point is, you know, as an object-oriented programmer, you're like, why are you pointing that out? Well, a lot of people when they're doing the component side, they sort of lose sight of the fact that WoE components are really just a hierarchy of objects. So there's no reason why you can't define an abstract superclass for your WoE components to encapsulate the common functionality and then make all the other components in your app a subclass of that.

It's also a really good place to stick in debugging hooks and other annotations that will help you during your analysis phase. And to reiterate, make sure those debugging hooks can be toggled easily as Max demonstrated. You don't want to pay that overhead all the time. There's even, I think, a woe component floating around somewhere that'll embed a stack trace in your page. It's kind of fun. You also want to consider caching your pages or using stateless components, the less state an individual component has, the more it can be reused across the rest of the application and the less cost there is associated with bringing it back into play as the user is moving across the application. And also, make static content static. If you have pages in your application that just aren't changing that often or aren't changing at all, push them out of static web pages and push the session ID into a cookie such that when the user navigates through the static content and then comes back into the site, they get reconnected to their session and see their session state again. You can use direct actions for this too, so it's very easy to embed URLs in that static content that will bring the user back into the application wherever they left or wherever you want them to. And the same thing goes for multiple applications. Using the multiple application approach to optimization, you can use the direct actions again to both navigate the user between different apps while preserving state as well as to control how they enter those individual applications. Also static content doesn't really have to be static. You can play some fun tricks like using proxy servers and things like that to cause content to be generated once dynamically and then cached for the next set of users.

You know, this is another one of those areas that a lot of people forget about. You're spending all your time developing web objects applications, writing code, writing EO models, testing databases, et cetera. You still need to go and optimize for fast browser display. And that, again, means checking the total size of the generated pages. Smaller pages display faster. They parse faster. They render faster. They're just faster. You want to batch display of longer sets of data, as was mentioned on a previous slide. No user out there is probably going to scroll past the first 30 to 50 hits when they're doing a search or a query, so batch it up and don't even fetch those other ones, much less generate the HTML. And again, generate those short orals. Look for a lot of opportunities to do anything you can do to reduce content size is going to make it go across the wire faster. You also want to do better things with images. And by better, I mean use smaller images. compress them more or use an appropriate format for the kind of content that's being presented. You also want to use common image names. So if you have the same image repeated across a site or if you have, say, a graph that's being rendered based on, say, time series data where it only updates every five minutes, generate a static image, put it somewhere, make sure everything uses that same image name. And, of course, use less images. You know, be more intelligent about the use of the image because every image is not only the cost of pushing the bits for the image across the wire, it's also the cost of a whole new decoding session in the browser for dealing with that image and a whole new connection through to the web server. And whatever you do, don't stick the images in the database. It's just not, doesn't make any sense. Web servers are really good at caching binary data and serving it up, caching static data and serving it up. As soon as you stick an image in a database, you go from web server file system, throws the data up the wire to the client, you're done, to web server, woe adapter, dispatch to an application, application calls a database, database pulls the data back, you end up with like five copies of the data in memory and a whole bunch of other overhead. Also, in now, pretty much all the browsers out there support dealing with compressed content. So if you have situations where you're just producing fairly large pages, with modern CPUs, it's almost always more efficient just to go ahead and compress the data on the server side, send it over the wire in the compressed form, and let the browser uncompress it. Apache has modgzip, which is very easy to install.

There's another area of optimization is if you generate crap HTML, it takes the browser longer to figure out what it should do with it. So if you generate well-structured HTML, the browsers render faster. And this is an interesting one because, of course, in the early days of HTML, it didn't matter if you closed your paragraph tags or your table tags or anything else. Because the browser would figure it out, thanks Microsoft. And as things evolved, it not only affects both the HTML processing and parsing because now the browser has to look ahead and then go, oh, well, that tag over there probably means this one over here needs to be closed. But it also confuses things on the WebObjects side.

WebObjects components really want to generate a hierarchy of tags that are nested in an insane fashion. So looking for things like overlap problems, using an HTML tool like WebLint to check the generated content. And you really got to check that generated content because, you know, a web object, a page is generated of many components all spewing forth HTML that then gets serialized and is one big response. And you need to check the content in the context of that whole response. 'Cause there can be overlap problems that are caused by component mis-nestings and things like that. Simplifying table structures is another great way to reduce content size and moving to CSS or having a site-wide common CSS document, CSS being cascading style sheets, which fortunately browsers seem to support though inconsistently, is another great way to both reduce the content size, speed up the rendering time, and make your site more flexible.

So there's also optimizing Direct2. The Direct2 stack is incredibly powerful. The whole notion of having rule-based content generation and data management and navigation management and user management and everything else. I don't know of any other tool out there that compares with WebObjects when it comes to this. But it is also overhead, and there's a different approach to optimizing it. And Max can certainly answer any questions in this regard.

So in the context of Direct2, the rules engine, it has this notion of significant keys and unbounded keys. Significant keys are the ones that are the focal points and then the ones that will be cached, etc. The unbound keys are the ones that will require calculation and a lot of faulting through the rule system to figure out the values of those things. That's very expensive. So you want to avoid that. You also want to optimize the data being accessed by property keys to a given task or page. So the web objects, the direct to stack has this very strong notion that the user is doing something somewhere. And you can optimize all of your data access around that notion. It gives you a lot of hints about what the user is doing at any given time. There's a number of debugging hooks, both in EO and direct to, and also down at the the lower layers, and a lot of which you can find in Project Wonder. And there's warm-up techniques you can do to cause the rule caching system to warm up its state such that subsequent evaluations of those rules will be much more efficient. Like one of the most common complaints we see about direct-to-web or direct-to-Java client is the first hit always takes a long time because it goes off and the rule cache is empty and it has to go off and evaluate all these rules to fill the rule cache. Well, the rule cache, most of it is actually going to be static results, so there's like entire huge sets of rules that just never need to be evaluated again because the results never change. And so you don't want your first user to have to pay that penalty.

And when you're building custom components, and this is true of both DirectTo as well as everything else, go for stateless. Stateless means that there's no session-specific data. It means the component can be shared across the app. It doesn't have to be archived and unarchived and reconstituted during request response. It's just a lot more efficient.

Then, also, you've got to look beyond the WebObjects application itself. Make sure your web server is doing its share of the work. And that means tuning the configuration. Like Apache has a mod status and one other module, which I can't remember the name of anyway, that out of the box can give you a lot of information about what your web server is doing. Plus, look to your web server, especially as your site grows. You'll want to look to your web server to be able to farm out content across multiple web servers, multiple boxes, and even up to the level of farming out to say an Akamai or the other content aggregators, because of course, once you do that, then any hit that doesn't hit your web server is more CPU cycles for the primary content generation.

Offloading all serving the content you can, like images, files, multimedia to other servers is a great thing. One of the challenges is always if you have a site that's secure, as soon as you go into the HTTPS, then that means all the images that are on that page have to be encrypted as well. Because web browsers don't like mixing encrypted and unencrypted content. This gets back to security being the antithesis of efficiency and convenience. So that makes for quite the adventure because now you're going to have to figure out how to pay the price of encrypting the content that's related to that page, including the static content. And of course, encrypted downloads is a really bad idea. Nothing like encrypting, say, 45 megabyte download for one user because everything has to be encrypted per individual user. Caching proxy servers.

This is a really neat technology. You can use something like Squid or the caching proxy server in Apache, such that the first user that hits your site will pay the price of the dynamic generation, but then that HTML page gets stuck in a caching proxy server that's in between the web server front line and your WebOptics application. Once that item is in there, you can then put timeouts on it, or you could have an external interface for invalidating it. Or the easiest thing to do is to just simply have a dynamic page which has the set of URLs that lead to what will be cached, and just change those URLs once it's invalidated. And that way, since it's an URL that hasn't been cached yet, the caching proxy server will go, "Oh, I gotta go get it. "Go get it, cache it, next user will be really fast."

Optimize deployment. Deployment is just such an adventure because it is just different than development. Max is coming on stage because he's done a lot of deployment. How large of a memory footprint can you live with for each of your applications? You've got to make sure that when you get this thing into production and you start hitting high loads, you don't hit the physical memory limit. Physical memory is so much faster than swapping to a hard drive that as soon as your app starts swapping or your server starts swapping, it's done for.

It's spiraling the bowl. You're looking to reboot pretty soon. You want to try for a heterogeneous mixture of applications across servers. So if you're running multiple application types and mix it up a bit, that way if one application goes pathological, I mean, it's going to take out your whole site. but you have some more wiggle room before it does so. And to be honest about it, like if the search component in say iTunes Music Store goes down and it was all on one server, the site's unusable anyway. So having a little component of the site up and running is probably really not that much advantageous if it's gonna cause the site to be unusable.

There's also tuning the adapter timeout values and making sure your wool worker threads settings are all set up correctly. Because as is the case with most things, the generic out-of-the-box configuration is pretty much guaranteed to be wrong for your application. This is also why WebObjects doesn't do synchronization of data between instances. We could do a generic solution for that, but it would be guaranteed that it would be inefficient for your specific business problem.

And you also want to determine ahead of time how you're going to monitor the system for problems. I mean, every component in the system. As you add more machines, as you add more complexities, you add firewalls and everything. These things need to be monitored. And you need to plan ahead for a catastrophe. And Max has got some great anecdotes on that, I'm sure.

And I'm going to turn it over to Max now to talk about this particularly fun, fun issue. Yeah, so we just wanted to finish up with the production quality deadlocks, which if any of you have been in high traffic sites, it's always one of those things that if you've got something, if you've got a recipe in it, it's definitely going to get baked in production, and you're going to find it in production. So just a few topics. One is that kill-minus-quit within the Java world, that will basically be full stack traces for the running app, for all the different threads that are currently running. One of the most common places of is is having initialization things that are happening in your dispatch request.

'Cause dispatch request is completely threaded, or will have multiple threads coming through there at any given point. So even if you have your app set in kind of single threaded mode, it's not doing concurrent requests, your dispatch request has to be threaded. Likewise, any of the code in there, if you have one method there, it's like, oh, let me go out, fetch something from an editing context, cache that value, and that value will then be used for any request that comes in. You can guarantee that when you start, you're gonna have two threads that are immediately gonna get in there and start doing EOF stuff, which is, yeah.

which you will run into serious problems with. Most of the deadlocks we have to track down are because of EOF, or we're not locking things correctly. Or the multiple EOF stacks in a single shared editing context. That one will kill you every single time. So you have to, if you wanna use those full-blown EOs in multiple different EOF stacks, you definitely have to create new shared editing contexts for each one of the stacks that you wanna use. If you, by default, if you don't do anything, just new object store coordinator, new EO editing context, fetch an EO, then you're gonna be dead in the water.

So the monitoring to detect wedged instances, that's a big one if you are having this problem. And also, it's very important to start your load testing before you start your applications. A lot of the times, we ran into several issues where we didn't detect a certain deadlock condition because we had all of our apps up and running, and then we're like, all right, now turn on the load test. Whereas if we would've had the load testing up, which is what you have in production if you bounce your apps, 'cause you constantly have users and they're clicking on everything. And then the app has some initialization deadlock and it starts up and it's like, "Ugh!" And it's wedged. And the dead time interval in the adapters can be a killer. 'Cause if you, 'cause of that interval, that basically says how long should we wait for, if we try an instance and it doesn't respond back, how long should we wait until we try it again? And so you can get this nice ripple effect where the wave will crash down and all your apps will basically register themselves as dead if they're starting up. And so then it will basically wedge all your web servers and then your web servers will finally come back. Your apps will be like, "Now we're ready." And then the web servers will go, "Well, here you go." And the wave will come sweeping over and the apps are like, "No, no, no, no, no more, no more." And so they'll wedge and the dead timeouts will set. And so you get this nice seesaw effect where all of a sudden things will be really fast and things will be really slow, really fast and really slow. So that's a, yeah. So it was just one last slide of these things.

So users are funny because they pay your bills, but they hate you. Because as soon as your app starts misbehaving, how do they respond? By clicking like spastic monkeys. fun. So, you know, quick summary here. So start thinking fast from the beginning, but don't overdo it. I mean, you want to instrument, you want to analyze, you want to track, you want to track your performance over time, but invariably you want to stay calm. And that's like a point that just, again, stay calm. Because when things start going wrong, the worst thing you can do is to start just, you know, throw your hands up in the air and start rebooting things at random without understanding the problem or getting your analysis tools up and running or gathering metrics or gathering evidence because simply doing the spastic monkey routine on the reboot button is not going to fix anything. It's just going to make it happen again sometime later.

There's a tremendous wealth of tools available. It's easy to forget exactly how much stuff is out there, but the industry as a whole has been doing web-based deployments now for more than a decade, and web application deployments now for a decade, too. So there's just a boatload of free and commercial products out there to do a lot of management and analysis, some of which are better than others. Always be aware of that security implication. There's a lot of really obvious optimizations that one can perform on a site that will make it completely insecure. Like the direct actions thing is one to watch out for. You got these direct actions in place. If you carry too much state in that URL, or the URLs, like there was one case where someone decided to separate their shopping cart out from their main application, and when they put the product in the shopping cart, they put the price in the URL, and they believed it. That wasn't good.

Having done this for so long, all of us having done this for so long, there's a tremendous number of community resources. There's Google. There's the Apple and the Omni list. There's, again, Google, which searches a lot of those lists and indexes everything else. There's Project Wonder and other random community projects that are out there, including a wealth of various random free Java projects that you can leverage. And finally, again, Google. If you get an error message that's coming back from something, almost always you can put that error message into quote marks in Google, hit return, and find 10 other people that are experiencing the same thing, one of which might have the answer.

So with that, for more information, that should have just said Google. There's sources of documentation and sample code. The documentation has been updated. I was reminded of one other thing. As far as performance analysis is concerned, Shark and the Chud tools now do Java as well. That works with WebObjects. Thank you.