I've got to say I'm just stunned there are this many people here because I saw the original schedule maybe a month ago and I said, "Oh look, I'm after lunch on Friday and I'm up against the Aqua Feedback Forum. It's just going to be me and a couple of close friends." So this, I'm stunned by this.

So I'm going to be talking about large-scale WebObjects applications and because the content of this is a little more subjective than some of the earlier sessions, I thought it's appropriate for me to let you know who I am. So Steve Meyer, consulting engagement manager from Apple iServices. I've actually done WebObjects and Next Apple related technologies here for just about two weeks short of ten years. The first six years of that I was doing what you'd now call co-contracting. development.

Two of those as a consultant, another four doing shrink wrap. And in the last four years, I've been with Apple doing the WebObjects development. And just recently, I've been working as the consulting manager. And what that means, outside of the business aspect of writing proposals and things like that, is I'm responsible for doing architectural reviews, design reviews at the beginning of projects, midway through projects.

Sometimes when customers have done projects, and they say, "Okay, can we have someone from Apple come in and just make sure we're on the right track before we go too far?" So I think I actually have a fairly good picture of what everybody has as a background for what they want to do.

Introduction. Why is it important that you're here? The thing is that even though we work in a business where we tend to think of, "Okay, everybody's on the web now," it's not even close to true. As a matter of fact, the big money isn't there yet. I mean, the government is just starting this.

State and local governments just starting this. Your conservative businesses, you know, Caterpillar, General Motors. You know, you look at the Dow, and they're the ones who actually have a lot of money and they're used to spending a lot of money. And I can tell you from the perspective of somebody who has done the, you know, small startup thing that consulting to those kind of people is a great way to pay the mortgage while you're writing the next killer app.

The second thing here is that large-scale projects really are different. You can't just say, "Oh, it's twice as hard, twice as many people." The standard problem there has to do with communication. But there's another thing that Ernie kind of barely touched on, which many people don't realize unless they stick around for a while.

And I like to use the analogy of A hill. And if you could go with the picture that everybody kind of uses when they want to indicate steep learning curve, though I want to use this in a completely different context. You've got this, starts out gentle, gets steeper and steeper and steeper. And if you think of that in terms of the complexity of a problem, what you find is that as everything in the tool space improves, you're cutting little steps into that hill.

And what that means is year by year by year, you're making hard problems simple. You know, that top step keeps rising. But there's always a top step. It means that at some point you reach the top, and what you don't realize is that while we're making the harder and harder things easier, we're making the first hard thing harder.

We're losing the middle ground. So what you end up with is a case where as soon as you get past the easy stuff, you're suddenly in the complex and you need all of the, what people say, baggage, but it's not, of large projects. So, what you're going to learn here, mostly, if I do my job, is how does WebObjects fit in a large project? What do you want to do special with WebObjects when you're in a large project? And of course, what I think is most important, what comes out of my experience of reviewing things, is what not to do. How to stay out of trouble.

So, that is a large project. If you have any of these things, you've got a large project. Now, that's kind of misleading because most of those things are related. I mean, no one is going to have a large project team working for a long time producing not much for a few people. So it all does kind of pile together. The one thing that's not on this list, and which kind of ties the whole thing together, big money. When I say a large-scale project, it's not less than a million dollars. Okay, just use that as a number.

It's my own number, but... So, I want to start out with just the general engineering practices. If there's anybody here who's run a large project, is a project manager, doesn't matter if you've ever used WebObjects at all, the next 10 minutes or so is going to be either boring or comforting to you. And for all those people who have never been on part of a large project, this will tell you what it is that is going to be expected of you. So, you always need these practices. And for the most part, WebObjects doesn't impose anything on it.

So, first obviously, project management. And I'm going to try to go through these relatively quickly. Staffing. We're not just talking about the development staffing here. We're not just talking about database admins or network people. We're talking about configuration management, talking about technical documentation, all that stuff. What we as developers typically call overhead. Which That's wrong. Communication. It's another nice way of hiding a dirty word to developers. That means meetings.

And I think it means real meetings in the sense of actually talking to each other face-to-face. Video conferencing can work. Audio conferencing doesn't work. Email, you're just wishing if you think that's going to get done. Scheduling. Not much to say there except that when you have a large project, you look out and you see you don't really have to deliver for nine months.

And it's very easy to just kind of cruise through the first six of those. So scheduling really should be--you need to test things basically a week at a time. If your granularity is any coarser than that, you're not really scheduling. And the last thing that you can't possibly track it.

Okay, source code control. This is kind of a religion with anybody who's been on a large project. You're not serious if you're not controlling your source. And also, it's not just source, although this says that every single document associated with a project, put it under the version control.

If nothing else, you know you've got it all in a repository and somebody's responsible for backing up the repository. You can never lose more than since the last time you checked in. In this case, I actually have a recommendation. I really like CVS. It's the most common thing for our developers to use. It's used internally by Apple. And the biggest advantage for a large project is this: the optimistic locking.

It means that you're not getting in each other's way. Let's see. Show of hands here. Who knows what optimistic locking is? Okay, that's only 25%. So I'm going to go through and explain that. The idea is that when you have a source code control system, a pessimistic system basically says that you can check out a document for viewing, but you can't modify it until you say, you know, I want to lock this thing. And as soon as you've done that, no one else can lock it. They've got it.

Steven Meyer And this presentation is actually a good example of what the problem with that is, is that there's like 45 slides in this. And I turned in the first draft about a week ago. And then I flew back home to Virginia, and I made a bunch of text changes to it. At the same time, we had a graphics professional changing about four of the graphics here. We came back, and since we didn't have any communication, we both changed the thing. control. Let's suppose we put it under a pessimistic system.

What would happen is he would have noticed that I was away. He would have checked it out. I would have been somewhere else not being able to communicate with him. I would have broken the lock anyway. And I'd be off and running. So in a large -- go on.

That's what's going to happen when you have a whole bunch of people who can't necessarily communicate all the time. You just end up breaking pessimistic locks all the time. The optimistic locking scheme is that everybody's allowed to check it out. It's on checking in that everybody has to sync up to make sure that they have the latest version. Now, great example. It turns out that between myself and the graphics guy, we'd only both touched one slide. So I could have put my stuff in with just one conflict and then we could have negotiated over that. So that's why I'm a big fan.

Bug You know, all the software is going to have bugs. You can't track it. You can't fix it. Simple enough. I don't have a specific recommendation here. We use a lot of different things, including some homegrown things. Where it says web access here, what I really mean is remote access. Because, again, if you've got a large team, sooner or later, somebody is going to need to be able to access that thing when they're on vacation, something like that.

The other elements here mostly just have to do with, they're what's good in supporting release management and some other things that will be coming up real soon.

Okay, that's pretty much it for general stuff. Now let's talk about what it is that you need to support WebObjects itself.

First, documentation. Now the project people are going to go, well, documentation is something you need for everything anyway. And the real reason I put it under the WebObjects thing is that WebObjects does have what I would call some special needs for documentation. Only some of these, though. Like requirements. Requirements are pretty much the same everywhere in that from your customers, what you're likely to get is relatively vague, high-level descriptions of functionality. Specifications is something else.

Specifications, what we want for specifications is we want to have actual prototype interfaces of all the screens, and we want a navigation map, and we want functional descriptions of what's supposed to occur between those. Now, the customers aren't necessarily going to give you that. And I'll get back to that in just a bit.

That's what we call a spec. It's not enough to just kind of say, oh, would we like to be an e-commerce system. The design documentation actually is kind of interesting because as we usually do design documentation, a lot of it looks like the specifications. One nice thing you can do is, you know, for each of the design mockups you've got, go ahead and associate with that the, you know, a screenshot of WoeBuilder for what that component looks like. And then you also, you want to have documentation on a component by component basis. You know, a one sheet on each component basically says here's what it is, inputs, outputs, actions, etc. The deployment documentation is for the most part fairly simple.

It's, you know, a lot of it is just a simple document. If you had to install the software from scratch, including really, you know, assume, install the web server, install the apps, install the database, etc. And if the thing crashes, how do you restart it? That's pretty much what it comes down to.

Documents like this, it's really good to have pictures. And I'll be showing you a picture in a little bit, you know, the kind of picture I'm talking about. The other two things, the coding and the user, for the most part, those are pretty standard documentation. They're only included here for completeness.

So, the hardware architecture you should expect to be working to. You start out, this is a large project, therefore, eventually you're going to have multiple web servers, multiple app servers, multiple data sources. Now, that's not the same thing as necessarily saying multiple databases or a single database replicated or whatever. That's more detail than I'm talking about right now. But, that's what we're looking at. And I simplified the picture a little bit. You notice that only web server one is connected to anything, and only really app server one is connected further downstream.

Otherwise, lots of lines all over everywhere. Up in front there, the DNS, that's referring to a DNS round robin that basically masks the fact to the net that you've got multiple web servers. So, they both appear to be the same thing. This is the kind of thing, like I said, you're going to expect to see something like this in your system admin/deployment documentation so that someone knows how to hook something up.

Now I'm actually going to zip by an animation that wasn't supposed to be in here and move on to the software architecture that that hardware architecture locks you into already. And that's actually partly a bad thing and partly a good thing. First thing, you know that because you have different hardware, separate hardware, multiple instances, all that, you start out designing for the problems you're going to have with data caching and synchronization. And I'm guessing that many of you were here yesterday when Dan did his presentation on that. So you should know the issues to look out for.

Also given that, once again, if you've separated, you know you've got multiple instances, you know you've got data synchronization issues, go ahead and break out the application into more than one application. I mean, don't dump all the functionality into one big thing. And typically that's done by, you know, you pick the things that are read-only because you can deal with their synchronization issues.

Because you can deal with their synchronization one way. You pick the things that are, you know, public, can see, that only maybe employees can see, and that only people who are just, you know, administering the database can see. And when you do that, you also get the benefit that you can then separately optimize configurations for each of those.

One thing that's really quite helpful for that is if you've got any particular function, you know, usually this is an admin function, where you're expecting it's going to take a long time, it's going to put a big load on something. You can schedule it so that that app only runs during low-level times.

So you'll never be, you know, hitting your system at the same time your users are. Okay, so you've got an architecture. Let's talk about some issues in application design. I'm not going to teach you how to design an application so much as show you what parts of application design are likely to cause you problems in a big app.

Okay, first, typical code organization looks like this. I mean, and you have a data framework or more than one framework. You've got multiple sources of different types. And it's good to get that off by itself. That'll have your EO model. That'll have your custom classes. You have this fixes framework, which always accumulates. And I'll talk about that in a little bit, but it happens. And then you'll have your architectural frameworks, your reusable components for the most part.

Then you'll have one or more applications. As a result, your team looks like that. You have an architecture team, you got a data modeling team, you got a couple app teams. Now it seems like every time I look at how someone has set up a team, they've piled all the experience people on to the data modeling and architecture.

And there's a real problem with that. Well, there's two real problems, actually. One has to do... One of them has to do with the fact that you're... Always going to be running the risk of having your data modeling and architecture being so sophisticated that your app teams can't use it.

The other problem with this is that, and this has got to kind of come up several times, you've got a maintainability problem there because at any particular point it's your senior people who are most likely to leave. So I always like to make sure that even if, and we're going with team here, and again, teams aren't necessarily very many people. When I talked about a large project way at the beginning, what is a large team? A large team is maybe 15 or 20 people, but the WebObjects part of that, we're talking 3, 4, 5. I mean, that's it.

If you have an app that takes 15 or 20 WebObjects developers, I only know one of those in my personal experience. And that was an effort that was going to replace a system that the federal government currently spends $300 million a year to maintain. So if you got one of those, call me.

Anyway, the other thing about this is you have to make sure that you have planned communication. And I'll get into this pretty shortly about what it is, why the communication is necessary, but the part that it's planned is what's important. Suppose you've got an architecture team. They're building reusable components.

If you have that once a month, you have to make the architecture team stand up and do a PowerPoint presentation on what they've done in the last month so the rest of the teams know what's available. You can't count on them to just, you know, send out messages, "Ooh, we've got this new, cool, reusable component. Come look at it." It doesn't work that way. So plan the communication. Plan the meetings.

Okay, data modeling. We're on the data modeling team. What do the data modeling teams need to watch out for? You've got to make sure that you're modeling the process, not just the static data where you can look at what the company does. You need to be sure how are the applications going to use this data.

And the biggest thing that happens when you don't do that, The next thing: you over-normalize the data. What that means is that you can never ever actually get to a meaningful collection of data to display to a user without having to follow a bunch of relationships or a few database folks do a bunch of table joins.

I've got to say, like, 80% of everything I see that I look at and I don't like the data modeling, that's it. And the nice thing about it is that there's some relatively simple fixes, and you tend to get big performance wins when you do it, and people are impressed. The bad thing is that it shouldn't happen in the first place. And this last one, being cautious with the entity inheritance.

I don't know why it is that it comes out this way, but every time I run into a project where someone has used entity inheritance, there's a problem on the project. There's different problems. But it just seems to be one of those things where, you know, if you see that someone's using entity inheritance, ask them why. And make sure everyone knows it's happening.

Abstraction of data interfaces is a great thing. It's an OO concept from way back. But, as you can see, I have a couple things that I don't like about it. Really the biggest one here is the first one. That if you try to abstract too much away about what your data sources are, you end up losing any of the advantages that the people who designed the data sources built into them. A great example is if you can put an interface on top of EOF, and if you forget to put prefetching of relationships into it, you've blown any chance of optimizing your data access.

Second thing is kind of similar. If you're trying to abstract things that just don't go together, you're going to end up with something that doesn't really work for you. That's actually a general case of the first one. And the last is, whenever you get into the mode of, I'm going to take something and build a system that will work with anything, you always base it on what you know.

And when you come up with something that doesn't match what you abstracted to, there's a really good chance instead of changing your abstraction, you're just going to hack in some code to make it work with just that one source. And then you've defeated the entire purpose of doing the abstraction.

Another thing that's great: Design for reuse. Although, see that never really happens in the real world either. What you have is design for use, redesign for reuse. Or, design for use, recognize you can reuse. So, You tend to want to kind of make a second sweep through the whole app once you've got it done and figure out which bits you want to use next time. But for a lot of people who have done even small apps before, they have with them kind of the collection. These are things that have been useful before. So there actually are, you know, it's not like you're always starting from scratch with reusable components.

And the best thing for, you know, when it comes to reuse with WebObjects is the small reusable components, small elements. And if you want to know a good way to design them, go look at what WoExtensions has as a framework. The other thing that, you know, clearly from a reuse standpoint is your data framework, your enterprise objects.

Basically, if you model the thing properly, you can carry it around, although you may need to modify it for the individual needs of applications, and I'll talk about that too. And lastly, when you do this, organize them into things that don't have dependencies on one another. It's a real shame when you find that there's a really cool table implementation, but it requires that you link in a framework that has somebody's own idea of what a set should be. I've found that a lot of times. Not only frustrating for the people who are trying to use it, but there's almost a guarantee that at some point someone's going to forget to drag along that utilities framework and it's just not going to work.

I get asked about this a lot from a design standpoint. People want to go things be fast. They hear, "Ooh, multi-threading. Let's do it." And, you know, multi-threading is hard. It's really hard. There's a few cases where you really want to use it. And it's mostly in terms of somebody, something that's listening and gets a request and spits the request off to somebody else to process and then starts listening again.

You can think of, you know, the WoW adapter is doing that. But you don't really want to think of your application as doing that. It's actually doing processing. And, like I said, it's really hard to write. It's really hard to debug. You notice I didn't even put test in the middle there.

Because the assumption is you're going to have to debug it. You're not going to get it right. And the cool thing about this is that multi-threaded things fail under load. And they come under load when you load test them. And that's at the end of your development cycle. So you're pretty much--you know, when you go down this road, you're pretty much guaranteed to get a hard to track down bug just as you want to go to production. Even if it's a simple solution, that's quite a lot of stress.

The benefits of multithreading. Tough to really be sure as you're designing you're going to get a chance to use them. Because unless you have the source to everything down the line that you're dealing with, you can't be sure that somebody isn't defeating the whole purpose. Whether it's occurring somewhere in the enterprise object framework, in the adapter for a particular kind of database, in the client libraries for that database, in the connections to the server. There's just too many places where you can spend a lot of time and find out that someone's just serializing it anyway.

However you have to do it, WebObjects does support it. I mean, by default, resources are requested and dealt with, you know, images, that sort of thing, in multiple threads. And you can turn on the request handling and have a ball, but... I warned you. That's all I'm willing to say there. I have an analogy here that I like to use when someone says, "I'd like to go to multithreading for performance." And that is that, think of the grocery store.

You've got a couple of checkout lines, a bunch of people piling up behind it. And what do they do when that happens? They open up another checkout stand, and people go move over to that, and you're good. To us, that's starting up another instance. Now, if they manage to actually fill up the whole store all the time, they open another store a mile down the road. For that, we're talking about bringing up another app server. What you never, ever have anybody do is go to the checkout clerks and say, why don't you process some of the customers in parallel? That's really kind of the end of my feeling on multithreading.

Now, when you're designing things, there are some warning signs. And this is actually--warning is a strong word, but I couldn't come up with one that I really wanted. This is when I go out and I review something, I see one of these things happening. I have three questions.

Do you need to be doing this to get the functionality you want? Is there any other way to do this? And right now, do you have it documented to the point where I could have walked in, not talked to you, and found out the answers to the questions I just asked you? And if the answer is yes to all these things, it's fine.

I'm pretty happy. But if the answer is no to any of these things, then there's work to be done. First of them kind of go together. You're subclassing a lot of Apple classes. You're kind of intertwined with Apple's classes. This is...when you go to training, this is what they show you. This is the ideal. You've got bottom four things are all ours, top thing's yours. And of course that top thing may be multiple frameworks, but it's all up there.

You're not... Really getting in the way of the WebObjects or the Eurocontrol or any of that, that's what I see a lot. Just, vroom, vroom, vroom. Now, I gotta say that this is the kind of thing that the Objective-C developers are much more likely to be able to get away with. So the thing is, Objective-C gives you a much bigger gun to shoot yourself in the foot with.

That's not saying I don't like it. As a language, I like it. But this kind of structure is something that's really common when you've found something that's kind of accumulated bits of Objective-C. And another problem here is those little thin blue lines, chances are they've got dependencies between them, and that violates one of those rules for usability I pointed out. That you can't, even if you decide you really like that one, you know, second blue line there, it's got some functionality you need, you may have to carry along the rest of it. And that's bad.

More warning signs. You've got large application in session classes. Now this is partly a design issue and partly a coding issue. Every once in a while you'll want to sweep through the application in session classes and see what cruft they've accumulated as people just put things in. Because it's a big global variable basically. You can stuff anything you want in there. But even from a design standpoint, application in session classes aren't necessarily supposed to be all that heavy.

You're using Objective-C in a Java project. And this is where I said, I like Objective-C, I like Java. I'm not going to get into the religion wars over that. But if you've chosen Java, and I see Objective-C anywhere in your project, I know you've drawn one of those blue lines.

Similarly, the next two actually are not all that bad. I mean, you're working with the EO Access framework. There's lots of benign reasons to be down at that level, probably more than anything else here. But still, at that point, you're down below the object level, and all I really want to do is make sure you justify your use. The last one is a new one. I only really found out about this in the last two months or so. I think it may be the result of frustrated Objective-C programmers. This is an option they have to do some very strange things while sticking with Java.

When you have custom key-val code, that means that you're doing things with the bindings or you're doing things with the value for key on your enterprise objects. It's a relatively benign solution. The other instance of this is if you're changing your enterprise objects so that they can tell, based on a key value, whether or not you're asking for a, you know, Let's say address from this particular page or that particular page. You know, if you just say put an asterisk on the end of the key, and you've got something that can parse that out.

That's relatively benign. Mostly it's a convenience. It keeps you from having to put methods inside each of those components to go and get the value from the enterprise object and manipulate it as that page wants it. I'm against it, but I'm not going to push it real hard. The thing is that this opens up. It actually does, and I kind of hate to say this because I'm telling you, don't build the bomb this way. You can put anything in bindings, and if you're overriding the take value for binding, you can do anything with it.

You can very easily just put a binding in and it says as long as the binding starts with three exclamation points, that means that the value that comes in is a block of Perl code, and I want you to send that out to the Perl interpreter and take that value.

And that actually I've seen once, not with Pearl, but with something else. And that's exactly what I mean by, you know, "Oh my God." That sort of thing is really hard to document, too, because WAD files are not where you're looking for code. And for, as we move more and more into, The broad developer community.

More people are going to be looking at this stuff in WebObjects Builder. And it's really hard to tell just as you're zipping down through the bindings. You know, this has an asterisk on the end. That means if it's a number coming in, we're going to divide it by two before we use it.

So, you know, I don't like that a lot. I think that there's a real possibility for abuse right now on that. So let's assume you've managed to get through design without any major efforts. You're on to prototyping. You've got to divide the prototyping into two things. This is a customer issue. The user interface prototyping.

Now, if you remember, way back at the beginning, I told you what a spec is. And user interface prototyping is writing the spec. And you have to have the customer know that. Make them understand that until you've locked down like 90% of the user interface to a level of detail that you've got screenshots, they haven't yet specified the app.

And all the work that you're doing up to that point may not be final. Technical thing. That's where you're trying to decide whether or not you can actually hit that airline reservation system, whether you can integrate the credit card processing with the Oracle database, that sort of thing. And you want to make the interface for that as ugly as you can.

You don't want to confuse the customer into thinking there's any possibility that the code you're writing now is anything but demonstrating that you can go do it. I like to say, when you're doing the demos, you want to demo the user interface on a Mac and demo the technical stuff on a PC. And if you do that, this throw-it-away thing works on its own.

When to construction. Here I'm mostly talking to the developer. Assume the project's going to continue after you're gone. This is a long project. It's got to be at least six months. And you've got to be responsible. and I have some things to say about responsibility in a bit. Overall though, the construction of a WebObjects application is no different than the rest of development. There's not much I can say about WebObjects technology-specific things as you're writing the code.

Two things, though, that are really helpful, and unfortunately I don't see enough of standardized. This idea of standardizing keywords in your comments, and I'll have an example on this. It's really helpful to come up with a list of keywords that someone can search through the code for, and that list of keywords ought to be put in your repository. You can hear that again. Document, document, document. Okay? How about that? Anybody seen that kind of comment before? Interesting thing about that.

The first, the first instance, that's actually two comments. I mean, separated by an indefinite amount of time. And the first, the first comment there I sort of made up, although I've got, I've seen something close to it. The frightening thing, the second comment I lifted verbatim from an app that's in production. All I did was change the three initials there at the end to my own, which may not have been all that necessary because when I asked around, nobody could tell me who that was.

That's what I want to see. And you don't have to add every line like that. But when you're doing something like, you know, clearly like that, that first thing there, it's a line of nothing but keywords. It's great to have, you know, fix me is a really common one.

I think we all use that occasionally. But if you standardize and say, yes, indeed, this is what we're going to use when we've got something that needs to be looked at. If you can put keywords for the dependencies of every version of things that you need to work with, that's really helpful. You notice you've got enough text there that no one's likely to go around and put a, hmm, line after it.

And I put my full name and a date. So at least, you know, people may not remember who I am even with a full name. But there's a date there so they have some clue as to when it was an issue. And the other nice thing actually about doing the whole, the keywords for versions of stuff, this is a 3-5 problem in theory. In practice, I've completely made this up.

What you need to do is every time you upgrade to 4, somebody searches through for all the 3-5 keywords, verifies whether or not these comments are right. And changes the 3-5 to 4, which means you've never got any issues that are more than one version old of any of your products. And you'll really like that. Trust me.

More construction things. At this point, the developers stopped wanting to go to meetings. Really, the meetings have a purpose. It's up to your management from a development standpoint, it's up to whoever it is you interface to the rest of the team to make sure that you're not going to the meetings you don't need to go to. Steven Meyer And one of the things that I hear every once in a while, and people try to convince me this is a bad thing, 12 of us trooped into a room, five minutes later we walked out again. What a waste of time. No, that's perfect.

If it took you an hour, that's bad. But to walk into a room, walk out in five minutes, that means the system's working. It means everybody showed up like they were supposed to. There's no open questions from people who didn't show up, and everything's on track. If you were to take a look at it, you'd find you'd take more time trying to coordinate among those 12 people whether or not they needed to have a meeting than to just have them all show up.

And this last item, clever is bad. You know, I tried to--I thought about making this less bold. And I talked to other people in my team about this concept and, you know, I thought that, you know, maybe it's just me. And after talking to them, I decided it deserves its own slide. Okay? Clever is bad. And keep in mind, this is clever in terms of coding. This is not intelligent design. This is clever coding.

And I have an example for you. I think probably the cleverest I've ever been, which is to say the grossest violation of this principle. Go back to 1992, I think it was. I was doing what was--what's now Cocoa Development. And I really wanted a print panel to work differently.

So, I went into the debugger, I disassembled the appropriate part of the print panel code, I examined that and figured out the memory locations of the private instance variables of it, and then with a little pointer arithmetic and a little guessing, I managed to get it so when I wanted to, I could stuff data into those places in memory that the print panel would go look to see what its state was, and I could change it. And it worked.

Uh, you know... This was code that I, in theory, owned. I was going to maintain. Nobody was paying me to do this. It was mine. This wasn't something I was leaving behind for somebody at my expense. Although, I'm standing here, and I've got an Apple badge on when I have a badge, which means I'm not maintaining it. I think I know who originally did have to maintain that, and I liked him.

Even though there's maybe 20 lines of code and I had a page and a half of documentation on that whole thing, when I look back on it, I was much prouder of that achievement then than I am now. And although it's hard for you to think forward and imagine how am I going to feel about that 10 years from now, it's a pretty good guideline.

How would you feel if a friend had to maintain this? Now, the three items there, I mean, why is it bad? Outside of, you know, my story. Basically, I had an even bolder version of this slide and I decided to tone it down because probably without my words it would seem a little too harsh. But first you can fool yourself. And by that I mean a couple of things. But you can have a problem with, you may not actually understand everything you're doing.

You know, I think the print panel worked. It worked in all the cases I tested. But, I mean, kind of wandering through the assembly code is not necessarily the best way to be sure. If I had to go back to that now, I mean, admittedly that's eight years ago.

But, I mean, if I had to go back to maybe even six months later, I may not know any idea what it was I was doing. And that kind of follows on. You can fool your team two different ways. You can fool them in coming up with something that's so clever they have no idea how to use it.

or you can fool them because six months later you're gone. You're the new CTO at showmethemoney.com. They have to try to maintain it. And last, this is something that people don't think of. You can fool us, and by this I mean the engineering team at Apple, because everybody knows there's no guarantee if you use private API. And of course, the print panel thing, I knew there was no guarantee.

But what you don't really get is that there's not really a guarantee if you use public API in a way that nobody ever thought you could. If you're depending upon the order in which certain things happen to get called, so you can stuff things into a request here and pick up the fact that those values are in the response over there, you run a risk.

The scariest thing that you can hear from a project management standpoint when someone's showing off something cool is, "Ooh, I bet Apple didn't know you could do that." Okay, so you got the app built, maybe. Testing. You need to have a real separate environment for this. You need to get your customers to duplicate the production as much as you possibly can.

Testing web apps is a boring thing to do. Use automated tools for both the regression testing and for load testing. Although it's occasionally, it's kind of neat to just get the whole, you know, accounting department to go at 4:00, come and hit the app. You've got to give them a break in their day.

You use the humans for functional testing, but the other thing you want to do is you want to have the occasional human stopping in doing your load testing because you're going to gather stats both from the inside of the app in terms of, you know, the woe stats things and from the outside in terms of what the overall, you know, what a browser would see in terms of response time. What you don't get is the human, "I thought this page would go quicker." So, while you're doing the load testing, just have a human sitting by doing some stuff and letting them know how they feel about it. And we're back to meetings.

Schedule the regular bug tracking meetings and make people show up. Because, once again, the developers are very busy trying to do things other than fix bugs that someone mentioned a week ago. So, make sure that... The right people attend each of those meetings, not everybody, but that you have them to really, by the time you're down to the testing phase, it's got to be daily.

Performance. Now it works. How fast can it go? I'm cheating here. There were like three different sessions you should have gone to if you wanted to know about performance. These are the questions everybody asks though. And the real point of this actually is not so much what there is to say here, it's where I'm saying it.

If you haven't gotten to this far, you can't answer those questions. And too often, I run into a case when I'm doing proposal type things that someone's going to say, "Well, I realize I haven't told you what the app does yet, but how many boxes do I need?" You can't do it.

I mean, I have tried to come up with heuristics for this sort of thing, to just say, well, if you happen to have, you know, a gigahertz Pentium machine running as your app server, then assume that you can support, you know, four transactions a second as long as no response is longer than X. Doesn't work. You really do need that because you're either CPU-bound or you're going to be memory-bound on these things, and you don't know which until you know how much memory you're going to have. So, never answer these questions until after you've started testing.

That given, assuming it's slow, what do you want to do about it? First bullet item there, I guess some people are going to say, "Well, that's obvious." But it isn't, because it's not just what's most commonly slow. It's what's really slow that kills you from a user perspective.

Let's say you have a whole bunch of pages that come back in two seconds and one page that comes back in ten. Now if that thing shows up ten percent of the time, you can do the math and you end up with a number that's three seconds or so for your response time within the app. That's not what the user sees at all.

And for that I'm going to go back to the whole grocery store thing, which is you've got your fifteen items or less and you're standing behind somebody with a hundred items. And behind you is a bunch of people who also have two, three items. Now from the checkout clerk's perspective, one person took a lot of time, other people just zipped right through.

You're back there fuming. So the approach that grocery stores take is they have 15 items or less. And I actually kind of advocate the other side of this, which is you should have lines for people with lots and lots of things and make them stand behind each other.

And it sounds like a joke, but it actually, it goes back to this, as well. I mean, as app design, it's a good thing sometimes to recognize when an operation is going to be expensive and shunt it off to the app server just handling the expensive operations. And this is where the human factor comes in, is people usually recognize that, you know, this is going to take a long time. I want to find every hotel in Orlando.

That's great. However, for the people who think they're doing something that's really short and they end up behind somebody who's looking for every hotel in Orlando, they're upset. So 90% of the time, when you're looking to improve the performance, it's data access. So you may look at modifying the data model.

Or you may look at customizing versions of the data model for each of your apps. Because the ReadWrite app may have a different need than the ReadOnly. And a lot of people don't think in terms of that because you've got your data framework. Well, you don't have to have just one data framework. You can have them optimized for the uses that they're going to have. So think about that. And lastly, manually fetching the data you need is something that we almost always resort to in one case or another.

And when I say manually, it's not all that manual. What we're really saying here is rather than configure a WoW display group with a data source to go and get data, go ahead and create a fetch spec yourself. Fetch objects with it and pass the array over to the display group. And what that can do is you can then set things like the prefetching at a much finer granularity for what's being used at the time.

A little bit more on that data access. This is something I see a little bit too much of. Don't have complicated bindings that just keep dotting along. Because in the middle there, you're not really sure, from a coding perspective, how you get from manager to spouse up to car. That isn't even necessarily all in one data source.

To find out what the spouse's car is, you may have to go up to some mainframe system. It's much better in this case to go ahead, figure out the relationships that you need, fetch the data for that, and just let — you know, if you put that thing inside a repetition, you'd be here forever.

So, Anticipate the needs of your interface elements and try to provide them with their data ahead of time so that they don't have to do it themselves. They're not smart about data access that way. And the last thing here, you'd think you'd never have this, but you do, is that customers will ask for, "I'd like the whole database, please." And sure, they understand that the fetch takes forever, and it's making the app go slow, but they don't realize that they're then transferring like 10 megabytes of data to their browser. And they wonder why their network is slow.

Why is my email not getting it every time someone tries to run the app? So, Limit this stuff. I mean, at the very least, go ahead and put it in a display group's turn on the batching. And if nothing else, that's going to show them when they see they've got a zillion pages of batches, what a silly idea it is to try to fetch all this data at once.

Okay, you've gotten to deploying. You need a deployment team. Same thing. Not necessarily a whole big team. Could be a small team. Could be a guy. Cool thing about this is if you go back to what I was saying about the builds, about the releases, about the testing, you've got an environment that's just like it that you're going to go to production on and your deployment team gets to practice every single time you put out a test build. So they really know what they're doing by that time. There shouldn't be any last minute, "Do we know Solaris?" kind of questions.

And gather as many stats as you can while you actually got this thing running. But you don't want to impact performance. I don't want to see a bunch of things logged out to the screen to let you know what's going on. I'm mostly here talking about, you know, you obviously get the woe statistics. But also, what you want to do is check the CPU loads on all the machines, web server, app server, data servers, etc. Check the memory on all these things.

I mean, it doesn't cost that much. Just go ahead and check these--checkpoint these things once an hour forever. Because if you ever need to know what happened, you've got knowledge. And, you know, knowledge is power. Last thing there, the application is not finished when you deploy. That's a function of the kind of customers we're talking about here and the kind of apps we're talking about.

You can expect a feature is going to not make it into version one. The only way it's not going to be later than it is anyway. Therefore, there's always a version 1.1. And from a management perspective, that means you need to be thinking about who your maintenance team is ahead of time. And you need to figure out not just your developers. Your maintenance team needs some of the same overhead as the rest. Some of this stuff can be shared. Some of it can't.

This is a sad thing for the management types. Probably a good thing for the development types. I didn't put this on the slide here. The biggest issue that I run into with deploying a big app "Attrition. This is the point when everyone leaves. And really when you're using something like WebObjects, you've got smart people and a high-demand technology.

At any moment, you have to be prepared to have somebody walk in and say, 'I'm gone in two weeks.' When you deploy, there's a real good chance you're going to lose like 40% of your folks in the next month, because there's a lot of people who want to see that make it. And then that's when they'll move on. And if you're not prepared for that, it's devastating.

Hey, what went wrong? This is actually not even my slide, essentially. This is a known thing within software development. If you never manage to get it done at all, it's management's fault. No doubt about it. No matter how bad your engineering might be, it's management's fault. They planned poorly. They didn't follow their plan. They picked the wrong people. They used Windows, whatever.

And the second half of that really is if you're stuck at version one, then your engineers did a lousy job. What's unsaid here, but I think we all understand, is that when they're not within earshot, it's the customer's fault. Really. But I'd like to switch that around, because despite all the things that are in the way, actually, Most of these apps actually do field, and they're successful. So let's think, what went right? Who gets the credit? And I got to say, I don't care how good your engineering is.

If you actually deploy version one, management gets the credit. And when I say that, I don't mean that deploying version one means you got all the features in, or even that you got it all in on time. It means that Things went wrong and the customer didn't fire you and the team didn't quit and you got there.

The second thing there is, you know, credit the engineering when the customer comes back and says, "I know I only wanted one language before. Now it needs four languages. I know I was going to use this service to validate credit cards, and it will have a response time of maybe five seconds with 100% reliability, but I can get a lot cheaper thing from somebody who will respond in two seconds with 75% reliability." And that's actually an example from my own experience. If you get that far, great. At that point, you want to congratulate the engineering folks. You don't want to congratulate them too early, though. That's when they leave.

And to be honest, to be fair, I want to be fair, also not on this bullet item is if it works, particularly if they happen to be an earshot, credit the customer. Never hurts, and it's actually true. A lot of times when I talk to engineering teams about their experiences, you know, how's it going, I hear, "My customer doesn't understand the problem. My customer can't make a decision." And the ever popular, "My customer is stupid." And see, it's not true.

The customer is not stupid. The customer has different priorities than you and doesn't necessarily have all the information to make a decision. So if I can apologize to folks who maybe missed lunch and are thinking that the candy is really a good idea right now, I'd like to go back to the grocery store just one more time as an analogy. I mean, if you've got an analogy, you've got a metaphor, just pound it into the ground.

Follow your customers sometime to the grocery store and think in terms of, you know, they're heading down the cereal aisle and they stop by the Frosted Flakes. And if you're going to have a, you know, a metropolitan area, chances are they've got choices there. They're going to have the Kellogg's Frosted Flakes, you know, the national brand.

They're going to have at least, you know, maybe a regional brand, but certainly a store brand of it. And, you know, they may pick up the boxes, they'll look at the nutritional information, something like that. But, you know, they'll put them down and then they've got a choice to face. You know, okay, the national brand comes with QuickTime, which is cool.

The local brand? It's cheaper. And what happens? Sometime within like 10 seconds, they pick up a box, put it in the basket, they move on. Nobody ever complains that the local brand doesn't have QuickTime. Nobody ever complains the Frosted Flakes are too expensive compared to the local brand. So when you're doing Those kinds of things with your customer, when you're trying to tell them bad news, schedule slippage, whatever, what you need to do is very simple.

You lay out the options for them. You give them the consequences of each of the options they have. You make it very clear, "We can do this your way, but there's a good chance if we lose this guy over here six months from now, we'll have to pull that feature because we can't maintain it." Give the customer that kind of information, the customer will make a decision. Once they've made the decision, document the decision. Put it in your repository.

So, in summary, big projects are hard. They are. But that's all big projects. That's not just WebObjects projects. Basically, if you've got anybody who's ever done a hard project in any other technology, they can manage a big project in WebObjects. And WebObjects helps because, you know, we're always cutting steps. So what is a hard project now and is a big project and requires a big team this year may not next year.

And last thing is, I mean, it's not really everybody is suited for the big projects. I mean, some people just, uh... They want to be too creative. They don't like the structure. And that's fine. There's lots of places for that. But, you know, if you can stick it out in there, it's--there's really an immense sense of pride in having done something like that. It's very much like building the tall building. And what you'll find is that no matter how many little projects you might do, it's always the big projects that you remember. And incidentally, it's always the big projects you put on your resume.

So, because we are so close to the end of this, there's like nothing else you can go to. If you haven't learned by now, all you've really got is the feedback forum to complain about the last 17 sessions. Most of the For More information you must have seen before, and the WebObjects lab is closed, so sorry about that. You must have seen these names by now, Tony and Ernest. If you got any questions about all this stuff, even something that's specifically for me, go ahead and send it to Ernie. Ernie will route it to me. And now we're on to Q&A.