Good afternoon. Thank you for coming today. So we're here to talk about graphics and media, graphics and media in Tiger. And for the next hour, Peter Graffagnino and I are going to give you an update on some of the technologies that we built into Tiger. Of course, there's a lot of stuff in Tiger, and we've only got an hour to talk, so we're just going to cover some of the highlights.

I want to start by reviewing just some of the simple view of the goals we set for graphics and media in Tiger, and really in Mac OS X overall. At the very highest level, our goal was to build the richest, most powerful set of graphics and media capabilities ever found in a computer operating system. We set out to do this about five years ago now.

So what does that involve? There are several key elements that we knew we had to build into the system. The first element was a professional-grade audio subsystem. So much of what we do these days involves audio, and that's everything from professional-type applications like Logic, all the way down to the consumer-oriented applications, and we knew we needed to have very robust technology for audio.

In addition, 2D graphics is clearly important. Macintosh has a long legacy of leadership in this area, and we wanted to make sure that that would continue in Mac OS X, and clearly we've done a great job there, and that's a very important part of the story for graphics and media.

The third element is 3D graphics. Now, of course, 3D graphics are absolutely critical for games, but increasingly, this technology is becoming vital for a lot of other areas, including video. And in Mac OS X, this is a very important part of our overall technology strategy for the UI. There's a lot of things we can do, leveraging the GPU and other aspects of 3D stack that are going to allow us to do amazing things in the operating system to date and going forward. The next major area is our imaging technologies. Now, these are the technologies that allow us to connect to cameras and printers and scanners. And increasingly, we're leveraging these technologies to be able to perform some incredible image processing.

And they're going to be very important in the overall system. Last but not least, of course, is video. Apple has a really strong position in video, and our goal is to ensure that we hold on to that and keep pushing that forward. There's a lot taking place. Now, these are sort of the right raw ingredients for an operating system, but this is not all that matters. The next thing we have to be concerned about is the speed of the individual technologies. Um... Excuse me, I screwed up.

These technologies have to be blindingly fast. And there's a few things we have to worry about there. The first one is we know we need to have the right architecture. Speed is, in large part, a function of architecture. It also is important that we have the right algorithms. And then lastly, we know that we need to have great implementations. And this is one of the things that you can count on us to do for you going forward.

This is a very important part of our service for all of you. The second element has to do with integration. Of course, we use these technologies independently. But increasingly, there's incredibly interesting things that can be done by combining the technologies together. And we want to make sure that we're providing an architecture that makes this easy and also provides high performance when you bring the technology together. And we have several things that we're going to show you today to demonstrate some of the progress we've made in this area.

The next important consideration-- -- is ease of use. The operating systems today are incredibly complicated. It's hard to realize this when you think of them sitting inside these little laptops, but a modern operating system has literally tens of thousands of APIs, hundreds, perhaps thousands, of distinct frameworks. The challenge for all of us is how to figure out how to plumb the depths of these things and get the functionality out and be able to wrangle it and use it in our applications.

This is something that we feel very strongly about. Ease of use and simplicity is not just for beginners, but experts can derive tremendous benefits with powerful tools that help them manage the complexity of their systems, and we'll talk more about that later. The last thing I want to talk about is the future.

Of course, we know that things are always changing in the world of technology, and today we announced a very interesting and very significant change in the core technology on which all your system, all your software, is eventually going to be based. One of the things that we wanted to do in our operating system is to be able to insulate you from as many of these disruptive technologies as we can.

And there's a second part of it, which is that if you can rely on the APIs that we provide, what we'll be able to do in the future is we'll be able to provide you tremendous benefits of performance and functionality in the future as the technology continues to evolve. So this is sort of the high-level goal that we set for the graphics and media team several years ago, and this is something we've been executing on for a long time now.

So the next thing I want to do is I want to start to go through the technologies in Tiger. We've made a lot of progress over the last year. We introduced most of these technologies to you here last year at WWDC. Last year we were just beginning, and now we're done. Things are out in the market, and we're really pleased with what's occurred, and we want to tell you about some of the things that we were able to accomplish.

The first topic I want to go into is QuickTime. Now, QuickTime is Apple's kind of granddaddy API for multimedia, digital media, and time-based content. This has been around since 1991. It's used by content creators and developers to create, manipulate, and publish digital media in a variety of different formats.

It's also -- it's built into the foundation of the operating system, but it's also used by virtually all of Apple's professional and consumer applications. But this technology is not just important. This is also very important to Apple. This is also very important to our third-party developer community. In fact, today, Apple licenses QuickTime for redistribution with software tools and titles, and every year we have more than 4,000 new titles that are licensed to distribute QuickTime with their products. So this has tremendous reach.

Now, because Apple provides QuickTime for Windows, we're able to go far beyond the Macintosh, and as you heard Steve mention today, our estimates are that since the beginning of QuickTime, we've been able to distribute more than a billion copies of QuickTime. So you can see that it reached very, very far.

In fact, it goes beyond the PC, because Apple has licensed the QuickTime file format for use in a variety of non-PC devices, and we've been able to incorporate the QuickTime format into the base of a number of industry standards. We find that QuickTime technology, is going way, way beyond the PC. This is very exciting, and I want you to know that the state of affairs with QuickTime today is it's very robust technology.

had the headaches, so you don't have to. This is probably the only multimedia system that you can find out there in the mainstream that has the ability to work with this kind of media in the way that you can do with QuickTime. It's really quite remarkable. You can learn a lot more about 264 and how it integrates into QuickTime in session 208 on Wednesday.

So the next area where we took a whole fresh look at QuickTime is in the area of audio. QuickTime and Quick-- the audio support in QuickTime 7 is now built on top of core audio. And this means we inherit a tremendous number of new capabilities that we haven't had in the past. And then combined with QuickTime's traditional capabilities, we're able to do all kinds of new things. And very simply, we've got rich support for a wide variety of surround formats.

We can go up to 192 kilohertz sample rates and beyond. We're able to support all the known sample formats, including floating point sampling formats. And we provide better accuracy-- synchronization accuracy-- than we've ever had before. You can learn more about the audio capabilities in session 201 on Tuesday.

The next big area that we wanted to tackle was sort of themed video on a texture. As you know, the GPU is a real important emerging development that's having dramatic impact on the graphics community, but it's also having dramatic impact on the way that we think about video.

QuickTime, up until QuickTime 7, had an assumption built into its core about the nature of video decompression and display. And the basic assumption was that the two ideas-- picture decode and picture display-- were an atomic unit. This was an intelligent decision, the right decision when we first made it, because it allowed us to ensure optimal transfer of the media through the subsystem at a time when memory bandwidth was at a premium.

But for those of you who have tried to take video and put it on a texture and have the video subsystem still work great, you found this is a very difficult task. This is difficult for us. I'm sure it was very difficult for you. And we set out in QuickTime 7 to crack this nut and solve it once and for all.

How did we do that? We introduced a new technology into the system called Core Video. This technology is underneath QuickTime now, and it allows us to split the picture decode and the picture display into two distinct phases. Core Video provides a very high performance service for timing and buffering and ultimately blitting of data to the display.

And then QuickTime is responsible for managing the decode and the scheduling of media to the point of the decode. And so what happens here is by making this split in the architecture, we've now created a wonderful opening for things like Core Image and OpenGL to be brought to bear on the video frames before they've actually reached the screen.

And we're going to show you some more about that in just a second. You can learn about how to use this in your own applications in Session 207 on Wednesday. So without any further ado, let me come over to the demo machine and show you a little bit about what you can do with this technology.

So I have a very simple application here that basically lets me bring up a clip. It's just a movie. And, uh, with this clip, what I want to show you first of all is that this is basically an application that's causing video to play, play, um, play the movie to a texture and then the texture is being displayed on the screen. And what I can do with this application is I can take, um, well, let me explain one other thing. So, um, for years people in the print publishing world have utilized color sync to do color managed, uh, workflows for print publishing.

This is something that's very important in professional workflows but has never really been available in a, uh, off the shelf sort of, uh, easy to download sort of a system in the past. What we are enabling with core video and QuickTime 7 is the ability to start to apply color sync style profiles, ICC profiles to video.

The color sync operations are hardware accelerated so that they can be utilized in parallel. They don't have any performance impacts. So, uh, I'm going to show you a little bit of a sample of this. So, I'm going to show you a little bit of a sample of this.

So, I'm going to show you a little bit of a sample And it's really as simple as dragging the profile on, it's applied the video. Here's another one which is sort of a typical sort of a popular kind of film effect, sort of desaturates the video and then increases the contrast. So that's pretty cool, and that's now possible with QuickTime 7.

Now, the next thing that we did was just sort of gratuitous and cute, but I want to show you this, because it gives you a little bit better sense of the kind of the power that's underneath this whole system. This is an application that lets me basically turn my video into a jigsaw puzzle.

So, I've got these little pieces, and if I can find the corners, I can maybe put the puzzle back together, and I don't think I'm doing so well. Well, the good news is I have a little feature here that helps me make this happen for you in real time.

Now... This is pretty cool. Let me tell you a little bit more about how this one works. So the next thing I'm going to do is I'm going to move, I can go to more pieces. We're now up to about, what are we up to here? 60 pieces. With this, if you look closely, what you can see is every piece has a little shadow on it. Of course, the pieces all have these rounded edges, and then there's the video playing behind it.

So what's happening is QuickTime is decoding to a texture, and then using Core Graphics, we've created a series of masks. I'm using Quartz 2D. We've created a series of masks, and then the video is being drawn through the mask, and the shadows on the puzzle pieces are being computed using Core Image. So you've got all these technologies working together, and it's really quite remarkable.

Obviously, this would be a very difficult thing to do without these technologies. And again, you can apply the auto-solve to get your groovy, funky animation. And you know, you can really take this thing to the extreme. There's a few hundred pieces here. We've got almost 500 puzzle pieces. And again, the movie's playing just fine. It's just working great. So that's an incredibly powerful new development in QuickTime 7.

Thank you. So, you know, one of the things that we learned talking with you over the last couple of years is that more and more of you are using Cocoa to write your applications. Many of you have really learned about the dramatic benefits that Cocoa can bring to your application development. Well, we wanted to make sure that QuickTime 7 was going to be a great client for you if you're writing in Cocoa. And so we told you about this last year, and we're done with it now.

We've built a wonderful toolkit that gives you very simple, powerful access to QuickTime through a set of Cocoa APIs. It speaks the language that Cocoa programmers speak. There are no sort of foreign data types and strange API conventions that you have to learn now. And it's very powerful.

In fact, we wanted to make sure it was going to be powerful enough to support an industrial-grade application. So we said we should be using this ourselves. So one of the other things we did in QuickTime 7 was we went back and we said, let's rewrite QuickTime Player, and let's write it on top of QtKit. And with QtKit, we'll automatically inherit a whole bunch of nice UI characteristics that come with every Cocoa application.

And because we knew we had to be able to support high performance playback, I can promise you that the QtKit framework is able to give you full access to the high performance application. And that's what we're doing. We're going to be using the high performance parts of the audio and video playback systems. There's no compromise here when it comes to performance. So in addition, we've been able to implement a whole bunch of new features because we've got an easy-to-use programming API. We added some simple movie recording. We added movie sharing functionalities. There's a whole new set of AV controls.

A very important feature for many of you is now when you're doing exports, you can get multiple exports running at the same time. And you can still use the same API. And that's what we're doing. And we're going to be using the same application for all of the applications in the system. And it works great simultaneously. And it's actually quite thread process friendly. So it'll back off when you're doing other things that are important to you.

Very, very cool feature. And then last of all, there's a new groovy full screen control and display capability that we're pretty excited about. Now this is a great testament to QtKit, but I don't want you to take my word for it. I'd like to bring a good friend here. He's a great friend. He's a great friend. He's a great friend. And we're going to get a good friend of ours up to the stage to tell you a little bit about what he's been doing. This is Michael Johnson from Pixar.

I was here about three years ago and gave a lunchtime talk where I showed you guys some sort of behind-the-scenes stuff about how we make movies at Pixar. And at the end of it there, I showed a little something I'd put together, and it was a little program that I wrote for Pete Docter, and it's what we refer to as being developed on a dare.

Pete actually asked for a program that he could just kind of tap out the timing of storyboards before he would go into Final Cut Pro, which is a much more complicated program than he wanted for that part of his process. So I went off and I wrote a little something with him. We kind of iterated on it for about two weeks, and every day we sort of put some new features in there.

And he really liked it, and we ended up actually using it for a whole bunch of things. We ended up throwing another engineer, Josh Annen, on with me, and we spent about six months doing a full-blown real production app inside of Pixar. And we have lots of happy users for this, and most of them are on movies I can't talk about.

And there's one that I can talk about, which it was actually used by Mark Andrews, unbeknownst to us, the engineers. Mark Andrews was trying to work on the final battle sequence. He's the head of story on The Incredibles, and he was having some troubles pinning this stuff up.

And he heard about this app that we had. He scanned his pictures in, he timed it out, he pitched it to Brad Bird, and it went really well for him. And so I asked Mark to come today, but unfortunately he was busy, so I have kind of the next best thing.

So Violet comes running in with the remote. "Mom, Mom, I got the remote!" She turns visible again. Here comes the robot. "A remote, a remote for what?" "Violet, I don't know!" She starts pressing buttons. Click, click, click, click. "To control the robot?" Click, click, "Why don't you come on thing?" Comes, steps over camera. Presses the button. Click.

Rocket Boosters, the robot, control the robot, takes off flying, crashes into a building. Bob's still stuck in the claw. Maybe that wasn't the right one. Here it comes. Gets up, starts charging him. Bob's still stuck. Dash it all. Here it comes. Give me that thing. You don't know how to use it. Give me that thing. Click. Presses a button. Click. In the background. Fwing.

Bob gets tossed up in the air. Oh, my God. He's free. That wasn't right. Bob grabs it. Give me that. Bob gets up off the ground and sees the robot's coming. Boom, boom. Closer to his family. Boom, boom. Nothing can stop that thing. The only thing that can stop it is itself.

So like I said, that's a little Cocoa app. We wrote it three years ago, and it was very successful. It was kind of too successful. And my group, we're really busy. We do a lot of support for sort of the story and editorial part of the pipeline. We have other things to do. We can't sort of just keep tweaking these things.

But the requests kind of kept piling up, and they wanted things like they wanted to do longer pitches, they wanted frame-accurate pitches, they wanted sound attached to the boards, they wanted faster load times. And then what's interesting is something we hadn't thought about, which is as the movie gets made, we go out of the storyboard phase, and we go into the previs phase, we go into a little modeling, then we have some animation and lighting. And they wanted this ability to actually read back these big, you know, 25 minutes worth of stuff back from editorial and play those in sequence with the rest of the boards.

And, you know, this is sort of an obvious answer here, right, which is use QuickTime, right? But, you know, I don't know. QuickTime is really... It's really hard, and we're really busy. We can't make that kind of investment. And, you know, and frankly, we're Cocoa programmers. We're a little spoiled. You know, we're simple people.

So about, oh, I don't know, eight months ago, a year ago, we started talking very concretely with Apple, actually, I guess longer ago than that, maybe a year and a half ago, about what we would like to see in a kit. And Apple was very, very responsive, and they came up with the QT kit, which is a really great 1.0.

It's a framework that I think is going to continue to evolve, but right now, the thing that's shipping there is really great for doing sort of movies of movies. And it did what we wanted, which is let us get to the power of QuickTime and remain true to our inner selves.

Cocoa programmers. And again, it's been about, because the QT kit, that's one thing. We built a couple apps with this stuff, but the one that I'm talking about here today is this thing called Pitch Doctor 2. And we've been working on it now for about five years. Five months. And another programmer and I, Josh Minor, and we're expecting lots of happy users. We're not there yet. We've got about another month before this thing ships. So I thought I would give you just a real quick demo here.

So again, here's Pitch Doctor 2. We just subclassed a bunch of stuff and made it darker because we like dark stuff. So here we are, and here's Pitch Doctor 2, very similar to sort of what you were seeing before, except this time we've actually got some previous stuff cut in there now.

And it's going in there, it's going back and forth between the boards, and... There's that sort of stuff, and that's sort of in the middle, when you're in the middle of the process there. But I brought along, actually, so if you look down at the bottom here, you can see this is about four minutes of footage. But where it starts to get really interesting for us is when we actually do stuff where we've got a real big chunk of the movie on there. So this is actually 6AB, which is the last 22 minutes.

It's a double reel of the movie. And so one of the other fun things that we can do is, in addition to doing video, we actually have a child window that we put up in there. And so we're actually doing a sketching stuff on top of there, and those are based on those boards.

And so we're getting all the advantages, all the sort of advantages of being a Cocoa programmer and child window stuff and all that kind of thing. And all the power of QuickTime and the fact that we can sort of move around here. I'm coming in here, going there.

And this is the kind of stuff that, for us, is just -- it makes all the difference in the world that we can sort of come in here, we go full screen on this stuff, we're playing it. We can work very, very tightly with huge chunks of the movie. And again, this is -- Pitch Doctor was about 28,000, 30,000 lines of code. Pitch Doctor 2, we're actually using this framework we wrote called the Film Online Kit. It's about 60,000 lines of code. And Pitch Doctor 2, the app, is 13,000 lines of code.

So this stuff is real. And it's really, really cool. So I hope you guys use it. And I'm giving another -- we can switch back over here. We have another talk on Friday, kind of sort of the -- you know, similar to the one I gave three years ago, sort of nominally called Cartoons and Code. And hope to see you there. It's at lunch. I don't know what room it is. But you can look at the pieces. Thank you.

Thank you very much, Michael. OK, so that's a quick view of QuickTime 7 and some of the things that we've been investigating there. And thanks again, Michael. That was great. So next, I'd like to bring up Peter Graffagnino and have him tell you a lot more about some of the other core technologies built into Tiger in the area of graphics and media. Thanks, Tim. There you go. Hi, everybody.

So I'm going to bring you through some of the technologies we have for you in Mac OS X for graphics and media. And kind of to restate our goal, I think you've heard a few of the talks this afternoon about how what we really want to do is make it possible for you to create applications that kind of surprise, delight, and inspire your users. You know, the very Apple qualities we all like. Another way to think about it is shortening the distance between your idea and kind of the code that you create to implement that.

Little corny, I know. So here's all the great core technology we have that Tim talked about. And the great thing to realize is our plan to give you all the tools you need is to build in professional-grade technology into the OS. Whether you're doing audio, 2D, 3D imaging, or video, it's all right there.

And there's professional applications built on top of all of these stacks that you can go out and buy. Or you don't have to learn about the low-level details. If you're just a Cocoa programmer, all of these technologies are enabled when you use NS Image, or use the sound API or 2D graphics.

So I'm going to take you on a tour through around these technologies and kind of tell you some of the basic facts about them in case you're new, and also kind of what's new in Tiger. So first up is Core Audio. When we designed Mac OS X, we kind of went back to the drawing board on platform audio to really bring it into the-- enable a whole new set of applications. And Core Audio is a professional-grade audio platform.

It's floating point, multi-channel, handles really high sampling frequencies, and it's incredibly low latency. We're able to accomplish this by using some of the thread scheduling primitives in the kernel to get incredibly low latency, which is real important for high-performance audio. New in Tiger is something we're calling aggregate devices, which is a way to take one physical device-- say it's an eight-channel device-- and make it look to software-- two of those, say-- and make it look like one 16-channel device. So you can aggregate physical devices into one logical device. That's really useful.

Also is a new thing called the Core Audio File Format, which is a new-- a professional-grade audio format, which goes beyond the 32-bit limit for addressing full 64 bits, multi-channel. And when you start dealing with surround sound soundtracks of really big, long feature films, for example, you can run out of 32 bits pretty quickly. So that's a really great format that's in there, and it's built in the QuickTime, and has lots of metadata support, so it works well with Spotlight as well.

The other real important thing about audio architecture is having a good plug-in story, and we've got that with Audio Units, which is a platform-wide standard format for doing audio plug-ins. So you can do either DSP-type plug-ins for just processing audio, or instrument plug-ins like you might see in GarageBand, for example.

And it's supported by all the premier audio applications, so we've been able to kind of standardize audio plug-ins across the platform, which is really great for people who develop audio plug-ins, and great for hosting applications as well. There's hundreds of effects available, and there's a licensing program, which if you go to that URL, and whether you write the individual audio plug-ins themselves, or you create a hosting application, you can license the logo and use it on your packaging if you want.

Another interesting application to go along with the audio units is something new in Tiger called AU Lab. And AU Lab is a visual developer tool for exploring core audio, and it's sort of the best practices test harness for your audio units. Audio units actually can have either a custom UI that's custom drawn or a generic UI just on their parameter description.

And in AU Lab, you can either look at it both ways, make sure it's working, you can do live mixing and playback. It's kind of a performance tool as well. So you can look at audio units in their full spectrum of uses, both real time and just within the application.

The other technology that we are shipping in Tiger is called OpenAL. OpenAL is a powerful API for 3D immersive audio, typically in games. It's cross-platform, Mac OS X, Linux Windows, and consoles as well. And we engaged with this project before Tiger and contributed to the open source, but now it's actually built into Tiger, so you can just use OpenAL directly. It supports multiple speakers for surround sound, and it's of course all built on core audio, and there are a number of game titles that are already shipping using OpenAL. A pointer to a couple of the audio sessions. We've got audio sessions tomorrow for you to check out.

Next technology is 2D graphics. Quartz 2D and PDF form the foundation of Mac OS X's 2D graphics API. It's a professional 2D drawing model. Obviously, PDF and PostScript have been used in the graphic arts industry forever. And it's the basic of the whole printing workflow on Mac OS X. It's integrated with ColorSync for color management, and in Tiger, it's integrated with Core Image for image processing as well.

There's some good tools to use with the Quartz 2D API. There's a Quartz debug, which really lets you get in there and see how frequently your windows are updating, what parts of your windows you're drawing. It's a great tool if you don't know about it. You can usually shave a lot of processing out of your app and optimize your redraw speeds.

ColorSync Utility is the utility we have in there. It can do some advanced color processing on PDF files. Also has a profile visualizer. You could take those profiles Tim was playing with, bring them up, and actually look at how the deforming color space. Great tool. And also Automator, which you've seen demoed before.

But it also has a-- we've provided a few PDF actions for Automator, so you can do batch processing to PDF files. And that's another opportunity for applications who can process PDF. If they can also do Automator actions, you can build up a pretty rich workflow of doing things to PDF files through Automator.

In fact, you can enjoy the benefits of Metro even if you're still running 10.0, because PDF has always been in OS X. But actually, there's a more interesting kind of point here that I'd like to make, which is kind of this point about if a problem's been really well solved in the industry, obviously, many years of graphic arts-- great slide, isn't it? You know, there's little value in really reinventing something like that.

And there's a limited amount of creative energy, you know, that anyone has or any team has, and there's just better things to do. And so the analogy here would be, you know, if you're writing applications on top of Mac OS X and all the technology and all the work we put in to all these core technologies, you know, don't go reinvent and write your own 2D graphics library and your own image processing and all that stuff. Just, you know, leverage the platform, you know, then spend your brain cells on doing things that are unique to your application. So that's my little sidebar for that.

I'm glad you agree. So next is 3D graphics, OpenGL. OpenGL is the industry standard API for 3D graphics and visualization. It's been around forever and really was the premier interactive 3D graphics in the industry. And on OS X, it's not just sort of an API that's on the side that burls a hole through the windowing system. It's built in at a fundamental level.

In fact, our whole windowing system, Quartz Extreme, is built on top of OpenGL. And OpenGL is running all the time, and it has been for a couple of releases now. And in Tiger, we're taking that even further with things like Core Image. We're putting image processing on top of the GPU. And with QuickTime and Core Video, where we're putting video display on top of OpenGL. And some OpenGL improvements themselves in Tiger, we have GLSL, which is the OpenGL shading language in Tiger. And we also have a bunch of performance optimizations.

And we've seen some really good wins with games just running from Panther to Tiger. So we're excited about that. The other great thing about the OpenGL story on Mac OS X is that we've got some platform-specific Mac OS X-only tools to work with OpenGL. You can work with OpenGL on other platforms, but you don't get the advantages of these tools.

Like OpenGL Profiler, which you can wheel up to your application and intercept all the OpenGL calls and see what you're doing and what's taking time. Or the Driver Monitor, where you can go in and see how texture and VRAM. is being allocated and see how long you're waiting for the graphics card. Or OpenGL Shader Builder, where you can prototype little fragment programs, vertex programs.

So a really great set of tools. And if you're not familiar with those, you should definitely go to the sessions and check those out. So there's a couple of sessions on OpenGL in 3D. There's going to be a games track. In fact, there's a whole games track this year at WWDC. And there's an OpenGL performance session on Thursday as well.

So next we come to imaging. The new technology in Tiger here is Core Image, which we talked about last year at WWDC, and it's been shipping in Tiger. It's a floating point image processing framework based on stream computing paradigm, which I talked about last year how that worked.

And it leverages both the GPU and the CPU for implementation. So whether you're on, say, a dual G5, or whether you're on an NVIDIA card or an ATI card, we can program any of those computational engines to do the work for you. And the nice thing about Core Image is the kernels are expressed in a hardware-independent manner, in a little mini OpenGL shading language-like language for doing image processing.

And this makes the whole thing very approachable. So you don't have to know how to wrangle OpenGL and P buffers, or program AlteVec, or whatever. You can either string together predefined kernels that we ship in the OS or kernels that other people have given you. Or you can write your own kernel, and you just write the inner bit of the loop, and everything else happens around it.

Image units are a plug-in architecture that goes along with core image, which is similar analog to core audio and audio units. And it's extensible image processing modules that you can do and install on the system and be hosted by applications. In fact, the new version of Motion that just shipped, Motion 2, will load image units. And again, there's a licensing program, whether you want to be a host app or you want to write image units, that you can go to license the logo.

Image.io is another really interesting technology that we put into Tiger-related imaging. We took together all the best implementations we had for JPEG, TIFF, and some of the open-source stuff that's out there, put a universal API over that, and support a wide variety of formats with a single API. And it's platform-optimized code, so it's AlteVec-tuned where it's important. It supports metadata, so all the Spotlight importers that deal with still image formats come through Image.io and get the metadata.

It has incremental loading, in fact, Safari is on top of Image.io for all of its image handling on Tiger. And there's a bunch of new data formats that we wanted to support as well. Floating-point images, both floating-point variants of TIFF and OpenEXR, which is a floating-point format, and camera RAW formats.

If you have high-end digital camera today, there's a bunch of RAW formats that Image.io will support as well. There's a bunch of sessions this year talking about Image.io as part of the... example code that they're doing. But what I wanted to point at was this one, Mac OS X Color Image Management Explained , because they actually go through quite a number of interesting cases with Image.io and color management, so I wanted to point you at that.

Next up on the core image side, we have a bunch of tools as well. We've got Core Image Funhouse, which we demoed last year at WWDC. You saw Phil make the electric zebra. The app's been cleaned up and made ready for you. In fact, the source code is in Tiger.

And you can explore the built-in image units. You can write your own, and Funhaus will load them so you can see your custom image unit in action. Quartz Composer is also a really good tool for exploring Core Image. You can wire it up in real time. You can be playing video. I'll give you a demo of this in a second.

It includes a real-time little Core Image kernel language pane that you can type into and see things change in real time. So to point to the imaging sessions, we've got Core Image tomorrow. And then on Wednesday, we've got Color Image Management and Essential Ingredients for Mac OS X Imaging, which is about camera connectivity and also some image I/O there as well.

Finally, video. Core video, I'm not going to spend a huge amount of time on it. Tim talked how it relates to QuickTime. It's an optimized architecture for video on a texture, includes all the data flow and synchronization primitives to make the GPU and the CPU have maximum parallelism when presenting video.

There's a facility to synchronize your display presentation of video with display refresh, and it forms the basis of the new QuickTime 7 display architecture, but you don't need to use QuickTime to take advantage of core video. If you're just reading, say, raw YUV data from a disk or something like that, you can just use the buffering services and draw it yourself.

So that's core video. And of course, what would Peter Graffagnino talk be without a little GPU action? So I thought I'd talk a little bit about the GPU this year. I mean, we've obviously put a lot of technology on top of the GPU with improvements in OpenGL and core image and all of that thing. But I wanted to kind of make a slightly different point this year. And that's what I call cutting edge versus trailing edge.

The industry, particularly in the area of graphics, is moving really quickly. And you've got to evolve or die. I mean, if you don't, you're going to die. If you're still on Quick Draw, let's face it, your application is going to slowly become obsolete. And people aren't going to use it because you're not keeping up with the times. But cutting edges are sharp.

And you've got to be careful when you're running with sharp objects. So there's some examples. Last year, we talked about Quartz 2D Extreme. Quartz 2D Extreme, the final version, is not in Tiger. We learned a bunch of stuff about the Quartz 2D API. First off, we learned many applications are not rendering bound.

And when they are rendering bound and they're using the Quartz 2D API well, we actually saw major performance gains with Quartz 2D on top of OpenGL. But for people who weren't using the API well, we actually saw some deficit when they would run accelerated. And so in the final analysis, with the extra resources required, it just wasn't worth doing Quartz 2D Extreme until we get everyone using the 2D API, hang on to their image refs so textures aren't uploaded. So definitely go to that 2D optimization session.

And learn a bunch of stuff about how to use Quartz 2D efficiently. The other thing to be careful of when dealing with GPU programming is subtle differences between GPU. For example, you saw Ted talk about division by zero on CPUs. Well, the same is true on GPUs, actually. If you divide by zero on an ATI card, you get a different answer than if you divide by zero on an NVIDIA card. And you just have to deal with these things in order to make progress.

But my big message is, you know, it's much better to be dealing with these little nicks and bruises than being on the cutting edge than it is, you know, taking the conservative route. Because if your competitor is out there on the cutting edge, you know, he's going to blaze the trail in front of you, and you want to be the one out there in the lead.

So that's it for all of the core technology that we have in the operating system, all industrial strength across the spectrum here. I have one more technology I want to talk about, which doesn't really fit into any of those categories, because it's kind of an integration technology across the whole suite of things in Mac OS X.

It's called Quartz Composer. Quartz Composer is a framework for real-time motion graphics, animations, and visualizations. The composition files that it creates just can play back in Tiger with a very simple runtime API. They can actually play back in QuickTime under Tiger. With the developer tools, there's a graphical editor that you can edit these compositions, and there's a really simple-to-use API for including them in your applications.

Some people make the analogy that it's kind of like Interface Builder, but for a little parameterized motion graphics things. So rather than try and explain it some more, let me go to the demo machine and show you a few things. Probably the most visible thing you may have seen with Quartz Composer is the screensaver in Mac OS X Tiger.

Let me bring it up here, which is the RSS Visualizer. So this is actually a parameterized screensaver. This is not rendered-- it's not a QuickTime movie or anything. It's actually reading the RSS feed, putting things on textures. There's a big sphere of cloudy stuff that's rotating around. And all that's just dynamically calculated.

And in fact, to prove that to you, what I can do is, if I option click on the Options tab, it actually brings up the composition. And so if you want to go mess with this on your system, you can actually look at the hidden parameters for the screensaver and figure out what's going on.

Let's see, another thing I wanted to show while I'm here is some of the core image integration with Quartz Composer. So let me open up this composition here. And you can see the data flow down here. What's going on is I'm taking a movie, and right now I'm just playing it on a billboard texture. This came from a guy on the internet who post-- after Tiger shipped, he went crazy and he did a bunch of video processing stuff, which I thought was pretty cool.

Sam Cass. So I'm going to disconnect the image from the text and put it through this mixer here. So I'm going to mix the original with the video as it's processed through this set of filters and put it back on there. And what he's done is he's added this sort of highlight glow effect, which I think is pretty wild. And you can actually go into the inspector here and look at some of the custom kernels.

This is actually the code that he wrote to process it. Some of these are built in. Some of these are custom one. And I can actually just go here. It's evaluating this in real time. I can say, OK, let's take the red and multiply it 2.0 times R.

So there it is with a little extra red. I could do, say, five times the red, or six times the red, or 10 times the red. But if you're coming up with an algorithm, you can just have stuff playing and just sit there typing with it. So it's a great environment for just kind of taming the GPU.

So let me quit that. And another thing we noticed, there's another guy who's been messing around with Quartz Composer over in Japan, and he's done some amazing stuff. Let me double click on that. These were up on his blog, and so we decided, well, let's just load them all, and let's just go into expose and show you some of the really cool things he's done.

One one in particular I like is this one. I mean, it's getting decent frame rate. Of course, there's like eight other compositions going, but that's kind of pretty. I mean, you should check out his stuff. Some of it's a little arty and probably wouldn't take a while to absorb, but it's a really great site if you caught that URL, if you didn't see me, and I will point you at it.

Let's see, so that's it for the Quartz Composer demo. If I can go back to slides. So now we do have one more demo for you. So this is juggling the most technology at once we could think of. And so what we're going to do is try to demonstrate all of the technologies working together in Tiger, both audio and visual. And so the concept here is we're going to use Quartz Composer with parameterized compositions to augment a live DJ performance. And so these visuals are not rendered.

We do use a QuickTime movie in there of some dancers, but it's all real-time animation that's responding to the performance. And I think this is going to enable some new possibilities and sort of real-time creation on the platform. So with that, let me invite up Ean Golden. Ean is a local DJ and producer, and he's going to show us what he's been able to do. Hey, Ean. Hey, Peter. Thank you.

Hey guys, I'm really excited to be here, and most excited about what you're going to create over the next year for me to play with. Because I am the end user, you guys are the creators, and I'm really, really stoked about all the technology that he just talked about, and what you can do with it, and then what I can do with it live.

I've been a professional DJ for eight years, and about two years ago I made a major change. I started DJing completely digitally. I got rid of the records, I got rid of the turntables, and I started using my PowerBook, a sound card, my headphones, and a program called Traktor from Native Instruments, which I want to show you a little bit about. So, Traktor is basically emulating your standard decks, if you will. On one side you've got one piece of music playing.

And on the other side, you've got a second piece of music playing. And it all comes together in the center in an internal mixer. Traktor's really cool. It's got some extra features. It's got really interesting looping functions and filters and all kinds of great stuff that I can manipulate the songs live with.

And the one major obstacle that Traktor, even though it created all this opportunity for creative potential-- I could take songs, I could remix them on the fly, I could do whatever I wanted-- the major obstacle it came up against was something I very fondly refer to as the geek effect, which is I was using a laptop.

And when you stick a performer on a stage with a laptop, oftentimes people don't feel like you're actually performing. They feel like you're up there checking your email or surfing the internet. And for all they know, you could be playing a podcast off of iTunes and not be doing anything. So the major battle was, OK, now how do I show people this dynamic, incredible, performance that we're able to do with this technology? And the answer was by integrating visuals, by, through digital means, showing on the screen what I was doing on the computer.

And I was able to do this, but it was through very tedious means. Basically, we had a second laptop running a great program, a standard VJ program called VDMX. And we used standard video clips. And we applied different kinds of effects to the video to reflect in real time what I was doing. But I came up against two really big obstacles. The first was processor power. Anybody who works with video knows the more effects you apply to video, as they start adding up, your processor hits the floor.

It just goes through the floor. You can't do all that much. The other problem was creative. I found a creative ceiling. I had all these ideas I wanted to do. When I turn a filter, I want the whole thing to spin around. And I want the screen to explode. And there's only so much you can do with video and effects before, again, the processor hits the floor. After doing a couple of shows with this technology, I was introduced to Quartz Composer.

And it became very apparent right off the bat this was the answer that I had been looking for. Quartz Composer solved both of these problems, number one, because it's not leveraging only the CPU, but the GPU. And not only is it freeing up the CPU by using the GPU, but now I was able to run it all on the same machine. I could run Tractor on the GPU. I could run Quartz Composer on the CPU. Quartz on the GPU.

And have both of these programs running the same environment dynamically. And the second major thing, the thing that really excited me, was Quartz Composer was not affecting video. It was creating content on the fly. It was essentially generating the same kind of material that I was generating in the stage, which gave me unlimited creative potential for what I wanted to do. I mean, really, the sky is the limit on that.

I could do anything, basically. However, it was really just a limitation of what I wanted to do. So I started working with the team at Apple to find an effective integration of these two technologies. And what we did for this particular demo is we've connected one piece of music in Tractor-- you see, it's on the left side-- to one aspect, or one layer, of this Quartz composition. Let's see. I'll show that to you here real quick.

So we've got a piece of video, or one aspect of the court's composition, connected to one piece of audio on one side, and we've got another aspect of the court's composition connected to a second piece of audio on the second side. And then we've also connected all the different looping and filters and effects to different parameters of these compositions.

So as I start to mix these two pieces of audio together live, and as that mix evolves, the video mix will also evolve in real time, sort of reflecting what I'm doing and creating a compelling example of the performance that's going on on the screen. So without much further ado, I'm going to do a short little demonstration of what that might look like in the real world. You know what would be good? If I stuck Quark's composer up on the screen first.

Explain that a little bit for you, what was going on. Maybe it was apparent, maybe it wasn't, but I'll do it pretty quick. So Ean had a device called Trigger Finger from M-Audio, which is just a pad that sends a bunch of MIDI signals into Traktor DJ Studio through Core MIDI and then playing out the audio. So that's just his basic setup, that's what he performs with.

And so what we did is we added Quartz Composer, we took the MIDI signals out of Traktor and out of the controller itself, and we built some compositions which used video and audio. And so what we did is we added Quartz Composer, we took the MIDI signals out of Traktor and out of the controller itself, and we built some compositions which used video and audio. So finally, I'm going to bring Tim back up on stage, and he's going to wrap us up. Thanks a lot, everybody.

Thanks. Okay, so we've given you a quick tour of some of the coolest technologies in Mac OS X Tiger. We think there's a lot of exciting stuff going on here, and we're really looking forward to talking with you this week and hearing about your experience with this. Of course, we've only been able to scratch the surface of all of the new features that our teams have built into the system.

There's literally hundreds of new features in the system for you to play around with and utilize in your applications. Now, we've got lots and lots of engineers here all week long, and I hope you'll have a chance to talk with them. I hope you'll not be shy, come up and talk with me or Peter about what you're up to. We really would love to hear this. We've got two feedback forums on the afternoon of Friday.

Come and tell us if you haven't had a chance to talk to us during the week. Come and talk to us at the end of the week and tell us what you're thinking. You know, there's one topic I haven't talked about today, and I thought I probably should mention something about it. And this was this little detail of transitions.

If things work the way I want them to work, you really aren't going to have to worry about these kinds of issues in terms of the graphics and media technologies. The way that I've spelled out our goals for Mac OS X in the area of graphics and media is that it's our job to provide you the right kinds of APIs so that we can insulate you from these kinds of changes.

We've done a lot of the work already. And by the time we get together next year, we'll have done the rest of the work. And, you know, my expectation is it's going to go really well, and we're very excited about it. If you write to the APIs, you'll derive all the benefits of our hard work, and we hope you'll take us up on that. Anyway, that's graphics and media in Tiger. Thank you very much for coming. Have a great week.