In 1998, Apple Computer became the first computer company to make open source development part of its ongoing software strategy. We've had great successes then, and today we'd like to give you an update and status on some of the projects, which includes Darwin itself, QuickTime Streaming Server, OpenPlay/NetSprocket. At this point, I'd like to bring on stage the director of CoreOS Engineering, Brett Halle.

Good morning. Certainly open source at Apple, I think one should realize, is becoming fairly pervasive through our project plans. And certainly this isn't the only session at all that's dealing with open source. But in this session, we're going to spend some time today and actually go over kind of more of the infrastructure and mechanisms and stuff behind what we're doing around open source.

And you can learn a lot about open source throughout the week and other sessions as well. There's a huge number of technical sessions for Darwin, something on the order of 20 different sessions, as well as some of the other technologies. So it's not just here we're going to be talking about open source. This is something that's pretty consistent throughout much of the technology discussions.

So we're going to spend a few minutes and talk about a number of the various projects around open source at Apple. In addition, we're going to give you a little idea of where we're going, particularly in some of the infrastructure and stuff that we're doing. We're going to talk about how we're going to support to kind of enable open source development in the community. And most importantly, how you can get involved and participate in the process.

So first, it's worth mentioning that our public source license, version 1.2, was released just a little while ago. And that's actually been extremely well received. Since the original release of our public source license, we've been working with the community to kind of refine it and make it work well for both our developers and Apple Computer. And I think we've done extremely well. And clearly the community believes that because we've been approved by the OSI as being a certified open source license. Certainly something we're very happy about.

I'd like to now actually make a bit of an announcement that we've added a new project to the open source projects at Apple. This is CDSA, the Common Data Security Architecture. Thank you. Sounds like that one person is very excited about this. This is an open group standard. It's basically the basis, the underlying foundation for a lot of the security features within Mac OS X.

Support for the key chain and other key services of the system. It provides a number of modular plug-ins for cryptography and certificate management and key management and things like that. This is now available as of today, I believe. We got all the stuff all up last night and this morning. It should be available now on our website for you to download and play with.

There's a ton of projects being ported to Mac OS X, and I think to a large degree that's because of the open source nature of Darwin and other parts of the system. It really fosters the philosophy of being able to get these kinds of services and capabilities under the system.

As of today, for example, as I understand it, on sale in Developer Depot is a CD of something like 1,200 ports, BSD ports from the ports collection. 1,000? 2,000! Oh my goodness, sorry. You know, tools and commands and applications, if you will, to run into the Darwin environment.

and David LeRoy. Also, I think that's one of the other aspects that we think is pretty important is being able to actually keep up with the community at large. There's an order of magnitude more people involved in the development process and there's a way to really leverage that community far beyond the walls of Apple. I think it's a pretty important capability and facility. Certainly, all of this really comes down to trying to make sure that we make the best possible products we can for our customers and our developers.

and much like there's quality benefits for us, I think that's true as well to the open source environment. The reason it works is because there's a lot of people working on the technologies and everybody benefits by the improvements in the quality and testing capabilities. For Apple, this community is growing pretty significantly. Since we released Darwin, we've had over 700,000 downloads, which is pretty darn significant.

Even the next door is clapping, I think this is great. They think it's pretty cool. We've had over 95,000 users registered on our site. And that's a pretty significant developer community. Our mailing list activity year over year has grown by two and a half times a year, it's exponential.

And we're at this point about 1,800, you know, items in the mailing list discussions every month, and we see this growing at a pretty ferocious rate. Clearly as we've reached major milestones of reaching to the developer preview releases and the public beta and now of course the final release, this ends up being pretty significant milestones of developer involvement and adoption. And as people continue to port their products and technologies to Mac OS X, this is a great forum for them to participate in. If you're not on the mailing list, we encourage you to. There's a lot of great discussion there.

There's a lot of activity going on, specific activity going on in the community well. Certainly we're actively involved in supporting the FreeBSD and changes and things that we have. There's a very vibrant relationship there, sharing information about security issues and collaborations that we're doing on packaging and other kinds of things. Also, we work very tightly with the IT department. We're very involved in the development of the freeBSD. We're very involved in the development of the freeBSD. We're very involved in the Apache group.

We've certainly had them on campus and sponsored their activities and try and provide hardware and other things to them as well as fixes to support the platform. The X386 project, MySQL, PAM, et cetera, there's a number of projects where there's been a lot of two-way activity. In addition, there's been quite a lot of interest in the open source developer community to actually port Darwin to the X86 platform. And we've, you know, been supporting that work.

That's not something that, you know, we're doing for any product direct interest. This is something the community is pretty excited about doing. And as that work continues to evolve, we take the fixes and changes incorporated in the source base so that people can take advantage of one shared source for all of that technology.

The streaming server has been ported to just about every significant platform that's out there. And as a result, you can make sure that it's possible that you can stream QuickTime content on just any, anything that you want. Also, certainly I'm sure you've seen recent announcements on things, new participants in NetSprocket and OpenPlay.

And there's been a lot of positive progress there. And we'll talk more about that in a few minutes. In addition, we've been really, you know, working with the community. As I said, there's been a lot of fixes and things that have come in, things like security fixes or other fixes that have come in.

We've actually, you know, watched those things. If there are opportunity, we fold them into software updates. So as you have seen, we've had, you know, three software updates since Mac OS X was released. And in those, when there's been opportunities to include security fixes or things from the community, we do. And like I said already, the support line has been kind of now kind of dealing with this new concept of getting fixes along with their, with bug reports. So there's been quite a lot of exciting activity there.

We have this as part of the infrastructure for working with the open source community. As I mentioned, it's not just Apple. We actually have people in the community who actually have write access to our CVS repositories and actually in a position to contribute directly to the code for Darwin and some of these other projects. And these folks we call Darwin Developers. These are partners in the open source arena for a number of specific open source projects.

They should be considered peers to the rest of the engineers and people at Apple in terms of their ability to help contribute code changes and kind of participate in the process of evolving the source code. They can help with questions and then certainly they're in a position because they have CVS commit access to be able to commit changes and modifications and things into the CVS repositories. Obviously from a standpoint of just managing the source code, it's not a very easy thing to do. But again, we do try and make sure that they are specific point people for each of the key technologies.

But again, it's worth noting that those people aren't necessarily people who work for or at Apple. These developers are folks that the community and the core team of Darwin have selected for technical ability, kind of well-known participation in the open source community, folks that are actually approved by the Apple executive management. So that they actually -- we are giving them access to our repositories, like I said.

There's quite a number of people who are involved in this. and here's a list of some of the key people if you're involved with. And one important announcement that I want to make today is that we've just recently in the past day signed Jordan Hubbard of FreeBSD to be part of the Darwin core team.

Jordan, if you can raise your hand. If you know, Jordan is actually one of the founders of FreeBSD and is certainly, I think, the real spiritual leader of the FreeBSD community, if you will. We really take involvement and collaboration with that community to be a very serious goal of Darwin and our open source project.

So we're really glad to have him on as a Darwin developer. In addition, there's a large number of other people here. I won't go through this. You can look at the list. But you'll actually get a chance to meet some of these folks here a little bit later in the presentation.

So from my particular section of this, I think one other thing to note, again, is that there's some really important benefits we see to open source. This is not an experiment for us. This is the way we do business for Mac OS X and for a large number of our, you know, technologies.

And this is something that we're really working to try and make work well for Apple Computer. We really, you know, think this is something that we're, you know, is a real benefit to you. We think it's a benefit to Apple, to the open source community, and to our customers.

And we really would like to, you know, have you guys be part of that more and more. So if you get an opportunity, please get on the mailing list, get involved, and we'll talk a little bit more about how you can do that and some of the specifics in just a few minutes.

You'll see, as mentioned today, with the growth of addition of CDSA now in our open source arsenal, that, you know, this is something that we're, you know, continuing to evolve and grow projects that we think are appropriate candidates for open sourcing at Apple. And, you know, if you're interested in it, if you think you're one of those people who really can make a contribution and want to be directly involved as a Darwin developer or just as part of the Darwin community, we'd love to have you.

So up next in the presentation, we have a number of things we're going to talk about. Dave Dozicki is going to come up next and talk about some of the support resources and kind of the mechanisms for supporting the community. He's also going to talk about Darwin itself and how you can get involved in building and working with that.

Chris LaCroix will talk a bit about the streaming community. He's going to talk about the streaming server. Lane Roth for NetSprockets and OpenPlay. And Ron Hayden will talk about our open documentation efforts with HeaderDoc. And we'll also do some Q&A in the end. And on that, I'll pass this on to Dave.

Alright, so we're going to start off talking about what infrastructure you guys have to work with us. First of all, let's start off with our Darwin repository. Our repository, where most of our projects live, is beyond our firewall. So, you guys can get at it fairly quickly. And our external Darwin developers have direct access to it, too. We also have an internal repository where some of our non-live projects live for a few specific reasons.

Either we've got source code from third parties that we can't release, there's some information we've received under NDA that we can't release, things like that. But we try to minimize that where we can and keep our projects in the external repository where you guys can get at them, read the history, read the comments, know why we did things the way we did.

We have a web form where you guys can come in and provide us with patches or feedback. Those will come to us and also they're available to our Darwin developers so we can get them back into our CVS repositories. And of course our Darwin developers work with us and you, the community, to incorporate changes and fixes into our code to make it better.

We have a bug tracking system inside of Apple, of course. We also thought it was good for you guys to be able to see what changes we're making or need to be made so you can help us out if we need it. You can browse and submit them via our website.

It is incorporated with our internal bug tracking system, so it's not a separate bug tracking system. It's just an import and an export of our bugs, so you don't need to worry about any disconnect in nature of the bugs that are reported there. They're what's in our bug tracking system. And we update it three times a day so you guys can see quick updates. These are a bunch of the resources we have. There's, of course, our main website.

There's also our page on CVS and how to use it at Apple with our repositories. We have a few main resources. We have our mailing list hosted on list.apple.com. That includes Darwin development, our main development mailing list. That includes Darwin OS users, our kind of discussion list for people that are actually using Darwin.

And there's also some mailing lists for the streaming server and the security project, also known as the CDSA project. And I think that's it for mailing lists. There's, of course, our bug tracking system if you want to see what's going on with our projects. a list of our Darwin developers is available.

We also have Hexley, which is a logo that was come up with by the community. Of course, created by John Hooper for the use by the Darwin community. There's also T-shirts available that you can talk to us later. You saw Brett, of course, wearing one and there are a few others of us around here wearing them.

and now I'll talk about Darwin itself. Darwin is the core of Mac OS X. It's the foundation. It's the same. You can swap in a kernel you build yourself from our repository with the kernel provided in our system. We consider that very important that you can do that. And, you know, if you ever run into any problems, let us know. But it works today and we plan on keeping it working.

So now you've probably seen this picture a lot. We'll show it to you again. Darwin is the foundation of Mac OS X. It provides the file systems, the networking, Mach, I/O Kit, drivers, and it also provides user land which would probably sit right next to QuickTime and all sorts of other-- or Cocoa actually.

Yeah, the foundation of MEGAUS 10. It's BSD based with a Mach kernel and I/O Kit. We think it's very important that it is BSD based because it provides a foundation for lots of your applications to be ported over in a quick way. It's extensible to provide support for a lot of file systems. We already have a bunch in the base system today, including HFS, UFS, NFS, UDF, MS-DOS.

But there's certainly more out there, and we'd like to see them. It provides dynamic and flexible configuration management. As you've seen in earlier presentations, we provide things like mobility, hot plug of disks, and et cetera. And it supports multiple platforms, and we'd certainly like to see more. Today we support PowerPC and x86. You can download binaries for both platforms from our website and be able to run them today on your system. And now we're going to have a demo by Wilfredo Sanchez.

Hey Brett. My name's Fred Sanchez, I'm a Darwin developer. What that means is basically that, like anyone else who works at Apple, for Brett in the CoreOS group, can you log in for me? I'm one of the engineers that works on Mac OS X. I'm one of the authors of some of the software that you'll see in your system when you boot up. In particular, I wrote a program called System Starter which turns your computer on from the sort of... Init has run, you've just turned the computer on up to the state where it's safe to bring up the login prompt for the multi-user.

Tomorrow there will be a talk on performance and Joe Sokol and Robert Budage will be like, "Why isn't the system started? It doesn't go fast enough." That's a personal affront to me so I had to fix that. And so I got a demo here of a thing I've been hacking on. Yes, sir. Flip the switch? Demo 1? Alright. I am going to bring up the very famous terminal application, which scares a lot of people but is a lot of fun for me.

If you're a Darwin developer, you spend an awful lot of time on the Unix command line prompt, so this shouldn't scare you too much. Basically, System Starter is like all good Apple software, it has a demo mode. So I'll show you how it normally works. It runs a series of startup items in serialized order.

So it knows this item has to start before this other item can go, and it runs one at a time in order. And what I did different here is if you turn on your computer, you might notice that it takes a while to run each one, one at a time.

And this might have looked different to you in that you see a whole bunch of things happening at once, and then you see a few more, and then you see a few more. So what I changed is that the startup items, rather than running serially in order, it'll run as many as it can possibly run all at once in parallel. And then as soon as it can run another one, it runs a few more. And so the debugging app what you see here is it's showing you like it's launching a whole bunch of commands and then waiting on the right one to exit for you.

And I do this because I have a vested interest in seeing Mac OS become a better system, mostly because I don't want to use it. I want to use Windows for the rest of my life. And I don't work at Apple, but I have access to the software and I can do this as well, and so can you. Now before you think I'm a liar, I actually used to work at Apple, which is why I wrote this, but now I don't, so it's all good. So that's my demo. Do I do the -- can you do that? Back to date. Thanks.

Can we switch back to the slides? So the roadmap for what's left in the week, we have a 193 leveraging BSD services in Mac OS X. It's going to be in the Civic Center Friday at 2:00 p.m. We have 140, the Darwin kernel, which is also going to be at the Civic Center on Friday at 3:30, which I highly recommend for those of you that are interested in what's going on inside the kernel. And then there's our feedback forum on -- also in the Civic Center at 5:00 p.m. So up next, we have Chris LaCroix from the QuickTime group.

Thanks Dave. I'm just going to really briefly go through, or kind of provide some status on where we're at with the QuickTime Streaming Server. First thing I want to do is kind of take care of some confusion that people have. There's a Darwin Streaming Server and there's a QuickTime Streaming Server. And people always ask, "Well, what's the difference between those two?" And there's actually nothing different at all other than the name.

QuickTime Streaming Server is just a branding, our brand name that we use for the Mac OS X version of the server. Other than that, they're built from the same source code. The Darwin platforms are FreeBSD, Linux, Windows NT, Solaris. It's all identical source code. They have identical functionality. They have the same user interface as F3.0. There's no difference.

and David LeRoy. So here's what, here are the platforms we currently build on. The ones on the top are platforms that Apple actually builds and tests on. So, those, and we actually provide binaries that you can download of all of those platforms. The ones underneath are platforms that other people have ported to. Mac OS was done by some guys in Brazil. There's a SGI IREX port.

Compaq actually ported it to their true 64 systems which are like massive 64-bit big hardware. And it's been ported to a bunch of other Linux I or Linux S or whatever the In 3.0, the newest features are we've added a web-based administration. That's one of the things that allowed 3.0 to be exactly the same on every platform. In versions 2 and earlier, we actually had a little Cocoa app for Mac OS X. We've gotten rid of that and everything is all web-based now. We've added some broadcast automation functionality which allows a broadcaster to automatically start up broadcasts to a server which then get reflected to clients.

Skip protection is new. Skip protection is just another brand name for a collection of quality of service enhancements that we've added to 3.0 and will continue adding in future versions. We've improved the authentication. We now have Digest MD5 authentication. and we've done a few minor API changes. So if you want to hear more about these, there's a session tomorrow that will go into a lot more detail.

and so here we're here's where we are with downloads. So we have over 150,000 downloads of the server itself. The client has over 150 million copies of the client in distribution, which is a pretty big number. And then the number I just looked up recently which kind of surprised me is that we have over 500 developers that are currently subscribed to the developer mailing list, which means that they're actively watching mail go by. So I guess we've got probably the biggest... We have the biggest development team for any streaming server in the world. I guess you can look at it that way.

and another thing I did is I just kind of did a search on Google for QuickTime Streaming Server, Darwin Streaming Server, just to kind of see what kind of people were using this. And here's a list of some of them I came up with. There are actually a bunch. I got like a massive number of pages in Google when I did the search. So NASA's using it someplace. I'm not sure exactly what department it is. Popwire actually does some pretty cool stuff.

They've got, they provide kind of internet radio station. And they've taken the playlist broadcaster portion of the server and modified that so that you could, it, provide crates of an internet radio station that has ad insertion and things like that. Vancouver Film School has standardized on the server.

They actually use it for, they've got a little internal TV show that they do and they put that up on the web in streaming format for quick time only. SurfNet TV is, I think that's the European site, I actually don't remember what they do, this is a week ago that I looked at this stuff. Ink Tomy has created a caching proxy that works with the server.

Compaq, as I mentioned, ported it to their 264 system and they include that as, one of the things they provide is a collection of open source software for that platform. And the Darwin Streaming Server is one of those items. And the Mount Erebus Volcano Observatory, they've got a camera that's up in the crater and is actually watching the volcano all the time. And they've got a live stream that you can go and watch. They use the Darwin Streaming Server for that.

Some of the resources are the mailing list that I just mentioned. If you go to lists.apple.com, just search for the word "stream" and you'll see two lists. You'll see a developer list and a user list. They're both actually good to subscribe to. They don't have a massive amount of traffic, somewhere between 5 and 20 messages a day, but they're all really focused and good messages.

So it's actually good. And all of our engineers are actually up on those lists too, so that's the best way to get a hold of those guys. You can get all the information about the product up on Apple's open source site. And then you can go to the QuickTime site if you want to find out more about QuickTime and more of the client side features of QuickTime and how they interact with the server.

and a couple of, or three, four more sessions. So there's a "Broadcating Live" with QuickTime streaming or broadcasting live at 5 o'clock. And then tomorrow we've got two sessions that are server-side only, starting at 9 o'clock. Just an overview of the server. I'll go into more detail about the server, how it works, the new features, things like that.

And then we'll have a bunch of the engineers come up in the 407 session and show you all the different ways you can customize the server. It's truly the most customizable streaming server in the world. And then we've got our feedback forum on Thursday at 3.30. And we'll also be at the Beer Bust at Apple. All the engineers there, so you can kind of look either in the server group or the QuickTime group. We'll be in one or the other. And that's it. Next is Lane Roth. Thank you.

All right, so I'm covering the OpenPlay effort within Darwin. The biggest news is that NetSprocket, which was released as open source a while back, is now part of the OpenPlay effort and it has been carbonized. All of the NetSprocket functionality, all those APIs are now part of OpenPlay and are being tested right now and getting all that functionality working.

Currently, the package that will be checked in to CVS shortly consists of two separate projects: the OpenPlay project and the NetSprocket project. This is a legacy of how this all came to be and how it was developed. It makes things a little bit tough to figure out. We are working to correct that.

We will combine both of those projects into a single project that will be called OpenPlay. This will alleviate some of the weirdness with having multiple directories that are the same name and some duplication of source files, so forth like The current NetSprocket source is available and will remain as such, but all future work will be on the combined OpenPlay project.

Okay, so the networking API is designed to be cross-platform and protocol agnostic. Currently, it is running under Mac OS X in a carbonized form under Mac OS as well as under Windows. There is a TCP/IP network module that is available for all of those platforms, and then Apple Talk is available for Mac OS.

There are three API layers to the new OpenPlay. There's the game interface, and that's the highest level interface, which was known previously as NetSprocket. That interface is what most of the games and other applications will use. The protocol interface was the former straight OpenPlay interface. It's a lower level interface just for connecting and passing packets back and forth. The networking interface is just a wrapper to go around the different networking APIs, and most people won't be using that.

It all comes together as you see. Your application can talk to any one of the three APIs within OpenPlay. You can also talk directly to any of the OS networking APIs as you feel needed. Inside of OpenPlay, the game interface talks to the protocol interface and that talks to the net module interface that is selected for whichever game you're running and it will talk directly to the networking API in the OS.

Okay, so the game interface, as I said, is the highest level interface. This is basically the NetSprocket API. It's the easiest way for you to add functionality, networking functionality to your game. The APIs handle hosting games, joining games, it handles all the game management, it handles your player management for just two or more players.

It also handles packet management, so you can send data back and forth very simply. You don't have to worry about a lot of the details like all the wrapper functions are in the cross platform segment are swapped in their byte order. You still are responsible for doing that for your specific data, but you don't have to worry about all the underlying data. That's all handled by those APIs. It's cross platform again, OS X, Mac OS, Windows.

The protocol interface is a lower level interface. This is where all the platform agnostic and networking wrappers are. Like I said, there's IP for all the platforms and there is Apple Talk for Mac OS. And then besides running on the mentioned platforms, this interface has a start into a Linux version and that is underway by the community.

The networking interface is just a very low level wrapper to go around your different protocols. It has a plug-in design so that in the future if it's needed, somebody can drop in a new protocol module and they can do that independent of the release package. So you can do it just if you have some private networking API that you need to do, you can have your own separate module and not have to check it into CVS. This single API can also be used by your program if you want to have that very low level control and still have the network diversity. And that's it.

and I manage the Darwin Documentation Project. So a couple years ago, Apple obviously did a big leap into open source with Darwin and then a few months ago we did another major leap into open source with our documentation. It was a recognition that the documentation is often as important as the source code and What the purpose of this project is, is to provide the tools and resources to allow developers to provide the documentation directly.

Which is very exciting for me. I've worked for a long time in technical publications and one of the things I've discovered is that frequently the writers are a bottleneck to the information that you need. The engineer may have the information, especially reference material. The engineer may have written down that information for the writer. The technical support people may have the information. Some of our customers using the product may have the information.

But because of the enormous number of things the writers have to cover, that may not get into the next release. And it's wrong. And it's a problem. So one of the things that we're learning from open source is how to remove that bottleneck and let the person who has the information provide the information as quickly as possible to the audience. This doesn't mean that we're not writing documentation anymore or that we're asking the community to provide it all.

We have a Darwin Technologies writing team which is concentrating on some of the big picture stuff. So we're doing an IOCAP fundamentals book, we're doing KEX tutorials, we're doing stuff on handling USB and HID and such. But the actual reference material is more and more frequently coming directly from the engineer and getting into people's hands.

So here's the things that we cover in the Darwin Documentation Project. First and probably most important is HeaderDoc. And what is important about HeaderDoc is that it's a very simple process. It's a very simple process. It's a very simple project. It's a very simple project. And the reason about this is that every person in this room who's doing a C or C++ project should be using HeaderDoc.

So as you can see, we have quite a lot of different projects and things that we're involved in. Apple is extremely committed to open source. As I've mentioned before, this isn't an experiment. This is a way we're doing business. And you'll see more and more involvement from us as each day goes by, as new projects come online as it's appropriate and we get more and more infrastructure in place.

We really want to encourage you to join the community of open source developers. There's, as I said, over 3,500 just on our mailing list alone. There's a lot of great information there and there's a lot of people who really know the software inside and out, including a lot of the internal Apple developers. This is a great way to get to them directly. And the people are quite willing to jump on and answer questions if you've got any. So please participate.

There's a lot of conferences that we're also involved in, so in the open source community and such. So we're going to be at MacHack. We have a number of different sessions there where we'll talk about the booting architecture and more information about Darwin and a number of other talks going on at MacHack.

At USENIX, we have one, a Boffin participation going on that Ernie will be at. And as well, we'll be at the O'Reilly Open Source Conference. I know I have a talk there, an executive forum talk, and we've got some others as well. So we're certainly looking at ways that we can be more involved in some of these external activities. And if you're going to be there, we're going to be there too. So on that, I'd like to have Jason come up, and we're going to do some Q&A.