Thank you. So yeah, as Steve said, we're going to have a cast of characters up here. And he forgot to mention one, which is John Anderson, at the very end, is going to do a demo of the next version of QuickTime Streaming Server Publisher, which has some great new features that, for the project that we're going to talk about, MI StreamNet, they're very excited about the new product because of some of its new capabilities. So the agenda is, first, me talking about some the how-to stuff setting up authentication streaming through firewalls etc and then we're going to go into this case study on the merit network so merit network they are a non-profit provider of internet access for most of the schools and universities in Michigan and so they're going to go ahead and and go through details on the history of the project some of the objectives, how they set up relays, how they've set it up so that to redirect clients to the right server, some of the testing and configuration and then future plans.

So let's dive right into some of the specifics on how to. So one of the things that a lot of people want to do is set up authentication. And there's two main reasons to do this. One is you want to authenticate who can broadcast through your server. So with a live streaming product, who can actually send streams through your server?

And we have this thing we call automatic announced broadcasts that save a lot of the hassle of the way it used to have to be done in terms of doing a live webcast through the server. So one of the reasons for setting up authentication is for providing a friendly way to do these automatic announced broadcasts. Then there's also just the authentication to the media itself. And the authentication that we've implemented, RTSP authentication is very similar to HTTP Digest authentication. So the protocols are similar. And the implementation is also similar to what you would see in Apache. So just like in Apache where you have an HT access file for authentication, in QuickTime Streaming Server you have a QT access file for setting up authentication to a specific directory on the streaming server.

And just like Apache with HT access, you have files, flat files on the disk that have the users and groups in them. So let's just take a look at this. For announced broadcasts, In QuickTime Streaming Server, we have a GUI for creating this password. And I've highlighted them here on the screen. You would click on Accept Incoming Broadcast and click on Set Password. So that allows you to create a username and password for an announced broadcast.

broadcast from a live encoder to the server. On the client side, I have two different screenshots here. These are three graphical broadcasting clients available today that support announced broadcasts. QuickTime Broadcaster is the one from Apple. It's free and it's very simple in its capabilities. One that I'm really excited about, there's a new product and Simon Clark is in the audience here, I encourage you to maybe catch him afterwards.

He's got a great new product called Wirecast that uses quartz transitions, supports multiple video inputs. It's an incredible piece of webcasting software and their screenshot is on the right. So they also support this idea of announced broadcast. So basically what you do is you provide a host name, a file location. You don't really have to put the SDP extension in. I like to put it in, you know, it depends on if you, it's a good idea to add the.stp extension, the username and password that you've set up on the, you know, in the previous area, and then when you webcast, what this does is it automatically generates this file on the server that's necessary for a live webcast, this.stp file, session description protocol file. So, in the old days, which still works the old way, you can do not an automatic webcast, a manual unicast, which means you have to manually export the STP file and manually transfer it to the server. That way still works, and that's the way that, for instance, if you were using one of the hardware encoders, VBRIC, JVC both make hardware MPEG-4 encoders that work great with our streaming server, but they don't automatically generate the STP file.

You have to get the STP file from the device and transfer it to the server to reflect the live webcast. So this way is a friendlier way. So if you're out in the field with your PowerBook and you wanna do a live webcast through the server, this is a very easy way to set it up.

Now under the hood, this all uses these files that we talked about before. So in this example, I'll try this laser pointer, I've done this cat QT access. So I'm in the movies folder on the server and I'm looking at this QT access file. This is what's in it. Limit write, require group broadcaster slash limit and then require end user. So this one means that to write to the server you have to be this user named broadcaster.

Require any user means any other user can just connect to the server. So there's no authentication. Anybody else can freely connect to the server to access the streams. And then this cat command is I'm catting the QT groups file. And actually, here's require group, not user. So anybody in the group broadcaster will be able to webcast to the server. So I've catted this QT Groups file and piped it into grep, being a Unix weenie here, and look for this name broadcaster and these are all the users that are going to be able to broadcast through the server in this setup. So this is the, if you want to set up your own authentication, you're going to have to do some command line stuff and that's just the way it currently works.

So basically what happens if you want to authenticate access to media, you are going to have to modify these files. So again, I've looked at the QT access file. This time I'm saying require group users. That means in this folder, I have this special directory off of my movies directory called authenticated. So now it's just like Apache. anything in this directory will now be require authentication because I've got a Qt access file in there that says require group users so now you have to be in the users group to be able to access media and it's going to put up this login dialog box when you try so if I look through my QT groups file grab users these are the users that are in that group called users and then here I've shown what's the contents of the directory are. So what happens with QuickTime Player, you launch QuickTime Player, you type in the URL to that authenticated media, and then it's going to ask you for a username and password before it streams the content to you. So to create these users, there's a command QT password, which allows you to create new users in the QT users file. To create the groups, it's just a text editor. It's a very simple format. It's just the group name followed by a space delimited list of users. So these files, you can have other QT groups and QT users files. They don't have to be the default files that are in the config folder of QuickTime Streaming Server. They can be other files. So you can set up your own little-- we see it in higher education in particular, where they want to give a faculty member the capability create their own users and groups file. And that works as well.

So this is kind of the old way to do things that we've been going over, which is what Apache has and what QuickTime Streaming Server has. A lot of folks want to tie into a more organizational authentication architecture. And QuickTime Streaming Server, like Apache, is written in modules. So there are modules that can be created or modified to do things like customize authentication. So one group that has done this is NYU. So NYU is involved in an Internet2 project called Shibboleth. Shibboleth is about cross-realm authentication, which means that, for instance, the NYU library might want to trust people from another university, we'll say Cornell University. They might say, "Well, we've got an agreement with Cornell Library so that we can both access each other's stuff." And so Shibboleth, that's what it's about, is this cross-realm authentication. And if you go to the Internet2 webpage, you can read all about Shibboleth. They have written an Apache module, because it's modular too, for Shibboleth that allows you to do this with websites. So what NYU has done is taken that infrastructure and extended it to the streaming server. They didn't do it directly with the streaming server because the Shibboleth authentication they use, it's not just a simple challenge and response kind of dialogue between the web server and the client. There's a lot that goes on there. And that sophisticated dialogue, the RTSP authentication was just too simple for that. It didn't provide them with all the infrastructure they needed. So they've set it up with Apache. So basically the flow here is you request a URL from Safari, your web browser. It goes through this Apache module to authenticate you against Shibboleth, which does this cross-realm authentication. So Shibboleth can talk to basically anything, LDAP, Kerberos, whatever you want. And then what they do is once you've authenticated, it talks to a MySQL database, stores the IP address of the client, the cookie, the shib cookie, and timestamps it, and then redirects to this custom module they've written for QuickTime Streaming Server. Because it's modular, they've written this custom module that understands these special requests, these special URLs. The special URL comes in, and then this custom module goes back to the MySQL database and It says, "Okay, what IP address are they coming from? What's the ship cookie? And when did they do this so you can't replay these things?" And then it says, "Yes or no," and delivers the stream or doesn't deliver the stream to the client. So, this is an example of how a customer has taken the open source part of the streaming server, the Darwin part of the streaming server, customized the module so that they can deliver custom authentication. And we're working with NYU to try to make this piece available through the Darwin CVS so it would be available to any QuickTime streaming server or Darwin streaming server. But you'd have to have all this other stuff working too for it to work. So we're in the process of trying to figure out how to make this available to a wider audience than NYU right now. So pretty interesting stuff.

Next topic, so we're done with authentication, let's talk about firewalls. If you have a publicly accessible streaming server, you should turn on port 80 streaming. And this means that you can't run a web server on that same port on that same IP address on that server. So it means that you might want to bind your streaming server to one IP address and the web server to a different IP address if you want them both to work over port 80 or just run your web server on 8080 or some other port. So what this does, port 80 streaming, is it tunnels all the RTSP/RTP traffic through HTTP port 80. And that means it gets through most firewalls. QuickTime 6 supports automatic configuration. So it will try RTSP over the standard streaming ports. And if they're not available, it will go back to port 80, fall back to port 80. And this is very recommended for these publicly accessible servers, because it allows you to get through firewalls. And you don't know where the client's going to be.

In your organization, another thing that you might want to do is open up certain ports. So this allows you to access streaming servers that might not have port 80 streaming enabled. So minimally, you can open up TCP port 554 and 7070. 7070 is what real servers use, and we can use it too. But by default, we use TCP port 554 for RTSP, which is a signaling protocol to the server.

And then UDP is the protocol that's used to actually stream the data. And you can open up a range of ports here. It depends on how busy your server is and how many clients you have connected to it. So that's what sets that range up, how many ports you actually need. But a typical range is 6970 through 9999. And then if you want to administer the server via a web browser, you need to open up TCP port 1220. And if you want to administer it through our server admin application, you need to open up TCP port 311. So those are the ports you need to have open.

Another way that people set up, again, inside an organization, security through a firewall is they have a special part of their network called a DMZ where they put proxy servers. And there is an RTSP proxy server that allows you to do this. And basically what you're doing here is you're saying the proxy server can talk to any server on the Internet and any client can talk to the proxy server, but clients can't directly talk to the servers out on the Internet. So it puts this go-between box in there, and we have a version of this that's posted to our Darwin site that you can download and use as well. So this is another option, more of an institutional option, again, for people to create firewall, to work with a firewall. Probably the least common. This is the least common of the three. Thank you.

So next we'll talk about the, really, I think the meat of this presentation, which is scaling streaming through relays and replication. So streaming relays are a way to conserve bandwidth, really. So when you're doing a live stream or a playlist stream where you're streaming stuff out in real time to a lot of different clients, what you can do is set up a relay so that a client doesn't have to go to the source server. So this is what we've used in Michigan. Typically a school district doesn't have a big OC3 line or something like that to the internet. They have a T3 or a T1 even, which isn't a very big pipe. So you're not going to be able to get a lot of 300 kilobit streams through a 1.5 megabit T1. There's not a lot of bandwidth there. So instead of trying to get 50 people to connect to a T1 on 300 kilobits, which won't work, because you'll way overload the capacity of the T1, you could put a relay inside the school.

Then what happens is the main server sends a stream to the relay and all the clients inside the school tune in on the local relay where they have plenty of bandwidth inside the school. So they provide these streaming points of presence. So this is the kind of model that Akamai is based on. So Akamai has this capability built in. You may have heard of Akamai which is an industrial strength replication and service out on the internet, but they also support this live capability. These relays can support unicast or multicast. So what I'm doing right now is I'm multicasting. Hopefully you're all listening and I'm speaking. So it's one person, lots of listeners. So that's what multicast is about. Whereas unicast is if we're all on a conference call, we'd be tying up, I don't know, a hundred phone lines. So in unicast, everyone makes a separate connection, just like we do on the phone network. when we make phone calls.

These can be used with live, are typically used with live or playlist streams. They can actually be used with on-demand media, but it's not a very common use of them. Another way to save bandwidth is to replicate your content. And this is what Akamai does in a big way, but there are little ways you can do this with a streaming server as well that are pretty interesting. And it's commonly used for video on demand. Moving that content closer to the client.

So relays, the basics of relays is every relay has one source and one or more destinations. The source for a relay can be one of these SDP files that comes from a live encoder. It can be an actual piece of media. And it needs to have an IP address and port numbers for that source. Where is it going to get that information from? And then you can have one or more destinations.

Where do you want to send this stuff out to? So that can be an IP address with specific port numbers, or it can be this thing called an announced destination, which means that it will do the same thing the broadcaster does. It will announce the broadcast to that destination, automatically generate the STP file, figure out the port numbers, etc. et cetera.

So a very simple configuration for a relay is where you have a main campus and a remote campus. On the main campus, again, you're trying to make most effective use of bandwidth. So you have a lot of clients out on the remote campus and then a lot of clients on the main campus that want to view this webcast. And then you have an encoder. I've labeled it as a broadcaster in this slide. So the broadcaster sends, creates this SDP file, sends a stream off to the main campus server.

The clients on the main campus would just be able to tune in on the SDP file. They're seeing what's called a reflected stream. They're getting a reflected stream off of the broadcaster. This allows you to scale up because most of these webcasting products, you know, encoders support say one to ten clients at the most. So you really need a streaming server in there to scale just internally to more than, you know, a handful of clients. In this case, what we've done is we've created a relay on the main campus server to send the stream to a remote campus, and that remote campus server reflects the stream to all the clients. The value of this is we only have one stream going over the Internet. Instead of all those clients on the remote campus creating their own connections back to the main campus server, they're just connecting locally. So that's the value of a relay. Simplistically, I described two basic kinds of relays. pull and push. So a pull relay, what this does is the relay is pulling a stream from a remote source and relaying it locally. So this would be, in this case, the look, the relay is set up on the remote campus. The remote campus is pulling it from MERIT, in this case, that's what I call it from MERIT, from a server called MIStreemNet1. And I give it the path to the SDP file. When you do this kind of pull relay you have to authenticate to the main server using the administrative username and password of the streaming server anybody that's in the admin group so it doesn't have to be the administrator of the server which would be I think a bad idea it can be anybody that's in that QT groups file listed in the admin group so it doesn't that user can be not a real user on the system this is one of those times when having it the users not as real users comes in handy actually. So this is the remote campus server, it's pulling in this content and relaying it locally. So that's the source, the destination is itself. It's very common for either the source or the destination to be the loopback address. So in this case, the destination is the server itself and we're calling the, giving it the same file name. Because it is the loopback address, we don't need a username and password. We don't need to authenticate because we're talking to ourselves. We figure that's secure.

Then the other kind of relay is a push relay. This is also very common. This is what we've used in Michigan. This is when local streams are pushed out to another server. You have a stream coming into the server and we want to push it out to multiple destinations. In this case, the source is the loopback address and this file name. The destination is a remote server and there are a few different types of destinations we can use here. I'll bring up my little crib notes here. There's unannounced UDP is what this is set up. So a specific IP address and port number.

This destination address can be a multicast address, as we'll see in a little bit. It can be a unicast address or it can be announced UDP. Announced UDP is just like the broadcaster. So it needs a username and password. It's automatically going to generate the SDP on the destination. So it acts basically just like a live broadcaster would act.

So beyond the basics, a relay can pull from a source and push to a destination. A real common scenario for this would be, let's go back to that scenario where we have the remote campus. On the remote campus we have this server out there and if you saw my picture, what it was doing was it was pulling the content from the main campus and relaying it locally as a unicast.

to itself. So it was just sending it back to itself so that people could tune in on that. But what if we wanted to relay it as a multicast to make even more effective use of bandwidth? Then what we would do is we would pull it from the remote server and push it to a multicast address on the local network. So you can pull and push, but as I said, it's very common for either the source or the destination to be the machine itself. We have found that when you're using multiple destinations, as they're doing in Michigan, that this unannounced UDP provides the most robust configuration.

The main reason is if the relay server becomes unavailable for whatever reason and then becomes available again, clients automatically reconnect when you set it up this way. So the unannounced UDP, because the packets are just flowing out on the network, if there's an interruption in the flow, as soon as the flow is restored, clients will continue to receive the stream without hands-off. Whereas if you're using an announced webcast, the clients have to reconnect.

Because with an announced webcast, that STP file gets deleted when the flow is interrupted. And it goes away, so that's why the clients need to reconnect. You can also reflect a multicast source with the streaming server. So this means if you had a rack of servers, each one serving a different subnet, you could multicast one stream to all those servers from your encoder, your broadcaster, and each one of those could reflect those as unicasts off to different parts of your campus.

And the relay destination can be a multicast address, and we're going to take a look at that one. So this is the most sophisticated setup that I'm going to go through, and then I'll turn it over to Merit. So in this case, what we have is the blue lines are unicast, and the green lines are multicast.

So in this case, what we want to do is take advantage of multicast to, again, reduce bandwidth usage on our networks. So we have these great multicast-enabled networks. You have to have your network multicast-enabled for any of this stuff to work. The Internet 1 is not multicast-enabled, so this won't work over the Internet. It does work over the internet too, however. It is a multicast network. So in this case what we're having is the broadcaster is sending a unicast stream to the main server. The main server then is reflecting that unicast, or relaying it actually, relaying that unicast as a multicast on the main campus. So now instead of sending out, in this case, three different streams, we're sending out one stream and people are just tuning in on that multicast. We're sending another unicast stream to the remote campus server and the remote campus server is doing the same thing.

It's relaying that incoming webcast as a multicast. So let's take a look at the configuration on this. So again, using server admin, this is how you would configure it. On the main campus server, we have a source. The source is ourselves and it's this file coming in from the, it's a stream coming in from the encoder. So the encoder has created this 020UDP.sdb file, and we're just bringing that in as a source. The destination is a multicast address. So I've used one that's in the experimental range here. So there are assigned multicast address, and the 239 addresses are basically wide open, and you can just use them to play with. So I've used the 239 address, and I'm sending it as unannounced UDP on a specific port number. So now I'm multicasting. As soon as I turn on this, I'm multicasting, but how the heck do you tune in on a multicast?

Well, this SDP file from the broadcast source, this is what it looks like. This is a typical SDP file. There's a couple things you need to change in this SDP file to make it a multicast SDP file. So what you need to do is there's a line here, the C equals inline. You have to change the address, which was the address in this case of the broadcaster. you have to change it to the multicast address. And you have to change the port numbers, look for the first M equals line, change it to the base port number you selected as your destination, and then add two for the next track, the video track. So now we have a multicast STP file.

If we put this on the streaming server and people tune in on it, the streaming server reflects the multicast as unicast, which is a useful thing, but not what we want in this case. To actually tune in on a multicast, you need to put the SDP file on any web server, and that's the easiest way to do it. And you can just drag and drop it on QuickTime Player too, but -- or e-mail it or whatever you want to do, but the easiest way is to put it on a web server, and then people use an HTTP URL to connect to the multicast SDP file.

On the main campus server we're relaying also to the remote campus. So in this case what we're doing is we're sending a unicast stream from the main campus to the destination which is the remote campus. So the destination is that remote server and we've decided again we're using unannounced UDP because it's more robust. And we know that if the stream somehow gets interrupted this will continue to go as soon as the things start back up again.

On the remote server, we're going to set up the same thing we have on the main campus server. We're going to tune in on that incoming stream, so the source is ourselves, and it's just on that IP address, listening to specific ports, so it's unannounced UDP. We're saying, "Look at ourselves, look at these two specific ports. That's our source. We know we're getting a stream there. We're acting basically as a client to that stream, and then we're going to set the destination, And in this case, I'm reusing the same multicast address.

Because these are two separate networks, I can do this. And I'm using the same multicast address. That's a good thing to do in this case, because then we can use the same SDP file. We don't have to edit it again. We can just use the same file we already edited, put it on a web server, and it just works. So that's a more sophisticated one.

I have a website that tries to untangle all this stuff for people and to help you do more sophisticated stuff with a streaming server. It's just soundscreen.com. So if you go to that website, you can get in touch with me through that page. Also, I've got a lot of instructional information on the how-tos on all these things. So with that, though, I want to turn it over here to Laurie Kirkmeyer, who is the senior engineer of networking R&D for Merit Network, and Patrick Bymhold, who is the systems engineer that supports education in the state of Michigan, and I've worked closely with them on their statewide streaming initiative. Thank you.

Hello. As we get into this, Patrick and I have worked very closely over the last year putting this together. And so we thought that we would sort of tag team this presentation because there's parts that we both worked on. So we'll see how this goes. As I say, I'm a senior engineer with Merit Network. Merit is a statewide network for sort of educational institutions in Michigan. The 15 universities are the board members of the MERIT board. We have over 200 affiliates connected to the statewide network, those affiliates being the ISDs, school districts, hospitals, libraries. The objectives we have here.

were to provide an infrastructure for statewide live webcasting. The impetus for this was from one of the, what's called the regional educational media centers, had a broadcast studio and they were doing satellite broadcasts of administrative broadcasts to school administrators on various topics, like No Child Left Behind and that's one example. And they wanted a way to provide that broadcast to users without the administrators having to drive to a central location to watch this satellite broadcast or having to go to receive it over an HC23 stream to a video conferencing room. So, um, um... The, so what was, what happened about a year and a half ago was Patrick came in and there was one broadcast where they wanted to explore new streaming technologies and so Patrick suggested using a QuickTime streaming server.

It's actually kind of a funny story. We were doing broadcast statewide. We were using Akamai because Apple's got a great relationship with Akamai and we couldn't support the projects the way we want it to, so merit, not merit. Wayne Reese has set up a streaming server doing their broadcasts. And it was working just smashingly, up until the point that the network manager for the organization walked in and he's got a map of the network, bandwidth usage, and he's seeing, you know, ups and downs, and then there's this huge spike when the broadcasts have come on, because they've got like 60 or 70 people that are tuning in the broadcasts. And his network manager goes, "That's unacceptable. needs to change, which we all do as we're a network manager because we have to make sure the traffic can flow perfectly. So we started having conversations about the possibilities, and I knew that QuickTime's got to be one of the most amazing streaming services available. And I said, cool, let's set up a statewide Akamai. And, yeah, this is where we're going to start talking. Right. So what happened was, we'll go to the next slide, this is the setup where Patrick installed a streaming server, let me try and point with this thing, over at what's called Wayne Resa, this regional educational service association. They installed one QuickTime streaming server for one of these educational broadcasts that was being put out by satellite and they simultaneously took a feed and put it on a QuickTime broadcaster and started, this was a combined machine running the streaming server as well. And so this is the Merit Network, MishNet is the name of our network. And we have various affiliates around the state of Michigan who were tuning into this. And so what happened was the stream went across this one link from Wayne Resa to Mishnet, which happened to be a DS3, which was fine initially until they started to get many people tuning into this point and completely overloaded their outbound connection to MishNet. And again, the network administrators were hammering on the door saying, "What the hell are you doing? What is causing all this bandwidth?" So it turned out that also with this solution where the stream was going out across MishNet, there were similar choke points at various affiliate networks where multiple people were tuning in. some of the school districts that are connected.

pressing too fast, I'm sure. It's all the coffee you had this morning. I know. That's right. We'll get back. As I say, some of these school districts are connected with a T1, and so you had four users on there, and you overloaded their inbound connection to their network. So, the solution was to deploy a hierarchy of streaming servers with... Let's go to this slide. And so, that's what we did. And so, now we can have a broadcaster at any location on the mission, on the network, on the internet basically, there's one single, this green dotted line, it's a single stream, that is central master relay server and merit network, we're providing the co-location space and sort of the bandwidth, free bandwidth basically for this master relay server. And so this master relay server in turn just redirects the broadcast out to a number of relay servers in the different school districts around the state of Michigan. So that's minimizing the bandwidth going across those expensive links or for the school districts the expensive links to our network.

So again, with regards to multicast and unicast, MissionNet is a multicast-enabled network. And so wherever we can, we would like to be able to send a multicast stream to the relay servers. And we're working with our affiliates to see whether that can be done. So now we're going to talk about, or at least start talking about, the things we've learned. This has been an amazing journal over the past year. A few more.

Yep. I'd like to take the opportunity right now to thank the person that has more information and has had about QuickTime streaming server, which is George Cook, than I think anybody does. So if you get a chance, take a look at his site. But we ended up with, right now, we have 16 locations, 15 locations in the state of Michigan, including Copper Country. It was a nice seven-hour drive. up to Houghton, Michigan to set up that server. And the thing that's been most informational about this whole project is working with 16 WAN managers, right? Because now I have 16 people that are responsible for their own networks. Every network has been completely different and so we've had some serious growing that we had to do. So go on to the... So, um, okay. Are you, is there something you want to add? Um, just talking about working with these 16 WAN managers, the other interesting piece to this is that you tend to have separate people wanting to deliver the content on a campus. You have the content folks who are wanting to get this stream in, and then you have the networking WAN folks. And deploying a QuickTime streaming server tends to bring those two groups closer together very quickly. And that's something which is good. I mean, it's definitely an education for the network administrators. And as Patrick mentioned, he has been doing the legwork, driving to all these locations, installing these servers, and has done an amazing job.

Okay, quickly about the architecture. So some of the things we wanted to do is make sure we could broadcast from anywhere. We want to be able to eventually have students that are working on projects in an educational institution anywhere in Michigan broadcast. We've got a program going on at 11 o'clock, or a program going on at noon. We have actually had conversations about universities doing school nights. So instead of students having to go to the university, they tune into the school night event that's taking place in their local school district from MI StreamNet. So broadcast from anywhere. We want to be able to have the parent node-- what does that say?

push information out to the relays regarding the content of the video, which we'll talk about in a minute, but I've got this thing now that I call the Microsoft effect. And it's bad for me to say that nobody's from Microsoft. We're in an Apple developer conference. I can say this part. But this whole notion of servers do a good communication in terms of announced relays over ICMP. Right? ICMP. Right.

No, the pinging between the-- The pinging between the-- Yeah. And so there's some organizations where actually ICMP's been turned off. And so we have servers that can't talk with each other anymore, and they just, oh, I'm going to stop broadcasting now because I'm the only one who can talk to the child. Replicate, we're going to talk about replication of content over the servers over our sync. And then the thing that Laura's going to get to is how we made all this thing come together, the Akamai effect with the redirection script. The children notes, we wanted them to be able to have access to the data, meaning the video coming in from the streaming server via the STP file.

And when you start, is anybody running streaming servers? Everybody? The thing that I have to say is don't make this any harder than it actually is. For some reason, the hardest time we had in setting up the servers we were setting up is we thought it had to be harder math. We thought maybe we needed a pocket protector, some incredible slide rule, something that, would allow this to make sense to us. But really, this is just very, very simple in the way that it can be configured. Okay. The last thing that I think is, from Merit's perspective, being an ISP, and we were concerned about the... Whilst we've got these 15 relay servers around the state and users get redirected to pull their streams from these local points of presence in their various school districts.

We were concerned about how many other users might tune in who don't have a local relay server. And so what we put in is a second QuickTime streaming server in the Merit data center, which would be the catch-all server for users who are not redirected to a local streaming server in their ISD. And so we're sort of working to see how much bandwidth we're willing to donate for that and how much demand there is. But again, this is a useful tool because we can begin to see if more users come in from another school district, we can begin to go to that target, that school district, and say, hey, if you put a streaming server in your district, then you wouldn't incur all the bandwidth costs across your link to our network, and there would be a great benefit. Okay. So the one note that we have is we've limited the shunt server so that it allows up to 50 users.

Anybody that's outside of the service area will not end up getting shunt server. And really what we want to do is if it's a 51st person goes and clicks on that shunt server, what we should really do is put up a web page which says, you should go buy a QuickTime, or get your network, to buy a QuickTime streaming server and put it in your school district and then you could watch this. Yeah.

So some of the stuff we learned. The first time we set this up, we used announced relay, which is the easiest way to set up a relay network. I don't know if anybody's looked at expanding their networks or they're doing hierarchical broadcasting. But announced relay, very simple, put in from QuickTime streaming server. I'm simply setting up a-- actually, I'm going to talk about that in a minute. The first time we set this up, we used announced relay. So let's go on to the next slide.

And in an announced relay, the parents responsible for getting the content out to the children's servers, what I call children's servers, so they know where the broadcast is coming from. So a parent announces the STP file, and each one of the children simply tune into the STP file because it's sitting inside their movies/library/movies directory. Next. And this was a great solution. Very, very simple in setting up. streaming server network. We also found a very interesting feature built into streaming server in that it-- If one of the children was not available, the streaming server would simply say, I can't broadcast anymore, and restart itself every two minutes. So we ended up with these live broadcasts where the video is being restarted every two minutes over and over and over again. You just stop for two minutes and you tune in and you see a little bit more and then it would stop. Yeah. And so we had two things that were taking place. Firewall configurations, right?

Or, again, the Microsoft effect, which goes back to firewall configurations because there was no ICMP. One of the things I would recommend in setting up a distributed network if you're going over firewalls is make sure that you have a one-to-one relationship. The easiest conversation for me with the network managers, the WAN managers, was set up your parent, set up your child, and set up the firewall in between so that the two can do anything they want to do with each other. There is no restriction. There is no port mapping. It's just a completely open connection between those two devices on the network.

So we worked at this for a little while, and we continued having problems. So we switched to-- unannounced, basically give a name, give a port number. So this is on the parent server. The parent simply got a relay set up. And we'll see some in just a sec. And it simply says, I'm sending data to this port number at this machine. It doesn't even care if the machine's out there. It's just, here's video. You're going to tune into it. You can watch the video to your heart's content.

If not, I don't care, really. I'm just giving you a video. Go on. So this is the way the network is currently configured. I didn't mention in the first part, we have QuickTime's broadcaster running on a machine at Wayne County Resort right now. And we've simply taken the STP file off, QuickTime broadcaster exported it and put it inside the movies directory in the parent. Each one of the children have been set up so that they simply tune into the STP file. And we're going to look at the STP file in a minute so you can see what that looks like. It's again very simple. We thought it had to be hard. Patrick went and I hardly ever use Excel. I used Excel and I got this spreadsheet that just would cover this whole stage and it's got this flowing set of formulas where you can figure out port mappings. Oh my gosh, I would open source this thing, it was so amazing. It didn't need to be that hard. I still have it. We thought we had to have a unique STP file for each. I thought about printing it, but I really don't need it now. We copy the broadcaster file up to the server. One of the relays are configured inside the streaming server and we'll look at that and then video goes down to the children notes. So this is the way we're currently configured and this has been the best experience that we've had. Go on. One other thing to note, I mean I like to think of the way this is set up is that the STP file is the key. If you don't have the STP file on the child relay server, you know, the master relay, the parent server is sending the data down, and it's only when you get the key, that key on that local relay server, that the stream actually gets into that local relay server and users can watch it. And so it's a good sort of analogy. country.

What's up? Oh, so some of the things we like about this. The fact that if the parent ends up going away, the children don't lose the connection to the viewers that are watching the broadcast when the parent comes back online. If for some reason it goes down, the children will automatically pick up in the broadcast that they were showing to the viewers. And the same -- this is a huge note with the little pointer up in the third line up there. The same SDP file we use for children to tune into the broadcast can be distributed throughout the children.

So with our sync, we're actually looking at the ability to say we're going to have 20 channels, right, and channel one equals and with the information that we're going to have regarding local host address 127.0.0.1, we will distribute that throughout the entire network affecting all the servers. And we see this thing growing. We've got 17 servers, 16 servers now. We've got two more coming online. So it'll be nice to be able to have this easy configuration be replicable throughout the state. Next.

So let's talk about the way this works. On the left-hand side, inside streaming server admin, is the relays themselves. And there's some information you associate with the relay itself. And then next is, on the right-hand side, the actual servers that you're redistributing this broadcast to. So go on.

I have a clicker, but it doesn't work with that thing. But I wanted this. So if you're doing... So we set up the server so we have a relay name. It doesn't really affect in terms of what the user is going to be tuning into. We have a relay point, basically 0.2.0.sdp. And this is the mount point or the broadcast connection point. And the server ends up pointing it back to itself because we have the broadcaster sending video to a specific IP address on the network. We're not going to share it with you because it's not fair for us to do that. But we tell the server, tune in to yourself to watch the broadcast 'cause it knows that this 020.stp file has all the information it needs to be able to get to the broadcast at Broadcaster Sunday. Go on to the next one. And then within the relay setup, We have each one of the servers defined, either by IP address or by name.

This actually doesn't have to be a name that is resolvable by DNS. It can actually be a name that's sitting in the host file. And the port that this is going to be broadcast to. And as I mentioned before, we're talking about channel 20. This could be that all the servers for channel 20 are sitting at 21, 326 throughout the entire network. So the server simply says, I'm broadcasting to this UDB base port. Go on to the next one. So remember this number. UDB base port actually ends up being two ports. 21326 is the beginning port. And you will notice we assigned this to audio. We didn't. Quick time bit. And then video is assigned to 21328. This does not mean you can start messing around with the port 21327. Basically three ports for broadcast is what you're going to be using. And I tried to get crazy with the spreadsheet. I tried to figure out a way to maximize my number of ports because when I was thinking I had to have a look at George laughing, he had a fun time with this. He really didn't tell me this. I knew he didn't tell me. I'm making that part up. But I started to get crazy with how I was configuring the ports, and it turned out that the 327 is a, I don't know, sync port, something that is used by the server in terms of communication.

So this information really isn't pertinent to what we're talking about regarding the broadcast. talk to myself, look at these ports, and then start spreading video to the viewers that are wanting to tune in. Yeah. Yeah. When are the next one? The crossed out lines, I mean, you can delete those out of the STP file. So, that was, which were the unique things. It turned out that, go back one, it turned out that these three lines right here are extremely important and George can talk about that. I didn't have them and we couldn't have a broadcast and of course he's like, "Oh, you're messing with these three lines." Guy's scary. I don't know.

Okay, I'll talk about live redirection. to, as we said, we want to redirect the users to the closest streaming server local to their school district. And so what we've put together is a CGI Perl script and what will be a database right now is a flat configuration file which contains a mapping between a set of IP addresses, IP ranges for that school district and then the address of the local relay server to redirect the user to. The users can open up a URL, call the CGI, the Perl script, and they automatically get data back to QuickTime Player or to QuickTime embedded in a browser, telling it to open up the stream from the local server.

We also added, what I worked on was some additional logging, so we could log the, as users get redirected or do not get redirected, so we could begin to see how successfully, how well the coverage was of local relay servers. And again, we could have the ability to redirect users to a live stream, to an STP file on on a local relay server or to replicated movies or MP4s, whatever media we have replicated on the local relay, this redirection script will work with any of those. This is just a one-line example. We've just got the child IP address, the relay address, and then the client address range start and end. All right.

And as the script is called, essentially this is, you can see there's a curl statement. We're just showing that when the script gets called by the browser, you can use curl, this command line to make this call. This is what gets returned, this XML. And this is passed to QuickTime, either the QuickTime player or to QuickTime inside of the browser. and does the necessary redirection, passing the redirection to the local relay IP address followed by the name of the STP file.

This is showing the embed command, how you would embed the call of the object, embed command to the call. to embed the call. This is our master relay server and we're calling this Perl script, my streamnet.perl with the name of the SDV file. And what results from that call is this a redirection to the local server. Thank you.

We're going to talk, yeah. So what we wanted to do was, as we rolled out these servers, we wanted to do a whole set of testing and load testing as much as possible to make sure we that it would run under load. Certainly we wanted to stress the redirection script. And so we developed-- there's a Python script which George put together to stress test the redirection script itself. And then we also use streaming load tool, which is part of QuickTime streaming server. Is it part of the release? Yeah, it's on every server, isn't it? So do you want to talk a little more about that? So two things I wanted to add. So we've been talking very positively about MIStreemNet. And it worked incredibly well the last two broadcasts that we had. But we actually were at a point where we had four broadcasts. And all of them ended up with some part of the network falling apart.

So we were needing to find ways-- brought George out-- to figure out what the heck was taking place. And if anybody thinks that Mac OS X server and QuickTime streaming server and an XServe is not an incredibly powerful tool to use, I just have to let you know what we saw these things capable of doing. MiStreamNet, our parent streaming server, We were simulating 500 broadcasts along with 17 relays. And it just ran for like hours and hours and hours and hours and hours. And this is great. We thought this is amazing. We got a 10 megabit connection to the network. Or 100 megabit. Yeah, 10 megabit. We got a 100 megabit connection to the network. And we got 500 users that are watching this video. Plus we're doing video out to relays. I think overall we simulated 2,000.

2,000, 3,000 users on the net because we started stream loading each one of the relay servers or the children's servers out on the stage. And it was great. It worked. The only problem was we had not gotten to the point where the server failed, which is what we were trying to find, until the day that we got Ken Schramm, who's been the most wonderful person to work with. He grew up in the video world, right? In the video world, if something fails, you get rid of it. There's no three strikes and you're out. is something fails in the video world, it's just done. Dead. Don't talk to me anymore. And we were bringing him up in this electronic world, which is a little bit different. And as soon as I got Ken on the phone, I'm like, Ken, check this out. We're doing these broadcasts. Working just great. We got things figured out because we reconfigured some stuff. And as soon as he got on the phone, he's like, oh, that looks wonderful. Bam. And my StreamNet 1 goes down. Just, and we had to go actually restart the machine. Well, it turned out, and the reason I'm bringing this up is Streaming Load Tools is a great application comes, a streaming server, you can simulate 25 to 50 users per connection hitting the server in real time. So it's sending out the bandwidth, each one of those connections.

And so, long story short, turned out that we ended up with a motherboard in our server that had a problem. Only related to power changes that were taking place in the environment that it was in. So it was like one of those things you would never find. But we ended up with a guy putting a new server in the server room that we had our MIS Stream.1 in. And he turned on a server. And that little switch changed the power enough for that configuration that the server was just like, come out of here. Done. See you tomorrow. So we got a new motherboard.

Things have been working perfect. ever since. Going back to talking about working with this video production person, with any of this, when you roll out a service like this, it's important that it works. You don't get many chances with this. You want to make sure when you roll this out that it runs because if you're dealing with video production folks from that world, they really have a very low tolerance for failure. I mean, and so it's important to test and... So streaming around with a load tool became a great solution.

Is the Python script available at the site that you have? So basically the Python script is what we use to simulate 100 users per second, hitting the redirect script to make sure we weren't having some kind of memory problem with Apache or the script we'd written to do the redirection. And the thing that ended up in the end giving us the information we needed was the server monitor tools.

Anybody using an XServe, The server monitor tools are incredible for what the information it can provide. Some things that I would recommend, all services we're running on the servers, anything that's not necessary is turned off. So anything that's taking place in terms of file copies we're doing over SCP. We've actually set up firewall rules so that the things that we need to be public, like 1220, we have that open to the world so we can get to it from anywhere. 660 is what we actually have for server settings on 10.2 Server. We're going to be up to 10.3 by the end of the summer, which will take us back to 3.11. And then because we're using SCP and SSH, we actually have the firewall set up so that there's only one machine on the internet that has the ability to administer these servers over SSH, which we're not going to tell you because you might try and get into it. So I would highly recommend to not run as much on the server as you possibly can. Lock this thing down because it's sitting out in a public place and this thing doesn't work. What's the next one?

This is where George talks. It is. We'll all say it one stage. So streaming mode tool comes with with both Darwin and QuickTime streaming server and it's a great tool for stress testing and as these folks said the main thing here was testing you know? And so and I see some other folks I know in the audience and and they know that that's that that is the critical component here is that when you're rolling out something a new set of services on on the network, especially something new like streaming. The important thing is do some stress testing. Make sure that you've got things under control before you do a broad-scale deployment. So some of the things we did was each instance of Streaming Load Tool we set up to do 50 concurrent clients.

We have multiple instances running. We have three different machines running multiple instances. And we've loaded the heck out of the server. And we're amazed at the performance. And I recently, Simon again from Wirecast was showing me a Swedish newspaper that is, really loading the heck out of the server and again another one to talk to about what they're doing there is even more amazing really. Okay so. Let's move on. Do some demos.

replication, these are the next steps. So the next things we're working on is replicating content. Mac OS X, the Unix layer has this great replication technology built in called rsync. So this is an rsync command that you can, that we're using right now in a test mode to synchronize content between the servers.

The important thing here is that this synchronized stuff all works with the redirection script. So the redirection script doesn't care if it's an SDP file, a movie file, an MP4 file, whatever it is, it does the redirection. We're also working on some monitoring tools and Lori is really excited about putting some of this stuff into a SQL database. I've just been doing flashy, quick time stuff so far. So this is something we're working on for local administrators.

The idea is the local WAN guys that are out in these school districts, we want to provide them with a tool that they they can use to monitor the server, but not give them full administrative privileges to the server. I think, again, monitoring, especially during the testing phase, being able to monitor and know how many streams are actually running on the child relays. If something does go wrong, you've got some logs that you can go back and really begin to figure out what was the state when-- We're also-- Patrick and I are planning on setting up Moodle, which is an open source course management system. It's a very nice one, beautifully on Mac OS X server for all these people that are becoming involved in the project. So as more people become involved, the faculty member wants to know how to produce content, place it on the server. There will be a website for them to go to to get educated on that. And so that's another work in progress. Quicktime streaming server publisher, we're really excited about the new version. One of the new capabilities that's coming in the next version is the ability for end users to use the tool. Right now it requires an administrative user which limits the, the scope of its use, but with the capability for end users to basically publish streaming content or downloadable content to their own site streaming folder through this tool is just phenomenal and we're very excited about that use because it will ex it'll extend the use of these child node servers so that they can be used for locally hosted content.

So we think that's a really great thing. So. Demo. With that we'll do a few demos and how are we - How long are we doing on time here? - We got 15 minutes. - We have 15 minutes left. - Okay, let's bring up the demo machine, we have it.

So, My StreamNet. This is the folks at Wayne Resa, who are the third partner of My StreamNet with Merit and Apple, have got an amazing web designer who's putting together this website. And this is the live streaming page of the website. And so we're giving the users the ability to view the stream within a quick time in a browser window.

Wait, let me make sure. And so -- No, that doesn't -- What we're streaming here right now is as a sort of a test, we're taking the Annenberg video, Annenberg educational video, which is being streamed in over satellite, and they're taking this, feeding it through QuickTime Broadcaster at Wayne Resa, and then streaming it out over QuickTime over my stream net. And so we have a great ability to test at any time. It's a good promotional piece for new users in the state of Michigan to see this, or educational users.

And so the other way, you can also click and view it in a QuickTime player window. The other piece, we're also saving archives. Maybe this isn't working. And so we have all the archives of previous broadcasts that we've done over the last year, and users can view these at any time. So do you want to show the... Sure. Are we working here? Yeah.

got their eye here myself. So another thing that we've been looking at is some of these administrative tools. So what I did was I created some wired QuickTime movies that talk to this thing called the admin protocol on the server. So on the back end of the server there's this thing called the admin protocol which is used to do all the administration and so I wrote some CGI's that talk to the admin protocol and then wired them into some QuickTime It sounds crazy, right? So the first thing we'll do is we'll bring up this little graph thing.

So this is graphing bandwidth usage coming from the server in real time. And I can change the interval of the graph. And I can change the bit rate. But while it's doing that, let's go ahead and we'll start up a stream from the server. So we have this movie. So we'll go ahead and start this up.

And you see that the bandwidth jumps up. Now, it jumped up a lot there in the beginning. That's because we're doing the skip protection. And we're sending a whole bunch of packets down. And then it will start to settle down as the buffer fills. So it's kind of a cool thing to be able to watch the bandwidth in real time, right? So that's one little hack. And what's neat about this is that the admin protocol that all this stuff is talking to, Laurie's really excited about logging all this stuff into a database. I like the cool, whizzy demos myself.

I like the demos as well. So here's another one that we've done. So I'm going to go ahead and set this to one minute. So it's polling now every one minute. This is the number of connections, RTSP connections, or we can look at HTTP tunnel connections. We can look at the RTP bandwidth. So let's crank this back up to five seconds. So now we're looking at the individual nodes around the network and seeing what the current bandwidth is. And this is the main-- This is the one. The main server here. So just some interesting things that you can do with wired QuickTime. So thank you, Totally Hip, for your tool, because it allows you to do all these things. And then we actually have this web page, but I left the password over there. If you want to see the web page, it's just talking to CGI's, and you can actually drill down on the individual sessions. You can see what IP address somebody is coming from and what piece of content they're watching with the admin protocol. So you can get that level of detail with this admin protocol. Very exciting in terms of how we log this stuff, how we track it in the future. Okay, so I think that's it for us for right now. And what I want to do is invite up John Anderson. And John is on the streaming server engineering team. And he is largely responsible for QuickTime streaming server publisher.

And there's a new version coming, as I said. What's exciting to us about this tool is to really leverage these child node servers to be able to take those servers and in the school district, allow faculty to give them a very easy use tool to publish their own content, locally hosted content to these streaming servers, which is something we found very effective at a lot of schools and now we have a great new tool to support it. Thank you. It's a little weird with the microphone. It's like karaoke or something. I'm going to go ahead and close this. As it was mentioned before, one of the big features in QTSS Publisher 2 is-- oh.

Oh. OK. As was mentioned before, one of the major features in QTSS Publisher 2 is that anybody can log in to their home directory and publish content. They don't have to be an administrator. So that's a great new feature. I'm going to go ahead and start this up and log in.

There's also been a lot of great performance improvements in the application, and, um... It has a new UI now. So if anybody is familiar with QTSS Publisher 1, you had all these different modes that you had to go through where you were uploading your media in one mode, and then you'd go to the settings and set that up, and you'd go to links. At this point, everything is all in one place. Another thing that's nice, that's new, is if you're binding QuickTime Streaming Server to specific IP addresses, so that, for example, you can run on port 80 and then run your web server on another IP address, you can actually pick which IP address you want to publish files through. So that's really nice. Also, if you have multiple virtual hosts set up in server admin for the web server, you can pick any of those sites here as well. Thank you. And then, of course, if you're an administrator, you get a choice.

You can work either out of the system library, or you can work out of your home folder as well. So that's kind of a nice feature. And so what I'm going to do here is I'm going to go ahead and open on this up and upload one of these movies. And it's real simple. All you do is just drag it into the library. That uploads the movie and in addition, it will hint the movie as well. So it will add hint tracks to the movie automatically on the server.

So now what I can do is I can click on this movie. Now, when we put files into the library, they actually go into a location outside of the QuickTime streaming server folder. And the reason for that is that you may want to make a movie available only via simulated live broadcasts or playlists.

And so that gives you the opportunity to do that. But what I'm going to do is I'm going to go ahead and check this checkbox, which is Publish Media to QuickTime Streaming Server. And when I hit Apply, what this is going to do is It's actually going to create a SIM link in the QuickTime streaming server folder. It will also go and create a ref movie that points to the, um, to the movie on the streaming server. So it does all of that automatically. You get down over here, you get a URL that you can copy and paste into the player, and then you can also just click play, and the movie's up and running. So it's very simple streaming.

And then of course if you want to set up a simulated live playlist, which is great for relays, something that you can then relay off to other servers, all I have to do is click on this plus button and I'll just name it playlist. Much like iTunes, you just create the playlist and drag the file to it, and that's it. If you want, you can change weights on this. So you can do a weighted random playlist, and that will actually play some tracks more than others. And then you just start it. Now the cool thing is, I'm going to go back to the file.

Not only is it publishing those files and making them available, you can change the annotations any time you want, and it will re-hint the movie when it's done. And in addition, down here, you get all of the HTML to embed the movie in a page. So basically, you can choose whether you want to, for example, open the movie in QuickTime Player or have it autoplay. And once you choose that, you have HTML right here that you can copy and paste into Dreamweaver or BBEdit or any other web page editor. In addition to that, I can actually create a web page directly from here.

And this is a development OS, so this may or may not work. We'll have to see. But I just go ahead and create a page, and we'll drag the file to it. And this will actually generate a page based on an XSLT template that has the movie in it. Now, of course, I haven't set a poster frame, so it's just defaulting to the first frame. But if I click on that, then that automatically brings up the movie. So-- So that's Publisher.