Hi everyone. My name is Antoine. I work as part of the rich media team in the Internet Technologies Group at Apple and today's session will be a hands on session, so mostly focused on code. All about HTML 5, Media support and Safari 4.0 and Safari in iPhone 3.0.

And all the code that we'll be going through today is actually already available to you from the attendee website, so you can just log on to www.apple.com/wwdc/attendee, go to session 205 and just download the code. So once this session is over you'll be able to reflect on what we learn today, look at the code, play around with it, modify it any way you want to and well basically you'll have a ball with it I guess. And well you'll be required to use Safari 4.0 to be able to view this content.

It's the minimal as far as that will be needed for this. I'll be joined on stage a little bit later by my colleague Eric Carlson, who has really been taking care a lot of the engineering in WebKit for media support and he'll be giving us some great demo's later on.

And the session is really about learning how to use these new technologies for practical integration because all these key technologies are really going to be able to give you a great media experience in the browser and there are really three things you need to worry about. The first thing is going to be the video element itself and that's what gives you media in your webpage.

You just drop in a video element, point to your resource and it will start playing right away. That's very easy. We'll also have the audio elements in HTML 5 and we do support that in Safari. We won't be talking about it much today, but all of the technology, attributes, properties and etcetera that we'll learn today about video, directly applies to audio.

So you'll be learning about two things side by side today. And so these video elements integrate really nicely into other web technologies. For example the DOM and as media plays and through the life cycle as your media playback on your webpage, you'll be getting events through the DOM as it plays back and we'll be leveraging those in code today to do a variety of things. We'll be tracking the media state in particular. And finally the other way around is when you want your script to interact with the video and you'll be using Media APIs again through the DOM.

So you'll get a pointer to the video element and you'll be able to program it right away using simple DOM calls in JavaScript. OK, before we get into the code which will really be the bulk of this session, I'd like to put the whole video effort, the video element effort, in context.

And the context is really that the web today is all about media. It can range from news website, using video for newscasts from websites like YouTube are all about user generated content, Internet radio websites are providing TV experiences and etcetera. And to give you an idea of how much media is on the web today, you just look, you just have to listen to Ryan Junee [assumed spelling] from YouTube who was recently sharing some metrics about how much content is uploaded to YouTube every day and as of today, which was actually a month ago so might be even higher, 20 hours of video are uploaded to YouTube every minute. So I can hear the gears spinning in everyone's heads and it's actually 3.3 years worth of content every single day.

So not only there are a lot of people watching stuff, I'm sure all of you here are watching YouTube pretty often, it's also about people creating media and that's what makes media so important on the web today. And so that's why we're doing the video element really. It's that media on the web right now is all about plug-ins, and plug-ins are cumbersome.

Plug-ins don't integrate well and plug-ins are inherently limited. And if you think about it, the most, the first type of media that we had on the web was images and do you use plug-ins for images, no you don't. Because the browser supports that. It's pretty obvious. And really the video element is all about this. It makes it so much easier for you to author video content. You just put in a video element, just like you would an image and that's really all you've got to do.

And it's also about integration with broad technologies. A plug-in is really just a black box. Whereas an element inherently will integrate with anything on the webpage. For example, technologies like CSS, cascading style sheets, so you can style video elements just like you would style any other part of your webpage.

But also the DOM so you can interact and program your content directly with JavaScript. But it's not just about how easy it's going to be for you developers and authors to create content, it's also about the experience that the users will get and having a high level technology like the video element, lets implementers like ourselves, take it and integrate it in a very optimized fashion in computers and devices.

Because you need to remember, it's not all about 8 core Mac Pro's, how about iPhone's for example. These are a lot more constrained. Very efficient at certain task though, so in with the video element on iPhone, we can take this and implement it in a very efficient way on a limited device as well as a super powerful powerhouse of a computer. And it's not just about this.

It's also, a video element tells the browser here's a video. A plug-in just says well here's a piece of content, something else will take care of, another process, you don't know whatever this is. It might turn out to be a video but it might turn out not to be a video.

But the browser knows that the video element is video and that means the browser can provide a built in experience for video playback that will be really rich. So for example on iPhone, when you play back a video element, you'll get the gorgeous full-screen user interface that you would get to play back any media on any native iPhone application.

And we'll take a look at this in a minute. And really a key point and I think it's a mantra that whole teams try to keep going after, is that media should be a first class citizen on the web. Just like images, just like a lot of other things, media should be the first class citizen on the web and I'll probably say that a few more times this session and it's really what keeps us going with this whole initiative.

And to do that we need to make it part of Open Standards. That's how technology innovation on the web should be done and HTML 5 is the key driving standard for all of this and it's a joint effort between the WHAT Working Group and all of the implementers out there, including Apple. Obviously at the forefront of this technology.

And so all of this started way back in March of 2007, with a f irst working draft of HTML 5 and very soon after that and later this year, we already had experimental support in WebKit [inaudible] on both Windows and obviously MacOS X with QuickTime support and other browsers did it as well. That's the whole point of adding web technologies to our standards.

So Firefox and Opera also had experimental versions. And WebKit is an open source framework, it's not just for Mac and Windows and people that work on the GTK Portal also made sure they have a nice media backing for media on that platform. And already in January 2008, less than a year after the first draft came out, Safari 3.1 was the first commercially available shipping browser to support video element, both on Mac and on Windows. And we kept going. Other contributers in the WebKit sphere, like the key people provided full support and earlier this year provided the first [inaudible] preview of Safari 4.0 which had greatly enhanced video support, matching the latest iteration of the specs.

And very, you've heard obviously about Chrome, which is a Google browser using WebKit again and these guys have made sure to provide also support for the video element. So you can rely on this browser as well. So you've seen almost every browser that you get to which one is the missing element here.

And that brings us to today and today we have Safari 4.0 and Safari on iPhone OS 3.0, both supporting the latest and greatest in video technology with the video element up to date with the last spec from the WHATWG and W3C. And Firefox 3.5 is about to come out and also support the video element.

So all of this made it possible for early adopters to start kicking in and use this technology and this will be the case of all of you after the session right, we'll know all there is to know about video. And a website like YouTube, not exactly a lightweight, if I can say so, actually showed right here a couple weeks ago in Google IO, an experimental version of YouTube using only HTML 5 and this speaks volumes about how ready this technology is.

Because their demo was literally the same experience that you would get with a plug-in. They could reimplement exact same features, except they were only using the video element and some JavaScript DOM APIs and they actually even have a few more features and you should go and check it out, youtube.com/html5 and I think this really shows that we're all on the right track.

Not only from a technology point of view, but also we already have early adopters that make a strong case for video on the web. So I think that pretty much sets the scene and I'd like to show you right now the demo that we'll be building. And so I was just talking about YouTube recently and YouTube gives you a video experience that is very custom to YouTube.

It, nothing else will look just like it right? So what we want to do here is we want to create our own custom media experience. So here if you've been to the CSS Effect session yesterday, in the same place at the same time, you seen a lot about Wilco.

And Wilco is this amazing band and they have provided us with a lot of access to build demo's for this WWDC and they have a DVD that's came out that's come out called, Ashes American Flags and we took a song of it for the purpose of this demo, you also get in the sample code so you can play around with it and you'll have the whole code to build this.

What this is is the video playing front and center on the page, but also providing custom control that look and feel like the rest of the Wilco website. So you notice the types are the same, using custom font, blocky, that's really the kind of design that Wilco wants and that's the kind of experience we want to provide. But we do a lot of pretty advanced things. Like for example we can Scrub, we can Seek and Jump through Times, we can control volume with a slider. You notice how smooth and easy interface is with traditions and nice you know, sophisticated fit and finish.

So that's what we're going to be building today. And although it might look maybe a bit complex, you will see it's actually fairly simple to implement all of this. And so our first step will be about what I call setting the stage. It'll be really just starting with our page without any video and just putting a video element and make video playback happen in our webpage. So this is what it's going to look like. I'm going to have this video right in front center of our website. We'll hover on it and we have the default QuickTime Lite controls.

And we'll be able to Seek, Scrub, Play, Pause and etcetera and these are going to be build in controls provided directly by a platform. You will be writing very little code here. OK and our webpage is very simple. It's just an HTML page linking to a script file and a style CSS file as well and we'll be going through adding code to all three of those files as we go along. So let's step into the code right away.

So here's I'm going to load my first step and I'm going to bring the HTML file in the browser and let's see what it looks like. OK, we have basically an empty area for our video. And if we look at our code, we just have a DIV with a class video player and that's where we'll put all of the structure we'll need for our video player. So right now it doesn't do much, links to the script and CSS files and I'd like to take a look at these. The script file doesn't do much, resize the window just so it fits nicely for the purpose of this demo. Nothing major here.

And the style just provides the basic styling, like the backdrops on the background and you know, some basic positioning and sizing for the video which will be width and height 100% and you can already see we'll be able to use styling for the video. OK so first thing we've got to do is remove this and add our little piece of the code to get started. Oops, OK, accept. Except that doesn't, oh sorry. Classic. So we drop in a video element and it looks exactly like an image element.

For example, an image points with a source attribute to the image on the web and here we're just using source attribute to point to our video. And if we save that and go straight back to the code to, sorry to the presentation, we have the first string of video sitting there but it doesn't know how to play and it doesn't do anything at all.

So what we want to do next is to make it play automatically and there's a very useful attribute here called autoplay and if we just turn it on, we got autoplay going right away and what autoplay does is as soon as the media has enough content loaded, it will start playing back and you don't have to do anything in particular here. OK so, we don't have any controls showing so it would be nice to have that going as well.

So let's just add a little bit more code, simple control attributes, reload the page and now we have our built in controls. And with a very small amount of code, you basically have your video experience right in your browser, right now. OK let's see what it looks like on iPhone. Because we got great support on iPhone OS 3.0 as well, that's brand new to that version.

So let's drop our HTML files right into the simulator here and let's look at this. So it seems that there is a video playback that might happen here, but it's a default appearance. We'd like it to look tailor made for our content. For example, we'd like to show a poster frame. We'd like to show a frame from the video that would make it clear to the user what kind of content they'd be looking at.

So lucky for us there is an attribute just for that. The poster attribute lets you point to an image on the network and that will be displayed instead of the video, until playback actually starts. So we reload and we have this great frame that we already prepared beforehand and if we start tap on it, we go into the built in, great looking media experience on iPhone, you can Scrub, you can Play and Pause and etcetera. And once again, we did almost no work and when we tap Done, we can just go back to our webpage.

Thank you. So that was really easy and in fact, I wouldn't be surprised some of you walked out right now, that's all you need to know right. OK well please stick around because you know, we'll be doing all the rest of the cool stuff. And what we'll learn about today at this stage, is just about the core attributes from the video element. So we know about source obviously, that's really simple.

We know about controls as well, which lets you put on the built in controller, autoplay starts playing automatically as soon as enough data is loaded without you needing to do any kind of scripting technique or anything like that and also the poster attribute which lets you point to an image that will just be displayed instead of the video until it starts playing.

And there are other attributes that we don't talk to you about that I'd like to mention briefly. If you don't have autoplay but you still want to keep loading the data right away, you can use autobuffer to start loading the data even though you don't play yet and loop, which just loops the video indefinitely.

So there is however the small problem about browsers or browser without any plural form, that do not provide built in support for this great technology and we actually have a good story and good scenario to handle these cases. Any of the code that you would have right now, it could be plug-in or it could be code that you want to display to say well you know, you really [inaudible] browser because this is just not a good experience, not good enough to enjoy this website.

Well you can take all that content and just put it as a child of your video element and that means that if a browser or the browser that doesn't support video, encounters this, it can just fall back to whatever content is underneath that. So in that case we'll just fall back to using a plug-in to display our movie. You could display an error message of some kind here as well.

And we won't go into a lot of detail today about this, we want to focus on things that actually work and Vicki Murley our web technologies evangelist will spend a lot of time talking about building compatible website using web technologies like this one, tomorrow in Russian Hill at 5 pm and she'll be focusing a fair bit about video. So go to this session if you're really interested about this and you'll learn all the various techniques we have for fallback.

OK so that was step one. Let's get into a little bit more serious coding. And step two will be about providing the basic media controls and starting to learn about events and the media APIs that we have. And so what we will have here is simply the Play and Pause buttons, with just a toggle and a Volume button and these will control the playback state and the mute state.

OK, so that's all we've got to do here. Pretty simple. So let's step back a little bit of theory here and you've got to listen up, even though it's theory, because it's really the key point that will be driving the entire presentation, the entire code we'll be building and what matters with media is the evenflow. The media engine is not something you can only talk to with APIs but it's also something that are going to keep you informed at all times of what's going on inside of it.

So in a typical webpage you load it, it has a video. As it gets loaded the video, we get a request to the cloud saying I need that video. The video data starts coming into the browser and as soon as it starts going in, we are already getting events and we can get those DOM events straight back into the JavaScript and these events might be progress events.

For example, I've got [inaudible] of data, it could be load events saying I've got everything or it can be an event that says I've got enough data and starts playing. So if that were the case we could go back into the page with media APIs and start playback.

So our video starts playing, but we keep getting data so we'll be getting more information from the cloud and more progress into the JavaScript, in which case we'll be able to provide more information back into the video, more interaction. And that goes on and on. So that's the key design power we'll be using throughout this session is about how you respond to events before you want to program your video and vice versa. OK so let's go back into the code for step two which is all about providing the simple media controls. So at this point we'll be focusing only on the desktop.

Let's load our second step. OK and let's see what this looks like in the browser. OK so at this stage we've got our video still and it's not autoplaying anymore. We removed all this and we've got buttons but they don't do anything at this stage. They hover and etcetera. That's just things you can do easily enough in CSS, so let's look at our HTML. We don't have the autoplayer controls anymore.

We got rid of those because we want to provide this experience ourselves. Sorry, and we've got some basic controls wrapped into a DIV, Play button and a Volume button. OK and in the CSS we'll notice that we'll drive the labels of the Play button basically using the content property and based on other selectors. So whenever the button will be under an element that has a playing class, or a pause class, it will update it's state. So the states will be CSS classes, they'll be driving the presentation.

And if you'll go into our HTML just to make sure we actually have a label saying that we can play because it's paused, we have the pause class put onto our video player and the video player, our container, will always have classes applied to it to update the visuals of the presentation. So that's an important point before we get going and I want to clear that up.

And in terms of scripts, we've just got a little bit more already in there. We have a video player class, that will be associated with a container, so it will be pointed to our video player here and we'll keep track of that container and a few other objects. For example the video.

So we get a pointer to this video and that will be the object in the DOM that will be triggering the events and be implementing all the media APIs that we'll be able to call. So all the code I will be writing will really often be interacting directly with the video.

Now we got to point it to the Play button and the Volume button. And we also have a handle event method implementing our video player and we do that because it's a built in feature of DOM events, if you implement this function on an object, whenever you have an event listener to use this object's event handler, this will be the entry point.

So that means that you can have event handling local, always scoped to your instance object, very powerful. OK and finally we just instantiated that video player with a reference to that container so we actually get it on screen and hook up the controls to our video. OK so at this stage we don't do much and start to put up some action there. So the first thing we've got to do obviously is we want to listen to click events on our buttons. OK so Play button and Volume button are now listening to events for click and using this, our instance object, as the event handler.

This means all the events will go through handle event. OK we're set up to switch based on event type, we'll be dealing with a variety of events here. So the first one we're interested in is click. OK so we have two different targets, Play button and the Volume button and based on that if we hit the Play button we want toggle play state, if it's the Volume button we want toggle mute.

So let's implement those methods, these will be very simple. First one is toggle play state and that's the first time we're going to see a property from the DOM APIs for Media Elements and that one is Pause. If we look at our video and query if it's paused, we can know what kind of action to take. If it's paused we want to play, if it wasn't paused we'd want to pause.

And to play it we simply call the play method on the video, if we want to pause we simply call the pause method on the video. OK, Mute, Toggle Mute, is going to be very similar. We just look at the muted property and assign it's exact opposite so that it will actually toggle. OK so we should actually have enough information now to get the basics of our UI working. So let's play.

It plays perfect. We can click again, it will pause and hitting this, will mute. OK but as you'll notice, the buttons are not actually in sync, the UI doesn't change when we click on it. OK we haven't actually done any work for that, so that will make sense. So let's go back into our code and what we need to do is update the classes on the container so that it changes from playing to pausing and etcetera so that the CSS have changed to update the UI.

OK so traditionally I think what you would do is just say OK, so when we toggle the play state I'm going to update the playing or the pause class in the container. Well that would be the wrong approach in that case. Because the playback state in the video may change at any given time from a variety of sources. It might not just be the user interaction that you're providing, it might be for example the network stalls, there's not enough data, it stops playing.

And in that case what do you do? You don't' know from that simple code that you need to refresh the UI to match that state. Well that's where we're going to be using those DOM events because the DOM events are always going to be feeding us information about the state of the media player. OK so let's go back. We're adding event listeners on our buttons with click events and we can do something extremely similar for the video.

We're just using DOM APIs. OK so our video implements that event listener like any other element and we'll listen to three events on it. Play for whenever we start playback, Pause whenever we move to pause state and Volume change whenever the volume changes. In our case, whenever we'll be muting. OK so we need to handle those events to actually update our UI.

So we'll start by putting a play and pause event handler and we'll also want an event handler for volume change. And you'll notice that all three of them do the exact same thing which is to call reflect state in CSS, all we'll do is make sure we have the right CSS classes on our container.

So let's implement that function. OK so reflect state in CSS starts with a video player class which we always want to have in our video player container right. If it's muted it will add the muted class into our classes array and if it's paused it will add the pause class or otherwise the playing class and once we're done compiling these classes, we join them with wide spaces and assign it to the class name of our container. OK so at this state, at this stage we'll have all our states updated directly when the playback state changes for example and here we go.

Play/Pause, that works. And Mute will display our strike through state that we have already set up in CSS. OK so we made these events here, the pause events will let us update the UI and etcetera, you always want to tie the user UI updates to an event. That's really key. OK so that means that we don't really need our pause class here because now we are intelligent enough to do all of this class management ourselves.

So what we need to do is instead of that, have a call to reflect state in CSS right here and do it whenever we instantiate our controller. OK so we reload, everything looks fine. That's great. So you'll notice this one feature we used to have that we don't have anymore.

We don't autoplay. We have to let the user actually click Play, which is you know, it's a decent experience but that's not what we want in our cases. So the only thing we're going to do here, well we just listen to another event, it's all about events. So a useful event to listen to is Can't Play Through.

That means that the engine has enough data, it thinks it can play through all to the end if the network conditions remain stable. OK so we'll listen to this, we'll just add yet another event handler, nothing major here and all we'll do is call play on a view. OK let's see how that goes. Reload our page and it start playing automatically.

And you'll notice here again, the UI was already wired up correctly but the call to play was not coming from the click interaction for example. And if we had put the UI sync into the actually click interaction, we would have had to write extra code to handle the case where the media starts playing automatically.

So relying on events will make sure that you always reflect the correct state. OK so that's done. That's all we need to do. And we just learned about a slew of new events and there are lots of media events and we're not going to cover all of them so all the ones you see dimmed are events that we haven't talked about or we might even not talk about.

And the ones in orange are the ones we already talked about. Can't play through, tells you that you can't basically autoplay. Pause and play, playback state has changed. And volume change, the volume's changed. Simple enough. And converse, we have APIs to control the engine, not just let it tell us about it's state, but also control it.

So play and pause methods for example and immediate [inaudible]. We'll also learn about some key coding techniques here. The first one is that you want user action to update the media engine and that's it. When we get a click on Play, we just play and when we get a click on Pause, we just pause. We don't update the user interface. We only update the user interface when the media engine tells us it's ready to do so. And that lets us update the UI at any time to be in sync with the media playback and to always ensure that's the case.

And finally, a really interesting design pattern is that our code in the JavaScript knows very little about the presentation. All it knows is about states and the CSS will be providing the logic and styling to display the right information. So the JavaScript doesn't know it needs to display a Play or Pause label for the Play/Pause toggle button, but the CSS does and that lets the script handle interaction and logic and the style, well handle style and presentation.

OK so let's go to step three now and crank it up a notch. We'd like to provide progress bars. That's obviously pretty standard thing you provide in a media playback experience. And what we'll have here is we'll have two bars. One that will display how much data has been loaded and one that will gradually fill up as we play, you can see it up on the left, as we play through the content and we want to have to make sure that this stops whenever we pause and this resumes whenever we play.

OK so let's get right back into the code. So let's load our step three here and let's look at what it does so far. Well there's little new. It has the same behavior, autoplay, play and pause still work, muting as well and it has basically a placeholder for our progress bars.

Our progress bars will always snap to the left and right edges of the Play and Pause buttons, which means that you know, it's kind of cool with CSS, you can make things very flexible. So if we end up having a bigger video later, we changed the assets, the styling in CSS will be very flexible and just update the size of the progress bar.

OK that'll be very efficient. So let me load the right code for this. I'm sorry. Let's look at some of the changes we have in our HTML here. We add a little extra structure to be able to deal with our progress bar. So we got one for the loaded bar and one for the play bar and also we changed our source and strictly for demo purposes, we have a CGI script that's going to throttle the data rate for our video, sorry the network rate for our video so that we can actually see progress as it's come in as opposed to something really fast for local video right.

So just ignore that. It's only for demo purposes. And we have CSS already in place for our progress bars and one interesting thing is we'll be updating basically the width for loaded and played dynamically and we made, you'll look at this yourself, we made it so that we'll be able to simple percentages to size these bars.

So the script will only say well, I'm half way through the bar will be 50% width. That will be very simple and once again we'll let CSS take care of the complexity of handling sizing and etcetera. OK so if we go back into our script, we've got a few minor modifications.

We've got a pointer to our loading bar and we've got a pointer to our play bar, well once you update those obviously. And something we're going to want to keep track of at all times is the duration of the video. Because whenever we'll be filling those bars, it will always be a ratio based on the duration of the content right.

OK so the first thing that we want to do is have our own instant property duration, set it to 0 and we want to cache this property, instead of always querying the engine whenever we'll have an interaction that requires knowing what the duration is, it would be nice if we could just cache this information whenever it changes and well obviously there's an event for that right.

Duration change will tell you whenever the video duration changes. OK so we need to handle this event just like we handled a bunch of other events. Let's put it right here and we'll call got duration, which is a new function. OK let's put it in right here. OK got duration and all we'll do is query the duration of our video and just cache it.

That let the engine do what it has to do and not have to worry about providing us the duration each time we ask it for it. We can do it with a cached value which will make it more efficient. So this is we don't really do anything yet. What we need to do is start tracking progress and this is again, all done with events right.

So another event listen for our video, one for progress and one for load and we need both because load just tells you when everything is completely loaded and you might think well we don't need that, we can just infer that from progress events right. When progress is 100% or whatever, that means we've loaded everything.

But in the case where your video is actually cached or if it's local, you wouldn't want to get progress events, it's instantaneous, so you want to have both events so you can deal both with network conditions and local conditions, which is our case. OK so we need to handle those events now, so once again just more event handling. Progress and load both call the same function. Got more data. OK it's all about data events.

So let's handle those right here and whenever we get more data, we'll look at the buffer property in our video and it just tells us how much data is there and it has various ranges depending on which parts of the video is loaded. In our case we'll just simplify it and just look at the first range, 0 and just get the end position.

That tells us how much data has been loaded and we divide that by the duration and then we multiply by 100 and add a % to it so that we can type it right away in our CSS so now we can just assign it to our loaded bar to width style, make it our presentation. And that's fairly easy. So let's look at this and look at our content and we should see a bar, there you go.

We see a bar filling in right here as data comes in. And once again, the code we write in the script is so simple, we're just updating the percentages by looking at a buffered amount of content we got and the duration. OK, fairly easy. So now we want to do something similar for the played bar, but instead of looking at the buffered content, we want to know where we are in the video. And so looking at progress and load events, we want to know about when time changes and once again there's an event for that which is the time of the event and whatever time of day you'll get a call back through events.

And what is great about this is you don't have to provide your own JavaScript time or anything like that. The video will have its own finely tuned timer that will [inaudible] these events when it matters and automatically when the video will play or pause, it will stop triggering these events so you don't need to even worry about what the state of the media is. OK so let's handle that event. Just one more to add to the list.

It's very repetitive but it's simple enough. And we have a time change function that we'll want to implement now. OK and it's very similar to the function that we had before for got more data, instead this time we'll look at the current time of video, a new property, divided by the duration, make that a percentage again and just apply it to the play bar width style. OK so it's just the same thing except two different elements with different data and there you go.

We've got a darker bar filling up, we got a loaded bar right here, dark you know, darker gray and we got our maroon colored playing bar right here filling up and we'll pause, it will automatically stop filling up, because the time of the events won't be going through anymore and soon as they start going through again, we'll be updating our playing bar. OK, well that's all we need to do for step three. We got our progress bars going, both for progress of loading, thank you.

And for actually progress in a playback experience. OK so I hope you're bearing with me here because we're going to take it once again a bit further now. Oops sorry, it got away, I got ahead of myself. Let's step back and actually look at what we learned about.

We already knew about the can't play through events and all those events are in white here, but we also learned about these new orange events, duration change, whenever duration changes, load and progress for progress events and completion of progress and time of this for whenever, whenever the actually playback time change. And we learned about some new API's, we learned about buffered property which tells us how much data has been loaded.

We learned about content which shows us in seconds how much data has been played and also the duration property which well simple enough, it just tells us what the duration of the media is. And a couple tips that we'll learn about. First of all it might make sense if you cached properties that you'll be requesting often enough and that will take a load off the media engine because the media engine is already quite busy you know, just doing the video playback which is already a lot of work.

And also make sure you simplify as much as possible, the kind of changes you need to make through CSS, through JavaScript, so that it's as simple as setting a percentage value for example. And let the CSS take care of the heavy lifting of making sure that percentage value will actually work in your workflow, in your layout, sorry. OK so OK I've got a little technical difficulty with the clicker here. OK, do we have a spare clicker maybe.

OK we don't. All right, so I'll do it from the podium if that's all right. Except that doesn't work either. OK, OK so it looks like the clicker has gone back to life. All right. All right, step four, thank you. Step four will be about scrubbing. We've got our progress bars and now we want to make them interactive.

We want to provide a scrubber that will be able to drag and click to move. So let's look at this. We'll have a little block here, I'll display the elapsed timer underneath and we'll just be able to click on that, drag it around. We'll be able to change our current time and click also to specific positions to just seek directly. OK again, let's go back straight to the code.

And let's take a look at what we have to get going. OK so before we start even the code, we already have our play head, our scrubber, already in place. The reason I didn't do it is that, doing code with you is it's simple enough. We'll just need to update it's position, just like we update the width of the play bar, we'll just update the position of the scrubber.

It's simple enough, so I don't think it was worth going through that in particular. OK let's look at the HTML. We have our scrubber right here. It has a thumb, which is the actual little block that we will be dragging and it has the time which will be displayed underneath to show the elapsed time and we'll have another time here for the duration, that'll be displayed on top of the progress bars. OK if we look at our script, we've got a few more things already in there.

First of all we've got pointers to our scrubber, to our lapsed time display and to our duration display and also we've got a few, we have a few utility methods already in there to pretty print times to provide a second, sorry a float number in seconds and just get in nicely human-readable string to display into our UI.

OK so first thing we're going to want to do is do something with the duration. We already know how to handle the duration. We listen to the duration change event and we already know we can cache it and etcetera. So to update the duration display, we just need to add a little bit more code here.

So let's just do that. Easy. So all we got to do is take our duration, put it into pretty print time to get a human readable version of it and pipe it into our inner text property of our duration display. And if we just reload that now, there you go. We already have the duration provided right away on the display here. That was very easy.

That was just another line of code. OK so we want to do something very similar to the elapsed time. We already have the events to deal with change of time, it's the time of day event and the articles are time change function. And so just like we updated the duration display, we want to update oops wrong one, I'm sorry, we want to update the elapsed time display inner text, to be the video current time.

OK load that and as the scrubber moves along, we've got the time displaying, the actual lapse time right underneath it. OK so that was simple enough. And in fact, if we look at this time change function, we can see that we also, that's also where we update the left position of our scrubber, which was already in the code.

OK [inaudible] code and we already have the play head/scrubber already moving along and the times being displayed. So now we need to take care of the interaction. We need to take care of the dragging interaction to do the scrubbing and what we want to do is map the mouse events to the change of current time. Basically right? OK so once again, I already have a little bit of built in code in here.

We already have a pointer to our scrubber and we already have a mouse down event listener attached to it and we already have code to deal with that mouse down event. When we get a mouse down, we'll start scrubbing. In fact, our drag interaction will be about mouse moves and mouse stops as well, to keep on going as we scrub. Scrubbing is really about moving the mouse.

So whenever we get a mouse move we'll update the scrub and we'll, whenever we get a mouse up we'll call Nscrubbing. OK. And I already have some skeleton code for these functions. So for example, start scrubbing, we'll add event listeners for mouse move and mouse up so we can try the move the mouse and it already has code to remove those event listeners whenever the scrubbing is done.

OK but we need still to fill in these blocks of code here. And the first thing we've got to do is like I said, our UI will always adapt to whatever metrics you provide in your CSS. So the playing the bar, the progress bar might actually be any size. You really don't want to make an assumption of the size of this because it might be changed to be anything. So I always say you know, make sure your script doesn't know too much about your presentation.

In that case, the script is providing a direction that's directly based on you know, spacial values, things that are on the user interface. So what we want to do is to make sure that we always know the size of this progress bar and to that we'll have a new utility function called update progress bar bounds and we'll want to call that whenever we need to know what the metrics of that progress bar is.

OK and we store those metrics in the progress bar bounds property on our instance and this will just come through a single call in the dump called getBoundingClientRect(), another new feature in Safari 4 and calls that in progress bar will tell us the position in the accordion system, the whole webpage, which is the same accordion system we'll be handling event with, so it's very relevant and also the width and height.

Width is really going to be what matters because what we want to be able to say is, well the duration of the video is this, the width of our playing bar is this and based on these two information, we can infer how many pixels re-sent in a second and that's basically what we're going to do now.

OK so when we start scrubbing, we're going to want to do a variety of different things. First thing we're going to do is we're going to update the progress bar bounds. Every time we have a user interaction, we want to make sure we have the right info about the bounds of the progress bar.

OK. So we got that covered. And the second thing we're doing is we're going to store a property on our object that entails how many seconds per pixel. So that's easy, we just take the duration and divide it by the available width to drag and that width is the same width as the progress bar minus the size of the scrubber, because we want the scrubber to fit nicely to the end of the progress bar.

OK so you'll notice here we're going to pause the video as well and why do we do that. There are very good reasons. The first one is that we'll get a lot better performance. The scrubbing and setting current time will be extremely fast in the media engine if the video is actually paused. So that's one good argument but the second argument is really about user experience.

Imagine we just let the video play as soon as we change the current time. Well obviously a scrubber already updates whenever the time changes, so imagine we move, stop moving but keep on holding the scrubber and the media starts playing. Well the scrubber will move but usually when you're dragging something it remains stationary under your mouse until you stop dragging. That's another reason why we want to pause.

And so that we can actually restore the playback state when we're done scrubbing, we want to store that in an instance property, we'll call it was playing when scrubbing began and make sure it's just the opposite or the pause state of our video. So then when we're done, we can resume playback if we were playing.

Because we're pausing right here. OK and finally we need to record a little bit more state about our current playback UI. The first thing we want to track is our current time when we started our interaction. So scrubbing start time is the current time when you mouse down on the scrubber and scrubbing start X is the position of the event when we start scrubbing.

So that will, with this information, will be able to infer the delta of time whenever you interact with the scrubber. OK so we got, all we need to do here and now we need to actually update the current time as we'll be dragging along. OK so let's prorate the interaction for this and it's very simple.

First of all we'll look at how many pixels we've interacted, events will tell us about mouse position, pixels. So we'll take the current position of the events only on X and subtract from that the start position that we had for the event. OK and that will tell us how many pixels we've dragged since we started.

OK. And now we'll take the delta N pixel and multiply it by the number of seconds per pixel and that tells us the delta in time, in seconds. And all we've got to do now is just add that to the start time that we had and assign it to current time. And that is all we need to actually do the scrubbing.

And you notice here again, we do not change the position of the scrubber, this will be done through the event flow. Whenever we change the current time here, we'll get a time of that event and the time of the event we'll call the function that will lay out the progress bar and the scrubber and the code is already there and you don't need to change it.

So once again, let the events change the UI. OK and we just need to do a little cleaning up when the interaction is over. So here all we've got to do is check on our was playing when scrubbing began property and make sure we resume playback if that was true. If it's not true, it was already paused and since we're already paused, we don't need to do anything.

So let's reload this and fortunately enough, we'll be able to scrub. Look at this. And look at the scrubbing performance. It's actually amazing. You're just scrubbing here and it just is so responsive. That's the kind of support you get from video in our browser. OK but we can't click anywhere yet and that's the next step and it's just going to be a lot more of what we've already been doing.

So the first thing we want to do is we want to have a click event listener on our progress bar. OK so actually it's on the loaded bar, we only want to scrub to a place we've already loaded, click in on there and we'll just need to deal with this. So let's go back to our click event handlers and let's just add another case so that when the loaded bar is the current target, we'll call this very long name but very descriptive, setVideoTimeFromClickinProgressBar.

OK and finally we'll just implement this. So we'll put the code close to where we had the scrubbing code because it really belongs in the same area and whenever we'll get that call, we'll update the progress bar bounds again, do a little math to figure out where in the loaded bar we are, the fraction that we are in the width of the loaded bar based on the X event and just assign a current time based on that.

And we'll also do a little clipping to make sure we don't assign a time that's less than 0 or more than the duration. Cool, and once again, this will just work because we don't need to update the position of the scrubber ourselves, it's already done with the time of day events handler. So we've got everything done directly and it's still blazingly fast.

OK, so we're done with that step. [Applause] Thanks. And that's really, that's it, that was the most advanced thing we'll do today and that still was a fairly easy. The hard bit I would say is just dealing with mouse interaction, but the media integration was dead easy right So let's go back, let's focus on some of the coding techniques that went through here again.

So first of all, because we want to let CSS take care of a lot of the presentation, you don't want to make your UI metrics you know, constants, you know, code, that just dirty, that's what a bad programmer would do. What you want to do is create a UI metrics in real time at all times and that's where we got the get [inaudible] method in the DOM for using this method you can simply get the actual position and size of any element in the DOM.

So very easy and a very powerful concept in programming. And also you want to pause while you scrub so that you get a lot better performance first and a better experience so that the scrubber always remains stationary under your mouse. And that leads us to the last step.

The whole logic is done. Now it's all about the fit and finish. We want our UI to be just amazing to the user and to do that we'll be using CSS Transitions, we provide for less implicit UI animations and if you were in our session that other people on our team yesterday presented, here at the same time on the CSS Effects yesterday, you seen a lot about this and you already know how cool it is. But for those who might not have been here, well you'll see what we can do with a little bit of CSS.

And what we'll do is that we'll hover the view only then will provide the controls and we have that with a nice little animation, an opacity and translation and do the same thing for the Volume slider and that really brings our UI to life. That's what a rich media experience is about on the web.

OK let's go right back into the code.

And load our fifth step to see where we're at. In fact we're exactly the same stage, save for the volume slider. I'm not going through the code for this because we're doing with the volume slider, the exact same thing we're doing with the scrubber here.

Instead of looking at current time, we're just looking at the volume change. So it's a nice and functional volume slider and you'll be able to look at it in the sample code. OK but we have no transitions whatsoever here. OK so that's what we're going to be doing ourselves. So first off we want to make sure our controls are off by default and we don't see those.

OK so let's look, oops sorry, this is not, so if we look at our controls right here, we want to make sure that by default they are faded out, so opacity 0 and slightly [inaudible] as well. So let's put that in, opacity 0, WebKit transform translate wide by 100%. 100% is actually 100% of the height of the controls. So since the control is already at the bottom of the page and they're 100 pixels high predict that translation will just lay them out underneath the video on the Y axis.

OK what we need is another state for whenever we're actually hovering over the controls, to actually show the controls. So let's add the over pseudo class in our video player and match controls only in that state to make opacity 1 and our translation to be 0. OK so let's take a look at this.

Controls are off by default and when we're over it it just snapped. OK so that's good enough but we don't want them to snap. We really wish we could have those really rich transitions right. So traditionally you'd be writing like as much code as we've written so far just to do this little animation in JavaScript and that would be a waste of time and also it wouldn't be very performant on more [inaudible] devices like iPhone.

Well what we're going to do here is just add two lines of CSS and with those two lines we're just going to say whenever we change the opacity of WebKit transform that elements, we want to transition that change during 0.5 seconds. And in fact, if you'll look at our content now, look at this, we have built in transitions without writing any logic simply declarative CSS to take care of that.

But we're not-- [Applause] Thanks. But our job is not quite done yet. If you notice, it actually sticks under so we really wish we could clip that. So if we go back to our video player, we'll just add a simple overflow rule, set that to event and now the controls really just pop in from within the video. OK great. So let's do something similar for the video, for the volume slider.

We'll just do more of the same. So we have a slider here, we'll set up the default state to be opacity 0, translate X by 88 pixels, set up the duration for the transition and the properties opacity and WebKit transform and just add another state for hovering on the volume. OK super simple.

Set up opacity 1, translate 0, reload. OK that was really easy. It was just doing just applying the same technique that we already learned. So now I'd like to do something better. Our video should be front and center. We really want to be able to focus on it when we just want to be watching the video, not actually playing with any other part of the webpage.

So to do that we actually have an X array [assumed spelling] element in our tree which is called Dimmer and it's basically going to be a layer of black content, it will be put in between the body and the video player to just dim the background of our page. And we already have a little bit of script to take care of that.

Whenever we'll click on our video, we'll make sure the dimmer is on and whenever we click on the dimmer, oops sorry, whenever we click on the dimmer we'll make sure the dimmer goes away and we'll do that by calling this function callSetNSPlayback and all that will do is turn on or off the enhanced playback class on our body. OK so what we need to do in the CSS is provide a little bit of styling for that dimmer.

So let's put it in. OK so we got our dimmer here. By default it will be opacity 0, it'll transition into opacity and whenever we got the enhanced playback class as a parent of the dimmer, we'll just make the opacity 0.9. OK let's look at this, click on this and we get a nice fade of the background. So already our video really pops out and we've got a much better experience.

But you know, since the beginning I made a big, big point about making sure we build our CSS so that at all times we'll be able to change the metrics of that container so that if we have a bigger video we can just expand it very easily and etcetera. So really we should just be able to change the width and height of our whole player and just have it snap all its controls and etcetera, as soon as we change it.

So let's just do that. Let's create a new state when we've got NS playback on for our video player. And that state will just change its position in width and height, to make sure it fills up the screen as much as possible to look good and relocate it as well. So let's do that.

Click on it and our video's a lot bigger. But it doesn't transition yet and surely applying a transition here is going to be really heavy duty. We're not just doing a little opacity fade here, we're going to redraw and resize a lot of content in our page, all that on top of video, that must be pretty heavy.

Well I'm pretty sure you know what I'm getting at here, turns out it's not at all. All we have to do is again, set up the transition width, height and top, a duration, one second which is the same duration as it takes to fade the content and look at this. We click on and it just resizes everything. Look at the play bar resizing concurrently, the code doesn't do anything here.

It's all based in CSS.

And that's really, really, really powerful and once again the script knows very little about this interaction. In fact the script knows barely anything. And that's all we have to do today building the code and that's all the features you'll get in sample code you'll be able to download from the attendee site. So let's wrap up on some techniques we used here.

First of all, we triggered transitions completely declaratively. Again you set states with CSS classes, using JavaScript but the transition and the heavy lifting is all done in the CSS and really, you really need to make sure CSS handles the complex presentation and let the script handle the logic. But we have a little bit more because this is all we have time to cover in you know an hour.

But we can do so much more with the powerful features in Safari 4 and I'd like to bring up Eric Carlson on stage, my esteemed colleague and the guy who implemented all this great technology in WebKit recently and he's going to show you how we can take this all to the next level.

So Eric, if you just move to the demo machine on the display, we'll be in business.

OK. So what we have here, we took the, we took the DVD that Wilco has just shipped and thought about what it would take to make a full DVD-like experience in the web. Since we have all this, since we have all this wonderful content.

So we encoded, we grabbed all the audio and video from the DVD and encoded it as an MPEG4 file. We did an SD encode to H.264 AAC. We ended up with a movie that's just almost an hour and a half. You can see from here. About 700 MB and because we, because Antoine wrote the demo to adapt to the media, there's absolutely nothing that has to change here to deal with this large file.

So we have a much bigger file, it's almost an hour and a half long, a very high data rate, because we're using QuickTime for media playback and we're playing it on Snow Leopard here, we have QuickTime X, we have the enhancement to QTKit and the experience is, is wonderful. We've got quick seeking, we've got good sync and we didn't have to do anything at all to our, to our code.

Now on a DVD, you often have extra information and in order for somebody using the DVD to get at the extra information, you usually have to leave the video and go to a menu and we didn't want to do that. We don't want to force the user to go out of context. So instead, we just put it on the backside of the video. And using a little bit of CSS, we won't actually go in, we won't actually go into the code here but the CSS is dead simple.

We're just applying rotation on it. We have the DIV with all of this information on the backside of the video and because the video element is just another element, you can see that the opacity on this backside lets the video play through and if you watch, what you're seeing is the video is actually being rotated and playing through backwards and it's not because we're asking QuickTime to flip the video around and play it backwards, it's because the video element is just another element and the and WebKit which is orchestrating the playback, is able to do the right thing. Because it knows how to deal with video.

So we have here a list of all of the songs on the album and when the mouse hovers over a title, we're changing the location of the DIV on the right with all of the meta data and images and we've got a simple CSS transition on it and of course it does the right thing. Click on a title, we instantly seek to it and there we are. We play it back. Reduce the size, look at the back again, everything scales.

Again, it's the power of CSS. But, so all of the, all of the meta data that you see there is actually just being, is in JSON structure and of course we can have lots of extra information too. So I'm going to use the keyboard shortcut here and you'll see that we also are able to put the album and song information in, in line with the video playback and as I seek from song to song, with the keyboard, we're easily able to show them over top of the video and then fade it out, again using CSS and because we're using CSS for the display here, it's really easy to keep it in context and give a really good user experience.

So we thought why not include extra information. So we have also, we also have lyrics for some of the songs. I'm going to use a keyboard shortcut here to seek to a section where there's some interesting information and because we're already responding in the code to time update events, it's simple to keep the lyrics in sync with the video, because we're using CSS for the display there's absolutely nothing to it.

We see them from the back, we see them from the front.

So what we've done here is we've taken the code that we built up in the first part of the session which, if you download the code later and look, you'll see that with comments and everything, there were about 300 lines of JavaScript, which isn't much and by adding less than another 300 lines, we're able to add all this extra sophistication and really create a very rich, DVD-like experience within the context of the page and really adding to the, adding to the DVD experience as well.

Thank you.

Yeah, I think basically what it boils down to is that mixing some simple things was really simple and making these really advanced things were actually not that much extra work and you can really take it to the next level with all the technology you provide us.

So I'm really curious to see what you're going to be doing with this. And I'd like to wrap up our session right now. So really I think what we discussed is all about how simple and yet powerful the integration of the video element is in your webpage. It's just another HTML element.

You just have an image with an IMG element, you just have a video with a video element. It's just that simple and without, with barely like typing another 16 characters you've got enough information to tell the engines to provide you with a great built in experience with a nice controls, both on iPhone and on the desktop. And you can do way, you can do way much more.

You can style and even use very sophisticated styling like transitions and transform and apply that to your video with CSS. Because once again, it's just another HTML element. And likewise the DOM just works with video. You'll get events whenever the video playback stops or resumes or the timing changes and etcetera.

And you'll be using, you have a variety of rich API's to control the video through user interaction. But more importantly, all this is ready to employ today. We have fallback mechanism and we've got the technology. And it's important in both iPhone and desktop, in fact if you think about it, on iPhone OS 3.0, this video is the only and best way to provide video experience in a browser. So that makes it very compelling to use video right away.

Because what do you want to do really is to give your video content available to anyone, both on iPhone and on desktop. And also remember that we have support and fallback for non WebKit browsers. Well more importantly it's not just about WebKit browser, it's really about you know, the less capable legacy browsers, right. And well that's all we got today.

And I'd like you to contact Vicki Murley, [email protected]. She's our web technologies evangelist. She knows all about this and she'll be able to direct you and help you with whatever questions you have. We have great documentation so obviously there's the HTML 5 spec, you can go and look that up. We're in sync, we will always continue to be in sync.

We always support the latest standards in Safari and WebKit. But we also have great documentation on developer [inaudible] Safari, we have the visual effects guide focusing more on CSS and the nice tuning effects that we had in our session. We also developed expansions for the video element for example. So go look at this.