Hello, and welcome to part 2 of Delivering Audio and Video Using Web Standards. My name is Jer Noble. I'm a Safari and WebKit engineer, and I'm just going to pick up right where Vicki left off from part 1, specifically, last part of the session we covered basic <video> element playback in your webpage, specifically how to add using the video tag, a <video> element to your site and a few JavaScript functions to control playback of your media. And from there, we're going to take all of the things she talked about, build upon them and add a whole lot of more additional functionality to the <video> element in your webpage.

So, where we left off? Just a quick recap for people who may have missed the last session, the adding of <video> element to your site is as easy as the video tag or the single source attribute. With this alone, you get HTML5 video embedded in your webpage. But for additional functionality, if you want to let your browsers choose between the most appropriate video content, adding multiple source tags to your video tag will allow you to do that. And finally, controlling the video playback is as easy as a Play and Pause of it, or Play and Pause function on your <video> element.

So, that's where we left off. The last demo was a video with a big Play/Pause button and, while, that's cool and all, we're going to take all of those stuff and pump up to awesome, the video playback experience. So we're going to add or we're going to fill out the custom JavaScript and HTML and CSS controllers so it reproduces most, if not all of the functionality of the built-in controller.

Once that's done, we're going to add some subtitles and closed captions to your video. We're going to cover how to add bumper clips to your video within playback, and we're also going to talk about how to take the video from your website and embed it someone else's website.

So step 1, so we have a big Play/Pause button. Something that exists in the built-in controllers that you might want to add through JavaScript and CSS is a progress meter, so you might be asking, "What's a progress meter?" Well, you have your big Play button, you have your piece of video. The progress meter would be something like this, that when the user hits Play, [background sound] it starts playing the video and the time continues. I know it's really cute, you'll be seeing a lot of it.

Anyway, so, to implement this JavaScript based playback meter, you'll need a few pieces of API. First of all, there's a timeupdate event. Now this is fired occasionally, 4 times a second or so, during regular playback, when you're playing back at a constant rate. Also, it gets fired more continuously as the user perhaps changed the time with the built-in timeline controller.

Once you catch this timeupdate event, you'll need to query the currentTime attribute to pull the current time out of the video. And to calculate a percentage of the way through the video that your playback has currently progressed, it will pull the duration attribute out of the video as well.

Now, there's an additional attribute called startTime that for almost all cases will be zero. So for normal playback, you got a media file on a server somewhere. You put it in a <video> element, the startTime will be zero. The cases where this is not the case is in HTTP live streaming, because the startTime is going to change based on when the user starts playing their content.

And as Vicki described in the first session, there's the durationchange event that fires when the browser notices that the duration of the video has changed. Again, in most cases, the durationchange event will only get fired once. The cases where it doesn't or it gets fired mobile times is again, like HTTP live streaming.

OK, so to implement this in JavaScript, first we're going to make an updateProgress function. We're going to ignore the start time and assume it 0, and calculate the percentage based on dividing the current time of the video by its duration. And again, we're going to take the simplest case of representing the progress meter by stuffing that percentage in the style attribute of a div called progressDiv.

Now this is the most, like again, the simplest case, but you could represent the current time as a string placed anywhere on your website. You could do it as a pie chart, a bar chart, the possibilities are endless here. So, we'll catch the timeupdate event from the <video> element and call updateProgress occasionally.

And to cover our bases, we'll also listen to the durationchange event. And when that changes, recalculate the width of the div, and it's simple as that. So you have working progress meter. Something else that's in the built-in controls that you might want to add to your custom controls is the way of showing how much of the video has loaded. In part 1, there was an indeterminate progress meter spinning, but we're going to show you how to add a determinate one.

So-- but why should you care? Again, Vicki kind of covered this in the first part of this. But if you don't display "what" ... to the user the state of their movie, whether it's loading or not, what it's doing. They might get bored waiting for a video to start playing if they're on a slow connection and it's a big movie. So, you can add an indeterminate state, but we're going to show you a way of doing a-- showing how much of the video as a percentage is loaded.

And again, if you're a good internet citizen and you have a page full of videos, and you don't want them all to start loading at once to-- so that the first one the user clicks on loads faster, you will have like a Click To Play button like we showed you last time, and it might look something like this. User clicks the Click To Play, the movie starts loading and at some point in the future, the user says, "Oh, I can play, awesome!" [background sound] and the movie plays, awesome. You know, I just can't get enough of that.

[Laughter] Anyway, OK. So, how would you implement this script to your other script? Well, an event that's fired occasionally by the video element is the progress event. It's fired no more often than about 1/3 a second and no less often than every byte that's loaded. Hopefully, you're not getting progress events for every byte that's loaded. It indicates your user is on a very slow connection. But you'll catch this event and then query the buffered attribute of the video element.

Now the buffered attribute is implemented in terms of this type called TimeRanges. It is a single object even though it's a plural name. It is like an array but not quite an array. It has a linked attribute and a couple of functions called start and end that allow you to pull out of that TimeRanges object the start and end points of a range of time that the browser has loaded and has a data available for immediate playback.

Now these ranges could be discontinuous. The user could have jumped ahead of the downloading stream and the user or the browser started downloading a new area of the movie, but they'll always be nonoverlapping and they'll always increase in time. You can be guaranteed that the last item in the timeranges object will be after the first item, and every item in between. So, to implement this piece of JavaScript, we're going to have a function called updateLoadState.

And for our purposes, again the very simple case, we just want to know what the maximum allowed time that's been loaded, and so we'll check to see if there is any area of the film that has currently been buffered, grab the last item, ask it for its end time, and use that as the maximum loaded value. And just like in the progress example, we'll calculate a simple percentage and set that as the width of a div somewhere else in our webpage.

OK, listen for the progress event, and every time the browser tells us that it's loaded a little bit more of the movie, we'll update the load state of our custom controller. OK, so you have a progress meter, you have a load state. Something else that's in the built-in controls that you might want to re-implement is a Scrubber a Scrubbing slider, and why do you need a Scrubber? Well if-- like most people, they'll have favorite parts of your movie and you might want your users to jump to any random point in the timeline. And as long as your server supports byte range requests, the browser can immediately pick up where your user requested at a current time and start loading times past that.

So, just like in the progress event, you'll need-- or the progress meter, you'll need the currentTime attribute this time to set the current time and cause the browser to jump to a specific point. There's also the duration attribute which like in the first example, we'll need to calculate a percentage. And there's another attribute called seekable.

Now, when buffered, it tells you what parts of the video are currently loaded, but there's no restriction on jumping past the end of the loaded state. The browser will notice, it will start downloading a new portion of the movie, except for the seekable attribute. For most cases, seekable will define the entire range of the video.

There'll be no restrictions on where the user can jump. In special cases like HTTP streaming, this attribute will contain a range-- or a set of ranges of time which define where the available data lies in your video content. Now-- and when you're live streaming for example, 30 seconds in the future hasn't been filmed yet, so there's no way for the user to jump past like what is considered now.

And perhaps the user has thrown away all the data from 30 seconds ago and won't let the users scrub back to there. In that case, the seekable attribute will define a very narrow range of time that you want to make your slider the min and maximum amounts on that slider.

Now, it's reasonably difficult to implement a slider with straight HTML and CSS. However, new in HTML5 is a new type of input that is of the type range normally rendered as like a normal slider from whatever your OS is. However, in Safari and WebKit, this input element is completely styleable through CSS. So we're going to use the input range element to allow the users to control the portion of the movie that they're currently playing back.

So for this element, it takes a minimum and a maximum. Because we're working at percentages already, we're just going to set min to 0 and max to 1 and use the value attribute of the input element to extract what the current state of the input is and, as the time changes, to shove a new value in and have the input element reflect that change.

And as the user is dragging the input slider around, it will generate these change events and that's what we'll use to set the current time. So this is what it would look like in JavaScript. On the scrubber side, we'll listen for the change event. Its value is something between 0 and 1, an easy percentage.

So we will just multiply it by the duration, set with the current time, and Bob's your uncle, you've got a working slider in your ... in your HTML video. And on the-- from the other direction, as the video's time changes, we'll want to occasionally update the value of the slider with its new position. And again, since it's a percentage arranged from 0 to 1, it's as easy as dividing the current time by the duration and setting that to the value of the slider.

And in order to demonstrate these techniques, all of the last 3 I'd like to call Eric up, and he has got a few demos to show you.

So, we have got our adorable baby here with a controller. We're going to be logging the events over on the right-hand side. We click Play, we click Pause, we see the events coming through. You can see the progress indicator here. We have got a thumb, we drag it back and forth.

You can see that we're getting seek events and timeupdate events just as you expect. One more thing that we added just for a little bit of extra bling, is as we move the mouse off of the video, we show and hide the controller. And that's as easy because we have CSS.

That's as easy as setting a couple of properties on the CSS. So you can see in the video controls attribute, we initially have the-- we've initially set the opacity of the controls to 0, and we've set the Y translate to 100 percent, which pushes it off the-- pushes it off the bottom of the screen.

And we've also set WebKit transitions on both opacity and WebKit transform. So that when the user hovers the mouse over the player, when we change the opacity to 1 to make it visible, and translate Y to 0 to bring it back up, that transition will take a half a second. So again, we move the mouse over it, it shows. We move the mouse off and it hides. And it's literally as simple as adding just that, that little bit of CSS.

Alright. Thanks, Eric. OK, so we have most of the basic functionality of the built-in controller now done through JavaScript, accessible through CSS. Style it however you want. For the next step, we're going to push past what the built-in controller can do and add functionality that it never could in the first place.

So subtitles, why should you care about subtitles? For one reason, perhaps you have Americans with Disabilities Act requirements that require your video and your entire website to be accessible to users for whom the audio is not very useful. So you might need to present closed captions on top of the video describing what is happening in the scene. So that's something called descriptive audio.

You might also have video content, the audio which isn't in the same language as your user's native language and you want to have translations appear on top of your video. In this case, she doesn't speak any English [background sound], we'll need to translate. That's mama, and mama again.

OK, so how are we going to do this? Well, before we start, I'd like to point out that this is an active area of development in the "what" working group. And they're currently writing the spec that will define how, in the future, these subtitles and its closed captions will be built in to future browsers. Now, however, even though this is a future part of the spec that's not even very well defined right now, all of this is perfectly capable with the HTML and video element as it exists today in Safari and other browsers.

So to start off, if you have a piece of video content that already has closed caption tracks built in, there are few methods you can use to detect that there is closed caption tracks and enable them or disable them in the video itself, and that's this webkitHasClosedCaptions and webkitClosedCaptionsVisible attributes of the video element. Now this allows you to make visible a single closed caption track in your video. And that's great for people who have existing content with closed captions.

However, it's not exactly dynamic. It allows you to turn on or off a single closed caption track. And if you want to support multiple languages of closed captions, you will need multiple videos. So we're going to show you a technique that will work with video that does not have anything built in and will work for any language you care to insert through this method.

So meanwhile, we're going to embed the subtitles directly in the HTML. For the purposes of, you know, very simple demo, this is the quickest way to get your subtitles built in to your video content. But there is no reason why you have to embed the subtitles in HTML. You could pull them down through JSON. You could use XML HTTP requests.

And as we saw yesterday in the Safari state of the union, someone from being implemented in pure HTML5, a way of doing live translations of selected text, there is no reason why you couldn't have translated the subtitles available completely dynamically through an API provided by a language translation service.

But for our case, we're just going to embed it in HTML, and we're going to listen for the same timeupdate event we use to track the progress meter. And as that event is fired, we'll check the current time, figure out what subtitle is supposed to be visible at that time and mark it as display. For a long, long list of subtitles, that might be something of an expensive proposition.

So if you have a set of subtitles, you can use the built-in setTimeout and clearTimeout functions to schedule a subtitle to be visible some time in the future and use the built-in properties of the video element to figure out when that subtitle will need to be displayed to the screen, making things slightly more performing.

Again for a simple case, we only have a few subtitles, so we're just going to calculate the correct subtitle every time we catch the timeupdate event. So here are some basic HTML code, you have a <video> element, and a few divs that have start and end attributes defining when they should be made visible through the playback of the video. In this case, a 1-second subtitle, and then another 1-second subtitle 1 second later.

For the CSS part of your HTML code, well by default, make these subtitles display absolute nothing, the bottom center of the screen. But again, because this is completely dynamic done in HTML and CSS, you are not limited to overlaying your video. You can have the subtitles appear in a different part of the same page, or even in a pop-up page maybe movable around by the user. For the JavaScript portion, we're going to listen to the timeupdate event.

And just like in the progress meter example, plot the current time, the time considered to be now. And then iterate through all of the elements marked as subtitles. And for each one of them pull out their start and their end attributes, compare them to now. And when we find a subtitle whose start and end times {current time}, set the display rule for its CSS style to be inherent overwriting the display none in our CSS style sheet.

Otherwise, we'll set it back to display none and that subtitle will hide. And to demonstrate this technique, I'd like to call Eric back up to do another demo.

OK, this time there is not much to show. You have seen the code, but let's see it in action.

We'll start our video playing. There she says mama, and she coos at the end. Because we're changing the display of these through the events that are emitted by the element, it works the same when we're playing it as when we drag on the thumb. Because the element as you can see from the events on the side, emits the events so we know when to show it and we know when to hide it. It makes your coding really simple. You just listen for the events to be emitted by that element and you take action based on that. Jer?

Thanks, Eric.

So, to reiterate, it's an entirely flexible technique that allows you to do closed captions, subtitles, apply styles as you saw on the example, to make your subtitles look however you like. And it works not only for subtitles, this could-- the same technique could work to cause events to fire as you progress through the movie.

Put overlays on top of your video at specific times. Cause different parts of your page to change as the movie progresses. We can use this technique for just about any dynamic action you want to occur on your webpage. So, let's recap. We've got working JavaScript controller, we have subtitles.

Something else you might want to do with your video playback is add a bumper clip. And you might be asking yourself, what is he talking about? Well, a bumper clip is a piece of video industry jargon that refers to a secondary piece of video inserted in between two other video clips.

Now, we might know this more colloquially as advertising. So, a lot of video playback sites are doing this now. YouTube, specifically I just saw the other day, inserting advertisement at the beginning of video playback. And you might want to know how to do this if through HTML5 video on your own site.

So, there's a few different techniques to getting your advertisements or your bumpers or completely non-advertising these things into your video playback. The easiest approach is just to burn in the advertising into your video content directly. Do that QuickTime Player Pro or if you have Final Cut Pro or even iMovie. You can embed one video in another with some nice transitions and do it that way.

That's great as far as it stands. It's a little static, so we're going to show you a couple of techniques to use if you want to do a more dynamic way of inserting video. So, if you're already doing HTTP Live Streaming, you can insert some video content into your live stream directly.

And for this technique, I advise you to stick around for the session following this one that talks more about HTTP Live Streaming. If you just want something to work completely client side with no service added requirements whatsoever, we're going to show you a technique that involves switching sources of the video content during playback to have the videos chained together one after the other. It would require APIs like this one.

So first of all, there's an event inspired by the video element as the playback reaches the end of your video, and that's the ended event, named very well, I might say. So, the source attribute is not only-- it's read/write, and what we're going to do is shove a new source into the video once the last video ended.

And then as the video sources change, 'cause the video act to play, and if you're on a fast connection you'll see a very seamless transition from video clip to video clip. So, it looks something like this. We have an array of sources, 4 or 3 different movies that we want to play in a row, and an index into that array pointing to the current playing movie, and the function which advances those self sources from one to the other. It simply increments the index, pulls the new source out, sets it to the video and then cause play to occur.

And in this case, we're going listen to the ended event and cause the advance to occur only when the last movie ends playback. There's no reason why you're limited to this. You can have the next video source come in the middle of your playback. You could have something played in the beginning and when the advertising or whatever your bumper is finishes playing, move on to the next video.

You can even have different links in the same webpage cause different sources to be loaded without taking the user out of the current page. And to demonstrate this technique one more time, we're going to have Eric come up.

In this case, we're just using the default controls so that it's easy to see what's happening. So, if you watch the current time thumb on the default controls, watch what happens as we play through. So we start our first movie.

It reaches the end. If you watch fast, you can see that the ended event was fired. Our next movie loads. It gets to the end and it goes back to another movie. Now, you may want to customize the experience, make it look to the user like they're just watching a single movie. In that case, you would want to use a default controller, but that would just use the same techniques using for the custom controls up to this point.

Let's take a look here. We'll see the same thing on the iPad. Not too much of a surprise. We start the movie, it reaches the end, it goes to the next movie. It plays through. Isn't she cute? And it goes to the last movie. Now, if you're paying attention in the last session, you'll recall that Vicki said that playback was strictly under-- playback is strictly under user control on iOS. That is a script is not allowed to start or stop a movie unless it's called from a user gesture.

And yet, what we have here is when the ended event fires, our function is called. We change the source to the next source. We call load on the element and we call play. The ended event isn't the user event and so you would think that this wouldn't work but it does because the user initiated playback on this element when we first started to play. So from that point on, the scripts are in fact allowed to start and stop playback without user control because the user initiated that initial playback. So, that's one important point that you want to keep in mind. Jer?

Awesome.

Thanks, Eric. OK, so we have a working JavaScript controller. We have subtitles. We have learned how to embed different clips and-- or not how to embed, we're getting to that, how to combine different clips together. The next step you might want to consider is if you have a video page or if you have a webpage with some video elements in it and your users come to that site and they see something that it's really awesome and they want to embed that same clip into their own site. This is called embedding.

And originally, it referred to inserting a plug-in into a webpage that was running native code but was hosted in the interpretative code of the webpage. However, it also allows you to insert one page into another, and this is the technique we're going to use to embed HTML5 video into a page that was maybe not originally coded to HTML5 standards.

Because you embedded one webpage into another, it should allow your users to copy and paste a very short line of code to get that video into their page instead of having to copy and paste like, you know, the entirety of your page web and your CSS and your JavaScript. We'll get this with just one single small line. So, the technique we're going to use for this is the <iframe> tag.

And the <iframe> tag takes a source, and to make it slightly more dynamic, we're going pass the argument of the video we would like the iframe to play back in the fragment portion of the URL. That's the one after the octothorpe, the hash mark, the sharp, the pound, that thing. And on the client side, it might look something like this.

They have an existing HTML page. They've got a body, they've got a bunch of content, and they want to put your video right in the middle of that, and they're going to do that with the <iframe> tag. As a source attribute, in this case, an embed-- a link to an embedded specific website.

And in our case, we're going to pass the argument as the entirety of the URL after the fragment portion. If you're interested in doing server side scripting, you can pass any number of normal query arguments in that same URL. If you're still doing it, like, all through JavaScript, you can still do arguments after the fragment and parse them out yourself. And this is the technique, actually this is the most simple technique in the slides here but Eric has a slightly more complicated technique that's going to involve passing arguments in this fragment.

However, on the HTML code on your side, if you're doing a very simple video tag embed, you'll have a video tag with no source content and an onload handler for the body. In this case, init, that will set everything up to the way the user who embedded your URL requested.

So, the JavaScript to get this to work that's hosted on your site, the init function might look something like this. We're going to pull out the video element by its ID and, in the simplest case, it will be as easy as setting the source to the substring of the fragment, everything after the hash mark.

And this alone will allow users to link to an embedded HTML5 video page to get a video playback. And to show you a slightly more complicated piece of sample code, Eric is going to-- has a great embedded video demo.

First, what we have is our-- the page that we've been using.

And what we've done is we've added a little bit of JavaScript that queries the movie in the page, pulls out the URL, pulls out the poster frame, if there is one, gets the width and the height of the video and puts it into this text area in a page.

And we've done that so that a user can come to the page. For example, a user with the site cutebabyfaces.com, which is nice, but it's not very cute without the face of the baby. Go to our blog, we'll open up our demo, our blog, and we'll just paste in the iframe that we pulled out of the webpage.

We can reload it and there's our cute little baby. Now, this is fine but this doesn't work so well with the layout of the page. The video is a bit wide for the layout that we have here. But because the width and the height of the iframe are parts of the iframe element, we can easily come in here and change this, down here, scale it by 75 percent in this case, reload our page and there she is. It works like it always did. In this case again, we're showing the default controls to make it slightly simpler but it would work just as well with custom controls.

Let's look at it on the iPhone and we'll reload the page. And you can see, again, it works just the same as it does on the desktop. So, allowing somebody to come to your site and use your content on their own site without actually having to understand how all the technology works.

Awesome.

Thanks, Eric. OK, while you're still up here, however, I'd like you to give the example we came up with that took all of these different techniques and pulled them on together to something that's slightly more refined than the examples that we've been using so far. As much as I like that baby, there's a limited audience, so something a little bit more well polished.

So, what we've done here is we have the same Dartmoor video that Vicki showed in the first example, and we have put our own custom controller on it. It's slightly different than the one that we saw last time. We have a button. We've added a button that allows us to scale it up and to scale it back.

And as Vicki noted in the first session, Safari 5 has full screen. So if we wanted to, we could take this element. We could ask WebKit, we could ask Safari to fill the entire screen with this. But when that happens, we take just the video element full screen and, in this case, we actually want to have-- we want to be able to show titles on it. And if we were to show that in full screen, because only that video element shows that those subtitles wouldn't work. We'll start it playing. Subtitles are helpful. Maybe we want to see them in a different language.

Notice that because this is HTML5 and the video element is just another element, we have flipped the element around using the CSS transform. We have another element on the back with our subtitles controls and it is not a hundred percent opaque so the video is showing through and you'll notice that it actually is-- the video is actually playing backwards because we've flipped it around. And that's not because QuickTime knows how to play this thing backwards, it's because it's a video, it's just another element. And so of course when you flip it around, it behaves the way that it should. Back to English.

I don't know, maybe you guys can tell me what that thing was. It's not a fossilized pig face. So this happens to be a fairly high data rate movie. It's 1280 by 544 pixels at 1875 kilobit, and look at the performance. It's really stunning. I take it big and small. You see playback is smooth. It's really spectacular.

Please contact Vicki Murley. She's our Safari Technologies Evangelist, as well as check the documentation on the apple.com developer page for Safari. Specifically, there's a whole section on HTML5 audio and video in the Safari HTML5 audio and video guide. As well as the, what working webpage-- the W3C also has a working draft of the HTML5 spec which you can check out. And the brand new developer forums for Safari are available at devforums.apple.com.

So, in the session directly following this one, it will be a whole talk on advances in HTTP streaming. And I highly encourage you to stick around for that one. Later today, there are-- as a two part session on CSS effects, we'll talk about all the special CSS3 stylings that we put into these webpages to make them super awesome.