Welcome everyone and thank you for coming to this session. My name is Athar Shah. And I'm a manager here at Apple in the core media software team. And we at Apple are really excited to talk to you today about two new media technologies. In particular, we're going to be talking about a new codec for video and image compression called HEVC and a file format for images that we're adopting called HEIF.

But before we get into the details, if you've downloaded the developer seed, the latest build, then on certain iOS devices you are already capturing and using these new technologies and file formats. And if you pay close attention, you'll notice that these files are a lot smaller than they used to be. Later on in the presentation, Gavin will talk to you about the specifics of our platforms and how we support these technologies in hardware and in software.

So we're going to be going over the landscape of media as it is today, why we need to change. We'll talk about what are HEVC and HEIF and why we decided to adopt them here at Apple. And then finally, we'll give an overview of how we've adopted these technologies within the Apple ecosystem and then also provide some guidance on how you can take advantage of these within your apps for your use cases.

So let's talk about media today. The world is becoming more and more visual. And both consumers and producers are generating more and more video and media related content. Not only that, the content is taking new forms like high resolution 4K video, HDR video, wide color or wide gamut video and so on. The nature of media is also changing with, you know, our personal favorite being live photos but there's lots of content out there, short-form video and so one.

Bandwidth continues to be at a premium. And certain applications and use cases like over-the-top video delivery and wireless networks place a premium on the amount of bandwidth. And anything we can do to reduce the bandwidth requirements really helps out those use cases. So we've been using H.264 and JPEG for a while.

There are limits to what these codecs can do in this evolving landscape with these new challenges. And that leads us to HEVC. We were looking for a next-generation codec that we could use both for movies and for photos or images. And we were looking for something that was going to give us significant benefits over what exists today.

So having evaluated our options, we decided to select HEVC. HEVC stands for High Efficiency Video Coding. It is a state-of-the-art industry standard that was adopted and approved in 2013. It was ratified by ISO as MPEG-H part 2 and by ITU as H.265. We're going to be calling it and referring to it as HEVC. And Apple is adopting it as its next-generation codec.

So we'll spend a couple of minutes talking about what makes HEVC such a great codec. Now HEVC is similar to H.264 in that it is a codec that processes videos and frames in blocks. And it uses temporal and spatial compression techniques to get the compression benefits. Now H.264 has a notion of macro blocks, which are 16 x 16 size blocks that are used within the codec for processing.

HEVC introduces notion of CTUs or coding tree units. And these it start down at 4 x 4 and go all the way up to 64 x 64. And it's when you can use these larger sizes is when you realize the greater compression benefit. And this is especially true when you're dealing with high resolution videos and images. You can really take advantage of this.

Similarly, H.264 had 4 x 4 and 8 x 8 DCT or discrete cosine transform, whereas HEVC not only uses a discrete cosine transform, it also uses a DST, distribute sine transform. And similarly, with the coding block sizes for the transform blocks, it also goes up to 32 x 32. To get better spatial compression, HEVC introduces additional directional modes. While H.264 had up to nine, HEVC has up to 35.

A key part of being able to do a high degree of compression involves motion estimation or motion compensation. And this is where you try to find -- you basically have a block in your current image that you're trying to predict from a previous image. And what you can do is say, you know what? This block is that same block from the past but moved over by a certain number of pixels. Sometimes you don't always land in a pixel's boundary. For example, you know, something could have moved five pixels over but it could have moved five-and-a-half pixels over or five-and-a-quarter pixels over.

When you need to do motion estimation at that half pixel or quarter pixel boundary, you need to be able to generate those pixels with accuracy and precision because those don't exist. You just have to pixels at the boundaries. And so as you can see, HEVC has advanced filtering which can be used to generate those sub-pel and quarter-pel pixels with much more accuracy and precision. And that leads to better motion estimation compensation and, hence, better compression.

Finally, if you're familiar with block-based codecs, we talked about H.264 having macro blocks, sometimes you can actually see in the coded video, artifacts around the edges of that block. We call those blocking artifacts. H.264 introduced a loop filter called the deblocking filter that helps get rid of a lot of those artifacts.

Now HEVC improves upon that filter, but it takes it a step further and introduces a second sequential step where we run a sample adaptive offset filter which gives us even better results. So now these are some of the highlights that give us a better compression in HEVC. And there is more.

So having said that, how much improvement are we talking about? And this is why we're excited about introducing this next generation codec at Apple. In the case of general video content, we're seeing an up to 40 percent compression improvement over H.264. So you can generate the same quality video as you were doing previously but reduced 40 percent less bandwidth. Or you could keep the same bandwidth and significantly improve the video quality. And we've done quite a bit of subjective and objective testing to fine-tune the video to realize these benefits.

In some use cases, we're seeing an even greater benefit. So we've really optimized our end-to-end pipeline with the iOS Camera Capture. And in this use case, we're seeing an up to 2x improvement over H.264. What this means is you're able to now store twice as many movies as you could previously with H.264.

Let's talk a little bit about the specifics of what we are supporting. Like H.264, HEVC also has the notion of profiles. And in HEVC, we're supporting the Main profile, the Main Still Picture profile, and finally the Main 10 profile. So with the Main 10 profile, we are now able to encode and decode video with 10-bit precision. This means that you can represent more gray scales and more colors and increase the overall quality of the video end-to-end across our pipeline.

With HEVC, we are requiring the hvc1 codec type for playback. This means that the parameter sets have to be stored in the decoder configuration record as opposed to within the samples or the payload itself. So make sure when you are creating HEVC assets that they are the hvc1 codec type to enable playback within the Apple ecosystem.

Finally, we're lucky in that we can take advantage of the fact that HEVC naturally fits into the existing file formats. So it fits nicely within the QuickTime movie file format as well as the ISO MPEG-4 file format. So we picked a technology that is well supported within the industry and within Apple's ecosystem. It has both hardware and software support. And later on in the presentation, Gavin will go into the details about where and how that's supported.

It works already with the file formats that we've talked about. So it's supporting the QuickTime movie file format and within the MPEG-4 file format. And finally, it's an ideal codec for both movies as well as still images, allowing us to use it in place of H.264 and JPEG.

Now as I've mentioned, HEVC can use the existing movie file formats. That's not the case for when we want to use it for photos or images. So we needed to find a different file format that we could use for images that would allow us to use HEVC as the codec and that's where HEIF comes in.

So before we get into HEIF, let's talk a little bit about the requirements that we wanted satisfied when adopting a new file format for images. We wanted support for state-of-the-art compression technology with HEVC being the primary consideration. We wanted explicit support for alpha and depth channels as primary asset types. We needed support for animation with animated GIF or JIF and Live Photo. In addition to just still image support, we wanted support for image sequences, to be able to compress and represent those in a file format such as what happens when you're dealing with bursts of photos.

And finally, with images getting larger and larger in size, it was important for us to be able to process these efficiently with low latency and a reasonable amount of performance. So when you've got a massive image, you want to be able to download it and start to preview it immediately.

And if you're interested in a particular smaller region in a much wider image, you want to be able to just decode and process the relevant tiles. And so we were looking for something that would allow us to take advantage of this kind of a pipeline. So with these requirements in mind, we looked at a few options and we selected HEIF.

HEIF stands for High Efficiency Image File Format. And like HEVC, it is also an industry standard that was ratified by ISO in 2015. It's based on the familiar ISO Base Media File Format. It is extremely feature-rich. And it supports both individual images and sequences. There are many use cases in addition to that, that are supported by this file format which allows for future extensibility.

It typically uses HEVC for compression. It has the option where you can use other encoders. But at Apple, when we're generating HEIF assets, they will only be using the HEVC encoder. So why do all this for images? We saw the benefits for movies and videos. And for images, we're also seeing tremendous gains over JPEG. So now using HEVC within HEIF, you're able to capture and store twice as many images as you were able to previously with JPEG.

Let's talk a little bit about the different formats of HEIF that we are supporting. So as I mentioned from an encode or asset creation perspective, we're going to be using the HEVC encoder, which means we're going to be generating the .heic file extension or .heic. So that's what we're doing from asset generation perspective. So we'll be able to create and play back those files. In terms of playback, we're also going to support decoding H.264 HEIF images and, more generally, if there's another codec that's supported within our system, we'll also be able to decode and display those HEIF files.

There is a great video that one of our colleagues [inaudible] has done which will provide more information into the HEIF file format. So -- and I think he'll be posted sometime later today or tomorrow. So please be sure to check that out if you want more details about the HEIF file format, the different atom types, and how to parse the file and process it.

So now that we've talked about these new technologies, what they are and why we decided to select them at Apple, we'd like to talk to you about how we are using them within the Apple ecosystem and then also provide some guidance on what you can do with these fantastic technologies in your apps and use cases. And to talk you through that, I'd like to hand things over to Gavin Thomson.

Thanks Athar. So my name is Gavin Thomson. I'm one of the engineering managers for the camera and photo organization at Apple. OK. So with -- we just learned a lot of the great benefits of HEIF and HEVC. But what are the ecosystem implications? So at Apple, we've been working over the last year to adopt HEIF and HEVC into the Apple ecosystem.

So today I would like to share, talk about that adoption and share the changes that were made to help with that transition. Our goal is to make the introduction of these new formats as transparent as possible. So there are three topics that I would like to talk about. First is creation: how and where we can create HEIF image and HEVC movie content.

There is access: how and where to access HEIF and HEVC movie content. And finally transfer: so what strategies do you need to consider when you want to move HEIF or HEVC content off a capture or supported device? So first I'd like to start with access. So we've discussed the extensibility of the HEIF format for images. What are some of the characteristics of the Apple generated HEIF images? So as we heard, the file format is an ISO Based Media format.

So those of you that have look at the internals of the MP4 or a QuickTime movie file, be very familiar with the internal structure as they're based on the same standard. So I want to reiterate here, HEIF is a container format. Not too similar to the QuickTime movie file format.

It's got many more options and a lot more flexibility than a JPEG file format. Now, our image payload is encoded using HEVC. And as we've learned, that provides great compression improvements over JPEG, up to 2X. We also encode the image payload as 512 x 512 tiles. And amongst other advantages, that provides great flexibility for fast incremental loading of high resolution content.

We also have a 320 x 240 embedded thumbnail, which is four times the resolution but only twice the size of our current 160 x 120 JPEG embedded thumbnail. Why can we do this? Because the thumbnail was also encoded using HEVC. We still support an image, EXIF metadata payload compatible with the payload that we capture in our JPEG format.

So HEIF with the HEVC encoded image will be identified in the file system with a new extension .heic or .heic as we call it. So some will know this is a breakaway from the DCF 8.3 filenaming convention. So if you have any assumptions with your file name parsers, a variant three character extension, we now have the default still capture format that has four characters.

So where is HEIF decode supported? It's supported on all of our -- it's available on all of our supported platforms that have macOS 10.3 or iOS 11 installed but there's a variety of hardware and software support. For iOS, we have hardware decode on the minimum config of the A9 chip.

An example, of which, is the iPhone 6S or the iPad Pro. On macOS, we have hardware decode support on the sixth-generation Intel Core, which is the Skylake family of processors. An example of that machine is the new MacBook with the touchbar, but we have software decode support on all of our supported iOS and macOS devices.

So where do we have HEIF image support? ImageIO was Apple's lowest level image framework. And it supports HEIF as a source for decode, incremental loading, metadata, and thumbnail abstraction. This is in line for their other supported image formats using exactly the same APIs. Core Image also supports HEIF as a source for real-time image manipulation. And the PhotoKit APIs which allow access to assets in the photo library also supports direct access to HEIF, but I'll talk about this framework in a little more detail in the coming slides.

Many of Apple's many based applications will also natively work with HEIF. Notable amongst those is photos, preview, Quick Look but there are also many others. OK. Let's move to our movie format and talk a little bit about Apple captured HEVC movies. So the changes here aren't as drastic as our newly supported image format.

We're still capturing using the QuickTime file format but with HEVC coded video frames. As we've learned for Apple captured video, we're getting twice the compression that we're getting from H.264. And, once again, just to call out again, we're using HEVC to encode both our images and our video formats.

We also support both an 8- and a 10-bit encoding. So those of you that really care about image quality or deep color, we have a non-real-time, a 10-bit software encoder on macOS to satisfy your needs. You'll be happy to learn, with this new format, we have retained the three character .mov extensions, so you won't have to update any of your filename parsers for this media format.

OK. So where do we have decode support for HEVC movies? So we have both 8- and 10-bit decode which is available on all our supported platforms with, once again, macOS 10.13 and iOS 11, but there's a variety of hardware and software support. So in iOS, we have both 8- and 10-bit hardware decode on the minimum config of the A9 chip. Once again, the iPhone 6S is an example there.

For macOS, we have 8-bit hardware decode in sixth-generation Intel Core processor which is the Skylake family of processors. And we have 10-bit hardware decode support on the seventh-generation Intel Core processors or the Kaby Lake family of processors. And we have both 8- and 10-bit software decode across all of our supported macOS and iOS platforms.

OK. So where do we support HEVC movies? AVFoundation is the primal framework to manage movies. And it supports play, create, and edit workflows for HEVC content. Once again, PhotoKit will vend original HEVC movies. WebKit will also playback HEVC movies but only on devices that have hardware acceleration and all macOS desktops. There's also support to encourage HLS streams using HEVC. This represents a great opportunity to improve network throughput. And, in fact, the session directly after this is going to talk about this initiative in much more detail.

Also, we have many Apple applications that will natively work with HEVC movies, you know, QuickTime player, Quick Look photos but also FaceTime. This is a great example of HEVC usage to greatly improve network throughput. So we have decode support for HEVC across all of our devices. What about playback? This is where the distinction between decodable and playable is very important and particularly so with HEVC.

This is unfamiliar territory for many of us. We don't have hardware acceleration across all supported devices for default capture format. And this is a problem we haven't had to deal with for a long time as H.264 hardware decode is fairly ubiquitous at this point. So all of our movie formats are decodable but on some software systems, there will be formats that are much slower than real-time and really only supported for export or transcode workflows.

So how do you make the determination that a format is suitable for playback on a given device? So AVFoundation, through its API, supports the notion of "isPlayable." And this indicates whether a device's video level supports the movie for playback. If true, you should experience smooth playback without incurring any significant power or [inaudible] cost for videos of extended durations.

For example, even though Apple captured 4K30 is decodable across all of our supported systems, it's unlikely to be marked as playable on some of our older hardware like the iPhone 5S. So this is a call out to developers. It's really, really important, at this junction, for you to be observing these playable state to ensure we provide the best possible user experience.

OK. For many developers, their first interaction with original HEIF and HEVC content will be through one of the public photo APIs. PhotoKit is the widely used photos API on iOS. And during his conference, they'll be some announcements about its availability on macOS. So look to the what's new photo session tomorrow for more details on that, but it will vend original HEIF and HEVC movie content.

Also, the deprecated AssetLibrary framework is still very popular in the development community. And it will also vend HEIF and HEVC content. On macOS, there is also the Media Library API. For this release, it will only be vending transcoded representations of HEIF and HEVC as JPEG and H.264 have been an equivalent resolution.

Because of the popularity of PhotoKit to access media on Apple platforms, I wanted to highlight some of the classes through which you can access HEIF and HEVC content. When requesting images, you use the PHImageManager. Through this class, you can request original HEIF images. Also, remember requesting video objects, you can also use the PHImageManager to request HEVC movies.

For managing resources, we have the PHAssetResourceManager which allows you to manage all the discrete resources in the photo library. Amongst those could be HEIF and HEVC content. And for edit workflows, we have the PHContentEditinginput. And it will support HEIF image or HEVC movies as input to an edit session.

So a point that I really want to stress here is that if you're already using Apple frameworks to manage media, the transition to HEIF or HEVC should be transparent. On the other hand, if you [inaudible] image or video stack, you might need to revisit that integration and possibly consider adopting one of the appropriate Apple frameworks. Amongst those are ImageIO for images, AVFoundation for videos, Core Image for video frame or image manipulation. We have UIKit for presentation.

We have PhotoKit to access the resources within the photo library. So usage of HEIF and HEVC through these frameworks will be transparent. I'm going to make a second call out to the session that we have Friday at 11:00 a.m. working with HEIF and HEVC. There are lots of great code examples of using HEIF and HEVC with these frameworks. I highly recommend it.

OK. That was access. Let's move on to creation. So where and how can we create HEIF image and HEVC movie content? So, as you can see, we currently only have HEIF encode support and hardware on iOS with minimum configuration being the A10 Fusion chip, an example, of which, is the iPhone 7 and the iPhone 7 Plus.

So the notable exception at this point in time is HEIF encode support on macOS. How do we create HEIF images? ImageIO supports HEIF as a destination. So you could consider transcoding your JPEG resources to HEIF for great storage or network benefits. Also, the AVFoundation capture APIs will support HEIF captured directly from the camera.

As Athar said, so all the Apple camera modes will default to HEIF with the bursts being the only exception. So if you have installed the seed builds on supported hardware and you're taking photos, you're capturing them in the HEIF format. It's also worth reemphasizing that only the HEIC variation of HEIF is supported from code through Apple frameworks. So HEIF images with the .hvc encoded image.

Let's move to movie creation. Here are at the table showing where we have HEVC encode support. So in iOS, we have 8-bit encode support with minimum configuration of the A10 Fusion chip, the iPhone 7 for example. On macOS, we have 8-bit hardware encode on the sixth-generation Intel Core processor or the Skylake family of processors. And also on macOS in software we have a 10-bit encode support.

How do we create HEVC movies? So AVFoundation is a framework for movie creation. And HEVC movies can be created through an AVFoundation export session. So you could export H.264 to HEVC for great storage or network optimizations. You can also capture HEIF movies directly through an AVFoundation capture session with a camera. Also, all the current movie camera modes will default to HEVC encoded movies. So, once again, if you've installed the seed builds and you're taking movies, you're capturing them as HEVC movies.

That was creation. Next we come to transfer. So what questions should we be considering when we want to transfer HEIF image of HEVC movies from a creation device or other supported devices? When transferring HEIF or HEVC off a supported device, you don't have the same ecosystem to code support that JPEG or H.264 provides. You might want to consider transcoding. There are a few approaches you might want to consider. The first and the simplest would be to always transcode. Another option might be to support a capabilities exchange. Let's start with looking at an example for workflow where you might always transcode.

In this example, you have your own social networking client that allows users to add HEIF or HEVC content to a timeline. So with this architecture, there is really no opportunity to evaluate the capabilities of all the receiving devices and possibly no server transcode support. So the option here is to always transcode.

So for this scenario, both supported and unsupported devices would receive the transcoded representation, for example a JPEG or H.264. Another approach that we might want to consider is a capabilities exchange. Let's take a look at an example of that workflow. So here you might have a application that is adopted, Apple's Multipeer Connectivity APIs. If you're exchanging media with a supported device, you don't want to incur the cost of transcode and high network latency of always sending a transcoded JPEG or H.264. So you could introduce your capabilities exchange in the initial handshake.

The sending device would evaluate the capabilities of the receiving device and decide whether to transcode or not. So the hope, over time, is that as the support for these formats grows, the amount of times that we need to transcode decreases. So this strategy is really suitable for both P2P and in client/server architectures.

So how is Apple handling many similar workflows? Here are a couple of examples. So it's mail. It's not really possible to evaluate the capabilities of all the receiving clients. And we don't have server transcode support so before sending HEIF or HEVC as a mail attachment, we always transcode.

Now for those developers that have share extensions, we'll also transcode before handing off a HEIF or HEVC. This simplifies that integration for the time being anyway. We've also adopted the capabilities exchange for a number of workflows. Example of those, P2P and AirDrop. So with these integrations, we always evaluate the capabilities of the receiver before deciding whether to transcode or not.

So in summary, there are a few points with regards to HEIF and HEVC that I'd really like to highlight. HEVC is Apple's next-generation codec, which we're going to use for both encoding images and videos and is providing up to two times the compression improvement for Apple captured content. We're adopting HEIF as our image file format. This image container provides us with a flexible format which we can use well into the future.

If you're using Apple frameworks within the Apple ecosystem, the transition to HEIF and HEVC should be mostly transparent. But if you need to move that content outside of that ecosystem, you should consider your transcoding options to provide the best backwards compatibility for our users. And finally, we really want developers to embrace HEIF and HEVC for creation and access workflows as we believe this will provide great benefits to not only developers but all of our customers. So for more details on this particular session, you can go to the following website, but we also have a number of sessions and labs. We can learn more about HEIF and HEVC. Just a few to highlight. The session following this is advances in HTTP Live Streaming.

We also have working with HEIF and HEVC at 11:00 a.m. on Friday. We have a great video, which you can learn more about the HEIF image format. Thanks for your time today. And we look forward to answering your questions at the labs and sessions for the rest of the week. Thank you.