Good morning. Thank you all for coming. I'm Michael Hopkins. I'm with the Core Audio Engineering team. And today we'll be talking about music on both iOS and Mac OS X. We'll be covering four main topics, the first of which is how to use audio units to do audio processing. We'll talk about a specific audio unit, the audio unit sampler. We'll be talking about how to do MIDI on your iOS device. And finally, we'll be playing back music sequences.

Mac OS X as well as iOS provide a wide range of facilities for developers making music applications. These can range from the simple soundboard application to a more complex application that edits musical content. This hypothetical example that you see here on the screen is one that's geared more at live playback. We have on your left-hand side a file player, which is playing back content stored on disk.

We have two live inputs, a guitar and a microphone. These inputs are each going through an audio processing effect, kind of like a guitar stomp box that you'd have. We also have a music device that is playing a MIDI, triggering samples on an electric piano sampler. Each of these inputs in turn is going through a mixer, which is changing the relative volumes of each track, doing something different. Michael Hopkins, Doug Scott Mac OS X as well as iOS and Mac OS X feature a state-of-the-art audio engine, enabling a wide range of facilities for developers making music applications.

These can range from the simple soundboard application to a more complex application that edits musical content stored on disk. Michael Hopkins, Doug Scott Mac OS X as well as iOS provide a wide range of facilities for developers making music applications. These can range from the simple soundboard application to a more complex application, doing something different. changing the relative volumes of each track, doing some panning, combining all that audio, and outputting it to the speaker.

So now let's talk about how to make such an application. It seems very daunting and complex, but actually one of the fundamental building blocks for doing such processing is available for you on Mac OS X and iOS, and that's the audio unit. This is a plug-in architecture for processing audio. It is very low latency and designed for real-time input, output, or simultaneous input and output.

These audio units can be organized in an audio unit graph, and they can be controlled via properties and parameters, which do things such as set the sample rate of the unit. And on the desktop, Mac OS X, they can be controlled by the user with a user interface, and we'll see that in a bit.

There are several types of audio units that ship as part of Mac OS X and iOS. These include effects units, which do things such as apply a delay effect to audio. We have music effects, which are similar, except that they also allow the input of MIDI data to control how that audio is rendered.

We have instruments, such as the new sampler unit, which we'll be showing you a bit later. We have generators, which take no input. They simply generate an output signal, such as a sine wave generator. We have panner units, such as our 3D panners, which spatialize audio into a 3D environment. We have converter units for taking audio from one format and converting it to another.

Mixers. Offline effects, which do play a role in the sound of the audio. We have processing that can't be accomplished in real time, such as an audio unit that takes a file and reverses its contents, playing it backwards, revealing satanic messages in our favorite '70s rock. We have output units.

And now that we've talked about these different types of units, I'd like to discuss organizing them in a graph. We have something called an AU graph, and these audio units can be added and organized in this structure. Each item, as shown in my example, is an AU node, and in this example we have a microphone input going to a reverb unit, and then the output from that reverb is going to the output. This graph is defining the topology and the connections between each audio unit in the graph.

Finally, I'd like to point out that the graph must end with a single output unit, as shown here. These graphs can be made much more complex. I've now modified the example to show that we can additionally have a MIDI device that is going to an instrument, and then the audio is passing through two separate effects units before being combined by the mixer unit and then output to the output unit.

Now when we go back to our diagram of our hypothetical audio application, we can see that this very complex behavior is actually being managed by the host application by using an AU graph with several specific audio units. For example, the file player unit is handling file playback. A distortion unit is taking the input from our guitar and modifying it. The reverb unit is being connected to the microphone input. A sampler unit is what's playing those MIDI-triggered samples. And all these are going through a mixer unit before going to the output.

Now that I've given a bit of background on this, it's important to understand how we can interact programmatically with these audio units on the system. There are two main ways. The first is properties, and these are key value pairs. The properties allow a host to configure specific states, such as sample rate or stream format.

and Michael Kahn. These properties are set on the audio unit and take effect when the audio unit is initialized. And then finally, parameters are designed to change the way that that real-time processing is accomplished. For example, in the case of a delay effect, we may want to change in real-time our delay time or feedback amount. Typically, parameters are set through a user interface, as I'll show in my demo shortly.

I'd now like to take a look at AU Lab, which is our reference host application running on Mac OS X. And we're going to take a look at an example document, which contains a graph that is configured in the manner that we saw in the slides previously. We have a microphone track. That's input coming from the microphone. And we have some meters that allow us to monitor the input coming through. Exactly like our diagram, that's going through a matrix reverb effect.

And then that track has a panner and a volume control. This is actually in the mixer unit. And when we change the UI here, we're actually changing how the mixer is panning and setting the volume of that track. We have our guitar track with a distortion effect, a sampler, and then the file player. The file player is configured to play a file. And I'd like to audition that for you now. Whoa. Wake all of you up, I hope.

Okay, so that's the file that we're playing back in our file player. Now I'm going to add some effects to that track. And when I'm doing this, I'm just adding additional nodes in that graph that this host application is running. So I'm going to go ahead and I'm going to choose a distortion effect.

As you can see in this pop-up menu, this allows me to pick both a type of effect. In this case, we can have converters or panners. I'm going to choose an effect. These effects are grouped by manufacturer. So that all of Apple, for example's effects are grouped in the same location. I'm going to pick an effect. This is the distortion unit.

And what you see here is a generic, excuse me, is a custom view, which is a very elaborate user interface that allows the end user to interact with that audio unit. And when I go ahead and change the delay, for example, of the distortion unit, dragging this control changes the parameters of that audio unit. So let me go ahead and pull up a generic view of that same audio unit so you can see what's going on here.

This AU distortion unit has several groups of parameters. It has a delay group, a ring modulation group, a decimation group, etc. And each of these groups has related parameters, which are specified by the name of the parameter. They have a minimum and a maximum value, a current value, which is editable by this slider. You'll also notice that as I'm changing that parameter value, that the custom view, which is looking at the same audio unit, is showing that change in real time.

And that also happens the other way as well. You'll notice that with a custom view, I'm able to control two different parameters simultaneously, because this view allows me to do it. X and Y and not just operate on a single parameter. So let's go ahead and hear the effect of this audio unit now. Ooh, that sounds horrible. Bypass. And now applying the effect. You can really hear that ring modulator working.

Now, at any time, I can add more audio units to this chain just by selecting another audio unit. And in this case, I'm going to choose a filter. Now the audio is going from the distortion effect through the filter, and using the custom view, I can change the The audio, now I have a really, I've really cut out the high frequencies of that audio file. So although these custom views are available in AU Lab, we can use them in any other host application, such as Logic or GarageBand. So that's an important distinction. All right. Go back to slides. That was AU Lab.

With iOS version 5, we've now taken a lot of these audio units that you've just seen running on the desktop and made them available to developers. These include several new effects, such as some filters, the high-pass and low-pass. We have a peak limiter. We have a dynamics processor that allows you to do some compression. And a new reverb unit, which is very high quality, so it's great to use in your application.

We have additionally some time pitch effects. The VeraSpeed, which does only pitch, and then SimpleTime and NotQuiteSoSimpleTime, which allow you to change time and pitch independently. The difference being that the NotQuiteSoSimpleTime version is higher quality at the expense of higher CPU usage. We have two new generators, a file player, as well as a scheduled slice player, which allows you to play back snippets of audio, schedule them from any thread for rendering later. And then finally, we have an AU sampler, which we'll talk about in greater detail in a bit.

So that's how audio units are used. But how can you as a developer take advantage of them in your application? Well, first of all, you need to know how to find and load those audio units. When I talk about an audio unit, I'm talking specifically about an instance of an audio unit. And that's created from something called an audio component. The audio component is a factory, which knows how to create those audio unit instances.

Each component is uniquely identified by an audio component description, which has three key fields: a type, a subtype, and a manufacturer. The type is the type of effect. In this case, AUFX stands for effect. And then the subtype and manufacturer are provided by the creator of that audio unit.

There's also a name, which the host maybe will use to display, and a version. So there's an audio component API, which can be used to find the load and do management on the state of that audio unit. It's available on both iOS and Mac OS X. And on the desktop, it replaces the component manager, while maintaining backwards compatibility with older components.

Let's look at how that works in an application. This is identical on both the desktop as well as our mobile platforms. An application first needs to find a specific audio unit, and it does that by creating an audio component description. It passes that component description to the audio component system by calling audio component find next.

The audio component system then looks through its list of audio components and returns the matching component back to the application. The application can then create an audio unit instance by calling audio component instance new, passing that component along to the system, which will then return back an audio unit.

Let's look at more detail about the audio component Find Next Call. And there are a couple of additional pieces of information I'd like to provide here. The first is that the first argument can be null, to specify that we want to start looking in that list at the beginning.

Or if we want to iterate through several components, we can provide a specific component after which search will begin. We have the facility for wildcard searches, which would allow you to say, find all instances of effects. In order to do that, we would zero out all fields in our component descriptor, except for the component type.

Audio components are registered on the system in two main ways, the first of which is statically. The system at boot time will look through all the bundles in these three specific directory locations, four bundles with a specific extension, and then those audio units will be loaded. Components can also be registered at runtime. When you use audio component register to do so, that component becomes private to your application process. So it's not a system-wide component. In order to make that call, you provide a description for the component that you're registering, a name, a version, and a factory function.

and Michael Hopkins, Doug Scott, and Michael Hopkins, Doug Scott, and Michael Hopkins, Looking into a bit more detail about how the audio component system works under the hood, as I showed earlier, all audio components have an audio component factory function, and this is used to create instances of that component.

It returns a pointer to an audio component plug-in interface structure. This structure has three function pointers. The first two are open and close, which are used to open and close, surprisingly, that component. And then we have a lookup function pointer that takes a selector. We'll examine this in a bit more detail shortly.

Now when our application is calling audio component instance new, what's actually happening is that the system is getting that component factory function, and then getting the audio component plug-in interface by calling that. That audio component plug-in interface contains a function pointer to the open method, which is then called, and that open method returns an instance of an audio unit, which goes back to the application.

Now let's say, for example, that application has taken that audio unit, added it into a graph, initialized it, and then is ready to render. The application calls audio unit render. The audio component system gets the component, looks up the factory function, gets the plug-in interface, gets a function pointer to the lookup method, which then it provides the selector K audio unit render select.

And then the function pointer for render is returned by the lookup method, which is then called by the system. That's all I have for audio units. I'd now like to call up Doug Scott, who will be speaking about the sampler audio unit. Doug? Good morning. I'm Doug Scott.

I am also a member of the Core Audio Engineering team. And I'd like to talk to you in some detail about the new sampler audio unit that Michael mentioned a minute ago. If you remember the diagram that Michael showed, this is the location of the sample unit in that diagram.

What makes this sampler an instrument type is that it sits at the top of any branch of an audiograph and generates audio in response to commands that are sent to it via the music device API, such as note-ons, note-offs, and anything that might change the quality of that sound, like adjusting the volume and things of that sort.

What makes this a sampler is that it organizes a set of audio files that you provide it by, it organizes them and shapes them and organizes them into a coherent, playable instrument. Examples of this might be a drum kit, where you've loaded a set of sampled percussion sounds to various places on the keyboard and then play them back either together or individually.

Or perhaps you're creating a very fancy acoustic piano where you've loaded a complete range of samples from end to end on the keyboard and want to play that back as if it were a real piano. Or you could just have an arbitrary set of sound effects that you want to be able to trigger in a game or an application.

Some of the features of the new sampler are it accepts samples in any of the formats that our audio library supports, which includes compressed formats. It shares any sample resources across all instances of the sampler that are present in your application. And files that exceed a certain size are automatically streamed for you. So what this means is that you can have a fairly large body of sample content within the sampler, but have a fairly low memory footprint because you're not spending a lot of time duplicating or storing information. in memory.

In addition, the sampler has a very lightweight preset format stored as a text file, a plist file actually, which makes it very easy to transfer that from your desktop where you can audition and create, as we will see in a few minutes, to other desktop machines or into your iOS application or game.

The structure of the sampler is such that it provides an extremely flexible instrument design, which allows you to create anything from the most bare-bones instrument that plays a single sound on a single key, all the way to incredibly rich and complex instrumental timbres, all depending upon the needs of your particular application. And in addition, this sampler has the capability of translating both DLS and SoundFont 2 preset format into its own native format, which allows you to work with the huge existing body of designed instruments as a starting place before you begin working on your own custom designs.

Let's look a little bit about how the sampler patch is organized. What we have is a hierarchy of zones and layers where each level of the system is dependent upon the layer above it. At the bottom, we have zones. Zones represent the way in which any individual audio file in the system is going to be played back within the context of the entire instrument.

That includes information about its root key, in other words, where it plays without any transposition, the range of keys over which that sample will play, the range of velocities over which it will play, plus information specific to how that sample is going to be played back, involving looping, perhaps if you wanted to do some fixed gain or detuning to make it match your other samples, that's all available to be set there as well. And there's many other properties that I can't even get into here that are available there.

The layer, besides being a collection of the zones below it, is also the place where all of what we often call articulation data is provided, which means anything that modifies the quality of the sound, such as filters, LFOs that might be controlling frequency or amplitude, envelopes, of course, which can control various other things in the system. All of this is defined at the layer level. Like I said, all the modulation connections.

Additionally, there are other parameters here specific to the layer, such as the ability to select -- to determine how your zones are being selected when they're picked, and an offset which allows you to affect all the zones which are present below it. And there's several other things -- many other things you can do here as well.

The instrument is the top level. It's a collection of the layers. It doesn't expose any parameters or properties at this time, but it is also the place where all of the samples that are referenced by your patch are kept. Here's an example of a simple patch. We have a single layer with multiple zones in it. The layer extends the entire length of the keyboard, and the zones basically divide up that keyboard space into four equal parts.

Here's the behavior of the sampler if a note on is received on a particular key. Sorry. The zone whose key range includes that key number will be activated. And that zone will generate a single active voice, which is the actual audio producing engine of the system. In contrast, let's look at a slightly more complicated patch, which is layered. It's layered layers. In this case, we have two, both of which extend the entire range of the keyboard. But each one has its own individual set of zones, so you're going to have different samples in each associated with each layer.

When a MIDI note-on is received by this instrument, Because we have two zones whose key ranges intercept that particular key number, we have two zones become activated. and you generate two voices. So the idea is that a complex instrument may very well generate a set of active voices, which will combine to produce the complex timbre of that instrument.

Here's another example of a complex patch. This one, instead of having overlapping layers, has two complete layers which divide the keyboard approximately into one-third and two-third. The bottom one is dedicated to producing the timbre of a double bass, which means its zones are going to contain samples that are double bass samples. And the upper half of the keyboard is a vibraphone preset, which has its own zones representing the samples for the vibraphone. So this is a traditional split keyboard patch.

Okay, what I'd like to do now is have Michael back up, and we're going to see how we can audition and edit one of these, a patch that is loaded into the sampler using its custom view. What we have on the screen right now is the custom view for the AU Sampler in its compact form. What's displaying by default is the keyboard, which allows you to audition the preset.

What we have loaded here first is an example of a simple preset, which has, like the simple example we gave, a single layer and several zones under it. If we open up the editor portion of the window, We can see on the left a list of whatever layers are present in the file, plus the zones underneath them.

When the layer is selected in that list, you can see on the right we have an editor for the settings that are available for the layer, and when we select each of the zones, we get an editor showing the characteristics of that zone. The large rectangles at the bottom are simply a display of the currently selected zone and its location on the keyboard. Horizontally, we're displaying the key numbers that it's active over, and vertically we're displaying its velocity range.

If we go back to the layer on the left, and then bring up the parameters button on the right-hand side of the screen, here's where we have access to all of what I was talking about as the articulation data, which includes envelope settings, filter settings, any low-frequency oscillators that might be available. All these are settable here. And as I said, each layer that you have in your instrument will have its own set of these that can be configured independently.

Okay, let's bring up a second preset, which is an example of a slightly more complicated one. In this case, we have a vibraphone, which has two layers, one of which has been dedicated to the attack portion of the vibraphone, the second to the sustain portion. Let's just audition that for a minute so they can hear that.

To let you hear the way this is divided up as a complex instrument, would you please mute the one that's for the sustain portion? And now if we play this back-- Oh, we got the wrong one. Let's try the other one. It's a little more obvious when we-- There we go.

So now all you're hearing is the layer which represents the attack portion of the sound. So this is an example of how the layers combine to produce a complex timbre. Each note on when this instrument is completely active produces two voices. And lastly, what I'd like to do is show you that the presets which are being loaded and auditioned here are simply small files which are available and accessible directly through the file system as .au preset files.

We're seeing the path, the default path to these here. And these are the files that can be moved from machine to machine or to an iOS application in order to be able to play these files in a game or in an application on your phone. So what I'm going to -- I'll be talking about that again in a minute. Thank you very much, Michael.

Let's talk a little bit about how you can configure the sampler. We have three methods for loading what I call a patch or a preset. The first is using the files that I just described, which are AU preset files, text files that can be loaded. The second is that you can build up a custom patch from a set of individual audio files that you provide. Or third, as I mentioned earlier, you can load an instrument definition from a DLS bank or a SoundFont 2 file. Let's look at each of those in a little more detail.

Firstly, loading a patch from an AU preset file. The first thing that's most important if you're under iOS is to make sure that all of the resources that are used by that patch, which means any of the samples associated with it, plus the patch itself, are part of your application bundle's resource directory so that they're accessible to the sampler when it's loading.

The second thing you do, and we'll be providing some sample code for this in a future seed, is to convert that preset file into a property list using a couple of standard API calls. I won't go into those now. And then thirdly, you take that property list that you have and load it into the sampler using the set property call. This is a property of the type that Michael talked about earlier, using this 1K audio unit property class info, and that will load that preset into the sampler.

Secondly, creating a patch from a set of audio files. Again, all of your files must be available to the sampler by being in your applications bundle if you are running iOS. And if you want these individual files to be loaded and produce a zone which has information such as we talked about, which is like key range and looping and things of that sort, you can provide that information to the sampler by editing the file's instrument chunk. This is something which is available in core audio file format, the CAF file format.

If you do that, when these files are loaded, that information will be transferred directly into the zone so that you can actually have a pre-configured set of files that when they're loaded, they'll, for example, map to your keyboard in an orderly fashion that you can control in advance.

And lastly, we can load patches directly from DLS sound banks. As I said, once again, make sure that all of your file content is available to the sampler if you're under iOS by putting it in your bundle. And we have a property that you can pass to the sampler where you pass the URL to your bank, plus a bank ID and an instrument ID, which is the way in which presets are identified in both DLS and sound font banks.

And that will load that instrument definition in. All of the properties that are present in the DLS or sound font bank will be translated into a form that the sampler understands itself. What I'd like to show you now is a quick demo. having the two presets that we saw earlier auditioned on a desktop machine loaded directly into an app.

Okay, there's not much to look at. I am an audio guy. All I have here is two buttons representing trombone and vibraphone. If you remember, those are the two presets that we auditioned on the desktop using AU Lab. If I select the trombone and have my audition keys... We have the exact same timbre brought over to an iOS app simply by bringing that AU preset file plus its associated samples over and loading them into my bundle and writing an app which made those function calls that I showed you a minute ago. The vibraphone. Ooh, a little loud. Oh, well.

Now I'd like to bring Michael back up to talk about the introduction of Core MIDI, which is now available in iOS 5. Now we're going to talk about this portion in the upper right-hand corner, which is getting MIDI to your iOS device. Under iOS 4.3 or later, you can use Core MIDI, which is a set of services where applications can communicate with MIDI devices, and it provides abstractions for interacting with a complete network.

As you can see in the diagram, we have, for example, a MIDI keyboard that's connected to an iPad via a camera connection kit. Any MIDI that occurs on that device will be sent to the MIDI server in its own process, and then it will communicate with your application via the core MIDI APIs. A bit of terminology about MIDI devices. When we use the term device, we're referring specifically to a piece of hardware. This piece of hardware can contain Endpoints, and these can be either source or destination endpoints, depending on whether they receive or send MIDI data.

When an application running on an iPad wants to get information about devices that are connected and receive notifications, for example, when something's plugged in, it first needs to create a client object, at which point the MIDI server will then call that application's MIDI notify proc and tell it when a device changes, connected or disconnected, for example, or if any properties on that specific device change.

So now we have a keyboard. We're plugging it in. The MIDI server process gets that information and communicates to the application via MIDI notify proc. Looking at those types of properties that we can get information about, those include name of, say, a device, or even a particular endpoint if it's named, manufacturer information, whether that device is offline or not, hardware settings, such as whether that piece of hardware supports general MIDI or can transmit clock. We can also get information about what patch it has loaded, et cetera.

Let's look at that in code. As I mentioned, we used a MIDI notify proc to register a handler with core MIDI that it will then call back. As you can see here in this code, we're specifically handling the KMIDIMESSAGE property changed. And depending on what that means to our app, for example, if we wanted to display a picture of a keyboard when it's connected, we can then do that here.

We then create a MIDI client, passing a string for the name of our client, our notification handler that we're registering with the MIDI server, and then we get back a MIDI client, which we'll use in further calls. So now the more interesting bit, getting MIDI data to our application. First, we create a MIDI input port.

Then the MIDI server will call our MIDI read proc when MIDI messages are received. And the information that we get is a packet list containing those MIDI events. So for example, when the user plays E on our keyboard, the MIDI server gets that note on, and then calls the MIDI read proc in our application, passing the packet list of data.

Let's look at that in more detail. MIDI packets are variable length. So that means that when we're getting this packet list, in addition to the timestamp, which is the time, an absolute time that that event occurred, we have a length that tells us how much data is in that packet. We can get the first packet by getting element zero in that array, but we need to use MIDI packet next to retrieve successive packets.

You'll see that in my programming example. We're registering our MIDI read proc. Here, we're processing packets by calling MIDI packet next. And those packets that we receive, their data is just standard MIDI, and it's up to your application to use the MIDI spec to decode those packets. We call MIDI input port create, getting our client that we previously created, specifying the name for our port, the callback proc, and then we get back our MIDI port, which we can then use to connect to destinations, for example.

Finally, if our application wants to send MIDI data, we use MIDI send with a packet list of the events that we're sending. We first need to create a MIDI output port by calling MIDI output port create. And then we specify where we want that data to go. And off it goes. It's that simple.

Now, we also have the capability of not only sending and receiving MIDI events from locally connected devices, we can also get devices that exist over a network. In this case, we have an API called MIDI Network Session that uses Bonjour to register peers. So we can use the MIDI Session default session object and set its enabled property, and that registers our peer with Bonjour. Saying, "Hey, I'm here. Anybody interested in communicating with me, go ahead." And we can also specify a connection policy limiting who can connect to us. So each peer that wants to communicate in this manner creates a MIDI Network Session object and enables a connection.

Now, on the other hand, when we want to make that connection, We use the MIDI network host API to specify which of those Bonjour peers we're connecting to. And the name, address, and port, we just use using the NS Network Session API. And there's a number of other facilities and network hosts, if you have other types of information other than the name, address, or port that you can use. Please see the header for more details.

Once you've created that MIDI network host, you simply call connect with host, and then you're done. At that point, your MIDI notification proc will get called, telling you what devices are available. You'll get your MIDI read proc called with MIDI data, and that's it. So that includes my talk on Core MIDI. I'd now like to return the presentation to Doug, who will be talking about the MIDI sequencer.

Thank you, Michael. I'd like to start this part of the presentation by doing another demo. Once again, my minimalist application. But what you'll need to do is imagine that you are creating a game. And in this game, you want the player to be driving around the countryside in an antique automobile.

After a while you realize this is sort of boring, so you decide you want to make sure that when the automobile is traveling up and down the hills, the speed of your music changes. As it goes up a hill... down the other side very quickly. Now you might be able to figure out how to do that using a set of pre-created audio files, but it might be quite complicated.

What about if your automobile were to travel up to the very top of a hill and crash into a church? That kind of a musical transition would likely be quite difficult to do with a set of pre-existing musical files. But using the music sequencing API, that operation is extremely simple. So let's talk about how that all works.

So the music sequencing API allows you to record, edit, save, and playback music events that are organized in a time-based fashion. By music events here, we're not just talking about MIDI. Well, we're talking about MIDI, but we're talking about any event which is synchronized in time with other events to control your musical performance. So that includes -- that does include all MIDI events, but we'll talk about some other things as well.

The complete set of API is declared in the header that you see on the screen now. Let's talk about the sort of parts that go into the sequencing API. First, we have the music sequence. This data type is the actual storage place for all of the information that is part of your musical performance. And that information is organized into tracks.

One track will always be a tempo track, which allows you to store information about the rate at which the music plays back, which is usually defined in beats per minute. All the other tracks in the file will be event tracks, which contain information, like we said, about music events, which we'll be talking about here in a second. Next, we have the music track itself. These are contained within the music sequencer. In other words, they belong to a particular sequencer. And they are a time-ordered collection of events.

These events can be of two types. MIDI events, like I said, which includes note on, note off, controller events, meta events relating to, you know, you can add tags and things of that sort, anything that is defined by the MIDI specification. And secondly, it allows you to store audio unit parameter automation data.

Now, this is something which allows you to take parameters, such as Mike described earlier, and give them time points so that over the course of a performance, the parameter values change in a way which is synchronized with your other events. This is very powerful when you're working with a fairly complex audio unit graph because it allows you to manipulate various parameters and make sure that everything happens precisely synchronized the way you would want it.

And in addition, we're able to add arbitrary user data events into the audio unit. So, this is a very powerful tool. It's very useful for you to use. And finally, it allows you to use the music track so that during playback, you can trigger things to happen in your application that are not directly related to the playback of the sequence.

And lastly, we have the music player itself, which, when it's associated with a given music sequence, allows you to convert that sequence into a performance. Now let's look a little more about what each of these components does for you. First, the music sequence. This is the place where you add, remove, and merge the music tracks.

In addition, this is the place where you can read and write MIDI files to disk. And in addition, it has a series of utilities that let you convert between beats and time. This is useful if you're trying to display your events in a user interface, where you want to maybe provide the user with an alternate way of displaying where the events are occurring.

The music track, as I said, part of the music sequence, allows you to add, move, and clear music events. It has properties on it which allow you to mute a track, solo it, control whether the track loops during performance, and several other things that you can do with it.

A very important aspect of a track is that it is responsible for associating the events that are contained within it with a particular destination. The destination in this case is the particular portion of the system that is going to respond to the events that are in that track. The two primary places where you might send things are to audio units that are part of a graph. For example, an instrument like the sampler. Note-on events would be sent to an instrument like that.

You might have parameters which you want to have control a sound effect, perhaps in the same way as Michael showed, where you might want to be adjusting your delay time, and have this all part of this synchronized performance that is being controlled by the sequence. MIDI events can also be sent out to external MIDI devices using the core MIDI system that Michael described.

This would allow you to take a sequence and not only play back music on the device, but also play it back on the device. You could also use the track as a way to control the playback. You could control the playback on the device itself, or your desktop or your device, but it could actually control an external device synchronously with the playback internally.

So you could create an entire ensemble of internal and external devices. And in addition, the track supports the ability to iterate through its events, which is something you would need to use if you were going to build an interactive program which displayed all the events that were in all of your tracks. And there's a thing called a music event iterator which allows you to do that.

And the music player, as I said, the part that actually creates, makes it become a performance, supports all the playback controls, starting, stopping, adjusting the position, and also adjusting the playback rate. In the app that I demonstrated for you, that was the function that I used to basically slow us down and speed us up as we traveled the mountains.

And we have additional utilities here for converting between host time to beats and vice versa, which is very important when you are reading in MIDI events from the outside world, which are tagged with host times, and you want to turn those into a format that the sequence understands, which is typically in beats. We'll talk about that more in a minute. So let's walk through what happens when you load a MIDI file into a sequence. This is useful because it shows the relationship between all the components that go into actually producing a performance with that sequence.

So first thing you do is we assume we have a sequence that we've created using our creation API that I don't actually describe here. We have a function music sequence file load, where you hand it your empty new sequence, the path to the MIDI file you want to load, and some other properties. First thing that happens when you do that is the tempo track is loaded in from the file if it's present. If it isn't, the sequence will create a default tempo track for you.

Next, any tracks contained within that MIDI file will be pulled in with their associated events, and the structure of that file in terms of the layout of the tracks and the organization of the events will be mirrored exactly in the sequence that's created. Next, the sequence creates a default AU graph, which always contains a single instrument at the top, plus a limiter to just help control the volume, and of course, the necessary output audio unit.

Lastly, all the events in all the tracks that are in that file have their destination set so that the events will control the instrument that is at the top of that graph. Any node-ons, any controller events, anything of that sort will be sent to that instrument. That's exactly what was happening in my demo. In that case, there's just a single track with the piano music in it, and the node-ons and node-offs from that piano music were being sent to an instance of a sampler that was built automatically by the load of the SMF file.

To play the sequence, we create a music player and then call music player set sequence to associate the two together. And music players start. We have, and of course we can stop, we can pause, and we can set position. And the function there, the last one, Music Player Set Play Rate Scaler, is what I use to adjust the speed of the playback.

There'll be a lot of instances, though, where loading a sequence from a file is not what you want to do. You want to have a lot more control, and perhaps you want to create something from scratch. So you might want to create a MIDI recorder sequencer application, for example, a drum machine where you can actually interactively add note-on and note-off events on the screen and then hear them played back. Or you might want to play a MIDI sequence that you know has multiple instruments in it, and so you need to have a graph which is going to have more instruments than just the default one.

Or you might want to create a sequence which has a large number of tracks which contain this automated parameter information if you wanted to create some very complex system where things are being controlled in real time. Way more information is being controlled than you could control having a user interface.

To create a custom sequence, we pretty much follow the same set of steps as you saw automated in the process of loading an SMF file. We create an empty sequence. We create our own custom graph that meets our particular needs. We add any number of tracks to that sequence that we need and add events to those tracks.

Usually, tracks are designed to group events which are targeted at a particular audio unit, just for clarity, but it's up to you how you organize. And then the last thing you do is make sure that each of your tracks is targeted to some destination so that those events will be recognized and played back when the sequence is performed.

Here's an example of a custom sequence. In this case, the sequence itself looks very much like the one we saw before. We have a tempo track and a couple of event tracks. But note that we have created a custom AU graph with two instruments in it, and each of the tracks has been set to play back through a different instrument.

Lastly, you might want to actually turn this into a system which can record a live performance, which would be coming in for MIDI events sent from the outside. And you can synchronize the recording of the live events with what's playing back by simply listening for MIDI events on a MIDI input using the system that Michael described just a few minutes ago.

The events that is, as the events come in, they are tagged with a system time, which can then be converted by the utility and the music player into the exact beat and measure that is currently active in the playback. And this allows those events to be placed precisely in the track at a place where you get a time-synchronized recording of your live data combined with whatever is playing back through the sequence.

All right. So in summary, what we've seen today as technologies for music. First of all, we saw how audio units can be used as building blocks in music processing and synthesis applications. We saw the AU Sampler, which allows you to create simple or very, very rich, complex instrumental patches for use for musical and sound effects purposes in iOS or Mac OS X games and applications.

We saw how Core MIDI allows you to send and receive audio -- sorry, send and receive MIDI under both iOS and Mac OS X, either to a locally connected MIDI device or via the network. And lastly, we saw how the music sequencing API allows you to add -- to record, add, create, edit, save, and playback music events to create a complex musical performance.

I'd like to invite you to a related session which follows upon this one right away, which is audio session management for iOS. This will be of great interest to anybody who is working with anything that uses audio seriously under iOS. We're very excited to provide you with these new technologies and are very much looking forward to see what you're able to do with them. Lastly, if you have questions and we're looking for more information, here's a list of resources for all of you.

Our media technology evangelist, plus where to look for documentation. Of course, there's a WWDC website which has information. And, of course, our wonderful Apple Developer Forums, where you'll find us lurking. Thank you very much for coming, and I really hope that you enjoy the rest of your stay here.