Hi my name is Quincy Jermyn, and I’m a software engineer on the visionOS Team. One of the features I use the most on Apple Vision Pro is Mac Virtual Display, which lets me work on my Mac through a virtual screen provided by Vision Pro. In macOS and visionOS 27, it’s even easier to preview content from Mac to Vision Pro. I’m excited to present Spatial Preview, a framework that lets people leverage the spatial computing capabilities of visionOS when working with content on their Mac.

You can check out the Spatial Preview framework in action with the Preview app for macOS, alongside other features to help people work with spatial content including, 3D content editing, photorealistic rendering, camera viewpoints, and spatial media output like Apple Immersive Video frames and spatial photos. Using Mac Virtual Display, people can share content seamlessly to Vision Pro.

On visionOS, the Quick Look app receives this content and now allows them to work immersively like moving around to the different cameras set up for this living room scene. These are tools that are very useful to get a sense of scale and layout of a 3D design, like this one!

With Spatial Preview, people can now extend content from their Mac into the world around them and these features aren’t just available in the Preview app. They’re also exposed as an API for you so you can augment new or existing macOS applications to take advantage of the power of Vision Pro.

These tools enable dynamic workflows with spatial content, including live synchronization and editing across devices. With real-time synchronization, apps like Cinema4D and SketchUp are transforming the creative process, unlocking real-time, collaborative 3D workflows like iterating on material changes live. And in this session I’m going to cover how you can too!

I’ll start with an overview of the framework designed for developers building content creation tools or any app that works with spatial content. Next, I’ll walk through an example workflow showing how to share and update documents like Apple Immersive Video frames in place. Lastly, I’ll show how you can create apps that fully immerse people in the 3D content they’re working on, live, by using Universal Scene Description, which is a format used to describe 3D scenes.

By the end of this session, you’ll have everything you need to leverage the Spatial Preview framework and enable people to work with content from their Macon visionOS. Now lets take a look at the components provided by the framework in order to get a sense of just how easy it is to set up.

The first step is selecting an endpoint that points to a device to share to. If someone’s actively using Mac Virtual Display it’s easy to use the device that’s already connected. Alternatively, you can add the Device Picker UI to your app so people can pick from any nearby Vision Pro on the same iCloud account. Next, create a Spatial Preview Session for the content. There are two types of Preview Sessions, Document Preview Session covers types like spatial photos, videos, and document types like PDF. USD Preview Sessions handle 3D content.

Once the session starts, Quick Look is launched on vision Pro and the content your app provides to the session appears. No code is required on visionOS to set this up! Now that you have an overview of how a Spatial Preview Session works, let’s look at an example of how to send and update documents with Document Preview.

In this example I’ll take a still from an Apple Immersive Video which is generated out of the Preview app on Mac. Then, I’ll use the Mac Virtual Display endpoint to start a Document Preview Session, I’ll then provide the image to the session so it appears in Quick Look. Let’s look at the code to set this up.

First, create a ConnectedSpatialEndpointObserver to obtain the Spatial Preview Endpoint from Mac Virtual Display. Then create a DocumentPreviewSession specifying the document type and a name for the session. With the returned endpoint start the session on the selected device and provide the content URL. Remember, Mac Virtual Display may not always be active so consider integrating the SpatialPreviewDevicePicker as a view in your UI to select a different device. Use a sheet in SwiftUI to control when it appears. Then, proceed to create the Document Preview Session just as you saw previously with the device endpoint selected in the UI.

It’s this easy to get started with Spatial Preview. Let’s see the result. Putting the code I wrote into a button in the UI launches the content on visionOS when pressed and I can view it in immersive mode, this is pretty cool! Let’s take this one step further and build a gallery of immersive architectural renderings.

When you call updateContents, you’re reusing the same scene that was launched when you started the session. If you make a new Document Preview Session and call start, that will launch a new scene. So for the gallery view I’m going to want to leverage updateContents to ensure I re-use the same scene. In this view, I’ll create a row of buttons for all the video frames in the gallery and make the buttons call updateContents on the session to swap to a different file when selected.

I also set up a task that observes the session state. If the scene is closed on visionOS, I’ll receive a state change here that the session has been invalidated. When you’re done with a session, call close to end it visionOS will automatically dismiss that scene. Now the gallery switches between immersive renderings live within the same scene using DocumentPreviewSession’s updateContents function. This is a great way for people to review design renders and get a better sense of the content they’re working on at scale.

Beyond Apple Immersive Video, many other content types work well with Spatial Preview, including Spatial Photos, PDFs, standard images and files and also 3D content! Now that you’ve seen how easy it is to send and update documents, let’s look at how to work with 3D content using USDKit with Spatial Preview.

If you’re new to USD, I recommend starting with the video “Understand USD fundamentals” and to learn how to use the new Swift USDKit framework, I recommend watching “Discover USDKit and what’s new in OpenUSD”. In this section I’m going to cover how to use USDKit to work with 3D content from your macOS app on visionOS.

I’ll go over how easy it is to set up a USD Preview Session and start navigating a 3D scene, looking through cameras and applying material overrides. I’ll then cover how to edit content, and how people can interact and make changes to USD on visionOS including adding annotations and moving objects around.

Last, I’ll go over the events and observable properties of a Spatial Preview Session, including events for animation playback and session synchronization progress, and how to customize the features available in a session. Creating a USD Preview Session is similar to a Document Preview Session except that the content is a USDKit stage.

When you call session start a scene opens on visionOS and the USD content appears in a volumetric view, people can then choose to go into an Immersive View to see it at full scale. Lets see what the code looks like to set this up. Just like with document sharing, begin by choosing a target device endpoint. Then use USDKit to load USD content into a stage, provide that stage to a USDPreviewSession, and then start the session on the selected device endpoint.

The content appears in 3D on visionOS in this bounded view. I can rotate the scene around to inspect it and then go Immersive to see it at full scale. There are some cameras set up in this scene, as you saw before, so selecting one here will move me to that viewpoint. I can also look more closely at the geometry by overriding the materials to be in Wireframe mode.

This is all functionality built into visionOS when using Spatial Preview, no additional setup is required from your Mac app. USD scenes can contain incredibly high fidelity content, which might be too complex to render on Vision Pro. By default Spatial Preview automatically optimizes the USD content before sharing, including mesh decimation, texture downsampling, and potential full scene reconstruction if necessary, all to ensure that it performs well on Vision Pro. If the scene needs to be reconstructed, it won’t be editable but people can still view it and add annotations.

To opt out of this optimization, pass the .unmodified parameter when creating the session. However, if optimization is disabled, complex scenes may not be shareable to Vision Pro and an error will be thrown from the start function. For guidance on how to reduce the rendering cost of your content, check out the Developer Documentation.

Now, let see how editing USD content works on macOS and visionOS. In a USD Preview Session, people can make changes from either device! A live USD stage is used to replicate content between macOS and visionOS when calling regular USDKit APIs. This is very useful if someone wants to make changes to the content on their Mac and see them spatially on visionOS, or capture the edits during a review session on visionOS.

Before jumping into the code, lets go over some details of what a USD stage is and some USD terminology. A stage is composed of one or many layers and layers contains USD prims. A USD prim is an object that can represent many different features like a 3D transformation, or a mesh. And variant sets can swap in alternative data on a prim.

I’ve set up some USD variants in this living room scene that change the position and rotation of the furniture using variantSets. When I select one of these variants in USDKit, the stage is changed and the edits that move the furniture to different locations are applied and synchronized between macOS and visionOS.

The buttons along the toolbar here switch between variants on macOS and the resulting furniture layout is reflected on visionOS. I can also select a variant in the Quick Look menu and see this change on Mac, this is a great way to quickly iterate on content from the Mac on visionOS. Now, let’s see how to listen for changes coming from visionOS.

Changes to this USD stage are automatically synchronized and you can observe the updates happening in a session like when someone adds an annotation using standard USD notices. Here I subscribe to ObjectsDidChange notices that give information about which USD prim paths have changed. I can iterate through those paths and find annotations to update them in the UI.

For a text annotation you’ll want information like author and a unique identifier for the annotation in addition to the actual text note. Setting up annotation data in this way allows annotations to show up on visionOS as long as they are children of a USD prim specified as a documentAnnotationGroup.

In order to have USD prims be editable on visionOS using gestures, they need to have the spatialEditable metadata set on them. People can also easily set this up on their USD asset using Preview on macOS. Let’s see this in action. When I apply an annotation on macOS it appears on visionOS, and if I comment here, ’I like layout A’, that is reflected back in the app on macOS. And since the furniture has spatialEditable metadata, I can move it. Say I want to move this chair out the window, that would show up on macOS as well. Let’s go over the events you can listen for and the options you can configure within a USD Preview session.

You can control what features are available in Quick Look on visionOS using the option set on USD Preview session start. By default, annotations, object manipulation and USD export are all enabled, by default. You can also subscribe to non-USD events like animation playback time via Spatial Preview API events.

You can monitor the sharing progress of a session using ProgressReporter, this is useful if you’d like to display a loading bar on macOS while large data is synchronized over to vision Pro. For example, here, I’ve hooked up the time and playback events provided by USD Preview Session. Now, I can hit the play button and see this hummingbird animate outside the window in both apps.

When you adopt Spatial Preview in your app, you can unlock so many new workflows for creators. And SharePlay support is built in on visionOS, so collaborators can join the same live session to review and edit spatial content simultaneously! Here, my friend Joy flagged an issue with my room layout, so I’ll adjust it in real time. Every participant’s view updates immediately, eliminating the back-and-forth of traditional asset review.

So how can you get started with Spatial Preview today? We recommend using the USDKit Swift APIs as Spatial Preview works seamlessly with it. If you already have USD in your Mac app, you can set up bridging to transfer edits between your USD installation and USDKit. For more information, check out the Spatial Preview Developer Documentation.

Next, explore how you can leverage each type of Preview session; Document Preview Session and USD Preview Session. They integrate easily into your Mac app whether you’re looking to preview and update documents or work with 3D content immersively, it’s only a few lines of code to get started. You can go even deeper with 3D using USDKit and Spatial Preview to support live editing of a USD stage, enabling real-time collaboration across devices, including over SharePlay.

And finally, take advantage of asset review tools like material overrides, camera viewpoints, object manipulation, variants and annotations. The team and I can’t wait to see how your apps will take advantage of Spatial Preview, whether you’re making a creative app, content review tool, or something entirely new. Thanks for watching!