Hi everyone, I'm Seema and I work on the App Store business management team. Today I'd like to talk about how to make the most of your product page optimization efforts on the App Store. We've designed this presentation for people familiar with the feature who are hoping to continue to improve their app or their game's performance across the App Store. This is a follow up to the previous session titled Get Started with Product Page Optimization. If you've not used product page optimization, and are interested in learning more, we recommend watching the Get Started with Product Page Optimization video which is available on the Apple Developer website.

Today we'll begin with the brief recap of product page optimization on the App Store. Then we'll share some success stories showcasing developers who have used the feature. Lastly, we'll share some key takeaways to keep in mind as you continue to use this feature. Starting with a brief summary.

Product page optimization is an App Store feature that gives you the ability to test up to three alternative treatments against your original product page assets. So you can find out what resonates most with your audience. It gives you a data-driven way to optimize your product page assets across the App Store so you can learn what's most effective at getting visitors to download or re-download your app or your game.

Treatments are shown to a percentage of people across all traffic sources who are selected at random in a privacy friendly way. People will see a particular treatment when they browse the today, games, and apps tabs, or on the search tab in relevant search results. Additionally, people coming to the App Store from external referral links such as social media posts or paid advertising can see the treatments.

So here's an overview of the feature. You can test your app icon, different screenshots, and app previews. You can create up to three treatments. You can choose the percentage of traffic that'll see the treatments, and you can select specific localizations in which to run your test. Now, as a part of a submission flow specifically designed for this feature, you'll be able to submit your product page test assets for review independent from the app binary. Now, as with any optimization feature, you need analytics to help you understand the performance of each test.

You can view your test progress in App Analytics, under the Product Page Optimization section. The Product Page Optimization dashboard includes indicators to help guide your analysis as your test progresses. These indicators tell you when a treatment is likely performing better or worse than your original product page, with at least 90% confidence.

Now, what do we mean by 90% confidence in a test result? It means that if we ran the experiment 10 times then at least 9 of those 10 would have a similar result. Meaning the difference between the baseline and the variant is not due to random chance. We've developed a unique methodology that lets you continuously monitor your test while it's running so you can identify exactly when you've collected enough data to make a decision. You can also stop the test at any time.

In summary, the Product Page Optimization dashboard provides you with the information you need to help you make data-driven decisions, and identify which treatment converts users the best. Now, it's important to note that product page optimization testing is not a one and done. It is an iterative ongoing process. So what do we mean by iterative? It is improving your app's conversion rate in the form of an ongoing optimization loop.

Here are a few suggested steps to include as a part of the optimization loop. Most testing efforts start with conducting research. For example, this could mean looking into internal usage data, customer surveys, or market research. Next is creating a hypothesis. After conducting your research, creating a hypothesis can help determine which test to run and what to expect from it.

Your research and hypothesis will inform what assets you will create for the test. And once your assets are ready you'll want to implement the test. At this point, you will upload your assets into App Store Connect and submit for review. Next, it's time to analyze the results. As the test is running, and once complete, you can see the results in App Analytics. Based on the results you may decide to apply one of the treatments to your product page or keep the control as is.

Use the results to inform next steps and iterate from there on. Now, I'd like to note that these are recommended steps and they might not apply to every testing scenario. For example, based on the results from one test you might not need to research for the next one. You might choose to start with the hypothesis and go from there. Keep this optimization loop in mind and once you test, continue iterating on that test, in addition to creating new areas to test.

Now let's take a look at some testing themes and examples where developers have successfully leveraged this feature. We've seen developers test a variety of themes when optimizing their product page assets. Here are the most common ones. Developers have tested various creative designs or inclusion and exclusion of their brand across their product page assets. As expected, testing seasonal content is a common testing theme, where developers wanna see if seasonal assets convert when relevant.

Developers are also testing reordering their screenshots to see which one converts better as their first screenshot. Another common testing type is testing orientation of screenshots and app previews to see if landscape or portrait converts users better. Developers are also learning from testing various callouts on their screenshots, to see which callout or text works better to convert users. Testing by asset type is also a common testing theme, where developers are testing to see if having an app preview converts users higher than not having one. There are many more themes that developers are exploring. These are the most common ones we came across.

From these themes, we will focus on the most popular ones and share developer success stories for each today. The first success story is where a developer tested branding through the design of their app icon. The second one is where a developer tested seasonal content using their screenshots. And the third success story is a developer that tested asset types like app previews. Starting with the success story from Peak Brain Training, a brain training app created by the developer PopReach.

Peak Brain Training is a freemium educational app that offers games designed to help you expand your mental skills and challenge yourself. The developer PopReach decided to start with testing their app icon to find out what creative converts best. Their hypothesis was that a robot character would convert better than the control.

In this case, they ran the test for 44 days with an average of 154,000 impressions per treatment during the test. Now, the reason we included sharing the duration, and the volume of impressions per treatment, is that when it comes to gaining insights from your test, the accuracy of those insights is directly dependent on time and impressions.

As much as there are many variables that impact a test reaching statistical significance, volume of impressions and time play a major role. So here's what they tested. Starting with the control as you see here on the left. Next, you see the icon treatment where the developer changed the background.

Here you see the icon treatment with the design of a brain. And here is the icon treatment with the robots. If you remember this is the one the developer thought would convert best as per their hypothesis. And here are the results. The icon with the brain performed better than the control, with an 8% higher conversion rate and over 98% confidence. So the takeaways are across four variants testing confirmed that the icon with the image of the brain converted users the best.

Results were different from the hypothesis, the robot icon performed worse than the control, which means that the developer was able to make a data-driven decision informed by the results from their product page optimization testing. After concluding the test, the developer applied the better performing icon to their next app version. Although they did this once they ensured the downstream performance of those users met their goals. The developer has plans to continue to iterate and test additional assets to inform their ongoing optimization efforts.

Next up we have a success story from Angry Birds 2 by Rovio. Angry Birds 2 is a freemium game in the puzzle action category, with an addictive slingshot mechanic, stunning graphics, challenging multi-stage levels, and scheming boss pigs. For their product page optimization test, Rovio wanted to determine whether their seasonal creative performed better than their evergreen creative. After doing some research, they came up with the hypothesis that a screenshot featuring seasonal gameplay would convert better.

They ran the test for 20 days, during the month of December, and saw an average of 2 million impressions per treatment during the test. Here's what they tested, the first image is the control where their evergreen art is their first screenshot. On the right is their treatment, where the first screenshot is different and features holiday themed gameplay.

As a reminder, this is the one they thought would convert higher and here are the results. To their surprise, the original product page with the Evergreen art performed better than the seasonal creative with a 1.5% higher conversion rate, and 100% confidence. While in this scenario, this particular seasonal art did not perform better, we recommend that you test seasonal creative for your app or game and let the data help decide what works best for you.

So the takeaways from this test are, across two variants testing confirmed that the control, meaning the one with the evergreen creative converted better. Results were different from the hypothesis. Now their hypothesis was informed by the fact that seasonal assets have converted better for them in other scenarios, but in this case, the results were otherwise. Again, I would like to remind everyone that these results are specific to this developer and this creative, and not to apply this as a general rule for your app or your game.

Important to note that in some cases the control may perform better than the treatment, which is a success. In this case the results from the test inform the developer to keep their first screenshot as is, which might not have been the case considering their hypothesis. The Angry Birds 2 team has conducted many tests. As the marketing team works closely with the product team to plan around their version release schedule. They make sure that they have assets prepared and ready to go prior to release, so they can start testing immediately post-release.

Our last success story is from Simply Piano by the developer Simply. Simply Piano is a subscription app used by millions that helps you learn how to play the piano in a fast, fun way, with any piano, or any keyboard. For their product page optimization test they wanted to determine if having an app preview would convert users at a higher rate. They hypothesized the original product page with an app preview would convert better. They ran the test for 12 days with an average of 430,000 impressions per treatment during the test.

Here you can see what they tested. The first image is the control, with the app preview which you can see playing here. This is the one that they thought would perform better than the treatment as per their hypothesis. The image on the right shows their test treatment, which includes a screenshot instead of an app preview.

And here are the results. The treatment with the screenshot, meaning no app preview, performed better, 3.3% higher conversion rate, and 100% confidence. In many cases, having an app preview can have a very positive impact on your conversion. But in this case, the particular video that they tested did not perform better.

As mentioned before, what's great about product page optimization is that it helps you determine whether a particular asset will improve your conversion rate with data. Now, when it comes to app previews, keep in mind, you can test up to three videos in the same test to see which video might perform best. So the takeaways from this Simply Piano test are, across two variants testing confirmed that the treatment, meaning the one without the app preview, converted higher.

Results were different from the hypothesis, similar to what we saw with the Angry Birds 2 example. Their next steps is to continue to iterate and test again as the findings were different than the hypothesis, which surprised them. So we're hoping that these examples help inform your thinking as you implement your product page optimization testing. Next, I'd like to share some key takeaways for you to keep in mind as you think through your ongoing optimization efforts.

When planning and implementing your test take an iterative approach. Revisit your assets over time to account for changes in your audience, seasonality, et cetera. This will ensure you get the biggest impact with your testing efforts. As you saw in our success story, sometimes the data proves that the hypothesis can be incorrect. More the reason to iterate and test again.

Limit the number of variables in the test by testing a single asset at a time. This will ensure you can easily identify what has brought the biggest impact to your conversion rate. We recommend running your test for at least one week and in seven day increments. For instance, if your top users are more active on the weekend and you start your test on a Monday and run it for three days, you're missing a key period of traffic and your results won't be scalable.

While testing, also aim for your treatment assets to have notable and significant differences in order to make an impact. If differences are minor or subtle that could influence the amount of time it takes for your test to reach confidence. You can see the results as the test progresses to help you anticipate how long each test is likely to take. Now, if your team is okay with a lower confidence level then you have the choice to stop the test at the confidence level of your choosing.

Here's some takeaways related to the submission process. When running optimization tests, plan and coordinate closely with the people responsible for the release schedules to ensure tests have the most time and the best chance to achieve necessary confidence. Product page optimization assets can be submitted independent of the app release. With that, we recommend having your product page assets ready to go for the next open window between app releases.

This will ensure you can maximize the time between app releases for testing. If you plan to test your app icon, remember to submit any app icon assets with the version of the binary being tested. To ensure a smooth submission process, be sure to follow the App Review Guidelines before submitting your product page assets.

And here are some additional considerations to keep in mind. Keep testing on the App Store independent from any other platform. Avoid replicating alternative platform results on the App Store. Apple and other platforms are diverse with marked differences in user behavior, and most importantly, user interface. Meaning that identical creatives would not be seen the same way. Look out for external factors, like traffic mix, addition of a new campaign, weekly and seasonal changes, and keep these in mind as you assess and measure the impact of your tests.

So that's all I had for today. In closing, I'd like to share that as a part of your optimization strategy, we highly encourage you to continue testing to see what works and does not work for your app or your game's audience. Have it be a simple test like reordering your screenshots or something more complex like testing app previews. Keep at it to ensure you're maximizing your app's presence on the App Store. If you're looking for more information or resources about what we discussed today you can visit the Apple Developer website. Thank you.