Building Modern Web Media Experiences: Picture-in-Picture and AV1 (Chrome Dev Summit 2018)

You are tired. I can feel it, so I
will cut to the chase. Media on the web matters. And I’m not just saying
it because I deeply believe in this. Let me share some
numbers with you. Almost 40,000 years of video
are watched every day in Chrome. You may want to take
some time to digest this. 30% of all time on
Chrome for Android is spent watching video,
and it is about 15% of all time on
Chrome for Android. It is huge. And I’m not the only
one to have noticed it. Media PWAs across the world
have seen business impact. Spotify globally
and Ghana in India are both in great success. And it’s just the beginning. In the next 20 minutes
or so, Angie and I will watch you [INAUDIBLE]. We think will help you
build great and modern media experiences. We’ll cover multitasking with
picture-in-picture, bandwidth saving with the 81 video codec,
a brand new way of switching codec seamlessly, and finally,
playback quality predictability with the media capabilities API. Let me tell you a little
bit about myself first and how I work. I love to multitask
on my computers doing many thing
at the same time– browsing, obviously,
writing some code, sharing news on social
media platforms, watching educational
videos, and so on. This is what I do
on a daily basis. And I’m quite sure I’m not
the only one to do that. But you may wonder,
Francois, this looks cool, but are you good at it? Does that make you
more productive? This does not matter. I love to do that, and I
want to be efficient at it. But it is not always easy. For instance, watching a
video while I’m coding, how’s that work? First, I have to open a
separate YouTube window, move it to a corner
of my screen, and making sure other
windows are not covering it. And only then I
can start to enjoy. But can you imagine
my frustration when window position
are not remembered, or when some new window open in
the middle of my little video? But that’s OK. That’s OK, because today,
a brand new web API called picture-in-picture solved
that very specific use case. Picture-in-picture,
also known as PIP, is a common feature
in television that allows users to watch
video in a floating window always on top of other
window, so that they can keep an eye on what they’re
watching while interacting with other sites’ obligations. The BBC website actually
picture-in-picture a month ago. And they’re quite happy with
the early result they got. Now, like I said earlier,
I like to write code. So let me show some code. To enter picture-in-picture
on the video element, you simply have to call
request picture-in-picture on a video element. And because this
call is asynchronous, it will return a promise that
can either resolve or reject. And I’ll explain why
it can reject in a bit. The important thing
to notice here is that the user has to
interact with the page first to be able to enter
picture-in-picture. In this example,
I use the button. Making this button
a toggle button is quite straightforward. By checking if the
video element is not the document that
picture-in-picture element, in other words, already
in picture-in-picture, I’ll proceed as before. If it is, let’s call document
dot exit picture-in-picture. Requesting picture-in-picture
may reject for several reasons. The most common ones being
the video metadata not loaded yet, picture-in-picture
not supported by the platform, or simply not
allowed by the user. The full list of reasons is
available in the documentation. Updating your website
when the video is playing in [INAUDIBLE]
picture-in-picture is crucial. And you may think that
waiting for the request picture-in-picture promise
to wait is good enough, but it is not. What if the video enters
picture-in-picture from another path, for instance? What if the user clicked
the browser context menu, for instance, or the browser
triggered picture-in-picture automatically, like Chrome
does on full-screen video on Android? This is why I strongly
recommend you update your layout and enter and leave
picture-in-picture event listeners. Now, having a 4K video
playing in a small window may not be what you want. So to adjust the
quality of the video based on the
picture-in-picture window size, you can simply check
the width and height attribute of the
picture-in-picture window available in the enter
picture-in-picture event. Too many
picture-in-picture, I know. Adding a resize
amount to this object would also let you know
when to user resize the picture-in-picture
window so that you can update the video quality. By the way, Angie
will work you later on how to change seamlessly
the video quality and the codec container
to help with that. And finally, defining the
availability of your custom picture-in-picture
button should be as easy as taking the
Boolean value of document dot picture-in-picture
enable, but it is not. Because you want your
website to be perfect. So you also have to check
if the HTML video attribute desirable picture-in-picture
is present. And finally, for
real this time, you have to check if the video
is actually ready to play. And only then, you’ll get
a perfect implementation for your custom
picture-in-picture button in your media player. I’m glad to say that we have
shaped picture-in-picture API last month Chrome 70 in
Linux, Mac and Windows. Chrome OS and Android
are coming soon. And we’re looking forward to
see all the browser vendors implement this API as well. You’ll find all documentation
and sample units at this URL. Now, what if I tell you this
is just the tip of the iceberg? In Chrome 71, currently, beta
will support media stream video in picture-in-picture. These two little
lines up java script do what you think it does. Your web cam video stream
in a picture-in-picture in picture window. And this already makes me happy. And in case you’re wondering,
those are real fake glasses. [LAUGHTER] But wait, there’s more. Soon, a brand new web API,
called screen capture, will allow a website in Chrome
to capture a screen to a media stream for recording or
sharing over the network. This API will enable a
number of web applications, including screen sharing. Imagine now if you combine these
APIs with picture-in-picture. Let’s have a look at this code. After getting the
screen, we display media. And the voice audio stream
will get user media. I create from scratch
a new media stream, but contains the
screen video stream, including my picture-in-picture
window and my voice as the audio stream. This code is simply
gorgeous, in my opinion. This is it. There’s nothing more. So let me show what this code
does with short demo I’ve created just for you. And can we switch
to demo, please? So on the left is me,
Francois, the hardcore gamer. On the right, it’s still me, but
this time, as a casual viewer. And what you’re going
to see on the left is me sharing my
screen, including the picture-in-picture
window, while I’m playing the diner game. So this is me. Hi, mom. And let’s say [INAUDIBLE]. So like I said, I’m
a hardcore game– OK, sorry, I lied. [LAUGHTER] So keep in mind that
the code you’ve just seen with 10 more
lines of java script involving the media we called
API and some web sockets are pretty much the
entire code for this demo. Can we switch back to
the slides, please? The demo is available
for you on Glitch if you want to play
with it later on. Now, some of you may
have noticed that the picture-in-picture window
contains only two buttons– a play/pause button
and a close button. Those are blue there. We’ve talked to other
brother vendors interested in picture-in-picture
about this, and we’re happy to
share that the media session API we’ve talked
about last year [INAUDIBLE].. We’ll be using a new feature to
add and customize some actions to picture-in-picture window. Think of seat backwards,
forward, previous track, next track, and even new ones. If you are already using
it for your mobile website, this will come for free. To illustrate these
upcoming possibilities, imagine a web app that shows the
poster image of a podcast show, for instance, in a
picture-in-picture window, all right up, and use
these window media controls to tailor
the media experience. I think that looks cool. And this is coming as well. So to summarize,
picture-in-picture is great for multitasking. And in near future,
it may also be used to record your
screen with your webcam or even to create a
custom media center always on top of other windows that
users can access easily. I think we all agree
that picture-in-picture improves a lot the user
experience in general. And you know what else
improves it a lot? Video codec. And talk about this,
let me introduce Angie Chiang, a software
engineer working on video compression. [MUSIC PLAYING] [APPLAUSE] ANGIE CHIANG: Thank you. Thank you, Francois. Hi, everyone. I’m Angie from
Google’s web engine. Today, I’m going
to share with you a new generation video codec,
AV1, was launched recently. So we have three
main goals for AV1. First off, we want AV1 to
provide state-of-the-art compression efficiency
among other codecs. Secondly, we want AV1 to
be accessible by everyone, so we made it an open source
project, and it’s royalty-free. Finally, we want to deploy
AV1 right away and quickly. So before I jump into
how we did or will do to achieve these goals,
let me explain a little bit why video compression
is important for users. To visualize the importance of
compression for video service, let me give you an example. Using Edge 264, a five
minute HD-compressed video will take about 300 megabytes. On the other hand, the
uncompressed version will take about 25
gigabytes, 80 times larger than the compressed version. This means, without
video compression, watching video online will
eat up all your internet bandwidth, not to mention the
sky-rocket costs of storage on the cloud. So more compression gives the
user better user experience. We have seen the proof
of this with VP9, the predecessor of AV1. YouTube did a comprehensive AV
experiment when launching VP9. Compared to H.264, performance
improved in a number of ways, ultimately result in
higher watch time. This is due to VP9’s
outperforming coding efficiency. With AV1, we have done it again. A new generate codec
provides 30% [INAUDIBLE] reduction over VP9 across a
variety of video qualities. This means, given the
[INAUDIBLE] quality, AV1’s video size will
be 30% smaller than VP9. This project integrated
over 100 algorithm tools, including the technologies
from open source projects, like Mozilla’s
[INAUDIBLE] project and Cisco’s SOAR project. And again, AV1 is an open source
project, and it’s royalty-free. So its development community
alliance for open media has attracted 40 companies
to join and to contribute their technologies into AV1. There are content providers,
streaming service providers, and power companies which
cover a wide spectrum of the ecosystem for
video on the web. Having Google, Apple,
Microsoft, and Mozilla means we can have AV1 to work
everywhere on the web platform. So we have been working
hard toward this goal. In this quarter, AV1 decoder
is supported by Chrome browser. And we started sorting
video content from YouTube. Firefox and Edge are also
launched in the beta platform last month. And we plan to integrate
the AV1 with web RTC and to deploy AV1
into Android platform in the following years. [INAUDIBLE] support is
also under development, first arriving in 2020 for
TVs and mobile handsets. For web developers,
using AV1 is easy. You simply need to code its type
supported function to make sure the browser support AV1. Then you can start
play the video. So AV1 is very exciting. But rolling out AV1 is not easy. To ease the process of
the codec transition. a new change type
function is supported on Chrome browser, which allow
the browser to switch one codec to the other seamlessly. So for example, when
AV1 is not performing on some low-end devices,
the string service can always fall back to
VP9, which is less complex and may have
[INAUDIBLE] support. Here is a simple call
for using change type. So initially, we add a
source buffer using VP9. And later on, we can call
a change type function to switch to AV1. Let me give you some
demo about change type. So we start playing
video using Edge 264. You can see that from
the top left corner, by clicking the button,
we can switch to AV1. Now, we are using AV1. By clicking the
button again, now, we switch back to Edge 264. You can’t tell there’s
a transition, right? I think this is really cool. Let’s go back to the slide. This is a support slide,
where you can find information about AV1 decoder
and change type. Please, welcome back
on stage Francois, who’s going to talk about
playback predictability with media capabilities API. [APPLAUSE] Francois. [MUSIC PLAYING] Thank you. FRANCOIS BEAUFORT: Thank you. Thank you. Thank you. Did you ever wish you
could predict the future? I know I did. Be some kind of fortune teller. I personally would love
to jump into my future, have a quick look,
and just come back. Out of curiosity, I
asked people around me what they would do with
this kind of power. Some said they would
love to know if they would become grandparents. Some would like to see
if their project is going to be successful. And some obviously
would simply look for upcoming lottery numbers. Now brace yourself. In a sense, the
media capabilities API allows you to
predict the future, but only for a media
playback on the web. Until recently, web
developer had to rely solely on web API such
as is type supported or can play type to discover
whether media could be decoded. While this told them whether
media could be played at all, it didn’t provide an
indication of whether the media playback would
drop lots of frame or rapidly drain device battery. In the absence of
the signal, developer had to create their
own heuristic, or just assume that if
a device can play back a combination of
codec and resolution, it could do so smoothly
and power efficiently. For users with less
capable devices, this often led to a
very poor experience. By using the media
capabilities API today, you can get more information
about the client’s video decoded performance
and make an informed decision about which codec and resolution
to deliver to the user. In other words, it helps
ensure adaptive video streaming [INAUDIBLE] selection that
will play back smoothly onto the specific device. Here’s how it works in Chrome. The media capabilities API use
metrics from previous playback to predict whether future
playback in the same codec and at the same resolution
will be smoothly decoded. When you ask this API about a
specific media configuration, it will return asynchronously
three Booleans. Is this configuration supported? This is the same result
returned by these types of body you can use it for
instance to detect whether everyone video correct
is supported by the way easily by going to be smooth? It is currently true if
less than 10% of frame have been previously dropped
for this media configuration. Is playback going to
be power efficient? Am I basically going to
drain the device battery? If true, if more
than 50% of frame have been decoded by the
hardware for this media configuration. Now, warning, this is not
some kind of magic API that will tell you what to play. You are in control and
have to make decisions about which media configuration
to play eventually based on the result of this API. Speaking of results, YouTube
experimented with the media capabilities API to prevent the
adaptive [INAUDIBLE] algorithm from selecting a resolution
that a device could not playback exclusively. For users who were part
of the experimental group, the mean time between
rebuffers, also known as MTBR, increased by 7.1%. While the average
resolution served to the aggregate group
measured by video height only declined by 0.5%. These results, as obscure
as it may be to some of you, basically show that some
users on low-end devices saw less frequently on YouTube
the frustrating buffering animation. So just for that, thank you. The media capabilities
API is available today in Chrome, Firefox, and
Safari Tech Preview. As you saw, you’ll find all
documentation and sample you need at this URL. Now, if you remember
one thing from this talk about those features, it is
[INAUDIBLE] the web matters. And the web platform
is the best place to serve efficient and
delightful major experiences. One more thing– you can
find all audio and video updates in Chrome by simply
searching for Chrome media updates on your
favorite search engine. This will allow you to stay up
to date with the amazing media features that Chrome is
adding to the web platform every release. And with that, I humbly
thank you for your time. [APPLAUSE] [MUSIC PLAYING]

Posts created 3303

5 thoughts on “Building Modern Web Media Experiences: Picture-in-Picture and AV1 (Chrome Dev Summit 2018)

  1. Can't wait till we're start seeing effecient hybrid our hardware av1 implementations. But until av1 dramatically improves their encoder, it will be limited to video giants like Netflix and YouTube. Hopefully we see the fruits of some promising developments on the encoder front next year

Leave a Reply

Your email address will not be published. Required fields are marked *

Begin typing your search term above and press enter to search. Press ESC to cancel.

Back To Top