Planet Igalia WebKithttps://planet.igalia.com/webkit/atom.xml2024-03-28T18:00:40+00:00Planet/2.0 +http://www.planetplanet.orgJani Hautakangas: Bringing WebKit back to Android.https://blogs.igalia.com/jani/bringing-webkit-back-to-android/2024-03-20T00:00:00+00:00
<p>It’s been quite a while since the last blog <a href="https://ferjm.github.io/wpe/webkit/android/2021/05/10/wpe-webkit-android.html">post</a> about WPE-Android, but that doesn’t mean WPE-Android hasn’t been in development. The focus has been on stabilizing the runtime and implementing the most crucial core features to make it easier to integrate new APIs into WPEView.</p>
<h2 id="main-building-blocks" tabindex="-1">Main building blocks <a class="header-anchor" href="https://blogs.igalia.com/jani/bringing-webkit-back-to-android/">#</a></h2>
<p>WPE-Android has three main building blocks:</p>
<ul>
<li><a href="https://github.com/Igalia/wpe-android-cerbero">Cerbero</a>
<ul>
<li>Cross-platform build aggregator that is used to build WPE WebKit and all of its dependencies to Android.</li>
</ul>
</li>
<li><a href="https://github.com/Igalia/WPEBackend-android">WPEBackend-Android</a>
<ul>
<li>Implements Android specific graphics buffer support and buffer sharing
between WebKit UIProcess and WebProcess.</li>
</ul>
</li>
<li><a href="https://github.com/Igalia/wpe-android">WPEView</a>
<ul>
<li>Allows displaying web content in activity layout using WPEWebKit.</li>
</ul>
</li>
</ul>
<p><source type="image/avif"><source type="image/webp"><img alt="WPE-Android high-level design" src="https://blogs.igalia.com/jani/img/alt3tCpXgU-441.png" width="441" height="271" /></source></source></p>
<h2 id="what-s-new" tabindex="-1">What’s new <a class="header-anchor" href="https://blogs.igalia.com/jani/bringing-webkit-back-to-android/">#</a></h2>
<p>The list of all work completed so far would be quite long, as there have been no official releases or public announcements, with the exception of the last blog <a href="https://ferjm.github.io/wpe/webkit/android/2021/05/10/wpe-webkit-android.html">post</a>. Since that update, most of the efforts have been focused on ‘under the hood’ improvements, including enhancing stability, adding support for some core WebKit features, and making general improvements to the development infrastructure.</p>
<p>Here is a list of new features worth mentioning:</p>
<ul>
<li>Based on WPE WebKit 2.42.1, the project was branched for the 2.42.x series. Future work will continue in the main branch for the next release.</li>
<li>Dropped support for 32-bit platforms (x86 and armv7). Only arm64-v8a and x86_64 are supported</li>
<li>Integration to Android main loop so that WPE WebKit GLib main loop is driven by Android main loop</li>
<li>Process-Swap On Navigation aka PSON</li>
<li>Added ASharedMemory support to WebKit SharedMemory</li>
<li>Hardware-accelerated multimedia playback</li>
<li>Fullscreen support</li>
<li>Cookies management</li>
<li>ANGLE based WebGL</li>
<li>Cross process fence insertion for composition synchronization with Surface Flinger</li>
<li>WebDriver support</li>
<li>GitHub Actions build bots</li>
<li>GitHub Actions WebDriver test bots</li>
</ul>
<h2 id="demos" tabindex="-1">Demos <a class="header-anchor" href="https://blogs.igalia.com/jani/bringing-webkit-back-to-android/">#</a></h2>
<h4 id="wpewebkit-powered-web-view-wpeview" tabindex="-1">WPEWebkit powered web view (WPEView) <a class="header-anchor" href="https://blogs.igalia.com/jani/bringing-webkit-back-to-android/">#</a></h4>
<p>Demo uses WPE-Android <a href="https://github.com/Igalia/wpe-android/tree/main/tools/minibrowser">MiniBrowser</a> sample application to show basic web page loading and touch-based scrolling, usage of a simple cookie manager to clear the page’s date usage, and finally loads the popular “<a href="https://webglsamples.org/aquarium/aquarium.html">Aquarium sample</a>” to show a smooth (60FPS) WebGL animation running thanks to HW acceleration support.</p>
<video width="400" controls="">
<source src="https://people.igalia.com/jani/videos/blog/minibrowser_20240320.mp4" type="video/mp4">
</source></video>
<h4 id="webdriver" tabindex="-1">WebDriver <a class="header-anchor" href="https://blogs.igalia.com/jani/bringing-webkit-back-to-android/">#</a></h4>
<p>Demo shows how to run <a href="https://www.w3.org/TR/webdriver1">WebDriver</a> test with with emulator.
Detailed instructions how to run WebDriver test can be found in <a href="https://github.com/Igalia/wpe-android/blob/main/README.md">README.md</a></p>
<p>Test requests a website through the Selenium remote webdriver. It then replaces the default behavior of <code>window.alert</code> on the requested page by injecting and executing a JavaScript snippet. After loading the page, it performs a <code>click()</code> action on the element that calls the alert. This results in the text ‘cheese’ being displayed right below the ‘click me’ link.</p>
<p>Test contains three building blocks:</p>
<ul>
<li><a href="https://github.com/Igalia/wpe-android/tree/main/tools/webdriver">WPE WebDriver</a> running on emulator</li>
<li>HTTP server serving test.html web page</li>
<li>Selenium python test script executing the test</li>
</ul>
<video width="1200" controls="">
<source src="https://people.igalia.com/jani/videos/blog/webdriver_20240320.webm" type="video/webm">
</source></video>
<h2 id="what-s-next" tabindex="-1">What’s next <a class="header-anchor" href="https://blogs.igalia.com/jani/bringing-webkit-back-to-android/">#</a></h2>
<p>We have ambitious plans for the coming months regarding the development of WPE Android, focusing mainly on implementing additional features, stabilization, and performance improvements.</p>
<p>As for implementing new features, now that the integration with the WPE WebKit runtime has reached a more robust state, it’s time to start adding more of the APIs that are still missing in WPEView compared to other webviews on Android and to enable other web-facing features supported by WebKit. This effort, along with adding support for features like HTTP/2 and the remote Web Inspector, will be a major focus.</p>
<p>As for stabilization and performance, having WebDriver support will be very helpful as it will enable us to identify and fix issues promptly and thus help make WPE Android more stable and feature-complete. We would also like to focus on conformance testing compared to other web views on Android, which should help us prioritize our efforts.</p>
<p>The broader goal for this year is to develop WPE-Android into an easy-to-use platform for third-party projects, offering a compelling alternative to other webviews on the Android platform. We extend many thanks to the <a href="https://nlnet.nl">NLNet Foundation</a>, whose <a href="https://nlnet.nl/project/WPE-Android/">support for WPE Android through a grant from the NGI Zero Core fund</a> will be instrumental in helping us achieve this goal!</p>
<h2 id="try-it-yourself" tabindex="-1">Try it yourself <a class="header-anchor" href="https://blogs.igalia.com/jani/bringing-webkit-back-to-android/">#</a></h2>
<p>WPE-Android is still considered a prototype, and it’s a bit difficult to build. However, if you want to try it out, you can follow the instructions in the <a href="https://github.com/Igalia/wpe-android/blob/main/README.md">README.md</a> file. Additionally, you can use the project’s <a href="https://github.com/Igalia/wpe-android/issues">issue</a> tracker to report problems or suggest new features.</p> Jani Hautakangashttps://blogs.igalia.com/jani/Carlos García Campos: A Clarification About WebKit Switching to Skiahttp://blogs.igalia.com/carlosgc/?p=10242024-02-20T18:11:43+00:00
<p>In the previous post I talked about the plans of the WebKit ports currently using Cairo to switch to Skia for 2D rendering. Apple ports don’t use Cairo, so they won’t be switching to Skia. I understand the post title was confusing, I’m sorry about that. The original post has been updated for clarity.</p> carlos garcia camposhttps://blogs.igalia.com/carlosgcCarlos García Campos: WebKitGTK and WPEWebKit Switching to Skia for 2D Graphics Renderinghttp://blogs.igalia.com/carlosgc/?p=10162024-02-19T13:27:39+00:00
<p>In recent years we have had an ongoing effort to improve graphics performance of the WebKit GTK and WPE ports. As a result of this we shipped features like threaded rendering, the DMA-BUF renderer, or proper vertical retrace synchronization (VSync). While these improvements have helped keep WebKit competitive, and even perform better than other engines in some scenarios, it has been clear for a while that we were reaching the limits of what can be achieved with a CPU based 2D renderer.</p>
<p>There was an attempt at making Cairo support GPU rendering, which did not work particularly well due to the library being designed around stateful operation based upon the PostScript model—resulting in a convenient and familiar API, great output quality, but hard to retarget and with some particularly slow corner cases. Meanwhile, other web engines have moved more work to the GPU, including 2D rendering, where many operations are considerably faster.</p>
<p>We checked all the available 2D rendering libraries we could find, but none of them met all our requirements, so we decided to try writing our own library. At the beginning it worked really well, with impressive results in performance even compared to other GPU based alternatives. However, it proved challenging to find the right balance between performance and rendering quality, so we decided to try other alternatives before continuing with its development. Our next option had always been Skia. The main reason why we didn’t choose Skia from the beginning was that it didn’t provide a public library with API stability that distros can package and we can use like most of our dependencies. It still wasn’t what we wanted, but now we have more experience in WebKit maintaining third party dependencies inside the source tree like ANGLE and libwebrtc, so it was no longer a blocker either.</p>
<p>In December 2023 we made the decision of giving Skia a try internally and see if it would be worth the effort of maintaining the project as a third party module inside WebKit. In just one month we had implemented enough features to be able to run all MotionMark tests. The results in the desktop were quite impressive, getting double the score of MotionMark global result. We still had to do more tests in embedded devices which are the actual target of WPE, but it was clear that, at least in the desktop, with this very initial implementation that was not even optimized (we kept our current architecture that is optimized for CPU rendering) we got much better results. We decided that Skia was the option, so we continued working on it and doing more tests in embedded devices. In the boards that we tried we also got better results than CPU rendering, but the difference was not so big, which means that with less powerful GPUs and with our current architecture designed for CPU rendering we were not that far from CPU rendering. That’s the reason why we managed to keep WPE competitive in embeeded devices, but Skia will not only bring performance improvements, it will also simplify the code and will allow us to implement new features . So, we had enough data already to make the final decision of going with Skia.</p>
<p>In February 2024 we reached a point in which our Skia internal branch was in an “upstreamable” state, so there was no reason to continue working privately. We met with several teams from Google, Sony, Apple and Red Hat to discuss with them about our intention to switch from Cairo to Skia, upstreaming what we had as soon as possible. We got really positive feedback from all of them, so we sent an email to the WebKit developers mailing list to make it public. And again we only got positive feedback, so we started to prepare the patches to import Skia into WebKit, add the CMake integration and the initial Skia implementation for the WPE port that already landed in main.</p>
<p>We will continue working on the Skia implementation in upstream WebKit, and we also have plans to change our architecture to better support the GPU rendering case in a more efficient way. We don’t have a deadline, it will be ready when we have implemented everything currently supported by Cairo, we don’t plan to switch with regressions. We are focused on the WPE port for now, but at some point we will start working on GTK too and other ports using cairo will eventually start getting Skia support as well.</p> carlos garcia camposhttps://blogs.igalia.com/carlosgcWPE WebKit Blog: Use Case: Server-side headless renderinghttps://wpewebkit.org/blog/2024-use-case-server-side-rendering.html2024-02-01T00:00:00+00:00
<div class="success-top">
<img alt="WPE and server-side headless rendering" align="center" src="https://wpewebkit.org/assets/img/logo-server-side-rendering.png" />
</div>
<p>In many distributed applications, it can be useful to run a light web browser on the server side to render some HTML content or process images, video and/or audio using JavaScript.</p>
<p>Some concrete use-cases can be:</p>
<ul>
<li>Video post-production using HTML overlays.</li>
<li>Easy 3D rendering with WebGL that can be broadcasted as a video stream.</li>
<li>Reusing the same JavaScript code between a frontend web application and the backend processing.</li>
</ul>
<p>WPE WebKit is the perfect solution for all those use cases as it offers a lightweight solution which can run on low-end hardware or even within a container. It provides a lot of flexibility at the moment of choosing the backend infrastructure as WPE WebKit can, for instance, run from within a container with a very minimal Linux configuration (no need for any windowing system) and with full hardware acceleration and zero-copy of the video buffers between the GPU and the CPU.</p>
<p>Additionally, the fact that WPE WebKit is optimized for lower-powered devices, makes it also the perfect option for server-side rendering when scaling commercial deployments while keeping cost under control, which is yet another important factor to take into account when considering cloud rendering.</p> WPE WebKit Bloghttps://wpewebkit.org/blog/WPE WebKit Blog: A New WPE Backend Using EGLStreamhttps://wpewebkit.org/blog/07-creating-wpe-backends.html2024-01-29T06:00:00+00:00
<h2 id="what-is-a-wpe-backend%3F" tabindex="-1">What is a WPE Backend?</h2>
<div class="sidebar">
<p>This article is a mashup of <a href="https://blogs.igalia.com/llepage/the-process-of-creating-a-new-wpe-backend/">The process of creating a new WPE
backend</a>
and <a href="https://blogs.igalia.com/llepage/use-eglstreams-in-a-wpe-webkit-backend/">Use EGLStreams in a WPE WebKit
backend</a>,
originally published at Loïc’s blog.</p>
</div>
<p>Depending on the target hardware WPE may need to use different techniques and
technologies to ensure correct graphical rendering. To be independent of any
user-interface toolkit and windowing system, WPE WebKit delegates the rendering
to a third-party API defined in the
<a href="https://github.com/WebPlatformForEmbedded/libwpe">libwpe</a> library. A concrete
implementation of this API is a “WPE backend”.</p>
<p>WPE WebKit is a multiprocess application, the end-user starts and controls the
web widgets in the application process (which we often call “the <abbr title="User Interface">UI</abbr> process” while the web engine itself uses
different subprocesses: <code>WPENetworkProcess</code> is in charge of managing network
connections and <code>WPEWebProcess</code> (or “web process”) in charge of the HTML and
JavaScript parsing, execution and rendering. The WPE backend is at a crossroads
between the UI process and one or more web process instances.</p>
<figure>
<a href="https://wpewebkit.org/assets/svg/part1-basics.md-1.svg" target="_blank"><img src="https://wpewebkit.org/assets/svg/part1-basics.md-1.svg" alt="Diagram showing a box for the WPE backend in between the UI process and WPEWebProcess" />
</a>
</figure>
<p>The WPE backend is a shared library that is loaded at runtime by the web
process and by the UI process. It is used to render the visual aspect of a web
page and transfer the resulting video buffer from the web process to the
application process.</p>
<h2 id="backend-interfaces" tabindex="-1">Backend Interfaces</h2>
<p>The WPE backend shared library must export at least one symbol called
<code>_wpe_loader_interface</code> of type <code>struct wpe_loader_interface</code> as defined <a href="https://github.com/WebPlatformForEmbedded/libwpe/blob/d7c669ca6f5ec0d544c264016d270669b336c931/include/wpe/loader.h#L57">in
the <em>libwpe</em>
API</a>.
Presently its only member is <code>load_object</code>, a callback function that receives a
string with an interface name and returns concrete implementations of the
following interfaces:</p>
<ul>
<li><a href="https://github.com/WebPlatformForEmbedded/libwpe/blob/d7c669ca6f5ec0d544c264016d270669b336c931/include/wpe/renderer-host.h#L48">struct wpe_renderer_host_interface</a></li>
<li><a href="https://github.com/WebPlatformForEmbedded/libwpe/blob/d7c669ca6f5ec0d544c264016d270669b336c931/include/wpe/renderer-backend-egl.h#L75">struct wpe_renderer_backend_egl_interface</a></li>
<li><a href="https://github.com/WebPlatformForEmbedded/libwpe/blob/d7c669ca6f5ec0d544c264016d270669b336c931/include/wpe/renderer-backend-egl.h#L93">struct wpe_renderer_backend_egl_target_interface</a></li>
<li><a href="https://github.com/WebPlatformForEmbedded/libwpe/blob/d7c669ca6f5ec0d544c264016d270669b336c931/include/wpe/renderer-backend-egl.h#L115">struct wpe_renderer_backend_egl_offscreen_target_interface</a></li>
<li><a href="https://github.com/WebPlatformForEmbedded/libwpe/blob/d7c669ca6f5ec0d544c264016d270669b336c931/include/wpe/view-backend.h#L63">struct wpe_view_backend_interface</a></li>
</ul>
<p>The names passed to the <code>.load_object()</code> function are the same as those of the
interface types, prefixed with an underscore. For example, a
<code>.load_object("_wpe_renderer_host_interface")</code> call must return a pointer to a
<code>struct wpe_renderer_host_interface</code> object.</p>
Example C code for a <code>load_object</code> callback.
<!-- load_object example <<<1 -->
<pre class="language-c"><code class="language-c"><span class="token keyword">static</span> <span class="token keyword">struct</span> <span class="token class-name">wpe_renderer_host_interface</span> <span class="token operator">=</span> <span class="token punctuation">{</span> <span class="token comment">/* ... */</span> <span class="token punctuation">}</span><span class="token punctuation">;</span>
<span class="token keyword">static</span> <span class="token keyword">struct</span> <span class="token class-name">wpe_renderer_backend_egl_interface</span> <span class="token operator">=</span> <span class="token punctuation">{</span> <span class="token comment">/* ... */</span> <span class="token punctuation">}</span><span class="token punctuation">;</span>
<span class="token keyword">static</span> <span class="token keyword">void</span><span class="token operator">*</span>
<span class="token function">my_backend_load_object</span><span class="token punctuation">(</span><span class="token keyword">const</span> <span class="token keyword">char</span> <span class="token operator">*</span>name<span class="token punctuation">)</span>
<span class="token punctuation">{</span>
<span class="token keyword">if</span> <span class="token punctuation">(</span><span class="token operator">!</span><span class="token function">strcmp</span><span class="token punctuation">(</span>name<span class="token punctuation">,</span> <span class="token string">"_wpe_renderer_host_interface"</span><span class="token punctuation">)</span><span class="token punctuation">)</span>
<span class="token keyword">return</span> <span class="token operator">&</span>my_renderer_host<span class="token punctuation">;</span>
<span class="token keyword">if</span> <span class="token punctuation">(</span><span class="token operator">!</span><span class="token function">strcmp</span><span class="token punctuation">(</span>name<span class="token punctuation">,</span> <span class="token string">"_wpe_renderer_backend_egl_interface"</span><span class="token punctuation">)</span><span class="token punctuation">)</span>
<span class="token keyword">return</span> <span class="token operator">&</span>my_renderer_backend_egl<span class="token punctuation">;</span>
<span class="token comment">/* ... */</span>
<span class="token keyword">return</span> <span class="token constant">NULL</span><span class="token punctuation">;</span>
<span class="token punctuation">}</span>
<span class="token keyword">struct</span> <span class="token class-name">wpe_loader_interface</span> _wpe_loader_interface <span class="token operator">=</span> <span class="token punctuation">{</span>
<span class="token punctuation">.</span>load_object <span class="token operator">=</span> my_backend_load_object<span class="token punctuation">,</span>
<span class="token punctuation">}</span><span class="token punctuation">;</span></code></pre>
<!-- 1>>> -->
<p>Each of these interfaces follow the same base structure: the <code>struct</code> members
are callback functions, all interfaces have <code>create</code> and <code>destroy</code> members which
act as instance constructor and destructor, plus any additional “methods”.
The pointer returned by the <code>create</code> callback will be passed as the <code>object</code>
“instance” of the other methods:</p>
<pre class="language-c"><code class="language-c"><span class="token keyword">struct</span> <span class="token class-name">wpe_renderer_host_interface</span> <span class="token punctuation">{</span>
<span class="token keyword">void</span><span class="token operator">*</span> <span class="token punctuation">(</span><span class="token operator">*</span>create<span class="token punctuation">)</span><span class="token punctuation">(</span><span class="token keyword">void</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
<span class="token keyword">void</span> <span class="token punctuation">(</span><span class="token operator">*</span>destroy<span class="token punctuation">)</span><span class="token punctuation">(</span><span class="token keyword">void</span> <span class="token operator">*</span>object<span class="token punctuation">)</span><span class="token punctuation">;</span>
<span class="token comment">/* ... */</span>
<span class="token punctuation">}</span><span class="token punctuation">;</span></code></pre>
<p>In the <strong>UI process</strong> side WPE WebKit will create:</p>
<ul>
<li>One “renderer host” instance, using <code>wpe_renderer_host_interface.create()</code>.</li>
<li>Multiple “renderer host client” instances, using <code>wpe_renderer_host_interface.create_client()</code>.
These are mainly used for IPC communication, one instance gets created for
each web process launched by WebKit.</li>
<li>Multiple “view backend” instances, using <code>wpe_view_backend_interface.create()</code>.
One instance is created for each rendering target in the web process.</li>
</ul>
<p>In each <strong>web process</strong>—there can be more than one—WPE WebKit
will create:</p>
<ul>
<li>One “renderer backend EGL” instance, using <code>wpe_renderer_backend_egl_interface.create()</code>.</li>
<li>Multiple “renderer backend EGL target” instances, using
<code>wpe_renderer_backend_egl_target_interface.create()</code>. An instance is created
for each new rendering target needed by the application.</li>
</ul>
How about <code>wpe_renderer_backend_egl_offscreen_target_interface</code>?
<div>
<p>The <code>rendererBackendEGLTarget</code> instances may be created by the <code>wpe_renderer_backend_egl_target_interface</code>, or
the <code>wpe_renderer_backend_egl_offscreen_target_interface</code> depending on the interfaces implemented in the backend.</p>
<p>Here we are only focusing on the <code>wpe_renderer_backend_egl_target_interface</code> that is relying on a classical EGL
display (defined in the <code>rendererBackendEGL</code> instance). The <code>wpe_renderer_backend_egl_offscreen_target_interface</code> may
be used in very specific use-cases that are out of the scope of this post. You can check its usage in the WPE WebKit
<a href="https://github.com/WebKit/WebKit/blob/f32cd0f7cb7961420ce08ae78b8f01f287bec199/Source/WebCore/platform/graphics/egl/GLContextLibWPE.cpp#L61">source code</a>
for more information.</p>
</div>
<p>These instances typically communicate with each others using Unix sockets for
<abbr title="Inter-Process Communication">IPC</abbr>. The IPC layer must be
implemented in the WPE backend itself because the <em>libwpe</em> interfaces only pass
around the file descriptors to be used as communication endpoints.</p>
<p>From a topological point of view, all those instances are organized as follows:</p>
<figure>
<a href="https://wpewebkit.org/assets/svg/part1-basics.md-2.svg" target="_blank"><img src="https://wpewebkit.org/assets/svg/part1-basics.md-2.svg" />
</a>
</figure>
<p>From an usage point of view:</p>
<ul>
<li>The <code>rendererHost</code> and <code>rendererHostClient</code> instances are only used to manage
IPC endpoints on the UI process side that are connected to each running
web process. They are not used by the graphical rendering system.</li>
<li>The <code>rendererBackendEGL</code> instance (one per web process) is only used to
connect to the native display for a specific platform. For example, on a
desktop Linux, the platform may be X11 where the native display would be the
result of calling <code>XOpenDisplay()</code>; or the platform may be Wayland and in
this case the native display would be the result of calling
<code>wl_display_connect()</code>; and so on.</li>
<li>The <code>rendererBackendEGLTarget</code> (on the web process side) and <code>viewBackend</code>
(on the UI process side) instances are the ones truly managing the web page
graphical rendering.</li>
</ul>
<h2 id="graphics-rendering" tabindex="-1">Graphics Rendering</h2>
<p>As seen above, the interfaces in charge of the rendering are
<code>wpe_renderer_backend_egl_target_interface</code> and <code>wpe_view_backend_interface</code>.
During their creation, WPE WebKit exchanges the file descriptors used to
establish a direct IPC connection between a <code>rendererBackendEGL</code> (in the
web process), and a <code>viewBackend</code> (in the UI process).</p>
<p>During the EGL initialization phase, when a new web process is launched, WebKit
will use the native display and platform provided by the
<code>wpe_renderer_backend_egl_interface.get_native_display()</code> and <code>.get_platform()</code>
functions to create a suitable OpenGL ES context.</p>
<p>When WebKit’s
<a href="https://github.com/WebKit/WebKit/blob/c22f641da18b8c4eee23b8021b37aeec69268675/Source/WebKit/Shared/CoordinatedGraphics/threadedcompositor/ThreadedCompositor.cpp#L220">ThreadedCompositor</a>
is ready to render a new frame (in the web process), it calls the
<code>wpe_renderer_backend_egl_target_interface.frame_will_render()</code> function to let
the WPE backend know that rendering is about to start. At this moment, the
previously created OpenGL ES context is made current to be used as the target
for GL drawing commands.</p>
<p>Once the threaded compositor has finished drawing, it will swap the front and
back EGL buffers and call the
<code>wpe_renderer_backend_egl_target_interface.frame_rendered()</code> function to signal
that the frame is ready. The compositor will then wait until the WPE backend
calls <code>wpe_renderer_backend_egl_target_dispatch_frame_complete()</code> to indicate
that the compositor may produce a new frame.</p>
<p>What happens inside the <code>.frame_will_render()</code> and <code>.frame_rendered()</code>
implementations is up to the WPE backend. As en example, it could
set up a <a href="https://www.khronos.org/opengl/wiki/Framebuffer_Object">Frame Buffer Object</a>
to have the web content draw offscreen, in a texture that can be passed
back to the UI process for further processing, or use extensions like
<a href="https://registry.khronos.org/EGL/extensions/KHR/EGL_KHR_stream.txt">EGLStream</a>,
or <a href="https://registry.khronos.org/EGL/extensions/MESA/EGL_MESA_image_dma_buf_export.txt">DMA-BUF exports</a>
to transfer the frame to the UI process without copying the pixel data.</p>
<figure>
<a href="https://wpewebkit.org/assets/svg/part1-basics.md-3.svg" target="_blank"><img src="https://wpewebkit.org/assets/svg/part1-basics.md-3.svg" />
</a>
</figure>
<p>Typically the backend sends each new frame to the corresponding view backend in
in its <code>.frame_rendered()</code> function. The application can use the frame until it
sends back an <abbr>IPC</abbr> message to the renderer target (in the web
process) to indicate that the frame is not in use anymore and may be be freed
or recycled. Although it is not a requirement to do it at this exact point,
usually when a renderer backend receives this message it calls the
<code>wpe_renderer_backend_egl_target_dispatch_frame_complete()</code> function to trigger
the rendering of a new frame. As a side effect, this mechanism also allows
controlling the pace at which new frames are produced.</p>
<h2 id="using-eglstream" tabindex="-1">Using EGLStream</h2>
<p>EGLStream is an EGL extension that defines a mechanism to transfer hardware
video buffers from one process to another efficiently, without getting them
out of GPU memory. Although the extension is supported only in Nvidia
hardware, it makes for a good example as it transparently handles some
complexities involved, like buffers with multiple planes.</p>
<p>This backend uses the EGLStream extension to transfer graphics buffers from the
web process, which acts as a producer, to the UI process acting as a consumer.
The producer extension
<a href="https://registry.khronos.org/EGL/extensions/KHR/EGL_KHR_stream_producer_eglsurface.txt">EGL_KHR_stream_producer_eglsurface</a>
allows creating a surface that may be used as target for rendering, then using
<a href="https://registry.khronos.org/EGL/sdk/docs/man/html/eglSwapBuffers.xhtml">eglSwapBuffers()</a>
finishes drawing and sends the result to the consumer. Meanwhile, in the
consumer side, the
<a href="https://registry.khronos.org/EGL/extensions/NV/EGL_NV_stream_consumer_eglimage.txt">EGL_NV_stream_consumer_eglimage</a>
extension is used to turn each buffer into an <code>EGLImage</code>.</p>
<p>The reference source code for this WPE backend is available in the
<a href="https://github.com/Igalia/WPEBackend-offscreen-nvidia">WPEBackend-offscreen-nvidia</a>
repository, which has been tested with WPE WebKit 2.38.x or 2.40.x, and
<em>libwpe</em> version 1.14.x.</p>
Behold, the Future Belongs to DMA-BUF!
<div>
<p>With the growing adoption of
<a href="https://docs.kernel.org/driver-api/dma-buf.html">DMA-BUF</a> for sharing memory
buffers on modern Linux platforms, the WPE WebKit architecture will be
evolving and, in the future, the need for a WPE Backend should disappear in
most cases.</p>
<p>Ongoing work on WPE WebKit removes the need to provide a WPE backend
implementation for most hardware platforms, with a generic implementation
using DMA-BUF provided as an integral, built-in feature of WebKit. It will
still be possible to provide external implementations for platforms that
might need to use custom buffer sharing mechanisms.</p>
<p>From the application developer point of view, in most cases writing
programs that use the WPE WebKit API will be simpler, with the complexity
of the communication among multiple processes handled by WebKit.</p>
</div>
<h3 id="stream-setup" tabindex="-1">Stream Setup</h3>
<p>The steps needed to set up EGLStream endpoints need to be done in a particular
order:</p>
<ol>
<li>Create the consumer.</li>
<li>Get the stream file descriptor for the consumer.</li>
<li>Send the stream file descriptor to the producer.</li>
<li>Create the producer.</li>
</ol>
<p><strong>First</strong>, the consumer needs to be created:</p>
<pre class="language-cpp"><code class="language-cpp">EGLStream <span class="token function">createConsumerStream</span><span class="token punctuation">(</span>EGLDisplay eglDisplay<span class="token punctuation">)</span> <span class="token punctuation">{</span>
<span class="token keyword">static</span> <span class="token keyword">const</span> EGLint s_streamAttribs<span class="token punctuation">[</span><span class="token punctuation">]</span> <span class="token operator">=</span> <span class="token punctuation">{</span>
EGL_STREAM_FIFO_LENGTH_KHR<span class="token punctuation">,</span> <span class="token number">1</span><span class="token punctuation">,</span>
EGL_CONSUMER_ACQUIRE_TIMEOUT_USEC_KHR<span class="token punctuation">,</span> <span class="token number">1000</span> <span class="token operator">*</span> <span class="token number">1000</span><span class="token punctuation">,</span>
EGL_NONE
<span class="token punctuation">}</span><span class="token punctuation">;</span>
<span class="token keyword">return</span> <span class="token function">eglCreateStreamKHR</span><span class="token punctuation">(</span>eglDisplay<span class="token punctuation">,</span> s_streamAttribs<span class="token punctuation">)</span><span class="token punctuation">;</span>
<span class="token punctuation">}</span></code></pre>
<p>The <code>EGL_STREAM_FIFO_LENGTH_KHR</code> parameter defines the length of the EGLStream
queue. If set to zero, the stream will work in “mailbox” mode and each time the
producer has a new frame it will empty the stream content and replace the frame
by the new one. If non-zero, the stream works work in “<abbr title="First-In,
First-Out">FIFO</abbr>” mode, which means that the stream queue can contain up
to <code>EGL_STREAM_FIFO_LENGTH_KHR</code> frames.</p>
<p>Here we configure a queue for one frame because in this case the specification
of <code>EGL_KHR_stream_producer_eglsurface</code> guarantees that calling
<code>eglSwapBuffers()</code> on the producer the call will block until the consumer
retires the previous frame from queue. This is used as implicit synchronization
between the UI process side and the web process side without needing to rely on
custom IPC, which would add a small delay between frames.</p>
<p>The <code>EGL_CONSUMER_ACQUIRE_TIMEOUT_USEC_KHR</code> parameter defines the maximum
timeout in microseconds to wait on the consumer side to acquire a frame when
calling <code>eglStreamConsumerAcquireKHR()</code>. It is only used with the
<code>EGL_KHR_stream_consumer_gltexture</code> extension because the
<code>EGL_NV_stream_consumer_eglimage</code> extension allows setting a timeout on each
call to <code>eglQueryStreamConsumerEventNV()</code> function.</p>
<p><strong>Second</strong>, to initialize the consumer using the <code>EGL_NV_stream_consumer_eglimage</code>
extension it is enough to call the <code>eglStreamImageConsumerConnectNV()</code> function.</p>
<p><strong>Once the consumer has been initialized</strong>, you need to send the EGLStream
file descriptor to the producer process. The usual way of achieving this would
be using IPC between the two processes, sending the file descriptor in a
<code>SCM_RIGHTS</code> message through an Unix socket—although with recent kernels
using <a href="https://lwn.net/Articles/808997/">pidfd_getfd()</a> may be an option if
both processes are related.</p>
<p>When the file descriptor is <strong>finally</strong> received, the producer endpoint can be
created using the <code>EGL_KHR_stream_producer_eglsurface</code> extension:</p>
<pre class="language-cpp"><code class="language-cpp"><span class="token keyword">const</span> EGLint surfaceAttribs<span class="token punctuation">[</span><span class="token punctuation">]</span> <span class="token operator">=</span> <span class="token punctuation">{</span>
EGL_WIDTH<span class="token punctuation">,</span> width<span class="token punctuation">,</span>
EGL_HEIGHT<span class="token punctuation">,</span> height<span class="token punctuation">,</span>
EGL_NONE
<span class="token punctuation">}</span><span class="token punctuation">;</span>
EGLStream eglStream <span class="token operator">=</span> <span class="token function">eglCreateStreamFromFileDescriptorKHR</span><span class="token punctuation">(</span>eglDisplay<span class="token punctuation">,</span> consumerFD<span class="token punctuation">)</span><span class="token punctuation">;</span>
EGLSurface eglSurface <span class="token operator">=</span> <span class="token function">eglCreateStreamProducerSurfaceKHR</span><span class="token punctuation">(</span>eglDisplay<span class="token punctuation">,</span> config<span class="token punctuation">,</span> eglStream<span class="token punctuation">,</span> surfaceAttribs<span class="token punctuation">)</span><span class="token punctuation">;</span></code></pre>
<p>As with <abbr title="Pixel Buffer">pbuffer</abbr> surfaces, the dimensions
need to be specified as surface attributes. When picking a frame buffer
configuration with <code>eglChooseConfig()</code> the <code>EGL_SURFACE_TYPE</code> attribute must
be set to <code>EGL_STREAM_BIT_KHR</code>. From this point onwards, rendering proceeds as
usual: the EGL surface and context are made active, and once the painting is
done a call to <code>eglSwapBuffers()</code> will “present” the frame, which in this case
means sending the buffer with the pixel data down the EGLStream to the
consumer.</p>
<figure>
<a href="https://wpewebkit.org/assets/svg/part2-eglstream.md-1.svg" target="_blank"><img src="https://wpewebkit.org/assets/svg/part2-eglstream.md-1.svg" />
</a>
</figure>
<h3 id="consuming-frames" tabindex="-1">Consuming Frames</h3>
<p>While on the producer side rendering treats the EGLStream surface like any
other, on the consumer some more work is needed to manager the lifetime of
the data received: frames have to be manually acquired and released once
they are not needed anymore.</p>
<p>The producer calls <code>eglQueryStreamConsumerEventNV()</code> repeatedly to retire the
next event from the stream:</p>
<ul>
<li><code>EGL_STREAM_IMAGE_ADD_NV</code> indicates that there is a buffer in the stream
that has not yet been bound to an <code>EGLImage</code>, and the application needs to
create a new one to which the actual data will be bound later.</li>
<li><code>EGL_STREAM_IMAGE_AVAILABLE_NV</code> indicates that a new frame is available
and that it can be bound to the previously created <code>EGLImage</code>.</li>
<li><code>EGL_STREAM_IMAGE_REMOVE_NV</code> indicates that a buffer has been retired from
the stream, and that its associated <code>EGLImage</code> may be released once the
application has finished using it.</li>
</ul>
<p>This translates roughly to the following code:</p>
<pre class="language-cpp"><code class="language-cpp"><span class="token keyword">static</span> <span class="token keyword">constexpr</span> EGLTime MAX_TIMEOUT_USEC <span class="token operator">=</span> <span class="token number">1000</span> <span class="token operator">*</span> <span class="token number">1000</span><span class="token punctuation">;</span>
EGLImage eglImage <span class="token operator">=</span> EGL_NO_IMAGE<span class="token punctuation">;</span>
<span class="token keyword">while</span> <span class="token punctuation">(</span><span class="token boolean">true</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>
EGLenum event <span class="token operator">=</span> <span class="token number">0</span><span class="token punctuation">;</span>
EGLAttrib data <span class="token operator">=</span> <span class="token number">0</span><span class="token punctuation">;</span>
<span class="token comment">// WARNING: The specification states that the timeout is in nanoseconds</span>
<span class="token comment">// (see: https://registry.khronos.org/EGL/extensions/NV/EGL_NV_stream_consumer_eglimage.txt)</span>
<span class="token comment">// but in reality it is in microseconds, at least with the version 535.113.01 of the NVidia drivers.</span>
<span class="token keyword">if</span> <span class="token punctuation">(</span><span class="token operator">!</span><span class="token function">eglQueryStreamConsumerEventNV</span><span class="token punctuation">(</span>display<span class="token punctuation">,</span> eglStream<span class="token punctuation">,</span> MAX_TIMEOUT_USEC<span class="token punctuation">,</span> <span class="token operator">&</span>event<span class="token punctuation">,</span> <span class="token operator">&</span>data<span class="token punctuation">)</span><span class="token punctuation">)</span>
<span class="token keyword">break</span><span class="token punctuation">;</span>
<span class="token keyword">switch</span> <span class="token punctuation">(</span>event<span class="token punctuation">)</span> <span class="token punctuation">{</span>
<span class="token keyword">case</span> EGL_STREAM_IMAGE_ADD_NV<span class="token operator">:</span> <span class="token comment">// Bind an incoming buffer to an EGLImage.</span>
<span class="token keyword">if</span> <span class="token punctuation">(</span>eglImage<span class="token punctuation">)</span> <span class="token function">eglDestroyImage</span><span class="token punctuation">(</span>display<span class="token punctuation">,</span> eglImage<span class="token punctuation">)</span><span class="token punctuation">;</span>
eglImage <span class="token operator">=</span> <span class="token function">eglCreateImage</span><span class="token punctuation">(</span>display<span class="token punctuation">,</span> EGL_NO_CONTEXT<span class="token punctuation">,</span> EGL_STREAM_CONSUMER_IMAGE_NV<span class="token punctuation">,</span>
<span class="token generic-function"><span class="token function">static_cast</span><span class="token generic class-name"><span class="token operator"><</span>EGLClientBuffer<span class="token operator">></span></span></span><span class="token punctuation">(</span>eglStream<span class="token punctuation">)</span><span class="token punctuation">,</span> <span class="token keyword">nullptr</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
<span class="token keyword">continue</span><span class="token punctuation">;</span> <span class="token comment">// Handle the next event.</span>
<span class="token keyword">case</span> EGL_STREAM_IMAGE_REMOVE_NV<span class="token operator">:</span> <span class="token comment">// Buffer removed, EGLImage may be disposed.</span>
<span class="token keyword">if</span> <span class="token punctuation">(</span>data<span class="token punctuation">)</span> <span class="token punctuation">{</span>
EGLImage image <span class="token operator">=</span> <span class="token generic-function"><span class="token function">reinterpret_cast</span><span class="token generic class-name"><span class="token operator"><</span>EGLImage<span class="token operator">></span></span></span><span class="token punctuation">(</span>data<span class="token punctuation">)</span><span class="token punctuation">;</span>
<span class="token function">eglDestroyImage</span><span class="token punctuation">(</span>display<span class="token punctuation">,</span> image<span class="token punctuation">)</span><span class="token punctuation">;</span>
<span class="token keyword">if</span> <span class="token punctuation">(</span>image <span class="token operator">==</span> eglImage<span class="token punctuation">)</span>
eglImage <span class="token operator">=</span> EGL_NO_IMAGE<span class="token punctuation">;</span>
<span class="token punctuation">}</span>
<span class="token keyword">continue</span><span class="token punctuation">;</span> <span class="token comment">// Handle the next event.</span>
<span class="token keyword">case</span> EGL_STREAM_IMAGE_AVAILABLE_NV<span class="token operator">:</span> <span class="token comment">// New frame available.</span>
<span class="token keyword">if</span> <span class="token punctuation">(</span><span class="token function">eglStreamAcquireImageNV</span><span class="token punctuation">(</span>display<span class="token punctuation">,</span> eglStream<span class="token punctuation">,</span> <span class="token operator">&</span>eglImage<span class="token punctuation">,</span> EGL_NO_SYNC<span class="token punctuation">)</span><span class="token punctuation">)</span>
<span class="token keyword">break</span><span class="token punctuation">;</span>
<span class="token keyword">default</span><span class="token operator">:</span>
<span class="token keyword">continue</span><span class="token punctuation">;</span> <span class="token comment">// Handle the next event.</span>
<span class="token punctuation">}</span>
<span class="token comment">/*** Use the EGLImage here ***/</span>
<span class="token function">eglStreamReleaseImageNV</span><span class="token punctuation">(</span>display<span class="token punctuation">,</span> eglStream<span class="token punctuation">,</span> eglImage<span class="token punctuation">,</span> EGL_NO_SYNC<span class="token punctuation">)</span><span class="token punctuation">;</span>
<span class="token punctuation">}</span></code></pre>
<p>The application is free to use each <code>EGLImage</code> as it sees fit. An obvious
example would be to use it as the contents for a texture, which then gets
painted in the “content” area of a web browser; or as the contents of the
screen for an in-game computer that the player can interact with, enabling
display of real, live web content as part of the gaming experience—now
<em>that</em> would be a deeply embedded browser!</p>
<h3 id="one-last-thing" tabindex="-1">One Last Thing</h3>
<p>There is a small showstopper to have EGLStream support working:
<a href="https://github.com/WebKit/WebKit/blob/cb07c70c253a35b0e09e46e6100e1cdcebab26e2/Source/WebCore/platform/graphics/egl/GLContextEGL.cpp#L135">currently</a>
when WPE WebKit uses surfaceless EGL contexts it sets the surface type to
<code>EGL_WINDOW_BIT</code> attribute, while <code>EGL_STREAM_BIT_KHR</code> would be needed
instead. <a href="https://github.com/Igalia/WPEBackend-offscreen-nvidia/blob/main/wpewebkit-patches/005-fix-surfaceless-egl-context-creation.patch">A small
patch</a>
is enough to apply this tweak:</p>
<pre class="language-diff"><code class="language-diff">diff --git a/Source/WebCore/platform/graphics/egl/GLContextEGL.cpp b/Source/WebCore/platform/graphics/egl/GLContextEGL.cpp
index d5efa070..5f200edc 100644
<span class="token coord">--- a/Source/WebCore/platform/graphics/egl/GLContextEGL.cpp</span>
<span class="token coord">+++ b/Source/WebCore/platform/graphics/egl/GLContextEGL.cpp</span>
@@ -122,9 +122,11 @@ bool GLContextEGL::getEGLConfig(EGLDisplay display, EGLConfig* config, EGLSurfac
<span class="token unchanged"><span class="token prefix unchanged"> </span><span class="token line"> attributeList[13] = EGL_PIXMAP_BIT;
</span><span class="token prefix unchanged"> </span><span class="token line"> break;
</span><span class="token prefix unchanged"> </span><span class="token line"> case GLContextEGL::WindowSurface:
</span></span><span class="token deleted-sign deleted"><span class="token prefix deleted">-</span><span class="token line"> case GLContextEGL::Surfaceless:
</span></span><span class="token unchanged"><span class="token prefix unchanged"> </span><span class="token line"> attributeList[13] = EGL_WINDOW_BIT;
</span><span class="token prefix unchanged"> </span><span class="token line"> break;
</span></span><span class="token inserted-sign inserted"><span class="token prefix inserted">+</span><span class="token line"> case GLContextEGL::Surfaceless:
</span><span class="token prefix inserted">+</span><span class="token line"> attributeList[13] = EGL_STREAM_BIT_KHR;
</span><span class="token prefix inserted">+</span><span class="token line"> break;
</span></span><span class="token unchanged"><span class="token prefix unchanged"> </span><span class="token line"> }
</span></span>
<span class="token unchanged"><span class="token prefix unchanged"> </span><span class="token line"> EGLint count;
</span></span></code></pre>
<!-- vim:set foldmethod=marker foldmarker=<<<,>>>: --> WPE WebKit Bloghttps://wpewebkit.org/blog/