After a year or two of hiatus I attended the GStreamer conference which
happened in beautiful Edinburgh. It was great to meet the friends from the
community again and learn about what’s going on in the multimedia world. The
quality of the talks was great, the videos are published online as usual in
Ubicast. I delivered a talk about the Multimedia support in WPEWebKit, you can
watch it there and the slides are also available.
One of the many interesting presentations was about GStreamer for cloud-based
live video. Usually anything with the word cloud would tend to draw my
attention away but for some reason I attended this presentation, and didn’t
regret it! The last demo presented by the BBC folks was about overlaying Web
content on native video streams. It’s an interesting use-case for live TV
broadcasting for instance. A web page provides dynamic notifications popping up
and down, the web page is rendered with a transparent background and blended
over the live video stream. The BBC folks implemented a GStreamer source
element relying on CEF for their Brave project.
So here you wonder, why am I talking about Chromium Embedded Framework (CEF)?
Isn’t this post about WPEWebKit? After seeing the demo from the Brave
developers I immediately thought WPE could be a great fit for this HTML overlay
use-case too! So a few weeks after the conference I finally had the time to
start working on the WPE GStreamer plugin. My colleague Žan Doberšek,
WPE’s founder hacker, provided a nice solution for the initial rendering issues
of the prototype, many thanks to him!
Here’s a first example, a basic web-browser with gst-play:
A GTK window opens up and the GNOME homepage should load. You can click on links
too! To overlay a web page on top of a video you can use a pipeline like this one:
$ gst-launch-1.0 glvideomixer name=m sink_1::zorder=0 sink_0::height=818 sink_0::width=1920 ! gtkglsink \
wpesrc location="file:///home/phil/Downloads/plunk/index.html" draw-background=0 ! m. \
uridecodebin uri="http://192.168.1.44/Sintel.2010.1080p.mkv" name=d d. ! queue ! glupload \
! glcolorconvert ! m.
which can be represented with this simplified graph:
The advantage of this approach is that many heavy-lifting tasks happen in the
GPU. WPE loads the page using its WPENetworkProcess external process, parses
everything (DOM, CSS, JS, …) and renders it as a EGLImage, shared with the
UIProcess (the GStreamer application, gst-launch in this case). In most
situations decodebin will use an hardware decoder. The decoded video frames are
uploaded to the GPU and composited with the EGLImages representing the web-page,
in a single OpenGL scene, using the glvideomixer element.
The initial version of the GstWPE plugin is now part of the gst-plugins-bad
staging area, where most new plugins are uploaded for further improvements later
on. Speaking of improvements, the following tasks have been identified:
The wpesrc draw-background property is not yet operational due to missing
WPEWebKit API for background-color configuration support. I expect to complete
this task very soon, interested people can follow this bugzilla ticket
Audio support, WPEWebKit currently provides only EGLImages to application
side. The audio session is rendered directly to GStreamer’s autoaudiosink in
WebKit, so there’s currently no audio sharing support in wpesrc.
DMABuf support as an alternative to EGLImages. WPEWebKit internally leverages
linux-dmabuf support already but doesn’t expose the file descriptors and plane informations.
Better navigation events support. GStreamer’s navigation events API was
initially designed for DVD menus navigation uses-cases mostly, the exposed
input events informations are not a perfect match for WPEWebKit which expects
hardware-level informations from keyboard, mouse and touch devices.
There are more ways and use-cases related with WPE, I expect to unveil another
WPE embedding project very soon. Watch this space! As usual many thanks to my
Igalia colleagues for sponsoring this work. We are always happy to hear what
others are doing with WPE and to help improving it, don’t hesitate to get in touch!
Multimedia applications based on GStreamer usually handle playback with the
playbin element. I recently added support for playbin3 in WebKit. This post
aims to document the changes needed on application side to support this new
generation flavour of playbin.
So, first of, why is it named playbin3 anyway? The GStreamer 0.10.x series had a
playbin element but a first rewrite (playbin2) made it obsolete in the GStreamer
1.x series. So playbin2 was renamed to playbin. That’s why a second rewrite is
nicknamed playbin3, I suppose :)
Why should you care about playbin3? Playbin3 (and the elements it’s using
internally: parsebin, decodebin3, uridecodebin3 among others) is the result of a
deep re-design of playbin2 (along with decodebin2 and uridecodebin) to better support:
gapless playback
audio cross-fading support (not yet implemented)
adaptive streaming
reduced CPU, memory and I/O resource usage
faster stream switching and full control over the stream selection process
This work was carried on mostly by Edward Hervey, he presented his work in
detail at 3 GStreamer conferences. If you want to learn more about this and the
internals of playbin3 make sure to watch his awesome presentations at the 2015
gst-conf, 2016 gst-conf and 2017 gst-conf.
Playbin3 was added in GStreamer 1.10. It is still considered experimental but in
my experience it works already very well. Just keep in mind you should use at
least the latest GStreamer 1.12 (or even the upcoming 1.14) release before
reporting any issue in Bugzilla. Playbin3 is not a drop-in replacement for
playbin, both elements share only a sub-set of GObject properties and signals.
However, if you don’t want to modify your application source code just yet, it’s
very easy to try playbin3 anyway:
$ USE_PLAYBIN3=1 my-playbin-based-app
Setting the USE_PLAYBIN environment variable enables a code path inside the
GStreamer playback plugin which swaps the playbin element for the playbin3
element. This trick provides a glance to the playbin3 element for the most lazy
people :) The problem is that depending on your use of playbin, you might get
runtime warnings, here’s an example with the Totem player:
$ USE_PLAYBIN3=1 totem ~/Videos/Agent327.mp4
(totem:22617): GLib-GObject-WARNING **: ../../../../gobject/gsignal.c:2523: signal 'video-changed' is invalid for instance '0x556db67f3170' of type 'GstPlayBin3'
(totem:22617): GLib-GObject-WARNING **: ../../../../gobject/gsignal.c:2523: signal 'audio-changed' is invalid for instance '0x556db67f3170' of type 'GstPlayBin3'
(totem:22617): GLib-GObject-WARNING **: ../../../../gobject/gsignal.c:2523: signal 'text-changed' is invalid for instance '0x556db67f3170' of type 'GstPlayBin3'
(totem:22617): GLib-GObject-WARNING **: ../../../../gobject/gsignal.c:2523: signal 'video-tags-changed' is invalid for instance '0x556db67f3170' of type 'GstPlayBin3'
(totem:22617): GLib-GObject-WARNING **: ../../../../gobject/gsignal.c:2523: signal 'audio-tags-changed' is invalid for instance '0x556db67f3170' of type 'GstPlayBin3'
(totem:22617): GLib-GObject-WARNING **: ../../../../gobject/gsignal.c:2523: signal 'text-tags-changed' is invalid for instance '0x556db67f3170' of type 'GstPlayBin3'
sys:1: Warning: g_object_get_is_valid_property: object class 'GstPlayBin3' has no property named 'n-audio'
sys:1: Warning: g_object_get_is_valid_property: object class 'GstPlayBin3' has no property named 'n-text'
sys:1: Warning: ../../../../gobject/gsignal.c:3492: signal name 'get-video-pad' is invalid for instance '0x556db67f3170' of type 'GstPlayBin3'
As mentioned previously, playbin and playbin3 don’t share the same set of
GObject properties and signals, so some changes in your application are required
in order to use playbin3.
If your application is based on the GstPlayer library then you should set the
GST_PLAYER_USE_PLAYBIN3 environment variable. GstPlayer already handles both
playbin and playbin3, so no changes needed in your application if you use GstPlayer!
Ok, so what if your application relies directly on playbin? Some
changes are needed! If you previously used playbin stream selection
properties and signals, you will now need to handle the GstStream and
GstStreamCollection APIs. Playbin3 will emit a stream collection
message on the bus, this is very nice because the collection
includes information (metadata!) about the streams (or tracks) the
media asset contains. In playbin this was handled with a bunch of
signals (audio-tags-changed, audio-changed, etc), properties (n-audio,
n-video, etc) and action signals (get-audio-tags, get-audio-pad, etc).
The new GstStream API provides a centralized and
non-playbin-specific access point for all these informations. To
select streams with playbin3 you now need to send a select_streams
event so that the demuxer can know exactly which streams should be
exposed to downstream elements. That means potentially improved
performance! Once playbin3 completed the stream selection it will emit
a streams selected message, the application should handle this
message and potentially update its internal state about the selected
streams. This is also the best moment to update your UI regarding the
selected streams (like audio track language, video track dimensions, etc).
Another small difference between playbin and playbin3 is about the source
element setup. In playbin there is a source read-only GObject property and a
source-setup GObject signal. In playbin3 only the latter is available, so your
application should rely on source-setup instead of the notify::source
GObject signal.
The gst-play-1.0 playback utility program already supports playbin3 so it
provides a good source of inspiration if you consider porting your application
to playbin3. As mentioned at the beginning of this post, WebKit also now
supports playbin3, however it needs to be enabled at build time using the CMake
-DUSE_GSTREAMER_PLAYBIN3=ON option. This feature is not part of the WebKitGTK+
2.20 series but should be shipped in 2.22. As a final note I wanted to
acknowledge my favorite worker-owned coop Igalia for allowing me to work on
this WebKit feature and also our friends over at Centricular for all the
quality work on playbin3.
Khronos Group has published two new extensions for Vulkan: VK_KHR_shader_float16_int8 and VK_KHR_shader_float_controls. In this post, I will talk about VK_KHR_shader_float_controls, which is the extension I have been implementing on Anvil driver, the open-source Intel Vulkan driver, as part of my job at Igalia. For information about VK_KHR_shader_float16_int8 and its implementation in Mesa, you can read Iago’s blogpost.
The Vulkan Working Group has defined a new extension VK_KHR_shader_float_controls, which allows applications to query and override the implementation’s default floating point behavior for rounding modes, denormals, signed zero and infinity. From the Vulkan application developer perspective, VK_shader_float_controls defines a new structure called VkPhysicalDeviceFloatControlsPropertiesKHR where the drivers expose the supported capabilities such as the rounding modes for each floating point data type, how the denormals are expected to be handled by the hardware (either flush to zero or preserve their bits) and if the value is a signed zero, infinity and NaN, whether it will preserve their bits.
This structure will be filled by the driver when calling vkGetPhysicalDeviceProperties2(), with a pointer to such structure as one of the pNext pointers of VkPhysicalDeviceProperties2 structure. With that, we know if the driver will support the SPIR-V capabilities we want to use in our shaders, if separate*Settings are true, remember to check the value of the property for the floating point bit-size types you are planning to work with.
The required bits to enable such capabilities in a SPIR-V shader are the following:
Enable the extension: OpExtension "SPV_KHR_float_controls"
Enable the desired capability. For example: OpCapability DenormFlushToZero
Specify where to apply it. For example, we would like to flush to zero all fp64 denormalss in the %main function of a shader: OpExecutionMode %main DenormFlushToZero 64. If we want to apply different modes, we would repeat that line with the needed ones.
Profit!
I implemented the support of this extensions for the Anvil’s supported GPUs (Broadwell, Skylake, Kabylake and newer), although we don’t support all the capabilities. For example on Broadwell, float16 denormals are not supported, and the support for flushing to zero the float16 denormals is not supported for all the instructions in the rest of generations.
If you are interested, the patches are now under review :-) As there are not real world code using this feature yet, please fill any bug you find about this in our bugzilla.
The last time I talked about my driver work was to announce the implementation of the shaderInt16 feature for the Anvil Vulkan driver back in May, and since then I have been working on VK_KHR_shader_float16_int8, a new Vulkan extension recently announced by the Khronos group, for which I have just posted initial patches in mesa-dev supporting Broadwell and later Intel platforms.
As you probably guessed by the name, this extension enables Vulkan to consume SPIR-V shaders that use of Float16 and Int8 types in arithmetic operations, extending the functionality included with VK_KHR_16bit_storage and VK_KHR_8bit_storage, which was limited to load/store operations. In theory, applications that do not need the range and precision of regular 32-bit floating point and integers, can use these new types to improve performance by increasing ALU throughput and reducing register pressure, which in some platforms can also lead to improved parallelism.
In the case of the Intel platforms initial testing done by Intel suggests that better ALU throughput is expected when issuing half-float instructions. Lower register pressure is also expected, at least for SIMD16 fragment and compute shaders, where we can pack all 16-channels worth of half-float data into a single GPU register, which could significantly improve performance for shaders that would otherwise need to spill registers to memory.
Another neat thing is that while VK_KHR_shader_float16_int8 is a Vulkan extension, its implementation is mostly API agnostic, so most of the work we did here should also help us have a proper mediump implementation for GLSL ES shaders in the future.
There are a few caveats to consider as well though: on some hardware platforms smaller bit-sizes have certain hardware restrictions that may lead to emitting worse shader code in some scenarios, and generally, Mesa’s compiler infrastructure (and the Intel compiler backend in particular) have a long history of being 32-bit only, so there are parts of the compiler stack that still work better for 32-bit code.
Because VK_KHR_shader_float16_int8 is a brand new feature, we don’t really have any real world use cases yet. This is on top of the fact that Mesa’s compiler backends have been mostly (or exclusively) 32-bit aware until now (and more recently 64-bit too), so going forward I would expect a lot of focus on making our compiler be as robust (and optimal) for 16-bit code as it is for 32-bit code.
While we are already aware of a few areas where we can do better and I am currently working on addressing a few of these, one of the major limiting factors we have at the moment is the fact that the only source of 16-bit shaders available to us is the Khronos CTS, which due to its particular motivation, is very different from real world shader workloads and it is not a valid source material to drive compiler optimization work. Unfortunately, it might take some time until we start seeing applications using these new features, so in the meantime we will need to find other ways to drive further work in this area, and I think our best option here might be GLSL ES’s mediump and lowp qualifiers.
GLSL ES mediump and lowp qualifiers have been around for a long time but they are only defined as hints to the shader compiler that lower precision is acceptable and we have never really used them to emit half-float code. Thankfully, Topi Pohjolainen from Intel has been working on this for a while, which would open up a much better scenario for improving our 16-bit compiler paths, so this is something I am really looking forward to.
Finally, as I say above, we could could definitely use more testing and feedback from real world use cases, so if you decide to use this feature in your next project and you hit any bugs, please be sure to file them in Bugzilla so we can continue to improve our implementation.
I tried, several months ago, the open source version of Intel MediaSDK and it was a complete mess. In order to review some patches for gst-msdk I tried it again. I am surprised how the situation has improved since then.
It will be built in its source directory: ~/msdk/MediaSDK/build
It will be installed in /opt/intel
$ git clone https://github.com/Intel-Media-SDK/MediaSDK.git
$ cd MediaSDK
$ mkdir build
$ cd build
$ cmake ..
$ make
$ make install
Build media-driver
$ cd ~/msdk
$ git clone https://github.com/intel/media-driver.git
$ git clone https://github.com/intel/gmmlib.git
$ mkdir build
$ cd build
$ cmake ../media-driver
$ make
Let’s install media-driver in /opt/intel too
$ cd ~/msdk/build
$ cp ./media_driver/iHD_drv_video.so /opt/intel
But don’t remove, rename or move the directori ~/msdk/build because iHD_drv_video.so links against libigdgmm.so.5 which is there. Thus either you keep the directory or you install that library in a path searchable by the linker, or set the environment variable LD_LIBRARY_PATH
A while back, I wrote a blog post explaining some of the basics of writing plugins for the VPP networking toolkit.
In that previous post, I explained a few mechanisms for hooking a plugin into VPP’s graph architecture so that your code can process incoming packets.
I also briefly mentioned something called DPOs (data path objects) but didn’t explain what they are or how they work. Since then, I’ve been reading and hacking on code that involves DPOs, so I’d like to attempt to explain them in this post.
(I’ll be assuming you’ve read the previous post or are already somewhat familiar with VPP, so if that’s not the case you may want to take a look at my previous post)
Data path objects
Here’s how DPOs are defined in their main header file (vpp.h):
A Data-Path Object is an object that represents actions that are applied to packets as they are switched through VPP’s data-path.
So a DPO is an object, which means that it’s a value that we can create and manipulate (via instances of of dpo_id_t) and also has some behavior (i.e., it has specialized methods or functions that do something).
By “as they are switched through”, this means that DPOs can be set to activate via rules set in VPP’s FIB (forwarding information base). For example, you can add a DPO that will act on IPv6 packets matching an address prefix that you choose.
The job of a FIB is to maintain forwarding information so that the switch knows which interfaces on which to forward packets along. With DPOs, you can add entries to the FIB that tell VPP to forward packets via your DPO to a VPP node of your choosing instead (where presumably you will act on the packets somehow).
You can see this at work by interacting with the FIB in VPP. Here’s an example CLI interaction:
The first two commands are part of the DS-Lite plugin and use some DPOs to set up a kind of IPv4 in IPv6 tunnel. You can see from the results of show fib
entry that the commands have populated the FIB with entries for the given addresses and their associated DPO: DS-Lite: AFTR:0.
The ability to tie into the FIB is why you may want to use DPOs instead of some of the mechanisms I mentioned in the previous blog post. For some applications, it could make sense to hook into a feature arc because you potentially want to look at all packets (e.g., a monitoring program like an IPFIX meter) or, say, all IP packets. But in other cases, you are only interested in packets going to a specific prefix (e.g., you are setting up an endpoint for a tunnel) and would like to take advantage of the FIB for that.
DPO API
In order to set up DPOs, you first create an interface of DPO functions for your own DPO type. I’ve been reading the DS-Lite implementation in VPP a lot recently so I’ll show some (simplified) examples from that.
The typical pattern to use DPOs is to first create your own DPO type and create an API of DPO functions to use with that type. The first part of this API is a constructor function for making instances of the DPO, like dslite_dpo_create:
The dpo_set function takes a protocol constant, an index_t, and a dpo_id_t (a struct that identifies a particular DPO). It’s used to initialize the DPO. You could directly ues dpo_set if you wanted by passing in the dslite_dpo_type, so dslite_dpo_create is effectively a partial application of dpo_set.
A use of the constructor in your API client’s code might look like this:
1
2
3
4
5
/* declaration & temp initialization of DPO */dpo_id_tmy_dpo=DPO_INVALID;/* initialize DPO for desired protocol */dslite_dpo_create(DPO_PROTO_IP6,0,&my_dpo);
The constructor takes a few arguments, namely the protocol to use, an index for the DPO, and a pointer to the DPO that’s going to be initialized.
The most interesting argument is the protocol, which in this case is DPO_PROTO_IP6. You pass in a protocol at construction time because:
DPOs can be specialized to work on packets with a specific protocol because the actions you take on them are specialized, and
DPOs can be used with more than one protocol type, for example both IPv4 and IPv6.
In particular, you can also send packets to different nodes depending on the protocol that is matched. This is set up with some additional data structures in the API code like this:
The code above basically constructs a table mapping DPO protocol to arrays of node names. The table doesn’t have to be exhaustive (relative to all protocols that DPOs work on), but it should cover whatever protocols you want to use with your particular DPO type.
The nodes specified in your mapping are actually registered for a DPO type by calling the dpo_register_new_type function:
This dslite_dpo_module_init function is called from the NAT plugin’s initialization function (the DS-Lite code is a part of the NAT code). If you write your own DPO API, you’ll need to register the new DPO type in your VPP plugin’s initialization code.
You might be wondering where the object-oriented aspect of DPOs come from, given the allusion in the name. When defining your DPO API, you also define a virtual function table struct (dpo_vft_t) that is passed to dpo_register_new_type call shown above. That table might look like this:
In which the fields are basically methods that you implement for the DPO type. For the DS-Lite example, these functions do very little so I won’t go into the details here.
Using DPOs in forwarding
Once you’ve defined your DPO type and API, you can use it to forward packets to your VPP graph nodes. In order to hook your DPO up to the FIB, which lets you switch packets to your nodes, you need to construct a DPO instance in your plugin code and then call an function that registers FIB entries.
This example code from the DS-Lite implementation illustrates some of this:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
/* recall constructor examples from earlier */dslite_dpo_create(DPO_PROTO_IP6,0,&dpo);/* FIB prefix data structure, used below */fib_prefix_tpfx={.fp_proto=FIB_PROTOCOL_IP6,.fp_len=128,.fp_addr.ip6.as_u64[0]=addr->as_u64[0],.fp_addr.ip6.as_u64[1]=addr->as_u64[1],};/* register FIB entry for DPO */fib_table_entry_special_dpo_add(0,&pfx,/* if you're writing a plugin you use this, some other DPO code uses other constants */FIB_SOURCE_PLUGIN_HI,FIB_ENTRY_FLAG_EXCLUSIVE,&dpo);
The excerpt above is doing a few things:
Constructs a DPO and putting it in the dpo variable,
Declares a FIB prefix (fib_prefix_t) used for switching (which could be an address for IP, or a label for MPLS), and
Adds an entry to the FIB using the DPO and FIB prefix.
With that information added, the FIB can start switching packets to the nodes specified in your DPO (in this case to dslite-ce-decap). When packets go to your node, they are processed like in any other VPP node that you write.
To explain what the above excerpt is doing a bit more concretely, recall that this example is taken from the DS-Lite code. DS-Lite is a mechanism for sending IPv4 traffic tunneled (i.e., encapsulated) over an IPv6 network.
The DPO above is associated with the server endpoint for a DS-Lite tunnel that does NAT on the inner (decapsulated) packet, which means the server only receives encapsulated packets addressed to its IPv6 address.
Therefore the prefix is an IPv6 address (put inside .fp_addr.ip6 above) and the prefix length is 128, the full length of the address.
In other networking setups, you may have a different kind of tunnel in which the client does NAT rather than the server. In this case, you might set a prefix length that isn’t the full length of an address, and instead corresponds to however you allocate your NAT addresses (see the MAP-E code in VPP for an example of this).
The point here is that DPOs let you use the typical prefix forwarding capabilities of IP (or MPLS, etc) to hook up packets to your VPP node.
Further reading
Hopefully this blog post made it a bit clearer why you might want to use DPOs in your own VPP code and how to start doing so. To learn more, I would suggest just reading examples in the code base (files ending with _dpo.c and _dpo.h are helpful, and then look for uses of those API functions).
The DS-Lite code is also relatively simple and easy to read. The main files for the DPO code in DS-Lite are dslite_dpo.c and dslite.c.
CoreFoundation (used by macOS and iOS, and thus Safari)
CFNet (used by iTunes on Windows… I think only iTunes?)
cURL (used by most Windows applications, also PlayStation)
libsoup (used by WebKitGTK+ and WPE WebKit)
One guess which of those we’re going to be talking about in this post. Yeah, of course, libsoup! If you’re not familiar with libsoup, it’s the GNOME HTTP library. Why is it called libsoup? Because before it was an HTTP library, it was a SOAP library. And apparently somebody thought that when Mexican people say “soap,” it often sounds like “soup,” and also thought that this was somehow both funny and a good basis for naming a software library. You can’t make this stuff up.
Anyway, libsoup is built on top of GIO’s sockets APIs. Did you know that GIO has Object wrappers for BSD sockets? Well it does. If you fancy lower-level APIs, create a GSocket and have a field day with it. Want something a bit more convenient? Use GSocketClient to create a GSocketConnection connected to a GNetworkAddress. Pretty straightforward. Everything parallels normal BSD sockets, but the API is nice and modern and GObject, and that’s really all there is to know about it. So when you point WebKitGTK+ at an HTTP address, libsoup is using those APIs behind the scenes to handle connection establishment. (We’re glossing over details like “actually implementing HTTP” here. Trust me, libsoup does that too.)
Things get more fun when you want to load an HTTPS address, since we have to add TLS to the picture, and we can’t have TLS code in GIO or GLib due to this little thing called “copyright law.” See, there are basically three major libraries used to implement TLS on Linux, and they all have problems:
OpenSSL is by far the most popular, but it’s, hm, shall we say technically non-spectacular. There are forks, but the forks have problems too (ask me about BoringSSL!), so forget about them. The copyright problem here is that the OpenSSL license is incompatible with the GPL. (Boring details: Red Hat waves away this problem by declaring OpenSSL a system library qualifying for the GPL’s system library exception. Debian has declared the opposite, so Red Hat’s choice doesn’t gain you anything if you care about Debian users. The OpenSSL developers are trying to relicense to the Apache license to fix this, but this process is taking forever, and the Apache license is still incompatible with GPLv2, so this would make it impossible to use GPLv2+ software except under the terms of GPLv3+. Yada yada details.) So if you are writing a library that needs to be used by GPL applications, like say GLib or libsoup or WebKit, then it would behoove you to not use OpenSSL.
GnuTLS is my favorite from a technical standpoint. Its license is LGPLv2+, which is unproblematic everywhere, but some of its dependencies are licensed LGPLv3+, and that’s uncomfortable for many embedded systems vendors, since LGPLv3+ contains some provisions that make it difficult to deny you your freedom to modify the LGPLv3+ software. So if you rely on embedded systems vendors to fund the development of your library, like say libsoup or WebKit, then you’re really going to want to avoid GnuTLS.
NSS is used by Firefox. I don’t know as much about it, because it’s not as popular. I get the impression that it’s more designed for the needs of Firefox than as a Linux system library, but it’s available, and it works, and it has no license problems.
So naturally GLib uses NSS to avoid the license issues of OpenSSL and GnuTLS, right?
Haha no, it uses a dynamically-loadable extension point system to allow you to pick your choice of OpenSSL or GnuTLS! (Support for NSS was started but never finished.) This is OK because embedded systems vendors don’t use GPL applications and have no problems with OpenSSL, while desktop Linux users don’t produce tivoized embedded systems and have no problems with LGPLv3. So if you’re using desktop Linux and point WebKitGTK+ at an HTTPS address, then GLib is going to load a GIO extension point called glib-networking, which implements all of GIO’s TLS APIs — notably GTlsConnection and GTlsCertificate — using GnuTLS. But if you’re building an embedded system, you simply don’t build or install glib-networking, and instead build a different GIO extension point called glib-openssl, and libsoup will create GTlsConnection and GTlsCertificate objects based on OpenSSL instead. Nice! And if you’re Centricular and you’re building GStreamer for Windows, you can use yet another GIO extension point, glib-schannel, for your native Windows TLS goodness, all hidden behind GTlsConnection so that GStreamer (or whatever application you’re writing) doesn’t have to know about SChannel or OpenSSL or GnuTLS or any of that sad complexity.
Now you know why the TLS extension point system exists in GIO. Software licenses! And you should not be surprised to learn that direct use of any of these crypto libraries is banned in libsoup and WebKit: we have to cater to both embedded system developers and to GPL-licensed applications. All TLS library use is hidden behind the GTlsConnection API, which is really quite nice to use because it inherits from GIOStream. You ask for a TLS connection, have it handed to you, and then read and write to it without having to deal with any of the crypto details.
As a recap, the layering here is: WebKit -> libsoup -> GIO (GLib) -> glib-networking (or glib-openssl or glib-schannel).
So when Epiphany fails to load a webpage, and you’re looking at a TLS-related error, glib-networking is probably to blame. If it’s an HTTP-related error, the fault most likely lies in libsoup. Same for any other GNOME applications that are having connectivity troubles: they all use the same network stack. And there you have it!
P.S. The glib-openssl maintainers are helping merge glib-openssl into glib-networking, such that glib-networking will offer a choice of GnuTLS or OpenSSL and obsoleting glib-openssl. This is still a work in progress. glib-schannel will be next!
P.S.S. libcurl also gives you multiple choices of TLS backend, but makes you choose which at build time, whereas with GIO extension points it’s actually possible to choose at runtime from the selection of installed extension points. The libcurl approach is fine in theory, but creates some weird problems, e.g. different backends with different bugs are used on different distributions. On Fedora, it used to use NSS, but now uses OpenSSL, which is fine for Fedora, but would be a license problem elsewhere. Debian actually builds several different backends and gives you a choice, unlike everywhere else. I digress.
I had the pleasure to attend the Web Engines Hackfest last week, hosted and organized by Igalia in its HQ in A Coruña. I’m really proud of what we are achieving with this event, thank so much to everybody involved in the organization but, specially, to the people attending. This year we had a lot of talent in a single place, hacking and sharing their expertise on the Web Platform; I really think we all have pushed the Web Platform forward during these days.
As you may already know, I’m part of the Web Platform team at Igalia working on the implementation of the CSS Grid Layout feature for Blink and WebKit web engines. This work has been sponsored by Bloomberg, as part of the collaboration we started several years ago to improve the Web Platform in many areas, from JS (V8, JSC and even ChakraCore) to different modules of the layout engine (eg. CSS features, editing/selection).
Since the day I received the invitation to attend the hackfest I knew that one of the tasks I wanted to hack on was the implementation of the CSS Grid feature in the new Chromium’s layout engine (still experimental), known as LayoutNG. Having Chistinan Biesinger, one of the Google engineers working on LayoutNG, here during 3 days was so good to let pass by the oportunity to, at least, start this task. I asked him to give a lighting talk during the layout breakout session about the current status of the LayoutNG project, a brief explanation of some of the most relevant details of its logic and its advantages with the current layout.
Layout Breakout Session
A small group of people interested on layout met to discuss about the future of layout in different web engines. I attended with some folks representing Igalia, and other people from Mozilla, Google, WebKit and ARM.
Christian described the key parts of the new LayoutNG, which provides a simpler code and generally better performance (although results are still preliminary since its under development phase). The concept of fragments gained relevance in this new layout model and currently, inline and block layout is basically complete; the multicolumn layout is quite advanced while Flexbox is still in the early stages (although a substantial portion of the layout tests are passing now).
We discussed about the different strategies that Firefox, Chrome and Safari are following to redesign the layout logic, which has a huge legacy codebase required to support the old web along the last years. However, browsers need to adapt to the new layout models that a modern Web Platform requires. Chrome with LayoutNG implies a clear bet, with a big team and strong determination; it seems it’ll be ready to ship in the first months of 2019. Firefox is also starting to implement a new layout design with Servo, although I couldn’t get details about its current status and plans. Finally, WebKit started a few months ago a new project called Next-Generation layout which tries to implement a new Layout Formatting Context (LFC) logic from scratch, getting rid f the huge technical debt acquired during the last years; although I couldn’t get confirmation, my opinion is that it’s still an experimental project.
We also had time to talk about the effort ARM is doing towards a better parallelization of the CSS parsing and style recalc logic, following a similar approach to Mozilla’s Sylo in Servo. It’s a very intresting initiative, but still quite experimental. There is some progress on specific codepahts, but still dealing with Oilpan (Blink’s Garbage Collector) which is the root cause of several issues that prevents to obtain an effective parallelization.
Hacking, hacking, hacking, ….
As I commented, this event is designed precisely to gather together some of the most brilliant minds in the Web Platform to discuss, analyze and hack on complex topics that are usually very difficult to handle when working remotely. I had a clear hacking task this time, so that’s why I decided to focus a bit more on coding. Although I already had assumed that implementing CSS Grid in LayoutNG would be a huge challenge, I decided to take it and at least start the task. I took as reference the Flexible Box implementation, which is under development right now and something Christian was partially involved on.
As it happened with the Flexbible Box implementation, the first step was to redesign the logic so that we can get rid of the dependency of the old Layout Tree, in this case, the LayoutGrid class. This has ben a complex and long task, which took me a quite big part of my time during the hackfest. The following diagrams show the redesign effort achieved, which I’d admit is still a preliminary approach that needs to be refined:
The next step was to implement an skeleton of the new layout-ng grid algorithm. Thanks to Christian’s direction, I quickly figure out how to do it and it looks like something like this:
Finally, I tried to implement the Grid layout’s algorithm, according to the CSS Grid Layout feature’s specification, using the new LayoutNG APIs. This is the more complex tasks, since I still have to learn how sizing and positioning functions are used on the new layout logic, specially how to use the new Fragments and ContainerBuilder concepts.
I submitted a WIP CL so that anybody can take a look, give suggestions or continue with the work. My plan is to devote some time to this challenge, every now and then, but I can’t set specifc goals or schedule for the time being. If anybody wants to speed up this task, perhaps it’d be possible to fund a project, which Igalia would be happy to participate.
Other Web Engines Hackfest stories
I also tried to attend some of the talks give during the hackfest and participate in a few breakout sessions. I’ll give now my impression of some of the ones I liked more.
I enjoyed a lot the one given by Camille Lamy, Colin Blundell and Robert Kroeger (Google) about the Chrome’s Servicification project. The new services design they are implementing is awesome and it will improve for sure Chrome modularity and codebase maintenance.
I participated in the MathML breakout session, which has somehow related to LayoutNG. Igalia launched a crowfunding campaign to implement the MathML specification in Chrome, using the new LayoutNG APIs. We thin that MathML could be a great success case for the new LayoutNG APIs, which has the goal of provide a stable API to implement new and complex layout models. This model will provide flexibility to the web engine, proving an easier way to implement new layout models without depending too much on the Chrome development cycle. In a way, this development model could be similar to a polyfill, but it’s integrated in the browser as native code instead of via external libraries.
To avoid this bug, downgrade to mesa-18.2.2-1.fc29:
$ sudo dnf downgrade mesa*
You can also update to mesa-18.2.4-2.fc29, but this build has not yet reached updates-testing, let alone stable, so downgrading is easier for now. Another workaround is to run your application with accelerated compositing mode disabled, to avoid OpenGL usage:
$ WEBKIT_DISABLE_COMPOSITING_MODE=1 epiphany
On the bright side of things, from all the bug reports I’ve received over the past two days I’ve discovered that lots of people use Epiphany and notice when it’s broken. That’s nice!
Huge thanks to Dave Airlie for quickly preparing the fixed mesa update, and to Jakub Jelenik for handling the same for GCC.
Last month, I attended the Web Engines Hackfest (hosted by Igalia in A Coruña, Spain) and also the WebKit Contributors Meeting (hosted by Apple in San Jose, California). These are easily the two biggest WebKit development events of the year, and it’s always amazing to meet everyone in person yet again. A Coruña is an amazing city, and every browser developer ought to visit at least once. And the Contributors Meeting is a no-brainer event for WebKit developers.
One of the main discussion points this year was Media Source Extensions (MSE). MSE is basically a way for browsers to control how videos are downloaded. Until recently, if you were to play a YouTube video in Epiphany, you’d notice that the video loads way faster than it does in other browsers. This is because WebKitGTK+ — until recently — had no support for MSE. In other browsers, YouTube uses MSE to limit the speed at which video is downloaded, in order to reduce wasted bandwidth in case you stop watching the video before it ends. But with WebKitGTK+, MSE was not available, so videos would load as quickly as possible. MSE also makes it harder for browsers to offer the ability to download the videos; you’ll notice that neither Firefox nor Chrome offer to download the videos in their context menus, a feature that’s been available in Epiphany for as long as I remember.
So that sounds like it’s good to not have MSE. Well, the downside is that YouTube requires it in order to receive HD videos, to avoid that wasted bandwidth and to make it harder for users to download HD videos. And so WebKitGTK+ users have been limited to 720p video with H.264 and 480p video with WebM, where other browsers had access to 1080p and 1440p video. I’d been stuck with 480p video on Fedora for so long, I’d forgotten that internet video could look good.
Unfortunately, WebKitGTK+ was quite late to implement MSE. All other major browsers turned it on several years ago, but WebKitGTK+ dawdled. There was some code to support MSE, but it didn’t really work, and was disabled. And so it came to pass that, in September of this year, YouTube began to require MSE to access any WebM video, and we had a crisis. We don’t normally enable major new features in stable releases, but this was an exceptional situation and users would not be well-served by delaying until the next release cycle. So within a couple weeks, we were able to release WebKitGTK+ 2.22.2 and Epiphany 3.30.1 (both on September 21), and GStreamer 1.14.4 (on October 2, thanks to Tim-Philipp Müller for expediting that release). Collectively, these releases enabled basic video playback with MSE for users of GNOME 3.30. And if you still use of GNOME 3.28, worry not: you are still supported and can get MSE if you update to Epiphany 3.28.5 and also have the aforementioned versions of WebKitGTK+ and GStreamer.
MSE in WebKitGTK+ 2.22.2 had many rough edges because it was a mad rush to get the feature into a minimally-viable state, but those issues have been polished off in 2.22.3, which we released earlier this week on October 29. Be sure you have WebKitGTK+ 2.22.3, plus GStreamer 1.14.4, for a good experience on YouTube. Unfortunately we can’t provide support for older software versions anymore: if you don’t have GStreamer 1.14.4, then you’ll need to configure WebKitGTK+ with -DENABLE_MEDIA_SOURCE=OFF at build time and suffer from lack of MSE.
Epiphany 3.28.1 uses WebKitSettings to turn on the “enable-mediasource” setting. Turn that on if your application wants MSE now (if it’s a web browser, it certainly does). This setting will be enabled by default in WebKitGTK+ 2.24. Huge thanks to the talented developers who made this feature possible! Enjoy your 1080p and 1440p video.
When building WebKitGTK+, it’s a good idea to stick to the default values for the build options. If you’re building some sort of embedded system and really know what you’re doing, then OK, it might make sense to change some settings and disable some stuff. But Linux distros are generally well-advised to stick to the defaults to avoid creating problems for users.
One exception is if you need to disable certain features to avoid newer dependencies when building WebKit for older systems. For example, Ubuntu 18.04 disables web fonts (ENABLE_WOFF2=OFF) because it doesn’t have the libbrotli and libwoff2 dependencies that are required for that feature to work, hence some webpages will display using subpar fonts. And distributions shipping older versions of GStreamer will need to disable the ENABLE_MEDIA_SOURCE option (which is missing from the below feature list by mistake), since that requires the very latest GStreamer to work.
Other exceptions are the ENABLE_GTKDOC and ENABLE_MINIBROWSER settings, which distros do want. ENABLE_GTKDOC is disabled by default because it’s slow to build, and ENABLE_MINIBROWSER because, well, actually I don’t know why, you always want that one and it’s just annoying to find it’s not built.
OK, but really now, other than those exceptions, you should probably leave the defaults alone.
The feature list that prints when building WebKitGTK+ looks like this:
-- ENABLE_ACCELERATED_2D_CANVAS .......... OFF
-- ENABLE_DRAG_SUPPORT ON
-- ENABLE_GEOLOCATION .................... ON
-- ENABLE_GLES2 OFF
-- ENABLE_GTKDOC ......................... OFF
-- ENABLE_ICONDATABASE ON
-- ENABLE_INTROSPECTION .................. ON
-- ENABLE_JIT ON
-- ENABLE_MINIBROWSER .................... OFF
-- ENABLE_OPENGL ON
-- ENABLE_PLUGIN_PROCESS_GTK2 ............ ON
-- ENABLE_QUARTZ_TARGET OFF
-- ENABLE_SAMPLING_PROFILER .............. ON
-- ENABLE_SPELLCHECK ON
-- ENABLE_TOUCH_EVENTS ................... ON
-- ENABLE_VIDEO ON
-- ENABLE_WAYLAND_TARGET ................. ON
-- ENABLE_WEBDRIVER ON
-- ENABLE_WEB_AUDIO ...................... ON
-- ENABLE_WEB_CRYPTO ON
-- ENABLE_X11_TARGET ..................... ON
-- USE_LIBHYPHEN ON
-- USE_LIBNOTIFY ......................... ON
-- USE_LIBSECRET ON
-- USE_SYSTEM_MALLOC ..................... OFF
-- USE_WOFF2 ON
And, asides from the exceptions noted above, those are probably the options you want to ship with.
Why are some things disabled by default? ENABLE_ACCELERATED_2D_CANVAS is OFF by default because it is experimental (i.e. not great :) and requires CairoGL, which has been available in most distributions for about half a decade now, but still hasn’t reached Debian yet, because the Debian developers know that the Cairo developers consider CarioGL experimental (i.e. not great!). Many of our developers use Debian, and we’re not keen on having two separate sets of canvas bugs depending on whether you’re using Debian or not, so best keep this off for now. ENABLE_GLES2 switches you from desktop GL to GLES, which is maybe needed for embedded systems with crap proprietary graphics drivers, but certainly not what you want when building for a general-purpose distribution with mesa. Then ENABLE_QUARTZ_TARGET is for building on macOS, not for Linux. And then we come to USE_SYSTEM_MALLOC.
USE_SYSTEM_MALLOC disables WebKit’s bmalloc memory allocator (“fast malloc”) in favor of glibc malloc. bmalloc is performance-optimized for macOS, and I’m uncertain how its performance compares to glibc malloc on Linux. Doesn’t matter really, because bmalloc contains important heap security features that will be disabled if you switch to glibc malloc, and that’s all you need to know to decide which one to use. If you disable bmalloc, you lose the Gigacage, isolated heaps, heap subspaces, etc. I don’t pretend to understand how any of those things work, so I’ll just refer you to this explanation by Sam Brown, who sounds like he knows what he’s talking about. The point is that, if an attacker has found a memory vulnerability in WebKit, these heap security features make it much harder to exploit and take control of users’ computers, and you don’t want them turned off.
USE_SYSTEM_MALLOC is currently enabled (bad!) in openSUSE and SUSE Linux Enterprise 15, presumably because when the Gigacage was originally introduced, it crashed immediately for users who set address space (virtual memory allocation) limits. Gigacage works by allocating a huge address space to reduce the chances that an attacker can find pointers within that space, similar to ASLR, so limiting the size of the address space prevents Gigacage from working. At first we thought it made more sense to crash than to allow a security feature to silently fail, but we got a bunch of complaints from users who use ulimit to limit the address space used by processes, and also from users who disable overcommit (which is required for Gigacage to allocate ludicrous amounts of address space), and so nowadays we just silently disable Gigacage instead if enough address space for it cannot be allocated. So hopefully there’s no longer any reason to disable this important security feature at build time! Distributions should be building with the default USE_SYSTEM_MALLOC=OFF.
which all looks pretty reasonable to me: certain features that require “newer” dependencies are disabled on the old distros, and NPAPI plugins are not supported in the enterprise distro, and JIT doesn’t work on odd architectures. I would remove the ENABLE_JIT=OFF lines only because WebKit’s build system should be smart enough nowadays to disable it automatically to save you the trouble of thinking about which architectures the JIT works on. And I would also remove the -DUSE_SYSTEM_MALLOC=ON line to ensure users are properly protected.
As part of my job at Igalia, I have been attending 2-3 events per year. My role mostly as a Chromium stack engineer is not usually much demanding regarding conference trips, but they are quite important as an opportunity to meet collaborators and project mates.
This month has been a bit different, as I ended up visiting Santa Clara LG Silicon Valley Lab in California, Igalia headquarters in A Coruña, and Dresden. It was mostly because I got involved in the discussions for the web runtime implementation being developed by Igalia for AGL.
AGL f2f at LGSVL
It is always great to visit LG Silicon Valley Lab (Santa Clara, US), where my team is located. I have been participating for 6 years in the development of the webOS web stack you can most prominently enjoy in LG webOS smart TV.
One of the goals for next months at AGL is providing an efficient web runtime. In LGSVL we have been developing and maintaining WAM, the webOS web runtime. And as it was released with an open source license in webOS Open Source Edition, it looked like a great match for AGL. So my team did a proof of concept in May and it was succesful. At the same time Igalia has been working on porting Chromium browser to AGL. So, after some discussions AGL approved sponsoring my company, Igalia for porting the LG webOS web runtime to AGL.
So we took the opportunity to continue discussions and progress in the development of the WAM AGL port. And, as we expected, it was quite beneficial to unblock tasks like AGL app framework security integration, and the support of AGL latest official release, Funky Flounder. Julie Kim from Igalia attended the event too, and presented an update on the progress of the Ozone Wayland port.
The organization and the venue were great. Thanks to LGSVL!
Web Engines Hackfest 2018 at Igalia
Next trip was definitely closer. Just 90 minutes drive to our Igalia headquarters in A Coruña.
Igalia has been organizing this event since 2009. It is a cross-web-engine event, where engineers of Mozilla, Chromium and WebKit have been meeting yearly to do some hacking, and discuss the future of the web.
This time my main interest was participating in the discussions about the effort by Igalia and Google to support Wayland natively in Chromium. I was pleased to know around 90% of the work had already landed in upstream Chromium. Great news as it will smooth integration of Chromium for embedders using Ozone Wayland, like webOS. It was also great to know the work for improving GPU performance reducing the number of copies required for painting web contents.
Other topics of my interest:
– We did a follow-up of the discussion in last BlinkOn about the barriers for Chromium embedders, sharing the experiences maintaining a downstream Chromium tree.
– Joined the discussions about the future of WebKitGTK. In particular the graphics pipeline adaptation to the upcoming GTK+ 4.
As usual, the organization was great. We had 70 people in the event, and it was awesome to see all the activity in the office, and so many talented engineers in the same place. Thanks Igalia!
The last event in barely a month was my first visit to the beautiful town of Dresden (Germany).
The goal was continuing the discussions for the projects Igalia is developing for AGL platform: Chromium upstream native Wayland support, and the WAM web runtime port. We also had a booth showcasing that work, but also our lightweight WebKit port WPE that was, as usual, attracting interest with its 60fps video playback performance in a Raspberry Pi 2.
I co-presented with Steve Lemke a talk about the automotive activities at LGSVL, taking the opportunity to update on the status of the WAM web runtime work for AGL (slides here). The project is progressing and Igalia should be landing soon the first results of the work.
Igalia booth at AGL AMM Europe 2018
It was great to meet all this people, and discuss in person the architecture proposal for the web runtime, unblocking several tasks and offering more detailed planning for next months.
Dresden was great, and I can’t help highlighting the reception and guided tour in the Dresden Transportation Museum. Great choice by the organization. Thanks to Linux Foundation and the AGL project community!
Next: Chrome Dev Summit 2018
So… what’s next? I will be visiting San Francisco in November for Chrome Dev Summit.
I can only thank Igalia for sponsoring my attendance to these events. They are quite important for keeping things moving forward. But also, it is also really nice to meet friends and collaborators. Thanks Igalia!
Just a quick update before boarding to Lyon for TPAC 2018.
This year 12 igalians will be at TPAC,
10 employees (Álex García Castro, Daniel Ehrenberg, Javier Fernández, Joanmarie Diggs, Martin Robinson, Rob Buis, Sergio Villar, Thibault Saunier and myself)
and 2 coding experience students (Oriol Brufau and Sven Sauleau).
We will represent Igalia in the different working groups and breakout sessions.
On top of that Igalia will have a booth in the solutions showcase
where we’ll have a few demos of our last developments like:
WebRTC, MSE, CSS Grid Layout, CSS Box Alignment, MathML, etc.
Showing them in some low-end boards like the Raspebrry Pi
using WPE an optimized WebKit port for embedded platforms.
In my personal case I’ll be attending the CSS Working Group (CSSWG)
and Houdini Task Force meetings to follow the work
Igalia has been doing on the implementation of different standards.
In addition, I’ll be giving a talk about how to contribute
to the evolution of CSS on the W3C Developer Meetup
that happens on Monday.
I’ll try to explain how easy is nowadays to provide feedback
to the CSSWG and have some influence on the different specifications.
Ever wanted to work on the design, specification and implementation of new web platform features? Does participating from anywhere in the world in a flat cooperative doing free software sound good? Igalia's hiring a web platform engineer! https://t.co/75A6I0zEfe
Last but not least, Igalia Web Platform Team is hiring,
we’re looking for people willing to work on web standards
from the implementation on the different browser engines,
to the discussions with the standard bodies or the definition of test suites.
If you’re attending TPAC and you want to work on a flat company focused on free software development,
probably you are a good candidate to join us.
Read the position announcement and don’t hesitate to talk to any of us there about that.
I was having a look through SpiderMonkey's source code today and found something interesting about how it represents heap objects and wanted to share.
I was first looking to see how to implement arbitrary-length integers ("bigints") by storing the digits inline in the allocated object. (I'll use the term "object" here, but from JS's perspective, bigints are rather values; they don't have identity. But I digress.) So you have a header indicating how many words it takes to store the digits, and the digits follow. This is how JavaScriptCore and V8 implementations of bigints work.
Incidentally, JSC's implementation was taken from V8. V8's was taken from Dart. Dart's was taken from Go. We might take SpiderMonkey's from Scheme48. Good times, right??
When seeing if SpiderMonkey could use this same strategy, I couldn't find how to make a variable-sized GC-managed allocation. It turns out that in SpiderMonkey you can't do that! SM's memory management system wants to work in terms of fixed-sized "cells". Even for objects that store properties inline in named slots, that's implemented in terms of standard cell sizes. So if an object has 6 slots, it might be implemented as instances of cells that hold 8 slots.
Truly variable-sized allocations seem to be managed off-heap, via malloc or other allocators. I am not quite sure how this works for GC-traced allocations like arrays, but let's assume that somehow it does.
Anyway, the point of this blog post. I was looking to see which part of SpiderMonkey reserves space for type information. For example, almost all objects in V8 start with a "map" word. This is the object's "hidden class". To know what kind of object you've got, you look at the map word. That word points to information corresponding to a class of objects; it's not available to store information that might vary between objects of that same class.
Interestingly, SpiderMonkey doesn't have a map word! Or at least, it doesn't have them on all allocations. Concretely, BigInt values don't need to reserve space for a map word. I can start storing data right from the beginning of the object.
But how can this work, you ask? How does the engine know what the type of some arbitrary object is?
The answer has a few interesting wrinkles. Firstly I should say that for objects that need hidden classes -- e.g. generic JavaScript objects -- there is indeed a map word. SpiderMonkey calls it a "Shape" instead of a "map" or a "hidden class" or a "structure" (as in JSC), but it's there, for that subset of objects.
But not all heap objects need to have these words. Strings, for example, are values rather than objects, and in SpiderMonkey they just have a small type code rather than a map word. But you know it's a string rather than something else in two ways: one, for "newborn" objects (those in the nursery), the GC reserves a bit to indicate whether the object is a string or not. (Really: it's specific to strings.)
For objects promoted out to the heap ("tenured" objects), objects of similar kinds are allocated in the same memory region (in kind-specific "arenas"). There are about a dozen trace kinds, corresponding to arena kinds. To get the kind of object, you find its arena by rounding the object's address down to the arena size, then look at the arena to see what kind of objects it has.
There's another cell bit reserved to indicate that an object has been moved, and that the rest of the bits have been overwritten with a forwarding pointer. These two reserved bits mostly don't conflict with any use a derived class might want to make from the first word of an object; if the derived class uses the first word for integer data, it's easy to just reserve the bits. If the first word is a pointer, then it's probably always aligned to a 4- or 8-byte boundary, so the low bits are zero anyway.
The upshot is that while we won't be able to allocate digits inline to BigInt objects in SpiderMonkey in the general case, we won't have a per-object map word overhead; and we can optimize the common case of digits requiring only a word or two of storage to have the digit pointer point to inline storage. GC is about compromise, and it seems this can be a good one.
Well, that's all I wanted to say. Looking forward to getting BigInt turned on upstream in Firefox!
One year more
and a new edition of the Web Engines Hackfest
was arranged by Igalia.
This time it was the tenth edition,
the first five ones using the WebKitGTK+ Hackfest name
and another five editions with the new broader name Web Engines Hackfest.
A group of igalians, including myself, have been organizing this event.
It has been some busy days for us, but we hope everyone enjoyed it
and had a great time during the hackfest.
This was the biggest edition ever, we were 70 people
from 15 different companies including Apple, Google and Mozilla
(three of the main browser vendors).
It seems the hackfest is getting more popular,
several people attending are repeating in the next editions,
so that shows they enjoy it.
This is really awesome and we’re thrilled about the future of this event.
Talks
The presentations are not the main part of the event,
but I think it’s worth to do a quick recap about the ones we had this year:
Behdad Esfahbod and Dominik Röttsches from Google
talked about Variable Fonts
and the implementation in Chromium.
It’s always amazing to check the possibilities of this new technology.
Camille Lamy, Colin Blundell and Robert Kroeger from Google
presented the Servicification effort
in the Chromium project.
Which is trying to modularize Chromium in smaller parts.
Žan Doberšek from Igalia gave an update on WPE WebKit.
The port is now official
and it’s used everyday in more and more low-end devices.
Thibault Saunier from Igalia complemented Žan’s presentation
talking about the GStreamer based WebRTC
implementation in WebKitGTK+ and WPE ports.
Really cool to see WebRTC arriving to more browsers and web engines.
Antonio Gomes and Jeongeun Kim from Igalia
explained the status of Chromium on Wayland
and it’s way to become fully supported upstream.
This work will help to use Chromium on embedded systems.
Youenn Fablet from Apple closed the event
talking about Service Workers
support on WebKit.
This is a key technology for Progressive Web Apps (PWA)
and is now available in all major browsers.
The slides of the talks are available on the website
and wiki.
The videos will be published soon in our YouTube channel.
Some pictures from Web Engines Hackfest 2018 (Flickr album)
Other topics
During the event there were breakout sessions about many different topics.
In this section I’m going to talk about the ones I’m more interested on.
Web Platform Tests (WPT)
This is a key topic to improve interoperability on the web platform.
Simon Pieters started the session with an introduction to WPT
just in case someone was not aware of the repository and how it works.
For the rest of the session we discussed
the status of WPT on the different browsers.
Chromium and Firefox are doing
an automatic two ways (import/export) synchronization process
so the tests can be easily shared between both implementations.
On the other side WebKit still has some kind of manual process over the table,
neither import or export is totally automatic,
there are some scripts that help with the process though.
Apart from that, WPT is a first-class citizen in Chromium,
and the encouraged way to do new developments.
In Firefox it’s still not there,
as the test suites are not run in all the possible configurations yet
(but they’re getting there).
Christian Biesinger gave an introduction to LayoutNG project
in Blink, where Google is rewriting Chromium’s layout engine.
He showed the main ideas and concepts behind this effort
and navigated the code showing some examples.
According to Christian things are getting ready
and LayoutNG could be shipping in the coming months
for inline and block layout.
On top of questions about LayoutNG, we briefly mentioned
how other browsers are also trying to improve the layout code:
Firefox with Servo layout
and WebKit with Layout Formatting Context (LFC) aka Layout Reloaded.
It seems quite clear that the current layout engines
are getting to their limits and people are looking for new solutions.
Chromium downstream
Several companies (Google included) have to maintain downstream forks
Chromium with their own customizations to fit their particular use cases
and hardware platforms.
Colin Blundell was explaining how it was the process of maintaining
the downstream version of Chrome for iOS.
After trying many different strategies
the best solution was rebasing their changes 2-3 times per day.
That way the conflicts they had to deal with were much simpler to resolve,
otherwise it was not possible for them to cope with all the upstream changes.
Note that he mentioned that one (rotatory) full-time resource
was required to perform this job in time.
It was good to share the experiences of different companies
that are facing very similar issues for this kind of work.
Thank you very much
Just to close this post, big thanks to all the people attending the event,
without you the hackfest wouldn’t have any sense at all.
People are key for this event where discussions and conversations
are one of the main parts of it.
Of course special acknowledgments to the speakers
for the hard work they put on their lovely talks.
Finally I couldn’t forget to thank the Web Engines Hackfest 2018 sponsors:
Google and Igalia.
Without their support this event won’t be possible.
Web Engines Hackfest 2018 sponsors: Google and Igalia
I am back home after my last trip for this year and a long and great week in A Coruña where I attended my first XDC, organized and sponsored by Igalia. It was a funny but also tiring week, especially for my colleagues of the organisation team (Sam, Chema, Juan and Antia) that were working … Continue reading XDC2018
This year, X.org Developers’ Conference (XDC 2018) happened in the Computer Science Faculty of University of Coruña, in the city of A Coruña, Spain during the last week of September, from Wednesday 26th to Friday 28th. XDC 2018 was a 3-day conference full of talks about all the technologies about free software graphics stack covering different topics like graphics driver development, testing and benchmarking, DRM, X, virtualization… Check the schedule for more info, slide decks and, once we have them edited, videos. For those loving statistics, we had 18 main track talks, 4 Workshops, 17 lightning talks, 3 social events, hallway tracks… Amazing!
This year we had 110 registered attendees and +40 students from the University of A Coruña attending it. Taking into account that some people couldn’t come at the very last minute, we estimate that we had ~140 attendees in total, which is probably the most successful XDC conference ever in terms of attendance. Honestly, we were a bit worried as coming to A Coruña is not so easy as other international hubs, so thanks everybody for coming to A Coruña!
This year the conference was organized by GPUL (Galician Linux User and Developer Group founded in 1998) together with University of A Coruña, Igalia and, of course, X.Org Foundation. The organization team was composed by 12 volunteers, some from Igalia and the rest from GPUL, who were taking care that everything went fine and fixed all the late minute issues that happened. I hope GPUL can keep organizing events for another 20 years! :-D
However, we are not perfect. Feel free to send us your feedback to both xdc2018@gpul.org and board@foundation.x.org, we would like to improve the organization of both next XDC conferences and our own local conferences.
Thanks to Igalia for allowing me organizing this event, for their platinum sponsorship and for sponsor Tuesday and Wednesday events.
Starting today, the comments section is not available any more in the articles
at the perezdecastro.org website. The main reason
for for taking this decision is that I have never felt particularly comfortable
leaving that to a third party service. I have had in mind to provide a
self-hosted solution, which never materialized; and given the
little amount of comments posted in the last years, it seemed to me that
removing the comments section altogether would be the best course of action.
This means that now there is absolutely nothing in the website which could
potentially track visitors. This is a small contribution against surveillance
capitalism
that I have been wanting to do for a while.
The only information stored are the usual HTTP server
logs; which contain the IP
addresses of visitors and the URLs they fetch from the server. Because I do
my own hosting I know for a fact that this data is not used for anything
other than troubleshooting, and only a few months of it are kept. Also,
IP addresses are anyway “public” information in the sense that any server
we connect to gets to know it, and that is part of how the Internet works.
Thanks for reading, and enjoy the ad- and tracker-free experience!
XDC 2018 is going to happen in the Computer Science Faculty of University of Coruña, in the city of A Coruña, Spain during the last week of September, from Wednesday 26th to Friday 28th. This is a 3-day conference full of talks about all the technologies about free software graphics stack covering different topics like graphics driver development, testing and benchmarking, DRM, X, virtualization… Check the schedule for more info.
As member of the organization team, I’m proud to announce that there are several surprises for the attendees :-)
Tuesday: welcome party, sponsored by Igalia. Traditionally, XDC organizes a networking event the day before the conference, in order to meet each other and share some drinks and food together. This year, the welcome party is going to happen at the Breen’s Tavern, Praza Maria Pita, 24, A Coruña, from 19:00 to 23:00. There will be beverages and food for free for registered attendees thanks to Igalia :-) There will be a registration desk there too.
Wednesday: guided tour and stand-up dinner, sponsored by Igalia. This is the first time XDC is happening in A Coruña, which is where Igalia’s headquarters are located. Igalia wants to show the city to the attendees so they can discover the hidden gems it has! There will be a bus waiting in front of the venue and will take us to the city center, where the 2-hour guided tour (in English) will start. After that, we will head to Rectorado UDC where the stand-up dinner is going to happen. The stand-up dinner is a kind of dinner where there are no chairs but only tables with food on them, facilitating chit-chatting among the attendees :-) Again, this two events are for free for registered attendees thanks to Igalia.
Saturday: sightseeing activity in Santiago de Compostela. Another XDC tradition is sightseeing activity for the attendees that are staying on Saturday. We are going to visit the city of Santiago de Compostela, Do you want to know more about the Way of Saint James? Walk through the old town which was designated as UNESCO World Heritage Site? We will meet in A Coruña’s train stationat 9:30, to take a train to Santiago. There, we will do a guided tour visiting the relevant parts of the old town, have lunch there and spend free time together during the afternoon until we come back by train around 18:00. The organization asks for 20 EUR to buy the train tickets and to book the guided tour, but the lunch is not included. If you want to attend it, just notify it in the registration desk at the venue.
Conference days: lunches are sponsorized by X.Org Foundation. This year, the X.Org Foundation is sponsoring the lunches for all the registered attendees in order to facilitate networking, discussions and spend more time together talking about free software instead of going far away for having lunch. We booked space in the nearest canteen of the campus (see itinerary) where the daily menu will be for free for registered attendees. Thanks X.Org Foundation!
The organization team wants also to help attendees. If you have any food allergy, dietary restriction or something else that we should know in advance, please contact us as soon as possible, either by xdc2018@gpul.org or at the registration desk.
Alarm management is a fundamental part of network monitoring. The motivation for defining a standard alarm interface for network devices isn’t new. In the early 90s, ITU-T standardized X.733 (OSI model). This continued in mobile networks with the standardization of Alarm IRP (Integration Reference Point) by 3GPP. In TCP/IP networks, SNMP is the preferred choice for network management, along with ad hoc tools (usually command-line scripts). In SNMP, object information is stored as MIBs (Management Information Base), formal descriptions of the network objects that can be managed. Usually MIBs have a tree structure.
The IETF didn’t early on standardize an alarm MIB. Instead, management systems interpreted the enterprise specific traps per MIB to build an alarm list. When finally RFC 3877 (Alarm Management Information Base MIB) was published, it had to address the existence of these enterprise traps and map them into alarms. This requirement led to a MIB that was not easy to use.
Introducing NETCONF and YANG
SNMP is still the dominant protocol for network management, although it has start showing its age. In the last years, several alternatives were proposed with the goal of replacing it. Among all proposals, the most promising alternative is NETCONF (RFC 6241: Network Configuration Protocol). NETCONF is, like SNMP, a network management protocol. It provides mechanisms to install, manipulate, and delete the configuration of network devices. NETCONF uses an RPC mechanism to execute its operations, whereas protocol messages are encoded in XML (or JSON).
The NETMOD WG (NETCONF Data Modeling Working Group) defines the semantics of operational data, configuration data, notifications and operations, using a data modeling language called YANG (See RFC 6020 and RFC 6021).
YANG is a very rich language. It allows to define much more complex data structures than other modeling languages such DTD or XML-Schema. For instance, YANG features a wide range of primitive data types (uint32, string, boolean, decimal64, etc), simple data (leaf), structured data elements (container, list, list-leaf), definition of customized types (typedef), definition of remote procedure calls, references (instance-ref, leaf-ref), notifications, etc.
Take the following model as example:
container students {
list student {
leaf name {
type string;
}
leaf data-birth {
type yang:date;
}
}
}
students {
student { name "Jane"; date-of-birth "01-01-1995"; }
student { name "John"; date-of-birth "31-03-1995"; }
}
That very same model could be written in DTD/XML form as:
<?xml version="1.0"?><!DOCTYPE note [<!ELEMENT students (student*)><!ELEMENT student (name,date-of-birth)><!ELEMENT name (#PCDATA)><!ELEMENT date-of-birth (#PCDATA)>
]>
<students><student><name>Jane</name><date_of_birth>01-01-1995</date_of_birth></student><student><name>John</name><date_of_birth>31-01-1995</date_of_birth></student></students>
One obvious difference is that field date-of-birth is encoded as a string in the DTD/XML model. On the contrary, it’s defined as a date in the YANG model. Supporting date as a native data type in the language improves value checking. If date-of-birth is not a valid date, our YANG library will report the error.
YANG also allows to compose several YANG modules into one single document. Data types from a different module can be accessed via namespace, as in the example above in the case of yang:date.
Covering all its aspects of YANG would require a blog post on its own, so I will end here this introduction. Summarizing, the two main ideas I’d like to highlight are the following:
NETCONF is a relatively new network management protocol, aimed to replace SNMP, tightly coupled with YANG.
YANG is the data modeling language used to define NETCONF’s data models.
YANG alarms
The YANG alarms module is defined in draft-vallin-netmod-alarm-module-02.txt. Its implementation in Snabb was sponsored, as most lwAFTR related work, by Deutsche-Telekom. The module specification is still a draft but even in this state it features enough functionality to make an implementation valuable.
The implementation uses Snabb’s native YANG library and Snabb’s config tool, a simple implementation of NETCONF. Both tools were mostly developed by Igalia (more precisely by my colleagues Andy Wingo and Jessica Tallon), also as part of the work of Snabb’s lwAFTR.
At a high level view, the YANG alarms module is organized in two parts:
Configuration data: stores all the attributes and variables that control how the module should operate. For example, max-alarm-status-changes controls the size of an alarm status-change list (default: 32); notify-status-changes, controls whether notifications are sent on alarms status updates.
State data: actually stores alarm information and consists of 4 containers: alarm-list, alarm-inventory, shelved-alarms and summary.
The main component of the state data container is the alarm-list container:
list alarm {
key "resource alarm-type-id alarm-type-qualifier";
uses common-alarm-parameters;
}
grouping common-alarm-parameters {
leaf resource {
type resource;
mandatory true;
}
leaf alarm-type-id {
type alarm-type-id;
mandatory true;
}
leaf alarm-type-qualifier {
type alarm-type-qualifier;
}
}
The alarm-list container stores all the active alarms managed in the system. But before going any further, we should define what an alarm is. Basically, an alarm is a persistent indication of a fault that clears only when its triggering condition has been resolved. An active alarm is always in at least these two states: raised or cleared.
When an alarm is raised a new entry is created in alarm-list. An alarm is identified by the triple: {resource, alarm-type-id, alarm-type-qualifier}, describing the resource that is affected, a type of alarm identifier and a qualifier that contains other optional information. Besides this information, an alarm also stores other information (omitted in the example for simplification) such as whether the alarm is-cleared, its last-changed timestamp, perceived-severity and a list of status changes. When an alarm is created, a new item is created in this list. If later the alarm increases or decreases its priority, or changes some other properties as per defined in the standard, a new status change is added to this list.
Most of the YANG Alarms module business logic is implemented in lib/yang/alarms.lua. This library provides an API that allows to define alarms and handle when to raise them or clear them. If we would like to monitor a special condition we just simply need to import the alarms module and create a check point. For instance:
When the condition is not met (self.next_arp wasn’t solved yet and self.next_arp_request_time has expired), an alarm is raised. But what if this check point is executed repeatedly, for instance every second until an operator fixes the alarm condition? To avoid saturating the alarm list, the standard specifies an elapse time of 2 minutes before the same alarm is raised again. This elapse is managed by the alarms library.
Besides a list of alarms, the module also defines these other containers:
alarm-inventory: It contains all possible alarm types for the system.
summary: Summary of numbers of alarms and shelved alarms.
shelved-alarms: A shelved alarm is ignored and won’t emit raise or clear events. Shelved alarms don’t emit notifications either. Shelving an alarm is a convenient way to silent an alarm.
When an alarm is raised, cleared or changes its status, a notification is sent. The alarms module specifies three types of notifications:
alarm-notification: Used to report a state change for an alarm. This alarm is emitted when an alarm is raised, clear or its status change.
alarm-inventory-changed: Used to report that the list of possible alarms has changed.
operator-action: Used to report that an operator acted upon an alarm.
Continuing with the ARP alarm example, here’s how a notification looks like when such alarm raises:
$ sudo ./snabb alarms listen lwaftr
{"event":"alarm-notification",
"resource":"16446", "alarm_type_id":"arp-resolution", "alarm_type_qualifier":"",
"perceived_severity":"critical", "alarm_text":"Make sure you can resolve..."}
Upon receiving a notification, an operator, or an external program, can act on the affected resource signaled by the alarm and fix the condition that triggered it. For instance, in the case of the lwAFTR being unable to resolve the next hop IPv4 address, such alarm indicates the host isn’t reachable (the host is down, or there’s no route to that address).
Lastly, the module also specifies one YANG action and two YANG RPCs:
set-operator-state: Allows an operator to change the state of an alarm. The specification defines 4 possible operator states: cleared-not-closed, cleared-closed, not-cleared-closed, not-cleared-closed, not-cleared-not-closed.
purge-alarms: Deletes entries from the alarm list according to the supplied criteria. It can be used to delete alarms that are in closed state or an older than a specified time.
compress-alarms: Compress entries in the alarm list by removing all but the latest state change for all alarms.
NETCONF side
Adding alarms support to Snabb, and more precisely to the lwAFTR, has brought in many good things. First of all, Snabb’s YANG library has added support for more data types such as empty, identityref and leafref. It has also improved parsing and validation of other data types such as ipv4-prefix, ipv6-prefix and enum, in addition to other minor improvements and bug fixes. For the moment, the lwAFTR is the poster child for alarms, but the mechanism is generic enough and it can be used by other data-planes.
A new program has been added to Snabb, not surprisingly being called alarms. It consists of five sub-commands:
listen: Listens to a Snabb instance which provides alarms support. The subprogram can send RPC requests calls to the server program or listen to notifications.
get-state: Sends an XPath request to a target Snabb instance that provides alarms state information.
set-operator-state: User interface to set-operator-state action.
purge-alarms: User interface to purge-alarms.
compress-alarms: User interface to compress-alarms.
Below there’s an excerpt of get-state subprogram and its output:
$sudo./snabbalarmsget-statelwaftr/alarm-list{alarm{alarm-type-idarp-resolution;alarm-type-qualifier'';resource21385;alarm-text"Make sure you can resolve external-interface.next-hop.ip address manually.""If it cannot be resolved, consider setting the MAC address of the next-hop directly.""To do it so, set external-interface.next-hop.mac to the value of the MAC address.";is-clearedfalse;last-changed2018-06-18T14:57:40Z;perceived-severitycritical;status-change{time2018-06-18T14:57:40Z;alarm-text"Make sure you can resolve external-interface.next-hop.ip address manually.""If it cannot be resolved, consider setting the MAC address of the next-hop directly.""To do it so, set external-interface.next-hop.mac to the value of the MAC address.";perceived-severitycritical;}time-created2018-06-18T14:57:40Z;}last-changed2018-06-18T14:57:40Z;number-of-alarms1;}
The alarms module keeps all its state into one Snabb instance, the leader process. As a reminder, since v3.0 the lwAFTR runs in a multiprocess architecture which consists of:
1 Leader, which manages changes in lwAFTR configuration file. For instance, changes in softwires (add, remove, update).
1 or N Workers, which runs a lwAFTR data-plane.
Both processes communicate via an IPC (Inter-process communication) mechanism, in this case a message channel implemented using sockets. When a worker raises an alarm, a message is sent to the leader via a worker. The leader polls the alarms-channel periodically, consuming all the stored messages. The result of processing a message is an action that alters the alarms state, for instance, adding a new alarm to the inventory, raising an alarm, clearing it, etc. All this logic is coded in lib/ptree/ptree.lua and lib/ptree/alarm_coded.lua.
Besides alarms, there are also notifications. A notification is a sort of simple message that is emitted under certain circumstances: when an alarm is raised, when its status change or when a new alarm-type is added to the inventory. Notifications are a native YANG element, not particular only to alarms.
In Snabb, the notifications mechanism is also implemented via sockets. In this case, a socket connects a lwAFTR leader to a series of peers that listen on the socket. When a notification is triggered, a new notification is added to the leader’s list of notifications. The leader process runs a fiber that constantly polls this list. If it finds new entries, the notifications got serialized to a JSON object and are sent through the socket. Once a notification is sent, it’s removed from the alarms state. This logic is implemented lib/ptree/ptree.lua and lib/yang/alarms.lua.
Summary and conclusions
YANG Alarms is a simple mechanism to notify erroneous conditions. The main strengths of this module are:
It’s encoded as a YANG module, with all the advantages which that represents (common vocabulary and semantics, reusable).
Signaling errors by simply printing out messages in stdout is not reliable, as they can be easily missed. Alarms are in-memory stored, they keep state which can be later consulted on demand.
Active notifications for the most important state changes. This allows to hook external programs, which do not need to constantly poll the artifact current state to check whether a change happened.
On the down side, I personally think that the amount of information tracked per alarm is excessive, making the YANG specification more complex than one may thought at first. Fortunately, programs interested in supporting this module do not need to implement all the features specified, being satisfied with just a subset of all the module’s features. At the moment of writing this, the YANG alarms proposal is still a draft but hopefully it will become an standard after several revisions.
About 2 weeks ago, I attended SIGGRAPH 2018. I am still very excited about the whole event, and I am very thankful that Igalia (the consultancy company I work for) and specifically the Graphics Team selected me to go, despite this being my first year at the company! 😀 SIGGRAPH is the biggest event for companies and individuals … Continue reading SIGGRAPH2018
This year, X.org Developers’ Conference (
