Planet Igalia

August 09, 2018

Manuel Rego

Changes on CSS Grid Layout in percentages and indefinite height

This is a blog post about a change of behavior on CSS Grid Layout related to percentage row tracks and gutters in grid containers with indefinite height. Igalia has just implemented the change in Chromium and WebKit, which can affect some websites out there. So here I am going to explain several things about how percentages work in CSS and all the issues around it, of course I will also explain the change we are doing in Grid Layout and how to keep your previous behavior in the new version with very simple changes.

Sorry for the length but I have been dealing with these issues since 2015 (probably earlier but that is the date of the first commit I found about this topic), and I went too deep explaining the concepts. Probably the post has some mistakes, this topic is not simple at all, but it represents a kind of brain dump of my knowledge about it.

Percentages and definite sizes

This is the easy part, if you have an element with fixed width and height resolving percentages on children dimensions is really simple, they are just computed against the width or height of the containing block.

A simple example:

<div style="width: 500px; height: 200px; border: dashed thick black;">
  <div style="width: 50%; height: 75%; background: magenta;"></div>
</div>

Example of percentage dimensions in a containing block with definite sizes Example of percentage dimensions in a containing block with definite sizes

Things are a bit trickier for percentage margins and paddings. In inline direction (width in horizontal writing mode) they work as expected and are resolved against the inline size. However in block direction (height) they are not resolved against the block size (as one can initially expect) but against the inline size (width) of the containing block.

Again a very simple example:

<div style="width: 500px; height: 200px; border: dashed thick black;">
  <div style="margin-left: 10%; margin-top: 10%;
              height: 150px; background: magenta;"></div>
</div>

Example of percentage margins in a containing block with definite sizes Example of percentage margins in a containing block with definite sizes

Note that there is something more here, in both Flexbox and Grid Layout specifications it was stated in the past that percentage margins and paddings resolve against their corresponding dimension, for example inline margins against inline axis and block margins against block axis.

This was implemented like that in Firefox and Edge, but Chromium and WebKit kept the usual behavior of resolving always against inline size. So for a while the spec had the possibility to resolve them in either way.

This was a source of interoperability issues between the different browsers but finally the CSS Working Group (CSSWG) resolved to keep the behavior for regular blocks also for flex and grid items. And both Firefox and Edge modified their behavior and all browsers have the same output nowadays.

Percentages and indefinite sizes

First question is, what is an indefinite size? The simple answer is that a definite size is a size that you can calculate without taking into account the contents of the element. An indefinite size is the opposite, in order to compute it you need to check the contents first.

But then, what happens when the containing block dimensions are indefinite? For example, a floated element has indefinite width (unless otherwise manually specified), a regular block has indefinite height by default (height: auto).

For heights this is very simple, percentages are directly ignored so they have no effect on the element, they are treated as auto.

For widths it starts to get funny. Web rendering engines have two phases to compute the width of an element. A first one to compute the minimum and maximum intrinsic width (basically the minimum and maximum width of its contents), and a second one to compute the final width for that box.

So let’s use an example to explain this properly. Before getting into that, let me tell you that I am going to use Ahem font in some examples, as it makes very easy to know the size of the text and resolve the percentages accordingly, so if we use font: 50px/1 Ahem; we know that the size of an X character is a square of 50x50 pixels.

<div style="float: left; font: 50px/1 Ahem;
            border: solid thick black; background: magenta; color: cyan;">
  XX XXXXX
</div>

Example of intrisic width without constraints Example of intrisic width without constraints

The browser first calculates the intrinsic width, as minimum it computes 250px (the size of the smallest word, XXXXX in this case), as maximum size it would be 400px (the size of the whole text without line breaking XX XXXXX). So after this phase the browser knows that the element should have a width between 250px and 400px.

Then during layout phase the browser will decide the final size, if there are no constraints imposed by the containing block it will use the maximum intrinsic width (400px in this case). But if you have a wrapper with a 300px width, the element will have to use 300px as width. If you have a wrapper smaller than the minimium intrinsic width, for example 100px, the element will still use the minimum 250px as its size. This is a quick and dirty explanation, but I hope it is useful to get the general idea.

Example of intrisic width width different constraints Example of intrisic width with different constraints

In order to resolve percentage widths (in the indefinite width situations) the browser does a different thing depending on the phase. During intrinsic size computations the percentage width is ignored (treated as auto like for the heights). But in the layout phase the width is resolved against the intrinsic size computed earlier.

Trying to summarize the above paragraphs, we can say that somehow the width is only indefinite while the browser is computing the intrinsic width of the element, afterwards during the actual layout the width is considered definite and percentages are resolved against it.

So now let’s see an example of indefinite dimensions and percentages:

<div style="float: left;
            border: solid thick black; background: magenta;">
  <div style="width: 50%; height: 50%; background: cyan;">Hello world!</div>
</div>

Example of percentage dimensions in a containing block with indefinite sizes Example of percentage dimensions in a containing block with indefinite sizes

First the size of the magenta box is calculated based on its contents, as it has not any constraint it uses the maximum intrinsic width (the length of Hello world!). Then as you can see the width of the cyan box is 50% of the text length, but the height is the same than if we use height: auto (the default value), so the 50% height is ignored.

Back-compute percentages

For margins and paddings things work more or less the same, remember that all of them are resolved against the inline direction (so they are ignored during intrinsic size computation and resolved later during layout).

But there is something special about this too. Nowadays all the browsers have the same behavior but that was not always the case, not so long time ago (before Firefox 61 which was released past June) things worked different in Firefox than the rest of browsers

Again let’s go to an example:

<div style="float: left; font: 50px/1 Ahem;
            border: solid thick black; background: magenta;">
  <div style="margin-left: 50%; height: 100px;
              background: cyan; color: blue;">XXXXX</div>
</div>

Example of percentage margins in a containing block with indefinite sizes Example of percentage margins in a containing block with indefinite sizes

In this example the size of the magenta box (the floated div) is the width of the text, 250px in this case. Then the margin is 50% of that size (125px), making that the size of the cyan box gets reduced to 125px too, which causes overflow.

But for these cases (percentage width margins and paddings and indefinite width container) Firefox did something extra that was called back-compute percentages. For that it something similar to the following formula:

Intrinsic width / (1 - Sum of percentages)

Which for this case would be 250px / (1 - 0.50) = 500px. So it takes as intrinsic size of the magenta box 500px, and then it resolves the 50% margin against it (250px). Thanks to this there is no overflow, and the margin is 50% of the containing block size.

Example of old Firefox behavior back-computing percentage margins Example of old Firefox behavior back-computing percentage margins

This Firefox behavior seems really smart and avoid overflows, but the CSSWG discussed about it and decided to use the other behavior. The main reason is what happens when you are around 100% percentages, or if you go over that value. The size of the box starts to be quite big (with 90% margin it would be 2500px), and when you go to 100% or over it you cannot use that formula so it considers the size as infinity (basically the viewport size in this example) and there is discontinuity in how percentages are resolved.

So after that resolution Firefox changed their implementation and removed the back-computing percentages logic, thus we have now interoperability in how percentage margins and paddings are resolved.

CSS Grid Layout and percentages

And now we arrive to CSS Grid Layout and how to resolve percentages in two places: grid tracks and grid gutters.

Of course when the grid container has definite dimensions there are no problems in resolving percentages against them, that is pretty simple.

As usual the problem starts with indefinite sizes. Originally this was not a controversial topic, percentages for tracks were behaving similar to percentage for dimensions in regular blocks. A percentage column was treated as auto for intrinsic size computation and later resolved against that size during layout. For percentage rows they were treated as auto. It does not mean that this is very easy to understand (actually it took me a while), but once you get it, it is fine and not hard to implement.

But when percentage support was added to grid gutters the big party started. Firefox was the first browser implementing them and they decided to use the back-compute technique explained in the previous point. Then when we add support in Chromium and WebKit we did something different than Firefox, we basically mimic the behavior of percentage tracks. As browsers started to diverge different discussions appear.

One of the first agreements on the topic was that both percentage tracks and gutters should behave the same. That invalidated the back-computing approach, as it was not going to work fine for percentage tracks as they have contents. In addition it was finally discarded even for regular blocks, as commented earlier, so this was out of the discussion.

However the debate moved to how percentage row tracks and gutters should be resolved, if similar to what we do for regular blocks or if similar to what we do for columns. The CSSWG decided they would like to keep CSS Grid Layout as symmetric as possible, so making row percentages resolve against the intrinsic height would achieve that goal

So finally the CSSWG resolved to modify how percentage row tracks and gutters are resolved for grid containers with indefinite height. The two GitHub issues with the last discussions are: #509 and #1921.

Let’s finish this point with a pair of examples to understand the change better comparing the previous and new behavior.

Percentage tracks:

<div style="display: inline-grid; border: solid thick;
            grid-template-columns: 75%; grid-template-rows: 50%;">
  <div style="background: magenta;">Testing</div>
</div>

Example of percentage tracks in a grid container with indefinite sizes Example of percentage tracks in a grid container with indefinite sizes

Here the intrinsic size of the grid container is the width and height of the text Testing, and then the percentages tracks are resolved against that size for both columns and rows (before that was only done for columns).

Percentage gutters:

<div style="display: inline-grid; grid-gap: 10%; border: solid thick;
            grid-template-columns: 200px 200px; grid-template-rows: 100px 100px;">
  <div style="background: magenta;"></div>
  <div style="background: cyan;"></div>
  <div style="background: yellow;"></div>
  <div style="background: lime;"></div>
</div>

Example of percentage gutters in a grid container with indefinite sizes Example of percentage gutters in a grid container with indefinite sizes

In this example we can see the same thing, with the new behavior both the percentage column and row gaps are resolved against the intrinsic size.

Change behavior for indefinite height grid containers

For a while all browsers were behaving the same (after Firefox dropped the back-computing approach) so changing this behavior would imply some kind of risks, as some websites might be affected by that and get broken.

For that reason we added a use counter to track how many websites where hitting this situation, using percentage row tracks in a indefinite height grid container. The number is not very high, but there is an increasing trend as Grid Layout is being adopted (almost 1% of websites are using it today).

And then Firefox changed the behavior for percentage row gutters to follow the new text on the spec, so they are resolved against the intrinsic height (this happened in version 62). However it did not change the behavior for percentage row tracks yet.

This was a trigger to retake the topic and go deeper on it, after analyzing it carefully and crafting a prototype implementation we sent an intent to implement and ship to blink-dev mailing list.

The intent was approved, but we were requested to analyze the sites that were hitting the use counter. After checking 178 websites only 8 got broken due to this change, we contacted them to try to get them fixed explaining how to keep the previous behavior (more about this in next point). You can find more details about this research in this mail.

Apart from that we added a deprecation message in Chromium 69, so if you have a website that is affected by this (it does not mean that it has to get broken but that it uses percentage row tracks in a grid container with indefinite height) you will get the following warning in the JavaScript console:

[Deprecation] Percentages row tracks and gutters for indefinite height grid containers will be resolved against the intrinsic height instead of being treated as auto and zero respectively. This change will happen in M70, around October 2018. See https://www.chromestatus.com/feature/6708326821789696 for more details.

Finally this week the patch has been accepted and merged in master, so since Chromium 70.0.3516 (current Canary) you will have the new behavior. Apart from that we also make the fix in WebKit that will be hopefully part of the next Safari releases.

In addition Firefox and Edge developers have been notified and we have shared the tests in WPT as usual, so hopefully those implementations will get updated soon too.

Update your website

Yes this change might affect your website or not, even if you get the deprecation warning it can be the case that your website is still working perfectly fine, but in some cases it can break quite badly. The good news is that the solution is really straightforward.

If you find issues in your website and you want to keep the old behavior you just need to do the following for grid containers with indefinite height:

  • Change percentages in grid-template-rows or grid-auto-rows to auto.
  • Modify percentages in row-gap or grid-row-gap to 0.

With those changes your website will keep behaving like before. In most cases you will realize that the percentages were unneeded and were not doing anything useful for you, even you would be able to drop the declaration completely.

One of these cases would be websites that have grid containers with just one single row of 100% height (grid-template-rows: 100%), many of the sites hitting the use counter are like this. All these are not affected by this change, unless the have extra implicit rows, but the 100% is not really useful at all there, they can simply remove the declaration.

Another sites that have issues are the ones that have for example two rows that sum up 100% in total (grid-template-rows: 25% 75%). These percentages were ignored before, so the contents always fit in each of the rows. Now the contents might not fit in each row and the results might not be the desired ones. Example:

<div style="display: grid; grid-template-rows: 25% 75%; border: solid thick;">
  <div style="background: magenta;">First<br>two lines</div>
  <div style="background: cyan;">Second</div>
</div>

Example of overlapping rows in the new behavior Example of overlapping rows in the new behavior

The sites that were more broken usually have several rows and used percentages only for a few of them or for all. And now the rows overflow the height of the grid container and they overlap other content on the website. There were cases like this example:

<div style="display: grid; grid-template-rows: 50%; border: solid thick;">
  <div style="background: magenta;">First</div>
  <div style="background: cyan; height: 200px;">Second</div>
  <div style="background: yellow; height: 100px;">Third</div>
</div>

Example of overflowing rows in the new behavior Example of overflowing rows in the new behavior

Closing

This topic has been a kind of neverending story for the CSSWG, but finally it seems we are reaching to an end. Let’s hope this does not get any further and things get settle down after all this time. We hope that this change is the best solution for web authors and everyone will be happy with the final outcome.

As usual I could not forget to highlight that all this work has been done by Igalia thanks to Bloomberg sponsorship as part of our ongoing collaboration.

Igalia and Bloomberg working together to build a better web Igalia and Bloomberg working together to build a better web

Thanks for reading that long, this ended up being much more verbose and covering more topics than originally planned. But I hope it can be useful to understand the whole thing. You can find all the examples from this blog post in this pen feel free to play with them.

And to finish this blog post I could only do it by quoting fantasai:

this is why I hate percentages in CSS

I cannot agree more with her. 😇

August 09, 2018 10:00 PM

August 07, 2018

Manuel Rego

CSS Logical Properties and Values in Chromium and WebKit

Since the beginning of the web we have been used to deal with physical CSS properties for different features, for example we all know how to set a margin in an element using margin-left, margin-right, margin-top and/or margin-bottom. But with the appearance of CSS Writing Modes features, the concepts of left, right, top and bottom have somehow lost their meaning.

Imagine that you have some right-to-left (RTL) content on your website your left might be probably the physical right, so if you are usually setting margin-left: 100px for some elements, you might want to replace that with margin-right: 100px. But what happens if you have mixed content left-to-right (LTR) and RTL at the same time, then you will need different CSS properties to set left or right depending on that. Similar issues are present if you think about vertical writing modes, maybe left for that content is the physical top or bottom.

CSS Logical Properties and Values is a CSS specification that defines a set of logical (instead of physical) properties and values to prevent this kind of issues. So when you want to set that margin-left: 100px independently of the direction and writing mode of your content, you can directly use margin-inline-start: 100px that will be smart enough. Rachel Andrew has a nice blog post explaining deeply this specification and its relevance.

Example of 'margin-inline-start: 100px' in different combinations of directions and writing modes Example of margin-inline-start: 100px in different combinations of directions and writing modes

Oriol Brufau, an active collaborator on the CSS Working Group (CSSWG), has been doing a Igalia Coding Experience implementing support for CSS Logical Properties and Values in Blink and WebKit. Maybe you were already aware of this as my colleague Frédéric Wang already talked about it in his last blog post reviewing the activities of Igalia Web Platform team in the past semester.

Some history

Chromium and WebKit have had support since a long time ago for some of the CSS logical properties defined by the spec. But they were not using the standard names defined in the specification but some -webkit- prefixed ones with different names.

For setting the dimensions of an element Chromium and WebKit have properties like -webkit-logical-width and -webkit-logical-height. However CSS Logical defines inline-size and block-size instead. There are also the equivalent ones for minimum and maximum sizes too. These ones have been already unprefixed at the beginning of 2017 and included in Chromium since version 57 (March 2017). In WebKit they are still only supported using the prefixed version.

But there are more similar properties for margins, paddings and borders in Chromium and WebKit that use start and end for inline direction and before and after for block direction. In CSS Logical we have inline-start and inline-end for inline direction and block-start and block-end for block direction, which are much less confusing. There was an attempt in the past to unprefix these properties but the work was abandoned and never completed. These ones were still using the -webkit- prefix so we decided to tackle them as the first task.

The post has been only talking about properties so far, but the same thing applies to some CSS values, that is why the spec is called CSS Logical Properties and Values. For example a very well-known property like float has the physical values left and right. The spec defines inline-start and inline-end as the logical values for float. However these were not supported yet in Chromium and WebKit, not even using -webkit- prefixes.

Firefox used to have some -moz- prefixed properties, but since Firefox 41 (September 2015) it is shipping many of the standard logical properties and values. Firefox has been using these properties extensively in its own tests, thus having them supported in Chromium will make easier to share them.

At the beginning of this work, Oriol wrote a document in which explaining the implementation plan where you can check the status of all these properties in Chromium and Firefox.

Unprefix existent properties

We originally send an intent to implement and ship for the whole spec, actually not all the spec but the parts that the CSSWG considered ready to implement. But Chromium community decided it was better to split it in two parts:

The work on the first part, making the old -webkit- prefixed properties to use the new standard names, has been already completed by Oriol and it is going to be included in the upcoming release of Chromium 69.

In addition to the Chromium work Oriol has just started to do this on WebKit too. Work is on early stages here but hopefully things will move forward in parallel to the Chromium stuff.

Adding support for the rest

Next step was to add support for the new stuff behind an experimental flag. This work is ongoing and you can check the current status in the latest Canary enabling the Experimental Web Platform features flag.

So far Oriol has added support for a bunch of shorthands and the flow-relative offset properties. You can follow the work in issue #850004 in Chromium bug tracker.

We will talk more about this in a future blog post once this task is completed and the new logical properties and values are shipped.

Tests!

Of course testing is a key part of all these tasks, and web-platform-tests (WPT) repository plays a fundamental role to ensure interoperability between the different implementations. Like we have been doing in Igalia lately in all our developments we used WPT as the primary place to store all the tests related to this work.

Oriol has been creating tests in WPT to cover all these features. Initial tests were based in the ones already available in Firefox and modified them to adapt to the rest of stuff that needs to be checked.

Note that in Chromium all the sideways writing modes test cases are failing as there is no support for sideways in Chromium yet.

Plans for the future

As explained before, this is an ongoing task but we already have some extra plans for it. These are some of the tasks (in no particular order) that we would like to do in the coming months:

  • Complete the implementation of CSS Logical Properties and Values in Chromium. This was explained in the previous point and is moving forward at a good pace.
  • Get rid of usage of -webkit- prefixed properties in Chromium source code. Oriol has also started this task and is currently work in progress.
  • Deprecate and remove the -webkit- prefixed properties. It still too early for that but we will keep an eye on the metrics and do it once usage has decreased.
  • Implement it in WebKit too, first by unprefixing the current properties (which has been already started) and later continuing with the new things. It would be really nice if WebKit follows Chromium on this. Edge also has plans to add support for this spec, so that would make logical properties and values available in all the major browsers.

Wrap up

Oriol has been doing a good job here as part of his Igalia Coding Experience. Apart from all the new stuff that is landing in Chromium, he has also been fixing some related bugs.

We have just started the WebKit tasks, but we hope all this work can be part of future Chromium and Safari releases in the short term.

And that is all for now, we will keep you posted! 😉

August 07, 2018 10:00 PM

July 31, 2018

Thibault Saunier

WebKitGTK and WPE gain WebRTC support back!

WebRTC is a w3c draft protocol that “enables rich, high-quality RTP applications to be developed for the browser, mobile platforms, and IoT devices, and allow them all to communicate via a common set of protocols”. The protocol is mainly used to provide video conferencing systems from within web browsers.

https://appr.tc running in WebKitGTK
https://appr.tc running in WebKitGTK

A brief history

At the very beginning of the WebRTC protocol, before 2013, Google was still using WebKit in chrome and they started to implement support using LibWebRTC but when they started the blink fork the implementation stopped in WebKit.

Around 2015/2016 Ericsson and Igalia (later sponsored by Metrological) implemented WebRTC support into WebKit, but instead of using LibWebRTC from google, OpenWebRTC was used. This had the advantage of being implemented on top of the GStreamer framework which happens to be used for the Multimedia processing inside WebKitGTK and WebKitWPE. At that point in time, the standardization of the WebRTC protocol was still moving fast, mostly pushed by Google itself, and it was hard to be interoperable with the rest of the world. Despite of that, the WebKit/GTK/WPE WebRTC implementation started to be usable with website like appr.tc at the end of 2016.

Meanwhile, in late 2016, Apple decided to implement WebRTC support on top of google LibWebRTC in their ports of WebKit which led to WebRTC support in WebKit announcement in June 2017.

Later in 2017 the OpenWebRTC project lost momentum and as it was considered unmaintained, we, at Igalia, decided to use LibWebRTC for WebKitGTK and WebKitWPE too. At that point, the OpenWebRTC backend was completely removed.

GStreamer/LibWebRTC implementation

Given that Apple had implemented a LibWebRTC based backend in WebKit, and because this library is being used by the two main web browsers (Chrome and Firefox), we decided to reuse Apple’s work to implement support in our ports based on LibWebRTC at the end of 2017. A that point, the two main APIs required to allow video conferencing with WebRTC needed to be implemented:

  • MediaDevices.GetUserMedia and MediaStream: Allows to retrieve Audio and Video streams from the user Cameras and Microphones (potentially more than that but those are the main use cases we cared about).
  • RTCPeerConnection: Represents a WebRTC connection between the local computer and a remote peer.

As WeKit/GTK/WPE heavily relies on GStreamer for the multimedia processing, and given its flexibility, we made sure that our implementation of those APIs leverage the power of the framework and the existing integration of GStreamer in our WebKit ports.

Note that the whole implementation is reusing (after refactoring) big parts of the infrastructure introduced during the previously described history of WebRTC support in WebKit.

GetUserMedia/MediaStream

To implement that part of the API the following main components were developed:

  • RealtimeMediaSourceCenterLibWebRTC: Main entry point for our GStreamer based LibWebRTC backend.
  • GStreamerCaptureDeviceManager: A class to list and manage local Video/Audio devices using the GstDeviceMonitor API.
  • GStreamerCaptureDevice: Implementation of WebKit abstraction for capture devices, basically wrapping GstDevices.
  • GStreamerMediaStreamSource: A GStreamer Source element which wraps WebKit abstraction of MediaStreams to be used directly in a playbin3 pipeline (through a custom mediastream:// protocol). This implementation leverages latest GstStream APIs so it is already one foot into the future.

The main commit can be found here

RTCPeerConnection

Enabling the PeerConnection API meant bridging previously implemented APIs and the LibWebRTC backend developed by Apple:

  • RealtimeOutgoing/Video/Audio/SourceLibWebRTC: Passing local stream (basically from microphone or camera) to LibWebRTC to be sent to the peer.
  • RealtimeIncoming/Video/Audio/SourceLibWebRTC: Passing remote stream (from a remote peer) to the MediaStream object and in turn to the GStreamerMediaStreamSource element.

On top of that and to leverage GStreamer Memory management and negotiation capabilities we implemented encoders and decoder for LibWebRTC (namely GStreamerVideoEncoder and GStreamerVideoDecoders). This brings us a huge number of Hardware accelerated encoders and decoders implementations, especially on embedded devices, which is a big advantage in particular for WPE which is tuned for those platforms.

The main commit can be found here

WebKitWebRTC dataflow diagram

Conclusion

While we were able to make GStreamer and LibWebRTC work well together in that implementation, using the new GstWebRTC component (that is now in upstream GStreamer) as a WebRTC backend would be cleaner. Many pieces of the current implementation could be reused and it would allow us to have a simpler infrastructure and avoid having several RTP stack in the WebKitGTK and WebKitWPE ports.

Most of the required APIs and features have been implemented, but a few are still under development (namely MediaDevices.enumerateDevices, canvas captureStream and WebAudio and MediaStream bridging) meaning that many Web applications using WebRTC already work, but some don’t yet, we are working on those!

A big thanks to my employer Igalia and Metrological for sponsoring our work on that!

by thiblahute at July 31, 2018 06:06 PM

July 21, 2018

Michael Catanzaro

On Flatpak Nightlies

Here’s a little timeline of some fun we had with the GNOME master Flatpak runtime last week:

  • Tuesday, July 10: a bad runtime build is published.  Trying to start any application results in error while loading shared libraries: libdw.so.1: cannot open shared object file: No such file or directory. Problem is the library is present in org.gnome.Sdk instead of org.gnome.Platform, where it is required.
  • Thursday, July 12:  the bug is reported on WebKit Bugzilla (since it broke Epiphany Technology Preview)
  • Saturday, July 14: having returned from GUADEC, I notice the bug report and bisect the issue to a particular runtime build. Mathieu Bridon fixes the issue in the freedesktop SDK and opens a merge request.
  • Monday, July 16: Mathieu’s fix is committed. We now have to wait until Tuesday for the next build.
  • Tuesday, Wednesday, and Thursday: we deal with various runtime build failures. Each day, we get a new build log and try to fix whatever build failure is reported. Then, we wait until the next day and see what the next failure is. (I’m not aware of any way to build the runtime locally. No doubt it’s possible somehow, but there are no instructions for doing so.)
  • Friday, July 20: we wait. The build has succeeded and the log indicates the build has been published, but it’s not yet available via flatpak update
  • Saturday, July 21: the successful build is now available. The problem is fixed.

As far as I know, it was not possible to run any nightly applications during this two week period, except developer applications like Builder that depend on org.gnome.Sdk instead of the normal org.gnome.Platform. If you used Epiphany Technology Preview and wanted a functioning web browser, you had to run arcane commands to revert to the last good runtime version.

This multi-week response time is fairly typical for us. We need to improve our workflow somehow. It would be nice to be able to immediately revert to the last good build once a problem has been identified, for instance.

Meanwhile, even when the runtime is working fine, some apps have been broken for months without anyone noticing or caring. Perhaps it’s time for a rethink on how we handle nightly apps. It seems likely that only a few apps, like Builder and Epiphany, are actually being regularly used. The release team has some hazy future plans to take over responsibility for the nightly apps (but we have to take over the runtimes first, since those are more important), and we’ll need to somehow avoid these issues when we do so. Having some form of notifications for failed builds would be a good first step.

P.S. To avoid any possible misunderstandings: the client-side Flatpak technology itself is very good. It’s only the server-side infrastructure that is problematic here. Clearly we have a lot to fix, but it won’t require any changes in Flatpak.

by Michael Catanzaro at July 21, 2018 02:12 PM

July 10, 2018

Pablo Saavedra

http503

Many times, during these last months, I thought to keep updated my blog writing a new post. Unfortunately, for one or another reason I always found an excuse to not do so. Well, I think that time is over because finally I found something useful and worthy the time spent time on the writing.

– That is OK but … what are you talking about?.
– Be patient Pablo, if you didn’t skip the headline of the post you already know about what I’m talking, probably :-).

Yes, I’m talking about how to setup a MacPro computer into a icecc cluster based on Linux hosts to take advantage of those to get more CPU power to build heavy software projects, like Chromium,  faster. The idea besides this is to distribute all the computational work over Linux nodes (fairly cheaper than any Mac) requested for cross-compiling tasks from the Mac host.

I’ve been working as a sysadmin at Igalia for the last couple of years. One of my duties here is to support and improve the developers building infrastructures. Recently we’ve faced long building times for heavy software projects like, for instance, Chromium. In this context, one of the main  issues that I had to solve is  how to build Chromium for MacOS in a reasonable time and avoiding to spend a lot of money in expensive bleeding edge Apple’s hardware to get CPU power.

This is what this post is about. This is an explanation about how to configure a Mac Pro to use a Linux based icecc cluster to boost the building times using cross-compilation. For simplicity, the explanation is focused in the singular case of just one single Linux host as icecc node and just one MacOS host requesting for compiling tasks but, in any case, you can extrapolate the instructions provided here to have many nodes as you need.

So let’s go with the the explanation but, first of all, a summary for those who want to go directly to the minimal and essential information …

TL;DR

On the Linux host:

# Configure the iceccd
$ sudo apt install icecc
$ sudo systemctl enable icecc-scheduler
$ edit /etc/icecc/icecc.conf
ICECC_MAX_JOBS="32"
ICECC_ALLOW_REMOTE="yes"
ICECC_SCHEDULER_HOST="192.168.1.10"
$ sudo systemctl restart icecc

# Generate the clang cross-compiling toolchain
$ sudo apt install build-essential icecc
$ git clone https://chromium.googlesource.com/chromium/tools/depot_tools.git ~/depot_tools
$ export PATH=$PATH:~/depot_tools
$ git clone https://github.com/psaavedra/chromium_clang_darwin_on_linux ~/chromium_clang_darwin_on_linux
$ cd ~/chromium_clang_darwin_on_linux
$ export CLANG_REVISION=332838  # or CLANG_REVISION=$(./get-chromium-clang-revision)
$ ./icecc-create-darwin-env
# copy the clang_darwin_on_linux_332838.tar.gz to your MacOS host

On the Mac:

# Configure the iceccd
$ git clone https://github.com/darktears/icecream-mac.git ~/icecream-mac/
$ sudo ~/icecream-mac/install.sh 192.168.1.10
$ launchctl load /Library/LaunchDaemons/org.icecream.iceccd.plist
$ launchctl start /Library/LaunchDaemons/org.icecream.iceccd.plist

# Set the ICECC env vars
$ export ICECC_CLANG_REMOTE_CPP=1
$ export ICECC_VERSION=x86_64:~/clang_darwin_on_linux_332838.tar.gz
$ export PATH=~/icecream-mac/bin/icecc/:$PATH

# Get the depot_tools
$ cd ~
$ git clone https://chromium.googlesource.com/chromium/tools/depot_tools.git
$ export PATH=$PATH:~/depot_tools

# Download and build the Chromium sources
$ cd chromium && fetch chromium && cd src
$ gn gen out/Default --args='cc_wrapper="icecc" \
  treat_warnings_as_errors=false \
  clang_use_chrome_plugins=false \
  use_debug_fission=false \
  linux_use_bundled_binutils=false \
  use_lld=false \
  use_jumbo_build=true'
$ ninja -j 32 -C out/Default chrome

… and now the detailed explanation

Installation and setup of icecream on Linux hosts

The installation of icecream on a  Debian based Linux host is pretty simple. The latest version (1.1) for icecc is available in Debian testing and sid for a while so everything that you must to do is install it from the APT repositories. For case of stretch, there is a backport available  in the apt.igalia.com repository publically available:

sudo apt install icecc

The second important part of a icecc cluster is the icecc-scheduler. This daemon is in charge to route the requests from the icecc nodes which requiring available CPUs  for compiling to the nodes of the icecc cluster allowed to run remote build jobs.

In this setup we will activate the scheduler in the Linux node (192.168.1.10). The key here is that only one scheduler should be up at the same time in the same network to avoid errors in the cluster.

sudo systemctl enable icecc-scheduler

Once the scheduler is configured and up, it is time to add icecc hosts to the cluster. We will start adding the Linux hosts following this idea:

  • The IP of the icecc scheduler is 192.168.1.10
  • The Linux host is allowed to run remote jobs
  • The Linux host is allowed to run up to 32 concurrent jobs (this is arbitrary decision and can be adjusted per each particular host)
    # edit /etc/icecc/icecc.conf
    ICECC_NICE_LEVEL="5"
    ICECC_LOG_FILE="/var/log/iceccd.log"
    ICECC_NETNAME=""
    ICECC_MAX_JOBS="32"
    ICECC_ALLOW_REMOTE="yes"
    ICECC_BASEDIR="/var/cache/icecc"
    ICECC_SCHEDULER_LOG_FILE="/var/log/icecc_scheduler.log"
    ICECC_SCHEDULER_HOST="192.168.1.10"

We will need to restart the service to apply those changes:

sudo systemctl restart icecc

Installing and setup of icecream on MacOS hosts

The next step is to install and configure the icecc service on our Mac.  The easy way to get icecc available on Mac is icecream-mac project from darktears. We will do the installation assuming the following facts:

  • The local user account in Mac is psaavedra
  • The IP of the icecc scheduler is 192.168.1.10
  • The Mac is not allowed to accept remote jobs
  • We don’t want run use the Mac as worker.

To get the icecream-mac software we will make a git-clone of the project on Github:

git clone https://github.com/darktears/icecream-mac.git /Users/psaavedra/icecream-mac/
sudo /Users/psaavedra/icecream-mac/install.sh 192.168.1.10

We will edit a bit the /Library/LaunchDaemons/org.icecream.iceccd.plist daemon definition as follows:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
  <dict>
    <key>Label</key>
    <string>org.icecream.iceccd</string>
    <key>ProgramArguments</key>
    <array>
      <string>/Users/psaavedra/icecream-mac/bin/icecc/iceccd</string>
      <string>-s</string>
      <string>192.168.1.10</string>
      <string>-m</string>
      <string>2</string>
      <string>--no-remote</string>
    </array>
    <key>KeepAlive</key>
    <true/>
    <key>UserName</key>
    <string>root</string>
  </dict>
</plist>

Note that we are setting 2 workers in the Mac. Those workers are needed to execute threads in the host client host for things like linking … We will reload the service with this configuration:

launchctl load /Library/LaunchDaemons/org.icecream.iceccd.plist
launchctl start /Library/LaunchDaemons/org.icecream.iceccd.plist

Getting the cross-compilation toolchain for the icecream-mac

We already have the icecc cluster configured but, before to start to build Chromium on MacOS using icecc, there is still something before to do. We still need a cross-compiled clang for Darwin on Linux and, to avoid incompatibilities between versions, we need a clang based on the very same version that your Chromium code to be compiled.

You can check and get the cross-compilation clang revision that you need as follows:

cd src
CLANG_REVISION=$(cat tools/clang/scripts/update.py | grep CLANG_REVISION | head -n 1 | cut -d "'" -f 2)
echo $CLANG_REVISION
332838

In order to simplify this step.  I made some scripts which make it easy the generation of this clang cross-compiled toolchain. On a Linux host:

  • Install build depends:
    sudo apt install build-essential icecc
  • Get the Chromium project depot tools
    git clone https://chromium.googlesource.com/chromium/tools/depot_tools.git ~/depot_tools  
    export PATH=$PATH:~/depot_tools
  • Download the psaavedra’s scripts (yes, my scripts):
    git clone https://github.com/psaavedra/chromium_clang_darwin_on_linux ~/chromium_clang_darwin_on_linux
    cd ~/chromium_clang_darwin_on_linux
  • You can use the get-chromium-clang-revision script to get the latest clang revision using in Chromium master:
    ./get-chromium-clang-revision
  • and then, to build the cross-compiled toolchain:
    ./icecc-create-darwin-env

    ; this script encapsulates the download, configure and build of the clang software.

  • A clang_darwin_on_linux_999999.tar.gz file will be generated.

Setup the icecc environment variables

Once you have the /Users/psaavedra/clang_darwin_on_linux_332838.tar.gz generated in your MacOS. You are ready to set the icecc environments variables.

export ICECC_CLANG_REMOTE_CPP=1
export ICECC_VERSION=x86_64:/Users/psaavedra/clang_darwin_on_linux_332838.tar.gz

The first variable enables the usage of the remote clang for C++. The second one establish toolchain to use by the x86_64 (Linux nodes) to build the code sent from the Mac.

Finally, remember to add the icecc binaries to the $PATH:

export PATH=/Users/psaavedra/icecream-mac/bin/icecc/:$PATH

You can check and get the cross-compiled clang revision that you need as follows:

cd src
CLANG_REVISION=$(cat tools/clang/scripts/update.py | grep CLANG_REVISION | head -n 1 | cut -d "'" -f 2)
echo $CLANG_REVISION
332838

… and building Chromium, at last

Reached this point, it’s time to build a Chromium using the icecc cluster and the cross-compiled clang toolchain previously created. These steps follows the official Chromium build procedure and only adapted to setup the icecc wrapper.

Ensure depot_tools is the path:

cd ~git clone https://chromium.googlesource.com/chromium/tools/depot_tools.git
export PATH=$PATH:~/depot_tools
ninja --version
# 1.8.2

Get the code:

git config --global core.precomposeUnicode truemkdir chromium
cd chromium
fetch chromium

Configure the build:

cd src
gn gen out/Default --args='cc_wrapper="icecc" treat_warnings_as_errors=false clang_use_chrome_plugins=false linux_use_bundled_binutils=false use_jumbo_build=true'
# or with ccache
export CCACHE_PREFIX=icecc
gn gen out/Default --args='cc_wrapper="ccache" treat_warnings_as_errors=false clang_use_chrome_plugins=false linux_use_bundled_binutils=false use_jumbo_build=true'

And build, at last:

ninja -j 32 -C out/Default chrome

icemon allows you to graphically monitoring the icecc cluster. Run it in remote from your Linux host if you don’t want install it in the MacOS:

ssh -X user@yourlinuxbox icemon

; with icemon you should see how each build task is distributed across the icecc cluster.

by Pablo Saavedra at July 10, 2018 08:15 AM

July 09, 2018

Frédéric Wang

Review of Igalia's Web Platform activities (H1 2018)

This is the semiyearly report to let people know a bit more about Igalia’s activities around the Web Platform, focusing on the activity of the first semester of year 2018.

Projects

Javascript

Igalia has proposed and developed the specification for BigInt, enabling math on arbitrary-sized integers in JavaScript. Igalia has been developing implementations in SpiderMonkey and JSC, where core patches have landed. Chrome and Node.js shipped implementations of BigInt, and the proposal is at Stage 3 in TC39.

Igalia is also continuing to develop several features for JavaScript classes, including class fields. We developed a prototype implementation of class fields in JSC. We have maintained Stage 3 in TC39 for our specification of class features, including static variants.

We also participated to WebAssembly (now at First Public Working Draft) and internationalization features for new features such as Intl.RelativeTimeFormat (currently at Stage 3).

Finally, we have written more tests for JS language features, performed maintenance and optimization and participated to other spec discussions at TC39. Among performance optimizations, we have contributed a significant optimization to Promise performance to V8.

Accessibility

Igalia has continued the standardization effort at the W3C. We are pleased to announce that the following milestones have been reached:

A new charter for the ARIA WG as well as drafts for ARIA 1.2 and Core Accessibility API Mappings 1.2 are in preparation and are expected to be published this summer.

On the development side, we implemented new ARIA features and fixed several bugs in WebKit and Gecko. We have refined platform-specific tools that are needed to automate accessibility Web Platform Tests (examine the accessibility tree, obtain information about accessible objects, listen for accessibility events, etc) and hope we will be able to integrate them in Web Platform Tests. Finally we continued maintenance of the Orca screen reader, in particular fixing some accessibility-event-flood issues in Caja and Nautilus that had significant impact on Orca users.

Web Platform Predictability

Thanks to support from Bloomberg, we were able to improve interoperability for various Editing/Selection use cases. For example when using backspace to delete text content just after a table (W3C issue) or deleting a list item inside a content cell.

We were also pleased to continue our collaboration with the AMP project. They provide us a list of bugs and enhancement requests (mostly for the WebKit iOS port) with concrete use cases and repro cases. We check the status and plans in WebKit, do debugging/analysis and of course actually submit patches to address the issues. That’s not always easy (e.g. when it is touching proprietary code or requires to find some specific reviewers) but at least we make discussions move forward. The topics are very diverse, it can be about MessageChannel API, CSSOM View, CSS transitions, CSS animations, iOS frame scrolling custom elements or navigating special links and many others.

In general, our projects are always a good opportunity to write new Web Platform Tests import them in WebKit/Chromium/Mozilla or improve the testing infrastructure. We have been able to work on tests for several specifications we work on.

CSS

Thanks to support from Bloomberg we’ve been pursuing our activities around CSS:

We also got more involved in the CSS Working Group, in particular participating to the face-to-face meeting in Berlin and will attend TPAC’s meeting in October.

WebKit

We have also continued improving the web platform implementation of some Linux ports of WebKit (namely GTK and WPE). A lot of this work was possible thanks to the financial support of Metrological.

Other activities

Preparation of Web Engines Hackfest 2018

Igalia has been organizing and hosting the Web Engines Hackfest since 2009, a three days event where Web Platform developers can meet, discuss and work together. We are still working on the list of invited, sponsors and talks but you can already save the date: It will happen from 1st to 3rd of October in A Coruña!

New Igalians

This semester, new developers have joined Igalia to pursue the Web platform effort:

  • Rob Buis, a Dutch developer currently living in Germany. He is a well-known member of the Chromium community and is currently helping on the web platform implementation in WebKit.

  • Qiuyi Zhang (Joyee), based in China is a prominent member of the Node.js community who is now also assisting our compilers team on V8 developments.

  • Dominik Infuer, an Austrian specialist in compilers and programming language implementation who is currently helping on our JSC effort.

Coding Experience Programs

Two students have started a coding experience program some weeks ago:

  • Oriol Brufau, a recent graduate in math from Spain who has been an active collaborator of the CSS Working Group and a contributor to the Mozilla project. He is working on the CSS Logical Properties and Values specification, implementing it in Chromium implementation.

  • Darshan Kadu, a computer science student from India, who contributed to GIMP and Blender. He is working on Web Platform Tests with focus on WebKit’s infrastructure and the GTK & WPE ports in particular.

Additionally, Caio Lima is continuing his coding experience in Igalia and is among other things working on implementing BigInt in JSC.

Conclusion

Thank you for reading this blog post and we look forward to more work on the web platform this semester!

July 09, 2018 10:00 PM

June 26, 2018

Gyuyoung Kim

How to develop Chromium with Visual Studio Code on Linux?

How have you been developing Chromium? I have often been asked what is the best tool to develop Chromium. I guess Chromium developers have been usually using vim, emacs, cscope, sublime text, eclipse, etc. And they have used GDB or console logs for debugging. But, in case of Windows, developers have used Visual Studio. Although Visual Studio supports powerful features to develop C/C++ programs, unfortunately, it couldn’t be used by other platform developers. However, recently I notice that Visual Studio Code also can support to develop Chromium with the nice editor, powerful debugging tools, and a lot of extensions. And, even it can work on Linux and Mac because it is based on Electron. Nowadays I’m developing Chromium with VS Code. I feel that VS Code is one of the very nice tools to develop Chromium. So I’d like to share my experience how to develop Chromium by using the Visual Studio Code on Ubuntu.

Default settings for Chromium

There is already an article about how to set up the environment to build Chromium in VS codeSo to start, you need to prepare the environment settings. This section just lists key points in the instructions.
* https://chromium.googlesource.com/chromium/src/+/lkcr/docs/vscode.md

  1. Download VS Code
    https://code.visualstudio.com/docs/setup/setup-overview
  2. Launch VS Code in chromium/src
    $ code .
  3. Install useful extensions
    1. Python – Linting, intellisense, code formatting, refactoring, debugging, snippets.
    2. c/c++ for visual studio code – Code formatting, debugging, Intellisense.
    3.  Toggle Header/Source Toggles between .cc and .h with F4. The C/C++ extension supports this as well through Alt+O but sometimes chooses the wrong file when there are multiple files in the workspace that have the same name.
    4. you-complete-me YouCompleteMe code completion for VS Code. It works fairly well in Chromium. To install You-Complete-Me, enter these commands in a terminal:
      $ git clone https://github.com/Valloric/ycmd.git ~/.ycmd 
      $ cd ~/.ycmd 
      $ git submodule update --init --recursive 
      $ ./build.py --clang-completer
    5. Rewrap –  Wrap lines at 80 characters with Alt+Q.
  4. Setup for Chromium
  5. Key mapping
    * Ctrl+P: opens a search box to find and open a file.
    * F1 or Ctrl+Shift+P: opens a search box to find a command
      (e.g. Tasks: Run Task).
    * Ctrl+K, Ctrl+S: opens the key bindings editor.
    * Ctrl+`: toggles the built-in terminal.
    * Ctrl+Shift+M: toggles the problems view (linter warnings, 
      compile errors and warnings). You'll swicth a lot between
      terminal and problem view during compilation.
    * Alt+O: switches between the source/header file.
    * Ctrl+G: jumps to a line.
    * F12: jumps to the definition of the symbol at the cursor
      (also available on right-click context menu).
    * Shift+F12 or F1: CodeSearchReferences, Return shows all
      references of the symbol at the cursor.
    * F1: CodeSearchOpen, Return opens the current file in
      Code Search.
    * Ctrl+D: selects the word at the cursor. Pressing it multiple
      times multi-selects the next occurrences, so typing in one
      types in all of them, and Ctrl+U deselects the last occurrence.
    * Ctrl+K+Z: enters Zen Mode, a fullscreen editing mode with
      nothing but the current editor visible.
    * Ctrl+X: without anything selected cuts the current line.
      Ctrl+V pastes the line.
  6. (Optional) Color setting
    • Press Ctrl+Shift+P, color, Enter to pick a color scheme for the editor

Additional settings after the default settings.

  1. Set workspaceRoot to .bashrc. (Because it will be needed for some extensions.)
    export workspaceRoot=$HOME/chromium/src
  2. Add new tasks to tasks.json in order to build Chromium by using ICECC.

 

{
  "taskName": "1-build_chrome_debug_icecc",
  "command": "buildChromiumICECC.sh Debug",
  "isShellCommand": true,
  "isTestCommand": true,
  "problemMatcher": [
    {
      "owner": "cpp",
      "fileLocation": [
        "relative",
        "${workspaceRoot}"
      ],
      "pattern": {
        "regexp": "^../../(.*):(\\d+):(\\d+):\\s+(warning|\\w*\\s?error):\\s+(.*)$",
        "file": 1,
        "line": 2,
        "column": 3,
        "severity": 4,
        "message": 5
      }
    },
    {
      "owner": "cpp",
      "fileLocation": [
        "relative",
        "${workspaceRoot}"
      ],
      "pattern": {
        "regexp": "^../../(.*?):(.*):\\s+(warning|\\w*\\s?error):\\s+(.*)$",
        "file": 1,
        "severity": 3,
        "message": 4
      }
    }
  ]
},
  1. Update “Chrome Debug” configuration in launch.json
    {
      "name": "Chrome Debug",
      "type": "cppdbg",
      "request": "launch",
      "targetArchitecture": "x64",
      "environment": [
        {"name":"workspaceRoot", "value":"${HOME}/chromium/src"}
      ],
      "program": "${workspaceRoot}/out/Debug/chrome",
      "args": ["--single-process"],  // The debugger only can work with the single process mode for now.
      "preLaunchTask": "1-build_chrome_debug_icecc",
      "stopAtEntry": false,
      "cwd": "${workspaceRoot}/out/Debug",
      "externalConsole": true,
    },

Screenshot after the settings

you will see this window after finishing all settings.

Build Chromium with ICECC in VS Code

  1. Run Task (F1 or Ctrl+Shift+P)
  2. Select 1-build_chrome_debug_icecc. VS Code will show an integrated terminal when building Chromium as below, On the ICECC monitor, you can see that VS Code builds Chromium by using ICECC.

 

Start debugging in VS Code

After completing the build, now is time to start debugging Chromium.

  1. Set a breakpoint
    1. F9 button or just click the left side of line number
  2. Launch debug
    1. Press F5 button
  3. Screen captures when debugger stopped at a breakpoint
    1. Overview
    2. Editor
    3. Call stack
    4. Variables
    5. Watch
    6. Breakpoints

Todo

In multiple processes model, VS Code can’t debug child processes yet (i.e. renderer process). According to C/C++ extension project site, they suggested us to add the below command to launch.json though, it didn’t work for Chromium when I tried.

"setupCommands": [
    { "text": "-gdb-set follow-fork-mode child" }
],

Reference

  1. Chromium VS Code setup: https://chromium.googlesource.com/chromium/src/+/lkcr/docs/vscode.md
  2. https://github.com/Microsoft/vscode-cpptools/blob/master/launch.md

by gyuyoung at June 26, 2018 12:55 AM

June 14, 2018

Diego Pino

Fast checksum computation

An Internet packet generally includes two checksums: a TCP/UDP checksum and an IP checksum. In both cases, the checksum value is calculated using the same algorithm. For instance, IP header checksum is computed as follows:

  • Set the packet’s IP header checksum to zero.
  • Fetch the IP header octets in groups of 16-bit and calculate the accumulated sum.
  • In case there’s an overflow while adding, sum the carry-bit to the total sum.
  • Once the sum is done, set the checksum to the one’s complement of the accumulated sum.

Here is an implementation of such algorithm in Lua:

local ffi = require("ffi")
local bit = require("bit")

local function checksum_lua (data, size)
   local function r16 (data)
      return ffi.cast("uint16_t*", data)[0]
   end
   local csum = 0
   local i = size
   -- Accumulated sum.
   while i > 1 do
      local word = r16(data + (size - i))
      csum = csum + word
      i = i - 2
   end
   -- Handle odd sizes.
   if i == 1 then
      csum = csum + data[size-1]
   end
   -- Add accumulated carry.
   while true do
      local carry = bit.rshift(csum, 16)
      if carry == 0 then break end
      csum = bit.band(csum, 0xffff) + carry
   end
   -- One's complement.
   return bit.band(bit.bnot(csum), 0xffff)
end

The IP header checksum is calculated only over the IP header octets. However, the TCP header is calculated over the TCP header, the packet’s payload plus an extra header called the pseudo-header.

A pseudo-header is a 12-byte data structured composed of:

  • IP source and destination addresses (8 bytes).
  • Protocol (TCP=0x6 or UDP=0x11) (2 bytes).
  • TCP length, calculated from IP header’s total length minus TCP or UDP header size (2 bytes).

You may wonder what’s the purpose of the pseudo-header? David P Reed, often considered the father of UDP, provides a great explanation in this thread: Purpose of pseudo header in TCP checksum. Basically, the original goal of the pseudo-header was to take into account IP addresses as part of the TCP checksum, since they’re relevant fields in an end-to-end communication. Back in those days, the original plan for TCP safe communications was to leave source and destination addresses clear but encrypt the rest of the TCP fields. That would avoid man-in-the middle attacks. However NAT, which is essentially a man-in-the-middle, happened thrashing away this original plan. In summary, the pseudo-header exists today for legacy reasons.

Lastly, is worth mentioning that UDP checksum is optional in IPv4, so you might see it set as zero many times. However, the field is mandatory in IPv6.

Verifying a packet’s checksum

Verifying a packet’s checksum is easy. On receiving a packet, the receiver sums all the relevant octets, including the checksum field. The result must be zero if the packet is correct, since the sum of a number and its one’s complement is always zero.

From a developer’s perspective there are several tools for verifying the correctness of a packet’s checksum. Perhaps my preferred tool is Wireshark, which features an option to check the validity of TCP checksums (Edit->Preferences->Protocols[TCP]. Mark Validate the TCP checksum if possible). When this option is enabled, packets with a wrong checksum are highlighted in a black background.

Bad checksums
Bad checksums

Seeing packets with wrong checksums is common when capturing packets with tcpdump and open them in Wireshark. The reason why checksums are not correct is that TCP checksumming is generally offloaded to the NIC, since it’s a relatively expensive operation (nowadays, NICs count with specialized hardware to do that operation fast). Since tcpdump captures outgoing packets before they hit the NIC, the checksum value hasn’t been calculated yet and likely contains garbage. It’s possible to check whether checksum offloading is enabled in a NIC by typing:

$ ethtool --show-offload <nic> | grep checksumming
rx-checksumming: on
tx-checksumming: on

Another option for verifying checksum values is using tshark:

$ tshark -r packets.pcap -V -o tcp.check_checksum:TRUE | grep -c "Error/Checksum"
20

Lastly, in case you’d like to fix wrong checksums in a pcap file is possible to do that with tcprewrite:

$ tcprewrite -C -i packets.pcap -o packets-fixed.pcap
$ tshark -r packets-fixed.pcap -V -o tcp.check_checksum:TRUE | grep -c "Error/Checksum"
0

Fast checksum computation

Since TCP checksum computation involves a large chunk of data improving its performance is important. There are, in fact, several RFCs dedicated exclusively to discuss this topic. RFC 1071 (Computing the Internet Checksum) includes a detailed explanation of the algorithm and also explores different techniques for speeding up checksumming. In addition, it features reference implementations in several hardware architectures such as Motorola 68020, Cray and IBM 370.

Perhaps the fastest way to recompute a checksum of a modified packet is to incrementally update the checksum as the packet gets modified. Take for instance the case of NAT which modifies origin and destination ports and addresses. Those operations affect both the TCP and IP checksums. In the case of the IP checksum, if the source address gets modified we can recompute the new IP checksum as:

new_checksum = ~(~checksum + (-pkt.source_ip) + new_source_ip);

Or more generically using the following formula:

HC = one-complement(one-complement(C) + (-m) + m)

This technique is covered in RFC 1071 and further polished over two other RFCs: RFC 1141 and RFC 1624 (Incremental Updating of the Internet Checksum).

If we decide to recompute the checksum, there are several techniques to do it fast. On its canonical form, the algorithm says octets are summed as 16-bit words. If there’s carry after an addition, the carry should be added to the accumulated sum. Truth is it’s not necessary to add octets as 16-bit words. Due to the associative property of addition, it is possible to do parallel addition using larger word sizes such as 32-bit or 64-bit words. In those cases the variable that stores the accumulative sum has to be bigger too. Once the sum is computed a final step folds the sum to a 16-bit word (adding carry if any).

Here’s an implementation in C using 32-bit words:

uint16_t checksum (uint8_t* data, uint16_t len)
{
    uint64_t sum = 0;
    uint32_t* p = (uint32_t*) data;
    uint16_t i = 0;
    while (len >= 4) {
        sum = sum + p[i++];
        len -= 4;
    }
    if (len >= 2) { 
        sum = sum + ((uint16_t*) data)[i * 4];
        len -= 2;
    }
    if (len == 1) {
        sum += data[len-1];
    }
    
    // Fold sum into 16-bit word.
    while (sum>>16) {
        sum = (sum & 0xffff) + (sum>>16);
    }
    return ntohs((uint16_t)~sum);
}

Using larger word sizes increases speed as it reduces the total number of operations. What about using this technique on 64-bit integers? It definitely would be possible, but it requires to handle carry in the body loop. In the algorithm above, 32-bit words are summed to a 64-bit word. Carry, if any, is stored in the higher part of sum, which later gets summed in the folding step.

Using SIMD instructions should allow us to sum larger sizes of data in parallel. For instance using AVX2’s VPADD (vector-packed addition) instruction it should be possible to sum 16x16-bit words in parallel. The issue here once again is handling the possible generated carry to the accumulated sum. So instead of a 16x16-bit vector a 8x32 vector is used instead. From a functional point of view this is equivalent to sum using 128-bit words.

Snabb features implementations of checksum computation, generic and using SIMD instructions. In the latter case there are versions for SSE2 and AVX2 instruction sets. Snabb’s philosophy is to do everything in software and rely as much less as possible in offloaded NIC functions. Thus checksum computation is something done in code. Snabb’s implementation using AVX2 instructions is available at src/arch/avx2.c (Luke pushed a very interesting implementation in machine code as well. See PR#899).

Going back to RFC 1071, many of the reference implementations do additions in the main loop taking into account the carry bit. For instance, in the Motorola 68020 implementation that is done using the ADDXL instruction. In X86 there’s an equivalent add-with-carry instruction (ADC). Basically this instruction performs a sum of two operands plus the carry-flag.

Another technique described in RFC 1071, and also used in the reference implementations, is loop unrolling. Instead of summing one word per loop, we could sum 2, 4 or 8 words instead. A loop that sums 64-bit words in strides of 8 means actually avoiding loops for packet sizes lower than 512 bytes. Unrolling a loop requires adding waterfall code after the loop to handle the edge-cases that control the bounds of the loop.

As an exercise to teach myself more DynASM and X86-64 assembly, I decided to rewrite the generic checksum algorithm and see if performance improved. The first implementation followed the canonical algorithm, summing words as 16-bit values. Performance was much better than the generic Lua implementation posted at the beginning of this article, but it wasn’t better than Snabb’s C implementation, which does loop unrolling.

After this initially disappointing result, I decided to apply some of the optimization techniques commented before. Summing octets as 32-bit words definitely improved performance. The advantage of writing the algorithm in assembly is that I could make use of the ADC instruction. That allowed me to use 64-bit words. Performance improved once again. Finally I tried out several loop unrolling. With a loop unrolling of 4 strides the algorithm proved to be better than the SSE2 algorithm for several packet sizes: 64 bytes, 570 bytes and 1520 bytes. However, it doesn’t beat the AVX2 implementation in the large packet case, but it shown better performance for small and medium sizes.

And here’s the final implementation:

; Prologue.
push rbp
mov rbp, rsp

; Accumulative sum.
xor rax, rax                ; Clear out rax. Stores accumulated sum.
xor r9, r9                  ; Clear out r9. Stores value of array.
xor r8, r8                  ; Clear out r8. Stores array index.
mov rcx, rsi                ; Rsi (2nd argument; size). Assign rsi to rcx.
1:
cmp rcx, 32                 ; If index is less than 16.
jl >2                       ; Jump to branch '2'.
add rax, [rdi + r8]         ; Sum acc with qword[0].
adc rax, [rdi + r8 + 8]     ; Sum with carry qword[1].
adc rax, [rdi + r8 + 16]    ; Sum with carry qword[2].
adc rax, [rdi + r8 + 24]    ; Sum with carry qword[3]
adc rax, 0                  ; Sum carry-bit into acc.
sub rcx, 32                 ; Decrease index by 8.
add r8, 32                  ; Jump two qwords.
jmp <1                      ; Go to beginning of loop.
2:
cmp rcx, 16                 ; If index is less than 16.
jl >3                       ; Jump to branch '2'.
add rax, [rdi + r8]         ; Sum acc with qword[0].
adc rax, [rdi + r8 + 8]     ; Sum with carry qword[1].
adc rax, 0                  ; Sum carry-bit into acc.
sub rcx, 16                 ; Decrease index by 8.
add r8, 16                  ; Jump two qwords.
3:
cmp rcx, 8                  ; If index is less than 8.
jl >4                       ; Jump to branch '2'.
add rax, [rdi + r8]         ; Sum acc with qword[0].
adc rax, 0                  ; Sum carry-bit into acc.
sub rcx, 8                  ; Decrease index by 8.
add r8, 8                   ; Next 64-bit.
4:
cmp rcx, 4                  ; If index is less than 4.
jl >5                       ; Jump to branch '3'.
mov r9d, dword [rdi + r8]   ; Fetch 32-bit from data + r8 into r9d.
add rax, r9                 ; Sum acc with r9. Accumulate carry.
sub rcx, 4                  ; Decrease index by 4.
add r8, 4                   ; Next 32-bit.
5:
cmp rcx, 2                  ; If index is less than 2.
jl >6                       ; Jump to branch '4'.
movzx r9, word [rdi + r8]   ; Fetch 16-bit from data + r8 into r9.
add rax, r9                 ; Sum acc with r9. Accumulate carry.
sub rcx, 2                  ; Decrease index by 2.
add r8, 2                   ; Next 16-bit.
6:
cmp rcx, 1                  ; If index is less than 1.
jl >7                       ; Jump to branch '5'.
movzx r9, byte [rdi + r8]   ; Fetch 8-bit from data + r8 into r9.
add rax, r9                 ; Sum acc with r9. Accumulate carry.

; Fold 64-bit into 16-bit.
7:
mov r9, rax                 ; Assign acc to r9.
shr r9, 32                  ; Shift r9 32-bit. Stores higher part of acc.
and rax, 0x00000000ffffffff ; Clear out higher-part of rax. Stores lower part of acc.
add eax, r9d                ; 32-bit sum of acc and r9.
adc eax, 0                  ; Sum carry to acc.
mov r9d, eax                ; Repeat for 16-bit.
shr r9d, 16
and eax, 0x0000ffff
add ax, r9w
adc ax, 0

; One's complement.
not rax                     ; One-complement of rax.
and rax, 0xffff             ; Clear out higher part of rax.

; Epilogue.
mov rsp, rbp
pop rbp

; Return.
ret

Benchmark results for several data sizes:

Data size: 44 bytes

Algorithm Time per csum Time per byte
Generic 87.77 ns 1.99 ns
SSE2 86.06 ns 1.96 ns
AVX2 83.20 ns 1.89 ns
New 52.10 ns 1.18 ns

Data size: 550 bytes

Algorithm Time per csum Time per byte
Generic 1058.04 ns 1.92 ns
SSE2 510.40 ns 0.93 ns
AVX2 318.42 ns 0.58 ns
New 270.79 ns 0.49 ns

Data size: 1500 bytes

Algorithm Time per csum Time per byte
Generic 2910.10 ns 1.94 ns
SSE2 991.04 ns 0.66 ns
AVX2 664.98 ns 0.44 ns
New 743.88 ns 0.50 ns

All in all, it has been a fun exercise. I have learned quite a lot about the Internet checksum algorithm. I also learned how loop unrolling can help improving performance in a dramatic way (more than I initially expected). I found very interesting as well how changing the context of a problem, in this case the target programming language, forces to think about the problem in a different way but it also enables the possibility of doing more optimizations that were not possible before.

June 14, 2018 06:00 AM

June 13, 2018

Jacobo Aragunde

Chromium official/release builds and icecc

You may already be using icecc to compile your Chromium, either by following some instructions like the ones published by my colleague Gyuyoung or using the popular icecc-chromium set of scripts. In those cases, you will probably get in some trouble if you try to generate an official build with that configuration.

First, let me refresh what an “official build” is called in Chromium. You may know that build optimization in Chromium builds depends on two flags:

  • is_debug
    Debug build. Enabling official builds automatically sets is_debug to false.
  • is_official_build
    Set to enable the official build level of optimization. This has nothing
    to do with branding, but enables an additional level of optimization above
    release (!is_debug). This might be better expressed as a tri-state
    (debug, release, official) but for historical reasons there are two
    separate flags.

  • The GN documentation is pretty verbose about this. To sum up, to get full binary optimization you should enable is_official_build which will also disable is_debug in the background. This is what other projects would call a release build.

    Back to the main topic, I was running an official build distributed via icecc and stumbled on some compilation problems:

    clang: error: no such file or directory: /usr/lib/clang/7.0.0/share/cfi_blacklist.txt
    clang: error: no such file or directory: ../../tools/cfi/blacklist.txt
    clang: error: no such file or directory: /path/to/src/chrome/android/profiles/afdo.prof
    

    These didn’t happen when icecc build was disabled, so I was certain to have found some limitations in the distributed compiler. The icecc-chromium set of scripts was already disabling a number of clang cleanup/sanitize tools, so I decided to take the same approach. First, I checked the GN args that could be related to these errors and identified two:

    • is_cfi
      Current value (from the default) = true
      From //build/config/sanitizers/sanitizers.gni:53

    Compile with Control Flow Integrity to protect virtual calls and casts.
    See http://clang.llvm.org/docs/ControlFlowIntegrity.html

    TODO(pcc): Remove this flag if/when CFI is enabled in all official builds.

  • clang_use_default_sample_profile
    Current value (from the default) = true
    From //build/config/compiler/BUILD.gn:117

    Some configurations have default sample profiles. If this is true and
    clang_sample_profile_path is empty, we’ll fall back to the default.

    We currently only have default profiles for Chromium in-tree, so we disable
    this by default for all downstream projects, since these profiles are likely
    nonsensical for said projects.

  • These two args were enabled, I just disabled them and got rid the compilation flags that were causing trouble: -fprofile-sample-use=/path/to/src/chrome/android/profiles/afdo.prof -fsanitize=cfi-vcall -fsanitize-blacklist=../../tools/cfi/blacklist.txt. I’ve learned that support for -fsanitize-blacklist is available in upstream icecc, but most distros don’t package it yet, so it’s safer to disable that.

    To sum up, if you are using icecc and you want to run an official build, you have to add a couple more GN args:

    clang_use_default_sample_profile = false
    is_cfi = false
    

    by Jacobo Aragunde Pérez at June 13, 2018 08:06 AM

    June 04, 2018

    Michael Catanzaro

    Security vulnerability in Epiphany Technology Preview

    If you use Epiphany Technology Preview, please update immediately and ensure you have revision 3.29.2-26 or newer. We discovered and resolved a vulnerability that allowed websites to access internal Epiphany features and thereby exfiltrate passwords from the password manager. We apologize for this oversight.

    The unstable Epiphany 3.29.2 release is the only affected release. Epiphany 3.29.1 is not affected. Stable releases, including Epiphany 3.28, are also not affected.

    There is no reason to believe that the issue was discovered or exploited by any attackers, but you might wish to change your passwords if you are concerned.

    by Michael Catanzaro at June 04, 2018 11:00 PM

    May 29, 2018

    Maksim Sisov

    Chromium with Ozone/Wayland: BlinkOn9, dmabuf and more refactorings…

    It has been quite a long while since we wrote blogs about our Chromium Ozone/Wayland effort, and there are a lot of news right now. Igalia participated in the BlinkOn9 conference and gave a talk (https://www.youtube.com/watch?v=DREywLVAVeo) about the Ozone/Wayland support, and had many discussions on how to continue with upstreaming desktop integration related patches for the Mus service.

    Even though, we had been able to make Chromium running with the Wayland backend natively, and upstreamed a lot of Ozone/Wayland related patches, some disagreements had had to be resolved before proceeding with upstreaming. To be precise, the future of the Mus service was unclear and it was decided to abandon it in favor of a platform desktop integration directly to Aura without Mus. Thanks for a good Ozone design, we had been able to quickly redesign our solution and upstream more patches to make it possible to run Chromium Ozone/Wayland from the ToT. Of course, it still has been missing some functionality patches, but the effort is going on steadily and we expect to have all the patches upstreamed in the following months. A lot of work still has to be done.

    Another point I would like to mention is about our UI/GPU split effort. This is the effort to make Chromium Ozone/Wayland to be run with a separate gpu process. For that, a proper communication channel between the browser process, where a wayland connection is established, must be established. We have had tried a nested compositor approach, but decided to abandon it in favor of a dmabuf based approach, which will also allow us to have gpu native memory buffers, rasterization and, perhaps, zero-copy features enabled on some platforms. The idea behind of the dmabuf approach is to reuse some of the Ozone/Drm codebase, and by utilizing drm render nodes, create shared GBM buffers, which file descriptors are then shared with the browser process, where Wayland creates a dmabuf based wl_buffer and attaches it to wayland windows. At this stage, we have already had a working PoC, and Ozone/Drm refactorings are being upstreamed now. It will also support a presentation feedback if such protocol is available on a system with a Wayland compositor.

    Last but not least, we would like to clarify about the accelerated media decode in Chromium Ozone/Wayland mentioned in https://www.phoronix.com/scan.php?page=news_item&px=Chromium-Igalia-Wayland-V4L2VDA.
    To make it clear, we are not currently working on the V4L2, but rather the patches have been just merged by another external contributor for simplicity of compiling our Chromium solution on ARM based boards, especially Renesas M3 R-car board.

    by msisov at May 29, 2018 08:51 AM

    May 27, 2018

    Michael Catanzaro

    Thoughts on Flatpak after four months of Epiphany Technology Preview

    It’s been four months since I announced Epiphany Technology Preview — which I’ve been using as my main browser ever since — and five months since I announced the availability of a stable channel via Flatpak. For the most part, it’s been a good experience. Having the latest upstream development code for everything is wonderful and makes testing very easy. Any user can painlessly download and install either the latest stable version or the bleeding-edge development version on any Linux system, regardless of host dependencies, either via a couple clicks in GNOME Software or one command in the terminal. GNOME Software keeps it updated, so I always have a recent version. Thanks to this, I’m often noticing problems shortly after they’re introduced, rather than six months later, as was so often the case for me in the past. Plus, other developers can no longer complain that there’s a problem with my local environment when I report a bug they can’t reproduce, because Epiphany Technology Preview is a canonical distribution environment, a ground truth of sorts.

    There have been some rough patches where Epiphany Technology Preview was not working properly — sometimes for several days — due to various breaking changes, and the long time required to get a successful SDK build when it’s failing. For example, multimedia playback was broken for all of last week, due to changes in how the runtime is built. H.264 video is still broken, since the relevant Flatpak extension is only compatible with the 3.28 runtime, not with master. Opening files was broken for a while due to what turned out to be a bug in mutter that was causing the OpenURI portal to crash. I just today found another bug where closing a portal while visiting Slack triggered a gnome-shell crash. For the most part, these sorts of problems are expected by testers of unstable nightly software, though I’m concerned about the portal bugs because these affect stable users too. Anyway, these are just bugs, and all software has bugs: they get fixed, nothing special.

    So my impression of Flatpak is still largely positive. Flatpak does not magically make our software work properly in all host environments, but it hugely reduces the number of things that can go wrong on the host system. In recent years, I’ve seen users badly break Epiphany in various ways, e.g. by installing custom mimeinfo or replacing the network backend. With Flatpak, either of these would require an incredible amount of dedicated effort. Without a doubt, Flatpak distribution is more robust to user error. Another advantage is that we get the latest versions of OS dependencies, like GStreamer, libsoup, and glib-networking, so we can avoid the many bugs in these components that have been fixed in the years since our users’ LTS distros froze the package versions. I appreciate the desire of LTS distros to provide stability for users, but at the same time, I’m not impressed when users report issues with the browser that we fixed two years ago in one dependency or another. Flatpak is an excellent compromise solution to this problem: the LTS distro retains an LTS core, but specific applications can use newer dependencies from the Flatpak runtime.

    But there is one huge downside to using Flatpak: we lose crash reports. It’s at best very difficult — and often totally impossible — to investigate crashes when using Flatpak, and that’s frankly more important than any of the gains I mention above. For example, today Epiphany Technology Preview is crashing pretty much constantly. It’s surely a bug in WebKit, but that’s all I can figure out. The way to get a backtrace from a crashing app in flatpak is to use coredumpctl to manually dump the core dump to disk, then launch a bash shell in the flatpak environment and manually load it up in gdb. The process is manual, old-fashioned, primitive, and too frustrating for me by a lot, so I wrote a little pyexpect script to automate this process for Epiphany, thinking I could eventually generalize it into a tool that would be useful for other developers. It’s a horrible hack, but it worked pretty well the day I tested it. I haven’t seen it work since. Debuginfo seems to be constantly broken, so I only see a bunch of ???s in my backtraces, and how are we supposed to figure out how to debug that? So I have no way to debug or fix the WebKit bug, because I can’t get a backtrace. The broken, inconsistent, or otherwise-unreliable debuginfo is probably just some bug that will be fixed eventually (and which I half suspect may be related to our recent freedesktop SDK upgrade. Update: Alex has debugged the debuginfo problem and it looks like that’s on track to be solved), but even once it is, we’re back to square one: it’s still too much effort to get the backtrace, relative to developing on the host system, and that’s a hard problem to solve. It requires tools that do not exist, and for which we have no plans to create, or even any idea of how to create them.

    This isn’t working. I need to be able to effortlessly get a backtrace out of my application, with no or little more effort than running coredumpctl gdb as I would without Flatpak in the way. Every time I see Epiphany or WebKit crash, knowing I can do nothing to debug or investigate, I’m very sorely tempted to switch back to using Fedora’s Epiphany, or good old JHBuild. (I can’t promote BuildStream here, because BuildStream has the same problem.)

    So the developer experience is just not good, but set that aside: the main benefits of Flatpak are for users, not developers, after all. Now, what if users run into a crash, how can they report the bug? Crash reports are next to useless without a backtrace, and wise developers refuse to look at crash reports until a quality backtrace has been posted. So first we need to fix the developer experience to work properly, but even then, it’s not enough: we need an automatic crash reporter, along the lines of ABRT or apport, to make reporting crashes realistically-achievable for users, as it already is for distro-packaged apps. But this is a much harder problem to solve. Such a tool will require integration with coredumpctl, and I have not the faintest clue how we could go about making coredumpctl support container environments. Yet without this, we’re asking application developers to give up their most valuable data — crash reports — in order to use Flatpak.

    Eventually, if we don’t solve crash reporting, Epiphany’s experiment with Flatpak will have to come to an end, because that’s more important to me than the (admittedly-tremendous) benefits of Flatpak. I’m still hopeful that the ingenuity of the Flatpak community will find some solutions. We’ll see how this goes.

    by Michael Catanzaro at May 27, 2018 11:39 PM

    May 21, 2018

    Andy Wingo

    correct or inotify: pick one

    Let's say you decide that you'd like to see what some other processes on your system are doing to a subtree of the file system. You don't want to have to change how those processes work -- you just want to see what files those processes create and delete.

    One approach would be to just scan the file-system tree periodically, enumerating its contents. But when the file system tree is large and the change rate is low, that's not an optimal thing to do.

    Fortunately, Linux provides an API to allow a process to receive notifications on file-system change events, called inotify. So you open up the inotify(7) manual page, and are greeted with this:

    With careful programming, an application can use inotify to efficiently monitor and cache the state of a set of filesystem objects. However, robust applications should allow for the fact that bugs in the monitoring logic or races of the kind described below may leave the cache inconsistent with the filesystem state. It is probably wise to do some consistency checking, and rebuild the cache when inconsistencies are detected.

    It's not exactly reassuring is it? I mean, "you had one job" and all.

    Reading down a bit farther, I thought that with some "careful programming", I could get by. After a day of trying, I am now certain that it is impossible to build a correct recursive directory monitor with inotify, and I am not even sure that "good enough" solutions exist.

    pitfall the first: buffer overflow

    Fundamentally, inotify races the monitoring process with all other processes on the system. Events are delivered to the monitoring process via a fixed-size buffer that can overflow, and the monitoring process provides no back-pressure on the system's rate of filesystem modifications. With inotify, you have to be ready to lose events.

    This I think is probably the easiest limitation to work around. The kernel can let you know when the buffer overflows, and you can tweak the buffer size. Still, it's a first indication that perfect is not possible.

    pitfall the second: now you see it, now you don't

    This one is the real kicker. Say you get an event that says that a file "frenemies.txt" has been created in the directory "/contacts/". You go to open the file -- but is it still there? By the time you get around to looking for it, it could have been deleted, or renamed, or maybe even created again or replaced! This is a TOCTTOU race, built-in to the inotify API. It is literally impossible to use inotify without this class of error.

    The canonical solution to this kind of issue in the kernel is to use file descriptors instead. Instead of or possibly in addition to getting a name with the file change event, you get a descriptor to a (possibly-unlinked) open file, which you would then be responsible for closing. But that's not what inotify does. Oh well!

    pitfall the third: race conditions between inotify instances

    When you inotify a directory, you get change notifications for just that directory. If you want to get change notifications for subdirectories, you need to open more inotify instances and poll on them all. However now you have N2 problems: as poll and the like return an unordered set of readable file descriptors, each with their own ordering, you no longer have access to a linear order in which changes occurred.

    It is impossible to build a recursive directory watcher that definitively says "ok, first /contacts/frenemies.txt was created, then /contacts was renamed to /peeps, ..." because you have no ordering between the different watches. You don't know that there was ever even a time that /contacts/frenemies.txt was an accessible file name; it could have been only ever openable as /peeps/frenemies.txt.

    Of course, this is the most basic ordering problem. If you are building a monitoring tool that actually wants to open files -- good luck bubster! It literally cannot be correct. (It might work well enough, of course.)

    reflections

    As far as I am aware, inotify came out to address the needs of desktop search tools like the belated Beagle (11/10 good pupper just trying to get his pup on). Especially in the days of spinning metal, grovelling over the whole hard-drive was a real non-starter, especially if the search database should to be up-to-date.

    But after looking into inotify, I start to see why someone at Google said that desktop search was in some ways harder than web search -- I mean we all struggle to find files on our own machines, even now, 15 years after the whole dnotify/inotify thing started. Part of it is that the given the choice between supporting reliable, fool-proof file system indexes on the one hand, and overclocking the IOPS benchmarks on the other, the kernel gave us inotify. I understand it, but inotify still sucks.

    I dunno about you all but whenever I've had to document such an egregious uncorrectable failure mode as any of the ones in the inotify manual, I have rewritten the software instead. In that spirit, I hope that some day we shall send inotify to the pet cemetery, to rest in peace beside Beagle.

    by Andy Wingo at May 21, 2018 02:29 PM

    May 16, 2018

    Andy Wingo

    lightweight concurrency in lua

    Hello, all! Today I'd like to share some work I have done recently as part of the Snabb user-space networking toolkit. Snabb is mainly about high-performance packet processing, but it also needs to communicate with management-oriented parts of network infrastructure. These communication needs are performed by a dedicated manager process, but that process has many things to do, and can't afford to make blocking operations.

    Snabb is written in Lua, which doesn't have built-in facilities for concurrency. What we'd like is to have fibers. Fortunately, Lua's coroutines are powerful enough to implement fibers. Let's do that!

    fibers in lua

    First we need a scheduling facility. Here's the smallest possible scheduler: simply a queue of tasks and a function to run those tasks.

    local task_queue = {}
    
    function schedule_task(thunk)
       table.insert(task_queue, thunk)
    end
    
    function run_tasks()
       local queue = task_queue
       task_queue = {}
       for _,thunk in ipairs(queue) do thunk() end
    end
    

    For our purposes, a task is just a function that will be called with no arguments.

    Now let's build fibers. This is easier than you might think!

    local current_fiber = false
    
    function spawn_fiber(fn)
       local fiber = coroutine.create(fn)
       schedule_task(function () resume_fiber(fiber) end)
    end
    
    function resume_fiber(fiber, ...)
       current_fiber = fiber
       local ok, err = coroutine.resume(fiber, ...)
       current_fiber = nil
       if not ok then
          print('Error while running fiber: '..tostring(err))
       end
    end
    
    function suspend_current_fiber(block, ...)
       -- The block function should arrange to reschedule
       -- the fiber when it becomes runnable.
       block(current_fiber, ...)
       return coroutine.yield()
    end
    

    Here, a fiber is simply a coroutine underneath. Suspending a fiber suspends the coroutine. Resuming a fiber runs the coroutine. If you're unfamiliar with coroutines, or coroutines in Lua, maybe have a look at the lua-users wiki page on the topic.

    The difference between a fibers facility and just coroutines is that with fibers, you have a scheduler as well. Very much like Scheme's call-with-prompt, coroutines are one of those powerful language building blocks that should rarely be used directly; concurrent programming needs more structure than what Lua offers.

    If you're following along, it's probably worth it here to think how you would implement yield based on these functions. A yield implementation should yield control to the scheduler, and resume the fiber on the next scheduler turn. The answer is here.

    communication

    Once you have fibers and a scheduler, you have concurrency, which means that if you're not careful, you have a mess. Here I think the Go language got the essence of the idea exactly right: Do not communicate by sharing memory; instead, share memory by communicating.

    Even though Lua doesn't support multiple machine threads running concurrently, concurrency between fibers can still be fraught with bugs. Tony Hoare's Communicating Sequential Processes showed that we can avoid a class of these bugs by treating communication as a first-class concept.

    Happily, the Concurrent ML project showed that it's possible to build these first-class communication facilities as a library, provided the language you are working in has threads of some kind, and fibers are enough. Last year I built a Concurrent ML library for Guile Scheme, and when in Snabb we had a similar need, I ported that code over to Lua. As it's a new take on the problem in a different language, I think I've been able to simplify things even more.

    So let's take a crack at implementing Concurrent ML in Lua. In CML, the fundamental primitive for communication is the operation. An operation represents the potential for communication. For example, if you have a channel, it would have methods to return "get operations" and "put operations" on that channel. Actually receiving or sending a message on a channel occurs by performing those operations. One operation can be performed many times, or not at all.

    Compared to a system like Go, for example, there are two main advantages of CML. The first is that CML allows non-deterministic choice between a number of potential operations in a generic way. For example, you can construct a operation that, when performed, will either get on one channel or wait for a condition variable to be signalled, whichever comes first. In Go, you can only select between operations on channels.

    The other interesting part of CML is that operations are built from a uniform protocol, and so users can implement new kinds of operations. Compare again to Go where all you have are channels, and nothing else.

    The CML operation protocol consists three related functions: try which attempts to directly complete an operation in a non-blocking way; block, which is called after a fiber has suspended, and which arranges to resume the fiber when the operation completes; and wrap, which is called on the result of a successfully performed operation.

    In Lua, we can call this an implementation of an operation, and create it like this:

    function new_op_impl(try, block, wrap)
       return { try=try, block=block, wrap=wrap }
    end
    

    Now let's go ahead and write the guts of CML: the operation implementation. We'll represent an operation as a Lua object with two methods. The perform method will attempt to perform the operation, and return the resulting value. If the operation can complete immediately, the call to perform will return directly. Otherwise, perform will suspend the current fiber and arrange to continue only when the operation completes.

    The wrap method "decorates" an operation, returning a new operation that, if and when it completes, will "wrap" the result of the completed operation with a function, by applying the function to the result. It's useful to distinguish the sub-operations of a non-deterministic choice from each other.

    Here our new_op function will take an array of operation implementations and return an operation that, when performed, will synchronize on the first available operation. As you can see, it already has the equivalent of Go's select built in.

    function new_op(impls)
       local op = { impls=impls }
       
       function op.perform()
          for _,impl in ipairs(impls) do
             local success, val = impl.try()
             if success then return impl.wrap(val) end
          end
          local function block(fiber)
             local suspension = new_suspension(fiber)
             for _,impl in ipairs(impls) do
                impl.block(suspension, impl.wrap)
             end
          end
          local wrap, val = suspend_current_fiber(block)
          return wrap(val)
       end
    
       function op.wrap(f)
          local wrapped = {}
          for _, impl in ipairs(impls) do
             local function wrap(val)
                return f(impl.wrap(val))
             end
             local impl = new_op_impl(impl.try, impl.block, wrap)
             table.insert(wrapped, impl)
          end
          return new_op(wrapped)
       end
    
       return op
    end
    

    There's only one thing missing there, which is new_suspension. When you go to suspend a fiber because none of the operations that it's trying to do can complete directly (i.e. all of the try functions of its impls returned false), at that point the corresponding block functions will publish the fact that the fiber is waiting. However the fiber only waits until the first operation is ready; subsequent operations becoming ready should be ignored. The suspension is the object that manages this state.

    function new_suspension(fiber)
       local waiting = true
       local suspension = {}
       function suspension.waiting() return waiting end
       function suspension.complete(wrap, val)
          assert(waiting)
          waiting = false
          local function resume()
             resume_fiber(fiber, wrap, val)
          end
          schedule_task(resume)
       end
       return suspension
    end
    

    As you can see, the suspension's complete method is also the bit that actually arranges to resume a suspended fiber.

    Finally, just to round out the implementation, here's a function implementing non-deterministic choice from among a number of sub-operations:

    function choice(...)
       local impls = {}
       for _, op in ipairs({...}) do
          for _, impl in ipairs(op.impls) do
             table.insert(impls, impl)
          end
       end
       return new_op(impls)
    end
    

    on cml

    OK, I'm sure this seems a bit abstract at this point. Let's implement something concrete in terms of these primitives: channels.

    Channels expose two similar but different kinds of operations: put operations, which try to send a value, and get operations, which try to receive a value. If there's a sender already waiting to send when we go to perform a get_op, the operation continues directly, and we resume the sender; otherwise the receiver publishes its suspension to a queue. The put_op case is similar.

    Finally we add some synchronous put and get convenience methods, in terms of their corresponding CML operations.

    function new_channel()
       local ch = {}
       -- Queues of suspended fibers waiting to get or put values
       -- via this channel.
       local getq, putq = {}, {}
    
       local function default_wrap(val) return val end
       local function is_empty(q) return #q == 0 end
       local function peek_front(q) return q[1] end
       local function pop_front(q) return table.remove(q, 1) end
       local function push_back(q, x) q[#q+1] = x end
    
       -- Since a suspension could complete in multiple ways
       -- because of non-deterministic choice, it could be that
       -- suspensions on a channel's putq or getq are already
       -- completed.  This helper removes already-completed
       -- suspensions.
       local function remove_stale_entries(q)
          local i = 1
          while i <= #q do
             if q[i].suspension.waiting() then
                i = i + 1
             else
                table.remove(q, i)
             end
          end
       end
    
       -- Make an operation that if and when it completes will
       -- rendezvous with a receiver fiber to send VAL over the
       -- channel.  Result of performing operation is nil.
       function ch.put_op(val)
          local function try()
             remove_stale_entries(getq)
             if is_empty(getq) then
                return false, nil
             else
                local remote = pop_front(getq)
                remote.suspension.complete(remote.wrap, val)
                return true, nil
             end
          end
          local function block(suspension, wrap)
             remove_stale_entries(putq)
             push_back(putq, {suspension=suspension, wrap=wrap, val=val})
          end
          return new_op({new_op_impl(try, block, default_wrap)})
       end
    
       -- Make an operation that if and when it completes will
       -- rendezvous with a sender fiber to receive one value from
       -- the channel.  Result is the value received.
       function ch.get_op()
          local function try()
             remove_stale_entries(putq)
             if is_empty(putq) then
                return false, nil
             else
                local remote = pop_front(putq)
                remote.suspension.complete(remote.wrap)
                return true, remote.val
             end
          end
          local function block(suspension, wrap)
             remove_stale_entries(getq)
             push_back(getq, {suspension=suspension, wrap=wrap})
          end
          return new_op({new_op_impl(try, block, default_wrap)})
       end
    
       function ch.put(val) return ch.put_op(val).perform() end
       function ch.get()    return ch.get_op().perform()    end
    
       return ch
    end
    

    a wee example

    You might be wondering what it's like to program with channels in Lua, so here's a little example that shows a prime sieve based on channels. It's not a great example of concurrency in that it's not an inherently concurrent problem, but it's cute to show computations in terms of infinite streams.

    function prime_sieve(count)
       local function sieve(p, rx)
          local tx = new_channel()
          spawn_fiber(function ()
             while true do
                local n = rx.get()
                if n % p ~= 0 then tx.put(n) end
             end
          end)
          return tx
       end
    
       local function integers_from(n)
          local tx = new_channel()
          spawn_fiber(function ()
             while true do
                tx.put(n)
                n = n + 1
             end
          end)
          return tx
       end
    
       local function primes()
          local tx = new_channel()
          spawn_fiber(function ()
             local rx = integers_from(2)
             while true do
                local p = rx.get()
                tx.put(p)
                rx = sieve(p, rx)
             end
          end)
          return tx
       end
    
       local done = false
       spawn_fiber(function()
          local rx = primes()
          for i=1,count do print(rx.get()) end
          done = true
       end)
    
       while not done do run_tasks() end
    end
    

    Here you also see an example of running the scheduler in the last line.

    where next?

    Let's put this into perspective: in a couple hundred lines of code, we've gone from minimal Lua to a language with lightweight multitasking, extensible CML-based operations, and CSP-style channels; truly a delight.

    There are a number of possible ways to extend this code. One of them is to implement true multithreading, if the language you are working in supports that. In that case there are some small protocol modifications to take into account; see the notes on the Guile CML implementation and especially the Manticore Parallel CML project.

    The implementation above is pleasantly small, but it could be faster with the choice of more specialized data structures. I think interested readers probably see a number of opportunities there.

    In a library, you might want to avoid the global task_queue and implement nested or multiple independent schedulers, and of course in a parallel situation you'll want core-local schedulers as well.

    The implementation above has no notion of time. What we did in the Snabb implementation of fibers was to implement a timer wheel, inspired by Juho Snellman's Ratas, and then add that timer wheel as a task source to Snabb's scheduler. In Snabb, every time the equivalent of run_tasks() is called, a scheduler asks its sources to schedule additional tasks. The timer wheel implementation schedules expired timers. It's straightforward to build CML timeout operations in terms of timers.

    Additionally, your system probably has other external sources of communication, such as sockets. The trick to integrating sockets into fibers is to suspend the current fiber whenever an operation on a file descriptor would block, and arrange to resume it when the operation can proceed. Here's the implementation in Snabb.

    The only difficult bit with getting nice nonblocking socket support is that you need to be able to suspend the calling thread when you see the EWOULDBLOCK condition, and for coroutines that is often only possible if you implemented the buffered I/O yourself. In Snabb that's what we did: we implemented a compatible replacement for Lua's built-in streams, in Lua. That lets us handle EWOULDBLOCK conditions in a flexible manner. Integrating epoll as a task source also lets us sleep when there are no runnable tasks.

    Likewise in the Snabb context, we are also working on a TCP implementation. In that case you want to structure TCP endpoints as fibers, and arrange to suspend and resume them as appropriate, while also allowing timeouts. I think the scheduler and CML patterns are going to allow us to do that without much trouble. (Of course, the TCP implementation will give us lots of trouble!)

    Additionally your system might want to communicate with fibers from other threads. It's entirely possible to implement CML on top of pthreads, and it's entirely possible as well to support communication between pthreads and fibers. If this is interesting to you, see Guile's implementation.

    When I talked about fibers in an earlier article, I built them in terms of delimited continuations. Delimited continuations are fun and more expressive than coroutines, but it turns out that for fibers, all you need is the expressive power of coroutines -- multi-shot continuations aren't useful. Also I think the presentation might be more straightforward. So if all your language has is coroutines, that's still good enough.

    There are many more kinds of standard CML operations; implementing those is also another next step. In particular, I have found semaphores and condition variables to be quite useful. Also, standard CML supports "guards", invoked when an operation is performed, and "nacks", invoked when an operation is definitively not performed because a choice selected some other operation. These can be layered on top; see the Parallel CML paper for notes on "primitive CML".

    Also, the choice operator above is left-biased: it will prefer earlier impls over later ones. You might want to not always start with the first impl in the list.

    The scheduler shown above is the simplest thing I could come up with. You may want to experiment with other scheduling algorithms, e.g. capability-based scheduling, or kill-safe abstractions. Do it!

    Or, it could be you already have a scheduler, like some kind of main loop that's already there. Cool, you can use it directly -- all that fibers needs is some way to schedule functions to run.

    godspeed

    In summary, I think Concurrent ML should be better-known. Its simplicity and expressivity make it a valuable part of any concurrent system. Already in Snabb it helped us solve some longstanding gnarly issues by making the right solutions expressible.

    As Adam Solove says, Concurrent ML is great, but it has a branding problem. Its ideas haven't penetrated the industrial concurrent programming world to the extent that they should. This article is another attempt to try to get the word out. Thanks to Adam for the observation that CML is really a protocol; I'm sure the concepts could be made even more clear, but at least this is a step forward.

    All the code in this article is up on a gitlab snippet along with instructions for running the example program from the command line. Give it a go, and happy hacking with CML!

    by Andy Wingo at May 16, 2018 03:17 PM

    Jessica Tallon

    IPFIX Performance

    IPFIX

    IPFIX is a standard for recording “flows”, which are packets with the same: source IP, destination IP, source port, destination port and protocol. It’s got a few uses but probably the most familiar is tracking usage for billing. The recording is done by something called a “probe” which gets the traffic in and builds a “flow table”, usually some form of hash-map which records a “flow” backed by additional data like how many bytes and packets have been transmitted. These flows are then exported by an “exporter”, usually just another process outside the performance critical “dataplane” (probe) to a collector which can store and display the information in a useful / pretty manner such as a web UI. IPFIX usually runs on a copy of the traffic to ensure not to hinder the traffic itself, which means we shouldn’t concern ourselves too much with what comes out at the end.

    So, why am I telling you about IPFIX? My colleagues Asumu and Andy have already implemented this in Snabb, and it already exists as a plugin in VPP. What it does is not too tricky and thus it makes for a great example to try to re-implement as a method of learning VPP. We’re able to test performance and see how things work while hacking on something familiar. There are definitely more features it could have and that the stock VPP one probably does implement. The Snabb implementation and our VPP plugin are fairly bare-bones, but they do record some basic information and export it.

    Measurements!

    We’re going to use a server with two network cards connected in a pair. This allows us to attach a Snabb app we wrote to one end which will send packets and have those packets be received by another card by VPP. The network card works by having ring buffer with a read pointer and a write pointer. You advance the read pointer to access the next packets and the network card will advance the write buffer when writing new packets it receives. The card keeps track of how often it finds itself writing over the unread packets in a counter. You can see some of the counters, including the rx missed counter showing how many packets we’re dropping by being too slow. This is what it looks when you ask VPP to show those counters:

    TenGigabitEthernet2/0/0            1     up   TenGigabitEthernet2/0/0
      Ethernet address 00:11:22:33:44:55
      Intel 82599
        carrier up full duplex speed 10000 mtu 9216 
        rx queues 1, rx desc 1024, tx queues 2, tx desc 1024
        cpu socket 0
    
        rx frames ok                                     2398280
        rx bytes ok                                    143896800
        rx missed                                         252816
        extended stats:
          rx good packets                                2398280
          rx good bytes                                143896800
          rx q0packets                                   2398280
          rx q0bytes                                   143896800
          rx size 64 packets                             2658721
          rx total packets                               2658694
          rx total bytes                               159520296
          rx l3 l4 xsum error                            2658762
          rx priority0 dropped                            253924
    local0                             0    down  local0
    

    We need to find the no-drop rate (NDR), which is the highest speed where we can keep up with the load. When the card shows that we’re dropping packets, we know we’re going too fast and need to back off. We’re able to find this speed by performing a binary search. Bisecting the speed and testing testing if we’re dropping packets, if so then we need to go lower, bisecting the remaining speed, otherwise bisect the higher range. This is an example of our snabb app performing the binary search, looking for the NDR:

    Applying 5.000000 Gbps of load.
        TX 7440477 packets (7.440477 MPPS), 446428620 bytes (5.000001 Gbps)
        RX 0 packets (0.000000 MPPS), 0 bytes (0.000000 Gbps)
        Loss: 0 ingress drop + 7440477 packets lost (100.000000%)
    Success.
    Applying 7.500000 Gbps of load.
        TX 11160650 packets (11.160650 MPPS), 669639000 bytes (7.499957 Gbps)
        RX 0 packets (0.000000 MPPS), 0 bytes (0.000000 Gbps)
        Loss: 0 ingress drop + 11160650 packets lost (100.000000%)
    Failed.
    Applying 6.250000 Gbps of load.
        TX 9300576 packets (9.300576 MPPS), 558034560 bytes (6.249987 Gbps)
        RX 0 packets (0.000000 MPPS), 0 bytes (0.000000 Gbps)
        Loss: 0 ingress drop + 9300576 packets lost (100.000000%)
    Failed.
    Applying 5.625000 Gbps of load.
        TX 8370519 packets (8.370519 MPPS), 502231140 bytes (5.624989 Gbps)
        RX 0 packets (0.000000 MPPS), 0 bytes (0.000000 Gbps)
        Loss: 0 ingress drop + 8370519 packets lost (100.000000%)
    Success.
    Applying 5.938000 Gbps of load.
        TX 8836291 packets (8.836291 MPPS), 530177460 bytes (5.937988 Gbps)
        RX 0 packets (0.000000 MPPS), 0 bytes (0.000000 Gbps)
        Loss: 0 ingress drop + 8836291 packets lost (100.000000%)
    Failed.
    Applying 5.782000 Gbps of load.
        TX 8604158 packets (8.604158 MPPS), 516249480 bytes (5.781994 Gbps)
        RX 0 packets (0.000000 MPPS), 0 bytes (0.000000 Gbps)
        Loss: 0 ingress drop + 8604158 packets lost (100.000000%)
    Success.
    Applying 5.860000 Gbps of load.
        TX 8720228 packets (8.720228 MPPS), 523213680 bytes (5.859993 Gbps)
        RX 0 packets (0.000000 MPPS), 0 bytes (0.000000 Gbps)
        Loss: 0 ingress drop + 8720228 packets lost (100.000000%)
    Success.
    Applying 5.899000 Gbps of load.
        TX 8778268 packets (8.778268 MPPS), 526696080 bytes (5.898996 Gbps)
        RX 0 packets (0.000000 MPPS), 0 bytes (0.000000 Gbps)
        Loss: 0 ingress drop + 8778268 packets lost (100.000000%)
    Success.
    Applying 5.919000 Gbps of load.
        TX 8808026 packets (8.808026 MPPS), 528481560 bytes (5.918993 Gbps)
        RX 0 packets (0.000000 MPPS), 0 bytes (0.000000 Gbps)
        Loss: 0 ingress drop + 8808026 packets lost (100.000000%)
    Failed.
    Applying 5.909000 Gbps of load.
        TX 8793130 packets (8.793130 MPPS), 527587800 bytes (5.908983 Gbps)
        RX 0 packets (0.000000 MPPS), 0 bytes (0.000000 Gbps)
        Loss: 0 ingress drop + 8793130 packets lost (100.000000%)
    Failed.
    Applying 5.904000 Gbps of load.
        TX 8785707 packets (8.785707 MPPS), 527142420 bytes (5.903995 Gbps)
        RX 0 packets (0.000000 MPPS), 0 bytes (0.000000 Gbps)
        Loss: 0 ingress drop + 8785707 packets lost (100.000000%)
    Success.
    Applying 5.907000 Gbps of load.
        TX 8790150 packets (8.790150 MPPS), 527409000 bytes (5.906981 Gbps)
        RX 0 packets (0.000000 MPPS), 0 bytes (0.000000 Gbps)
        Loss: 0 ingress drop + 8790150 packets lost (100.000000%)
    Failed.
    Applying 5.906000 Gbps of load.                                                                                                                                                                                                   
        TX 8788687 packets (8.788687 MPPS), 527321220 bytes (5.905998 Gbps)
        RX 0 packets (0.000000 MPPS), 0 bytes (0.000000 Gbps)
        Loss: 0 ingress drop + 8788687 packets lost (100.000000%)
    Failed.
    Applying 5.905000 Gbps of load.
        TX 8787182 packets (8.787182 MPPS), 527230920 bytes (5.904986 Gbps)
        RX 0 packets (0.000000 MPPS), 0 bytes (0.000000 Gbps)
        Loss: 0 ingress drop + 8787182 packets lost (100.000000%)
    Success.
    5.905

    The output shows applying a specific load and the success or failed state is based on if the rx missed counter from VPP is showing packets dropped or not. If we apply a load and it fails then we attempt it three times. The output of the failed retries has been removed for brevity.

    The sample data is a test data which includes a lot of flows, we have 6 different sample data sets which we test three times to be able to avarage the data. The sample datasets range from the smallest possible ethernet frame size, 64 bytes to the largest standard ethernet frame size of 1522 bytes.

    Reaching peek performance

    The initial results without any performance tweaking on our VPP plugin are:

     

    The first thing that grabbed my interest was that we weren’t getting to full speed with larger packets, something that shouldn’t be too tricky. As it turns out we had added some ELOG statements, a light-weight method of logging in VPP. These are pretty useful for getting a good look inside your function so that you can figure out what happens and when. Turns out they’re not as lightweight as I’d have liked. Removing those allowed us to get better performance across the board, including reaching line-rate (10 Gigabit per second) for frame sizes of 200 and over.

    Then my attention turned to the 64 and 100 frame sizes. I was wondering how long we were actually taking on average for each packet and how it compared to the stock VPP plugin. It turned out that we’re taking around 131 nanoseconds when the stock one takes around 67 nanoseconds. That’s a big difference, and for those who read my first networking post, you’ll remember I mentioned that you have around 67.2 nanoseconds per packet. We were definitely exceeding our budget. Taking a bit of a closer look, it’s taking us around 103 ns just to look up in our flow table whether a record exists or if it’s a new flow.

    I decided to look at what the stock plugin does. VPP has a lot of different data-types built in, and they had decided to go down a different route than us. They’re using a vector where the hash of the flow key is the index and the value is an index into a pool. VPP pools are fast blocks of memory to fit fixed sized stuff into, you add things and get back an index and use that to fetch it. We’re using a bihash which is a bounded-index hash map, it’s pretty handy as we just have the key as the flow key and then backing it the flow entry containing for example: amount of packets, bytes, and the start and end times for the flow. We had a similar problem in Snabb where our hashing algorithm was slow for the large key. I thought maybe the same thing was occurring but in actuality, the stock VPP was using the same xxhash algorithm that bihashes use by default and that was getting good performance. Not that then.

    The bi-hash works by having a vector of buckets, the amount of which are specified when you initialise it. When you add an item it selects the bucket based on the hash. Each bucket maintains it’s own cache able to fit a specific number of bi-hash entries (both key and value). The size of the cache is based a constant, defined for various key and value sized bi-hashes. The more buckets you have, the fewer keys per bucket and thus, more caches per bi-hash entries. The entries are stored on the heap in pages which the bucket manages. When an item isn’t in the cache of the bucket then, it performs a linear search over the allocated pages until it’s found. Here’s what the bucket data structure looks like:

    typedef struct
    {
      union
      {
        struct
        {
          u32 offset;
          u8 linear_search;
          u8 log2_pages;
          u16 cache_lru;
        };
        u64 as_u64;
      };
      BVT (clib_bihash_kv) cache[BIHASH_KVP_CACHE_SIZE];
    } BVT (clib_bihash_bucket);

    We were only defining 1024 buckets which meant that we missing the cache a lot and falling back to slow linear searches. Increasing this value up to 32768, we start getting much better performance:

    Now we seem to be outperforming both the stock VPP plugin and Snabb. The improvements over Snabb could be due to a number of things, however VPP does something rather interesting which may give it the edge. In VPP there is a loop which can handle two packets at once, it’s identical to the single packet loop but apparently operating on two packets at a time provides performance improvements. At a talk in FOSDEM they claimed it helped, and in some parts of VPP they have loops which handle 4 packets per iteration. I think that in the future exploring why we’re seeing the VPP plugin perform better compared to Snabb would be an interesting endeavour.

    by tsyesika at May 16, 2018 09:55 AM

    May 07, 2018

    Iago Toral

    Intel Mesa Vulkan driver now supports shaderInt16

    The Vulkan specification includes a number of optional features that drivers may or may not support, as described in chapter 30.1 Features. Application developers can query the driver for supported features via vkGetPhysicalDeviceFeatures() and then activate the subset they need in the pEnabledFeatures field of the VkDeviceCreateInfo structure passed at device creation time.

    In the last few weeks I have been spending some time, together with my colleague Chema, adding support for one of these features in Anvil, the Intel Vulkan driver in Mesa, called shaderInt16, which we landed in Mesa master last week. This is an optional feature available since Vulkan 1.0. From the spec:

    shaderInt16 specifies whether 16-bit integers (signed and unsigned) are supported in shader code. If this feature is not enabled, 16-bit integer types must not be used in shader code. This also specifies whether shader modules can declare the Int16 capability.

    It is probably relevant to highlight that this Vulkan capability also requires the SPIR-V Int16 capability, which basically means that the driver’s SPIR-V compiler backend can actually digest SPIR-V shaders that declare and use 16-bit integers, and which is really the core of the functionality exposed by the Vulkan feature.

    Ideally, shaderInt16 would increase the overall throughput of integer operations in shaders, leading to better performance when you don’t need a full 32-bit range. It may also provide better overall register usage since you need less register space to store your integer data during shader execution. It is important to remark, however, that not all hardware platforms (Intel or otherwise) may have native support for all possible types of 16-bit operations, and thus, some of them might still need to run in 32-bit (which requires injecting type conversion instructions in the shader code). For Intel platforms, this is the case for operations associated with integer division.

    From the point of view of the driver, this is the first time that we generally exercise lower bit-size data types in the driver compiler backend, so if you find any bugs in the implementation, please file bug reports in bugzilla!

    Speaking of shaderInt16, I think it is worth mentioning its interactions with other Vulkan functionality that we implemented in the past: the Vulkan 1.0 VK_KHR_16bit_storage extension (which has been promoted to core in Vulkan 1.1). From the spec:

    The VK_KHR_16bit_storage extension allows use of 16-bit types in shader input and output interfaces, and push constant blocks. This extension introduces several new optional features which map to SPIR-V capabilities and allow access to 16-bit data in Block-decorated objects in the Uniform and the StorageBuffer storage classes, and objects in the PushConstant storage class. This extension allows 16-bit variables to be declared and used as user-defined shader inputs and outputs but does not change location assignment and component assignment rules.

    While the shaderInt16 capability provides the means to operate with 16-bit integers inside a shader, the VK_KHR_16bit_storage extension provides developers with the means to also feed shaders with 16-bit integer (and also floating point) input data, such as Uniform/Storage Buffer Objects or Push Constants, from the applications side, plus, it also gives the opportunity for linked shader stages in a graphics pipeline to consume 16-bit shader inputs and produce 16-bit shader outputs.

    VK_KHR_16bit_storage and shaderInt16 should be seen as two parts of a whole, each one addressing one part of a larger problem: VK_KHR_16bit_storage can help reduce memory bandwith for Uniform and Storage Buffer data accessed from the shaders and / or optimize Push Constant space, of which there are only a few bytes available, making it a precious shader resource, but without shaderInt16, shaders that are fed 16-bit input data are still required to convert this data to 32-bit internally for operation (and then back again to 16-bit for output if needed). Likewise, shaders that use shaderInt16 without VK_KHR_16bit_storage can only operate with 16-bit data that is generated inside the shader, which largely limits its usage. Both together, however, give you the complete functionality.

    Conclusions

    We are very happy to continue expanding the feature set supported in Anvil and we look forward to seeing application developers making good use of shaderInt16 in Vulkan to improve shader performance. As noted above, this is the first time that we fully enable the shader compiler backend to do general purpose operations on lower bit-size data types and there might be things that we can still improve or optimize. If you hit any issues with the implementation, please contact us and / or file bug reports so we can continue to improve the implementation.

    by Iago Toral at May 07, 2018 08:39 AM

    April 30, 2018

    Samuel Iglesias

    Ubucon Europe 2018 experience and talk slides!

    Last weekend I attended Ubucon Europe 2018 in the city center of Gijón, Spain, at the Antiguo Instituto. This 3-day conference was full of talks, sometimes 3 or even 4 at the same time! The covered topics went from Ubuntu LoCo teams to Industrial Hardware… there was at least one talk interesting for all attendees! Please check the schedule if you want to know what happened at Ubucon ;-)

    Ubucon Europe 2018

    I would like to say thank you to all the organization team. They did a great job taking care of both the attendees and the speakers, they organized a very nice traditional espicha that introduced good cider and the Asturian gastronomy to the people coming from any part of the world. The conference was amazing and I am very pleased to have participated in this event. I recommend you to attend it next year… lots of fun and open-source!

    Regarding my talk “Introduction to Mesa, an open-source graphics API”, there were 30 people attending to it. It was great to check that so many people were interested on a brief introduction about how Mesa works, how the community collaborate together and how anyone can become contributor to Mesa, even as a non-developer. I hope I can see them contributing to Mesa in the coming future!

    You can find the slides of my talk here.

    Ubucon Europe 2018 talk

    April 30, 2018 12:00 PM

    April 25, 2018

    Frédéric Wang

    April 17, 2018

    Víctor Jáquez

    How to setup a gst-build environment with Intel’s VA-API stack

    gst-build is far superior than gst-uninstalled scripts for developing GStreamer, mainly because its meson and ninja usage. Nonetheless, to integrate external dependencies it is not as easy as in gst-uninstalled.

    This guide aims to show how to integrate GStreamer-VAAPI dependencies, in this case with the Intel VA-API driver.

    Dependencies

    For now we will need meson master, since it this patch is required. The pull request is already merged but it is unreleased yet.

    Installation

    clone gst-build repository

    $ git clone git://anongit.freedesktop.org/gstreamer/gst-build
    $ cd gst-build
    

    apply custom patch for gst-build

    The patch will add the repositories for libva and intel-vaapi-driver.

    $ wget https://people.igalia.com/vjaquez/gst-build-vaapi/0001-add-libva-and-intel-vaapi-driver-as-subprojects.patch
    $ git am 0001-add-libva-and-intel-vaapi-driver-as-subprojects.patch
    

    0001-add-libva-and-intel-vaapi-driver-as-subprojects.patch

    configure

    Running this command, all dependency repositories will be cloned, symbolic links created, and the build directory configured.

    $ meson bulid
    

    apply custom patches for libva, intel-vaapi-driver and gstreamer-vaapi

    libva

    This patch is required since the headers files uninstalled paths doesn’t match with the ones in the “include” directives.

    $ cd libva
    $ wget https://people.igalia.com/vjaquez/gst-build-vaapi/0001-build-add-headers-for-uninstalled-setup.patch
    $ git am 0001-build-add-headers-for-uninstalled-setup.patch
    $ cd -
    

    0001-build-add-headers-for-uninstalled-setup.patch

    intel-vaapi-driver

    The patch handles libva dependency as a subproject.

    $ cd intel-vaapi-driver
    $ wget https://people.igalia.com/vjaquez/gst-build-vaapi/0001-meson-support-libva-as-subproject.patch
    $ git am 0001-meson-support-libva-as-subproject.patch
    $ cd -
    

    0001-meson-support-libva-as-subproject.patch

    gstreamer-vaapi

    Note to myself: this patch must be split and merged in upstream.

    $ cd gstreamer-vaapi
    $ wget https://people.igalia.com/vjaquez/gst-build-vaapi/0001-build-meson-libva-gst-uninstall-friendly.patch
    $ git am 0001-build-meson-libva-gst-uninstall-friendly.patch
    $ cd -
    

    0001-build-meson-libva-gst-uninstall-friendly.patch updated: 2018/04/24

    build

    $ ninja -C build
    

    And wait a couple minutes.

    run uninstalled environment for testing

    $ ninja -C build uninstalled
    [gst-master] $ gst-inspect-1.0 vaapi
    Plugin Details:
      Name                     vaapi
      Description              VA-API based elements
      Filename                 /opt/gst/gst-build/build/subprojects/gstreamer-vaapi/gst/vaapi/libgstvaapi.so
      Version                  1.15.0.1
      License                  LGPL
      Source module            gstreamer-vaapi
      Binary package           gstreamer-vaapi
      Origin URL               http://bugzilla.gnome.org/enter_bug.cgi?product=GStreamer
    
      vaapih264enc: VA-API H264 encoder
      vaapimpeg2enc: VA-API MPEG-2 encoder
      vaapisink: VA-API sink
      vaapidecodebin: VA-API Decode Bin
      vaapipostproc: VA-API video postprocessing
      vaapivc1dec: VA-API VC1 decoder
      vaapih264dec: VA-API H264 decoder
      vaapimpeg2dec: VA-API MPEG2 decoder
      vaapijpegdec: VA-API JPEG decoder
    
      9 features:
      +-- 9 elements
    
    

    by vjaquez at April 17, 2018 02:43 PM

    Iago Toral

    Frame analysis of a rendering of the Sponza model

    For some time now I have been working on a personal project to render the well known Sponza model provided by Crytek using Vulkan. Here is a picture of the current (still a work-in-progress) result:


    Sponza rendering

    This screenshot was captured on my Intel Kabylake laptop, running on the Intel Mesa Vulkan driver (Anvil).

    The following list includes the main features implemented in the demo:

    • Depth pre-pass
    • Forward and deferred rendering paths
    • Anisotropic filtering
    • Shadow mapping with Percentage-Closer Filtering
    • Bump mapping
    • Screen Space Ambient Occlusion (only on the deferred path)
    • Screen Space Reflections (only on the deferred path)
    • Tone mapping
    • Anti-aliasing (FXAA)

    I have been thinking about writing post about this for some time, but given that there are multiple features involved I wasn’t sure how to scope it. Eventually I decided to write a “frame analysis” post where I describe, step by step, all the render passes involved in the production of the single frame capture showed at the top of the post. I always enjoyed reading this kind of articles so I figured it would be fun to write one myself and I hope others find it informative, if not entertaining.

    To avoid making the post too dense I won’t go into too much detail while describing each render pass, so don’t expect me to go into the nitty-gritty of how I implemented Screen Space Ambient Occlussion for example. Instead I intend to give a high-level overview of how the various features implemented in the demo work together to create the final result. I will provide screenshots so that readers can appreciate the outputs of each step and verify how detail and quality build up over time as we include more features in the pipeline. Those who are more interested in the programming details of particular features can always have a look at the Vulkan source code (link available at the bottom of the article), look for specific tutorials available on the Internet or wait for me to write feature-specifc posts (I don’t make any promises though!).

    If you’re interested in going through with this then grab a cup of coffe and get ready, it is going to be a long ride!

    Step 0: Culling

    This is the only step in this discussion that runs on the CPU, and while optional from the point of view of the result (it doesn’t affect the actual result of the rendering), it is relevant from a performance point of view. Prior to rendering anything, in every frame, we usually want to cull meshes that are not visible to the camera. This can greatly help performance, even on a relatively simple scene such as this. This is of course more noticeable when the camera is looking in a direction in which a significant amount of geometry is not visible to it, but in general, there are always parts of the scene that are not visible to the camera, so culling is usually going to give you a performance bonus.

    In large, complex scenes with tons of objects we probably want to use more sophisticated culling methods such as Quadtrees, but in this case, since the number of meshes is not too high (the Sponza model is slightly shy of 400 meshes), we just go though all of them and cull them individually against the camera’s frustum, which determines the area of the 3D space that is visible to the camera.

    The way culling works is simple: for each mesh we compute an axis-aligned bounding box and we test that box for intersection with the camera’s frustum. If we can determine that the box never intersects, then the mesh enclosed within it is not visible and we flag it as such. Later on, at rendering time (or rather, at command recording time, since the demo has been written in Vulkan) we just skip the meshes that have been flagged.

    The algorithm is not perfect, since it is possible that an axis-aligned bounding box for a particular mesh is visible to the camera and yet no part of the mesh itself is visible, but it should not affect a lot of meshes and trying to improve this would incur in additional checks that could undermine the efficiency of the process anyway.

    Since in this particular demo we only have static geometry we only need to run the culling pass when the camera moves around, since otherwise the list of visible meshes doesn’t change. If dynamic geometry were present, we would need to at least cull dynamic geometry on every frame even if the camera stayed static, since dynamic elements may step in (or out of) the viewing frustum at any moment.

    Step 1: Depth pre-pass

    This is an optional stage, but it can help performance significantly in many cases. The idea is the following: our GPU performance is usually going to be limited by the fragment shader, and very specially so as we target higher resolutions. In this context, without a depth pre-pass, we are very likely going to execute the fragment shader for fragments that will not end up in the screen because they are occluded by fragments produced by other geometry in the scene that will be rasterized to the same XY screen-space coordinates but with a smaller Z coordinate (closer to the camera). This wastes precious GPU resources.

    One way to improve the situation is to sort our geometry by distance from the camera and render front to back. With this we can get fragments that are rasterized from background geometry quickly discarded by early depth tests before the fragment shader runs for them. Unfortunately, although this will certainly help (assuming we can spare the extra CPU work to keep our geometry sorted for every frame), it won’t eliminate all the instances of the problem in the general case.

    Also, some times things are more complicated, as the shading cost of different pieces of geometry can be very different and we should also take this into account. For example, we can have a very large piece of geometry for which some pixels are very close to the camera while some others are very far away and that has a very expensive shader. If our renderer is doing front-to-back rendering without any other considerations it will likely render this geometry early (since parts of it are very close to the camera), which means that it will shade all or most of its very expensive fragments. However, if the renderer accounts for the relative cost of the shader execution it would probably postpone rendering it as much as possible, so by the time it actually renders it, it takes advantage of early fragment depth tests to avoid as many of its expensive fragment shader executions as possible.

    Using a depth-prepass ensures that we only run our fragment shader for visible fragments, and only those, no matter the situation. The downside is that we have to execute a separate rendering pass where we render our geometry to the depth buffer so that we can identify the visible fragments. This pass is usually very fast though, since we don’t even need a fragment shader and we are only writing to a depth texture. The exception to this rule is geometry that has opacity information, such as opacity textures, in which case we need to run a cheap fragment shader to identify transparent pixels and discard them so they don’t hit the depth buffer. In the Sponza model we need to do that for the flowers or the vines on the columns for example.

    Depth pre-pass output

    The picture shows the output of the depth pre-pass. Darker colors mean smaller distance from the camera. That’s why the picture gets brighter as we move further away.

    Now, the remaining passes will be able to use this information to limit their shading to fragments that, for a given XY screen-space position, match exactly the Z value stored in the depth buffer, effectively selecting only the fragments that will be visible in the screen. We do this by configuring the depth test to do an EQUAL test instead of the usual LESS test, which is what we use in the depth-prepass.

    In this particular demo, running on my Intel GPU, the depth pre-pass is by far the cheapest of all the GPU passes and it definitely pays off in terms of overall performance output.

    Step 2: Shadow map

    In this demo we have single source of light produced by a directional light that simulates the sun. You can probably guess the direction of the light by checking out the picture at the top of this post and looking at the direction projected shadows.

    I already covered how shadow mapping works in previous series of posts, so if you’re interested in the programming details I encourage you to read that. Anyway, the basic idea is that we want to capture the scene from the point of view of the light source (to be more precise, we want to capture the objects in the scene that can potentially produce shadows that are visible to our camera).

    With that information, we will be able to inform out lighting pass so it can tell if a particular fragment is in the shadows (not visible from our light’s perspective) or in the light (visible from our light’s perspective) and shade it accordingly.

    From a technical point of view, recording a shadow map is exactly the same as the depth-prepass: we basically do a depth-only rendering and capture the result in a depth texture. The main differences here are that we need to render from the point of view of the light instead of our camera’s and that this being a directional light, we need to use an orthographic projection and adjust it properly so we capture all relevant shadow casters around the camera.

    Shadow map

    In the image above we can see the shadow map generated for this frame. Again, the brighter the color, the further away the fragment is from the light source. The bright white area outside the atrium building represents the part of the scene that is empty and thus ends with the maximum depth, which is what we use to clear the shadow map before rendering to it.

    In this case, we are using a 4096×4096 texture to store the shadow map image, much larger than our rendering target. This is because shadow mapping from directional lights needs a lot of precision to produce good results, otherwise we end up with very pixelated / blocky shadows, more artifacts and even missing shadows for small geometry. To illustrate this better here is the same rendering of the Sponza model from the top of this post, but using a 1024×1024 shadow map (floor reflections are disabled, but that is irrelevant to shadow mapping):

    Sponza rendering with 1024×1024 shadow map

    You can see how in the 1024×1024 version there are some missing shadows for the vines on the columns and generally blurrier shadows (when not also slightly distorted) everywhere else.

    Step 3: GBuffer

    In deferred rendering we capture various attributes of the fragments produced by rasterizing our geometry and write them to separate textures that we will use to inform the lighting pass later on (and possibly other passes).

    What we do here is to render our geometry normally, like we did in our depth-prepass, but this time, as we explained before, we configure the depth test to only pass fragments that match the contents of the depth-buffer that we produced in the depth-prepass, so we only process fragments that we now will be visible on the screen.

    Deferred rendering uses multiple render targets to capture each of these attributes to a different texture for each rasterized fragment that passes the depth test. In this particular demo our GBuffer captures:

    1. Normal vector
    2. Diffuse color
    3. Specular color
    4. Position of the fragment from the point of view of the light (for shadow mapping)

    It is important to be very careful when defining what we store in the GBuffer: since we are rendering to multiple screen-sized textures, this pass has serious bandwidth requirements and therefore, we should use texture formats that give us the range and precision we need with the smallest pixel size requirements and avoid storing information that we can get or compute efficiently through other means. This is particularly relevant for integrated GPUs that don’t have dedicated video memory (such as my Intel GPU).

    In the demo, I do lighting in view-space (that is the coordinate space used takes the camera as its origin), so I need to work with positions and vectors in this coordinate space. One of the parameters we need for lighting is surface normals, which are conveniently stored in the GBuffer, but we will also need to know the view-space position of the fragments in the screen. To avoid storing the latter in the GBuffer we take advantage of the fact that we can reconstruct the view-space position of any fragment on the screen from its depth (which is stored in the depth buffer we rendered during the depth-prepass) and the camera’s projection matrix. I might cover the process in more detail in another post, for now, what is important to remember is that we don’t need to worry about storing fragment positions in the GBuffer and that saves us some bandwidth, helping performance.

    Let’s have a look at the various GBuffer textures we produce in this stage:

    Normal vectors

    GBuffer normal texture

    Here we see the normalized normal vectors for each fragment in view-space. This means they are expressed in a coordinate space in which our camera is at the origin and the positive Z direction is opposite to the camera’s view vector. Therefore, we see that surfaces pointing to the right of our camera are red (positive X), those pointing up are green (positive Y) and those pointing opposite to the camera’s view direction are blue (positive Z).

    It should be mentioned that some of these surfaces use normal maps for bump mapping. These normal maps are textures that provide per-fragment normal information instead of the usual vertex normals that come with the polygon meshes. This means that instead of computing per-fragment normals as a simple interpolation of the per-vertex normals across the polygon faces, which gives us a rather flat result, we use a texture to adjust the normal for each fragment in the surface, which enables the lighting pass to render more nuanced surfaces that seem to have a lot more volume and detail than they would have otherwise.

    For comparison, here is the GBuffer normal texture without bump mapping enabled. The difference in surface detail should be obvious. Just look at the lion figure at the far end or the columns and and you will immediately notice the addditional detail added with bump mapping to the surface descriptions:

    GBuffer normal texture (bump mapping disabled)

    To make the impact of the bump mapping more obvious, here is a different shot of the final rendering focusing on the columns of the upper floor of the atrium, with and without bump mapping:

    Bump mapping enabled
    Bump mapping disabled

    All the extra detail in the columns is the sole result of the bump mapping technique.

    Diffuse color

    GBuffer diffuse texture

    Here we have the diffuse color of each fragment in the scene. This is basically how our scene would look like if we didn’t implement a lighting pass that considers how the light source interacts with the scene.

    Naturally, we will use this information in the lighting pass to modulate the color output based on the light interaction with each fragment.

    Specular color

    GBuffer specular texture

    This is similar to the diffuse texture, but here we are storing the color (and strength) used to compute specular reflections.

    Similarly to normal textures, we use specular maps to obtain per-fragment specular colors and intensities. This allows us to simulate combinations of more complex materials in the same mesh by specifying different specular properties for each fragment.

    For example, if we look at the cloths that hang from the upper floor of the atrium, we see that they are mostly black, meaning that they barely produce any specular reflection, as it is to be expected from textile materials. However, we also see that these same cloths have an embroidery that has specular reflection (showing up as a light gray color), which means these details in the texture have stronger specular reflections than its surrounding textile material:

    Specular reflection on cloth embroidery

    The image shows visible specular reflections in the yellow embroidery decorations of the cloth (on the bottom-left) that are not present in the textile segment (the blue region of the cloth).

    Fragment positions from Light

    GBuffer light-space position texture

    Finally, we store fragment positions in the coordinate space of the light source so we can implement shadows in the lighting pass. This image may be less intuitive to interpret, since it is encoding space positions from the point of view of the sun rather than physical properties of the fragments. We will need to retrieve this information for each fragment during the lighting pass so that we can tell, together with the shadow map, which fragments are visible from the light source (and therefore are directly lit by the sun) and which are not (and therefore are in the shadows). Again, more detail on how that process works, step by step and including Vulkan source code in my series of posts on that topic.

    Step 4: Screen Space Ambient Occlusion

    With the information stored in the GBuffer we can now also run a screen-space ambient occlusion pass that we will use to improve our lighting pass later on.

    The idea here, as I discussed in my lighting and shadows series, the Phong lighting model simplifies ambient lighting by making it constant across the scene. As a consequence of this, lighting in areas that are not directly lit by a light source look rather flat, as we can see in this image:

    SSAO disabled

    Screen-space Ambient Occlusion is a technique that gathers information about the amount of ambient light occlusion produced by nearby geometry as a way to better estimate the ambient light term of the lighting equations. We can then use that information in our lighting pass to modulate ambient light accordingly, which can greatly improve the sense of depth and volume in the scene, specially in areas that are not directly lit:

    SSAO enabled

    Comparing the images above should illustrate the benefits of the SSAO technique. For example, look at the folds in the blue curtains on the right side of the images, without SSAO, we barely see them because the lighting is too flat across all the pixels in the curtain. Similarly, thanks to SSAO we can create shadowed areas from ambient light alone, as we can see behind the cloths that hang from the upper floor of the atrium or behind the vines on the columns.

    To produce this result, the output of the SSAO pass is a texture with ambient light intensity information that looks like this (after some blur post-processing to eliminate noise artifacts):

    SSAO output texture

    In that image, white tones represent strong light intensity and black tones represent low light intensity produced by occlusion from nearby geometry. In our lighting pass we will source from this texture to obtain per-fragment ambient occlusion information and modulate the ambient term accordingly, bringing the additional volume showcased in the image above to the final rendering.

    Step 6: Lighting pass

    Finally, we get to the lighting pass. Most of what we showcased above was preparation work for this.

    The lighting pass mostly goes as I described in my lighting and shadows series, only that since we are doing deferred rendering we get our per-fragment lighting inputs by reading from the GBuffer textures instead of getting them from the vertex shader.

    Basically, the process involves retrieving diffuse, ambient and specular color information from the GBuffer and use it as input for the lighting equations to produce the final color for each fragment. We also sample from the shadow map to decide which pixels are in the shadows, in which case we remove their diffuse and specular components, making them darker and producing shadows in the image as a result.

    We also use the SSAO output to improve the ambient light term as described before, multipliying the ambient term of each fragment by the SSAO value we computed for it, reducing the strength of the ambient light for pixels that are surrounded by nearby geometry.

    The lighting pass is also where we put bump mapping to use. Bump mapping provides more detailed information about surface normals, which the lighting pass uses to simulate more complex lighting interactions with mesh surfaces, producing significantly enhanced results, as I showcased earlier in this post.

    After combining all this information, the lighting pass produces an output like this. Compare it with the GBuffer diffuse texture to see all the stuff that this pass is putting together:

    Lighting pass output

    Step 7: Tone mapping

    After the lighting pass we run a number of post-processing passes, of which tone mapping is the first one. The idea behind tone mapping is this: normally, shader color outputs are limited to the range [0, 1], which puts a hard cap on our lighting calculations. Specifically, it means that when our light contributions to a particular pixel go beyond 1.0 in any color component, they get clamped, which can distort the resulting color in unrealistic ways, specially when this happens during intermediate lighting calculations (since the deviation from the physically correct color is then used as input to more computations, which then build on that error).

    To work around this we do our lighting calculations in High Dynamic Range (HDR) which allows us to produce color values with components larger than 1.0, and then we run a tone mapping pass to re-map the result to the [0, 1] range when we are done with the lighting calculations and we are ready for display.

    The nice thing about tone mapping is that it gives the developer control over how that mapping happens, allowing us to decide if we are interested in preserving more detail in the darker or brighter areas of the scene.

    In this particular demo, I used HDR rendering to ramp up the intensity of the sun light beyond what I could have represented otherwise. Without tone mapping this would lead to unrealistic lighting in areas with strong light reflections, since would exceed the 1.0 per-color-component cap and lead to pure white colors as result, losing the color detail from the original textures. This effect can be observed in the following pictures if you look at the lit area of the floor. Notice how the tone-mapped picture better retains the detail of the floor texture while in the non tone-mapped version the floor seems to be over-exposed to light and large parts of it just become white as a result (shadow mapping has been disabled to better showcase the effects of tone-mapping on the floor):

    Tone mapping disabled
    Tone mapping enabled

    Step 8: Screen Space Reflections (SSR)

    The material used to render the floor is reflective, which means that we can see the reflections of the surrounding environment on it.

    There are various ways to capture reflections, each with their own set of pros and cons. When I implemented my OpenGL terrain rendering demo I implemented water reflections using “Planar Reflections”, which produce very accurate results at the expense of requiring to re-render the scene with the camera facing in the same direction as the reflection. Although this can be done at a lower resolution, it is still quite expensive and cumbersome to setup (for example, you would need to run an additional culling pass), and you also need to consider that we need to do this for each planar surface you want to apply reflections on, so it doesn’t scale very well. In this demo, although it is not visible in the reference screenshot, I am capturing reflections from the floor sections of both stories of the atrium, so the Planar Reflections approach might have required me to render twice when fragments of both sections are visible (admittedly, not very often, but not impossible with the free camera).

    So in this particular case I decided to experiment with a different technique that has become quite popular, despite its many shortcomings, because it is a lot faster: Screen Space Reflections.

    As all screen-space techniques, the technique uses information already present in the screen to capture the reflection information, so we don’t have to render again from a different perspective. This leads to a number of limitations that can produce fairly visible artifacts, specially when there is dynamic geometry involved. Nevertheless, in my particular case I don’t have any dynamic geometry, at least not yet, so while the artifacts are there they are not quite as distracting. I won’t go into the details of the artifacts introduced with SSR here, but for those interested, here is a good discussion.

    I should mention that my take on this is fairly basic and doesn’t implement relevant features such as the Hierarchical Z Buffer optimization (HZB) discussed here.

    The technique has 3 steps: capturing reflections, applying roughness material properties and alpha blending:

    Capturing reflections

    I only implemented support for SSR in the deferred path, since like in the case of SSAO (and more generally all screen-space algorithms), deferred rendering is the best match since we are already capturing screen-space information in the GBuffer.

    The first stage for this requires to have means to identify fragments that need reflection information. In our case, the floor fragments. What I did for this is to capture the reflectiveness of the material of each fragment in the screen during the GBuffer pass. This is a single floating-point component (in the 0-1 range). A value of 0 means that the material is not reflective and the SSR pass will just ignore it. A value of 1 means that the fragment is 100% reflective, so its color value will be solely the reflection color. Values in between allow us to control the strength of the reflection for each fragment with a reflective material in the scene.

    One small note on the GBuffer storage: because this is a single floating-point value, we don’t necessarily need an extra attachment in the GBuffer (which would have some performance penalty), instead we can just put this in the alpha component of the diffuse color, since we were not using it (the Intel Mesa driver doesn’t support rendering to RGB textures yet, so since we are limited to RGBA we might as well put it to good use).

    Besides capturing which fragments are reflective, we can also store another piece of information relevant to the reflection computations: the material’s roughness. This is another scalar value indicating how much blurring we want to apply to the resulting reflection: smooth metal-like surfaces can have very sharp reflections but with rougher materials that have not smooth surfaces we may want the reflections to look a bit blurry, to better represent these imperfections.

    Besides the reflection and roughness information, to capture screen-space reflections we will need access to the output of the previous pass (tone mapping) from which we will retrieve the color information of our reflection points, the normals that we stored in the GBuffer (to compute reflection directions for each fragment in the floor sections) and the depth buffer (from the depth-prepass), so we can check for reflection collisions.

    The technique goes like this: for each fragment that is reflective, we compute the direction of the reflection using its normal (from the GBuffer) and the view vector (from the camera and the fragment position). Once we have this direction, we execute a ray marching from the fragment position, in the direction of the reflection. For each point we generate, we take the screen-space X and Y coordinates and use them to retrieve the Z-buffer depth for that pixel in the scene. If the depth buffer value is smaller than our sample’s it means that we have moved past foreground geometry and we stop the process. If we got to this point, then we can do a binary search to pin-point the exact location where the collision with the foreground geometry happens, which will give us the screen-space X and Y coordinates of the reflection point. Once we have that we only need to sample the original scene (the output from the tone mapping pass) at that location to retrieve the reflection color.

    As discussed earlier, the technique has numerous caveats, which we need to address in one way or another and maybe adapt to the characteristics of different scenes so we can obtain the best results in each case.

    The output of this pass is a color texture where we store the reflection colors for each fragment that has a reflective material:

    Reflection texture

    Naturally, the image above only shows reflection data for the pixels in the floor, since those are the only ones with a reflective material attached. It is immediately obvious that some pixels lack reflection color though, this is due to the various limitations of the screen-space technique that are discussed in the blog post I linked above.

    Because the reflections will be alpha-blended with the original image, we use the reflectiveness that we stored in the GBuffer as the base for the alpha component of the reflection color as well (there are other aspects that can contribute to the alpha component too, but I won’t go into that here), so the image above, although not visible in the screenshot, has a valid alpha channel.

    Considering material roughness

    Once we have captured the reflection image, the next step is to apply the material roughness settings. We can accomplish this with a simple box filter based on the roughness of each fragment: the larger the roughness, the larger the box filter we apply and the blurrier the reflection we get as a result. Because we store roughness for each fragment in the GBuffer, we can have multiple reflective materials with different roughness settings if we want. In this case, we just have one material for the floor though.

    Alpha blending

    Finally, we use alpha blending to incorporate the reflection onto the original image (the output from the tone mapping) ot incorporate the reflections to the final rendering:

    SSR output

    Step 9: Anti-aliasing (FXAA)

    So far we have been neglecting anti-aliasing. Because we are doing deferred rendering Multi-Sample Anti-Aliasing (MSAA) is not an option: MSAA happens at rasterization time, which in a deferred renderer occurs before our lighting pass (specifically, when we generate the GBuffer), so it cannot account for the important effects that the lighting pass has on the resulting image, and therefore, on the eventual aliasing that we need to correct. This is why deferred renderers usually do anti-aliasing via post-processing.

    In this demo I have implemented a well-known anti-aliasing post-processing pass known as Fast Approximate Anti Aliasing (FXAA). The technique attempts to identify strong contrast across neighboring pixels in the image to identify edges and then smooth them out using linear filtering. Here is the final result which matches the one I included as reference at the top of this post:

    Anti-aliased output

    The image above shows the results of the anti-aliasing pass. Compare that with the output of the SSR pass. You can see how this pass has effectively removed the jaggies observed in the cloths hanging from the upper floor for example.

    Unlike MSAA, which acts on geometry edges only, FXAA works on all pixels, so it can also smooth out edges produced by shaders or textures. Whether that is something we want to do or not may depend on the scene. Here we can see this happening on the foreground column on the left, where some of the imperfections of the stone are slightly smoothed out by the FXAA pass.

    Conclusions and source code

    So that’s all, congratulations if you managed to read this far! In the past I have found articles that did frame analysis like this quite interesting so it’s been fun writing one myself and I only hope that this was interesting to someone else.

    This demo has been implemented in Vulkan and includes a number of configurable parameters that can be used to tweak performance and quality. The work-in-progress source code is available here, but beware that I have only tested this on Intel, since that is the only hardware I have available, so you may find issues if you run this on other GPUs. If that happens, let me know in the comments and I might be able to provide fixes at some point.

    by Iago Toral at April 17, 2018 12:45 PM

    April 16, 2018

    Jacobo Aragunde

    Updated Chromium on the GENIVI platform

    I’ve devoted some of my time at Igalia to get a newer version of Chromium running on the GENIVI Development Platform (GDP).

    Since the last update, there have been news regarding Wayland support in Chromium. My colleagues Antonio, Maksim and Frédéric have worked on a new Wayland backend following modern Chromium architecture. You can find more information in their own blogs and talks. I’m linking the most recent talk, from FOSDEM 2018.

    Everyone can already try the new, Igalia-developed backend on their embedded devices using the meta-browser layer. I built it along with the GDP but discovered that it cannot run as it is, due to the lack of ivi-shell hooks in the new Chromium backend. This is going to be fixed in the mid-term, so I decided not to spend a lot of time researching this and chose a different solution for the current GDP release.

    The LG SVL team recently announced the release of an updated Ozone Wayland backend for Chromium, based on the legacy implementation provided by Intel, as a part of the webOS OSE project. This is an update on the backend we were already running on the GDP, so it looked like a good idea to reuse their work.

    I added the meta-lgsvl-browser layer to the GDP, which provides recipes for several Chromium flavors: chromium-lge is the one that builds upon the legacy Wayland backend and currently provides Chromium version 64.

    The chromium-lge browser worked out-of-the-box on Raspberry Pi, but I faced some trouble with the other supported platforms. In the case of ARM64 platforms, we were finding a “relocation overflow” problem. This is something that my colleagues had already detected when trying the new Wayland backend on the R-Car gen. 3 platform, and it can be fixed by enabling compiler optimization flags for binary size.

    In the case of Intel platforms, compilation failed due to a build-system assertion. It looks like Clang’s Control Flow Integrity feature is enabled by default on x64 Linux builds, but we aren’t using the Clang compiler. The solution consists just in disabling this feature, like the upstream meta-browser project was already doing.

    The ongoing work is shared in this pull request. I hope to be able to make it for the next GDP release!

    Finally, this week my colleague Xavi is taking part in the GENIVI All Member Meeting. If you are interested in browsers, make sure you attend his talk, “Wayland Support in Open Source Browsers“, and visit our booth during the Member Showcase! (EDIT: check out the slides in our slideshare page!)

    by Jacobo Aragunde Pérez at April 16, 2018 11:04 AM

    April 15, 2018

    Manuel Rego

    CSSWG F2F Berlin 2018

    Last week I was in Berlin for the CSS Working Group (CSSWG) face-to-face meeting representing Igalia, member of the CSSWG since last year. Igalia has been working on the open web platform for many years, where we help our customers with the implementation of different standards on the open source web engines. Inside the CSSWG we play the implementors role, providing valuable feedback around the specifications we’re working on.

    It was really nice to meet all the folks from the CSSWG there, it’s amazing to be together with such a brilliant group of people in the same room. And it’s lovely to see how easy is to talk with any of them, you all rock!

    CSSWG F2F Berlin 2018 by Rossen Atanassov CSSWG F2F Berlin 2018 by Rossen Atanassov

    This is a brief post about my highlights from there, of course totally subjective and focused on the topics I’m more interested.

    CSS Grid Layout

    We were discussing two issues of the current specification related to the track sizing algorithm and its behavior in particular cases. Some changes will be added in the specification to try to improve them and we’ll need to update the implementations accordingly.

    On top of that, we discussed about the Level 2 of the spec. It’s already defined that this next level will include the following features:

    • The awaited subgrids feature: There was the possibility of allowing subgrids in both axis (dual-axis) or only in one of them (per-axis), note that the per-axis approach covers the dual-axis if you define the subgrid in both axis.

      There are clear uses cases for the per-axis approach but the main doubt was about how hard it’d be to implement. Mats Palmgren from Mozilla posted a comment on the issue explaining that he has just created a prototype for the feature following the per-axis idea, so the CSSWG resolved to remove the dual-axis one from the spec.

    • And aspect-ratio controlled gutters: Regarding this topic, the CSSWG decided to add a new ar unit. We didn’t discuss anything more but we need to decide what we’ll do in the situations where there’s no enough free space to fulfill the requested aspect-ratio, should we ignore it or overflow in that case?

      Talking to Rachel Andrew about the issue, she was not completely sure of what would be the preferred option from the authors point of view. I’ve just added some examples to the issue so we can discuss about them there and gather more feedback, please share your thoughts.

    Tests

    This was a discussion I wanted to have with the CSSWG people in order to understand better the current situation and possible next steps for the CSSWG test suites.

    Just to add some context, the CSSWG test suites are now part of the web-platform-tests (WPT) repository. This repository is being used by most browser vendors to share tests, including tests for new CSS features. For example, at Igalia we’re currently using WPT test suites in all our developments.

    The CSSWG uses the CSS Test Harness tool which has a build system that adds some special requirements for the test suites. One of them causes that we need to duplicate some files in the repository, which is not nice at all.

    Several people in the CSSWG still rely on this tool mainly for two things:

    • Run manual tests and store their results: Some CSS features like media queries or scrolling are hard to automate when writing tests, so several specs have manual tests. Probably WebDriver can help to automate this kind of tests, maybe not all though.
    • Extract status reports: To verify that a spec fulfills the CR exit criteria, the current tooling has some nice reports, it also provides info about the test coverage of the spec.

    So we cannot get rid of the CSS Test Harness system at this point. We discussed about possible solutions but none of them were really clear, also note that the lack of funding for this kind of work makes it harder to move things forward.

    I still believe the way to go would be to improve the WPT Dashboard (wpt.fyi) so it can support the 2 features listed above. If that’s the case maybe the specific CSS Test Harness stuff won’t be needed anymore, thus the weird requirements for people working on the test suites will be gone, and there would be a single tool for all the tests from the different working groups.

    As a side note wpt.fyi needs some infrastructure improvements, for example Microfost was not happy as Ahem font (which is used a lot in CSS tests suites) is still not installed on the Windows virtual machines that extract test results for wpt.fyi.

    Floats, floats, floats

    People are using floats to simulate CSS Shapes on browsers that don’t have support yet. That is causing that some special cases related to floats happen more frecuently, and it’s hard to decide what’s the best thing to do on them.

    The CSSWG was discussing what would be the best solution when the non-floated content doesn’t fit in the space left by the floated elements. The problem is quite complex to explain, but imagine the following picture where you have several floated elements.

    An example of float layout An example of float layout

    In this example there are a few floated elements restricting the area where the content can be painted, if the browser needs to find the place to add a BFC (like a table) it needs to decide where to place it avoiding overlapping any other floats.

    There was a long discussion, and it seems the best choice would be that the browser tests all the options and if there’s no overlapping then puts the table there (basically Option 1 in the linked illustration). Still there are concerns about performance, so there’s still more work to be done here. As a result of this discussion a new CSS Floats specification will be created to describe the expected behavior in this kind of scenarios.

    Monica Dinculescu created a really cool demo to explain how float layout works, with the help of Ian Kilpatrick who knows it pretty well as he has been dealing with lots of corner cases while working in LayoutNG.

    TYPO Labs

    The members of the CSSWG were invited to the co-located TYPO Labs event. I attended on Friday when Elika (fantasai), Myles and Rossen gave a talk. It was nice to see that CSS Grid Layout was mentioned in the first talk of the day, as an useful tool for typographers. Variable fonts and Virtual Reality were clearly hot topics in several talks.

    Elika (fantasai), Myles and Rossen in the CSSWG talk at TYPO Labs Elika (fantasai), Rossen and Myles in the CSSWG talk at TYPO Labs

    It’s funny that the last time I was in Berlin was 10 years ago for a conference related to TYPO3, totally unrelated but with a similar name. 😄

    Other

    Some pictures of Berlin Some pictures of Berlin

    And that’s mostly all that I can remember now, I’m sure I’m missing many other important things. It was a fantastic week and I even find some time for walking around Berlin as the weather was really pleasant.

    April 15, 2018 10:00 PM