Planet Igalia

October 08, 2019

Andy Wingo

thoughts on rms and gnu

Yesterday, a collective of GNU maintainers publicly posted a statement advocating collective decision-making in the GNU project. I would like to expand on what that statement means to me and why I signed on.

For many years now, I have not considered Richard Stallman (RMS) to be the head of the GNU project. Yes, he created GNU, speaking it into existence via prophetic narrative and via code; yes, he inspired many people, myself included, to make the vision of a GNU system into a reality; and yes, he should be recognized for these things. But accomplishing difficult and important tasks for GNU in the past does not grant RMS perpetual sovereignty over GNU in the future.

ontological considerations

More on the motivations for the non serviam in a minute. But first, a meta-point: the GNU project does not exist, at least not in the sense that many people think it does. It is not a legal entity. It is not a charity. You cannot give money to the GNU project. Besides the manifesto, GNU has no by-laws or constitution or founding document.

One could describe GNU as a set of software packages that have been designated by RMS as forming part, in some way, of GNU. But this artifact-centered description does not capture movement: software does not, by itself, change the world; it lacks agency. It is the people that maintain, grow, adapt, and build the software that are the heart of the GNU project -- the maintainers of and contributors to the GNU packages. They are the GNU of whom I speak and of whom I form a part.

wasted youth

Richard Stallman describes himself as the leader of the GNU project -- the "chief GNUisance", he calls it -- but this position only exists in any real sense by consent of the people that make GNU. So what is he doing with this role? Does he deserve it? Should we consent?

To me it has been clear for many years that to a first approximation, the answer is that RMS does nothing for GNU. RMS does not write software. He does not design software, or systems. He does hold a role of accepting new projects into GNU; there, his primary criteria is not "does this make a better GNU system"; it is, rather, "does the new project meet the minimum requirements".

By itself, this seems to me to be a failure of leadership for a software project like GNU. But unfortunately when RMS's role in GNU isn't neglect, more often as not it's negative. RMS's interventions are generally conservative -- to assert authority over the workings of the GNU project, to preserve ways of operating that he sees as important. See for example the whole glibc abortion joke debacle as an example of how RMS acts, when he chooses to do so.

Which, fair enough, right? I can hear you saying it. RMS started GNU so RMS decides what it is and what it can be. But I don't accept that. GNU is about practical software freedom, not about RMS. GNU has long outgrown any individual contributor. I don't think RMS has the legitimacy to tell this group of largely volunteers what we should build or how we should organize ourselves. Or rather, he can say what he thinks, but he has no dominion over GNU; he does not have majority sweat equity in the project. If RMS actually wants the project to outlive him -- something that by his actions is not clear -- the best thing that he could do for GNU is to stop pretending to run things, to instead declare victory and retire to an emeritus role.

Note, however, that my personal perspective here is not a consensus position of the GNU project. There are many (most?) GNU developers that still consider RMS to be GNU's rightful leader. I think they are mistaken, but I do not repudiate them for this reason; we can work together while differing on this and other matters. I simply state that I, personally, do not serve RMS.

selective attrition

Though the "voluntary servitude" questions are at the heart of the recent joint statement, I think we all recognize that attempts at self-organization in GNU face a grave difficulty, even if RMS decided to retire tomorrow, in the way that GNU maintainers have selected themselves.

The great tragedy of RMS's tenure in the supposedly universalist FSF and GNU projects is that he behaves in a way that is particularly alienating to women. It doesn't take a genius to conclude that if you're personally driving away potential collaborators, that's a bad thing for the organization, and actively harmful to the organization's goals: software freedom is a cause that is explicitly for everyone.

We already know that software development in people's free time skews towards privilege: not everyone has the ability to devote many hours per week to what is for many people a hobby, and it follows of course that those that have more privilege in society will be more able to establish a position in the movement. And then on top of these limitations on contributors coming in, we additionally have this negative effect of a toxic culture pushing people out.

The result, sadly, is that a significant proportion of those that have stuck with GNU don't see any problems with RMS. The cause of software freedom has always run against the grain of capitalism so GNU people are used to being a bit contrarian, but it has also had the unfortunate effect of creating a cult of personality and a with-us-or-against-us mentality. For some, only a traitor would criticise the GNU project. It's laughable but it's a thing; I prefer to ignore these perspectives.

Finally, it must be said that there are a few GNU people for whom it's important to check if the microphone is on before making a joke about rape culture. (Incidentally, RMS had nothing to say on that issue; how useless.)

So I honestly am not sure if GNU as a whole effectively has the demos to make good decisions. Neglect and selective attrition have gravely weakened the project. But I stand by the principles and practice of software freedom, and by my fellow GNU maintainers who are unwilling to accept the status quo, and I consider attempts to reduce GNU to founder-loyalty to be mistaken and without legitimacy.

where we're at

Given this divided state regarding RMS, the only conclusion I can make is that for the foreseeable future, GNU is not likely to have a formal leadership. There will be affinity groups working in different ways. It's not ideal, but the differences are real and cannot be papered over. Perhaps in the medium term, GNU maintainers can reach enough consensus to establish a formal collective decision-making process; here's hoping.

In the meantime, as always, happy hacking, and: no gods! No masters! No chief!!!

by Andy Wingo at October 08, 2019 03:34 PM

October 02, 2019

Paulo Matos

A Brief Look at the WebKit Workflow

As I learn about the workflow for contributing to JSC (the JavaScript Compiler) in WebKit, I took a few notes as I went along. However, I decided to write them as a post in the hope that they are useful for you as well. If you use git, a Unix based system, and want to start contributing to WebKit, keep on reading.

More…

by Paulo Matos at October 02, 2019 01:00 PM

September 15, 2019

Brian Kardell

Beyond Browser Vendors

Beyond Browser Vendors

I'd like to explain what I think is potentially the most exciting set of changes that have taken place in the latter stages of the Web's history: Why we are now collectively considerably more in control of our own destinies and have moved "beyond browser vendors".

For most of the history of the Web, browser vendors have played an exceptionally critical role in standardization. This isn't a secret, in fact, it's widely discussed. What is less widely discussed is why that is. Frequently, most of our discussions give this very political origins, or assume it is about power struggles. But in fact, much of it has been for pretty considerably less intriguing reasons: It was a sort of predictable, natural outflow of the reality that we had created.

The first popular web browser (Mosiac) was proprietary. For a good chunk of our history, either all or most of the deployed browsers were built upon proprietary stuff, managed entirely by a single company.

This matters a lot because it meant that, practically speaking, everyone who is not a browser, including the myriad of organizations that participate in standards efforts are, effectively asking a small number of companies to invest money and manage priorities and budgets very particularly.

Mundane Realities

What's very, very difficult to appreciate from the outside is just how much is impacted, historically, by the fairly boring and entirely mundane fact above and that browser vendors aren't actually that special.

That is, they ultimately operate very much like most other organizations: They have a budget, carved out of some larger company. That company has some larger goals.

Their organization has an org chart with managers, teams, and projects, and QAs. They hire staff with specialized skills (these can be quite specialized). They have tasks and processes and annual reviews, and goals, and so on.

Code and approaches are adapted, they look for ways to improve performance, extend ideas, make things more modular, and so on.

Each quarter, then, managers have to decide how to maximize the budget in a way that includes an intersection of things in the pipeline (or that they might be interested in introducing), which best align with larger organizational goals,for which they have free resources with the right skills to do that work this quarter.

The Under Discussed, Big Impacts

If you can imagine a conveyor belt of ideas that ultimately land as tasks to be implemented, you can understand some of the problem. Even given the exact same inputs, different people can make pretty different, but entirely reasonable and defensible choices. And the inputs aren't remotely the same. Perhaps it takes one vendor twice as long to implement because this feature is much harder in their architecture, or they simply have less resources, or perhaps they're in the middle of reworking their layout, or... There are myriad reasons why these things get prioritized and worked differently.

If you were to consider nothing else at all, you can see why things get gummed up. There is an absolute gauntlet that something has to run in order to become a standard, even when there isn't particular controversy.

Practically speaking, this leads to misses. It leads to things backing up. The more backed up things get, the less likely a browser vendor is to entertain new ideas and spend a lot of time discussing things that doesn't directly align with their existing investments and priorities.

Things that come from another vendor, on the other hand, are more likely to place more direct pressure on them to respond.

Worse still, this creates kind of a feedback loop that ultimately winds up with a reality that yields near total practical domination of Web standards by browser vendors for reasons that aren't really especially political ones. These outcomes are not because of controvery, nor even particularly intentional.

A different story...

"But we do manage to get things done, even big things, rather quickly, when we really want to, right?" Is a refrain I've heard a lot. What about CSS Grid, for example? Grid seemed to drop everywhere at pretty much the same time, in 2017. It seemed like it happened very quickly to many of us, and I've heard this comment a number of times. If this isn't and illustration of vendors aligning on big ideas, prioritizing things, and getting things done fast when they want to, I don't know what is... Right? Here's the interesting part: It isn't that at all.

And what it is is far more interesting and exciting.

The backstory of CSS Grid

In 1996, Bert Bos, Dave Raggett, and Håkon Lie were the authors of a proposal called "Frame-Based Layout" which attempted to tackle the idea of a grid-oriented layout controlled by CSS. That completely failed to get uptake and implementations.

At the end of 2005, Bert Bos put some of these ideas forward again in CSS Advanced Layout. This kind of then became CSS Template Layout in 2009. Not one browser shipped any of this. It was important to the print industry though, so they invested and made their own renderers for print.

Fast foward to 2011, Microsoft reframes this work as Grid Layout and pretty much just documented what they shipped in IE10. But then, that's it. It sat there, in no small part because it had some real problems, having been mostly developed by Microsoft for specific use cases.

But then something changed, what? Well, a lot of things, but very importantly, the model of it all.

Open

Something happened along the way in our history: Both the standards process, and the the Web Platform itself got increasingly open.

In fact, today all of the implementations are open, and that's a big part of how we moved grid.

Most of the work on CSS Grid in both WebKit and Chromium (Blink) was done, not by Google or Apple, but by teams at Igalia.

Think about that for a minute: The prioritization of its work was determined in 2 browsers not by a vendor, but by an investment from Bloomberg who had the foresight to fund this largely uncontroversial work.

This isn't a unique story, it's just a really important and highly visible one that's fun to hold up. In fact, just in the last 6 months engineers as Igalia have worked on CSS Containment, ResizeObserver, BigInt, private fields and methods, responsive image preloading, CSS Text Level 3, bringing MathML to Chromium, normalizing SVG and MathML DOMs and a lot more.

Igalia loves open, and open standards - it's what we do. We participate in TC39, W3C, WHATWG and Khronos. We work on V8, JavaScriptCore, SpiderMonkey. Our teams work on Chromium, Gecko/Servo, WebKit. We work very closely with browser vendors. Openness can really change the economics involved and remove some previously difficult hurdles. This allows us all (including browsers vendors) to address more of those things than we could before.

The possibilities

This is the really exciting reason that I came to work at Igalia: Igalia potentially entirely shifts the standards paradigm. All this stuff still has a lot of costs to manage along the way. Browsers still play a hugely important role in reviews, managing questions of maintenance and so on - but, finally, we can begin to imagine a world in where we can more collectively prioritize and cooperate.

Imagine the possibilities of all of the things we could get done together. Lots and lots of stuff is just stuck- and many of the causes for this are ultimately because of things that are more organizational than because something is particularly controversial. We can help break the cycle and give those uncontroversial things implementation priority.

All we have to do is decide a thing is important enough, and there are lots of ways to do that. We're already doing a lot and I think we've just begun to scratch the surface of what is possible.

Imagine, for example, a world in which organizations or groups very interested in moving a set of topics simply came together and helped collectively prioritize and fund the efforts of those things, for example. We could get things done. Things that have lingered. Honestly, the possibilities seem endless: From discrete proposals that are critically important to a small group to broad ideas like color systems that touch everything from design to accessibility... Or, good print support from the same rendering engines? New core capabilities to help make things more extensible that are just moving too slow?

Our possible future is very exciting, and more than ever in our collective hands to determine! If you're interested in learning more, or have questions, you can contact Igalia, or come and find me at TPAC if you happen to be here.

September 15, 2019 03:00 PM

Eleni Maria Stea

GUADEC 2019 took place in my city! Also: my helloworld in Rust.

This year (2019) GUADEC took place in my city, Thessaloniki. Although I’ve never had any direct contributions to GNOME I wouldn’t miss the opportunity to attend the conference: the traveling time to the venue was about 15 minutes! Igalia was among the sponsors and I had the opportunity to meet some colleagues of mine that … Continue reading GUADEC 2019 took place in my city! Also: my helloworld in Rust.

by hikiko at September 15, 2019 01:07 PM

August 26, 2019

Alberto Garcia

The status of WebKitGTK in Debian

Like all other major browser engines, WebKit is a project that evolves very fast with releases every few weeks containing new features and security fixes.

WebKitGTK is available in Debian under the webkit2gtk name, and we are doing our best to provide the most up-to-date packages for as many users as possible.

I would like to give a quick summary of the status of WebKitGTK in Debian: what you can expect and where you can find the packages.

  • Debian unstable (sid): The most recent stable version of WebKitGTK (2.24.3 at the time of writing) is always available in Debian unstable, typically on the same day of the upstream release.
  • Debian testing (bullseye): If no new bugs are found, that same version will be available in Debian testing a few days later.
  • Debian stable (buster): WebKitGTK is covered by security support for the first time in Debian buster, so stable releases that contain security fixes will be made available through debian-security. The upstream dependencies policy guarantees that this will be possible during the buster lifetime. Apart from security updates, users of Debian buster will get newer packages during point releases.
  • Debian experimental: The most recent development version of WebKitGTK (2.25.4 at the time of writing) is always available in Debian experimental.

In addition to that, the most recent stable versions are also available as backports.

  • Debian stable (buster): Users can get the most recent stable releases of WebKitGTK from buster-backports, usually a couple of days after they are available in Debian testing.
  • Debian oldstable (stretch): While possible we are also providing backports for stretch using stretch-backports-sloppy. Due to older or missing dependencies some features may be disabled when compared to the packages in buster or testing.

You can also find a table with an overview of all available packages here.

One last thing: as explained on the release notes, users of i386 CPUs without SSE2 support will have problems with the packages available in Debian buster (webkit2gtk 2.24.2-1). This problem has already been corrected in the packages available in buster-backports or in the upcoming point release.

by berto at August 26, 2019 01:13 PM

Miyoung Shin

Why should we use a precise-width integer type?

Introduction
A few months ago, I’ve finished the task of replacing imprecise-width integer types like (unsigned) short, int, long, long long by precise-width integer types like (u)int16/int32/int64_t in Blink.. I’ve been working on the Onion Soup project and it was my first task of this project. When I took this task, I thought it would be comparatively simple and quite mechanical work, but this was not the case. It was sometimes required to understand whole code flow in-depth even out of blink or to explain the benefit of such changes to reviewers.

I’d like to introduce why we should prefer using the precise-width integer type and what I did for this task.

Why should we use the precise-width integer type?
We should keep in mind the fact that the c++ standards don’t specify the specific size of each integer type.

Data Model Table

Data model sizeof(int) sizeof(long) sizeof(long long) example
LP32 16b 32b 64b Win16
ILP32 32b 32b 64b Win32, i386 OSX & Linux
LP64 32b 64b 64b x86-64 OSX & Linux
LLP64 32b 32b 64b Win64

Chromium supports various platforms with one repository and shows definitely the variable of different sizes with (unsigned) long between Android-Kitkat/MacOS/Win7 buildbots and other buildbots for try-bots. It means that it would have potential security issues like the stack overflow if we treat that long has 64 bits precise since the size of long is 64 bits like long long on almost all of build systems. And we should avoid mixing the use of the precise-width integer type and the imprecise-width integer type to store the same value.

Actually, after working on this I got a report mentioning that my changes to move from imprecise-width to precise-width types fixed a security issue. This made me realize that more existing issues could still be fixed this way, and I started fixing other ones reported via ClusterFuzz after that.

Google also recommends this in its own coding styleguide, see here.

Summary of my contributions in Blink
1) Decided the order of the changing type.

unsigned short -> uint16_t / short -> int16_t
unsigned long long -> uint64_t / long long -> int64_t
unsigned long -> uint32_t or uint64_t, long -> int32_t or int64_t

2) Wrote the patch including only related codes.

3) Found what is the proper type.
Which type is proper between uint32_t and uint64_t when I should change unsigned long?
Every time I asked myself the question since I had to consider the fact that unsigned long was 32 bit on Mac, Android-kitkat and Window7 buildbot and was 64 bit on others so far. So I needed to understand its code flow and the use cases not only in Blink but also out of Blink. In general, it would be best to avoid integer-width conversions where possible and to only convert to larger types when needed.

4) Utilized the template instead of the functions for each type
I reduced the code quantity with the template and avoided the build error by duplicated definition.

5) Used base::Optional when the initial value is ambiguous.
In general the variable’s initial value would be 0 or -1, but base::Optional is more readable and simple than comparing 0 or -1 to check the initial value. Actually, I needed to reach a consensus with the review since it could change the code logic.

bool HasValidSnapshotMetadata() const { return snapshot_size_ >= 0; }
long long snapshot_size_;

Instead,

bool HasValidSnapshotMetadata() const { return snapshot_size_.has_value(); }
base::Optional<uint64_t> snapshot_size_;

6) Used CheckedNumeric to check the overflow of the variable.

static bool SizeCalculationMayOverflow(unsigned width,
                                         unsigned height,
                                         unsigned decoded_bytes_per_pixel) {
    unsigned long long total_size = static_cast<unsigned long long>(width) *
                                    static_cast<unsigned long long>(height);
    if (decoded_bytes_per_pixel == 4)
      return total_size > ((1 << 29) - 1);
    return total_size > ((1 << 28) - 1);
}

The above code was the old one before I changed and it was for checking if the variable has the overflow, I took some time to fully understand it.
I’ve used CheckedNumeric since the reviewer let me know this utility. As you can see from the code below, it has been changed to be simple and readable.

  inline bool SizeCalculationMayOverflow(unsigned width,
                                         unsigned height,
                                         unsigned decoded_bytes_per_pixel) {
    base::CheckedNumeric<int32_t> total_size = width;
    total_size *= height;
    total_size *= decoded_bytes_per_pixel;
    return !total_size.IsValid();
  }

7) Oops! MSVC!
The below code shows the different results due to compilers.

class Foo {
    uint16_t a;
    unsigned b : 16;
};
std::cout << sizeof(Foo);

The results are from clang : 4 bytes, gcc : 4 bytes and msvc : 8 bytes

So, it caused the build break with MSVC toolchain on Windows buildbots as the class size was increased. It took a long time to analyze the cause and solve this issue.

In conclusion, it was very simple to fix. See the code below.

class Foo {
    unsigned a : 16;
    unsigned b : 16;
};

8) Brought out the potential issue with with the overflow.
Blink can handle the width and height within 4 bytes range. The old code showed the abnormal layout result with overflowed width/height but it did not crash. I added the assertion code to check if there is the overflow when changing the type. Finally, we prevent the overflow and solve to layout the object’s position and size within the maximum value of 4 bytes.

9) Added PRESUBMIT check.
Now that we cleaned up the Blink code from the imprecise-width types, a PRESUBMIT hook was added to prevent them from being reintroduced to the sourcebase. So I added the PRESUBMIT check to ensure new uses do not come in again.

10) Why do we still use int and unsigned?
This is the comment of the reviewer and I also agree with it.

Thee C++ style guide states "We use int very often, for integers we know are not going to be too big, e.g., loop counters. Use plain old int for such things. You should assume that an int is at least 32 bits, but don't assume that it has more than 32 bits. If you need a 64-bit integer type, use int64_t or uint64_t."
The style guide does state not to use the imprecise-width integers like long long/long/short, but int is the imprecise-width type that is explicitly preferred. Among other reasons, this is likely for readability and the fact that ints are by default the "native"/fastest type, and potentially large concerns over code churn. More there is not any benefit to replace int to int32t and potentially large concerns over code churn.

I’m happy to contribute to Chromium & Blink with more stable code through changing to the precise-width integer type and to learn more about this particular domain.

References
[1] https://en.cppreference.com/w/cpp/language/types
[2] http://nickdesaulniers.github.io/blog/2016/05/30/data-models-and-word-size/

by mshin at August 26, 2019 07:59 AM

August 22, 2019

Alejandro Piñeiro

ARB_gl_spirv and ARB_spirv_extension support for i965 landed Mesa master

And something more visible thanks to that: now the Intel Mesa driver exposes OpenGL 4.6 support, the most recent version of OpenGL.

As perhaps you could recall, the i965 Intel driver became 4.6 conformant last year. You have more details about that, and what being conformant means in this Iago blog post. On that blog post Iago mentioned that it was passing with an early version of the ARB_gl_spirv support, that we were improving and interating during this time so it could be included on Mesa master. At the same time, the CTS tests were only testing the specifics of the extensions, and we wanted a more detailed testing, so we also were adding more tests on the piglit test suite, written manually for ARB_gl_spirv or translated from existing GLSL tests.

Why did it take so long?

Perhaps some would wonder why it took so much time. There were several reasons, but the main one, was related to the need to add a lot of code related to linking on NIR. On a previous blog post ( Introducing Mesa intermediate representations on Intel drivers with a practical example) I mentioned that there were several intermediate languages on Mesa.

So, for the case of the Intel driver, for GLSL, we had a chain like this:

GLSL -> AST -> Mesa IR -> NIR -> Intel backend IR

Now ARB_gl_spirv introduces the possibility to use SPIR-V instead of GLSL. Thanks to the Vulkan support on Mesa, there is a SPIR-V to NIR pass, so the chain for that case would be something like this:

SPIR-V -> NIR -> Intel backend IR

So, at first sight, this seems like it should be more simple, as there is one intermediate step less. But there is a problem. On Vulkan there is no need of a really complex shader linking. Basically gathering some info from the NIR shader. But OpenGL requires more. Even though the extension doesn’t required error validation, the spec points that queries and other introspection utilities should work naturally, as long as they don’t require names (names are considered debug info on SPIR-V). So for example, on Vulkan you can’t ask the shader about how big an SSBO is in order to allocate the space needed for it. It is assumed that you know that. But in OpenGL you can, and as you could do that using just the SSBO binding, ARB_gl_spirv requires that you still support that.

And here resided the main issue. On Mesa most of the linking is done at the Mesa IR level, with some help when doing the AST to Mesa IR pass. So the new intermediate language chain lacked it.

The first approach was trying to convert just the needed NIR stuff back to Mesa IR and then call the Mesa IR linker, but that was working only on some limited cases. Additionally, for this extension the linker rules change significantly. As mentioned, on SPIR-V names are optional. So everything needs to work without names. In fact, the current support just ignores names, for simplicity, and to ensure that everything works without it. So we ended up writing a new linker, based on NIR, and based on ARB_gl_spirv needs.

The other main reason for the delay was the significant changes on the SPIR-V to NIR pass, and NIR in general, to support SSBO/UBO (derefs were added), and also the native support of transform feedback, as Vulkan added a new extension, that we wanted to use. Here I would like to thank Jason Ekstrand for his support and patience ensuring that all those changes were compatible with our ARB_gl_spirv work.

So how about removing Mesa IR linking?

So, now that it is done, perhaps some people would wonder if this work could be used to remove Mesa IR on the GLSL intermediate language chain. Specially taking into account that there were already some NIR based linking utilities. My opinion is that no 😉

There are several reasons. The main one is that as mentioned ARB_gl_spirv linking rules are different, specifically on the lack of names. GLSL rules are heavily based on names. During the review it was suggested that need could be abstracted somehow, and reuse code. But, that doesn’t solve the issue that current Mesa IR supports the linkage of several different versions of GLSL, and more important, the validation and error checking done there, that is needed for GLSL, but not for ARB_gl_spirv (so it is not done). And that is a really big amount of work needed. In my opinion, redoing that work on NIR would not bring a significant advantage. So the current NIR linker would be just the implementation of a linker focused on the ARB_gl_spirv new rules, and would just share utility NIR based linking methods, or specific subfeatures, as far as possible (like the mentioned transform feedback support).

Final words

If you are interested on more details about the work done to implement ARB_gl_spirv/ARB_spirv_extensions support, you can check the presentation I gave on FOSDEM 2018 (slides, video) and the update I gave the same year on XDC (slides, video).

And finally I would like to thank all the people involved. First, thanks to Nicolai Hähnle for starting the work, that we used as basis.

The Igalia team that worked on it at some point were: Eduardo Lima, Alejandro Piñeiro, Antía Puentes, Neil Roberts, some help from Iago Toral and Samuel Iglesias at the beginning and finally thanks to Arcady Goldmints-Orlov, that dealt with handling the review feedback for the two ~80 patches MR (Mesa and Piglit test suite MR) that I created, when I became needed elsewhere.

And thanks a lot to the all the the reviewers, specially Timothy Arceri, Jason Ekstrand and Caio Marcelo.

Finally, big thanks to Intel for sponsoring our work on Mesa, Piglit, and CTS, and also to Igalia, for having me working on this wonderful project.

by infapi00 at August 22, 2019 11:54 AM

August 19, 2019

Brian Kardell

Top Level Await: 2 Minute Standards

Top Level Await: 2 Minute Standards

My first post in a new effort #StandardsIn2Min to provide short, but useful information about developing standards.

Previously, in order to use await, code needed to be inside a function marked as async. This meant you couldn't use await at the 'top-level' (outside of any function notation). At first, this might seem like a minor annoyance. Just put the code you want to await into an async function() {....}() and call it, right?

Yes, and (mostly) no. While being able to use await at the top-level is generally useful - the real value in it (the problem that it solves) has a lot to do with modules.

Modules are already asynchronous, and have a declarative import and export, and those also expressed at the top-level. The practical implication of this was that if you wanted provide a module which relied on some asynchronus task in order to be useful - for example, connecting to a database - you had really no good options. You might make every method in your own API internally dependent on that promise. But, this is extra complicated. All of your methods need to return promises, whether that makes sense, or not. If errors do occur, they are late, and in the wrong place. Or, you could make your API export that promise somehow. "Somehow" because there are several models for how you can choose to do this. Worse, whichever promise exporting model you chose, users of your module are left with precisely the same problem, creating a kind of domino effect.

That's what the new top-level await proposal (stage 3 as of the time of this writing) really solves. With it, you can write something like this.

// products.js 
import { fictionalDb } from './fictionaldb.js' 
import { config } from './db-config.js'
 
// connect() is promise returning
let connection = await fictionalDb.connect(config)

export default {
	recent: function () {   
	   // use the connection to return recent products
	}
	
	discontinued: function () {
	   // use the connection to return discontinued products	
	}
}

It seems incredibly small - one word on line 6. However, the magic and value really comes in how this module can now be used: You just import it like any other module.

If you're curious about how it works

The real magic here is largely in the definition of a standard protocol that imports and exports can reason about internally. To explain: You can think about modules as really including both their actual expressed exports, and, a new implicit internal promise used by the module system's internal protocol. While you're expressing your dependencies without that, the module system will (roughly) expand this internally as an implicit Promise.all around anything waiting for export.

For example, given the code:

import { a } from './a.mjs';
import { b } from './b.js';
import { c } from './c.js';

console.log(a, b, c)

The module system, internally, sees (again, roughly) this:

import { _internalPromise as aPromise, a } from './a.js';
import { _internalPromise as bPromise, b } from './b.js';
import { _internalPromise as cPromise, c } from './c.js';

// the module system creates this promise 
// and uses it to know your module is ready.
export const _internalPromise = 
	Promise.all([aPromise, bPromise, cPromise])
		.then(() => {
   			console.log(a, b, c);
		});

The net result is that modules are fetched in parallel and executed in order until the first await, and then waits until they are all complete before resolving the module itself.

August 19, 2019 03:00 PM

2 Minute Standards

2 Minute Standards

A short explanation of a new effort

For a while now, we've been running some 'outreach groups'. Originally started informally by my colleague at Igalia, Dan Ehrenberg, it's become a set of monthly meetings with various groups of stakeholders in TC39 and the Web at large which I really look forward to. The groups are meetings of people who work on Tooling, people who work on Frameworks and Libraries and Educators, with folks who work on standards. The goals of all of them, at the end of the day, is to figure out how we all work better, together.

The Educators Group has talked a bit about a number of efforts, but there's one conversation we've been having that I think is pretty interesting. I dont know of any developer who isn't interested in what is going on in standards, but the economics of this is currently kind of hard. We're interested in a whole bunch of other things too. There's lots of good content - too much in fact, coming at us from a million directions and it's all competing for our time and attention. We have jobs, and families, and pets, hobbies and lives. I love a lot of long form things, I'm excited when I see them in my stream, but they pile up and I have to choose. I only have so many slots of time where I can actually afford to consume 10, 15, 30 or 40 minutes worth of information. It can feel a little overwhelming sometimes.

So, we're going to try something new: 2 Minute Standards. Think of it as "standards news, in brief". The basic idea is pretty simple: Like a news segment, it's got to contain pertinent facts, more than a tweet - but it's air time is limited, as the name would suggest, to 2 Minutes. That can come in a number of forms: An article that is 500 words or less, a two minute video, or a short web comic/zine. The idea though is to ultimately give developers some channel with reasonable economics for staying 'generally informed'. In the end, that's frequently what we want anyways: We can't know everything, we just need to know enough to know what it is, why it is, and that it is there when we're ready to learn more. We'd like to be able to roughly understand it if it comes up. A maximum of 2 minutes per day, and almost certainly not even that much, seems... unintimidating and managable.

So, that's the idea. We're still discussing many of the details about how to make this really 'click' in a bigger way: How to create as a sort of channel you can follow? For now, it seems better to start with something simple and perhaps to begin to get feedback on the idea. To this end, we'll post things with the hashtag #StandardsIn2Min, and others can too. If you see something fly by with the hashtag you can always find it again later on the twitter feed and, hopefully, expect that it will take no more than 2 minutes to consume. However, we will also manage the Twitter account @StandardsIn2Min, where we'll work to curate those hashtags too and create a manageable (honestly, probably slow) pace that never asks for more than two minutes of your attention in a day to keep up. Let us know what you think about the idea!

Thanks to the educators group for all of their help discussing and creating, and especially to Laurie Barth and Jory Burson for helping get this post put together quickly!

August 19, 2019 03:00 PM

August 12, 2019

Jacobo Aragunde

Spatial navigation operations for the webview tag

The modern web is a space meant to be browsed with a mouse or a touchscreen, but think of a smart TV, set-top box, game console or budget phone. What happens when your target hardware doesn’t have those devices and provides only directional input instead?

TV, remote, phone

If you are under control of the contents, you certainly can design your UI around directional input, providing custom focus management based on key events, and properly highlighting the focused item. But that doesn’t work when you confront external content provided by the millions of websites out there, not even if you are targeting a small subset of them like the most popular social networks and search engines.

Virtual cursors are one solution. They certainly address the problem and provide full functionality, but they are somehow inconvenient: traversing the screen can take too long, and precision can be an issue.

Tab navigation is always available and it certainly works well in sites designed for that, like forms, but it’s quite limited; it can also show unexpected behavior when the DOM tree doesn’t match the visual placement of the contents.

Spatial navigation is a feature designed to address this problem. It will:

  • respond to directional input taking into account the position of the elements of the screen
  • deal with scroll when we reach the limits of the viewport or a scrollable element, in a way that content can be easily read (e.g. no abrupt jumps to the next focusable element)
  • provide special highlight for the focused element.

You can see it in action in this BlinkOn lightning talk by Junho Seo, one of the main contributors. You can also test it yourself with the --enable-spatial-navigation flag in your Chrome/Chromium browser.

A non-trivial algorithm makes sure that input matches user expectations. That video shows a couple of corner cases that demonstrate this isn’t easy.

When creating a web UI for a device that makes use of cursor navigation, it’s usual that you have to deal with the two kinds of content mentioned before: the one created specifically for the device, which already takes cursor input into account in its design, and external content from websites that don’t follow those principles. It would be interesting to be able to isolate external content and provide spatial navigation for it.

The webview extension tag is one way to provide isolation for an external site. To support this use case, we implemented new API operations for the tag to enable or disable spatial navigation inside the webview contents independently from the global settings, and to check its state. They have been available for a while, since Chromium version 71.

There are some interesting details in the implementation of this operation. One is that it is required some IPC between different browser processes, because webview contents run on a different renderer (guest), which must be notified of the status changes in spatial navigation made by its “parent” (host). There is no need to do IPC in the other direction because we cache the value in the browser process for the implementation of isSpatialNavigationEnabled to use it.

Another interesting detail was, precisely, related with that cache; it made our test flaky because sometimes the calls to isSpatialNavigationEnabled happened before the call to setSpatialNavigationEnabled had completed its IPC communication. My first attempt to fix this added an ACK reply via IPC to confirm the cache value, but it was considered an overkill for a situation that would never trigger in a real-world scenario. It even was very hard to reproduce from the test! It was finally fixed with some workarounds in the test code.

A more general solution is proposed for all kinds of iframes in the CSS WG draft of the spatial navigation feature. I hope my previous work makes the implementation for the iframe tag straightforward.

by Jacobo Aragunde Pérez at August 12, 2019 06:00 PM

August 05, 2019

Philippe Normand

Review of the Igalia Multimedia team Activities (2019/H1)

This blog post takes a look back at the various Multimedia-related tasks the Igalia Multimedia team was involved in during the first half of 2019.

GStreamer Editing Services

Thibault added support for the OpenTimelineIO open format for editorial timeline information. Having many editorial timeline information formats supported by OpenTimelineIO reduces vendor lock-in in the tools used by video artists in post-production studios. For instance a movie project edited in Final Cut Pro can now be easily reimported in the Pitivi free and open-source video editor. This accomplishment was made possible by implementing an OpenTimelineIO GES adapter and formatter, both upstreamed respectively in OpenTimelineIO and GES by Thibault.

Another important feature for non-linear video editors is nested timeline support. It allows teams to decouple big editing projects in smaller chunks that can later on be assembled for the final product. Another use-case is about pre-filling the timeline with boilerplate scenes, so that an initial version of the movie can be assembled before all teams involved in the project have provided the final content. To support this, Thibault implemented a GES demuxer which transparently enables playback support for GES files (through file://path/to/file.xges URIs) in any GStreamer-based media player.

As if this wasn’t impressive enough yet, Thibault greatly improved the GES unit-tests, fixing a lot of memory leaks, race conditions and generally improving the reliability of the test suite. This is very important because the Gitlab continuous integration now executes the tests harness for every submitted merge request.

For more information about this, the curious readers can dive in Thibault’s blog post. Thibault was invited to talk about these on-going efforts at SIGGRAPH during the OpenTimelineIO BOF.

Finally, Thibault is mentoring Swayamjeet Swain as part of the GSoC program, the project is about adding nested timeline support in Pitivi.

GStreamer VA-API

Víctor performed a good number of code reviews for GStreamer-VAAPI contributors, he has also started investigating GStreamer-VAAPI bugs specific to the AMDGPU Gallium driver, in order to improve the support of AMD hardware in multimedia applications.

As part of the on-going GStreamer community efforts to improve continuous integration (CI) in the project, we purchased Intel and AMD powered devices aimed to run validation tests. Running CI on real end-user hardware will help ensure regressions remain under control.

Servo and GStreamer-rs

As part of our on-going involvement in the Servo Mozilla project, Víctor has enabled zero-copy video rendering support in the GStreamer-based Servo-media crate, along with bug fixes in Glutin and finally in Servo itself.

A prior requirement for this major milestone was to enable gst-gl in the GStreamer Rust bindings and after around 20 patches were merged in the repository, we are pleased to announce Linux and Android platforms are supported. Windows and macOS platforms will also be supported, soon.

The following screencast shows how hardware-accelerated video rendering performs on Víctor’s laptop:

WebKit’s Media Source Extensions

As part of our on-going collaboration with various device manufacturers, Alicia and Enrique validated the Youtube TV MSE 2019 test suite in WPEWebKit-based products.

Alicia developed a new GstValidate plugin to improve GStreamer pipelines testing, called validateflow. Make sure to read her blog post about it!.

In her quest to further improve MSE support, especially seek support, Alicia rewrote the GStreamer source element we use in WebKit for MSE playback. The code review is on-going in Bugzilla and on track for inclusion in WPEWebKit and WebKitGTK 2.26. This new design of the MSE source element requires the playbin3 GStreamer element and at least GStreamer 1.16. Another feature we plan to work on in the near future is multi-track support; stay tuned!

WebKit WebRTC

We announced LibWebRTC support for WPEWebKit and WebKitGTK one year ago. Since then, Thibault has been implementing new features and fixing bugs in the backend. Bridging between the WebAudio and WebRTC backend was implemented, allowing tight integration of WebAudio and WebRTC web apps. More WebKit WebRTC layout tests were unskipped in the buildbots, allowing better tracking of regressions.

Thibault also fixed various performance issues on Raspberry Pi platforms during apprtc video-calls. As part of this effort he upstreamed a GstUVCH264DeviceProvider in GStreamer, allowing applications to use already-encoded H264 streams from webcams that provide it, thus removing the need for applications to encode raw video streams.

Additionally, Thibault upstreamed a device provider for ALSA in GStreamer, allowing applications to probe for Microphones and speakers supported through the ALSA kernel driver.

Finally, Thibault tweaked the GStreamer encoders used by the WebKit LibWebRTC backend, in order to match the behavior of the Apple implementation and also fixing a few performance and rendering issues on the way.

WebKit GStreamer Multimedia maintenance

The whole team is always keeping an eye on WebKit’s Bugzilla, watching out for multimedia-related bugs reported by the community members. Charlie recently fixed a few annoying volume bugs along with an issue related with youtube.

I rewrote the WebKitWebSrc GStreamer source element we use in WebKit to download HTTP(S) media resources and feed the data to the playback pipeline. This new element is now based on the GStreamer pushsrc base class, instead of appsrc. A few seek-related issues were fixed on the way but unfortunately some regressions also slipped in; those should all be fixed by now and shipped in WPEWebKit/WebKitGTK 2.26.

An initial version of the MediaCapabilities backend was also upstreamed in WebKit. It allows web-apps (such as Youtube TV) to probe the user-agent for media decoding and encoding capabilities. The GStreamer backend relies on the GStreamer plugin registry to provide accurate information about the supported codecs and containers. H264 AVC1 profile and level information are also probed.

WPEQt

Well, this isn’t directly related with multimedia, but I finally announced the initial release of WPEQt. Any feedback is welcome, QtWebKit has phased out and if anyone out there relies on it for web-apps embedded in native QML apps, now is the time to try WPEQt!

by Philippe Normand at August 05, 2019 01:30 PM

August 01, 2019

Julie Kim

Giving a talk at Automotive Linux Summit 2019

Last year, I’ve worked on Chromium and Web Application Manager on AGL platform with some fellow Igalian colleagues. Our goal was to support Chromium with upstream Wayland port and Web Application Manager. Although we faced a lot of challenges, such as porting them to AGL framework, handling SMACK on Chromium multiprocess architecture, supporting wayland ivi protocol on Chromium and so on, we completed our goal and showed a demo at CES 2019.

This year, I gave a talk about our work on AGL at several events such as AGL All Member Meeting, BlinkOn10 in Canada and Automotive Linux Summit 2019.

For Automotive Linux Summit 2019 in Tokyo last week, my colleague, Lorenzo, and I attended the event and managed our Igalia booth.


We showed some demos for Chromium and Web Application Manager on AGL on several reference boards such as Renesas m3 board, Intel Minnowboard, and RPi3. I was very pleased to meet many people and share our works with them.

I gave a talk about “HTML5 apps on AGL platform with the Web Application Manager“.


As you might know, Igalia has worked on Chromium Wayland port for several years. We tried various approaches and finally fully upstreamed our changes to upstream last year. Chromium on AGL has been implemented based on it with supporting ivi-extension.
LG opensourced Web Application Manager used for their products and Igalia implemented Web Runtime solution for AGL on the top of it. My colleague, Jacobo, uploaded a post about ‘Introducing the Chromium-based web runtime for the AGL platform‘. You can find more information there.

I also participated in the AGL Developer Panel in ALS.


We generally talked about the current status and the plans for the next step. Thanks to Walt Minor, Automotive Grade Linux Development Manager at The Linux Foundation, everything went well and I believe all of us enjoyed the time.

I think AGL is one of open source projects growing fast and it shows continuous improving. When I looked around demo booths, I could see many companies tried various interesting ideas on the top of the AGL.

Last, I’d like to say “Thanks a lot, people who organized this event, came by our Igalia booth, and listened to my talk”.

Automotive Grade Linux Logo

by jkim at August 01, 2019 06:31 AM

July 30, 2019

Manuel Rego

Talking about CSS Containment at CSSconf EU 2019

Back in January I wrote a blog post introducing CSS Containment where I tried to explain this specification and the main motivation and uses cases behind it. I’ve been working on css-contain implementation in Chromium during the last months as part of the ongoing collaboration between Igalia and Bloomberg, I believe this is not a very well known topic yet, and I have the chance to speak about it in the impressive CSSconf EU.

First of all it was a really great surprise to get my talk proposal accepted, Igalia has been attending CSSconf EU regularly but we never spoke there before. It’s one of the biggest CSS events and the setup they prepare (together with JSConf) is overwhelming, I’m very grateful to be part of the speakers lineup. Also the organization was very good and really paid attention to every detail.

CSSconf EU 2019 stage CSSconf EU 2019 stage

My talk’s title was Improving Website Performance with CSS Containment, during the talk I did an introduction to the spec, explained the contain property and the implications of the different types of containment. On top of that I showed a few examples about how the performance of a website can be improved thanks to css-contain. The video of my talk is published in YouTube and you can also get the slides from this blog.

After the presentation I got some positive feedback from the audience, it seems that despite of being a somehow deep topic people liked it. Thank you very much!

Last but not least, just to close this blog post I’d like to say thanks one more time to Bloomberg. They have been supporting Igalia’s work on CSS Containment implementation, which is helping to improve performance of their applications.

Igalia logo Bloomberg logo
Igalia and Bloomberg working together to build a better web

July 30, 2019 10:00 PM

July 08, 2019

Brian Kardell

Moving Math Forward, Thoughts and Experiments

Moving Math Forward, Thoughts and Experiments

In this piece, I explain details on thoughts on experiments and conversations around moving Math forward on the Web.

For some context, my team is actively working on implementing MathML in Chromium (the engine Chrome, Edge, Samsung, Brave, UC Browser and lots of other share), more specifically MathML Core. MathML has long had implementations in Safari and Firefox and has an intersting history that leaves it in a very unique spot.

MathML Core, defines the comparatively small, but well defined and mostly interoperable subset of MathML - with an aim of resolving uniqueness. It provides details about how it actually fits into the layers platform in well defined ways and to explain their existing magic in terms of the platform - to expose it if is it new. As such, in indirectly aims to provide a basis for moving many things forward.

A thought experiment

Recently conversations in the web platform at large have begun about moving HTML forward and polyfilling elements. Since we are actively working on a 'core' ourselves, we thought this would be an interesting discussion and experiment: We asked "What if MathML Core was largely 'done' and you would import other support to move forward?" Those things could potentially be shipped by a browser, but there's no guarantee they ever would. However, they could also be loaded as polyfills and in standard means that help omit many of the problems that traditional polyfills create.

To explain this, we took some elements that aren't in MathML Core, but are specified elsewhere, as an interesting stand in. One such example is an element called mfenced. mfenced is just a "more convenient way" of writing some math that has a lot of 'fences' (parens, pipes, braces, and so on). The original spec even said "It is strictly equivalent to this expanded, longer form".

Today, 'polyfills' exist for it if your browser doesn't support it, and today, those work by (if your browser doesn't support it) destroying your tree and replacing it with the equivalent version, at some undefined point in time.

We experimentally changed very little in our implementation and it now considers any MathML 'unknown element' as on the whitelist of elements which can have a Shadow DOM. We used a simple stand-in using Mutation Observer to allow any of those to be defined without a dash, and wrote a polyfill for mfenced which simply expands it into Shadow DOM.

For purposes of discussion, we illustrated how we could, in theory, use MathML Core, import some new vocabulary (potentially a future proposal, or just some slang) and easily expand it into some Shadow DOM, leaving you with the tree that you authored and all of the things just fitting together nicely and predictably. Below is a screenshot from our actual implementation experiment.

screenshot of element and shadow dom in experimental browser build

We think that these sorts of experiments are useful and informative and help us ask better questions about how we are fitting into the platform itself: Have we answered the necessary questions? How does the introduction of a ShadowRoot impact rendering and namespacing, for example? Could developers really 'plug in' to all the necessary bits? Could they properly layer atop? To test this, we also created elements that expand shorthands like TeX and ASCII Math. In the end, can we ultimately resolve a great a stable core that allows us to move math forward using these things similar in ways similar to Knuth's TeX book/engine - but in a way that is properly "of the platform"? We think the answer is yes.

While we're not entirely sure where larger conversations about how HTML will move forward will go, and we aren't claiming that this implementation is anything more than a quick experiment for discussion and testing, the MathML community itself is very excited that such talks are even happening in HTML and very interested in being a part of that conversation, and sorting out how we can move forward together as One Platform™, with less 'specialness'.

July 08, 2019 03:00 PM

One Platform™

One Platform™

Let's talk about making SVG and MathML less special.

As I mentioned in Harold Crick and the Web Platform, the HTML specification contains a section on "Other Embedded Content" which specifically mentions SVG and MathML. In that piece I explained their unique histories - how they wound up being "special" and how I felt this leaves them in a kind of unique place in the platform that deserves some similarly "special" attention in order to resolve.

What I mean by this, in part, is that we need to think about how to make them less special, so that we are reasoning about things as One Platform™ as evenly as possible. I'd like to illustrate and talk about two simple examples:

What is this element?

The HTML parser parses these things specially. While this is a kind of advantage we don't expect to be repeated, it actually came at a cost of historically disadvantaging the actual elements that we had to work with. They lacked basic/general interfaces that are otherwise common among any other element in HTML. There's really no great reason for this and we should unweird it - it's not hard. Last year, this work began by providing a mixin that could be used in SVG, and we'd like to do the same for MathML which lacked any kind of IDL at all.

What does this mean in practice? Well, it means that if you grab a reference to a MathML element, for example, and try to set it's .style property - what happens? It blows up. Why? Because if you ask it what is it's .constructor, the answer is just Element. But you can style it with CSS just fine. Common things like data-* attribute and tab indexes just don't work, trying to access any of over 100 common API properties or methods that Just Work anywhere else will throw, or not function. All of this is unnecessarily weird, for not great reasons.

This is a really basic thing and it's not at all hard to do technically. It seems unlikely that the parser will change, and everyone agrees that MathML is a reasonable thing to send to a browser (whether they believe that it is necessary that the browser itself renders that or not), so I am not sure why this should be controversial. This is a useful normalization of basic, low level platform things. It feels like everyone should support this.

Generally, let's make them both as unspecial as possible - which leads to the other thing...

How do we move forward?

HTML has had, and continues to have, a whole lot of thought and efforts to define "how to move forward". Figuring out the underpinnings of both the platform features necessary and a process enables the Web to really work and flourish is a big effort. A wide range of efforts over the last several years like Custom Elements, Shadow DOM, Houdini, AOM, modules (and types of modules), ways to share stylesheets, Import Maps and built-in modules are all, in some ways proposing elements of possible solutions to the basic question "How do we move forward?" A lot of this takes a long time and advancing frequently requires compromises in the interim.

Recently, new conversations have begun attempting to assemble these into a larger possible vision and discuss resolving compromises made along to really answer this question.

However, currently, SVG and MathML, exist at least partially outside many of those years of conversations and thought. It feels important to at least stop and consider the question: How do these move forward? And, if that doesn't look like the rest of the platform -- why not? Figuring out how to align with HTMLs "forward plans" seems very valuable.

I don't have concrete answers here, but I'd like to start making these things part of the conversation. For example...

Polyfilling Elements, Custom Elements and Built-ins

One of the more interesting recent conversations recently is around moving forward and polyfilling elements. It is, of course, very early. This issue and related efforts are intended to open conversation by offering some larger picture that we can talk about. One interesting conversation, I think, would be to think about how this might relate to SVG and MathML. The topics of Custom Elements and Shadow DOM, for example, are two that are thus far historically outside of SVG and MathML.

One way of looking at it in the thread offers that we might say that HTML itself is largely (not completely) 'done'. Not standards, just the core of HTML itself. It offers that what you could do then is import other support, and that support could also potentially be shipped down natively by the browser or polyfilled.

But, how does this apply to SVG and MathML? Is there real use in asking these questions? We think so.

While we're not entirely sure where larger conversations about how HTML will move forward will go, as part of our efforts to implement MathML Core in Chromium (the engine Chrome, Edge, Samsung, Brave, UC Browser and lots of apps share), we are asking lots of questions about how things fit into the platform more naturally. In this, we've recently done some interesting experimenting with some of these concepts in order to ask potentially useful questions. We believe these are both interesting 'at large', and actually very useful in helpng us make sure we're thinking about all of the right and necessary things.

If you're interested, you can read more on some experiments (with implementation) and discussions around them regarding Shadow DOM, Import Maps and potentially prefix-less custom elements for MathML and how we feel this is both compelling and helpful.

While the MathML Community Group is aware that these questions are far from settled, they were universally excited to see such talks begin in HTML and very interested in making sure that they're thinking about those things too: Being a part of that conversation, and sorting out how we can move forward together as One Platform™, with less 'specialness'.

If you think this is interesting or useful, please share your support and let someone know.

July 08, 2019 03:00 PM

June 30, 2019

Eleni Maria Stea

Depth-aware upsampling experiments (Part 6: A complete approach to upsample the half-resolution render target of a SSAO implementation)

This is the final post of the series where I explain the ideas tried in order to improve the upsampling of the half-resolution SSAO render target of the VKDF sponza demo that was written by Iago Toral. In the previous posts, I performed experiments to explore different upsampling ideas and I explained the logic behind … Continue reading Depth-aware upsampling experiments (Part 6: A complete approach to upsample the half-resolution render target of a SSAO implementation)

by hikiko at June 30, 2019 08:05 AM

Depth-aware upsampling experiments (Part 5: Sample classification tweaks to improve the SSAO upsampling on surfaces)

This is another post of the series where I explain the ideas I try in order to improve the upsampling of the half-resolution SSAO render target of the VKDF sponza demo that was written by Iago Toral. In a previous post (3.2), I had classified the sample neighborhoods in surface neighborhoods and neighborhoods that contain … Continue reading Depth-aware upsampling experiments (Part 5: Sample classification tweaks to improve the SSAO upsampling on surfaces)

by hikiko at June 30, 2019 08:04 AM

June 28, 2019

Michael Catanzaro

On Version Numbers

I’m excited to announce that Epiphany Tech Preview has reached version 3.33.3-33, as computed by git describe. That is 33 commits after 3.33.3:

Epiphany about dialog displaying the version number

I’m afraid 3.33.4 will arrive long before we  make it to 3.33.3-333, so this is probably the last cool version number Epiphany will ever have.

I might be guilty of using an empty commit to claim the -33 commit.

I might also apologize for wasting your time with a useless blog post, except this was rather fun. I await the controversy of your choice in the comments.

by Michael Catanzaro at June 28, 2019 04:07 PM

June 26, 2019

Jacobo Aragunde

Introducing the Chromium-based web runtime for the AGL platform

Igalia has been working with AGL (Automotive Grade Linux) to provide a web application runtime to their platform, based on Chromium. We delivered the first phase of this project back in January, and it’s been available since the Flounder 6.0.5 and Guppy 7.0.0 releases. This is a summary of how it came to be, and the next steps into the future.

Igalia stand showing the AGL web runtime

Web applications as first-class citizens

The idea of web applications as first-class citizens is not new in AGL. The early versions of the stack, based on Tizen 3.0, already implemented this idea, via the Crosswalk runtime. It provided extended JS APIs for developers to access to system services inside a web application runtime.

The abandon of the Tizen 3.0 effort by its main developers meant the chance for AGL to start fresh, redefining its architecture to become what it is today, but the idea of web applications was still there. The current AGL architecture still supports web apps, and the system APIs are available through WebSockets so both native and web applications can use them. But, until now, there wasn’t a piece of software to run them other than a general-purpose browser.

Leveraging existing projects

One of the strengths of open source is speeding up the time-to-market by allowing code reuse. We looked into the WebOS OSE platform, developed by LGe with contributions from Igalia. It provides the Web Application Manager (WAM) component, capable of booting up a Chromium-based runtime with a low footprint and managing application life cycle. Reusing it, we were able to deliver a web application runtime more quickly.

Our contribution back to the webOS OSE project is the use of the new Wayland backend, independently developed by Igalia and available upstream in new Chromium versions. It was backported to the Chromium version used by WebOS OSE, and hopefully it will be part of future releases of the platform based on newer versions of Chromium.

Development process

My colleague Julie made a great presentation of the AGL web runtime in the last AGL All Member Meeting in March. I’m reusing her work for this overview.

Wayland

The version of Chromium provided by the WebOS OSE platform makes use of an updated version of the Intel Ozone-Wayland backend. This project has been abandoned for a while, and never upstreamed due to deep architecture differences. In the last couple of years, Igalia has implemented a new backend independently, using the lessons learned from integrating and maintaining Ozone-Wayland in previous projects, and following the current (and future) architecture of the video pipeline.

Compared process structure of Wayland implementations

As we mentioned before, we backported the new implementation of the Wayland backend to WebOS OSE, and added IVI-shell integration patches on top of it.

AGL life cycle

The WAM launcher process was modified to integrate with AGL life cycle callbacks and events. In particular, it registers event callbacks for HomeScreen, WindowManager and notification for ILMControl, activates WebApp window, when it gets Event_TapShortcut, and manages WebApp states for Event_Active/Event_Inactive. LGe, also a member of AGL, provided the initial work based on the Eel release, which we ported over to Flounder and kept evolving and maintaining.

WAM Launcher process integration diagram

AGL security model

In the AGL security model, access to system services is controlled with SMACK labels. A process with a certain label can only access a subset of the system API. The installation manifest for AGL applications and services is in charge of the relation between labels and services.

Access to system APIs in AGL happens through WebSockets so, from the point of view of the browser, it’s just a network request. Since WAM reuses the Chromium process model, the network interaction happening in any running webapp would actually be performed by the browser process. The problem is that there is only one browser process in the system, it’s running in the background and its labels don’t relate with the ones from the running applications. As a solution, we configure web applications to channel their networking through a proxy, and the WAM launcher creates a proxy process with the proper SMACK label for every webapp.

Proxy process integration diagram

Current implementation is based on the Tinyproxy project, but we are pending to review this model and find a more efficient solution.

Next steps

We are clearing any Qt dependencies from WAM, replacing them with standard C++ or the boost library. This work is interesting for AGL to be able to create a web-only version of the platform, and also for WebOS OSE to make the platform lighter, so we will be contributing it back. In this regard, the final goal is that AGL doesn’t require any patches on top of WebOS OSE to be able to use WAM.

Also, in the line of creating a web-only AGL, we will provide new demo apps so the Qt dependency wouldn’t be needed anywhere in the platform.

And, after the AGL face-to-face meeting that Igalia hosted in Coruña, we will be integrating more subsystems available in the platform, and reworking the integration with the security framework to be more robust and efficient. You can follow our progress in the AGL Jira with the label WebAppMgr.

Try it now!

The web application runtime is available in Flounder (starting in 6.0.5) and Guppy releases, and it will be in Halibut (current master). It can be tested adding the agl-html5-framework feature to the agl setup script. I have a bunch of test web applications available for testing in the repository wam-demo-applications, but there will be official demos in AGL soon.

Youtube running as an AGL app

Youtube running as an AGL application

This project has been made possible by the Linux Foundation, through a sponsorship from Automotive Grade Linux. Thanks!

AGL logo

Igalia

by Jacobo Aragunde Pérez at June 26, 2019 05:00 PM

Andy Wingo

fibs, lies, and benchmarks

Friends, consider the recursive Fibonacci function, expressed most lovelily in Haskell:

fib 0 = 0
fib 1 = 1
fib n = fib (n-1) + fib (n-2)

Computing elements of the Fibonacci sequence ("Fibonacci numbers") is a common microbenchmark. Microbenchmarks are like a Suzuki exercises for learning violin: not written to be good tunes (good programs), but rather to help you improve a skill.

The fib microbenchmark teaches language implementors to improve recursive function call performance.

I'm writing this article because after adding native code generation to Guile, I wanted to check how Guile was doing relative to other language implementations. The results are mixed. We can start with the most favorable of the comparisons: Guile present versus Guile of the past.


I collected these numbers on my i7-7500U CPU @ 2.70GHz 2-core laptop, with no particular performance tuning, running each benchmark 10 times, waiting 2 seconds between measurements. The bar value indicates the median elapsed time, and above each bar is an overlayed histogram of all results for that scenario. Note that the y axis is on a log scale. The 2.9.3* version corresponds to unreleased Guile from git.

Good news: Guile has been getting significantly faster over time! Over decades, true, but I'm pleased.

where are we? static edition

How good are Guile's numbers on an absolute level? It's hard to say because there's no absolute performance oracle out there. However there are relative performance oracles, so we can try out perhaps some other language implementations.

First up would be the industrial C compilers, GCC and LLVM. We can throw in a few more "static" language implementations as well: compilers that completely translate to machine code ahead-of-time, with no type feedback, and a minimal run-time.


Here we see that GCC is doing best on this benchmark, completing in an impressive 0.304 seconds. It's interesting that the result differs so much from clang. I had a look at the disassembly for GCC and I see:

fib:
    push   %r12
    mov    %rdi,%rax
    push   %rbp
    mov    %rdi,%rbp
    push   %rbx
    cmp    $0x1,%rdi
    jle    finish
    mov    %rdi,%rbx
    xor    %r12d,%r12d
again:
    lea    -0x1(%rbx),%rdi
    sub    $0x2,%rbx
    callq  fib
    add    %rax,%r12
    cmp    $0x1,%rbx
    jg     again
    and    $0x1,%ebp
    lea    0x0(%rbp,%r12,1),%rax
finish:
    pop    %rbx
    pop    %rbp
    pop    %r12
    retq   

It's not quite straightforward; what's the loop there for? It turns out that GCC inlines one of the recursive calls to fib. The microbenchmark is no longer measuring call performance, because GCC managed to reduce the number of calls. If I had to guess, I would say this optimization doesn't have a wide applicability and is just to game benchmarks. In that case, well played, GCC, well played.

LLVM's compiler (clang) looks more like what we'd expect:

fib:
   push   %r14
   push   %rbx
   push   %rax
   mov    %rdi,%rbx
   cmp    $0x2,%rdi
   jge    recurse
   mov    %rbx,%rax
   add    $0x8,%rsp
   pop    %rbx
   pop    %r14
   retq   
recurse:
   lea    -0x1(%rbx),%rdi
   callq  fib
   mov    %rax,%r14
   add    $0xfffffffffffffffe,%rbx
   mov    %rbx,%rdi
   callq  fib
   add    %r14,%rax
   add    $0x8,%rsp
   pop    %rbx
   pop    %r14
   retq   

I bolded the two recursive calls.

Incidentally, the fib as implemented by GCC and LLVM isn't quite the same program as Guile's version. If the result gets too big, GCC and LLVM will overflow, whereas in Guile we overflow into a bignum. Also in C, it's possible to "smash the stack" if you recurse too much; compilers and run-times attempt to mitigate this danger but it's not completely gone. In Guile you can recurse however much you want. Finally in Guile you can interrupt the process if you like; the compiled code is instrumented with safe-points that can be used to run profiling hooks, debugging, and so on. Needless to say, this is not part of C's mission.

Some of these additional features can be implemented with no significant performance cost (e.g., via guard pages). But it's fair to expect that they have some amount of overhead. More on that later.

The other compilers are OCaml's ocamlopt, coming in with a very respectable result; Go, also doing well; and V8 WebAssembly via Node. As you know, you can compile C to WebAssembly, and then V8 will compile that to machine code. In practice it's just as static as any other compiler, but the generated assembly is a bit more involved:


fib_tramp:
    jmp    fib

fib:
    push   %rbp
    mov    %rsp,%rbp
    pushq  $0xa
    push   %rsi
    sub    $0x10,%rsp
    mov    %rsi,%rbx
    mov    0x2f(%rbx),%rdx
    mov    %rax,-0x18(%rbp)
    cmp    %rsp,(%rdx)
    jae    stack_check
post_stack_check:
    cmp    $0x2,%eax
    jl     return_n
    lea    -0x2(%rax),%edx
    mov    %rbx,%rsi
    mov    %rax,%r10
    mov    %rdx,%rax
    mov    %r10,%rdx
    callq  fib_tramp
    mov    -0x18(%rbp),%rbx
    sub    $0x1,%ebx
    mov    %rax,-0x20(%rbp)
    mov    -0x10(%rbp),%rsi
    mov    %rax,%r10
    mov    %rbx,%rax
    mov    %r10,%rbx
    callq  fib_tramp
return:
    mov    -0x20(%rbp),%rbx
    add    %ebx,%eax
    mov    %rbp,%rsp
    pop    %rbp
    retq   
return_n:
    jmp    return
stack_check:
    callq  WasmStackGuard
    mov    -0x10(%rbp),%rbx
    mov    -0x18(%rbp),%rax
    jmp    post_stack_check

Apparently fib compiles to a function of two arguments, the first passed in rsi, and the second in rax. (V8 uses a custom calling convention for its compiled WebAssembly.) The first synthesized argument is a handle onto run-time data structures for the current thread or isolate, and in the function prelude there's a check to see that the function has enough stack. V8 uses these stack checks also to handle interrupts, for when a web page is stuck in JavaScript.

Otherwise, it's a more or less normal function, with a bit more register/stack traffic than would be strictly needed, but pretty good.

do optimizations matter?

You've heard of Moore's Law -- though it doesn't apply any more, it roughly translated into hardware doubling in speed every 18 months. (Yes, I know it wasn't precisely that.) There is a corresponding rule of thumb for compiler land, Proebsting's Law: compiler optimizations make software twice as fast every 18 years. Zow!

The previous results with GCC and LLVM were with optimizations enabled (-O3). One way to measure Proebsting's Law would be to compare the results with -O0. Obviously in this case the program is small and we aren't expecting much work out of the optimizer, but it's interesting to see anyway:


Answer: optimizations don't matter much for this benchark. This investigation does give a good baseline for compilers from high-level languages, like Guile: in the absence of clever trickery like the recursive inlining thing GCC does and in the absence of industrial-strength instruction selection, what's a good baseline target for a compiler? Here we see for this benchmark that it's somewhere between 420 and 620 milliseconds or so. Go gets there, and OCaml does even better.

how is time being spent, anyway?

Might we expect V8/WebAssembly to get there soon enough, or is the stack check that costly? How much time does one stack check take anyway? For that we'd have to determine the number of recursive calls for a given invocation.

Friends, it's not entirely clear to me why this is, but I instrumented a copy of fib, and I found that the number of calls in fib(n) was a more or less constant factor of the result of calling fib. That ratio converges to twice the golden ratio, which means that since fib(n+1) ~= φ * fib(n), then the number of calls in fib(n) is approximately 2 * fib(n+1). I scratched my head for a bit as to why this is and I gave up; the Lord works in mysterious ways.

Anyway for fib(40), that means that there are around 3.31e8 calls, absent GCC shenanigans. So that would indicate that each call for clang takes around 1.27 ns, which at turbo-boost speeds on this machine is 4.44 cycles. At maximum throughput (4 IPC), that would indicate 17.8 instructions per call, and indeed on the n > 2 path I count 17 instructions.

For WebAssembly I calculate 2.25 nanoseconds per call, or 7.9 cycles, or 31.5 (fused) instructions at max IPC. And indeed counting the extra jumps in the trampoline, I get 33 cycles on the recursive path. I count 4 instructions for the stack check itself, one to save the current isolate, and two to shuffle the current isolate into place for the recursive calls. But, compared to clang, V8 puts 6 words on the stack per call, as opposed to only 4 for LLVM. I think with better interprocedural register allocation for the isolate (i.e.: reserve a register for it), V8 could get a nice boost for call-heavy workloads.

where are we? dynamic edition

Guile doesn't aim to replace C; it's different. It has garbage collection, an integrated debugger, and a compiler that's available at run-time, it is dynamically typed. It's perhaps more fair to compare to languages that have some of these characteristics, so I ran these tests on versions of recursive fib written in a number of languages. Note that all of the numbers in this post include start-up time.


Here, the ocamlc line is the same as before, but using the bytecode compiler instead of the native compiler. It's a bit of an odd thing to include but it performs so well I just had to include it.

I think the real takeaway here is that Chez Scheme has fantastic performance. I have not been able to see the disassembly -- does it do the trick like GCC does? -- but the numbers are great, and I can see why Racket decided to rebase its implementation on top of it.

Interestingly, as far as I understand, Chez implements stack checks in the straightfoward way (an inline test-and-branch), not with a guard page, and instead of using the stack check as a generic ability to interrupt a computation in a timely manner as V8 does, Chez emits a separate interrupt check. I would like to be able to see Chez's disassembly but haven't gotten around to figuring out how yet.

Since I originally published this article, I added a LuaJIT entry as well. As you can see, LuaJIT performs as well as Chez in this benchmark.

Haskell's call performance is surprisingly bad here, beaten even by OCaml's bytecode compiler; is this the cost of laziness, or just a lacuna of the implementation? I do not know. I do know I have this mental image that Haskell is a good compiler but apparently if that's the standard, so is Guile :)

Finally, in this comparison section, I was not surprised by cpython's relatively poor performance; we know cpython is not fast. I think though that it just goes to show how little these microbenchmarks are worth when it comes to user experience; like many of you I use plenty of Python programs in my daily work and don't find them slow at all. Think of micro-benchmarks like x-ray diffraction; they can reveal the hidden substructure of DNA but they say nothing at all about the organism.

where to now?

Perhaps you noted that in the last graph, the Guile and Chez lines were labelled "(lexical)". That's because instead of running this program:

(define (fib n)
  (if (< n 2)
      n
      (+ (fib (- n 1)) (fib (- n 2)))))

They were running this, instead:

(define (fib n)
  (define (fib* n)
    (if (< n 2)
        n
        (+ (fib* (- n 1)) (fib* (- n 2)))))
  (fib* n))

The thing is, historically, Scheme programs have treated top-level definitions as being mutable. This is because you don't know the extent of the top-level scope -- there could always be someone else who comes and adds a new definition of fib, effectively mutating the existing definition in place.

This practice has its uses. It's useful to be able to go in to a long-running system and change a definition to fix a bug or add a feature. It's also a useful way of developing programs, to incrementally build the program bit by bit.


But, I would say that as someone who as written and maintained a lot of Scheme code, it's not a normal occurence to mutate a top-level binding on purpose, and it has a significant performance impact. If the compiler knows the target to a call, that unlocks a number of important optimizations: type check elision on the callee, more optimal closure representation, smaller stack frames, possible contification (turning calls into jumps), argument and return value count elision, representation specialization, and so on.

This overhead is especially egregious for calls inside modules. Scheme-the-language only gained modules relatively recently -- relative to the history of scheme -- and one of the aspects of modules is precisely to allow reasoning about top-level module-level bindings. This is why running Chez Scheme with the --program option is generally faster than --script (which I used for all of these tests): it opts in to the "newer" specification of what a top-level binding is.

In Guile we would probably like to move towards a more static way of treating top-level bindings, at least those within a single compilation unit. But we haven't done so yet. It's probably the most important single optimization we can make over the near term, though.

As an aside, it seems that LuaJIT also shows a similar performance differential for local function fib(n) versus just plain function fib(n).

It's true though that even absent lexical optimizations, top-level calls can be made more efficient in Guile. I am not sure if we can reach Chez with the current setup of having a template JIT, because we need two return addresses: one virtual (for bytecode) and one "native" (for JIT code). Register allocation is also something to improve but it turns out to not be so important for fib, as there are few live values and they need to spill for the recursive call. But, we can avoid some of the indirection on the call, probably using an inline cache associated with the callee; Chez has had this optimization since 1984!

what guile learned from fib

This exercise has been useful to speed up Guile's procedure calls, as you can see for the difference between the latest Guile 2.9.2 release and what hasn't been released yet (2.9.3).

To decide what improvements to make, I extracted the assembly that Guile generated for fib to a standalone file, and tweaked it in a number of ways to determine what the potential impact of different scenarios was. Some of the detritus from this investigation is here.

There were three big performance improvements. One was to avoid eagerly initializing the slots in a function's stack frame; this took a surprising amount of run-time. Fortunately the rest of the toolchain like the local variable inspector was already ready for this change.

Another thing that became clear from this investigation was that our stack frames were too large; there was too much memory traffic. I was able to improve this in the lexical-call by adding an optimization to elide useless closure bindings. Usually in Guile when you call a procedure, you pass the callee as the 0th parameter, then the arguments. This is so the procedure has access to its closure. For some "well-known" procedures -- procedures whose callers can be enumerated -- we optimize to pass a specialized representation of the closure instead ("closure optimization"). But for well-known procedures with no free variables, there's no closure, so we were just passing a throwaway value (#f). An unhappy combination of Guile's current calling convention being stack-based and a strange outcome from the slot allocator meant that frames were a couple words too big. Changing to allow a custom calling convention in this case sped up fib considerably.

Finally, and also significantly, Guile's JIT code generation used to manually handle calls and returns via manual stack management and indirect jumps, instead of using the platform calling convention and the C stack. This is to allow unlimited stack growth. However, it turns out that the indirect jumps at return sites were stalling the pipeline. Instead we switched to use call/return but keep our manual stack management; this allows the CPU to use its return address stack to predict return targets, speeding up code.

et voilà

Well, long article! Thanks for reading. There's more to do but I need to hit the publish button and pop this off my stack. Until next time, happy hacking!

by Andy Wingo at June 26, 2019 10:34 AM

June 22, 2019

Eleni Maria Stea

Depth-aware upsampling experiments (Part 4: Improving the nearest depth where we detect discontinuities)

This is another post of the series where I explain some ideas I tried in order to improve the upscaling of the half-resolution SSAO render target of the VKDF sponza demo that was written by Iago Toral. In the previous post, I had classified the sample neighborhoods in surface neighborhoods and neighborhoods that contain depth … Continue reading Depth-aware upsampling experiments (Part 4: Improving the nearest depth where we detect discontinuities)

by hikiko at June 22, 2019 09:44 AM

June 18, 2019

Manuel Rego

Speaking at CSS Day 2019

This year I got invited to speak at CSS Day, this is an amazing event that every year brings to Amsterdam great speakers from all around the globe to talk about cutting edge topics related to CSS.

Pictures of speakers and MC at CSS Day 2019 Speakers and MC at CSS Day 2019

The conference happened in the beautiful Compagnietheater. Kudos to the organization as they were really kind and supportive during the whole event. Thanks for giving me the opportunity to speak there.

Compagnietheater in Amsterdam Compagnietheater in Amsterdam

For this event I prepared a totally new talk, focused on explaining what it takes to implement something like CSS Grid Layout in a browser engine. I took an idea from Jen Simmons and implemented a new property grid-skip-areas during the presentation, this was useful to explain the different things that happen during the whole process. Video of the talk is available on YouTube, and the slides are available if you are interested too; however, note that some of them won’t work on your browser unless you built the linked patches.

The feedback after the talk was really good, everyone seemed to like it (despite the fact that I showed lots of slides with C++ code) and find it useful to understand what’s going on behind the scenes. Thank you very much for your kind words! 😊


Somehow with this talk I was explaining the kind of things we do at Igalia and how we can help to fill the gaps left by browser vendors in the evolution of the Web Platform. Igalia has now a solid position inside the Web community, which makes us an excellent partner in order to improve the platform aligned with your specific needs. If you want to know more don’t hesitate to read our website or contact us directly.

I hope you enjoy it! 😀

June 18, 2019 10:00 PM

Samuel Iglesias

My latest VK-GL-CTS contributions

Even if you are not a gamer, odds are that you already heard about Vulkan graphics and compute API that provides high-efficency, cross-platform access to modern GPUs. This API is designed by the Khronos Group and it is supported by a new set of drivers specifically designed to implement the different functions and features defined by the spec (at the time of writing this post, it is version 1.1).

Vulkan

In order to guarantee that the drivers work according to the spec, drivers need to pass a conformance test suite that ensures they do what it is expected from them. VK-GL-CTS is the name of the conformance test suite used for certify the conformance on both Vulkan and OpenGL APIs and… it is open-source!

VK-GL-CTS

As part of my daily job at Igalia, I contribute to VK-GL-CTS from fixing some bugs, improving existing tests or even writing new tests for a variety of extensions. In this post I am going to describe some of the work I have been doing in the last few months.

VK_EXT_host_query_reset

This extension gives you the opportunity to reset queries outside a command buffer, which is a fast way of doing it once your application has finished reading query’s data. All that you need is to call vkResetQueryPoolEXT() function. There are several Vulkan drivers supporting already this extension on GNU/Linux (NVIDIA, open-source drivers AMDVLK, RADV and ANV) and probably more in other platforms.

I have implemented tests for all the different queries: occlusion queries, pipeline timestamp queries and statistics queries. Transform feedback stream queries tests landed a bit later.

VK_EXT_discard_rectangles

VK_EXT_discard_rectangles provides a way to define rectangles in framebuffer-space coordinates that discard rasterization of all points, lines and triangles that fall inside (exclusive mode) or outside (inclusive mode) of their area. You can regard this feature as something similar to scissor testing but it operates orthogonally to the existing scissor test functionality.

It is easier to understand with an example. Imagine that you want to do the following in your application: clear the color attachment to red color, draw a green quad covering the whole attachment but defining a discard rectangle in order to restrict the rasterization of the quad to the area defined by the discard rectangle.

For that, you define the discard rectangles at pipeline creation time for example (it is possible to define them dynamically too); as we want to restrict the rasterization of the quad to the area defined by the discard rectangle, then we set its mode to VK_DISCARD_RECTANGLE_MODE_INCLUSIVE_EXT.

VK_EXT_discard_rectangles inclusive mode example

If we want to discard the rasterization of the green quad inside the area defined by the discard rectangle, then we set VK_DISCARD_RECTANGLE_MODE_EXCLUSIVE_EXT mode at pipeline creation time and that’s all. Here you have the output for this case:

VK_EXT_discard_rectangles exclusive mode example

You are not limited to define just one discard rectangle, drivers supporting this extension should support a minimum of 4 of discard rectangles but some drivers may support more. As this feature works orthogonally to other tests like scissor test, you can do fancy things in your app :-)

The tests I developed for VK_EXT_discard_rectangles extension are already available in VK-GL-CTS repo. If you want to test them on an open-source driver, right now only RADV has implemented this extension.

VK_EXT_pipeline_creation_feedback

VK_EXT_pipeline_creation_feedback is another example of a useful feature for application developers, specially game developers. This extension gives a way to know at pipeline creation, if the pipeline hit the provided pipeline cache, the time consumed to create it or even which shaders stages hit the cache. This feature gives feedback about pipeline creation that can help to improve the pipeline caches that are shipped to users, with the final goal of reducing load times.

Tests for VK_EXT_pipeline_creation_feedback extension have made their way into VK-GL-CTS repo. Good news for the ones using open-source drivers: both RADV and ANV have implemented the support for this extension!

Conclusions

Since I started working in the Graphics team at Igalia, I have been contributing code to Mesa drivers for both OpenGL and Vulkan, adding new tests to Piglit, improve VkRunner among other contributions.

Now I am contributing to increase VK-GL-CTS coverage by developing new tests for extensions, fixing existing tests among other things. This work also involves developing patches for Vulkan Validation Layers, fixes for glslang and more things to come. In summary, I am enjoying a lot doing contributions to the open-source ecosystem created by Khronos Group as part of my daily work!

Note: if you are student and you want to start contributing to open-source projects, don’t miss our Igalia Coding Experience program (more info in our website).

Igalia

June 18, 2019 06:45 AM

June 14, 2019

Michael Catanzaro

An OpenJPEG Surprise

My previous blog post seems to have resolved most concerns about my requests for Ubuntu stable release updates, but I again received rather a lot of criticism for the choice to make WebKit depend on OpenJPEG, even though my previous post explained clearly why there are are not any good alternatives.

I was surprised to receive a pointer to ffmpeg, which has its own JPEG 2000 decoder that I did not know about. However, we can immediately dismiss this option due to legal problems with depending on ffmpeg. I also received a pointer to a resurrected libjasper, which is interesting, but since libjasper was removed from Ubuntu, its status is not currently better than OpenJPEG.

But there is some good news! I have looked through Ubuntu’s security review of the OpenJPEG code and found some surprising results. Half the reported issues affect the library’s companion tools, not the library itself. And the other half of the issues affect the libmj2 library, a component of OpenJPEG that is not built by Ubuntu and not used by WebKit. So while these are real security issues that raise concerns about the quality of the OpenJPEG codebase, none of them actually affect OpenJPEG as used by WebKit. Yay!

The remaining concern is that huge input sizes might cause problems within the library that we don’t yet know about. We don’t know because OpenJPEG’s fuzzer discards huge images instead of testing them. Ubuntu’s security team thinks there’s a good chance that fixing the fuzzer could uncover currently-unknown multiplication overflow issues, for instance, a class of vulnerability that OpenJPEG has clearly had trouble with in the past. It would be good to see improvement on this front. I don’t think this qualifies as a security vulnerability, but it is certainly a security problem that would facilitate discovering currently-unknown vulnerabilities if fixed.

Still, on the whole, the situation is not anywhere near as bad as I’d thought. Let’s hope OpenJPEG can be included in Ubuntu main sooner rather than later!

by Michael Catanzaro at June 14, 2019 02:43 PM