Planet Igalia Chromium

August 26, 2019

Miyoung Shin

Why should we use a precise-width integer type?

Introduction
A few months ago, I’ve finished the task of replacing imprecise-width integer types like (unsigned) short, int, long, long long by precise-width integer types like (u)int16/int32/int64_t in Blink.. I’ve been working on the Onion Soup project and it was my first task of this project. When I took this task, I thought it would be comparatively simple and quite mechanical work, but this was not the case. It was sometimes required to understand whole code flow in-depth even out of blink or to explain the benefit of such changes to reviewers.

I’d like to introduce why we should prefer using the precise-width integer type and what I did for this task.

Why should we use the precise-width integer type?
We should keep in mind the fact that the c++ standards don’t specify the specific size of each integer type.

Data Model Table

Data model sizeof(int) sizeof(long) sizeof(long long) example
LP32 16b 32b 64b Win16
ILP32 32b 32b 64b Win32, i386 OSX & Linux
LP64 32b 64b 64b x86-64 OSX & Linux
LLP64 32b 32b 64b Win64

Chromium supports various platforms with one repository and shows definitely the variable of different sizes with (unsigned) long between Android-Kitkat/MacOS/Win7 buildbots and other buildbots for try-bots. It means that it would have potential security issues like the stack overflow if we treat that long has 64 bits precise since the size of long is 64 bits like long long on almost all of build systems. And we should avoid mixing the use of the precise-width integer type and the imprecise-width integer type to store the same value.

Actually, after working on this I got a report mentioning that my changes to move from imprecise-width to precise-width types fixed a security issue. This made me realize that more existing issues could still be fixed this way, and I started fixing other ones reported via ClusterFuzz after that.

Google also recommends this in its own coding styleguide, see here.

Summary of my contributions in Blink
1) Decided the order of the changing type.

unsigned short -> uint16_t / short -> int16_t
unsigned long long -> uint64_t / long long -> int64_t
unsigned long -> uint32_t or uint64_t, long -> int32_t or int64_t

2) Wrote the patch including only related codes.

3) Found what is the proper type.
Which type is proper between uint32_t and uint64_t when I should change unsigned long?
Every time I asked myself the question since I had to consider the fact that unsigned long was 32 bit on Mac, Android-kitkat and Window7 buildbot and was 64 bit on others so far. So I needed to understand its code flow and the use cases not only in Blink but also out of Blink. In general, it would be best to avoid integer-width conversions where possible and to only convert to larger types when needed.

4) Utilized the template instead of the functions for each type
I reduced the code quantity with the template and avoided the build error by duplicated definition.

5) Used base::Optional when the initial value is ambiguous.
In general the variable’s initial value would be 0 or -1, but base::Optional is more readable and simple than comparing 0 or -1 to check the initial value. Actually, I needed to reach a consensus with the review since it could change the code logic.

bool HasValidSnapshotMetadata() const { return snapshot_size_ >= 0; }
long long snapshot_size_;

Instead,

bool HasValidSnapshotMetadata() const { return snapshot_size_.has_value(); }
base::Optional<uint64_t> snapshot_size_;

6) Used CheckedNumeric to check the overflow of the variable.

static bool SizeCalculationMayOverflow(unsigned width,
                                         unsigned height,
                                         unsigned decoded_bytes_per_pixel) {
    unsigned long long total_size = static_cast<unsigned long long>(width) *
                                    static_cast<unsigned long long>(height);
    if (decoded_bytes_per_pixel == 4)
      return total_size > ((1 << 29) - 1);
    return total_size > ((1 << 28) - 1);
}

The above code was the old one before I changed and it was for checking if the variable has the overflow, I took some time to fully understand it.
I’ve used CheckedNumeric since the reviewer let me know this utility. As you can see from the code below, it has been changed to be simple and readable.

  inline bool SizeCalculationMayOverflow(unsigned width,
                                         unsigned height,
                                         unsigned decoded_bytes_per_pixel) {
    base::CheckedNumeric<int32_t> total_size = width;
    total_size *= height;
    total_size *= decoded_bytes_per_pixel;
    return !total_size.IsValid();
  }

7) Oops! MSVC!
The below code shows the different results due to compilers.

class Foo {
    uint16_t a;
    unsigned b : 16;
};
std::cout << sizeof(Foo);

The results are from clang : 4 bytes, gcc : 4 bytes and msvc : 8 bytes

So, it caused the build break with MSVC toolchain on Windows buildbots as the class size was increased. It took a long time to analyze the cause and solve this issue.

In conclusion, it was very simple to fix. See the code below.

class Foo {
    unsigned a : 16;
    unsigned b : 16;
};

8) Brought out the potential issue with with the overflow.
Blink can handle the width and height within 4 bytes range. The old code showed the abnormal layout result with overflowed width/height but it did not crash. I added the assertion code to check if there is the overflow when changing the type. Finally, we prevent the overflow and solve to layout the object’s position and size within the maximum value of 4 bytes.

9) Added PRESUBMIT check.
Now that we cleaned up the Blink code from the imprecise-width types, a PRESUBMIT hook was added to prevent them from being reintroduced to the sourcebase. So I added the PRESUBMIT check to ensure new uses do not come in again.

10) Why do we still use int and unsigned?
This is the comment of the reviewer and I also agree with it.

Thee C++ style guide states "We use int very often, for integers we know are not going to be too big, e.g., loop counters. Use plain old int for such things. You should assume that an int is at least 32 bits, but don't assume that it has more than 32 bits. If you need a 64-bit integer type, use int64_t or uint64_t."
The style guide does state not to use the imprecise-width integers like long long/long/short, but int is the imprecise-width type that is explicitly preferred. Among other reasons, this is likely for readability and the fact that ints are by default the "native"/fastest type, and potentially large concerns over code churn. More there is not any benefit to replace int to int32t and potentially large concerns over code churn.

I’m happy to contribute to Chromium & Blink with more stable code through changing to the precise-width integer type and to learn more about this particular domain.

References
[1] https://en.cppreference.com/w/cpp/language/types
[2] http://nickdesaulniers.github.io/blog/2016/05/30/data-models-and-word-size/

by mshin at August 26, 2019 07:59 AM

August 12, 2019

Jacobo Aragunde

Spatial navigation operations for the webview tag

The modern web is a space meant to be browsed with a mouse or a touchscreen, but think of a smart TV, set-top box, game console or budget phone. What happens when your target hardware doesn’t have those devices and provides only directional input instead?

TV, remote, phone

If you are under control of the contents, you certainly can design your UI around directional input, providing custom focus management based on key events, and properly highlighting the focused item. But that doesn’t work when you confront external content provided by the millions of websites out there, not even if you are targeting a small subset of them like the most popular social networks and search engines.

Virtual cursors are one solution. They certainly address the problem and provide full functionality, but they are somehow inconvenient: traversing the screen can take too long, and precision can be an issue.

Tab navigation is always available and it certainly works well in sites designed for that, like forms, but it’s quite limited; it can also show unexpected behavior when the DOM tree doesn’t match the visual placement of the contents.

Spatial navigation is a feature designed to address this problem. It will:

  • respond to directional input taking into account the position of the elements of the screen
  • deal with scroll when we reach the limits of the viewport or a scrollable element, in a way that content can be easily read (e.g. no abrupt jumps to the next focusable element)
  • provide special highlight for the focused element.

You can see it in action in this BlinkOn lightning talk by Junho Seo, one of the main contributors. You can also test it yourself with the --enable-spatial-navigation flag in your Chrome/Chromium browser.

A non-trivial algorithm makes sure that input matches user expectations. That video shows a couple of corner cases that demonstrate this isn’t easy.

When creating a web UI for a device that makes use of cursor navigation, it’s usual that you have to deal with the two kinds of content mentioned before: the one created specifically for the device, which already takes cursor input into account in its design, and external content from websites that don’t follow those principles. It would be interesting to be able to isolate external content and provide spatial navigation for it.

The webview extension tag is one way to provide isolation for an external site. To support this use case, we implemented new API operations for the tag to enable or disable spatial navigation inside the webview contents independently from the global settings, and to check its state. They have been available for a while, since Chromium version 71.

There are some interesting details in the implementation of this operation. One is that it is required some IPC between different browser processes, because webview contents run on a different renderer (guest), which must be notified of the status changes in spatial navigation made by its “parent” (host). There is no need to do IPC in the other direction because we cache the value in the browser process for the implementation of isSpatialNavigationEnabled to use it.

Another interesting detail was, precisely, related with that cache; it made our test flaky because sometimes the calls to isSpatialNavigationEnabled happened before the call to setSpatialNavigationEnabled had completed its IPC communication. My first attempt to fix this added an ACK reply via IPC to confirm the cache value, but it was considered an overkill for a situation that would never trigger in a real-world scenario. It even was very hard to reproduce from the test! It was finally fixed with some workarounds in the test code.

A more general solution is proposed for all kinds of iframes in the CSS WG draft of the spatial navigation feature. I hope my previous work makes the implementation for the iframe tag straightforward.

by Jacobo Aragunde Pérez at August 12, 2019 06:00 PM

August 01, 2019

Julie Kim

Giving a talk at Automotive Linux Summit 2019

Last year, I’ve worked on Chromium and Web Application Manager on AGL platform with some fellow Igalian colleagues. Our goal was to support Chromium with upstream Wayland port and Web Application Manager. Although we faced a lot of challenges, such as porting them to AGL framework, handling SMACK on Chromium multiprocess architecture, supporting wayland ivi protocol on Chromium and so on, we completed our goal and showed a demo at CES 2019.

This year, I gave a talk about our work on AGL at several events such as AGL All Member Meeting, BlinkOn10 in Canada and Automotive Linux Summit 2019.

For Automotive Linux Summit 2019 in Tokyo last week, my colleague, Lorenzo, and I attended the event and managed our Igalia booth.


We showed some demos for Chromium and Web Application Manager on AGL on several reference boards such as Renesas m3 board, Intel Minnowboard, and RPi3. I was very pleased to meet many people and share our works with them.

I gave a talk about “HTML5 apps on AGL platform with the Web Application Manager“.


As you might know, Igalia has worked on Chromium Wayland port for several years. We tried various approaches and finally fully upstreamed our changes to upstream last year. Chromium on AGL has been implemented based on it with supporting ivi-extension.
LG opensourced Web Application Manager used for their products and Igalia implemented Web Runtime solution for AGL on the top of it. My colleague, Jacobo, uploaded a post about ‘Introducing the Chromium-based web runtime for the AGL platform‘. You can find more information there.

I also participated in the AGL Developer Panel in ALS.


We generally talked about the current status and the plans for the next step. Thanks to Walt Minor, Automotive Grade Linux Development Manager at The Linux Foundation, everything went well and I believe all of us enjoyed the time.

I think AGL is one of open source projects growing fast and it shows continuous improving. When I looked around demo booths, I could see many companies tried various interesting ideas on the top of the AGL.

Last, I’d like to say “Thanks a lot, people who organized this event, came by our Igalia booth, and listened to my talk”.

Automotive Grade Linux Logo

by jkim at August 01, 2019 06:31 AM

July 30, 2019

Manuel Rego

Talking about CSS Containment at CSSconf EU 2019

Back in January I wrote a blog post introducing CSS Containment where I tried to explain this specification and the main motivation and uses cases behind it. I’ve been working on css-contain implementation in Chromium during the last months as part of the ongoing collaboration between Igalia and Bloomberg, I believe this is not a very well known topic yet, and I have the chance to speak about it in the impressive CSSconf EU.

First of all it was a really great surprise to get my talk proposal accepted, Igalia has been attending CSSconf EU regularly but we never spoke there before. It’s one of the biggest CSS events and the setup they prepare (together with JSConf) is overwhelming, I’m very grateful to be part of the speakers lineup. Also the organization was very good and really paid attention to every detail.

CSSconf EU 2019 stage CSSconf EU 2019 stage

My talk’s title was Improving Website Performance with CSS Containment, during the talk I did an introduction to the spec, explained the contain property and the implications of the different types of containment. On top of that I showed a few examples about how the performance of a website can be improved thanks to css-contain. The video of my talk is published in YouTube and you can also get the slides from this blog.

After the presentation I got some positive feedback from the audience, it seems that despite of being a somehow deep topic people liked it. Thank you very much!

Last but not least, just to close this blog post I’d like to say thanks one more time to Bloomberg. They have been supporting Igalia’s work on CSS Containment implementation, which is helping to improve performance of their applications.

Igalia logo Bloomberg logo
Igalia and Bloomberg working together to build a better web

July 30, 2019 10:00 PM

June 26, 2019

Jacobo Aragunde

Introducing the Chromium-based web runtime for the AGL platform

Igalia has been working with AGL (Automotive Grade Linux) to provide a web application runtime to their platform, based on Chromium. We delivered the first phase of this project back in January, and it’s been available since the Flounder 6.0.5 and Guppy 7.0.0 releases. This is a summary of how it came to be, and the next steps into the future.

Igalia stand showing the AGL web runtime

Web applications as first-class citizens

The idea of web applications as first-class citizens is not new in AGL. The early versions of the stack, based on Tizen 3.0, already implemented this idea, via the Crosswalk runtime. It provided extended JS APIs for developers to access to system services inside a web application runtime.

The abandon of the Tizen 3.0 effort by its main developers meant the chance for AGL to start fresh, redefining its architecture to become what it is today, but the idea of web applications was still there. The current AGL architecture still supports web apps, and the system APIs are available through WebSockets so both native and web applications can use them. But, until now, there wasn’t a piece of software to run them other than a general-purpose browser.

Leveraging existing projects

One of the strengths of open source is speeding up the time-to-market by allowing code reuse. We looked into the WebOS OSE platform, developed by LGe with contributions from Igalia. It provides the Web Application Manager (WAM) component, capable of booting up a Chromium-based runtime with a low footprint and managing application life cycle. Reusing it, we were able to deliver a web application runtime more quickly.

Our contribution back to the webOS OSE project is the use of the new Wayland backend, independently developed by Igalia and available upstream in new Chromium versions. It was backported to the Chromium version used by WebOS OSE, and hopefully it will be part of future releases of the platform based on newer versions of Chromium.

Development process

My colleague Julie made a great presentation of the AGL web runtime in the last AGL All Member Meeting in March. I’m reusing her work for this overview.

Wayland

The version of Chromium provided by the WebOS OSE platform makes use of an updated version of the Intel Ozone-Wayland backend. This project has been abandoned for a while, and never upstreamed due to deep architecture differences. In the last couple of years, Igalia has implemented a new backend independently, using the lessons learned from integrating and maintaining Ozone-Wayland in previous projects, and following the current (and future) architecture of the video pipeline.

Compared process structure of Wayland implementations

As we mentioned before, we backported the new implementation of the Wayland backend to WebOS OSE, and added IVI-shell integration patches on top of it.

AGL life cycle

The WAM launcher process was modified to integrate with AGL life cycle callbacks and events. In particular, it registers event callbacks for HomeScreen, WindowManager and notification for ILMControl, activates WebApp window, when it gets Event_TapShortcut, and manages WebApp states for Event_Active/Event_Inactive. LGe, also a member of AGL, provided the initial work based on the Eel release, which we ported over to Flounder and kept evolving and maintaining.

WAM Launcher process integration diagram

AGL security model

In the AGL security model, access to system services is controlled with SMACK labels. A process with a certain label can only access a subset of the system API. The installation manifest for AGL applications and services is in charge of the relation between labels and services.

Access to system APIs in AGL happens through WebSockets so, from the point of view of the browser, it’s just a network request. Since WAM reuses the Chromium process model, the network interaction happening in any running webapp would actually be performed by the browser process. The problem is that there is only one browser process in the system, it’s running in the background and its labels don’t relate with the ones from the running applications. As a solution, we configure web applications to channel their networking through a proxy, and the WAM launcher creates a proxy process with the proper SMACK label for every webapp.

Proxy process integration diagram

Current implementation is based on the Tinyproxy project, but we are pending to review this model and find a more efficient solution.

Next steps

We are clearing any Qt dependencies from WAM, replacing them with standard C++ or the boost library. This work is interesting for AGL to be able to create a web-only version of the platform, and also for WebOS OSE to make the platform lighter, so we will be contributing it back. In this regard, the final goal is that AGL doesn’t require any patches on top of WebOS OSE to be able to use WAM.

Also, in the line of creating a web-only AGL, we will provide new demo apps so the Qt dependency wouldn’t be needed anywhere in the platform.

And, after the AGL face-to-face meeting that Igalia hosted in Coruña, we will be integrating more subsystems available in the platform, and reworking the integration with the security framework to be more robust and efficient. You can follow our progress in the AGL Jira with the label WebAppMgr.

Try it now!

The web application runtime is available in Flounder (starting in 6.0.5) and Guppy releases, and it will be in Halibut (current master). It can be tested adding the agl-html5-framework feature to the agl setup script. I have a bunch of test web applications available for testing in the repository wam-demo-applications, but there will be official demos in AGL soon.

Youtube running as an AGL app

Youtube running as an AGL application

This project has been made possible by the Linux Foundation, through a sponsorship from Automotive Grade Linux. Thanks!

AGL logo

Igalia

by Jacobo Aragunde Pérez at June 26, 2019 05:00 PM

Andy Wingo

fibs, lies, and benchmarks

Friends, consider the recursive Fibonacci function, expressed most lovelily in Haskell:

fib 0 = 0
fib 1 = 1
fib n = fib (n-1) + fib (n-2)

Computing elements of the Fibonacci sequence ("Fibonacci numbers") is a common microbenchmark. Microbenchmarks are like a Suzuki exercises for learning violin: not written to be good tunes (good programs), but rather to help you improve a skill.

The fib microbenchmark teaches language implementors to improve recursive function call performance.

I'm writing this article because after adding native code generation to Guile, I wanted to check how Guile was doing relative to other language implementations. The results are mixed. We can start with the most favorable of the comparisons: Guile present versus Guile of the past.


I collected these numbers on my i7-7500U CPU @ 2.70GHz 2-core laptop, with no particular performance tuning, running each benchmark 10 times, waiting 2 seconds between measurements. The bar value indicates the median elapsed time, and above each bar is an overlayed histogram of all results for that scenario. Note that the y axis is on a log scale. The 2.9.3* version corresponds to unreleased Guile from git.

Good news: Guile has been getting significantly faster over time! Over decades, true, but I'm pleased.

where are we? static edition

How good are Guile's numbers on an absolute level? It's hard to say because there's no absolute performance oracle out there. However there are relative performance oracles, so we can try out perhaps some other language implementations.

First up would be the industrial C compilers, GCC and LLVM. We can throw in a few more "static" language implementations as well: compilers that completely translate to machine code ahead-of-time, with no type feedback, and a minimal run-time.


Here we see that GCC is doing best on this benchmark, completing in an impressive 0.304 seconds. It's interesting that the result differs so much from clang. I had a look at the disassembly for GCC and I see:

fib:
    push   %r12
    mov    %rdi,%rax
    push   %rbp
    mov    %rdi,%rbp
    push   %rbx
    cmp    $0x1,%rdi
    jle    finish
    mov    %rdi,%rbx
    xor    %r12d,%r12d
again:
    lea    -0x1(%rbx),%rdi
    sub    $0x2,%rbx
    callq  fib
    add    %rax,%r12
    cmp    $0x1,%rbx
    jg     again
    and    $0x1,%ebp
    lea    0x0(%rbp,%r12,1),%rax
finish:
    pop    %rbx
    pop    %rbp
    pop    %r12
    retq   

It's not quite straightforward; what's the loop there for? It turns out that GCC inlines one of the recursive calls to fib. The microbenchmark is no longer measuring call performance, because GCC managed to reduce the number of calls. If I had to guess, I would say this optimization doesn't have a wide applicability and is just to game benchmarks. In that case, well played, GCC, well played.

LLVM's compiler (clang) looks more like what we'd expect:

fib:
   push   %r14
   push   %rbx
   push   %rax
   mov    %rdi,%rbx
   cmp    $0x2,%rdi
   jge    recurse
   mov    %rbx,%rax
   add    $0x8,%rsp
   pop    %rbx
   pop    %r14
   retq   
recurse:
   lea    -0x1(%rbx),%rdi
   callq  fib
   mov    %rax,%r14
   add    $0xfffffffffffffffe,%rbx
   mov    %rbx,%rdi
   callq  fib
   add    %r14,%rax
   add    $0x8,%rsp
   pop    %rbx
   pop    %r14
   retq   

I bolded the two recursive calls.

Incidentally, the fib as implemented by GCC and LLVM isn't quite the same program as Guile's version. If the result gets too big, GCC and LLVM will overflow, whereas in Guile we overflow into a bignum. Also in C, it's possible to "smash the stack" if you recurse too much; compilers and run-times attempt to mitigate this danger but it's not completely gone. In Guile you can recurse however much you want. Finally in Guile you can interrupt the process if you like; the compiled code is instrumented with safe-points that can be used to run profiling hooks, debugging, and so on. Needless to say, this is not part of C's mission.

Some of these additional features can be implemented with no significant performance cost (e.g., via guard pages). But it's fair to expect that they have some amount of overhead. More on that later.

The other compilers are OCaml's ocamlopt, coming in with a very respectable result; Go, also doing well; and V8 WebAssembly via Node. As you know, you can compile C to WebAssembly, and then V8 will compile that to machine code. In practice it's just as static as any other compiler, but the generated assembly is a bit more involved:


fib_tramp:
    jmp    fib

fib:
    push   %rbp
    mov    %rsp,%rbp
    pushq  $0xa
    push   %rsi
    sub    $0x10,%rsp
    mov    %rsi,%rbx
    mov    0x2f(%rbx),%rdx
    mov    %rax,-0x18(%rbp)
    cmp    %rsp,(%rdx)
    jae    stack_check
post_stack_check:
    cmp    $0x2,%eax
    jl     return_n
    lea    -0x2(%rax),%edx
    mov    %rbx,%rsi
    mov    %rax,%r10
    mov    %rdx,%rax
    mov    %r10,%rdx
    callq  fib_tramp
    mov    -0x18(%rbp),%rbx
    sub    $0x1,%ebx
    mov    %rax,-0x20(%rbp)
    mov    -0x10(%rbp),%rsi
    mov    %rax,%r10
    mov    %rbx,%rax
    mov    %r10,%rbx
    callq  fib_tramp
return:
    mov    -0x20(%rbp),%rbx
    add    %ebx,%eax
    mov    %rbp,%rsp
    pop    %rbp
    retq   
return_n:
    jmp    return
stack_check:
    callq  WasmStackGuard
    mov    -0x10(%rbp),%rbx
    mov    -0x18(%rbp),%rax
    jmp    post_stack_check

Apparently fib compiles to a function of two arguments, the first passed in rsi, and the second in rax. (V8 uses a custom calling convention for its compiled WebAssembly.) The first synthesized argument is a handle onto run-time data structures for the current thread or isolate, and in the function prelude there's a check to see that the function has enough stack. V8 uses these stack checks also to handle interrupts, for when a web page is stuck in JavaScript.

Otherwise, it's a more or less normal function, with a bit more register/stack traffic than would be strictly needed, but pretty good.

do optimizations matter?

You've heard of Moore's Law -- though it doesn't apply any more, it roughly translated into hardware doubling in speed every 18 months. (Yes, I know it wasn't precisely that.) There is a corresponding rule of thumb for compiler land, Proebsting's Law: compiler optimizations make software twice as fast every 18 years. Zow!

The previous results with GCC and LLVM were with optimizations enabled (-O3). One way to measure Proebsting's Law would be to compare the results with -O0. Obviously in this case the program is small and we aren't expecting much work out of the optimizer, but it's interesting to see anyway:


Answer: optimizations don't matter much for this benchark. This investigation does give a good baseline for compilers from high-level languages, like Guile: in the absence of clever trickery like the recursive inlining thing GCC does and in the absence of industrial-strength instruction selection, what's a good baseline target for a compiler? Here we see for this benchmark that it's somewhere between 420 and 620 milliseconds or so. Go gets there, and OCaml does even better.

how is time being spent, anyway?

Might we expect V8/WebAssembly to get there soon enough, or is the stack check that costly? How much time does one stack check take anyway? For that we'd have to determine the number of recursive calls for a given invocation.

Friends, it's not entirely clear to me why this is, but I instrumented a copy of fib, and I found that the number of calls in fib(n) was a more or less constant factor of the result of calling fib. That ratio converges to twice the golden ratio, which means that since fib(n+1) ~= φ * fib(n), then the number of calls in fib(n) is approximately 2 * fib(n+1). I scratched my head for a bit as to why this is and I gave up; the Lord works in mysterious ways.

Anyway for fib(40), that means that there are around 3.31e8 calls, absent GCC shenanigans. So that would indicate that each call for clang takes around 1.27 ns, which at turbo-boost speeds on this machine is 4.44 cycles. At maximum throughput (4 IPC), that would indicate 17.8 instructions per call, and indeed on the n > 2 path I count 17 instructions.

For WebAssembly I calculate 2.25 nanoseconds per call, or 7.9 cycles, or 31.5 (fused) instructions at max IPC. And indeed counting the extra jumps in the trampoline, I get 33 cycles on the recursive path. I count 4 instructions for the stack check itself, one to save the current isolate, and two to shuffle the current isolate into place for the recursive calls. But, compared to clang, V8 puts 6 words on the stack per call, as opposed to only 4 for LLVM. I think with better interprocedural register allocation for the isolate (i.e.: reserve a register for it), V8 could get a nice boost for call-heavy workloads.

where are we? dynamic edition

Guile doesn't aim to replace C; it's different. It has garbage collection, an integrated debugger, and a compiler that's available at run-time, it is dynamically typed. It's perhaps more fair to compare to languages that have some of these characteristics, so I ran these tests on versions of recursive fib written in a number of languages. Note that all of the numbers in this post include start-up time.


Here, the ocamlc line is the same as before, but using the bytecode compiler instead of the native compiler. It's a bit of an odd thing to include but it performs so well I just had to include it.

I think the real takeaway here is that Chez Scheme has fantastic performance. I have not been able to see the disassembly -- does it do the trick like GCC does? -- but the numbers are great, and I can see why Racket decided to rebase its implementation on top of it.

Interestingly, as far as I understand, Chez implements stack checks in the straightfoward way (an inline test-and-branch), not with a guard page, and instead of using the stack check as a generic ability to interrupt a computation in a timely manner as V8 does, Chez emits a separate interrupt check. I would like to be able to see Chez's disassembly but haven't gotten around to figuring out how yet.

Since I originally published this article, I added a LuaJIT entry as well. As you can see, LuaJIT performs as well as Chez in this benchmark.

Haskell's call performance is surprisingly bad here, beaten even by OCaml's bytecode compiler; is this the cost of laziness, or just a lacuna of the implementation? I do not know. I do know I have this mental image that Haskell is a good compiler but apparently if that's the standard, so is Guile :)

Finally, in this comparison section, I was not surprised by cpython's relatively poor performance; we know cpython is not fast. I think though that it just goes to show how little these microbenchmarks are worth when it comes to user experience; like many of you I use plenty of Python programs in my daily work and don't find them slow at all. Think of micro-benchmarks like x-ray diffraction; they can reveal the hidden substructure of DNA but they say nothing at all about the organism.

where to now?

Perhaps you noted that in the last graph, the Guile and Chez lines were labelled "(lexical)". That's because instead of running this program:

(define (fib n)
  (if (< n 2)
      n
      (+ (fib (- n 1)) (fib (- n 2)))))

They were running this, instead:

(define (fib n)
  (define (fib* n)
    (if (< n 2)
        n
        (+ (fib* (- n 1)) (fib* (- n 2)))))
  (fib* n))

The thing is, historically, Scheme programs have treated top-level definitions as being mutable. This is because you don't know the extent of the top-level scope -- there could always be someone else who comes and adds a new definition of fib, effectively mutating the existing definition in place.

This practice has its uses. It's useful to be able to go in to a long-running system and change a definition to fix a bug or add a feature. It's also a useful way of developing programs, to incrementally build the program bit by bit.


But, I would say that as someone who as written and maintained a lot of Scheme code, it's not a normal occurence to mutate a top-level binding on purpose, and it has a significant performance impact. If the compiler knows the target to a call, that unlocks a number of important optimizations: type check elision on the callee, more optimal closure representation, smaller stack frames, possible contification (turning calls into jumps), argument and return value count elision, representation specialization, and so on.

This overhead is especially egregious for calls inside modules. Scheme-the-language only gained modules relatively recently -- relative to the history of scheme -- and one of the aspects of modules is precisely to allow reasoning about top-level module-level bindings. This is why running Chez Scheme with the --program option is generally faster than --script (which I used for all of these tests): it opts in to the "newer" specification of what a top-level binding is.

In Guile we would probably like to move towards a more static way of treating top-level bindings, at least those within a single compilation unit. But we haven't done so yet. It's probably the most important single optimization we can make over the near term, though.

As an aside, it seems that LuaJIT also shows a similar performance differential for local function fib(n) versus just plain function fib(n).

It's true though that even absent lexical optimizations, top-level calls can be made more efficient in Guile. I am not sure if we can reach Chez with the current setup of having a template JIT, because we need two return addresses: one virtual (for bytecode) and one "native" (for JIT code). Register allocation is also something to improve but it turns out to not be so important for fib, as there are few live values and they need to spill for the recursive call. But, we can avoid some of the indirection on the call, probably using an inline cache associated with the callee; Chez has had this optimization since 1984!

what guile learned from fib

This exercise has been useful to speed up Guile's procedure calls, as you can see for the difference between the latest Guile 2.9.2 release and what hasn't been released yet (2.9.3).

To decide what improvements to make, I extracted the assembly that Guile generated for fib to a standalone file, and tweaked it in a number of ways to determine what the potential impact of different scenarios was. Some of the detritus from this investigation is here.

There were three big performance improvements. One was to avoid eagerly initializing the slots in a function's stack frame; this took a surprising amount of run-time. Fortunately the rest of the toolchain like the local variable inspector was already ready for this change.

Another thing that became clear from this investigation was that our stack frames were too large; there was too much memory traffic. I was able to improve this in the lexical-call by adding an optimization to elide useless closure bindings. Usually in Guile when you call a procedure, you pass the callee as the 0th parameter, then the arguments. This is so the procedure has access to its closure. For some "well-known" procedures -- procedures whose callers can be enumerated -- we optimize to pass a specialized representation of the closure instead ("closure optimization"). But for well-known procedures with no free variables, there's no closure, so we were just passing a throwaway value (#f). An unhappy combination of Guile's current calling convention being stack-based and a strange outcome from the slot allocator meant that frames were a couple words too big. Changing to allow a custom calling convention in this case sped up fib considerably.

Finally, and also significantly, Guile's JIT code generation used to manually handle calls and returns via manual stack management and indirect jumps, instead of using the platform calling convention and the C stack. This is to allow unlimited stack growth. However, it turns out that the indirect jumps at return sites were stalling the pipeline. Instead we switched to use call/return but keep our manual stack management; this allows the CPU to use its return address stack to predict return targets, speeding up code.

et voilà

Well, long article! Thanks for reading. There's more to do but I need to hit the publish button and pop this off my stack. Until next time, happy hacking!

by Andy Wingo at June 26, 2019 10:34 AM

June 18, 2019

Manuel Rego

Speaking at CSS Day 2019

This year I got invited to speak at CSS Day, this is an amazing event that every year brings to Amsterdam great speakers from all around the globe to talk about cutting edge topics related to CSS.

Pictures of speakers and MC at CSS Day 2019 Speakers and MC at CSS Day 2019

The conference happened in the beautiful Compagnietheater. Kudos to the organization as they were really kind and supportive during the whole event. Thanks for giving me the opportunity to speak there.

Compagnietheater in Amsterdam Compagnietheater in Amsterdam

For this event I prepared a totally new talk, focused on explaining what it takes to implement something like CSS Grid Layout in a browser engine. I took an idea from Jen Simmons and implemented a new property grid-skip-areas during the presentation, this was useful to explain the different things that happen during the whole process. Video of the talk is available on YouTube, and the slides are available if you are interested too; however, note that some of them won’t work on your browser unless you built the linked patches.

The feedback after the talk was really good, everyone seemed to like it (despite the fact that I showed lots of slides with C++ code) and find it useful to understand what’s going on behind the scenes. Thank you very much for your kind words! 😊


Somehow with this talk I was explaining the kind of things we do at Igalia and how we can help to fill the gaps left by browser vendors in the evolution of the Web Platform. Igalia has now a solid position inside the Web community, which makes us an excellent partner in order to improve the platform aligned with your specific needs. If you want to know more don’t hesitate to read our website or contact us directly.

I hope you enjoy it! 😀

June 18, 2019 10:00 PM

June 10, 2019

Javier Fernández

A new terminal-style line breaking with CSS Text

The CSS Text 3 specification defines a module for text manipulation and covers, among a few other features, the line breaking behavior of the browser, including white space handling. I’ve been working lately on some new features and bug fixing for this specification and I’d like to introduce in this posts the last one we made available for the Web Platform users. This is yet another contribution that came out the collaboration between Igalia and Bloomberg, which has been held for several years now and has produced many important new features for the Web, like CSS Grid Layout.

The feature

I guess everybody knows the white-space CSS property, which allows web authors to control two main aspects of the rendering of a text line: collapsing and wrapping. A new value break-spaces has been added to the ones available for this property, which allows web authors to emulate a terminal-like line breaking behavior. This new value operates basically like pre-wrap, but with two key differences:

  • any sequence of preserved white space characters takes up space, even at the end of the line.
  • a preserved white space sequence can be wrapped at any character, moving the rest of the sequence, intact, to the line bellow.

What does this new behavior actually mean ? I’ll try to explain it with a few examples. Lets start with a simple but quite illustrative demo which tries to emulate a meteorology monitoring system which shows relevant changes over time, where the gaps between subsequent changes must be preserved:



 #terminal {
  font: 20px/1 monospace;
  width: 340px;
  height: 5ch;
  background: black;
  color: green;
  overflow: hidden;
  white-space: break-spaces;
  word-break: break-all;
 }


    

Another interesting use case for this feature could be a logging system which should preserve the text formatting of the logged information, considering different window sizes. The following demo tries to describe this such scenario:



body { width: 1300px; }
#logging {
  font: 20px/1 monospace;
  background: black;
  color: green;

  animation: resize 7s infinite alternate;

  white-space: break-spaces;
  word-break: break-all;
}
@keyframes resize {
  0% { width: 25%; }
  100% { width: 100%; }
}

Hash: 5a2a3d23f88174970ed8 Version: webpack 3.12.0 Time: 22209ms Asset Size Chunks Chunk Names pages/widgets/index.51838abe9967a9e0b5ff.js 1.17 kB 10 [emitted] pages/widgets/index img/icomoon.7f1da5a.svg 5.38 kB [emitted] fonts/icomoon.2d429d6.ttf 2.41 kB [emitted] img/fontawesome-webfont.912ec66.svg 444 kB [emitted] [big] fonts/fontawesome-webfont.b06871f.ttf 166 kB [emitted] img/mobile.8891a7c.png 39.6 kB [emitted] img/play_button.6b15900.png 14.8 kB [emitted] img/keyword-back.f95e10a.jpg 43.4 kB [emitted] . . .

Use cases

In the demo shown before there are several cases that I think it’s worth to analyze in detail.

A breaking opportunity exists after any white space character

The main purpose of this feature is to preserve the white space sequences length even when it has to be wrapped into multiple lines. The following example tries to describe this basic use case:



.container {
  font: 20px/1 monospace;
  width: 5ch;
  white-space: break-spaces;
  border: 1px solid;
}

XX XX

The example above shows how the white space sequence with a length of 15 characters is preserved and wrapped along 3 different lines.

Single leading white space

Before the addition of the break-spaces value this scenario was only possible at the beginning of the line. In any other case, the trailing white spaces were either collapsed or hang, hence the next line couldn’t start with a sequence of white spaces. Lets consider the following example:



.container {
  font: 20px/1 monospace;
  width: 3ch;
  white-space: break-spaces;
  border: 1px solid;
}

XX XX

Like when using pre-wrap, the single leading space is preserved. Since break-spaces allows breaking opportunities after any white space character, we break after the first leading white space (” |XX XX”). The second line can be broken after the first preserved white space, creating another leading white space in the next line (” |XX | XX”).

However, lets consider now a case without such first single leading white space.



.container {
  font: 20px/1 monospace;
  width: 3ch;
  white-space: break-spaces;
  border: 1px solid;
}

XXX XX

Again, it s not allowed to break before the first space, but in this case there isn’t any previous breaking opportunity, so the first space after the word XX should overflow (“XXX | XX”); the next white space character will be moved down to the next line as preserved leading space.

Breaking before the first white space

I mentioned before that the spec states clearly that the break-space feature allows breaking opportunities only after white space characters. However, it’d be possible to break the line just before the first white space character after a word if the feature is used in combination with other line breaking CSS properties, like word-break or overflow-wrap (and other properties too).



.container {
  font: 20px/1 monospace;
  width: 4ch;
  white-space: break-spaces;
  overflow-wrap: break-word;
  border: 1px solid;
}

XXXX X

The two white spaces between the words are preserved due to the break-spaces feature, but the first space after the XXXX word would overflow. Hence, the overflow-wrap: break-word feature is applied to prevent the line to overflow and introduce an additional breaking opportunity just before the first space after the word. This behavior causes that the trailing spaces are moved down as a leading white space sequence in the next line.

We would get the same rendering if word-break: break-all is used instead overflow-wrap (or even in combination), but this is actualy an incorrect behavior, which has the corresponding bug reports in WebKit (197277) and Blink (952254) according to the discussion in the CSS WG (see issue #3701).

Consider previous breaking opportunities

In the previous example I described a combination of line breaking features that would allow breaking before the first space after a word. However, this should be avoided if there are previous breaking opportunities. The following example is one of the possible scenarios where this may happen:



.container {
  font: 20px/1 monospace;
  width: 4ch;
  white-space: break-spaces;
  overflow-wrap: break-word;
  border: 1px solid;
}

XX X X

In this case, we could break after the second word (“XX X| X”), since overflow-wrap: break-word would allow us to do that in order to avoid the line to overflow due to the following white space. However, white-space: break-spaces only allows breaking opportunities after a space character, hence, we shouldn’t break before if there are valid previous opportunities, like in this case in the space after the first word (“XX |X X”).

This preference for previous breaking opportunities before breaking the word, honoring the overflow-wrap property, is also part of the behavior defined for the white-space: pre-wrap feature; although in that case, there is no need to deal with the issue of breaking before the first space after a word since trailing space will just hang. The following example uses just the pre-wrap to show how previous opportunities are selected to avoid overflow or breaking a word (unless explicitly requested by word-break property).



.container {
  font: 20px/1 monospace;
  width: 2ch;
  white-space: pre-wrap;
  border: 1px solid;
}

XX
overflow-wrap:
break-word
word-break:
break-all

In this case, break-all enables breaking opportunities that are not available otherwise (we can break a word at any letter), which can be used to prevent the line to overflow; hence, the overflow-wrap property doesn’t take any effect. The existence of previous opportunities is not considered now, since break-all mandates to produce the longer line as possible.

This new white-space: break-spaces feature implies a different behavior when used in combination with break-all. Even though the preference of previous opportunities should be ignored if we use the word-break: break-all, this may not be the case for the breaking before the first space after a word scenario. Lets consider the same example but using now the word-break: break-all feature:



.container {
  font: 20px/1 monospace;
  width: 4ch;
  white-space: break-spaces;
  overflow-wrap: break-word;
  word-break: break-all;
  border: 1px solid;
}

XX X X

The example above shows that using word-break: break-all doesn’t produce any effect. It’s debatable whether the use of break-all should force the selection of the breaking opportunity that produces the longest line, like it happened in the pre-wrap case described before. However, the spec states clearly that break-spaces should only allow breaking opportunities after white space characters. Hence, I considered that breaking before the first space should only happen if there is no other choice.

As a matter of fact, specifying break-all we shouldn’t considering only previous white spaces, to avoid breaking before the first white space after a word; the break-all feature creates additional breaking opportunities, indeed, since it allows to break the word at any character. Since break-all is intended to produce the longest line as possible, this new breaking opportunity should be chosen over any previous white space. See the following test case to get a clearer idea of this scenario:



.container {
  font: 20px/1 monospace;
  width: 4ch;
  white-space: break-spaces;
  overflow-wrap: break-word;
  word-break: break-all;
  border: 1px solid;
}

X XX X

Bear in mind that the expected rendering in the above example may not be obtained if your browser’s version is still affected by the bugs 197277(Safari/WebKit) and 952254(Chrome/Blink). In this case, the word is broken despite the opportunity in the previous white space, and also avoiding breaking after the ‘XX’ word, just before the white space.

There is an exception to the rule of avoiding breaking before the first white space after a word if there are previous opportunities, and it’s precisely the behavior the line-break: anywhere feature would provide. As I said, all these assumptions were not, in my opinion, clearly defined in the current spec, so that’s why I filed an issue for the CSS WG so that we can clarify when it’s allowed to break before the first space.

Current status and support

The intent-to-ship request for Chrome has been approved recently, so I’m confident the feature will be enabled by default in Chrome 76. However, it’s possible to try the feature in older versions by enabling the Experimental Web Platform Features flag. More details in the corresponding Chrome Status entry. I want to highlight that I also implemented the feature for LayoutNG, the new layout engine that Chrome will eventually ship; this achievement is very important to ensure the stability of the feature in future versions of Chrome.

In the case of Safari, the patch with the implementation of the feature landed in the WebKit’s trunk in r244036, but since Apple doesn’t announce publicly when a new release of Safari will happen or which features it’ll ship, it’s hard to guess when the break-spaces feature will be available for the web authors using such browser. Meanwhile, It’s possible to try the feature in the Safari Technology Preview 80.

Finally, while I haven’t see any signal of active development in Firefox, some of the Mozilla developers working on this area of the Gecko engine have shown public support for the feature.

The following table summarizes the support of the break-spaces feature in the 3 main browsers:

Chrome Safari Firefox
Experimental M73 STP 80 Public support
Ship M76 Unknown Unknown

Web Platform Tests

At Igalia we believe that the Web Platform Tests project is a key piece to ensure the compatibility and interoperability of any development on the Web Platform. That’s why a substantial part of my work to implement this relatively small feature was the definition of enough tests to cover the new functionality and basic use cases of the feature.

white-space overflow-wrap word-break
pre-wrap-008
pre-wrap-015
pre-wrap-016
break-spaces-003
break-spaces-004
break-spaces-005
break-spaces-006
break-spaces-007
break-spaces-008
break-spaces-009
break-word-004
break-word-005
break-word-006
break-word-007
break-word-008
break-all-010
break-all-011
break-all-012
break-all-013
break-all-014
break-all-015

Implementation in several web engines

During the implementation of a browser feature, even a small one like this, it’s quite usual to find out bugs and interoperability issues. Even though this may slow down the implementation of the feature, it’s also a source of additional Web Platform tests and it may contribute to the robustness of the feature itself and the related CSS properties and values. That’s why I decided to implement the feature in parallel for WebKit (Safari) and Blink (Chrome) engines, which I think it helped to ensure interoperability and code maturity. This approach also helped to get a deeper understanding of the line breaking logic and its design and implementation in different web engines.

I think it’s worth mentioning some of these code architectural differences, to get a better understanding of the work and challenges this feature required until it reached web author’s browser.

Chrome/Blink engine

Lets start with Chrome/Blink, which was especially challenging due to the fact that Blink is implementing a new layout engine (LayoutNG). The implementation for the legacy layout engine was the first step, since it ensures the feature will arrive earlier, even behind an experimental runtime flag.

The legacy layout relies on the BreakingContext class to implement the line breaking logic for the inline layout operations. It has the main characteristic of handling the white space breaking opportunities by its own, instead of using the TextBreakIterator (based on ICU libraries), as it does for determining breaking opportunities between letters and/or symbols. This design implies too much complexity to do even small changes like this, especially because is very sensible in terms of performance impact. In the following diagram I try to show a simplified view of the classes involved and the interactions implemented by this line breaking logic.

The LayoutNG line breaking logic is based on a new concept of fragments, mainly handled by the NGLineBreaker class. This new design simplifies the line breaking logic considerably and it’s highly optimized and adapted to get the most of the TextBreakIterator classes and the ICU features. I tried to show a simplified view of this new design with the following diagram:

In order to describe the work done to implement the feature for this web engine, I’ll list the main bugs and patches landed during this time: CR#956465, CR#952254, CR#944063,CR#900727, CR#767634, CR#922437

Safari/WebKit engine

Although as time passes this is less probable, WebKit and Blink still share some of the layout logic from the ages prior to the fork. Although Blink engineers have applied important changes to the inline layout logic, both code refactoring and optimizations, there are common design patterns that made relatively easy porting to WebKit the patches that implemented the feature for the Blink’s legacy layout. In WebKit, the line breaking logic is also implemented by the BreakingContext class and it has a similar architecture, as it’s described, in a quite simplified way, in the class diagram above (it uses different class names for the render/layout objects, though) .

However, Safari supports for the mac and iOS platforms a different code path for the line breaking logic, implemented in the SimpleLineLayout class. This class provides a different design for the line breaking logic, and, similar to what Blink implements in LayoutNG, is based on a concept of text fragments. It also relies as much as possible into the TextBreakIterator, instead of implementing complex rules to handle white spaces and breaking opportunities. The following diagrams show this alternate design to implement the line breaking process.

This SimpleLineLayout code path in not supported by other WebKit ports (like WebKitGtk+ or WPE) and it’s not available either when using some CSS Text features or specific fonts. There are other limitations to use this SimpleLineLayout codepath, which may lead to render the text using the BreakingContext class.

Again, this is the list of bugs that were solved to implement the feature for the WebKit engine: WK#197277, WK#196169, WK#196353, WK#195361, WK#177327, WK#197278

Conclusion

I hope that at this point these 2 facts are clear now:

  • The white-space: break-spaces feature is a very simple but powerful feature that provides a new line breaking behavior, based on unix-terminal systems.
  • Although it’s a simple feature, on the paper (spec), it implies a considerable amount of work so that it reaches the browser and it’s available for web authors.

In this post I tried to explain in a simple way the main purpose of this new feature and also some interesting corner cases and combinations with other Line Breaking features. The demos I used shown 2 different use cases of this feature, but there are may more. I’m sure the creativity of web authors will push the feature to the limits; by then, I’ll be happy to answer doubts, about the spec or the implementation for the web engines, and of course fix the bugs that may appear once the feature is more used.

Igalia logo
Bloomberg logo

Igalia and Bloomberg working together to build a better web

Finally, I want to thank Bloomberg for supporting the work to implement this feature. It’s another example of how non-browser vendors can influence the Web Platform and contribute with actual features that will be eventually available for web authors. This is the kind of vision that we need if we want to keep a healthy, open and independent Web Platform.

by jfernandez at June 10, 2019 08:11 PM

May 23, 2019

Andy Wingo

bigint shipping in firefox!

I am delighted to share with folks the results of a project I have been helping out on for the last few months: implementation of "BigInt" in Firefox, which is finally shipping in Firefox 68 (beta).

what's a bigint?

BigInts are a new kind of JavaScript primitive value, like numbers or strings. A BigInt is a true integer: it can take on the value of any finite integer (subject to some arbitrarily large implementation-defined limits, such as the amount of memory in your machine). This contrasts with JavaScript number values, which have the well-known property of only being able to precisely represent integers between -253 and 253.

BigInts are written like "normal" integers, but with an n suffix:

var a = 1n;
var b = a + 42n;
b << 64n
// result: 793209995169510719488n

With the bigint proposal, the usual mathematical operations (+, -, *, /, %, <<, >>, **, and the comparison operators) are extended to operate on bigint values. As a new kind of primitive value, bigint values have their own typeof:

typeof 1n
// result: 'bigint'

Besides allowing for more kinds of math to be easily and efficiently expressed, BigInt also allows for better interoperability with systems that use 64-bit numbers, such as "inodes" in file systems, WebAssembly i64 values, high-precision timers, and so on.

You can read more about the BigInt feature over on MDN, as usual. You might also like this short article on BigInt basics that V8 engineer Mathias Bynens wrote when Chrome shipped support for BigInt last year. There is an accompanying language implementation article as well, for those of y'all that enjoy the nitties and the gritties.

can i ship it?

To try out BigInt in Firefox, simply download a copy of Firefox Beta. This version of Firefox will be fully released to the public in a few weeks, on July 9th. If you're reading this in the future, I'm talking about Firefox 68.

BigInt is also shipping already in V8 and Chrome, and my colleague Caio Lima has an project in progress to implement it in JavaScriptCore / WebKit / Safari. Depending on your target audience, BigInt might be deployable already!

thanks

I must mention that my role in the BigInt work was relatively small; my Igalia colleague Robin Templeton did the bulk of the BigInt implementation work in Firefox, so large ups to them. Hearty thanks also to Mozilla's Jan de Mooij and Jeff Walden for their patient and detailed code reviews.

Thanks as well to the V8 engineers for their open source implementation of BigInt fundamental algorithms, as we used many of them in Firefox.

Finally, I need to make one big thank-you, and I hope that you will join me in expressing it. The road to ship anything in a web browser is long; besides the "simple matter of programming" that it is to implement a feature, you need a specification with buy-in from implementors and web standards people, you need a good working relationship with a browser vendor, you need willing technical reviewers, you need to follow up on the inevitable security bugs that any browser change causes, and all of this takes time. It's all predicated on having the backing of an organization that's foresighted enough to invest in this kind of long-term, high-reward platform engineering.

In that regard I think all people that work on the web platform should send a big shout-out to Tech at Bloomberg for making BigInt possible by underwriting all of Igalia's work in this area. Thank you, Bloomberg, and happy hacking!

by Andy Wingo at May 23, 2019 12:13 PM