Update on what happened in WebKit in the week from January 26 to February 2.

A calm week for sure! The highlight this week is the fix for scrolling not starting when the main thread is blocked.

Cross-Port 🐱

Graphics 🖼️

Maintaining the Bridges

Thoughts and analogies about infrastructure...

I live in Pittsburgh, Pennsylvania — “The Steel City,” once the beating heart of American steelmaking. In 1902, U.S. Steel’s first full year of operation, it produced 67% of all steel in the United States. By 1943, the company employed more than 340,000 people. We burned so much coal that Pittsburgh earned the nickname “Hell with the lid off.” Streetlights sometimes ran at noon because the sky was that dark.

The city’s geography didn’t make things any easier. Pittsburgh is carved by mountains, valleys, and the three rivers — the Allegheny and Monongahela merging to form the Ohio. That topography, combined with the industrial boom, meant we built a lot of bridges. It helps that when your city is literally manufacturing the materials, you get a hometown discount.

One of them, the Hot Metal Bridge — just a mile or two from my house — once carried ladle cars full of molten iron between the blast furnaces and mills of J&L Steel. During World War II, 15% of America’s steelmaking capacity crossed that bridge, up to 180 tons per hour.

These bridges were originally built by private companies with a clear profit motive: to move coal, ore, steel, or workers. Others were toll bridges, run by private companies the way you’d run a turnpike or ferry.

But more bridges meant more industry, which meant more people, which meant more bridges. You can see where this goes.

Even by the late 1800s we were beginning to publicly fund them. By the 1920s–1930s Allegheny County’s bridge program bought out many of the private bridges and replaced many of them. By the time the New Deal and Interstate era arrived, the private‑toll era was basically over - and since then over 90% of Pittsburgh's public bridges were funded by federal programs (we still have some private industry use bridges).

So what does any of this have to do with software?

Aside from giving me an excuse to talk about my city (which I enjoy), Pittsburgh’s bridges are a useful metaphor for the infrastructure we rely on in tech, in two important ways:

1. Becoming a public good Private investment built early bridges, just like private companies built much of what we've got now in terms of browser engines, search index, foundational libraries and so on - but eventually they stopped becoming optional. I think we're only now starting to really understand that we need a lot of this to be a public good in the same kind of way. These are the roads and bridges of the modern world.

2. Building something new is exciting. Maintaining it, not so much. A lot of my city's physical infrastructure is aging and some of it has been neglected. It's somehow way easier to get people to build new things than to take care of the old stuff. The public notices a new bridge! The ribbon-cutting gets a photo op and celebration. The maintenance budgets and crews struggle to even get funding.

In fact, even when things are fairly well funded, it doesn't mean they're kept up to date. While researching to write this piece I realized that a lot of the Wikipedia data about Pittsburgh (and many topics!) is actually really out of date. It's cool to write the article with these cool facts, but it's not so cool to do the work to keep it up... Or, maybe thats just not what you want to do anymore. Or maybe you were incarcerated, or you died, or you went to Mars - idk.

The point is that writing the thing in the first place is only half the battle. If most of your entry on a city was written two decades ago, a lot of what it details about the economics, population, jobs, and so on are probably not very accurate!

It's no different with software. It's cool and fun to build a new thing or add a new feature to an existing thing, but keeping them maintained is annoying. New mechanisms arrive that you might need to adapt to. Underlying code bit rots. All of it needs release teams and Q&A and reviews and fixes and updates at global scales, even if no new features were added. But very few people actually want to do that, and almost nobody wants to pay for it.

More Public Funding

I'd really love for societies around the world to come to the realization that a lot of the online things we've built are, like roads and bridges, now necessary - and figure out how we can publicly fund enough of them that important things without an obvious and direct profit motive can get done. MathML and SVG are two easy examples of this, but there are plenty more. Maybe XSLT is another example. Perhaps if we had good funding for those things, their ongoing survival wouldn't be questioned.

I feel like there is a lot of room here for improvement from the status quo. It doesn't even have to start with governments. Any ways that we expand the pool of funding avilable and diversifying, it helps.

Image source

For the past year, I've been working on implementing the Temporal proposal for date and time handling in JavaScript, in JavaScriptCore (JSC). JavaScriptCore is the JavaScript engine that's part of the WebKit browser engine. When I started, Temporal was partially implemented, with support for the Duration, PlainDate, PlainDateTime, and Instant types. However, many test262 tests related to Temporal didn't pass, and there was no support for PlainMonthDay, PlainYearMonth, or ZonedDateTime objects. Further, there was no support for the relativeTo parameter, and only the "iso8601" calendar was supported.

Duration precision (landed) #

Conceptually, a duration is a 10-tuple of time components, or a record with the fields "years", "months", "weeks", "days", "hours", "seconds", "milliseconds", "microseconds", and "nanoseconds".

One way durations are used is to represent the difference between two dates. For example, to find the length of time from a given date until the end of 2027, I could write the following JS code:

> const duration = (new Temporal.PlainDate(2026, 1, 26)).until(new Temporal.PlainDate(2027, 12, 31), { largestUnit: "years" })
> duration
Temporal.Duration <P1Y11M5D> {
  years: 1,
  months: 11,
  days: 5
}

The until method in this case returns a duration comprising one year, eleven months, and five days. Because durations can represent differences between dates, they can also be negative:

> const duration = (new Temporal.PlainDate(2027, 12, 31)).until(new Temporal.PlainDate(2026, 1, 26), { largestUnit: "years" })
> duration
Temporal.Duration <P1Y11M5D> {
  years: -1,
  months: -11,
  days: -5
}

When converted to nanoseconds, the total of days, hours, minutes, seconds, milliseconds, microseconds, and nanoseconds for a duration may be a number whose absolute value is as large as 10⁹ × 2⁵³. This number is too large to represent either as a 32-bit integer or as a 64-bit double-precision value. (If you're wondering about the significance of the number 2⁵³, see the MDN documentation on JavaScript's MAX_SAFE_INTEGER.)

To understand why we need to be able to work with such large numbers, consider totaling the number of nanoseconds in a duration. Following on the previous example’s definition of the variable duration:

> duration.total({unit: "nanoseconds", relativeTo: new Temporal.PlainDate(2025, 12, 15)})
60912000000000000

There are 60912000000000000 nanoseconds, or about 6.1e16, in a period of one year, eleven months, and five days. Since we want to allow this computation to be done with any valid start and end date, and valid years in Temporal range from -271821 to 275760, the result can get quite large. (By default, Temporal follows the ISO 8601 standard for calendars, which entails using a proleptic Gregorian calendar. Also note that this example uses a PlainDate, which has no time zone, so computations are not affected by daylight savings time; when computing with the Temporal ZonedDateTime type, the specification ensures that time zone math is done properly.)

To make it easier for implementations to fulfill these requirements, the specification represents durations internally as Internal Duration Records and converts between JavaScript-level duration objects and Internal Duration Records (which I'll call "internal durations") as needed. An internal duration pairs the date component of the duration (the years, months, weeks, and days fields) with a "time duration", which is a single integer that falls within an accepted range, and can be as large as 2⁵³ × 10⁹ - 1.

Implementations don't have to use this representation, as long as the results are observably the same as what the specification dictates. However, the pre-existing implementation didn't suffice, so I re-implemented durations in a way that closely follows the approach in the specification.

This work has been landed in JSC.

New date types #

Temporal's date types include PlainDate, PlainDateTime, Instant, ZonedDateTime, PlainMonthDay, and PlainYearMonth. The latter two represent partial dates: either a pair of a month and a day within that month, or a pair of a year and month within that year. Partial dates are a better solution for representing dates where not all of the fields are known (or not all of the fields matter) than full dates with default values for the missing bits.

Temporal's ZonedDateTime type represents a date along with a time zone, which can either be a numeric offset from UTC, or a named time zone.

I implemented PlainMonthDay and PlainYearMonth with all their operations. ZonedDateTime is fully implemented and the first pull request in a series of PRs for it has been submitted.

The relativeTo parameter #

What if you want to convert a number of years to a number of days? Temporal can do that, but there's a catch. Converting years to days depends on what year it is, when using the ISO 8601 calendar (similar to the Gregorian calendar), because the calendar has leap years. Some calendars have leap months as well, so converting years to months would depend on what year it is as well. Likewise, converting months to days doesn't have a consistent answer, because months vary in length.

For that reason, the following code will throw an exception, because there's not enough information to compute the result:

> const duration = Temporal.Duration.from({ years: 1 })
> duration.total({ unit: "days" })
Uncaught RangeError: a starting point is required for years total

The above definition of duration can still be made to work if we pass in a starting point, which we can do using the relativeTo parameter:

> duration.total({ unit: "days", relativeTo: "2025-01-01" })
365
> duration.total({ unit: "days", relativeTo: "2024-01-01" })
366

The string passed in for the relativeTo parameter is automatically converted to either a PlainDate or a ZonedDateTime, depending on which format it conforms to.

I implemented support for the relativeTo parameter on all the operations that have it; once the implementations for all the date types land, I'll be submitting this work as a series of pull requests.

Calendars #

Representing dates with non-ISO8601 calendars is still very much a work in progress. The ICU library can already do the basic date computations, but much glue code is necessary to internally represent dates with non-ISO8601 calendars and call the correct ICU functions to do the computations. This work is still underway. The Temporal specification does not require support for non-ISO8601 calendars, but a separate proposal, Intl Era Month Code, proposes a set of calendars to be supported by conformant implementations.

Testing Temporal #

The JavaScript test suite is called test262 and every new proposal in JavaScript must be accompanied by test262 tests. Not all JS implementations are required to support internationalization, so Temporal tests that involve non-ISO calendars or named time zones (other than the UTC time zone) are organized in a separate intl402 subdirectory in test262.

The test262 suite includes 6,764 tests for Temporal, with 1,791 of these tests added in 2025. Igalia invested hundreds of hours on increasing test coverage over the past year.

Status of work #

All of this work is behind a flag in JSC in Technology Preview, so to try it out, you'll have to pass the --useTemporal=1 flag.

All of the implementation work discussed above (except for non-ISO calendars) is complete, but I've been following an incremental approach to submitting the code for review by the JSC code owners. I've already landed about 40 pull requests over the course of 2025, and expect to be submitting at least 25 more to complete the work on PlainYearMonth, ZonedDateTime, and relativeTo.

Based on all the code that I've implemented, 100% of the non-intl402 test262 tests for Temporal pass, while the current HEAD version of JSC passes less than half the tests.

My colleagues at Igalia and I look forward to a future JavaScript standard that fully integrates Temporal, enabling JavaScript programs to handle dates more robustly and efficiently. Consistent implementation of the proposal across browsers is a key step towards this future. Step by step, we're getting closer to this goal.

We thank Bloomberg for sponsoring this work.

A while back, I wrote about Building Node.js on Windows using the new ClangCL support, which was done on an actual x64 Windows machine

On my way to FOSDEM where tomorrow I’ll be talking about Servo in the Browser and web platform devroom. See you there!

Recently, I landed a change that moves the Single Executable Application (SEA) build process directly into Node.js core - a hobby project I’d been tinkering with for some time

Update on what happened in WebKit in the week from January 19 to January 26.

The main event this week has been the creation of the branch for the upcoming stable series, accompanied by the first release candidate before 2.52.0. But there's more: the WPE port gains hyphenation support and the ability to notify of graphics buffer changes; both ports get graphics fixes and a couple of new Web features, and WPE-Android also gets a new stable release.

Cross-Port 🐱

Graphics 🖼️

WPE WebKit 📟

WPE Platform API 🧩

Releases 📦️

Now that 2025 is over, it’s time to look back and feel proud of the path we’ve walked. Last year has been really exciting in terms of contributions to GStreamer and WebKit for the Igalia Multimedia team.

With more than 459 contributions along the year, we’ve been one of the top contributors to the GStreamer project, in areas like Vulkan Video, GstValidate, VA, GStreamer Editing Services, WebRTC or H.266 support.

In Vulkan Video we’ve worked on the VP9 video decoder, and cooperated with other contributors to push the AV1 decoder as well. There’s now an H.264 base class for video encoding that is designed to support general hardware-accelerated processing.

GStreaming Editing Services, the framework to build video editing applications, has gained time remapping support, which now allows to include fast/slow motion effects in the videos. Video transformations (scaling, cropping, rounded corners, etc) are now hardware-accelerated thanks to the addition of new Skia-based GStreamer elements and integration with OpenGL. Buffer pool tuning and pipeline improvements have helped to optimize memory usage and performance, enabling the edition of 4K video at 60 frames per second. Much of this work to improve and ensure quality in GStreamer Editing Services has also brought improvements in the GstValidate testing framework, which will be useful for other parts of GStreamer.

Regarding H.266 (VVC), full playback support (with decoders such as vvdec and avdec_h266, demuxers and muxers for Matroska, MP4 and TS, and parsers for the vvc1 and vvi1 formats) is now available in GStreamer 1.26 thanks to Igalia’s work. This allows user applications such as the WebKitGTK web browser to leverage the hardware accelerated decoding provided by VAAPI to play H.266 video using GStreamer.

Igalia has also been one of the top contributors to GStreamer Rust, with 43 contributions. Most of the commits there have been related to Vulkan Video.

In addition to GStreamer, the team also has a strong presence in WebKit, where we leverage our GStreamer knowledge to implement many features of the web engine related to multimedia. From the 1739 contributions to the WebKit project done last year by Igalia, the Multimedia team has made 323 of them. Nearly one third of those have been related to generic multimedia playback, and the rest have been on areas such as WebRTC, MediaStream, MSE, WebAudio, a new Quirks system to provide adaptations for specific hardware multimedia platforms at runtime, WebCodecs or MediaRecorder.

We’re happy about what we’ve achieved along the year and look forward to maintaining this success and bringing even more exciting features and contributions in 2026.

Welcome back to the second post in this series looking at how we can improve the performance of RISC-V code from LLVM.

Previously in part 1 we looked at how we can use LNT to analyze performance gaps, then identified and fixed a missed fmsub.d opportunity during instruction selection, giving a modest 1.77% speedup on a SPEC CPU 2017 benchmark.

In this post we’ll be improving another SPEC benchmark by 7% by teaching the loop vectorizer to make smarter cost modelling decisions. It involves a relatively non-trivial analysis, but thanks to LLVM’s modular infrastructure we can do it in just a handful of lines of code. Let’s get started.

Analysis

Just like last time, all fruitful performance work begins by analysing some workloads. In the last post we had already run some comparisons of SPEC CPU 2017 benchmarks on LNT, so we can return to those results and pick another benchmark to focus on. Here’s one that’s 12% slower than GCC:

531.deepsjeng_r is a chess engine that tied first in the World Computer Chess Championships back in 2009. It consists of a lot bitwise arithmetic and complex loops, since the state of the game is encoded in 64 element arrays: one element for each square on the board. Unlike 508.namd_r from last time, there’s no floating point arithmetic.

Drilling into the profile and its list of functions, right off the bat we can see that one function is much slower on LLVM. On GCC qsearch(state_t*, int, int, int, int) makes up 9.1% of the overall cycles, but on LLVM it’s 16.1%. And if we click in on the function and view the cumulative total of cycles spent in user mode, Clang takes 74.6 billion cycles to do what takes GCC only 37.7 billion cycles.

So there’s probably something we can improve upon here, but it’s not immediately obvious from staring at the disassembly. qsearch is a pretty big function with a couple hundred instructions, so switching to the CFG view gives us a better overview.

On LLVM’s side we see the offending loop that’s consuming so many cycles: It’s long, vectorized, and completely if-predicated: there’s no control flow inside the loop itself. This is typical of a loop that’s been auto-vectorized by the loop vectorized. If you look at the load and store instructions you can see that they are masked with the v0.t operand, stemming from the original control flow that was flattened.

But on the GCC side there’s no equivalent vectorized loop. The loop is in there somewhere, but all the loops are still in their original scalar form with the control flow intact. And if we look at the edges coming from the loop headers, we can see that most of the time it visits one or two basic blocks and then branches back up to the header. Most of the blocks in the loop are completely cold.

Unfortunately the sources for deepsjeng aren’t open source so we can’t share them in this post, but the very rough structure of the loop is something like this:

for (i = 0; i < N; i++) {
    if (foo[i] == a) {
        if (bar[i] == b) {
            if (baz[i] == c) {
                qux[i] = 123;
                // lots of work here...
            }
        }
    }
}

For any given iteration, it’s statistically unlikely that we enter the first if statement. It’s even more unlikely that the second if’s condition is also true. And even more so for the third nested if where we eventually have lots of work to compute.

In a scalar loop this doesn’t matter because if an if statement’s condition is false, then we don’t execute the code inside it. We just branch back to the start of the loop. But with a vectorized loop, we execute every single instruction regardless of the condition.

This is the core of the performance gap that we’re seeing versus GCC: Given that the majority of the work in this loop is so deeply nested in the control flow, it would have been better to have not vectorized it given that we need to if-convert it.

Cost modelling

One of the hardest problems when making an optimizing compiler is to know when an optimization is profitable. Some optimizations are a double edged sword that can harm performance just as much as they can improve it (if not more), and loop vectorization falls squarely into this category. So rather than blindly applying optimizations at any given opportunity, LLVM has detailed cost models for each target to try and estimate how expensive or cheap a certain sequence of instructions is, which it can then use to evaluate whether or not a transform will be a net positive.

It’s hard to overstate the amount of effort in LLVM spent fine tuning these cost models, applying various heuristics and approximations to make sure different optimizations don’t shoot themselves in the foot. In fact there are some optimizations like loop distribute that are in-tree but disabled by default due to the difficulty in getting the cost model right.

So naturally, we would expect that the loop vectorizer already has a sophisticated solution for the problem we’re seeing in our analysis: Given any predicated block that’s if-converted during vectorization, we would expect the scalar cost for that block to be made slightly cheaper because the scalar block may not always be executed. And the less likely it is to be executed, the cheaper it should be — the most deeply nested if block should be discounted more than the outermost if block.

So how does the loop vectorizer handle this?

/// A helper function that returns how much we should divide the cost of a
/// predicated block by. Typically this is the reciprocal of the block
/// probability, i.e. if we return X we are assuming the predicated block will
/// execute once for every X iterations of the loop header so the block should
/// only contribute 1/X of its cost to the total cost calculation, but when
/// optimizing for code size it will just be 1 as code size costs don't depend
/// on execution probabilities.
///
/// TODO: We should use actual block probability here, if available. Currently,
///       we always assume predicated blocks have a 50% chance of executing.
inline unsigned
getPredBlockCostDivisor(TargetTransformInfo::TargetCostKind CostKind) {
  return CostKind == TTI::TCK_CodeSize ? 1 : 2;
}

We’ve come across a load bearing TODO here. Either the block is executed or its not, so it’s a fifty/fifty chance.

On its own this hardcoded probability doesn’t seem like an unreasonable guess. But whilst 50% may be an accurate estimate as to whether or not a branch will be taken, it’s an inaccurate estimate as to whether or not a block will be executed. Assuming that a branch has a 1/2 chance of being taken, the most deeply nested block in our example ends up having a 1/2 * 1/2 * 1/2 = 1/8 chance of being executed.

for (i = 0; i < N; i++) {
    if (foo[i] == a) {
        // 1/2 chance of being executed
        if (bar[i] == b) {
            // 1/4 chance of being executed
            if (baz[i] == c) {
                // 1/8 chance of being executed
                // ...
            }
        }
    }
}

The fix to get the loop vectorizer to not unprofitably vectorize this loop will be to teach getPredBlockCostDivisor to take into account control flow between blocks.

It’s worth mentioning the fact that a hardcoded constant managed to work well enough up until this point is the sign of an good trade off. 1% of the effort for 90% of the benefit. A patch can go off the rails very easily by trying to implement too much in one go, so deferring the more complex cost modelling here till later was an astute choice. Incremental development is key to making progress upstream.

VPlan cost modeling

To get a better picture of how the loop vectorizer is calculating the cost for each possible loop, lets start with a simplified LLVM IR reproducer:

; for (int i = 0; i < 1024; i++)
;   if (c0)
;     if (c1)
;       p1[p0[i]] = 0; // extra work to increase the cost in the predicated block

define void @nested(ptr noalias %p0, ptr noalias %p1, i1 %c0, i1 %c1) {
entry:
  br label %loop

loop:
  %iv = phi i32 [ 0, %entry ], [ %iv.next, %latch ]
  br i1 %c0, label %then.0, label %latch

then.0:
  br i1 %c1, label %then.1, label %latch

then.1:
  %gep0 = getelementptr i32, ptr %p0, i32 %iv
  %x = load i32, ptr %gep0
  %gep1 = getelementptr i32, ptr %p1, i32 %x
  store i32 0, ptr %gep1
  br label %latch

latch:
  %iv.next = add i32 %iv, 1
  %done = icmp eq i32 %iv.next, 1024
  br i1 %done, label %exit, label %loop

exit:
  ret void
}

We can run opt -p loop-vectorize -debug on this example to see how the loop vectorizer decides if it’s profitable to vectorize the loop or not:

$ opt -p loop-vectorize -mtriple riscv64 -mattr=+v nested.ll -disable-output -debug
...
LV: Found an estimated cost of 0 for VF 1 For instruction:   %iv = phi i32 [ 0, %entry ], [ %iv.next, %latch ]
LV: Found an estimated cost of 0 for VF 1 For instruction:   br i1 %c0, label %then.0, label %latch
LV: Found an estimated cost of 0 for VF 1 For instruction:   br i1 %c1, label %then.1, label %latch
LV: Found an estimated cost of 0 for VF 1 For instruction:   %gep0 = getelementptr i32, ptr %p0, i32 %iv
LV: Found an estimated cost of 1 for VF 1 For instruction:   %x = load i32, ptr %gep0, align 4
LV: Found an estimated cost of 0 for VF 1 For instruction:   %gep1 = getelementptr i32, ptr %p1, i32 %x
LV: Found an estimated cost of 1 for VF 1 For instruction:   store i32 0, ptr %gep1, align 4
LV: Found an estimated cost of 0 for VF 1 For instruction:   br label %latch
LV: Found an estimated cost of 1 for VF 1 For instruction:   %iv.next = add i32 %iv, 1
LV: Found an estimated cost of 1 for VF 1 For instruction:   %done = icmp eq i32 %iv.next, 1024
LV: Found an estimated cost of 0 for VF 1 For instruction:   br i1 %done, label %exit, label %loop
LV: Scalar loop costs: 3.
...
Cost of 1 for VF vscale x 4: induction instruction   %iv.next = add i32 %iv, 1
Cost of 0 for VF vscale x 4: induction instruction   %iv = phi i32 [ 0, %entry ], [ %iv.next, %latch ]
Cost of 1 for VF vscale x 4: exit condition instruction   %done = icmp eq i32 %iv.next, 1024
Cost of 0 for VF vscale x 4: EMIT vp<%4> = CANONICAL-INDUCTION ir<0>, vp<%index.next>
Cost of 0 for VF vscale x 4: EXPLICIT-VECTOR-LENGTH-BASED-IV-PHI vp<%5> = phi ir<0>, vp<%index.evl.next>
Cost of 0 for VF vscale x 4: EMIT-SCALAR vp<%avl> = phi [ ir<1024>, vector.ph ], [ vp<%avl.next>, vector.body ]
Cost of 1 for VF vscale x 4: EMIT-SCALAR vp<%6> = EXPLICIT-VECTOR-LENGTH vp<%avl>
Cost of 0 for VF vscale x 4: vp<%7> = SCALAR-STEPS vp<%5>, ir<1>, vp<%6>
Cost of 0 for VF vscale x 4: CLONE ir<%gep0> = getelementptr ir<%p0>, vp<%7>
Cost of 0 for VF vscale x 4: vp<%8> = vector-pointer ir<%gep0>
Cost of 2 for VF vscale x 4: WIDEN ir<%x> = vp.load vp<%8>, vp<%6>, vp<%3>
Cost of 0 for VF vscale x 4: WIDEN-GEP Inv[Var] ir<%gep1> = getelementptr ir<%p1>, ir<%x>
Cost of 12 for VF vscale x 4: WIDEN vp.store ir<%gep1>, ir<0>, vp<%6>, vp<%3>
Cost of 0 for VF vscale x 4: EMIT vp<%index.evl.next> = add nuw vp<%6>, vp<%5>
Cost of 0 for VF vscale x 4: EMIT vp<%avl.next> = sub nuw vp<%avl>, vp<%6>
Cost of 0 for VF vscale x 4: EMIT vp<%index.next> = add nuw vp<%4>, vp<%0>
Cost of 0 for VF vscale x 4: EMIT branch-on-count vp<%index.next>, vp<%1>
Cost of 0 for VF vscale x 4: vector loop backedge
Cost of 0 for VF vscale x 4: EMIT-SCALAR vp<%bc.resume.val> = phi [ ir<0>, ir-bb<entry> ]
Cost of 0 for VF vscale x 4: IR   %iv = phi i32 [ 0, %entry ], [ %iv.next, %latch ] (extra operand: vp<%bc.resume.val> from scalar.ph)
Cost of 0 for VF vscale x 4: EMIT vp<%3> = logical-and ir<%c0>, ir<%c1>
Cost for VF vscale x 4: 17 (Estimated cost per lane: 2.1)
...
LV: Selecting VF: vscale x 4.
LV: Minimum required TC for runtime checks to be profitable:0
LV: Interleaving is not beneficial.
LV: Found a vectorizable loop (vscale x 4) in nested.ll
LV: Vectorizing: innermost loop.
LEV: Unable to vectorize epilogue because no epilogue is allowed.
LV: Loop does not require scalar epilogue
LV: Loop does not require scalar epilogue
Executing best plan with VF=vscale x 4, UF=1

First we see it work out the cost of the original scalar loop, or as the vectorizer sees it, the loop with a vectorization factor (VF) of 1. It goes through each instruction calling into TargetTransformInfo, and arrives at a total scalar cost of 3. You might have noticed though, if you went through and manually summed up the individual instruction costs you would have gotten a total cost of 4. However the load and store instructions belong to the predicated then.1 block, so they have their cost divided by 2 from getPredBlockCostDivisor.

For the vectorized loop, the loop vectorizer uses VPlan to cost the one plan for a range of different VFs¹. VPlan is an IR specific to the loop vectorizer to help represent various vectorization strategies, which is why you see all the EMIT and WIDEN “recipes” in the output. It calculates a total cost for the loop and divides it by the estimated number of lanes — we’re working with scalable vectors on RISC-V so the target needs to make an estimate of what vscale is — and arrives at 2.1 per lane. There’s no predication discount applied here because it’s a vectorized loop. 2.1 is cheaper than 3, so it ultimately picks the vectorized loop.

BlockFrequencyInfo

Computing an accurate probability that a given block will be executed is a non-trivial task, but thankfully LLVM already has an analysis we can use for this called BlockFrequencyInfo.

BlockFrequencyInfo computes how often a block can be expected to execute relative to other blocks in a function. It in turn uses another analysis called BranchProbabilityInfo to work out how likely a branch to a specific block is going to be taken. And because BranchProbabilityInfo uses profiling information when available, it can give you much more accurate block frequencies when compiling with PGO. Otherwise it will fall back to guessing the probability of a branch being taken, which is just 50/50 a lot of the time, but sometimes influenced by interesting heuristics too: like the probability of icmp eq i32 %x, 0 is 0.375 instead of 0.5, and floats have a near zero chance of being NaN.

Plugging BlockFrequencyInfo into the loop vectorizer is straightforward, all we need to do is tell the pass manager that we want to access BlockFrequencyInfo from LoopVectorizePass:

PreservedAnalyses LoopVectorizePass::run(Function &F,
                                         FunctionAnalysisManager &AM) {
   ...
   BFI = &AM.getResult<BlockFrequencyAnalysis>(F);
   ...
}

(BlockFrequencyAnalysis is the pass that computes the analysis result BlockFrequencyInfo, if you’re wondering why the names are different)

Then we can use it to lookup the relative frequencies of whatever block and work out the probability of it being executed in the loop:

uint64_t LoopVectorizationCostModel::getPredBlockCostDivisor(
    TargetTransformInfo::TargetCostKind CostKind, const BasicBlock *BB) {
  if (CostKind == TTI::TCK_CodeSize)
    return 1;

  uint64_t HeaderFreq =
      BFI->getBlockFreq(TheLoop->getHeader()).getFrequency();
  uint64_t BBFreq = BFI->getBlockFreq(BB).getFrequency();
  return HeaderFreq / BBFreq;
}

The frequencies returned from BlockFrequencyInfo are relative to the the entry block of a function. So if a block has a frequency of 50 and the entry block has a frequency of 100, then you can expect that block to execute 50 times for every 100 times the entry block is executed.

You can use this to work out probabilities of a block being taken in a function, so in this example that block has a 50/100 = 50% chance of being executed every time the function is executed. However this only works in the case that the CFG has no loops: otherwise a block may be executed more times than the entry block and we’d end up with probabilities greater than 100%.

If we want to calculate the probability of a block being executed inside a loop though, that’s fine since the loop vectorizer currently only vectorizes inner-most loops², i.e. loops that contain no other loops.

We can consider the frequencies of each block in the loop relative to the frequency of the header block. To give a brief loop terminology recap, the header is the first block inside the loop body which dominates all other blocks in the loop, and is the destination of all backedges. So the header is guaranteed to have a frequency greater than or equal to any other block in the loop — this invariant is important as we’ll see later.

Then to calculate the probability of a block in a loop being executed, we divide the block frequency by the header frequency. To work out how much we should divide the cost of the scalar block by, we return the inverse of that.

Trying out this change on our sample loop, first we’ll see the debug output from BlockFrequencyInfo as it’s computed:

$ opt -p loop-vectorize -mtriple riscv64 -mattr=+v nested.ll -disable-output -debug
...
block-frequency-info: nested
 - entry: float = 1.0, int = 562949953421312
 - loop: float = 32.0, int = 18014398509481984
 - then.0: float = 16.0, int = 9007199254740992
 - then.1: float = 8.0, int = 4503599627370496
 - latch: float = 32.0, int = 18014398509481984
 - exit: float = 1.0, int = 562949953421312

loop is the header block and then.1 is the nested if block, and with BlockFrequencyInfo’s frequency we get a probability of 8/32 = 0.25. So we would expect then.1’s scalar cost to be divided by 4:

...
LV: Found an estimated cost of 0 for VF 1 For instruction:   %iv = phi i32 [ 0, %entry ], [ %iv.next, %latch ]
LV: Found an estimated cost of 0 for VF 1 For instruction:   br i1 %c0, label %then.0, label %latch
LV: Found an estimated cost of 0 for VF 1 For instruction:   br i1 %c1, label %then.1, label %latch
LV: Found an estimated cost of 0 for VF 1 For instruction:   %gep0 = getelementptr i32, ptr %p0, i32 %iv
LV: Found an estimated cost of 1 for VF 1 For instruction:   %x = load i32, ptr %gep0, align 4
LV: Found an estimated cost of 0 for VF 1 For instruction:   %gep1 = getelementptr i32, ptr %p1, i32 %x
LV: Found an estimated cost of 1 for VF 1 For instruction:   store i32 0, ptr %gep1, align 4
LV: Found an estimated cost of 0 for VF 1 For instruction:   br label %latch
LV: Found an estimated cost of 1 for VF 1 For instruction:   %iv.next = add i32 %iv, 1
LV: Found an estimated cost of 1 for VF 1 For instruction:   %done = icmp eq i32 %iv.next, 1024
LV: Found an estimated cost of 0 for VF 1 For instruction:   br i1 %done, label %exit, label %loop
LV: Scalar loop costs: 2.
...
Cost for VF vscale x 4: 17 (Estimated cost per lane: 2.1)
...
LV: Selecting VF: 1.
LV: Vectorization is possible but not beneficial.

then.1s scalar cost is now 2/4 = 0, so the total cost of the scalar loop is now 2 and the loop vectorizer no longer decides to vectorize. If we try this out on 538.deepsjeng_r, we can see that it no longer vectorizes that loop in qsearch either. Success!

Running it again on LNT showed a ~7% speedup in execution time. Not just as fast as GCC yet, but a welcome improvement for only a handful of lines of code.

Upstreaming

Now that we know the fix we want to land, we can start to think about how we want to upstream this into LLVM.

If we run llvm-lit --update-tests llvm/test/Transforms/LoopVectorize, we actually get quite a few unexpected test changes. One of the side effects of using BlockFrequencyInfo is that tail folded loops no longer discount the scalar loop if it wasn’t predicated to begin with. A tail folded loop is a loop where the scalar epilogue is folded into the vector loop itself by predicating the vector operations:

// non-tail folded loop:
// process as many VF sized vectors that fit in n
for (int i = 0; i < n - (n % VF); i += VF)
  x[i..i+VF] = y[i..i+VF];
// process the remaining n % VF scalar elements
for (int i = n - (n % VF); i < n; i++)
  x[i] = y[i];

// tail folded loop:
for (int i = 0; i < n; i += VF)
  x[i..i+VF] = y[i..i+VF] mask=[i<n, i+1<n, ..., i+VF-1<n];

However because this block is technically predicated due to the mask on the vector instructions, the loop vectorizer applied getPredBlockCostDivisor to the scalar loop cost even if the original scalar loop had no control flow in its body. BlockFrequencyInfo here can detect that if the block had no control flow, its probability of being executed is 1 and so the scalar loop cost isn’t made cheaper than it needs to be. I split off and landed this change separately, since it makes the test changes easier to review.

Now that the remaining changes in llvm/test/Transforms/LoopVectorize looked more contained, I was almost ready to open a pull request. I just wanted to quickly kick the tyres on llvm-test-suite with a few other targets, since this wasn’t a RISC-V specific change. The plan was to quickly collect some stats on how many loops were vectorized, check for any anomalies when compared to beforehand, and then be on our way:

$ cd llvm-test-suite
$ ninja -C build
...
[222/7278] Building C object External/...nchspec/CPU/500.perlbench_r/src/pp.c.o
FAILED: External/SPEC/CINT2017rate/500.perlbench_r/CMakeFiles/500.perlbench_r.dir/root/cpu2017/benchspec/CPU/500.perlbench_r/src/pp.c.o 
/root/llvm-test-suite/build.x86_64-ReleaseLTO-a/tools/timeit --summary External/SPEC/CINT2017rate/500.perlbench_r/CMakeFiles/500.perlbench_r.dir/root/cpu2017/benchspec/CPU/500.perlbench_r/src/pp.c.o.time /root/llvm-project/build/bin/clang -DDOUBLE_SLASHES_SPECIAL=0 -DNDEBUG -DPERL_CORE -DSPEC -DSPEC_AUTO_BYTEORDER=0x12345678 -DSPEC_AUTO_SUPPRESS_OPENMP -DSPEC_CPU -DSPEC_LINUX -DSPEC_LINUX_X64 -DSPEC_LP64 -DSPEC_SUPPRESS_OPENMP -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -D_LARGE_FILES -I/root/cpu2017/benchspec/CPU/500.perlbench_r/src -I/root/cpu2017/benchspec/CPU/500.perlbench_r/src/dist/IO -I/root/cpu2017/benchspec/CPU/500.perlbench_r/src/cpan/Time-HiRes -I/root/cpu2017/benchspec/CPU/500.perlbench_r/src/cpan/HTML-Parser -I/root/cpu2017/benchspec/CPU/500.perlbench_r/src/ext/re -I/root/cpu2017/benchspec/CPU/500.perlbench_r/src/specrand -march=x86-64-v3 -save-temps=obj     -O3 -fomit-frame-pointer -flto -DNDEBUG   -w -Werror=date-time -save-stats=obj -save-stats=obj -fno-strict-aliasing -MD -MT External/SPEC/CINT2017rate/500.perlbench_r/CMakeFiles/500.perlbench_r.dir/root/cpu2017/benchspec/CPU/500.perlbench_r/src/pp.c.o -MF External/SPEC/CINT2017rate/500.perlbench_r/CMakeFiles/500.perlbench_r.dir/root/cpu2017/benchspec/CPU/500.perlbench_r/src/pp.c.o.d -o External/SPEC/CINT2017rate/500.perlbench_r/CMakeFiles/500.perlbench_r.dir/root/cpu2017/benchspec/CPU/500.perlbench_r/src/pp.c.o -c /root/cpu2017/benchspec/CPU/500.perlbench_r/src/pp.c
PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace, preprocessed source, and associated run script.
Stack dump:
0.      Program arguments: /root/llvm-project/build/bin/clang-19 -cc1 -triple x86_64-unknown-linux-gnu -O3 -emit-llvm-bc -flto=full -flto-unit -save-temps=obj -disable-free -clear-ast-before-backend -main-file-name pp.c -mrelocation-model pic -pic-level 2 -pic-is-pie -mframe-pointer=none -relaxed-aliasing -fmath-errno -ffp-contract=on -fno-rounding-math -mconstructor-aliases -funwind-tables=2 -target-cpu x86-64-v3 -debugger-tuning=gdb -fdebug-compilation-dir=/root/llvm-test-suite/build.x86_64-ReleaseLTO-a -fcoverage-compilation-dir=/root/llvm-test-suite/build.x86_64-ReleaseLTO-a -resource-dir /root/llvm-project/build/lib/clang/23 -Werror=date-time -w -ferror-limit 19 -fgnuc-version=4.2.1 -fskip-odr-check-in-gmf -vectorize-loops -vectorize-slp -stats-file=External/SPEC/CINT2017rate/500.perlbench_r/CMakeFiles/500.perlbench_r.dir/root/cpu2017/benchspec/CPU/500.perlbench_r/src/pp.stats -faddrsig -fdwarf2-cfi-asm -o External/SPEC/CINT2017rate/500.perlbench_r/CMakeFiles/500.perlbench_r.dir/root/cpu2017/benchspec/CPU/500.perlbench_r/src/pp.c.o -x ir External/SPEC/CINT2017rate/500.perlbench_r/CMakeFiles/500.perlbench_r.dir/root/cpu2017/benchspec/CPU/500.perlbench_r/src/pp.bc
1.      Optimizer
2.      Running pass "function<eager-inv>(float2int,lower-constant-intrinsics,chr,loop(loop-rotate<header-duplication;prepare-for-lto>,loop-deletion),loop-distribute,inject-tli-mappings,loop-vectorize<no-interleave-forced-only;no-vectorize-forced-only;>,infer-alignment,loop-load-elim,instcombine<max-iterations=1;no-verify-fixpoint>,simplifycfg<bonus-inst-threshold=1;forward-switch-cond;switch-range-to-icmp;switch-to-arithmetic;switch-to-lookup;no-keep-loops;hoist-common-insts;no-hoist-loads-stores-with-cond-faulting;sink-common-insts;speculate-blocks;simplify-cond-branch;no-speculate-unpredictables>,slp-vectorizer,vector-combine,instcombine<max-iterations=1;no-verify-fixpoint>,loop-unroll<O3>,transform-warning,sroa<preserve-cfg>,infer-alignment,instcombine<max-iterations=1;no-verify-fixpoint>,loop-mssa(licm<allowspeculation>),alignment-from-assumptions,loop-sink,instsimplify,div-rem-pairs,tailcallelim,simplifycfg<bonus-inst-threshold=1;no-forward-switch-cond;switch-range-to-icmp;switch-to-arithmetic;no-switch-to-lookup;keep-loops;no-hoist-common-insts;hoist-loads-stores-with-cond-faulting;no-sink-common-insts;speculate-blocks;simplify-cond-branch;speculate-unpredictables>)" on module "External/SPEC/CINT2017rate/500.perlbench_r/CMakeFiles/500.perlbench_r.dir/root/cpu2017/benchspec/CPU/500.perlbench_r/src/pp.bc"
3.      Running pass "loop-vectorize<no-interleave-forced-only;no-vectorize-forced-only;>" on function "Perl_pp_coreargs"
 #0 0x0000556ff93ab158 llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) (/root/llvm-project/build/bin/clang-19+0x2d5c158)
 #1 0x0000556ff93a8835 llvm::sys::RunSignalHandlers() (/root/llvm-project/build/bin/clang-19+0x2d59835)
 #2 0x0000556ff93abf01 SignalHandler(int, siginfo_t*, void*) Signals.cpp:0:0
 #3 0x00007f305ce49df0 (/lib/x86_64-linux-gnu/libc.so.6+0x3fdf0)
 #4 0x0000556ffaa0dbfb llvm::LoopVectorizationCostModel::expectedCost(llvm::ElementCount) (/root/llvm-project/build/bin/clang-19+0x43bebfb)
 #5 0x0000556ffaa22a0d llvm::LoopVectorizationPlanner::computeBestVF() (/root/llvm-project/build/bin/clang-19+0x43d3a0d)
 #6 0x0000556ffaa36f3b llvm::LoopVectorizePass::processLoop(llvm::Loop*) (/root/llvm-project/build/bin/clang-19+0x43e7f3b)
 #7 0x0000556ffaa413eb llvm::LoopVectorizePass::runImpl(llvm::Function&) (/root/llvm-project/build/bin/clang-19+0x43f23eb)
 ...
...

A crash when building for X86. No assertion message, but a backtrace that points to the loop vectorizer cost model. Unfortunately this did not turn out to be simple to debug and instead turned into a whole other ordeal, so I’ll leave the details of that rabbit hole to the next post. But in the meantime, here are some hints if you want to guess what went wrong:

• The crash stems from a SIGFPE signal
• It only occurs when building on X86. Building on AArch64 is unaffected, even when cross-compiling to X86
• It only occurs with LTO

Hopefully this also gives a bit of insight into the type of upstream work that we carry out at Igalia. If you have an LLVM or RISC-V project that we could help with, feel free to reach out.

In June 2025, I joined the Igalia Coding Experience program. My role was to implement the TC39 proposal Legacy RegExp Features in SpiderMonkey, the JavaScript engine in Mozilla Firefox. This wasn't my first proposal implementation. I'd already implemented the Error.isError and Iterator.range TC39 proposals in SpiderMonkey, but implementing the Legacy RegExp Features proposal involved delving deeper into the Mozilla codebase, and new challenges for me.

To begin with, I created an implementation plan with a timeline of how I was going to approach the proposal. Additionally, I added links to the codebase where I thought I was going to make changes as per the specification, which helped me have a clear starting point and path for integrating the feature. It also meant I could get feedback from SpiderMonkey developers before actually beginning the implementation.

The Legacy RegExp features proposal disables legacy static properties and RegExp.prototype.compile for instances of proper subclasses of RegExp as well as for cross-realm regexps.

The following operations are modified in SpiderMonkey:

RegExp.prototype.compile(pattern, flags) #

This method reinitializes an existing RegExp object with a new pattern and/or flags. It modifies the RegExp object in place rather than creating a new one.

Modification: The proposal modifies RegExp.prototype.compile to throw errors for objects that are not direct instances of the RegExp as well as for cross-realm mismatches. The compile() method initializes a RegExp object similar to the way a RegExp literal is created, bypassing any preprocessing of the pattern that might be done by a RegExp subclass's constructor, and potentially breaking a subclass's custom "exec" method. Thus, compile is disallowed for subclasses. It is now forbidden for a RegExp compile method to be applied to a RegExp object belonging to a different realm, as this would typically result in static properties of the incorrect realm being updated.

Example of newly restricted behaviour:

(base) $ ./mach run
 0:00.29 /Users/default/firefox/obj-aarch64-apple-darwin25.2.0/dist/bin/js
js> let g = newGlobal();
js> let re = g.RegExp("x");
js> RegExp.prototype.compile.call(re);
typein:3:26 TypeError: RegExp operation not permitted on object from different realm
Stack:
  @typein:3:26
js>

To explain each line of the JavaScript code in detail:

• let g = newGlobal() creates a new JavaScript global object in SpiderMonkey, similar to opening a new window in a browser. Each global object has its own realm. A realm is a JavaScript execution context that contains its own set of global objects and built-in functions. Every object in SpiderMonkey has a realm pointer which identifies which realm it belongs to.

• let re = g.RegExp(“x”) creates a new RegExp object from g's realm, with a distinct instance of the RegExp constructor. Although the object behaves like one created from RegExp("x"), the two are not wholly compatible with one another.

• RegExp.prototype.compile.call(re) invokes the compile() method with the regexp initialized above for a realm returned from newGlobal(). Per step 5 of the modified RegExp.prototype.compile() algorithm in the proposal, this results in a TypeError exception being thrown.

Initially, I added my changes in regexp_compile_impl(), but when testing with ./mach try auto, the feature failed test262 cross-realm tests when run with the ion eager and --more-compartments flag. Debug output showed that when invoking the RegExp.prototype.compile(re) both the receiver or (this`) of the RegExp.prototype.compile() method, and the RegExp object were in the same realm while they weren’t. In other words, the cross-realm check was passing, when it should have been failing, according to the test expectations.

By the time execution reached regexp_compile(), the CallNonGenericMethod<IsRegExpObject, regexp_compile_impl> wrapper had already processed the "receiver" or "this" of the compile method. According to the CallNonGenericMethod documentation, if args.thisv() is not of the correct type, it will attempt to unwrap this and if successful, call the implementation function on the unwrapped this. For a bit of context on this, SpiderMonkey has a concept of Wrapper objects, which decorate an object in a sort of proxy membrane to provide security boundary enforcement. For instance, ensuring that a method can be invoked or a field can be written to from the presently entered compartment. Unwrapping an object means removing that proxy membrane, to access the actual object, similar to how you’d unwrap a gift. This can be done using js::CheckedUnwrapStatic().

With --more-compartments, CallNonGenericMethod in regexp_compile() was automatically unwrapping cross-compartment proxies through CallMethodIfWrapped before calling regexp_compile_impl().

This unwrapping process also switched the JSContext to the target object's realm. This meant that by the time my realm checks executed in regexp_compile_impl(), both cx->realm() and the RegExp object's realm pointed to the same realm (the object's home realm), making them appear equal even in genuine cross-realm call scenarios where the original call came from a different realm.

So I moved the same-realm testing and [[LegacyFeaturesEnabled]] bit testing to regexp_compile(), just before CallNonGenericMethod is called and added js::CheckedUnwrapStatic() to unwrap any proxy wrappers before checking the realm. This ensures we’re checking the realm of the actual RegExp object and not the compartment wrappers around it.

Subclass Instances #

As mentioned above, the RegExp method RegExp.prototype.compile() re-initializes a RegExp using a newly created matcher for the specified pattern and flags. The proposal adds some restrictions to this which prevent oddities such as subclasses not functioning as expected (for instance, by not preprocessing the pattern and adding context used by their exec() implementation). More importantly, when applied to a cross-realm object, this would result in execution modifying the static RegExp members for the incorrect realm.

The proposal modifies the behavior so that legacy static properties are only updated when direct instances of the built-in RegExp constructor are used, not subclass instances or cross-realm objects, using similar logic to RegExp.prototype.compile():

7. If SameValue(thisRealm, rRealm) is true, then
   • i. If the value of R’s [[LegacyFeaturesEnabled]] internal slot is true, then
     • a. Perform UpdateLegacyRegExpStaticProperties(%RegExp%, S, lastIndex, e, capturedValues).
   • ii. Else,
     • a. Perform InvalidateLegacyRegExpStaticProperties(%RegExp%).

The properties are specced and implemented as accessors with a getter and no setter, except for RegExp.input (and its alias RegExp.$_), which remains writable. Inside each of the accessors, if the receiver this and the %RegExp% realm intrinsic (the standard RegExp constructor) are not the same, we throw a TypeError.

(base) $ ./mach run
 0:00.28 /Users/default/firefox/obj-aarch64-apple-darwin25.2.0/dist/bin/js
js> /a(b)c/.exec("abc"); 
["abc", "b"]
js> RegExp.$1 
"b"
js> new RegExp("a(b)").exec("ab"); 
["ab", "b"]
js> RegExp.$1                       
"b"
js> new (class extends RegExp {})("a(b)").exec("ab"); 
["ab", "b"]
js> RegExp.$1                                          
typein:6:1 TypeError: RegExp static property 'static_paren1_getter' is invalid
Stack:
  @typein:6:1
js> 

/a(b)c/.exec("abc"); RegExp.$1  // should return "b"
new RegExp("a(b)").exec("ab"); RegExp.$1  // "b"
new (class extends RegExp {})("a(b)").exec("ab"); RegExp.$1 //  throws

Normalisation of RegExp Static Properties #

RegExp static properties are now defined as configurable and non-enumerable. This is so that the associated features may be easily removed by using the JavaScript delete operator. This is important for consistency with modern ECMA262 and for allows for applications to further reduce the number of side-affect producing globals, including VM native methods.

In SpiderMonkey, the legacy static properties are defined in RegExp.cpp. To implement the proposal, I enclosed the properties with a NIGHTLY_BUILD directive, removing the JS_PROP_PERMANEN and JS_PROP_ENUMERATE flags to make them configurable and non-enumerable for the Nightly environment, where they can be tested by the community. Outside of Nightly, we continue supporting the old implementation for beta/release environments.

Then, I updated the test262 AnnexB RegExp tests to support the change and to limit the tests to Nightly.

Understanding the Implementation: Challenges and Solutions #

1. Creative Bit Packing #

Once the legacy RegExp statics were normalised, the next step was adding a LegacyFeaturesEnabled internal slot. This slot keeps a reference to its constructor and is checked whenever legacy features are accessed. If the RegExp is a subclass instance or is is associated with a different realm, the slot indicates that legacy features should throw an error.

Initially, I added the slot to the RegExpObject :

static const unsigned LEGACY_FEATURES_ENABLED_SLOT = 3; 

This presented a couple of problems for me:

• The number of reserved slots must match the allocation kind defined in FOR_EACH_OBJECT_ALLOCKIND(D). The number of reserved slots increased to 5, which meant that I had to choose between OBJECT6 or OBJECT8. During implementation, I somehow missed OBJECT6 and went with OBJECT8.

• I knew that I’d get some pushback in code review, as my changes increased the size of the RegExp Object by 32 bytes (four 8-byte slots). I could see that there was a way for Boolean flags to share a slot but I didn't know how to implement my changes without breaking the JIT.

I decided to leave the implementation as is and wait for SpiderMonkey engineers / reviewers to give me feedback and their preference on how to add the Boolean.

During code review, my reviewer Iain pointed out that since we’re only storing a single bit of information (whether legacy features are enabled or not), and the existing FLAGS_SLOT only uses 8 bits, I could store the legacy features in the unused higher bits.

The slot implementation includes a getter, bool legacyFeaturesEnabled(), that reads the bit from the FLAGS_SLOT; and a setter, setLegacyFeaturesEnabled(bool), that writes the bit to the FLAGS_SLOT.

The new approach involved defining some constants based on the size of RegExp Flags so that the code keeps working if RegExpFlags gets bigger in future:

static const size_t RegExpFlagsMask = JS::RegExpFlag::AllFlags;
static const size_t LegacyFeaturesEnabledBit = Bit(8);

static_assert((RegExpFlagsMask & LegacyFeaturesEnabledBit) == 0,
              "LegacyFeaturesEnabledBit must not overlap");

RegExpFlagsMask has a bit set to 1 if that bit is part of the RegExpFlags, and 0 otherwise. The lowest 8 bits are currently set to other RegExp flags, which leaves us with the highest bits to pack our slot in.

We perform two operations: raw & RegExpFlagsMask, which gets only the traditional RegExp flags; and raw & ~RegExpFlagsMask, which gets everything apart from the RegExp flags.Those are bits 0-7. We use bit 8 to store LegacyFeaturesEnabled. When we read the flags, we mask off any bits that are not part of the RegExpFlags.

return JS::RegExpFlags(raw & RegExpFlagsMask);

When we write to the flags, we combine the new value of the RegExpFlags bits (flags.value()) with the old value of the other bits in (raw & RegExpFlagsMask).

uint32_t newValue = flags.value() | (raw & ~RegExpFlagsMask);
setFixedSlot(FLAGS_SLOT, Int32Value(newValue));

When we read the LegacyFeaturesEnabledBit, we check if it’s set. When we write it, we take the existing raw value and either set or clear the LegacyFeaturesEnabledBit.

2. Lazy Evaluation #

The proposal specifies RegExp properties as internal slots of the RegExp Object, and the abstract operations UpdateLegacyRegExpStaticProperties (C, S, startIndex, endIndex, capturedValues) and InvalidateLegacyRegExpStaticProperties(C) were initially confusing. The confusion came from a specification detail: we need to eagerly update the properties at a specific point in time, as opposed to SpiderMonkey’s lazily evaluated implementation.

It was the first time I had come across lazy evaluation and thought, naively, that it would be possible to change the implementation to eagerly update static properties after a successful match. This didn't work for a few reasons.

First, lazy evaluation is heavily embedded in the JIT, so the idea of just changing that was… ambitious. Second, lazy evaluation is a way to defer regexp evaluation until RegExp properties are accessed. Third, there’s no observable difference to the end user whether the RegExp properties were lazily or eagerly evaluated. Lastly, internal slots are a way for ECMA262 to describe the internal state of the object.

So, UpdateLegacyRegExpStaticProperties (C, S, startIndex, endIndex, capturedValues) wasn’t needed, as it codifies already existing behaviour in SpiderMonkey. For InvalidateLegacyRegExpStaticProperties(C), my mentor suggested implementing it as a boolean flag in RegExpStatics.

When a subclass or cross-realm regexp executes, this flag is set to true, preventing legacy static properties from being accessed. The flag is cleared after normal RegExp executions, allowing legacy features to work for standard RegExp instances.

Because InvalidateLegacyRegExpStaticProperties(C) marks the values of the static properties as unavailable by setting the internal slots to empty, in step 4 of the accessors GetLegacyRegExpStaticProperty(C, thisValue, internalSlotName), we throw a TypeError if the static properties are invalidated.

Then, we add the equivalent code in the JIT path and so that when a regexp is executed, we lazily store enough information to be able to rerun the regexp later if the RegExpStatics are accessed.

3. Gating the implementation behind a preference #

The first step to implementing a TC39 proposal in SpiderMonkey is adding a preference for it. This allows the feature to be enabled or disabled at runtime, which is important in gating the feature until it has been tested enough for release.

With this proposal, it was awkward, because this was not a new syntax or library method, but behavioral modifications to the existing RegExp static properties and the compile() method.

At first, I enclosed my changes in an #ifdef NIGHTLY_BUILD directive so that they are only available in the nightly environment. But given the potential for web compatibility risks, we needed to put the changes behind a preference. That way, we can flip the feature back in case we break something.

This created an awkward situation: the static RegExp properties themselves (like RegExp.$1, RegExp.input) are defined in regexp_static_props, which is baked into the static RegExp JSClass and embedded in the binary at compile time. I ended up wrapping these property definitions in an #ifdef NIGHTLY_BUILD, meaning they only exist in Nightly builds.

But the behavior of these properties — that is, whether accessing them should throw errors for subclasses and cross-realm regexps — is gated behind a runtime preference. This is even more awkward, because it will change behaviour in Nightly even without the preference enabled.

Thus, the preference only controls whether the new throwing behavior is active. As Iain noted, there wasn't a particularly clean way to avoid this. We'd need two parallel RegExp classes and then have to switch between them at runtime based on the pref, which seemed like overkill.

The compromise was to ship the properties in Nightly, use the preference to control the new behavior, and rely on extra-careful testing.

4. Wild Goose Chase #

Around August, when I had the initial implementation working without memory optimization or centralized legacy and realm checks, I was updating legacy regexp statics in RegExpBuiltinExec() only when matches succeeded.

RegExpBuiltinExec() has two execution paths: a forTest path for RegExp.prototype.test (where we can skip allocating a result object) and a normal path for full execution. I had legacy feature validation in both paths, but only for successful matches.

My mentor suggested we needed to update the legacy regexp statics not just on success, but also on failure. That made sense from a spec perspective, so I spent the next week and a half trying to figure out how to implement this. I was looking into the execution paths, trying to understand where and how to trigger updates on failed matches.

After about a week, we realized that they had misread the proposal! Oops. Turns out, SpiderMonkey doesn't update legacy regexp properties on failure at all: it just returns the last successful result. I'd been chasing a solution to a problem that didn't actually exist in the implementation.

Next Steps and Final Thoughts #

The "Legacy RegExp features in JavaScript" proposal is, at the time of this writing, in stage 3 of the TC39 process, meaning the proposal is stable and no further changes can be made to it. There are potential backward compatibility risks and any attempt to use a disabled feature will throw a Type Error. More on that can be found in the Breaking Hazards portion of the proposal.

Before implementing this proposal I had briefly interacted with C++ on a production level codebase when working on the Error.isError proposal, but working on legacy RegExp properties was a deeper dive into C++ and browser internals, which was difficult but also very much appreciated!

Working on this proposal exposed gaps in my knowledge but also gave me confidence in navigating large C++ codebases. I’m particularly grateful to my mentor, and Daniel Minor and Iain Ireland (from the SpiderMonkey team) for pointing me in the right direction and brainstorming solutions with me.

You may also like: #

A New RegExp Engine in SpiderMonkey

Implementing Iterator.range in SpiderMonkey

Update on what happened in WebKit in the week from December 26 to January 19.

We're back! The first periodical of 2026 brings you performance optimizations, improvements to the memory footprint calculation, new APIs, the removal of the legacy Qt5 WPE backend, and as always, progress on JSC's Temporal implementation.

Cross-Port 🐱

JavaScriptCore 🐟

Graphics 🖼️

WPE WebKit 📟

WPE Platform API 🧩

This is a brief blog post to highlight the growth of the Servo community in recent years, particularly since Igalia took over the project maintenance in 2023.

Note that this doesn’t talk about the technical achievements, though there have been tons of them in the last years. A picture is worth a thousand words so just take a look at this slide from my latest Servo talk which shows how google.com was rendered with Servo at the beginning of 2023 vs September 2025.

PRs numbers #

So like we did last year, let’s take a look at the PRs merged on the main Servo repository on GitHub since 2018.

• PRs: total number of PRs merged.
• Contributors: average number of contributors per month.
• Contributors ≥ 10: average number of contributors that have merged more than 10 PRs per month.

As a clarification, these numbers don’t include PRs from bots (dependabot and Servo WPT Sync).

Checking this we can see we are close to double the numbers from last year! The numbers in 2025 are way bigger than in the previous years (even checking the numbers from 2018-2019), showing a healthy community working on Servo.

The next chart is a different view of the same data but split per month, with the number of PRs landed every month, the number of contributors and the number of contributors with more than 10 patches. It shows the evolution over the years and the high activity last year.

Number of contributors #

Now let’s focus on the last 3 years, since the project reactivation, and the numbers of contributors to the Servo project.

The number of contributors to Servo has tripled since 2023, reaching 146 different contributors in 2025.

If we analyze the rest of the data in this table, we can see that the percentage of contributors that do a single PR to Servo in a year has been reduced, meaning that Servo contributors are now usually doing more than one PR to the project.

If we check the number of contributors that have done more than 10 PRs in a year, we see the percentage almost doubling from 15% to 29% in the last 3 years.

And for the top contributors doing more than 100 PRs in a year, we have gone from 1 in 2023 and 3 in 2024 to 8 last year, which represent the 5% of the Servo contributors, showing a good team of very active contributors to the project.

WPT pass-rate #

Let’s take a look at WPT evolution in 2025.

You can check more information about WPT pass-rates at Servo’s website (where you can also find an explanation of the Score number).

Note that these numbers differ from wpt.fyi because we’re still not running all the WPT tests in Servo, so the total numbers here are smaller.

It’s not easy to extract conclusions from this data, but it shows the Servo project keeps progressing and supporting more web platform features as time passes.

Sometimes these numbers grow artificially as new tests are added to WPT for features that Servo already supports (for example, the biggest jump last year was in October getting 188,281 new subtests passing without any change in Servo, just because new tests were added to WPT).

GitHub stars #

We are about to reach 35,000 stars on GitHub. It’s good to see the project has not stopped growing since the beginning, and the curve has become steeper in recent years.

Other #

If we check to the official project roles, we have now:

• 5 administrators
• 17 TSC members
• 25 maintainers
• 18 contributors

We have also started doing Servo releases, we have done 3 so far.

Also the TSC has setup sponsorship tiers for donations. We got 4 bronze sponsors in 2025 and we hope to increase the number of sponsorships in 2026.

Regarding donations, we have defined a funding process to request usage of that money. We are currently using it to sponsor Josh Matthews’ contributions, and pay for self-hosted runners to speed up CI times.

Servo has been present in several events last year, we ended up giving 10 talks all around the globe.

Wrap-up #

The idea here was to do a quick recap of the Servo stats in 2025. Taking a look at these numbers every now and then is useful, and gives you a different perspective about the status of the project, that one can easily ignore during the day-to-day tasks.

In general things have grown a lot in 2025, who knows what would happen in 2026, but we hope we can at least keep similar numbers or maybe even keep growing them further. That would be really great news for the Servo project.

Igalia is really proud of what the whole Servo community has achieved together in the recent years, and we hope for a bright future for the project going forward.

As an aside note, by the end of the month I’ll be at FOSDEM talking about Servo, other Servo folks like Delan Azabani and Martin Robinson will also be there. If you are around, don’t hesitate to say hi and ask anything about the project.

A History of Extensions for Embedders — and Where We’re Heading

Chromium’s Extensions platform has long been a foundational part of the desktop browsing experience. Major Chromium-based browsers—such as Chrome and Microsoft Edge—ship with full support for the Chrome Extensions ecosystem, and user expectations around extension availability and compatibility continue to grow.

In contrast, some Chromium embedders— for instance, products built directly on the //content API without the full //chrome stack—do not naturally have access to Extensions. Similarly, the traditional Chrome for Android app does not support Extensions. While some embedders have attempted to enable limited Extensions functionality by pulling in selected pieces of the //chrome layer, this approach is heavyweight, difficult to maintain, and fundamentally incapable of delivering full feature parity.

At Igalia we have been willing to help on the long term-goal of making Extensions usable on lightweight, //content-based products, without requiring embedders to depend on //chrome. This post outlines the background of that effort, the phases of work so far, the architectural challenges involved, and where the project is headed.

Note: ChromeOS supporting extensions (ChromeOS has announced plans to incorporate more of the Android build stack) is not the same thing as Chrome-Android App supporting extensions. The two codepaths and platform constraints differ significantly. While the traditional Chrome app on Android phones and tablets still does not officially support extensions, recent beta builds of desktop-class Chrome on Android have begun to close this gap by enabling native extension installation and execution.

Tracking bug: https://issues.chromium.org/issues/356905053

Extensions Architecture — Layered View

The following diagram illustrates the architectural evolution of Extensions support for Chromium embedders.

Traditional Chromium Browser Stack

At the top of the stack, Chromium-based browsers such as Chrome and Edge rely on the full //chrome layer. Historically, the Extensions platform has lived deeply inside this layer, tightly coupled with Chrome-specific concepts such as Profile, browser windows, UI surfaces, and Chrome services.

+-----------------------+
|      //chrome         |
|  (UI, Browser, etc.)  |
+-----------------------+
|     //extensions      |
+-----------------------+
|      //content        |
+-----------------------+

This architecture works well for full browsers, but it is problematic for embedders. Products built directly on //content cannot reuse Extensions without pulling in a large portion of //chrome, leading to high integration and maintenance costs.

Phase 1 — Extensions on Android (Downstream Work)

In 2023, a downstream project at Igalia required extension support on a Chromium-based Android application. The scope was limited—we only needed to support a small number of specific extensions—so we implemented:

• basic installation logic,
• manifest handling,
• extension launch/execution flows, and
• a minimal subset of Extensions APIs that those extensions depended on.

This work demonstrated that Extensions can function in an Android environment. However, it also highlighted a major problem: modifying the Android //chrome codepath is expensive. Rebasing costs are high, upstream alignment is difficult, and the resulting solution is tightly coupled to Chrome-specific abstractions. The approach was viable only because the downstream requirements were narrow and controlled.

I shared this experience at BlinkOn Lightning Talk: “Extensions on Android”.

Phase 2 — Extensions for Embedders
( //content + //extensions + //components/extensions )

Following Phase 1, we began asking a broader question:

Can we provide a reusable, upstream-friendly Extensions implementation that works for embedders without pulling in the //chrome layer?

Motivation

Many embedders aim to remain as lightweight as possible. Requiring //chrome introduces unnecessary complexity, long build times, and ongoing maintenance costs. Our hypothesis was that large portions of the Extensions stack could be decoupled from Chrome and reused directly by content-based products.

One early idea was to componentize the Extensions code by migrating substantial parts of //chrome/*/extensions into //components/extensions.

+-------------------------+
| //components/extensions |
+-------------------------+
|      //extensions       |
+-------------------------+
|       //content         |
+-------------------------+

Proof-of-concept : Wolvic

We tested this idea through Wolvic , a VR browser used in several commercial
solutions. Wolvic has two implementations:

• a Gecko-based version, and
• a Chromium-based version built directly on the //content API.

Originally, Extensions were already supported in Wolvic-Gecko, but not in Wolvic-Chromium. To close that gap, we migrated core pieces of the Extensions machinery into //components/extensions and enabled extension loading and execution in a content-only environment.

By early 2025, this work successfully demonstrated that Extensions could run without the //chrome layer.

Demo video::
https://youtube.com/shorts/JmQnpC-lxR8?si=Xf0uB6q__j4pmlSj

Design document:
https://docs.google.com/document/d/1I5p4B0XpypR7inPqq1ZnGMP4k-IGeOpKGvCFS0EDWHk/edit?usp=sharing

However, this work lived entirely in the Wolvic repository, which is a fork of Chromium. While open source, this meant that other embedders could not easily benefit without additional rebasing and integration work.

This raised an important question:

Why not do this work directly in the Chromium upstream so that all embedders can benefit?

Phase 3 — Extensions for Embedders
(//content + //extensions)

Following discussions with the Extensions owner (rdevlin.cronin@chromium.org), we refined the approach further.

Rather than migrating functionality into //components, the preferred long-term direction is to move Extensions logic directly into the //extensions layer wherever possible.

+-----------------------+
|      Embedder UI      | (minimal interfaces)
+-----------------------+
|      //extensions     |
+-----------------------+
|       //content       |
+-----------------------+

This approach offers several advantages:

• clearer layering and ownership,
• fewer architectural violations,
• reduced duplication between Chrome and embedders,
• a cleaner API surface for integration.

We aligned on this direction and began upstream work accordingly.

Tracking bug: https://issues.chromium.org/issues/358567092

Our goals for Content Shell + //extensions are:

1. Embedders should only implement a small set of interfaces, primarily for UI surfaces (install prompts, permission dialogs) and optional behaviors.
2. Full Web Extensions APIs support
   w3c standard : https://w3c.github.io/webextensions/specification/
3. Chrome Web Store compatibility
   Embedders should be able to install and run extensions directly from the Chrome Web Store.

Short-term Goal: Installation Support

Our immediate milestone is to make installation work entirely using //content + //extensions.

Current progress:

• .zip installation support already lives in //extensions
• Migrating Unpacked directory installation from //chrome to //extensions
  (including replacing Profile with BrowserContext abstractions)
• Moving .crx installation code from //chrome → //extensions
  
  As part of this effort, we are introducing clean, well-defined interfaces for install prompts and permission confirmations:
• Chrome will continue to provide its full-featured UI
• Embedders can implement minimal, custom UI as needed

What Comes Next:

Once installation is fully supported, we will move on to:

• Chrome Web Store integration flows
• Core WebExtensions APIs required by commonly used extensions

Main Engineering Challenge — Detaching from the Chrome Layer

The hardest part of this migration is not moving files—it is breaking long-standing dependencies on the //chrome layer.

The Extensions codebase is large and historically coupled to Chrome-only concepts such as:

• Profile
• Browser
• Chrome-specific WebContents delegates
• Chrome UI surfaces
• Chrome services (sync, signin, prefs)

Each migration requires careful refactoring, layering reviews, and close collaboration with component owners. While the process is slow, it has already resulted in meaningful architectural improvements.

What’s Next?

In the next post, We’ll demonstrate:

A functioning version of Extensions running on top of
//content + //extensions only — capable of installing and running extensions app.

from Igalia side, we continue working on ways to make easier integrating Chromium on other platforms, etc. This will mark the first end-to-end, //chrome-free execution path for extensions in content-based browsers.

Stay tuned!

How much energy is consumed when querying an LLM? We're largely in the dark when it comes to proprietary models, but for open weight models that anyone can host on readily available, albeit eye-wateringly expensive, hardware this is something that can be measured and reported, right? In fact, given other people are doing the hard work of setting up and running benchmarks across all kinds of different hardware and software configurations for common open weight models, can we just re-use that to get a reasonable figure in terms of Watt-hours (Wh) per query?

For the kind of model you can run locally on a consumer GPU then of course there's some value in seeing how low the per-query energy usage might be on a large scale commercial setup. But my main interest is in larger and more capable models, the kind that you wouldn't realistically run locally and end up using in a pay-per-token manner either directly with your host of choice or through an intermediary like OpenRouter. In these cases where models are efficiently served with a minimum of 4-8 GPUs or even multi-node clusters it's not easy to get a feel for the resources you're using. I'm pretty happy that simple back of the envelope maths shows that whether providers are properly amortising the cost of their GPUs or not, it's implausible that they're selling per-token API access for open models at below the cost of electricity. That gives a kind of upper bound on energy usage, and looking at the pennies I spend on such services it's clearly a drop in the ocean compared to my overall energy footprint. But it's not a very tight bound, which means it's hard to assess the impact of increasing my usage.

We can look at things like Google's published figures on energy usage for Gemini but this doesn't help much. They don't disclose the length of the median prompt and its response, or details of the model used to serve that median query meaning it's not helpful for either estimating how it might apply to other models or how it might apply to your own usage (which may be far away from this mysterious median query). Mistral released data on the per query environmental impact (assuming for a 400 token query), but the size of the Mistral Large 2 model is not disclosed and they don't calculate a Wh per query figure. CO2 and water per query are very helpful to evaluate a particular deployment, but the actual energy used is a better starting point that can be applied to other providers assuming different levels of carbon intensity. If one of the API providers were to share statistics based on a real world deployment of one of the open models with a much higher degree of transparency (i.e. sharing stats on the number of queries served during the period, statistics on their length, and measured system power draw) that would be a useful source of data. But today we're looking at what we can conclude from the InferenceMAX benchmark suite published results.

I'd started looking at options for getting good figures thinking I might have to invest in the hassle and expense of renting a multi-GPU cloud instance to run my own benchmarks, then felt InferenceMAX may make that unnecessary. After writing this up along with all my provisos I'm perhaps tempted again to try to generate figures myself. Anyway, read on for a more detailed look at that benchmark suite. You can scroll past all the provisos and jump ahead to the figures giving the Wh/query figures implied by the benchmark results across different GPUs, different average input/output sequence lengths, and for gpt-oss 120B and DeepSeek-R1-0528. But I hope you'll feel a bit guilty about it.

If you see any errors, please let me know.

High-level notes on InferenceMAX

InferenceMAX benchmark suite has the stated goal to "provide benchmarks that both emulate real world applications as much as possible and reflect the continuous pace of software innovation." They differentiate themselves from other benchmarking efforts noting "Existing performance benchmarks quickly become obsolete because they are static, and participants often game the benchmarks with unrealistic, highly specific configurations."

The question I'm trying to answer is "what is the most 'useful AI' I can expect for a modern GPU cluster in a realistic deployment and how much energy does it consume". Any benchmark is going to show peak throughput higher than you'd expect to achieve in real workload and there's naturally a desire to keep it pinned on a specific model for as long as it isn't totally irrelevant in order to enable comparisons as hardware and software evolves with a common point of reference. But although I might make slightly different choices about what gets benchmarked and how, the InferenceMAX setup at first look seems broadly aligned with what I want to achieve.

They benchmark DeepSeek-R1-0528 (both at the native fp8 quantisation and at fp4) which is a 671B parameter model with 37B active weights released ~7 months ago and seems a fair representative of a large MoE open weight model. gpt-oss-120b is also benchmarked, providing a point of comparison for a much smaller and efficient to run model. Different input sequence length and output sequence length (ISL and OSL - the number of input and output tokens) are tested: 1k/1k, 1k/8k, 8k/1k, which provides coverage of different query types. Plus tests against a wide range of GPUs (including the 72-GPU GB200 NVL72 cluster) and sweeps different settings.

At the time of writing you might reasonably consider to be 'InferenceMAX' is split into around three pieces:

• The frontend website you can see at inferencemax.semianalysis.com (not currently open source but planned to be)
• The script for executing queries against the LLM serving infrastructure and collecting stats (currently in a seperate repo but planned to be incorporated into the main InferenceMAX repository),
• The wrapper/runner scripts and GitHub actions workflows that live in the main InferenceMAX repository.
  • This is actively contributed to by at least Nvidia and AMD engineers.

GitHub Actions is used to orchestrate the runs, ultimately producing a zip file containing JSON with the statistics of each configuration (e.g. here). The benchmark_serving.py script is invoked via the run_benchmark_serving wrapper in benchmark_lib.sh which hardcodes some options and passes through some others from the workflow YAML. The results logged by benchmark_serving.py are processed in InferenceMAX's process_result.py helper which will produce JSON in the desired output format. Together, these scripts provide statistics like throughput (input and output token), end to end latency, interactivity (output tokens per second) etc.

Further studying the benchmark setup

So, let's look at the benchmarking logic in more detail to look for any surprises or things that might affect the accuracy of the Wh-per-query figure I want to generate. I'll note that InferenceMAX is an ongoing project that is actively being developed. These observations are based on a recent repo checkout, but of course things may have changed since then if you're reading this post some time after it was first published.

Looking through I made the following observations. Some represent potential issues (see the next subheading for a list of the upstream issues I filed), while others are just notes based on aspects of the benchmark I wanted to better understand.

• One of the required arguments to the benchmark serving script is --random-range-ratio. This is set by default to 0.8 in benchmark-tmpl.yml and in benchmark-multinode-tmpl.yml and is not overridden elsewhere.
  • This argument is ultimately used in sample_random_requests. It uses np.random.randint to sample input/output lengths between the range_ratio * {input,output}_len and {input,output}_len.
  • Taken together, this logic means for for a workload advertised as having 8k input or output tokens (8192), the benchmark will actually run with an average ~7373 (0.9*num_tokens, due to the length being a random number between 0.8*num_tokens and num_tokens) tokens.
  • Because the throughput figures are calculated using the actual input and output token lengths, the figure does represent what was observed, it's just the workload doesn't quite match the description. The reported end to end latency for instance will be misleadingly lower than you would get for a workload that actually did have the expected input / output sequence lengths.
• The various request functions in backend_request.func.py will set output.success = False if they don't get a HTTP 200 status code back for a request. There is no logic to retry a refused request and metrics will be calculated skipping any failed requests. This means an overloaded server will perform better on this benchmark for metrics like E2E latency and TTFT if it refuses requests rather than accept them and serve them slowly. As the number of failed requests isn't included in the results json it's not easy to tell if this is a factor for any benchmarks.
• Many of the various scripts in the benchmarks/ subdirectory set a max-model-len parameter or the similar --max_seq_len parameter for trt-llm (e.g. the b200 config which if I'm not mistaken will ultimately be set from the max_model_len defined in generate_sweep_configs.py. This parameter is documented in vllm and in TensortRT-LLM and controls the maximum supported length of a request, including both the prompt and any generated output. Setting it 20 or 200 tokens above the sum of the benchmarked ISL+OSL to minimise memory use does not seem like a realistic real-world deployment, which seems the wrong choice given the InferenceMAX complaint that in other suites "participants often game the benchmarks with unrealistic, highly specific configurations". Benchmarks naturally show a 'best case', but if you're generating figures like $ per M tokens it's a figure that makes little sense if it reflects a configuration you wouldn't feasibly use/sell.
• Throughput is calculated in benchmark_serving.py based on the total number of tokens divided by the duration of the benchmark. This is then normalised on a per-GPU basis in process_result.py. No problems here, I just wanted to clarify the source of the figure.
• In terms of the source of the input tokens themselves, we can see that --dataset-name random is always passed to benchmark_serving.py. This leads to sample_random_requests being called, which will pick random token ids and create a list of tokens of the desired length (mapping these randomly picked IDs to tokens).
  • The --ignore-eos flag is passed to the benchmark_serving.py script which will in turn set this option in the JSON when making the LLM request. backend_request_func.py sets this and also sets max_tokens to the desired output_len which should ensure that the response has that exact desired number of output tokens. ignore_eos means that the LLM server will keep generating tokens even after seeing the end of sequence token.
  • It's interesting that some of the benchmark configurations enable multi-token prediction, and presumably find it beneficial even given the totally random token inputs. Is it possible that such configurations benefit from undesirable looped outputs (due to a combination of random inputs and continuing to sample tokens past the EOS marker) that potentially are very predictable and give an extra boost?
• The --num-prompts parameter controls the total number of requests that are issued. The benchmark script is written so it will wait for all of these to complete (either successfully or unsuccessfully). This is typically set to the concurrency times 10, but some benchmark setups set it higher (presumably as the default figure finishes too quickly for good results).
• In terms of how requests are submitted with a certain level of concurrency:
  • See above for a discussion of the total number of requests
  • --request-rate inf is always passed, so there's no limit on submitting requests up to the concurency limit.
  • It precomputes a list of requests to submit and then uses a semaphore to limit concurrency but otherwise continuously submits requests up to the concurrency limit, and then waits until they call complete.
• There are no tests that the configuration is serving the model with the expected quality currently, but there's an issue tracking at least adding a simple quality benchmark. Although none of the explored settings should impact the quality of output, it's always possible they trigger a bug and in this case it's not interesting to benchmark.
• It would be helpful for reproducibility if more complete system information for the benchmark runners was released. This is being worked on.
• You should of course consider whether the tested input and output sequence lengths correspond to a workload you are interested in (thank you to Aaron Zhao for reminding me to mention this. This benchmarking approach also doesn't consider caching. Both factors could be highly relevant if trying to estimate energy cost for a long context chat or 'agentic' flow. But I'm happy enough with the tested workloads as a starting point, and my main focus here is trying to get a degree of comfort with the reported numbers for the ISL/OSL combinations they've chosen to test.

Filed issues

I ended up filing the following issues upstream:

• FIXED Token throughput per MW is described as reflecting the generated tokens but is actually processed+generated tokens
  • The companion article introducing InferenceMAX has previously defined throughput as the rate at which the GPU generates tokens yet the figure displayed in the UI was the total number of output and input tokens per second. The definition in the article has now been fixed, and changes to the UI make it more obvious based on context that throughput refers to input+output tokens (as y-axis metric options now exist to show "input token throughput per GPU" and "output token throughput per GPU").
  • This talking head video from Nvidia seems to make the same error, talking about the number of tokens 'generated' per second per GPU when looking at the relevant results these sem to be the total throughput (i.e. output plus the much faster to process input tokens).
• Presented input/output token throughput per GPU for disaggregated setups not usefully comparable to standard multi-gpu setups
  • In disaggregated setups you have some number of GPUs dedicated to prefill (processing input tokens) and some number dedicated to decode (generating output tokens). In this case, the reported input/output throughput figures refer to the input or output throughput per prefill GPU or per decode GPU. It doesn't make sense (IMHO) to plot this figure against the input/output throughput figures for a non-disaggregated setup. To make it comparable, the input/output throughput per GPU should be calculated by averaging across the whole cluster rather than just the GPUs dedicated to prefill or decode respectively.
• Standard deviatation of interactivity (std_intvty) in result json is incorrectly calculated
  • Not a big issue as the figure isn't used anywhere. Interactivity (tokens/second) metrics are calculated from the recorded time per output token. 1000/$tpot_metric is correct for the mean, median, and p99 figures but mathematically incorrect for the standard deviation. e.g. a small standard deviation for time per output token will result in a huge standard deviation being computed for interactivity.
• FIXED Reference kW figures no longer shown in frontend for each GPU
  • At some point updates to the frontend logic meant that the per-GPU kW figures used in calculating the token throughput per utility MW were no longer displayed. This has now been fixed.
• How will full workflow run output be retained beyond 90 days
  • The benchmark frontend helpfully links to the GitHub Actions run that generated the displayed results and has a datepicker to view previous results. Clicking through to GitHub means you can download the original .zip of the JSON format benchmark results which is something I take advantage of in the analysis later in this article. According to GitHub docs, the maximum retention period for Actions artifacts and logs is 90 days for a public repo. It would be good to have a mechanism so that this information is backed up rather than lost.
• Contents of CONFIG_DIR path as used in launch_gb200-nv.sh is undisclosed
  • Most benchmark configuration lives in the main repository, but unfortunately one of the Nvidia DeepSeek R1 configurations relies on a config dir that's not publicly available meaning it can't be audited or reproduced. This is a case where tightening up benchmark rules and review process can hopefully avoid it happening in the future.
• Reconsider allowing setting max_model_len / max_seq_len to isl+osl+tiny_margin
  • As explained above, a number of benchmarks set max_model_len (or for Nvidia's TensorRT, --max_seq_len) to some figure that is just above ISL+OSL. Although some degree of tuning is expected, to me this goes against the idea that "We want server configs to reflect real world deployments as much as possible" and the stated goal "to provide benchmarks that both emulate real world applications as much as possible and reflect the continuous pace of software innovation". It's hard to imagine a realistic deployment that would configure their serving engine in a way such that it errors if input+output tokens passes ~2k tokens for instance. Looking at the DeepSeek R1 0528 providers on OpenRouter, the vast majority offer greater than 128k context.
  • By my understanding, with PagedAttention the KV cache is dynamically allocated anyway so this setting would largely impact other data structures. Plus vllm at least contains a startup check that there is sufficient VRAM to serve at least one request at the maximum configured context. I would really like to see what impact this setting has on benchmarks.
  • The repository maintainers renamed my issue to a title that doesn't reflect my report. I'm hopeful they will review my recent comment and title it back.
• Some reported metrics will be inflated if a serving engine sheds load
  • This covers the observation made above that failed requests are simply skipped. As the number of failed requests isn't tracked, it's not easy to see if a particular configuration may appear better (better E2E latency, lower time to first token) as a reset of shedding load rather than queueing.
  • The repository maintainers renamed this issue to "[feature suggestion for vllm/vllm benchmark_serving]" and closed it. I'm hopeful they will read my response and reconsider on the grounds that:
    • The benchmark_serving script isn't doing anything "wrong" necessarily. It is simply making an implementation choice with potential impact on results that the InferenceMAX harness isn't tracking.
    • The script is planned to be added to the repo soon anyway.
• Benchmarked ISL and OSL averages 0.9*target_length meaning results are over-optimistic.
  • This is the problem mentioned above where the introduced variance in input/output sequence length has an average lower than the headline rate. As noted, this means specifically the end to end latency figure is misleading, but also impacts tokens/second and throughput to the extent that the cost of serving a query doesn't scale with O(n).
  • This will be fixed by PR 339 which upstreams the benchmark_serving.py script and in that modified branch changes sample_random_requests to sample a range with multiplier between 1 - RANGE_RATIO and 1 + RANGE_RATIO.

In the best case, you'd hope to look at the benchmark results, accept they're probably represent a higher degree of efficiency than you'd likely get on a real workload, that an API provider might achieve 50% of that and double the effective cost per query to give a very rough upper estimate on per-query cost But that only really works if the reported benchmark results roughly match the achievable throughput in a setup configured for commercial serving. Given the tuning to specific isl/osl values, I'm not at all confident thats the case and I don't know how wide the gap is.

Generating results

Firstly I wrote a quick script to check some assumptions about the data and look for anything that seems anomalous. Specifically:

• Check that total throughput per GPU matches what you'd expect based on the input token and output token throughput per GPU, even in the disaggregated case. i.e. the total thoughput per GPU averaged over the whole cluster should equal the sum of the input and output throughput per GPU provided those figures are averaged over the whole cluster.
• The ratio of input token throughput to output token throughput should be almost equal to the to the ratio of input to output tokens in the benchmark's workload. If not, there is something surprising that needs investigating.

Based on the information available in the generated result JSON and the reported all-in power per GPU (based on SemiAnalysis' model), we can calculate the Watt hours per query. First calculate the joules per token (watts per GPU divided by the total throughput per GPU). This gives a weighted average of the joules per token for the measured workload (i.e. reflecting the ratio of isl:osl). Multiplying joules per token by the tokens per query (isl+osl) gives the joules per query, and we can just divide by 3600 to get Wh.

There is some imprecision because we're constructing the figure for e.g. 8192/1024 ISL based on measurements with an average 0.9*8192 input and 0.9*1024 output length. The whole calculation would be much simpler if the benchmark harness recorded the number of queries executed and in what time, meaning we can directly calculate the Wh/query from the Wh for the system over the benchmark duration divided by the number of queries served (and remembering that in the current setup each query is on average 90% of the advertised sequence length).

This logic is wrapped up in a simple script.

There's been a recent change to remove the 'full sweep' workflows in favour of only triggering a subset of runs when there is a relevant change. But I grabbed my results from before this happened, from a December 15th 2025 run. However when finalising this article I spotted Nvidia managed to land some new NVL72 DeepSeek R1 0528 configurations just before Christmas, so I've merged in those results as well, using a run from December 19th. All data and scripts are collected together in this Gist.

Results

As well as giving the calculated Wh per query, the script also gives a comparison point of minutes of PS5 gameplay (according to Sony, "Active Power Consumption" ranges from ~217W to ~197W depending on model - we'll just use 200W). The idea here is to provide some kind of reference point for what a given Wh figure means in real-world times, rather than focusing solely on the relative differences between different deployments. Comparisons to "minutes of internet streaming" seem popular at the moment, presumably as it's because an activity basically everyone does. I'm steering away from that because I'd be comparing one value that's hard to estimate accurately and has many provisos to another figure that's hard to estimate accurately and has many provisos, which just injects more error and uncertainty into this effort to better measure/understand/contextualise energy used for LLM inference.

I'm now going to cherry-pick some results for discussion. Firstly for DeepSeek R1 0528 with 8k/1k ISL/OSL, we see that the reported configurations that give a usable level of interactivity at fp8 report between 0.96-3.74 Wh/query (equivalent to 0.29-1.12 minutes of PS5 gaming). The top row which is substantially more efficient is the newer GB200 NVL72 configuration added at the end of last year. It's not totally easy to trace the configuration changes given they're accompanied by a reworking of the associated scripts, but as far as I can see the configuration ultimately used is this file from the dynamo repository. Looking at the JSON the big gain comes from significantly higher prefill throughput (with output throughput per GPU remaining roughly the same). This indicates the older results (the second row) were bottlenecked waiting for waiting for prefill to complete.

Now taking a look at the results for an fp4 quantisation of the same workload, the result is significantly cheaper to serve with similer or better interactivity and the NVL72 setup Nvidia submitted does have a significant advantage over the 4/8 GPU clusters. This time we see 0.63-1.67 Wh/query (equivalent to 0.19-0.50 minutes of PS5 power draw while gaming). Serving at a lower quantisation impacts the quality of results of course, but the improved efficiency, including on smaler 4 GPU setups helps demonstrate why models like Kimi K2 thinking are distributed as "native int4", with benchmark results reported at this quantisation and quantisation aware training used to maintain quality of result.

Looking now at the 1k/8k workload (i.e. generating significant output) and the cost is 15.0-16.3 Wh/query (equivalent to 4.49-4.89 minutes of PS5 power draw while gaming). As expected this is significantly higher than the 8k/1k workload as prefill (processing input tokens) is much cheaper per token than decode (generating output tokens)

Again, fp4 has a significant improvement in efficiency:

As you'd expect for a much smaller model at native fp4 quantisation, GPT-OSS-120B is much cheaper to serve. e.g. for 8k/1k:

Or for 1k/8k:

Conclusion

Well, this took rather a lot more work than I thought it would and I'm not yet fully satisfied with the result. Partly we have to accept a degree of fuzziness about marginal energy usage of an individual query - it's going to depend on the overall workload of the system so there's going to be some approximation when you try to cost a single query.

I'm glad that InferenceMAX exists and am especially glad that it's open and publicly developed, which is what has allowed me to dive into its implementation to the extent I have and flag concerns/issues. I feel it's not yet fully living up to its aim of providing results that reflect real world application, but I hope that will improve with further maturation and better rules for benchmark participants. Of course, it may still make most sense to collect benchmark figures myself and even if doing so, being able to refer to the benchmarked configurations and get an indication of what hardware can achieve what performance is helpful in doing so. Renting a 72-GPU cluster is expensive and as far as I can see not typically available for a short time, so any benchmarking run by myself would be limited to 4-8 GPU configurations. If the gap in efficiency is huge for such setups vs the NVL72 then these smaller setups are maybe less interesting.

If I found the time to run benchmarks myself, what would I be testing? I'd move to DeepSeek V3.2. One of the big features of this release was the movement to a new attention mechanism which scales much closer to linearly with sequence length. With e.g. Kimi Linear and Qwen3-Next, other labs are moving in a similar direction experimentally at least. I'd try to set up 8 GPU configuration with sglang/vllm configured in a way that it would be capable of serving a commercial workload with varied input/output sequence lengths and test this is the case (Chutes provide their deployed configs which may be another reference point). I'd want to see how much the effective Wh per million input/output tokens varies depending on the different isl/osl workloads. These should be relatively similar given the linear attention mechanism, and if so it's a lot easier to estimate the rough energy cost of a series of your own queries of varied length. I would stick with the random input tokens for the time being.

So where does that leave us? All of this and we've got figures for two particular models, with one benchmark harness, a limited set of input/output sequence lengths, and a range of potential issues that might impact the conclusion. I think this is a useful yardstick / datapoint, though I'd like to get towards something that's even more useful and that I have more faith in.

• 2026-01-09:
  • Fix broken link.
  • Add note that more complete system info would be helpful for reproducibility.
  • Add note about variety of input/output sequence lengths tested.
• 2026-01-07: Initial publication date.

The Secret Life of Custom Elements

Twenty years ago last month, Google published an analysis of "slightly over a billion documents," a snapshot of the web that helped shape the early direction of HTML5. It followed a lineage of smaller, more personal studies — individuals poking at the web to answer some narrow question, often with datasets that would easily fit on a thumb drive today. For about half those two decades, I’ve been arguing that we need more study of the web, not less. The platform evolves faster than our understanding of it, and the only way to know what the web actually is — not what we imagine it to be — is to look.

Every month the HTTP Archive quietly captures a snapshot of the web as it actually exists—not the idealized web that we hope for, but the messy, improvised, duct‑taped reality of millions of sites in the wild. I’ve been collecting and studying these elements for the last six years.

This new dataset is the largest I’ve ever worked with: Billions of pages, hundreds of thousands of distinct non-standard element names, and a long tail that stretches into places no standards body has ever seriously examined. And unlike the Google study, which looked for patterns in class names, this dataset captures the long tail of non‑standard elements — the names people invent for actual elements when the platform doesn’t give them what they need.

What emerges is a portrait of the web as it is lived: messy, inventive, repetitive, global, and full of reinvention. It’s also a mirror held up to the platform itself.

But, it's also much more complex to study than I could have imagined a decade ago, and I really wish that the W3C (and member orgs which include academia) had taken up the charge to begin to figure out how to really study the web and use that information to inform standards work.

What's difficult about it...

One problem is that the dataset itself has some fairly extreme bias. The crawl doesn't hit anything that isn't on the public internet - that means it excludes intranets which are massive. In fact, most of my career was spent working on intranets. The crawl captures only home pages, plus the target of whatever it interprets as the largest link on that page. It also can't get to anything that requires login - which means that for a site like twitter or bluesky or mastodon, you're going to get something very unrepresentative of any of those. So, one challenge I'd love to see us trying to tackle is how to get even better data representation. It's hard to "pave cowpaths" if they're in a country we can't even see into.

Initially I had this idea that we could watch for the adoption of tags - imagining that we'd get some that would become very popular, just like we did with JavaScript libraries and frameworks. However, it turns out that this is not the signal it might first appear to be. An element appearing in tens of thousands or even hundreds of thousands of pages is often simply because they are part of a larger successful system. If Wix or Shopify create some custom elements that work behind the WYSIWYG tooling, and lots of people use it to create their pages - then suddenly that element gets very very popular - even if it isn't actually particularly good. In fact, we can see shifts in the data where the teams themselves changed their minds and another version supplants the first very quickly because it's simply internal.

Then, I thought that perhaps what we can do with the dataset instead, is to squint at it and look a little more abstractly at what people are naming their elements and see if people are re-solving similar problems. Do we find, for example, multiple non-standard element names that appear to be about tabs? Yes! Clearly that is indicative that we need a native element, right? Maybe. It's a bit more nuanced than that. Here are the most commonly re-created/repeated non-standard element themes:

• Navigation
• Headers and footers
• Carousels and sliders
• Modals
• Search bars
• Product cards
• Login forms
• Cookie banners
• Accordions
• Tabs
• Toasts
• Breadcrumbs

While we don't have several of these in standard HTML, we do have native <header>, <footer>, <nav>, <dialog>, and <search> elements, and even accordions via the name attribute of <details>. And yet, the wild still contains hundreds or thousands of custom elements with names like <app-header>, <site-footer>, <main-nav>, <modal-dialog>, <search-box>, and <accordion-panel>.

Native primitives may exist, but not at the same level of abstraction as these. <header> and <footer> in HTML are structural, not behavioral. <dialog> is behavioral, but not styled. <search> exists, but doesn’t solve autocomplete, filtering, or results.

So developers build those - and, if you stop and think about it, not all non-standard elements are equally as undesirable. Many of them will be simple decorations or thin wrappers that do use their native counterparts. Where there is definitely some interesting thing to study is where there is clear generic need where the platform doesn't provide anything close. Above, tabs, for example.

Observations..

Here are many observations from the data, in no real particular order of importance.

Forms and Inputs: Tweaked, Wrapped, and Re‑Wrapped

Forms and inputs are a great example of the constant re-invention I just described. Sometimes it's because the native element is insufficient, but that's not necessarily the case. In some cases they're just slight wrappers. Among them are lots and lots of "pickers" and "selecters" that show up...

• <custom-select>
• <date-picker>
• <variant-picker>
• <quantity-selector>

There is already a lot of ongoing work to make native form elements (including selects) require less code and just be more stylable and flexible, and the data at least suggests that such efforts will be very welcome.

Hidden Machinery

A surprising number of elements aren’t UI components at all. They’re runtime markers:

• <ng-container>
• <router-outlet>
• <astro-island>
• <ion-router-outlet>
• <next-route-announcer>

These exist because frameworks need declarative boundaries for hydration, routing, rendering or template expansion. I suppose it is debatable wither these are an indicator of “missing HTML features”, or just how much.

Carousels (and sliders... and toasts)

I don't love carousels, but it's hard to deny that they are popular. There are dozens of distinct and identifiable carousel/slider elements in the dataset and they appear a lot. I really dislike a few bits of Google's attempt to make CSS-only carousels possible, but it's pretty clear why they chose to tackle that problem. I guess it is worth stressing again the bias in the dataset here - if there is a page I most expect to see a carousel, it is exactly the primary one the archive crawls. So, while it is the most popular in the dataset, I don't know that it is the most popular all-around. You can see why Google winds up with their proposals though, toasts are on that top list too.

Structural semantics?

There are a few broad categories where the main point seems to be "semantics". That is, very often many of these don't actually do anything, beyond provide some hooks, mainly for styling. They aren't actually even custom elements sometimes (or maybe even often) - just non-standard elements.

e-commerce

Dozens of these surround e-commerce. There are tens of thousands of sites that use elements with names (and variants).

• <product-card>
• <product-title>
• <product-price>
• <product-rating>
• <product-variant>
• <product-gallery>
• <product-description>
• <product-badge>

• <price-money>
• <sale-price>
• <compare-at-price>
• <discount-amount>
• <currency-display>

• <stock-status>
• <pickup-availability>
• <delivery-estimate>
• <inventory-level>

• <cart-items>
• <cart-count>
• <checkout-button>
• <order-summary>

Very interestingly they are often used alongside actual machine readable semantics via jsonLD in the same markup.

While the vast majority of these elements appear because of common tooling, the fact that there are dozens of variants of similar names appearing on smaller numbers of sites indicates there is something widely interesting here. It's hard to say what it is other than that it would be nice to have a common structural semantic that would work for both purposes.

I guess the biggest surprise here is that if it's true, why hasn't such a thing arisen already? It is entirely within the community's power to develop such a thing. Perhaps the answer is that there is just so much variance it isn't easily plausible. Maybe templating would somehow allow us to achieve a common pattern which achieved this based on the shared jsonLD semantics.

Publishing & Editorial Semantics

CMSes and news sites often invent tags for editorial structure, and many of these are sticking around.

• <article-header>
• <article-summary>
• <article-author>
• <article-date>
• <article-tags>
• <article-tag>
• <article-category>
• <byline>
• <dateline>
• <pullquote>
• <footnote>

• <tag-list>
• <category-label>
• <topic-header>

These reflect the needs of journalism and long‑form content.

Social & Community Semantics

These show up in comment systems, forums, and social platforms.

• <comment>
• <comment-list>
• <comment-item>
• <comment-author>
• <comment-content>
• <comment-date>
• <comment-form>

• <user-avatar>
• <user-name>
• <profile-card>

These encode relationships and interactions, not UI patterns.

• <event-date>
• <event-location>
• <event-schedule>
• <event-details>

Again, these are domain objects, not widgets - and they have well established schema.org or microformats as well.

• <invoice>
• <invoice-line>
• <invoice-total>
• <invoice-summary>

Before the web came along, there were already national and international standards around electronically trading informtation like invoices - and when XML was sold, invoices were a common example. Here we are again.

"Namespaced" Elements

Several elements like `o:p`, `rdf:rdf`, `dc:format`, `cc:work`, `fb:like`, `g:plusone` appear in the top 100. These basically were thinking of an XHTML future (namespacing) that never really arrived. However, HTML has always allowed it - so that's just the tag name. In many ways, it's just as good. Interestingly, these may be some of the better examples of what I'd like to see happen - they are widely understood.

Conversely, while hugely successful, the share buttons are more an indication of a desire than something we could actually standardize in precisely that way. They also point to a desire _in time_. Google Plus doesn't even exist anymore, `fb:like` is from a time when Facebook was at the top of the most interesting places to be. Maybe one of the things we've learned is that this is way handier to do at the browser/OS levels? I suppose the Web Share API was a part of thinking how we'd deal with this.

The fact that they both still appear so much is also kind of an indication of age of the page and slow replacement of underlying tools.

Typos, Encoding Errors, and the Weird Stuff

One of the most delightful parts of the dataset is the long tail of what are almost certainly just typos:

• <prodcut-card>
• <navgation>
• <contianer>

The fact that these can appear on tens of thousands of sites because they are part of common tooling helps re-enforce that not every non-standard element is a signal. :)

In conclusion...

I wish that I could say "Ah ha - the data says very clearly that these are the specific things we should definitely 'just write down' now" in the way that I imagined a decade ago, but I don't think we're there yet. I guess if I had to give three things I'd like to see happen from here they'd be:

1. We need lots more effort in thinking about how to study these things. I would love to see real investment in this space. This year, at last, the W3C is hiring someone to study the web. I'm not yet sure what that looks like but I look forward to trying to discuss more with them.

2. We need a real community effort - an Underwriters Labs for custom elements, with participation and funding from orgs with money. We don't necessarily need "the one true tabs" as much as we need a place to find what I expect will be a very few sets of tabs as custom elements which we can trust like we trust native elements. Given a little bit of time, I have faith that this will naturally sort itself into a few 'winners'.

3. That community effort might also include things which won't ever have native implmentations, but which lay down some kind of light semantic meaning or compound styling structure that we all begin to agree on - like product cards or breadcrumbs.

A lot of this is pretty adjacent/close to the ideas behind OpenUI and it's possible some of this could just happen there. However, due mainly to limits and participation, OpenUI has really not really produced custom elements or worked to somehow list or grade and promote them (though we did study them quite a bit in the tabs research). The effort led by Brad Frost to think about a "global design system" in particular might be closer to some of these ideas.

More than a year ago, I set out to revive require(esm) in Node.js and

In earlier posts, I wrote about reviving require(esm) and its iteration process

Recently I landed experimental support for require()-ing synchronous ES modules in Node.js, a feature that has been long overdue

Update on what happened in WebKit in the week from December 16 to December 25.

Right during the holiday season 🎄, the last WIP installment of the year comes packed with new releases, a couple of functions added to the public API, cleanups, better timer handling, and improvements to MathML and WebXR support.

Cross-Port 🐱

Improved timers, by making some of them use the timerfd API. This reduces timer “lateness”—the amount of time elapsed between the configured trigger time, and the effective one—, which in turn improves the perceived smoothness of animations thanks to steadier frame delivery timings. Systems where the timerfd_create and timerfd_settime functions are not available will continue working as before.

Graphics 🖼️

WPE WebKit 📟

WPE Platform API 🧩

WPE Android ↗ 🤖

Releases 📦️

Community & Events 🤝

I’ve long made it clear that I don’t particularly care for the whole Shadow DOM thing.  I believe I understand the problems it tries to solve, and I fully acknowledge that those are problems worth solving.  There are just a bunch of things about it that don’t feel right to me, like how it can break accessibility in a number of ways.

One of those things is how it breaks stuff like the commandFor attribute on <button>s, or the popoverTarget attribute, or a variety of ARIA attributes such as aria-labelledby.  This happens because a Shadow DOMmed component creates a whole separate node tree, which creates a barrier (for a lot of things, to be clear; this is just one class of them).

At least, that’s been the case.  There’s now a proposal to fix that, and prototype implementations in both Chrome and Safari!  In Chrome, it’s covered by the Experimental Web Platform features flag in chrome://flags.  In Safari, you open the Develop > Feature Flags… dialog, search for “referenceTarget”, and enable both flags.

(Disclosure: My employer, Igalia, with support from NLnet, did the WebKit implementation, and also a Gecko implementation that’s being reviewed as I write this.)

If you’re familiar with Shadow DOMming, you know that there are attributes for the <template> element like shadowRootClonable that set how the Shadow DOM for that particular component can be used.  The proposal at hand is for a shadowRootReferenceTarget attribute, which is a string used to identify an element within the Shadowed DOM tree that should be the actual target of any references.  This is backed by a ShadowRoot.referenceTarget API feature.

Take this simple setup as a quick example.

 <label for="consent">I agree to join your marketing email list for some reason</label>
<sp-checkbox id="consent">
	<template>
		<input id="setting" type="checkbox" aria-checked="indeterminate">
		<span id="box"></span>	
	</template> </sp-checkbox> 

Assume there’s some JavaScript to make that stuff inside the Shadow DOM work as intended.  (No, nothing this simple should really be a web component, but let’s assume that someone has created a whole multi-faceted component system for handling rich user interactions or whatever, and someone else has to use it for job-related reasons, and this is one small use of that system.)

The problem is, the <label> element’s for is pointing at consent, which is the ID of the component.  The actual thing that should be targeted is the <input> element with the ID of setting .  We can’t just change the markup to <label for="setting"> because that <input> is trapped in the Shadow tree, where none in the Light beyond may call for it.  So it just plain old doesn’t work.

Under the Reference Target proposal, one way to fix this would look something like this in HTML:

 <label for="consent">I agree to join your marketing email list for some reason</label>
<sp-checkbox id="consent">
	<template shadowRootReferenceTarget="setting">
		<input id="setting" type="checkbox" aria-checked="indeterminate">
		<span id="box"></span>	
	</template> </sp-checkbox> 

With this markup in place, if someone clicks/taps/otherwise activates the label, it points to the ID consent .  That Shadowed component takes that reference and redirects it to an effective target  —  the reference target identified in its shadowRootReferenceTarget attribute.

You could also set up the reference with JavaScript instead of an HTML template:

 <label for="consent">I agree to join your marketing email list for some reason</label>
<sp-checkbox id="consent"></sp-checkbox> 

class SpecialCheckbox extends HTMLElement {
	checked = "mixed";
	constructor() {
		super();
		this.shadowRoot_ = this.attachShadow({ 
			referenceTarget: "setting"
		});
		
		// lines of code to Make It Go
	
	}
} 

Either way, the effective target is the <input> with the ID of setting .

This can be used in any situation where one element targets another, not just with for .  The form and list attributes on inputs would benefit from this.  So, too, would the relatively new popoverTarget and commandFor button attributes.  And all of the ARIA targeting attributes, like aria-controls and aria-errormessage and aria-owns as well.

If reference targets are something you think would be useful in your own work, please give it a try in Chrome or Safari or both, to see if your use cases are being met.  If not, you can leave feedback on issue 1120 to share any problems you run into.  If we’re going to have a Shadow DOM, the least we can do is make it as accessible and useful as possible.

Have something to say to all that? You can add a comment to the post, or email Eric directly.

This article is a continuation of the series on WPE performance considerations. While the previous article touched upon fairly low-level aspects of the DOM tree overhead, this one focuses on more high-level problems related to managing the application’s workload over time. Similarly to before, the considerations and conclusions made in this blog post are strongly related to web applications in the context of embedded devices, and hence the techniques presented should be used with extra care (and benchmarking) if one would like to apply those on desktop-class devices.

The workload #

Typical web applications on embedded devices have their workloads distributed over time in various ways. In practice, however, the workload distributions can usually be fitted into one of the following categories:

1. Idle applications with occasional updates - the applications that present static content and are updated at very low intervals. As an example, one can think of some static dashboard that presents static content and switches the page every, say, 60 seconds - such as e.g. a static departures/arrivals dashboard on the airport.
2. Idle applications with frequent updates - the applications that present static content yet are updated frequently (or are presenting some dynamic content, such as animations occasionally). In that case, one can imagine a similar airport departures/arrivals dashboard, yet with the animated page scrolling happening quite frequently.
3. Active applications with occasional updates - the applications that present some dynamic content (animations, multimedia, etc.), yet with major updates happening very rarely. An example one can think of in this case is an application playing video along with presenting some metadata about it, and switching between other videos every few minutes.
4. Active applications with frequent updates - the applications that present some dynamic content and change the surroundings quite often. In this case, one can think of a stock market dashboard continuously animating the charts and updating the presented real-time statistics very frequently.

Such workloads can be well demonstrated on charts plotting the browser’s CPU usage over time:

As long as the peak workload (due to updates) is small, no negative effects are perceived by the end user. However, when the peak workload is significant, some negative effects may start getting noticeable.

In case of applications from groups (1) and (2) mentioned above, a significant peak workload may not be a problem at all. As long as there are no continuous visual changes and no interaction is allowed during updates, the end-user is unable to notice that the browser was not responsive or missed some frames for some period of time. In such cases, the application designer does not need to worry much about the workload.

In other cases, especially the ones involving applications from groups (3) and (4) mentioned above, the significant peak workload may lead to visual stuttering, as any processing making the browser busy for longer than 16.6 milliseconds will lead to lost frames. In such cases, the workload has to be managed in a way that the peaks are reduced either by optimizing them or distributing them over time.

First step: optimization #

The first step to addressing the peak workload is usually optimization. Modern web platform gives a full variety of tools to optimize all the stages of web application processing done by the browser. The usual process of optimization is a 2-step cycle starting with measuring the bottlenecks and followed by fixing them. In the process, the usual improvements involve:

• using CSS containment,
• using shadow DOM,
• promoting certain parts of the DOM to layers and manipulating them with transforms,
• parallelizing the work with workers/worklets,
• using the visibility CSS property to separate painting from layout,
• optimizing the application itself (JavaScript code, the structure of the DOM, the architecture of the application),
• etc.

Second step: pre-rendering #

Unfortunately, in practice, it’s not uncommon that even very well optimized applications still have too much of a peak workload for the constrained embedded devices they’re used on. In such cases, the last resort solution is pre-rendering. As long as it’s possible from the application business-logic perspective, having at least some web page content pre-rendered is very helpful in situations when workload has to be managed, as pre-rendering allows the web application designer to choose the precise moment when the content should actually be rendered and how it should be done. With that, it’s possible to establish a proper trade-off between reduction in peak workload and the amount of extra memory used for storing the pre-rendered contents.

Pre-rendering techniques #

Nowadays, the web platform provides at lest a few widely-adapted APIs that provide means for the application to perform various kinds of pre-rendering. Also, due to the ways the browsers are implemented, some APIs can be purposely misused to provide pre-rendering techniques not necessarily supported by the specification. However, in the pursuit of good trade-offs, all the possibilities should be taken into account.

Before jumping into particular pre-rendering techniques, it’s necessary to emphasize that the pre-rendering term used in this article refers to the actual rendering being done earlier than it’s visually presented. In that sense, the resource is rasterized to some intermediate form when desired and then just composited by the browser engine’s compositor later.

Pre-rendering offline #

The most basic (and limited at the same time) pre-rendering technique is one that involves rendering offline i.e. before the browser even starts. In that case, the first limitation is that the content to be rendered must be known beforehand. If that’s the case, the rendering can be done in any way, and the result may be captured as e.g. raster or vector image (depending on the desired trade-off). However, the other problem is that such a rendering is usually out of the given web application scope and thus requires extra effort. Moreover, depending on the situation, the amount of extra memory used, the longer web application startup (due to loading the pre-rendered resources), and the processing power required to composite a given resource, it may not always be trivial to obtain the desired gains.

Pre-rendering using canvas #

The first group of actual pre-rendering techniques happening during web application runtime is related to Canvas and OffscreenCavas. Those APIs are really useful as they offer great flexibility in terms of usage and are usually very performant. However, in this case, the natural downside is the lack of support for rendering the DOM inside the canvas. Moreover, canvas has a very limited support for painting text — unlike the DOM, where CSS has a significant amount of features related to it. Interestingly, there’s an ongoing proposal called HTML-in-Canvas that could resolve those limitations to some degree. In fact, Blink has a functioning prototype of it already. However, it may take a while before the spec is mature and widely adopted by other browser engines.

When it comes to actual usage of canvas APIs for pre-rendering, the possibilities are numerous, and there are even more of them when combined with processing using workers. The most popular ones are as follows:

• rendering to an invisible canvas and showing it later,
• rendering to a canvas detached from the DOM and attaching it later,
• rendering to an invisible/detached canvas and producing an image out of it to be shown later,
• rendering to an offscreen canvas and producing an image out of it to be shown later.

When combined with workers, some of the above techniques may be used in the worker threads with the rendered artifacts transferred to the main for presentation purposes. In that case, one must be careful with the transfer itself, as some objects may get serialized, which is very costly. To avoid that, it’s recommended to use transferable objects and always perform a proper benchmarking to make sure the transfer is not involving serialization in the particular case.

While the use of canvas APIs is usually very straightforward, one must be aware of two extra caveats.

First of all, in the case of many techniques mentioned above, there is no guarantee that the browser will perform actual rasterization at the given point in time. To ensure the rasterization is triggered, it’s usually necessary to enforce it using e.g. a dummy readback (getImageData()).

Finally, one should be aware that the usage of canvas comes with some overhead. Therefore, creating many canvases or creating them often, may lead to performance problems that could outweigh the gains from the pre-rendering itself.

Pre-rendering using eventually-invisible layers #

The second group of pre-rendering techniques happening during web application runtime is limited to the DOM rendering and comes out of a combination of purposeful spec misuse and tricking the browser engine into making it rasterizing on demand. As one can imagine, this group of techniques is very much browser-engine-specific. Therefore, it should always be backed by proper benchmarking of all the use cases on the target browsers and target hardware.

In principle, all the techniques of this kind consist of 3 parts:

1. Enforcing the content to be pre-rendered being placed on a separate layer backed by an actual buffer internally in the browser,
2. Tricking the browser’s compositor into thinking that the layer needs to be rasterized right away,
3. Ensuring the layer won’t be composited eventually.

When all the elements are combined together, the browser engine will allocate an internal buffer (e.g. texture) to back the given DOM fragment, it will process that fragment (style recalc, layout), and rasterize it right away. It will do so as it will not have enough information to allow delaying the rasterization of the layer (as e.g. in case of display: none). Then, when the compositing time comes, the layer will turn out to be invisible in practice due to e.g. being occluded, clipped, etc. This way, the rasterization will happen right away, but the results will remain invisible until a later time when the layer is made visible.

In practice, the following approaches can be used to trigger the above behavior:

• for (1), the CSS properties such as will-change: transform, z-index, position: fixed, overflow: hidden etc. can be used depending on the browser engine,
• for (2) and (3), the CSS properties such as opacity: 0, overflow: hidden, contain: strict etc. can be utilized, again, depending on the browser engine.

The scrolling trick

While the above CSS properties allow for various combinations, in case of WPE WebKit in the context of embedded devices (tested on NXP i.MX8M Plus), the combination that has proven to yield the best performance benefits turns out to be a simple approach involving overflow: hidden and scrolling. The example of such an approach is explained below.

Suppose the goal of the application is to update a big table with numbers once every N frames — like in the following demo: random-numbers-bursting-in-table.html?cs=20&rs=20&if=59

With the number of idle frames (if) set to 59, the idea is that the application does nothing significant for the 59 frames, and then every 60th frame it updates all the numbers in the table.

As one can imagine, on constrained embedded devices, such an approach leads to a very heavy workload during every 60th frame and hence to lost frames and unstable application’s FPS.

As long as the numbers are available earlier than every 60th frame, the above application is a perfect example where pre-rendering could be used to reduce the peak workload.

To simulate that, the 3 variants of the approach involving the scrolling trick were prepared for comparison with the above:

• random-numbers-bursting-in-table-prerendered-1.html?cs=20&rs=20&if=59
• random-numbers-bursting-in-table-prerendered-2.html?cs=20&rs=20&if=59
• random-numbers-bursting-in-table-prerendered-3.html?cs=20&rs=20&if=59

In the above demos, the idea is that each cell with a number becomes a scrollable container with 2 numbers actually — one above the other. In that case, because overflow: hidden is set, only one of the numbers is visible while the other is hidden — depending on the current scrolling:

With such a setup, it’s possible to update the invisible numbers during idle frames without the user noticing. Due to how WPE WebKit accelerates the scrolling, changing the invisible numbers, in practice, triggers the layout and rendering right away. Moreover, the actual rasterization to the buffer backing the scrollable container happens immediately (depending on the tiling settings), and hence the high cost of layout and text rasterization can be distributed. When the time comes, and all the numbers need to be updated, the scrollable containers can be just scrolled, which in that case turns out to be ~2 times faster than updating all the numbers in place.

To better understand the above effect, it’s recommended to compare the mark views from sysprof traces of the random-numbers-bursting-in-table.html?cs=10&rs=10&if=11 and random-numbers-bursting-in-table-prerendered-1.html?cs=10&rs=10&if=11 demos:

While the first sysprof trace shows very little processing during 11 idle frames and a big chunk of processing (21 ms) every 12th frame, the second sysprof trace shows how the distribution of load looks. In that case, the amount of work during 11 idle frames is much bigger (yet manageable), but at the same time, the formerly big chunk of processing every 12th frame is reduced almost 2 times (to 11 ms). Therefore, the overall frame rate in the application is much better.

Results

Despite the above improvement speaking for itself, it’s worth summarizing the improvement with the benchmarking results of the above demos obtained from the NXP i.MX8M Plus and presenting the application’s average frames per second (FPS):

Clearly, the positive impact of pre-rendering can be substantial depending on the conditions. In practice, when the rendered DOM fragment is more complex, the trick such as above can yield even better results. However, due to how tiling works, the effect can be minimized if the content to be pre-rendered spans multiple tiles. In that case, the browser may defer rasterization until the tiles are actually needed. Therefore, the above needs to be used with care and always with proper benchmarking.

Conclusions #

As demonstrated in the above sections, when it comes to pre-rendering the contents to distribute the web application workload over time, the web platform gives both the official APIs to do it, as well as unofficial means through purposeful misuse of APIs and exploitation of browser engine implementations. While this article hasn’t covered all the possibilities available, the above should serve as a good initial read with some easy-to-try solutions that may yield surprisingly good results. However, as some of the ideas mentioned above are very much browser-engine-specific, they should be used with extra care and with the limitations (lack of portability) in mind.

As the web platform constantly evolves, the pool of pre-rendering techniques and tricks should keep evolving as well. Also, as more and more web applications are used on embedded devices, more pressure should be put on the specification, which should yield more APIs targeting the low-end devices in the future. With that in mind, it’s recommended for the readers to stay up-to-date with the latest specification and perhaps even to get involved if some interesting use cases would be worth introducing new APIs.

Servo is a greenfield web browser engine that supports many platforms. Automated testing for the project requires building Servo for all of those platforms, plus several additional configurations, and running nearly two million tests including the entire Web Platform Tests. How do we do all of that in under half an hour, without a hyperscaler budget for compute and an entire team to keep it all running smoothly, and securely enough to run untrusted code from contributors?

We’ve answered these questions by building a CI runner orchestration system for GitHub Actions that we can run on our own servers, using ephemeral virtual machines for security and reproducibility. We also discuss how we implemented graceful fallback from self-hosted runners to GitHub-hosted runners, the lessons we learned in automating image rebuilds, and how we could port the system to other CI platforms like Forgejo Actions.

This is a transcript post for a talk I gave internally at Igalia.

Let's talk about how Servo can have fast CI for not a lot of money.

Servo is a greenfield web browser engine. And being a web browser engine, it has some pretty demanding requirements for its CI setup.

We have to build and run Servo for many platforms, including three desktop platforms and two mobile platforms.

We have to run many, many tests, the main bulk of which is the entire Web Platform Tests suite, which is almost 2 million subtests. We also have several smaller test suites as well, like the WebGPU tests and the DevTools tests and so on.

We have to build Servo in many different configurations for special needs. So we might build Servo with the oldest version of Rust that we still support, just to make sure that still works. We might build Servo as a library in the same way that it would be consumed by embedders. We have to lint the codebase. We have to build with optimizations for nightly and monthly releases. We have to build with other optimizations for benchmarking work, and so on.

And as you'll see throughout this talk, we do this on GitHub and GitHub Actions, but GitHub Actions is a very frustrating CI service, and it has many baffling limitations. And as time goes on, I think Servo being on GitHub and GitHub Actions will be more for the network effects we had early on than for any particular merits of these platforms.

On GitHub Actions, GitHub provides their own first-party runners.

And these runners are very useful for small workloads, as well as the logic that coordinates workloads. So this would include things like taking a tryjob request for "linux", and turning that into a run that just builds Servo for Linux. Or you might get a try job request for "full", and we'll turn that into a run that builds Servo for all of the platforms and runs all of the tests.

But for a project of our scale, these runners really fall apart once you get into any workload beyond that.

They have runners that are free of charge. They are very tiny, resource constrained, and it seems GitHub tries to cram as many of these runners onto each server as they possibly can.

They have runners that you can pay for, but they're very very expensive. And I believe this is because you not only pay a premium for like hyperscaler cloud compute rates, but you also pay a premium on top of that for the convenience of having these runners where you can essentially just flip a switch and get faster builds. So they really make you pay for this.

Not only that, but using GitHub hosted runners you can't really customize the image that runs on the runners besides being able to pull in containers, which is also kind of slow and not useful all the time. If there are tools and dependencies that you need that aren't in those images, you need to install them every single time you run a job or a workflow, which is such a huge waste of time, energy, and money, no matter whose money it is.

There are also some caching features on GitHub Actions, but they're honestly kind of a joke. The caching performs really poorly, and there's not a lot you can cache with them. So in general, they're not really suitable for doing things like incremental builds.

So we have all these slow builds, and they're getting slower, and we want to make them faster. So we considered several alternatives.

The first that comes to mind are these third-party runner providers. These are things like Namespace, Warpbuild, Buildjet. There's so many of them. The caveat with these is that they're almost always... almost as expensive per hour as GitHub's first-party runners. And I think this is because they try to focus on providing features like better caching, that allow you to accumulate less hours on their runners. And in doing so, they don't really have any incentive to also compete on the hourly rate.

There is one exception: there's a system called RunsOn. It's sort of an off-the-shelf thing that you can grab this software and operate it yourself, but you do have to use AWS. So it's very tightly coupled to AWS. And both of these alternatives, they often lack support for certain platforms on their runners. Many of them are missing Windows or missing macOS or missing both. And RunsOn is missing macOS support, and probably won't get macOS support for the foreseeable future.

We considered offloading some of the work that these free GitHub hosted runners do onto our own servers with, let's call them like, "proxy jobs". The idea would be that you'd still use free GitHub hosted runners, but you'd do the work remotely on another server. The trouble with this is that then you're still using these free GitHub hosted runners, which take up your quota of your maximum concurrent free runners.

And it's also tricky to do this without losing access to the broader ecosystem of prebuilt Actions. So these Actions are like steps that you can pull in that will let you do useful things like managing artifacts and installing dependencies and so on. But it's one thing to say, okay, my workload is this shell script, and I'm going to run it remotely now. That's easy enough to do, but it's a lot harder to say, okay, well, I've got this workflow that has a bunch of scripts and a bunch of external Actions made by other people, and I'm going to run all of this remotely. I'm going to make sure that all of these Actions are also compatible with being run remotely. That's a lot trickier. That said, you should probably avoid relying too heavily on this ecosystem anyway. It's a great way to get locked into the platform, and working with YAML honestly really sucks.

So we could set up an entire CI service. There are some CI services like Jenkins, Bamboo... honestly, most CI services nowadays have support for built-in container orchestration. So they can spin up containers for each runner, for each job, autonomously. And while they can do containers, none of them have really solved the problem of virtual machine orchestration out of the box. And this is a problem for us, because we want to use virtual machines for that security and peace of mind, which I'll explain in more detail a little bit later. And seeing as we lack the dedicated personnel to operate an entire CI service and have that be on the critical path — we didn't want to have someone on call for outages — this was probably not the best option for us at the moment.

What we decided to do was set up some self-hosted runners.

These solve most of our problems, primarily because we can throw as much hardware and resources at these runners as we want. We can define the contention ratios.

And better yet, we can also customize the images that the runners use. By being able to customize the runner images, we can move a lot of steps and a lot of work that used to be done on every single workflow run, and do it only once, or once per build. We only rebuild the runner images, say, once a week. So that significantly saves... it cuts down on the amount of work that we have to do.

This is not just installing tools and dependencies, but it's also enabling incremental builds quite powerfully, which we'll see a little bit later on.

How much faster is this system with the self-hosted runners? Well, it turns out, quite a lot faster.

We've got this typical workflow that you use when you're testing your commits, when you're making a pull request. And if you kick off a tryjob like this, several of the jobs that we've now offloaded onto self-hosted runners are now taking 70, 80, even 90% less time than they did on GitHub hosted runners, which is excellent.

And even for the web platform tests, we found more modest time savings with the capacity that we've allocated to them. But as a result, even though the savings are a bit more modest, what's worth highlighting here is that we were able to go from twenty runners — we had to parallelise this test suite across twenty runners before — and now we only need three.

Some things that make our system unique and that we're pretty proud of — and these things apply to all versions of our system, which we've been developing over the last 12 to 18 months.

The first is that we build on top of the native CI service of the Git forge. So in this case, it's GitHub and GitHub Actions. It could be Forgejo and Forgejo Actions in the future, and we're working on that already.

We also want to give users more or less a transparent user experience, the idea being that users should not notice any changes in their day-to-day work, besides the fact that their builds have gotten faster. And I think for the most part, we have achieved that.

Our system supports all of the platforms that GitHub Actions supports for their runners, including Linux, Windows, and macOS, and we could even support some of the other platforms that Forgejo Actions supports in the future, including BSD.

We have the ability to set up a job so that it can try to use self-hosted runners if they're available, but fall back to GitHub hosted runners if there's none available for whatever reason, like maybe they're all busy for the foreseeable future, or the servers are down or something like that. We have the ability to fall back to GitHub hosted runners, and this is something that was quite complicated to build, actually. And we have a whole section of this talk explaining how that works, because this is something that's not actually possible with the feature set that GitHub provides in their CI system.

It's secure enough for a large public project like Servo, where we don't necessarily know all of our contributors all that well personally. And this is in large part because we use virtual machines, instead of just containers, for each run and for each job. My understanding is that it is possible to build a system like this securely with containers, but in practice it's a lot harder to get that right as a security boundary than if you had the benefit of a hypervisor.

And of course it is all completely self-hosted, which makes it about as cheap as it gets, because your operating expenses are really just the costs of bare metal compute.

Now, how cheap is that? Well, in our deployment in Servo, we spend about 300 EUR per month on servers that do general-purpose runners, and these handle most of our workload.

If we were to compare that to if we were running on GitHub-hosted runners, there'd be almost an order of magnitude increase in costs, somewhere like 1400 EUR to over 2000 EUR per month if we were doing the same work on GitHub-hosted runners.

There are also significant increases if we went with third-party runner providers as well, although not quite as much.

But in truth, this is actually kind of an unfair comparison, because it assumes that we would need the same amount of hours between if we were running on self-hosted runners or if we were running on GitHub hosted runners. And something that you'll see throughout this talk is that this is very much not the case. We spend so many fewer hours running our jobs, because we have to do so much less work on them. So in reality, the gap between our expenses with these two approaches would actually be a lot wider.

There are three ways to use our CI runner system.

The first one is the way that GitHub designed self-hosted runners to be used. The way they intend you to use self-hosted runners is that when you define a job, you use this runs-on setting to declare what kind of runners your job should run on. And you can use labels for GitHub hosted runners, or you can define arbitrary labels for your own runners and select those instead. But you have to choose one or the other in advance. And if you do that, you can't have any kind of fallback, which was a bit of a drawback for us, especially early on. One benefit of this, though, is that you get the natural ability to queue up jobs. Because if, at the time of queuing up a new job, if there's no self-hosted runners available yet, the job will stay in a queued state until a self-hosted runner becomes available. And it's nice that you essentially get that for free. But you have no ability to have this kind of graceful fallback.

So we built some features to allow you to do graceful fallback. And how this works is that each of the servers that operates these runners has a web API as well. And you can hit that web API to check if there are runners available and reserve them. Reserving them is something you have to do if you're doing graceful fallback. But I'll explain that in more detail a bit later on. And in doing so, because you have a job that can check if there are runners available, you can now parameterize the runs-on setting and decide "I want to use a self-hosted runner, or a GitHub hosted runner". It's unclear if this is going to be possible on Forgejo Actions yet, so we may have to add that feature, but it's certainly possible on GitHub Actions.

Now, the downside of this is that you do lose the ability, that natural ability, to queue up jobs, and I'll explain why that is a bit later. But in short, we have a queue API that mitigates this problem, because you can hit the queue API, and it can have a full view of the runner capacity, and either tell you to wait, or forward your request once capacity becomes available.

Some things that you can do with our system, that you can't do with GitHub hosted runners. One of them is check out the repo significantly faster.

Something about GitHub Actions and how it works is that, if you run a job, you run a workflow in a repo, you don't actually get a checkout of the repo. You don't get a clone of the repo out of the box, unless you explicitly add a step that does a checkout. And this is fine for the most part, it works well enough for most users and most repos.

But the Servo repo has over 130,000 files across 6,000 directories, and that's just the tracked files and directories. And as a result, even if we use things like shallow clones, there's just kind of no getting around the fact that cloning this repo and checking it out is just slow. It's unavoidably slow.

And it's especially slow on Windows and macOS, where the filesystem performance is honestly often pretty poor compared to Linux. So we want to make our checkouts faster.

Well, what we can do is, we can actually move the process of cloning and checking out the repo from the build process, and move that into the image build process. So we only do it once, when we're building the runner images.

Then what we can do is go into the jobs that run on self-hosted runners, and switch out the stock checkout action with our own action. And our own action will just use the existing clone of the repo, it will fetch the commit that we actually need to build on, then switch to it, and check it out.

And as a result, we can check out the Servo repo pretty reliably in a couple of seconds, instead of having to check out the entire repo from scratch.

Something that flows on from that, though, is that if we are baking a copy of the Servo repo into our runner images, well, what if we just bake a copy of the built artifacts as well? Like, why don't we just build Servo, and bake that into the image?

And this will allow us to do incremental builds, because Servo is a Rust project, and we use Cargo, and Cargo supports incremental builds. As a result, by doing this, when you run a job on our CI system, most of the time we only have to rebuild a handful of crates that have changed, and not have to rebuild all of Servo from scratch.

Now, this is not perfect. Sometimes we'll have some compilation units that get falsely rebuilt, but this works well enough, for the most part, that it's actually a significant time savings.

I also probably wouldn't trust this for building artifacts that you actually want to release in a finished product, just because of the kinds of bugs that we've seen in Cargo's incremental build support.

But for everything else, just your typical builds where you do like commit checks and such, this is very very helpful.

Servo has this system deployed, and it's had it deployed for at least the last year or so.

Nowadays we have three servers which are modestly large, and we use these for the vast majority of our workload. We also have two smaller servers that we use for specific benchmarking tasks.

The stack on these servers, if you could call it that, is largely things that I personally was very familiar with, because I built this. So we've got NixOS for config management, the hypervisor is libvirt and Linux KVM, and the storage is backed by ZFS. The process of actually building the images, and managing the lifecycle of the virtual machines though, is done with some custom orchestration tooling that we've written.

This tooling consists of two services. The monitor service, which runs on every server that operates self-hosted runners, and the queue service, which is single and global.

So the monitor service does the vast majority of the work. It rebuilds virtual machine images, these templates. It clones the templates into virtual machines for each runner, for each job. It registers the runners with the CI service using its API, so that it can receive jobs. And it can also label the runners to tie them to specific jobs when asked to reserve them. This is optional, but it is a required step if we're doing graceful fallback.

Now, with graceful fallback, you do lose the ability to naturally queue up jobs. So what we've put on top of that is a single queue service that sits in front of the cluster, and it essentially acts as a reverse proxy. It's quite thin and simple, and there's a single one of them, not one per server, for the same reason as the general principle of, like, when you go to a supermarket, it's more efficient to have a single large queue, a single long queue of customers that gets dispatched to a bunch of shopkeepers. That's more efficient than having a sort of per-shopkeeper queue, especially when it comes to, like, moving jobs dynamically in response to the availability.

So we've got a whole section here about how graceful fallback works. I might cut this from shorter versions of the talk, but let's jump into it.

On GitHub Actions, every job has this runs-on setting, that defines what kind of runner it needs to get assigned to. It has to define this in advance, before the job runs.

And annoyingly, once you choose, "I want to run on a GitHub hosted runner" or "I want to run on a self-hosted runner", once you decide that, there's no turning back. Now, if you've decided that my job needs to run on a self-hosted runner, it can only run on a self-hosted runner. And now the outcome of your job, and the outcome of your workflow, now depends on that job actually getting a self-hosted runner with the matching criteria and succeeding. There's no way for that to become optional anymore.

And you might say, okay, well, maybe I can work around this by spinning up my self-hosted runners, but just giving them the same labels that GitHub assigns to their runners, maybe it'll do something smart? Like maybe... it will run my self-hosted runners if they're available and fall back. But no, the system has no sense of priority and you can't even define any sense of priority of like, I want to try this kind of runner and fall back to another kind of runner. This is simply not possible with the GitHub Action system.

So it may seem like it's impossible to do graceful fallback, but we found a way.

[self-hosted label] | no => [GitHub-hosted label] 2. $ echo " /> [self-hosted label] | no => [GitHub-hosted label] 2. $ echo " /> [self-hosted label] | no => [GitHub-hosted label] 2. $ echo "label=${label}" | tee -a $GITHUB_OUTPUT - Use the step output in runs-on runs-on: ${{ needs.decision.outputs.label }} - But two decisions must not be made concurrently">

What we can do is add a decision job, for each workload job that may need to run on self-hosted runners. And we prepend this job, and its job is to choose a runner label.

So how it essentially works is: it checks if a runner is available, and based on that, either chooses a self-hosted label or a GitHub hosted label. And it chooses it and sets it in an output. And this output can get pulled in... in our workload job, the job that actually does our work, because now you can parameterize the runs-on setting, so that it takes the value from this previous decision job.

Unfortunately, it seems like this may not be possible in Forgejo Actions just yet, so we might have to develop support for that, but it's certainly possible in GitHub Actions today, and it has been for quite a while.

The one caveat with this approach is that you need to be careful to only do this decision process one at a time. You should not do two instances of this process concurrently and interleave them.

- runs-on: ${{ needs.decision.outputs.label }} runs-on: 6826776b-5c18-4ef5-8129-4644a698ae59 - Initially do this directly in the decision job (servo#33081)" /> - runs-on: ${{ needs.decision.outputs.label }} runs-on: 6826776b-5c18-4ef5-8129-4644a698ae59 - Initially do this directly in the decision job (servo#33081)" /> - runs-on: ${{ needs.decision.outputs.label }} runs-on: 6826776b-5c18-4ef5-8129-4644a698ae59 - Initially do this directly in the decision job (servo#33081)">

The reason for this is something you'll see if you think about a lot of the answers that you get on Stack Overflow and the GitHub forums. If you look up, "how do I solve this problem? How do I define a job where I can fall back from self-hosted runners to GitHub hosted runners?"

And most of the answers there, they'll have a problem where they check if there are any runners that are available, and then, they will make the decision, committing to either a self-hosted runner or a GitHub hosted runner. The trouble is that in between, if another decision job comes in and tries to make the same kind of decision, they can end up "earmarking" the same runner for two jobs. But each runner is only meant to run one job, and it can only run one job, so one of the jobs will get left without a runner.

So we can start to fix this by actually reserving the runners when we're doing graceful fallback. And how we've done it so far, is that we've used the GitHub Actions API to label the runner when we want to reserve it, and we label it with a unique ID. Then the workload job can put that unique ID, that label, in its runs-on setting. Instead of a general runner label, it can tie itself to this specific, uniquely identified runner label.

And we did it this way initially, because it allowed us to do it inside the decision job, at first. I think in the future, we will have to move away from this, because on Forgejo Actions, the way runner labels work is quite different. They're not something that you can sort of update after the fact. In fact, they're actually kind of defined by the runner process. So this approach for reserving runners won't work on Forgejo Actions. We'll probably have to do that internally on the runner servers. But in the meantime, we use labeling. Yeah, so at first we did this inside the decision job.

There are some problems with this. One of them is that now the decision job has to have a GitHub token that has permissions to manage runners and update their labels. And this is something that we'd like to avoid if possible, because we'd like our jobs to have least privilege that they need to do their work.

And something I didn't mention is that reserving runners in this way doesn't actually solve the problem on its own, because you've now transformed the problem to being, okay, we're going to check if the runner is not yet reserved. We're going to check if there's an unreserved runner, and then, we're going to label the runner. But in between, there's a moment where another process doing the same thing could make the same decision. And as a result, if we just did this, we could end up with a situation where one runner gets assigned two unique labels, but it can only fulfill one of them. So we have that same problem again.

So you might say, okay, well, it looks like GitHub Actions has this neat job concurrency feature. I mean, they say you can use this to define a job in a way where only one of them will run at a time, and you can't run them concurrently, so let's try using that to solve this problem.

What you'll find, though, is that if you try to solve the problem with job concurrency, as soon as there's even the slightest bit of contention, you'll just have your runs starting to fail spuriously, and you'll have to keep rerunning your jobs, and it'll be so much more annoying.

And the reason for this is that, if you look more closely at the docs, job concurrency essentially has a queue limited to one job. So at a maximum, you can have one job, one instance of the job that's running, one instance of the job that's queued, and then if another instance of the job comes in while you have one running and one queued, then those extra jobs just get cancelled, and the build fails. So unfortunately, job concurrency does not solve this problem.

So to solve these problems, what we did is we moved that decision and runner reservation process out of the decision jobs, and into the servers that operate the runners themselves. And we do this with an API that runs on the servers.

One smaller problem you might notice though, is that there's still a possibility that you could reserve a runner, but then after you've reserved the runner, it might fail to actually run. And this has become a lot less likely in our system, in our experience nowadays, now that we've ironed out most of the bugs, but it can still happen from time to time, usually due to infrastructure or network connectivity failures.

And we wanted to solve this by setting a time limit on how long a job can be queued for, because if it can't actually get a runner in practice, it will get stuck in that queued state indefinitely. But unfortunately, while we can set a time limit on how long a job can run for once it's been assigned, we can't actually set a time limit on how long a job can be queued for.

So we have to rely on the default built-in limit for all jobs, where I think there's like a limit of a day or two? So like, 24 or 48 hours or so, for how long a job can be queued? And this is just a really long time. So as a result, whenever this happens, you essentially have to cancel the job manually, or if you don't have permission to do that, you have to go ask someone who has permission, to go and cancel your job for you, which is really annoying.

So we solved that using timeout jobs. A timeout job is a sibling to your workload job, and it acts like a watchdog, and it ensures that it actually got a runner when you expect to get a self-hosted runner.

And how that works is, we wait a short amount of time, just like a minute or two, which should be long enough for the runner to actually start running the job, and then we query the API of the CI service to check if the workload job actually started running, or if it's still in that queued state. If it's still queued after two minutes, we cancel the run.

Unfortunately, we can't just cancel the job run. We do have to cancel the whole workflow run, which is quite annoying. But, you know, it's GitHub, nothing is surprising anymore.

Thankfully, we only have to run this when we actually get a self-hosted runner, so we can make it conditional. But over time, in Servo's deployment, we have actually stopped using these, to free up some of those GitHub-hosted runner resources.

One challenge that comes in making these timeout jobs is identifying the workload jobs uniquely, so we can look up whether it's still queued or whether it's started running. There are unique IDs for each job run. These are just like an incrementing number, and you'd think we'd be able to use this number to look up the workload job uniquely and robustly.

Unfortunately, you can't know the run ID of the job [correction 2025-12-24] until it starts, and it may not ever start... or at least you may not know it until the workflow runs, and there can be many instances of the job in the workflow because of workflow calls. Workflow calls are a feature that essentially allows you to inline the contents of a workflow in another as many times as you like. And as a result, you can have multiple copies, multiple instances of a job that run independently within one workflow run. So we definitely need a way to uniquely look up our instance of the workload job.

The trouble is that the only job relationship you can do in GitHub Actions is a needs relationship, and that's inappropriate for our situation here, because we can say that the workload job needs the decision job, we can say the timeout job needs the decision job — and in fact we do both of these, we "need" to do both of these — but we can't say that the timeout job needs the workload job, because of how needs works.

How needs works is that if job A needs job B, then job B has to actually get assigned a runner, and run, and complete its run — it has to finish — before job A can even start. And in this situation, we're making a timeout job to catch situations where the workload job never ends up running, so if we expressed a needs relationship between them, then the timeout job would never run, in these cases at least.

And even if we could express a needs relationship between jobs, like maybe we could walk the job tree, and go from the timeout job, through the decision job via the needs relationship, and then walk back down to the workload job using the same kind of needs relationship... unfortunately, none of these needs relationships are actually exposed in the API for a running workflow. So like, they're used for scheduling, but when you actually go and query the API, you can't tell what job needs what other job. They're just three jobs, and they're unrelated to one another.

in the name: name: Linux [${{ needs.decision.outputs.unique-id }}] name: Linux [6826776b-5c18-4ef5-8129-4644a698ae59] - Query the CI service API for all jobs in the workflow run - Check the status of the job whose name contains [${{ needs.decision.outputs.unique-id }}] - Yes, really, we have to string-match the name :)))" /> in the name: name: Linux [${{ needs.decision.outputs.unique-id }}] name: Linux [6826776b-5c18-4ef5-8129-4644a698ae59] - Query the CI service API for all jobs in the workflow run - Check the status of the job whose name contains [${{ needs.decision.outputs.unique-id }}] - Yes, really, we have to string-match the name :)))" /> in the name: name: Linux [${{ needs.decision.outputs.unique-id }}] name: Linux [6826776b-5c18-4ef5-8129-4644a698ae59] - Query the CI service API for all jobs in the workflow run - Check the status of the job whose name contains [${{ needs.decision.outputs.unique-id }}] - Yes, really, we have to string-match the name :)))">

So how we had to end up solving this is, we had to tie these jobs together by generating a unique ID, a UUID, and putting it in the friendly name, like the display name of the job, like this.

And to query the CI service to find out if that job is still queued, we need to query it for the whole workflow run, and just look at all of the jobs, and then find the job whose name contains that unique ID.

Then we can check the status, and see if it's still queued. This is really, really silly. Yes, we really do have to string match the name, which is bananas! But this is GitHub Actions, so this is what we have to do.

One thing I didn't mention is that being able to reserve runners needs to be kind of a privileged operation, because we don't just want an arbitrary client on the internet to be able to erroneously or maliciously reserve runners. Even if they may not be able to do a whole lot with those runners, they can still deny service.

So to use the monitor API to do graceful fallback and to request and reserve a runner, normally we would require knowledge of some kind of shared secret, like an API token, and that's what we've done for most of the life of this system.

The trouble with this is that there are many kinds of workflow runs that don't have access to the secrets defined in the repo. A big one is pull_request runs. Most of the time, pull_request runs don't have access to secrets defined in the repo. And there is another kind of run called a pull_request_target run, and those do have access to secrets, but they also have some pretty gnarly security implications that mean that in general, you wanna avoid using these for pull requests anyway.

So if you're stuck with pull_request runs for your pull requests, does that mean that you can't use self-hosted runners? How do we allow pull_request runs to request and reserve runners in a way that it can prove that its request is genuine and authorized?

What we do is we use artifacts. We upload a small artifact that encodes the details of the request and publish the artifact against the run. So in the artifact, we'd say, "I want two Ubuntu runners" or "one Windows runner" or something like that.

And then we would hit the monitor API, we hit a different endpoint that just says, go to this repo, go to this run ID, and then check the artifacts! You'll see my request there! And this does not require an API token, it's not a privileged operation. What's privileged is publishing the artifact, and that's unforgeable; the only entity who can publish the artifact is the workflow itself.

All we have to do then, all we have to be careful to do, is to delete the artifact after we reserve the runner, so that it can't be replayed by a malicious client. And in the event that deleting the artifact fails, we can also set the minimum auto-delete period of, I think at the moment it's 24 hours, just in case that fails.

Graceful fallback normally means that you lose the ability to naturally queue jobs for self-hosted runners. And this happens because when you hit the API, requesting, you know, are there any available runners, please reserve one for me. If there aren't any available at the time of the request, we will immediately fall back to running on a GitHub hosted runner.

But the thing is, is that our self-hosted runners are generally so much faster! A GitHub hosted run might take five times as long as the equivalent self-hosted run, and if that was the case, it would actually be beneficial to wait up to 80% of that time, and we'd still probably save time.

So to increase the utilization of our self-hosted runners, what we can do is run a small queue service that sits in front of the monitors, and it essentially acts as a reverse proxy. It will take the same kind of requests for reserving runners as before, but it will have a global view of the availability and the runner capacity at any given moment.

And based on that, it will either respond to the client saying, go away and please try again later, or it will forward the request onto one of the monitors based on the available capacity.

We also have some lessons here about how to automate building virtual machine images, essentially. And these are lessons that we've learned over the last year or two.

GitHub Actions uses Packer for their first-party runner images, so they use Packer to build those images. And our monitor service also automates building the images, but we don't use Packer.

Initially, we had a handful of scripts that we just kicked off manually whenever we needed to update our images, but we've now fully automated the process. And we've done this using some modules in our monitor service, so there's some high-level Rust code that drives these image rebuilds, and it even uses reflink copies to take advantage of copy-on-write time savings and space savings with ZFS.

Now, one of the complications we ran into when building images, over the past year or so, is that we used to pull a sort of clever trick where, we would do as much of the process of configuring the virtual machine image on the host, actually, as possible, rather than doing configuration inside the guest, and having to spin up the guest so that we can configure it. And we were able to do this by essentially mounting the root file system of the guest image, on the host, and then injecting files, like injecting tools and scripts and other things that are needed by the image and needed to configure the image.

But we stopped being able to do this eventually because, well, essentially because of macOS. macOS has no file system that's usable for building Servo that can also be mounted on Linux with write support. Because like, just think about them, right? We've got HFS+, which Linux can write to but only if there's no journaling and you can't install macOS on HFS+ without journaling. There's APFS, which Linux has no support for. There's exFAT, which has no support for symlinks, so a lot of tools like uv will break. There's NTFS, which we thought would be our savior, but when we tried to use it, we ran into all sorts of weird build failures, which we believe are due to some kind of filesystem syncing race condition or something like that, so NTFS was unusable as well.

In the end, what we had to do is, if a guest image needed any tools or dependencies, it had to download them from a web server on the host.

We eventually settled on a consistent approach to automating the process of installing the OSes and configuring them for each runner template.

And the approach that we used was to install the OS using whatever method is easiest, and to bootstrap the config using any native config management system, if there is one included with the operating system.

But once we've kicked off that process, we then use the native config management as little as possible. And we do this because a lot of the config management tools that are built into these operating systems are quite quirky and annoying, and they are built for needs that we don't have, the primary need being that they can manage the configuration of a system over time, keeping it up to date with any changes. The thing about these runner images, though, is that each runner image only needs to get built once, it only needs to get configured once, and then after after that it'll be cloned for each runner, and then it will only run once, and this kind of sort of "one shot" use case is something that's kind of overkill with config management systems.

We can just do it with the usual scripting and automation facilities of the operating system. How does that look like in practice?

Well for Linux, we install the OS using pre-built Ubuntu cloud images, that we just download from the mirrors.

We bootstrap the config using cloud-init, which is such a painful... it's so painful to use cloud-init. We use it because it's included with the operating system, so that means it's the fastest possible way to get started.

We use it as little as possible: we just configure the logs to go to a TTY, we configure the network, so we can connect to the network, and then once we're on the network, we just curl and run a bash script which does the rest.

The same goes for Windows.

We install the operating system using an automated answers file called autounattend.xml. There's a nice little generator here which you can use if you don't want to have to set up a whole Windows server to set up unattended installations. You generate that XML file, and you can also use that XML file to bootstrap the config.

Again we use it as little as possible, because writing automations as XML elements is kind of a pain. So we essentially just set up a scheduled task to run the next stage of the config, we install the network driver, we import some registry settings, and we reboot. That's it. The rest of it is done with a PowerShell script.

The same goes for macOS.

Now, unfortunately, installing the OS and bootstrapping the config does have to be done by hand. And this is because if you want to automate a macOS installation, your two options, more or less, are enterprise device management solutions, which cost a lot of money and mean that you have to have a macOS server around to control and orchestrate the servers. But if you don't want to use one of those enterprise solutions, what most open systems that are faced with this problem end up doing is to throw OpenCV at the problem. I've seen several projects use OpenCV to OCR the setup wizard, which is... it's certainly a bold strategy. It's not really for me.

What I decided to do instead is just install the OS by hand, and pipe curl into sh to kick off the config management process. And this is something that we only really have to do once, because we do it once, and then we take a snapshot of it, and then we never have to do it again, at least until the next version of macOS comes out.

So this bootstrap script just does a handful of minimal things: it enables automatic login, it sets up a LaunchAgent to ensure that we can run our own code on each boot, and then it does a handful of other things which it honestly doesn't really have to do in this script. We could probably do these things in the zsh script which we then curl and run. And that's where the remainder of the work is done.

So looking forward, some things that we'd like to do with this system.

The first is to decouple it from Servo. So we built this CI system quite organically over the past 12 to 18 months, and we built it around Servo's needs specifically, but we think this system could be more broadly useful for other projects. We'll just have to abstract away some of the Servo specific bits, so that it can be more easily used on other projects, and that's something we're looking into now.

Something else that we'll have to do sooner or later is add support for macOS runners on Apple Silicon, that is ARM64, and the reason we have to do this is that macOS 26, which is the most recent version of macOS that came out in September, that's just a couple months ago, that is the last version of macOS that will support x86 CPUs. And at the moment, our macOS runners run on x86 CPUs, on the host and in the guest.

This is a little bit complicated because at the moment, our macOS runners actually run on Linux hosts, using a Linux-KVM-based sort of Hackintosh-y kind of solution. And there is no counterpart for this for arm64 hosts and guests, and I'm not sure there ever will be one. So we're going to have to port the system so that it can run on macOS hosts, so we can use actual Mac hardware for this, which is easy enough to do, and that's in progress.

But we're also going to have to port the system so it can run with other hypervisors. And this is because, although libvirt supports macOS hosts, the support for the macOS Hypervisor framework and Virtualization framework is not mature enough to actually run macOS guests in libvirt. And I'm not sure how long that will take to develop, so in the meantime, we've been looking at porting the system so that when you're on a Mac, you can run with UTM instead, and that's been working fairly well so far.

We're also looking at porting the system so that it can run with Forgejo Actions and not just GitHub Actions. So Forgejo Actions is an open alternative to GitHub Actions that tries to be loosely compatible with GitHub Actions, and in our experience, from experimentation so far, we found that it mostly is loosely compatible. We think we'll only have to make some fairly minor changes to our system to make it work on both CI systems.

That said, this CI system could potentially be more broadly applicable to other CI services as well, because virtual machine orchestration is something that we haven't really seen any CI services have a great built-in solution for. So if this interests you and you want to use it on your project on some other CI service, then we'd appreciate knowing about that, because that could be something we would explore next.

The remaining ideas are things that I think we could look into to make our runners more efficient.

The big one is autoscaling. So at the moment when you set up a server to operate some self-hosted runners, you essentially have to statically pre-configure how many runners of each kind of runner you want to be kept operating. And this has worked well enough for us for the most part, but it does mean that there's some kind of wasted resources sometimes, when the moment-to-moment needs of the jobs that are queued up aren't well fitted to the composition of your runner configuration. So if we had the ability to dynamically respond to demand, or some kind of autoscaling, I think we could improve our runner utilization rates a little bit, and sort of get more out of the same amount of runner capacity, the same amount of server capacity.

There's a couple ideas here, also, about reducing boot times for the runners, which can be quite helpful if you have a big backlog of jobs queued up for these servers, and this is because time spent booting up each runner, each virtual machine, is time that cannot be spent doing real work.

So two ways we can think of to reduce these boot times are, to have hot spares ready to go, the idea being that, if you can spin up more runners than you actually intend to run concurrently, and just have them sitting around, then you can kind of amortize the boot times, and sort of get the boot process process out of the way. And the way you do this is by spinning up a whole bunch of runners, say maybe you spin up like twenty runners, even though you only intend to run four of them concurrently.

And what you do is you give these runners a token amount of RAM to start with. You give them like one gig of RAM instead of 16 or 32 gigs of RAM. And then when a job comes in, and you actually want to assign the runner out so that it can do the work, then you dynamically increase the RAM from one gig, or that token amount, to the actual amount, like 16 gigs or 32 gigs. And this should be fairly easy to do in practice. This is actually supported in libvirt using a feature known as memory ballooning. But there are some minor caveats, like you do lose the ability to do certain optimizations, like you can't do huge pages backing on the memory anymore. But for the most part, this should be fairly technically simple to implement.

Something that could be more interesting in the longer term is microVMs, things like Firecracker, which as I understand it, these microVMs can sort of take the concept of paravirtualization to its logical extreme. And what it means is that on kernels that support being run as microVMs, you can boot them in like one or two seconds, instead of 20 or 30 or 60 seconds. And this could save a great deal of time, at least for jobs that run on Linux and BSD. I don't know if I said Linux and macOS, but I meant Linux and BSD.

So yea, we now have a system that we use to speed up our builds in Servo's CI, and it works fairly well for us. And we think that it's potentially useful for other projects as well.

So if you're interested to find out more, or you're interested to find out how you can use the system in your own projects, go to our GitHub repo at servo/ci-runners, or you can go here for a link to the slides. Thanks! article > section:not(#specificity) { position: relative; display: flex; flex-flow: column nowrap; padding: 1em 0; } article > section:not(#specificity) > * { flex: 0 0 auto; } article > section:not(#specificity) > ._slide { max-height: 50vh; flex: 0 0 auto; position: sticky; top: 0; text-align: center; } article > section:not(#specificity) > ._slide > img { max-width: 100%; max-height: 50vh; } article > section:not(#specificity) > ._spacer { flex: 1 0 1em; }

Update on what happened in WebKit in the week from December 8 to December 15.

In this end-of-year special have a new GMallocString helper that makes management of malloc-based strings more efficient, development releases, and a handful of advancements on JSC's implementation of Temporal, in particular the PlainYearMonth class.

Cross-Port 🐱

JavaScriptCore 🐟

Releases 📦️

As I write often, maintaining a downstream of Chromium is not easy. A lot of effort falls on the shoulders of the teams embedding Chromium, or creating products on top of the upstream Chromium project.

We covered this in the previous chapters of my series of blog posts about maintaining Chromium downstreams. Now, this post is going to be a bit different.

I start with a question:

What can upstream Chromium do to help downstreams?

This very same question was discussed in the Web Engines Hackfest breakout session that originated most of these posts. In this blog post, I will share some of the most interesting answers that came up in that session.

Better componentization #

One of the ideas was to move code around more aggressively to make it easier to reuse. Specifically, refactoring to move more and more code from //chrome to //components.

Chromium has gone a long way in that direction. Each of these changes allows downstreams to directly use only the components they need, instead of working on top of //chrome. But there is still room for improvement.

Some parts of //chrome are still not refactored and could be very useful, especially for downstreams shipping a browser. Some examples:

• Tabs implementation.
• Profiles.
• Synchronization.

Improve extensibility #

In the same direction, supporting easier ways to provide alternative implementations, and add custom software components, was considered important.

Some examples:

• Making it easier to support Chrome extensions without using //chrome, would allow implementing new browsers without bundling the Chromium UI.
• Going further in the direction of what has been done with Ozone: the Chromium platform abstraction layer that helps to implement the support for a variety of OS (including Linux and X11). Similar steps could be taken at other levels to improve OS integration (system hardware encryption, accelerated video codecs, system IPC, and so on).

Downstream advocacy #

A very interesting proposal was to create the role of downstream advocates in the Chrome community.

They would act as an entry point for downstream projects wanting to interact with the Chrome community and be an official communication channel for downstreams to report their needs.

This would also increase awareness of the different ways Chromium is used by downstreams.

Today there are two channels that are somewhat similar: the Chromium Embedders mailing list and the #embedders Slack channel.

A two-way problem #

So far, three different problems raised by downstreams have been covered, and they seem like fair requests to the Chromium community.

But there is also work to do on the downstreams side.

Can downstreams contribute more of their work to upstream? Not only in code, but also in all the maintenance activities.

There is also code written for very specific downstream needs that could land upstream, as long as it does not become a burden to the common project. That means ownership and enough work bandwidth need to be in place.

Where are we now? #

There is a major change in the Chromium community: the creation of the Supporters of Chromium Based Browsers. What does it mean for embedders? Could it be a good way to channel requirements from the different downstream projects?

Two years after the Web Engines Hackfest session, we can see some improvements. But the general question is still valid:

What can upstream Chromium do to help downstreams?

The last post #

The next post in this series will be the last one. It will cover some typical problems downstream projects are facing today.

References #

• Web Engines Hackfest 2023 - Maintaining Chromium downstream breakout session.
• Supporters of Chromium-Based Browsers.

At the time of writing, GCC beats Clang on several SPEC CPU 2017 benchmarks on RISC-V¹:

LLVM developers upstream have been working hard on the performance of generated code, in every part of the pipeline from the frontend all the way through to the backend. So when we first saw these results we were naturally a bit surprised. But as it turns out, the GCC developers have been hard at work too.

Sometimes a bit of healthy competition isn’t a bad thing, so this blog post is the first in a series looking at the work going on upstream to improve performance and catch up to GCC.

Please note that this series focuses on RISC-V. Other targets may have more competitive performance but we haven’t measured them yet. We’ll specifically be focusing on the high-performance application processor use case for RISC-V, e.g. compiling for a profile like RVA23. Unfortunately since we don’t have access to RVA23 compatible hardware just yet we’ll be benchmarking on a SpacemiT-X60 powered Banana Pi BPI-F3 with -march=rva22u64_v. We don’t want to use -mcpu=spacemit-x60 since we want to emulate a portable configuration that an OS distribution might compile packages with. And we want to include the vector extension, as we’ll see in later blog posts that optimization like auto-vectorization can have a major impact on performance.

Where to start?

It goes without saying that a vague task like “make LLVM faster” is easier said than done. The first thing is to find something to make fast, and while you could read through the couple dozen million lines of code in LLVM until inspiration strikes, it’s generally easier to start the other way around by analyzing the code it generates.

Sometimes you’ll get lucky by just stumbling across something that could be made faster when hacking or poring through generated assembly. But there’s an endless amount of optimizations to be implemented and not all of them are equally impactful. If we really want to make large strides in performance we need to take a step back and triage what’s actually worth spending time on.

LNT, LLVM’s nightly testing infrastructure, is a great tool for this task. It’s both a web server that allows you to analyze benchmark results, and a command line tool to help run the benchmarks and submit the results to said web server.

As the name might imply, it’s normally used for detecting performance regressions by running benchmarks daily with the latest revision of Clang, flagging any tests that may have become slower or faster since the last revision.

But it also allows you to compare benchmark results across arbitrary configurations. You can run experiments to see what effects a flag has, or see the difference in performance on two pieces of hardware.

Moreover, you can pass in different compilers. In our case, we can do two “runs” with Clang and GCC. Here’s how we would kick these off:

for CC in clang riscv64-linux-gnu-gcc
do
  lnt runtest test-suite bpi-f3-rva22u64_v-ReleaseLTO \
    --sandbox /var/lib/lnt/ \
    --test-suite=path/to/llvm-test-suite \
    -DTEST_SUITE_SPEC2017_ROOT=path/to/cpu2017 \
    --cc=$CC \
    --cflags="-O3 -flto -march=rva22u64_v" \
    --cxxflags="-O3 -flto -march=rva22u64_v" \
    --benchmarking-only \
    --build-threads=16 \
    # cross-compile and run on another machine over ssh
    --toolchain=rva22u64_v.cmake \	
    --remote-host=bpi-f3 \
    # fight noise, run each benchmark 3 times on the same core
    --exec-multisample=3 \
    --run-under="taskset -c 5" \
    # submit the results to a web server for easy viewing
    --submit=https://mylntserver.com/submitRun
done

This command does a lot of heavy lifting. First off it invokes CMake to configure a new build of llvm-test-suite and SPEC CPU 2017 with -O3 -flto -march=rva22u64_v. But because compiling the benchmarks on the Banana Pi BPI-F3 would be painfully slow, we’ve specified a CMake toolchain file to cross-compile to riscv64-linux-gnu from an x86-64 build machine. Here’s what the toolchain file looks like:

# rva22u64_v.cmake
set(CMAKE_SYSTEM_NAME Linux)
set(CMAKE_C_COMPILER_TARGET riscv64-linux-gnu)
set(CMAKE_CXX_COMPILER_TARGET riscv64-linux-gnu)
set(ARCH riscv64)

If you’ve got your cross toolchain sysroots set up in the right place in /usr/riscv64-linux-gnu/, things should “just work” and CMake will magically build RISC-V binaries. On Debian distros you can install the *-cross packages for this:

$ apt install libc6-dev-riscv64-cross libgcc-14-dev-riscv64-cross
libstdc++-12-dev-riscv64-cross

(You could also use mmdebstrap, or see Alex Bradbury’s guide to this)

After the benchmarks are built it rsyncs over the binaries to the remote machine, and then sshes into it to begin running the benchmarks. It will expect the sandbox path where the binaries are built on the remote to also exist on the host, so something like /var/lib/lnt should work across both. The BPI-F3 can also produce some noisy results, so the --exec-multisample=3 and -run-under="taskset -c 5" tell it to run the benchmarks multiple times and pin them to the same core.

Finally it generates a report.json file and submits it to the web server of choice. Navigate to the web interface and you’ll be shown two “machines”, LNT’s parlance for a specific combination of hardware, compiler and flags. You should see something like: bpif3-rva22u64_v-ReleaseLTO__clang_DEV__riscv64 and bpif3-rva22u64_v-ReleaseLTO__gcc_DEV__riscv64. Clicking into one of these machines will allow you to compare it against the other.

Profiling

Once on the LNT web interface you’ll be presented with a list of benchmarks with a lot of red percentages beside them. We now know what is slower, but next we need to know why they’re slower. We need to profile these benchmarks to see where all the cycles are spent and to figure out what Clang is doing differently from GCC.

LNT makes this easy, all you need to do is add --use-perf=profile to the lnt runtest invocation and it will perform an additional run of each benchmark wrapped in perf record. Profiling impacts run time so it runs it separately to avoid interfering with the final results. If you want to override the default events that are sampled you can specify them with --perf-events=cycles:u,instructions:u,....

LNT will take care of copying back the collected profiles to the host machine and encoding them in the report, and in the web interface you’ll notice a “Profile” button beside the benchmark. Click on that and you’ll be brought to a side by side comparison of the profiles from the two machines:

From here you can dive in and see where the benchmark spends most of its time. Select a function from the dropdown and choose one with a particularly high percentage: This is how much it makes up overall of whatever counter is active in the top right, like cycles or instructions. Then do the same for the other run and you’ll be presented with the disassemblies side-by-side below. Most importantly, information about the counters is displayed inline with each instruction, much like the output of perf annotate.

You might find the per-instruction counter cycle information to be a bit too fine-grained, so personally I like to use the “Control-Flow Graph” view mode in the top left. This groups the instructions into blocks and lets you see which blocks are the hottest. It also shows the edges between branches and their destinations which makes identifying loops a lot easier.

A real example

Lets take a look at how we can use LNT’s web interface to identify something that GCC does but Clang doesn’t (but should). Going back to the list of SPEC benchmark results we can see 508.namd_r is about 17% slower, so hopefully we should find something to optimize in there.

Jumping into the profile we can see there’s a bunch of functions that all contribute a similar amount to the runtime. We’ll just pick the hottest one at 14.3%, ComputeNonbondedUtil::calc_pair_energy_fullelect(nonbonded*). It’s a pretty big function, but in GCC’s profile 71% of the dynamic instruction count comes from this single, albiet large block.

Looking at Clang’s profile on the opposite side we see a similar block that accounts for 85% of the function’s instruction count. This slightly higher proportion is a small hint that the block that Clang’s producing is sub-optimal. If we take the hint and stare at it for long enough, one thing starts to stand out is that Clang generates a handful of fneg.d instructions which GCC doesn’t:

	fneg.d  fa0, fa0
	fneg.d  ft0, ft0
	fneg.d  ft2, ft2
	fmul.d  fa3, ft5, fa3
	fmul.d  fa0, fa3, fa0
	fmul.d  ft0, fa3, ft0
	fmul.d  fa3, fa3, ft2
	fmadd.d fa2, fa4, fa2, fa0
	fmadd.d ft6, fa4, ft6, ft0
	fmadd.d fa4, fa4, ft1, fa3

fneg.d rd, rs1 negates a double and fmul.d multiplies two doubles. fmadd.d rd, rs1, rs2, rs3 computes (rs1*rs2)+rs3, so here we’re doing some calculation like (a*b)+(c*-d).

These fneg.ds and fmadd.ds are missing on GCC. Instead it emits fmsub.d, which is entirely absent from the Clang code:

	fmul.d fa1,fa4,fa1
	fmul.d ft10,fa4,fa5
	fmsub.d ft10,ft7,fa0,ft10
	fmsub.d fa5,ft7,fa5,fa1
	fmul.d fa1,fa4,fa1
	fmsub.d fa1,ft7,fa0,fa1

fmsub.d rd, rs1, rs2, rs3 computes (rs1*rs2)-rs3, so GCC is instead doing something like (a*b)-(c*d) and in doing so avoids the need for the fneg.d. This sounds like a missed optimization in LLVM, so lets take a look at fixing it.

Writing the (right) fix

The LLVM RISC-V scalar backend is pretty mature at this stage so it’s surprising that we aren’t able to match fmsub.d. But if you take a look in RISCVInstrInfoD.td, you’ll see that the pattern already exists:

// fmsub: rs1 * rs2 - rs3
def : Pat<(any_fma FPR64:$rs1, FPR64:$rs2, (fneg FPR64:$rs3)),
          (FMSUB_D FPR64:$rs1, FPR64:$rs2, FPR64:$rs3, FRM_DYN)>;

We’ll need to figure out why this pattern isn’t getting selected, so lets start by extracting the build commands so we can look under the hood and dump the LLVM IR:

$ cmake -B build -C cmake/caches/ReleaseLTO.cmake --toolchain=...
$ ninja -C build 508.namd_r -t clean
$ ninja -C build 508.namd_r -v
...
[44/45] : && llvm-project/build.release/bin/clang++ --target=riscv64-linux-gnu -march=rva22u64_v -O3 -fomit-frame-pointer -flto -DNDEBUG -fuse-ld=lld ... -o External/SPEC/CFP2017rate/508.namd_r/508.namd_r

This is an LTO build so the code generation step is actually happening during link time. To dump the IR we can copy and paste the link command from the verbose output and append -Wl,--save-temps to it, which in turn tells the Clang driver to pass --save-temps to the linker².

$ llvm-project/build.release/bin/clang++ -Wl,--save-temps --target=riscv64-linux-gnu -march=rva22u64_v -O3 -fomit-frame-pointer -flto -DNDEBUG -fuse-ld=lld ... -o External/SPEC/CFP2017rate/508.namd_r/508.namd_r 
$ ls External/SPEC/CFP2017rate/508.namd_r/508.namd_r*
External/SPEC/CFP2017rate/508.namd_r/508.namd_r
External/SPEC/CFP2017rate/508.namd_r/508.namd_r.0.0.preopt.bc
External/SPEC/CFP2017rate/508.namd_r/508.namd_r.0.2.internalize.bc
External/SPEC/CFP2017rate/508.namd_r/508.namd_r.0.4.opt.bc
External/SPEC/CFP2017rate/508.namd_r/508.namd_r.0.5.precodegen.bc

The bitcode is dumped at various stages, and 508.namd_r.0.5.precodegen.bc is the particular stage we’re looking for. This is after all the middle-end optimisations have run and is as close as we’ll get before the backend begins. It contains the bitcode for the entire program though, so lets find the symbol for the C++ function and extract just that corresponding LLVM IR function:

$ llvm-objdump -t 508.namd_r | grep calc_pair_energy_fullelect
...
000000000004562e l     F .text	0000000000001c92 _ZN20ComputeNonbondedUtil26calc_pair_energy_fullelectEP9nonbonded
$ llvm-extract -f 508.namd_r.0.5.precodegen.bc --func _ZN20ComputeNonbondedUtil26calc_pair_energy_fullelectEP9nonbonded \
  | llvm-dis > calc_pair_energy_fullelect.precodegen.ll

Now quickly grep the disassembled LLVM IR to see if we can find the source of the fnegs:

  %316 = fneg double %315
  %neg = fmul double %mul922, %316
  %317 = tail call double @llvm.fmuladd.f64(double %mul919, double %314, double %neg)

This looks promising. We have a @llvm.fmuladd that’s being fed by a fmul of a fneg, which is similar to the (a*b)+(c*-d) pattern in the resulting assembly. But looking back to our TableGen pattern for fmsub.d, we want (any_fma $rs1, $rs2, (fneg $rs3)), i.e. a llvm.fmuladd fed by a fneg of a fmul.

One thing about floating point arithmetic is that whilst it’s generally not associative, we can hoist out the fneg from the fmul since all negation does is flip the sign bit. So we can try to teach InstCombine to hoist the fneg outwards like (fmul x, (fneg y)) -> (fneg (fmul x, y)). But if we go to try that out we’ll see that InstCombine already does the exact opposite:

Instruction *InstCombinerImpl::visitFNeg(UnaryOperator &I) {
  Value *Op = I.getOperand(0);
  // ...
  Value *OneUse;
  if (!match(Op, m_OneUse(m_Value(OneUse))))
    return nullptr;

  if (Instruction *R = hoistFNegAboveFMulFDiv(OneUse, I))
    return replaceInstUsesWith(I, R);
  // ...
}

Instruction *InstCombinerImpl::hoistFNegAboveFMulFDiv(Value *FNegOp,
                                                      Instruction &FMFSource) {
  Value *X, *Y;
  if (match(FNegOp, m_FMul(m_Value(X), m_Value(Y)))) {
    // Push into RHS which is more likely to simplify (const or another fneg).
    // FIXME: It would be better to invert the transform.
    return cast<Instruction>(Builder.CreateFMulFMF(
        X, Builder.CreateFNegFMF(Y, &FMFSource), &FMFSource));
  }

InstCombine usually has good reasons for canonicalizing certain IR patterns, so we need to seriously reconsider if we want to change the canonical form. InstCombine affects all targets and it could be the case that some other backends have patterns that match fmul (fneg x, y), in which case we don’t want disturb them. However for RISC-V we know what our patterns for instruction selection are and what form we want our incoming IR to be in. So a much better place to handle this in is in RISCVISelLowering.cpp, which lets us massage it into shape at the SelectionDAG level, in a way that’s localized to just our target. “Un-canonicalizing” the IR is a common task that backends end up performing, and this is what the resulting patch ended up looking like:

--- a/llvm/lib/Target/RISCV/RISCVISelLowering.cpp
+++ b/llvm/lib/Target/RISCV/RISCVISelLowering.cpp
@@ -20248,6 +20248,17 @@ SDValue RISCVTargetLowering::PerformDAGCombine(SDNode *N,
       return V;
     break;
   case ISD::FMUL: {
+    using namespace SDPatternMatch;
+    SDLoc DL(N);
+    EVT VT = N->getValueType(0);
+    SDValue X, Y;
+    // InstCombine canonicalizes fneg (fmul x, y) -> fmul x, (fneg y), see
+    // hoistFNegAboveFMulFDiv.
+    // Undo this and sink the fneg so we match more fmsub/fnmadd patterns.
+    if (sd_match(N, m_FMul(m_Value(X), m_OneUse(m_FNeg(m_Value(Y))))))
+      return DAG.getNode(ISD::FNEG, DL, VT,
+                         DAG.getNode(ISD::FMUL, DL, VT, X, Y));
+

And if we rebuild our benchmark after applying it, we can see we the fmsub.ds getting matched, saving a couple of instructions:

@@ -983,18 +983,15 @@
        fld     ft2, 48(a5)
        fld     ft3, 64(a5)
        fld     ft4, 72(a5)
-       fneg.d  fa0, fa0
-       fneg.d  ft0, ft0
-       fneg.d  ft2, ft2
        fmul.d  fa3, ft5, fa3
        fmul.d  fa0, fa3, fa0
        fmul.d  ft0, fa3, ft0
        fmul.d  fa3, fa3, ft2
        fld     ft2, 0(s1)
        fmul.d  fa4, ft5, fa4
-       fmadd.d fa2, fa4, fa2, fa0
-       fmadd.d ft6, fa4, ft6, ft0
-       fmadd.d fa4, fa4, ft1, fa3
+       fmsub.d fa2, fa4, fa2, fa0
+       fmsub.d ft6, fa4, ft6, ft0
+       fmsub.d fa4, fa4, ft1, fa3

All in all this ended up giving a 1.77% improvement in instruction count for the 508.namd_r benchmark. It’s still not nearly as fast as GCC, but we’re a little bit closer than before we started.

What’s next?

Hopefully this has given you an overview of how to identify opportunities for optimization in LLVM, and what a typical fix might look like. The analysis is really the most important part, but if you don’t feel like setting up an LNT instance yourself locally Igalia runs one at cc-perf.igalia.com³. We run llvm-test-suite and SPEC CPU 2017 nightly built with Clang and GCC on a small set of RISC-V hardware⁴, but hopefully to be expanded in future. Feel free to use it to investigate some of the differences between Clang and GCC yourself, and maybe you’ll find some inspiration for optimizations.

In the next post in this series I’ll talk about a performance improvement that recently landed related to cost modelling.

Update on what happened in WebKit in the week from December 1 to December 8.

In this edition of the periodical we have further advancements on the Temporal implementation, support for Vivante super-tiled format, and an adaptation of the DMA-BUF formats code to the Android port.

Cross-Port 🐱

JavaScriptCore 🐟

Graphics 🖼️

WPE WebKit 📟

WPE Platform API 🧩

Some years ago I had mentioned some command line tools I used to analyze and find useful information on GStreamer logs. I’ve been using them consistently along all these years, but some weeks ago I thought about unifying them in a single tool that could provide more flexibility in the mid term, and also as an excuse to unrust my Rust knowledge a bit. That’s how I wrote Meow, a tool to make cat speak (that is, to provide meaningful information).

The idea is that you can cat a file through meow and apply the filters, like this:

cat /tmp/log.txt | meow appsinknewsample n:V0 n:video ht: \
ft:-0:00:21.466607596 's:#([A-za-z][A-Za-z]*/)*#'

which means “select those lines that contain appsinknewsample (with case insensitive matching), but don’t contain V0 nor video (that is, by exclusion, only that contain audio, probably because we’ve analyzed both and realized that we should focus on audio for our specific problem), highlight the different thread ids, only show those lines with timestamp lower than 21.46 sec, and change strings like Source/WebCore/platform/graphics/gstreamer/mse/AppendPipeline.cpp to become just AppendPipeline.cpp“, to get an output as shown in this terminal screenshot:

Cool, isn’t it? After all, I’m convinced that the answer to any GStreamer bug is always hidden in the logs (or will be, as soon as I add “just a couple of log lines more, bro” ).

Currently, meow supports this set of manipulation commands:

• Word filter and highlighting by regular expression (fc:REGEX, or just REGEX): Every expression will highlight its matched words in a different color.
• Filtering without highlighting (fn:REGEX): Same as fc:, but without highlighting the matched string. This is useful for those times when you want to match lines that have two expressions (E1, E2) but the highlighting would pollute the line too much. In those case you can use a regex such as E1.*E2 and then highlight the subexpressions manually later with an h: rule.
• Negative filter (n:REGEX): Selects only the lines that don’t match the regex filter. No highlighting.
• Highlight with no filter (h:REGEX): Doesn’t discard any line, just highlights the specified regex.
• Substitution (s:/REGEX/REPLACE): Replaces one pattern for another. Any other delimiter character can be used instead of /, it that’s more convenient to the user (for instance, using # when dealing with expressions to manipulate paths).
• Time filter (ft:TIME-TIME): Assuming the lines start with a GStreamer log timestamp, this filter selects only the lines between the target start and end time. Any of the time arguments (or both) can be omitted, but the - delimiter must be present. Specifying multiple time filters will generate matches that fit on any of the time ranges, but overlapping ranges can trigger undefined behaviour.
• Highlight threads (ht:): Assuming a GStreamer log, where the thread id appears as the third word in the line, highlights each thread in a different color.

The REGEX pattern is a regular expression. All the matches are case insensitive. When used for substitutions, capture groups can be defined as (?CAPTURE_NAMEREGEX).

The REPLACEment string is the text that the REGEX will be replaced by when doing substitutions. Text captured by a named capture group can be referred to by ${CAPTURE_NAME}.

The TIME pattern can be any sequence of numbers, : or . . Typically, it will be a GStreamer timestamp (eg: 0:01:10.881123150), but it can actually be any other numerical sequence. Times are compared lexicographically, so it’s important that all of them have the same string length.

The filtering algorithm has a custom set of priorities for operations, so that they get executed in an intuitive order. For instance, a sequence of filter matching expressions (fc:, fn:) will have the same priority (that is, any of them will let a text line pass if it matches, not forbidding any of the lines already allowed by sibling expressions), while a negative filter will only be applied on the results left by the sequence of filters before it. Substitutions will be applied at their specific position (not before or after), and will therefore modify the line in a way that can alter the matching of subsequent filters. In general, the user doesn’t have to worry about any of this, because the rules are designed to generate the result that you would expect.

Now some practical examples:

Example 1: Select lines with the word “one”, or the word “orange”, or a number, highlighting each pattern in a different color except the number, which will have no color:

$ cat file.txt | meow one fc:orange 'fn:[0-9][0-9]*'
000 one small orange
005 one big orange

Example 2: Assuming a pictures filename listing, select filenames not ending in “jpg” nor in “jpeg”, and rename the filename to “.bak”, preserving the extension at the end:

$ cat list.txt | meow 'n:jpe?g' \
   's:#^(?<f>[^.]*)(?<e>[.].*)$#${f}.bak${e}'
train.bak.png
sunset.bak.gif

Example 3: Only print the log lines with times between 0:00:24.787450146 and 0:00:24.790741865 or those at 0:00:30.492576587 or after, and highlight every thread in a different color:

$ cat log.txt | meow ft:0:00:24.787450146-0:00:24.790741865 \
   ft:0:00:30.492576587- ht:
0:00:24.787450146 739 0x1ee2320 DEBUG …
0:00:24.790382735 739 0x1f01598 INFO …
0:00:24.790741865 739 0x1ee2320 DEBUG …
0:00:30.492576587 739 0x1f01598 DEBUG …
0:00:31.938743646 739 0x1f01598 ERROR …

This is only the begining. I have great ideas for this new tool (as time allows), such as support for parenthesis (so the expressions can be grouped), or call stack indentation on logs generated by tracers, in a similar way to what Alicia’s gst-log-indent-tracers tool does. I might also predefine some common expressions to use in regular expressions, such as the ones to match paths (so that the user doesn’t have to think about them and reinvent the wheel every time). Anyway, these are only ideas. Only time and hyperfocus slots will tell…

By now, you can find the source code on my github. Meow!

Standards Queues

The hardest part of web standards isn’t even the technology — it’s the queues. And that’s the real problem I keep coming back to.

Pools, Queues, and Bottlenecks

As programmers, we’re familiar with these kinds of problems: if things enter faster than they leave, they back up. We often need to prioritize among the backlog. The standards process is like several of those queues stacked together.

Ideas enter the system far faster than they leave — and they can come from anywhere. But to progress, you need implementers. They are finite, already busy, and often advancing their own priorities. On top of that, every proposal competes for wide review in privacy, security, architecture, accessibility, and internationalization. Each of those specialties is even more finite, even more busy, and even more backed up.

So an idea lands in hundreds — even thousands — of inboxes, waiting for attention. We might not even notice it as it whips past among all the others. Even if we do, it might just get starred in email or left open in a tab for “later.” Sometimes that idea is a book, or an explainer, or suddenly has 20 replies. Instead of needing five minutes to read and consider, it becomes intimidating.

At some point it just sits. It might wait weeks, months, or even years before someone comments. Why? Because everyone has jobs with other tasks. The queues are full.

And the longer it sits, the more things change around it. The more it unloads from memory. The more intimidating it becomes to return to. It has to get through a whole lot of asynchronous back-and-forth between implementers, spec writers, and test writers before reaching baseline usability.

Along the way, if coordination isn’t strong (and historically it hasn’t been), once work is invested it’s hard to throw away. It’s hard to propose breaking changes or add stop energy.

Real Impacts

This is why something like :focus-visible can take seven years. Realistically, it required only a few days of effective discussion. The tests and development weren’t that hard.

The hard part was agreeing on what it should do and which tests it should pass. Most of that difficulty came from the fact that you couldn’t get everyone to sit down and focus concurrently. Implementations — and thus real focus — were years apart.

Checking Fitness

Getting “unstuck” isn’t just about moving something forward. One major appeal of the standards process is wide review, but for this to work we need ways for things to fail early enough to shift efforts.

Sometimes failure happens after months or even years. That’s frustrating and demoralizing. It’s like standing in a long DMV line, inching forward, only to discover at the end that you’re in the wrong building.

All of this is made worse by the fact that queues keep getting fuller.

Things that help

Interop

The Interop project illustrates the very end of the process, and in many ways the simplest.

Without intervention, each implementer historically built their own priority queue from all possible shortcomings of their browser engine. There’s a huge pool of things to choose from. I’ve written before about how WPT and the dashboard aren’t the best way to view this, but there are currently almost 23k subtests that fail in every browser (or almost 11k that fail and aren’t marked tentative).

Interop coordinates efforts to choose an achievable set of things from this gigantic pool that meet strict criteria: all that’s left is implementation work. It’s been hugely successful because it ensures delivery. It also helps us deliver early when people are excited about common priorities. In those cases, the impact is huge — we can go from mostly words to three interoperable implementations in one year. Amazing.

Still, every year a huge number of things remain in the pool that we can’t afford to take up. The pool keeps growing.

Joint Meetings

The WHATWG has started holding regular joint meetings with groups like OpenUI and CSSWG. This is valuable because it allows the right people to agree on an agenda and discuss directly, rather than leaving issues to sit unnoticed or requiring endless pings for attention.

W3C's TPAC is an annual event with five days of meetings (both W3C and WHATWG), many of them joint with wide-review specialists. These are dedicated times to get a lot of people in the same rooms for extended periods. The availability for hallway conversations also matters a lot: You can corner people there in ways that are much harder when everyone is remote. More progress happens at TPAC than in half the rest of the year combined.

Timely coordination — and investment to make it plausible — is still the single biggest problem we face in standards. I'd love to see us find ways to improve that.

Update on what happened in WebKit in the week from November 24 to December 1.

The main highlights for this week are the completion of `PlainMonthDay` in Temporal, moving networking access for GstWebRTC to the WebProcess, and Xbox Cloud Gaming now working in the GTK and WPE ports.

Cross-Port 🐱

Multimedia 🎥

JavaScriptCore 🐟

WebKitGTK 🖥️

WPE WebKit 📟

WPE Platform API 🧩

Infrastructure 🏗️

Community & Events 🤝

Although analysing performance by way of instruction counting has obvious limitations, it can be helpful (especially when combined with appropriate analysis scripts) to get rapid feedback on the impact of code generation changes or to explore hypotheses about why code from one compiler might be performing differently from another - for instance, by looking at instruction mix in the most executed translation blocks. In this post we'll look at how to capture the necessary data to perform such an analysis using a QEMU plugin. Future posts will give details of the analysis scripts I've used, and walk through an example or two of putting them to use.

Modifying QEMU

Over the past few years, QEMU's plugin API has developed a fair bit. QEMU includes several plugins, and hotblocks provides almost what we want but doesn't allow configurability of the number of blocks it will print information on. I submitted a small patch series (and submitted it a second time addressing this and other minor issues found along the way. The series has now been accepted by the maintainer.

To build QEMU with this patch:

git clone https://github.com/qemu/qemu && cd qemu
git checkout v10.1.2
cat - <<'EOF' > hotblocks.patch
index 98404b6885..8ecf033997 100644
--- a/contrib/plugins/hotblocks.c
+++ b/contrib/plugins/hotblocks.c
@@ -73,28 +73,29 @@ static void exec_count_free(gpointer key, gpointer value, gpointer user_data)
 static void plugin_exit(qemu_plugin_id_t id, void *p)
 {
     g_autoptr(GString) report = g_string_new("collected ");
-    GList *counts, *it;
+    GList *counts, *sorted_counts, *it;
     int i;

     g_string_append_printf(report, "%d entries in the hash table\n",
                            g_hash_table_size(hotblocks));
     counts = g_hash_table_get_values(hotblocks);
-    it = g_list_sort_with_data(counts, cmp_exec_count, NULL);
+    sorted_counts = g_list_sort_with_data(counts, cmp_exec_count, NULL);

-    if (it) {
+    if (sorted_counts) {
         g_string_append_printf(report, "pc, tcount, icount, ecount\n");

-        for (i = 0; i < limit && it->next; i++, it = it->next) {
+        for (i = 0, it = sorted_counts; (limit == 0 || i < limit) && it;
+             i++, it = it->next) {
             ExecCount *rec = (ExecCount *) it->data;
             g_string_append_printf(
-                report, "0x%016"PRIx64", %d, %ld, %"PRId64"\n",
+                report, "0x%016"PRIx64", %d, %ld, %"PRIu64"\n",
                 rec->start_addr, rec->trans_count,
                 rec->insns,
                 qemu_plugin_u64_sum(
                     qemu_plugin_scoreboard_u64(rec->exec_count)));
         }

-        g_list_free(it);
+        g_list_free(sorted_counts);
     }

     qemu_plugin_outs(report->str);
@@ -170,6 +171,13 @@ int qemu_plugin_install(qemu_plugin_id_t id, const qemu_info_t *info,
                 fprintf(stderr, "boolean argument parsing failed: %s\n", opt);
                 return -1;
             }
+        } else if (g_strcmp0(tokens[0], "limit") == 0) {
+            char *endptr = NULL;
+            limit = g_ascii_strtoull(tokens[1], &endptr, 10);
+            if (endptr == tokens[1] || *endptr != '\0') {
+                fprintf(stderr, "unsigned integer parsing failed: %s\n", opt);
+                return -1;
+            }
         } else {
             fprintf(stderr, "option parsing failed: %s\n", opt);
             return -1;
diff --git a/docs/about/emulation.rst b/docs/about/emulation.rst
index 4a7d1f4178..e8793b0f9c 100644
--- a/docs/about/emulation.rst
+++ b/docs/about/emulation.rst
@@ -463,6 +463,18 @@ Example::
   0x000000004002b0, 1, 4, 66087
   ...

+Behaviour can be tweaked with the following arguments:
+
+.. list-table:: Hot Blocks plugin arguments
+  :widths: 20 80
+  :header-rows: 1
+
+  * - Option
+    - Description
+  * - inline=true|false
+    - Use faster inline addition of a single counter.
+  * - limit=N
+    - The number of blocks to be printed. (Default: N = 20, use 0 for no limit).

 Hot Pages
 .........
EOF
patch -p1 < hotblocks.patch
./configure --prefix=$(pwd)/inst --target-list="riscv32-linux-user riscv64-linux-user"
make -j$(nproc)
cd ..

Using this plugin to capture statistics from running a binary under qemu-user

Assuming you have an appropriate sysroot, you can run a binary and have the execution information emitted to stderr by doing something like:

QEMUDIR=$HOME/qemu/build
SYSROOT=$HOME/rvsysroot
$QEMUDIR/qemu-riscv64 \
  -L $SYSROOT \
  -plugin $QEMUDIR/contrib/plugins/libhotblocks.so,limit=0,inline=on \
  -d plugin,nochain \
  my_rv64_binary

This produces output like:

collected 2229 entries in the hash table
pc, tcount, icount, ecount
0x00007fffee7012ba, 1, 1, 3737
0x00007fffee7012be, 1, 3, 3737
0x00007ffff741e738, 1, 23, 1074
0x00007fffee71bb38, 1, 5, 884
0x00007ffff741bb2e, 1, 11, 662
...

This listing indicates the address of the translation block, the number of times it's been translated, the number of instructions it contains, and the number of times it was executed. Note that a translation block is not the same as a basic block in the compiler. A translation block can span multiple basic blocks in the case of fallthrough, and this can also mean an instruction may show up in multiple translation blocks.

At least for my use cases, I need something a bit more involved than this. In order to add collection of these statistics to an existing benchmark harness I need a wrapper script that transparently collects these statistics to a file. It's also helpful to capture the runtime address of executable mappings for loaded libraries, allowing translation blocks to be attributed easily to either the binary itself or libc, libm etc. We have gdb connect to QEMU's gdbserver in order to dump those mappings. Do ensure you're using a recent version of QEMU (the version suggested in the patch application instructions is definitely good) for this as I wasted quite some time running into a bug with file descriptor numbers that caused odd breakage.

This qemu-forwarder.sh script will capture the plugin's output in a .qemu_out file and the mappings in a .map file, both of which can be later consumed by a detailed analysis script.

#!/bin/sh
QEMUDIR=$HOME/qemu/build
SYSROOT=$HOME/rvsysroot
QEMU="$QEMUDIR/qemu-riscv64 \
  -L $SYSROOT \
  -plugin $QEMUDIR/contrib/plugins/libhotblocks.so,limit=0,inline=on \
  -d plugin,nochain"

SUFFIX=""
if [ -e "$1.qemu_out" ]; then
  NUM=1
  while [ -e "$1.qemu_out.$NUM" ]; do
    NUM=$((NUM + 1))
  done
  SUFFIX=".$NUM"
fi

GDB_SOCK=$(mktemp -u)
setarch $(uname -m) -R $QEMU -g $GDB_SOCK -D $1.qemu_out$SUFFIX "$@" &
QEMU_PID=$!

RETRY_COUNT=0
while ! [ -e "$GDB_SOCK" ]; do
  RETRY_COUNT=$((RETRY_COUNT + 1))
  if [ $RETRY_COUNT -eq 10 ]; then
    echo "Timed out waiting for gdb socket to be created"
    exit 1
  fi
  sleep 0.1
  if ! kill -0 $QEMU_PID 2>/dev/null; then
    echo "QEMU process died before gdb socket was created"
    wait $QEMU_PID
    exit $?
  fi
done

gdb -batch \
  -ex "set pagination off" \
  -ex "target remote $GDB_SOCK" \
  -ex "break main" \
  -ex "continue" \
  -ex "set logging file $1.map$SUFFIX" \
  -ex "set logging enabled on" \
  -ex "info proc mappings" \
  -ex "detach" > /dev/null 2>&1
wait $QEMU_PID

The above will work under LLVM's lit, though you will need to use a recent enough version that doesn't strip HOME from the environment (or else edit the script accordingly). It also produces output in sequentially numbered files, again motivated by the desire to run under this script from lit as used by llvm-test-suite's SPEC configuration which can involve multiple invocations of the same binary for a given benchmark (e.g. 500.perlbench_r).

Analysing the output

A follow-up post will introduce the scripting I've built around this.

Recording and analysing results from running SPEC

Assuming you have qemu-forwarder.sh, in your llvm-test-suite directory:

CONF=clang-head-test
CLANG_BIN_DIR=$HOME/llvm-project/build/release/bin
CFLAGS="-march=rv64gc_zba_zbb_zbs"
cat - <<EOF > $CONF.cmake
set(CMAKE_SYSTEM_NAME Linux)

set(CMAKE_SYSROOT $HOME/rvsysroot)

set(CMAKE_C_COMPILER $CLANG_BIN_DIR/clang)
set(CMAKE_CXX_COMPILER $CLANG_BIN_DIR/clang++)

set(CMAKE_C_COMPILER_TARGET riscv64-linux-gnu)
set(CMAKE_CXX_COMPILER_TARGET riscv64-linux-gnu)
set(CMAKE_C_FLAGS_INIT "$CFLAGS")
set(CMAKE_CXX_FLAGS_INIT "$CFLAGS")

set(CMAKE_LINKER_TYPE LLD)

set(CMAKE_FIND_ROOT_PATH_MODE_PROGRAM NEVER)
set(CMAKE_FIND_ROOT_PATH_MODE_LIBRARY ONLY)
set(CMAKE_FIND_ROOT_PATH_MODE_INCLUDE ONLY)
set(CMAKE_FIND_ROOT_PATH_MODE_PACKAGE ONLY)
EOF
cmake -G Ninja \
  -B build.$CONF \
  --toolchain=$CONF.cmake \
  -DTEST_SUITE_SPEC2017_ROOT=~/cpu2017 \
  -DTEST_SUITE_SUBDIRS=External/SPEC \
  -DTEST_SUITE_COLLECT_CODE_SIZE=OFF \
  -DTEST_SUITE_COLLECT_COMPILE_TIME=OFF \
  -DTEST_SUITE_USER_MODE_EMULATION=ON \
  -DTEST_SUITE_RUN_UNDER=$(pwd)/qemu-forwarder.sh
cmake --build build.$CONF
$CLANG_BIN_DIR/llvm-lit -v --filter-out='.+_s|specrand' build.$CONF

The 526.blender_r test takes twice as long as the others, so you may wish to skip it by instead executing something like:

$CLANG_BIN_DIR/llvm-lit -v --filter-out='.+_s|specrand|blender' build.$CONF

If you want to re-run tests you must delete the previous .qemu_out and .map files, which can be done with:

[ -n "build.$CONF" ] && find "build.$CONF" -type f -name "*.qemu_out*" -exec sh -c '
    for q_file do
        base_path="${q_file%.qemu_out*}"
        rm -f "$q_file" "${base_path}.map"*
    done
' sh {} +

In order to compare two SPEC builds, you can use something like the following hacky script. Using the captured translation block execution data to generate a plain executed instruction count is overkill as the example tests/tcg/plugin/insn.c can easily dump for this for you directly. But by collecting the data upfront, you can easily dive right into a more detailed analysis when you see a surprising difference in executed instruction counts without rerunning the binary.

#!/usr/bin/env python3

from pathlib import Path
from collections import defaultdict
import sys

def collect_totals(root_dir):
    totals = defaultdict(int)
    root_path = Path(root_dir)/"External"

    for file_path in root_path.rglob("*.qemu_out*"):
        benchmark_name = file_path.parts[4]

        try:
            with file_path.open("r") as f:
                file_total = 0
                for line in f:
                    parts = line.strip().split(',')
                    # Only sum lines that match the expected format.
                    if len(parts) == 4 and parts[2].strip().isdigit():
                        # icount * ecount.
                        file_total += int(parts[2]) * int(parts[3])
                totals[benchmark_name] += file_total
        except Exception as e:
            print(f"Error reading {file_path}: {e}")

    return totals

if __name__ == "__main__":
    if len(sys.argv) != 3:
        print("Usage: spec-compare-helper <dir_a> <dir_b>")
        sys.exit(1)

    dir_a, dir_b = sys.argv[1], sys.argv[2]
    totals_a = collect_totals(dir_a)
    totals_b = collect_totals(dir_b)

    benchmarks = sorted(set(totals_a.keys()) | set(totals_b.keys()))

    print(f"{'Benchmark':<20} {'DirA':>15} {'DirB':>15} {'Diff (%)':>10}")
    print("=" * 60)

    for benchmark in benchmarks:
        val_a = totals_a.get(benchmark, 0)
        val_b = totals_b.get(benchmark, 0)
        diff_pct = ((val_b - val_a) / val_a * 100) if val_a else float("inf")

        print(f"{benchmark:<20} {val_a:>15} {val_b:>15} {diff_pct:>9.2f}%")

Which produces output looking something like this:

Benchmark                       DirA            DirB   Diff (%)
============================================================
500.perlbench_r         180245097594    182078714777      1.02%
502.gcc_r               220874510659    219647717585     -0.56%
505.mcf_r               131589945456    134271153130      2.04%
508.namd_r              220648061019    216682202888     -1.80%
510.parest_r            291341820355    291844973715      0.17%
511.povray_r             31911866906     31103201809     -2.53%
519.lbm_r                94166321698     86910581403     -7.71%
520.omnetpp_r           138002605692    137676301622     -0.24%
523.xalancbmk_r         283566182007    284735075518      0.41%
525.x264_r              380165035845    379862173371     -0.08%
526.blender_r           660528270138    659361380750     -0.18%
531.deepsjeng_r         355058534962    349621355155     -1.53%
538.imagick_r           238573643488    238560676372     -0.01%
541.leela_r             421886351310    405423320484     -3.90%
544.nab_r               415595728542    391443973852     -5.81%
557.xz_r                132548718317    130229753780     -1.75%

It's worth highlighting that as we're running this under user-mode emulation, the dynamic instruction count naturally never counts any instructions on the kernel side that you would see if profiling a real system.

• 2025-12-15: Note that the qemu patches have now been accepted in the maintainer's tree.
• 2025-12-02: Initial publication date.

You can now easily customize find-in-page with the new ::search-text pseudo-element, that is shipping in Chromium 144.0.7547. 🚀

Find more details on the blog post by Stephen Chenney. Thanks to Bloomberg for sponsoring this work.

Recently I jotted down some notes on LLM inference vs training costs for DeepSeek and I wanted to add on an additional datapoint for training cost based on the recently released Olmo3 models from the Allen Institute for AI ("Ai2"). The model family has 7B and 32B parameter models, with 'Think' variants available for 7B and 32B but so far only a 7B 'Instruct' non-reasoning version (but watch this space). What's particularly interesting about the Olmo models to me is that beyond providing open weights, the training scripts and datasets are openly available as well.

Going by the reported benchmarks at least it's competitive with less open models at a similar size, and importantly they've increased the supported context length from the rather limiting 4k tokens supported by the Olmo 2 series to a much more usable 64k tokens. Given the relatively small size these models are less capable than relatively chunky models like DeepSeek R1/V3.x or Kimi K2, but I've been impressed by the capability of 32B dense models for basic queries, and from my non-scientific testing both the 32B and 7B Olmo3 variants seem to do a reasonable job of summarising things like discussion threads. You can experiment yourself at playground.allenai.org.

Energy required for training Olmo 3

One of the neat things about this level of openness is that it should act as a floor in terms of performance for future models of this size class assuming they're appropriately funded and don't take too many risks chasing novelty. Rerunning the training process with an updated dataset and some minor tweaks is something you could imagine doing on some regular cadence, ideally as a shared endeavour. Imagining this effort in the future, how much energy is required? The initial version of the detailed Olmo 3 technical report unfortunately has little to say on this. We can get a back of the envelope figure in terms of GPU hours for pre-training based on the reported 7700 tokens per second per GPU for the 7B base model and 1900 tokens per second for the 32B base model and the ~6T token dataset. But even better than that, we can just ask the Ai2 folks (sometimes the internet really does work wonderfully!). After asking on their public Discord I was rapidly furnished with this helpful answer:

For some detailed numbers, we measured power consumption throughout training, along with total GPU hours. We used ~234k H100 hours to pretrain the 7B, and ~1.05m H100 hours to pretrain the 32B. 1900 TPS is generally what our trainer is capable of, but with restarts, evaluations, checkpointing, and occasional network issues, the 32B took 1.05m hours. We measured an average power consumption of ~621W while pretraining the 7B and ~649W while pretraining the 32B, and this means that our GPUs consumed ~146MWh for the 7B and ~681MWh for the 32B. We'll include more detailed GPU hour information in a future version of the paper, including for post-training!

Ai2 Olmo 3 team on their Discord.

So that's 0.681 GWh in GPU power draw for pretraining the 32B model and 0.146 GWh in GPU power draw for pretraining the 7B model. As noted in the quote, this is inclusive of restarts, checkpointing etc. But perhaps won't include previous early stage experimentation. I look forward to an updated technical report with full details, but pretraining should cover the bulk of the compute requirements (as a reference point, today's DeepSeek V3.2 paper found it notable that the post-training compute budget exceeded 10% of the pretraining cost).

The 0.681 GWh figure doesn't account for full system power and cooling cost. I'd love to be corrected, but I believe a 1.5x-2x multiplier would be an assumption towards the upper end. But for the sake of this yardstick comparison let's look at a few comparisons based on the reported number:

• 0.681 GWh of electricity would cost about £180k at UK residential rates (capped at 26.35p per kWh currently). Substantially less in the USA.
• A larger leisure centre with a pool consumes ~2.5 GWh of energy per year. I don't know if the idea of a "leisure centre" translates outside of the UK, but basically it's a swimming pool plus gym, squash/tennis courts etc.
  • The linked page claims ~2 GWh of energy in gas and 0.5 GWh in electricity. For the gas, to compare like with like you'd need to consider the source of energy for the electricity used for Olmo training.
• 0.681 GWh is ~0.11% of LHC's annual 600 GWh energy consumption or ~0.05% of CERN's annual consumption.
• We can estimate a Boeing 787-9 flying from London Heathrow to SFO consumes jet fuel containing ~0.58 GWh of energy.
  • Calculated with 8638km distance, 5.62kg fuel/km (taking the most economic 787-9 long haul figure from this table on Wikipedia and 11.95kWh/kg specific energy of jet fuel).
  • This is a yardstick rather than a direct comparison. A direct comparison to the GWh of electricity used for the GPU compute of the LLM would depend on the source of the electricity. If it was e.g. gas rather than solar/hydro/wind then you'd want to compare the number of GWh consumed to create that electricity which would of course be higher.
  • As a further point of reference, FlightAware indicates 5 separate direct LHR to SFO flights scheduled per day.

More efficient LLM training

We can hope for new breakthroughs, more efficient hardware, better datasets and so on. But here is some work I noticed in the area. Fair warning: this isn't my field, and we have to recognise applying a research result to a production training run is sure to have challenges even if the research suggests the trade-offs are worthwhile. So consider this vague gesticulating about seemingly interesting work that is going on and find someone who knows what they're talking about to confirm the degree to which it is interesting/viable.

• Mixture of Experts (MoE) models are substantially cheaper to train which is one reason the industry has moved in that direction. The next Ai2 Olmo model is expected to be MoE. The Qwen blog has a graph comparing the relative training cost in GPU hours of the dense Qwen3-32B vs Qwen3-30B-A3b vs Qwen3-Next-80B-A3B, where the latter makes further architectural changes, reporting a 10.7x reduction. ~2.5x of that is going to come from the reduced corpus size (15T tokens down from 36T), but that still leaves plenty of improvement from other factors.
• Maybe it will be shown viable to train in lower precision such as MXFP8 or even NVFP4, which would allow much more throughput for a similar energy budget. Nvidia have worked to demonstrate this can be effective for both formats (see also this work from MIT).
• Also from Nvidia, Nemotron Elastic showed a model architecture that allows deriving smaller models without doing a separate pre-training runs.

Finally, the cheapest way to train an LLM from scratch is...to find a way to avoid the need to. For models like Olmo 3 that release the base model and checkpoints, people can apply their own post-training or perform additional pre-training.

Bonus comparison point: Apertus

Apertus is a Swiss project to produce an open LLM, with 70B and 8B models released so far. Their full tech report notes the following "Once a production environment has been set up, we estimate that the model can be realistically trained in approximately 90 days on 4096 GPUs, accounting for overheads. If we assume 560 W power usage per Grace-Hopper module in this period, below the set power limit of 660 W, we can estimate 5 GWh power usage for the compute of the pretraining run."

• 2025-12-04: Add link to "Four Over Six" NVFP4 training paper.
• 2025-12-02: Added clarifying note about energy via gas in the leisure centre comparison.
• 2025-12-01: Initial publication date.

I recently had reason to do a quick comparison of the performance of the Hetzner AX102 dedicated server and the high-end 'dedicated' CCX53 VPS on Hetzner Cloud and thought I may as well write up the results for posterity. I'm incapable of starting a post without some kind of disclaimer so here comes the one for this post: naturally the two products have major differences in terms of flexibility (spin-up/down at will, vs pay a small setup fee and endure a wait time depending on hardware availability). So depending on your use case, your requirements with respect to that flexibility may override any cost differential.

Specs

All costs are exclusive of VAT, assuming the lowest cost data center location, and inclusive of IPv4 address.

AX102:

• 16 core Ryzen 9 7950X3D (32 threads)
• 128GB DDR5 RAM
• 2 x 1.92TB NVMe
• 104 EUR/month, 39 EUR one-off setup fee.

CCX53

• Unknown AMD CPU exposing 32vCPU (physical cores? threads?)
• 128GB RAM
• 600GB NVMe
• 192.49 EUR/month maximum charge. 0.3085 EUR per hour (if you keep the same VPS active over the month it won't exceed the monthly price cap, so you effectively get a small discount on the per-hour cost).

Benchmark

Building Clang+LLVM+LLD, everyone's favourite workload! Both systems are running an up to date Arch Linux (more details on setting this up on the CCX53 in the appendix below) with clang 21.1.6. The dedicated machine has the advantage of RAID 0 across the two SSDs, but also has encrypted rootfs configured. I didn't bother to set that up for the CCX53 VPS.

sudo pacman -Syu --needed clang lld cmake ninja wget
LLVM_VER=21.1.6
wget https://github.com/llvm/llvm-project/releases/download/llvmorg-${LLVM_VER}/llvm-project-${LLVM_VER}.src.tar.xz
tar -xvf llvm-project-${LLVM_VER}.src.tar.xz
cd llvm-project-${LLVM_VER}.src

cmake -G Ninja \
  -DLLVM_ENABLE_PROJECTS='clang;lld' \
  -DLLVM_TARGETS_TO_BUILD="all" \
  -DLLVM_CCACHE_BUILD=OFF \
  -DCMAKE_C_COMPILER=clang \
  -DCMAKE_CXX_COMPILER=clang++ \
  -DLLVM_ENABLE_LLD=ON \
  -DCMAKE_BUILD_TYPE=Release \
  -DLLVM_ENABLE_ASSERTIONS=ON \
  -S llvm \
  -B build
time cmake --build build

printf "### Version info ###\n"
clang --version | head -n 1

On both machines, ninja shows 5575 build steps.

Results:

• AX102
  • 10m27s (627s)
• CCX53
  • 14m11s (851s, about 1.36x the AX102)

Running the clang and LLVM tests with ./build/bin/llvm-lit -s --order=lexical llvm/test clang/test (which shows 9402 tests) gives:

• AX102
  • 3m39s (219s)
• CCX53
  • 4m28s (268s, about 1.24x the AX102)

I ran these multiple times, and in the case of the CCX53 across two different VMs in different regions and saw only a few percentage points variance.

Focusing on the results for build clang/llvm/lld, let's figure out the cost for 1000 from-scratch builds. Not so much as it's a representative workload, but because it gives an easy to compare metric that captures both the difference in price and in performance. So calculating time_per_build_in_hours * 1000 * cost_per_hour:

• AX102
  • (626.6 / 3600) * 1000 * (104/720) = 25.14 EUR
  • Or if you include the setup fee and assume it's amortised over 12 months:
    • (626.6/3600) * 1000 * ((104 + (39/12))/720) = 25.93 EUR
• CCX53
  • (850.6 / 3600) * 1000 * (192.49/720) = 63.17 EUR
  • Or using the 0.3085 EUR/hr price which you would pay if you didn't run for the whole month:
    • (850.6 / 3600) * 1000 * 0.3085 = 72.89 EUR

Appendix: CCX53 Arch Linux setup

This could be scripted, but I just created the VPS via their web UI. Then after it was provisioned, used that web UI to have it boot into a rescue system. Then do an Arch bootstrap that roughly mirrors the one I use on a dedicated build machine except that we don't bother with encrypting the rootfs. The CCX* server types at least use UEFI so we can keep using efistub for boot.

First get a bootstrap environment and enter it:

wget http://mirror.hetzner.de/archlinux/iso/latest/archlinux-bootstrap-x86_64.tar.zst
tar -xvf archlinux-bootstrap-x86_64.tar.zst --numeric-owner
sed -i '1s;^;Server=https://mirror.hetzner.de/archlinux/$repo/os/$arch\n\n;' root.x86_64/etc/pacman.d/mirrorlist
mount --bind root.x86_64/ root.x86_64/ # See <https://bugs.archlinux.org/task/46169>
printf "About to enter bootstrap chroot\n===============================\n"
./root.x86_64/bin/arch-chroot root.x86_64/

Now set info that will be used throughout the process:

export NEW_HOST_NAME=archvps
export PUBLIC_SSH_KEY="ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOfpPQ1j+XLsapAhONAQmvu6TZGT5y8jeziM4Vio1NrA asb@plurp"
export NEW_USER=asb

And now proceed to set up the disks, create filesystems, perform an initial bootstrap and chroot into the new rootfs:

pacman-key --init
pacman-key --populate archlinux
pacman -Sy --noconfirm xfsprogs dosfstools

sfdisk /dev/sda <<EOF
label: gpt

start=1MiB, size=255MiB, type=uefi
start=256MiB, type=linux
EOF

mkfs.fat -F32 /dev/sda1
mkfs.xfs /dev/sda2

mount /dev/sda2 /mnt
mkdir /mnt/boot
mount /dev/sda1 /mnt/boot
pacstrap /mnt base linux linux-firmware efibootmgr \
  xfsprogs dosfstools \
   python3 \
  openssh sudo net-tools git man-db man-pages vim
genfstab -U /mnt >> /mnt/etc/fstab

printf "About to enter newrootfs chroot\n===============================\n"
arch-chroot /mnt

Do final configuration from within the chroot:

sed /etc/locale.gen -i -e "s/^\#en_GB.UTF-8 UTF-8.*/en_GB.UTF-8 UTF-8/"
locale-gen
# Ignore "System has not been booted with systemd" and "Failed to connect to bus" error for next command.
systemd-firstboot --locale=en_GB.UTF-8 --timezone=UTC --hostname="$NEW_HOST_NAME"
ln -s /dev/null /etc/udev/rules.d/80-net-setup-link.rules # disable persistent network names

# No longer need to disable large fallback image as Arch stopped generating it
# by default

printf "efibootmgr before changes:\n==========================\n"
efibootmgr -u
# Set up efistub
efibootmgr \
  --disk /dev/sda \
  --part 1 \
  --create \
  --label 'Arch Linux' \
  --loader /vmlinuz-linux \
  --unicode "root=/dev/sda2 rw initrd=\initramfs-linux.img" \
  --verbose
printf "efibootmgr after changes:\n=========================\n"
efibootmgr -u

mkswap --size=8G --file /swapfile
cat - <<EOF > /etc/systemd/system/swapfile.swap
[Unit]
Description=Swap file

[Swap]
What=/swapfile

[Install]
WantedBy=multi-user.target
EOF
systemctl enable swapfile.swap

cat - <<EOF > /etc/systemd/network/10-eth0.network
[Match]
Name=eth0

[Network]
DHCP=yes
Address=$(ip -6 addr show dev eth0 scope global | grep "scope global" | cut -d' ' -f6)
Gateway=$(ip route show | head -n 1 | cut -d' ' -f 3)
Gateway=fe80::1
EOF
systemctl enable systemd-networkd.service systemd-resolved.service systemd-timesyncd.service
printf "PasswordAuthentication no\n" > /etc/ssh/sshd_config.d/20-no-password-auth.conf
systemctl enable sshd.service
useradd -m -g users -G wheel -s /bin/bash "$NEW_USER"
usermod --pass='!' root # disable root login
chmod +w /etc/sudoers
printf "%%wheel ALL=(ALL) ALL\n" >> /etc/sudoers
chmod -w /etc/sudoers
mkdir "/home/$NEW_USER/.ssh"
printf "%s\n" "$PUBLIC_SSH_KEY" > "/home/$NEW_USER/.ssh/authorized_keys"
chmod 700 "/home/$NEW_USER/.ssh"
chmod 600 "/home/$NEW_USER/.ssh/authorized_keys"
chown -R "$NEW_USER:users" "/home/$NEW_USER/.ssh"

Now set password:

passwd "$NEW_USER"

Then ctrl-d twice and set a symlink for resolv.conf:

ln -sf ../run/systemd/resolve/stub-resolv.conf root.x86_64/mnt/etc/resolv.conf

Finally, reboot.

Remember to ssh-keygen -R $THE_IP_ADDRESS so you don't get ssh host verification errors.

• 2025-11-30: Initial publication date.

Tl;dr: Based on published data from DeepSeek, we can estimate it takes something like ~70 days of inference traffic (served by DeepSeek themselves, ignoring any other providers) to match the GPU hours used for the final training run for V3 and R1.

Simon Willison recently reshared some figures on inference costs for LLMs. I couldn't agree more with the comment further down that thread "The big AI labs continue to be infuriatingly opaque about the actual figures for their total electricity and water consumption".

A number of responses wonder about the cost of training. If you accept the reported figures for serving a query, what impact does it have if you amortise the energy spent training the model over the served queries? Mistral did this for their lifecycle analysis but they grouped together "training and inference" and kept confidential the ratio of energy for training vs inference by reporting a figure that combined the training cost with 18 months of usage. The thread reminded me of another datapoint available for DeepSeek that seemed worth writing up. I think this gives some helpful intuition for the amortised cost of training for a widely used model of that size, but to state the obvious any attempt to apply that intuition to other models is totally reliant on how widely used it is.

DeepSeek have published figures both on training and on inference for DeepSeek's website and API users. I will attempt to consistently refer to the figure for training as "final run training cost" to reflect the fact the number of GPU hours used in experimentation and failed attempts isn't reported. For final run training for DeepSeek-R1:

• 2.788M H800 GPU hours for V3 which serves as the R1 base (see Table 1 in the V3 technical report).
• 0.147M H800 GPU hours for building R1 on top of V3 (see Supplementary Table 4 in the supplementary information for the R1 Nature article).
• Total: 2.935M H800 GPU hours

Now for inference, back in February DeepSeek wrote up details of their inference system giving details of cost of serving, profit margin, and load over a 24h period. So yes, we're extrapolating from this datapoint and assuming it's representative. Given the worldwide inference of DeepSeek R1/V3 is surely much larger (being openly licensed there are many vendors who are serving it), I'm not overly worried about this aspect. Their reported average inference serving infrastructure occupancy is 226.75 nodes (each node containing 8 H800 GPUs), meaning 43536 H800 GPU hours per day. At that rate, it will take ~67.5 days of traffic for the same number of H800 GPU hours to be used for inference as for the final training run.

All this to say, for a widely used model of DeepSeek R1 scale when looking at the cost of inference, accounting for the amortised final run training cost is more likely to be a multiplier of 2x or less rather than something much larger. In terms of energy, this does assume that the power draw of the H800 GPUs while running inference is similar to the draw during training. And to underline again, the reported training cost surely doesn't include experimentation, aborted runs etc.

• 2025-11-29: Initial publication date.

Interoperability makes the web better for everyone, allowing users to have a great experience regardless of their choice of browser. We have been working on MathML Core making across browser engines as part of an agreement with the Sovereign Tech Fund. There are some exciting developments and new features!

Interoperability makes the web better for everyone, allowing users to have a great experience regardless of their choice of browser. We have many standards that shape how the internet should work, drafted from consensus between different engine makers and third parties. While having specs on how everything should function is great, we still need to align the different browser implementations. This can be tricky as all of them have their peculiarities, and not all browsers agree on what is a priority for them. The goal of the Interop program is to select a few important features that all engines will prioritize, so users and editors can finally benefit from them.

A few months ago I joined Igalia's web platform team (and I'm really happy about it!). Thanks to an agreement with the Sovereign Tech Fund, this year we will be working on MathML and other important Interop areas.

This post contains MathML examples. Each formula is represented twice. Your browser renders the left one from the HTML code, while on the right there is a pre-printed SVG as a reference of how it should look. Keep in mind that most of these features are either experimental or have just landed, so you may need the latest version of a browser to view them correctly.

A bit of history

MathML was first published in 1998, and it grew to be a gigantic project that sought to define how mathematical notation should be rendered. However, due to its complexity, the implementations of the browser engines were wildly different and incomplete. This meant that editors could not rely on it, since users would see very different content depending on what they were browsing with.

<math>
  <msubsup>
    <mo>∫</mo>
    <mn>0</mn>
    <mn>1</mn>
  </msubsup>
  <mrow>
    <msup>
      <mi>x</mi>
      <mn>2</mn>
    </msup>
    <mo>+</mo>
    <mn>1</mn>
  </mrow>
</math>

∫ 0 1 x 2 + 1

---

This is why MathML Core was born. It is a small subset of MathML 3 that is feasible to implement in browsers. It is based on the parts of the specification that are used in practice, adding important implementation details and testing.

To illustrate why this is important, Chromium had support for some parts of MathML when it was forked from WebKit. However, it proved to be very difficult to maintain and complete, so it was removed in 2013. My colleague Frédéric Wang led the effort to create a new implementation based on MathML Core, which was shipped in 2023, a huge milestone for the standard.

We are in a very exciting moment in the MathML history, since all three major browser engines have overlapping support. However, there is still work to be done to align the different implementations so they follow the MathML Core specification. The goal is that one could write formulas on a website and have it look the same everywhere (like Wikipedia, which is now transitioning to native MathML instead of prerendered SVGs).

So, what have we been working on?

RTL mirroring

Some scripts are written from right to left, including Arabic. Browsers should be able to correctly render text and math in either direction, making use of the Unicode BiDi specification and the rtlm font feature. However, the existing implementations either didn't support mirroring or had hacky behaviour that didn't work correctly for all cases. Read this explainer that Frédéric made for a great visualization of the differences.

<link rel="stylesheet" href="https://fred-wang.github.io/MathFonts/XITS/mathfonts.css"/>

<math>
  <mrow>
    <mo>{</mo>
    <mfrac>
      <mn>5</mn>
      <mn>6</mn>
    </mfrac>
    <mo>)</mo>
  </mrow>
  <msqrt>
    <mfrac>
      <mn>3</mn>
      <mn>4</mn>
    </mfrac>
  </msqrt>
  <msub displaystyle="true">
    <mo>∲</mo>
    <mi>C</mi>
  </msub>
</math>

<math dir="rtl">
  <mrow>
    <mo>{</mo>
    <mfrac>
      <mn>٥</mn>
      <mn>٦</mn>
    </mfrac>
    <mo>)</mo>
  </mrow>
  <msqrt>
    <mfrac>
      <mn>٣</mn>
      <mn>٤</mn>
    </mfrac>
  </msqrt>
  <msub displaystyle="true">
    <mo>∲</mo>
    <mi>ج</mi>
  </msub>
</math>

{ 5 6 ) 3 4 ∲ C { ٥ ٦ ) ٣ ٤ ∲ ج

---

There are two cases when it comes to mirroring. If there is a corresponding mirrored character (e.g. opening parenthesis to closing parenthesis), it is called character-level mirroring or Unicode BiDi, and the browser just needs to swap one character for the other. Sadly, this doesn't apply to every operator.

Take the contour clockwise integral. If we just mirror the symbol by applying a reflection symmetry about a vertical line, the arrow is suddenly pointing in the other direction, making it counterclockwise. This changes the meaning of the formula!

To avoid this, the rtlm font feature can use glyph-level mirroring to provide a different set of correctly mirrored glyphs. Glyphs plural since a math symbol can have different size variants to accommodate multiple contents. Not only that, when the variants are not enough, there are glyphs for assembling arbitrarily long operators.

<link rel="stylesheet" href="https://fred-wang.github.io/MathFonts/XITS/mathfonts.css"/>

<math>
  <msqrt>
    <mspace height="0.8em" width="0.8em" style="background: tomato"></mspace>
  </msqrt>
  <msqrt>
     <mspace height="1.5em" width="0.8em" style="background: gold"></mspace>
  </msqrt>
  <msqrt>
    <mspace height="2.5em" width="0.8em" style="background: mediumseagreen"></mspace>
  </msqrt>
  <msqrt>
    <mspace height="4.5em" width="0.8em" style="background: cornflowerblue"></mspace>
  </msqrt>
</math>

<math dir="rtl">
  <msqrt>
    <mspace height="0.8em" width="0.8em" style="background: tomato"></mspace>
  </msqrt>
  <msqrt>
     <mspace height="1.5em" width="0.8em" style="background: gold"></mspace>
  </msqrt>
  <msqrt>
    <mspace height="2.5em" width="0.8em" style="background: mediumseagreen"></mspace>
  </msqrt>
  <msqrt>
    <mspace height="4.5em" width="0.8em" style="background: cornflowerblue"></mspace>
  </msqrt>
</math>

---

No browser engine supported glyph-level mirroring for MathML operators, so we had to implement it in all of them. Thankfully harfbuzz, the underlying font rendering library used by Chromium and Firefox, already supported it. WebKit is a work in progress, since there is more complexity because of different ports using different backends. As for character-level mirroring, Chromium and WebKit did it right, but Firefox applied reflection symmetry instead of replacing the correct pair. The changes in Firefox and Chromium are now stable and ready to be used!

math-shift and math-depth

Details are important, especially when rendering complex and layered formulas. One may think that a few pixels do not make that much of a difference. However, when you have multiple levels of nesting, offsets, and multiple elements, a slight change can make everything look ugly at best, wrong at worst.

Enter math-shift: compact. Look at this example from the MathML Core spec:

<math display="block">
  <msqrt>
    <msup>
      <mi>x</mi>
      <mn style="color: mediumseagreen">2</mn>
    </msup>
  </msqrt>
  <mo>≠</mo>
  <msup>
    <mi>x</mi>
    <mn style="color: cornflowerblue">2</mn>
  </msup>
</math>

x 2 ≠ x 2

---

At first glance, you may not see anything too different. But looking closely, the green "2" on the left is a bit lower than then blue one on the right. It is trying to fit under the square root bar. This is what LaTeX calls cramped mode.

Chromium already supported the definition given by MathML Core, so that left Firefox and WebKit, both of which used hardcoded rules for specific cases in C++ objects. MathML Core takes another approach, and incentivizes using CSS styling rules instead.

Another interesting property is math-depth. It is used to make nested elements, such as those inside fractions, scripts or radicals a bit smaller. That way, if you have an exponent of an exponent of an exponent (of an exponent...), each one is displayed a bit tinier than the last.

<math display="block">
  <msup>
    <mi>A</mi>
    <msup>
      <mi style="color: cornflowerblue">A</mi>
      <msup>
        <mi style="color: mediumseagreen">A</mi>
        <mi style="color: tomato">A</mi>
      </msup>
    </msup>
  </msup>
  <mo>+</mo>
  <mroot>
    <mi>A</mi>
    <mi style="color: mediumseagreen">A</mi>
  </mroot>
  <mo>+</mo>
  <mfrac>
    <mrow>
      <mi>A</mi>
      <mo>+</mo>
      <mfrac>
        <mi style="color: cornflowerblue">A</mi>
        <mi style="color: cornflowerblue">A</mi>
      </mfrac>
    </mrow>
    <mi>A</mi>
  </mfrac>
</math>

A A A A + A A + A + A A A

---

In this case, Firefox and Chromium already had compliant implementations, so only WebKit needed to catch up. Support for math-depth and the scriptlevel attribute (which allows to modify this depth) has now landed, while a patch for font-size: math (which sets the size of the element based on its depth) is on the way.

Other work

Rendering unknown elements as mrow

MathML 3 defined 195 elements. MathML Core focuses on about 30, leaving the rest to styling or polyfills. This means deprecating some features that were previously implemented in some browsers, like mfenced, semantics, and maction, as it would be too difficult to make them interoperable right now. To prevent breaking existing content too much, they are rendered like an mrow.

font-family: math

Selecting a good math font is essential for rendering. Stretchy operators, math symbols, and italics are not available with every font, so without one they are presented very poorly. font-family: math is a CSS property that specifies that the content should use a suitable font for mathematics. Previously browsers had a hardcoded list of CSS fallbacks, but now this has been standardized and implemented.

Android doesn't come with a math font installed, so it mixes symbols from different fonts, producing a rather unappealing result:

mathvariant and text-transform: math-auto

Single letter identifiers inside a <mi> tag are treated as variables, and so they should be rendered with fancy italics. This is still supported by MathML Core. However, MathML 3 allows a plethora of transformations using mathvariant, from bold to gothic text. The new spec says that while italic transformation should still happen by default, other text should use the specific Unicode codepoint directly, as it just adds too much complexity for the browser implementation.

text-transform: math-auto is a CSS property applied by default to <mi> elements that enables the italic transformation for them. Setting the new mathvariant attribute to normal will make the text-transform of the element be none, removing the italic styling.

DisplayOperatorMinHeight and Cambria Math

Microsoft made a mistake in Cambria Math, one of the math fonts used in Windows. They switched the DisplayOperatorMinHeight and DelimitedSubFormulaMinHeight, so operators weren't being displayed correctly. Some browsers had a workaround for this, but a more general fix was implemented in harfbuzz, so we removed the workarounds in favour of relying on the upstream library instead.

Animation for math-* properties

When implementing math-shift in Firefox, we noticed that the spec said the new properties are not supposed to be animatable. In the new CSS spec, most properties are defined as animatable (fun!). After some discussion with the MathML Working Group, we decided to change the spec, and we are adding this feature to the browser engines.

x 2

What's next?

Many of these improvements have already shipped, but our work continues on making mathematics more interoperable in browsers. This includes some exciting new features ahead:

• Updates to the operator dictionary: MathML Core revamped the existing list of operators and their default layouts. Additionally, there is a new compact form that removes redundancies.
• More improvements to operator stretching and spacing: There are still some inconsistencies between browsers and some long standing bugs that we would love to tackle.
• Handling positioned elements and forbidding floats in MathML: Like flex or grid, MathML doesn't create floating children for elements with a math display type. However, they can still have out of flow positioned children. At the moment this isn't consistent across browsers and it is something we want to improve.

Working on MathML is very rewarding, specially because of the people that have helped along the way. I'd like to specially thank my colleague @fredw, reviewers from Mozilla, Apple and Google, and the W3C Math Working Group. Also @delan for reviewing the first draft of this post.

We are very grateful to the Sovereign Tech Fund for supporting this work!

Update on what happened in WebKit in the week from November 17 to November 24.

In this week's rendition, the WebView snapshot API was enabled on the WPE port, further progress on the Temporal and Trusted Types implementations, and the release of WebKitGTK and WPE WebKit 2.50.2.

Cross-Port 🐱

JavaScriptCore 🐟

Releases 📦️

XDC 2025 happened at the end of September, beginning of October this year, in Kuppelsaal, the historic TU Wien building in Vienna. XDC, The X.Org Developer’s Conference, is truly the premier gathering for open-source graphics development. The atmosphere was, as always, highly collaborative and packed with experts across the entire stack.

I was thrilled to present, together with my workmate Ella Stanforth, on the progress we have made in enhancing the Raspberry Pi GPU driver stack. Representing the broader Igalia Graphics Team that work on this GPU, Ella and I detailed the strides we have made in the OpenGL driver, though part of the improvements affect also the Vulkan driver.

The presentation was divided in two parts. In the first one, we talked about the new features that we were implementing, or are under implementation, mainly to make the driver more closely aligned with OpenGL 3.2. Key features explained were 16-bit Normalized Format support, Robust Context support, and Seamless cubemap implementation.

Beyond these core OpenGL updates, we also highlighted other features, such as NIR printf support, framebuffer fetch or dual source blend, which is important for some game emulators.

The second part was focused on specific work done to improve the performance. Here, we started with different traces from the popular GFXBench application, and explained the main improvements done throughout the year, with a look at how much each of these changes improved the performance for each of the benchmarks (or in average).

At the end, for some benchmarks we nearly doubled the performance compared to last year. I won’t explain here each of the changes done, But I encourage the reader to watch the talk, which is already available.

For those that prefer to check the slides instead of the full video, you can view them here:

Outside of the technical track, the venue’s location provided some excellent down time opportunities to have lunch at different nearby places. I need to highlight here one that I really enjoyed: An’s Kitchen Karlsplatz. This cozy Vietnamese street food spot quickly became one of my favourite places, and I went there a couple of times.

On the last day, I also had the opportunity to visit some of the most recomendable sightseeings spots in Vienna. Of course, one needs more than a half-day to do a proper visit, but at least it helps to spark an interest to write it down to pay a full visit to the city.

Meanwhile, I would like to thank all the conference organizers, as well as all the attendees, and I look forward to see them again.

This blog post summarizes a talk given by Mikhail R. Gadelha at the RISC-V Summit North America 2025.

You can also watch the presentation here. And download the full presentation here.

Introduction #

In this post, I will walk through the results of a 10-month RISE project we completed in September, focused on improving the performance of the LLVM toolchain for RISC-V.

Since the moment I originally submitted this talk, we have actually squeezed out a bit more performance: what used to be a 15% speed-up is now up to 16% on SPEC CPU® 2017. Small change, but still measurable.

The project targeted the Banana Pi BPI-F3 board, which uses the SpacemiT X60: an in-order, 8-core RISC-V processor supporting the RVA22U64 profile and RVV 1.0, 256-bit vectors.

Our high-level goal was straightforward: to reduce the performance gap between LLVM and GCC for RISC-V. However, there wasn't (and still isn't) one single fix; LLVM is an extremely active codebase where improvements and regressions happen constantly. Instead, we focused on three major contributions:

1. A full scheduling model for the SpacemiT X60.

2. Improvements to vectorization across calls.

3. IPRA support for the RISC-V backend.

Let's walk through each contribution.

1. SpacemiT-X60 Scheduling Model #

By far, our main contribution to the LLVM project was the scheduling model for the Spacemit-X60, but before we delve deeper into the changes, let's understand what a scheduling model is.

Instruction scheduling directly impacts performance, especially on in-order processors. Without accurate instruction latencies, and instruction resource usage, the compiler can make poor scheduling decisions.

In our slides, there is an example where:

load -> uses ft0
fadd -> depends on ft0
fmul -> independent

The above is a possible naive code generated by the compiler, which has latency(load + fadd + fmul). Thanks to an accurate scheduling model the compiler can reason that it is better to emit the following code with latency(max(load, fmul) + fadd).

load -> uses ft0
fmul -> independent of preceding load
fadd -> depends on load

This was just an illustrative example, not something LLVM typically emits, but it demonstrates how missing scheduling information leads to unnecessary stalls.

The biggest piece of the work was to actually collect the data on every single instruction supported by the board. To build a correct model, we:

• wrote custom microbenchmarks to measure latency for every instruction.
• used throughput data from camel-cdr's RVV benchmark results.
• tracked all combinations of LMUL × SEW, which leads to:
  • 201 scalar instructions.
  • 82 FP instructions.
  • 9185 RVV "instructions" (combinations).

This resulted in a very, very large spreadsheet, and it took some time to analyse and then produce a series of upstream patches to reflect that data in LLVM.

With scheduling enabled on RVA22U64 (scalar-only), we got up to 16.8% execution time improvements (538.imagick_r) and no regressions. The combined results show a -4.75% geometric mean improvement on execution time.

When we enable vectors (RVA22U64_V), we got up to 16% improvement (508.namd_r) and no regressions. The combined results show a -3.28% geometric mean improvement on execution time.

One interesting result: scheduling nearly eliminated the gap between scalar and vector configurations on the x60; only one SPEC benchmark (x264) still favored the vectorized build.

This shows there may be more work in improving our ability to find profitable vectorization.

2. Improving Vectorization Across Calls #

During benchmarking, we found strange cases where scalar code was faster than vectorized code. The root cause: register spills, especially around function call boundaries.

Example: 544.nab_r #

In this function, the SLP vectorizer would look only at the basic blocks performing loads/stores, and ignore the blocks containing function calls in-between blocks.

Because those call blocks weren't considered when computing profitability, the vectorizer assumed vectorization was cheap, but in reality, it caused expensive vector register spills.

We modified SLP to walk all blocks in the region and estimate the cost properly. This helped: +9.9% faster on 544.nab_r, but hurt compilation time: +6.9% slower on 502.gcc_r. After discussion, Alexey Bataev (SLP vectorizer maintainer) created a refined version that fixed the issue and avoided compilation slowdowns. This shows how the open-source community is important and that collaboration can take us further.

With the refined patch, we got up to 11.9% improvement (544.nab_r), no regressions, with negligible compile-time regressions. The combined results show a -3.28% geometric mean improvement on execution time.

Note that we only show results for RVA22U64_V, because the regression only happened when vectors were enabled. Now, the execution time is on par or better than the scalar-only execution time.

3. IPRA Support (Inter-Procedural Register Allocation) #

IPRA tracks which registers are actually used across call boundaries. Without it, LLVM spills registers conservatively, including registers that aren’t truly live.

Let's consider the illustrative example above, and assume s0 and s1 are not live in this function. LLVM will still save and restore these registers when IPRA is disabled. By enabling IPRA, we reduce register pressure and create shorter function prologues and epilogues.

With IPRA enabled on RVA22U64, we got up to 3.2% execution time improvements (538.lbm_r) and no regressions. The combined results show a -0.50% geometric mean improvement on execution time.

When we enable vectors (RVA22U64_V), we got up to 3.4% improvement (508.deepsjeng_r) and no regressions. The combined results show a -0.39% geometric mean improvement on execution time.

Small but consistent wins; however, IPRA can’t be enabled by default due to an open bug, though it does not affect SPEC.

LLVM vs GCC #

This comparison isn’t perfectly apples-to-apples because LLVM has X60-specific scheduling and GCC does not. Still, it's useful to see progress.

On RVA22U64, LLVM can be 27% faster on some benchmarks but also 27% slower on others (notably x264).

The results when we enable vectors (RVA22U64_V) are similar: LLVM can be 8% faster in some benchmarks but also 9.2% slower in others.

My colleague Luke Lau is currently investigating these results to try to address the cases where we are slower.

What we learned #

1. Adding a scheduling model can give meaningful performance improvements, especially for in-order cores.

2. A default in-order scheduling model may be needed. Other backends do this already, and we have a PR open for that at PR#167008.

3. Many contributions don’t show results until the entire system comes together. When the project started, I spent some time modeling individual instruction, but only when the full model was integrated did we see actual improvements.

4. Vectorization must be tuned carefully; incorrect cost modeling leads to regressions.

Thank you #

Thank you for reading!

The GStreamer Conference is an annual gathering that brings together developers, contributors, and users of the GStreamer multimedia framework. It serves as a platform for sharing knowledge, discussing the latest advancements, and fostering collaboration within the open-source multimedia community.

This year’s conference was held in London at the impressive Barbican Center, located within the Barbican Estate, a residential complex rebuilt after World War II in the brutalist architectural style.

It was a pleasure to meet in person with the colleagues, from different companies and backgrounds, I usually collaborate with remotely, sharing and discussing their projects and results.

Recently, UbiCast, which generously records and streams the conference, has uploaded all talks of this year conference to their site, exclusively for the GStreamer Conference.

In this blog post, I’ll share the talks delivered by my fellow igalians:

Animate Your Subtimelines in GES #

GstVA and GStreamer-VAAPI updates #

Time Remapping and GES: Implementation Details and Latest Updates #

soothe: a proposal for encoder testing #

GstWebRTC in WebKit, current status & plans #

VVC/H.266 in GStreamer #

Video Reshaping with Skia #

Vulkan Video: pipeline update #

Following the GStreamer Conference, we hosted our Autumn hackfest at Amazon’s offices in the City of London. This time I worked on GStreamer Vulkan.

This year, two other conferences typically held in the US, FOMS and Demuxed, also took place in London. I attended FOMS, where I discovered the vibrant MOQ project.

Finally, I’d like to thank Centricular for organizing the event, especially Tim-Philipp Müller, and even more particularly Igalia for sponsoring it and allowing me to participate in this project that’s close to my heart.

And that’s all, mates. Cheers!

Update on what happened in WebKit in the week from November 10 to November 17.

This week's update is composed of a new CStringView internal API, more MathML progress with the implementation of the "scriptlevel" attribute, the removal of the Flatpak-based SDK, and the maintanance update of WPEBackend-fdo.

Cross-Port 🐱

Releases 📦️

Infrastructure 🏗️

Update on what happened in WebKit in the week from November 3 to November 10.

This week brought a hodgepodge of fixes in Temporal and multimedia, a small addition to the public API in preparation for future work, plus advances in WebExtensions, WebXR, and Android support.

Cross-Port 🐱

Multimedia 🎥

JavaScriptCore 🐟

WPE WebKit 📟

WPE Android ↗ 🤖

Community & Events 🤝

This was the first year I attended Kernel Recipes and I have nothing but say how much I enjoyed it and how grateful I’m for the opportunity to talk more about kworkflow to very experienced kernel developers. What I mostly like about Kernel Recipes is its intimate format, with only one track and many moments to get closer to experts and people that you commonly talk online during your whole year.

In the beginning of this year, I gave the talk Don’t let your motivation go, save time with kworkflow at FOSDEM, introducing kworkflow to a more diversified audience, with different levels of involvement in the Linux kernel development.

At this year’s Kernel Recipes I presented the second talk of the first day: Kworkflow - mix & match kernel recipes end-to-end.

The Kernel Recipes audience is a bit different from FOSDEM, with mostly long-term kernel developers, so I decided to just go directly to the point. I showed kworkflow being part of the daily life of a typical kernel developer from the local setup to install a custom kernel in different target machines to the point of sending and applying patches to/from the mailing list. In short, I showed how to mix and match kernel workflow recipes end-to-end.

As I was a bit fast when showing some features during my presentation, in this blog post I explain each slide from my speaker notes. You can see a summary of this presentation in the Kernel Recipe Live Blog Day 1: morning.

Introduction

Hi, I’m Melissa Wen from Igalia. As we already started sharing kernel recipes and even more is coming in the next three days, in this presentation I’ll talk about kworkflow: a cookbook to mix & match kernel recipes end-to-end.

This is my first time attending Kernel Recipes, so lemme introduce myself briefly.

• As I said, I work for Igalia, I work mostly on kernel GPU drivers in the DRM subsystem.
• In the past, I co-maintained VKMS and the v3d driver. Nowadays I focus on the AMD display driver, mostly for the Steam Deck.
• Besides code, I contribute to the Linux kernel by mentoring several newcomers in Outreachy, Google Summer of Code and Igalia Coding Experience. Also, by documenting and tooling the kernel.

And what’s this cookbook called kworkflow?

Kworkflow (kw)

Kworkflow is a tool created by Rodrigo Siqueira, my colleague at Igalia. It’s a single platform that combines software and tools to:

• optimize your kernel development workflow;
• reduce time spent in repetitive tasks;
• standardize best practices;
• ensure that deployment data flows smoothly and reliably between different kernel workflows;

It’s mostly done by volunteers, kernel developers using their spare time. Its features cover real use cases according to kernel developer needs.

Basically it’s mixing and matching the daily life of a typical kernel developer with kernel workflow recipes with some secret sauces.

First recipe: A good GPU driver for my AMD laptop

So, it’s time to start the first recipe: A good GPU driver for my AMD laptop.

Before starting any recipe we need to check the necessary ingredients and tools. So, let’s check what you have at home.

With kworkflow, you can use:

• kw device: to get information about the target machine, such as: CPU model, kernel version, distribution, GPU model,

• kw remote: to set the address of this machine for remote access

• kw config: you can configure kw with kw config. With this command you can basically select the tools, flags and preferences that kw will use to build and deploy a custom kernel in a target machine. You can also define recipients of your patches when sending it using kw send-patch. I’ll explain more about each feature later in this presentation.

• kw kernel-config manager (or just kw k): to fetch the kernel .config file from a given machine, store multiple .config files, list and retrieve them according to your needs.

Now, with all ingredients and tools selected and well portioned, follow the right steps to prepare your custom kernel!

First step: Mix ingredients with kw build or just kw b

• kw b and its options wrap many routines of compiling a custom kernel.
  • You can run kw b -i to check the name and kernel version and the number of modules that will be compiled and kw b --menu to change kernel configurations.
  • You can also pre-configure compiling preferences in kw config regarding kernel building. For example, target architecture, the name of the generated kernel image, if you need to cross-compile this kernel for a different system and which tool to use for it, setting different warning levels, compiling with CFlags, etc.
  • Then you can just run kw b to compile the custom kernel for a target machine.

Second step: Bake it with kw deploy or just kw d

After compiling the custom kernel, we want to install it in the target machine. Check the name of the custom kernel built: 6.17.0-rc6 and with kw s SSH access the target machine and see it’s running the kernel from the Debian distribution 6.16.7+deb14-amd64.

As with building settings, you can also pre-configure some deployment settings, such as compression type, path to device tree binaries, target machine (remote, local, vm), if you want to reboot the target machine just after deploying your custom kernel, and if you want to boot in the custom kernel when restarting the system after deployment.

If you didn’t pre-configured some options, you can still customize as a command option, for example: kw d --reboot will reboot the system after deployment, even if I didn’t set this in my preference.

With just running kw d --reboot I have installed the kernel in a given target machine and rebooted it. So when accessing the system again I can see it was booted in my custom kernel.

Third step: Time to taste with kw debug

• kw debug wraps many tools for validating a kernel in a target machine. We can log basic dmesg info but also tracking events and ftrace.
  • With kw debug --dmesg --history we can grab the full dmesg log from a remote machine, if you use the --follow option, you will monitor dmesg outputs. You can also run a command with kw debug --dmesg --cmd="<my command>" and just collect the dmesg output related to this specific execution period.
  • In the example, I’ll just unload the amdgpu driver. I use kw drm --gui-off to drop the graphical interface and release the amdgpu for unloading it. So I run kw debug --dmesg --cmd="modprobe -r amdgpu" to unload the amdgpu driver, but it fails and I couldn’t unload it.

Cooking Problems

Oh no! That custom kernel isn’t tasting good. Don’t worry, as in many recipes preparations, we can search on the internet to find suggestions on how to make it tasteful, alternative ingredients and other flavours according to your taste.

With kw patch-hub you can search on the lore kernel mailing list for possible patches that can fix your kernel issue. You can navigate in the mailing lists, check series, bookmark it if you find it relevant and apply it in your local kernel tree, creating a different branch for tasting… oops, for testing. In this example, I’m opening the amd-gfx mailing list where I can find contributions related to the AMD GPU driver, bookmark and/or just apply the series to my work tree and with kw bd I can compile & install the custom kernel with this possible bug fix in one shot.

As I changed my kw config to reboot after deployment, I just need to wait for the system to boot to try again unloading the amdgpu driver with kw debug --dmesg --cm=modprobe -r amdgpu. From the dmesg output retrieved by kw for this command, the driver was unloaded, the problem is fixed by this series and the kernel tastes good now.

If I’m satisfied with the solution, I can even use kw patch-hub to access the bookmarked series and marking the checkbox that will reply the patch thread with a Reviewed-by tag for me.

Second Recipe: Raspberry Pi 4 with Upstream Kernel

As in all recipes, we need ingredients and tools, but with kworkflow you can get everything set as when changing scenarios in a TV show. We can use kw env to change to a different environment with all kw and kernel configuration set and also with the latest compiled kernel cached.

I was preparing the first recipe for a x86 AMD laptop and with kw env --use RPI_64 I use the same worktree but moved to a different kernel workflow, now for Raspberry Pi 4 64 bits. The previous compiled kernel 6.17.0-rc6-mainline+ is there with 1266 modules, not the 6.17.0-rc6 kernel with 285 modules that I just built&deployed. kw build settings are also different, now I’m targeting a arm64 architecture with a cross-compiled kernel using aarch64-linu-gnu- cross-compilation tool and my kernel image calls kernel8 now.

If you didn’t plan for this recipe in advance, don’t worry. You can create a new environment with kw env --create RPI_64_V2 and run kw init --template to start preparing your kernel recipe with the mirepoix ready.

I mean, with the basic ingredients already cut…

I mean, with the kw configuration set from a template.

And you can use kw remote to set the IP address of your target machine and kw kernel-config-manager to fetch/retrieve the .config file from your target machine. So just run kw bd to compile and install a upstream kernel for Raspberry Pi 4.

Third Recipe: The Mainline Kernel Ringing on my Steam Deck (Live Demo)

Let’s show you how easy is to build, install and test a custom kernel for Steam Deck with Kworkflow. It’s a live demo, but I also recorded it because I know the risks I’m exposed to and something can go very wrong just because of reasons :)

Report: how was the live demo

For this live demo, I took my OLED Steam Deck to the stage. I explained that, if I boot mainline kernel on this device, there is no audio. So I turned it on and booted the mainline kernel I’ve installed beforehand. It was clear that there was no typical Steam Deck startup audio when the system was loaded.

As I started the demo in the kw environment for Raspberry Pi 4, I first moved to another environment previously used for Steam Deck. In this STEAMDECK environment, the mainline kernel was already compiled and cached, and all settings for accessing the target machine, compiling and installing a custom kernel were retrieved automatically.

My live demo followed these steps:

1. With kw env --use STEAMDECK, switch to a kworkflow environment for Steam Deck kernel development.

2. With kw b -i, shows that kw will compile and install a kernel with 285 modules named 6.17.0-rc6-mainline-for-deck.

3. Run kw config to show that, in this environment, kw configuration changes to x86 architecture and without cross-compilation.

4. Run kw device to display information about the Steam Deck device, i.e. the target machine. It also proves that the remote access - user and IP - for this Steam Deck was already configured when using the STEAMDECK environment, as expected.

5. Using git am, as usual, apply a hot fix on top of the mainline kernel. This hot fix makes the audio play again on Steam Deck.

6. With kw b, build the kernel with the audio change. It will be fast because we are only compiling the affected files since everything was previously done and cached. Compiled kernel, kw configuration and kernel configuration is retrieved by just moving to the “STEAMDECK” environment.

7. Run kw d --force --reboot to deploy the new custom kernel to the target machine. The --force option enables us to install the mainline kernel even if mkinitcpio complains about missing support for downstream packages when generating initramfs. The --reboot option makes the device reboot the Steam Deck automatically, just after the deployment completion.

8. After finishing deployment, the Steam Deck will reboot on the new custom kernel version and made a clear resonant or vibrating sound. [Hopefully]

Finally, I showed to the audience that, if I wanted to send this patch upstream, I just needed to run kw send-patch and kw would automatically add subsystem maintainers, reviewers and mailing lists for the affected files as recipients, and send the patch to the upstream community assessment. As I didn’t want to create unnecessary noise, I just did a dry-run with kw send-patch -s --simulate to explain how it looks.

What else can kworkflow already mix & match?

In this presentation, I showed that kworkflow supported different kernel development workflows, i.e., multiple distributions, different bootloaders and architectures, different target machines, different debugging tools and automatize your kernel development routines best practices, from development environment setup and verifying a custom kernel in bare-metal to sending contributions upstream following the contributions-by-e-mail structure. I exemplified it with three different target machines: my ordinary x86 AMD laptop with Debian, Raspberry Pi 4 with arm64 Raspbian (cross-compilation) and the Steam Deck with SteamOS (x86 Arch-based OS). Besides those distributions, Kworkflow also supports Ubuntu, Fedora and PopOS.

Now it’s your turn: Do you have any secret recipes to share? Please share with us via kworkflow.

Useful links

• Talk Recording of Kworkflow at Kernel Recipes 2025 on Igalia’s Channel
• Talk Abstract, Recording and Slide Deck of Kworkflow at Kernel Recipes 2025 on Kernel Recipes Website
• Talk Slide Deck for Download with some Videos instead of GIFs

Update on what happened in WebKit in the week from October 27 to November 3.

A calmer week this time! This week we have the GTK and WPE ports implementing the RunLoopObserver infrastructure, which enables more sophisticated scheduling in WebKit Linux ports, as well as more information in webkit://gpu. On the Trusted Types front, the timing of check was changed to align with spec changes.

Cross-Port 🐱

Graphics 🖼️

Previously on meyerweb, I crawled through a way to turn parenthetical comments into sidenotes, which I called “asidenotes”.  As a recap, these are inline asides in parentheses, which is something I like to do.  The constraints are that the text has to start inline, with its enclosing parentheses as part of the static content, so that the parentheses are present if CSS isn’t applied, but should lose those parentheses when turned into asidenotes, while also adding a sentence-terminating period when needed.

At the end of that post, I said I wouldn’t use the technique I developed, because the markup was too cluttered and unwieldy, and there were failure states that CSS alone couldn’t handle.  So what can we do instead?  Extend HTML to do things automatically!

If you’ve read my old post “Blinded By the DOM Light”, you can probably guess how this will go.  Basically, we can write a little bit of JavaScript to take an invented element and Do Things To It.  What things?  Anything JavaScript makes possible.

So first, we need an element, one with a hyphen in the middle of its name (because all custom elements require an interior hyphen, similar to how all custom properties and most custom identifiers in CSS require two leading dashes).  Something like:

<aside-note>(actual text content)</aside-note>

Okay, great!  Thanks to HTML’s permissive handling of unrecognized elements, this completely new element will be essentially treated like a <span> in older browsers.  In newer browsers, we can massage it.

class asideNote extends HTMLElement {
	connectedCallback() {
		let marker = document.createElement('sup');
		marker.classList.add('asidenote-marker');
		this.after(marker);
	}
}
customElements.define("aside-note",asideNote);

With this in place, whenever a supporting browser encounters an <aside-note> element, it will run the JS above.  Right now, what that does is insert a <sup> element just after the <aside-note>.

“Whoa, wait a minute”, I thought to myself at this point. “There will be browsers (mostly older browser versions) that understand custom elements, but don’t support anchor positioning.  I should only run this JS if the browser can position with anchors, because I don’t want to needlessly clutter the DOM.  I need an @supports query, except in JS!” And wouldn’t you know it, such things do exist.

class asideNote extends HTMLElement {
	connectedCallback() {
		if (CSS.supports('bottom','anchor(top)')) {
			let marker = document.createElement('sup');
			marker.classList.add('asidenote-marker');
			this.after(marker);
		}
	}
}

That will yield the following DOM structure:

<aside-note>(and brower versions)</aside-note><sup></sup>

That’s all we need to generate some markers and do some positioning, as was done in my previous post.  To wit:

@supports (anchor-name: --main) {
	#thoughts {
		anchor-name: --main;
	}
	#thoughts article {
		counter-reset: asidenotes;
	}
	#thoughts article sup {
		font-size: 89%;
		line-height: 0.5;
		color: inherit;
		text-decoration: none;
	}
	#thoughts article aside-note::after,
	#thoughts article aside-note + sup::before {
		content: counter(asidenotes);
	}
	#thoughts article aside-note {
		counter-increment: asidenotes;
		position: absolute;
		anchor-name: --asidenote;
		top: max(anchor(top), calc(anchor(--asidenote bottom, 0px) + 0.67em));
		bottom: auto;
		left: calc(anchor(--main right) + 4em);
		max-width: 23em;
		margin-block: 0.15em 0;
		text-wrap: balance;
		text-indent: 0;
		font-size: 89%;
		line-height: 1.25;
		list-style: none;
	}
	#thoughts article aside-note::before {
		content: counter(asidenotes);
		position: absolute;
		top: -0.4em;
		right: calc(100% + 0.25em);
	}
	#thoughts article aside-note::first-letter {
		text-transform: uppercase;
	}
}

I went through a lot of that CSS in the previous post, so jump over there to get details on what all that means if the above has you agog.  I did add a few bits of text styling like an explicit line height and slight size reduction, and changed all the asidenote classes there to aside-note elements here, but nothing is different with the positioning and such.

Let’s go back to the JavaScript, where we can strip off the leading and trailing parentheses with relative ease.

class asideNote extends HTMLElement {
	connectedCallback() {
		if (CSS.supports('bottom','anchor(top)')) {
			let marker = document.createElement('sup');
			marker.classList.add('asidenote-marker');
			this.after(marker);
			let inner = this.innerText;
			if (inner.slice(0,1) == '(' && inner.slice(-1) == ')') {
				inner = inner.slice(1,inner.length-1);}
			this.innerText = inner;
		}
	}
}

This code looks at the innerText of the asidenote, checks to see if it both begins and ends with parentheses (which all asidenotes should!), and then if so, it strips them out of the text and sets the <aside-note>’s innerText to be that stripped string.  I decided to set it up so that the stripping only happens if there are balanced parentheses because if there aren’t, I’ll see that in the post preview and fix it before publishing.

I still haven’t added the full stop at the end of the asidenotes, nor have I accounted for asidenotes that end in punctuation, so let’s add in a little bit more code to check for and do that:

class asideNote extends HTMLElement {
	connectedCallback() {
		if (CSS.supports('bottom','anchor(top)')) {
			let marker = document.createElement('sup');
			marker.classList.add('asidenote-marker');
			this.after(marker);
			let inner = this.innerText;
			if (inner.slice(0,1) == '(' && inner.slice(-1) == ')') {
				inner = inner.slice(1,inner.length-1);}
			if (!isLastCharSpecial(inner)) {
				inner += '.';}
			this.innerText = inner;
		}
	}
}
function isLastCharSpecial(str) {
	const punctuationRegex = /[!/?/‽/.\\]/;
	return punctuationRegex.test(str.slice(-1));
}

And with that, there is really only one more point of concern: what will happen to my asidenotes in mobile contexts?  Probably be positioned just offscreen, creating a horizontal scrollbar or just cutting off the content completely.  Thus, I don’t just need a supports query in my JS.  I also need a media query.  It’s a good thing those also exist!

class asideNote extends HTMLElement {
	connectedCallback() {
		if (CSS.supports('bottom','anchor(top)') &&
			window.matchMedia('(width >= 65em)').matches) {
			let marker = document.createElement('sup');
			marker.classList.add('asidenote-marker');
			this.after(marker);

Adding that window.matchMedia to the if statement’s test means all the DOM and content massaging will be done only if the browser understands anchor positioning and the window width is above 65 ems, which is my site’s first mobile media breakpoint that would cause real layout problems.  Otherwise, it will leave the asidenote content embedded and fully parenthetical.  Your breakpoint will very likely differ, but the principle still holds.

The one thing about this JS is that the media query only happens when the custom element is set up, same as the support query.  There are ways to watch for changes to the media environment due to things like window resizes, but I’m not going to use them here.  I probably should, but I’m still not going to.

So: will I use this version of asidenotes on meyerweb?  I might, Rabbit, I might.  I mean, I’m already using them in this post, so it seems like I should just add the JS to my blog templates and the CSS to my stylesheets so I can keep doing this sort of thing going forward.  Any objections?  Let’s hear ’em!

Have something to say to all that? You can add a comment to the post, or email Eric directly.

Update on what happened in WebKit in the week from October 21 to October 28.

This week has again seen a spike in activity related to WebXR and graphics performance improvements. Additionally, we got in some MathML additions, a fix for hue interpolation, a fix for WebDriver screenshots, development releases, and a blog post about memory profiling.

Cross-Port 🐱

Graphics 🖼️

WPE WebKit 📟

WPE Platform API 🧩

Releases 📦️

Community & Events 🤝

It’s not really a secret I have a thing for sidenotes, and thus for CSS anchored positioning.  But a thing I realized about myself is that most of my sidenotes are likely to be tiny asides commenting on the main throughline of the text, as opposed to bibliographic references or other things that usually become actual footnotes or endnotes.  The things I would sidenote currently get written as parenthetical inline comments (you know, like this).  Asidenotes, if you will.

Once I had realized that, I wondered: could I set up a way to turn those parenthetical asides into asidenotes in supporting browsers, using only HTML and CSS?  As it turns out, yes, though not in a way I would actually use.  In fact, the thing I eventually arrived at is pretty terrible.

Okay, allow me to explain.

To be crystal clear about this, here’s how I would want one of these parenthetical asides to be rendered in browsers that don’t support anchored positioning, and then how to render in those that do (which are, as I write this, recent Chromium browsers and Safari Technology Previews; see theanchor() MDN page for the latest):

My thinking is, the parenthetical text should be the base state, with some HTML to flag the bit that’s an asidenote, and then CSS is applied in supporting browsers to lay out the text as an asidenote.  There is a marker pair to allow an unambiguous association between the two, which is tricky, because that marker should not be in the base text, but should appear when styled.

I thought for a minute that I would wrap these little notes in <aside>s, but quickly realized that would probably be a bad idea for accessibility and other reasons.  I mean, I could use CSS to cast the <aside> to an inline box instead of its browser-default block box, but I’d need to label each one separately, be very careful with roles, and so on and so on.  It was just the wrong tool, it seemed to me.  (Feel free to disagree with me in the comments!)

So, I started with this:

<span class="asidenote">(Feel free to disagree with me in the comments!)</span>

That wasn’t going to be enough, though, because I can certainly position this <span>, but there’s nothing available to leave a maker behind when I do!  Given the intended result, then, there needs to be something in the not-positioned text that serves in that role (by which I mean a utility role, not an ARIA role).  Here’s where my mind went:

<span class="asidenote">(by which I mean a utility role, not an ARIA role)</span><sup></sup>

The added <sup> is what will contain the marker text, like 1 or a or whatever.

This seemed like it was the minimum viable structure, so I started writing some styles.  These asidenotes would be used in my posts, and I’d want the marker counters to reset with each blog post, so I built the selectors accordingly:

@supports not (anchor-name: --main) {
	#thoughts article .asidenote + sup {
		display: none;
	}
} 
@supports (anchor-name: --main) {
	#thoughts {
		anchor-name: --main;
	}
	#thoughts article {
		counter-reset: asidenotes;
	}
	#thoughts article :is(.asidenote::before, .asidenote + sup::before) {
		content: counter(asidenotes);
	}
}

So far, I’ve set a named anchor on the <main> element (which has an id of thoughts) that encloses a page’s content, reset a counter on each <article>, and inserted that counter as the ::before content for both the asidenotes’ <span>s and the <sup>s that follow them.  That done, it’s time to actually position the asidenotes:

	#thoughts article .asidenote {
		counter-increment: asidenotes;
		position: absolute;
		anchor-name: --sidenote;
		top: max(calc(anchor(--sidenote bottom, 0px) + 0.67em), anchor(top));
		bottom: auto;
		left: calc(anchor(--main right) + 4em);
		max-width: 23em;
		margin-block: 0.15em 0;
		text-wrap: balance;
		text-indent: 0;
	}

Here, each class="asidenote" element increments the asidenotes counter by one, and then the asidenote is absolutely positioned so its top is placed at the larger value of two-thirds of an em below the bottom of the previous asidenote, if any; or else the top of its implicit anchor, which, because I didn’t set an explicit named anchor for it in this case, seems to be the place it would have occupied in the normal flow of the text.  This latter bit is long-standing behavior in absolute positioning of inline elements, so it makes sense.  I’m just not sure it fully conforms to the specification, though it’s particularly hard for me to tell in this case.

Moving on!  The left edge of the asidenote is set 4em to the right of the right edge of --main and then some formatting stuff is done to keep it balanced and nicely sized for its context.  Some of you will already have seen what’s going to happen here.

Yep, the parentheses came right along with the text, and in general the whole thing looks a little odd.  I could certainly argue that these are acceptable design choices, but it’s not what I want to see.  I want the parentheses to go away when laid out as a asidenote, and also capitalize the first letter if it isn’t already, plus close out the text with a full stop.

And this is where the whole thing tipped over into “I don’t love this” territory.  I can certainly add bits of text before and after an element’s content with pseudo-elements, but I can’t subtract bits of text (not without JavaScript, anyway).  The best I can do is suppress their display, but for that, I need structure.  So I went this route with the markup and CSS:

<span class="asidenote"><span>(</span>by which I mean a utility role, not an ARIA role<span>)</span></span><sup></sup>

	#thoughts article .asidenote span:is(:first-child, :last-child) {
		display: none;
	}

I could have used shorter elements like <b> or <i>, and then styled them to look normal, but nah.  I don’t love the clutter, but <span> makes more sense here.

With those parentheses gone, I can uppercase the the first visible letter and full-stop the end of each asidenote like so:

	#thoughts article .asidenote::first-letter {
		text-transform: uppercase;
	}
	#thoughts article .asidenote::after {
		content: ".";
	}

Then I do a little styling of the asidenote’s marker:

	#thoughts article .asidenote::before {
		content: counter(asidenotes);
		position: absolute;
		top: -0.4em;
		right: calc(100% + 0.25em);
	}
} /* closes out the @supports block */

…and that’s more or less it (okay, yes, there are a few other tweaks to the markers and their sizes and line heights and asidenote text size and blah blah blah, but let’s not clutter up the main points by slogging through all that).  With that, I get little asides that are parenthetical in the base text, albeit with a bunch of invisible-to-the-user markup clutter, that will be progressively enhanced into full asidenotes where able.

There’s an extra usage trap here, as well: if I always generate a full stop at the end, it means I should never end my asidenotes with a question mark, exclamation point, interrobang, or other sentence-ending character.  But those are things I like to do!

So, will I use this on meyerweb?  Heck to the no.  The markup clutter is much more annoying than the benefit, it fumbles on some pretty basic use cases, and I don’t really want to go to the lengths of creating weird bespoke text macros  —  or worse, try to fork and extend a local Markdown parser to add some weird bespoke text pattern  —  just to make this work.  If CSS had a character selector that let me turn off the parentheses without needing the extras <span>s, and some kind of outside-the-element generated content, then maybe yes.  Otherwise, no, this is not how I’d do it, at least outside this post.  At the very least, some JavaScript is needed to remove bits of text and decide whether to append the full stop.

Given that JS is needed, how would I do it?  With custom elements and the Light DOM, which I’ll document in the next post.  Stay tuned!

Have something to say to all that? You can add a comment to the post, or email Eric directly.

One of the main constraints that embedded platforms impose on the browsers is a very limited memory. Combined with the fact that embedded web applications tend to run actively for days, weeks, or even longer, it’s not hard to imagine how important the proper memory management within the browser engine is in such use cases. In fact, WebKit and WPE in particular receive numerous memory-related fixes and improvements every year. Before making any changes, however, the areas to fix/improve need to be narrowed down first. Like any C++ application, WebKit memory can be profiled using a variety of industry-standard tools. Although such well-known tools are really useful in the majority of use cases, they have their limits that manifest themselves when applied on production-grade embedded systems in conjunction with long-running web applications. In such cases, a very useful tool is a debug-only feature of WebKit itself called malloc heap breakdown, which this article describes.

Industry-standard memory profilers #

When it comes to profiling memory of applications on linux systems, the 2 outstanding tools used usually are Massif (Valgrind) and Heaptrack.

Massif (Valgrind) #

Massif is a heap profiler that comes as part of the Valgrind suite. As its documentation states:

It measures how much heap memory your program uses. This includes both the useful space, and the extra bytes allocated for book-keeping and alignment purposes. It can also measure the size of your program’s stack(s), although it does not do so by default.

Using Massif with WebKit is very straightforward and boils down to a single command:

Malloc=1 valgrind --tool=massif --trace-children=yes WebKitBuild/GTK/Debug/bin/MiniBrowser '<URL>'

• The Malloc=1 environment variable set above is necessary to instruct WebKit to enable debug heaps that use the system malloc allocator.

Given some results are generated, the memory usage over time can be visualized using massif-visualizer utility. An example of such a visualization is presented in the image below:

While Massif has been widely adopted and used for many years now, from the very beginning, it suffered from a few significant downsides.

First of all, the way Massif instruments the profiled application introduces significant overhead that may slow down the application up to 2 orders of magnitude. In some cases, such overhead makes it simply unusable.