In Part 2 I explained how ARM32 has two ISAs available (ARM and Thumb), how their feature sets differ in important ways and how software can and often does make use of both even in the same program.

Today we’re going one level higher and look at the ABIs (Application Binary Interface). Whereas an ISA defines what bit sequences make the CPU do what operations, an ABI is a set of conventions software is meant to follow so that it is interoperable with other software targetting the same ABI.

An ABI is adopted by compilers and operating systems. The designers of the instruction set often define some base ABI that compilers only need to extend to define a few missing bits.

One of the biggest underlying reasons getting perf to work in ARM32 is hard is that ARM has deprecated and gone over multiple base ABIs in its lifetime. Furthermore: the details on how frame pointers should work used to be completely non-standard and that has created fragmentation.

As of writing, these are the standard base ABIs defined for ARM:

• ~1987: APCS (Arm Procedure Call Standard (obsolete)). Only applies to ARM.
• ~1997: TPCS (Thumb Procedure Call Standard (obsolete)). Thumb-only counterpart of APCS.
• 1998: ATPCS (ARM-THUMB Procedure Call Standard). Unified ABI for both ARM and Thumb.
• 2003: AAPCS (Procedure Call Standard for the Arm® Architecture). The current base ABI. It applies to both ARM and Thumb.

In this post we’ll learn some typical examples of features an ABI would standardize, ultimately landing in frame pointers as one of such features.

The stack

At its most basic, a CPU does not need to have a concept of a stack. The ARM ISA exemplifies this well: the instructions used for stack manipulation are extremely generic—they can work on any register, support any kind of stack (full descending, full ascending, empty descending, empty ascending) and also work as general “write multiple registers to memory pointed by register” instructions.

But of course, a stack is a very useful abstraction that any non-trivial program will need as it allows to free up registers by saving their data for later use. Different pieces of code (e.g. functions) need to agree on a definition for the stack and rules to use it so they don’t accidentally overwrite each other’s data.

As such, ABIs define the stack; namely: what register points to the current top of the stack, whether it grows towards greater adresses (ascending stack) or toward lower adresses (descending stack), whether the “top of the stack” should be understood as “the last element pushed” (full stack) or “the position a newly pushed element would use” (empty stack), any alignment requirements, and rules for writing to the stack—for example: writes outside of the stack may be considered undefined behavior even in the absence of function calls so that interrupt handlers can make use of the same stack.

On ARM32, virtually all the ABIs¹ agree on using a full-descending stack, whose top position is tracked by the r13 register, which for this reason is also give the alias sp (stack pointer). The stack is 4 byte (32 bit) aligned. The AAPCS ABI further requires 8 byte alignment at the public interface (e.g. when calling a library function). Storing data outside of the valid region of the stack (i.e. in addresses lower than sp) is undefined behavior.

Function calls

An ABI defines what a function call is under the hood; namely: how arguments are passed and how are values returned. This part of ABIs is important enough to have its own name: calling convention.

Function arguments

A simple calling convention could require all function arguments be to pushed into the stack. The callee could then access them by addressing the stack pointer. Notably, the cdecl and stdcall calling conventions used in MS Windows 32-bit x86 work in this manner.

Modern calling conventions—including ARM32 AAPCS and all x86-64 calling conventions—use registers in addition to the stack for passing arguments, trading complexity in the ABI for better performance. This is especially a win for functions with few arguments. In many cases, programmers and compilers can strategically do computations directly in the same registers that will be later used as arguments, further reducing the overhead of function calls.

In AAPCS, r0-r3 are filled with the first few arguments. Arguments with types smaller than 32 bit (e.g. char, int16_t) are extended to 32 bits before the call. When an argument has a 64-bit type, it takes two consecutive registers. For more details, see section 6.5, Parameter Passing in the AAPCS specification. Remaining arguments are pushed into the stack.

Callee-saved and caller-saved registers

ABIs often separate user registers in two groups:

• Callee-saved registers
• Caller-saved registers, also known as scratch registers

A function is allowed to write to callee-saved registers at any point, but it must ensure they have their original value on return. This is normally accomplished by pushing (saving) the previous value to the stack before writing and popping it before return, hence the name «callee-saved».

On the other hand, a function is allowed to freely write (scratch) any and all caller-saved registers without preserving their old value anywhere. This also means that, before doing any function call, you must be careful to save any important data in these registers, hence the name «caller-saved».

Having a healthy mix of callee-saved and caller-saved is good for performance, as it reduces the number of stack manipulation operations necessary in typical functions.

In AAPCS, r0-r3 and r12 are caller-saved (i.e. scratch registers). All other general purpose registers are callee-saved.

Returning from a function: return pointers

The calling convention also needs to specify how returning from a function works. The caller must store a return pointer (an address to executable code immediately following the function call) in a well-defined place.

In many calling conventions, including the common ones for x86 and x86-64, the return pointer is passed in the stack. ARM calling conventions are a bit smarter: instead they place the return pointer in a so-called link register (lr). This means that simple leaf functions (functions that do not call other functions) can operate on their arguments and return a value without ever touching the stack.

All ARM calling conventions use r14 as the link register.

Of course, many functions will need to call to other functions before returning, so how does a link register handle this? Quite simply: lr is a callee-saved register! A function that will perform function calls will typically store lr in the stack somewhere in its preamble, do as many function calls as it wants, then restore the old value of lr from the stack.

The l in the bl and blx branch instructions in ARM stands for “link”. It means that the current value of pc will be loaded in lr before the branch. This is the most common way to do function calls in ARM and Thumb.

Special purpose registers

An ABI may reserve certain CPU registers for specific purposes, even if those registers would otherwise be general purpose registers according to the ISA. Compilers do to take this into account when generating code.

AAPCS leaves r9 free to use for any ABI extending AAPCS (e.g. for a specific operating system).

AAPCS defines r12 as the Intra-Procedure-call scratch register (ip). Functions are allowed to use it as a regular scratch register, but its real purpose is to enable writing simple veneers without needing stack manipulation.

Side-tangent: Veneers…? What is a veneer?

(You can safely skip this section If you just want to understand the minimum concepts necessary for frame pointers. Keep reading if you have a more general interest in ARM assembly.)

Instructions in ARM have a fixed size of 32 bits which is as wide as pointer. A consequence of this is that calling functions in arbitrary locations of the program requires a variable number of instructions. Most branches in ARM are done with signed offsets with a range of ±32 MiB (26 bits, of which only the most significant 24 bits are actually encoded in the instruction because ARM instructions are 4-byte-aligned). Thumb is even scrappier: the typical Thumb bl (branch with link) instruction only supports offsets of ±4MiB encoded in 22 bits, which is already more than the 16 bits per instruction of Thumb-1 and it is only possible because the bl ASM instruction gets turned into two machine Thumb-1 instructions designed for this purpose, each containing half of the offset).

Compilers and linkers cope with this limitation by assuming calls will be near enough for the offset to fit. When the offset does not fit, they synthesize some code (the veneer, also sometimes called a trampoline or range extension thunk) in a location close enough for the offset to fit. The veneer will then load the entire address in some register—likely ip— and then use a «branch to address in register» instruction (e.g. bx ip).

From this example the utility of having a scratch register not used for argument passing (ip) becomes apparent: if a scratch register like ip did not exist, the veneer would need to spill some callee-saved register to the stack just to build an address for the bx instruction. Furthermore, the callee couldn’t return directly into the original caller, but it would have to return back into the veneer so that the values from the stack could be restored.

Also of note is that is the fact that the blx instruction (branch with link and exchange ISA, provided an offset or register) did not exist until ARMv5. Before, there was only bx (branch and exchange ISA), which operated on a register. This resulted in a different number of instructions required for the branch depending on the ISA of the target (which may not be known until run-time in the case of shared libraries), and by extension made veneers necessary for interworking between ARM and Thumb.

Frame pointers

A stack frame is the chunk of the stack used by a specific function call, not including nested calls.

A frame pointer points to a fixed location (e.g. the start or end) of the stack frame. When we say that a program uses frame pointers, we refer to that program dedicating a register—the frame pointer register (usually shortened as fp)—to store the frame pointer at all times.

Having a frame pointer allows assembly code to refer to locations in the stack with simple offset from the frame pointer. So, for example, a certain variable saved in the stack will be at fp-4 and will remain at fp-4 even if code later pushes to the stack. This is more useful for humans than compilers, since it’s possible to access the same data from sp as long as you keep track of the required offset changing every time something is pushed or popped.

Something much more useful occurs if code always pushes the old value of the frame pointer register and the return pointer to the stack at call boundaries: it becomes possible to inspect the stack at runtime: the frame pointer register points to the current frame record, from which the previous frame pointer and return pointer are accessible in a fixed offset. This process can be repeated, traversing the stack like a linked list, one stack frame at a time. Before main(), a sentinel value (typically a null pointer) should be pushed instead of a frame pointer so that the end of the linked list can be detected.

Note that the usage of frame pointers is an ABI matter: The above breaks if you call a function in a library that doesn’t use the same exact convention for frame pointers. If that function skips updating the frame pointer register, its stack frames will become invisible. Or worse: if it writes some random value to it, the entire chain of callers will no longer be recoverable.

In x86 ABIs, the Base Pointer register (bp in 16-bit, ebp in 32-bit, rbp in 64-bits) is commonly used to store the frame pointer.

Frame pointers were important for obtaining backtraces in early debuggers, but this is less necessary in modern toolchains. Nowadays compilers can generate debug information (debuginfo) in a format like DWARF that debuggers can use to map any value of the pc register to what function it is part of and how deep in the stack the return pointer is at that particular instruction. This is the same process they use to map local variables to positions in the stack. The process can be repeated until the entire stack trace is obtained.

Since frame pointers became less necessary for debuggers, compiler optimizations that skip updating the frame pointer register or even use it as a general-purpose register have become very common. Omitting frame pointers as an optimization has been the default in gcc for all platforms since 2017, after it had been enabled in most target platforms individually. ARM32 already had this enabled in 2010 and earlier.

While debuginfo is a good alternative to frame pointers to get backtraces in a debugger, other development tools are not so lucky. Recovering call chains from frame pointers is a very lightweight affair: it’s a singly linked list traversal where all the nodes are within consecutive memory. On the other hand, getting a call chain through debuginfo requires having separate debuginfo sections for each object file (e.g. .so library) mapped in memory, search the wanted pc value in their tables and decode the debug tags info to find the address of the return pointer; then repeat this for every other stack frame.

Some tools have tried to adapt to this new normal of not having frame-pointers by adding support for DWARF unwinding. perf is one of them. This is however still unsatisfactory on low-end hardware: perf record -g --call-graph=dwarf cog about:blank will bring a Raspberry Pi 3 to its knees, freezing the entire system until it dies on the hands of the OOM killer. High-end hardware, like desktop PCs, can usually handle using DWARF, but it still results in much higher overhead.

Omitting frame pointers as an optimization has been increasingly reconsidered in recent years, with Fedora 38+, Ubuntu 24.04+ and Arch Linux modifying their build recipes to attempt to build all packages with frame -fno-omit-frame-pointer. Modern CPUs (especially 64-bit CPUs) have enough registers that dedicating one for this purpose is not the hit to performance it once was. Some people would argue that even if there is a slight performance hit, the gains from users and developers being able to use profilers greatly outweight it.

With this modern understanding of what frame pointers are, we will also try to build all packages with frame pointers enabled in an ARM32 environment, so that we can also use a profiler there.

Problem: what are the ARM32 frame pointer ABIs, really?

APCS frames

The (old and deprecated) APCS spec specified an optional frame pointer ABI. When you pass the (now deprecated) -mapcs-frame flag to gcc, you tell it to use the frame pointer structures defined in APCS. Here is a descriptive diagram from page 629 of the Acorn Archimedes‘ Programmer Reference Manual (1987), whose Appendix C is “ARM Procedure Call Standard”:

Here is a simple function that we will use to compare the frame pointer ABIs:

extern void show_sum(int a);
extern void show_diff(int b);

int do_operations(int a, int b) {
    int sum = a + b;
    show_sum(sum);
    int diff = a - b;
    show_diff(diff);
    int xor = a ^ b;
    return xor;
}

This is the assembly generated by gcc:

$ arm-linux-gcc -mcpu=cortex-a53 -marm -mapcs-frame -gdwarf-4 -fno-omit-frame-pointer -O2 -g -c add.c && arm-linux-gnu-objdump -S example.o
int do_operations(int a, int b) {
   0:	e1a0c00d 	mov	ip, sp
   4:	e92dd830 	push	{r4, r5, fp, ip, lr, pc}
   8:	e1a04001 	mov	r4, r1
   c:	e1a05000 	mov	r5, r0
  10:	e24cb004 	sub	fp, ip, #4
    int sum = a + b;
    show_sum(sum);
  14:	e0800001 	add	r0, r0, r1
  18:	ebfffffe 	bl	0 <show_sum>
    int diff = a - b;
    show_diff(diff);
  1c:	e0450004 	sub	r0, r5, r4
  20:	ebfffffe 	bl	0 <show_diff>
    int xor = a ^ b;
    return xor;
}
  24:	e0250004 	eor	r0, r5, r4
  28:	e89da830 	ldm	sp, {r4, r5, fp, sp, pc}

Here ip is used as a temporary to hold the value of sp before the push. The resulting stack matches the one in the old APCS document as we could expect:

fp points here: | pc at time of push (i.e. savemask pointer) |
                | saved lr                                   |
                | ip (i.e. the saved sp before the push)     |
                | saved fp                                   |

We can obtain the call chain by looking at the fp register and traversing the linked list of frame records: the saved lr contains the return function for that stack frame, and by extension, who called that function. We continue traversing the stack upwards until we find a NULL pointer indicating the end of the linked list.

AAPCS frames, clang

It may seem weird that I just spend that much time above showing a very old ABI (APCS). Especially because if you look at the modern AAPCS spec today you’ll see frame pointers also defined.

However, if you scroll to the change history, you’ll also notice it was only added in January 2020, whereas AAPCS had been around for 17 years before. This is the real source of problems: as far as I can tell, during these 17 years compilers had no official guidance on any frame pointer ABI for AAPCS.

Making things worse, GCC and clang ended up with similar but mutually incompatible frame pointer ABIs. The official AAPCS frame pointer ABI sided with clang’s. It looks like this:

$ clang -target arm-linux-gnueabihf -g -gdwarf-4 -mcpu=cortex-a53 -c -O2 -fno-omit-frame-pointer -c example.c && arm-linux-gnu-objdump -S example.o
int do_operations(int a, int b) {
   0:	e92d4830 	push	{r4, r5, fp, lr}
   4:	e28db008 	add	fp, sp, #8
   8:	e1a05000 	mov	r5, r0
    int sum = a + b;
   c:	e0810000 	add	r0, r1, r0
  10:	e1a04001 	mov	r4, r1
    show_sum(sum);
  14:	ebfffffe 	bl	0 <show_sum>
    int diff = a - b;
  18:	e0450004 	sub	r0, r5, r4
    show_diff(diff);
  1c:	ebfffffe 	bl	0 <show_diff>
    int xor = a ^ b;
  20:	e0240005 	eor	r0, r4, r5
    return xor;
  24:	e8bd8830 	pop	{r4, r5, fp, pc}

Only two words are pushed in the stack: fp and lr. The fp register is updated to point to the the most recently saved fp in the stack.

GCC frames

If you pass -marm -fno-omit-frame-pointer to gcc (as of 15.2), this is what the resulting frame pointer ABI looks like:

$ arm-linux-gcc -mcpu=cortex-a53 -marm -gdwarf-4 -fno-omit-frame-pointer -O2 -g -c example.c && arm-linux-gnu-objdump -S example.o
int do_operations(int a, int b) {
   0:	e92d4830 	push	{r4, r5, fp, lr}
   4:	e1a04001 	mov	r4, r1
   8:	e1a05000 	mov	r5, r0
   c:	e28db00c 	add	fp, sp, #12
    int sum = a + b;
    show_sum(sum);
  10:	e0800001 	add	r0, r0, r1
  14:	ebfffffe 	bl	0 <show_sum>
    int diff = a - b;
    show_diff(diff);
  18:	e0450004 	sub	r0, r5, r4
  1c:	ebfffffe 	bl	0 <show_diff>
    int xor = a ^ b;
    return xor;
}
  20:	e0250004 	eor	r0, r5, r4
  24:	e8bd8830 	pop	{r4, r5, fp, pc}

This is very similar to what ended up becoming the official AAPCS frame pointer ABI and to what clang does except that fp points to the saved lr, rather than to the saved fp directly. This is what it ends up looking like:

What about Thumb?

While I’ve learned a lot about Thumb while working on this, I would be very cautious trying to make use of frame pointers in Thumb. They’re even messier and currently broken in GCC. For more details, you can keep reading this section.

r7 as frame pointer register

Thumb in stack traces complicates things significantly. Note that r11 is a “high register” in Thumb, which limits in how many different instructions it can be used efficiently. This is especially a problem in the original Thumb-1, which doesn’t have an instruction to push/pop low registers directly and you would typically need 3 additional instructions in the prologue and 3 additional instructions in the epilogue.

The above is much less of a concern with the now ubiquitous Thumb-2, which has 32-bit instructions for push/pop and arithmetic on high registers. However, as a consequence of this historical problem, you will see that in a default Linux setup both gcc and clang use r7 as frame pointer register in Thumb code.

$ clang -target arm-linux-gnueabihf -mcpu=cortex-a53 -mthumb -gdwarf-4 -fno-omit-frame-pointer -O2 -g -c example.c && arm-linux-gnu-objdump -S example.o
int do_operations(int a, int b) {
   0:	b5b0      	push	{r4, r5, r7, lr}
   2:	af02      	add	r7, sp, #8
   4:	4605      	mov	r5, r0
    int sum = a + b;
   6:	4408      	add	r0, r1
   8:	460c      	mov	r4, r1
    show_sum(sum);
   a:	f7ff fffe 	bl	0 <show_sum>
    int diff = a - b;
   e:	1b28      	subs	r0, r5, r4
    show_diff(diff);
  10:	f7ff fffe 	bl	0 <show_diff>
    int xor = a ^ b;
  14:	ea84 0005 	eor.w	r0, r4, r5
    return xor;
  18:	bdb0      	pop	{r4, r5, r7, pc}

The layout clang uses is the same as the official AAPCS ABI, but using r7 instead of the r11. Assuming there is no mixing of ARM and Thumb — the call chain can be recovered with the same traversal algorithm.

On GCC, however, Thumb frame pointers are unviable for call chain recovery, as the GCC sets r7 to the most recent position in the stack, which makes it impossible to know the position of the saved lr and the previous node pointer within the stack without additional information.

Using r11 for both ARM and Thumb

Using different frame pointer registers for different modes is problematic, as code in one mode (typically ARM) will frequently clobber the register used by the other mode (typically Thumb). Both r7 and r11 are caller-saved registers, so the information is there in the stack, but the stack and register values is not enough to locate them.

A simpler approach is to use r11 for both modes, which is much less awkward in Thumb-2. This is what the official AAPCS spec specifies. Recent versions of clang can be told to do this with -mframe-chain=aapcs (or alternatively, -mframe-chain=aapcs+leaf if frame pointers on leaf functions are also desired).

$ clang -target arm-linux-gnueabihf -mcpu=cortex-a53 -mthumb -gdwarf-4 -fno-omit-frame-pointer -mframe-chain=aapcs -O2 -g -c example.c && arm-linux-gnu-objdump -S example.o
int do_operations(int a, int b) {
   0:	e92d 4830 	stmdb	sp!, {r4, r5, fp, lr}
   4:	f10d 0b08 	add.w	fp, sp, #8
   8:	4605      	mov	r5, r0
    int sum = a + b;
   a:	4408      	add	r0, r1
   c:	460c      	mov	r4, r1
    show_sum(sum);
   e:	f7ff fffe 	bl	0 <show_sum>
    int diff = a - b;
  12:	1b28      	subs	r0, r5, r4
    show_diff(diff);
  14:	f7ff fffe 	bl	0 <show_diff>
    int xor = a ^ b;
  18:	ea84 0005 	eor.w	r0, r4, r5
    return xor;
  1c:	e8bd 8830 	ldmia.w	sp!, {r4, r5, fp, pc}

Frame pointer sadness

There are problems with frame pointers in ARM other than the incompatible ABIs. In fact, I kept finding more and more sadness the more I worked on this post.

GCC frames can’t be unwinded in leaf functions

While -fno-omit-frame-pointer will make GCC push fp to the stack, it doesn’t disable the “link register save elimination” optimization. This messes up with leaf functions. Consider the following trivial function:

void simple_nop(void) {
    __asm__("nop");
}

$ arm-linux-gcc -mcpu=cortex-a53 -marm -gdwarf-4 -fno-omit-frame-pointer -O2 -g -c simple_nop.c -S && cat simple_nop.s
simple_nop:
	@ args = 0, pretend = 0, frame = 0
	@ frame_needed = 1, uses_anonymous_args = 0
	@ link register save eliminated.
	str	fp, [sp, #-4]!
	add	fp, sp, #0
	.syntax divided
@ 31 "example.c" 1
	nop
@ 0 "" 2
	.arm
	.syntax unified
	add	sp, fp, #0
	@ sp needed
	ldr	fp, [sp], #4
	bx	lr
	.size	simple_nop, .-simple_nop

Notice that, unlike in the previously shown GCC frames, here the fp register is made to point to the saved fp in the stack, rather than the saved lr. This means that an unwinder will (1) wrongly read a next node pointer instead of the function return address that it would expect, (2) will interpret whatever random variable was pushed in the stack immediately before as the next node in the call chain.

I’m surprised I only caught this long after having written an unwinder for GCC frames and having seen it produce sensible flamecharts, but I’m pretty sure this is not just a synthetic problem, since I can see the same problematic assembly by objdump’ing GStreamer libraries.

The deprecated APCS frames (-mapcs-frame) are not affected by this bug.

Getting the top frame is racy during the prologue

At the top of a stack trace you expect to see the currently running function, at the currently running instruction. This is normally obtained by reading the value of the pc register. Then, by looking at the fp register and inspecting the topmost frame record we can see who called this function and traverse the chain.

However, this also means there is a small window of time from the moment a call is done with bl and until the fp register is updated to point to the newly created frame record in which the pc register points to the callee and the caller is recorded in the lr register, but hasn’t been put in a frame record yet.

This results on the caller function seemingly disappearing from the stack trace for a brief moment, as seen here in a instruction-per-instruction simulation:

* 1000(main+0)
* 1004(main+4)
* 1008(main+8)
* 1012(main+12)
* 2000(leaf1+0)
* 2004(leaf1+4)
* 1012(main+12) <- 2008(leaf1+8)
* 1012(main+12) <- 2012(leaf1+12)
* 1012(main+12) <- 2016(leaf1+16)

I can’t think of an alternative call chain traversal algorithm that solves this problem using only the stack and registers. While it seems tempting to also make use of the lr register directly to reconstruct the missing frame, that causes other problems later when the callee returns. I’m also not the first person to point out this problem.

Conclusions

Frame pointers in ARM32 are messy — especially compared to ARM64 or x86_64. A standard ABI for them has only been specified few years ago, with support for it is pretty spotty and divided into incompatible alternatives.

The status of frame pointers for ARM32 in GCC is much worse than I thought when I started working on this: my impression now is that the ostensibly deprecated -mapcs-frame is the only frame pointer ABI GCC handles in a reliable way, and I wouldn’t be too surprised if it turned out that any usefulness of the non-APCS frame pointers in ARM32 with GCC is coincidental rather than intentional. This is certainly not helped by the long lasting specification void in this area.

The default frame pointer ABI of GCC is only reliable during non-leaf functions and as long as Thumb is not used. With that big caveat, they’re still potentially useful. Furthermore, the GCC-specific deficiencies I’ve explained here could be fixed, especially now that there is some official ABI guidance.

Clang has positively surprised me during my investigation. I started looking at it only for comparison, but was happy to see support for the new AAPCS frame pointer ABI in Thumb as well, as well as a general good handling of edge cases.

On top of that, seeing the limitations of frame pointers has also made me a bit curious about the alternative approaches for improved accuracy. Most of those require lookup tables and therefore are generally going to be slower, but how much they can be optimized is still an area of ongoing research.

Additional resources

Analyzing code that uses the stack by hand is error prone, so eventually I wrote a simulator for testing frame pointer ABIs and the algorithms to traverse them.

If you want to look at the compiler sources, you’ll be mostly intersted in:

• clang: llvm/lib/Target/ARM/ARMFrameLowering.cpp. Contains the code for generating prologues and epilogues. Search FramePtr to find the relevant code.
• gcc: gcc/config/arm/arm.c. Look for HARD_FRAME_POINTER_REGNUM. The prologue contents are emitted in arm_expand_prologue().

1. The earliest variant of APCS—referred to as APCS-A—mapped sp to r12 instead of r13. fp was also mapped to r10 instead of r11. This is explained in page 1762 of the RISC OS Programmer Reference Manual (1989), which describes what would later be called APCS-2.
   I’m also leaving aside many complexities of APCS variants, most notably chunked stacks, where the program stack wouldn’t be in contiguous memory, but instead in a linked list of chunks, useful for multi-threading in processors without a Memory Management Unit. ︎

Update on what happened in WebKit in the week from August 17 to August 24.

Another periodical packed with updates on the graphics, multimedia, and tooling fronts. Which sure has to do with preparing for the upcoming 2.54.x release series, that now has release candidates published. Also, do not miss a new stable release being published with fixes for security issues, and make sure to update.

Cross-Port 🐱

Multimedia 🎥

Graphics 🖼️

Releases 📦️

Enhanced Spell Checking on the Web

Here's some interesting stuff about browsers you might not know, and a new API proposal that might interest you...

You probably already know that if you have an editable area in a page with the spellcheck=true attribute set on it, then when a user clicks inside that area, it marks words as spelling errors, and you can style spelling errors with CSS. What you might not know is that feature is carefully designed and implemnted to avoid the huge fingerprinting risk of letting people know what's in your dictionary. A lot of the actual details are left to implementations and this means that not only will each OS/language highlight different spelling errors, but so will different browsers on the same machine, or even different profiles in the same browser!

But, if you think about websites and apps, they are highly likely to be accepting input related to something fairly specific, and highly unlikely to be in a common dictionary. Right? Online education is a great example where you'll find students writing words in different domains that are very common in that domain, but aren't in the common dictionary. Mathematics has its own words, so does chemistry, astronomy, and law. But schools are just one example. There are sites related to the stock market - lots of things there might look like gibberish otherwise. Or your hospital and doctor reports. The sites and apps for wine afficianados, or coffee lovers. There are sites where you submit fan fiction. There are forums where you discuss web development or cloud infrastructure. All of these will have common words that aren't in a general purpose dictionary.

That's a problem because it means we're marking lots of words as misspelled, but they aren't. Too many false positives and we easily overlook the ones that really are missspelled. In fact, in some cases it can even feel like more of an annoyance than a help.

So, there is a proposal currently in HTML for a new API called SpellCheck Custom Dictionary. For v1 the proposal is dead simple. It introduces a new document.spellCheckCustomDictionary with exactly two methods: .addWords(listOfWords) and .removeWords(listOfWords) - where listOfWords is an array of Strings. Those words are simply added to (or removed from) a Set internally which is consulted before consulting any other dictionaries (remember, there can be several) for the lifetime of that document. It doesn't affect other pages or the profile dictionary, or the browser dictionary or the OS dictionary, or any other things.

This is currently implemented by Igalia, with funding from Bloomberg Tech. It is currently available in Chrome Canary behind the experimental web platform features flag, and is scheduled to be shipping in on September 8, 2026 behind the same flag.

Because of its simplicity, it also works with JSON pretty nicely, you can just fetch a JSON array and call .addWords with the result. Of course, in practice you'll probably want to do this as a kind of progressive enhancement, not load the JSON immediately, and maybe handle and report various errors. For a whole lot of developers and use cases, it seems to me that a declarative API would be ideal. But we're not there yet. We have more to figure out with regard to how to express more than exact matches and so on. So, we'll get there... I hope.

In the meantime, however, I've made a tiny, but handy library which speculatively polyfills the basic outline of a declarative form in a safe way. With this, you just include the script and then you can add a <link> tag (or tags) pointing to JSON serializations of arrays of words, and adorn them with a data-spellcheck-dictionary marker-attribute.

<script src="dist/spellcheck-dictionary-loader.min.js"></script>
<link href="terms/domain-terms.json" data-spellcheck-dictionary>
<link href="terms/stocks.json" data-spellcheck-dictionary>

The script will wait until the initial parse is complete and then check for support. If the browser supports the custom dictionary API, it will load and add the words.

There is a /src and /dist version of this and it's very tiny (441 bytes over the wire with Brotli) and I expect it works for most common stuff.

It's "rough" though and not exactly how we'd probably expect a real implementation to work. For example, if you remove the link, it doesn't remove the words. If you change the href, it doesn't do anything, and so on. A real implementation would probably do those things. So, if you're interested in playing with something more complete, I've also included a -complete version of each: src/spellcheck-dictionary-loader-complete.js and dist/spellcheck-dictionary-loader-complete.min.js. The latter is still only 770 bytes over the wire with Brotli, but it's up to you how important that use is to you.

You can get all of the versions from this repo where there is also a start on several domain dictionaries to play with. What I think would be really great is to have some shared dictionaries that we work on together, so feel free to send PRs for additional words, this is really only a mostly generated starting list for demonstration purposes.

Please, let us know what you think - and thanks again to Bloomberg Tech for the funding!

Update on what happened in WebKit in the week from August 10 to August 17.

Following an extra packed periodical, this week we get back to a more regular pace with two nice bugfixes, and a new tool to analyze WebKit performance on Linux!

Cross-Port 🐱

Graphics 🖼️

Update on what happened in WebKit in the week from July 28 to August 10.

Quite a packed pair of weeks this time! The range of updates is big, but some highlights are the handful of Layer-Based SVG Engine updates, performance improvements, and the new WPE APIs. Finally, the Web Engines Hackfest recordings are now published!

Cross-Port 🐱

Multimedia 🎥

Graphics 🖼️

WPE WebKit 📟

Community & Events 🤝

Introduction

The XHCI specification has a optional debugging capabilities that is available via the extended capabilities set. This set could be enabled by vendor to pass along additional information around the hardware - especially information that would help in debugging. In this post, we are going to identify a USB controller (on a target) that supports this debug capability and try to enable/verify it on the host side. At last, we are going to also look at the debug capabilites associated with early boot phase of the kernel

Requirements

• USB 3.1 Superspeed cable A-A or A-C cable
• Target with supported xHCI debug capability
• Host machine

Checking if xHCI debug capability is supported

We need to verify if the target machine has xHCI debug capability supported. We can simply find out by peeking into our sysfs device information:

($) find "/sys/devices" -type f -name "dbc"

If this command returns a device, we can be sure that the xHCI controller is present with debug capability.

Enable the xHCI debug capability on target

($) sudo bash -c "echo enable > /sys/devices/pci0000:00/0000:00:0d.0/dbc"

You should expect the following confirmation via dmesg on the target:

[ 3400.545281] xhci_hcd 0000:04:00.3: DbC connected
[ 3400.905302] xhci_hcd 0000:04:00.3: DbC configured

On the host machine which is connected via the USB 3.1 SuperSpeed cable, we can verify that the USB host controller recognises the debug device. Running dmesg on host:

[110351.933236] usb 4-2: new SuperSpeed USB device number 6 using xhci_hcd
[110351.944701] usb 4-2: New USB device found, idVendor=1d6b, idProduct=0010, bcdDevice= 0.10
[110351.944732] usb 4-2: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[110351.944742] usb 4-2: Product: Linux USB Debug Target
[110351.944750] usb 4-2: Manufacturer: Linux Foundation
[110351.944756] usb 4-2: SerialNumber: 0001
[110351.990401] usbcore: registered new interface driver usb_debug
[110351.990438] usbserial: USB Serial support registered for debug
[110351.990452] usbserial: USB Serial support registered for xhci_dbc
[110351.990502] usb_debug 4-2:1.0: xhci_dbc converter detected
[110351.990786] usb 4-2: xhci_dbc converter now attached to ttyUSB0

You can attach any serial console utility to interact with target device using xHCI debug capabilities.

Linux kernel early boot setup

xHCI is also helpful to output early kernel boot stage logs. This phase is the intialization process just after the bootloaders hands over control. To enable early kernel debug, we need to configure the kernel with proper kconfig and pass debug parameters in the kernel commandline.

Build the kernel with following kconfig enabled:

CONFIG_EARLY_PRINTK_USB_XDBC

And for the kernel commandline, append:

earlyprintk=xdbc

Early boot IO remap

Now that you have had gentle introduction to this debugging capability, we are going briefly mention about fixed boot-time mappings known as fixmaps. The early boot has a special set of operations that are carried out before the standard virtual memory and device drivers are active. In this phase, hardware registers are allowed to be accessed via fixed boot time mapping and page tables.

Generally the xHCI debug capability reside under the extended capabilities given by vendor and can be mapped in fixed boot time mappings. However, we have such devices coming up, that have extended capabilites size much bigger than the fixmaps.

In this case, the PCIe memory range would fail to get IO remapped using early_ioremap() call.

     kernel: xhci_dbc:early_xdbc_parse_parameter: dbgp_num: 0
     kernel: ------------[ cut here ]------------
    ay 19 13:42:57  kernel: WARNING: CPU: 0 PID: 0 at mm/early_ioremap.c:139 __early_ioremap+0xae/0x180
     kernel: Modules linked in:
     kernel: CPU: 0 UID: 0 PID: 0 Comm: swapper Not tainted 6.18.25-valve1-gcab630e7af50 #13 PREEMPT(undef)  4b70872d9de6788b7f2d10dce46ada89e6cd177b
     kernel: RIP: 0010:__early_ioremap+0xae/0x180
     kernel: Code: 60 ba 3f a0 4c 89 ca 48 81 e3 00 f0 ff ff 48 81 e2 00 f0 ff ff 48 29 d3 48 89 14 24 48 89 da 48 c1 ea 0c 89 d5 83 fa 40 76 04 <0f> 0b eb a2 6b c0 c0 4d 89 ce 41 81 e6 ff 0f 00 00 44 8d b8 ff 05
     kernel: RSP: 0000:ffffffff9fa03cb8 EFLAGS: 00010006 ORIG_RAX: 0000000000000000
     kernel: RAX: 0000000000000000 RBX: 0000000000100000 RCX: 0000000000100000
     kernel: RDX: 0000000000000100 RSI: 0000000000100000 RDI: 0000000080200000
     kernel: RBP: 0000000000000100 R08: 0000000000000000 R09: 0000000080200000
     kernel: R10: 0000000000000004 R11: ffffffff9fa03ad0 R12: 8000000000000163
     kernel: R13: 0000000000000000 R14: 0000000080200000 R15: 0000000000000000
     kernel: FS:  0000000000000000(0000) GS:0000000000000000(0000) knlGS:0000000000000000
     kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
     kernel: CR2: ffff888000000413 CR3: 000000020ed02000 CR4: 00000000000000b0
     kernel: Call Trace:
     kernel:  <TASK>
     kernel:  ? early_xdbc_parse_parameter+0x32c/0x360
     kernel:  ? setup_early_printk+0x4f5/0x520
     kernel:  ? do_early_param+0x44/0x70
     kernel:  ? parse_args+0x233/0x420
     kernel:  ? __pfx_do_early_param+0x10/0x10
     kernel:  ? parse_early_options+0x29/0x30
     kernel:  ? __pfx_do_early_param+0x10/0x10
     kernel:  ? parse_early_param+0x64/0xc0
     kernel:  ? setup_arch+0x542/0xbc0
     kernel:  ? _printk+0x6b/0x90
     kernel:  ? start_kernel+0x66/0x9a0
     kernel:  ? x86_64_start_reservations+0x24/0x30
     kernel:  ? x86_64_start_kernel+0xcc/0xd0
     kernel:  ? common_startup_64+0x13e/0x141
     kernel:  </TASK>
     kernel: ---[ end trace 0000000000000000 ]---

And this is the exact place that is triggered at in mm/early_ioremap.c

	/*
	 * Mappings have to fit in the FIX_BTMAP area.
	 */
	nrpages = size >> PAGE_SHIFT;
	if (WARN_ON(nrpages > NR_FIX_BTMAPS))
		return NULL;

We shall call this issue as extra extended capabilities issue for lack of a better work!

Solution to extra extended capbilities for xHCI

Now that we are seeing such hardware in the market where the fixmaps can be a limiting factor, we have to find a solution for it. This has been a focus of my work and after a few iterations, we have this patch for review! Feel free for review comments or follow it for curiousity. There might be few more followups on this particular approach – so stay tuned for that.

Thank you for reading!

Update on what happened in WebKit in the week from July 14 to July 27.

This two-week update includes plenty of changes to the Skia compositor, changes to multimedia support, three blog posts, and assorted improvements.

Cross-Port 🐱

Multimedia 🎥

Graphics 🖼️

Community & Events 🤝

In the past, when you needed to adapt WPE WebKit to a new platform, or integrate it with your own system/application not based on Wayland, you had to develop a specific backend. I wrote two blog posts in the past about this topic (One about the process of creating a new WPE backend and another about using EGLStreams in a WPE backend) and, to be honest, it was not really straightforward.

Since WPE WebKit 2.50, a new system has been designed to replace all this by a more modern and intuitive approach, making a lot easier to integrate WPE WebKit into your application. The new API is called WPE Platform and allows you to define the equivalent of the old backend system into your own executable. At the moment of this post (version 2.52), the WPE Platform is still in development and the official release is foreseen for the next stable version 2.54. Nevertheless, it is already stable enough to start playing with it.

This post is going to explain step by step how you can use WPE WebKit as a web view inside a simple X11 window using EGL with full hardware acceleration and zero-copy of the graphical hardware buffers while avoiding the complexity of writing an external backend. The X11/EGL window itself will be managed by the GLFW library.

N.B. GLFW is only used here as a convenient, cross-platform way of creating a window and an EGL context with a minimal amount of code. It is not the topic of this post. What matters here is the generic contract to follow to implement a custom WPE Platform. It consists basically in three GObject classes and a small set of virtual methods that would look exactly the same if we had chosen SDL, Qt, or directly a raw X11 Window instead of GLFW.

The reference project for this post is blog-the-wpe-platform-api. It implements a complete and minimal WPE Platform taking into account the basic user’s interactions (keyboard, scrolling, mouse, etc…).

The version 0.0 of this code implements the bases to initialize a web view using WPE (it will need a Wayland compositor to run). While the version 1.0 also integrates all the components needed to implement the WPE Platform. From the implementation point of view, the only difference is the usage of a custom WPEDisplay instead of the default one:

// Create and connect our GLFW-backed WPE display
WPEDisplay* display = wpe_glfw_display_new();
GError* error = NULL;
if (!wpe_display_connect(display, &error))
{
    ...
    return EXIT_FAILURE;
}

// Create the WebKitWebView
WebKitWebView* web_view = WEBKIT_WEB_VIEW(g_object_new(WEBKIT_TYPE_WEB_VIEW, "display", display, NULL));
...

Building and running the example project #

If your operating system has a development package for libWPEWebKit-2.0 version 2.52 or above, the easiest way is to install this package from your distribution. Else, you can build WPE WebKit by yourself:

• or by using the webkit-container-sdk reading the instructions provided on the project,
• or by building the library locally.

If you already have WPE WebKit installed, you can go directly to the build of the example project.

Build WPE WebKit locally #

The following instructions are for building the library locally out of any container. You first need to clone the WPE WebKit project, version 2.52 or above:

mkdir wpe-webkit
cd wpe-webkit
git clone --depth 1 -b wpewebkit-2.52.5 https://github.com/WebKit/WebKit.git

Then install clang and all the needed development dependencies by calling: ./WebKit/Tools/wpe/install-dependencies.

Download the build-wpe.sh and set_dev_env.sh scripts and copy them to your working folder containing the WebKit source code. Then execute:

chmod 755 ./build-wpe.sh ./set_dev_env.sh
./build-wpe.sh --configure
./set_dev_env.sh

It will build and install WPE WebKit into ./dist-wpe. The set_dev_env.sh script will set the environment variables to use the files in ./dist-wpe for pkg-config and for the runtime.

Build the example project #

Clone the example project:

git clone https://github.com/neodesys/blog-the-wpe-platform-api.git wpe-glfw-platform

Install the GLFW development dependency (package libglfw3-dev on Ubuntu/Debian).

Configure and build it:

cd wpe-glfw-platform
meson setup build
ninja -C build

You can now run it by calling ./build/wpe-glfw [url].

N.B. In the version 2.52.5 of WPE WebKit, the Skia multithreaded hardware-accelerated compositor is not fully stable with some specific GPUs. In particular, with NVidia graphic cards, it may crash when the allocated surfaces are resized to resolutions bigger than HD. If this is your case, you can disable the Skia hardware-accelerated compositor by setting the environment variable WEBKIT_SKIA_ENABLE_CPU_RENDERING=1. In some cases, you don’t need to disable the whole hardware-acceleration for the compositor. Sometimes just configuring the hardware compositor to use only one thread is enough. You can do that by setting the environment variable WEBKIT_SKIA_PAINTING_THREADS=1.

From a WPE Backend to a WPE Platform #

A WPE Backend sits at the crossroads between the WPEWebProcess, in charge of running the ThreadedCompositor, and the application process, which presents the resulting frames. Both processes must load the same backend shared library, and that library has to implement an IPC layer to move each handle from one process to the other.

graph LR;
    subgraph SA[<b>WPEWebProcess</b>]
        A(ThreadedCompositor)
    end
    subgraph SB[<b>Application Process</b>]
        C(User Application)
    end
    A -->|draw| B([WPE Backend shared library<br/>Loaded by both processes]) --> C

The WPE Platform API removes this shared library and all the IPC burden. WPE WebKit still spawns a WPEWebProcess to run the ThreadedCompositor but the transfer of the rendered frames from the WPEWebProcess to the application process is now handled internally by WPE WebKit itself. As an application developer, you no longer need to implement any IPC: you only receive a ready-to-use WPEBuffer object, backed by a DMA buffer or by shared memory, directly into your application process.

graph LR;
    subgraph SA[<b>WPEWebProcess</b>]
        A(ThreadedCompositor)
    end
    subgraph SB[<b>Application Process</b>]
        B[WPEDisplay<br/>WPEToplevel<br/>WPEView]
        C(User Application)
    end
    A -->|"WPEBuffer (handled internally)"| B --> C

What used to be a shared library exposing five libwpe interfaces is now just three plain GObject classes (WPEDisplay, WPEToplevel and WPEView) that you need to subclass and link directly into your application binary.

The three pillars of a WPE Platform #

• WPEDisplay is the entry point. It owns the connection to the native graphical system (here, an EGL display bootstrapped through GLFW) and acts as a factory for the toplevel window and the web views.
• WPEToplevel is roughly the equivalent of a native window. It owns the actual GLFW window, translates native window events (keyboard, mouse, scroll, focus and resize) into WPEEvent instances, and exposes the usual window operations to the WPE Platform API (resizing, setting the window title and switching to fullscreen).
• WPEView is where the web page is actually drawn. It receives a new WPEBuffer each time the ThreadedCompositor has produced a frame and is responsible for presenting it on screen.

graph TB;
    subgraph SA[<b>WPEGLFWDisplay</b>]
        A["connect(): create the EGL display through GLFW"]
    end
    A -->|create_toplevel| B[<b>WPEGLFWToplevel</b><br/>owns the GLFW window]
    A -->|create_view| C[<b>WPEGLFWView</b><br/>renders one WPEBuffer per frame]
    B -.attach view and broadcast window events.-> C

A single WPEDisplay can create several toplevel windows, and a toplevel window can have several views attached to it (think of tabs sharing one native window). In our example application we only create a single toplevel window with a single view.

Implementing WPEGLFWDisplay: bootstrapping EGL through GLFW #

The WPEGLFWDisplay::connect(…) override is called only once, when wpe_display_connect(...) is invoked from main(). It initializes GLFW, requests an EGL/GLES2 context for every window that will be created afterwards, and creates a tiny hidden bootstrap window used only to force GLFW to set up its EGL connection:

// Request EGL + GLES 2 context for all subsequent window creations
glfwWindowHint(GLFW_CONTEXT_CREATION_API, GLFW_EGL_CONTEXT_API);
glfwWindowHint(GLFW_CLIENT_API, GLFW_OPENGL_ES_API);
glfwWindowHint(GLFW_CONTEXT_VERSION_MAJOR, 2);
glfwWindowHint(GLFW_CONTEXT_VERSION_MINOR, 0);
glfwWindowHint(GLFW_VISIBLE, GLFW_FALSE);

// Create a tiny hidden window so GLFW can initialize its EGL connection
self->init_window = glfwCreateWindow(1, 1, "", NULL, NULL);
...
self->egl_display = glfwGetEGLDisplay();

If we want to be able to transfer the frames with zero-copy, keeping them in the GPU memory, we will need a valid WPEDRMDevice. It is also initialized during the display connection, using the EGL_EXT_device_query extension to fetch the device associated with the current EGL display:

EGLDeviceEXT egl_device = EGL_NO_DEVICE_EXT;
if (eglQueryDisplayAttribEXT(self->egl_display, EGL_DEVICE_EXT, (EGLAttrib*)&egl_device) &&
    egl_device != EGL_NO_DEVICE_EXT)
{
    const char* drm_device = eglQueryDeviceStringEXT(egl_device, EGL_DRM_DEVICE_FILE_EXT);
    const char* drm_render_node = eglQueryDeviceStringEXT(egl_device, EGL_DRM_RENDER_NODE_FILE_EXT);
    self->drm_device = wpe_drm_device_new(drm_device, drm_render_node);
}

If WPEGLFWDisplay::get_drm_device(…) returns NULL, the produced frames will be transferred to the application using shared memory, which implies copying the frames content back and forth between the GPU and the CPU.

Implementing WPEGLFWToplevel: the window and its events #

WPEGLFWToplevel is where the actual GLFW window is created. It is important to manage this creation in the WPEGLFWToplevel::constructed(…) override rather than in the init() function because WPEToplevel::constructed(...) is resetting the toplevel registered dimensions.

static void wpe_glfw_toplevel_constructed(GObject* object)
{
    // It is important to initialize the window in the `constructed` virtual
    // method and not in the `init` method because the parent class
    // (WPETopLevel) resets the toplevel window size in this call.
    G_OBJECT_CLASS(wpe_glfw_toplevel_parent_class)->constructed(object);

    WPEGLFWToplevel* self = WPE_GLFW_TOPLEVEL(object);

    // All window hints have already been configured by the display
    self->window = glfwCreateWindow(DEFAULT_WIDTH, DEFAULT_HEIGHT, "", NULL, NULL);
    ...

    // Communicate the initial window size to the WPETopLevel, so when
    // the WPEView is attached, it can be resized immediately to the correct
    // dimensions.
    wpe_toplevel_resized(WPE_TOPLEVEL(self), DEFAULT_WIDTH, DEFAULT_HEIGHT);
}

Once the window exists, each GLFW events callback (for the keyboard, mouse, window resizing, etc…) is translating the window events into the corresponding WPEEvent and broadcasts those events to every view currently attached to this toplevel window.

The collection and dispatch of the GLFW events themselves are ensured by calling glfwPollEvents(). As this function is managing the events for all the GLFW windows, it is configured in the WPEGLFWDisplay as a Glib source. This way all GLFW window events are collected, dispatched, translated into WPEEvent and broadcasted to each view from the main application thread running the Glib main loop.

The rest of the class is a set of straightforward virtual method overrides mapping WPE window operations onto GLFW calls to allow the WPE Platform API to set the window title, change the toplevel window size or switch to fullscreen.

Implementing WPEGLFWView: turning a WPEBuffer into pixels #

WPEGLFWView is the class doing the actual OpenGL ES drawing. The main interesting override is render_buffer(…), called by WPE WebKit every time a new frame is ready to be presented. The rest of the code is basically some boilerplate used to render a texture on a plane.

The WPEBuffer provided by WPE WebKit can be a wrapper for a DMA buffer allowing to draw the frame without copying it to the main memory, or it can be the wrapper of a classical block of shared memory if the DRM device was not available in WPEGLFWDisplay::get_drm_device(…).

So, when drawing, we first try to get the EGLImage wrapped by the WPEBuffer and, if not available, we fall back to the shared memory:

// Try to import the WPEView content through an EGLImage to allow a
// zero-copy transfer. This is only going to work if the EGLDisplay is
// associated with a valid DRM device returned by
// WPEDisplay::get_drm_device().
gpointer egl_image = wpe_buffer_import_to_egl_image(buffer, NULL);
if (egl_image)
{
    self->glEGLImageTargetTexture2DOES(GL_TEXTURE_2D, (GLeglImageOES)egl_image);
    glUniform1f(self->uniform_swap_rb, 0.f);
}
else
{
    // Else, fall back to the SHM buffer. In this case the WPEView content
    // is copied through the CPU into shared memory.
    GBytes* pixels = wpe_buffer_import_to_pixels(buffer, &shm_err);
    ...
    gconstpointer data = g_bytes_get_data(pixels, NULL);
    ...
    glTexImage2D(GL_TEXTURE_2D, 0, GL_RGBA, (GLsizei)buf_w, (GLsizei)buf_h, 0, GL_RGBA, GL_UNSIGNED_BYTE, data);
    glUniform1f(self->uniform_swap_rb, 1.f);
}

All the resources (the EGLImage or the GBytes pixels array) are held by the WPE WebKit ThreadedCompositor and so, once the drawing is finished, we must signal that the buffer has been rendered and can be recycled:

wpe_view_buffer_rendered(view, buffer);
wpe_view_buffer_released(view, buffer);

The first call (wpe_view_buffer_rendered(...)) triggers the rendering of the next frame, while the second call (wpe_view_buffer_released(...)) informs that the internal EGLImage or GBytes pixels array can be re-used for a future frame content, avoiding the allocation of new buffers for each new frame.

N.B. The drawing loop in the example is very simplified for the purpose of this post because we are not repainting the window content when it is damaged for example. We are only doing the drawing sequentially at one place when receiving a new frame from the web view. In a real application, we may want to keep the current WPEBuffer for intermediate repainting, until receiving the next frame. In this case, we would call wpe_view_buffer_rendered(...) for buffer A once it has been drawn but we would call wpe_view_buffer_released(...) only after receiving buffer B with the next frame content. So, buffer A may be used more than once to repaint the window content like shown in the following sequence diagram.

sequenceDiagram
    participant A as WPEWebProcess
    participant B as Application Process
    activate A
    A ->> A: Render frame 1 in buffer A
    A ->> B: WPEBuffer A
    deactivate A
    activate B
    B ->> B: Draw frame 1 from buffer A
    B ->> A: wpe_view_buffer_rendered(A)
    deactivate B
    activate A
    A ->> A: Render frame 2 in buffer B
    activate B
    B ->> B: Repaint buffer A
    deactivate B
    A ->> B: WPEBuffer B
    deactivate A
    activate B
    B ->> A: wpe_view_buffer_released(A)
    B ->> B: Draw frame 2 from buffer B
    B ->> A: wpe_view_buffer_rendered(B)
    deactivate B
    activate A
    A ->> A: Render frame 3 in buffer A
    A ->> B: WPEBuffer A
    deactivate A
    activate B
    B ->> A: wpe_view_buffer_released(B)
    B ->> B: Draw frame 3 from buffer A
    deactivate B

The rendering in the current example is not optimized either because the ThreadedCompositor must wait for the complete presentation of the current frame with glfwSwapBuffers(...) blocking until the drawing is finished. We can perfectly imagine a multithreaded view where the render_buffer(...) override just posts the current WPEBuffer to a separate drawing thread. This way the WPE WebKit drawing of the next frame and the presentation of the current frame on screen would run in parallel instead of waiting for each other.

Glad to see the Web Engines Hackfest covered in the local tech magazine Código Cero. Of course I’m biased, but I’m very happy with the growth of the event and the really interesting discussions that happen there every year.

See full interview in Galician (PDF version).

Update on what happened in WebKit in the week from June 30 to July 13.

The summer continues with many updates to the new SVG engine (LBSE), improvements to the new Skia-based compositor, some small API additions, and ever-important stable releases with security fixes.

Cross-Port 🐱

Graphics 🖼️

WPE WebKit 📟

Releases 📦️

A New Browser Vision?

In which I share some thoughts about the state of things, and how maybe we could hope to change them.

Back in 2020, after Microsoft gave up on their own engine, I began writing about a topic I called "Web Ecosystem Health". What ultimately makes a healthy system that will last? Over time I became convinced that it is all much more fragile than we realize. In 2021 I wrote Web Rise beginning to detail some of this.

There have been several more articles and a whole series of at least 20 podcast episodes with guests of all types talking about many, many different aspects of this. But, at the heart of it is really how it is all funded.

The other day I asked on social media "Imagine that one of the big 3 web engine stewards, for some reason, decided they would stop. What do you think would happen? Would someone step in and save the project? Who? Or would it just die?" My colleague Eric Meyer (who is on holiday and had no idea I was posting this, or why) replied

My prediction is that people would step in to save it, but the effort would falter and wane over the next several years as contributors lost momentum and interest until finally shuddering to a halt.

And that's kind of why I asked - because I was landing in a sort of similar situation. I mean, I've said it before, but I think maybe even more radically now, we need to diversify investment. Maybe even diverisfy ownership, somehow in a bigger way.

The Supporters of Chromium-Based Browsers (https://socbb.org/ - which I've recently heard pronounced "Sock Baby" and have now latched onto) is an interesting collaborative initiative under the Linux Foundation that some of our discussions helped inspire. Basically, it's a common pool of money that is paid into by Google, Microsoft, Meta and Opera which then tries to fund work and grow contributions from outside those organizations. Together, they decide how the money is spent.

I think it's still "small" though and I wonder how much you could scale it up. I was thinking about this with regard to Servo. Servo is a really interesting project. It gets people excited. It is written in Rust, it's the first one to come without a long history of baggage that it has to deal with. There is so much promise there.

But, it's also not really remotely ready to compete with Blink or WebKit or Gecko, and none of those are standing still. In fact, while Servo can close ground quickly in some cases, there are just far, far fewer people paddling it forward. It takes a leap of faith to believe that we could make it really competitive, and to provide the resources to do it. Again, very few orgs even could carry the load on their own - and it's still kind of fragile if it's just one org too. But what if we could collectively own it. Something more like the Mozilla Foundation, but... better? What if we could get a lot of companies to invest in that dream with the promise that a kind of collective ownership could really change things and that while no one could do it alone, probably we could all bear to take a chance on something with a lot of interesting upside. What makes Servo interesting here is that doens't already have a powerful and weathy steward. Diverisity of ownership it could ensure that the web would remain despite changes to buisness models and so on. It would help give them some kind of a louder voice - but also present really practical compelling reasons for building concensus and compromise -- because no one org is king.

It would be an interesting new challenge in governance and so on, but... It could be really interesting.

While working on Vulkan Video encode support in Mesa, I needed to capture H.265 encoding traces. gfxreconstruct already handled H.264 video and several other extensions, but VK_KHR_video_encode_h265 was explicitly blocked. Here’s how I unblocked it and what I learned about gfxreconstruct’s code generation machinery along the way.

gfxreconstruct is LunarG’s suite of tools for capturing and replaying graphics API calls. It intercepts Vulkan (and D3D12) calls at the layer level, serializes them into a compressed binary trace file, and can later replay that trace verbatim. This is useful for driver regression testing, GPU bring up, architecture simulation, and bug reporting.

The blocking mechanism #

gfxreconstruct has two independent mechanisms that prevent an extension from being captured.

The first is a runtime blocklist in framework/encode/vulkan_entry_base.cpp. A static array called kVulkanUnsupportedDeviceExtensions lists extension name strings that the layer strips from vkEnumerateDeviceExtensionProperties results. If your extension is on that list, applications cannot even see it when the capture layer is loaded, so they never attempt to use it and nothing gets recorded.

The second is a generation-time exclusion list in the Python code generator. gfxreconstruct does not hand-write capture and replay handlers for each Vulkan function. Instead, it parses the Khronos XML registry (vk.xml, video.xml) and auto-generates thousands of lines of C++ for encoding, decoding, and consuming API calls. The generator has exclusion lists that tell it which extensions and struct families to skip entirely.

VK_KHR_video_encode_h265 was on both lists.

Removing the runtime block #

This part is trivial: open framework/encode/vulkan_entry_base.cpp and delete the line

VK_KHR_VIDEO_ENCODE_H265_EXTENSION_NAME,

from kVulkanUnsupportedDeviceExtensions. One line. After rebuilding, the layer reports the extension to applications. But that is not enough: without generated capture/replay code, intercepted calls would have no handlers.

Removing the generator exclusions #

The code generator lives under framework/generated/. The file khronos_generators/vulkan_generators/vulkan_base_generator.py maintains two exclusion lists:

• _remove_extensions: extensions whose entire API surface the generator should skip.
• _remove_video_extensions: lower-level codec standard headers (vulkan_video_codec_h265std*) that define the StdVideoH265* struct families.

From _remove_extensions I removed:

VK_KHR_video_decode_h265
VK_KHR_video_encode_h265
VK_KHR_video_maintenance2

From _remove_video_extensions I removed:

vulkan_video_codec_h265std
vulkan_video_codec_h265std_decode
vulkan_video_codec_h265std_encode

I included VK_KHR_video_maintenance2 because its structs interact with video session parameters relevant to H.265 sessions.

The XML patch: when the generator breaks #

After removing the exclusions, I ran the generator and hit an error. The generator could not resolve a len attribute in video.xml for StdVideoH265HrdParameters.

The problematic members are pSubLayerHrdParametersNal and pSubLayerHrdParametersVcl. Both are pointers, and video.xml specifies len="*_max_sub_layers_minus1 + 1" for them. The expression references *_max_sub_layers_minus1, a field that lives in an outer struct (VPS or SPS), not in StdVideoH265HrdParameters itself.

gfxreconstruct’s code generator resolves len expressions by walking the current struct’s members, but it cannot follow cross-struct references. This is a reasonable limitation: the generator would need to understand the full semantics of the Vulkan Video specification to know which outer struct provides the length field.

The fix is a small XML tree patch in gencode.py. Before the generator runs, I strip the len attribute from both members:

hrd_type = video_tree.find('types/type[@name="StdVideoH265HrdParameters"]')
if hrd_type is not None:
    for member_name in ('pSubLayerHrdParametersNal', 'pSubLayerHrdParametersVcl'):
        for member in hrd_type.findall('member'):
            name_elem = member.find('name')
            if name_elem is not None and name_elem.text == member_name:
                member.attrib.pop('len', None)

Without a len attribute, the generator falls back to treating each pointer as pointing to a single element. This is safe for practical capture scenarios where only one sub-layer is in use.

Regenerating the code #

With the generator changes in place, regenerating is a single command:

uv run --with pyparsing python3 framework/generated/generate_vulkan.py

uv run --with pyparsing ensures the pyparsing dependency is available without a manual pip install.

The generator overwrites all files under framework/generated/generated_vulkan_*.cpp and framework/generated/generated_vulkan_*.h. The diff was substantial: hundreds of new functions for encoding and decoding StdVideoH265* structs, plus all the video session parameter handling infrastructure.

The bigger picture #

The interesting part of this exercise was understanding gfxreconstruct’s architecture. The capture layer is not a monolithic block of hand-written interceptors. It is a generator pipeline: Python scripts consume the Khronos XML registry and emit C++ that handles serialization, deserialization, and replay for every struct and function in the Vulkan API surface.

This means enabling a new extension is mostly a matter of telling the generator to stop ignoring it, then fixing any edge cases where the XML description does not match the generator’s assumptions. The actual capture and replay logic comes for free once the generator produces code for the extension’s types and entry points.

If you are considering enabling other video extensions (H.264 encode is still blocked), the same recipe applies: remove from the runtime blocklist, remove from _remove_extensions and _remove_video_extensions, regenerate, and fix any XML len expression issues that surface.

Update on what happened in WebKit in the week from June 22 to June 29.

After a small break after the Web Engines Hackgest, we're back with another round of updates, this time with a couple of exciting improvements to the SVG engine, a WebRTC fix, and support for WebP images with the toDataURL() API.

Cross-Port 🐱

Multimedia 🎥

Graphics 🖼️

Earlier this month, I returned to CSS Day for the first time since 2018 to deliver my first in-person talk since 2022.  “Forging Our Own Paths” should be available at some point; in the meantime, for the six or seven people in my audience who might need to do something similar, I’d like to share a small macOS workflow I developed to make syntax-highlighting code blocks in situ in Keynote a lot simpler.  The end result is to have an entry (or entries) in the Services submenu of the contextual (right-click) menu for highlighted text.  All this is adapted from an old blog post I found copied in a few places, and which needed some updates to make things work in 2026.

First, install highlight.  I used brew install highlight, and the rest of this piece assumes you’ve done it that way.  If you install it another way, such that it ends up in a different location than Homebrew would give it, you’ll need to modify a variable value later on, which I’ll point out when we get there.

Next, you need to install the following (also available as a gist) as a shell script called keynote-highlight:

#!  /bin/bash
set -e
while getopts 'h:o:i:s:t:' OPTION; do
	case "$OPTION" in
		h)
			highlighthome="$OPTARG"
			;;
		o)
			outputrtf="$OPTARG"
			;;
		i)
			inputrtf="$OPTARG"
			;;
		s)
			syntax="$OPTARG"
			;;
		t)
			theme="$OPTARG"
			;;
		?)
			echo "script usage incorrect?" >&2
			exit 1
			;;
	esac
done
shift "$(($OPTIND -1))"

#=============================

inputrtf="$(pbpaste -pboard -prefer public.rtf)"

regex="fcharset0 ([a-zA-Z0-9 ]+);"
if [[ "$inputrtf" =~ $regex ]]
then
	fontface=${BASH_REMATCH[1]}
else
	fontface="Courier"
fi

regex="fs([0-9]{1,5})"
if [[ "$inputrtf" =~ $regex ]]
then
	fontsize=${BASH_REMATCH[1]}
	fontsize2=$((fontsize/2))
else
	fontsize2="12"
fi

if [ -z "$theme" ]; then
	theme="candy"
fi

if [ -z "$highlighthome" ]; then
	highlighthome="/opt/homebrew/bin/highlight"
fi

highlighted=$("$highlighthome" --out-format="rtf" --syntax="$syntax" --style="$theme" --font="$fontface" --font-size="$fontsize2" --no-trailing-nl --stdout)
echo "$highlighted"

Put the script wherever you store your shell scripts, and make sure it’s both executable and can be invoked from the command line.  I believe, without any real basis for doing so, that if you already have syntax-highlight installed, which is (among other things) a wrapper around highlight, you could use it by modifying the highlighthome variable assignment to point to it rather than highlight, as well as modifying a variable in an upcoming bit of code.  But, as I say, I’m just guessing about that.

Once the shell script is installed and ready to execute, launch Automator and create a new Quick Action.  Call it “Syntax Highlight CSS” or something similar.  If you want to set up highlighting for other kinds of code, like HTML or any of the nearly 250 languages (!!!) highlight supports, each language has to be given its own Quick Action.  Thus, if you want them all next to each other in the Services menu, pick an appropriate naming scheme.  For this one, we’re doing CSS, but later you’ll see how you can quickly set up this same thing for other formats.

At the top of the right-hand panel in the new Quick Action workflow, check the “Workflow receives current” dropdown to make sure it’s set to either “Automatic (rich text)” or “rich text”, the latter if you plan to never, ever use this in any non-RTF setting.  I go with the Automatic option.  If you want to restrict the action to a particular application, like Keynote, change the dropdown that says “any application” to pick a specific application.  I leave mine to be available in any application, just in case I’m ever syntax highlighting code in TextEdit or something.  I also set the color to “Red”, because clearly that makes it go faster.

With all those things set, the first thing to add to the workflow is a “Copy to Clipboard” action.  That’s it for this step, just add that and leave it alone.

Now, add a “Run AppleScript” action.  Paste the following (also available as a gist) into the textbox that contains the boilerplate skeleton (replace the skeleton):

on run {input, parameters}
	set highlightHome to "/opt/homebrew/bin/highlight"
	set syntaxType to "css"
	set themeName to "navy"
	set command to "PATH_TO_SCRIPT/keynote-highlight -h " & highlightHome & " -s " & syntaxType & " -t " & themeName
	do shell script "/bin/bash -c 'pbpaste | " & command & " | pbcopy'"
	delay 0.1
	tell application "System Events" to keystroke "v" using command down
end run

Change the PATH_TO_SCRIPT in there to wherever you put the shell script, save the workflow, and it should be ready to go!

…unless your copy of highlight lives somewhere else or you’re trying out using syntax-highlight in its place, or you have a different theme you’d like to use.  In either case, change the value of the corresponding variable in the AppleScript.  As for the syntaxType variable, that’s what you change if you want to highlight HTML or Pascal or BASIC or whatever else, but since we’re doing CSS, leave it as is.

At this point, everything should be ready to go.  In your Keynote slides, wherever you want to syntax-highlight some CSS, drag-select (or select-all) the CSS text in question.  Just be sure you have the text actually highlighted; just selecting the outer text box that holds the text isn’t sufficient.  Right-click on the selected text to bring up the Context menu, and in there open the “Services” submenu.  “Syntax Highlight CSS” (or whatever you called yours) should be in that submenu.  Select it, and after a second or two, the un-highlighted CSS should be replaced with the same thing, except syntax-highlighted.

Well, “the same thing” in the sense of being the same font face and font size it was before you syntax-highlighted it.  If you used a line spacing other than 1.0, it will be reset to 1.0.  This is due to a limitation in highlight, which doesn’t accept line-height values as an argument, and thus will always return text with 1.0 spacing.  It’s likely that other fancy adjustments like kerning will also be reset to default, though I didn’t test them all.  I just know that highlight only accepts font name and size as styling parameters, so those were the only ones I could affect.

I did try to capture the output of highlight and do find-and-replace to restore the line height and tab sizes.  Alas, this was ultimately unsuccessful.  I’m fairly confident this is solvable, but a great deal less confident that it’s solvable by me.  Part of the problem seems to be how highlight returns the RTF, and part of it seems to be some kind of recursive munging of the RTF result (maybe?), and in the end I just gave up.

If you want to set up something similar for HTML, then you need only duplicate the workflow to a different name  —  say, “Syntax Highlight HTML”  —  and then change the value of the AppleScript syntaxType variable from css to html in the new workflow.  That’s all.  Similar steps should be taken to set up a workflow for any other recognized language.

There are a few things to note.

1. If you try to syntax-highlight a block of text that contains multiple font faces or sizes, all of the selected text will be reset to the first face and size, or the script will simply fail to work.  As far as I can tell, preserving the face and size on a per-line (or per-character) basis would be difficult to achieve and probably produce terrible RTF.  As a workaround, highlight each bit of differently-sized or -faced text on its own, and invoke the service.
2. There is no checking to see if the text you highlight matches the type of highlighter to apply to it.  If you try to use “Syntax Highlight CSS” on some HTML, the CSS parser will be used.  So be careful.
3. Picking a new theme can be a cumbersome process, involving a fair amount of trial and error.  It looks like Syntax Highlight has a standalone application that can be used for quick previewing, but I haven’t tried it so I can’t vouch for or against it.
4. I did all this in macOS Sequoia.  I would hope it still works in Tahoe, but if not, let me know in the comments and I’ll add a warning note and a heavy sigh.

There are probably more efficient or more elegant ways to do the individual bits of both scripts, but this works for me, so I figured I’d pass it on to anyone else who’d like to use it.  Improvements, or pointers to solid information that can help me overcome the limitations I mentioned, are always welcome!

Have something to say to all that? You can add a comment to the post, or email Eric directly.

Full room for the WHATNOT meeting that is happening live during the Web Engines Hackfest.

See https://github.com/whatwg/html/issues/12441.

Update on what happened in WebKit in the week from June 9 to June 16.

The major highlight this week is the Web Engines Hackfest! Despite it, there are a variety of updates as well, such as various improvements to input handling in WPE WebKit and WebKitGTK, WPE menu rendering changes, and a plethora of other smaller improvements.

Cross-Port 🐱

WPE WebKit 📟

Community & Events 🤝

Fine. What Is the Web?

In which I am drawn into an unexpected sort of conversation...

I like to feel like I'm working on — or at least toward — something concrete. When things begin to seem too academic or esoteric, or feel disconnected from what appear to be obvious realities, I find it much less interesting. There are clear examples of things I've worked on (or am working on): custom elements, :has, :focus-visible, Custom Properties or even Container Queries. All of these are very concrete, and as such, now that we have them we're also starting to be able to see how successful they are (or aren't). Things take a long time, so in the end, even very concrete proposals can start to feel a bit esoteric when they're so far out ahead of our skis, leaning more and more onto foundations that aren't yet solid.

Anyway... In contrast to this, if you asked me to professionally come up with a definition for "What is the web, exactly?" I have this almost visceral feeling that it's esoteric and I don't want to spend my limited energy on it. I want to run away from it. Far away. Who cares? It means whatever we collectively want it to mean. That's not my jam. It's stuff with URLs. It is not a thing I relish discussing.

But...

Circumstances have put me in a time and place where the question keeps coming up — and I hate to admit it, but I think there are a few reasons to engage with it.

The question surfaced concretely around what belongs as a W3C Recommendation, and more broadly, what belongs at the W3C at all. Its catalog is pretty diverse, actually. Is it all equally "the Web"? The stuff in the browser certainly seems like a special kind of thing. It carries special obligations around privacy, security, internationalization, and accessibility. It runs on just about every device imaginable. But then there's stuff like ActivityPub, JSON-LD, or XML — the browser doesn't do much with those. It could, maybe, but that would come with its own considerations. And yet they're totally relevant, and they're totally the web.

Then there's a whole category of things that have emerged over the past decade pointing toward something... more. Special kinds of apps that come preinstalled (on your TV or your smart toaster), or Electron apps you install yourself, or "super apps" that use web tech for UI while talking to things that aren't really the "drive-by" web we know from the browser. Different rules, but no standards. Yet.

Which of these do the words "web platform" and "web" actually apply to?

If we had a few more names, would it help us organize our thoughts, sharpen our priorities, and shape the overall architecture? Probably.

My current thinking

My current thinking is: I don't know that it is worth defining "the web" very specifically. "The Web Platform," however, I think is best used to describe what lives inside a web engine. And the embedded stuff? I feel like we need a group dedicated to that — an Embedded Web that tries to define something fairly minimal, grounded in the same concerns as the main web engines. We're working on getting people together to talk about this, because it really does affect what we prioritize and the direction we take things.

I recently recorded a podcast on this topic with Dan Appelquist and Eric Meyer called Is this the web?.

I'd love to hear your thoughts.

Update on what happened in WebKit in the week from June 1 to June 8.

Another great week, this time we have a performance improvement implemented in the Skia-based compositor, an excellent writeup about how to investigate and isolate memory leaks in WPE WebKit, a couple of multimedia fixes, and a variety of improvements and fixes across WebKit ports.

Cross-Port 🐱

Multimedia 🎥

Graphics 🖼️

Community & Events 🤝

As part of Igalia’s collaboration with Raspberry Pi, I have previously blogged about several improvements we landed for the Broadcom VideoCore GPU (known as V3D), with the goal of extracting the best possible performance from the hardware. However, performance is not the whole story. On embedded devices, power consumption is just as important: reducing unnecessary activity helps lower heat generation, improve energy efficiency, and preserve performance over time by avoiding thermal throttling.

That is why, over the last few months, we have been working on adding Runtime Power Management support to the upstream V3D DRM driver, allowing the GPU to be powered and clocked according to its actual usage.

Why Runtime Power Management?

In the Linux kernel, Runtime Power Management (known as Runtime PM) is the mechanism that allows individual devices to be suspended and resumed dynamically while the system as a whole remains running. Instead of keeping a device fully powered all the time, the kernel can put the device into a low-power state when it is idle and bring it back when it is needed again.

In the graphics context, it is easy to see why runtime PM can be useful. A GPU is not necessarily active all the time: it may be heavily used while rendering a scene, but remain idle for long periods afterwards. If the driver keeps the GPU clocked during those idle periods, the system keeps spending energy on a block that is not doing useful work at all.

For embedded platforms, this is even more pressing. Reducing unnecessary power consumption helps decrease heat generation and improve overall energy efficiency. Even if the board is not battery-powered, avoiding needless power usage can reduce the need for cooling and leave more thermal budget available for other blocks.

The Problem: an idle GPU with an enabled clock

Until now, the V3D driver had a very simple power model: the GPU clock was enabled during probe and remained enabled for the entire lifetime of the driver. In practice, this meant that once the driver was loaded, the V3D clock stayed on until the driver was removed, regardless of whether the GPU was actively executing jobs. This was simple and functional, but it meant that an idle GPU was not idle from a power-management point of view.

On Raspberry Pi platforms, this is easy to observe with vcgencmd. Even with no GPU workload running, the V3D clock would still report an enabled frequency:

$ vcgencmd measure_clock v3d
frequency(0)=960016128

If the GPU is idle, the driver should be able to let the hardware become idle as well. Runtime PM provides the kernel infrastructure for that, but enabling it in the V3D driver required a bit more than simply adding suspend and resume callbacks.

Making the Raspberry Pi firmware clocks obey

At first glance, adding Runtime PM to V3D might look like a driver-local change, but in practice, things were a bit more subtle.

On Raspberry Pi platforms, some clocks are managed by the Raspberry Pi firmware. From the V3D driver’s point of view, this is supposed to be mostly transparent: the driver uses the standard Linux clock framework, and the clock provider takes care of talking to the firmware underneath. However, this abstraction only works if calls to clk_prepare_enable() and clk_disable_unprepare() are translated into actual firmware requests to enable and disable the clock.

Surprisingly, that was not happening. The Raspberry Pi firmware clock driver did not implement the prepare/unprepare hooks, so these calls did not actually ask the firmware to enable or disable the clock. We fixed that by translating the common clock framework operations into the corresponding Raspberry Pi firmware commands [1][2][3].

However, there was still one firmware-specific caveat: on current firmware versions, RPI_FIRMWARE_SET_CLOCK_STATE does not fully power off the clock as expected. To work around this limitation and achieve meaningful power savings, the clock rate also needs to be set to the minimum before disabling the clock. This behavior may change in future firmware releases, but for now the clock driver needs to account for it explicitly.

With the firmware clock limitation addressed, the V3D driver could start relying on the usual kernel clock APIs as part of its Runtime PM flow. The next step was to reorganize the driver so that powering the GPU up and down became part of its operation.

Introducing Runtime PM to V3D

With the clock side behaving as expected, we could move the V3D driver itself to a Runtime PM model [7][8][9].

This required a small refactor of the probe path to separate power-independent setup from GPU-powered initialization. Resources that do not require the GPU to be powered are allocated during probe, while any initialization that depends on the GPU being clocked is handled during runtime resume. Runtime suspend then disables the clock again when the device becomes idle. The resulting flow is simple:

With that in place, the change becomes visible from userspace. While a GPU workload such as glmark2 is running, the V3D clock is enabled:

$ vcgencmd measure_clock v3d
frequency(0)=960016128

After the workload finishes and the GPU becomes idle, the clock can drop back to zero:

$ vcgencmd measure_clock v3d
frequency(0)=0

This is the behavior we wanted: the GPU remains available when there is work to do, but it no longer keeps its clock enabled while idle.

Results

To evaluate the effect of Runtime PM, we measured the board’s power consumption with an external power meter in three scenarios: an idle desktop session with labwc running, an idle system without the compositor, and a full glmark2 run. Each condition was sampled at 100 Hz for around 300 seconds.

The first case represents a mostly idle graphical session, where labwc, the compositor used by Raspberry Pi OS, may still wake the GPU occasionally. The second is a baseline with no graphical workload, while the third is a sustained GPU benchmark intended to keep the GPU active.

The numbers behave the way one would hope. When the GPU is genuinely idle, the clock can be gated off and the savings show up as a clear drop: average draw falls from 3.30 W to 3.19 W with labwc running, and from 3.18 W to 3.09 W with no compositor at all. Both idle scenarios end up with savings of about 0.1 W (around 3%). Under glmark2, where the GPU is doing useful work for most of the run, the difference shrinks to about 0.015 W (0.3%), which is expected, as Runtime PM mainly affects the periods where the GPU becomes idle.

The distribution of idle samples with labwc running also shows the effect clearly. With Runtime PM enabled, the distribution shifts toward lower power states. This indicates that the board spends more time in lower-power idle states once the V3D clock is no longer kept enabled unnecessarily.

The effect is even cleaner with no compositor running. The samples collapse into two very narrow peaks with no overlap between them: without Runtime PM, the board sits at a stable 3.18 W; with Runtime PM, it sits at a stable 3.09 W.

For glmark2, the time-series data shows that both configurations follow the same general workload pattern. Runtime PM does not significantly change the power profile while the GPU is busy, which is the intended behavior. The benefit appears when the workload leaves idle gaps or finishes, allowing the clock to be disabled again.

Overall, these measurements show that Runtime PM reduces power consumption where it matters most: when the GPU is idle. The absolute savings are modest at the board level, since the measurement includes the whole Raspberry Pi rather than the GPU power block alone, but the reduction is consistent with the intended change. The V3D clock no longer remains enabled for the full lifetime of the driver, and that translates into measurable reductions in idle power consumption.

Conclusion

Runtime PM support for V3D is one of those changes that is easy to overlook when everything is working correctly: userspace does not need to do anything differently, applications keep using the GPU as before, and the improvement happens underneath, in the way the kernel manages the hardware.

Beyond improving raw GPU performance, our work at Igalia is also about making the upstream graphics stack behave better as a system: more efficient when idle, more robust across firmware interfaces, and better aligned with the expectations of the Linux kernel infrastructure.

References

[1] clk: bcm: rpi: Turn firmware clock on/off when preparing/unpreparing - kernel/git/torvalds/linux.git - Linux kernel source tree

[2] clk: bcm: rpi: Maximize V3D clock - kernel/git/torvalds/linux.git - Linux kernel source tree

[3] clk: bcm: rpi: Manage clock rate in prepare/unprepare callbacks - kernel/git/torvalds/linux.git - Linux kernel source tree

[4] clk: bcm: rpi: Mark VEC clock as CLK_IGNORE_UNUSED - kernel/git/torvalds/linux.git - Linux kernel source tree

[5] pmdomain: bcm: bcm2835-power: Increase ASB control timeout - kernel/git/torvalds/linux.git - Linux kernel source tree

[6] pmdomain: bcm: bcm2835-power: Replace open-coded polling with readl_poll_timeout_atomic() - kernel/git/torvalds/linux.git - Linux kernel source tree

[7] drm/v3d: Use devm_reset_control_get_optional_exclusive() - kernel/git/next/linux-next.git - The linux-next integration testing tree

[8] drm/v3d: Allocate all resources before enabling the clock - kernel/git/next/linux-next.git - The linux-next integration testing tree

[9] drm/v3d: Introduce Runtime Power Management - kernel/git/next/linux-next.git - The linux-next integration testing tree

Last week the Embedded Recipes conference was held in Nice, France. Igalia was sponsoring the event, and like last year, my colleague Martín and myself were attending. Unlike last year, we weren’t presenting, which for me means less stress and more opportunities for hallway conversations.

The event was extremely well organized, in a really cool venue (Parc Phœnix, in Nice) for the second year in a row. Kudos to the team at BayLibre!

The selection of talks was overall quite interesting and relevant. Here are a few of my personal highlights:

• Yocto Project and the Cyber Resilience Act where Paul Barker (Yocto Project) gave a few relevant definitions (Product with Digital Elements, stewards vs manufacturers) and discussed how this affects the Yocto project, which is essentially tooling, and what the project plans on implementing to remain compliant. [Recording]

• U-Boot on boot core as an always-on debug tool where Marek Vasut presented a clever use of the separate Cortex-M33 core used for boot only to run U-Boot to get access to the Cortex-A core that runs Linux. This is intended for development purposes only, not to be deployed in production. Really cool if you’re into that sort of low-level bringup work. [Recording].

• A Distributed Phone CI for postmarketOS where Pablo Correa Gómez walked the audience through the different attempts at implementing a CI pipeline running on physical phones by the postmarketOS project. This involved advanced custom hardware design, and the use of CI-Tron as the orchestrator. [Recording].

• Four NPUs, One Stack, Zero Blobs: Edge AI Acceleration in Mainline where Tomeu Vizoso presented his work on the kernel userspace APIs and Mesa drivers to enable a truly open-source AI stack. [Recording].

My personal interests normally tend to drive me towards higher-level concerns and constructs, which is why I feel I learnt so much in just two days, being immersed in a sea of hardware and low-level software to control it.

The social event on the beach at the end of the first day was a perfect opportunity for networking, to meet old friends and new folks alike.

I flew home after the end of the two days of conference, but fellow Igalians stayed/arrived to attend the following colocated events: Yocto Project Workshop, GStreamer Spring Hackfest, Display Next Hackfest.

I sure hope to be attending again next year.

Update on what happened in WebKit in the week from May 19 to June 1.

The main feature of this week are new releases: stable ones with many security fixes, and development ones with the new Skia-based compositor enabled. Additionally, there was work on Web-facing features, optimizations, spell checking support for the WPE port, and more.

Cross-Port 🐱

WPE WebKit 📟

Releases 📦️

Infrastructure 🏗️

Depending on the web application, the WPE WebKit memory usage trend can vary. When simple web applications are being processed, the memory consumption tends to be virtually stable (the same) no matter the period. However, when more complicated web applications are being executed, the memory usage usually grows over time while going back to normal from time to time e.g., when GC / memory pressure mechanism releases all kinds of caches and not-needed memory. Therefore, memory growth itself is not unusual. Nevertheless, as the memory leaks happen in WPE at times, the memory growth is worth investigating — especially if very rapid or unbounded.

This article presents a structured playbook for investigating such a memory growth and memory leaks in WPE. Rather than diving straight into debugging tools, it starts from first principles: confirming the problem is real, choosing the right environment to work in, and narrowing down the leaking area before any heavy tooling is involved. The goal is to reach actual debugging as fast as possible, regardless of whether the environment is an embedded device or a desktop machine, and regardless of how quickly the problem reproduces.

Playbook #

The high-level list of recommended steps to follow is presented below. In a nutshell, the steps 1, 2, and 3 are meant to choose and follow the fastest possible investigation path so that actual debugging of the problem (step 4) can be started as soon as possible.

1. Confirming the problem
2. Identifying the best setup for reproducing the problem
3. Narrowing down
   • on embedded when the problem takes a long time to reproduce.
   • on embedded when the problem reproduces quickly,
   • on desktop when the problem takes a long time to reproduce,
   • on desktop when the problem reproduces quickly,
4. Debugging

1. Confirming the problem #

The ultimate first step when working with alleged memory leak is to check whether the observed memory growth is actually abnormal. In the case of web browsers in general, the memory growth alone may not necessarily mean something is leaking. There may be many regular reasons why the browser’s memory usage is growing, but the usual suspects are:

• JavaScript-level memory allocations — due to the very nature of JavaScript, the memory it allocates causes the overall web content process memory growth up until the garbage collector (GC) kicks in. Then (from the RSS perspective) some memory is usually freed. However, as it’s not easy to predict when the GC will be invoked (e.g., when the browser processes an application that performs heavy rendering), it’s possible that memory will grow but remain garbage-collectible.
• JavaScript Just-in-Time (JIT) compilation — when not explicitly disabled or limited, the processing of any web application that has JavaScript code associated with it will cause the browser to continuously compile the JavaScript code in the background so that it executes such code faster in runtime at the expense of memory that is required for storing compiled artifacts.
• Caches — as the WPE operates, it caches things such as web resources, style resolution artifacts, textures, glyph atlases, layer tiles, display lists, rasterization artifacts, and many others. Naturally, the cache sizes are limited, however, if many caches are growing at the same time, they may create an impression of a leak. The difference in that case is, the caches stop growing at some point.

Due to the above, to confirm the memory growth is abnormal, one should usually try the following first:

1. Triggering memory pressure to force the browser to trigger GC and evict as many cache entries as possible,
2. Rerunning the browser with JIT disabled to rule out the JIT-related memory growth — unless the application code is very small.

If the memory growth doesn’t stop with JIT disabled or its level does not go back to normal after triggering memory pressure, the growth can be assumed to be abnormal, and one can proceed to the next step.

2. Identifying the best setup for reproducing the problem #

When the memory growth is atypical, it needs to be narrowed down in a way that the final debugging is possible. For both narrowing down and the debugging, one should aim at the most flexible development environment along with the smallest possible web application that reproduces the problem quickly. What it means in practice is — desktop environment along with small demo web application that reproduces the problem. Whilst it’s not always possible to have such an environment, the 3 general rules are as follows:

1. Desktop environment is usually better than embedded one in terms of working with memory leaks as it offers minimal overhead (e.g., in terms of compilation times) and huge flexibility in choosing the industry standard tools for profiling/debugging.
2. Small web application is always better than a big one as long as it still reproduces the same problem in the same amount of time. In such case, a small application minimizes the amount of noise that usually stands in the way of profiling/debugging.
3. A web application that reproduces the problem quickly is always better than the one that needs much more time for it. The worst thing that can happen in the case of narrowing down memory leaks, is when the memory growth is noticeable or starts after a very long time such as hours/days+.

Given the above, at this point one should go through the below steps:

1. Check if the setup is trivial enough already — if the web application reproduces the problem quickly in a desktop environment and is simple enough, one should immediately jump to the Debugging section.
2. Check if the problem can be reproduced on desktop assuming it originally reproduces on embedded.
3. Check if the problem can be reproduced faster if it’s not reproducing fast enough.
4. Check if the web application could be simplified.

Once the setup is simplified as much as possible, one should proceed to one of narrowing down sections depending on the setup. Also, if the setup is still not ideal, one should actively seek opportunities for simplifying the setup even during narrowing down as it’s likely that some new information will eventually open new possibilities in terms of simplifying setup.

3. Narrowing down #

When the problem has been confirmed but there are not enough clues to tell exactly which parts leak, the debugging cannot be started right away. In such case, it’s necessary to narrow down the problem to the browser/application area that can be easily debugged.

While in some cases narrowing down is not even necessary, quite often it takes orders of magnitude more time than actual debugging, and hence one should pay special attention to this step.

3a. Narrowing down on embedded when the problem takes a long time to reproduce #

This is the toughest situation one can find themselves in. When a problem takes a long time to reproduce (hours/days+), every iteration/test comes automatically with a significant cost. Moreover, when the environment is an embedded one, rebuilding WPE is usually more time-consuming and the amount of tooling is usually limited — or requires some work to bring it to the image at least.

Due to the above, narrowing down the problem in this setup requires a structured approach with extra care. In such case, the things to check should be approached in steps defined as follows:

1. Things to check without rerunning the WebKit
   • in case of embedded devices, extra care is needed when attaching a memory profiler. On low-end devices, memory profilers tend to slow down the application hard enough to trigger otherwise non-existent problems.
2. Things to check without rebuilding the WebKit
   • in case of embedded devices, one should prefer limiting JIT over disabling it as without it, the JS execution may be slow enough to trigger unexpected scenarios.
3. Things to check if rebuilding WebKit

Ideally, while checking various things along the above steps, one should batch as many checks as possible within individual tests.

3b. Narrowing down on embedded when the problem reproduces quickly #

When the problem reproduces quickly, the limitations of embedded environment are not that relevant. In this scenario, one should prioritize getting debug symbols (RelWithDebInfo build) into the image and utilizing them by running the browser with whatever profilers are available. For the specific things to check, one should seek inspiration in the following groups:

1. Things to check without rebuilding the WebKit.
2. Things to check if rebuilding WebKit.

3c. Narrowing down on desktop when the problem takes a long time to reproduce #

This situation is similar to 3a and hence one should follow the things to check from the following groups:

1. Things to check without rerunning the WebKit.
2. Things to check without rebuilding the WebKit.
3. Things to check if rebuilding WebKit.

However, this time, there are some extra opportunities around tooling:

1. There should be many more tools available already in the system or available to be installed.
2. Tools such as memory profilers that could slow down the application making it unusable on embedded, may turn out to be working well when the desktop-class processing power is available.

With the above in mind, it’s worth trying all the tools available with priority because if at least one tool works well, one can save hours of narrowing down.

3d. Narrowing down on desktop when the problem reproduces quickly #

This is technically the simplest possible scenario, so basically, all the possibilities are available. The most time-consuming activity in this case is very likely rebuilding WebKit itself — although it should still be relatively fast. In such case, just after a few quick checks with the Web Inspector, it’s recommended to get debug symbols (RelWithDebInfo build) and start with tools such as memory profilers.

Other than the above, one should go through the following groups on things to check:

1. Things to check without rebuilding the WebKit.
2. Things to check if rebuilding WebKit.

4. Debugging #

The WPE debugging is twofold and depends on whether the problem is within the engine (usually C/C++ code) or the web application (JavaScript code).

When problem lies in the engine #

Debugging WPE WebKit is the same as debugging any other C/C++ application on Linux (or Mac if the issue is cross-port and one prefers an Apple port to work with), and hence is outside the scope of this article. Some WebKit-specific information can be found in the WebKit Documentation article on building and debugging page and therefore is recommended as a first step.

When problem lies in web application #

When the problem lies in JavaScript code, the situation is usually fairly straightforward. The majority of bugs in this area should be reproducible across various browser engines and hence a full variety of tooling should be available. If the WebKit is preferred or if the problem reproduces only there, the tooling available is still very useful and helps debugging problems quickly. The ultimate tool in such case is the Web Inspector. On official WebKit’s web page there’s entire index of articles on Web Inspector. Among those, the most interesting read is about Timelines Tab where the most useful debugging can be done. Once the features of Timelines Tab are understood, the next important article is the memory debugging guide. It dives into the most important Timelines Tab subsections and showcases the work with heap snapshots which is a key. To supplement it, it’s very important to know the heap snapshot delta feature which is basically about button:

that allows one to inspect the delta-snapshot between 2 snapshots. It’s critical as it answers the question on what JS objects were added between the base snapshot and the later one. If some objects are piling up, it immediately shows which ones.

One important note on snapshots is that in some cases when using Web Inspector is not possible, one can generate the snapshots manually from the web engine’s C++ code by just calling GarbageCollectionController::singleton().dumpHeap(); at some appropriate moment. In this case, the dump will be written to standard output. It can be then turned into a file and imported from any Web Inspector using Import button.

As the Timelines Tab with its subsections should be able to answer on what happens, to understand why it actually happens, the last missing piece is the JS debugger within Web Inspector. It’s not very different to debuggers in other engines, but it’s worth checking a dedicated article on it just to understand the capabilities.

Appendix #

Things to check without rerunning the webkit #

Even if the WPE is running with default settings in release mode, there are plenty of useful things that can be checked while the browser is still running:

1. Identifying which WebKit process allocates abnormally,
   • there are multiple ways to do this, but usually it’s as easy as using ps utility.
2. Identifying how fast the process in question allocates the memory,
   • this is useful to know at least for comparison purposes, but it may hint some problems already if the numbers correlate with what web application does.
3. Checking logs from stdout, stderr, and journal (using journalctl).
4. Checking detailed process memory statistics.
5. Triggering and checking the impact of memory pressure on given processes RSS,
   • in short, memory pressure triggers the cleanup of the majority of caches along with GC. Therefore, if this is able to bring memory back to normal level, then the problem is about caches, JS Heap / GC, or fragmentation.
6. Attaching memory profilers if available,
   • even if the debug symbols are not present, this may be useful to see what data is being captured and how the web application behaves when slowed down by profiler.
7. Attaching other tools if available,
   • even if the debug symbols are not present, various tools offer different perspectives on what the browser is doing. In some cases, such information may reveal some anomalies that may be related to the main issue.
8. Cross-checking with other browsers,
   • if other browsers show a similar pattern of memory usage, it’s very likely the problem lies in web application itself. Otherwise, it strongly suggests a bug in the WPE.
9. Cross-checking with other ports,
   • if any other WebKit port shows a similar pattern of memory usage, it allows one to narrow down the area in the code a bit based on what port it is:
     • if the same behavior is visible in any of Apple ports, the problem is most likely related to cross-platform code,
     • if the same behavior is visible only in GTK port, then the problem is most likely related to GLib-related part, coordinated graphics part, GStreamer-related part, or others that are shared.

Things to check without rebuilding the webkit #

1. Tweaking and checking the logs from WPE,
   • while generic logs may hint some unusual behavior, more specific ones such as GC logs (JSC_logGC=1) may be used to check how the individual JS heap sizes evolve over time and how GC behaves. If it’s JavaScript leaking the memory, this log will quickly provide the evidence.
2. Enabling Remote Web Inspector and checking:
   • both breakdown and trend of memory usage in the memory timeline after doing a bit of recording,
   • the effects of takeHeapSnapshot() invoked from JS console:
     • as this function usually triggers GC internally, it may be used to check how much RSS memory is reclaimed by GC in isolation (followed up by scavenger),
     • as this function takes a JS heap snapshot, it then can be used to explore manually if its contents point towards something interesting.
3. Disabling JIT and checking the memory usage,
   • if the memory usage is stable with JIT disabled, one should proceed to the step below.
4. Limiting JIT and checking the memory usage,
   • there are at least a few places (levels) where JIT compilation engine allocates memory. If limiting doesn’t resolve the issue completely, it’s likely the engine itself leaks some memory around temporary helper-heaps such as AssemblerData etc.
5. Experimenting with environment variables and runtime preferences,
   • some environment variables and runtime preferences change the behavior of the web engine significantly. If changing one of them makes the problem go away, it usually helps to narrow down the problematic area quickly.
6. Running WPE with system malloc (environment variable Malloc=1) and checking the memory usage,
   • when one suspects bmalloc/libpas issues with fragmentation or scavenger, it’s worth running a browser with system malloc to compare the memory evolution over time against the bmalloc/libpas.
7. Limiting device memory and checking the memory usage,
   • if triggering memory pressure is not possible, an alternative solution is to limit the device memory so that the browser is under constant memory pressure.
8. Running WPE with sysprof and checking:
   • stack traces — to see what parts of engine are particularly active as it may hint some problematic area,
   • WebKit marks — to see what the engine is doing as well as quantitative data in marks such as EventLoopRun etc. as in those cases the numeric value trends may reveal resource pile up.

Things to check if rebuilding webkit #

1. Building WPE in release mode with debug symbols and re-trying memory profilers or other tools if the debug symbols were not present before,
   • if some desired tools such as heaptrack, valgrind, perf, or strace were not available before, it’s the right moment to get/build them as well,
   • once the debug symbols are in, one should try:
     • attaching memory profilers if available, and inspecting detailed memory allocation reports,
     • attaching other tools if available, and investigating the traces.
2. Building and running with Google perftools,
   • as WPE allows switching to system malloc as an allocator, it’s possible to use custom malloc implementation with instrumentation such as gperftools. For that, the recommended read is this article from fellow Igalian, Pablo Saavedra.
3. Building and running with sanitizers,
   • if the problem is about low-level leak, address/leak sanitizer should be able to help pointing out the problematic area.
4. Building and running with memory sampler,
   • the data produced by memory sampler is roughly the same as inspector’s memory timeline, however, it’s much more convenient as it doesn’t need web inspector at all.
5. Building and running with node statistics,
   • when memory growth seems to be related to DOM mutations, it’s worth enabling and reporting node statistics periodically — in some cases, it may directly suggest what the problem is about.
6. Building and running with malloc heap breakdown,
   • when all other means fail, a very good last-resort approach for investigating memory usage statistics via a debug-only WebKit feature called Malloc Heap Breakdown. The details can be found in the dedicated article about it.
7. Building and running with libpas statistics,
   • On very rare occasions such as memory fragmentation or allocation issues, it may be worth checking the libpas (low-level memory allocation and management library) statistics as WPE uses it by default on the vast majority of platforms.

Individual instructions #

Checking detailed process memory statistics #

As WPE WebKit uses multi-process architecture, there are multiple processes that can be checked, although the most interesting one is usually the Web Content Process. Once the PID of the given process is determined (e.g., using ps utility) the usual steps to check detailed memory statistics are:

• cat /proc/<PID>/status or cat /proc/<PID>/statm for very basic statistics,
• pmap -X <PID> - for detailed statistics (if available),
• cat /proc/<PID>/smaps_rollup and cat /proc/<PID>/smaps for detailed statistics (requires CONFIG_PROC_PAGE_MONITOR kernel configuration option).

Triggering memory pressure from OS #

WPE uses a so-called Memory Pressure Monitor to observe the memory usage in the system and to react if there’s not much memory left. The default thresholds are specified in MemoryPressureMonitor.cpp and usually are 90% for non-critical and 95% for critical response. Depending on the response, WPE schedules GC and clears internal caches immediately.

As the above is usually on by default, one can leverage it to trigger GC (along with cache cleanups) by filling up the available memory in the OS to 95+%. There are many ways to allocate memory, yet the simplest is using stress:

• e.g. stress --vm 1 --vm-bytes 1024M --vm-keep to allocate 1024 MB.

Attaching memory profilers #

When attaching any memory profiler, unless one wants to profile only native allocations (Skia, GStreamer, ICU, etc.), the key is to use Malloc=1 environment variable on WPE startup so that bmalloc uses system malloc instead of libpas. Also, if WebKit is using a sanboxed mode in given configuration, it’s usually necessary to use WEBKIT_DISABLE_SANDBOX_THIS_IS_DANGEROUS=1 as well. Then the commands are as follows:

• to attach heaptrack:
  • heaptrack -p <PID> so e.g. heaptrack -p $(pgrep WPEWebProcess) (see this article for details),
• to run with valgrind’s massif (as attaching to running process is not possible):
  • valgrind --tool=massif --trace-children=yes <WPE-BROWSER-COMMAND> (see this article for details).

Attaching other tools #

If memory profilers are unusable or unavailable, it’s worth checking if other tools are present and experimenting a bit with them if so. In some cases, tools other than memory profilers may give some hints on further investigation or reveal a suspicious pattern within application execution. Some ideas for experiments with various tools are listed below:

• strace:
  • strace -c -p $(pgrep WPEWebProcess) — strace called with -c gives a nice summary of system calls executed by the traced application. It can be useful to check the overall syscall usage pattern to see if there are any anomalies.
  • strace -p $(pgrep WPEWebProcess) -e trace=mmap,munmap,mremap,madvise -tt — strace focused on mmap()-related system calls may be useful to debug libpas.
• perf:
  • perf record -F 999 -ag -p $(pgrep WPEWebProcess) -- sleep 60 — regular recording with perf can be very useful, especially if symbols are available. With that, one can generate flamegraphs and investigate what’s going on in the browser. While it’s not about profiling memory, it may be helpful to narrow down at least a bit.
  • perf record -F 999 -e syscalls:sys_enter_mmap,syscalls:sys_enter_munmap,syscalls:sys_enter_mremap:sys_enter_madvise -ag -p $(pgrep WPEWebProcess) -- sleep 60 — perf focused on mmap()-related system calls is much more superior than e.g. strace as it also records stack traces. Therefore, if debug symbols are present, and if the memory growth is very rapid, it’s very likely the libpas mmap() stacktraces will lead to the growth origin statistically.
  • perf trace -e mmap,munmap,mremap,madvise -p $(pgrep WPEWebProcess) — this is very much similar to strace focused on mmap()-related system calls as it shows a live preview of what’s happening.
• sysprof:
  • sysprof-cli -f — while running system-wide sysprof won’t make WPE push marks into it, the profiling trace may still be useful to some degree, especially if debug symbols are available.

Disabling JIT #

This can be done using an environment variable:

• JSC_useJIT=false.

Limiting JIT #

Limiting JIT can be achieved via environment variables:

• JSC_jitMemoryReservationSize=<BYTES> to limit JIT memory usage (the limit is semi-strict as some JIT compilation engine buffers are limited by this value indirectly),
• JSC_useFTLJIT=false to disable FTL tier,
• JSC_useDFGJIT=false to disable DFG and FTL tiers,
• JSC_useBaselineJIT=false to disable Baseline, DFG, and FTL tiers.

Tweaking WPE logs #

WPE is a fairly complex piece of software and hence it offers various logging capabilities related to WebKit itself, as well as to related libraries. The vast majority of logging can be controlled via environment variables:

• WEBKIT_DEBUG=all to enable all logging channels,
• WEBKIT_DEBUG=Layout,Media=debug,Events=debug to enable selected logging channels,
• JSC_logGC=2 to enable JS garbage collector logs,
• GST_DEBUG=4 to enable gstreamer (multimedia-related) logs (see the documentation),
• G_MESSAGES_DEBUG=all to enable GLib-level logs.

If MiniBrowser (or similar browser) is used, one can also set a runtime preference to enable JS console.log(...) logging to the standard output:

• --features=+LogsPageMessagesToSystemConsole.

Enabling remote web inspector #

Enabling WPE’s remote web inspector is a twofold process:

1. The first step is to run WPE with the proper environment variable so that it starts listening on IP:PORT using tcp socket:

• WEBKIT_INSPECTOR_SERVER=IP:PORT is the most reasonable option as it uses inspector:// protocol that can be utilized by WebKit-native browsers such as GNOME Web (Epiphany) or Safari,
• WEBKIT_INSPECTOR_HTTP_SERVER=IP:PORT is a less preferable alternative that uses HTTP protocol and technically works from any browser. However, no seamless integration is guaranteed in this case.

2. The second step is to connect from a regular web browser to the WPE:

• using inspector://IP:PORT/ if native inspector server was started,
• using http://IP:PORT/ if HTTP inspector server was started,
• forwarding the ports using socat tcp-l:PORT,fork,reuseaddr tcp:IP:PORT if the WPE is running in unreachable network.

Experimenting with environment variables and runtime preferences #

The most outstanding environment variables changing the behavior of WPE are the following:

• WPE_DISPLAY — assuming the new WPE platform API is used, this environment variable allows one to switch the pre-defined platform implementation thus changing a platform-facing part of graphics pipeline. The valid options are:
  • WPE_DISPLAY=wpe-display-headless — for headless implementation,
  • WPE_DISPLAY=wpe-display-drm — for direct rendering manager integration,
  • WPE_DISPLAY=wpe-display-wayland — for wayland integration,
• WEBKIT_SKIA_ENABLE_CPU_RENDERING — when set to 1, rendering the DOM contents to the layers is done using Skia CPU backend instead of GPU one.

The most outstanding runtime preferences changing the behavior of WPE are the following:

• CanvasUsesAcceleratedDrawing — when disabled, 2D canvas will use Skia CPU backend instead of GPU one,
• LayerBasedSVGEngine — when enabled, WPE uses a different SVG engine internally,
• AcceleratedCompositing — when disabled, WPE uses experimental, non-composited mode that bypasses all of the compositor work.

Limiting device memory #

On the majority of embedded devices, the device memory can be limited by:

1. Interrupting the boot sequence (usually holding some key such as z upon booting),
2. Invoking the command to change the limit and booting, e.g.:

   > global linux.bootargs.console="console=ttymxc0,115200n8 mem=2G"
   > boot
   

Running WPE with sysprof #

Regardless of whether it’s done on desktop (using wkdev-sdk) or on embedded device, the command is always as simple as:

• sysprof-cli -f -- <WPE-INVOCATION>.

See the documentation entry for more details.

Building WPE in release mode with debug symbols #

On desktop, the simplest way to get release with debug symbols is to utilize CMake’s build type by using -DCMAKE_BUILD_TYPE=RelWithDebInfo within WPE build command, so:

• ./Tools/Scripts/build-webkit --wpe --release --cmakeargs="-DCMAKE_BUILD_TYPE=RelWithDebInfo".

On embedded, when Yocto is used, one should tweak settings such as:

IMAGE_GEN_DEBUGFS = "1"                                                         
IMAGE_FSTYPES_DEBUGFS = "tar.bz2"
DEBUG_BUILD = "1"
EXTRA_IMAGE_FEATURES_append = " dbg-pkgs"

and potentially INHIBIT_PACKAGE_STRIP to control whether debug symbols should be kept with the binary or not. This may be necessary occasionally as some tools have problems reading .gnu_debuglink and therefore work only with symbols included in the binaries.

Building and running with sanitizers #

WebKit works pretty well with all kinds of sanitizers. To build with any of them a CMake-level helper called ENABLE_SANITIZERS can be used by specifying -DENABLE_SANITIZERS=address, -DENABLE_SANITIZERS=leak etc. With that, the command for building e.g. on desktop could look like:

• ./Tools/Scripts/build-webkit --wpe --debug --cmakeargs=-DENABLE_SANITIZERS=address.

For more details, one can refer to this article from fellow Igalian, Fujii.

Building and running with memory sampler #

When WPE is built with -DENABLE_MEMORY_SAMPLER=ON, the simple memory sampler can be started along with the browser using environment variable:

• WEBKIT_SAMPLE_MEMORY=1 (accompanied by WEBKIT_DISABLE_SANDBOX_THIS_IS_DANGEROUS=1 if needed).

With that, the memory of various WPE processes is sampled every second, and saved to the files under /tmp directory continuously.

Building and running with node statistics #

Node statistics are a debug-only feature that can be enabled by:

• changing 0 of #define DUMP_NODE_STATISTICS 0 to 1 in Source/WebCore/dom/Element.h,
• adding dumpStatistics() call, to e.g. Node constructor in Source/WebCore/dom/Node.cpp.

Building and running with libpas statistics #

Libpas statistics are a debug-only feature that can be enabled by changing 0 of #define PAS_ENABLE_STATS 0 to 1 in Source/bmalloc/libpas/src/libpas/pas_config.h and then running WPE with environment variable PAS_STATS_ENABLE=1.

Motivation

Like most people, I've been playing with agents to see where they're helpful, where they're not, and what kind of workflows are a good match for me. One area I've found friction is in iterating on a piece of code - written by me or otherwise. I can describe the relevant section and my question/request in command line chat, but it would be better to do so directly inline and have the LLM pick it up. at-agent is a super minimal approach for picking out such directives from your worktree and processing them.

This is very much a "worse is better" approach. A separate structured channel for attaching questions or requests to spans of code would have advantages. But that requires an interface for creating and editing such annotations as well as logic for handling edits after the annotation was made. Sticking @agent comments directly inline means it's trivial to intermingle manual edits with requests for action, it's trivial to keep comments attached to the region they were intended for, and reviewing and editing them at the file or repository level is easy through git diff and your text editor of choice.

You could get away with just asking your LLM of choice to "find all comments prefixed with @agent in this codebase, treat them as directions to you, action them, and then remove them". But I still believe in building on solid primitives, and limited as this little script is, I'd rather lean on its deterministic behaviour and reduce the number of round trips and tool calls the LLM needs to handle successfully. So far it's been helpful for some kinds of tasks and a handy tool to have in the toolbelt.

There are surely no end of IDE-integrated solutions and vim plugins that offer something similar. aider also supports 'AI' comments but relies on the model to remove them after the fact.

Interface

at-agent doesn't try to understand language-specific comment syntax. It looks for a whole line that starts with optional whitespace, then at least two punctuation characters such as //, ##, or ///, then whitespace and @agent. A following ! marks an action request rather than a question, and the rest of the line is the request text. Subsequent lines with the same comment prefix are consumed as part of the same @agent directive.

What this means is that you might write something like this:

Example: // @agent explain why std::vector isn't used here. How does the
Example: // custom vector compare in terms of reallocation strategy?

The Example: prefix is only there to keep this blog post from containing live annotations. Without it, running at-agent over the Markdown file would treat the example itself as a real request and remove it.

Or for an action request:

Example: ## @agent! split this into a helper and update the two other callers

@agent is a comment/explanation request, and @agent! is a request to make an edit. Only whole-line annotations are supported. If you want to nest a request inside a long comment, just tweak the prefix appropriately, e.g.:

Example: // This is a multi-line comment. We want to place a directive in it.
Example: /// @agent explain the paragraph below, with a worked example
         /// alongside appropriate source code snippets.
Example: // Normal comment continues here.

Running the tool removes the annotation lines from the working tree and emits something like:

The user manually added these annotations for you, the agent, to react to.
They are either comment/explanation requests, which ask you to explain nearby
code or answer the user's question without editing files, or action requests,
which ask you to investigate and make the requested code change.

The @agent remark lines have already been removed from the working tree.
Relevant line numbers refer to the files after remark removal.

1. example.cc
   kind: comment/explanation request
   relevant line number: 42
   text:
     @agent explain why std::vector isn't used here. How does the
     custom vector compare in terms of reallocation strategy?
   nearby context before removal:
     ...

Usage

at-agent can be pointed at specific files listed in command line arguments (passing files selects args mode automatically), or via --discover=rg recursively grep the current working directory, or via --discover=inodes (or indeed, no args) walk the current working directory recursively to find inodes flagged as being modified recently and potentially containing @agent directives. For that default mode, the idea is you set your text editor to append to that list of inodes as you edit files and leave @agent directives in them. This is particularly helpful to avoid expensive greps on large trees. There is also --dry-run, which reports directives without removing annotation lines.

The use of inode numbers rather than path names means that this works comfortably in the scenario where you are editing files in your normal environment, but an agent is running in a sandbox and so may have paths mounted at a different location.

You can just add something like this to your .vimrc (the filter is imprecise, but this doesn't matter as at-agent will just do nothing for files that have no valid directives):

let s:at_agent_dir = expand('~/.local/state/at-agent')

function! s:NoteAtAgentInode() abort
  let l:file = expand('%:p')
  if empty(l:file) || !isdirectory(s:at_agent_dir) || search('\s@agent', 'nw') == 0
    return
  endif

  call system(
        \ 'stat -c %i -- ' . shellescape(l:file) .
        \ ' >> ' . shellescape(s:at_agent_dir . '/inodes'))
endfunction

augroup at_agent
  autocmd!
  autocmd BufWritePost * call s:NoteAtAgentInode()
augroup END

So a simple flow would be:

1. Leave notes in the files while editing.
2. Save as normal in vim.
3. Run at-agent from the repository root in the agent session.

There are all kinds of ways this could be integrated into a harness, but I have a fondness for no integration at all, meaning it's easy to switch between different options. e.g. just give a prompt such as "Run at-agent and action its output.".

If using shandbox or similar, the list of inodes that potentially contain directives needs to be exposed at the expected location. You can add something like this to .shandbox_meta/init (or ~/.config/shandbox/default-init):

#!/bin/sh
host_at_agent_dir="$HOME/.local/state/at-agent"
mkdir -p "$host_at_agent_dir"

"$SHANDBOX_SELF" add-mount --read-write \
  "$host_at_agent_dir" \
  ".local/state/at-agent"

• 2026-05-31: Initial publication date.

My colleague Chris Griffith, with whom I collaborated to put The Effects of Nuclear Weapons, Third Edition (1977) online, is also a spaceflight enthusiast (and an urban trails hiker: check out his new book!).  He recently asked me how I would mark up a table with a split diagonal header cell; specifically, this one from the Apollo 16 documentation:

My immediate thought was to throw two spans in the header cell and position or grid them within that cell, but the accessibility of that seemed… questionable.  It’s also what Wikipedia already does, and we here at meyerweb are nothing if not obsessed with finding new ways to do niche stuff.  So I tried something different.  But is its accessibility any better?

If you want to see it as a live example, it’s over at Codepen.  Most of the text in the table is what macOS Preview OCRed out of the original image, which I kept intact because I think it’s funny.  Anyway, here is the original markup I came up with for the table head, which you should not use:

<thead>
	<tr>
		<th scope="row">SCIENTIFIC DISCIPLINE</th>
		<th scope="col">GEOLOGY</th>
		<th scope="col">GEOPHYSICS</th>
		<th scope="col">GEOCHEMISTRY</th>
	</tr>
	<tr>
		<th scope="col">EXPERIMENT</th>
	</tr>
 </thead>

So one row for the headers across the top of the table, including the top-left label that goes with them, and then another row with the header that relates to the row headers for the rows below.  That is to say, the row-scoped table header in each of the rows in the table’s bodies (it has more than one), like this:

<tbody>
	<tr>
		<th scope="row">CONTINGENCY SAMPLE COLLECTION</th>
		[…]
	</tr>

The thing is, when I ran the idea past accessibility experts like Alice Boxhall and Adrian Roselli, they identified a problem: Not having a full row of cells, as is the case for the second header row, fails WCAG 1.3.3.  The suggested fix was to rowspan most of the cells in the first row, like this:

 <thead>
	<tr>
		<th scope="row">SCIENTIFIC DISCIPLINE</th>
		<th scope="col" rowspan="2">GEOLOGY</th>
		<th scope="col" rowspan="2">GEOPHYSICS</th>
		<th scope="col" rowspan="2">GEOCHEMISTRY</th>
	</tr>
	<tr>
		<th scope="col">EXPERIMENT</th>
	</tr>
 </thead>

With that, table navigation wasn’t perfect, but it seemed decent, so we could move forward.

In terms of presentation, to get the upper-left header cell to do the split-diagonal thing, I relatively position the <thead> and then absolutely position the second row in the table head to sit over top of the first, pinned to the bottom left corner.

thead {
	position: relative;
}
thead tr:nth-child(2) th {
	  position: absolute;
	  bottom: 0;
	  left: 0;
}

I fiddled around for a bit with trying to use a grid instead, but it didn’t really add anything that positioning didn’t already provide and threw some other wrenches into the works, like having to convert the entire table into a grid so the columns would stay aligned, so I decided to just stick with the positioning.

Then I throw a linear gradient background into the first row’s first cell to draw the diagonal, and everything’s thus more or less as intended, visually speaking.  (That diagonal could also be an SVG, in fact probably should be in production, but I was seeing how an all-CSS solution might work so a gradient is where things stand.)

There are some layout caveats with this approach, but they’re pretty much the same as other solutions I saw: primarily, the two bits of text that the diagonal visually separates can stick out of their respective halves of the split cell, or even overlap each other.  Also, you might need to explicitly set a minimum height of the first header row, in order to not exacerbate the overlap risk just described.

And then there’s a really big caveat: Safari, as of this writing, doesn’t handle the layout at all well, because it doesn’t apply relative positioning to <thead> (or <tfoot> or <tbody>, but at least it does <tr>s).  I went to file a bug and found there’s already one open, so maybe this will be fixed in the near future.  I figured out a way to get at least close to the intended result while still allowing line-wrapping in the column header cells, but it mangled the layout in Firefox and Chrome.  In the end, to work around the problem, I delved into browser strangeness (at the suggestion of Marius Gundersen) and settled on the following:

/* this is gross and I hate it but it works to fix 
   Safari’s layout of the table’s top headers */
@supports (font: -apple-system-body) {
	thead tr:nth-child(1) th {
		white-space: nowrap;
	}
	thead tr:nth-child(2) th {
		position: static;
		display: block;
		margin-block: -1.5lh 0;
		padding-block: 0;
		text-align: start;
		transform: translateY(0.25lh);
	}
}

Thanks, I hate it!  But it works, and I try to be pragmatic.

Anyway, the point being, what I’ve done here feels more accessible to me, and basic testing by both me and Adrian didn’t reveal any major problems, but I still worry about the positioning dorking things up for the users of screen readers I don’t have access to.  So I throw it to the audience, particularly the accessibility-technology-using part of the audience: does this solution fall down for you, or is it good enough? Please let me know!

Have something to say to all that? You can add a comment to the post, or email Eric directly.

shandbox is a simple Linux sandboxing script that serves my needs well. Perhaps it works for you too? No dependencies between a shell and util-linux (unshare and nsenter).

In short, it aims to provide fairly good isolation for personal files (i.e. your $HOME) while being very convenient for day to day use. It's designed to be run as an unprivileged user - as long as you can make new namespaces you should be good to go. By default /home/youruser/sandbox shows up as /home/sandbox within the sandbox, and other than standard paths like /usr, /etc, /tmp, and so on it's left for you to either copy things into the sandbox or expose them via a mount. There's a single shared sandbox (i.e. processes within the sandbox can see and interact with each other, and the exposed sandbox filesystem is shared as well), which trades off some ease of use for the security you might get with a larger number of more targeted sandboxes. On the other hand, you only gain security from a sandbox if you actually use it and this is a setup that offers very low friction for me. The network is not namespaced (although this is something you could change with a simple edit). If you do want more than one sandbox environment, see the relevant section below.

Usability is both subjective and highly dependent on your actual use case, so the tradeoffs may or may not align with what is interesting for you! Bubblewrap is an example of a mature alternative unprivileged sandboxing tool that offers a lot of configurability as well as options with greater degrees of sandboxing. Beyond that, look to Firecracker based solutions or gvisor. shandbox obviously aims to provide a reasonable sandbox as much as Linux namespaces alone are able to offer, but if you're looking for a security property stronger than "makes it harder for something to edit or access unwanted files" it's down to you to both carefully review its implementation and consider alternatives. The recent spate of disclosed local privilege escalation vulnerabilities is helpful to keep in mind as a reminder of the limits of this namespacing based approach.

Usage example

$ shandbox run uvx pycowsay
initialised sandbox at /home/asb/sandbox
created default ssh config at /home/asb/sandbox/.ssh/config
to add an init hook, create an executable script at: /home/asb/sandbox/.shandbox_meta/init
started (pid 1589289)
Installed 1 package in 5ms

  ------------
< Hello, world >
  ------------
   \   ^__^
    \  (oo)\_______
       (__)\       )\/\
           ||----w |
           ||     ||
$ shandbox status
running (pid 1589364)

log:
  2026-02-11 13:02:51 stopped
  2026-02-11 13:05:06 started (pid 1589289)
$ shandbox add-mount ~/repos/medley
mounted /home/asb/repos/medley -> /home/sandbox/medley
$ shandbox run ls -lh /home/sandbox/medley/README.md
-rw-r--r-- 1 sandbox users 2.7K Feb 11 20:02 /home/sandbox/medley/README.md
$ shandbox run touch /home/sandbox/medley/write-attempt
touch: cannot touch '/home/sandbox/medley/write-attempt': Read-only file system
$ shandbox remove-mount /home/sandbox/medley
unmounted /home/sandbox/medley
$ shandbox add-mount --read-write ~/repos/medley
mounted /home/asb/repos/medley -> /home/sandbox/medley
$ shandbox run touch /home/sandbox/medley/write-attempt
$ shandbox list-mounts
/home/sandbox                /dev/mapper/root[/home/asb/sandbox]
/home/sandbox/medley         /dev/mapper/root[/home/asb/repos/medley]

shandbox enter will open a shell within the sandbox for easy interactive usage. As a convenience, if the current working directory is in $HOME/sandbox (e.g. $HOME/sandbox/foo) then the working directory within the sandbox for shandbox run or shandbox enter will be set to the appropriate path within the sandbox (/home/sandbox/foo in this case). i.e., the case where this mapping is trivial. Environment variables are not passed through.

You can also explicitly control the working directory used by shandbox run or shandbox enter by setting SB_PWD to an absolute in-sandbox path. If SB_PWD isn't set, paths within the sandbox home are translated to /home/sandbox/..., and some host paths that are directly visible in the sandbox (such as /tmp, /usr, /etc, and similar) are used as-is.

Functionality overview

• shandbox new <dir>: Initialise a sandbox directory, setting up the .shandbox_meta layout and a default .ssh/config suitable for use with share-ssh. If ${XDG_CONFIG_HOME:-$HOME/.config}/shandbox/default-init exists it is copied to .shandbox_meta/init.
• shandbox start: Start the sandbox, creating the necessary namespaces and mount layout. Fails if the sandbox is already running. If the selected $SANDBOX_DIR hasn't been initialised yet, it is initialised first. If present, the init script in .shandbox_meta/init is always run.
• shandbox stop: Stop the sandbox by killing the process holding the namespaces. Fails if the sandbox is not running.
• shandbox status: Print whether the sandbox is running and if it is, the pid. Also print the last 20 lines of the log.
• shandbox enter: Open bash within the sandbox, starting the sandbox first if it's not already running.
• shandbox enter-root: Open bash within the outer "root" namespace. This is mostly useful for debugging the namespace or mount layout.
• shandbox run <command> [args...]: Run a command inside the sandbox. The current working directory is translated to an in-sandbox path when this is straightforward, and SB_PWD can be used to override it explicitly. Starts the sandbox first if it isn't already running.
• shandbox add-mount [--read-write] <host-path> [<sandbox-path>]: Bind-mount a host path into the running sandbox. Mounts are read-only by default; pass --read-write to allow writes. The sandbox must already be running. Both directories and individual files are supported, and if no sandbox path is provided the host path basename is mounted under /home/sandbox.
• shandbox remove-mount <sandbox-path>: Remove a previously added bind mount from the running sandbox.
• shandbox list-mounts [--all]: List mounts visible from the sandbox. By default this is restricted to mounts under /home/sandbox; --all shows the full namespace mount table.
• shandbox share-ssh <socket-name> <ssh-target> [ssh args...]: Expose a host-side ssh ControlMaster connection inside the sandbox without copying private keys or ssh-agent state into the sandbox. The sandbox directory must already have been initialised with shandbox new. See below.

Minimum requirements and Ubuntu compatibility

Two core requirement are the ability to create a new user namespace, and a recent enough util-linux release (2.41 or newer should work). The earliest Ubuntu release known to work is 25.10 (25.04 won't work, as its util-linux is too old).

Recent Ubuntu releases restrict unprivileged user namespaces through AppArmor, meaning additional settings are required. Chromium's AppArmor user namespace restrictions notes describe this policy and workarounds.

To change the relevant setting non-persistently:

sudo sysctl -w kernel.apparmor_restrict_unprivileged_userns=0

You can alternatively add an AppArmor profile covering the path you install shandbox to. e.g. put this at /etc/apparmor.d/shandbox and then do sudo service apparmor reload:

abi <abi/4.0>,
include <tunables/global>

profile shandbox /usr/local/bin/shandbox flags=(unconfined) {
  userns,
}

Self-contained sandbox directories

A sandbox is represented by a normal directory, defaulting to $HOME/sandbox. The files visible as /home/sandbox live directly in that directory, and shandbox's own state lives under .shandbox_meta inside it. That means a sandbox is self-contained: you can create another one with shandbox new ~/other-sandbox, select it by setting SANDBOX_DIR (using the absolute path it prints, or a shell-expanded path such as ~/other-sandbox), and it will have its own root layout, runtime directory, pid files, log, init hook, and ssh socket directory.

For example:

$ shandbox new ~/other-sandbox
$ SANDBOX_DIR=~/other-sandbox shandbox run pwd
/home/sandbox
$ shandbox new ~/throwaway-sandbox
$ SANDBOX_DIR=~/throwaway-sandbox shandbox status
stopped

Sandboxes in different SANDBOX_DIR have independent state and home directories. The contents of .shandbox_meta is hidden from inside the sandbox by mounting an empty tmpfs over it. I don't personally use separate sandboxes outside of testing purposes. But it's simple functionality to provide and it's easy to imagine cases where this is useful.

Sharing ssh connections

One aspect of this I'm pretty pleased with is the mechanism for exposing an ssh connection without having to share any key material or password, or set up credentials specifically for the sandbox. shandbox share-ssh will create an ssh ControlMaster and expose the control socket in the sandbox home directory. The sandbox can use this connection for as long as that ssh process lives.

e.g.:

$ shandbox share-ssh buildbox user@example.com
shandbox share-ssh: connecting (user@example.com) using /home/asb/sandbox/.ssh/sockets/ext%buildbox
shandbox share-ssh: connected
shandbox share-ssh: from inside the sandbox, use ssh ext%buildbox

Then from inside the sandbox:

ssh ext%buildbox

The ext%... name format is recognised thanks to a config fragment installed in ~/.ssh/config within the sandbox.

Init hooks

The main way of customising sandbox setup outside of hacking on the shandbox script yourself is through an "init script" which will be called for every shandbox start (implicit or explicit). Just place your script in .shandbox_meta/init, and if you want a default one that is copied into that location for you when creating a new sandbox then put it in $XDG_CONFIG_HOME/.shandbox/default-init.

As the script is executed for each shandbox start, you should either ensure it is idempotent or have it create and check for some marker file so it exits early for subsequent invocations.

The following environment variables are passed through:

• SHANDBOX_SELF: Path to the shandbox script being run.
• SANDBOX_DIR: The host-side sandbox directory.
• SB_HOME: The in-sandbox home path.
• SB_PATH: The path used for sandboxed commands.

A trivial example that adds a default mount:

#!/bin/sh

"$SHANDBOX_SELF" add-mount ~/repos/src src

Implementation approach

The core sandboxing functionality is provided by the Linux namespaces functionality exposed by unshare and nsenter. The script's implementation should be quite readable but I'll try to summarise some key points here.

The goal is that:

• Within the sandbox, you appear as an unprivileged user, with uid and gid equal to your usual Linux user.
• It should be possible to expose additional files or directories to the sandbox once it's running.
• Applications running within the sandbox have no way (modulo bugs or vulnerabilities in the kernel or accessible applications) of reaching files on the host filesystem that aren't explicitly exposed.
  • To underline: This is a goal, it is not a guarantee.
• It's possible to launch multiple processes within the sandbox which can all see each other, and have the same shared sandboxed filesystem.
• This is all doable as an unprivileged user.

To implement that:

• Two sets of namespaces are used to provide this isolation: the outer 'shandbox_root' has the user mapped to root within the namespace and retains access to standard / (allowing us to mount additional paths into after the sandbox has started). The inner 'shandbox_user' represents a new user namepsace mapping our uid/gid to an unprivileged user, but other namespaces are shared with 'shandbox_root'. Sandboxed processes are launched within the namespaces of 'shandbox_user'.
• The process IDs of the initial process within 'sandbox_root' and 'sandbox_user' are saved and recalled so the script can use nsenter to enter the namespace. On newer systems this uses util-linux's getino to store a pid:inode pid reference while on older systems it stores pid plus process start time.
• To help make it easier to tell when you're in the sandbox, a dummy /etc/passwd is bind-mounted naming the current user as sandbox.
• When shandbox start is executed, the necessary directories are bind mounted in a directory that will be used as root (/) for the user sandbox in $SANDBOX_DIR/.shandbox_meta/root. This happens within the sandbox_root namespace, which then uses unshare again to create a new user namespace with an unprivileged user, executing within a chroot.
• A small private /dev is created rather than exposing the host /dev wholesale. Basic devices such as /dev/null, /dev/zero, /dev/random, and /dev/tty are provided, along with a private devpts instance.
• 'sandbox_root' retains access to the host filesystem, which is necessary to allow mounting additional paths after the fact. Without this requirement, we could likely rewrite shandbox start to use pivot_root.
• The host-side .shandbox_meta directory is hidden inside the sandbox by mounting an empty unreadable tmpfs over /home/sandbox/.shandbox_meta.
• When /etc/ssh/ssh_config.d exists, shandbox stages a user-owned copy of that directory and bind-mounts it over the original inside the sandbox. This avoids OpenSSH refusing to process included config snippets that appear as owned by nobody in the inner user namespace.

• 2026-05-30: Added note about Ubuntu compatibility and util-linux requirements.
• 2026-05-26: Update article to reflect a wide range of improvements to the script.
  • share-ssh functionality.
  • init hooks
  • Easy to use support for multiple independent sandboxes (with sandbox state now localised to a single directory).
  • list-mounts
• 2026-02-11: Initial publication date.

This is a cross-post from my personal blog Five years of JavaScript on WebAssembly.

This post summarizes my talk at Wasm I/O 2026: Five Years of JavaScript on WebAssembly. The recording will be uploaded to the 2026 playlist.

By the end, you should have a clear picture of how JavaScript on WebAssembly evolved from an experiment into a production toolchain, the design decisions that shaped it, and where I think it's heading next.

The talk is structured into 5 main sections, each representing a key event during the past 5 years (2021 - 2026) of development of JavaScript on WebAssembly and Javy, which is the leading character of the talk.

April 23rd 2021 — Inception #

Javy's git log shows that on April 23rd 2021 I made the first commit:

Author: Saúl Cabrera 
Date:   Fri Apr 23 09:41:56 2021 -0400

    First commit. Several important things to note:

A fair and usually common question that gets asked is why do you need to run JavaScript on WebAssembly? Isn't WebAssembly meant to run in the browser and alongside JavaScript? It all depends on the use-case. WebAssembly's inherent secure-by-default and near native performance properties are very appealing for use-cases that involve executing untrusted code server side.

Thus, in 2021 I had set myself the objective of creating a toolchain to make it extremely easy and accessible to target WebAssembly. Additionally, the toolchain should also:

1. Produce the smallest possible WebAssembly modules, to minimize the operational infrastructure cost of managing the native binaries produced by compiling Wasm to machine code and to be competitive with other WebAssembly toolchains.

2. Offer reasonably complete JavaScript spec support.

3. Offer decent runtime performance.

QuickJS is a natural fit for all three goals, and has powered Javy ever since.

2022 — Production ready toolchain #

During 2022 Javy became production ready: it was put in the hands of developers and started serving real production use-cases.

The first production ready version included an extremely simple interface for building WebAssembly modules:

javy build index.js -o index.wasm

And it also included a feature to generate extremely small WebAssembly modules, in the range of 1kb - 16kb. During Wasm I/O 2023, Jeff Charles presented the underlying technical details that make this possible.

2023 — Governance model #

2023 brought no major technical milestones, but an equally important shift happened: Javy moved to a hosted-project model under the Bytecode Alliance.

While Javy was initially designed for a specific use-case, it eventually found its way into multiple companies operating in the WebAssembly space. This adoption prompted us to rethink its governance model, shifting towards an open, community-driven model that ensures a healthy environment for the project's growth and development.

2024 — A profiler for JavaScript on WebAssembly #

During 2024, we set out to solve the problem of profiling JavaScript programs on WebAssembly. It is critical for developers to understand how their programs behave inside WebAssembly in order to perform optimizations where applicable. This post intentionally skips over all the technical details behind the research and implementation work; however, if you're interested in reading further, you can check out the first post of my series, A profiler for JavaScript on WebAssembly, Part 1, which accompanies the work happening upstream to materialize this research. The series is not complete, but the first part lays the foundation for why this is important and the challenges behind it.

2025 — Toolchain extensibility #

Javy's design philosophy was always to provide a minimal and lightweight JavaScript on Wasm implementation by default, allowing for extensibility where appropriate. The very initial version of an extensible runtime required users to assemble their own runtime from a set of official Rust crates. This approach was not ergonomic and required a non-trivial knowledge of Javy's internals to get right.

During 2025, we overhauled the extensibility story, making the Javy CLI the canonical starting point of extensibility, promoting the CLI from a simple entry point for the basic use-case into a build tool capable of orchestrating JavaScript-on-WebAssembly builds for third-party use cases.

The official RFC behind this work contains all the technical details.

2026 — A preliminary look at potential throughput improvements #

WebAssembly, as a Harvard Architecture, does not allow just-in-time code generation yet. Thus, any dynamic language running on WebAssembly which requires just-in-time generation as a means to improve throughput is limited to interpreter mode only. This is the case for Javy today: it principally relies on QuickJS' interpreter for code execution. Even though QuickJS' interpreter is very well optimized, there is still room for improvement. Profiles, taken from the work done in 2024 and being upstreamed this year, show that optimizing for:

• Field accesses
• Closure creation
• Function calls

has the potential to considerably improve throughput. More concretely, a recent proof-of-concept of compiling QuickJS bytecode to Wasm, conducted during late 2025, showed that optimizing function calls can improve performance by ~1.2x over the interpreter, and optimizing closure creation, particularly by relying on the Wasm GC proposal, can improve performance by ~1.5x-~3.5x over the interpreter. Note that this work is in its very early days, and the benchmarks were conducted on a very small sample of programs. Our plan is to upstream the compiler as an experimental feature during the upcoming 6-12 months.

Au revoir #

What started in 2021 as — "can we make JavaScript on WebAssembly easy?" — has turned into a question we now get to answer in more interesting ways: not just easy, but small, fast, extensible, and soon, possibly, faster.

Get Clamped: Unwinding Some Difficult CSS History

Sometimes features take a long and winding road on the way to a good solution. We could use your help testing improvements for something that's been somehow developing for about a quarter of a century...

Back in the mid-2000s there was a great debate on where the web was going. The web itself had really exploded and people were really starting to use it as an application delivery platform. Most of the world, including W3C members, seemed to kind of assume that we would get a web for apps instead of documents, and there were several pieces being developed for this. Mozilla had XUL, Microsoft had XAML, Adobe had Flex, even Oracle wound up with this JFX thing. Back then, anyone using these would have definitely noticed that the layout models were different from the web which was then still all abspos and floats. David Baron wrote about this in 2006.

This all brewed for a while until CSS picked up a similar concept. As often happens, they were experimenting with implementations before anyone submitted a public draft. David Baron wrote to the www-style mailing list in June 2008

Much of the specification is implemented in both Gecko and Webkit (with prefixes), and this implementation forms the basis of the formatting model of XUL

This created -webkit-box which was an earlier take on Flexbox.

This is not flexbox's story.

It is instead the strange story of the origins of line clamping, if you can believe it! That's because given this internal ability, Apple added some support for line clamping which used it. Internally, WebKit supported -apple-line-clamp (and later -khtml-line-clamp). This became -webkit-line-clamp and it accidentally escaped the lab and made it into the public releases of Safari.

Given it, one could suddenly have a few lines that would show ellipsis if rendering went beyond that.

.things {
  overflow: hidden;
  display: -webkit-box;
  -webkit-line-clamp: 2;
  -webkit-box-orient: vertical;
}

As far as I can tell, it was never actually publicized by Apple. Despite this, it was useful enough that as people learned about it, they shared (as we do) and eventually so many people used it that it became something of a de facto standard. This is before the WebKit/Blink fork, and so it wound up in Chrome and later in other chromium flavored browsers, all with the -webkit- prefixes. In the Project Spartan/EdgeHTML era Microsoft found that they needed to support it for compatibility and added support for the -webkit- prefix in April 2018. Firefox added support in July 2019. The fact that it was internally flex related at all changed in most browsers a while back, but this weird winding road led through lot of pains.. Chris Coyier wrote a piece about the state of it in 2013. A few years later, in 2016 Nils Rasmusson wrote CSS Line-Clamp — The Good, the Bad and the Straight-up Broken.

Today, according to chromestatus, it's in over 40% of page loads, and on over 30% of pages in the HTTPArchive crawl — and popularity is still growing.

Interoperability and Correction

Despite all of that, lots of it was not consistent or interoperable.

Lots of it didn't even really make sense internally.

It wasn't well designed.

So, there has been a real effort to create a good solution here for the past few years. Igalia (primarily my colleague Andreu Botella) has been working on it, and started with an explainer that is still pretty good and includes images and lots more info. It is now part of CSS Overflow Level 4, and there is an updated implementation in Chromium — you can try it today by enabling experimental web platform features at chrome://flags/#enable-experimental-web-platform-features. If you flip that flag, you're using it!

Guess what it lets you do now....

.thingr {
  line-clamp: 3;
}

It's so simple by default. So good, right?

You can also use the auto value to do this based on some kind of measured value — a max-height, for example:

.thingr {
  max-height: 400px;
  line-clamp: auto;
}

Great, how can I help?

Excellent, I'm glad you asked!

Well, for one you can try out the new unprefixed line-clamp, it's way nicer. You can read about it in the explainer.

Ask questions (you can send them to Andreu on bluesky or mastodon), or report issues if you find them.

Maybe say "thanks" to Bloomberg Tech for funding the work!

But more importantly: This work also involved reconciling as much as we can in order to share code and tests and standard, well-defined behavior for -webkit-line-clamp and friends. We believe that all of these decisions are good, but we need some time for research and feedback given the scale of existing deployments. Remember — this thing is on over 40% of page loads. That means even a small percentage of regressions is a lot of real pages and real users. Check your sites, let us know if you experience issues related to this, and help us make sure a quarter century of history ends with something everyone can be proud of.

Next week, several Igalians will be heading to Nice for a bunch of open source gatherings and hackfests.

• Display Next Hackfest
• Embedded Recipes
• GStreamer Spring Hackfest
• PipeWire Hackfest
• BlueZ F2F
• Media Summit

I’ll attend the Media Summit remotely (I’m interested in the discussion about the usage of Vulkan Video API in embedded devices), then I’ll travel, as other multimedia folks, to the GStreamer Hackfest. While other Igalians will be at the Display Next Hackfest and Embedded Recipes.

My goal for the hackfest is to chat with other GStreamer developers about hardware-accelerated encoders and how to test them, especially those using VA-API and Vulkan.

Here’s what my multimedia colleagues are planning for the hackfest:

Stéphane Cerveau will talk about new features in GstPipelineStudio and its future integration with GstPrinceOfParser, along with several core GStreamer improvements.

Alicia Boya will tackle subtle timing issues, out-of-order and race conditions in frame processing, and other GStreamer bugs.

Xabier Rodríguez Calvar plans to close the issues around static compilation of gstreamer-rs, including selected plugins with system-deps.

Thibault Saunier will aim to finish upstreaming some new features and focus on WebAssembly support.

Recently I’ve found myself repeating the same explanations to different people about how to test a V8 patch in Node.js, how to patch the Node.js fork run in V8’s integration CI, or how to get these patches into their repositories/CIs

Update on what happened in WebKit in the week from May 11 to May 18.

For this week we have quite a collection of news! Ranging a variety of improvements to dialog.requestClose(), rendering fixes, the new Skia-based compositor enabled by default, and proper versioning and improvements to the WebKit Container SDK, there's news for everyone.

Cross-Port 🐱

Graphics 🖼️

Infrastructure 🏗️

Thanks to the Debian 64-bit RISC-V port it's really easy to build a sysroot appropriate for cross-compiling Clang/LLVM and its separate test suite. Either use my rootless-deboostrap-wrapper script or the command I documented in LLVM's cross-compilation instructions, being sure to see the note on working around a Ninja dependency issue. For a bootable QEMU image, Debian-based recipes are similarly straightforward. But we don't have the luxury of a precompiled distribution for 32-bit RISC-V and so we'll lean on Yocto to produce the needed sysroot by building from source. I cover three cases:

1. building a sysroot for cross-compiling projects like LLVM
2. doing the same but in a way that requires fewer build steps
3. building an image approximating my debootstrap image recipes.

In this article I use release 6.0 ('Wrynose') and the bitbake-setup helper tool. For documentation, I found the Yocto quick build guide, and bitbake-setup docs, and image customisation guide helpful.

I'm not a Yocto developer, so if you are reading this and think there are other approaches to consider or alternative ways of solving the problem that are better, please do drop me a note!

Common setup

I'm running on Arch Linux which isn't one of the tested Yocto host distributions, but seemed to work just fine.

I found I needed to enable the en_US locale:

sudo sed /etc/locale.gen -i -e "s/^\#en_US.UTF-8 UTF-8.*/en_US.UTF-8 UTF-8/"
sudo locale-gen

And install the following additional packages:

sudo pacman -S inetutils chrpath cpio diffstat rpcsvc-proto flex bison zstd

Now we will check out bitbake into a work directory and set a directory to be used to hold downloaded files:

mkdir yocto-work && cd yocto-work
git clone https://git.openembedded.org/bitbake
./bitbake/bin/bitbake-setup settings set default dl-dir $HOME/.cache/yocto/dl

Producing a sysroot based on core-image-minimal

As is often the case, the workload I'm interested in here is LLVM. If you're looking to build a sysroot to cross-compile something else, you may need a slightly different package list.

In this first stanza, we use bitbake-setup to initialise our development environment. Because there isn't a predefined machine target for riscv32 in bitbake/default-registry/configurations/poky-wrynose.conf.json, we avoid selecting machine and will address it later. Importantly, we set a SSTATE_DIR which will be used for the shared state cache, avoiding rebuilding packages when not necessary (I'm not totally sure when this isn't exposed in bitbake-setup settings like dl-dir is). Another relevant variable is BB_HASHSERVE_BB_DIR which controls where the hash equivalence database is stored. But with current bitbake-setup this defaults to SSTATE_DIR, so there's no need to set it explicitly.

./bitbake/bin/bitbake-setup init --non-interactive \
  --skip-selection machine \
  ./bitbake/default-registry/configurations/poky-wrynose.conf.json \
  poky \
  distro/poky

printf 'SSTATE_DIR = "%s"\n' "$HOME/.cache/yocto/sstate" >> bitbake-builds/site.conf

With that done, we can source the generated definitions to enter the build environment (note we're using the default setup directory, you can override it to something other than poky-wrynose by using --setup-dir-name) and use enable-fragment to set the qemuriscv32 machine:

. bitbake-builds/poky-wrynose/build/init-build-env
bitbake-config-build enable-fragment machine/qemuriscv32

Now configure the build, indicating the additional libraries that need to be present and run bitbake to actually produce it:

cat >> conf/local.conf <<'EOF'
IMAGE_INSTALL:append = " \
  glibc-dev \
  libgcc \
  libgcc-dev \
  libatomic \
  libatomic-dev \
  libstdc++ \
  libstdc++-dev \
"
EOF

bitbake core-image-minimal

This results in 4624 build tasks and takes quite some time to complete if you haven't run it before (i.e. aren't hitting in the sstate cache). The next section of this article explores how to produce the needed output while building much less, but let's finish the job and extract a rootfs from what was built. I would like to now follow advice in the documentation and run runqemu-extract-sdk on the rootfs archive (I submitted a little patch upstream) to fix this command for .zst which was applied:

runqemu-extract-sdk tmp/deploy/images/qemuriscv32/core-image-minimal-qemuriscv32.rootfs.tar.zst ~/rv32sysroot

At this point, you have a sysroot that's almost directly usable for cross-compiling Clang/LLVM (with --target=riscv32-poky-linux) but there are three finalisation steps we will perform:

• Add an additional symlink to the tree so that upstream Clang's search procedure for the GCC install finds the correct directory. The combination of these two downstream patches which Yocto applies to its own Clang builds would make this unnecessary. I'm not sure if upstreaming has ever been pursued.
• Convert all absolute symlinks to relative ones. Yocto provides a Python script for this, which is in our $PATH after sourcing build/init-build-env.
• (Optional) Apply workaround for a ninja issue that would otherwise mean incremental builds don't work.

mkdir -p "$HOME/rv32sysroot/usr/lib/gcc"
ln -s ../riscv32-poky-linux "$HOME/rv32sysroot/usr/lib/gcc/riscv32-poky-linux"
sysroot-relativelinks.py "$HOME/rv32sysroot"
ln -s usr/include "$HOME/rv32sysroot/include"

Producing a sysroot with fewer build steps

The core-image-minimal recipe above is straightforward, but does a lot more work than strictly necessary. We can reduce this by instead adding a dependency-only recipe that explicitly lists the needed build-time dependencies and contains logic to produce the sysroot.

First, create a layer:

. bitbake-builds/poky-wrynose/build/init-build-env
bitbake-layers create-layer --add-layer ../layers/meta-rv32-llvm-sysroot

Then add the recipe:

recipe_dir="../layers/meta-rv32-llvm-sysroot/recipes-devtools/rv32-llvm-deps-sysroot"
mkdir -p "$recipe_dir"

cat > "$recipe_dir/rv32-llvm-deps-sysroot.bb" <<'EOF'
SUMMARY = "Dependency-only recipe to export an RV32 sysroot"
LICENSE = "MIT-0"

INHIBIT_DEFAULT_DEPS = "1"
EXCLUDE_FROM_WORLD = "1"
PACKAGE_ARCH = "${MACHINE_ARCH}"

DEPENDS = "virtual/libc libgcc virtual/${MLPREFIX}compilerlibs zlib zstd-native"

inherit deploy nopackages

do_configure[noexec] = "1"
do_compile[noexec] = "1"
do_install[noexec] = "1"
do_populate_sysroot[noexec] = "1"

do_deploy() {
    export_dir="${WORKDIR}/${PN}-export"
    rm -rf "$export_dir"
    mkdir -p "$export_dir"
    cp -a "${RECIPE_SYSROOT}/." "$export_dir/"

    sysroot-relativelinks.py "$export_dir"

    mkdir -p "$export_dir/usr/lib/gcc"
    ln -s ../riscv32-poky-linux "$export_dir/usr/lib/gcc/riscv32-poky-linux"
    ln -s usr/include "$export_dir/include"

    tar -C "$export_dir" -cf - . | \
      zstd -T0 -f -o "${DEPLOYDIR}/${PN}-${MACHINE}.tar.zst"
}
addtask deploy after do_prepare_recipe_sysroot before do_build
EOF

The do_deploy function implements the sysroot preparation logic that largely mirrors the previous section. Otherwise, DEPENDS specifies the needed dependencies (of these, virtual/${MLPREFIX}compilerlibs is a bit magic: this resolves to the compiler runtime provider which pulls in things like libstdc++).

Build the sysroot with:

bitbake rv32-llvm-deps-sysroot

This performs ~948 build tasks and will produce the sysroot tarball at tmp/deploy/images/qemuriscv32/rv32-llvm-deps-sysroot-qemuriscv32.tar.zst.

You can use it by doing something like:

SYSROOT="$HOME/rv32depssysroot"
rm -rf "$SYSROOT"
mkdir -p "$SYSROOT"
tar --zstd -C "$SYSROOT" -xf \
  tmp/deploy/images/qemuriscv32/rv32-llvm-deps-sysroot-qemuriscv32.tar.zst

The sysroot is slightly larger than the one in the section above because it contains large unstripped static archives like usr/lib/libstdc++.a.

Producing a featureful image bootable in QEMU

We could probably quibble on the definition of "featureful" as listed in the subheading above. For me, this means an image that boots using systemd and you can ssh into, roughly approximating what you get from my debootstrap recipes. But by adding other packages to the image recipe you can certainly make it more featureful.

First, let's start to set up the build environment and directories we'll use for additional recipes. We use distro/poky-altcfg which is just Poky with systemd as the init manager.

cd yocto-work
./bitbake/bin/bitbake-setup init --non-interactive \
  --setup-dir-name poky-wrynose-systemd \
  --skip-selection machine \
  ./bitbake/default-registry/configurations/poky-wrynose.conf.json \
  poky \
  distro/poky-altcfg

. bitbake-builds/poky-wrynose-systemd/build/init-build-env
bitbake-config-build enable-fragment machine/qemuriscv32
bitbake-layers create-layer --add-layer ../layers/meta-rv32-qemu-image

mkdir -p \
  ../layers/meta-rv32-qemu-image/recipes-core/images \
  ../layers/meta-rv32-qemu-image/recipes-core/rv32-qemu-config/files

Some may prefer to split different aspects of image configuration into independent recipes, but I opt to combine it into one for simplicity (in this case, just configuring systemd-networkd dhcp and adding a config file that will enable sudo for our user account):

cat > ../layers/meta-rv32-qemu-image/recipes-core/rv32-qemu-config/rv32-qemu-config.bb <<'EOF'
SUMMARY = "Configuration for RV32 systemd images"
LICENSE = "MIT-0"
LIC_FILES_CHKSUM = "file://${COMMON_LICENSE_DIR}/MIT-0;md5=f41b3a5f969eb450434cf0e4f33449b9"


SRC_URI = " \
    file://20-wired.network \
    file://90-rv32-qemu \
"

RDEPENDS:${PN} = "systemd-networkd sudo"
FILES:${PN} = " \
    ${sysconfdir}/systemd/network/20-wired.network \
    ${sysconfdir}/sudoers.d/90-rv32-qemu \
"

S = "${UNPACKDIR}"

do_install() {
    install -d ${D}${sysconfdir}/systemd/network
    install -m 0644 ${S}/20-wired.network ${D}${sysconfdir}/systemd/network/20-wired.network

    install -d ${D}${sysconfdir}/sudoers.d
    install -m 0440 ${S}/90-rv32-qemu ${D}${sysconfdir}/sudoers.d/90-rv32-qemu
}
EOF

cat > ../layers/meta-rv32-qemu-image/recipes-core/rv32-qemu-config/files/20-wired.network <<'EOF'
[Match]
Type=ether

[Network]
DHCP=yes
EOF

cat > ../layers/meta-rv32-qemu-image/recipes-core/rv32-qemu-config/files/90-rv32-qemu <<'EOF'
%sudo ALL=(ALL) ALL
EOF

Now, we create the image recipe that will:

• Use the above config recipe as well as pull in other needed packages.
• Add a user account and set passwords of root and user to root and user respectively. This follows the approach in the Yocto docs.
• Make it so runqemu will configure things so we can connect ssh in via a Unix domain socket (as done in the debootstrap-based article). Alternatively you can choose to set QB_SLIRP_OPT = "-netdev user,id=net0,hostfwd=tcp:127.0.0.1:2222-:22" if you'd rather just connect to localhost:2222.

ROOT_PASSWORD_HASH="$(printf "%q" "$(openssl passwd -6 root)")"
USER_PASSWORD_HASH="$(printf "%q" "$(openssl passwd -6 user)")"

cat > ../layers/meta-rv32-qemu-image/recipes-core/images/rv32-qemu-systemd-ssh-image.bb <<EOF
SUMMARY = "Bootable RV32 QEMU image with systemd, networkd, and SSH access"
LICENSE = "MIT-0"

inherit image extrausers

IMAGE_FSTYPES = "ext4"
IMAGE_FEATURES = "allow-root-login"

QB_DEFAULT_FSTYPE = "ext4"
QB_CMDLINE_IP_SLIRP = "ip=none"
QB_SLIRP_OPT = "-netdev user,id=net0,hostfwd=unix:/tmp/yoctorv32.sock-:22"
SERIAL_CONSOLES = "115200;ttyS0"

ROOT_PASSWORD_HASH = "$ROOT_PASSWORD_HASH"
USER_PASSWORD_HASH = "$USER_PASSWORD_HASH"

IMAGE_INSTALL = "packagegroup-core-boot"
IMAGE_INSTALL += "os-release"
IMAGE_INSTALL += "systemd-networkd"
IMAGE_INSTALL += "systemd-serialgetty"
IMAGE_INSTALL += "rv32-qemu-config"
IMAGE_INSTALL += "openssh"
IMAGE_INSTALL += "sudo"
IMAGE_INSTALL += "bash"
IMAGE_INSTALL += "iproute2"
IMAGE_INSTALL += "iputils"
IMAGE_INSTALL += "procps"

EXTRA_USERS_PARAMS = " \\
    groupadd sudo; \\
    usermod -p '\${ROOT_PASSWORD_HASH}' root; \\
    useradd -m -d /home/user -s /bin/bash -G sudo -p '\${USER_PASSWORD_HASH}' user; \\
"
EOF

Now write necessary configuration and build (disabling a number of distro features that would lead to larger build time). The following results in 4305 build tasks on my machine:

printf 'DL_DIR = "%s"\n' "$HOME/.cache/yocto/dl" >> conf/local.conf
printf 'SSTATE_DIR = "%s"\n' "$HOME/.cache/yocto/sstate" >> conf/local.conf

cat >> conf/local.conf <<'EOF'
PACKAGE_CLASSES = "package_ipk"
EXTRA_IMAGE_FEATURES = ""
IMAGE_FEATURES = ""

DISTRO_FEATURES:remove = "x11 wayland opengl alsa bluetooth wifi 3g nfc pcmcia usbgadget usbhost nfs zeroconf pulseaudio gobject-introspection-data"
SERIAL_CONSOLES = "115200;ttyS0"
EOF

bitbake rv32-qemu-systemd-ssh-image

Finally we can boot the image (snapshot means changes to the filesystem image won't persist, just drop this if that isn't what you desire):

DEPLOY="$PWD/tmp/deploy/images/qemuriscv32"
rm /tmp/yoctorv32.sock
runqemu "$DEPLOY/rv32-qemu-systemd-ssh-image-qemuriscv32.rootfs.qemuboot.conf" nographic slirp snapshot

And connect via ssh with something like ssh root@unix/tmp/yoctorv32.sock.

• 2026-05-18: Add second part to article, covering building a bootable image.
• 2026-05-18: Updated to use the Wrynose Yocto release.
• 2026-05-11: Initial publication date.

At BlinkOn 20 in 2025, I introduced our experimental work on bringing Blink to Apple tvOS. You can also find a blog post covering that initial work here: https://blogs.igalia.com/gyuyoung/2026/05/09/introduce-blink-for-apple-tvos/.

If you’re interested in the background and early prototype, you can find more details in my previous post on Blink for iOS and related work: https://blogs.igalia.com/gyuyoung/2024/08/08/chrome-ios-browser-on-blink/.

Over the past year, we have continued developing this effort, and I recently had a chance to share an update along with a demo running on a real Apple TV device at BlinkOn 21 in 2026. In this post, I’d like to walk through what has changed since the initial prototype, what works today, and what challenges still remain.

A quick recap

Apple TV runs tvOS, which is derived from iOS, but it comes with important differences. Most notably, tvOS does not provide a WebKit WebView for third-party applications. This means that any application requiring web functionality needs to embed its own web engine. This constraint was one of the key motivations behind exploring whether Blink, originally being ported to iOS, could also be adapted to tvOS.

While the idea sounds straightforward, the reality is more complicated. tvOS lacks several low-level system APIs required for Chromium’s multi-process architecture, which makes it impossible to use the standard process model. On top of that, BrowserEngineKit, which the iOS Blink effort relies on, is not available on tvOS. There are also platform restrictions such as the lack of JIT support, and the input model is fundamentally different since Apple TV relies on a remote control rather than touch or pointer-based interaction. Because of these constraints, our goal has not been to build a full-featured browser, but rather to enable Blink-based web capabilities in a way that works within the limitations of the platform.

Progress over the last year

Over the past year, we have made steady progress toward that goal. One of the most significant milestones is that we have upstreamed the initial tvOS implementation. This means that the work is no longer just an isolated experiment, but part of the upstream Chromium codebase. As part of this effort, we enabled content_shell running on the tvOS simulator and on actual Apple TV devices. Moving from simulator-only execution to running on real hardware was an important step, as it allowed us to validate real-world behavior and platform integration.

We have also improved platform integration in several areas. Crashpad support has been added, and input handling for the Apple TV Remote has been significantly improved. The latter is particularly important because navigating web content with a remote requires a focus-based interaction model, which is quite different from what Blink typically assumes on desktop or mobile platforms.

On the web platform side, we have enabled a number of features that make it possible to run more realistic content. WebAssembly now works in interpreted mode, which allows execution within the constraints of the platform. We have also enabled VP9 software decoding and verified hardware-accelerated decoding for H.264 and H.265. These improvements are essential for media playback scenarios and were necessary to support the demo content.

To support ongoing development, we also set up reference bots for tvOS builds and tests. This helps ensure that the port can be maintained over time and reduces the risk of regressions as upstream Chromium continues to evolve.

Demo on a real device

This demo shows playing a YouTube video in content_shell running on a real Apple TV device. The demo also showed that we can navigate the video using the remote controller, which highlights the progress we’ve made in adapting Blink to the tvOS interaction model. While simple, this demonstration is an important milestone because it proves that Blink can run real-world web content on actual hardware.

Current limitations

Despite this progress, several challenges remain. One of the most noticeable issues is build stability. The tvOS port has been broken frequently, mainly because both the iOS and tvOS ports are still experimental and not always considered in upstream changes. In particular, configurations such as the WebAssembly interpreter mode are not consistently handled by all changes, leading to breakage.

Testing is another area where limitations are evident. Since the port relies on a single-process model, running web tests is currently not supported, which makes it harder to validate correctness and compatibility. There are also platform-level gaps, such as missing accessibility support and the absence of certain UI components like file choosers and color pickers. As a result, some web pages do not behave as expected on tvOS.

Next steps

Looking ahead, our focus is on improving the robustness and maintainability of the port. This includes stabilizing the build, expanding test coverage, and investigating ways to run web tests in a single-process environment. We also plan to keep the port up to date with the latest tvOS SDK and continue maintaining it in upstream Chromium.

Closing thoughts

Over the past year, Blink for tvOS has evolved from an initial experiment into a working upstream port that can run on real devices. While it is still early and many challenges remain, the progress so far shows that it is possible to bring Blink-based web capabilities to a constrained platform like tvOS. We will continue exploring this space and see how far this effort can go.

Finally, I would like to thank all the contributors, reviewers, and sponsors who made this work possible.

Igalia Contributors

• Abhijeet Kandalkar
• Gyuyoung Kim
• Jeongeun Kim (Julie)
• Raphael Kubo da Costa

Update on what happened in WebKit in the week from May 4 to May 11.

This week we have a bag of exciting updates, such as fixes to crashes, better YouTube playback, a handful of advancements to WebXR, and the development releases of WebKitGTK and WPE WebKit 2.53.2.

Cross-Port 🐱

Networking 📶

Multimedia 🎥

Graphics 🖼️

Releases 📦️

At BlinkOn 20 in 2025, I gave a short lightning talk about an experimental project called Blink for Apple tvOS. Although the presentation took place about a year ago, I wanted to take some time to provide more context on why we started this work, what challenges we encountered along the way, and where the project stands today in this blog again.

Motivation

Apple TV runs tvOS, which is based on iOS, but it differs in some important ways. One of the most notable differences is that tvOS does not provide a WebKit WebView for third-party applications. This limitation has significant implications, as applications that need web functionality must embed their own web engine.

A well-known example is the YouTube app on Apple TV, which uses a custom web engine called Cobalt. This engine is based on an outdated Chromium fork, and maintaining such a fork becomes increasingly difficult over time, especially as the web platform continues to evolve.

At the same time, the Chromium community has been exploring Blink-based implementations on Apple platforms, including the experimental Blink for iOS project. As that work progressed, it naturally led to a new question: whether Blink could also be brought to tvOS. Beyond that, we also started wondering if it would be possible to eventually upstream Blink support for tvOS. These questions became the starting point of this project.

Challenges

Although tvOS is derived from iOS, porting Blink to this platform turned out to be far from straightforward. One of the biggest challenges comes from the lack of support for the multi-process architecture that Chromium relies on. Several low-level system APIs, such as fork(), mach_msg(), and posix_spawn_*(), are not available on tvOS, which makes it impossible to adopt the standard process model.

Another major limitation is the absence of BrowserEngineKit, which the Blink port on iOS uses for process management and integration. Without this framework, alternative approaches are required to make the system work on tvOS.

In addition, tvOS does not allow JIT compilation due to platform restrictions, which directly affects the execution model of V8. This requires running JavaScript in a more restricted mode compared to other platforms.

The input model also differs significantly. Apple TV primarily relies on a remote control, which leads to a focus-based navigation model rather than pointer-based interaction. This affects how web content needs to be handled and navigated.

Given all these constraints, it became clear that the goal of this project should not be to build a fully-featured browser. Instead, we focused on enabling Blink-based web capabilities on tvOS in a way that is both practical and maintainable.

Current progress

To get Blink running on tvOS, we made a number of changes across both the build system and the runtime. On the build side, we introduced tvOS-specific configurations, including a new toolchain, the IS_IOS_TVOS build flag in C++ and Objective-C code, and a target_platform = "tvos" setting in GN.

On the runtime side, the lack of multi-process support required us to enable a single-process mode. We also removed or disabled code paths that depend on BrowserEngineKit and ensured that unsupported low-level system APIs are not used in the tvOS build.

Several modifications were also necessary in core components. For example, JIT was disabled in V8, and build configurations were adjusted for third-party libraries such as ANGLE, V8, and Dawn to make them compatible with tvOS.

At the same time, we worked on integrating platform-specific features. This includes support for hardware-accelerated graphics, media codecs, and Crashpad, as well as improvements to input handling to better support remote-based interaction.

As a result of these efforts, content_shell is now able to run in our internal repository, and work toward upstreaming is currently in progress.

Demo

To demonstrate the current state of the project, we prepared a simple demo showing Blink running on tvOS. In this demo, a YouTube video is played inside content_shell on the tvOS simulator, which illustrates that Blink is capable of rendering and running real-world web content in this environment.

Next steps

There is still a significant amount of work ahead. In the short term, our focus is on making content_shell build and run in upstream Chromium, setting up a reference bot for tvOS builds, and passing relevant unit tests, browser tests, and web platform tests.

In other words, we are currently transitioning from a prototype that “works” to something that is stable, maintainable, and ready for upstream integration.

Closing thoughts

One of the most interesting aspects of this project is how different tvOS is, despite being closely related to iOS. Even relatively small platform restrictions can have large architectural implications when working with a complex system like Blink.

While tvOS is a constrained environment, that is precisely what makes it an interesting engineering challenge. We will continue exploring how far we can take this effort and whether Blink on tvOS can eventually become part of upstream Chromium.

Acknowledgements

This work would not have been possible without the support of many contributors, including Blink and Chromium reviewers, the Google YouTube team, and many collaborators in the community.

Thank you all!

Igalia Contributors

• Abhijeet Kandalkar
• Gyuyoung Kim
• Jeongeun Kim (Julie)
• Raphael Kubo da Costa

Quite some time ago I shared a script and methodology for performing a cross-architecture debootstrap in a rootless way. I had a short note on producing an image bootable in QEMU, but it was fairly minimal. This page provides a cookbook / quick reference on producing such images across various Debian target architectures supported by QEMU. The goal is that the starting point here "gets the basics right" for local experimentation, but of course you are encouraged to evolve the recipe for your needs.

The basic process is to:

1. Build a root filesystem with rootless-debootstrap-wrapper.
2. Configure just enough networking, DNS, serial login, and SSH.
3. Create a 30 GiB ext4 filesystem image directly with mkfs.ext4.
4. Boot it with qemu-system-*, passing the Debian kernel and initrd directly.

We use Debian trixie for amd64, arm64, armhf, ppc64el, riscv64, and s390x. We use sid for ppc64 big endian and loong64. I ran all of this on a current Arch Linux install.

Common setup

sudo pacman -S debootstrap fakeroot qemu-user-static qemu-user-static-binfmt \
  qemu-emulators-full e2fsprogs socat debian-archive-keyring debian-ports-archive-keyring

Put rootless-debootstrap-wrapper somewhere in your PATH, then create a working directory:

mkdir -p qemu-debian-images
cd qemu-debian-images
mkdir -p "$HOME/debcache"

Paste the following into your terminal, which will be called to do the common guest-side configuration. The main thing that's slightly non-standard in this setup are the systemd drop-in overrides which allow authorised SSH keys to be specified by teh systemd credential mechanism. If that's not something you're interested in doing, you can skip the parts touch /etc/systemd/system/ssh* altogether.

configure_qemu_rootfs() {
  rootfs=$1
  console=$2
  suite=$3
  hostname=$4

  "$rootfs/_enter" sh <<EOF
mkdir -p /etc/systemd/network /etc/ssh/sshd_config.d

cat > /etc/systemd/network/10-qemu.network <<'INNER'
[Match]
Type=ether

[Network]
DHCP=yes
INNER

cat > /etc/ssh/sshd_config.d/20-qemu-login.conf <<'INNER'
PermitRootLogin yes
PasswordAuthentication yes
INNER
rm -f /etc/ssh/ssh_host_*_key /etc/ssh/ssh_host_*_key.pub

cat > /etc/systemd/system/ssh.service.d/10-ephemeral-authorized-keys.conf <<'INNER'
[Service]
ImportCredential=ssh.ephemeral-authorized_keys-all
ExecStart=
ExecStart=/usr/sbin/sshd -D \$SSHD_OPTS -o "AuthorizedKeysFile .ssh/authorized_keys" -o "AuthorizedKeysCommand /usr/bin/cat \${CREDENTIALS_DIRECTORY}/ssh.ephemeral-authorized_keys-all" -o "AuthorizedKeysCommandUser root"
INNER

cat > /etc/systemd/system/sshd-vsock@.service.d/10-ephemeral-authorized-keys.conf <<'INNER'
[Service]
ImportCredential=ssh.ephemeral-authorized_keys-all
ExecStart=
ExecStart=-/usr/sbin/sshd -i \$SSHD_OPTS -o "AuthorizedKeysFile .ssh/authorized_keys" -o "AuthorizedKeysCommand /usr/bin/cat \${CREDENTIALS_DIRECTORY}/ssh.ephemeral-authorized_keys-all" -o "AuthorizedKeysCommandUser root"
INNER

/usr/bin/systemd-firstboot --locale=C.UTF-8 --hostname=${hostname} --force
ln -sf ../locale.conf /etc/default/locale
printf '127.0.1.1 %s\n' "$hostname" >> /etc/hosts
printf 'uninitialized\n' > /etc/machine-id
mkdir -p /var/lib/dbus
rm -f /var/lib/dbus/machine-id
ln -sf /etc/machine-id /var/lib/dbus/machine-id

systemctl enable systemd-networkd systemd-resolved systemd-timesyncd ssh
systemctl enable serial-getty@${console}.service

ln -sf ../run/systemd/resolve/resolv.conf /etc/resolv.conf
printf 'root:root\n' | chpasswd
adduser --gecos ",,," --disabled-password user
usermod -aG sudo user
printf 'user:user\n' | chpasswd
EOF

  if [ "$suite" = trixie ]; then
    cat >> "$rootfs/etc/apt/sources.list" <<'EOF'
deb https://security.debian.org/debian-security trixie-security main
deb https://deb.debian.org/debian trixie-updates main
EOF
  fi
}

This should not be exposed on any public network without further configuration. You can ssh in to either the root user or user via ssh, using password root or user respectively. The commands below expose ssh via a unix domain socket. One potential gotcha: this unix domain socket must not have any - in its name as that collides with the splitting done for the hostfwd argument. The examples given below avoid this issue. The boot commands pass net.ifnames=0, so the single QEMU network device is consistently named eth0 and matched by the networkd config above (I found this more reliable than ln -sf /dev/null /etc/udev/rules.d/80-net-setup-link.rules).

For simplicity we make use of mkfs.ext4's ability to populate the image from a directory. Pleasingly, mkfs.xfs gained a similar ability in the xfsprogs 6.17.0 release in Oct 2025. If you have a new enough version, and you prefer an XFS rootfs over ext4 you can tweak the recipes below to do the following for the final image population step:

fakeroot -i "$ROOTFS/.fakeroot.env" \
  mkfs.xfs -f -q -L rootfs \
    -d file,name="$WORK/rootfs.img",size=30g \
    -p "$ROOTFS",atime=0

amd64 / x86-64

Build:

WORK=$PWD/amd64-trixie-qemu
ROOTFS=$WORK/rootfs
mkdir -p "$WORK"

rootless-debootstrap-wrapper \
  --arch=amd64 \
  --suite=trixie \
  --mirror=https://deb.debian.org/debian \
  --cache-dir="$HOME/debcache" \
  --target-dir="$ROOTFS" \
  --include=linux-image-amd64,zstd,dbus,systemd-resolved,systemd-timesyncd,openssh-server,sudo

configure_qemu_rootfs "$ROOTFS" ttyS0 trixie qemu-amd64-trixie
cp "$ROOTFS"/boot/vmlinuz-* "$WORK/kernel"
cp "$ROOTFS"/boot/initrd.img-* "$WORK/initrd"
fakeroot -i "$ROOTFS/.fakeroot.env" \
  mkfs.ext4 -q -L rootfs -d "$ROOTFS" "$WORK/rootfs.img" 30G

Boot:

cd amd64-trixie-qemu
qemu-system-x86_64 \
  -accel kvm \
  -machine q35 \
  -cpu host \
  -smp 2 \
  -m 8G \
  -drive file=rootfs.img,if=none,id=hd,format=raw \
  -device virtio-blk-pci,drive=hd \
  -netdev user,id=net,hostfwd=unix:/tmp/qemu_amd64.sock-:22 \
  -device virtio-net-pci,netdev=net \
  -object rng-random,filename=/dev/urandom,id=rng \
  -device virtio-rng-pci,rng=rng \
  -kernel kernel \
  -initrd initrd \
  -nographic \
  -append "rw root=LABEL=rootfs console=ttyS0 net.ifnames=0"

The above assumes you are running on a x86-64 host, hence enables KVM. If not, then drop -accel kvm and use -cpu max instead of -cpu host.

arm64 / AArch64

Build:

WORK=$PWD/arm64-trixie-qemu
ROOTFS=$WORK/rootfs
mkdir -p "$WORK"

rootless-debootstrap-wrapper \
  --arch=arm64 \
  --suite=trixie \
  --mirror=https://deb.debian.org/debian \
  --cache-dir="$HOME/debcache" \
  --target-dir="$ROOTFS" \
  --include=linux-image-arm64,zstd,dbus,systemd-resolved,systemd-timesyncd,openssh-server,sudo

configure_qemu_rootfs "$ROOTFS" ttyAMA0 trixie qemu-arm64-trixie
cp "$ROOTFS"/boot/vmlinuz-* "$WORK/kernel"
cp "$ROOTFS"/boot/initrd.img-* "$WORK/initrd"
fakeroot -i "$ROOTFS/.fakeroot.env" \
  mkfs.ext4 -q -L rootfs -d "$ROOTFS" "$WORK/rootfs.img" 30G

Boot:

cd arm64-trixie-qemu
qemu-system-aarch64 \
  -machine virt \
  -cpu cortex-a57 \
  -smp 2 \
  -m 8G \
  -drive file=rootfs.img,if=none,id=hd,format=raw \
  -device virtio-blk-device,drive=hd \
  -netdev user,id=net,hostfwd=unix:/tmp/qemu_arm64.sock-:22 \
  -device virtio-net-device,netdev=net \
  -object rng-random,filename=/dev/urandom,id=rng \
  -device virtio-rng-device,rng=rng \
  -kernel kernel \
  -initrd initrd \
  -nographic \
  -append "rw root=LABEL=rootfs console=ttyAMA0 net.ifnames=0"

armhf / 32-bit ARM

For this one I had to add the relevant virtio modules to the initrd.

Build:

WORK=$PWD/armhf-trixie-qemu
ROOTFS=$WORK/rootfs
mkdir -p "$WORK"

rootless-debootstrap-wrapper \
  --arch=armhf \
  --suite=trixie \
  --mirror=https://deb.debian.org/debian \
  --cache-dir="$HOME/debcache" \
  --target-dir="$ROOTFS" \
  --include=linux-image-armmp,zstd,dbus,systemd-resolved,systemd-timesyncd,openssh-server,sudo

configure_qemu_rootfs "$ROOTFS" ttyAMA0 trixie qemu-armhf-trixie
printf '%s\n' virtio_mmio virtio_blk virtio_net >> "$ROOTFS/etc/initramfs-tools/modules"
"$ROOTFS/_enter" update-initramfs -u -k all
cp "$ROOTFS"/boot/vmlinuz-* "$WORK/kernel"
cp "$ROOTFS"/boot/initrd.img-* "$WORK/initrd"
fakeroot -i "$ROOTFS/.fakeroot.env" \
  mkfs.ext4 -q -L rootfs -d "$ROOTFS" "$WORK/rootfs.img" 30G

Boot:

cd armhf-trixie-qemu
qemu-system-arm \
  -machine virt \
  -cpu cortex-a15 \
  -smp 2 \
  -m 4G \
  -drive file=rootfs.img,if=none,id=hd,format=raw \
  -device virtio-blk-device,drive=hd \
  -netdev user,id=net,hostfwd=unix:/tmp/qemu_armhf.sock-:22 \
  -device virtio-net-device,netdev=net \
  -object rng-random,filename=/dev/urandom,id=rng \
  -device virtio-rng-device,rng=rng \
  -kernel kernel \
  -initrd initrd \
  -nographic \
  -append "rw root=LABEL=rootfs console=ttyAMA0 net.ifnames=0"

riscv64

Build:

WORK=$PWD/riscv64-trixie-qemu
ROOTFS=$WORK/rootfs
mkdir -p "$WORK"

rootless-debootstrap-wrapper \
  --arch=riscv64 \
  --suite=trixie \
  --mirror=https://deb.debian.org/debian \
  --cache-dir="$HOME/debcache" \
  --target-dir="$ROOTFS" \
  --include=linux-image-riscv64,zstd,dbus,systemd-resolved,systemd-timesyncd,openssh-server,sudo

configure_qemu_rootfs "$ROOTFS" ttyS0 trixie qemu-riscv64-trixie
cp "$ROOTFS"/boot/vmlinux-* "$WORK/kernel"
cp "$ROOTFS"/boot/initrd.img-* "$WORK/initrd"
fakeroot -i "$ROOTFS/.fakeroot.env" \
  mkfs.ext4 -q -L rootfs -d "$ROOTFS" "$WORK/rootfs.img" 30G

Boot:

cd riscv64-trixie-qemu
qemu-system-riscv64 \
  -machine virt \
  -cpu rv64 \
  -smp 2 \
  -m 8G \
  -drive file=rootfs.img,if=none,id=hd,format=raw \
  -device virtio-blk-device,drive=hd \
  -netdev user,id=net,hostfwd=unix:/tmp/qemu_riscv64.sock-:22 \
  -device virtio-net-device,netdev=net \
  -object rng-random,filename=/dev/urandom,id=rng \
  -device virtio-rng-device,rng=rng \
  -bios /usr/share/qemu/opensbi-riscv64-generic-fw_dynamic.bin \
  -kernel kernel \
  -initrd initrd \
  -nographic \
  -append "rw root=LABEL=rootfs console=ttyS0 net.ifnames=0"

The above assumes you have opensbi installed in /usr/share/qemu (it is put here by the qemu-system-riscv-firmware package on Arch).

ppc64el

Build:

WORK=$PWD/ppc64el-trixie-qemu
ROOTFS=$WORK/rootfs
mkdir -p "$WORK"

rootless-debootstrap-wrapper \
  --arch=ppc64el \
  --suite=trixie \
  --mirror=https://deb.debian.org/debian \
  --cache-dir="$HOME/debcache" \
  --target-dir="$ROOTFS" \
  --include=linux-image-powerpc64le,zstd,dbus,systemd-resolved,systemd-timesyncd,openssh-server,sudo

configure_qemu_rootfs "$ROOTFS" hvc0 trixie qemu-ppc64el-trixie
cp "$ROOTFS"/boot/vmlinux-* "$WORK/kernel"
cp "$ROOTFS"/boot/initrd.img-* "$WORK/initrd"
fakeroot -i "$ROOTFS/.fakeroot.env" \
  mkfs.ext4 -q -L rootfs -d "$ROOTFS" "$WORK/rootfs.img" 30G

Boot:

cd ppc64el-trixie-qemu
qemu-system-ppc64 \
  -machine pseries \
  -cpu power9 \
  -smp 2 \
  -m 8G \
  -drive file=rootfs.img,if=none,id=hd,format=raw \
  -device virtio-blk-pci,drive=hd \
  -netdev user,id=net,hostfwd=unix:/tmp/qemu_ppc64el.sock-:22 \
  -device virtio-net-pci,netdev=net \
  -object rng-random,filename=/dev/urandom,id=rng \
  -device virtio-rng-pci,rng=rng \
  -kernel kernel \
  -initrd initrd \
  -nographic \
  -append "rw root=LABEL=rootfs console=hvc0 net.ifnames=0"

s390x (SystemZ)

Build:

WORK=$PWD/s390x-trixie-qemu
ROOTFS=$WORK/rootfs
mkdir -p "$WORK"

rootless-debootstrap-wrapper \
  --arch=s390x \
  --suite=trixie \
  --mirror=https://deb.debian.org/debian \
  --cache-dir="$HOME/debcache" \
  --target-dir="$ROOTFS" \
  --include=linux-image-s390x,zstd,dbus,systemd-resolved,systemd-timesyncd,openssh-server,sudo

configure_qemu_rootfs "$ROOTFS" ttysclp0 trixie qemu-s390x-trixie
cp "$ROOTFS"/boot/vmlinuz-* "$WORK/kernel"
cp "$ROOTFS"/boot/initrd.img-* "$WORK/initrd"
fakeroot -i "$ROOTFS/.fakeroot.env" \
  mkfs.ext4 -q -L rootfs -d "$ROOTFS" "$WORK/rootfs.img" 30G

Boot:

cd s390x-trixie-qemu
qemu-system-s390x \
  -machine s390-ccw-virtio \
  -smp 2 \
  -m 8G \
  -drive file=rootfs.img,if=none,id=hd,format=raw \
  -device virtio-blk-ccw,drive=hd \
  -netdev user,id=net,hostfwd=unix:/tmp/qemu_s390x.sock-:22 \
  -device virtio-net-ccw,netdev=net \
  -object rng-random,filename=/dev/urandom,id=rng \
  -device virtio-rng-ccw,rng=rng \
  -kernel kernel \
  -initrd initrd \
  -nographic \
  -append "rw root=LABEL=rootfs console=ttysclp0 net.ifnames=0"

ppc64 big-endian

This is a Debian ports target, so we use sid and the ports mirror.

Build:

WORK=$PWD/ppc64-sid-qemu
ROOTFS=$WORK/rootfs
mkdir -p "$WORK"

rootless-debootstrap-wrapper \
  --arch=ppc64 \
  --suite=sid \
  --mirror=https://deb.debian.org/debian-ports \
  --cache-dir="$HOME/debcache" \
  --target-dir="$ROOTFS" \
  --keyring=/usr/share/keyrings/debian-ports-archive-keyring.gpg \
  --include=linux-image-powerpc64,zstd,dbus,systemd-resolved,systemd-timesyncd,openssh-server,sudo

configure_qemu_rootfs "$ROOTFS" hvc0 sid qemu-ppc64-sid
cp "$ROOTFS"/boot/vmlinux-* "$WORK/kernel"
cp "$ROOTFS"/boot/initrd.img-* "$WORK/initrd"
fakeroot -i "$ROOTFS/.fakeroot.env" \
  mkfs.ext4 -q -L rootfs -d "$ROOTFS" "$WORK/rootfs.img" 30G

Boot:

cd ppc64-sid-qemu
qemu-system-ppc64 \
  -machine pseries \
  -cpu power9 \
  -smp 2 \
  -m 8G \
  -drive file=rootfs.img,if=none,id=hd,format=raw \
  -device virtio-blk-pci,drive=hd \
  -netdev user,id=net,hostfwd=unix:/tmp/qemu_ppc64.sock-:22 \
  -device virtio-net-pci,netdev=net \
  -object rng-random,filename=/dev/urandom,id=rng \
  -device virtio-rng-pci,rng=rng \
  -kernel kernel \
  -initrd initrd \
  -nographic \
  -append "rw root=LABEL=rootfs console=hvc0 net.ifnames=0"

loong64 / LoongArch

For this one, we need EDK2 which you can obtain from Debian's qemu-efi-loongarch64 package (QEMU_EFI.fd).

Build:

WORK=$PWD/loong64-sid-qemu
ROOTFS=$WORK/rootfs
mkdir -p "$WORK"

rootless-debootstrap-wrapper \
  --arch=loong64 \
  --suite=sid \
  --mirror=https://deb.debian.org/debian \
  --cache-dir="$HOME/debcache" \
  --target-dir="$ROOTFS" \
  --include=linux-image-loong64,zstd,dbus,systemd-resolved,systemd-timesyncd,openssh-server,sudo

configure_qemu_rootfs "$ROOTFS" ttyS0 sid qemu-loong64-sid
cp "$ROOTFS"/boot/vmlinuz-* "$WORK/kernel"
cp "$ROOTFS"/boot/initrd.img-* "$WORK/initrd"
fakeroot -i "$ROOTFS/.fakeroot.env" \
  mkfs.ext4 -q -L rootfs -d "$ROOTFS" "$WORK/rootfs.img" 30G

Boot:

cd loong64-sid-qemu
cp ../QEMU_EFI.fd .
qemu-system-loongarch64 \
  -machine virt,firmware=QEMU_EFI.fd \
  -smp 2 \
  -m 8G \
  -drive file=rootfs.img,if=none,id=hd,format=raw \
  -device virtio-blk-pci,drive=hd \
  -netdev user,id=net,hostfwd=unix:/tmp/qemu_loong64.sock-:22 \
  -device virtio-net-pci,netdev=net \
  -object rng-random,filename=/dev/urandom,id=rng \
  -device virtio-rng-pci,rng=rng \
  -kernel kernel \
  -initrd initrd \
  -nographic \
  -append "rw root=LABEL=rootfs console=ttyS0 net.ifnames=0"

Logging in

As noted above, you can log in with root/root or user/user. The launch commands above run QEMU with -nographic causing your terminal to be connected to the guest serial console. Ctrl-c alone won't kill the virtual machine, so it's helpful to know:

• Ctrl-a x exits QEMU immediately.
• Ctrl-a c switches between the guest serial console and the QEMU monitor. From the monitor, quit exits QEMU and system_powerdown asks the guest to shut down cleanly.
• Ctrl-a h prints QEMU's help for the other Ctrl-a shortcuts.

Once the guest is booted, you can connect via ssh to the Unix domain socket that forwards to guest port 22. Assuming you're on a recent system with systemd-ssh-proxy (and the ssh config file it adds) present, this can be done with e.g.:

ssh root@unix/tmp/qemu_amd64.sock

Without systemd-ssh-proxy, you can specify ProxyCommand instead:

# For socat:
ssh -o "ProxyCommand=socat - UNIX-CONNECT:/tmp/qemu_amd64.sock" root@vm
# Or for OpenBSD netcat:
ssh -o "ProxyCommand=nc -U /tmp/qemu_amd64.sock" root@vm

If you'd rather use a TCP port, replace the -netdev part of the qemu launch command with something like the following and connect to localhost:2222:

-netdev user,id=net,hostfwd=tcp:127.0.0.1:2222-:22

The systemd-provided config for use of systemd-ssh-proxy disables host identity checks, which is what you typically want with this setup. If using one of the ProxyCommand options above you may want to add -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null to your `ssh invocation.

Alternative: SSH over vsock

It's possible to avoid QEMU user-mode networking and use ssh via AF_VSOCK. This can even work without any additional image changes as systemd-ssh-generator in the guest will generate an appropriate socket-activated sshd service if vsock is present. On the host, you'll need to pick a numeric address for the vsock ('guest CID') that isn't already in use on the system, and change the qemu command line to add the appropriate vsock device with that CID assigned. The vsock device used depends on the machine being emulated - e.g. whether to attach on PCI or the virtio device bus.

For amd64, ppc64el, ppc64, and loong64, add:

-device vhost-vsock-pci,guest-cid=42

For arm64, armhf, and riscv64, add:

-device vhost-vsock-device,guest-cid=42

For s390x, use:

-device vhost-vsock-ccw,guest-cid=42

Assuming your host has systemd-ssh-proxy and its OpenSSH config installed, you can connect with:

ssh root@vsock/42

Using an injected SSH key

Images set up using the recipes above allow a public key to be specified at boot time using the systemd system credential mechanism. Just append the following to the qemu launch command and you can ssh in using that key:

-smbios "type=11,value=io.systemd.credential.binary:ssh.ephemeral-authorized_keys-all=$(base64 -w0 ~/.ssh/id_ed25519.pub)"

• 2026-05-11:
  • Add notes on ssh over AF_VSOCK.
  • Add note about ssh host key checking.
  • Add support for injecting ssh keys using systemd's credential mechanism.
• 2026-05-10:
  • Add note about serial console shortcuts.
  • Use systemd-ssh-proxy.
• 2026-05-09:
  • Tweak shell commands slightly (no cp -f) and use Type=ether for systemd-networkd match.
• 2026-05-07:
  • Add note about how to use an XFS rootfs.
  • Get rid of vestigial errexit usage in common setup script.
• 2026-05-05: Use net.ifnames=0 command line argument rather than ln -sf /dev/null /etc/udev/rules.d/80-net-setup-link.rules.
• 2026-05-04: Initial publication date.

Browsers and Language Features

Web browsers do an astounding amount of stuff with language - and we're always trying to do more. Like most things that we get "for free" and don't give a lot of attention to, there is a lot more to it than you might realize.

Recently, I've been bouncing around looking at several browser language features. Today, our web browsers can listen to us and transcribe our words, they can speak to us, they can do spell checking, and grammar checking. At a surface level there is just a seeming "need to understand words", right? But what exactly that means is actually really variant!

Speech to Text... but more.

Today, your browser can talk to you via the Web Speech API. That's not new. Recently there was a W3C workshop on Voice Interaction. All of the presentations are available on the W3C's playlist. For some reason the sound on some of them seems pretty bad, and unfortunately many very good discusions aren't recorded. One presentation Solving Lead vs. Lead: Consistent Pronunciation for Web Content was very interesting to me. In my 2017 post Greetings, Professor Falken I talked about how the underlying speech system was able to gather a lot from context. For example, even back then, all of the browsers and OSes I could try got these "right" in that they were not naively read but rather read as the correct "forms".

1. Pi is about 3.14
2. We loaded the 4x4
3. Please meet me at 3.14pm EST
4. My birthday is 2/17/1974

Going on 10 years later, today's speech systems would get most "lead" (the heavy element) vs "lead" (being out in front) examples correct because of context. But, in the talk and later discussion, there are plenty of examples where you shouldn't really rely on that. For example, if we're trying to teach someone how something is said. That isn't really just about academics, instances abound.

In that same 2017 post I also mentioned that what the speech subsystems didn't get right was "Greetings, Professor Falken". They didn't pronounce it like the movie. I overcame that in the post by feeding it a misspelling, but this was sort of non-deterministic too, and solved by trial and error. Sarah (the presenter of the W3C talk) lays into a lot of examples like this - we discussed way more examples than I was initially considering where either there was no great context, or where no amount of context is actually likely to help. A lot of these cases are proper nouns or regional pronunciations. Montpelier, VT and Montpelier in the south of France are pronounced very differently. Barre, VT and Martin Barre the lead guitarist for Jethro Tull are pronounced very differently. These are somewhat famous examples. Ana can be pronounced several ways - which one is this? How do you pronounce the names of fictional characters? Or companies and products? And so on.

Sarah is from ETS, who also participated in an earlier "Spoken Presentation Task Force" which produced a proposal for Spoken HTML - and in theory Web Speech should support SSML too. So, that's the proposed solution for that kind of problem. Keep that in the back of your mind for now.

As you can imagine, this is all true in the reverse direction as well. That is, if you are transcribing speech there are sound-alike words and so on - which do I transcribe? “cache” or “cash”? Is it "Barre" or "Barry" or "Berry" Vermont? Do I write "4 by 4" or "4x4" or "Four by Four"? Again, today's models will do very well, generally - but you'll have problems with most of those same things, including especially all of those proper nouns.

At the end of the day all of the listening part is statistics based. The listening machine is this many % confident that it heard X, a little less confident that it heard Y and so on... Then it's sort of a lot like an LLM. So, given context it can do better. Contextual biasing is a simple, common way to improve the result: Just tell it a list of words you might be more likely to use and it'll bias toward them (optionally, provide a weight as to how much more likely it is to be this word, vs something that sounds similar). So, in our example above, if your page is a discussion forum about Vermont politics, it's probably going to hear what is pronounced like "berry" but we want it to write "Barre" and not "Barry" or "bury" or "berry" or anything else, even if it's just a single word without a larger context. Like a host asks "where was that?" and someone replies "Barre".

That functionality was recently added to the Web Speech API in Chrome.

More Different Than You Might Think...

Listening, and speaking both feel so related, but the approach to the two is actually very different. In one it's just a word and optionally a number, and in the other is more complex - you need specialized knowledge about SSML and IPA (International Phonetic Alphabet), some mapping and... in practice more. That's because while SSML seems very deterministic compared to an arbitrary number and statistics, in practice, results vary. In theory, providing the IPA to pronounce "tomato" both of the popular ways- /təˈmɑːtəʊ/ and /təˈmeɪtoʊ/ (think of the old song Let's Call the Whole Thing Off) should make them be pronounced as expected. However, sending this to different speech engines yields unpredictable results. There isn't a way to check the authoritativeness. If the engine doesn't support IPA, it may read contents of the SSML itself, if it exists. If you Give it a name like Saoirse Ronan - even with SSML and a very good engine that supports phonemes and IPA - it often still won't actually pronounce it properly unless it has a good Irish voice... And there aren't actually a lot of them.

Contextual biasing is clearly no help on the pronunciation side of things. But, flip it around and you might think that IPA would be a good input as to how to hear words too - but I've not really found anyone who even tries to do that.

And now for something completely different

As I hinted at the beginning, those are only two examples. We also have things like spell checking - that's the part that marks things as spelled incorrectly - or grammar checking. You can independently style these two cases with the CSS :spelling-error and :grammar-error pseudo-classes independently, as written about by my colleague Stephen Chenney who did the work at Igalia thanks to funding from Bloomberg.

Note that neither of these suggests words in any way. None of them deal with the concepts of autosuggest or autocorrect or autofill or hints or maybe soon even stuff to indicate and generate text rewrites in the future.

None of these are the same thing. Many of them also inevitably have this same general specialized domains problem. For example, if I am editing some Hitchhiker's Guide to the Galaxy wiki and it contains a quote of Vogon poetry:

Oh freddled gruntbuggly, Thy micturations are to me, (with big yawning) As plurdled gabbleblotchits, On a lurgid bee, That mordiously hath blurted out, Its earted jurtles, grumbling Into a rancid festering confectious organ squealer. [drowned out by moaning and screaming]

None of those words are actually spelling errors in this context, and highlighting them as such entirely ruins the experience - there are so many false positives, you miss the real errors.

And almost all of these problems are also somehow still differently handled.

For example, spell check dictionaries are located at potentially several levels. Sometimes you have one in a browser, sometimes the browser is just integrated with the OS level one. Sometimes a browser has 1 per profile. In Android, virtual keyboards have their own dictionaries.

But what does that even mean, "dictionaries"?

Just as is in the cases of speech to text, or text to speech, it can mean something different. A lot of things use a thing called Hunspell which packs up language "dictionaries" that work for all languages and more efficiently can encode complexities like plural rules, autosuggestion help and all sorts of things. For example, here is the LibreOffice en_GB.dic. In this file you'll find simple words like ablaze but most words have some kind of 'affixes' and you'll find similar words in runs that look something like this

spoon-feed/SG
spoon/D6GSM
spoonbill/MS
Spooner/M
spoonerism/SM
spoonful/MS
spoonier
spooniest
spooniness/M	Noun: uncountable
spoonsful
spoony/SMY

These connect to affix definitions in an parallel ".aff" file. This is full of entries like

SFX n e ations [^ckt]e 
SFX n 0 ations [^e]r 
SFX n e ations [iou]te 
SFX n y ations py 
SFX n ke cation ke 
SFX n ke cation's ke 
SFX n ke cations ke 
SFX n y ication [^p]y 
SFX n y ication's [^p]y 

In other words... It's complicated, but covers a lot.

If you ever right clicked something and selected "learn word" or "add to dictionary" or something, you're doing the equivalent of adding ablaze in the ".dic" file - just a simple string. But, unlike Hunspell or other complex formats, it's simple enough that even an end user can do it.

Bloomberg is also sponsoring our work on a proposal called the SpellCheckCustomDictionary API. We've been working on an explainer. Realistically maybe we should call it SpellCheckExclusions to be clearer that all it really does is allow a site to provide a list of words to not match as :spelling-error. We also let you do that in groups so that any active spellchecking can just re-run once.

There was a lot of debate and thought about how much of this should be shared or centralized, but starting with a simple list that can be well defined in terms of standard exclusion seems like a nice first step. It does mean that you would need to potentially add both "Gandalf" and "Gandalf's" as valid, and "hobbit" and "hobbits" and "hobbit's" and "hobbits'". But at least this is simple and understandable, works in every language and widens the pool of developers who might do it. A nice trade-off here might be to allow some sense of regexp in this list - though, for practical purposes it should probably be a limited subset (() for grouping, | for alternation, ? for optional, plus maybe an :i sigil for case insensitivity). This would potentially help make it easier for some authors to express more with less and help shrink the memory initially required, while still not getting too specialized and fairly easy to make performant enough.

It would also be nice to follow this with a purely declarative solution.

I'm pleased that this went to Stage 1 in WHATWG this week and looking forward to figuring out how we move this forward.

Anyway, it's been really interesting and fun to dig into all of this and it's always eye-opening to look behind another curtain... I'm looking forward to continuing these conversations and ultimately getting something useful into all of the browsers! Thanks again to Bloomberg Tech for the sponsorship!