Last month I woke up to a fresh regression on llvm-test-suite where the automotive-bitcount benchmark was 46% slower on RISC-V.

What had happened was that SLP began vectorizing the following sequence in a hot loop:

andi a3, a1, 0xf
slli a4, a1, 0x38
slli a5, a1, 0x34
srli a4, a4, 0x3c
add a3, a3, s4
add a4, a4, s4
srli a5, a5, 0x3c
slli a6, a1, 0x30
add a5, a5, s4
srli a6, a6, 0x3c
add a6, a6, s4
slli a7, a1, 0x2c
srli a7, a7, 0x3c
lbu a3, 0x0(a3)
lbu a4, 0x0(a4)
lbu a5, 0x0(a5)
lbu a6, 0x0(a6)
add a7, a7, s4
slli t0, a1, 0x28
slli t1, a1, 0x24
srli t0, t0, 0x3c
srli t1, t1, 0x3c
add t0, t0, s4
add t1, t1, s4
lbu a7, 0x0(a7)
lbu t0, 0x0(t0)
lbu t1, 0x0(t1)
srliw t2, a1, 0x1c
add t2, t2, s4
add a3, a3, a4
lbu a4, 0x0(t2)
add a5, a5, a6
add a7, a7, t0
add a3, a3, a5
add a7, a7, t1
add a3, a3, a7
add a2, a2, a4
addi a1, a1, 0xd
add a2, a2, a3
bne a1, a0, 0xaf8 <main+0x2b0>

To this indexed gather vluxei64.v of 8 elements on RISC-V:

vsetvli zero, 0x8, e64, m4, ta, ma
vmv.v.x v16, a2
vsrl.vv v16, v16, v12
vand.vi v16, v16, 0xf
vsetvli zero, zero, e8, mf2, ta, ma
vluxei64.v v20, (s4), v16
vsetvli zero, zero, e32, m2, ta, ma
vzext.vf4 v16, v20
vwaddu.wv v8, v8, v16
addi a1, a1, -0x1
addi a2, a2, 0xd
bnez a1, 0xb04 <main+0x2bc>

On the surface the new vectorized code looks pretty reasonable, but in reality the vectorized code takes 51.9M cycles whilst the original scalar only takes 12.3M. vluxei64.v and other indexed memory operations are pretty slow on most available RISC-V hardware today, typically only processing elements one at a time, and to make matters worse we have to do a bunch of 64-bit pointer arithmetic at a wide LMUL too.

So there was somewhere where we weren’t taking this into account in the cost model, either in the RISC-V backend or in the SLP vectorizer. Just as I was about to crank up the cost for vluxei64.v I took a step back and looked at the original C source for that function.

static char bits[256] = {0, 1, 1, 2, 1, 2, 2, 3, 1, 2, 2, 3, 2, 3, 3, 4, ...};

static int ntbl_bitcount(long x) {
  return bits[(int)(x & 0x0000000FUL)] +
         bits[(int)((x & 0x000000F0UL) >> 4)] +
         bits[(int)((x & 0x00000F00UL) >> 8)] +
         bits[(int)((x & 0x0000F000UL) >> 12)] +
         bits[(int)((x & 0x000F0000UL) >> 16)] +
         bits[(int)((x & 0x00F00000UL) >> 20)] +
         bits[(int)((x & 0x0F000000UL) >> 24)] +
         bits[(int)((x & 0xF0000000UL) >> 28)];
}

As it turns out, this function is just popcount: it computes the number of bits set in a 32-bit integer x. And we have an instruction for this on RISC-V, cpop! So why aren’t we lowering to it?

AggressiveInstCombine can recognise some forms of popcount that directly manipulate the bits of the input, but the twist in arithmetic-bitcount is that it uses a table lookup on precomputed values instead.

x comes in as a long but is cast to a 32-bit int. The function then breaks up the 32-bit int into 4-bit nibbles, shifting and masking it with 0xF, and looks up the popcount of each 4-bit nibble. Then it returns the sum of the 4-bit popcounts.

Doing it per-nibble is a nice approach as it keeps the accessed part of the lookup table small at just 16 bytes, so it should all fit inside a single cache line. But even if all 8 loads are L1 cache hits, it’s not as fast as a native cpop on RISC-V. So how can we match this pattern? Here’s the IR before we reach AggressiveInstCombine:

@bits = internal unnamed_addr constant [256 x i8] c"\00\01\01\02\01\02\02\03\01\02\02\03\02\03\03\04\01\02\02\03\02\03\03\04\02\03\03\04\03\04\04\05\01\02\02\03\02\03\03\04\02\03\03\04\03\04\04\05\02\03\03\04\03\04\04\05\03\04\04\05\04\05\05\06\01\02\02\03\02\03\03\04\02\03\03\04\03\04\04\05\02\03\03\04\03\04\04\05\03\04\04\05\04\05\05\06\02\03\03\04\03\04\04\05\03\04\04\05\04\05\05\06\03\04\04\05\04\05\05\06\04\05\05\06\05\06\06\07\01\02\02\03\02\03\03\04\02\03\03\04\03\04\04\05\02\03\03\04\03\04\04\05\03\04\04\05\04\05\05\06\02\03\03\04\03\04\04\05\03\04\04\05\04\05\05\06\03\04\04\05\04\05\05\06\04\05\05\06\05\06\06\07\02\03\03\04\03\04\04\05\03\04\04\05\04\05\05\06\03\04\04\05\04\05\05\06\04\05\05\06\05\06\06\07\03\04\04\05\04\05\05\06\04\05\05\06\05\06\06\07\04\05\05\06\05\06\06\07\05\06\06\07\06\07\07\08", align 1

define dso_local signext range(i32 0, 2041) i32 @ntbl_bitcount(i64 noundef %x) local_unnamed_addr #0 {
entry:
  %conv = and i64 %x, 15
  %arrayidx = getelementptr inbounds nuw i8, ptr @bits, i64 %conv
  %0 = load i8, ptr %arrayidx, align 1, !tbaa !4
  %conv1 = zext i8 %0 to i32
  %1 = lshr i64 %x, 4
  %conv3 = and i64 %1, 15
  %arrayidx5 = getelementptr inbounds nuw i8, ptr @bits, i64 %conv3
  %2 = load i8, ptr %arrayidx5, align 1, !tbaa !4
  %conv6 = zext i8 %2 to i32
  %add = add nuw nsw i32 %conv1, %conv6
  %3 = lshr i64 %x, 8
  %conv9 = and i64 %3, 15
  %arrayidx11 = getelementptr inbounds nuw i8, ptr @bits, i64 %conv9
  %4 = load i8, ptr %arrayidx11, align 1, !tbaa !4
  %conv12 = zext i8 %4 to i32
  %add13 = add nuw nsw i32 %add, %conv12
  %5 = lshr i64 %x, 12
  %conv16 = and i64 %5, 15
  %arrayidx18 = getelementptr inbounds nuw i8, ptr @bits, i64 %conv16
  %6 = load i8, ptr %arrayidx18, align 1, !tbaa !4
  %conv19 = zext i8 %6 to i32
  %add20 = add nuw nsw i32 %add13, %conv19
  %7 = lshr i64 %x, 16
  %conv23 = and i64 %7, 15
  %arrayidx25 = getelementptr inbounds nuw i8, ptr @bits, i64 %conv23
  %8 = load i8, ptr %arrayidx25, align 1, !tbaa !4
  %conv26 = zext i8 %8 to i32
  %add27 = add nuw nsw i32 %add20, %conv26
  %9 = lshr i64 %x, 20
  %conv30 = and i64 %9, 15
  %arrayidx32 = getelementptr inbounds nuw i8, ptr @bits, i64 %conv30
  %10 = load i8, ptr %arrayidx32, align 1, !tbaa !4
  %conv33 = zext i8 %10 to i32
  %add34 = add nuw nsw i32 %add27, %conv33
  %11 = lshr i64 %x, 24
  %conv37 = and i64 %11, 15
  %arrayidx39 = getelementptr inbounds nuw i8, ptr @bits, i64 %conv37
  %12 = load i8, ptr %arrayidx39, align 1, !tbaa !4
  %conv40 = zext i8 %12 to i32
  %add41 = add nuw nsw i32 %add34, %conv40
  %13 = lshr i64 %x, 28
  %idxprom45 = and i64 %13, 15
  %arrayidx46 = getelementptr inbounds nuw i8, ptr @bits, i64 %idxprom45
  %14 = load i8, ptr %arrayidx46, align 1, !tbaa !4
  %conv47 = zext i8 %14 to i32
  %add48 = add nuw nsw i32 %add41, %conv47
  ret i32 %add48
}

You could write one big optimisation to recognise the whole chain, but it’s often better practice to do things as smaller, composable chunks. Lets start with a smaller chunk of the IR, focusing on a single table lookup:

  %gep = getelementptr [256 x i8], ptr @bits, i64 %x
  %ctpop = load i8, ptr %gep

What’s important to note is that any read outside of @bits is undefined behaviour, so we’re free to assume that %x lies somewhere within it. So given a load from some global variable @bits and an index into it %x, if every element in the global variable maps to bits[i] = popcount(i), then we can replace bits[%x] with popcount(%x). Here’s the implementation in AggressiveInstCombine, landed in #231225:

static bool tryToRecognizeTableBasedCtpop(LoadInst *LI, Type *AccessType,
                                          GlobalVariable *GVTable,
                                          Value *GepIdx, const APInt &GEPScale,
                                          const DataLayout &DL,
                                          const TargetTransformInfo &TTI) {
  Type *IdxTy = GepIdx->getType();
  unsigned IdxBits = IdxTy->getIntegerBitWidth();
  if (!LI->isSimple() || !isPowerOf2_32(IdxBits) ||
      TTI.getPopcntSupport(IdxBits) != TargetTransformInfo::PSK_FastHardware)
    return false;

  // GEPScale = the i8 in getelementptr [256 x i8]
  // AccessSize = the i8 in load i8
  uint64_t TableSize = DL.getTypeAllocSize(GVTable->getValueType());
  unsigned AccessSize = DL.getTypeStoreSize(AccessType);
  if (GEPScale != AccessSize)
    return false;
  uint64_t NumEntries = TableSize / AccessSize;
  
  // Is this a popcount table?
  for (uint64_t I = 0; I < NumEntries; I++) {
    auto *C = dyn_cast_or_null<ConstantInt>(ConstantFoldLoadFromConst(
        GVTable->getInitializer(), AccessType, I * GEPScale, DL));
    if (!C || C->getValue() != popcount(I))
      return false;
  }

  // Zero extend or truncate as needed
  replaceWithPopCount(*LI, GepIdx);
  return true;
}

With a quick profitability query and a check to make sure we’re loading from a popcount table, we collapse a lookup into a call to the dedicated popcount intrinsic @llvm.ctpop (plus a truncate to get it back to the right size):

  %ctpop = call i64 @llvm.ctpop(i64 %x)
  %ctpop.trunc = trunc i64 %ctpop to i8

When applied to the IR from the benchmark we’re left with a chain of ctpops all operating on 4-bit chunks at a time, summed together:

define dso_local signext range(i32 0, 2041) i32 @ntbl_bitcount(i64 noundef %x) local_unnamed_addr #0 {
entry:
  %conv = and i64 %x, 15
  %0 = tail call range(i64 0, 5) i64 @llvm.ctpop.i64(i64 %conv)
  %conv1 = trunc nuw nsw i64 %0 to i32
  %1 = lshr i64 %x, 4
  %conv3 = and i64 %1, 15
  %2 = tail call range(i64 0, 5) i64 @llvm.ctpop.i64(i64 %conv3)
  %conv6 = trunc nuw nsw i64 %2 to i32
  %add = add nuw nsw i32 %conv6, %conv1
  %3 = lshr i64 %x, 8
  %conv9 = and i64 %3, 15
  %4 = tail call range(i64 0, 5) i64 @llvm.ctpop.i64(i64 %conv9)
  %conv12 = trunc nuw nsw i64 %4 to i32
  %add13 = add nuw nsw i32 %add, %conv12
  %5 = lshr i64 %x, 12
  %conv16 = and i64 %5, 15
  %6 = tail call range(i64 0, 5) i64 @llvm.ctpop.i64(i64 %conv16)
  %conv19 = trunc nuw nsw i64 %6 to i32
  %add20 = add nuw nsw i32 %add13, %conv19
  %7 = lshr i64 %x, 16
  %conv23 = and i64 %7, 15
  %8 = tail call range(i64 0, 5) i64 @llvm.ctpop.i64(i64 %conv23)
  %conv26 = trunc nuw nsw i64 %8 to i32
  %add27 = add nuw nsw i32 %add20, %conv26
  %9 = lshr i64 %x, 20
  %conv30 = and i64 %9, 15
  %10 = tail call range(i64 0, 5) i64 @llvm.ctpop.i64(i64 %conv30)
  %conv33 = trunc nuw nsw i64 %10 to i32
  %add34 = add nuw nsw i32 %add27, %conv33
  %11 = lshr i64 %x, 24
  %conv37 = and i64 %11, 15
  %12 = tail call range(i64 0, 5) i64 @llvm.ctpop.i64(i64 %conv37)
  %conv40 = trunc nuw nsw i64 %12 to i32
  %add41 = add nuw nsw i32 %add34, %conv40
  %13 = lshr i64 %x, 28
  %idxprom45 = and i64 %13, 15
  %14 = tail call range(i64 0, 5) i64 @llvm.ctpop.i64(i64 %idxprom45)
  %conv47 = trunc nuw nsw i64 %14 to i32
  %add48 = add nuw nsw i32 %add41, %conv47
  ret i32 %add48
}

We’re still using 8 ctpops to compute 1 ctpop though, so the next step is to merge these together.

If we have ctpop(X) + ctpop(Y) we can fold that into ctpop(X | Y) if X and Y don’t share any set bits. For example, if X = 0b11110000 and Y = 0b00001111, because they’re disjoint we have ctpop(X) + ctpop(Y) = ctpop(0b11111111) = 8. This combine already exists in InstCombine:

 // ctpop(A) + ctpop(B) => ctpop(A | B) if A and B have no bits set in common.
 if (match(LHS, m_OneUse(m_Ctpop(m_Value(A)))) &&
     match(RHS, m_OneUse(m_Ctpop(m_Value(B)))) &&
     haveNoCommonBitsSet(A, B, SQ.getWithInstruction(&I)))
   return replaceInstUsesWith(
       I, Builder.CreateIntrinsic(Intrinsic::ctpop, {I.getType()},
                                  {Builder.CreateDisjointOr(A, B)}));

But our motivating IR is slightly different. We have ctpop(X & 0xf) + ctpop((X >> 4) & 0xf) + ctpop((X >> 8) & 0xf) + .... Each 4-bit nibble overlaps in the same least significant bits which prevents the fold above, but:

• the individual bits that we extract from X are distinct.
• it doesn’t matter where the bits are placed in ctpop. ctpop(X) and ctpop(X << 4) give the same result as long as no bits of X are shifted out.
• we know that we can fold ctpop(X) + ctpop(Y) -> ctpop(X | Y) when the bits are distinct.

So what we can do is shift back up the bits of X up so that they’re now in distinct bit positions, which allows us to do the earlier fold that combines the add:

ctpop(X & 0x0f) + ctpop((X >> 4) & 0xf)
=>
; shift the bits back up so they don't overlap
ctpop(X & 0x0f) + ctpop(((X >> 4) & 0xf) << 4)
=>
ctpop(X & 0x0f) + ctpop(X & 0xf0)
=>
ctpop(X & (0x0f | 0xf0))
=>
ctpop(X & 0xff)

We can generalise this to arbitrary shifts and masks, and then we get this code in InstCombine (#231321):

/// Given (X >> C) & M, returns {X, M << C} (the source and a mask of what bits
/// of the source are in \p V).
static std::pair<Value *, APInt> matchMaskedBits(Value *V) {
  unsigned BitWidth = V->getType()->getScalarSizeInBits();
  APInt Mask = APInt::getAllOnes(BitWidth);
  Value *Src;
  const APInt *M;
  uint64_t ShAmt;
  if (match(V, m_And(m_Value(Src), m_APInt(M)))) {
    V = Src;
    Mask = *M;
  }
  if (match(V, m_LShrOrSelf(m_Value(Src), ShAmt)) && ShAmt < BitWidth) {
    V = Src;
    Mask <<= ShAmt;
  }
  return {V, Mask};
}

/// ctpop((X >> C1) & M1) + ctpop((X >> C2) & M2)
/// =>
/// ctpop(X & ((M1 << C1) | (M2 << C2))) if M1 << C1 and M2 << C2 are disjoint
static Value *foldAddOfMaskedCtpops(BinaryOperator &I,
                                    InstCombiner::BuilderTy &Builder) {
  Value *A, *B;
  if (!match(I.getOperand(0), m_OneUse(m_Ctpop(m_Value(A)))) ||
      !match(I.getOperand(1), m_OneUse(m_Ctpop(m_Value(B)))))
    return nullptr;

  auto [XA, MaskA] = matchMaskedBits(A);
  auto [XB, MaskB] = matchMaskedBits(B);
  if (XA != XB || MaskA.intersects(MaskB))
    return nullptr;

  Value *Masked = Builder.CreateAnd(XA, MaskA | MaskB);
  return Builder.CreateUnaryIntrinsic(Intrinsic::ctpop, Masked);
}

If we try this out: Nothing changes! We don’t actually match the ctpop(X & 0x0f) + ctpop((X >> 4) & 0xf) pattern here because there’s truncates in between.

  %conv = and i64 %x, 15
  %0 = tail call range(i64 0, 5) i64 @llvm.ctpop.i64(i64 %conv)
  %conv1 = trunc nuw nsw i64 %0 to i32
  %1 = lshr i64 %x, 4
  %conv3 = and i64 %1, 15
  %2 = tail call range(i64 0, 5) i64 @llvm.ctpop.i64(i64 %conv3)
  %conv6 = trunc nuw nsw i64 %2 to i32
  %add = add nuw nsw i32 %conv6, %conv1
  ...

We can strip off the individual truncates by pulling them outside of the adds in InstCombine (#231332):

  // trunc(ctpop(A)) + trunc(ctpop(B)) => trunc(ctpop(A) + ctpop(B))
  if (match(LHS, m_OneUse(m_Trunc(m_Value(A, m_Ctpop(m_Value()))))) &&
      match(RHS, m_OneUse(m_Trunc(m_Value(B, m_Ctpop(m_Value()))))) &&
      A->getType() == B->getType())
    return new TruncInst(Builder.CreateAdd(A, B), I.getType());

Which now gives us the right shape for our ctpop pattern:

  %conv = and i64 %x, 15
  %0 = tail call range(i64 0, 5) i64 @llvm.ctpop.i64(i64 %conv)
  %1 = lshr i64 %x, 4
  %conv3 = and i64 %1, 15
  %2 = tail call range(i64 0, 5) i64 @llvm.ctpop.i64(i64 %conv3)
  %3 = add nuw nsw i64 %2, %0
  %add = trunc nuw nsw i64 %3 to i32
  ...

And finally the entire function folds away to one call:

define dso_local signext range(i32 0, 33) i32 @ntbl_bitcount(i64 noundef %x) local_unnamed_addr #0 {
entry:
  %0 = and i64 %x, 4294967295
  %1 = call range(i64 0, 33) i64 @llvm.ctpop.i64(i64 %0)
  %add48 = trunc nuw nsw i64 %1 to i32
  ret i32 %add48
}

Which gives us a single RISC-V instruction:

ntbl_bitcount:                          # @ntbl_bitcount
	.cfi_startproc
# %bb.0:                                # %entry
	cpopw	a0, a0
	ret

As an added bonus, ntbl_bitcount is inlined within the loop its called from. So if we’re compiling with Zvbb (which rva23u64 contains), the loop vectorizer is able to kick in and we get a vectorized reduction using vcpop.v:

# %bb.1:                                # %vector.ph
	vsetvli	a2, zero, e64, m2, ta, ma
	vmv.v.x	v10, a0
	vid.v	v8
	li	a0, 13
	vmadd.vx	v8, a0, v10
	vmv.v.i	v10, 0
	li	a0, -1
	srli	a0, a0, 32
.LBB1_2:                                # %vector.body
                                        # =>This Inner Loop Header: Depth=1
	vsetvli	a2, a1, e64, m2, ta, ma
	vand.vx	v12, v8, a0
	vcpop.v	v12, v12
	sh1add	a3, a2, a2
	vsetvli	zero, zero, e64, m2, tu, ma
	vadd.vv	v10, v12, v10
	sh2add	a3, a3, a2
	vsetvli	zero, zero, e64, m2, ta, ma
	vadd.vx	v8, v8, a3
	sub	a1, a1, a2
	bnez	a1, .LBB1_2
# %bb.3:                                # %for.cond.cleanup.loopexit
	vsetvli	a0, zero, e64, m2, ta, ma
	vmv.s.x	v8, zero
	vredsum.vs	v8, v10, v8
	vmv.x.s	a0, v8
	ret

The final vcpop.v reduction is a mighty 115x times faster than the original SLP vectorized version as measured on a SpacemiT-K3. Not bad!

Now with all this said, automotive-bitcount as a benchmark is very artificial. It just consists of a bunch of different implementations of popcount, each executed in a loop. However out of the 8 implementations, with these patches LLVM is able to lower 6 of them to llvm.ctpop intrinsics. In fact LLVM’s AggressiveInstCombine can already recognise various implementations of popcount, and there’s a whole zoo of different formats that tryToRecognizePopCount/tryToRecognizePopCount1/tryToRecognizePopCount2n3 detect:

int popcount(unsigned int i) {
  i = i - ((i >> 1) & 0x55555555);
  i = (i & 0x33333333) + ((i >> 2) & 0x33333333);
  i = ((i + (i >> 4)) & 0x0F0F0F0F);
  return (i * 0x01010101) >> 24;
}

int popcnt(unsigned x) {
  x = x - ((x >> 1) & 0x55555555);
  x = (x & 0x33333333) + ((x >> 2) & 0x33333333);
  x = (x + (x >> 4)) & 0x0F0F0F0F;
  x = x + (x >> 8);
  x = x + (x >> 16);
  return x & 0x0000003F;
}

int popcnt(unsigned x) {
  x = x - ((x >> 1) & 0x55555555);
  x = x - 3*((x >> 2) & 0x33333333);
  x = (x + (x >> 4)) & 0x0F0F0F0F;
  x = x + (x >> 8);
  x = x + (x >> 16);
  return x & 0x0000003F;
}

// Ref. Hackers Delight
int popcount32(unsigned int i) {
  uWord = (uWord & 0x55555555) + ((uWord>>1) & 0x55555555);
  uWord = (uWord & 0x33333333) + ((uWord>>2) & 0x33333333);
  uWord = (uWord & 0x0F0F0F0F) + ((uWord>>4) & 0x0F0F0F0F);
  uWord = (uWord & 0x00FF00FF) + ((uWord>>8) & 0x00FF00FF);
  return  (uWord & 0x0000FFFF) + (uWord>>16);
}

int popcount64(unsigned long i) {
  uWord = (uWord & 0x5555555555555555) + ((uWord>>1) & 0x5555555555555555);
  uWord = (uWord & 0x3333333333333333) + ((uWord>>2) & 0x3333333333333333);
  uWord = (uWord & 0x0F0F0F0F0F0F0F0F) + ((uWord>>4) & 0x0F0F0F0F0F0F0F0F);
  uWord = (uWord & 0x00FF00FF00FF00FF) + ((uWord>>8) & 0x00FF00FF00FF00FF);
  uWord = (uWord & 0x0000FFFF0000FFFF) + ((uWord>>16) & 0x0000FFFF0000FFFF);
  return  (uWord & 0x00000000FFFFFFFF) + (uWord>>32) & 0x00000000FFFFFFFF;
}

Try as you might to be clever with your magic constants and fancy bit shifts, LLVM sees through it all and steamrolls over it it with relish.

I invite anyone to try and tackle the remaining two, bitcount and bit_shifter, since even though this series of patches stemmed from an artificial benchmark the table lookup popcount pattern shows up in a few places in the real world, notably Redis and GCC. So sometimes chasing down these benchmarks can be for both fun and profit!

The proposal period for Interop 2027 closed in late September, which means the Interop team is now in the process of working through all the proposals and ultimately deciding which will be taken on next year.  Part of that process is looking at “developer signals”, which is a loose amalgamation of “State of…” survey results, feedback gathered by developer relations teams, blog posts, YouTube videos, and so on.

A relatively new tool is Jake Archibald’s ranking tool, where anyone with a Github login can add any of the ~150 proposals to a ranking list, then order them in the priority you would give them if the decision were solely yours.  The results will be anonymized and shared with the Interop team.

A number of people (like Ana, Declan, Paweł, Sara, Stuart, and Tyler) have blogged about their rankings, and I love a good web standards bandwagon, so here’s my bloggery!  And if you’d like to hear me talk about all this, head on over to Igalia Chats to give it a listen.  And maybe even a subscribe!

To be crystal clear: these are my personal rankings, and do not predict or even affect how I will vote as Igalia’s representative to Interop.  In fact, it’s almost certain that I will, at some point in the process, vote to cut more than one of the things I’ve ranked here, or at least will not vote to advance them to the next step in the process.  That’s how Interop goes, given that there are an order of magnitude more proposals than the teams have the resources to accept.

First, let’s go with my Top Five, in order.

#1: JavaScript Temporal

You might think I’m picking this one because Igalia has been a large part of getting the standard finished and support into browsers, but I actually ranked it first because it is it just such a huge step forward.  I don’t write much JS, but even I know the Date object is terrible.  The Temporal API is so much better, it’s hard to overstate.  It understands time zones, summer times, and even historical changes like the exact moment a given country changed time zones.  I can’t find anything else in the list that I think is more deserving of all-browser all-out effort to implement and make interoperable.

#2: Improve CSS sibling-count() and sibling-index() functions interop.

This gets my second-place ranking because it is, to me, the essence of the perfect Interop proposal.  In Jake’s ranking tool, the summary for this proposal is literally “Fix remaining test failures.” The proposal itself says “These functions are shipped in all engines, but all have [WPT] failures.” That’s what Interop is for, in my opinion.  The thing exists, but needs a bit more work to be truly interoperable.

#3: CSS box-decoration-break without a vendor prefix

This proposal gets the bronze medal from me for similar reasons the previous proposal grabbed silver.  In this case, we have a long-supported feature that still has a vendor prefix clinging to it.  Unprefix it, already!

#4: HTML headingoffset and headingreset attributes

This one’s emblematic of another role Interop has taken on over the years, which is to get an up-and-coming feature interoperable from the outset.  Here, the ability to use HTML attributes to shift how heading levels are treated is already behind flags in two browsers.  Having the browser teams agree to all work on the same thing at the same time would help find any problems early, and get them fixed before the feature comes out from behind the flags.  I’ll also admit my personal ranking of this is a little bit influenced by the fact that it would help me fix problems on this very web site  —  indeed, in this very blog post — but it would help a lot of others as well, such as Github.

#5: CSS line-clamp property and CSS margin-trim

Okay, I cheated a little here by combining two proposals for this article.  In my actual ranking in Jake’s tool, I ranked them #5 and #6, because I think if you’re going to clamp lines, you’re probably also going to want to trim margins.  I like the capabilities line-clamping enables (and I’d think that even if Igalia weren’t one of the drivers of its implementation), and I imagine that if you’re limiting an element’s height to a certain number of lines of text, you’re probably trying to make things on the page adhere to a vertical rhythm and you don’t want inner margins possibly messing things up.  Maybe I’m off base conflating the two, but it feels right to me.

So there you go, that’s my top five or six rankings.  I ended up ranking twenty proposals in all, triggering a friendly “whoooa there, big fella” warning from the ranking tool.  Here are the rest of what I put on the list, grouped by high-level area and then sub-sorted in roughly alphabetical order, with exceptions to group like topics together:

• Trusted Types  —  Support is already pretty good, but there are gaps, and this helps a lot with security.
• DOM moveBefore()  —  I ran into the lack of this when I was messing around with HTML-in-canvas.  I’d love to see it cross-browser.
• DOM Viewport Segments API  —  This seems useful, given the existence of devices like Galaxy Fold and iPhone Duo.
• Improve interoperability of the WebCodecs API  —  Browsers use this, services use this, interoperability is sub-standard.  Let’s fix it.
• CSS anchor positioning with transforms and animations  —  Animating anchor positioning seems like a thing people will want to do, so let’s make sure browsers are consistent about it.
• CSS color gamut mapping  —  It turns out that the visual appearance of color varies greatly depending on a thousand factors, and gamut mapping would help resolve at least some of them.
• CSS corner-shape property  —  It’s been too hard for too long to create scalloped, beveled, and other not-ovoid corners.
• CSS polygon() vertex rounding — I’d love to be able to round the corners of a polygon, rather than have to construct an arc path or even a shape() value.  This is of a piece with corner-shape, in my view.
• CSS border-shape property  —  And while we’re at it, how about we make it easier to shape the whole of the border area?
• CSS gap decorations  —  Ever since we got Grid layout, I’ve wanted to be able to draw decorations into the gaps, similar to column-rule in CSS Multicolumn.  Doing it now requires essentially spacer-GIF hackery, and I’d rather not have to do that.
• CSS grid lanes (masonry)  —  Masonry (or waterfall) layouts are surprisingly common, and being able to do it in CSS is finally coming, so let’s make sure it’s interoperable.
• CSS linked parameters  —  You know that thing where you can’t have your page’s CSS change the stroke or fill colors of an external SVG?  This would fix that.  Put it in my vein!
• CSS transitioning/animating display and content-visibility  —  The desire to transition from and to display: none has existed for about as long as CSS transitions have existed, and I’d love to see it interoperable so people can use it comfortably.
• CSS var() in size container queries  —  If wanting to say @container (width<= var(--min-width)) has kept you using a CSS preprocessor, this would free you.

That’s where I ended.  There were a few I didn’t rank because I’m pretty confident they’re already going to get strong ranking support.  There were a few I wanted to rank, but couldn’t in good conscience because they had no standard, no evidence of browser interest, or neither of those.  And in all honesty, there were some that didn’t get on my list because I think they’re bad or terribly formed ideas.

How about you  —  what are your favorites?  Go rank them, blog them, and spread the word however you can.

Have something to say to all that? You can add a comment to the post, or email Eric directly.

Update on what happened in WebKit in the week from October 5 to October 12.

A CSS invalidation bugfix, moving steps for optgroup and hr elements, and a handful of fixes to the Layer-Based SVG Engine compose this week's bag of updates, changing to a calmer pace compared to the packed weeks we've had recently!

Cross-Port 🐱

Graphics 🖼️

Testing in the web platform is done in many different ways, but a very important one is through the Web Platform Tests (WPT) repository, which contains around 130,000 tests (note that almost 54,000 are from test262, testing JavaScript), which make a total of about 2.25 million subtests. This shared test suite covers most of the web specifications, or at least that’s the aim. All this is relevant because the different web engines run these tests constantly to ensure compatibility and avoid regressions.

WPT is great, but until very recently it couldn’t properly test platform accessibility (a11y) APIs, which means it was leaving out a very important part of the web, and not finding interop issues, catching regressions and identifying implementation gaps in that area. Browser engines have usually been testing a11y through internal tests, which is nice, but not enough in many cases. WPT can now test platform a11y APIs, and, as we’ll see later in this blog post, these new tests have already found bugs in web engines, and even in the specs.

Testing a11y on the web platform #

Igalia has been exploring ways to solve this problem for quite a while. A first attempt was a project called Acacia, which was sponsored by the Sovereign Tech Fund. Given all platform a11y APIs have Python bindings, we were working on a Python library that would allow us to inspect them directly. After many conversations with different parties, that project was put on hold, and we moved to find a solution directly inside WPT based on all we’ve learnt in that project.

That solution is now a reality, a new type of WPT tests called aamtest, which aim to test the AAM (Accessibility API Mappings) for the different platform a11y APIs. My colleague Valerie Young introduced this new type in March (RFC 204: WPT testing for AAMs) and started to write new tests based on that directly in WPT. As of today, there are 190 aamtest tests in WPT, covering 4 specs: ARIA, Core-AAM, HTML-AAM and MathML-AAM. We’re glad to mention that the recent work on this project has been funded by the NLnet Foundation.

Let’s use a diagram by my other coworker Alice Boxhall to try to clarify a little bit more about what these new tests do. If you want a full description, the diagram is fully explained in the previously linked RFC.

In this diagram you can see the different components related to the a11y support in browsers, and how they interact with each other. The green dashed line is the new proposed way to directly test the platform a11y APIs from WPT.

It’s worth mentioning that back in June, Valerie gave a great talk at the Web Engines Hackfest covering all the details related to this work; don’t miss it if you want to dig deeper.

What does an aamtest look like? #

These tests are written in Python and have different sections covering each platform a11y API. They’re all inside an aamtests folder, which is how WPT identifies this kind of test and knows it has to enable a11y in the browser for them.

Let’s take a look at one example (core-aam/aamtests/role/blockquote.py) and explain the different parts.

# Testing: https://w3c.github.io/core-aam/#role-map-blockquote

TEST_HTML = "<div role='blockquote' id='test'>content</div>"

def test_atspi(atspi, session, inline):
    session.url = inline(TEST_HTML)

    # Spec:
    # Role: ROLE_BLOCK_QUOTE

    node = atspi.find_node("test", session.url)
    assert atspi.Accessible.get_role(node) == atspi.Role.BLOCK_QUOTE

def test_axapi(axapi, session, inline):
    session.url = inline(TEST_HTML)

    # Spec:
    # AXRole: AXGroup
    # AXSubrole: <nil>

    node = axapi.find_node("test", session.url)
    role = axapi.AXUIElementCopyAttributeValue(node, "AXRole", None)[1]
    assert role == "AXGroup"
    subrole = axapi.AXUIElementCopyAttributeValue(node, "AXSubrole", None)[1]
    assert subrole == None

def test_ia2(ia2, session, inline):
    session.url = inline(TEST_HTML)

    # Spec:
    # Role: ROLE_SYSTEM_GROUPING
    # Role: IA2_ROLE_BLOCK_QUOTE

    node = ia2.find_node("test", session.url)
    assert ia2.get_role(node) == "IA2_ROLE_BLOCK_QUOTE"
    assert ia2.get_msaa_role(node) == "ROLE_SYSTEM_GROUPING"


def test_uia(uia, session, inline):
    session.url = inline(TEST_HTML)

    # Spec:
    # Control Type: Group
    # Localized Control Type: blockquote

    node = uia.find_node("test", session.url)
    assert node.CurrentControlType == uia.ControlType.Group
    assert node.CurrentLocalizedControlType == "blockquote"

As you can see, these are quite simple; in this case it’s a test to check that role="blockquote" is properly exposed.

The first thing in the test is some HTML code that we’ll use to test that the role is properly exposed; in this case it’s quite simple, consisting of a DIV element with role="blockquote":

TEST_HTML = "<div role='blockquote' id='test'>content</div>"

Next you can see 4 different functions prefixed with test_, one per platform a11y API:

• test_atspi(): AT-SPI (Linux).
• test_axapi(): AX API (macOS).
• test_ia2(): IA2 (Windows).
• test_uia(): UIA (Windows).

All of them start with the same line, which inlines the HTML to be able to test it (this uses WebDriver). Then they have a comment with the mapping information from the spec. And after that, they use the Python library to check that the role is properly exposed to the platform a11y APIs.

Let’s take a closer look at the AT-SPI one:

    node = atspi.find_node("test", session.url)
    assert atspi.Accessible.get_role(node) == atspi.Role.BLOCK_QUOTE

First it finds the a11y node corresponding to the DOM element with ID test, and then uses the a11y library to get the role of that element. Finally it verifies that it’s the expected one.

You can run these tests locally with the WPT tooling, but you can also check the results for the different platforms on the wpt.fyi website. For example, you can see the results for this particular test.

We have also tweaked wpt.fyi so the tests show as missing for the platforms they don’t belong to. You can see that AT-SPI is only tested on Linux (Chrome and Firefox), while IA2 and UIA are tested on Windows (Edge). These tests cannot run on WPT’s macOS (Safari) CI infrastructure yet, though the WebKit team is working on adding support for them; for now, you can run them locally if you have a Mac.

All this is great, allowing us to directly test the platform a11y APIs, and verify that things are properly exposed to assistive technology users. And being in WPT, it also helps us to check that implementations are compatible with each other, and that there are no regressions that go unnoticed.

Adding events support #

In the last few months, I’ve had the chance to help with this project. And I’ve worked on adding one missing piece, the capability of testing a11y events. When something changes on a page (like a form entry that is marked as disabled), platform a11y APIs emit events to notify assistive technologies, and thus the users, about it.

In order to be able to test those events, we have added a new method called expect_event() to the testing infrastructure, which adds an event listener for a given event in a node, executes an action to trigger it, waits for the event and returns it. That way we can make the test suite more comprehensive by also covering events.

Let’s see an example of how it works in the wild (core-aam/aamtests/event/aria_checked.py).

# Testing: https://w3c.github.io/aria/core-aam/#event-aria-checked

TEST_HTML = "<div id='target' role='checkbox' aria-checked='false'>"

def test_atspi(atspi, session, inline):
    session.url = inline(TEST_HTML)

    # Spec:
    # object:state-changed:checked

    node = atspi.find_node("target", session.url)
    assert "STATE_CHECKED" not in atspi.get_state_list_helper(node)

    def toggle_aria_checked():
        session.execute_script(
            "target.ariaChecked = target.ariaChecked === 'true' ? 'false' : 'true'",
        )

    event = atspi.expect_event(
        "object:state-changed:checked", dom_id="target",
        action=toggle_aria_checked,
    )

    assert event.detail1 == 1
    assert "STATE_CHECKED" in atspi.get_state_list_helper(node)

    event = atspi.expect_event(
        "object:state-changed:checked", dom_id="target",
        action=toggle_aria_checked,
    )

    assert event.detail1 == 0
    assert "STATE_CHECKED" not in atspi.get_state_list_helper(node)

In this case the test checks the event triggered when the aria-checked attribute is modified.

Again it has quite simple HTML, and a similar structure to the previous one, though there are a couple of new things.

We define a function toggle_aria_checked() that modifies the value of the aria-checked attribute. That function is passed as an action to expect_event() to which we also pass the event name object:state-changed:checked and the ID of the node in which we’re going to wait for the event. When the event is triggered, we get it and we can check different things on it to verify that it’s working correctly. Then we run it twice to check that the event is also triggered when you do the opposite change to the aria-checked attribute.

How useful are these tests? #

One question you might have is why we are doing all this, knowing web engines already have internal tests covering this (as mentioned before). That’s true, but just with the first tests we’ve added, we have discovered interop issues and missing features in different places. Let’s review a few examples.

While working on events support, we started with a simple example changing the value of aria-disabled and catching the expected events, which for AT-SPI is expected to trigger 2 events: object:state-changed:enabled and object:state-changed:sensitive. We realized that Firefox wasn’t doing that, as it was only triggering one of them (enabled). This was a small issue and easy to fix, so we went ahead and fixed the issue in Firefox; the patch landed even before the support for testing events in aamtest was merged in WPT.

Another example, while testing an ARIA “must” statement about aria-errormessage when aria-invalid="false", we found again that Firefox wasn’t fulfilling the spec. In this case, this was a known old bug that we also fixed upstream.

And it’s not only Firefox; for example, when we added tests for Core-AAM attributes, we discovered that Chromium wasn’t properly exposing aria-sort="none", which again we fixed upstream.

Last but not least, while working on these tests we have also found spec issues, not only implementation ones. For example, adding a test for the <abbr> element, we found a wrong statement in the HTML-AAM spec, which has already been fixed.

There are more examples like these; these are just a few. In the early days of these new tests, they have already proven to be very valuable for identifying issues and improve the a11y support in different engines. It’s really nice to see how useful they are while things are coming along.

Future work #

This doesn’t finish here; this work can be considered the foundation, which is now in place and people can build things on top of it.

Now that WPT has a new type of tests to cover a11y platform APIs, we expect that aamtest tests start to be used more and more to cover a11y support for new and existing web platform features. Our recent work has been focused on the Linux platform, but there is more work to do covering other platforms, completing tests for platforms that aren’t being tested yet. If you want to do that, you can check the documentation.

Not only that, with all these tests, we’re finding interoperability issues here and there, so we hope people start to fix them where convenient. Apart from that, there is some flakiness when running these tests in the WPT CI for both Chromium and Firefox, as they’re the first ones testing against the platform a11y APIs. This needs more investigation to come up with a solution.

All in all, we’re very happy to have reached this point and to see these tests already having a positive impact on the web platform. We’d like to thank everyone who has been helping make this happen, whether through reviews, feedback, participation in discussions, and so on.

Update on what happened in WebKit in the week from September 28 to October 5.

Another week populated with updates! This time we sweep through a new crypto backend, more layer-based SVG engine updates, better filters, and some miscellaneous updates.

Cross-Port 🐱

Graphics 🖼️

WPE WebKit 📟

In the previous post, Extension Migration Progress Update – Part 2, we demonstrated how a Chrome extension can be installed from the Chrome Web Store using the Extension Shell.

This time, we’d like to take the demo one step further.

Instead of focusing on the installation process, the following two videos show extensions that have already been installed in the Extension Shell and are actually running on real web pages.

These demos are intended to show that the migrated extension APIs are not only sufficient to install extensions, but can also support extensions interacting with real-world web content.

Demo 1: Knockoff – Amazon Shopping Filter

The first demo uses Knockoff – Amazon Shopping Filter, a Chrome extension that filters products on Amazon.

In the video, the extension is installed in the Extension Shell and then used while browsing an Amazon search results page.

Extension: Knockoff – Amazon Shopping Filter
Test page: Amazon – LED Strip Lights search

The video demonstrates the extension interacting with the Amazon page and modifying the displayed content based on its functionality.

Demo 2: Allow Copy & Enable Right Click – Text Unlock

The second demo uses Allow Copy & Enable Right Click – Text Unlock.

This extension allows users to copy text and use the right-click context menu on websites that would otherwise prevent these actions.

In the video, the extension is used on a real article page where copying and right-clicking are restricted.

Extension: Allow Copy & Enable Right Click – Text Unlock
Test page: Native vs Cross-Platform App Development – What Should Businesses Choose in 2026?

The video shows the extension modifying the behavior of the web page and enabling the previously restricted interactions.

From Installation to Real-World Usage

These demos represent the next step in our Extension Shell work.

In Part 2, we showed that an extension could be installed directly from the Chrome Web Store. With these demos, we are showing what happens after installation: the extension can run on real websites and interact with their web content.

Together, the demos cover the basic flow we want to support:

Chrome Web Store → Extension installation → Extension execution → Interaction with real web content

There is still more work to do as we continue migrating extension APIs from //chrome to //extensions, but these demos provide a concrete example of how the migrated APIs can be used by real-world Chrome extensions outside of the Chrome browser UI.

Following the last blog post, I would now document in more detail how we fixed the visual jank in the Android phones using WPE, it clearly shows testing an embedded engine in multiple devices is really valuable.

Originally, the Pixel 7 looked broken: a horizontal bar of stale content between tile rows, black corners appearing around spinners and rounded elements, and images that sometimes came up as solid rectangles of garbage. We tested multiple options to analyze the problem: it was not a tearing problem, the frames were correctly isolated, it was the frames rendering being wrong, which is much more complex to look into and much harder to profile.

What eventually gave us the answer was testing different rendering configurations. Skia can rasterize on the CPU or on the GPU, we support both in WPE, and in both cases it can use a pool of worker threads, so when we ran some tests we verified:

• CPU with workers worked well, no jank.
• GPU on the main thread with no workers at all worked well, no jank.
• GPU with workers was broken: jank, every time.

So neither GPU rasterization nor parallelism was the problem on its own, it was the combination was causing the visual artifacts. The reason was that each painting worker has its own GrDirectContext, with its own GL context, sharing the underlying EGL display and textures with the context the compositor uses to put the frame together. That kind of sharing depends on the driver getting context synchronization right, and on Mali ((the Android driver for the Pixel 7 phone) it does not. We confirmed it by writting a small standalone reproducer outside WebKit.

The fix for all of that, which is Carlos’ WebKit PR #65286, uses a Skia feature that exists precisely for this kind of problem: deferred display lists. The feature comes from the idea of recording raster work in one place and executing it in the GPU process. In WPE we are currently still using a compositor thread with a GPU context, but the idea is the same. The approach is that you describe the destination surface up front, as a GrSurfaceCharacterization holding the surface properties, and you build a GrDeferredDisplayListRecorder from that description. The canvas you get from the recorder accepts and validates all the usual drawing, but does not send any GL commands at all, and at the end detach() gives you a GrDeferredDisplayList. The worker thread does all the expensive CPU side of painting (building the draw operations, resolving paints and geometry) with no GPU context required. Then the compositor thread, in SkiaBackingStore::Tile::update, creates or reuses an SkSurface compatible with the deferred operations object and calls skgpu::ganesh::DrawDDL with it. The compositor’s context is the only thing that ever renders tile content with the GPU context.

It is also very interesting how this is implemented. A thread with no GPU context can not sample a texture, so any image that already lives on the GPU (including the glyph and image atlases) can not be referenced directly while recording. Instead, they are turned into promise images. This is what makes atlas access safe from several recording threads at once. It is also the reason why, the first time we tested, every image was rendered as a black rectangle, which turned out to be an unrelated Android problem with the format used to upload those atlas textures, and cost us a couple of days wondering why the fix did not work.

Once it was activated, the Pixel 7 jank was gone, the bars, the black corners and the garbage in general. Also, parallel painting is still parallel, only the submission is serialized. There was a price, though: moving all the real rasterization to the compositor thread puts it in the same queue as frame presentation, and the first version felt slower. The cause was a fresh render target being allocated for every tile update, so adding a cache to recycle the backing buffer solved the problem.

Nowadays we are using DDL and the skia compositor by default, we plan to remove TextureMapper support, and the Skia compositor is what we provide in main.

The last part of the story is a warning note about benchmarks. When we ran a proper A/B testing with MotionMark, the overall score said deferred display lists were 7% slower, and that number was something we needed to understand: the whole difference came from a single subtest, Suits, which scored 76 against 19. The option without DDL was rendering incorrectly in Mali, so the benchmark in the end was rewarding broken frames in Android. The rendering was just find in Linux drivers where we tested it, so the decision took the whole situation into account. We did not want to add more rendering options (that usually means more complex maintenance), and the regression was measured against an architecture we knew was not completely safe.

Next time I could talk about how we are currently planning to recover that performance that is going to show pretty soon in the benchmarks, and it has taken some time because it requires patching Skia.

Update on what happened in WebKit in the week from September 21 to September 28.

Following the stable releases last week, we have the first security advisory covering fixes included in them; the flurry of fixes and polish for the Skia compositor continues; SharedArrayBuffer gets re-enabled, and more.

Cross-Port 🐱

JavaScriptCore 🐟

Graphics 🖼️

WebKitGTK 🖥️

WPE WebKit 📟

WPE Platform API 🧩

Releases 📦️

Ideals

I thought it would be interesting to talk a little bit about the standards process given a few events that lined up last week in my mind...

Last week my colleague Rakhi wrote a great piece about how standardization at the edges sometimes (maybe even often) works backwards because there are fallable humans all along the way - writing the spec, interpreting the spec, writing tests, and so on. The very next day, another such case was brought to the CSS Working Group. A lot of people found this very interesting.

Around the same time, we published a new episode of Igalia Chats about Global Privacy Control, which, at the end of the day, is mainly a boolean header that a browser can send. As we discussed in the Chat, it probably feels suprising to learn that the efforts to do something like this wind up spanning _many years_ (there are some helpful links there to the history of Do Not Track as well) and what makes people believe that this time it will be different (that the pressure kind of came from outside first is a big one). Anyway, it's kind of fascinating if you don't know any of this stuff.

Then, on Thursday, the SpellCheckCustomDictionary proposal advanced to stage 2 in WHATWG. If you're not familliar with it, I wrote a short post about it not too long ago. The reason I mention this is that it's been stuck for months at stage 1, because we needed to talk a lot about Internationalization concerns.

This is a potentially important discussion, to be sure. However, only "potentially" so because while spell check has existed in browsers for more than 10 years, for the most part, the `lang` attribute remains unconsulted. Where it is, it's not really specified how that should work in terms of general spell checking.

To me, that feels worth talking about, because it sometimes feels like there are impractical gaps that it's not clear how to fill. On the one hand, it feels unreasonable to expect a small improvement to spell checking to be held to a higher standard than the mature specification and existing implementations. On the other hand, I fully agree that these are important questions. The difficulty, at some level, is that all of these uniquely important questions are also somehow dramatically underfunded. Accessibility, Internationalization, Privacy - or even just integrating and normalizing MathML and SVG into One Platform being just a few obvious examples.

It's hard to live up to ideals if we're not funding them well enough, or if they aren't pragmatic enough (for example, demanding the the spec is right).

I don't really know how to fix the funding bit except to continue to lobby vendors to allocate more budget to those things (or, in come cases, maybe just any budget at all) and to suggest that we need more experiments in collectively funding and prioritizing those ideals in implementation work. Lots of features are like fast food or desserts - someone needs to make sure that we eat our vegetables.

GstPipelineStudio 0.6.0 #

Let the Pipeline Build Itself #

GstPipelineStudio 0.6.0 is out! After 0.5.1 taught GPS to open dot files, this release teaches it to suggest: the next element for any free source pad, or even a complete pipeline from a media file.

The Right-Click I Always Wanted #

Almost since I started using GStreamer, I have wanted this feature. It comes straight from my days with GraphEdit and DirectShow: connecting black boxes taught me what each filter did, but I was never quite sure I had picked the best one for a given pin. So I relied on the right-click menu to show me my options.

GStreamer makes that question even harder. A typical development setup exposes more than 1,500 features, with five or more decoders able to handle H.264 alone. Which one should I use? Can I try another? Newcomers and seasoned developers alike ask this every day.

Right-click any free source pad and GPS now offers three choices:

• Suggest compatible elements: list the elements that can link to this pad
• Auto-select next element: let GPS pick the best candidate
• Auto-select all elements: let GPS build the whole branch, up to the last compatible element or the sink

The same mechanism works from a media file. Open a URI and, much like decodebin3, GPS lays out a working pipeline with every decode branch. Except here you can see it, and change it. Try it, it’s really fun!

A Fresh, Living Interface #

GPS 0.6.0 has a new look. The panels have been reworked, and libadwaita now drives the UI on every platform, Windows included, so GPS looks and behaves the same wherever you run it.

Color themes arrive too. Pick the one that suits you: the same pipeline, a different mood.

The interface also comes alive during playback. A heartbeat shows at a glance that your pipeline is running, so you are no longer watching a static diagram but a live system.

The graph view is also easier to handle. You can now select several elements at once, then move or delete them in a single batch, which makes large and complex pipelines far easier to reorganize. The underlying graphmanager library is becoming mature enough to fly on its own soon.

A Catalog of Classic Pipelines #

Not sure where to start? The new Pipelines tab lists classic GStreamer pipelines, adapted to your platform: test sources, timestamp overlays, scaling, framerate limiting, tee, compositor and more. Pick one, and it opens ready to play and tweak.

And More #

Here are the other improvements in 0.6.0:

• A much faster logger that keeps up with very verbose output such as GST_DEBUG=*:7, laying the groundwork for log filtering
• French and Spanish translations, managed with Weblate. Contributions for your language are welcome!

What’s Next #

Log filtering comes next. It could take GPS to another level, letting you validate a pipeline or track down a problem from a friendly interface instead of digging through raw logs. Pipeline statistics and smarter auto-suggestion are also on the way.

We had a great time building this release, and we hope you’ll have as much fun playing with it, and with the fabulous GStreamer framework. Feel free to propose new features with an RFC here or send merge requests here.

As usual, if you would like to learn more about GstPipelineStudio, GStreamer, or any other open multimedia framework, please contact us!

And here we are three years after I wrote about the Google Pixel Fold being announced, followed now with the announcement of the iPhone Duo...(I have questions about the naming by the way).

Four years ago I was talking about web primitives in the platform for the Surface Duo. My how time flies.

There are CSS media features, a Viewport Segments API, a Device Posture API but Chromium based browsers are the only ones currently supporting these things.

I haven't been able to find any signal yet on whether Safari will support these things in the web platform as the developer docs focus on application development.

If you're interested in trying out the platform features, you can emulate the Surface Duo and Galaxy Z Fold in the developer tools.

And if you're thinking, do I really have to have my website adapt to two screens? The answer is no. Adding a design to an application or dual screen makes sense if you have an experience that has two simulataneous contexts that are useful e.g. a list of email messages/inbox on one screen, an open message, email thread or email composer on the other.

Here's one of my talks from 2022 if you're interested in learning more about what's available in the browser for dual screen/foldable devices.

Happy building :)

Update on what happened in WebKit in the week from September 14 to September 21.

Following up to last week, we have a handful of exciting fixes to WebKitGTK and WPE WebKit covering Sysprof, rendering of CSS "filter" and "backdrop-filter", and the Layer-Based SVG Engine. We also have optimizations, stable releases, and new blog posts!

Cross-Port 🐱

Graphics 🖼️

Releases 📦️

Community & Events 🤝

In this instalment of the “Why don’t we do a demo?” series I’ll continue through the software development part of the demo and I’ll end up with an MVP that will be later shaped into a finished product. Check out the previous parts (The plan, Software development) to have more context on how we got here.

Problem 13: Bringing up the touch-display in the console device

I chose the display hardware based on what was readily supported in Zephyr so, in theory, I’m expecting it to work out of the box. However, things are rarely that simple.

There’s a basic LVGL sample as a starting point to check the hardware. In principle, if that works, all I have to do then is to design and implement the user interface. The sample works but, as the docs explain, I’ll need to do a small manual tweak in the board:

Touch controller IRQ line is not connected by default. You will need to solder the 5 INT jumper to use it. You will also need to adjust driver configuration and its Device Tree entry to make use of it.

Besides, while the basic sample works, there are other problems when I try to draw more complex layouts.

Solution

Soldering the jumper isn’t really an obstacle:

The pads are accessible and big enough to be easy to solder using a solder pen or silver paste syringe with a steady hand:

After soldering, I tested the sample application again and I could check that the touchscreen works by clicking on the test button:

But now, if I continue building upon this, creating a proper interface with more widgets and a more complex layout, even if it’s just for testing for now, the display shows a blank screen.

This is a fairly common problem and it was easy to search for info about it, everything pointed to the small default size of the memory region for the LVGL heap (CONFIG_LV_Z_MEM_POOL_SIZE). After some experiments I settled on a value that worked for the application needs. To err on the safe side, the strategy is to allocate as much memory as possible to LVGL after the memory needs of all the other subsystems are met. The network stack and web server are the other parts that need large-ish memory allocations, but the LVGL memory pool will be the largest by far, at 32 KB.

Building the application with this configuration shows that I’m now using almost all the RAM:

Memory region         Used Size  Region Size  %age Used
           FLASH:      780028 B         1 MB     74.39%
             RAM:      251936 B       256 KB     96.11%
        IDT_LIST:          0 GB        32 KB      0.00%

But I can continue adding more widgets and creating more complex GUI layouts.

Problem 14: Touch-display problems

The initial LVGL “hello world” test above works as expected when tested on the display but, once I start adding more components to the interface, something seems wrong. Buttons aren’t pressed when I click on them, and touching random areas of the display trigger some widgets in other parts of the screen.

After a while, I noticed this behavior isn’t random. When I click something on the right side, the widget receiving the action is on the left side. When I click on the top area, the action is triggered at the bottom. The touch coordinates seem to be flipped. This probably went unnoticed because the layout of the “hello world” sample is just a single button in the center of the display, that is, it’s symmetrical in both axes, so any touch on the button will push it, flipped coordinates or not.

Solution

This one’s really easy, just a matter of removing the pointer axes inversion properties from the display device tree. Done and merged.

Problem 15: GUI design

The design and implementation of the graphical interface involves two different problems:

• How to fit all the information and controls in a tiny display and in a platform like this.
• How to implement it effectively.

In terms of design, I’m restricted by the screen real state (3.5 inches), but also by the hardware (memory and display speed). I certainly can’t make anything as responsive and feature-rich as what we’re used to in smartphones, so I need to keep it simple but functional.

With regards to implementation, the main obstacle is the complexity of the LVGL api and the amount of trial-and-error needed while tweaking and testing the layout, which can be very time-consuming, even when testing on the simulator. While LVGL isn’t more complex than other graphical libraries, the amount of widget configuration variables, particularly related to styling, means I’ll still need to do a lot of tests to get the positioning and behavior just right.

Solution

I started designing the simplest possible layout, a main container with subcontainers arranged in a single column, where every component is a widget box containing the controls for a peripheral slot. It’d look something like this:

When a slot isn’t connected to any peripheral, I can make everything except for the “Status” label and the “Scan” button invisible, I can shrink the containers vertically. The scrollbar would let us move through the available slots if they don’t all fit in the main container.

Now I need to grow the initial LVGL example code to implement this layout. Normally, this would involve a lot of iterations of changing details in the code, building and testing until I get everything looking and working the way I want. Fortunately, I found a tool (EEZ Studio) that’ll make part of the process much easier and faster, as it has an LVGL visual editor and code generator that will let me sketch the layout visually, do all the style changes and then generate the code. I’m not taking the generated code as is, as I need to integrate it into the application and I’ll want to do some things differently, but this is a huge time saver compared to doing the whole process by hand.

Problem 16: Display refresh speed

The GUI is looking good now, I can test it on the board and control the peripherals through the display, but I noticed that when the peripheral controls don’t fit into one screen and I need to scroll down to reach the additional controls, the scroll redisplay is really slow. Normally, re-drawing a label or a button happens fast enough, but re-drawing the whole screen happens too slowly, you can clearly notice parts of the screen being updated as you scroll, with the naked eye. This drags down the responsiveness and makes the display practicaly unusable in some scenarios.

All in all, this adds yet another restriction: no scrolling.

Solution

The best solution I could find for this is to draw a single controller widget container in the screen at a time, and switch from one controller to another using tabs. In each controller container, I also need to make sure all peripheral widgets fit in a single screen so I can safely disable and avoid scrolling altogether:

That’ll do, in theory. Implementing it isn’t too much of a hassle after having done the previous, non-tabbed version. The GUI logic is simple enough, and keeping a hierarchical and instance-based model for the data structures, where widgets are nested inside containers (nested within other containers), and certain parts can be modeled and duplicated, makes it easy to control each widget group and make them reactive to events. For instance, remove the peripheral widgets of a controller when the controller isn’t connected, or disable the “Scan” button while the scan is already ongoing.

void setup_slot(int r, int s)
{
    lv_obj_t *tab = controller_widgets[r].tab;
    struct slot_widget *slot = &controller_widgets[r].slots[s];
    lv_obj_t *label;

    slot->slot_idx = s;
    slot->controller_idx = r;

    slot->slot_cont = lv_obj_create(tab);
    [...]

    slot->slot_status_cont = lv_obj_create(slot->slot_cont);
    [...]

    slot->status_label = lv_label_create(slot->slot_status_cont);
    lv_obj_set_size(slot->status_label, LV_SIZE_CONTENT, LV_SIZE_CONTENT);
    lv_label_set_text_fmt(slot->status_label, "Slot %d: Disconnected", s + 1);
    slot->scan_btn = lv_btn_create(slot->slot_status_cont);
    label = lv_label_create(slot->scan_btn);
    lv_obj_set_size(label, LV_SIZE_CONTENT, LV_SIZE_CONTENT);
    lv_label_set_text(label, "Scan");
    lv_obj_add_event_cb(slot->scan_btn, controller_scan_btn_cb,
        LV_EVENT_CLICKED, (void *)slot);
    [...]
}

void setup_controller(int r)
{
    char controller_str[30];
    lv_obj_t *label_cont;
    lv_obj_t *label;
    lv_obj_t *tab;
    int i;

    snprintf(controller_str, sizeof(controller_str), "Controller %d", r + 1);
    controller_widgets[r].tab = lv_tabview_add_tab(tabview_container, controller_str);
    tab = controller_widgets[r].tab;
    [...]

    /* Container for label and button */
    controller_widgets[r].label_cont = lv_obj_create(tab);
    label_cont = controller_widgets[r].label_cont;
    lv_obj_set_width(label_cont, LV_PCT(100));
    lv_obj_set_height(label_cont, LV_SIZE_CONTENT);
    [...]

    /* Label and Scan button */
    controller_widgets[r].controller_label = lv_label_create(label_cont);
    lv_label_set_text(controller_widgets[r].controller_label, "Status: Disconnected");
    controller_widgets[r].scan_btn = lv_btn_create(label_cont);
    lv_obj_set_style_pad_top(controller_widgets[r].scan_btn, 5,
        LV_PART_MAIN | LV_STATE_DEFAULT);
    lv_obj_set_style_pad_bottom(controller_widgets[r].scan_btn, 5,
        LV_PART_MAIN | LV_STATE_DEFAULT);
    label = lv_label_create(controller_widgets[r].scan_btn);
    lv_obj_set_size(label, LV_SIZE_CONTENT, LV_SIZE_CONTENT);
    lv_label_set_text(label, "Scan");
    lv_obj_add_event_cb(controller_widgets[r].scan_btn, console_scan_btn_cb,
        LV_EVENT_CLICKED, (void *)r);

    /* Create slot widgets */
    for (i = 0; i < MAX_PERIPHERALS_PER_CONTROLLER; i++) {
        setup_slot(r, i);
    }
}

void startup_screen(void)
{
    [...]

    /* Main container */
    tabview_container = lv_tabview_create(lv_screen_active());
    lv_obj_set_size(tabview_container, LV_PCT(100), LV_PCT(100));
    lv_tabview_set_tab_bar_position(tabview_container, LV_DIR_TOP);
    lv_tabview_set_tab_bar_size(tabview_container, 28);
    lv_obj_t *content = lv_tabview_get_content(tabview_container);
    lv_obj_clear_flag(content, LV_OBJ_FLAG_SCROLLABLE);

    [...]

    /* Populate tabs */
    for (i = 0; i < MAX_CONTROLLERS; i++) {
        setup_controller(i);
    }

    [...]
}

The sketch looks like this on the simulator:

Problem 17: Battery management

The peripheral devices are battery powered but so far we’re just trusting the the batteries have enough charge and that everything will just work. At the very least we should periodically monitor the battery charge, set some operational safety measures in the firmware and broadcast the battery level to the connected devices.

Solution

The XIAO nRF54L15 provides a mechanism to measure the battery level on demand using a load switch and an ADC.

To get a battery voltage measurement I need to enable the load switch, wait for it to turn on, sample the ADC connected to its output and convert the value to millivolts, doing the necessary adjustments to the reading (considering the voltage divider applied at the output).

Once I have this, there are two things I’ll want to do with it:

• Shutdown the firmware when the voltage is below a safe threshold. The SoC has a brownout reset generator and a glitch detector, but I’d rather detect and manage this via firmware.
• Publish the battery level, in percentage, using the standard BAS service.

This can be done with very little code:

/* Data of ADC io-channels specified in devicetree. */
static const struct adc_dt_spec adc_channels[] = {
    DT_FOREACH_PROP_ELEM(DT_PATH(zephyr_user), io_channels,
                         DT_SPEC_AND_COMMA)};

/* Battery load switch (for battery level measurement) */
static const struct device *const vbat_reg = DEVICE_DT_GET(DT_NODELABEL(vbat_pwr));

/* ADC data */
uint16_t buf;
struct adc_sequence sequence = {
    .buffer = &buf,
    .buffer_size = sizeof(buf),
};

void shutdown(void)
{
    gpio_remove_callback(button.port, &button_cb_data);
    gpio_remove_callback(button_board.port, &button_board_cb_data);
    led_state = 0;
    gpio_pin_set_dt(&led, led_state);
    gpio_pin_set_dt(&led_board, led_state);
    if (led_indication_enabled && !atomic_get(&indication_ongoing))
        k_work_schedule(&led_indicate_work, K_NO_WAIT);
    k_sleep(K_SECONDS(5));

    sys_poweroff();
    /* Should never reach here */
    while (1) {
        LOG_ERR("Shouldn't reach here");
        k_sleep(K_FOREVER);
    }
}

/* BAS handler */
void battery_voltage_check(struct k_work *work)
{
    int err;
    int32_t val_mv;
    uint8_t val_pct;

    regulator_enable(vbat_reg);
    k_sleep(K_MSEC(100));
    err = adc_read_dt(&adc_channels[ADC_CHANNEL_ID], &sequence);
    if (err < 0) {
        LOG_ERR("Could not read (%d)", err);
        return;
    }

    /*
     * If using differential mode, the 16 bit value
     * in the ADC sample buffer should be a signed 2's
     * complement value.
     */
    if (adc_channels[ADC_CHANNEL_ID].channel_cfg.differential)
        val_mv = (int32_t)((int16_t)buf);
    else
        val_mv = (int32_t)buf;
    err = adc_raw_to_millivolts_dt(&adc_channels[ADC_CHANNEL_ID], &val_mv);
    /* conversion to mV may not be supported, skip if not */
    if (err < 0) {
        LOG_ERR(" value in mV not available");
        return;
    }
    else {
        /* Correct the voltage taking into account the voltage divider */
        val_mv *= 2;
    }
    regulator_disable(vbat_reg);

    if (val_mv >= BATTERY_VOLTAGE_MAX) {
        val_pct = 100;
    } else if (val_mv <= BATTERY_VOLTAGE_MIN) {
        val_pct = 0;
        LOG_ERR("Critical battery voltage: %d mV - Shutting down!", val_mv);
        shutdown();
    } else {
        val_pct = ((val_mv - BATTERY_VOLTAGE_MIN) * 100) /
            (BATTERY_VOLTAGE_MAX - BATTERY_VOLTAGE_MIN);
    }

    bt_bas_set_battery_level(val_pct);
    LOG_DBG("Battery level: %d%%", val_pct);
}

Now I only have to schedule the battery_voltage_check() function periodically. With this, the controller devices can subscribe to the peripheral BAS service and get a notification every time the battery level changes.

Problem 18: web GUI

In the previous blog post I set up a web server on the console board and ran a basic test. Now that I can control the devices through the display graphical interface, it’d be interesting to add a secondary web-based interface so that we can control them remotely as well.

The main requirements are:

• It must be lightweight and simple.
• It must be responsive and hassle-free.
• Both interfaces must be synchronized, i.e. any change in one of the interfaces must be updated in the other one in real time.

Solution

The first technical decision I need to take is how I’ll implement the web page in terms of the Zephyr web server api in a way that’s both lightweight and simple but also dynamic. A simple way to design a dynamic page would be to define a dynamic resource handler that will render different contents depending on the client requests. That means the firmware will be responsible of the content generation, which can be a resource-intensive task. Besides, the client won’t be aware of the interface changes unless it does a new request.

A much better option is to have a live connection between the server and the client through a websocket. As an additional improvement, I’ll offload the formatting and processing to the client as much as possible, so what the firmware will do is to serve a single template-based web page and the necessary javascript code for the client to do the formatting and page updating, and communicate the GUI changes from server to client and vice-versa using json messages through an established websocket.

This will minimize the amount of data to store in the firmware (only a small html page and some javascript code) and the GUI updates will be carried out using small json-formatted payloads.

The html contents (excluding the CSS definitions) can be as small as this:

<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="UTF-8">
    <style>
    <!-- css definitions -->
    </style>
    <script src="main.js"></script>
    <title>Igalia Zephyr Demo</title>
  </head>
  <body>
    <main>
      <h1>Igalia Zephyr Demo</h1>

      <div id="controller_container">
      </div>

    </main>
  </body>

  <template id="controller_template">
    <div id="controller" class="controller_info">
      <div style="display:flex; align-items:center;">
        <h2>Controller device</h2>
        <input id="controller_scan" type="button" value="Scan" style="margin-left:20px;">
      </div>
      <ul>
        <li>
          Connection state: <span id="controller_state"></span>
          <span id="controller_scan_animation" class="waitdots"></span>
        </li>
      </ul>
      <!-- Slots divs go here -->
    </div>
  </template>

  <template id="slot_template">
    <div id="slot" class="slot_info">
      <div style="display:flex; align-items:center;">
        <h3>Slot</h3>
        <input id="slot_scan" type="button" value="Scan" style="margin-left:20px;">
      </div>
      <ul>
        <li>
          Connection state: <span id="slot_state"></span>
          <span id="slot_scan_animation" class="waitdots"></span>
        </li>
        <li id="slot_dev_id_li">
          Id: <span id="slot_dev_id"></span>
        </li>
        <li id="slot_battery_li">
          Battery: <span id="slot_battery"></span>
        </li>
        <li id="slot_led_li">
          LED state:
          <span id="slot_led_state" style="display:inline-block; width:50px"></span>
          <input id="slot_toggle_led" type="button" value="Toggle LED">
        </li>
      </ul>
    </div>
  </template>
</html>

And the javascript code that will run in the client will control the layout:

class Controller {
    constructor(idx, container_id) {
        let templ = document.getElementById('controller_template');
        let clone = templ.content.cloneNode(true);
        this.div = clone.querySelector('#controller');
        this.id = `controller_${idx}`;
        this.div.id = this.id;
        this.div.querySelector("h2").innerHTML = `Controller ${idx + 1}`;

        this.scan = this.div.querySelector('#controller_scan');
        this.scan.id = `${this.id}_scan`;

        this.state = this.div.querySelector('#controller_state');
        this.state.id = `${this.id}_state`;

        this.dots = this.div.querySelector('#controller_scan_animation');
        this.dots.id = `${this.id}_scan_animation`;

        // Insert controller div in the container
        let container = document.getElementById(container_id);
        container.appendChild(this.div);

        // Create and insert slot divs in the controller div
        this.slots = [
            new Slot(idx, 0),
            new Slot(idx, 1),
            new Slot(idx, 2)
        ];
        for (let s of this.slots) {
            this.div.appendChild(s.div);
        }

        // Initial element states
        this.state.innerHTML = 'Disconnected';
        this.dots.style.display = 'none';

        this.scan.addEventListener('click', (event) => {
            console.log(`Request scan on ${this.id}`);
            postAction(`{"target": -1, "action": ${Action.SCAN}, "param": ${idx}}`);
        })
    [...]
}

window.addEventListener('DOMContentLoaded', (ev) => {
    const ws = new WebSocket('/');

    controllers = [
        new Controller(0, "controller_container"),
        new Controller(1, "controller_container")];

    ws.onmessage = (event) => {
    const json = JSON.parse(event.data);
        for (let controller_idx in json) {
            if (controller_idx >= controllers.length)
                continue;
            controllers[controller_idx].update(json[controller_idx]);
        }
    }
})

Now, I need to define a suitable json schema to format the data passed between the server and the client. I’ll be using a fixed-size buffer in the firmware for the payload, so I need to make sure that the format produces compact enough messages. For simplicity, I also want every message to carry the whole peripheral state for all the devices, rather than partial updates.

This is the schema I came up with:

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "type": "object",
  "patternProperties": {
    "^[0-9]+$": {
      "type": "object",
      "description": "Controller status",
      "required": ["state", "slots"],
      "properties": {
        "state": {
          "type": "integer",
          "description": "Current state of the controller"
        },
        "slots": {
          "type": "array",
          "items": {
            "oneOf": [
              {
                "type": "null"
              },
              {
                "type": "object",
                "required": ["state", "id", "led", "bat"],
                "properties": {
                  "state": {
                    "type": "integer",
                    "description": "Current state of the slot"
                  },
                  "id": {
                    "type": "string",
                    "description": "String identifier of the connected peripheral"
                  },
                  "led": {
                    "type": "integer",
                    "description": "Peripheral LED state"
                  },
                  "bat": {
                    "type": "integer",
                    "minimum": 0,
                    "maximum": 100,
                    "description": "Peripheral battery level percentage"
                  }
                },
                "additionalProperties": false
              }
            ]
          },
          "minItems": 3,
          "maxItems": 3,
          "description": "Array containing the status of the controller slots"
        }
      },
      "additionalProperties": false
    }
  },
  "additionalProperties": false
}

With which I can format a message like this one:

{
    "0": {
        "state": 3,
        "slots": [
            null,
            {
                "state": 1,
                "id": "ZD-01",
                "led": 1,
                "bat": 97
            },
            null
        ]
    },
    "1": {
        "state": 0,
        "slots": [null,null,null]
    }
}

Once up and running, here’s how the web GUI looks like:

Problem 19: BLE security

At this point, the application is functionally complete in the sense that it showcases all the features I planned. However, so far we only cared about functionalities for a minimum viable product. There are many things we can improve and do properly rather than just for a simple demo, the most important of them is to enable BLE security and privacy.

Right now, all the BLE services I enabled are open. Anyone can connect to them using any client, which is a good first approach for testing and debugging. However, the demo will be installed in a conference or a fair with thousands of people, and it’d be totally vulnerable to any user that (maliciously or not) wants to interact with it without permission.

What I want to do is to allow the demo devices to establish connections between each other only, preventing other devices from joining. I don’t care much about data privacy, although that’d be an added bonus.

Solution

For these requirements, what I need to do is to enforce BLE security in all connections, which will give us protection to MITM attacks if I set up a way to require human interaction during the pairing process.

The basic security strategy I’ll implement is:

• All BLE services provided require authentication.
• Connection between any pair of devices requires L4 security (pairing).
• Pairing requires human interaction on the target device.
• The pairing process is allowed only during a limited time window. Additional pairing requests can be started manually.
• The amount of paired devices in each device is limited.
• Connection requires device proximity.

Since the only interactive user interface I have in the peripherals is a button (4 buttons in the controller devices), which I’m already using to toggle the LEDs, I need to multiplex it somehow to allow using it as a “keypress” for pairing confirmation, and also to start a pairing advertising process and, optionally, to “forget” all previous pairings, in case we ever need to do that. I chose to encode the different actions based on the duration of the button press. That way, a short press is normally used for toggling the LED, but it’s also used as a pairing confirmation during the pairing process. A long press is used to restart the advertising process, and a very long press to forget all previous pairings.

The LED also needs to encode the current status of the device. Since the advertising and pairing processes are only temporary, I’m setting the LED to blink during both of them: slowly during advertising and fast during the pairing window, where the user needs to press the button to confirm the pairing.

Implementing these security features means the connection handling is considerably more complex than before. Fortunately, the state-machine-based architecture I had in place is easy to extend and modify to incorporate the changes.

This should be enough to keep the application secure. Maybe even too paranoid, but I’d rather have no surprises during the deployment. For convenience, I’ll pre-pair all the devices so that they’ll be ready to work right after booting. A possible improvement for this setup would be to use directed advertising to reconnect to previously bond devices, but I’ll leave that for future iterations.

With this we’re done with the firmware / software part of the project, thanks for reading this far! The next thing to do is to turn this into a presentable demo and figure out the materials, assembly method and the setup logistics. I’ll go through the details in the next post of this series.

WPE WebKit and WebKitGTK 2.54 have been released with a bunch of improvements and new APIs as usual, but there’s one point that kept the Igalia WebKit graphics team busy for the whole cycle: the new Skia-based compositor. The replacement of Cairo with Skia for content rendering has been a success and it’s already well integrated and optimized. We thought we could try to use Skia for the composition too and replace TextureMapper with Skia. TextureMapper was introduced in 2010 for the Qt port and later adopted by other ports. It uses the OpenGL ES API and maintains a collection of shader programs to paint different content. Nowadays TextureMapper is mostly the same code and shader programs, and it’s unmaintained and missing features. However, the performance was good and it has served us really well all these years. So, this time the goal was not to get better results in benchmarks, but to modernize the implementation, reduce the amount of code to maintain ourselves (like all shader programs) and make it easier to implement the missing features and fix existing bugs. This post is a summary of all the work we have done this cycle to implement the new Skia compositor.

SkiaCompositingLayer

The first step was adding an SkiaCompositingLayer class to replace TextureMapperLayer and adapt all the code to use one or the other depending on an environment variable. The initial implementation was based on the TextureMapper one for the things that are common like iterating the layer tree, computing transformations, etc. The way layers produced their contents didn’t change, so we were receiving textures for tiled content, video buffers, WebGL, accelerated 2D canvas, etc. SkiaCompositingLayer created a Ganesh Skia surface to draw those textures using SkCanvas::drawImageRect(). This initial implementation was enough to run the default MotionMark test suite, since it doesn’t use other composition features. Even though performance was not the goal, we had to make sure we didn’t regress. This initial implementation was neutral in MotionMark. We needed tests to implement those features and measure performance at the same time, so we decided to add a new set of tests to MotionMark, just extending the existing tests to require composition, which makes sure that filters, masks, path clipping, transformations, etc. were done by the compositor.

Filters

We first tried implementing filters using an intermediate surface like TextureMapper does. It worked, but the MotionMark score in the filters test was much worse. We realized that with Skia we could implement most of the filters without using an intermediate surface. All filter types except blur and drop shadow can be simplified to an SkColorFilter with SkImageFilter::asAColorFilter() which can be implemented without an intermediate surface, just by setting the color filter in the SkPaint we pass to SkCanvas::drawImageRect(). This not only fixed the performance regression, but also gave better results than TextureMapper, which always needs an intermediate surface.

Masks

There are two different kinds of masks: image mask, where the source mask is an image already, and clip path, where the mask is represented by a path to be clipped. In TextureMapper both are implemented the same way using intermediate surfaces. The mask is painted into a surface and then the masked layer creates an intermediate surface where its contents are first painted and then the mask contents on top using DstIn blend mode. Skia has APIs that allowed us to implement both cases in a much simpler and more efficient way. In the case of image masks, where we already have an image, we paint the mask contents once and keep it cached, and then the masked layer creates an SkShader for the image mask that is passed to SkCanvas::clipShader() without having to paint into an intermediate surface. Clip path masks are even easier, because we can just take the path we get and build an SkPath we can pass to SkCanvas::clipPath(), without having to paint the mask as an image at all or use any other intermediate surface. Once again, masks were not only easier to implement but they ended up being more performant too.

3D contexts

The implementation of 3D layer contexts is fairly independent of TextureMapper and OpenGL, so we could just take it almost as it was, using SkPath to build the clips and a few other adaptations. We could also fix existing bugs like the z-ordering that has always been broken in TextureMapper.

Blend modes

TextureMapper never supported blend modes and they were easy to implement with Skia just using the SkPaint property for it. This made several layout tests start passing.

Batched painting

After implementing all the features we were at a point in which we had the same or better performance in all tests except for three MotionMark compositing tests that were giving much worse results. Those tests use small layers and give a high result which means we end up adding a lot of layers to the scene before we start skipping frames. The root cause was the large number of layers filling the command queue of Ganesh. Skia Ganesh queues the GL drawing operations instead of sending them to the GPU right away. When the surface is flushed for whatever reason, the queued GL drawing operations are then processed and sent to the GPU. This allows Skia to apply nice optimizations like merging several tasks and reducing the amount of draw operations we end up sending to the GPU. In those tests where a lot of layers are created and painted to the compositor Skia surface the internal command queue ends up being huge too. Processing and analyzing such a long queue to optimize what we send to the GPU required more CPU work than what we save by optimizing the GL draw operations. Skia provides an API that allows us to do the batching ourselves. Since the compositor already has information to decide what operations could be merged together, we could reduce the internal queue size in many cases. We can merge SkCanvas::drawImageRect() operations as long as they share the same color filter, blend modes and sampling options. In the best case scenario we could reduce the whole internal queue to just one operation. This time the change improved the results of those tests getting them to about 93% of the TextureMapper score, but still a bit behind.

Promise images

The Skia Ganesh backend requires that an SkImage backed by a texture is created for the current thread GrDirectContext, even if it’s borrowing an existing texture. In WebKit all textures are created with a sharing GL context so that they can be accessed and destroyed from different threads with the same sharing GL context. So, for a layer whose content is an image we had to create a texture in the compositing thread to upload the pixels if the image was not accelerated, or for accelerated images get the texture identifier of the image, and then create another SkImage from the compositing thread borrowing the texture for the current GrDirectContext. The Skia Ganesh backend provides an API to create promise images, which can be created from any thread but targeting a specific thread, providing a fulfill callback that will be called on the target thread when the SkImage is first used to retrieve the wrapped texture. This way we can create the SkImage from the main thread for the compositing thread without using OpenGL at creation time. For non-accelerated images we realized we don’t need to manually create the texture and upload the pixels in the compositor, we can just pass the unaccelerated SkImage to the compositor SkCanvas and Skia will handle it internally much more efficiently than we did. And this change improved those compositing tests much further than we expected. The reason turned out to be the batching from the previous section: Skia merges the entries of an image set by comparing texture proxy pointers, and until now we were wrapping the texture in a new SkImage on every frame for every layer, so hundreds of layers drawing the very same image produced hundreds of different proxies that Skia could not merge. Passing the same SkImage every time collapses all of them into a single draw operation, which is the best case we described above. With batched painting and promise images together we could beat TextureMapper significantly.

Deferred Display Lists (DDL)

When we switched to Skia for painting, we kept the threaded rendering model, just using a separate smaller queue for GPU rendering workers. The GPU workers created their own GrDirectContext to paint the layer tiles. The resulting textures were re-wrapped in the compositing thread for the compositor GrDirectContext using fences for the proper synchronization. We knew this was not the recommended way to use Skia Ganesh from multiple threads, but with TextureMapper we had no other option. However, with the Skia compositor we can do it the recommended way by using a single GrDirectContext in the compositing thread and use Deferred Display Lists (DDL) and promise images to paint the tiles. With DDL, GPU workers no longer use GL at all and they don’t need a GrDirectContext, they paint tiles into a display list that records the GL drawing operations, but without touching GL. For image drawing operations recorded into the DDL, promise images are used too. Since this is now all CPU work we can remove the smaller GPU worker queue and use a single queue with more workers. The compositor replays the DDL into an SkSurface that is then passed to the compositor SkCanvas.

This change fixed rendering glitches on Android and was performance neutral for the whole composition suite and for most of the MotionMark tests, but in MotionMark 1.3 at 15fps it cost 29% in Suits and 14% in Leaves, while improving Images by 9%. Correctness and the other benefits of DDL made us accept those regressions.

Damage

TextureMapper already supported using damage information to optimize the painting while compositing, but it has always been disabled at run time because there were issues we never managed to fix. With the Skia compositor we decided to start from scratch and properly handle the damage information while compositing to render only the parts of the frame that actually changed. I’m not going to go into detail here because Nikolas Zimmermann has written an amazing blog post about it with all the details.

Current situation

The Skia compositor is finished and enabled by default in 2.54. Even though it was not the main goal, it performs better than TextureMapper in most of the benchmarks we run: the composition suite we added is 45% faster, and MotionMark 1.3.1 is 35% faster. The exception is MotionMark 1.3 at 15fps with GPU rendering, which comes out flat, because the Suits and Leaves tests are still about 26% and 10% behind due to the deferred display lists trade-off described above.

We are already working on fixing existing issues in composition that we never fixed in TextureMapper. In the main branch TextureMapper is now disabled by default at build time, and support will be removed soon for the GTK and WPE ports. In 2.54 it’s still a run-time decision so if you find any issue with 2.54, you can check if it’s a Skia compositor regression by trying TextureMapper with WEBKIT_USE_SKIA_FOR_COMPOSITION=0 environment variable.

Future plans

We are already working on further improvements like using promise images for all external textures we have to pass to the compositor. We will explore the possibility of using Vulkan with the Ganesh backend instead of GL and eventually try the new Graphite backend. And of course we will continue fixing any existing issues related to the compositor.

Table of Contents

1. Challenges
2. 2021
   1. DXVK Time!
   2. Measuring Performance
3. 2022
   1. Low Resolution Z (LRZ)
   2. GPU Hangs
   3. Turnip supports Vulkan 1.3
   4. Reviewing VK_EXT_fragment_density_map
4. 2023
5. 2024
   1. Android (Waste)Lands
   2. Turnip Is Vulkan 1.4 Conformant and Preemption Support
6. 2025
   1. More VR Extensions
   2. Half-Life: Alyx
   3. Are We Performant Yet?
   4. Preventing Regressions
7. 2026
   1. Performance
   2. D3D12 Woes
   3. Alyx ☆ Rare GPU Hangs
8. Present

It’s been more than 5 years since I started working on Turnip, the open source Mesa 3D driver for Adreno GPUs, with Igalia’s graphics team. Looking back, it’s amazing how much we have achieved together, and how much the driver has improved since then. Here’s my take on the evolution through those years, some of the challenges we faced and how we overcame them. Before I start, I want to thank all those great people from Igalia, Valve, and the Mesa community I’ve had the pleasure to work with.

This post is a bit one-sided as it contains my particular view of the evolution of Turnip through those years and it doesn’t touch on a lot of important features implemented, issues debugged, and improvements made by others.

Challenges

There are several overarching challenges that the Turnip driver has had and still has to this day:

• There was no hardware documentation, we had to reverse engineer everything, with the most painful part being hardware bugs that necessitate workarounds. I cannot say I became as good as I would have wanted in all that, but I’m grateful that in Turnip we had people with absolutely amazing HW reverse-engineering skills!
• Hardware wasn’t really built to run desktop games, at least at first. It became much better in A7XX generation, but there are still some bumpy parts. Running desktop games is specially important for hardware like the Steam Frame and others.

2021

Way back, when I started to contribute to the Turnip driver, we were fixing CTS tests, running “trivial”, by today’s standards, applications like “Genshin Impact”, “TauCeti Vulkan Technology Benchmark”, “3DMark”. I have a few posts describing some issues debugged back then:

• Turnips in the wild (Part 1) - Fixing “Genshin Impact”;
• Turnips in the wild (Part 2) - Fixing “Genshin Impact” and “TauCeti Benchmark”.

If you’d asked me back then about running AAA PC games, I would have nervously laughed at best =)

DXVK Time!

But soon, we found a way to at least test how PC games render on Adreno GPUs with Turnip; we weren’t able to run games on our development boards, but the driver, after a bit of massaging, had enough features to work with DXVK (Vulkan-based translation layer for Direct3D 8/9/10/11). So the solution at that time was to run the game on the PC, record Vulkan API calls with GFXReconstruct and a Vulkan profile that constrained the desktop GPU’s capabilities to those of the Adreno GPU.

A bit later I was able to “play” simple DirectX games on the development board by playing the game on a PC and in real-time replaying translated Vulkan API calls on the board. You can read more on this in “Testing Vulkan drivers with games that cannot run on the target device”.

That was a great start; we weren’t able to run games directly on our development boards, but we were now able to test them and begin to implement more features necessary for DXVK.

Measuring Performance

In parallel with trying to support DXVK, fixing issues, and reverse engineering hardware features, we found ourselves needing to better understand performance. We chose to support Mesa’s u_trace framework and integrate with Perfetto. At that time the performance measurement support in Mesa was rough, with only Freedreno (OpenGL driver for Adreno GPUs) supporting u_trace. Spoiler: as of now, most Mesa drivers either support or are in the process of merging support for u_trace/Perfetto integration.

2022

At the end of 2021 Turnip became Vulkan 1.1 conformant. We also started testing lots of single frame D3D11 captures of games we had, which uncovered plenty of new issues.

With more testing came more issues!

At that point I met our three two main adversaries over the years, in full force:

• Low-Resolution-Z (LRZ) depth optimization;
• GPU hangs, with the worst of them completely shutting down the SoC.

Low Resolution Z (LRZ)

Low-Resolution-Z is an extremely important optimization to get right in order to get reasonable performance out of our GPU (especially for VR games), but I believe it’s also one of the most complicated (at least from a software POV) depth-related hardware optimizations across various GPUs.

Play 1 / 16

Conceptually it’s relatively simple - create a low resolution depth buffer during primitive binning pre-pass, throw out primitives that completely fail LRZ tests during binning. While during tiling, prevent a lot of overdraw by testing against this already formed low resolution depth buffer. But, in practice, there are lots of implicit restrictions on what can be done without disabling LRZ (to prevent correctness issues), and being too pessimistic when disabling LRZ can lead to unacceptable performance.

GPU Hangs

GPU hangs, on the other hand, are a plague that every graphics driver developer is intimately familiar with. However, at that time there were also unrecoverable hangs, which, due to certain issues in firmware or GPU HW itself, caused the entire SoC to shut down. They were an extreme pain to debug.

I tried many methods over time to debug them:

• At first I tried Vulkan-level breadcrumbs via “Graphics Flight Recorder”, it was useful at that time but far from perfect and abandoned by Google;
• Later, I implemented driver-level breadcrumbs that allowed finding the source of hangs at a finer level, and more importantly, synchronously going through breadcrumbs to debug unrecoverable hangs;
• Prototyped editing a captured submission to the GPU.

That made debugging somewhat bearable.

Turnip supports Vulkan 1.3

Meanwhile Turnip gained Vulkan 1.3 support! It was necessary for DXVK and VKD3D-Proton.

Reviewing VK_EXT_fragment_density_map

The third Harbinger of the Apocalypse has arrived: VK_EXT_fragment_density_map, the first stepping stone of the extensions essential for VR, was implemented by Connor Abbott. Since then I have been on the hook reviewing increasingly complicated interactions between VK_EXT_fragment_density_map and the growing host of extensions.

The fragment density map (Foveated Rendering) explanation can be found in the following blog posts from Qualcomm and Meta:

• Eye Tracked Foveated Rendering
• Improving Foveated Rendering with the Fragment Density Map Offset Extension for Vulkan

2023

Turnip began to support Adreno 7XX GPUs; previously we only supported the single 6XX generation, albeit with several sub-generations. The proprietary driver supported Vulkan since the 4XX generation, but it was only Vulkan 1.0, and 4XX/5XX generations were not powerful enough and didn’t have enough features to support anything with Vulkan.

New generation means more issues to fix!

To help with that I implemented:

• rddecompiler which makes it possible to decompile captured raw submissions to the GPU into editable C code, coupled with the ability to replay them, print from shaders, and print from the command stream - resulted in a much faster debug loop;
• A debug option that helps find where we use stale register values TU_DEBUG_STALE_REGS_RANGE.

/* pkt4: GRAS_SC_SCREEN_SCISSOR[0].TL = { X = 0 | Y = 0 } */
pkt4(cs, REG_A6XX_GRAS_SC_SCREEN_SCISSOR_TL(0), (2), 0);
/* pkt4: GRAS_SC_SCREEN_SCISSOR[0].BR = { X = 32767 | Y = 32767 } */
pkt(cs, 2147450879);
/* pkt4: VFD_INDEX_OFFSET = 0 */
pkt4(cs, REG_A6XX_VFD_INDEX_OFFSET, (2), 0);
/* pkt4: VFD_INSTANCE_START_OFFSET = 0 */
pkt(cs, 0);
/* pkt4: SP_FS_OUTPUT[0].REG = { REGID = r0.x } */
pkt4(cs, REG_A6XX_SP_FS_OUTPUT_REG(0), (1), 0);

After a lot of command stream and shader reverse-engineering - at the end of the year the Adreno 7XX generation was in decent shape in Turnip.

2024

Work continued with Adreno 750 now being the main target. We had a lot of issues to debug and fix:

And many, many more games.

Android (Waste)Lands

While Turnip, as far as I can remember, was officially only used on some Google Chromebooks, there apparently was a dedicated community of people who tried and actually ran desktop games on Android via Termux. The community has grown since then, but it was great to have a real user already using the driver to play games and having better outcomes than with the proprietary driver, which often doesn’t get updated after a phone’s release.

While fascinating, those Android setups were hard to debug, so we used them only a few times.

Turnip Is Vulkan 1.4 Conformant and Preemption Support

Not my achievements at all, but two major milestones for Turnip were:

• We caught up with the Vulkan releases and were day 1 conformant to Vulkan 1.4 on A7XX.
• Preemption support - this is a crucial feature for VR and needed by the Steam Frame. The VR compositor has to be able to preempt games that are being rendered at that moment, otherwise we can miss a frame or several, which feels extremely bad for a user moving their head in the VR environment. An Adreno GPU can be preempted at two boundaries: at the drawcall/dispatch boundary in direct (sysmem) rendering, and at the tile boundary in tiling (gmem) rendering.

2025

More VR Extensions

Even more extensions came in from Connor: VK_QCOM_multiview_per_view_viewports, VK_QCOM_multiview_per_view_render_areas, VK_VALVE_fragment_density_map_layered, VK_QCOM_fragment_density_map_offset, VK_QCOM_subpass_shader_resolve, VK_EXT_custom_resolve. Those are essential for pushing VR performance to its limit. The biggest issue with FDM related extensions is that they don’t have good CTS tests; the complexity comes from the fact that a conforming driver implementation may simply choose not to reduce quality in regions specified by the density map. Even the size of those regions is not known to a game using those extensions!

As a result I wrote lots of tests that exercise various combinations of those extensions and require visual inspection for them to pass 🫠🫠🫠.

Those tests also measured performance, which helped us fix disparities with the proprietary driver.

Half-Life: Alyx

What is going to use the above extensions to push Steam Frame to its limits? Of course, Half-Life: Alyx.

Even with previously mentioned tests in place, Half-Life: Alyx found plenty of both rendering and performance issues.

Are We Performant Yet?

We already had feedback from Android users running games that Turnip performance is sometimes better than the proprietary driver, but sometimes noticeably worse. But how to compare them? Turnip has Perfetto support, the proprietary driver has Snapdragon Profiler, but it’s hard to use and even then - we’d just see that some particular renderpass is faster or not. There could be hundreds or thousands of draw calls in a renderpass!

Staring at the command stream from the proprietary driver stopped yielding any insights, so the next step was to take a game trace that can run both on Turnip and Qualcomm’s driver and compare them draw by draw, measuring all performance counters along the way. It was done by lots of command stream patching, but the end result was an ability to compare key registers at every draw and every performance counter in existence.

The resulting table was huge:

This helped us to close several gaps in performance, however the most useful part of the comparison was not the counters, but the register comparison. The counters, aside from execution time and LRZ stats, were surprisingly hard to convert into any useful insight.

Preventing Regressions

With the driver gaining capabilities, but not gaining many more users, we needed a way to prevent regressions while introducing more and more complex features. And while Vulkan Conformance Test Suite is being tirelessly improved upon year after year, it’s still far from enough.

It was time to introduce a CI system that would be able to detect visual regressions in rendering and performance regressions. We already had a number of d3d11 and d3d9 captures to start working with, and so it came to be:

Above, you can see one of the results where the rendering regressed. In most cases we run only a single frame capture instead of longer multi-frame traces. This was an explicit choice due to the observation that it’s better to have a wider selection of captures, than trying to cover any single game better (VR games are an exception here). In many cases only one or two captures out of hundreds regressed due to some issue; the regression was caused by some very specific pattern the game had, which wouldn’t appear in others.

Every night we test several driver configurations:

• Fixed DXVK/VKD3D versions + forced direct (sysmem) rendering;
• Fixed DXVK/VKD3D versions + forced tiling (gmem) rendering;
• Upstream DXVK/VKD3D versions;
• Turnip compiled with ubsan (undefined behaviour sanitizer).

We also run important MRs through that CI to find regressions early on.

Every nightly run generates a performance datapoint, so we can see the line going down day by day. Don’t mind the bump where we had concurrent binning “optimization” enabled 🫠 (that’s a sad tale of an incredibly complicated HW feature which failed to deliver performance gains).

At the moment we are testing more than 600 different game frames per driver configuration; the APIs span D3D8-D3D12, Vulkan, and OpenGL. A single configuration runs in under an hour on just two Adreno 750 devices.

With CI in place we became increasingly confident in making changes to the driver.

See more in my XDC 2025 talk:

2026

Steam Frame was announced at the end of 2025; now we still had plenty of things to polish.

Performance

We were now working much more on performance, and the main culprit in bad performance is often Low-Resolution-Z being fully or partially disabled, especially in VR. We improved Perfetto tracepoints a lot: added new ones, fixed tracepoints with complex renderpass suspend-resume setups, and added performance warnings:

Now it was much nicer to work with, and something external developers could reasonably use.

As a result, while working on the performance of Half-Life: Alyx and some other VR games, we improved LRZ support a lot.

We’ve also compared Turnip against Qualcomm’s driver on a new GPU Performance Microbenchmark (gpu-ratemeter) to squash the rest of the performance differences.

There was also a lot of compiler work, done by great compiler engineers working on Turnip.

D3D12 Woes

As we have been testing more D3D12 games running through VKD3D-Proton, we started to find interesting issues. Those generally are: implicit D3D12 features/requirements, or some behaviour out of the D3D12 spec but which all desktop GPUs do, or in one case Turnip having a higher limit than desktop drivers. We saw at least:

• UE5 not working correctly with wave128 (only Adreno has such wide waves) up until a few months ago: vkd3d-proton PR #3265
• D3D12 not having a proper query/limit for number of elements in buffers, implicitly supporting more than Turnip advertises: mesa MR !41477
• One game relying on “fair” execution of dispatches when implementing its own spin locks in compute shaders: mesa MR !41562
• UE5 relying on higher memory allocation alignment than Turnip has: vkd3d-proton PR #3231
• Games not checking for D3D12_FEATURE_DATA_D3D12_OPTIONS21::ExecuteIndirectTier before using D3D12_EXECUTE_INDIRECT_TIER_1_1 commands

Alyx ☆ Rare GPU Hangs

It’s “good” when the GPU hangs at a predictable place, it’s bad when the GPU hangs randomly, and it’s even worse when the GPU stops hanging when you try to isolate the issue in any way to debug it.

One of such hangs happened in Half-Life: Alyx, when moving through a specific location the GPU hung, sometimes, and sometimes it didn’t for a long while. I’ve tried:

• At first it seemed to happen only on the stable Turnip branch, so I’ve tried to bisect the issue;
• After a while I found out that it happened in any branch;
• I tried to stare at GPU coredumps - no luck;
• Tried to get a gfxreconstruct trace to reproduce the hang; it might hang once or twice out of many replays.

What is almost impossible to do in such cases is test whether disabling a certain driver feature helps; you always have doubts - it didn’t hang this time because of the feature I disabled, or I’m just unlucky. This happened several times during the investigation, the hang would disappear for an hour, and then reproduce several times in a row.

Previously I wrote that we have a mechanism to capture and replay raw submissions that are sent to the GPU. Yes, I tried that too, we can capture the submission that hanged the GPU. Guess what? The captured submission executed absolutely normally and didn’t hang, not on the first execution, nor on thousands….

At that point I still didn’t have a single clue what’s going wrong, aside from some kind of hardware errata being involved. It was time to improve the debug tooling even further. Before, the captured .rd submission could be replayed once with one replay invocation, but the issue at hand demanded lots of iterations, and more ergonomic editing of the command stream than we previously had. So I’ve made improvements (they still are work-in-progress) to:

• Loop specific submission any number of times;
• Quickly disable specific draw call ranges in the submission;
• Automatically bisect which draw/dispatch causes hang/fault.

Only with each bisection step doing 50000 iterations was I able to narrow things down to a few draw calls. Looking at them closely still yielded only more head scratches though. However, with things narrowed down that far, I had something to poke other, more knowledgeable people with.

After some back and forth, it appeared that I hadn’t thoroughly checked all shader debug options we had, or rather, I checked the option, but due to the rarity of the hang I misidentified it as not helpful!

In the end it appeared that there were two hardware errata that needed to be implemented in our shader compiler. And we got “lucky” that they were revealed by one of the most important games to run on the headset.

Present

Driver work never ends, there are still games to debug, features to implement, and VR performance to improve. The fact that Steam Frame runs Linux, is based on open-source software, and isn’t locked down means that Steam Frame would be used in many ways unforeseen by us. I hope that Steam Frame release would bring improvements to the VR ecosystem and spearhead PC gaming on Linux running on ARM platforms.

Update on what happened in WebKit in the week from September 7 to September 14.

In this week's edition, the PNG image decoder received a much needed cleanup, as well as another swipe of unassorted graphics improvements, most notably in the Layer-Based SVG Engine. Finally, we also have an exciting and informative article about how to use the WPE Platform API on Raspberry Pi.

Cross-Port 🐱

Graphics 🖼️

Community & Events 🤝

• Bug: 318938 – Support touch event asynchronous scrolling
• Commit: 319347@main

The WPE and GTK ports have supported touch events for a long time, but asynchronous scrolling only worked for wheel events. Scrolling driven by touch still depended on the web process main thread. This change puts touch events on the asynchronous scrolling path too.

Let’s start with the background: what asynchronous scrolling is and why it needs to be built differently here.

What is asynchronous scrolling? #

Why it is needed #

A naive implementation of scrolling looks like this:

1. The UI process receives an input event (wheel or touch).
2. It sends the event to the web process main thread.
3. The main thread runs the page’s JavaScript event listeners.
4. If nothing called preventDefault(), the scroll position is updated.
5. The page is rendered at the new scroll position.

Step 3 is the problem. The main thread is easily blocked for hundreds of milliseconds by JavaScript execution or layout, and scrolling is frozen for that whole time. Your finger moves, the screen does not. That is what synchronous scrolling feels like.

Asynchronous scrolling moves the work off the main thread: the scrolling thread updates the scroll position, and the compositor thread composites and presents the frame. Neither needs the main thread, so scrolling keeps running at 60fps even when the main thread is busy.

The scrolling tree and event regions #

Two data structures make this possible.

The scrolling tree is a tree of the scrollable areas of a page. The main frame, overflow: scroll elements, position: fixed/sticky elements and so on each become a node holding its own scroll position and a reference to its layer. The scrolling thread updates scroll positions by looking only at this tree, and the compositor thread then draws the layers at their new positions — the main thread is not involved in either step.

But there are places where scrolling on its own would be wrong, because the page might call preventDefault() from an addEventListener("touchstart", ...) handler. That is what event regions are for.

An event region records, per layer and at rendering time, “this rectangle has a listener for this kind of event”. EventRegion keeps separate regions for touchstart, touchmove, pointerdown, mousedown and friends, and for any given point it yields a TrackingType:

enum class TrackingType : uint8_t {
    NotTracking = 0,   // No listener. The event does not even need to be delivered.
    Asynchronous = 1,  // Passive listeners only. Scroll now, notify the page later.
    Synchronous = 2    // A non-passive listener may call preventDefault(). We must wait.
};

The passive distinction is what makes Asynchronous possible. preventDefault() cancels an event only if the listener was registered with passive: false; from a passive listener it does nothing. And on window, document and document.body, touchstart/touchmove (and wheel) default to passive: true — see MDN: Using passive listeners. So Asynchronous is the case “there are listeners, but none of them can cancel the scroll”: start scrolling now, deliver the event to the main thread afterwards.

Because this information travels to the scrolling thread along with the layer tree, an incoming input event can be classified without waking the main thread. That is the heart of asynchronous scrolling.

Why the iOS implementation could not be reused #

The iOS port already implements asynchronous scrolling for touch events, but it could not be reused, because the process layout is different.

iOS:
  UI process:  platform layer tree + scrolling tree + touch event input
  Web process: main thread (DOM, layout)

WPE / GTK (Coordinated Graphics):
  UI process:  touch event input only
  Web process: main thread (DOM, layout)
               + EventDispatcher thread / scrolling thread
               + platform layer tree + scrolling tree

On iOS both the platform layer tree and the scrolling tree live in the UI process — the very process that receives touch events — so the classification and the scroll both happen right there. On WPE and GTK we use Coordinated Graphics, and both trees live in the web process instead. The classification therefore has to happen after sending the event to the web process, but before touching the main thread.

Fortunately the same problem was already solved for wheel events. The web process has an EventDispatcher thread that receives wheel events from the UI process without going through the main thread and consults the scrolling tree directly. This change builds the same shape for touch events.

How a touch becomes a scroll in WPE #

One more piece of background: in WPE a touch does not scroll the page directly. Touch events are first offered to the page; only if the page does not consume them does the UI process turn the touch sequence into scrolling.

The decision point is PageClientImpl::doneWithTouchEvent(). If the page handled the event, gesture detection is cancelled with wpe_gesture_controller_cancel() so the engine does not also act on it. If it was not handled, the event is fed to the WPE platform gesture controller via ViewPlatform::handleGesture(), and a recognized WPE_GESTURE_DRAG is turned into a synthetic scroll event pushed back into the page as a wheel event:

GRefPtr<WPEEvent> simulatedScrollEvent = adoptGRef(wpe_event_scroll_new(
    m_wpeView.get(), WPE_INPUT_SOURCE_TOUCHSCREEN, 0, static_cast<WPEModifiers>(0), dx, dy, TRUE, FALSE, x, y));
page().handleNativeWheelEvent(WebKit::NativeWebWheelEvent::create(simulatedScrollEvent.get(), phase));

That TRUE is precise_deltas: touch-driven scrolling in WPE reaches the engine as precise-delta wheel events, which becomes relevant later.

The important consequence is this: the UI process cannot start scrolling until it knows whether the page is going to consume the touch. That answer used to come from the web process main thread — so when the main thread was busy, scrolling did not start. That is the problem this change fixes.

The change #

Enabling touch event regions #

A new ENABLE(COORDINATED_TOUCH_EVENTS) is introduced in PlatformEnableGlib.h, and it turns on ENABLE(TOUCH_EVENT_REGIONS) whenever touch events are enabled on WPE/GTK. The AlwaysUseTouchEventRegions preference now defaults to true under that flag, so Document::shouldUseTouchEventRegions() returns true and touch regions are actually recorded on the layers during rendering.

UI process: send to the EventDispatcher instead of the main thread #

The old WebPageProxy::handleTouchEvent() consulted a touchEventTracking state kept in the UI process and sent Messages::WebPage::TouchEvent, i.e. straight to the web process main thread.

The new version delegates all classification to the web process and only queues events and delivers answers. One event is in flight at a time; the next is sent when the reply arrives. The flood of touchmove events produced while a finger moves is coalesced into the newest queued event when that is also a touchmove, and the coalesced events are flushed to doneWithTouchEvent() together with the reply.

The destination is now Messages::EventDispatcher::TouchEvent.

Web process: classification on the EventDispatcher thread #

EventDispatcher::touchEvent() runs on the EventDispatcher thread, where it looks up the page’s scrolling tree, asks it for a TrackingType, and splits three ways:

• NotTracking — no listeners. Reply handled = false immediately, without bothering the main thread at all. The UI process can start scrolling right away.
• Asynchronous — passive listeners only, so nothing can cancel the event. Reply handled = false first so scrolling starts, then deliver the event to the main thread.
• Synchronous — a non-passive listener may call preventDefault(), so wait for the main thread result as before.

Replying without a main thread round trip is possible because the new TouchEvent message is declared AnyThread in EventDispatcher.messages.in. The iOS equivalent is MainThreadCallback, which always replies from the main thread.

If there is no scrolling tree for the page yet, the event goes to the main thread as before.

Classifying a touch in the scrolling tree #

The classification itself is ScrollingTreeCoordinated::eventTrackingTypeForTouchEvent(). It works in two stages.

First, for each newly pressed touch point: convert the point from view to contents coordinates, hit test the layer tree down from the root contents layer, take the frontmost layer whose event region contains the point, and query that region. It is queried for many event types, because a touch fires more than the DOM touch* events — pointer*, compatibility mouse events and gesture* too, and a non-passive listener for any of them forces synchronous handling. The results are folded into a small TouchEventTracking struct with four fields: start, move, end and force-change.

Second, the tracking type of the event as a whole is derived from the touch point states, merging the per-field values. Merging picks the stronger of two types (NotTracking < Asynchronous < Synchronous), so if any single point needs synchronous handling, the whole event is synchronous.

TouchEventTracking persists for the lifetime of a touch sequence and is reset once all points are released, so the hit test done at touchstart is reused for the following touchmove/touchend. That guarantees a sequence never flips from synchronous to asynchronous halfway through just because a finger moved off a listener’s area.

<input type=range> #

A slider handles touches internally even with no JavaScript listener, so looking at the event region alone would classify it as NotTracking. HTMLInputElement::updateTouchEventHandler() now sets the HasInternalTouchEventHandling flag on EventTarget for range inputs, and StyleAdjuster turns that flag into the full set of touch region types for the element.

Keeping the animation running on the scrolling thread #

The last piece is in ScrollingEffectsController::handleWheelEvent(). As shown above, WPE synthesizes wheel events from touch gestures with precise deltas. Precise-delta events only need immediateScrollBy() to move the scroll position — but then nothing drives screen updates while the main thread is busy.

The fix is that, while a scroll gesture is in progress, a scroll animation is also started — from one ULP short of the destination (std::nextafter()) to the destination. Visually it finishes instantly — the real scroll is still done by immediateScrollBy() — but a scroll animation is now running, which starts display link monitoring and keeps compositing driven regardless of the main thread.

The event flow, summarized #

Before:

After (no listeners, or passive listeners only):

Where a non-passive listener exists, we still wait for the main thread as before. The spec requires preventDefault() to be honoured, so that is unavoidable.

Layout test updates #

As a side effect, the tests under fast/events/touch/ had to be updated.

Event regions are computed during a rendering update and propagated to the scrolling tree via the platform layer tree. Which means a test like this:

target.addEventListener("touchstart", handler);
tapSoon(20, 20);   // ← the region has not been updated yet!

taps immediately after registering the listener, while the scrolling tree still believes there is no listener and returns NotTracking. The event never reaches the main thread and the test fails.

A new UIHelper.renderingComplete() was added for this:

static async renderingComplete()
{
    // Wait for the platform layer tree to be updated
    await UIHelper.animationFrame();
    await UIHelper.animationFrame();
}

Two animation frames are needed because the first one runs the rendering update that computes the regions, and a second is needed for the result to reach the layer tree.

Summary #

• On WPE and GTK both the layer tree and the scrolling tree live in the web process (Coordinated Graphics), so the iOS touch asynchronous scrolling implementation could not be reused directly.
• Instead, touch events were given the same shape that already works for wheel events: ask the scrolling tree from the EventDispatcher thread.
• The keys were enabling touch event regions, and making the IPC reply AnyThread so it can be sent without waiting for the main thread.
• Anywhere the page has no non-passive listener, scrolling now starts regardless of what the main thread is doing.

Acknowledgements #

Many thanks to Alejandro G. Castro and Carlos Garcia Campos for their insightful reviews of this work, and to Claude for writing this blog post.

It has been a while since I wrote a post here. The last one was from December 2012, almost fourteen years, which is a bit embarrassing for someone who is still hacking on WebKit. I need to try to fix it, and this post is the first attempt.

wpe-android started in 2017 as an experiment of running WPE WebKit on Android. The engine communicated with the system through libwpe, the abstraction library WPE used before WPEPlatform. It used an Android backend living in its own separate library, libWPEBackend-android, and the Java side exposed the API with the WPEView, WPEContext and the other proxy classes. It was a useful experiment. It was enough to show the engine running in a phone. The problem is a good part of the Android specific code, in particular the rendering path sending buffers to a SurfaceControl, was placed in the application side instead of in the platform layer of the engine. Also Android developers had to learn an API that did not look like anything they already had in the platform, it was close to the Linux WPE API.

Since then we have been implementing WPEPlatform, a new platform abstraction that has been added inside WebKit. It gave us the chance to fix both problems at the same time. During this spring we implemented a proper Android backend on top of it, so now the display system of Android looks like a regular WPE platform to the engine. WPEDisplayAndroid registers as a display extension point and sets up an AHardwareBuffer configuration, WPEToplevelAndroid follows the ANativeWindow lifecycle from the SurfaceHolder callbacks, WPEViewAndroid submits the frames with SurfaceTransaction and explicit sync fences, and WPEScreenSyncObserverAndroid uses AChoreographer so the engine renders aligned with vsync. On top of that we added a thin JNI bridge that mirrors the WebKit and WPEPlatform objects in Java, and a new convenience API that looks like android.webkit.WebView, with WebView, WebContext, WebSettings, CookieManager, WebViewClient and WebChromeClient. This way people can reuse the Android documentation and the knowledge they already have instead of learning something new.

We ported the tools used to the new API, the MiniBrowser, the media player sample, the instrumented tests and WebDriver. We were able to delete the legacy stack.

After that, the only thing keeping libwpe in the build was process management: Android cannot simply fork() and exec(), since auxiliary processes are Android services that need to be launched explicitly. To solve this, we implemented a WPEProcessManager API for WPEPlatform inside WebKit, which landed upstream and is now used by the port.

With that in place, we were able to remove libwpe entirely, and WPE WebKit now talks to Android the same way it talks to any other platform. In parallel, we kept improving other parts of the platform, implementing new API features and fixing bugs along the way. One notable example is a scroll jank issue on the Mali driver, which was resolved largely thanks to the Skia DDL architecture upstream (a story that deserves its own write-up).

There is still plenty work to do, for our small team this project is still a very valuable experiment and we would like to keep exploring the options.

The code is here if you want to give it a try or help:

https://github.com/Igalia/wpe-android

I will try to write about our work around WebKit and wpe-android in more detail in the next weeks, hopefully the next post will not take another fourteen years …

WPE Platform API (also known as “new API”) is a redesigned, GObject-based platform-integration layer for WPE WebKit that replaces the older libwpe backend model. A few months ago, Kate and Simon published two closely related blog posts about it. The first one focuses more on the API and browser implementation details, while the second one focuses more on writing and integrating the browser within Linux distribution.

This article builds on top of the above ones, and showcases how to build and try a minimal WPE browser using WPE Platform API on Raspberry Pi. Moreover, as the WPE Platform API still evolves to some degree, this article also explains how to use and stick to the latest WPE WebKit from main branch. This way one can play with all the latest features straight on embedded hardware.

Before going further, one should be aware that in case of a simple release build (instead of one using latest main branch) it’s better to follow official instructions instead of this article.

Setup #

This article focuses on a certain setup using Raspberry Pi 3B but it should be fairly easy to adapt the config to any other Raspberry Pi model.

As for the work environment: the Linux-based host with ability to run containers was used along with WebKit Container SDK. The SDK version was precisely 2.53-v6-d535e88 as it uses Ubuntu 24.04.4 LTS that works well with Yocto scarthgap.

The Yocto scarthgap has been used to increase the chances that the config and commands demonstrated in this article will remain buildable for many years to follow.

Preparing the image #

The preparation of the image starts with a series of commands that create a main directory and clone important Yocto repositories along with some meta layer repositories. At this point already, it’s important to have the working directory shared between host and SDK.

# host
mkdir wpe-upstream
cd wpe-upstream
git clone https://git.yoctoproject.org/git/poky -b scarthgap
git clone git@github.com:openembedded/meta-openembedded.git -b scarthgap
git clone https://git.yoctoproject.org/git/meta-raspberrypi -b scarthgap
git clone https://github.com/Igalia/meta-webkit -b scarthgap
source poky/oe-init-build-env build

Once the build directory is created, it’s necessary to configure the meta layers in the build/conf/bblayers.conf file the following way:

# POKY_BBLAYERS_CONF_VERSION is increased each time build/conf/bblayers.conf
# changes incompatibly
POKY_BBLAYERS_CONF_VERSION = "2"

BBPATH = "${TOPDIR}"
BSPDIR := "${@os.path.abspath(os.path.dirname(d.getVar('FILE', True)) + '/../..')}"

BBFILES ?= ""
BBLAYERS ?= " \
  ${BSPDIR}/poky/meta \
  ${BSPDIR}/poky/meta-poky \
  ${BSPDIR}/poky/meta-yocto-bsp \
  ${BSPDIR}/meta-openembedded/meta-oe \
  ${BSPDIR}/meta-openembedded/meta-multimedia \
  ${BSPDIR}/meta-openembedded/meta-python \
  ${BSPDIR}/meta-raspberrypi \
  ${BSPDIR}/meta-webkit \
  "

With the above, the recipes from the meta layers cloned earlier will be considered by bitbake.

Next, the most important configuration step is appending the following to build/conf/local.conf:

MACHINE = "raspberrypi3-64" 
MACHINE_FEATURES:append = " vc4graphics"
GPU_MEM_256 = "128"
GPU_MEM_512 = "196"
GPU_MEM_1024 = "396"
DISTRO_FEATURES:append = " opengl egl wayland"
EXTRA_IMAGE_FEATURES = "debug-tweaks"
IMAGE_FEATURES:append = " ssh-server-dropbear hwcodecs"
IMAGE_INSTALL:append = " wpewebkit wpe-browser"
PREFERRED_VERSION_wpewebkit = "latest"
LICENSE_FLAGS_ACCEPTED = "synaptics-killswitch"

With that, wpewebkit latest will be preferred and installed in the image along with a dummy browser called wpe-browser.

To make the wpewebkit latest work, one needs to create meta-webkit/recipes-browser/wpewebkit/wpewebkit_latest.bb:

SUMMARY = "Lightweight WebKit port for embedded devices with OpenGL-ES acceleration"
DESCRIPTION = "WPE WebKit port pairs the WebKit engine with OpenGL-ES (OpenGL for Embedded Systems), \
               allowing embedders to create simple and performant systems based on Web platform technologies. \
               It is designed with hardware acceleration in mind, relying on EGL, and OpenGL ES."
HOMEPAGE = "https://wpewebkit.org/"
BUGTRACKER = "https://bugs.webkit.org/"
LICENSE = "BSD-2-Clause & LGPL-2.0-or-later"
LIC_FILES_CHKSUM = "file://Source/WebCore/LICENSE-LGPL-2.1;md5=a778a33ef338abbaf8b8a7c36b6eec80 "

REQUIRED_DISTRO_FEATURES = "opengl"

DEPENDS:append = " \
    libsoup \
    bison-native gperf-native harfbuzz-native libxml2-native ccache-native ninja-native ruby-native \
    fontconfig freetype glib-2.0 harfbuzz icu jpeg pcre sqlite3 zlib libpng libtasn1 \
    libwebp libxml2 libxslt virtual/egl virtual/libgles2 libepoxy libgcrypt \
    unifdef-native \
"

inherit cmake features_check pkgconfig perlnative python3native

export WK_USE_CCACHE = "NO"

PACKAGECONFIG ??= "accessibility avif dfg-jit gbm gpu-process \
                   jit jpegxl libbacktrace \
                   mediasource mediastream \
                   remote-inspector \
                   sysprof \
                   ${@' system-sysprof' \
                       if bb.utils.contains('BBFILE_COLLECTIONS', 'meta-gnome', True, False, d) \
                       else '' } \
                   unified-builds video webaudio woff2 wpe-platform \
                   ${@bb.utils.contains('DISTRO_FEATURES', 'systemd', 'journald', '' ,d)} \
                  "

PACKAGECONFIG[reduce-size] = "-DCMAKE_BUILD_TYPE=MinSizeRel,-DCMAKE_BUILD_TYPE=Release,,"
PACKAGECONFIG[release-with-debug-info] = "-DCMAKE_BUILD_TYPE=RelWithDebInfo,-DCMAKE_BUILD_TYPE=Release,,"

# WPE features
PACKAGECONFIG[accessibility] = "-DUSE_ATK=ON,-DUSE_ATK=OFF,atk at-spi2-atk"
PACKAGECONFIG[avif] = "-DUSE_AVIF=ON,-DUSE_AVIF=OFF,libavif"
PACKAGECONFIG[bubblewrap] = "-DENABLE_BUBBLEWRAP_SANDBOX=ON -DBWRAP_EXECUTABLE=${bindir}/bwrap -DDBUS_PROXY_EXECUTABLE=${bindir}/xdg-dbus-proxy,-DENABLE_BUBBLEWRAP_SANDBOX=OFF,bubblewrap xdg-dbus-proxy libseccomp"
PACKAGECONFIG[developer-mode] = "-DDEVELOPER_MODE=ON,-DDEVELOPER_MODE=OFF,wayland-native wayland-protocols wpebackend-fdo"
PACKAGECONFIG[deviceorientation] = "-DENABLE_DEVICE_ORIENTATION=ON,-DENABLE_DEVICE_ORIENTATION=OFF,"
PACKAGECONFIG[dfg-jit] = "-DENABLE_DFG_JIT=ON,-DENABLE_DFG_JIT=OFF,"
PACKAGECONFIG[documentation] = "-DENABLE_DOCUMENTATION=ON,-DENABLE_DOCUMENTATION=OFF, gi-docgen-native gi-docgen"
PACKAGECONFIG[encryptedmedia] = "-DENABLE_ENCRYPTED_MEDIA=ON,-DENABLE_ENCRYPTED_MEDIA=OFF,libgcrypt"
PACKAGECONFIG[experimental-features] = "-DENABLE_EXPERIMENTAL_FEATURES=ON,-DENABLE_EXPERIMENTAL_FEATURES=OFF,libavif libjxl"
PACKAGECONFIG[gamepad] = "-DENABLE_GAMEPAD=ON,-DENABLE_GAMEPAD=OFF,libmanette"
PACKAGECONFIG[gbm] = "-DUSE_GBM=ON,-DUSE_GBM=OFF,libdrm"
PACKAGECONFIG[geolocation] = "-DENABLE_GEOLOCATION=ON,-DENABLE_GEOLOCATION=OFF,geoclue"
PACKAGECONFIG[gpu-process] = "-DENABLE_GPU_PROCESS=ON,-DENABLE_GPU_PROCESS=OFF,"
PACKAGECONFIG[hyphen] = "-DUSE_LIBHYPHEN=ON,-DUSE_LIBHYPHEN=OFF,hyphen"
PACKAGECONFIG[introspection] = "-DENABLE_INTROSPECTION=ON,-DENABLE_INTROSPECTION=OFF, gobject-introspection-native"
PACKAGECONFIG[jit] = "-DENABLE_JIT=ON -DENABLE_C_LOOP=OFF,-DENABLE_JIT=OFF -DENABLE_C_LOOP=ON,"
PACKAGECONFIG[jpegxl] = "-DUSE_JPEGXL=ON,-DUSE_JPEGXL=OFF,libjxl"
PACKAGECONFIG[journald] = "-DENABLE_JOURNALD_LOG=ON,-DENABLE_JOURNALD_LOG=OFF,"
PACKAGECONFIG[lcms] = "-DUSE_LCMS=ON,-DUSE_LCMS=OFF,"
PACKAGECONFIG[spellcheck] = "-DENABLE_SPELLCHECK=ON,-DENABLE_SPELLCHECK=OFF,enchant"
PACKAGECONFIG[wpe-legacy-api] = "-DENABLE_WPE_LEGACY_API=ON,-DENABLE_WPE_LEGACY_API=OFF,libwpe virtual/wpebackend,"
PACKAGECONFIG[libbacktrace] = "-DUSE_LIBBACKTRACE=ON,-DUSE_LIBBACKTRACE=OFF,libbacktrace"
PACKAGECONFIG[minibrowser] = "-DENABLE_MINIBROWSER=ON,-DENABLE_MINIBROWSER=OFF,wayland-native wayland-protocols wpebackend-fdo"
PACKAGECONFIG[mediasource] = "-DENABLE_MEDIA_SOURCE=ON,-DENABLE_MEDIA_SOURCE=OFF,gstreamer1.0 gstreamer1.0-plugins-good"
PACKAGECONFIG[mediastream] = "-DENABLE_MEDIA_STREAM=ON,-DENABLE_MEDIA_STREAM=OFF,gstreamer1.0 gstreamer1.0-plugins-bad"
PACKAGECONFIG[pdfjs] = "-DENABLE_PDFJS=ON,-DENABLE_PDFJS=OFF,"

PACKAGECONFIG[speech-synthesis] = "-DENABLE_SPEECH_SYNTHESIS=ON,-DENABLE_SPEECH_SYNTHESIS=OFF,flite"

PACKAGECONFIG[sysprof] = "-DUSE_SYSPROF_CAPTURE=ON, -DUSE_SYSPROF_CAPTURE=OFF,"
PACKAGECONFIG[system-sysprof] = "-DUSE_SYSTEM_SYSPROF_CAPTURE=ON, -DUSE_SYSTEM_SYSPROF_CAPTURE=OFF, sysprof"
PACKAGECONFIG[video] = "-DENABLE_VIDEO=ON,-DENABLE_VIDEO=OFF,gstreamer1.0 gstreamer1.0-plugins-base"
PACKAGECONFIG[webaudio] = "-DENABLE_WEB_AUDIO=ON,-DENABLE_WEB_AUDIO=OFF,gstreamer1.0 gstreamer1.0-plugins-base gstreamer1.0-plugins-good"
PACKAGECONFIG[woff2] = "-DUSE_WOFF2=ON,-DUSE_WOFF2=OFF,woff2"
PACKAGECONFIG[remote-inspector] = "-DENABLE_REMOTE_INSPECTOR=ON,-DENABLE_REMOTE_INSPECTOR=OFF,"
PACKAGECONFIG[webrtc] = "-DENABLE_WEB_RTC=ON,-DENABLE_WEB_RTC=OFF,libvpx libevent libopus openh264"
PACKAGECONFIG[qtwpe] = "-DENABLE_WPE_QT_API=ON ${CMAKE_QT_OECONF},-DENABLE_WPE_QT_API=OFF,qtbase-native qtbase qtdeclarative libepoxy wpebackend-fdo ${QT_BUILD_DEPS}"
PACKAGECONFIG[unified-builds] = "-DENABLE_UNIFIED_BUILDS=ON,-DENABLE_UNIFIED_BUILDS=OFF,"
PACKAGECONFIG[thunder] = "-DENABLE_THUNDER=ON,-DENABLE_THUNDER=OFF,virtual/open-cdm"
PACKAGECONFIG[webxr] = "-DENABLE_WEBXR=ON,-DENABLE_WEBXR=OFF,openxr"

# Build option for WPE API 1.1
PACKAGECONFIG[wpe-1-1-api] = "-DENABLE_WPE_1_1_API:BOOL=ON,-DENABLE_WPE_1_1_API:BOOL=OFF,"

# Build option for WPE platform API
PACKAGECONFIG[wpe-platform] = "-DENABLE_WPE_PLATFORM=ON,-DENABLE_WPE_PLATFORM=OFF,libinput libxkbcommon wayland-native"

EXTRA_OECMAKE = " -DPORT=WPE -G Ninja"

# TODO: documentation and introspection are disabled by default because the are
# causing cross-compiling build errors
# PACKAGECONFIG:append = " ${@bb.utils.contains('DISTRO_FEATURES', 'api-documentation', 'documentation', '' ,d)} introspection"

# If SSE code compiles, assume it runs successfully (it can't actually run
# because of cross compiling)
EXTRA_OECMAKE:append:x86 = " -DHAVE_SSE2_EXTENSIONS_EXITCODE=0"
# Javascript JIT is not supported on ppc/arm/RISCV32/mips64
PACKAGECONFIG:remove:powerpc = "jit"
PACKAGECONFIG:remove:powerpc64 = "jit"
PACKAGECONFIG:remove:powerpc64le = "jit"
PACKAGECONFIG:remove:armv4 = "jit"
PACKAGECONFIG:remove:armv5 = "jit"
PACKAGECONFIG:remove:armv6 = "jit"
PACKAGECONFIG:remove:armv7a = "jit"
PACKAGECONFIG:remove:armv7ve = "jit"
PACKAGECONFIG:remove:riscv32 = "jit"
PACKAGECONFIG:remove:riscv64 = "jit"
PACKAGECONFIG:remove:mipsarchn64 = "jit"
PACKAGECONFIG:remove:mipsarchn32 = "jit"
PACKAGECONFIG:remove:loongarch64 = "jit"

# Javascript JIT is not supported on x86
PACKAGECONFIG:remove:x86 = "jit"

LDFLAGS:append:riscv64 = " -pthread"

FULL_OPTIMIZATION:remove = "-g"

LEAD_SONAME = "libWPEWebKit.so"
PACKAGES =+ "${PN}-web-inspector-plugin ${PN}-qtwpe-qml-plugin"
FILES:${PN} += "${libdir}/wpe-webkit*/injected-bundle/libWPEInjectedBundle.so"
FILES:${PN}-web-inspector-plugin += "${datadir}/wpe-webkit-*/inspector.gresource"
# nooelint: oelint.vars.insaneskip - ignored for convenience. We need to recheck if problem persist
INSANE_SKIP:${PN}-web-inspector-plugin = "dev-so"

# nooelint: oelint.vars.insaneskip - ignored for convenience. We need to recheck if problem persist
INSANE_SKIP:${PN}-qtwpe-qml-plugin = "dev-so"

# JSC JIT on ARMv7 is better supported with Thumb2 instruction set.
ARM_INSTRUCTION_SET:armv7a = "thumb"
ARM_INSTRUCTION_SET:armv7r = "thumb"
ARM_INSTRUCTION_SET:armv7m = "thumb"
ARM_INSTRUCTION_SET:armv7ve = "thumb"

# Extra runtime depends
# nooelint: oelint.vars.dependsordered - ignored for convenience
RDEPENDS:${PN} += "\
    ${@bb.utils.contains('PACKAGECONFIG', 'remote-inspector', '${PN}-web-inspector-plugin', '', d)} \
    ${@bb.utils.contains('PACKAGECONFIG', 'gst_gl', 'gstreamer1.0-plugins-base-opengl', '', d)} \
    ${@bb.utils.contains('PACKAGECONFIG', 'mediasource', 'gstreamer1.0-plugins-good-isomp4', '', d)} \
    ${@bb.utils.contains('PACKAGECONFIG', 'webaudio', 'gstreamer1.0-plugins-good-wavparse', '', d)} \
    ${@bb.utils.contains('PACKAGECONFIG', 'video', 'gstreamer1.0-plugins-base-app \
                                                    gstreamer1.0-plugins-base-audioconvert \
                                                    gstreamer1.0-plugins-base-audioresample \
                                                    gstreamer1.0-plugins-base-gio \
                                                    gstreamer1.0-plugins-base-playback \
                                                    gstreamer1.0-plugins-base-typefindfunctions \
                                                    gstreamer1.0-plugins-base-videoconvertscale \
                                                    gstreamer1.0-plugins-base-volume \
                                                    gstreamer1.0-plugins-good-audiofx \
                                                    gstreamer1.0-plugins-good-audioparsers \
                                                    gstreamer1.0-plugins-good-autodetect \
                                                    gstreamer1.0-plugins-good-avi \
                                                    gstreamer1.0-plugins-good-deinterlace \
                                                    gstreamer1.0-plugins-good-interleave \
                                                    ', '', d)} \
    libgles2 \
"

RDEPENDS:${PN}-web-inspector-plugin += "\
    shared-mime-info \
"

# Extra runtime recommends
RRECOMMENDS:${PN} += "\
    ca-certificates \
    ttf-dejavu-sans \
    ttf-dejavu-sans-mono \
    ttf-dejavu-serif \
    ${PN}-qtwpe-qml-plugin \
    ${@bb.utils.contains('PACKAGECONFIG', 'video', 'gstreamer1.0-plugins-base-meta gstreamer1.0-plugins-good-meta gstreamer1.0-plugins-bad-meta', '', d)} \
"

DEFAULT_PREFERENCE = "-1"

FILESEXTRAPATHS:prepend := "${THISDIR}/${PN}:"

# https://commits.webkit.org/319279@main
PR = "r319279"
SRCREV = "93472ec12ee947b30c7ec3c176ca0e2fb6ba6ace"
SRC_URI = "git://github.com/WebKit/WebKit.git;protocol=https;branch=main \
           file://0001-libpas-Only-include-stdatomic.h-when-compiling-with.patch \
           file://0002-WebDriver-Guard-LOG_CHANNEL-check-with-LOG_DISABLED.patch \
           "
S = "${WORKDIR}/git"

This file is self-contained on purpose; the idea is not to rely on any includes so that unpredictable behavior doesn’t happen in the future.

While the above file contains a lot of interesting details, the most interesting practical part is the last few lines. The SRCREV is set to point to the latest commit from the WebKit main branch (at the time of writing). Along with that comes SRC_URI that specifies two patches required to make WPE WebKit compile.

The problem with the patches is that advancing SRCREV will likely make them unusable due to merge conflicts. Therefore, when advancing the revision, it’s recommended to remove patches from SRC_URI and face the compilation problems from scratch as it’s very likely there will be new compilation problems anyway. Fortunately, nowadays LLMs can be used to fix any compilation problems by preparing custom patches just like the below ones (created by LLM as well). For example, most of the modern models should manage to prepare proper patches just by pointing them to the above recipe and the temp directory with the latest logs from build commands.

Assuming one uses the wpewebkit_latest.bb above, the first patch needs to be created in: meta-webkit/recipes-browser/wpewebkit/wpewebkit/0001-libpas-Only-include-stdatomic.h-when-compiling-with.patch with the following content:

From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001
From: Pawel Lampe <plampe@igalia.com>
Date: Mon, 17 Aug 2026 00:00:00 +0000
Subject: [PATCH] [libpas] Only include stdatomic.h when compiling with Clang

Some libpas .c files (e.g. jit_heap.c) are compiled as C++ via
set_source_files_properties(... PROPERTIES LANGUAGE CXX) in
Source/bmalloc/CMakeLists.txt, for TZone heap support. pas_utils.h
unconditionally does `#include <stdatomic.h>`, but GCC's own
<stdatomic.h> relies on the C11-only `_Atomic` keyword and has no
support for being included from C++ translation units (this was only
addressed in much newer GCC releases). Compiling any of the
CXX-tagged libpas .c files with GCC 13 therefore fails with:

    error: '_Atomic' does not name a type

The header is only actually needed here for the Clang-specific
__c11_atomic_* intrinsics guarded by `#elif PAS_COMPILER(CLANG)`
further down in this file; the non-Clang (GCC) path uses the
__atomic_* builtins instead and does not need any of the types or
macros from <stdatomic.h>. Guard the include accordingly so GCC
builds (both plain C and the CXX-tagged libpas sources) are
unaffected by GCC's non-C++-aware <stdatomic.h>.

Upstream-Status: Pending
Signed-off-by: Pawel Lampe <plampe@igalia.com>
---
 Source/bmalloc/libpas/src/libpas/pas_utils.h | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/Source/bmalloc/libpas/src/libpas/pas_utils.h b/Source/bmalloc/libpas/src/libpas/pas_utils.h
index 962634080930..9b98fda54f43 100644
--- a/Source/bmalloc/libpas/src/libpas/pas_utils.h
+++ b/Source/bmalloc/libpas/src/libpas/pas_utils.h
@@ -42,7 +42,15 @@
 #endif
 
 #include <limits.h>
+#if PAS_COMPILER(CLANG)
+/* GCC's <stdatomic.h> relies on the C-only _Atomic keyword and is not
+ * usable when this header is included from a translation unit compiled
+ * as C++ (some libpas .c files are compiled as C++, see bmalloc's
+ * CMakeLists.txt). It is only needed here for the Clang-specific
+ * __c11_atomic_* intrinsics below; the GCC path uses __atomic_* builtins
+ * instead. */
 #include <stdatomic.h>
+#endif
 #include <stdbool.h>
 #include <stdint.h>
 #include <string.h>
--
2.43.0

The second patch should be: meta-webkit/recipes-browser/wpewebkit/wpewebkit/0002-WebDriver-Guard-LOG_CHANNEL-check-with-LOG_DISABLED.patch with:

From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001
From: Pawel Lampe <plampe@igalia.com>
Date: Mon, 17 Aug 2026 00:00:00 +0000
Subject: [PATCH] [WebDriver] Guard LOG_CHANNEL check with
 LOG_DISABLED/RELEASE_LOG_DISABLED

WebDriverService::handleRequest() unconditionally checks
LOG_CHANNEL(WebDriverClassic).state to decide whether it is worth
building the request/response log strings. However,
Source/WebDriver/Logging.h only declares the WebDriverClassic (and
other WebDriver) log channels inside:

    #if !LOG_DISABLED || !RELEASE_LOG_DISABLED

On a release build (NDEBUG, so LOG_DISABLED is true) without journald
support and without OS_LOG/Android (so RELEASE_LOG_DISABLED is also
true) - the common configuration for an embedded Linux build without
the "journald" PACKAGECONFIG - that guard is false, so the channel is
never declared, and this direct, unguarded use of LOG_CHANNEL() fails
to compile:

    error: 'LOG_CHANNEL_PREFIXWebDriverClassic' was not declared in this scope

RELEASE_LOG_INFO() itself already collapses to a no-op in that
configuration (see wtf/Assertions.h), so guard this manual
LOG_CHANNEL() state check with the same condition used to declare the
channel in Logging.h.

Upstream-Status: Pending
Signed-off-by: Pawel Lampe <plampe@igalia.com>
---
 Source/WebDriver/WebDriverService.cpp | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/Source/WebDriver/WebDriverService.cpp b/Source/WebDriver/WebDriverService.cpp
index 07ae385f33f4..2612d239a611 100644
--- a/Source/WebDriver/WebDriverService.cpp
+++ b/Source/WebDriver/WebDriverService.cpp
@@ -381,6 +381,7 @@ bool WebDriverService::findCommand(HTTPMethod method, const String& path, Comman
 void WebDriverService::handleRequest(HTTPRequestHandler::Request&& request, Function<void (HTTPRequestHandler::Response&&)>&& replyHandler)
 {
     Function<void (HTTPRequestHandler::Response&&)> actualReplyHandler = WTF::move(replyHandler);
+#if !LOG_DISABLED || !RELEASE_LOG_DISABLED
     if (LOG_CHANNEL(WebDriverClassic).state != WTFLogChannelState::Off) {
         RELEASE_LOG_INFO(WebDriverClassic, "HTTP request %s %s (body=%zu bytes)", request.method.utf8().data(), request.path.utf8().data(), request.dataLength);
         actualReplyHandler = [startTime = MonotonicTime::now(), replyHandler = WTF::move(actualReplyHandler)](HTTPRequestHandler::Response&& response) mutable {
@@ -388,6 +389,7 @@ void WebDriverService::handleRequest(HTTPRequestHandler::Request&& request, Func
             replyHandler(WTF::move(response));
         };
     }
+#endif
 
     auto method = toCommandHTTPMethod(request.method);
     if (!method) {
--
2.43.0

Once the patches are added, wpewebkit latest should build correctly. However, to make use of it one needs a browser.

To demonstrate how easy the browser for WPE WebKit with Platform API can be, a very minimalistic one will be prepared below.

The first step is to create a directory:

mkdir meta-webkit/recipes-browser/wpe-browser/

Then a file: meta-webkit/recipes-browser/wpe-browser/main.cpp that implements the whole browser:

#include <wpe/webkit.h>

int main(int argc, const char *argv[]) {
    g_autoptr(GMainLoop) loop = g_main_loop_new(nullptr, false);
    g_autoptr(WebKitWebView) view = WEBKIT_WEB_VIEW(g_object_new(WEBKIT_TYPE_WEB_VIEW,
        nullptr));
    webkit_web_view_load_uri(view,
        (argc > 1) ? argv[1] : "https://wpewebkit.org");
    g_main_loop_run(loop);
    return EXIT_SUCCESS;
}

Then a file: meta-webkit/recipes-browser/wpe-browser/CMakeLists.txt that describes how to build it:

cmake_minimum_required(VERSION 3.16)
project(wpe-browser CXX)

set(CMAKE_CXX_STANDARD 17)

include(GNUInstallDirs)

find_package(PkgConfig REQUIRED)

# The Wayland WPE Platform already depends on wpe-platform-2.0
pkg_check_modules(WebKitDeps REQUIRED
    IMPORTED_TARGET
    wpe-webkit-2.0
    wpe-platform-wayland-2.0
)

add_executable(wpe-browser main.cpp)

target_link_libraries(wpe-browser
    PRIVATE
        PkgConfig::WebKitDeps
)

install(TARGETS wpe-browser RUNTIME DESTINATION ${CMAKE_INSTALL_BINDIR})

and finally a recipe file: meta-webkit/recipes-browser/wpe-browser/wpe-browser_1.0.bb that allows bitbake to build the browser and install it in the image:

SUMMARY = "Minimal WPE WebKit browser launcher"
DESCRIPTION = "A minimal launcher built on the WPE Platform API, displaying \
               a URL given as its only argument (defaults to https://wpewebkit.org). \
               Based on https://simonpena.com/blog/2026/03/20/getting-started-with-wpe-webkit/"
HOMEPAGE = "https://simonpena.com/blog/2026/03/20/getting-started-with-wpe-webkit/"
LICENSE = "MIT"
LIC_FILES_CHKSUM = "file://${COMMON_LICENSE_DIR}/MIT;md5=0835ade698e0bcf8506ecda2f7b4f302"

FILESEXTRAPATHS:prepend := "${THISDIR}:"

SRC_URI = "file://main.cpp \
           file://CMakeLists.txt \
          "

S = "${WORKDIR}"

DEPENDS = "wpewebkit"
RDEPENDS:${PN} += "wpewebkit"

inherit cmake pkgconfig features_check

REQUIRED_DISTRO_FEATURES = "opengl wayland"

After all those steps, everything is ready to perform a build. This time the commands are executed in the SDK:

source poky/oe-init-build-env build
bitbake core-image-weston
cd tmp/deploy/images/raspberrypi3-64/
# flash to SD card based on preference

Trying the image #

Once the image is built and flashed to SD card, and the SD card has been used to boot the Raspberry Pi, one can SSH into it, basically by:

ssh root@<IP>

Then, the browser should work out of the box. The first command to try is the one that doesn’t need the network and therefore is the simplest:

XDG_RUNTIME_DIR=/run/user/1000 WAYLAND_DISPLAY=wayland-1 wpe-browser 'webkit://gpu/stdout'

If the network is available, it’s worth starting simple and loading some HTTP page:

XDG_RUNTIME_DIR=/run/user/1000 WAYLAND_DISPLAY=wayland-1 wpe-browser 'http://info.cern.ch'

If that works as well, one can try the HTTPS one:

XDG_RUNTIME_DIR=/run/user/1000 WAYLAND_DISPLAY=wayland-1 wpe-browser 'https://igalia.com'

If there’s an issue with certificates, it’s likely due to broken date/time, so the correct one needs to be set using a command like:

date -s '2026-09-02 22:34:56'

Conclusions #

Since the Platform API is the future of WPE, it’s worth using it already. As the above sections demonstrate, it hasn’t ever been easier to create a WPE-powered browser for embedded hardware. Moreover, nowadays as LLMs are at play, using latest sources from main branch and quickly patching on demand is a possibility worth utilizing. With that, a broad sea of experimenting possibilities becomes wide open. However, it must not be forgotten that while main branch is great for testing new web platform features implemented in WebKit, it’s not necessarily ideal for testing performance. In such case, it’s better to rely on releases and proper browser engine fine tuning for particular hardware one plays with.

Update on what happened in WebKit in the week from August 31 to September 7.

This was another week focused on ironing out graphics issues in preparation for the upcoming 2.54.x release series. Did we say release? Here we have another set of packaged release candidates as well!

Cross-Port 🐱

Graphics 🖼️

Releases 📦️

This post walks through the process of bringing back an extension-focused //extensions/shell in Chromium and enabling the “Add to Chrome” button on the Chrome Web Store.

If you haven’t read it yet, I recommend starting with Extension Migration Progress Update – Part 1, where I described the initial stage of migrating extension installation formats and APIs out of the //chrome layer.

After migrating the extension installation formats — CRX, ZIP, and Unpacked — into the //extensions layer, the next natural step was to enable embedders to install extensions directly from the Chrome Web Store.

To achieve this, the Web Store Private APIs were an essential piece of the puzzle.

Migrating Web Store Private APIs

The Web Store Private APIs had a relatively large dependency on //chrome. We started by migrating the relevant files and gradually replacing Chrome-specific dependencies with alternative implementations where necessary.

In some cases, this meant introducing new interfaces, while in others it meant moving code to a more appropriate layer.

After working through these dependencies, the final changes were eventually landed upstream: Chromium CL – Web Store Private API migration.

The changes are included in M152.

However, after this migration was completed, we encountered an unexpected obstacle.

Where did extensions/shell go?

Chromium originally had a lightweight embedder called //extensions/shell, previously built as the app_shell target.

The idea was simple: provide an environment where extensions could be installed and run without bringing up the entire Chrome browser.

However, the shell had originally been created about a decade ago for a different purpose. It was intended to provide an independent environment for ChromeOS platform applications without requiring the full Chrome browser.

Platform apps were eventually deprecated, and the shell was later repurposed primarily as a lightweight environment for extension browser tests.

Over time, the extension browser tests were moved into the //chrome layer. The shell was no longer used by any real product, and maintaining it became technical debt. As a result, app_shell was eventually removed upstream.

Why were the browser tests moved from //extensions to //chrome?

I asked an Extension owner about the reasoning behind this change.

The key point was that the old shell-based browser tests did not provide a sufficiently realistic environment for testing extensions.

There were two major problems.

First, many extension APIs delegate part of their implementation to the embedder through interfaces such as ExtensionsApiClient. Testing only the //extensions implementation with the shell meant that the actual embedding logic was not exercised. Bugs in the delegated implementation could therefore go unnoticed.

Second, extensions depend on much more than the //extensions layer itself. UI, networking, rendering, process management, and other browser functionality all affect how extensions behave. The //chrome layer provides an important part of that environment.

Therefore, the upstream direction was to move extension browser tests into the real Chrome browser environment rather than maintaining a separate shell.

That reasoning makes sense for testing.

In fact, we agreed that bringing the old shell back simply as a test environment would not be the right approach.

But I had a different question:

What if we use a lightweight shell not as a test environment, but as a reference implementation and a demo environment for embedders?

The answer was that Chromium currently does not plan to maintain such an embedder. The concern is that a simplified shell could itself become an inaccurate reference implementation. If its behavior diverged from the real Chrome integration, it could create more confusion rather than provide useful guidance.

The suggested direction was instead to use extension-based Web Platform Tests (WPT) to provide consistency, while considering actual browsers as the reference implementations.

That approach makes sense from the upstream perspective.

However, for our migration work, we still needed something slightly different.

We needed a small, working environment where we could demonstrate that extensions can actually be supported outside of the //chrome layer.

So we decided to bring back an extension-focused shell — not as an upstream testing framework, but as a practical environment for integration and demonstration.

Although this shell is not part of upstream Chromium, we are currently maintaining the implementation internally in the Igalia repository, with plans to open-source it in the future. It serves as a lightweight environment for demonstrating and validating Extension support outside of the //chrome layer, rather than as a replacement for Chromium’s browser tests or an upstream reference implementation.

Rebuilding the Extension Shell

The first step was to establish the basic structure of the shell.

1. Building the skeleton

We created a custom ShellBrowserContext directly derived from content::BrowserContext, bootstrapped the Views environment, and added a simple placeholder window.

On top of that, we connected the extension system through:

• ShellExtensionsBrowserClient — the entry point for implementing embedder-specific Extensions behavior for the shell.
• ShellExtensionSystem — the service responsible for core extension management, including loading and unloading extensions and parsing manifests.

This gave us the basic environment needed to initialize Chromium’s extension infrastructure without depending on the full Chrome browser.

2. Building the installation pipeline

Next, the shell needed to actually install extensions.

We connected ExtensionExternalInstaller, which orchestrates the overall installation flow — including downloading, unpacking, and manifest validation — to support CRX and ZIP installation.

We also implemented ShellContentUtilityClient, which registers the sandboxed utility-process service needed to safely handle untrusted data, such as ZIP decompression, out of process.

We then completed the renderer-side wiring so that an installed Manifest V3 extension could actually execute its service worker and access chrome-extension:// resources.

Several pieces of the extension installation infrastructure also needed to be connected, including:

• Blocklist — a KeyedService that tracks which extensions are blocklisted, backed by Safe Browsing.
• InstallStageTracker — tracks installation stages and failure events specifically for policy force-installed extensions.
• InstallTracker — broadcasts installation start, completion, and failure events to observers.
• InstallVerifier — confirms that an installed extension is either verified through a Web Store signature or allowlisted by enterprise policy.
• SharedModuleService — supports the import/export mechanism that extensions use to pull in shared modules from other extensions.

At this point, we had a shell that could actually install and initialize extensions.

3. Making it look like a browser

The next step was to make the shell usable as an actual browser rather than just an extension test environment.

We used the existing content/shell implementation and ShellPlatformDelegateViews as references, without directly depending on them.

The shell now has a basic browser UI with:

• Back
• Forward
• Reload
• Stop
• Address bar
• A views::WebView containing a real content::WebContents

This allowed us to browse normal web pages as well as run installed extensions.

4. Enforcing network-level policies

With the browser environment in place, we also started validating extension APIs that operate at the network level.

In particular, we tested declarativeNetRequest and webRequest, which uncovered a few issues that needed to be fixed in the shell integration.

After these steps, the basic foundation of an embedder that can install and run extensions was finally becoming stable.

And that naturally led to the next question:

Can we install an extension directly from the Chrome Web Store?

The answer was initially no.

Enabling the “Add to Chrome” Button

When opening an extension page on the Chrome Web Store, the “Add to Chrome” button was disabled.

The reason was another missing piece of Chrome-specific functionality.

The Web Store uses the chrome.webstorePrivate extension API to perform the installation flow.

Behind this API is WebstorePrivateAPIDelegate, which delegates browser-specific operations to the embedder. These operations include things such as installation approval, login state, Safe Browsing checks, and the installation confirmation UI.

Since our shell was not Chrome, these pieces were not implemented.

Implementing the missing pieces

To make the Web Store installation flow work, we implemented the required delegate functionality with safe defaults and added the necessary extension infrastructure to the shell.

This included:

• WebstorePrivateAPIDelegate — provides the embedder-specific implementation required by the chrome.webstorePrivate API to handle Web Store extension installation.
• ShellExtensionsAPIClient — provides shell-specific implementations of browser-level services required by extension APIs.
• ShellExtensionInstallPromptClient — provides the shell-specific UI handling for extension installation prompts.
• WebstorePrivateAPI factory — creates and provides the WebstorePrivateAPI service for extensions running in the shell.
• ManagementAPI factory — creates and provides the ManagementAPI service, which exposes extension management functionality to extensions.

The shell also needed to provide appropriate User-Agent metadata, which was delegated through:

embedder_support::GetUserAgentMetadata().

After wiring these pieces together, the Chrome Web Store was finally able to recognize our shell as an environment capable of handling extension installation.

Demo

And this is where things became much more interesting.

The “Add to Chrome” button is now enabled, and clicking it can trigger the extension installation flow in our extension shell.

This is an important milestone for the migration.

We now have a lightweight browser environment that can:

1. Launch independently of Chrome.
2. Browse normal web pages.
3. Install extensions.
4. Run Manifest V3 extensions.
5. Exercise extension APIs.
6. Install an extension directly from the Chrome Web Store.

What’s next?

Getting an extension installed is only part of the story.

The next step is making the installed extensions behave like they do in a real browser.

There are still several APIs that have traditionally depended on the //chrome layer. Permissions, cookies, and other browser integration points are among the areas that need to be migrated or provided by the embedder.

Once these pieces are migrated, we should be able to go beyond simply installing an extension from the Web Store and demonstrate a more complete end-to-end experience:

Chrome Web Store → Extension installation → Extension startup → Extension API usage

This is the next stage of the migration work.

I’ll continue documenting the progress as more of these APIs move into the //extensions layer and the extension shell becomes a more complete demonstration environment for embedders.

Last week I attended FOSForums 2026 and presented about Linux page_owner.

The conference

FOSForums (Free and Open Source Forums) is a free conference focused on free and open source software, hardware, and the communities around them.

Now in its 4th edition, since 2023 it has been held at the Institute of Computing at Unicamp – one of Brazil’s top public research universities – and has featured speakers from Igalia, Red Hat, Linaro, RISC-V International, Toradex, Unicamp (State University of Campinas), USP (University of São Paulo), Eldorado Research Institute, and others.

The partners of the conference for this year are RISC-V International, Unicamp and its Institute of Computing and Institute for the Conservation of Free Technologies.

It’s still a relatively new and small conference – around 30 attendees this year – but a notably high share of them are students, both undergraduate and graduate. That makes it an especially relevant place to share knowledge, identify talent, and support people early in their tech careers – contributing back to the community.

This year’s schedule covered the FinOps Foundation, open science, continuous integration with RISC-V/Linux/KernelCI, Linux filesystem performance on flash devices, and Linux page_owner (my talk), plus a hands-on session with BitDogLab (an open-source educational hardware platform). A talk on turning bug reports into kernel patches was also planned but unfortunately couldn’t be delivered – keep an eye on Heitor’s blog!

The talk

Picking up the thread on page_owner (see the blog post series), I wanted to put together a more introductory, user-friendly version of the talk Improving page_owner for profiling and monitoring memory usage per allocation stack trace presented at Linux Plumbers Conference 2025.

I opened with an introduction about Igalia – who we are, what we work with, and what we do – and how to join us through our open positions and our Coding Experience program.

The technical part covered the problem at hand, “Where is memory going?”, and then walked through page_owner from definition to application: what it is, what it is for, how it works, how to use it, then basic and advanced examples.

The session recording is available on YouTube (currently within the live stream; later in the FOSForums 2026 playlist), and the slides are available here.

Acknowledgments

Thank you to the FOSForums organizing committee for the great conference, and Igalia for the opportunity to present this work.

Update on what happened in WebKit in the week from August 24 to August 31.

After such a packed edition last week, we can relax with an even more packed installment this week! The team has been moving full steam ahead with the Layer-Based SVG Engine, and graphics improvements in general, but we also had a handful of other updates, such as SDK updates, a new WebsitePolicies API, and more.

Cross-Port 🐱

Graphics 🖼️

Infrastructure 🏗️

In Part 2 I explained how ARM32 has two ISAs available (ARM and Thumb), how their feature sets differ in important ways and how software can and often does make use of both even in the same program.

Today we’re going one level higher and look at the ABIs (Application Binary Interface). Whereas an ISA defines what bit sequences make the CPU do what operations, an ABI is a set of conventions software is meant to follow so that it is interoperable with other software targetting the same ABI.

An ABI is adopted by compilers and operating systems. The designers of the instruction set often define some base ABI that compilers only need to extend to define a few missing bits.

One of the biggest underlying reasons getting perf to work in ARM32 is hard is that ARM has deprecated and gone over multiple base ABIs in its lifetime. Furthermore: the details on how frame pointers should work used to be completely non-standard and that has created fragmentation.

As of writing, these are the standard base ABIs defined for ARM:

• ~1987: APCS (Arm Procedure Call Standard (obsolete)). Only applies to ARM.
• ~1997: TPCS (Thumb Procedure Call Standard (obsolete)). Thumb-only counterpart of APCS.
• 1998: ATPCS (ARM-THUMB Procedure Call Standard). Unified ABI for both ARM and Thumb.
• 2003: AAPCS (Procedure Call Standard for the Arm® Architecture). The current base ABI. It applies to both ARM and Thumb.

In this post we’ll learn some typical examples of features an ABI would standardize, ultimately landing in frame pointers as one of such features.

The stack

At its most basic, a CPU does not need to have a concept of a stack. The ARM ISA exemplifies this well: the instructions used for stack manipulation are extremely generic—they can work on any register, support any kind of stack (full descending, full ascending, empty descending, empty ascending) and also work as general “write multiple registers to memory pointed by register” instructions.

But of course, a stack is a very useful abstraction that any non-trivial program will need as it allows to free up registers by saving their data for later use. Different pieces of code (e.g. functions) need to agree on a definition for the stack and rules to use it so they don’t accidentally overwrite each other’s data.

As such, ABIs define the stack; namely: what register points to the current top of the stack, whether it grows towards greater adresses (ascending stack) or toward lower adresses (descending stack), whether the “top of the stack” should be understood as “the last element pushed” (full stack) or “the position a newly pushed element would use” (empty stack), any alignment requirements, and rules for writing to the stack—for example: writes outside of the stack may be considered undefined behavior even in the absence of function calls so that interrupt handlers can make use of the same stack.

On ARM32, virtually all the ABIs¹ agree on using a full-descending stack, whose top position is tracked by the r13 register, which for this reason is also give the alias sp (stack pointer). The stack is 4 byte (32 bit) aligned. The AAPCS ABI further requires 8 byte alignment at the public interface (e.g. when calling a library function). Storing data outside of the valid region of the stack (i.e. in addresses lower than sp) is undefined behavior.

Function calls

An ABI defines what a function call is under the hood; namely: how arguments are passed and how are values returned. This part of ABIs is important enough to have its own name: calling convention.

Function arguments

A simple calling convention could require all function arguments be to pushed into the stack. The callee could then access them by addressing the stack pointer. Notably, the cdecl and stdcall calling conventions used in MS Windows 32-bit x86 work in this manner.

Modern calling conventions—including ARM32 AAPCS and all x86-64 calling conventions—use registers in addition to the stack for passing arguments, trading complexity in the ABI for better performance. This is especially a win for functions with few arguments. In many cases, programmers and compilers can strategically do computations directly in the same registers that will be later used as arguments, further reducing the overhead of function calls.

In AAPCS, r0-r3 are filled with the first few arguments. Arguments with types smaller than 32 bit (e.g. char, int16_t) are extended to 32 bits before the call. When an argument has a 64-bit type, it takes two consecutive registers. For more details, see section 6.5, Parameter Passing in the AAPCS specification. Remaining arguments are pushed into the stack.

Callee-saved and caller-saved registers

ABIs often separate user registers in two groups:

• Callee-saved registers
• Caller-saved registers, also known as scratch registers

A function is allowed to write to callee-saved registers at any point, but it must ensure they have their original value on return. This is normally accomplished by pushing (saving) the previous value to the stack before writing and popping it before return, hence the name «callee-saved».

On the other hand, a function is allowed to freely write (scratch) any and all caller-saved registers without preserving their old value anywhere. This also means that, before doing any function call, you must be careful to save any important data in these registers, hence the name «caller-saved».

Having a healthy mix of callee-saved and caller-saved is good for performance, as it reduces the number of stack manipulation operations necessary in typical functions.

In AAPCS, r0-r3 and r12 are caller-saved (i.e. scratch registers). All other general purpose registers are callee-saved.

Returning from a function: return pointers

The calling convention also needs to specify how returning from a function works. The caller must store a return pointer (an address to executable code immediately following the function call) in a well-defined place.

In many calling conventions, including the common ones for x86 and x86-64, the return pointer is passed in the stack. ARM calling conventions are a bit smarter: instead they place the return pointer in a so-called link register (lr). This means that simple leaf functions (functions that do not call other functions) can operate on their arguments and return a value without ever touching the stack.

All ARM calling conventions use r14 as the link register.

Of course, many functions will need to call to other functions before returning, so how does a link register handle this? Quite simply: lr is a callee-saved register! A function that will perform function calls will typically store lr in the stack somewhere in its preamble, do as many function calls as it wants, then restore the old value of lr from the stack.

The l in the bl and blx branch instructions in ARM stands for “link”. It means that the current value of pc will be loaded in lr before the branch. This is the most common way to do function calls in ARM and Thumb.

Special purpose registers

An ABI may reserve certain CPU registers for specific purposes, even if those registers would otherwise be general purpose registers according to the ISA. Compilers do to take this into account when generating code.

AAPCS leaves r9 free to use for any ABI extending AAPCS (e.g. for a specific operating system).

AAPCS defines r12 as the Intra-Procedure-call scratch register (ip). Functions are allowed to use it as a regular scratch register, but its real purpose is to enable writing simple veneers without needing stack manipulation.

Side-tangent: Veneers…? What is a veneer?

(You can safely skip this section If you just want to understand the minimum concepts necessary for frame pointers. Keep reading if you have a more general interest in ARM assembly.)

Instructions in ARM have a fixed size of 32 bits which is as wide as pointer. A consequence of this is that calling functions in arbitrary locations of the program requires a variable number of instructions. Most branches in ARM are done with signed offsets with a range of ±32 MiB (26 bits, of which only the most significant 24 bits are actually encoded in the instruction because ARM instructions are 4-byte-aligned). Thumb is even scrappier: the typical Thumb bl (branch with link) instruction only supports offsets of ±4MiB encoded in 22 bits, which is already more than the 16 bits per instruction of Thumb-1 and it is only possible because the bl ASM instruction gets turned into two machine Thumb-1 instructions designed for this purpose, each containing half of the offset).

Compilers and linkers cope with this limitation by assuming calls will be near enough for the offset to fit. When the offset does not fit, they synthesize some code (the veneer, also sometimes called a trampoline or range extension thunk) in a location close enough for the offset to fit. The veneer will then load the entire address in some register—likely ip— and then use a «branch to address in register» instruction (e.g. bx ip).

From this example the utility of having a scratch register not used for argument passing (ip) becomes apparent: if a scratch register like ip did not exist, the veneer would need to spill some callee-saved register to the stack just to build an address for the bx instruction. Furthermore, the callee couldn’t return directly into the original caller, but it would have to return back into the veneer so that the values from the stack could be restored.

Also of note is that is the fact that the blx instruction (branch with link and exchange ISA, provided an offset or register) did not exist until ARMv5. Before, there was only bx (branch and exchange ISA), which operated on a register. This resulted in a different number of instructions required for the branch depending on the ISA of the target (which may not be known until run-time in the case of shared libraries), and by extension made veneers necessary for interworking between ARM and Thumb.

Frame pointers

A stack frame is the chunk of the stack used by a specific function call, not including nested calls.

A frame pointer points to a fixed location (e.g. the start or end) of the stack frame. When we say that a program uses frame pointers, we refer to that program dedicating a register—the frame pointer register (usually shortened as fp)—to store the frame pointer at all times.

Having a frame pointer allows assembly code to refer to locations in the stack with simple offset from the frame pointer. So, for example, a certain variable saved in the stack will be at fp-4 and will remain at fp-4 even if code later pushes to the stack. This is more useful for humans than compilers, since it’s possible to access the same data from sp as long as you keep track of the required offset changing every time something is pushed or popped.

Something much more useful occurs if code always pushes the old value of the frame pointer register and the return pointer to the stack at call boundaries: it becomes possible to inspect the stack at runtime: the frame pointer register points to the current frame record, from which the previous frame pointer and return pointer are accessible in a fixed offset. This process can be repeated, traversing the stack like a linked list, one stack frame at a time. Before main(), a sentinel value (typically a null pointer) should be pushed instead of a frame pointer so that the end of the linked list can be detected.

Note that the usage of frame pointers is an ABI matter: The above breaks if you call a function in a library that doesn’t use the same exact convention for frame pointers. If that function skips updating the frame pointer register, its stack frames will become invisible. Or worse: if it writes some random value to it, the entire chain of callers will no longer be recoverable.

In x86 ABIs, the Base Pointer register (bp in 16-bit, ebp in 32-bit, rbp in 64-bits) is commonly used to store the frame pointer.

Frame pointers were important for obtaining backtraces in early debuggers, but this is less necessary in modern toolchains. Nowadays compilers can generate debug information (debuginfo) in a format like DWARF that debuggers can use to map any value of the pc register to what function it is part of and how deep in the stack the return pointer is at that particular instruction. This is the same process they use to map local variables to positions in the stack. The process can be repeated until the entire stack trace is obtained.

Since frame pointers became less necessary for debuggers, compiler optimizations that skip updating the frame pointer register or even use it as a general-purpose register have become very common. Omitting frame pointers as an optimization has been the default in gcc for all platforms since 2017, after it had been enabled in most target platforms individually. ARM32 already had this enabled in 2010 and earlier.

While debuginfo is a good alternative to frame pointers to get backtraces in a debugger, other development tools are not so lucky. Recovering call chains from frame pointers is a very lightweight affair: it’s a singly linked list traversal where all the nodes are within consecutive memory. On the other hand, getting a call chain through debuginfo requires having separate debuginfo sections for each object file (e.g. .so library) mapped in memory, search the wanted pc value in their tables and decode the debug tags info to find the address of the return pointer; then repeat this for every other stack frame.

Some tools have tried to adapt to this new normal of not having frame-pointers by adding support for DWARF unwinding. perf is one of them. This is however still unsatisfactory on low-end hardware: perf record -g --call-graph=dwarf cog about:blank will bring a Raspberry Pi 3 to its knees, freezing the entire system until it dies on the hands of the OOM killer. High-end hardware, like desktop PCs, can usually handle using DWARF, but it still results in much higher overhead.

Omitting frame pointers as an optimization has been increasingly reconsidered in recent years, with Fedora 38+, Ubuntu 24.04+ and Arch Linux modifying their build recipes to attempt to build all packages with frame -fno-omit-frame-pointer. Modern CPUs (especially 64-bit CPUs) have enough registers that dedicating one for this purpose is not the hit to performance it once was. Some people would argue that even if there is a slight performance hit, the gains from users and developers being able to use profilers greatly outweight it.

With this modern understanding of what frame pointers are, we will also try to build all packages with frame pointers enabled in an ARM32 environment, so that we can also use a profiler there.

Problem: what are the ARM32 frame pointer ABIs, really?

APCS frames

The (old and deprecated) APCS spec specified an optional frame pointer ABI. When you pass the (now deprecated) -mapcs-frame flag to gcc, you tell it to use the frame pointer structures defined in APCS. Here is a descriptive diagram from page 629 of the Acorn Archimedes‘ Programmer Reference Manual (1987), whose Appendix C is “ARM Procedure Call Standard”:

Here is a simple function that we will use to compare the frame pointer ABIs:

extern void show_sum(int a);
extern void show_diff(int b);

int do_operations(int a, int b) {
    int sum = a + b;
    show_sum(sum);
    int diff = a - b;
    show_diff(diff);
    int xor = a ^ b;
    return xor;
}

This is the assembly generated by gcc:

$ arm-linux-gcc -mcpu=cortex-a53 -marm -mapcs-frame -gdwarf-4 -fno-omit-frame-pointer -O2 -g -c add.c && arm-linux-gnu-objdump -S example.o
int do_operations(int a, int b) {
   0:	e1a0c00d 	mov	ip, sp
   4:	e92dd830 	push	{r4, r5, fp, ip, lr, pc}
   8:	e1a04001 	mov	r4, r1
   c:	e1a05000 	mov	r5, r0
  10:	e24cb004 	sub	fp, ip, #4
    int sum = a + b;
    show_sum(sum);
  14:	e0800001 	add	r0, r0, r1
  18:	ebfffffe 	bl	0 <show_sum>
    int diff = a - b;
    show_diff(diff);
  1c:	e0450004 	sub	r0, r5, r4
  20:	ebfffffe 	bl	0 <show_diff>
    int xor = a ^ b;
    return xor;
}
  24:	e0250004 	eor	r0, r5, r4
  28:	e89da830 	ldm	sp, {r4, r5, fp, sp, pc}

Here ip is used as a temporary to hold the value of sp before the push. The resulting stack matches the one in the old APCS document as we could expect:

fp points here: | pc at time of push (i.e. savemask pointer) |
                | saved lr                                   |
                | ip (i.e. the saved sp before the push)     |
                | saved fp                                   |

We can obtain the call chain by looking at the fp register and traversing the linked list of frame records: the saved lr contains the return function for that stack frame, and by extension, who called that function. We continue traversing the stack upwards until we find a NULL pointer indicating the end of the linked list.

AAPCS frames, clang

It may seem weird that I just spend that much time above showing a very old ABI (APCS). Especially because if you look at the modern AAPCS spec today you’ll see frame pointers also defined.

However, if you scroll to the change history, you’ll also notice it was only added in January 2020, whereas AAPCS had been around for 17 years before. This is the real source of problems: as far as I can tell, during these 17 years compilers had no official guidance on any frame pointer ABI for AAPCS.

Making things worse, GCC and clang ended up with similar but mutually incompatible frame pointer ABIs. The official AAPCS frame pointer ABI sided with clang’s. It looks like this:

$ clang -target arm-linux-gnueabihf -g -gdwarf-4 -mcpu=cortex-a53 -c -O2 -fno-omit-frame-pointer -c example.c && arm-linux-gnu-objdump -S example.o
int do_operations(int a, int b) {
   0:	e92d4830 	push	{r4, r5, fp, lr}
   4:	e28db008 	add	fp, sp, #8
   8:	e1a05000 	mov	r5, r0
    int sum = a + b;
   c:	e0810000 	add	r0, r1, r0
  10:	e1a04001 	mov	r4, r1
    show_sum(sum);
  14:	ebfffffe 	bl	0 <show_sum>
    int diff = a - b;
  18:	e0450004 	sub	r0, r5, r4
    show_diff(diff);
  1c:	ebfffffe 	bl	0 <show_diff>
    int xor = a ^ b;
  20:	e0240005 	eor	r0, r4, r5
    return xor;
  24:	e8bd8830 	pop	{r4, r5, fp, pc}

Only two words are pushed in the stack: fp and lr. The fp register is updated to point to the the most recently saved fp in the stack.

GCC frames

If you pass -marm -fno-omit-frame-pointer to gcc (as of 15.2), this is what the resulting frame pointer ABI looks like:

$ arm-linux-gcc -mcpu=cortex-a53 -marm -gdwarf-4 -fno-omit-frame-pointer -O2 -g -c example.c && arm-linux-gnu-objdump -S example.o
int do_operations(int a, int b) {
   0:	e92d4830 	push	{r4, r5, fp, lr}
   4:	e1a04001 	mov	r4, r1
   8:	e1a05000 	mov	r5, r0
   c:	e28db00c 	add	fp, sp, #12
    int sum = a + b;
    show_sum(sum);
  10:	e0800001 	add	r0, r0, r1
  14:	ebfffffe 	bl	0 <show_sum>
    int diff = a - b;
    show_diff(diff);
  18:	e0450004 	sub	r0, r5, r4
  1c:	ebfffffe 	bl	0 <show_diff>
    int xor = a ^ b;
    return xor;
}
  20:	e0250004 	eor	r0, r5, r4
  24:	e8bd8830 	pop	{r4, r5, fp, pc}

This is very similar to what ended up becoming the official AAPCS frame pointer ABI and to what clang does except that fp points to the saved lr, rather than to the saved fp directly. This is what it ends up looking like:

What about Thumb?

While I’ve learned a lot about Thumb while working on this, I would be very cautious trying to make use of frame pointers in Thumb. They’re even messier and currently broken in GCC. For more details, you can keep reading this section.

r7 as frame pointer register

Thumb in stack traces complicates things significantly. Note that r11 is a “high register” in Thumb, which limits in how many different instructions it can be used efficiently. This is especially a problem in the original Thumb-1, which doesn’t have an instruction to push/pop low registers directly and you would typically need 3 additional instructions in the prologue and 3 additional instructions in the epilogue.

The above is much less of a concern with the now ubiquitous Thumb-2, which has 32-bit instructions for push/pop and arithmetic on high registers. However, as a consequence of this historical problem, you will see that in a default Linux setup both gcc and clang use r7 as frame pointer register in Thumb code.

$ clang -target arm-linux-gnueabihf -mcpu=cortex-a53 -mthumb -gdwarf-4 -fno-omit-frame-pointer -O2 -g -c example.c && arm-linux-gnu-objdump -S example.o
int do_operations(int a, int b) {
   0:	b5b0      	push	{r4, r5, r7, lr}
   2:	af02      	add	r7, sp, #8
   4:	4605      	mov	r5, r0
    int sum = a + b;
   6:	4408      	add	r0, r1
   8:	460c      	mov	r4, r1
    show_sum(sum);
   a:	f7ff fffe 	bl	0 <show_sum>
    int diff = a - b;
   e:	1b28      	subs	r0, r5, r4
    show_diff(diff);
  10:	f7ff fffe 	bl	0 <show_diff>
    int xor = a ^ b;
  14:	ea84 0005 	eor.w	r0, r4, r5
    return xor;
  18:	bdb0      	pop	{r4, r5, r7, pc}

The layout clang uses is the same as the official AAPCS ABI, but using r7 instead of the r11. Assuming there is no mixing of ARM and Thumb — the call chain can be recovered with the same traversal algorithm.

On GCC, however, Thumb frame pointers are unviable for call chain recovery, as the GCC sets r7 to the most recent position in the stack, which makes it impossible to know the position of the saved lr and the previous node pointer within the stack without additional information.

Using r11 for both ARM and Thumb

Using different frame pointer registers for different modes is problematic, as code in one mode (typically ARM) will frequently clobber the register used by the other mode (typically Thumb). Both r7 and r11 are caller-saved registers, so the information is there in the stack, but the stack and register values is not enough to locate them.

A simpler approach is to use r11 for both modes, which is much less awkward in Thumb-2. This is what the official AAPCS spec specifies. Recent versions of clang can be told to do this with -mframe-chain=aapcs (or alternatively, -mframe-chain=aapcs+leaf if frame pointers on leaf functions are also desired).

$ clang -target arm-linux-gnueabihf -mcpu=cortex-a53 -mthumb -gdwarf-4 -fno-omit-frame-pointer -mframe-chain=aapcs -O2 -g -c example.c && arm-linux-gnu-objdump -S example.o
int do_operations(int a, int b) {
   0:	e92d 4830 	stmdb	sp!, {r4, r5, fp, lr}
   4:	f10d 0b08 	add.w	fp, sp, #8
   8:	4605      	mov	r5, r0
    int sum = a + b;
   a:	4408      	add	r0, r1
   c:	460c      	mov	r4, r1
    show_sum(sum);
   e:	f7ff fffe 	bl	0 <show_sum>
    int diff = a - b;
  12:	1b28      	subs	r0, r5, r4
    show_diff(diff);
  14:	f7ff fffe 	bl	0 <show_diff>
    int xor = a ^ b;
  18:	ea84 0005 	eor.w	r0, r4, r5
    return xor;
  1c:	e8bd 8830 	ldmia.w	sp!, {r4, r5, fp, pc}

Frame pointer sadness

There are problems with frame pointers in ARM other than the incompatible ABIs. In fact, I kept finding more and more sadness the more I worked on this post.

GCC frames can’t be unwinded in leaf functions

While -fno-omit-frame-pointer will make GCC push fp to the stack, it doesn’t disable the “link register save elimination” optimization. This messes up with leaf functions. Consider the following trivial function:

void simple_nop(void) {
    __asm__("nop");
}

$ arm-linux-gcc -mcpu=cortex-a53 -marm -gdwarf-4 -fno-omit-frame-pointer -O2 -g -c simple_nop.c -S && cat simple_nop.s
simple_nop:
	@ args = 0, pretend = 0, frame = 0
	@ frame_needed = 1, uses_anonymous_args = 0
	@ link register save eliminated.
	str	fp, [sp, #-4]!
	add	fp, sp, #0
	.syntax divided
@ 31 "example.c" 1
	nop
@ 0 "" 2
	.arm
	.syntax unified
	add	sp, fp, #0
	@ sp needed
	ldr	fp, [sp], #4
	bx	lr
	.size	simple_nop, .-simple_nop

Notice that, unlike in the previously shown GCC frames, here the fp register is made to point to the saved fp in the stack, rather than the saved lr. This means that an unwinder will (1) wrongly read a next node pointer instead of the function return address that it would expect, (2) will interpret whatever random variable was pushed in the stack immediately before as the next node in the call chain.

I’m surprised I only caught this long after having written an unwinder for GCC frames and having seen it produce sensible flamecharts, but I’m pretty sure this is not just a synthetic problem, since I can see the same problematic assembly by objdump’ing GStreamer libraries.

The deprecated APCS frames (-mapcs-frame) are not affected by this bug.

Getting the top frame is racy during the prologue

At the top of a stack trace you expect to see the currently running function, at the currently running instruction. This is normally obtained by reading the value of the pc register. Then, by looking at the fp register and inspecting the topmost frame record we can see who called this function and traverse the chain.

However, this also means there is a small window of time from the moment a call is done with bl and until the fp register is updated to point to the newly created frame record in which the pc register points to the callee and the caller is recorded in the lr register, but hasn’t been put in a frame record yet.

This results on the caller function seemingly disappearing from the stack trace for a brief moment, as seen here in a instruction-per-instruction simulation:

* 1000(main+0)
* 1004(main+4)
* 1008(main+8)
* 1012(main+12)
* 2000(leaf1+0)
* 2004(leaf1+4)
* 1012(main+12) <- 2008(leaf1+8)
* 1012(main+12) <- 2012(leaf1+12)
* 1012(main+12) <- 2016(leaf1+16)

I can’t think of an alternative call chain traversal algorithm that solves this problem using only the stack and registers. While it seems tempting to also make use of the lr register directly to reconstruct the missing frame, that causes other problems later when the callee returns. I’m also not the first person to point out this problem.

Conclusions

Frame pointers in ARM32 are messy — especially compared to ARM64 or x86_64. A standard ABI for them has only been specified few years ago, with support for it is pretty spotty and divided into incompatible alternatives.

The status of frame pointers for ARM32 in GCC is much worse than I thought when I started working on this: my impression now is that the ostensibly deprecated -mapcs-frame is the only frame pointer ABI GCC handles in a reliable way, and I wouldn’t be too surprised if it turned out that any usefulness of the non-APCS frame pointers in ARM32 with GCC is coincidental rather than intentional. This is certainly not helped by the long lasting specification void in this area.

The default frame pointer ABI of GCC is only reliable during non-leaf functions and as long as Thumb is not used. With that big caveat, they’re still potentially useful. Furthermore, the GCC-specific deficiencies I’ve explained here could be fixed, especially now that there is some official ABI guidance.

Clang has positively surprised me during my investigation. I started looking at it only for comparison, but was happy to see support for the new AAPCS frame pointer ABI in Thumb as well, as well as a general good handling of edge cases.

On top of that, seeing the limitations of frame pointers has also made me a bit curious about the alternative approaches for improved accuracy. Most of those require lookup tables and therefore are generally going to be slower, but how much they can be optimized is still an area of ongoing research.

Additional resources

Analyzing code that uses the stack by hand is error prone, so eventually I wrote a simulator for testing frame pointer ABIs and the algorithms to traverse them.

If you want to look at the compiler sources, you’ll be mostly intersted in:

• clang: llvm/lib/Target/ARM/ARMFrameLowering.cpp. Contains the code for generating prologues and epilogues. Search FramePtr to find the relevant code.
• gcc: gcc/config/arm/arm.c. Look for HARD_FRAME_POINTER_REGNUM. The prologue contents are emitted in arm_expand_prologue().

1. The earliest variant of APCS—referred to as APCS-A—mapped sp to r12 instead of r13. fp was also mapped to r10 instead of r11. This is explained in page 1762 of the RISC OS Programmer Reference Manual (1989), which describes what would later be called APCS-2.
   I’m also leaving aside many complexities of APCS variants, most notably chunked stacks, where the program stack wouldn’t be in contiguous memory, but instead in a linked list of chunks, useful for multi-threading in processors without a Memory Management Unit. ︎

Update on what happened in WebKit in the week from August 17 to August 24.

Another periodical packed with updates on the graphics, multimedia, and tooling fronts. Which sure has to do with preparing for the upcoming 2.54.x release series, that now has release candidates published. Also, do not miss a new stable release being published with fixes for security issues, and make sure to update.

Cross-Port 🐱

Multimedia 🎥

Graphics 🖼️

Releases 📦️

Enhanced Spell Checking on the Web

Here's some interesting stuff about browsers you might not know, and a new API proposal that might interest you...

You probably already know that if you have an editable area in a page with the spellcheck=true attribute set on it, then when a user clicks inside that area, it marks words as spelling errors, and you can style spelling errors with CSS. What you might not know is that feature is carefully designed and implemnted to avoid the huge fingerprinting risk of letting people know what's in your dictionary. A lot of the actual details are left to implementations and this means that not only will each OS/language highlight different spelling errors, but so will different browsers on the same machine, or even different profiles in the same browser!

But, if you think about websites and apps, they are highly likely to be accepting input related to something fairly specific, and highly unlikely to be in a common dictionary. Right? Online education is a great example where you'll find students writing words in different domains that are very common in that domain, but aren't in the common dictionary. Mathematics has its own words, so does chemistry, astronomy, and law. But schools are just one example. There are sites related to the stock market - lots of things there might look like gibberish otherwise. Or your hospital and doctor reports. The sites and apps for wine afficianados, or coffee lovers. There are sites where you submit fan fiction. There are forums where you discuss web development or cloud infrastructure. All of these will have common words that aren't in a general purpose dictionary.

That's a problem because it means we're marking lots of words as misspelled, but they aren't. Too many false positives and we easily overlook the ones that really are missspelled. In fact, in some cases it can even feel like more of an annoyance than a help.

So, there is a proposal currently in HTML for a new API called SpellCheck Custom Dictionary. For v1 the proposal is dead simple. It introduces a new document.spellCheckCustomDictionary with exactly two methods: .addWords(listOfWords) and .removeWords(listOfWords) - where listOfWords is an array of Strings. Those words are simply added to (or removed from) a Set internally which is consulted before consulting any other dictionaries (remember, there can be several) for the lifetime of that document. It doesn't affect other pages or the profile dictionary, or the browser dictionary or the OS dictionary, or any other things.

This is currently implemented by Igalia, with funding from Bloomberg Tech. It is currently available in Chrome Canary behind the experimental web platform features flag, and is scheduled to be shipping in on September 8, 2026 behind the same flag.

Because of its simplicity, it also works with JSON pretty nicely, you can just fetch a JSON array and call .addWords with the result. Of course, in practice you'll probably want to do this as a kind of progressive enhancement, not load the JSON immediately, and maybe handle and report various errors. For a whole lot of developers and use cases, it seems to me that a declarative API would be ideal. But we're not there yet. We have more to figure out with regard to how to express more than exact matches and so on. So, we'll get there... I hope.

In the meantime, however, I've made a tiny, but handy library which speculatively polyfills the basic outline of a declarative form in a safe way. With this, you just include the script and then you can add a <link> tag (or tags) pointing to JSON serializations of arrays of words, and adorn them with a data-spellcheck-dictionary marker-attribute.

<script src="dist/spellcheck-dictionary-loader.min.js"></script>
<link href="terms/domain-terms.json" data-spellcheck-dictionary>
<link href="terms/stocks.json" data-spellcheck-dictionary>

The script will wait until the initial parse is complete and then check for support. If the browser supports the custom dictionary API, it will load and add the words.

There is a /src and /dist version of this and it's very tiny (441 bytes over the wire with Brotli) and I expect it works for most common stuff.

It's "rough" though and not exactly how we'd probably expect a real implementation to work. For example, if you remove the link, it doesn't remove the words. If you change the href, it doesn't do anything, and so on. A real implementation would probably do those things. So, if you're interested in playing with something more complete, I've also included a -complete version of each: src/spellcheck-dictionary-loader-complete.js and dist/spellcheck-dictionary-loader-complete.min.js. The latter is still only 770 bytes over the wire with Brotli, but it's up to you how important that use is to you.

You can get all of the versions from this repo where there is also a start on several domain dictionaries to play with. What I think would be really great is to have some shared dictionaries that we work on together, so feel free to send PRs for additional words, this is really only a mostly generated starting list for demonstration purposes.

Please, let us know what you think - and thanks again to Bloomberg Tech for the funding!

Update on what happened in WebKit in the week from August 10 to August 17.

Following an extra packed periodical, this week we get back to a more regular pace with two nice bugfixes, and a new tool to analyze WebKit performance on Linux!

Cross-Port 🐱

Graphics 🖼️

Update on what happened in WebKit in the week from July 28 to August 10.

Quite a packed pair of weeks this time! The range of updates is big, but some highlights are the handful of Layer-Based SVG Engine updates, performance improvements, and the new WPE APIs. Finally, the Web Engines Hackfest recordings are now published!

Cross-Port 🐱

Multimedia 🎥

Graphics 🖼️

WPE WebKit 📟

Community & Events 🤝

Introduction

The XHCI specification has a optional debugging capabilities that is available via the extended capabilities set. This set could be enabled by vendor to pass along additional information around the hardware - especially information that would help in debugging. In this post, we are going to identify a USB controller (on a target) that supports this debug capability and try to enable/verify it on the host side. At last, we are going to also look at the debug capabilites associated with early boot phase of the kernel

Requirements

• USB 3.1 Superspeed cable A-A or A-C cable
• Target with supported xHCI debug capability
• Host machine

Checking if xHCI debug capability is supported

We need to verify if the target machine has xHCI debug capability supported. We can simply find out by peeking into our sysfs device information:

($) find "/sys/devices" -type f -name "dbc"

If this command returns a device, we can be sure that the xHCI controller is present with debug capability.

Enable the xHCI debug capability on target

($) sudo bash -c "echo enable > /sys/devices/pci0000:00/0000:00:0d.0/dbc"

You should expect the following confirmation via dmesg on the target:

[ 3400.545281] xhci_hcd 0000:04:00.3: DbC connected
[ 3400.905302] xhci_hcd 0000:04:00.3: DbC configured

On the host machine which is connected via the USB 3.1 SuperSpeed cable, we can verify that the USB host controller recognises the debug device. Running dmesg on host:

[110351.933236] usb 4-2: new SuperSpeed USB device number 6 using xhci_hcd
[110351.944701] usb 4-2: New USB device found, idVendor=1d6b, idProduct=0010, bcdDevice= 0.10
[110351.944732] usb 4-2: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[110351.944742] usb 4-2: Product: Linux USB Debug Target
[110351.944750] usb 4-2: Manufacturer: Linux Foundation
[110351.944756] usb 4-2: SerialNumber: 0001
[110351.990401] usbcore: registered new interface driver usb_debug
[110351.990438] usbserial: USB Serial support registered for debug
[110351.990452] usbserial: USB Serial support registered for xhci_dbc
[110351.990502] usb_debug 4-2:1.0: xhci_dbc converter detected
[110351.990786] usb 4-2: xhci_dbc converter now attached to ttyUSB0

You can attach any serial console utility to interact with target device using xHCI debug capabilities.

Linux kernel early boot setup

xHCI is also helpful to output early kernel boot stage logs. This phase is the intialization process just after the bootloaders hands over control. To enable early kernel debug, we need to configure the kernel with proper kconfig and pass debug parameters in the kernel commandline.

Build the kernel with following kconfig enabled:

CONFIG_EARLY_PRINTK_USB_XDBC

And for the kernel commandline, append:

earlyprintk=xdbc

Early boot IO remap

Now that you have had gentle introduction to this debugging capability, we are going briefly mention about fixed boot-time mappings known as fixmaps. The early boot has a special set of operations that are carried out before the standard virtual memory and device drivers are active. In this phase, hardware registers are allowed to be accessed via fixed boot time mapping and page tables.

Generally the xHCI debug capability reside under the extended capabilities given by vendor and can be mapped in fixed boot time mappings. However, we have such devices coming up, that have extended capabilites size much bigger than the fixmaps.

In this case, the PCIe memory range would fail to get IO remapped using early_ioremap() call.

     kernel: xhci_dbc:early_xdbc_parse_parameter: dbgp_num: 0
     kernel: ------------[ cut here ]------------
    ay 19 13:42:57  kernel: WARNING: CPU: 0 PID: 0 at mm/early_ioremap.c:139 __early_ioremap+0xae/0x180
     kernel: Modules linked in:
     kernel: CPU: 0 UID: 0 PID: 0 Comm: swapper Not tainted 6.18.25-valve1-gcab630e7af50 #13 PREEMPT(undef)  4b70872d9de6788b7f2d10dce46ada89e6cd177b
     kernel: RIP: 0010:__early_ioremap+0xae/0x180
     kernel: Code: 60 ba 3f a0 4c 89 ca 48 81 e3 00 f0 ff ff 48 81 e2 00 f0 ff ff 48 29 d3 48 89 14 24 48 89 da 48 c1 ea 0c 89 d5 83 fa 40 76 04 <0f> 0b eb a2 6b c0 c0 4d 89 ce 41 81 e6 ff 0f 00 00 44 8d b8 ff 05
     kernel: RSP: 0000:ffffffff9fa03cb8 EFLAGS: 00010006 ORIG_RAX: 0000000000000000
     kernel: RAX: 0000000000000000 RBX: 0000000000100000 RCX: 0000000000100000
     kernel: RDX: 0000000000000100 RSI: 0000000000100000 RDI: 0000000080200000
     kernel: RBP: 0000000000000100 R08: 0000000000000000 R09: 0000000080200000
     kernel: R10: 0000000000000004 R11: ffffffff9fa03ad0 R12: 8000000000000163
     kernel: R13: 0000000000000000 R14: 0000000080200000 R15: 0000000000000000
     kernel: FS:  0000000000000000(0000) GS:0000000000000000(0000) knlGS:0000000000000000
     kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
     kernel: CR2: ffff888000000413 CR3: 000000020ed02000 CR4: 00000000000000b0
     kernel: Call Trace:
     kernel:  <TASK>
     kernel:  ? early_xdbc_parse_parameter+0x32c/0x360
     kernel:  ? setup_early_printk+0x4f5/0x520
     kernel:  ? do_early_param+0x44/0x70
     kernel:  ? parse_args+0x233/0x420
     kernel:  ? __pfx_do_early_param+0x10/0x10
     kernel:  ? parse_early_options+0x29/0x30
     kernel:  ? __pfx_do_early_param+0x10/0x10
     kernel:  ? parse_early_param+0x64/0xc0
     kernel:  ? setup_arch+0x542/0xbc0
     kernel:  ? _printk+0x6b/0x90
     kernel:  ? start_kernel+0x66/0x9a0
     kernel:  ? x86_64_start_reservations+0x24/0x30
     kernel:  ? x86_64_start_kernel+0xcc/0xd0
     kernel:  ? common_startup_64+0x13e/0x141
     kernel:  </TASK>
     kernel: ---[ end trace 0000000000000000 ]---

And this is the exact place that is triggered at in mm/early_ioremap.c

	/*
	 * Mappings have to fit in the FIX_BTMAP area.
	 */
	nrpages = size >> PAGE_SHIFT;
	if (WARN_ON(nrpages > NR_FIX_BTMAPS))
		return NULL;

We shall call this issue as extra extended capabilities issue for lack of a better work!

Solution to extra extended capbilities for xHCI

Now that we are seeing such hardware in the market where the fixmaps can be a limiting factor, we have to find a solution for it. This has been a focus of my work and after a few iterations, we have this patch for review! Feel free for review comments or follow it for curiousity. There might be few more followups on this particular approach – so stay tuned for that.

Thank you for reading!

Update on what happened in WebKit in the week from July 14 to July 27.

This two-week update includes plenty of changes to the Skia compositor, changes to multimedia support, three blog posts, and assorted improvements.

Cross-Port 🐱

Multimedia 🎥

Graphics 🖼️

Community & Events 🤝

In the past, when you needed to adapt WPE WebKit to a new platform, or integrate it with your own system/application not based on Wayland, you had to develop a specific backend. I wrote two blog posts in the past about this topic (One about the process of creating a new WPE backend and another about using EGLStreams in a WPE backend) and, to be honest, it was not really straightforward.

Since WPE WebKit 2.50, a new system has been designed to replace all this by a more modern and intuitive approach, making a lot easier to integrate WPE WebKit into your application. The new API is called WPE Platform and allows you to define the equivalent of the old backend system into your own executable. At the moment of this post (version 2.52), the WPE Platform is still in development and the official release is foreseen for the next stable version 2.54. Nevertheless, it is already stable enough to start playing with it.

This post is going to explain step by step how you can use WPE WebKit as a web view inside a simple X11 window using EGL with full hardware acceleration and zero-copy of the graphical hardware buffers while avoiding the complexity of writing an external backend. The X11/EGL window itself will be managed by the GLFW library.

N.B. GLFW is only used here as a convenient, cross-platform way of creating a window and an EGL context with a minimal amount of code. It is not the topic of this post. What matters here is the generic contract to follow to implement a custom WPE Platform. It consists basically in three GObject classes and a small set of virtual methods that would look exactly the same if we had chosen SDL, Qt, or directly a raw X11 Window instead of GLFW.

The reference project for this post is blog-the-wpe-platform-api. It implements a complete and minimal WPE Platform taking into account the basic user’s interactions (keyboard, scrolling, mouse, etc…).

The version 0.0 of this code implements the bases to initialize a web view using WPE (it will need a Wayland compositor to run). While the version 1.0 also integrates all the components needed to implement the WPE Platform. From the implementation point of view, the only difference is the usage of a custom WPEDisplay instead of the default one:

// Create and connect our GLFW-backed WPE display
WPEDisplay* display = wpe_glfw_display_new();
GError* error = NULL;
if (!wpe_display_connect(display, &error))
{
    ...
    return EXIT_FAILURE;
}

// Create the WebKitWebView
WebKitWebView* web_view = WEBKIT_WEB_VIEW(g_object_new(WEBKIT_TYPE_WEB_VIEW, "display", display, NULL));
...

Building and running the example project #

If your operating system has a development package for libWPEWebKit-2.0 version 2.52 or above, the easiest way is to install this package from your distribution. Else, you can build WPE WebKit by yourself:

• or by using the webkit-container-sdk reading the instructions provided on the project,
• or by building the library locally.

If you already have WPE WebKit installed, you can go directly to the build of the example project.

Build WPE WebKit locally #

The following instructions are for building the library locally out of any container. You first need to clone the WPE WebKit project, version 2.52 or above:

mkdir wpe-webkit
cd wpe-webkit
git clone --depth 1 -b wpewebkit-2.52.5 https://github.com/WebKit/WebKit.git

Then install clang and all the needed development dependencies by calling: ./WebKit/Tools/wpe/install-dependencies.

Download the build-wpe.sh and set_dev_env.sh scripts and copy them to your working folder containing the WebKit source code. Then execute:

chmod 755 ./build-wpe.sh ./set_dev_env.sh
./build-wpe.sh --configure
./set_dev_env.sh

It will build and install WPE WebKit into ./dist-wpe. The set_dev_env.sh script will set the environment variables to use the files in ./dist-wpe for pkg-config and for the runtime.

Build the example project #

Clone the example project:

git clone https://github.com/neodesys/blog-the-wpe-platform-api.git wpe-glfw-platform

Install the GLFW development dependency (package libglfw3-dev on Ubuntu/Debian).

Configure and build it:

cd wpe-glfw-platform
meson setup build
ninja -C build

You can now run it by calling ./build/wpe-glfw [url].

N.B. In the version 2.52.5 of WPE WebKit, the Skia multithreaded hardware-accelerated compositor is not fully stable with some specific GPUs. In particular, with NVidia graphic cards, it may crash when the allocated surfaces are resized to resolutions bigger than HD. If this is your case, you can disable the Skia hardware-accelerated compositor by setting the environment variable WEBKIT_SKIA_ENABLE_CPU_RENDERING=1. In some cases, you don’t need to disable the whole hardware-acceleration for the compositor. Sometimes just configuring the hardware compositor to use only one thread is enough. You can do that by setting the environment variable WEBKIT_SKIA_PAINTING_THREADS=1.

From a WPE Backend to a WPE Platform #

A WPE Backend sits at the crossroads between the WPEWebProcess, in charge of running the ThreadedCompositor, and the application process, which presents the resulting frames. Both processes must load the same backend shared library, and that library has to implement an IPC layer to move each handle from one process to the other.

graph LR;
    subgraph SA[<b>WPEWebProcess</b>]
        A(ThreadedCompositor)
    end
    subgraph SB[<b>Application Process</b>]
        C(User Application)
    end
    A -->|draw| B([WPE Backend shared library<br/>Loaded by both processes]) --> C

The WPE Platform API removes this shared library and all the IPC burden. WPE WebKit still spawns a WPEWebProcess to run the ThreadedCompositor but the transfer of the rendered frames from the WPEWebProcess to the application process is now handled internally by WPE WebKit itself. As an application developer, you no longer need to implement any IPC: you only receive a ready-to-use WPEBuffer object, backed by a DMA buffer or by shared memory, directly into your application process.

graph LR;
    subgraph SA[<b>WPEWebProcess</b>]
        A(ThreadedCompositor)
    end
    subgraph SB[<b>Application Process</b>]
        B[WPEDisplay<br/>WPEToplevel<br/>WPEView]
        C(User Application)
    end
    A -->|"WPEBuffer (handled internally)"| B --> C

What used to be a shared library exposing five libwpe interfaces is now just three plain GObject classes (WPEDisplay, WPEToplevel and WPEView) that you need to subclass and link directly into your application binary.

The three pillars of a WPE Platform #

• WPEDisplay is the entry point. It owns the connection to the native graphical system (here, an EGL display bootstrapped through GLFW) and acts as a factory for the toplevel window and the web views.
• WPEToplevel is roughly the equivalent of a native window. It owns the actual GLFW window, translates native window events (keyboard, mouse, scroll, focus and resize) into WPEEvent instances, and exposes the usual window operations to the WPE Platform API (resizing, setting the window title and switching to fullscreen).
• WPEView is where the web page is actually drawn. It receives a new WPEBuffer each time the ThreadedCompositor has produced a frame and is responsible for presenting it on screen.

graph TB;
    subgraph SA[<b>WPEGLFWDisplay</b>]
        A["connect(): create the EGL display through GLFW"]
    end
    A -->|create_toplevel| B[<b>WPEGLFWToplevel</b><br/>owns the GLFW window]
    A -->|create_view| C[<b>WPEGLFWView</b><br/>renders one WPEBuffer per frame]
    B -.attach view and broadcast window events.-> C

A single WPEDisplay can create several toplevel windows, and a toplevel window can have several views attached to it (think of tabs sharing one native window). In our example application we only create a single toplevel window with a single view.

Implementing WPEGLFWDisplay: bootstrapping EGL through GLFW #

The WPEGLFWDisplay::connect(…) override is called only once, when wpe_display_connect(...) is invoked from main(). It initializes GLFW, requests an EGL/GLES2 context for every window that will be created afterwards, and creates a tiny hidden bootstrap window used only to force GLFW to set up its EGL connection:

// Request EGL + GLES 2 context for all subsequent window creations
glfwWindowHint(GLFW_CONTEXT_CREATION_API, GLFW_EGL_CONTEXT_API);
glfwWindowHint(GLFW_CLIENT_API, GLFW_OPENGL_ES_API);
glfwWindowHint(GLFW_CONTEXT_VERSION_MAJOR, 2);
glfwWindowHint(GLFW_CONTEXT_VERSION_MINOR, 0);
glfwWindowHint(GLFW_VISIBLE, GLFW_FALSE);

// Create a tiny hidden window so GLFW can initialize its EGL connection
self->init_window = glfwCreateWindow(1, 1, "", NULL, NULL);
...
self->egl_display = glfwGetEGLDisplay();

If we want to be able to transfer the frames with zero-copy, keeping them in the GPU memory, we will need a valid WPEDRMDevice. It is also initialized during the display connection, using the EGL_EXT_device_query extension to fetch the device associated with the current EGL display:

EGLDeviceEXT egl_device = EGL_NO_DEVICE_EXT;
if (eglQueryDisplayAttribEXT(self->egl_display, EGL_DEVICE_EXT, (EGLAttrib*)&egl_device) &&
    egl_device != EGL_NO_DEVICE_EXT)
{
    const char* drm_device = eglQueryDeviceStringEXT(egl_device, EGL_DRM_DEVICE_FILE_EXT);
    const char* drm_render_node = eglQueryDeviceStringEXT(egl_device, EGL_DRM_RENDER_NODE_FILE_EXT);
    self->drm_device = wpe_drm_device_new(drm_device, drm_render_node);
}

If WPEGLFWDisplay::get_drm_device(…) returns NULL, the produced frames will be transferred to the application using shared memory, which implies copying the frames content back and forth between the GPU and the CPU.

Implementing WPEGLFWToplevel: the window and its events #

WPEGLFWToplevel is where the actual GLFW window is created. It is important to manage this creation in the WPEGLFWToplevel::constructed(…) override rather than in the init() function because WPEToplevel::constructed(...) is resetting the toplevel registered dimensions.

static void wpe_glfw_toplevel_constructed(GObject* object)
{
    // It is important to initialize the window in the `constructed` virtual
    // method and not in the `init` method because the parent class
    // (WPETopLevel) resets the toplevel window size in this call.
    G_OBJECT_CLASS(wpe_glfw_toplevel_parent_class)->constructed(object);

    WPEGLFWToplevel* self = WPE_GLFW_TOPLEVEL(object);

    // All window hints have already been configured by the display
    self->window = glfwCreateWindow(DEFAULT_WIDTH, DEFAULT_HEIGHT, "", NULL, NULL);
    ...

    // Communicate the initial window size to the WPETopLevel, so when
    // the WPEView is attached, it can be resized immediately to the correct
    // dimensions.
    wpe_toplevel_resized(WPE_TOPLEVEL(self), DEFAULT_WIDTH, DEFAULT_HEIGHT);
}

Once the window exists, each GLFW events callback (for the keyboard, mouse, window resizing, etc…) is translating the window events into the corresponding WPEEvent and broadcasts those events to every view currently attached to this toplevel window.

The collection and dispatch of the GLFW events themselves are ensured by calling glfwPollEvents(). As this function is managing the events for all the GLFW windows, it is configured in the WPEGLFWDisplay as a Glib source. This way all GLFW window events are collected, dispatched, translated into WPEEvent and broadcasted to each view from the main application thread running the Glib main loop.

The rest of the class is a set of straightforward virtual method overrides mapping WPE window operations onto GLFW calls to allow the WPE Platform API to set the window title, change the toplevel window size or switch to fullscreen.

Implementing WPEGLFWView: turning a WPEBuffer into pixels #

WPEGLFWView is the class doing the actual OpenGL ES drawing. The main interesting override is render_buffer(…), called by WPE WebKit every time a new frame is ready to be presented. The rest of the code is basically some boilerplate used to render a texture on a plane.

The WPEBuffer provided by WPE WebKit can be a wrapper for a DMA buffer allowing to draw the frame without copying it to the main memory, or it can be the wrapper of a classical block of shared memory if the DRM device was not available in WPEGLFWDisplay::get_drm_device(…).

So, when drawing, we first try to get the EGLImage wrapped by the WPEBuffer and, if not available, we fall back to the shared memory:

// Try to import the WPEView content through an EGLImage to allow a
// zero-copy transfer. This is only going to work if the EGLDisplay is
// associated with a valid DRM device returned by
// WPEDisplay::get_drm_device().
gpointer egl_image = wpe_buffer_import_to_egl_image(buffer, NULL);
if (egl_image)
{
    self->glEGLImageTargetTexture2DOES(GL_TEXTURE_2D, (GLeglImageOES)egl_image);
    glUniform1f(self->uniform_swap_rb, 0.f);
}
else
{
    // Else, fall back to the SHM buffer. In this case the WPEView content
    // is copied through the CPU into shared memory.
    GBytes* pixels = wpe_buffer_import_to_pixels(buffer, &shm_err);
    ...
    gconstpointer data = g_bytes_get_data(pixels, NULL);
    ...
    glTexImage2D(GL_TEXTURE_2D, 0, GL_RGBA, (GLsizei)buf_w, (GLsizei)buf_h, 0, GL_RGBA, GL_UNSIGNED_BYTE, data);
    glUniform1f(self->uniform_swap_rb, 1.f);
}

All the resources (the EGLImage or the GBytes pixels array) are held by the WPE WebKit ThreadedCompositor and so, once the drawing is finished, we must signal that the buffer has been rendered and can be recycled:

wpe_view_buffer_rendered(view, buffer);
wpe_view_buffer_released(view, buffer);

The first call (wpe_view_buffer_rendered(...)) triggers the rendering of the next frame, while the second call (wpe_view_buffer_released(...)) informs that the internal EGLImage or GBytes pixels array can be re-used for a future frame content, avoiding the allocation of new buffers for each new frame.

N.B. The drawing loop in the example is very simplified for the purpose of this post because we are not repainting the window content when it is damaged for example. We are only doing the drawing sequentially at one place when receiving a new frame from the web view. In a real application, we may want to keep the current WPEBuffer for intermediate repainting, until receiving the next frame. In this case, we would call wpe_view_buffer_rendered(...) for buffer A once it has been drawn but we would call wpe_view_buffer_released(...) only after receiving buffer B with the next frame content. So, buffer A may be used more than once to repaint the window content like shown in the following sequence diagram.

sequenceDiagram
    participant A as WPEWebProcess
    participant B as Application Process
    activate A
    A ->> A: Render frame 1 in buffer A
    A ->> B: WPEBuffer A
    deactivate A
    activate B
    B ->> B: Draw frame 1 from buffer A
    B ->> A: wpe_view_buffer_rendered(A)
    deactivate B
    activate A
    A ->> A: Render frame 2 in buffer B
    activate B
    B ->> B: Repaint buffer A
    deactivate B
    A ->> B: WPEBuffer B
    deactivate A
    activate B
    B ->> A: wpe_view_buffer_released(A)
    B ->> B: Draw frame 2 from buffer B
    B ->> A: wpe_view_buffer_rendered(B)
    deactivate B
    activate A
    A ->> A: Render frame 3 in buffer A
    A ->> B: WPEBuffer A
    deactivate A
    activate B
    B ->> A: wpe_view_buffer_released(B)
    B ->> B: Draw frame 3 from buffer A
    deactivate B

The rendering in the current example is not optimized either because the ThreadedCompositor must wait for the complete presentation of the current frame with glfwSwapBuffers(...) blocking until the drawing is finished. We can perfectly imagine a multithreaded view where the render_buffer(...) override just posts the current WPEBuffer to a separate drawing thread. This way the WPE WebKit drawing of the next frame and the presentation of the current frame on screen would run in parallel instead of waiting for each other.

Glad to see the Web Engines Hackfest covered in the local tech magazine Código Cero. Of course I’m biased, but I’m very happy with the growth of the event and the really interesting discussions that happen there every year.

See full interview in Galician (PDF version).

Update on what happened in WebKit in the week from June 30 to July 13.

The summer continues with many updates to the new SVG engine (LBSE), improvements to the new Skia-based compositor, some small API additions, and ever-important stable releases with security fixes.

Cross-Port 🐱