Next week Igalia is hosting a new edition of the Web Engines Hackfset in A Coruña.

As last year we’ll be back at Palexco an amazing venue and we have around 100 people registered to participate onsite. You can check the full schedule of the event at the wiki page.

We hope it’s going to be a great week for everyone, and we’re looking forward to the event!

Talks

On Monday 5th there will be five talks:

• JavaScript Modules: Past, Present, and Future by Nicolò Ribaudo
• Inside Kotlin/Wasm (or how your language could benefit from new proposals, like GC, EH, TFR) by Zalim Bashorov
• Status of the WPE & GTK WebKit ports by Žan Doberšek
• Servo 2023 by Delan Azabani
• Ladybird: Building a new browser from scratch by Andreas Kling

I’m really happy about the set of talks, and particularly excited about the possibility to see the presentations about some less known web rendering engines like WPE, Servo and LibWeb. BTW, the talks will be live streamed in the Web Engines Hackfest YouTube channel.

Breakout Sessions

Apart from that we’ll have breakout sessions as usual. But this year, remote participation will be allowed on these sessions; you don’t need to register or anything like that, just join the room on the GitHub issues of each breakout session at the planned time.

More sessions might be scheduled during the event so keep an eye to the hackfest wiki page and issues.

Sponsors

Last, but not least. Thanks to the Web Engines Hackfest sponsors Arm, Google and Igalia; without your support this event won’t be possible.

Last year I talked about the newly added support for Apple’s Visual Format Language in Emeus, which allows to quickly describe layouts using a cross between ASCII art and predicates. For instance, I can use:

H:|-[icon(==256)]-[name_label]-|
H:[surname_label]-|
H:[email_label]-|
H:|-[button(<=icon)]
V:|-[icon(==256)]
V:|-[name_label]-[surname_label]-[email_label]-|
V:[button]-|

and obtain a layout like this one:

Thanks to the contribution of my colleague Martin Abente Lahaye, now Emeus supports extensions to the VFL, namely:

• arithmetic operators for constant and multiplication factors inside predicates, like [button1(button2 * 2 + 16)]
• explicit attribute references, like [button1(button1.height / 2)]

This allows more expressive layout descriptions, like keeping aspect ratios between UI elements, without requiring hitting the code base.

Of course, editing VFL descriptions blindly is not what I consider a fun activity, so I took some time to write a simple, primitive editing tool that lets you visualize a layout expressed through VFL constraints:

Here’s a couple of videos showing it in action:

At some point, this could lead to a new UI tool to lay out widgets inside Builder and/or Glade.

As of now, I consider Emeus in a stable enough state for other people to experiment with it — I’ll probably make a release soon-ish. The Emeus website is up to date, as it is the API reference, and I’m happy to review pull requests and feature requests.

It’s no longer surprising that you can style parent elements using :has() pseudo-class in Chrome. According to the Chrome Platform Status metrics (as of May 1st, 2023), approximately 4% of page loads in Chrome use :has() pseudo-class in their style rules. :has() pseudo class" />:has() pseudo class" width="100%" />

As I mentioned in a previous post, the problems related to :has() pseudo-class can be categorized into two categories:

• Issues to be addressed in testing :has() on an element
• Issues to be addressed in handling style invalidation with :has()

In another post, I introduced how Blink, the rendering engine used in Chrome, addresses the challenges related to testing :has() pseudo-class on an element.

In this post, I will explain how Blink handles style invalidation with the :has() pseudo-class.

Performance and variation

Style invalidation is a process of identifying and marking elements that require their style to be recalculated in response to a mutation in the DOM. The style engine traverses the DOM tree and designates the affected elements for recalculation.

Efficient handling of style invalidation is crucial as it directly impacts meeting the 60fps criterion, which is essential for smooth and responsive rendering. Several performance factors can be considered in this context:

• The quantity of elements requiring traversal
• The overhead associated with verifying the relationship between a DOM mutation and an element
• The number of elements incorrectly marked for recalculation

These factors are affected by how selectors are defined within style rules. Selectors can take various forms and can be combined in infinite ways, leading to a wide range of possibilities. This selector variation greatly adds to the complexity of the issue.

The :has() pseudo-class complicates the issue by introducing additional complexities and enables a wider range of variations in selectors:

• variations in argument relationships
• variations in the position of the :has() pseudo-class (subject/non-subject)
• complexity of logical combinations inside :has()
• various mutation scenarios
  • changes in class or attribute
  • element insertion or removal
  • state change triggered by user action)
  • …

To provide a basic understanding of style invalidation with :has() pseudo-class, this post will focus on a simple scenario: .a:has(.b).

Style invalidation in Blink

Before going through how Chrome handles style invalidation with :has(), it is essential to grasp the two style invalidation approaches in Blink.

‘Invalidation Sets’ approach in Blink

To get a brief understanding of Blink’s ‘Invalidation Sets’ approach, we can refer to design documents such as “CSS Style Invalidation in Blink”, “Invalidation Sets Design Doc.” and “Sibling Invalidation Design Doc”. Additionally, relevant information can be found in the comments within the rule_feature_set.h and invalidation_set.h files.

While it is impractical to cover every intricate detail of Blink’s style invalidation in this post, I will provide some highlights that aid in comprehending the approach.

• To identify elements that need to be invalidated, Blink uses what we call ‘Invalidation sets’ to store meta-data of selectors in style sheets.

  <style>.a .b .c { ... }</style>
  <!--
  .a[>] { .c }  // For an element's class 'a' mutation,
                // find descendants having class 'c'
                // and follow .c[>] invalidation set.
                // - mark the element to be reclaculated.
  .b[>] { .c }  // For an element's class 'b' mutation,
                // find descendants having class 'c'
                // and follow .c[>] invalidation set.
                // - mark the element to be reclaculated.
  .c[>] { $ }   // For an element's class 'c' mutation,
                // mark the element to be reclaculated.
  -->
  
  <div id="mutation_target">
    <div class="b">
      <div class="c"></div>
      <div></div>
    </div>
    <div>
      <div class="c"></div>
      <div></div>
    </div>
  </div>
  
  <script>
    mutation_target.classList.toggle("a");
  </script>
  

• Invalidation sets serve as a mutation filter, allowing the style engine to disregard irrelevant DOM mutations.

  <style>.a .b .c { ... }</style>
  
  <div id="mutation_target">
    <div class="b">
      <div class="c"></div>
      <div></div>
    </div>
    <div>
      <div class="c"></div>
      <div></div>
    </div>
  </div>
  
  <script>
    mutation_target.classList.toggle("d"); // Do not trigger
                                           // any traversal
  </script>
  

• Invalidation sets provide a means to identify a smaller set of elements that are potentially affected by a DOM mutation.

  <style>.a .b .c { ... }</style>
  
  <div id="mutation_target">
    <div class="b">
      <div class="c"></div> <!-- marked -->
      <div></div>
    </div>
    <div>
      <div class="c"></div> <!-- marked -->
      <div></div>
    </div>
  </div>
  
  <script>
    mutation_target.classList.toggle("a");
  </script>
  

• Although the invalidation process is not perfect and may mark some elements unnecessarily for recalculation, it is still significantly more efficient than recalculating everything.

  <style>.a .b .c { ... }</style>
  
  <div id="mutation_target">
    <div class="b">
      <div class="c"></div> <!-- necessary marking -->
      <div></div>
    </div>
    <div>
      <div class="c"></div> <!-- unnecessary marking -->
      <div></div>
    </div>
  </div>
  
  <script>
    mutation_target.classList.toggle("a");
  </script>
  

• The invalidation sets approach aims to identify elements that require invalidation by examining the changed element itself, its descendants, next siblings. This determination is based on the relationship defined by CSS combinators within selectors.

  <style> .a { ... } </style>
  <!--
  .a[>] { $ }   // For an element's class 'a' mutation,
                // mark the element to be reclaculated.
  -->
  
  <style> .b .c { ... } </style>
  <!--
  .b[>] { .c }  // For an element's class 'b' mutation,
                // find descendants having class 'c'
                // and follow .c[>] invalidation set.
                // - mark the element to be reclaculated.
  .c[>] { $ }   // For an element's class 'c' mutation,
                // mark the element to be reclaculated.
  -->
  
  <style> .d ~ .e { ... } </style>
  <!--
  .d[+] { .e }  // For an element's class 'd' mutation,
                // find next siblings having class 'e'
                // and follow .e[>] invalidation set.
                // - mark the element to be reclaculated.
  .e[>] { $ }   // For an element's class 'e' mutation,
                // mark the element to be reclaculated.
  -->
  
  <style> .f ~ .g .h { ... } </style>
  <!--
  .f[+] { .g }  // For an element's class 'f' mutation,
                // find next siblings having class 'g'
                // and follow .g[>] invalidation set.
                // - find descendants having class 'h'
                // - and follow .h[>] invalidation set.
                //   - mark the element to be reclaculated.
  .g[>] { .h }  // For an element's class 'g' mutation,
                // find descendants having class 'h'
                // and mark those to be recalculated.
  .h[>] { $ }   // For an element's class 'h' mutation,
                // mark the element to be reclaculated.
  -->
  

In summary, the ‘Invalidation Sets’ approach involves extracting meta-data from selectors in style rules and creating invalidation sets. These sets enable the skipping of irrelevant mutations, resulting in a smaller set of elements that require style recalculation.

‘Dynamic Restyle Flags’ approach in Blink

While the use of invalidation sets allows for skipping irrelevant mutations, it may not provide significant benefits in certain cases involving pseudo-state mutations. These cases include pseudo-states that are frequently changed in many elements (e.g., :hover) or pseudo-states that require relatively intensive operations to check (e.g., :nth-child()).

To address such scenarios, the Blink style engine implements the ‘Dynamic Restyle Flags’ approach. During style recalculation, the style engine sets these flags on DOM elements while testing selectors. This enables the style engine to determine whether an element is potentially affected by the pseudo-state change.

:hover dynamic restyle flag setting

<style> .a:hover { ... } </style>
<!--
.a[>] { $ }     // For an element's class 'a' mutation,
                // mark the element to be reclaculated.
:hover[>] { $ } // For an element's ':hover' state mutation,
                // mark the element to be reclaculated.
-->

<div> ... </div>
<div id="mutation_target"> ... </div>
<div> ... </div>

During the initial loading, the style engine attempts to match the style rule .a:hover { ... } against all elements. Every matching on each element fails at the first part (.a) of the selector. As a result, the style engine does not set the :hover dynamic restyle flag on any of the <div> elements.

When the mouse pointer hovers over the #mutation_target element after the initial loading, the :hover state of the element changes. Despite the presence of an invalidation set :hover[>] { $ }, the style engine does not trigger the style invalidation step because the element does not have the :hover dynamic restyle flag set.

If we add the a class to the #mutation_target element, it becomes invalidated due to the invalidation set .a[>] { $ }.

During the subsequent style recalculation, the style engine tests :hover selector against #mutation_target. At this point, regardless of the selector testing result, style engine sets the :hover dynamic restyle flag to indicate that the :hover state change of #mutation_target can affect an element’s style.

After the dynamic restyle flag set, the :hover state change on the #mutation_target can trigger the style invalidation.

nth-child() dynamic restyle flag setting

Let’s check how Blink handles style invalidation with :nth-child(). In terms of :nth-child() pseudo-state change, we need to focus on the element insertion and removal operations, as these can impact the nth-child() state of the inserted/removed element’s siblings.

In theory, every element possesses every possible :nth-child() pseudo-state, as they all belong to the DOM tree. Once a node becomes part of the DOM, all :nth-child() states are fixed.

However, maintaining all possible :nth-child() pseudo-states for every DOM node would be inefficient due to these reasons:

• The calculation of :nth-child() is a heavy operation.
• An insertion or removal operation can trigger :nth-child() state change for every next sibling element of the inserted or removed element, even if the affected :nth-child() pseudo-states are not needed.

To address this inefficiency, the style engine only calculates the :nth-child() pseudo-state when testing selectors during the style recalculation process.

The problem arises from the need to optimize restyle performance by minimizing unnecessary element invalidation. To achieve this, the style engine must determine whether the :nth-child() state of next siblings has changed in response to the mutation. However, since the state is not stored within the element nodes themselves, checking for state changes becomes complex and slows down the invalidation process.

To circumvent the need for :nth-child() state checking during the style invalidation step, the style engine introduces the :nth dynamic restyle flags. These flags are set on the parent element of the element being tested for :nth-child() pseudo-class. Their purpose is to indicate that the children of the parent element could be affected by :nth-child state change.

<style> .a:nth-child(2n) { ... } </style>
<!--
.a[>] { $ }         // For an element's class 'a' mutation,
                    // mark the element to be reclaculated.
:nth-child[>] { $ } // For an element's :nth-child state change,
                    // mark the element to be recalculated.
-->

<div id="nth_parent">
  <div></div>
  <div id="insertion_reference"></div>
  <div></div>
</div>

During the initial loading, the style engine attempts to match the style rule .a:nth-child(2n) { ... } to all elements. However, since none of the elements satisfy the first part of the selector(.a), the style engine does not set the :nth dynamic restyle flag on any elements in the DOM tree.

If an element is inserted before #insertion_reference, causing a change in the :nth-child(2n) state for certain elements, the style engine does not perform any invalidation steps because #nth_parent does not have the :nth dynamic restyle flag set.

However, when we add the class value a to the #insertion_reference, the style engine invalidates and recalculates the style of the element. While matching style rules, the style engine tests the :nth-child(2n) pseudo-class on the inserted element and sets the :nth dynamic restyle flag of its parent (#nth_parent).

Once the :nth dynamic restyle flag is set on #nth_parent, subsequent insertions can trigger sibling invalidation. When an element is inserted before #insertion_reference, the style engine marks the inserted element to be recalculated and checks if the parent element has the flag set. If the flag set, the style engine invalidates all next siblings of the inserted element to recalculate their styles.

The interesting aspect of this scenario is that multiple elements are involved in this process:

• The inserted element
• The element that has the flags set
• The elements to be invalidated due to the :nth-child(2n) state change

In summary, the ‘Dynamic Restyle Flags’ approach utilizes flags within element nodes to indicate their relationship to a mutation. By checking these flags, the style engine can skip irrelevant mutations on elements, even if there are corresponding invalidation sets for those mutations.

The style invalidation for .a:has(.b)

As described above, the invalidation sets approach was designed to identify the subject element from descendants, next siblings and next sibling descendants.

<style> .a .b { ... } </style>

<div id="target">
  <div>
    <div class="b"></div>
  </div>
</div>

<script>
target.classList.toggle('a');
</script>

When the class value a is added to the #target, the style engine will search for the .b element among the descendant elements of #target. Once the .b is found, it will be invalidated.

We can change the subject of the relationship between .a and .b by using the :has() pseudo-class:

• In the selector .a .b, the subject is .b.
• In the selector .a:has(.b), the subject is .a.

<style> .a:has(.b) { ... } </style>

<div id="subject" class="a">
  <div id="not_subject">
    <div id="relevant_target"></div>
  </div>
</div>
<div>
  <div id="irrelevant_target"></div>
</div>

<script>
relevant_target.classList.toggle('b');
</script>

In the case of the style rule .a:has(.b) { ... }, the invalidation direction is different from .a .b { ... }. When the class value b is added to the #relevant_target, the style engine needs to traverse the ancestors of the #relevant_target to find the subject element .a.

By addressing the following questions, we can get insights into how Blink ensures efficient performance in style invalidation for the style rule .a:has(.b) { ... }:

1. How does Blink avoid unnecessary ancestor traversal for irrelevant mutations?
2. How does Blink prevent the invalidation of ancestors that are not the subject element?
3. How does Blink avoid unnecessary ancestor traversal for relevant mutations on irrelevant elements?

Extracting meta-data from :has() argument selectors

When extracting meta-data from a selector, it is important to note that only the meta-data from :has() argument selectors has an impact on the :has() state change.

For example:

<style> .a:has(.b) { ... } </style>

<div id="subject" class="a">
  <div>
    <div id="relevant_target"></div>
  </div>
</div>

<script>
relevant_target.classList.toggle('c');
relevant_target.classList.toggle('a');
</script>

In the above scenario, the mutations involving the toggling of class value c and a should not trigger ancestor traversal. This is because the class value a does not affect the :has(.b) state and the class value c is not used in the style rule.

The only class value that influences the :has(.b) state is b, which is a part of the argument selector.

By extracting the meta-data from the argument selector, we can make meta-data sets that serve as mutation filters. These sets enable the style engine to avoid unnecessary ancestor traversal for mutations that do not involve the meta-data contained in the set.

<style>
  .a:has(.b) { ... }
  .c:has(.d ~ .e .f) { ... }
</style>
<!--
 The meta-data set { .b .d .e .f } is extracted from the
 argument selectors. This allows the style engine to
 specifically target mutations that involve changing the
 class values within the set.
-->

In blink, RuleFeatureSet class provides the functionality to StyleEngine class:

class CORE_EXPORT RuleFeatureSet {
  ...
  ValuesInHasArgument classes_in_has_argument_;
  ...
};
...
bool RuleFeatureSet::AddValueOfSimpleSelectorInHasArgument(
    const CSSSelector& selector) {
  if (selector.Match() == CSSSelector::kClass) {
    classes_in_has_argument_.insert(selector.Value());
    return true;
  }
  ...
}
...
bool RuleFeatureSet::NeedsHasInvalidationForClass(
    const AtomicString& class_name) const {
  return classes_in_has_argument_.Contains(class_name);
}
...

Set ‘Affected by :has()’ flag

In order to determine whether an ancestor element is a subject element and needs to be invalidated during ancestor traversal, the style engine sets an ‘Affected by :has()’ dynamic restyle flag.

This flag indicates that the element, when flagged, can be affected by a :has() state change.

In the example scenario

<style> .a:has(.b) { ... } </style>
<!--
The meta-data set { .b } is extracted.
-->

<div id="subject" class="a">
  <div id="not_subject">
    <div id="relevant_target"></div>
  </div>
</div>

<script>
relevant_target.classList.toggle('b');
</script>

When the class value b is added to #relevant_target, the style engine initiates ancestor traversal since b is included in the meta-data set from the argument selector.

During the ancestor traversal, the style engine encounters four ancestors: #not_subject, #subject, <body> and <html>. Among these ancestors, only the #subject is the subject element of the selector .a:has(.b) and needs to have its style recalculated after the mutation.

One possible approach is to invalidate all ancestors, regardless of whether they are subject or not. This approach is simple but inefficient.

Another approach is to extract meta-data, such as the class value a, from the selector and use it as a filter to identify the subject element. This approach can potentially reduce the number of invalidations, but implementing it requires careful consideration of selector variation, such as .a:is(:has(.b)).

There is a simpler approach: adding a dynamic restyle flag that indicates an element is an anchor element of a :has() argument selector (“Anchor element” refers to the element that can be affected by the state change of :has()).

In the case of .a:has(.b) { ... }, the anchor element for :has(.b) is the subject element of .a:has(.b), as :has(.b) is in the subject position. During ancestor traversal, the style engine can invalidate an ancestor element only if the ancestor element has the dynamic restyle flag set.

<style> .a:has(.b) { ... } </style>

<div id="subject" class="a"> <!-- AffectedBySubjectHas flag set -->
  <div id="not_subject">
    <div id="relevant_target"></div>
  </div>
</div>

Similar to the process of setting other dynamic restyle flags, the style engine sets the AffectedBySubjectHas flag when testing selectors. During the initial loading, when the style engine attempts to match the style rule .a:has(.b) { ... } to all elements, it tests the selector :has(.b) on #subject. Since the style engine needs to test a :has() on #subject, it sets the AffectedBySubjectHas flag on the element to indicate that it is the subject element affected by a :has() state change.

The dynamic restyle flags about :has() is in HasInvalidationFlags class and Element class provides the setter/getter functionality to SelectorChecker class and StyleEngine class

struct HasInvalidationFlags {
  ...
  unsigned affected_by_subject_has : 1;
  ...
};
...
bool SelectorChecker::CheckPseudoClass(
    const SelectorCheckingContext& context,
    MatchResult& result) const {
  ...
    case CSSSelector::kPseudoHas:
      if (mode_ == kResolvingStyle) {
        if (context.in_rightmost_compound) {
          element.SetAffectedBySubjectHas();
        } else {
        ...
  ...
}
...
void StyleEngine::InvalidateElementAffectedByHas(
    Element& element,
    bool for_element_affected_by_pseudo_in_has) {
  ...
  if (element.AffectedBySubjectHas()) {
    element.SetNeedsStyleRecalc(
        StyleChangeType::kLocalStyleChange,
        StyleChangeReasonForTracing::Create(
            blink::style_change_reason::kStyleInvalidator));
  }
  ...
}

Set ‘Ancestors affected by :has()’ Flag

The problem with the ‘Affected by :has() flag’ approach is that, the style engine needs to traverse ancestors for a relevant mutation even if there is no ancestor that has the flag set.

<style> .a:has(.b) { ... } </style>
<!--
Extract class value meta-data set { .b }
-->

<div id="subject" class="a"> <!-- AffectedBySubjectHas -->
  <div>
    <div id="relevant_target"></div>
  </div>
</div>
<div id="not_subject">
  <div>
    <div id="irrelevant_target"></div>
  </div>
</div>

When the style engine detects the mutation of adding the class value b, it will traverse ancestors because the class value is in the meta-data set. In the case of the mutation on #relevant_target, the style engine will find #subject during the ancestor traversal and invalidate it. However, in the case of the mutation on #irrelevant_target, the style engine does not need to traverse ancestors since there is no subject element in the ancestors of #irrelevant_target.

To skip the unnecessary ancestor traversal on #irrelevant_target, we can add an additional dynamic restyle flag indicating that the style engine can find a :has() argument’s anchor element in its ancestors.

<style> .a:has(.b) { ... } </style>
<!--
Extract class value meta-data set { .b }
-->

<div id="subject" class="a"> <!-- AffectedBySubjectHas -->
  <div>                               <!-- AncestorsAffectedByHas -->
    <div id="relevant_target"></div>  <!-- AncestorsAffectedByHas -->
  </div>
</div>
<div id="not_subject">
  <div>
    <div id="irrelevant_target"></div>
  </div>
</div>

While matching the style rule .a:has(.b) { ... }, the style engine tests :has(.b) on #subject. To test the pseudo-class, the style engine tries to test its argument :relative-anchor .b on the descendants of #subject. While testing the argument selector, the style engine sets the AncestorsAffectedBySubjectHas flag for each descendant. This flag indicates that the style engine can find a :has() anchor element among the ancestors of the element that have the flag set.

After the matching process, when the style engine detects a mutation on #irrelevant_target, it can skip the ancestor traversal because the element doesn’t have the AncestorsAffectedBySubjectHas flag set.

Similar to the AffectedBySubjectHas flag, this flag is in HasInvalidationFlags class and Element class provides the setter and getter functionality to SelectorChecker class and StyleEngine class.

struct HasInvalidationFlags {
  ...
  unsigned ancestors_or_ancestor_siblings_affected_by_has : 1;
  ...
};
...
void SetAffectedByHasFlagsForElementAtDepth(
    CheckPseudoHasArgumentContext& argument_context,
    Element* element,
    int depth) {
  if (depth > 0) {
    element->SetAncestorsOrAncestorSiblingsAffectedByHas();
  } else {
  ...
}
...
bool SelectorChecker::CheckPseudoHas(
    const SelectorCheckingContext& context,
    MatchResult& result) const {
  ...
    for (CheckPseudoHasArgumentTraversalIterator iterator(
             *has_anchor_element, argument_context);
         !iterator.AtEnd(); ++iterator) {
      if (update_affected_by_has_flags) {
        SetAffectedByHasFlagsForElementAtDepth(
            argument_context,
            iterator.CurrentElement(),
            iterator.CurrentDepth());
      }
      ...
    }
}
...
void StyleEngine::InvalidateAncestorsOrSiblingsAffectedByHas(
    const PseudoHasInvalidationTraversalContext& traversal_context) {
  ...
  Element* element = traversal_context.FirstElement();
  ...
  while (element) {
    traverse_to_parent |=
        element->AncestorsOrAncestorSiblingsAffectedByHas();
    ...
    if (!traverse_to_parent)
      return;

    element = element->parentElement();
    traverse_to_parent = false;
  }
}

Summary

In summary, Blink utilizes several approaches to handle style invalidation for .a:has(.b) { ... }:

1. Mutation filters:
   Meta-data is extracted from the :has() argument selectors to create mutation filters. These filters determine which mutations are relevant for :has() state change.
2. :has() anchor element flag:
   A dynamic restyle flag is set on elements to indicate that the elements can be affected by a :has() state change.
3. :has() anchor element position flag:
   Other dynamic restyle flags are set on elements to indicate that the style engine can find a :has() anchor element by traversing tree from the elements for :has() style invalidation.

To support the variations of :has() style invalidation mentioned above, we can add some modifications to the above basic approaches. Here is the overview:

• Dynamic restyle flags for finding a :has() anchor element from the previous siblings or the previous siblings of ancestors.
• Dynamic restyle flag for non-subject :has() anchor element.
• Extraction of invalidation sets from compound selectors containing :has() and :has() arguments: to handle logical combinations inside :has()
• Adoption of dynamic restyle flags from parent and previous siblings of the inserted element so that the style engine can find a :has() anchor element by traversing the tree from the inserted element.
• Dynamic restyle flag for :hover inside :has(). (The flag indicates that a :hover state change on the element can affect a :has() state change)
• …

It would be better to cover more details in a separated post later.

Similar to the other Blink style invalidation approaches, the :has() style invalidation approach is not perfect, but better than recalculating everything.

I’ll conclude by sharing the test result from a simple performance test.
(Reference: has-restyle-performance.html)

The test compares the restyle performance with and without :has(), and it shows reasonable results:

• Subject :has() invalidation performance in a tree (size: 19531)
  
• Non-subject :has() invalidation performance in a tree (size: 19531)
  

This blog post is a part of the Chrome :has() pseudo-class support project, implemented by Igalia and funded by eye/o.

One of the things I’ve been recently working on at Igalia is the desktop portals implementation, the middleware layer of API for application and toolkit developers that allows sandboxed applications to interact with the host system. Sandboxing technologies like Flatpak and Snap expose the portal D-Bus interfaces inside the sandbox they manage, to handle user-mediated interactions like opening a file that exists outside of the locations available to the sandboxed process, or talking to privileged components like the compositor to obtain a screenshot.

Outside of allowing dynamic permissions for sandboxed applications, portals act as a vendor-neutral API for applications to target when dealing with Linux as an OS; this is mostly helpful for commercial applications that are not tied to a specific desktop environment, but don’t want to re-implement the layer of system integration from the first principles of POSIX primitives.

The architecture of desktop portals has been described pretty well in a blog post by Peter Hutterer, but to recap:

• desktop portals are a series of D-Bus interfaces
• toolkits and applications call methods on those D-Bus interfaces
• there is a user session daemon called xdg-desktop-portal that provides a service for the D-Bus interfaces
• xdg-desktop-portal implements some of those interface directly
• for the interfaces that involve user interaction, or interaction with desktop-specific services, we have separate services that are proxied by xdg-desktop-portal; GNOME has xdg-desktop-portal-gnome, KDE has xdg-desktop-portal-kde; Sway and wlroot-based compositors have xdg-desktop-portal-wlr; and so on, and so forth

There’s also xdg-desktop-portal-gtk, which acts a bit as a reference portal implementation, and a shared desktop portal implementation for a lot of GTK-based environments. Ideally, every desktop environment should have their own desktop portal implementation, so that applications using the portal API can be fully integrated with each desktop’s interface guidelines and specialised services.

One thing that is currently messy is the mechanism by which xdg-desktop-portal finds the portal implementations available on the system, and decides which implementation should be used for a specific interface.

Up until the current stable version of xdg-desktop-portal, the configuration worked this way:

1. each portal implementation (xdg-desktop-portal-gtk, -gnome, -kde, …) ships a ${NAME}.portal file; the file is a simple INI-like desktop entry file with the following keys:
   • DBusName, which contains the service name of the portal, for instance, org.freedesktop.impl.portal.desktop.gnome for the GNOME portals; this name is used by xdg-desktop-portal to launch the portal implementation
   • Interfaces, which contains a list of D-Bus interfaces under the org.freedesktop.impl.portal.* namespace that are implemented by the desktop-specific portal; xdg-desktop-portal will match the portal implementation with the public facing D-Bus interface internally
   • UseIn, which contains the name of the desktop to be matched with the contents of the $XDG_CURRENT_DESKTOP environment variable
2. once xdg-desktop-portal starts, it finds all the .portal files in a well-known location and builds a list of portal implementations currently installed in the system, containing all the interfaces they implement as well as their preferred desktop environment
3. whenever something calls a method on an interface in the org.freedesktop.portal.* namespace, xdg-desktop-portal will check the current desktop using the XDG_CURRENT_DESKTOP environment variable, and check if the portal that has a UseIn key that matches the current desktop
4. once there’s a match, xdg-desktop-portal will activate the portal implementation and proxy the calls made on the org.freedesktop.portal interfaces over to the org.freedesktop.impl.portal ones

This works perfectly fine for the average case of a Linux installation with a single session, using a single desktop environment, and a single desktop portal. Where things get messy is the case where you have multiple sessions on the same system, each with its own desktop and portals, or even no portals whatsoever. In a bad scenario, you may get the wrong desktop portal just because the name sorts before the one you’re interested in, so you get the GTK “reference” portals instead of the KDE-specific ones; in the worst case scenario, you may get a stall when launching an application just because the wrong desktop portal is trying to contact a session service that simply does not exist, and you have to wait 30 seconds for a D-Bus timeout.

The problem is that some desktop portal implementations are shared across desktops, or cover only a limited amount of interfaces; a mandatory list of desktop environments is far too coarse a tool to deal with this. Additionally, xdg-desktop-portal has to have enough fallbacks to ensure that, if it cannot find any implementation for the current desktop, it will proxy to the first implementation it can find in order to give a meaningful answer. Finally, since the supported desktops are shipped by the portal themselves, there’s no way to override this information by packagers, admins, or users.

After iterating over the issue, I ended up writing the support for a new configuration file. Instead of having portals say what kind of desktop environment they require, we have desktop environments saying which portal implementations they prefer. Now, each desktop should ship a ${NAME}-portals.conf INI-like desktop entry file listing each interface, and what kind of desktop portal should be used for it; for instance, the GNOME desktop should ship a gnome-portals.conf configuration file that specifies a default for every interface:

[preferred]
default=gnome

On the other hand, you could have a Foo desktop that relies on the GTK portal for everything, except for specific interfaces that are implemented by the “foo” portal:

[preferred]
default=gtk
org.freedesktop.impl.portal.Screenshot=foo
org.freedesktop.impl.portal.Screencast=foo

You could also disable all portals except for a specific interface (and its dependencies):

[preferred]
default=none
org.freedesktop.impl.portal.Account=gtk
org.freedesktop.impl.portal.FileChooser=gtk
org.freedesktop.impl.portal.Lockdown=gtk
org.freedesktop.impl.portal.Settings=gtk

Or, finally, you could disable all portal implementations:

[preferred]
default=none

A nice side effect of this work is that you can configure your own system, by dropping a portals.conf configuration file inside the XDG_CONFIG_HOME/xdg-desktop-portal directory; this should cover all the cases in which people assemble their desktop out of disparate components.

By having desktop environments (or, in a pinch, the user themselves) owning the kind of portals they require, we can avoid messy configurations in the portal implementations, and clarify the intended behaviour to downstream packagers; at the same time, generic portal implementations can be adopted by multiple environments without necessarily having to know which ones upfront.

In a way, the desktop portals project is trying to fulfill the original mission of freedesktop.org’s Cross-desktop Group: a set of API that are not bound to a single environment, and can be used to define “the Linux desktop” as a platform.

Of course, there’s a lot of work involved in creating a vendor-neutral platform API, especially when it comes to designing both the user and the developer experiences; ideally, more people should be involved in this effort, so if you want to contribute to the Linux ecosystem, this is an area where you can make the difference.

For the last four years I’ve served as a member of the X.Org Foundation Board of Directors, but some days ago I stepped down after my term ended and not having run for re-election.

I started contributing to Mesa in 2014 and joined the amazing freedesktop community. Soon after, I joined the X.Org Foundation as an regular member in order to participate in the elections and get access to some interesting perks (VESA, Khronos Group). You can learn more about what X.Org Foundation does in Ricardo’s blogpost.

But everything changed in 2018. That year, Chema and I organized XDC 2018 in A Coruña, Spain.

The following year, I ran for the yearly election of X.Org Foundation’s board of directors (as it is a two years term, we renew half of the board every year)… and I was elected! It was awesome! Almost immediately, I started coordinating XDC, and looking for organization proposals for the following XDC. I documented my experience organizing XDC 2018 in an attempt to make the job easier for future organizers, reducing the burden that organizing such a conference entails.

In 2021, I was re-elected and everything continued without changes (well, except the pandemic and having our first 2 virtual XDCs: 2020 and 2021).

Unfortunately, my term finished this year… and I did not re-run for election. The reasons were a mix of personal life commitments (having 2 kids change your life completely) and new professional responsibilities. After those changes, I could not contribute as much as I wanted, and that was enough to me to pass the torch and let others contribute to the X.Org Foundation instead. Congratulations to Christopher Michale and Arek Hiler, I’m pretty sure you are going to do great!

Surprisingly enough, I am closing the cycle as it started: organizing X.Org Developers Conference 2023 in A Coruña, Spain from 17th to 19th October 2023.

I leave the board of directors but I won friends and great memories. In case you are interested on participating to the community via the board of directors, prepare your candidancy for next year!

See you in A Coruña!

Last year, the Linux Foundation announced the creation of the Linux Foundation Europe.

The goal of the Linux Foundation Europe is, in a nutshell, to promote Open Source in Europe not only to individuals (via events and courses), but to companies (guidance and hosting projects) and European organizations. However, this effort needs the help of European experts in Open Source.

Thus, the Linux Foundation Europe (LFE) has formed an advisory board called the Linux Foundation Europe Advisory Board (LFEAB), which includes representatives from a cross-section of 20 leading European organizations within the EU, the UK, and beyond. The Advisory Board will play an important role in stewarding Linux Foundation Europe’s growing community, which now spans 100 member organizations from across the European region.

Early this year, I was invited to join the LFEAB as an inaugural member. I would not be in this position without the huge amount of work done by the rest of my colleagues at Igalia since the company was founded in 2001, which has paved the way for us to be one of the landmark consultancies specialized in Open Source, both globally and in Europe.

My presence in the LFEAB will help to share our experience, and help the Linux Foundation Europe to grow and spread Open Source everywhere in Europe.

I’m excited to participate in the Linux Foundation Europe Advisory Board! I and the rest of the LFEAB will be at the Open Source Summit Europe, send me an email if you want to meet me to know more about LFEAB, about Igalia or about how you can contribute more to Open Source.

Happy hacking!

Says Who?

Thoughts on standards and the new baseline effort.

If you've been around me, or my writing, for any time at all, you've probably heard me ask "but what really makes it a standard"?

It is, for example, possible to have words approved in a standards body for which there are, for all intents and purposes, not much in the way of actual implementation or users. Conversely, it is possible to have things that had nothing to do with standards bodies and yet have dozens or hundreds of interoperable implementations and open licenses and are, in reality, much more universal.

At the end of the day, any real judgement kind of involves looking at the reality on the ground. It is a standard... when it is a standard.

I come from Pittsburgh, and in the Steel City, outside the locker room of the Steelers (our amazing football team) it says...

See? It's right there on the wall.

Still, if this makes you uncomfortable, think about an english dictionary. The words in it are simply recognized as standards... because they are.

Where is the invisible line?

At some point, it seems, a thing crosses an invisible line and then it's standard. But only after a gradual process to reach that point. The very end of that process is really rather boring because it's really just stating the obvious.

But where is that magical line when it comes to "the reality on the ground" for web standards?

There's a new effort by the WebDX Community Group called "Baseline" which attempts to idenitfy it and I'm excited because feels like it could be really valuable in several ways.

One that I am most keen on is using it to create a really high signal-to-noise channel for developers to subscribe to. If we define a line right, then something reaching that line is very newsworthy and pretty rare, so we can all afford to pay some attention to it. Imagine an RSS Feed and social media accounts that posted very rarely and only told you this Very Serious Amazing News. Yes, please, give that access to my notifcations! I feel like that would make everything feel a lot less overwhelming and also probably markedly speed real adoption at scale.

The really tricky thing here seems to be, mainly, that it's just really hard to define where that line is in a way everyone agrees with that is actually still useful as more than a kind of historical artifact. That's not to say that such an artifact isn't useful to future learners, but again, by that point this will just be common knowledge.

Stage {x}?

The new Baseline idea has a definition (as of the time of this writing, that is "supported in the last 2 major releases of certain popular browsers (Firefox, Samsung Internet, Safari and Chrome). There was a lot of debate about it before arriving at that. It also currently has a whole slew of issues about why that definition isn't great.

But maybe that's because there are actually several different lines and all of them are interesting in different ways. Think about a progress meter: there can be lots of lines along the way to "done".

The thing I like about the ECMA "Stages" model is that it's easy to visualize like that, and has no clever names: Just 0, 1, 2, 3, 4. Each of those is a 'line' you pass on the way to done. Maybe that kind of model works to discuss here too - we just need more numbers, because those are about ECMA 'done-ness' and not something like what baseline is trying to convey.

Something reaching stage 4 is a huge day, but it doesn't mean the on-the-ground-reality of "all users have support on their devices". In theory, at least, that could still take years to reach.

Conceptually speaking, we could imagine more interesting "lines" (plural) a thing would cross on the way to that day.

For example, the day we learn there is a final engine implementation in experimental builds passing tests is an interesting line. Maybe that's "Stage 5" in my analogy.

The day when it "ships without caveat in the last of the 'steward' browsers" is, in my opinion, a super interesting line (that is, the steward browser that primarily maintains the engine itself). That seems like an especially newsworthy day we should pay attention to because many people will be working on projects that won't ship for a long time, and maybe it's worth considering using it. Maybe that's like "Stage 6" in my analogy.

But that also doesn't mean all of the downstream browsers have released support - that can take time. If you're just starting on a months or year long project, that's probably pretty safe. There's always also a risk that downstream browsers can choose not to adopt a new feature for some reason. Many downstream browsers have considerable support differences with certain APIs (web speech, for example). Not much you can do about that, but what does it mean? Is Web Speech a standard? Is it “baseline”? There are at least tens of millions of browser instances out there that lack support, but a few billion that don't.

Even in the steward's own browser (ie, Chrome, Firefox or Safari), it's not as if releasing a new version is a lightswitch that updates all of the devices in the world. There are lots of things that prevent or delay updating: Corporate controls, OS limits/settings. In some cases, a user interaction is simply required and for whatever reason, people just... don't, for long stretches of time.

So, what should baseline use? Any of them? There probably several useful 'lines' (or stages, if that is easier to imagine) worth discussing. I guess one of those can be called "baseline" - I'm just not really sure where that is in this spectrum. I'm curious for more your thoughts!

Feel free to hit me up on any of the social medias. Tweet, toot or skeet at me if you like. Or, even better: If you're interested in contributing to the thinking around this, it's part of the Web Platform DX Community Group which you can participate in. This work is being tracked and discussed mainly in its Feature-Set Repository on GitHub. Participation is welcome.

I tend to keep quite a lot of notes on the development related (sometimes at work, sometimes not) I do on a week-by-week basis, and thought it might be fun to write up the parts that were public. This may or may not be of wider interest, but it aims to be a useful aide-mémoire for my purposes at least. Weeks with few entries might be due to focusing on downstream work (or perhaps just a less productive week - I am only human!).

Week of 15th May 2023

• Corrected Clang codegen support for half FP types when the zhinx extension is available (D150777.
• Rebased and committed patches to implement MC layer support for the bfloat16 extensions (unblocked now a new PDF was posted in the riscv-bfloat16 repo). Zfbfmin, zvfbfmin, zvfbfwma. Also made a trivial typo fix to the spec.
• Looked at cleaning up the usage of report_fatal_error in the RISC-V backend and also fixed a bug encountered while looking at this. D150674.
• Participated in the ongoing discussion about adding atomics lowering to the RISC-V psABI, including a slightly altered lowering of some primitives in order to allow for forwards compatibility with "table A.7".
• Usual mix of upstream LLVM reviews, and a number of RISC-V psABI or ASM manual reviews.
• LLVM Weekly #489.
• Missed some weeks - busy with EuroLLVM etc.

Week of 17th April 2023

• Still pinging for an updated riscv-bfloat1y spec version that incorporates the fcvt.bf16.s encoding fix.
• Bumped the version of the experimental Zfa RISC-V extension supported by LLVM to 0.2 (D146834). This was very straightforward as after inspecting the spec history, it was clear there were no changes that would impact the compiler.
• Filed a couple of pull requests against the riscv-zacas repo (RISC-V Atomic Compare and Swap extension).
  • #8 made the dependency on the A extension explicit.
  • #7 attempted to explicitly reference the extension for misaligned atomics, though it seems won't be merged. I do feel uncomfortable with RISC-V extensions that can have their semantics changed by other standard extensions without this possibility being called out very explicitly. As I note in the PR, failure to appreciate this might mean that conformance tests written for zacas might fail on a system with zacas_zam. I see a slight parallel to a recent discussion about RISC-V profiles.
• Fixed the canonical ordering used for ISA naming strings in RISCVISAInfo (this will mainly affect the string stored in build attributes). This was fixed in D148615 which built on the pre-committed test case.
• A whole bunch of upstream LLVM reviews. As noted in D148315 I'm thinking we should probably relaxing the ordering rules for ISA strings in -march in order to avoid issues due to spec changes and incompatibilities between GCC and Clang.
• LLVM Weekly #485.

Week of 10th April 2023

• Some days off due to the Easter holidays, so less to report this week.
• Updated RISC-V bfloat16 patches (Zfbfmin, Zvfbfmin, Zvfbfwma), incorporating new fcvt.bf16.s encoding. Also filed an issue about the way in which the dependencies of the vector bfloat16 extensions is specified.
• Blogged about updating the Wren language benchmarks.
• Variety of upstream LLVM reviews.
• LLVM Weekly #484.

Week of 3rd April 2023

• Some days off due to the Easter holidays, so less to report this week.
• Posted MC layer (assembler/disassembler) patches for the bfloat16 extensions: Zfbfmin, Zvfbfmin, Zvfbfwma.
  • Also posted a PR to the riscv-bfloat16 spec to clarify the vector extension dependencies.
  • Pinged on my bug report about the fcvt.bf16.s encoding clash. Once this is resolved, the LLVM MC layer patches can land.
• Updated authorship information for riscv-toolchain-conventions which now has a range of contributors beyond myself.
• Usual mix of upstream LLVM reviews. This included some discussion on changing the shadow call stack register to x3 which spilled into the psABI PR where I suggested some alternate wording.
• LLVM Weekly #483.

• 2023-05-22: Added notes for the week of 15th May 2023.
• 2023-04-24: Added notes for the week of 17th April 2023.
• 2023-04-17: Added notes for the week of 10th April 2023.
• 2023-04-10: Initial publication date.

as usual — long time, no blog.

my only excuse is that I was busy with other things: new job, new office, holidays… you know, whatever happens between coding. :-)

it’s that time of year again, and we’re nearing another Clutter release — this time it’s a special one, though, as it is 1.0.0. which also means that the API will be frozen for the entire duration of the 1.x branch: only additions and deprecations will be allowed. no worries about stagnation, though — we are already planning for 2.0, even though it’ll take at least a couple of years to get there)).

since we’re in the process of finalizing the 1.0 API I thought about writing something about what changed, what was added and what has been removed for good.

let’s start with the Effects API. the Effects were meant to provide a high level API for simple, fire-and-forget animations ((even though people always tried to find new ways to abuse the term “fire-and-forget”)). they were sub-obtimal in the memory management — you had to keep around the EffectTemplate, the effects copied the timelines — and they weren’t extensible — writing your own effect would have been impossible without reimplementing the whole machinery. after the experiments done by Øyvind and myself, and after looking at what the high-level languages provided, I implemented a new implicit animation API — all based around a single object, with the most automagic memory management possible:

/* resize the actor in 250 milliseconds using a cubic easing
 * and attach a callback at the end of the animation
 */
 ClutterAnimation *animation =
   clutter_actor_animate (actor, 250, CLUTTER_EASE_IN_CUBIC,
                          "width", 200,
                          "height", 200,
                          "color", &new_color,
                          NULL);
  g_signal_connect (animation, "completed",
                    G_CALLBACK (on_animation_complete),
                    NULL);

this should make a lot of people happy. the easing modes in particular are the same shared among various animation framworks, like tweener and jQuery

what might make some people slightly less happy is the big API churn that removed both ClutterLabel and ClutterEntry and added ClutterText. the trade-off, though, is clearly in favour of ClutterText, as this is a base class for both editable and non-editable text displays; it supports pointer and keyboard selection, and multi-line as well as single-line editing.

another big changed happened on the low level COGL API, with the introduction of vertex buffers — which allow you to efficiently store arrays of vertex attributes; and, more importantly, with the introduction of the Materials which decouple the drawing operations with the fill operations. it also adds support for multi-texturing, colors and other GL features — on both GL and GLES.

after unifying Label and Entry, we also decided to unify BehaviourPath and BehaviourBspline; after that we added support for creating paths using SVG-like descriptions and for “replaying” a Path on a cairo_t. well, the Cairo integration is also another feature — clutter-cairo has been deprecated and its functionality moved inside ClutterCairoTexture.

one of the last minute additions has been ClutterClone an efficient way to clone generic actors without using FBOs — which also supercedes the CloneTexture actor.

the Pango integration has been extended, and the internal Pango API exposed and officially supported — now you can display text using the Pango renderer and glyphs cache inside your own custom actors without using internal/unstable API.

thanks to Johan Dahlin and Owen Taylor, Clutter now generates GObject-Introspection data at compile time, so that runtime language bindings will be ready as soon as 1.0.0 hits the internets.

finally, there’s a ton of bug fixes in how we use GL, how we render text, how we relayout actors, etc.

hope you’ll have fun with Clutter!

as promised during GUADEC, I’m going to blog a bit more about the development of GSK — and now that I have some code, it’s actually easier to do.

so, let’s start from the top, and speak about GDK.

in April 2008 I was in Berlin, enjoying the city, the company, and good food, and incidentally attending the first GTK+ hackfest. those were the days of Project Ridley, and when the plan for GTK+ 3.0 was to release without deprecated symbols and with all the instance structures sealed.

in the long discussions about the issue of a “blessed” canvas library to be used by GTK app developers and by the GNOME project, we ended up discussing the support of the OpenGL API in GDK and GTK+. the [original bug][bug-opegl] had been opened by Owen about 5 years prior, and while we had ancillary libraries like GtkGLExt and GtkGLArea, the integration was a pretty sore point. the consensus at the end of the hackfest was to provide wrappers around the platform-specific bits of OpenGL inside GDK, enough to create a GL context and bind it to a specific GdkWindow, to let people draw with OpenGL commands at the right time in the drawing cycle of GTK+ widgets. the consensus was also that I would look at the bug, as a person that at the time was dealing with OpenGL inside tool kits for his day job.

well, that didn’t really work out, because cue to 6 years after that hackfest, the bug is still open.

to be fair, the landscape of GTK and GDK has changed a lot since those days. we actually released GTK+ 3.0, and with a lot more features than just deprecations removal; the whole frame cycle is much better, and the paint sequence is reliable and completely different than before. yet, we still have to rely on poorly integrated external libraries to deal with OpenGL.

right after GUADEC, I started hacking on getting the minimal amount of API necessary to create a GL context, and being able to use it to draw on a GTK widget. it turns out that it wasn’t that big of a job to get something on the screen in a semi-reliable way — after all, we already had libraries like GtkGLExt and GtkGLArea living outside of the GTK git repository that did that, even if they had to use deprecated or broken API. the complex part of this work involved being able to draw GL inside the same infrastructure that we currently use for Cairo. we need to be able to synchronise the frame drawing, and we need to be able to blend the contents of the GL area with both content that was drawn before and after, likely with Cairo — otherwise we would not be able to do things like drawing an overlay notification on top of the usual spinning gears, while keeping the background color of the window:

luckily, thanks to Alex, the amount of changes in the internals of GDK was kept to a minimum, and we can enjoy GL rendering running natively on X11 and Wayland, using GLX or EGL respectively.

on top of the low level API, we have a GtkGLArea widget that renders all the GL commands you submit to it, and it behaves like any other GTK+ widgets.

today, Matthias merged the topic branch into master, which means that, barring disastrous regressions, GTK+ 3.16 will finally have native OpenGL support — and we’ll be one step closer to GSK as well.

right now, there’s still some work to do — namely: examples, performance, documentation, porting to MacOS and Windows — but the API is already fairly solid, so we’d all like to get feedback from the users of libraries like GtkGLExt and GtkGLArea, to see what they need or what we missed. feedback is, as usual, best directed at the gtk-devel mailing list, or on the #gtk+ IRC channel.

As I’ve been asked by different people about data from older releases of GTK+, after the previous article on Who Wrote GTK+ 3.18, I ran the git-dm script on every release and generated some more data:

Here you can see the history of the GTK releases, since 2.0.

These numbers are to be taken with a truckload of salt, especially the ones from the 2.x era. During the early 2.x cycle, releases did not follow the GNOME timed release schedule; instead, they were done whenever needed:

Starting with 2.14, we settled to the same cycle as GNOME, as it made releasing GNOME and packaging GTK+ on your favourite distribution a lot easier.

This disparity in the length of the development cycles explains why the 2.12 and 2.14 cycles, which lasted a year, represent an anomaly in terms of contributors (148 and 140, respectively) and in terms of absolute lines changed.

The reduced activity between 2.20 and 2.24.0 is easily attributable to the fact that people were working hard on the 2.90 branch that would become 3.0.

In general, once you adjust by release time, it’s easy to see that the number of contributors is pretty much stable at around 90:

Another interesting data point would be to look at the ecosystem of companies spawned around GTK+ and GNOME, and how it has changed over the years — but that’s part of a larger discussion that would probably take more than a couple of blog posts to unpack.

I guess the larger point is that GTK+ is definitely not dying; it’s pretty much being worked on by the same amount of people — which includes long timers as well as newcomers — as it was during the 2.x cycle.

See the previous article for an introduction to GSK.

In order to render with GSK we need to get acquainted with two classes:

• GskRenderNode, a single element in the rendering tree
• GskRenderer, the object that effectively turns the rendering tree into rendering commands

GskRenderNode

The usual way to put things on the screen involves asking the windowing system to give us a memory region, filling it with something, and then asking the windowing system to present it to the graphics hardware, in the hope that everything ends up on the display. This is pretty much how every windowing system works. The only difference lies in that “filling it with something”.

With Cairo you get a surface that represents that memory region, and a (stateful) drawing context; every time you need to draw you set up your state and emit a series of commands. This happens on every frame, starting from the top level window down into every leaf object. At the end of the frame, the content of the window is swapped with the content of the buffer. Every frame is drawn while we’re traversing the widget tree, and we have no control on the rendering outside of the state of the drawing context.

With GSK we change this process with a small layer of indirection; every widget, from the top level to the leaves, creates a series of render nodes, small objects that each hold the drawing state for their contents. Each node is, at its simplest, a collection of:

• a rectangle, representing the region used to draw the contents
• a transformation matrix, representing the parent-relative set of transformations applied to the contents when drawing
• the contents of the node

Every frame, thus, is composed of a tree of render nodes.

The important thing is that the render tree does not draw anything; it describes what to draw (which can be a rasterization generated using Cairo) and how and where to draw it. The actual drawing is deferred to the GskRenderer instance, and will happen only once the tree has been built.

After the rendering is complete we can discard the render tree. Since the rendering is decoupled from the widget state, the widgets will hold all the state across frames — as they already do. Each GskRenderNode instance is, thus, a very simple instance type instead of a full GObject, whose lifetime is determined by the renderer.

GskRenderer

The renderer is the object that turns a render tree into the actual draw commands. At its most basic, it’s a simple compositor, taking the content of each node and its state and blending it on a rendering surface, which then gets pushed to the windowing system. In practice, it’s a tad more complicated than that.

Each top-level has its own renderer instance, as it requires access to windowing system resources, like a GL context. When the frame is started, the renderer will take a render tree and a drawing context, and will proceed to traverse the render tree in order to translate it into actual render commands.

As we want to offload the rendering and blending to the GPU, the GskRenderer instance you’ll most likely get is one that uses OpenGL to perform the rendering. The GL renderer will take the render tree and convert it into a (mostly flat) list of data structures that represent the state to be pushed on the state machine — the blending mode, the shading program, the textures to sample, and the vertex buffer objects and attributes that describe the rendering. This “translation” stage allows the renderer to decide which render nodes should be used and which should be discarded; it also allows us to create, or recycle, all the needed resources when the frame starts, and minimize the state transitions when doing the actual rendering.

Going from here to there

Widgets provided by GTK will automatically start using render nodes instead of rendering directly to a Cairo context.

There are various fallback code paths in place in the existing code, which means that, luckily, we don’t have to break any existing out of tree widget: they will simply draw themselves (and their children) on an implicit render node. If you want to port your custom widgets or containers, on the other hand, you’ll have to remove the GtkWidget::draw virtual function implementation or signal handler you use, and override the GtkWidget::get_render_node() virtual function instead.

Containers simply need to create a render node for their own background, border, or custom drawing; then they will have to retrieve the render node for each of their children. We’ll provide convenience API for that, so the chances of getting something wrong will be, hopefully, reduced to zero.

Leaf widgets can remain unported a bit longer, unless they are composed of multiple rendering elements, in which case they simply need to create a new render node for each element.

I’ll provide more example of porting widgets in a later article, as soon as the API will have stabilized.

I’m going to talk about the evolution of GTK+ rendering, from its humble origins of X11 graphics contexts, to Cairo, to GSK. If you are interested in this kind of stuff, you can either attend my presentation on Saturday at 11 in the Grace Room, or you can just find me and have a chat.

I’m also going to stick around during the BoF days — especially for the usual GTK+ team meeting, which will be on the 15th.

See you all in Karlsruhe.

In both web and NodeJS worlds, the main runtime for executing program logic is the Javascript runtime. Because of that, a huge number of applications and user interfaces are using it. As any software component, Javascript code uses resources of the system, that are not unlimited. We should be careful when using CPU time, application storage, or memory.

In this blog post we are going to focus on the latter.

Where’s my memory!

Usually the objects allocated by a web page are not a lot, so they do not eat a huge amount of memory for a modern and beefy computer. But we find problems like:

• Oh, but I don’t have a single web page loaded. I like those 40-80 tabs all open for some reason… Well, no, there’s no reason for that! But that’s another topic.
• Many users are not using beefy phones or computers. So using memory has an impact on what they can do.

The user may not be happy with the web application developer implementation choices. And this developer may want to be… more efficient. Do something.

Where’s my memory! The cloud strikes back

Now… Think about the cloud providers. And developers implementing software using NodeJS in the cloud. The contract with the provider may limit the available memory… Or get money depending on the actual usage.

So… An innocent script that takes 10MB, but is run thousands or millions or times for a few seconds. That is expensive!

These developers will need to make their apps… again, more efficient.

A new hope

In performance problems, we usually want to have reliable data of what is happening, and when. Memory problems are no different. We need some observability of the memory usage.

Chromium and NodeJS share their Javascript runtime, V8, and it provides some tools to help with memory investigation.

In this post we are going to focus on the family of tools around a V8 feature named heap snapshot, that allows capturing the memory usage at any time in a Javascript execution context.

About the heap

This is a fast recap on how Javascript heap works, you can skip it if you want

In V8 Javascript runtime, variables, no matter their scope, are allocated on a heap. No matter if it is a number, a string, an object or a function, all of them are stored there. Not only that, in V8 even the code is stored in the heap.

But, in Javascript, memory is freed lazily, with a garbage collection. This means that, when an object is not used anymore, its memory is not immediately disposed. Garbage collector will explore which objects are disposable later, and free them when it is convenient.

How do we know if an object is still used? The idea is simple: objects are used if they can be accessed. To find out which ones, the runtime will take the root objects, and explore recursively all the object references. Any object that has not been found in that exploration can be discarded.

OK, and what is a root object? In a script it can be the objects in the global context. But also Javascript objects referred from native objects.

More details of how the V8 garbage collector works are out of the scope of this post. If you want to learn more, this post should provide a good overview of current implementation: Trash talk: the Orinoco garbage collector.

Heap snapshot: how does it work?

OK, so we know all the Javascript memory allocation goes through the heap. And, as I said, heap snapshot is a tool to investigate memory problems.

The name is quite explicit about how it works. Heap snapshot will stop the Javascript execution, traverse all the heap, analyze it, and dump it in a meaningful format that can be investigated.

What kind of information does it have?

• Which objects are in the heap, and their types.
• How much memory each object takes.
• The references between them, so we can understand which object is keeping another one from being disposed.
• In some of the tools, it can also store the stack trace of the code that allocated that memory.

The format of those snapshots is using JSON, and it can be opened from Chromium developer tools for analysis.

Heap snapshots from Chromium

In the Chromium browser, heap snapshots can be obtained from the Chrome developer tools, accessed through the Inspect right button menu option.

This is common to any browser based in Chromium exposing those developer tools locally or remotely.

Once the developer tools are visible, there is the Memory tab:

We can select three profiling types:

• Heap snapshot: it just captures the heap at the specific moment it is captured.
• Allocation instrumentation on timeline: this records all the allocations over time, in a session, allowing to check the allocation that happened in a specific time range. This is quite expensive, and suitable only for short profiling sessions.
• Allocation sampling: instead of capturing all allocations, this one records them with sampling. Not as accurate as allocation instrumentation, but very lightweight, allowing to give a good approximation for a long profiling session.

In all cases, we will get a profiling report that we can analyze later.

Heap snapshots from NodeJS

Using Chromium dev tools UI

In NodeJS, we can attach the Chrome dev tools passing --inspect through the command line or the NODE_OPTIONS environment variable. This will attach the inspector to NodeJS, but it does not stop execution. The variant --inspect-brk will break on debugger at start of the user script.

How does it work? It will open a port in localhost:9229, and then this can be accessed from Chromium browser URL chrome://inspect. The UI allows users to select which hosts to listen to for Node sessions. The end point can be modified using --inspect=[HOST:]PORT, --inspect-brk=[HOST:]PORT or with the specific command line argument --inspect-port=[HOST:]PORT.

Once you attach dev tools inspector, you can access the Memory tab as in the case of Chromium

There is a problem, though, when we are using NODE_OPTIONS. All instances of NodeJS will take the same parameter, so they will try to attach to the same host and port. And only the first instance will get the port. So it is less useful than we would expect for a session running multiple NodeJS processes (as it can be just running NPM or YARN to run stuff).

Oh, but there are some tricks!:

• If you pass port 0 it will allocate a port (and report it through the console!). So you can inspect any arbitrary session (more details).
• In POSIX systems such as Linux, the inspector will be enabled if the process receives SIGUSR1. This will run in default localhost:9229 unless a different setting is specified with --inspect-port=[HOST:]PORT (more details).

Using command line

Also, there are other ways to obtain heap snapshots directly, without using developer tools UI. NodeJS allows to pass different command line parameters for programming heap snapshot capture/profiling:

• --heapsnapshot-near-heap-limit=N will dump a heap snapshot when the V8 heap is close to its maximum size limit. The N parameter is the number of times it will dump a new snapshot. This is important because, when V8 is reaching the heap limit, it will take measures to free memory through garbage collection, so in a pattern of growing usage we will hit the limit several times.
• --heapsnapshot-signal=SIGNAL will dump heap snapshots every time the NodeJS process gets the UNIX signal SIGNAL.

We can also record a heap profiling session from the start of the process to the end (same kind of profiling we obtain from Dev Tools using Allocation sampling option) using command line option --heap-prof. This will sample continuously the memory allocations, and can be tuned using different command line parameters as documented here.

Analysis of heap snapshots

The scope of this post is about how to capture heap snapshots in different scenarios. But… once you have them… You will want to use that information to actually understand memory usage. Here are some good reads about how to use heap snapshots.

First, from Chrome DevTools documentation:

• Memory terminology: it gives a great tour on how memory is allocated, and what heap snapshots try to represent.
• Fix memory problems: this one provides some examples of how to use different tools in Chromium to understand memory usage, including some heap snapshot and profiling examples.
• View snapshots: a high level view of the different heap snapshot and profiling tools.
• How to Use the Allocation Profiler Tool: this one specific to the allocation profiler.

And then, from NodeJS, you have also a couple of interesting things:

• Memory Diagnostics: some of this has been covered in this post, but still has an example of how to find a memory leak using Comparison.
• Heap snapshot exercise: this is an exercise including a memory leak, that you can hunt with heap snapshot.

Recap

• Memory is a valuable resource that Javascript (both web and NodeJS) application developers may want to profile.
• As usual, when there are resource allocation problems, we need reliable and accurate information about what is happening and when.
• V8 heap snapshots provide such information, integrated with Chromium and NodeJS.

Next

In a follow up post, I will talk about several optimizations we worked on, that make V8 heap snapshot implementation faster. Stay tuned!

Thanks!

This work has been thanks to the sponsorship from Igalia and Bloomberg.

It’s an exciting time for Mesa as its next major release is unveiled this week. Igalia has played an important role in this milestone, with Eric Engestrom managing the release and 11 other Igalians contributing over 110 merge requests. A sample of these contributions are detailed below.

radv: Implement vk.check_status

As part of an effort to enhance the reliability of GPU resets on amdgpu, Tony implemented a GPU reset notification feature in the RADV Vulkan driver. This new function improves the robustness of the RADV driver. The driver can now check if the GPU has been reset by a userspace application, allowing the driver to recover their contexts, exit, or engage in some other appropriate action.

You can read more about Tony’s changes in the link below

• https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22253

turnip: KGSL backend rewrite

With a goal of improving feature parity of the KGSL kernel mode driver with its drm counterpart, Mark has been rewriting the backend for KGSL. These changes leverage the new, common backend Vulkan infrastructure inside Mesa and fix multiple bugs. In addition, they introduce support for importing/exporting sync FDs, pre-signalled fences, and timeline semaphore support.

If you’re interested in taking a deeper dive into Mark’s changes, you can read the following MR:

• https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/21651

turnip: a7xx preparation, transition to C++

Danylo has adopted a significant role for two major changes inside turnip: 1)contributing to the effort to migrate turnip to C++ and 2)supporting the next generation a7xx Adreno GPUs from Qualcomm. A more detailed overview of Danylo’s changes can be found in the linked MRs below:

• https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/21931
• https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22148

v3d/v3dv various fixes & CTS conformance

Igalia maintains the v3d OpenGL driver and v3dv Vulkan drive for broadcom videocore GPUs which can be found on devices such as the Raspberry Pi. Iago, Alex and Juan have combined their expertise to implement multiple fixes for both the v3d gallium driver and the v3dv vulkan driver on the Raspberry Pi. These changes include CPU performance optimizations, support for 16-bit floating point vertex attributes, and raising support in the driver to OpenGL 3.1 level functionality. This Igalian trio has also been addressing fixes for conformance issues raised in the Vulkan 1.3.5 conformance test suite (CTS).

You can dive into some of their Raspberry Pi driver changes here:

• https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22131
• https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/21361
• https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/20787

ci, build system, and cleanup

In addition to managing the 23.1 release, Eric has also implemented many fixes in Mesa’s infrastructure. He has assisted with addressing a number of CI issues within Mesa on various drivers from v3d to panfrost. Eric also dedicated part of his time to general clean-up of the Mesa code (e.g. removing duplicate functions, fixing and improving the meson-based build system, and removing dead code).

If you’re interested in seeing some of his work, check out some of the MRs below:

• https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22410
• https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/21504
• https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/21558
• https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/20180

In the JavaScript world, browser implementations have focused on JIT compilation as a high-performance implementation technique. Recently, new applications of JS, such as on cloud compute and edge compute platforms, have driven interest in non-JIT implementations of the language. For these kinds of use cases, fast startup and predictable performance can make traditional implementation approaches appealing. An example implementation is QuickJS, which compiles JS to a bytecode format and interprets the bytecodes. Another approach is Manuel Serrano's work on Hopc, which is a performant AOT JS compiler that uses Scheme as an intermediate language.

Another direction that is gaining interest is compiling JavaScript to WebAssembly (Wasm). The motivations for this approach are explained very clearly in Lin Clark's article on making JS run fast on Wasm, and some of my Igalia colleagues are spearheading this effort with the SpiderMonkey JS engine in collaboration with Bytecode Alliance partners.

There is still an open question of if we can apply these techniques for AOT compilation of JS to compile JS to Wasm in a practical way (though the componentize-js effort appears to be building up to this using partial evaluation). One way to test this out would be to apply the previously mentioned Hopc compiler. Hopc compiles to Scheme which, via the Bigloo Scheme implementation, compiles to C. Using the standard C toolchain for Wasm (i.e., Emscripten), we can compile that C code to Wasm.

To even attempt this, we would have to first make sure Bigloo Scheme's C output can be compiled to Wasm, which is the main topic of this blog post.

Bigloo on Wasm #

In theory, it's simple to get Bigloo working with Wasm because it can emit C code, which you can compile with the C compiler of your choice. For example, you could use Emscripten's emcc to generate the final executable. In practice, it's more complicated than that.

For one, if you only compile the user Bigloo code to Wasm, it will fail to execute. The binary relies on several libraries that make up the runtime system, which themselves have to be compiled to Wasm in order to link a final executable.

The diagram below illustrates the compilation pipeline. The purple boxes at the lower right are the runtime libraries that need to be linked in.

As a result, we will need to build Bigloo twice: once natively and once to Wasm. The latter build to Wasm will create the needed runtime libraries. This approach is also suggested in the Emscripten documentation for building projects that use self-execution.

I've scripted this approach in a Dockerfile that contains a reproducible setup for reliably compiling Bigloo to Wasm. You can see that starting at line 21 an ordinary native Bigloo is built, with a number of features disabled that won't work well in Wasm. Starting at line 42 a very similar build is done using the emconfigure wrapper that handles the configure script process for Emscripten. The options passed mirror the native build, but with some extra options needed for Wasm.

Like many projects that use Emscripten, some modifications are needed to get Bigloo to compile properly. For example, making C types more precise, backporting Emscripten compatibility patches for included libraries, and adjusting autoconf tests to return a desired result with Emscripten.

1 + 1 = 4? #

There are some tricky details that you need to get right to have working Wasm programs in the end. For example, when I first got a working docker environment to run Bigloo-on-Wasm programs, I got the following result:

$ cat num.scm # this is a Bigloo scheme module
(module num)
(display (+ 1 1)) (newline)
$ /opt/bigloo/bin/bigloo -O3 num.scm -o num.js -cc /emsdk/upstream/emscripten/emcc # compile to wasm, more arguments are needed in practice, this is a simplified example
$ emsdk/node/14.18.2_64bit/bin/node num.js # execute the compiled wasm in nodejs
4

(Side note: if you haven't used a Wasm toolchain before, you may be confused why the output is num.js. Wasm toolchains often produce JS glue code that you use to load the actual Wasm code in a browser/JS engine.)

The Scheme program num.scm is supposed to print the result of "1 + 1". The wasm binary helpfully prints... 4. Other programs that I tried, like printing "hello world", resulted in the IO system trying to print random parts of Wasm's linear memory.

The proximal reason for this failure was that the value tagging code in the Bigloo runtime was being configured incorrectly. If you look at the Bigloo tagging code, you see these cpp definitions:

#  define TAG_SHIFT PTR_ALIGNMENT
#  define TAG_MASK ((1 << PTR_ALIGNMENT) - 1)
#  define TAG_MASKOBJECT TAG_MASK
#  define TAG_MASKPOINTER TAG_MASK

#  define TAG(_v, shift, tag) \
     ((long)(((unsigned long)(_v) << shift) | tag))
#  define UNTAG(_v, shift, tag) \
     ((long)((long)(_v) >> shift))

/* ... */

#  define TAG_INT 0                   /*  integer tagging       ....00 */

The TAG operation is used throughout compiled Bigloo code to tag values into the unityped Scheme value representation. The default tagging scheme (see TAG_SHIFT) is a typical one that depends on the pointer alignment, which depends on the word size (4 bytes on 32-bit, 8 bytes on 64-bit). The PTR_ALIGNMENT definition is defined to be the log base 2 of the word size. This means 2 bits of the value are used for a tag on 32-bit platforms and 3 bits are used on 64-bit platforms.

In the case of numbers, the tag is 0 (TAG_INT above) so a discrepancy in tagging will produce a mis-shifted number value. That's exactly why the num.js program printed 4 above. It's the right answer 2 shifted by one bit.

The reason for that shift is that I was compiling native Bigloo in a 64-bit configuration since that's the architecture of the host machine. Wasm, however, is specified to have a 32-bit address space (unless the memory64 proposal is used). This discrepancy caused values to get shifted with 2 bits in some places, and 3 bits in others during tagging/untagging. After figuring this out, it was relatively easy to compile Bigloo with an i686 toolchain.

Function pointers #

After fixing 32-bit/64-bit discrepancies, simple Bigloo programs would run in a Wasm engine. On more complex examples, however, I was running into function pointer cast errors like the following:

$ emsdk/node/15.14.0_64bit/bin/node bigloo-compiled-program.js
RuntimeError: function signature mismatch
    at <anonymous>:wasm-function[184]:0x3b430
    at <anonymous>:wasm-function[1397]:0x3400bb
    at <anonymous>:wasm-function[505]:0x118046
    at <anonymous>:wasm-function[325]:0xd693e
    at <anonymous>:wasm-function[4224]:0x71dd82
    at Ya (<anonymous>:wasm-function[18267]:0x143d987)
    at ret.<computed> (/test-output.js:1:112711)
    at Object.doRewind (/test-output.js:1:114339)
    at /test-output.js:1:114922
    at /test-output.js:1:99074

This is a documented issue that comes up when porting systems to Emscripten. It's not Emscripten's fault, because oftentimes the programs are relying on undefined behavior (UB) in C.

In particular, CPPReference says the following about function pointer casts:

Any pointer to function can be cast to a pointer to any other function type. If the resulting pointer is converted back to the original type, it compares equal to the original value. If the converted pointer is used to make a function call, the behavior is undefined (unless the function types are compatible)

which means that generally function pointer casts are undefined unless the source and target types are compatible. Bigloo has many cases where function pointers need to be cast. For example, the representation of Scheme procedures contains a field for a C function pointer:

   /* procedure (closures) */
   struct procedure {
      header_t header;    
      union scmobj *(*entry)(); // <-- function pointer for the procedure entrypoint
      union scmobj *(*va_entry)();
      union scmobj *attr;
      int arity;
      union scmobj *obj0;
   } procedure;

The function pointer's C type cannot precisely capture the actual behavior even with a uniform value representation, as the arity of the Scheme procedure needs to be represented. C does not prevent you from calling the function with whatever arity you like though, as you can see in the Gstreamer API code:

obj_t proc = cb->proc; // A Scheme procedure object
switch( cb->arity ) {
  case 0:
     PROCEDURE_ENTRY( proc ) ( proc, BEOA ); // Extract the function entry pointer and call it
     break;
     
  case 1:
     PROCEDURE_ENTRY( proc ) ( proc, convert( cb->args[ 0 ], BTRUE ), BEOA );
     break;

  /* ... */
}

In practice, this kind of UB shouldn't cause problems for a typical C compiler because its output (assembly/machine code) is untyped. What matters is whether the calling convention is followed (it should be fine in Bigloo since these functions uniformly take and return scmobj* pointers).

Since Wasm has a sound static type system, it doesn't allow such loose typing of functions and will crash with a runtime type check if the types do not match. It's possible to work around this by using the EMULATE_FUNCTION_POINTER_CASTS Emscripten option to generate stubs that emulate the cast, but it adds significant overheads as the Emscripten docs note (emphasis mine):

Use EMULATE_FUNCTION_POINTER_CASTS. When you build with -sEMULATE_FUNCTION_POINTER_CASTS, Emscripten emits code to emulate function pointer casts at runtime, adding extra arguments/dropping them/changing their type/adding or dropping a return type/etc. This can add significant runtime overhead, so it is not recommended, but is be worth trying.

The overhead is clear in the generated code, because the option adds dummy function parameters to virtualize calls. Here's an example showing the decompiled Wasm code with emulated casts:

You can see that the function being called with call_indirect has a huge number of arguments (the highlighted (param i64 ...) shows the type of the function being called, and the (i64.const 0) ... above the call are the concrete arguments). There are more than 70 arguments here, and most of them are unused and are present only for the virtualization of the call. This can add up to a huge binary size cost, since there can also be a large number of functions in the Wasm module's indirect function table:

The screenshot from the V8 debugger above is showing the contents of the running module. In this case the module's table (highlighted in red) has over 20,000 function entries. Calls to many of these will incur the emulation overhead. It's not clear to me that there is any good way to avoid this cost without significantly changing the representation of values in Bigloo.

What about Hopc? #

After getting Bigloo to compile to Wasm, I did go back to the initial motivation of this blog post and tried to get Hopc (the JS to Scheme compiler) working in order to have a whole pipeline to compile JS to Wasm. While I was able to get a working build, I had some trouble producing a final Wasm program that could serve as a demo without crashing. At some point, some of the runtime initialization code hits a call_indirect on a null function pointer and crashes.

I suspect that even if I could resolve the crashes, there would be more work needed to make this practical for the use cases I described at the beginning. The best code size I've been able to get for a minimal JS program compiled to Wasm using this pipeline was 29MB, which is rather large. For comparison, Guy Bedford quoted in the JS to Wasm article linked earlier suggested 5-6MB was a reasonable number for a Spidermonkey embedding.

There may be opportunities to reduce this overhead. For example, disabling asyncify and function pointer cast emulation reduces the binary to 9.8MB, albeit a non-working one. Asyncify appears to be required to use the default BDW garbage collector due to the use of emscripten_scan_registers(). There is some discussion of possibly making the asyncify use optional (and possibly using Binaryen's "spill pointers" pass), but it looks like this will take more time to materialize. To avoid the asyncify overhead at the Bigloo level, it could be interesting to look into alternative GC configurations that don't use BDW at all. For the function pointer issue, maybe future changes that leverage the Wasm GC proposal (which has a ref.cast instruction that can cast a function reference to a more precise type) could provide a workaround.

Wrap-up #

It was fun to explore this possibility for AOT compiling JS to Wasm, and more generally it was a good exercise in porting a programming language to Wasm. While there were some tricky problems, the Emscripten tools were good at handling many parts of the process automatically.

I also had to debug a bunch of crashing Wasm code too, and found that the debug support was better than I expected. Passing the debug mode flag -g to emcc helped in getting useful stack traces and in utilizing the Chrome debugger. Though I did wish I had access to a rr-style time travel debugger to continue backwards from a crash site.

With regard to Hopc, I think it could be worth exploring further if the runtime crashes in Wasm can be resolved and if the binary size could be brought down using some of the approaches I suggested above. For the time being though, if you wanted to compile Scheme to Wasm you have an option available now with Bigloo. The Bigloo setup can compile some non-trivial Scheme programs too, such as this demo page that uses Olin Shivers' maze program compiled to Wasm: https://people.igalia.com/atakikawa/wasm/bigloo/maze.html

For another path to use Scheme on Wasm, also check out my colleagues' work to compile Guile to Wasm.

Header image credit: https://www.pexels.com/photo/close-up-photo-of-codes-1089440/

An­nounc­ing decimal128.js, an NPM pack­age for IEEE 754 Dec­i­mal128 float­ing-point dec­i­mal num­bers

I’m hap­py to an­nounce decimal128.js, an NPM pack­age I made for sim­u­lat­ing IEEE 754 Dec­i­mal128 num­bers in JavaScript.

(This is my first NPM pack­age. I made it in Type­Script; it’s my first go at the lan­guage.)

What?

Dec­i­mal128 is an IEEE stan­dard for float­ing-point dec­i­mal num­bers. These num­bers aren’t the bi­na­ry float­ing-point num­bers that you know and love (?), but dec­i­mal num­bers. You know, the kind we learn about be­fore we’re even ten years old. In the bi­na­ry world, things like 0.1 + 0.2 aren’t ex­act­ly* equal to 0.3, and cal­cu­la­tions like 0.7 * 1.05 work out to ex­act­ly 0.735. These kinds of num­bers are what we use when do­ing all sorts of every­day cal­cu­la­tions, es­pe­cial­ly those hav­ing to do with mon­ey.

Dec­i­mal128 en­codes dec­i­mal num­bers into 128 bits. It is a fixed-width en­cod­ing, un­like ar­bi­trary-pre­ci­sion num­bers, which, of course, re­quire an ar­bi­trary amount of space. The en­cod­ing can rep­re­sent of num­bers with up to 34 sig­nif­i­cant dig­its and an ex­po­nent of –6143 to 6144. That is a tru­ly vast amount of space if one keeps the in­tend­ed use cas­es in­volv­ing hu­man-read­able and -writable num­bers (read: mon­ey) in mind.

Why?

I’m work­ing on ex­tend­ing the JavaScript lan­guage with dec­i­mal num­bers (pro­pos­al-dec­i­mal). One of the de­sign de­ci­sions that has to be made there is whether to im­ple­ment ar­bi­trary-pre­ci­sion dec­i­mal num­bers or to im­ple­ment some kind of ap­prox­i­ma­tion there­of, with Dec­i­mal128 be­ing the main con­tender. As far as I could tell, there was no im­ple­men­ta­tion of Dec­i­mal128 in JavaScript, so I made one.

The in­ten­tion isn’t to sup­port the full Dec­i­mal128 stan­dard, nor should one ex­pect to achieve the per­for­mance that, say, a C/C++ li­brary would give you in user­land JavaScript. (To say noth­ing of hav­ing ma­chine-na­tive dec­i­mal in­struc­tions, which is tru­ly ex­ot­ic.) The in­ten­tion is to give JavaScript de­vel­op­ers some­thing that gen­uine­ly strives to ap­prox­i­mate Dec­i­mal128 for JS pro­grams.

In par­tic­u­lar, the hope is that this li­brary of­fers the JS com­mu­ni­ty a chance to get a feel for what Dec­i­mal128 might be like.

How to use

Just do

$ npm install decimal128

and start us­ing the pro­vid­ed Decimal128 class.

Is­sues?

If you find any bugs or would like to re­quest a fea­ture, just open an is­sue and I’ll get on it.

Wolvic Behind-the-Scenes

A little post about power dynamics, frustrations, mistakes and things I’m still learning along the way about the challenges of making The World’s Greatest Cross Device Open Source Browser for XR (or any browser, really).

The next minor (not patch) release of Wolvic will very likely include a fairly major API addition, but if it does, you won’t even find it in the release notes. Still I find myself wanting to write about it.

We are … {drum roll} … very probably ^[1] re-adding support for the WebVR API.

Sigh.

If you're not familiar: The WebVR API was first conceived in 2014 and shipped in Firefox in 2015. It ran as an origin trial in Chrome itself from 2016 until 2018, but was enabled in some downstream browsers. It has been superseded since 2018 by WebXR, which supports a superset of its use cases and addresses some UX issues caused by WebVR. It was removed from Firefox and Wolvic in early 2022.

And now, in spite of all of that, we're very probably ^[1] re-enabling it in Wolvic. For now, at least.

Record scratch
Freeze Frame
Yup, that’s me.
You’re probably wondering how I got here…

Reality is Messy

To reiterate, this technology

• Was supposed to be experimental
• Was superseded five years ago by WebXR, which is under active development and/or partly shipping in multiple browser engines;
• Has an easy replacement (many cases just need a library update).

However, none of that really matters if you’re a “young” browser and your users complain in the app stores, because that really hampers your ability to grow.

And at least currently, too many do.

“user-agent” cuts browsers both ways

I’ve written before (a few times actually) about our trials with the user agent string, and sites just blocking our browser from getting immersive content and why we have to tell A Few Good Lies on behalf of the user. We are the user agent, that’s our job.

But also, if we don’t tell those lies, people will give us bad ratings in the stores.

It makes sense, right? I don’t really blame them. At the end of the day users just don’t care who did The Bad Thing™.

They don’t care that websites shouldn’t try use the UA string like that, or any of that other unseen stuff. What they care about is that from their perspective it just

doesn’t work in your shitty upstart browser. ☆ 1 star.

For a young browser, app store reviews matter a lot^[2]. And reviews will typically skew negative, because happy users don’t really have a good motivation to give you a positive review; they just want to do whatever they do in the browser, not jump over to the app store to say

It got right out of my way. ☆☆☆☆☆ five stars

.

Unhappy users, though: they’ll leave a review, because they’re frustrated and this is the only real chance they have to do anything about it.

On the one hand, that kind of mechanism can be generally helpful: It gives users at least some way to apply pressure on apps to make users happy. Stores also give us ways to reply and ask questions, and to let them know their issue is fixed so that hopefully they can update their review. That happens, and in an ideal world, maybe that’s fine.

At the same time: You might not want to tell everyone exactly what you were doing...

However, users don’t always want to give details on what precise thing they were doing that led to them to getting frustrated enough to leave a bad review.

So, for example, a surprising number of sites which offer streaming video don’t use the <video> element. Often, instead they’ll use some library which rolls its own video streaming support, usually with device feature targeting, based on some lower-level APIs. So, they'll offer you an immersive experience if they think that's possible. If support for those APIs is dropped, it looks like video streaming broke, with no warning. Libraries might have updates available, but sites don't always get updates like that.

So... For example, if your upstart browser appears to have problems with some sexually explicit streaming video sites, those people might be highly motivated to tell you that your browser is shitty or broken.

But also, that’s about all they will tell you.

Again, I get it! I mean, they don’t care why it’s broken, and they don’t want to tell you what they were looking at. I can respect both of those things!

But… that also means it’s very hard to actually do anything about it.

Look, the web is really terribly, mindbogglingly huge place and certainly by far the vast majority of it appears to be working just fine. If we can identify a thing that isn’t working, we’d like to fix it. But we’re unlikely to just stumble across it ourselves.

The worst part about this is that I imagine it will be very hard to get those people to reconsider because it’s impossible to know when to tell them their issue is fixed, and I assume at some point they’ll just stop looking.

Wolvic is adding a new way to report a broken site, optionally anonymously. Maybe that will help this specific problem, but I’m not sure.

Help a browser out...

I hope it’s interesting to come on this adventure of building The World’s Greatest Cross Device Open Source Browser for XR with me.

I hope if you know of any WebVR stuff floating around out there you’ll encourage people to update their libraries already so we can hopefully all get rid of it someday.

If you have Wolvic, maybe you’ll think about popping over and give it an honest rating if you haven’t already - or tell someone to try it out.

Footnotes

1. Assuming that we ultimately judge that this won't actually cause even worse problems after some more research. Part of the reason we were eager to remove it early in the life of Wolvic was that there are so many of exactly the kinds of problems described here with websites or libraries trying to make a good accommodation but making a poor assumption. We suspect that many things were checking for WebVR before WebXR, which would prevent the preferred APIs from being used and further entrench the focus on the deprecated ones!
2. I think about these rating things in stores every time I’m shopping. It feels like they generally not great. They are a dull instrument and full of problems and bad incentives and ways to game. We should be able to do better than that. For example, a small number of reviews from people I have some reason to trust is worth thousands from random internet accounts I have plenty of reason to mistrust. I’m not sure how we build that yet, but maybe one of you can figure it out. Please :) If you don’t mind, and share it with the rest of the class. Thanks.

An easy 360º solution for realtime multimedia communication.

First part is available in Part 1 - The story so far...

Part 2 - The GstWebRTC API #

The new gstwebrtc-api Javascript library developed at Igalia offers a full integration of the GStreamer webrtcsrc/webrtcsink protocols within a web browser or a mobile WebView.

You can easily and transparently interconnect a realtime streaming web application with GStreamer native components. Some interesting use-cases may be:

• ingestion of audio/video from a mobile phone to a backend infrastructure with low latency,
• broadcasting of low-latency media to a web page,
• video conferencing with native desktop clients and web clients,
• mobile video heavy post-processing (like AI) on server-side with direct feedback on client-side,
• mixing videos from different remote sources on the server-side and broadcasting back with low end-to-end latency,
• etc...

A full-featured demo showing how to use the gstwebrtc-api is available on the source code repository in the index.html file.

Following the instructions in the repository’s README, you can launch the demo and interact with GStreamer pipelines.

This demo opens a simple web page that, on one hand, offers to stream out the device webcam as a producer and, on the other hand, automatically detects any new remote producer available on the signalling network and offers to connect to them to consume their WebRTC streams.

The gstwebrtc-api is very simple and articulates around 3 groups of functions:

1 - Management of the connection to the signalling server. #

The client is automatically connected to the signalling server when the DOMContentLoaded event is triggered and, in case of unwanted disconnection, it is automatically reconnected. The signalling server address and reconnection timeout can be configured in the gstWebRTCConfig configuration.

const connectionListener = {
    connected: function(clientId)
    {
        // New connection established with the unique identifier: clientId
    },

    disconnected: function()
    {
        // Disconnected from the signalling server
    }
};

gstWebRTCAPI.registerConnectionListener(connectionListener);
gstWebRTCAPI.unregisterConnectionListener(connectionListener);
gstWebRTCAPI.unregisterAllConnectionListeners();

2 - Management of the producer role. #

Only one producer session can be created at once. While a producer session is active, it is offering the provided MediaStream to the signalling network. The producer session can be stopped at any moment by calling the close() method on it.

const gstWebRTCAPI.SessionState = Object.freeze({
    idle: 0,       // Session has just been created, you need to call start() or connect()
    connecting: 1, // Session is connecting to the other peer
    streaming: 2,  // Session is actively streaming (out for a producer, in for a consumer)
    closed: 3      // Session is definitively closed and object can be recycled
});

class gstWebRTCAPI.ProducerSession
{
    get stream() -> MediaStream; // The stream broadcasted out by this producer
    get state() -> SessionState; // Current producer session state
    start() -> boolean;          // Must be called to start the session after registering eventual events listeners
    close();                     // Closes definitively the session
}

// The provided MediaStream is the stream that will be broadcasted out by the created producer session
gstWebRTCAPI.createProducerSession(stream) -> ProducerSession;

The ProducerSession class inherits from EventTarget and emits the following events:

• error when a network or streaming error occurs,
• stateChanged each time the session state changes,
• closed when the session has been closed,
• clientConsumerAdded each time a new remote consumer connects to the session,
• clientConsumerRemoved each time a remote consumer disconnects from the session.

3 - Management of the consumer role #

You can connect to any remote producer using its unique identifier. You can get the list of remote producers by calling gstWebRTCAPI.getAvailableProducers(). You can also register a listener to stay informed when a producer appears or disappears.

gstWebRTCAPI.Producer = Object.freeze({
    id: {non-empty string},
    meta: {non-null object}
});

gstWebRTCAPI.getAvailableProducers() -> Producer[];

const producersListener = {
    producerAdded: function(producer)
    {
        // A new producer is available (producer is a gstWebRTCAPI.Producer object)
    },

    producerRemoved: function(producer)
    {
        // A producer has been closed (producer is a gstWebRTCAPI.Producer object)
    }
};

gstWebRTCAPI.registerProducersListener(producersListener);
gstWebRTCAPI.unregisterProducersListener(producersListener);
gstWebRTCAPI.unregisterAllProducersListeners();

class gstWebRTCAPI.ConsumerSession
{
    get peerId() -> string;         // Unique identifier of the producer peer to which this session is (or must be)
                                    // connected (always non-empty)
    get sessionId() -> string;      // Unique identifier of this session provided by the signalling server during
                                    // connection (empty until connection succeeds)
    get state() -> SessionState;    // Current consumer session state
    get streams() -> MediaStream[]; // List of remote streams received by this consumer session
    connect() -> boolean;           // Must be called to connect the session after registering eventual events
                                    // listeners
    close();                        // Closes definitively the session
}

gstWebRTCAPI.createConsumerSession(producerId) -> ConsumerSession;

The ConsumerSession class inherits from EventTarget and emits the following events:

• error when a network or streaming error occurs,
• stateChanged each time the session state changes,
• closed when the session has been closed,
• streamsChanged each time the underlying media streams change when media tracks are added or removed.

You can find a reference usage of the gstwebrtc-api in the index.html file.

Conclusion #

By using the new gstwebrtc-api in your web application in conjunction with the new webrtcsrc and webrtcsink GStreamer elements, you can now easily build a 360º realtime audio/video communication product.

You can transparently integrate web components and native components for desktop and/or backend applications.

The web integration is as easy as including a single script with a few straightforward functions (see index.html), whereas you can write your desktop or backend application using any wrapping language (C/C++, Rust, Python, .Net, Java, Javascript, Shell Script, etc.) and/or UI framework (GTK, QT, WxWidgets, Tcl/Tk, Avalonia, etc.) supported by GStreamer.

If you want to integrate multimedia and realtime communication into your project, don't hesitate to contact us at info@igalia.com or on our website: https://www.igalia.com/technology/multimedia. We will be pleased to offer our expertise to unblock your developments, reduce your innovation risks, or to help you add astonishing features to your products.

ESExtractor, how to integrate a dependency-free library to the Khronos CTS #

Since the Vulkan CTS is now able to test and check Vulkan Video support including video decoding, it was necessary to define the kind of media container to be used inside the test cases and the library to extract the necessary encoded data.

In a first attempt, the FFMpeg media toolkit had been chosen to extract the video packets from the A/V ISO base media format chosen as a container reference. This library was provided as a binary package and loaded dynamically at each test run.

As Vulkan video aims to test only video contents, it was not necessary to choose a complex media container, so first all the videos were converted to the elementary stream format for H264 and H265 contents. This is a very elementary format based on MPEG start codes and NAL unit identification.

To avoid an extra multimedia solution integrable only with binaries, a first attempt to replace FFmpeg was, to use GStreamer and an in-house helper library called demuxeres. It was smaller but needed to be a binary still to avoid the glib/gstreamer system dependencies (self contained library). it was a no-go still because the binary package would be awkward to support on the various platforms targetted by the the Khronos CTS.

So at Igalia, we decided to implement a minimal, dependency-free, custom library, written in C++ to be compliant with the Khronos CTS and simple to integrate into any build system.

This library is called ESExtractor

What is ESExtractor ? #

ESExtractor aims to be a simple elementary stream extractor. For the first revision it was able to extract video data from a file in the NAL standard. The first official release was 0.2.4. In this release, only the NAL was supported with both the H264 and H265 streams supported.

As Vulkan Video aims to support more than H264 and H265 including format such as AV1 or VP9, the ESExtractor had to support multiple format. A redesign has been started to support multiple format and is now available in the version 0.3.2.

How ESExtractor works #

A simple C interface is provided to maximise portability and use by other languages.

es_extractor_new #

ESExtractor extractor = es_extractor_new(filePath, "options"));

This interface returns the main object which will give you access to the packets according to the file path and the options given in the arguments. This interface returns a ready to use object where the stream has been initially parsed to determine the kind of video during the object creation.

Then you can check the video format with:

ESEVideoFormat eVideoFormat = es_extractor_video_format(extractor);

or the video codec with:

ESEVideoCodec eVideoCodec = es_extractor_video_codec(extractor);

It supports H264, H265, AV1 and VP9 for now.

es_extractor_read_packet #

This API is the main function to retrieve the available packets from the file. Each time this API is called, the library will return the next available packet according to the format and the specific alignment (ie NAL) and a status to let the application decide what to do next. The packet should be freed using es_extractor_clear_packet.

Has CI powered by github #

To test the library usage, we have implementing a testing framework in addition to a CI infrastructure As github offers a very powerful worklow, we decided to use this platform to test the library on various architectures and platforms. The CI is now configured to release packages for 64 and 32 bits on Linux and Windows.

As usual, if you would like to learn more about Vulkan Video, ESExtractor or any other open multimedia framework, please contact us!

In my unending quest to move the configuration for everything I manage into version control, I was always annoyed with configuring DNS providers. I run or use some services for domains that often require adding various records beyond the "standard" ones. Some examples are using DNS-01 for ACME stuff, configuring mail handling / verification, adding sub-domains and redirects, and so on. And backing this config up was always a chore. Put your hand up if you've thought to back this up before.

I should say at this point that I tend to pay for DNS providers to implement stuff for me. I assume that I don't have time to run an authoritative server, but I could be wrong...

So... the amount of DNS configuration I end up managing is more than I'd like. Not to mention that entering and saving config from DNS providers is absolutely crap. For the few I've used in the past, they all had different web forms for adding/managing records, and the forms all had limitations around acceptable input and how they were presented. I remember one that required splitting the record into two because of some arbitrary limit on number of characters for an input field. Fun, right?

This led me on a massive hunt a few years ago to find something better. That's when I discovered LuaDNS. This is a DNS provider that load configuration, written in Lua, from your git repository. Do I really need to say more??

Ok fine, here are some reasons why I like it:

Example: Configuring a domain for email service

This is an example of configuring a domain to use migadu for email service:

set_defaults(_a)

-- migadu email setup
-- https://admin.migadu.com/domains/109975/dns/instructions
txt(_a, 'hosted-email-verify=aaaaaaaa')
txt(_a, 'v=spf1 include:spf.migadu.com -all')
txt('_dmarc.' .. _a, 'v=DMARC1; p=quarantine;')

mx(_a, 'aspmx1.migadu.com', 0)
mx(_a, 'aspmx2.migadu.com.', 1)

cname('key1._domainkey.' .. _a, 'key1.' .. _a .. '._domainkey.migadu.com.')
cname('key2._domainkey.' .. _a, 'key2.' .. _a .. '._domainkey.migadu.com.')
cname('key3._domainkey.' .. _a, 'key3.' .. _a .. '._domainkey.migadu.com.')

srv('_submissions._tcp.' .. _a, 'smtp.migadu.com', 465)
srv('_imaps._tcp.' .. _a, 'imap.migadu.com', 993)

The set_defaults(_a) at the top is a template I created. It does some configuration that is applicable for all domains, like setting 'www' CNAME <domain>. _a is a placeholder for the domain.

Yes, this config snippet could itself be a template, which I could then use like this in a domain config file: set_migadu(_a).

Version control

LuaDNS can be triggered by a webhook to load configuration from a git repo. This means that all of the Lua config written can be in git, with branches, or whatever you want. And pushed anywhere, triggering automation at LuaDNS to pull and deploy it.

API

This provider has an API, that many ACME clients support for DNS-01, but I use it too for updating records with changed IP addresses. Yeah, I know many providers have this too, but none(?) of them have the other benefits above.

Oh, there is one more benefit: I can't afford to lose any more hair :D

For anyone (rightfully) wondering: I wasn't paid or anything by LuaDNS for this, I'm just a happy paying customer.

Wren is a "small, fast, class-based, concurrent scripting language", originally designed by Bob Nystrom (who you might recognise as the author of Game Programming Patterns and Crafting Interpreters. It's a really fun language to study - the implementation is compact and easily readable, and although class-based languages aren't considered very hip these days there's a real elegance to its design. I saw Wren's performance page hadn't been updated for a very long time, and especially given the recent upstream interpreter performance work on Python, was interested in seeing how performance on these microbencharks has changed. Hence this quick post to share some new numbers.

New results

To cut to the chase, here are the results I get running the same set of benchmarks across a collection of Python, Ruby, and Lua versions (those available in current Arch Linux).

Method Call:

Delta Blue:

Binary Trees:

Recursive Fibonacci:

I've used essentially the same presentation and methodology as in the original benchmark, partly to save time pondering the optimal approach, partly so I can redirect any critiques to the original author (sorry Bob!). Benchmarks do not measure interpreter startup time, and each benchmark is run ten times with the median used (thermal throttling could potentially mean this isn't the best methodology, but changing the number of test repetitions to e.g. 1000 seems to have little effect).

The tests were run on a machine with an AMD Ryzen 9 5950X processor. wren 0.4 as of commit c2a75f1 was used as well as the following Arch Linux packages:

• lua52-5.2.4-5
• lua53-5.3.6-1
• lua-5.4.4-3,
• luajit-2.1.0.beta3.r471.g505e2c03-1
• mruby-3.1.0-1
• python-3.10.10-1
• python-3.11.3-1 (taken from Arch Linux's staging repo)
• ruby2.7-2.7.7-1
• ruby-3.0.5-1

The Python 3.10 and 3.11 packages were compiled with the same GCC version (12.2.1 according to python -VV), though this won't necessarily be true for all other packages (e.g. the lua52 and lua53 packages are several years old so will have been built an older GCC).

I've submitted a pull request to update the Wren performance page.

Old results

The following results are copied from the Wren performance page (archive.org link ease of comparison. They were run on a MacBook Pro 2.3GHz Intel Core i7 with Lua 5.2.3, LuaJIT 2.0.2, Python 2.7.5, Python 3.3.4, ruby 2.0.0p247.

Method Call:

DeltaBlue:

Binary Trees:

Recursive Fibonacci:

Observations

A few takeaways:

• LuaJIT's bytecode interpreter remains incredibly fast (though see this blog post for a methodology to produce an even faster interpreter).
• The performance improvements in Python 3.11 were well documented and are very visible on this set of microbenchparks.
• I was more surprised by the performance jump with Lua 5.4, especially as the release notes give few hints of performance improvements that would be reflected in these microbenchmarks. The LWN article about the Lua 5.4 release however did note improved performance on a range of benchmarks.
• Wren remains speedy (for these workloads at least), but engineering work on other interpreters has narrowed that gap for some of these benchmarks.
• I haven't taken the time to compare the January 2015 version of Wren used for the benchmarks vs present-day Wren 0.4. It would be interesting to explore that though.
• A tiny number of microbenchmarks have been used in this performance test. It wouldn't be wise to draw general conclusions - this is just a bit of fun.

Appendix: Benchmark script

Health warning: this is incredibly quick and dirty (especially the repeated switching between the python packages to allow testing both 3.10 and 3.11):

#!/usr/bin/env python3

# Copyright Muxup contributors.
# Distributed under the terms of the MIT license, see LICENSE for details.
# SPDX-License-Identifier: MIT

import statistics
import subprocess

out = open("out.md", "w", encoding="utf-8")


def run_single_bench(bench_name, bench_file, runner_name):
    bench_file = "./test/benchmark/" + bench_file
    if runner_name == "lua5.2":
        bench_file += ".lua"
        cmdline = ["lua5.2", bench_file]
    elif runner_name == "lua5.3":
        bench_file += ".lua"
        cmdline = ["lua5.3", bench_file]
    elif runner_name == "lua5.4":
        bench_file += ".lua"
        cmdline = ["lua5.4", bench_file]
    elif runner_name == "luajit2.1 -joff":
        bench_file += ".lua"
        cmdline = ["luajit", "-joff", bench_file]
    elif runner_name == "mruby":
        bench_file += ".rb"
        cmdline = ["mruby", bench_file]
    elif runner_name == "python3.10":
        bench_file += ".py"
        subprocess.run(
            [
                "sudo",
                "pacman",
                "-U",
                "--noconfirm",
                "/var/cache/pacman/pkg/python-3.10.10-1-x86_64.pkg.tar.zst",
            ],
            check=True,
        )
        cmdline = ["python", bench_file]
    elif runner_name == "python3.11":
        bench_file += ".py"
        subprocess.run(
            [
                "sudo",
                "pacman",
                "-U",
                "--noconfirm",
                "/var/cache/pacman/pkg/python-3.11.3-1-x86_64.pkg.tar.zst",
            ],
            check=True,
        )
        cmdline = ["python", bench_file]
    elif runner_name == "ruby2.7":
        bench_file += ".rb"
        cmdline = ["ruby-2.7", bench_file]
    elif runner_name == "ruby3.0":
        bench_file += ".rb"
        cmdline = ["ruby", bench_file]
    elif runner_name == "wren0.4":
        bench_file += ".wren"
        cmdline = ["./bin/wren_test", bench_file]
    else:
        raise SystemExit("Unrecognised runner")

    times = []
    for _ in range(10):
        bench_out = subprocess.run(
            cmdline, capture_output=True, check=True, encoding="utf-8"
        ).stdout
        times.append(float(bench_out.split(": ")[-1].strip()))
    return statistics.median(times)


def do_bench(name, file_base, runners):
    results = {}
    for runner in runners:
        results[runner] = run_single_bench(name, file_base, runner)
    results = dict(sorted(results.items(), key=lambda kv: kv[1]))
    longest_result = max(results.values())
    out.write(f"**{name}**:\n")
    out.write('<table class="chart">\n')
    for runner, result in results.items():
        percent = round((result / longest_result) * 100)
        out.write(
            f"""\
  <tr>
    <th>{runner}</th><td><div class="chart-bar" style="width: {percent}%;">{result:.3f}s&nbsp;</div></td>
  </tr>\n"""
        )
    out.write("</table>\n\n")


all_runners = [
    "lua5.2",
    "lua5.3",
    "lua5.4",
    "luajit2.1 -joff",
    "mruby",
    "python3.10",
    "python3.11",
    "ruby2.7",
    "ruby3.0",
    "wren0.4",
]
do_bench("Method Call", "method_call", all_runners)
do_bench("Delta Blue", "delta_blue", ["python3.10", "python3.11", "wren0.4"])
do_bench("Binary Trees", "binary_trees", all_runners)
do_bench("Recursive Fibonacci", "fib", all_runners)
print("Output written to out.md")

An easy 360º solution for realtime multimedia communication.

Part 1 - The story so far... #

It's been a few years that we've been able to communicate in realtime from one web browser to another using the WebRTC protocol. The same protocol also allows broadcasting or ingestion of multimedia streams with a very low latency, in general less than half a second. All web browsers have started integrating the protocol from 2013/2014 and today, in 2023, we have a pretty stable and efficient support everywhere.

The GStreamer multimedia framework has also started integrating WebRTC from 2017 through the webrtcbin plugin. Using this plugin you can perfectly connect to a web browser and stream audio and video in realtime. Unfortunately, webrtcbin is a low-level component. It implements the peer-to-peer connection handshake (using ICE and external STUN servers), packets rerouting if direct connection is not possible (using external TURN servers) and then maintains the underlying RTP session which transports the actual audio and video data.

To build a 100% working product you need to write a lot of code above webrtcbin. Not only do you have to design your own signalling protocol but also implement a signalling server, take care of packet loss and retransmission, manage network congestion and adapt the encoding bitrates to maintain an acceptable user experience over networks of different qualities.

With all that in mind, in 2021 Mathieu Duponchelle started implementing a new GStreamer element called webrtcsink. This element, based on webrtcbin and written in Rust, allows to produce a WebRTC stream and to maintain the underlying connections to multiple remote peers with retransmission of lost packets, control of the network congestion and adaptative encoding bitrates. The webrtcsink project comes with a signalling server that helps normalizing communication between peers.

N.B. as far as signalling normalization is concerned, it is interesting to point out the WHIP protocol that is exclusively used for media ingestion.

Since then, at Igalia, we continuously worked to improve webrtcsink. Last year, in particular, Thibault Saunier has implemented the complete Google Congestion Control Algorithm from its RFC. And recently he has ported the whole project to gst-plugins-rs in order to make the plugin available with the official GStreamer distribution and foster a broader collaboration from the GStreamer community.

So far, webrtcsink is the best looking starting point for building a 360º solution including bi-directional and realtime communications between multiple peers using transparently web browsers and/or native code.

Improvements #

The webrtcsink element is dedicated to broadcast a WebRTC stream to multiple remote clients (this is a sink as its name suggests), and the original signalling server is designed with this objective in mind.

For one of our clients, we needed to be able to also consume a remote WebRTC stream and interact with web browsers and mobile WebViews on Android and iOS.

For that we decided at Igalia to:

• Improve the signalling protocol allowing:
  • to use a single WebSocket connection (and single ID) per connected peer,
  • to endorse multiple roles at the same time (listener, producer and consumer),
  • to have producer and consumer sessions isolated and independent.
• Develop a webrtcsrc element dedicated to consume WebRTC streams produced by the webrtcsink element (or from a web browser).
• Develop gstwebrtc-api, a Javascript API allowing developers to communicate transparently with webrtcsink and webrtcsrc elements from a web browser.

Signalling #

The signalling protocol relies on the same original concepts but with the improvements listed above.

Each peer connects to the signalling server using a WebSocket (can be secured over SSL/TLS) and receives a unique identifier used for further commands. This unique identifier remains until the WebSocket connection is closed.

By default, all peers inherit the role of consumer and can connect to a remote WebRTC stream to consume. To do so, a peer needs to create a session with a remote producer peer. This session has its own unique identifier and is in charge of exchanging SDP and ICE messages between peers until the WebRTC link can be established.

A peer can explicitly require to become a producer, in which case it is announced as this to all other connected peers and gets ready to receive WebRTC connection requests from remote consumer peers.

To finish with, a peer can also explicitly require to become a listener, in which case it will receive a message each time a new producer appears on the signalling network or disappears. The list of currently available producers can also be required independently of the peer roles.

The webrtcsink element always registers as a producer whereas the webrtcsrc element is a consumer. The gstwebrtc-api allows to activate and desactivate the producer mode from the API.

The novelty here is that, from now on, it is possible to activate and deactivate the listener and producer roles without having to open several WebSocket connections or to disconnect from the signalling server. You can also create more than one consumer sessions but you are still limited to one producer session per peer.

To sum-up: if you are creating a web application you can connect to as many remote streams as you want but you can only produce one unique stream (which can have several video and audio tracks). If you are creating a native application with GStreamer, you can use as many webrtcsrc elements as you want to connect to several remote streams and one webrtcsink element to produce one WebRTC stream.

With these combinations you can easily create low-latency streaming applications, media ingestion tools or any kind of video conferencing software, among other examples.

Webrtcsrc #

The new webrtcsrc element developed in Rust by Thibault Saunier offers a full-featured WebRTC consumer. The element connects to the signalling server and manages automatically the WebRTC session with a remote peer identified by its unique identifier.

It also registers the gstwebrtc(s):// schemes to be used directly with uridecodebin, playbin or playbin3 elements. Connecting to a remote WebRTC stream has never been so easy:

gst-launch-1.0 playbin uri=gstwebrtc://127.0.0.1:8443?peer-id=e54e5d6b-f597-4e8f-bc96-2cc3765b6567

or directly:

gst-play-1.0 gstwebrtc://127.0.0.1:8443?peer-id=e54e5d6b-f597-4e8f-bc96-2cc3765b6567

The structure of a webrtcsrc URI is as follows:

• gstwebrtc:// for a non-secured WebSocket connection to the signalling server
• or gstwebrtcs:// for a secured connection over SSL/TLS.
• The signalling server address (hostname or IP address and port).
• The remote producer peer to connect to specified with ?peer-id=[uuid].

The webrtcsrc element also provides access to its internal signaller object, so you can communicate with the signalling server and, for example, listen to all producers created on the signalling network without needing to implement the signalling protocol by yourself. The following example shows how to create an instance of the signaller to receive information about new producers.

class Application;
extern Application* myApp;

GstElement* elem = gst_element_factory_make("webrtcsrc", nullptr);
if (elem)
{
    GObject* signaller = nullptr;
    g_object_get(elem, "signaller", &signaller, nullptr);
    gst_object_unref(elem);

    if (signaller)
    {
        GObject* listener = G_OBJECT(g_object_new(G_OBJECT_TYPE(signaller),
                                                  "uri", "ws://127.0.0.1:8443",
                                                  "role", "listener", nullptr));
        g_object_unref(signaller);

        if (listener)
        {
            g_signal_connect_swapped(listener, "error", G_CALLBACK(+[](Application* app, const char* error) {
                // Manage errors...
            }), myApp);

            g_signal_connect_swapped(listener, "producer-added",
                G_CALLBACK(+[](Application* app, const char* producerId, const GstStructure* producerMeta) {
                    // Manage a new producer...
                    // You can, for example, create a new pipeline/branch using a webrtcsrc
                    // element and the producerId value to consume the new remote stream
                }), myApp);

            g_signal_connect_swapped(listener, "producer-removed",
                G_CALLBACK(+[](Application* app, const char* producerId, const GstStructure* producerMeta) {
                    // Cleanup a removed producer...
                    // Any local consumer pipeline/branch previously created
                    // with a webrtcsrc element will receive an EOS event
                }), myApp);
        }
    }
}

I’ve been a bit over a month now on my new 14” MacBook Pro, and I have complaints.  Not about the hardware, which is solid yet lightweight, super-quiet yet incredibly fast and powerful, long-lived on battery, and decent enough under the fingertips.  Plus, all the keyboard keys Just Work, unlike the MBP it replaced!  So that’s nice.

No, my complaints are entirely about the user environment.  At first I thought this was because I skipped directly from OS X 10.14 to macOS 13, and simply wasn’t used to How The Kids Do Things These Days®, but apparently I would’ve felt the same even if I’d kept current with OS updates.  So I’m going to gripe here in hopes someone who knows more than me will have recommendations to ameliorate my annoyance.

DragThing Dismay

This isn’t on Apple, but still, it’s a huge loss for me.  I know I already complained about the lack of DragThing, but I really, really do miss what it did for me.  You never know what you’ve got ’til it’s gone, right?  But let me be clear about exactly what it did for me, which so far as I can tell no macOS application does, nor does macOS itself.

The way I used DragThing was to have a long shelf down the right side of my monitor containing small-but-recognizable icons representing my most-used folders (home directory, Downloads, Documents, Applications, a few other folders) and a number of applications.  It stayed there all the time, and the icons were always there whether or not the application was running.

When I launched, say, Firefox, then there would be a little indicator next to its application icon in DragThing to indicate it was running.  When I quit Firefox, the indicator went away but the Firefox icon stayed.  And also, if I launched an application that wasn’t in the DragThing shelf, it did not add an icon for that application to the shelf. (I used the Dock at the bottom of the screen to show me that.)

There are super-powered application switchers available for macOS, but as far as I’ve seen, they only list the applications actually running.  Launch an application, its icon is added.  Quit an application, its icon disappears.  None of these switchers let me keep persistent static one-click shortcuts to launch a variety of applications and open commonly-used folders.

Dock Folder Disgruntlement

Now I’m on to macOS itself.  Given the previous problem, the Dock is the only thing available to me, and I have gripes about it.  One of the bigger ones is rooted in folders kept on the Dock, to the right of the bar that divides them from the application icons.  When I click on them, I get a popup (wince) or a Stack (shudder) instead of them just opening the target folder in the Finder.

In the Before Times, I could create an alias to the folder and drop that in the Dock, the icon in the Dock would look like the target folder, and clicking on the alias opened the folder’s window.  If I do that now, the click-to-open part works, but the aliases all look like blank text documents with tiny arrows.  What the hell?

If I instead add actual folders (not aliases) to the Dock, holding down ⌥⌘ (option-command) when I click them does exactly what I want.  Only, I don’t want to have to hold down modifier keys, especially when using the trackpad.  I’ve mostly adapted to the key combo, but even on desktop I still sometimes click a folder and blink in irritation at the popup thingy for a second before remembering that things are stupider now.

Translucency Tribulation

The other problem with the Dock is that mine is too opaque.  That’s because the nearly-transparent Finder menu bar was really not doing it for me, so acting on a helpful tip, I went and checked the “Reduce Transparency” option in the Accessibility settings.  That fixed the menu bar nicely, but it also made the Dock opaque, which I didn’t actually want.  I can pretty easily live with it, but I do wish I could make just the menu bar opaque (without having to resort to desktop wallpaper hacks, which I suspect do not do well with changes of display resolution).

Shortcut Stupidity

And while I’m on the subject of the menu bar: no matter the application or even the Finder itself, dropdown menus from the menu bar render the actions you can do in black and the actions you can’t do in washed-out gray.  Cool.  But also, all the keyboard shortcuts are now a washed-out gray, which I keep instinctively thinking means they’ve been disabled or something.  They’re also a lot more difficult for my older eyes to pick out, and I have to flick my eyes back and forth to make sure a given keyboard shortcut corresponds to a thing I actually can do.  Seriously, Apple, what the hell?

Trash Can Troubles

I used to have the Trash can on the desktop, down in the lower right corner, and now I guess I can’t.  I vaguely recall this is something DragThing made possible, so maybe that’s another reason to gripe about the lack of it, but it’s still bananas to me that the Trash can is not there by default.  I understand that I may be very old.

Preview Problems

On my old machine, Preview was probably the most rock-solid application on there.  On the new machine, Preview occasionally hangs on closing heavily-commented PDFs when I choose not to save changes.  I can force-quit it and so far haven’t experienced any data corruption, but it’s still annoying.

Those are the things that have stood out the most to me about Ventura.  How about you?  What bothers you about your operating system (whichever one that is) and how would you like to see it fixed?

Oh, and I’ll follow this up soon with a post about what I like in Ventura, because it’s not all frowns and grumbles.

Have something to say to all that? You can add a comment to the post, or email Eric directly.

I tend to keep quite a lot of notes on the development related (sometimes at work, sometimes not) I do on a week-by-week basis, and thought it might be fun to write up the parts that were public. This may or may not be of wider interest, but it aims to be a useful aide-mémoire for my purposes at least. Weeks with few entries might be due to focusing on downstream work (or perhaps just a less productive week - I am only human!).

Week of 27th March 2023

• Submitted a backport request to 16.0.1 for my recent fixes to llvm-objdump (and related tools) when encountering unrecognised RISC-V base or ISA extension versions, or unrecognised ISA extension names.
• Landed a tweak to the RISC-V ISA manual to make it clear that HINT encodings aren't "reserved" in terms of being part of the defined "reserved instruction-set category". Thanks to Andrew Waterman for suggesting a simpler fix than my first attempt.
• I'm now on Mastodon, at @asb@fosstodon.org and @llvmweekly@fosstodon.org.
• Proposed alternate wording for the RISC-V LLVM doc updates reflecting recent ISA versioning discussions.
• Set agenda for and ran the usual biweekly RISC-V LLVM contributor sync up call.
• Added the Renesas R9A06G150 to my commercially available RISC-V silicon list.
• Posted and landed patches to implement MC layer and codegen support for the experimental Zicond (integer conditional operations) extension (D146946, D147147). This is essentially the same as the XVentanaCondOps extension.
• Advertised the ongoing discussion about changing the shadow call stack register on RISC-V through an RFC.
• It was pointed out to me that the expansion of seq_cst atomic ops to RISC-V lr/sc loops was slightly stronger than required by the mapping table in the ISA manual. Specifically, sc.{w|d}.rl is sufficient rather than sc.{w|d}.aqrl. Fixed with D146933.
• Filed issue about the fcvt.bf16.s instruction encoding colliding with fround.h (instructions from zfbfmin and zfa respectively).
• Usual mix of upstream LLVM reviews. We were now able to bump the versions of the standard ISA extensions LLVM claims to support. As noted in my previous RFC, LLVM was reporting the wrong version information for the A/F/D extensions.
• LLVM Weekly #482.

Week of 20th March 2023

• Landed patches to fix RISC-V ISA extension versioning related issues in llvm-objdump and related tools (D146070, D146113, and D146114). Also patches to fix an ABI bug with _Float16 lowering on RISC-V (D142326, D145074).
• Opened a few issues on the aichat (command line ChatGPT client) repo: one for maximum line width, another for word wrapping, and a suggestion on converting one-shot questions to conversations.
• Added the HPM6750 to my commercially available RISC-V silicon list.
• My tutorial and lightning talk proposals were accepted for EuroLLVM!
• Some bits and pieces related to the RISC-V bfloat16 spec and also Zfh.
  • Drafted a Zfbfinxmin extension definition (primarily for symmetry with the existing z*inx[min] extensions.
  • Fixing a missed predicate for PseudoQuietFCMP.
  • Minor clarification to the riscv-bfloat16 spec.
• Usual mix of upstream LLVM reviews.
• LLVM Weekly #481.

Week of 13th March 2023

• Most importantly, added some more footer images for this site from the Quick Draw dataset. Thanks to my son (Archie, 5) for the assistance.
• Reviewed submissions for EuroLLVM (I'm on the program committee).
• Added note to the commercially available RISC-V silicon post about a hardware bug in the Renesas RZ/Five.
• Finished writing and published what's new for RISC-V in LLVM 16 article and took part in some of the discussions in the HN and Reddit threads (it's on lobste.rs too, but that didn't generate any comments).
• Investigated an issue where inline asm with the m constraint was generating worse code on LLVM vs GCC, finding that LLVM conservatively lowers this to a single register, while GCC treats m as reg+imm, relying on users indicating A when using a memory operand with an instruction that can't take an immediate offset. Worked with a colleague who posted D146245 to fix this.
• Set agenda for and ran the biweekly RISC-V LLVM contributor sync call as usual.
• Bisected reported LLVM bug #61412, which as it happens was fixed that evening by D145474 being committed. We hope to backport this to 16.0.1.
• Did some digging on a regression (compiler crash) for -Oz, bisecting it to the commit that enabled machine copy propagation by default. I found the issue was due to machine copy propagation running after the machine outliner, and incorrectly determining that some register writes in outlined functions were not live-out. I posted and landed D146037 to fix this by running machine copy propagation earlier in the pipeline, though a more principled fix would be desirable.
• Filed a PR against the riscv-isa-manual to disambiguate the use of the term "reserved" for HINT instructions. I've also been looking at the proposed bfloat16 extension recently and filed an issue to clarify if Zfbfinxmin will be defined (as all the other floating point extensions so far have an *inx twin.
• Almost finished work to resolve issues related to overzealous error checking on RISC-V ISA naming strings (with llvm-objdump and related tools being the final piece).
  • Landed D145879 and D145882 to expand RISCVISAInfo test coverage and fix an issue that surfaced through that.
  • Posted a pair of patches that makes llvm-objdump and related tools tolerant of unrecognised versions of ISA extensions. D146070 resolves this for the base ISA in a minimally invasive way, while D146114 solves this for other extensions, moving the parsing logic to using the parseNormalizedArchString function I introduced to fix a similar issue in LLD. This built on some directly committed work to expand testing.
• The usual assortment of upstream LLVM reviews.
• LLVM Weekly #480.

Week of 6th March 2023

• Had a really useful meeting with a breakout group from the RISC-V LLVM sync-up calls about the long-standing issues related to ISA extension versioning, error handling for this, and other related issues.
  • Related to this, posted D145879 and D145882 to flesh out testing of RISCVISAInfo::parseArchString ahead of further improvements, and to start to fix identified issues.
• Incorporated more clarifications submitted to me about the commercially available RISC-V SoCs post, particularly around SiFive cores.
• Shared some thoughts on attracting more people to the LLVM Foundation strategic planning sessions.
• Posted and committed an LLVM patch to migrate the RISC-V backend to using shared MCELFStreamer code for attribute emission. The initial implementation was largely derived from Arm's version of the same feature but a later refactoring managed to move this logic to common code, which we can now reuse.
• Some small tasks related to the RISC-V LLVM build-bot: trying to find a path forwards for a simple patch to enable RISC-V support in libcxx, and clarifying how often the staging buildmaster configuration is updated.
• Posted a docs patch to clarify Clang's -fexceptions in follow-up to discussion in issue 61216.
• Did some final preparations for the LLVM 16.0.0 release - committing some cleaned up release notes.
• A variety of upstream LLVM reviews, and left some thoughts on using sub-modules for RVV intrinsics tests.
• LLVM Weekly #479.

Week of 27th February 2023

• Completed (to the point I was happy to publish at least) my attempt to enumerate the commercially available RISC-V SoCs. I'm very grateful to have received a whole range of suggested additions and clarifications over the weekend, which have all been incorporated.
• Ran the usual biweekly RISC-V LLVM sync-up call. Topics included outstanding issues for LLVM 16.x (no major issues now my backport request to fix and LLD regression was merged), an overview off _Float16 ABI lowering fixes, GP relaxation in LLD, my recent RISC-V buildbot, and some vectorisation related issues.
• Investigated and largely resolved a issues related to ABI lowering of _Float16 for RISC-V. Primarily, we weren't handling the cases where a GPR+FPR or a pair of FPRs are used to pass small structs including _Float16.
  • Part of this work involved rebasing my previous patches to refactor our RISC-V ABI lowering tests in Clang. Now that a version of my improvements to update_cc_test_check.py --function-signature (required for the refactor) landed as part of D144963, this can hopefully be completed.
  • Committed a number of simple test improvements related to half floats. e.g. 570995e, 81979c3, 34b412d.
  • Posted D145070 to add proper coverage for _Float16 ABI lowering, and D145074 to fix it. Also D145071 to set the HasLegalHalfType property, but the semantics of that are less clear.
  • Posted a strawman psABI patch for __bf16, needed for the RISC-V bfloat16 extension.
• Attended the Cambridge RISC-V Meetup.
• After seeing the Helix editor discussed on lobste.rs, retried my previously shared large Markdown file test case. Unfortunately it's still unusably slow to edit, seemingly due to a tree-sitter related issue.
• Cleaned up the static site generator used for this site a bit. e.g. now my fixes (#157, #158, #159) for the traverse() helper in mistletoe where merged upstream, I removed my downstream version.
• The usual mix of upstream LLVM reviews.
• Had a day off for my birthday.
• Publicly shared this week log for the first time.
• LLVM Weekly #478.

Week of 20th February 2023

• Iterated on D144353 (aiming to fix LLD regression related to merging RISC-V attributes) based on review feedback and committed it.
  • Created an issue to track this as a regression, aiming for a backport into 16.0.0, and requested that backport.
  • Also some related discussion in ClangBuiltLinux issues #1777 and #1808.
• Committed my llvm-zorg patch to add the qemu-user based RISC-V builder, after finalising provisioning the machine to run it. The builder is live on the LLVM staging buildmaster as clang-rv64gc-qemu-user-single-stage.
  • Worked to resolve remaining test failures and stability issues. One recurrent issue was an assert in ___pthread_mutex_lock when executing ccache. Setting inode_cache=false in the local ccache config seems to avoid this.
  • Posted a couple of patches - D144464 and D144465 to tweak the LLVM docs on setting up a builder, based on my experience doing so.
• Chased for reviews and clarification about pre-commit test requirements for my libcxx RISC-V test fix patch, D134158.
• Committed a couple of further test updates for Wasm in LLVM ahead of some upcoming patches. 771261f 1ae8597.
• Left some quick notes on the LLVM RFC shepherds proposal.
• A variety of upstream LLVM reviews, and received a useful clarification on the RISC-V psABI and the ratification lifecycle.
• LLVM Weekly #477.

Week of 13th February 2023

• After a fair bit of investigation and thinking about reported compatibility issues between GNU and LLVM tools (particularly binutils ld and lld) due to RISC-V extension versioning, posted an RFC outlining the major issues and a proposed fix for what I consider to be a regression in lld.
  • Landed a few LLVM patches cleaning up tests related to this. 8b50048 574d0c2, d05e1e9.
  • Posted D144353, a proposed fix for the LLD regression due to overzealous checking of extensions/versions when merging RISC-V attributes.
• Organised agenda for and ran the bi-weekly RISC-V LLVM contributor call. Key discussion items were the extension versioning related compatibility issue mentioned below and support for emulated TLS (where I'd left some comments the previous week).
• Updated my patch (D143172) to register and configure a RISC-V qemu-user based builder with LLVM's staging buildmaster, based on review feedback.
• A variety of upstream LLVM reviews. Also landed D143407, marking Zawrs as non-experimental.
• LLVM Weekly #476.

Week of 6th February 2023

• Left feedback on the proposed RISC-V psABI patch clarifying treatment of empty structs or unions in the FP calling convention. This is a follow-up to the issue I filed on this issue (where I have D142327 queued up for LLVM to fix our incorrect handling).
• Responded to a question on LLVM's Discourse about zicsr and zifencei support in LLVM. As noted, the issue is that we haven't moved RV32I/RV64I 2.1 yet which split out Zicsr and Zifencei. Unfortunately this is a backwards-incompatible change so requires some care.
• Worked with a colleague trying to reproduce an assertion failure in his committed patch adding support for WebAssembly externref in Clang that appeared only on an MSan buildbot. The sanitizers project guidance is useful for this, but I ended up rolling a slightly hacky script as I stepped through each part of the multi-stage build and test sequence.
• Left my thoughts on a proposed RISC-V preprocessor define to specify support and for and performance of misaligned loads/stores. I like the idea of the define, but prefer sticking to the Zicclsm terminology introduced in the RISC-V profiles.
• Posted patch D143507 to mark RISC-V Zawrs as non-experimental, after confirming there were no relevant changes between the implemented 1.0-rc3 spec and the ratified version.
• A series of WebAssembly GC type related patches remains a work in progress downstream, but I landed a couple of related minor test cleanups. 604c9a0, 3a80dc2.
• Many upstream RISC-V LLVM reviews.
• LLVM Weekly #475.

• 2023-04-03: Added notes for the week of 27th March 2023.
• 2023-03-27: Added notes for the week of 20th March 2023.
• 2023-03-20: Added notes for the week of 13th March 2023.
• 2023-03-13: Added notes for the week of 6th March 2023.
• 2023-03-06: Added notes for the week of 27th February 2023.
• 2023-02-27: Added in a forgotten note about trivial buildbot doc improvements.
• 2023-02-27: Initial publication date.

Initial accelerated compositing support

When accelerated compositing support was added to WebKitGTK, there was only X11. Our first approach was quite simple, we sent the web view widget Xwindow ID to the web process to be used as rendering target using GLX. This was very efficient, but soon we realized it broke the GTK rendering model so it was not possible to use a web view inside a GtkOverlay, for example, to show status messages on top. The solution was to use a redirected Xcomposite window in the web process, and use its ID as the render target using GLX. The pixmap ID of the redirected Xcomposite window was sent to the UI process to be painted in the web view widget using a Cairo Xlib surface. Since the rendering happens in the web process, this approach required to use Xdamage to monitor when the redirected Xcomposite window was updated to schedule a web view redraw.

Wayland support

To support accelerated compositing under Wayland we initially added a nested Wayland compositor running in the UI process. The web process connected to the nested Wayland compositor and created a surface to be used as the rendering target using EGL. The good thing about this approach compared to the X11 one, is that we can create an EGLImage from Wayland buffers and use a GDK GL context to paint the contents in the web view. This is more efficient than X11 because we can use OpenGL both in web and UI processes.
WPE, when using the fdo backend, uses the same approach of running a nested Wayland compositor, but in a more efficient way, using DMABUF instead of Wayland buffers when available. So, we decided to use libwpe in the GTK port only for rendering under Wayland, and eventually remove our Wayland compositor implementation.
Before the removal of the custom Wayland compositor we had all these possible combinations:

• UI Process
• X11: Cairo Xlib surface
• Wayland: EGL
• Web Process
• X11: GLX using redirected Xwindow
• Wayland (nested Wayland compositor): EGL using Wayland surface
• Wayland (libwpe): EGL using libwpe to get the Wayland surface

To reduce a bit the differences, and to make it easier to support WebGL with ANGLE we decided to change X11 to prefer EGL if possible, falling back to GLX only if EGL failed.

GTK4

GTK4 was released and we added support for it. The fact that GTK4 uses GL by default should make the rendering more efficient in accelerated compositing mode. This is definitely true under Wayland, because we are using a GL context already, so we just keep passing a texture to GTK to paint the contents in the web view. However, in the case of X11 we still have a Cairo Xlib surface that GTK paints into a Cairo image surface to be uploaded to the GPU. With GTK4 now we have two more combinations in the UI process side X11 + GTK3, X11 + GTK4, Wayland + GTK3 and Wayland + GTK4.

Reducing all the combinations to (almost) one: DMABUF

All these combinations to support the different platforms made it quite difficult to maintain, every time we get a bug report about something not working in accelerated compositing mode we have to figure out the combination actually used by the reporter, GTK3 or GTK4? X11 or Wayland? using EGL or GLX? custom Wayland compositor or libwpe? driver? version? etc.

We are already using DMABUF in WebKit for different things like WebGL and media rendering, so we thought that we could also use it for sharing the rendered buffer between the web and UI processes. That would be a more efficient solution but it would also drastically reduce the amount of combinations to maintain. The web process always uses the surfaceless platform, so it doesn’t matter if it’s under Wayland or X11. Then we create a surfaceless context as the render target and use EGL and GBM APIs to export the contents as a DMABUF buffer. The UI process imports the DMABUF buffer using EGL and GBM too, to be passed to GTK as a texture that is painted in the web view.

This theoretically recudes all the previous combinations to just one (note that we removed GLX support entirely, making EGL a requirement for accelerated compositing), but there’s a problem under X11: GTK3 doesn’t support EGL on X11 and GTK4 defaults to EGL but falls back to GLX if it doesn’t find an EGL config that perfectly matches the screen visual. In my system it never finds that EGL config because mesa doesn’t expose any 32 bit depth config. So, in the case of GTK3 we have to manually download the buffer to CPU and paint normally using Cairo, but in the case of GTK4 + GLX, GTK uploads the buffer again to be painted using GLX. I don’t think it’s possible to force GTK to use EGL from the API, but at least you can use GDK_DEBUG=gl-egl.

WebKitGTK 2.41.1

WebKitGTK 2.41.1 is the first unstable release of this cycle and already includes the DMABUF support that is used by default. We encourage everybody to try it out and provide feedback or report any issue. Please, export the contents of webkit://gpu and attach it to the bug report when reporting any problem related to graphics. To check if the issue is a regression of the DMABUF implementation you can use WEBKIT_DISABLE_DMABUF_RENDERER=1 to use the WPE renderer or X11 instead. This environment variable and the WPE render/X11 code will be eventually removed if DMABUF works fine.

WPE

If this approach works fine we plan to use something similar for the WPE port and get rid of the nested Wayland compositor there too.

The two videos I was using Whisper on have been published, so you can see for yourself how the captioning worked out.  Designed as trade-show booth reel pieces, they’re below three minutes each, so watching both should take less than ten minutes, even with pauses to scrutinize specific bits of captioning.

As I noted in my previous post about this, I only had to make one text correction to the second video, plus a quick find-and-replace to turn “WPE WebKit” into “WPEWebKit”.  For the first video, I did make a couple of edits beyond fixing transcription errors; specifically, I added the dashes and line breaking in this part of the final SubRip Subtitle (SRT) file uploaded to YouTube:

00:00:25,000 --> 00:00:32,000
- Hey tell me, is Michael coming out?
- Affirmative, Mike's coming out.

This small snippet actually embodies the two things where Whisper falls down a bit: multiple voices, and caption line lengths.

Right now, Whisper doesn’t even try to distinguish between different voices, the technical term for which is “speaker diarisation”.  This means Whisper ideal for transcribing, say, a conference talk or a single-narrator video.  It’s lot less useful for things like podcasts, because while it will probably get (nearly) all the words right, it won’t even throw in a marker that the voice changed, let alone try to tell which bits belong to a given voice.  You have to go into the output and add those yourself, which for an hourlong podcast could be… quite the task.

There are requests for adding this to Whisper scattered in their GitHub discussions, but I didn’t see any open pull requests or mention of it in the README, so I don’t know if that’s coming or not.  If you do, please leave a comment!

As for the length of captions, I agree with J David Eisenberg: Whisper too frequently errs on the side of “too long”.  For example, here’s one of the bits Whisper output:

00:01:45,000 --> 00:01:56,000
Here is the dash.js player using MSE, running in a page, and using Widevine DRM to decrypt and play rights-managed video with EME, all fluidly.

That’s eleven seconds of static subtitling, with 143 characters of line length.  The BBC recommends line lengths at or below 37 characters, and Netflix suggests a limit of 42 characters, with actual hard limits for a few languages.  You can throw in line breaks to reduce line length, but should never have more than three lines, which wouldn’t be possible with 143 characters.  But let’s be real, that 11-second caption really should be split in twain, at the absolute minimum.

Whisper does not, as of yet, have a way to request limiting caption lengths, either in time or in text.  There is a fairly detailed discussion of this over on Whisper’s repository, with some code graciously shared by people working to address this, but it would be a lot better if Whisper accepted an argument to limit the length of any given bit of output.  And also if it threw in line breaks on its own, say around 40 characters in English, even when not requested.

The last thing I’d like to see improved is speed.  It’s not terribly slow as is, to be clear.  Using the default model size (small), which is what I used for the videos I wrote about, Whisper worked at about 2:1 speed: a two-minute video took about a minute to process.  I tried the next size up, the medium model, and it worked at roughly 1:1.5 speed, taking about an hour fifteen to process a 46-minute video.

The thing is, all that is running solely on the CPU, which in my case is a 12-core M2.  According to this pull request, problems in one of Whisper’s dependencies, PyTorch, means GPU utilization is essentially unavailable on the hardware I have. (Thanks to Chris Adams for the pointer.) I expect that will be cleared up sooner or later, so the limitation feels minor.

Overall, it’s a powerful tool, with accuracy I still find astounding, only coming up short in quality-of-life features that aren’t critical in some applications (transcribing a talk) or relatively easily worked around in others (hand-correcting caption length in short videos; using a small script to insert line breaks in longer videos).  The lack of speaker diarisation is the real letdown for me, and definitely the hardest to work around, so I hope it gets addressed soon.

Have something to say to all that? You can add a comment to the post, or email Eric directly.

What happened was, I was hanging out in an online chatter channel when a little birdy named Bruce chirped about OpenAI’s Whisper and how he was using it to transcribe audio.  And I thought, Hey, I have audio that needs to be transcribed.  Brucie Bird also mentioned it would output text, SRT, and WebVTT formats, and I thought, Hey, I have videos I’ll need to upload with transcription to YouTube!  And then he said you could run it from the command line, and I thought, Hey, I have a command line!

So off I went to install it and try it out, and immediately ran smack into some hurdles I thought I’d document here in case someone else has similar problems.  All of this took place on my M2 MacBook Pro, though I believe most of the below should be relevant to anyone trying to do this at the command line.

The first thing I did was what the GitHub repository’s README recommended, which is:

$ pip install -U openai-whisper

That failed because I didn’t have pip installed.  Okay, fair enough.  I figured out how to install that, setting up an alias of python for python3 along the way, and then tried again.  This time, the install started and then bombed out:

Collecting openai-whisper
  Using cached openai-whisper-20230314.tar.gz (792 kB)
  Installing build dependencies ...  done
  Getting requirements to build wheel ...  done
  Preparing metadata (pyproject.toml) ...  done
Collecting numba
  Using cached numba-0.56.4.tar.gz (2.4 MB)
  Preparing metadata (setup.py) ...  error
  error: subprocess-exited-with-error

…followed by some stack trace stuff, none of which was really useful until ten or so lines down, where I found:

RuntimeError: Cannot install on Python version 3.11.2; only versions >=3.7,<3.11 are supported.

In other words, the version of Python I have installed is too modern to run AI.  What a world.

I DuckDucked around a bit and hit upon pyenv, which is I guess a way of installing and running older versions of Python without having to overwrite whatever version(s) you already have.  I’ll skip over the error part of my trial-and-error process and give you the commands that made it all work:

$ brew install pyenv

$ pyenv install 3.10

$ PATH="~/.pyenv/shims:${PATH}"

$ pyenv local 3.10

$ pip install -U openai-whisper

That got Whisper to install.  It didn’t take very long.

At that point, I wondered what I’d have to configure to transcribe something, and the answer turned out to be precisely zilch.  Once the install was done, I dropped into the directory containing my MP4 video, and typed this:

$ whisper wpe-mse-eme-v2.mp4

Here’s what I got back.  I’ve marked the very few errors.

[00:00.000 --> 00:07.000]  In this video, we'll show you several demos showcasing multi-media capabilities in WPE WebKit,
[00:07.000 --> 00:11.000]  the official port of the WebKit engine for embedded devices.
[00:11.000 --> 00:18.000]  Each of these demos are running on the low-powered Raspberry Pi 3 seen in the lower right-hand side of the screen here.
[00:18.000 --> 00:25.000]  Infotainment systems and media players often need to consume digital rights-managed videos.
[00:25.000 --> 00:32.000]  They tell me, is Michael coming out?  Affirmative, Mike's coming out.
[00:32.000 --> 00:45.000]  Here you can see just that, smooth streaming playback using encrypted media extensions, or EME, with PlayReady 4.
[00:45.000 --> 00:52.000]  Media source extensions, or MSE, are used by many players for greater control over playback.
[00:52.000 --> 01:00.000]  YouTube TV has a whole conformance test suite for this, which WPE has been passing since 2021.
[01:00.000 --> 01:09.000]  The loan exceptions here are those tests requiring hardware support not available on the Raspberry Pi 4, but available for other platforms.
[01:09.000 --> 01:16.000]  YouTube TV has a conformance test for EME, which WPE WebKit passes with flying colors.
[01:22.000 --> 01:40.000]  Music
[01:40.000 --> 01:45.000]  Finally, perhaps most impressively, we can put all these things together.
[01:45.000 --> 01:56.000]  Here is the dash.js player using MSE, running in a page, and using Widevine DRM to decrypt and play rights-managed video with EME all fluidly.
[01:56.000 --> 02:04.000]  Music
[02:04.000 --> 02:09.000]  Remember, all of this is being played back on the same low-powered Raspberry Pi 3.
[02:27.000 --> 02:34.000]  For more about WPE WebKit, please visit WPE WebKit.com.
[02:34.000 --> 02:42.000]  For more information about EGALIA, or to find out how we can help with your embedded device needs, please visit us at EGALIA.com.  

I am, frankly, astonished.  This has no business being as accurate as it is, for all kinds of reasons.  There’s a lot of jargon and very specific terminology in there, and Whisper nailed pretty much every last bit of it, first time in, no special configuration, nothing.  I didn’t even bump up the model size from the default of small.  I felt a little like that Froyo guy in the animated Hunchback of Notre Dame meme yelling about sorcery or whatever.

True, the output isn’t absolutely perfect.  Let’s review the glitches in reverse order.  The last two errors, turning “Igalia” into “EGALIA”, seems fair enough given I didn’t specify that there would be languages other than English involved.  I routinely have to spell it for my fellow Americans, so no reason to think a codebase could do any better.

The space inserted into “WPEWebKit” (which happens throughout) is similarly understandable.  I’m impressed it understood “WebKit” at all, never mind that it was properly capitalized and not-spaced.

The place where it says Music and I marked it as an error: This is essentially an echoing countdown and then a white-noise roar from rocket engines.  There’s a “music today is just noise” joke in here somewhere, but I’m too hip to find it.

Whisper turning “lone” into “loan” doesn’t particularly faze me, given the difficulty of handling soundalike words.  Hell, just yesterday, I was scribing a conference call and mistakenly recorded “gamut” as “gamma”, and those aren’t even technically homophones.  They just sound like they are.

Rounding out the glitch tour, “Hey” got turned into “They”, which (given the audio quality of that particular part of the video) is still pretty good.

There is one other error I couldn’t mark because there’s nothing to mark, but if you scrutinize the timeline, you’ll see a gap from 02:09.000 and 02:27.000.  In there, a short clip from a movie plays, and there’s a brief dialogue between two characters in not-very-Dutch-accented English there.  It’s definitely louder and more clear than the 00:25.000 –> 00:32.000 bit, so I’m not sure why Whisper just skipped over it.  Manually transcribing that part isn’t a big deal, but it’s odd to see it perform so flawlessly on every other piece of speech and then drop this completely on the floor.

Before posting, I decided to give Whisper another go, this time on a different video:

$ whisper wpe-gamepad-support-v3.mp4

This was the result, with the one actual error marked:

[00:00.000 --> 00:13.760]  In this video, we demonstrate WPE WebKit's support for the W3C's GamePad API.
[00:13.760 --> 00:20.080]  Here we're running WPE WebKit on a Raspberry Pi 4, but any device that will run WPE WebKit
[00:20.080 --> 00:22.960]  can benefit from this support.
[00:22.960 --> 00:28.560]  The GamePad API provides a JavaScript interface that makes it possible for developers to access
[00:28.560 --> 00:35.600]  and respond to signals from GamePads and other game controllers in a simple, consistent way.
[00:35.600 --> 00:40.320]  Having connected a standard Xbox controller, we boot up the Raspberry Pi with a customized
[00:40.320 --> 00:43.040]  build route image.
[00:43.040 --> 00:48.560]  Once the device is booted, we run cog, which is a small, single window launcher made specifically
[00:48.560 --> 00:51.080]  for WPE WebKit.
[00:51.080 --> 00:57.360]  The window cog creates can be full screen, which is what we're doing here.
[00:57.360 --> 01:01.800]  The game is loaded from a website that hosts a version of the classic video arcade game
[01:01.800 --> 01:05.480]  Asteroids.
[01:05.480 --> 01:11.240]  Once the game has loaded, the Xbox controller is used to start the game and control the spaceship.
[01:11.240 --> 01:17.040]  All the GamePad inputs are handled by the JavaScript GamePad API.
[01:17.040 --> 01:22.560]  This GamePad support is now possible thanks to work done by Igalia in 2022 and is available
[01:22.560 --> 01:27.160]  to anyone who uses WPE WebKit on their embedded device.
[01:27.160 --> 01:32.000]  For more about WPE WebKit, please visit wpewebkit.com.
[01:32.000 --> 01:35.840]  For more information about Igalia, or to find out how we can help with your embedded device
[01:35.840 --> 01:39.000]  needs, please visit us at Igalia.com.  

That should have been “buildroot”.  Again, an entirely reasonable error.  I’ve made at least an order of magnitude more typos writing this post than Whisper has in transcribing these videos.  And this time, it got the spelling of Igalia correct.  I didn’t make any changes between the two runs.  It just… figured it out.

I don’t have a lot to say about this other than, wow.  Just WOW.  This is some real Clarke’s Third Law stuff right here, and the technovertigo is Marianas deep.

Have something to say to all that? You can add a comment to the post, or email Eric directly.

Table of Contents

1. How the tool is used

In previous posts, “Graphics Flight Recorder - unknown but handy tool to debug GPU hangs” and “Debugging Unrecoverable GPU Hangs”, I demonstrated a few tricks of how to identify the location of GPU fault.

But what’s the next step once you’ve roughly pinpointed the issue? What if the problem is only sporadically reproducible and the only way to ensure consistent results is by replaying a trace of raw GPU commands? How can you precisely determine the cause and find a proper fix?

Sometimes, you may have an inkling of what’s causing the problem, then and you can simply modify the driver’s code to see if it resolves the issue. However, there are instances where the root cause remains elusive or you only want to change a specific value without affecting the same register before and after it.

The optimal approach in these situations is to directly modify the commands sent to the GPU. The ability to arbitrarily edit the command stream was always an obvious idea and has crossed my mind numerous times (and not only mine – proprietary driver developers seem to employ similar techniques). Finally, the stars aligned: my frustration with a recent bug, the kernel’s new support for user-space-defined GPU addresses for buffer objects, the tool I wrote to replay command stream traces not so long ago, and the realization that implementing a command stream editor was not as complicated as initially thought.

The end result is a tool for Adreno GPUs (with msm kernel driver) to decompile, edit, and compile back command streams: “freedreno,turnip: Add tooling to edit command streams and use them in ‘replay’”.

The primary advantage of this command stream editing tool lies the ability to rapidly iterate over hypotheses. Another highly valuable feature (which I have plans for) would be the automatic bisection of the command stream, which would be particularly beneficial in instances where only the bug reporter has the necessary hardware to reproduce the issue at hand.

How the tool is used

# Decompile one command stream from the trace
./rddecompiler -s 0 gpu_trace.rd > generate_rd.c

# Compile the executable which would output the command stream
meson setup . build
ninja -C build

# Override the command stream with the commands from the generator
./replay gpu_trace.rd --override=0 --generator=./build/generate_rd

Reading dEQP-VK.renderpass.suballocation.formats.r5g6b5_unorm_pack16.clear.clear.rd...
gpuid: 660
Uploading iova 0x100000000 size = 0x82000
Uploading iova 0x100089000 size = 0x4000
cmdstream 0: 207 dwords
generating cmdstream './generate_rd --vastart=21441282048 --vasize=33554432 gpu_trace.rd'
Uploading iova 0x4fff00000 size = 0x1d4
override cmdstream: 117 dwords
skipped cmdstream 1: 248 dwords
skipped cmdstream 2: 223 dwords

The decompiled code isn’t pretty:

/* pkt4: GRAS_SC_SCREEN_SCISSOR[0].TL = { X = 0 | Y = 0 } */
pkt4(cs, REG_A6XX_GRAS_SC_SCREEN_SCISSOR_TL(0), (2), 0);
/* pkt4: GRAS_SC_SCREEN_SCISSOR[0].BR = { X = 32767 | Y = 32767 } */
pkt(cs, 2147450879);
/* pkt4: VFD_INDEX_OFFSET = 0 */
pkt4(cs, REG_A6XX_VFD_INDEX_OFFSET, (2), 0);
/* pkt4: VFD_INSTANCE_START_OFFSET = 0 */
pkt(cs, 0);
/* pkt4: SP_FS_OUTPUT[0].REG = { REGID = r0.x } */
pkt4(cs, REG_A6XX_SP_FS_OUTPUT_REG(0), (1), 0);
/* pkt4: SP_TP_RAS_MSAA_CNTL = { SAMPLES = MSAA_FOUR } */
pkt4(cs, REG_A6XX_SP_TP_RAS_MSAA_CNTL, (2), 2);
/* pkt4: SP_TP_DEST_MSAA_CNTL = { SAMPLES = MSAA_FOUR } */
pkt(cs, 2);
/* pkt4: GRAS_RAS_MSAA_CNTL = { SAMPLES = MSAA_FOUR } */
pkt4(cs, REG_A6XX_GRAS_RAS_MSAA_CNTL, (2), 2);

Shader assembly is editable:

const char *source = R"(
  shps #l37
  getone #l37
  cov.u32f32 r1.w, c504.z
  cov.u32f32 r2.x, c504.w
  cov.u32f32 r1.y, c504.x
  ....
  end
)";
upload_shader(&ctx, 0x100200d80, source);
emit_shader_iova(&ctx, cs, 0x100200d80);

However, not everything is currently editable, such as descriptors. Despite this limitations, the existing functionality is sufficient for the majority of cases.

LLVM 16.0.0 was just released today, and as I did for LLVM 15, I wanted to highlight some of the RISC-V specific changes and improvements. This is very much a tour of a chosen subset of additions rather than an attempt to be exhaustive. If you're interested in RISC-V, you may also want to check out my recent attempt to enumerate the commercially available RISC-V SoCs and if you want to find out what's going on in LLVM as a whole on a week-by-week basis, then I've got the perfect newsletter for you.

In case you're not familiar with LLVM's release schedule, it's worth noting that there are two major LLVM releases a year (i.e. one roughly every 6 months) and these are timed releases as opposed to being cut when a pre-agreed set of feature targets have been met. We're very fortunate to benefit from an active and growing set of contributors working on RISC-V support in LLVM projects, who are responsible for the work I describe below - thank you! I coordinate biweekly sync-up calls for RISC-V LLVM contributors, so if you're working in this area please consider dropping in.

Documentation

LLVM 16 is the first release featuring a user guide for the RISC-V target (16.0.0 version, current HEAD. This fills a long-standing gap in our documentation, whereby it was difficult to tell at a glance the expected level of support for the various RISC-V instruction set extensions (standard, vendor-specific, and experimental extensions of either type) in a given LLVM release. We've tried to keep it concise but informative, and add a brief note to describe any known limitations that end users should know about. Thanks again to Philip Reames for kicking this off, and the reviewers and contributors for ensuring it's kept up to date.

Vectorization

LLVM 16 was a big release for vectorisation. As well as a long-running strand of work making incremental improvements (e.g. better cost modelling) and fixes, scalable vectorization was enabled by default. This allows LLVM's loop vectorizer to use scalable vectors when profitable. Follow-on work enabled support for loop vectorization using fixed length vectors and disabled vectorization of epilogue loops. See the talk optimizing code for scalable vector architectures (slides) by Sander de Smalen for more information about scalable vectorization in LLVM and introduction to the RISC-V vector extension by Roger Ferrer Ibáñez for an overview of the vector extension and some of its codegen challenges.

The RISC-V vector intrinsics supported by Clang have changed (to match e.g. this and this) during the 16.x development process in a backwards incompatible way, as the RISC-V Vector Extension Intrinsics specification evolves towards a v1.0. In retrospect, it would have been better to keep the intrinsics behind an experimental flag when the vector codegen and MC layer (assembler/disassembler) support became stable, and this is something we'll be more careful of for future extensions. The good news is that thanks to Yueh-Ting Chen, headers are available that provide the old-style intrinsics mapped to the new version.

Support for new instruction set extensions

I refer to 'experimental' support many times below. See the documentation on experimental extensions within RISC-V LLVM for guidance on what that means. One point to highlight is that the extensions remain experimental until they are ratified, which is why some extensions on the list below are 'experimental' despite the fact the LLVM support needed is trivial. On to the list of newly added instruction set extensions:

• Experimental support for the Zca, Zcf, and Zcd instruction set extensions. These are all 16-bit instructions and are being defined as part of the output of the RISC-V code size reduction working group.
  • Zca is just a subset of the standard 'C' compressed instruction set extension but without floating point loads/stores.
  • Zcf is also a subset of the standard 'C' compressed instruction set extension, including just the single precision floating point loads and stores (c.flw, c.flwsp, c.fsw, c.fswsp).
  • Zcd, as you might have guessed, just includes the double precision floating point loads and stores from the standard 'C' compressed instruction set extension (c.fld, c.fldsp, c.fsd, c.fsdsp).
• Experimental assembler/disassembler support for the Zihintntl instruction set extension. This provides a small set of instructions that can be used to hint that the memory accesses of the following instruction exhibits poor temporal locality.
• Experimental assembler/disassembler support for the Zawrs instruction set extension, providing a pair of instructions meant for use in a polling loop allowing a core to enter a low-power state and wait on a store to a memory location.
• Experimental support for the Ztso extension, which for now just means setting the appropriate ELF header flag. If a core implements Ztso, it implements the Total Store Ordering memory consistency model. Future releases will provide alternate lowerings of atomics operations that take advantage of this.
• Code generation support for the Zfhmin extension (load/store, conversion, and GPR/FPR move support for 16-bit floating point values).
• Codegen and assembler/disassembler support for the XVentanaCondOps vendor extension, which provides conditional arithmetic and move/select operations.
• Codegen and assembler/disassembler support for the XTHeadVdot vendor extension, which implements vector integer four 8-bit multiple and 32-bit add.

LLDB

LLDB has started to become usable for RISC-V in this period due to work by contributor 'Emmer'. As they summarise here, LLDB should be usable for debugging RV64 programs locally but support is lacking for remote debug (e.g. via the gdb server protocol). During the LLVM 16 development window, LLDB gained support for software single stepping on RISC-V, support in EmulateInstructionRISCV for RV{32,64}I, as well as extensions A and M, C, RV32F and RV64F, and D.

Short forward branch optimisation

Another improvement that's fun to look more closely at is support for "short forward branch optimisation" for the SiFive 7 series cores. What does this mean? Well, let's start by looking at the problem it's trying to solve. The base RISC-V ISA doesn't include conditional moves or predicated instructions, which can be a downside if your code features unpredictable short forward branches (with the ensuing cost in terms of branch mispredictions and bloating branch predictor state). The ISA spec includes commentary on this decision (page 23), noting some disadvantages of adding such instructions to the specification and noting microarchitectural techniques exist to convert short forward branches into predicated code internally. In the case of the SiFive 7 series, this is achieved using macro-op fusion where a branch over a single ALU instruction is fused and executed as a single conditional instruction.

In the LLVM 16 cycle, compiler optimisations targeting this microarchitectural feature were enabled for conditional move style sequences (i.e. branch over a register move) as well as for other ALU operations. The job of the compiler here is of course to emit a sequence compatible with the micro-architectural optimisation when possible and profitable. I'm not aware of other RISC-V designs implementing a similar optimisation - although there are developments in terms of instructions to support such operations directly in the ISA which would avoid the need for such microarchitectural tricks. See XVentanaCondOps, XTheadCondMov, the previously proposed but now abandoned Zbt extension (part of the earlier bitmanip spec) and more recently the proposed Zicond (integer conditional operations) standard extension.

Atomics

It's perhaps not surprising that code generation for atomics can be tricky to understand, and the LLVM documentation on atomics codegen and libcalls is actually one of the best references on the topic I've found. A particularly important note in that document is that if a backend supports any inline lock-free atomic operations at a given size, all operations of that size must be supported in a lock-free manner. If targeting a RISC-V CPU without the atomics extension, all atomics operations would usually be lowered to __atomic_* libcalls. But if we know a bit more about the target, it's possible to do better - for instance, a single-core microcontroller could implement an atomic operation in a lock-free manner by disabling interrupts (and conventionally, lock-free implementations of atomics are provided through __sync_* libcalls). This kind of setup is exactly what the +forced-atomics feature enables, where atomic load/store can be lowered to a load/store with appropriate fences (as is supported in the base ISA) while other atomic operations generate a __sync_* libcall.

There's also been a very minor improvement for targets with native atomics support (the 'A' instruction set extension) that I may as well mention while on the topic. As you might know, atomic operations such as compare and swap that are lowered to an instruction sequence involving lr.{w,d} (load reserved) and sc.{w,d} (store conditional). There are very specific rules about these instruction sequences that must be met to align with the architectural forward progress guarantee (section 8.3, page 51), which is why we expand to a fixed instruction sequence at a very late stage in compilation (see original RFC). This means the sequence of instructions implementing the atomic operation are opaque to LLVM's optimisation passes and are treated as a single unit. The obvious disadvantage of avoiding LLVM's optimisations is that sometimes there are optimisations that would be helpful and wouldn't break that forward-progress guarantee. One that came up in real-world code was the lack of branch folding, which would have simplified a branch in the expanded cmpxchg sequence that just targets another branch with the same condition (by just folding in the eventual target). With some relatively simple logic, this suboptimal codegen is resolved.

; Before                 => After
.loop:                   => .loop
  lr.w.aqrl a3, (a0)     => lr.w.aqrl a3, (a0)
  bne a3, a1, .afterloop => bne a3, a1, .loop
  sc.w.aqrl a4, a2, (a0) => sc.w.aqrl a4, a2, (a0)
  bnez a4, .loop         => bnez a4, .loop
.aferloop:               =>
  bne a3, a1, .loop      =>
  ret                    => ret

Assorted optimisations

As you can imagine, there's been a lot of incremental minor improvements over the past ~6 months. I unfortunately only have space (and patience) to highight a few of them.

A new pre-regalloc pseudo instruction expansion pass was added in order to allow optimising the global address access instruction sequences such as those found in the medany code model (and was later broadened further). This results in improvements such as the following (note: this transformation was already supported for the medlow code model):

; Before                            => After
.Lpcrel_hi1:                        => .Lpcrel_hi1
auipc a0, %pcrel_hi1(ga)            => auipc a0, %pcrel_hi1(ga+4)
addi a0, a0, %pcrel_lo(.Lpcrel_hi1) =>
lw a0, 4(a0)                        => lw a0, %pcrel_lo(.Lpcrel_hi1)(a0)

A missing target hook (isUsedByReturnOnly) had been preventing tail calling libcalls in some cases. This was fixed, and later support was added for generating an inlined sequence of instructions for some of the floating point libcalls.

The RISC-V compressed instruction set extension defines a number of 16-bit encodings that map to a 32-bit longer form (with restrictions on addressable registers in the compressed form of course). The conversion 32-bit instructions 16-bit forms when possible happens at a very late stage, after instruction selection. But of course over time, we've introduced more tuning to influence codegen decisions in cases where a choice can be made to produce an instruction that can be compressed, rather than one that can't. A recent addition to this was the RISCVStripWSuffix pass, which for RV64 targets will convert addw and slliw to add or slli respectively when it can be determined that all the users of its result only use the lower 32 bits. This is a minor code size saving, as slliw has no matching compressed instruction and c.addw can address a more restricted set of registers than c.add.

Other

At the risk of repeating myself, this has been a selective tour of some additions I thought it would be fun to write about. Apologies if I've missed your favourite new feature or improvement - the LLVM release notes will include some things I haven't had space for here. Thanks again for everyone who has been contributing to make the RISC-V in LLVM even better.

If you have a RISC-V project you think me and my colleagues and at Igalia may be able to help with, then do get in touch regarding our services.

• 2023-03-19: Clarified that Zawrs and Zihintntl support just involves the MC layer (assembler/disassembler).
• 2023-03-18: Initial publication date.

Last year I wrote a series of blog posts (1, 2, 3) about stack walk profiling Chromium using Windows native tools around ETW.

A fast recap: ETW support for stack walking in V8 allows to show V8 JIT generated code in the Windows Performance Analyzer. This is a powerful tool to analyze work loads where Javascript execution time is significant.

In this blog post, I will cover the usage of this very same tool, but to analyze NodeJS execution.

Enabling stack walk JIT information in NodeJS

In an ideal situation, V8 engines would always generate stack walk information when Windows is profiling. This is something we will want to consider in the future, as we prove enabling it has no cost if we are not in a tracing session.

Meanwhile, we need to set the V8 flag --enable-etw-stack-walking somehow. This will install hooks that, when a profiling session starts, will emit the JIT generated code addresses, and the information about the source code associated to them.

For a command line execution of NodeJS runtime, it is as simple as passing the command line flag:

node --enable-etw-stack-walking

This will work enabling ETW stack walking for that specific NodeJS session… Good, but not very useful.

Enabling ETW stack walking for a session

What’s the problem here? Usually, NodeJS is invoked indirectly through other tools (based or not in NodeJS). Some examples are Yarn, NPM, or even some Windows scripts or link files.

We could tune all the existing launching scripts to pass --enable-etw-stack-walking to the NodeJS runtime when it is called. But that is not much convenient.

There is a better way though, just using NODE_OPTIONS environment variable. This way, stack walking support can be enabled for all NodeJS calls in a shell session, or even system wide.

Bad news and good news

Some bad news: NodeJS was refusing --enable-etw-stack-walking in NODE_OPTIONS. There is a filter for which V8 options it accepts (mostly for security purposes), and ETW support was not considered.

Good news? I implemented a fix adding the flag to the list accepted by NODE_OPTIONS. It has been landed already, and it is available from NodeJS 19.6.0. Unfortunately, if you are using an older version, then you may need to backport the patch.

Using it: linting TypeScript

To explain how this can be used, I will analyse ESLint on a known workload: TypeScript. For simplicity, we are using the lint task provided by TypeScript.

This example assumes the usage of Git Bash.

First, clone TypeScript from GitHub, and go to the cloned copy:

git clone https://github.com/microsoft/TypeScript.git
cd TypeScript

Then, install hereby and the dependencies of TypeScript:

npm install -g hereby
npm ci

Now, we are ready to profile the lint task. First, set NODE_OPTIONS:

export NODE_OPTIONS="--enable-etw-stack-walking"

Then, launch UIForETW. This tool simplifies capturing traces, and will provide good defaults for Javascript ETW analysis. It provides a very useful keyboard shortcut, <Ctrl>+<Win>+R, to start and then stop a recording.

Switch to Git Bash terminal and do this sequence:

• Write (without pressing <Enter>): hereby lint
• Press <Ctrl>+<Win>+R to start recording. Wait 3-4 seconds as recording does not start immediately.
• Press <Enter>. ESLint will traverse all the TypeScript code.
• Press again <Ctrl>+<Win>+R to stop recording.

After a few seconds UIForETW will automatically open the trace in Windows Performance Analyzer. Thanks to settings NODE_OPTIONS all the child processes of the parent node.exe execution also have stack walk information.

Randomascii inclusive (stack) analysis

Focusing on node.exe instances, in Randomascii inclusive (stack) view, we can see where time is spent for each of the node.exe processes. If I take the bigger one (that is the longest of the benchmarks I executed), I get some nice insights.

The worker threads take 40% of the CPU processing. What is happening there? I basically see JIT compilation and garbage collection concurrent marking. V8 offloads that work, so there is a benefit from a multicore machine.

Most of the work happens in the main thread, as expected. And most of the time is spent parsing and applying the lint rules (half for each).

If we go deeper in the rules processing, we can see which rules are more expensive.

Memory allocation

In total commit view, we can observe the memory usage pattern of the process running ESLint. For most of the seconds of the workload, allocation grows steadily (to over 2GB of RAM). Then there is a first garbage collection, and a bit later, the process finishes and all the memory is deallocated.

More findings

At first sight, I observe we are creating the rules objects for all the execution of ESLint. What does it mean? Could we run faster reusing those? I can also observe that a big part of the time in main thread leads to leaves doing garbage collection.

This is a good start! You can see how ETW can give you insights of what is happening and how much time it takes. And even correlate that to memory usage, File I/O, etc.

Builtins fix

Using NodeJS, as is today, will still show many missing lines in the stack. I did those tests, and could do a useful analysis, because I applied a very recent patch I landed in V8.

Before the fix, we would have this sequence:

• Enable ETW recording
• Run several NodeJS tests.
• Each of the tests creates one or more JS contexts.
• That context then sends to ETW the information of any code compiled with JIT.

But there was a problem: any JS context has already a lot of pre-compiled code associated: builtins and V8 snapshot code. Those were missing from the ETW traces captured.

The fix, as said, has been already landed to V8, and hopefully will be available soon in future NodeJS releases.

Wrapping up

There is more work to do:

• WASM is still not supported.
• Ideally, we would want to have --enable-etw-stack-walking set by default, as the impact while not tracing is minimal.

In any case, after these new fixes, capturing ETW stack walks of code executed by NodeJS runtime is a bit easier. I hope this gives some joy to your performance research.

One last thing! My work for these fixes is possible thanks to the sponsorship from Igalia and Bloomberg.

We, Igalia’s multimedia team, would like to share with you our list of achievements along the past 2022.

WebKit Multimedia

WebRTC

Phil already wrote a first blog post, of a series, on this regard: WebRTC in WebKitGTK and WPE, status updates, part I. Please, be sure to give it a glance, it has nice videos.

Long story short, last year we started to support Media Capture and Streams in WebKitGTK and WPE using GStreamer, either for input devices (camera and microphone), desktop sharing, webaudio, and web canvas. But this is just the first step. We are currently working on RTCPeerConnection, also using GStreamer, to share all these captured streams with other web peers. Meanwhile, we’ll wait for the second episode of Phil’s series

MediaRecorder

We worked in an initial implementation of MediaRecorder with GStreamer (1.20 or superior). The specification goes about allowing a web browser to record a selected stream. For example, a voice-memo or video application which could encode and upload a capture of your microphone / camera.

Gamepad

While WebKitGTK already has Gamepad support, WPE lacked it. We did the implementation last year, and there’s a blog post about it: Gamepad in WPEWebkit, with video showing a demo of it.

Capture encoded video streams from webcams

Some webcams only provide high resolution frames encoded in H.264 or so. In order to support these resolutions with those webcams we added the support for negotiate of those formats and decode them internally to handle the streams. Though we are just at the beginning of more efficient support.

Flatpak SDK maintenance

A lot of effort went to maintain the Flatpak SDK for WebKit. It is a set of runtimes that allows to have a reproducible build of WebKit, independently of the used Linux distribution. Nowadays the Flatpak SDK is used in Webkit’s EWS, and by many developers.

Among all the features added during the year we can highlight added Rust support, a full integrity check before upgrading, and offer a way to override dependencies as local projects.

MSE/EME enhancements

As every year, massive work was done in WebKit ports using GStreamer for Media Source Extensions and Encrypted Media Extensions, improving user experience with different streaming services in the Web, such as Odysee, Amazon, DAZN, etc.

In the case of encrypted media, GStreamer-based WebKit ports provide the stubs to communicate with an external Content Decryption Module (CDM). If you’re willing to support this in your platform, you can reach us.

Also we worked in a video demo showing how MSE/EME works in a Raspberry Pi 3 using WPE:

WebAudio demo

We also spent time recording video demos, such as this one, showing WebAudio using WPE on a desktop computer.

GStreamer

We managed to merge a lot of bug fixes in GStreamer, which in many cases can be harder to solve rather than implementing new features, though former are more interesting to tell, such as those related with making Rust the main developing language for GStreamer besides C.

Rust bindings and GStreamer elements for Vonage Video API / OpenTok

OpenTok is the legacy name of Vonage Video API, and is a PaaS (Platform As a Service) to ease the development and deployment of WebRTC services and applications.

We published our work in Github of Rust bindings both for the Client SDK for Linux and the Server SDK using REST API, along with a GStreamer plugin to publish and subscribe to video and audio streams.

GstWebRTCSrc

In the beginning there was webrtcbin, an element that implements the majority of W3C RTCPeerConnection API. It’s so flexible and powerful that it’s rather hard to use for the most common cases. Then appeared webrtcsink, a wrapper of webrtcbin, written in Rust, which receives GStreamer streams which will be offered and streamed to web peers. Later on, we developed webrtcsrc, the webrtcsink counterpart: an element which source pads push streams from web peers, such as another browser, and forward those Web streams as GStreamer ones in a pipeline. Both webrtcsink and webrtcsrc are written in Rust.

Behavior-Driven Development test framework for GStreamer

Behavior-Driven Development is gaining relevance with tools like Cucumber for Java and its domain specific language, Gherkin to define software behaviors. Rustaceans have picked up these ideas and developed cucumber-rs. The logical consequence was obvious: Why not GStreamer?

Last year we tinkered with GStreamer-Cucumber, a BDD to define behavior tests for GStreamer pipelines.

GstValidate Rust bindings

There have been some discussion if BDD is the best way to test GStreamer pipelines, and there’s GstValidate, and also, last year, we added its Rust bindings.

GStreamer Editing Services

Though not everything was Rust. We work hard on GStreamer’s nuts and bolts.

Last year, we gathered the team to hack GStreamer Editing Services, particularly to explore adding OpenGL and DMABuf support, such as downloading or uploading a texture before processing, and selecting a proper filter to avoid those transfers.

GstVA and GStreamer-VAAPI

We helped in the maintenance of GStreamer-VAAPI and the development of its near replacement: GstVA, adding new elements such as the H.264 encoder, the compositor and the JPEG decoder. Along with participation on the debate and code reviewing of negotiating DMABuf streams in the pipeline.

Vulkan decoder and parser library for CTS

You might have heard about Vulkan has now integrated in its API video decoding, while encoding is currently work-in-progress. We devoted time on helping Khronos with the Vulkan Video Conformance Tests (CTS), particularly with a parser based on GStreamer and developing a H.264 decoder in GStreamer using Vulkan Video API.

You can check the presentation we did last Vulkanised.

WPE Android Experiment

In a joint adventure with Igalia’s Webkit team we did some experiments to port WPE to Android. This is just an internal proof of concept so far, but we are looking forward to see how this will evolve in the future, and what new possibilities this might open up.

If you have any questions about WebKit, GStreamer, Linux video stack, compilers, etc., please contact us.

I use Home Assistant (HA) to control my WiFi-enabled thermostat, which in turn is walled off from the Internet. So no matter how "smart" my thermostat wants to be, it's forced to live in isolation and follow orders from HA. For the past year, I've been trying out crude methods for detecting when humans are home, or not, so that the HVAC can be set accordingly. The last attempt required using the "ping" integration in HA. You basically provide a list of IPs, add users that are associated with the IP / device tracker for it, then you can conditionally run automation based on the status of those things.

This has worked pretty well, however there are some pain points that have been really digging at me:

1. Setting this up requires using static DHCP for devices on the network that I want to associate with someone being home, like a cell phone. This is tricky since some phones use random MACs, and it's cumbersome to add new people to the "who could be home?" list.

2. If people come over, I don't want the HVAC to shut off if the regular inhabitants leave. E.g. if my partner and I have to leave the house, and my mother-in-law stays at home, I'll get in trouble if the heat shuts off because HA thinks everyone is gone. Don't ask me how I know.

3. Setting this all up requires basically configuring HA to track inhabitants and guests in my home. Even though I've gone through great lengths to try to prevent HA from sending any data externally, I still don't like that this data is being stored anywhere. I actually don't care if my partner or some particular guest is home and I'm not, it's easy enough to ask if I suddenly do. But seeing that someone is home is fine. I've heard that some folks use the "cloud" to manage HA and its data. Ouch.

So I present my latest attempt to alleviate those 3 things as much as possible:

• Monitor IP ranges and detect if any devices in those ranges are "online" (responding to ping)
• Use DHCP to "put" devices that humans carry or use when they are home into the appropriate monitored IP range
• HA can use this to determine if any humans are connected (or not), and thus at home (or not)
• Guests who are on WiFi are counted as "home" by HA without me having to do anything more than just give them the guest WiFi password (which I would have done previously anyways).
• HA doesn't have any information about specific users and their home/not home status. And it doesn't care if devices use a consistent MAC or not.

All three pain points above are addressed... although maybe not 100% solved in every situation. But it still seems like an improvement.

At the heart of this is a script that runs ping in parallel on an arbitrary number of IP addresses given to it. I'm sure it could be improved... Like, I doubt this would turn out well if the list gets too large. And the run time of the script can be kinda long if there are no devices and it ends up retrying the max number of times. The retry is there because I've seen that some devices might not respond to ping immediately (they were sleeping, or had a WiFi reconnect event at the wrong time, etc...) So if you have any ideas, let me know!

#!/bin/bash
#
# Pings the given addresses and prints "up" if any of
# them respond to the ping, else it prints "down". A
# non-zero return value is an error.
# A simple backoff delay is used between attempts to
# ping the group when none of them responded to the
# last attempt.

function pings(){
    local PIDS=()
    local STATUS=()
    local IP=( "$@" )

    for ip in "${IP[@]}"; do
        ping  -c1 -W1 "$ip" &>/dev/null &
        PIDS+=($!)
    done

    i=0
    for p in "${PIDS[@]}"; do
        wait "${p}"
        STATUS[i]=$?
        ((i+=1))
    done

    for s in "${STATUS[@]}"; do
        if [[ $s -eq 0 ]]; then
            return 0
        fi
    done

    return 1
}

if [ -z "$1" ]; then
    echo "Parameter required!!"
    exit 1
fi

ADDRS=( "$@" )
try_interval=1  # starting interval, before backoff
tries=0
max_tries=4

while ! pings "${ADDRS[@]}"; do
    if [[ $tries -ge $max_tries ]]; then
        echo down
        exit 0
    fi

    sleep "$try_interval"
    ((tries+=1))
    ((try_interval*=2))
done

echo up
exit 0

This script should be saved in some directory that HA can access. Next, we need to actually run the script in a way that HA can consume the output directly and use it. A binary sensor seemed like the best choice:

binary_sensor:
  - platform: command_line
    name: 'network_users'
    device_class: presence
    scan_interval: 30 # seconds
    command_timeout: 30 # seconds
    command: 'bash -c "tools/mass_ping.sh 192.168.20.{100..150} 192.168.40.37"'
    payload_on: 'up'
    payload_off: 'down'

The parameters to that script can be ranges that the shell can expand, so you don't have to list out every IP. Triggering every 30 seconds may be too often, I may back that off later to be a minute or more... so I'll run this for a bit to see if I start to form any opinions about changing it.

Next we need to create a "device tracker" so that this can be associated with some "user" in HA for presence detection. I used device_tracker.see in a scheduled automation to add and update a device tracker thing in HA specifically for network users:

- alias: Update presence tracking sensors
  description: This is done periodically so that device trackers for each sensor have
    updated values after HA startup / config reload
  trigger:
  - platform: time_pattern
    minutes: /1
  condition: []
  action:
  - parallel:
    - service: device_tracker.see
      data:
        dev_id: network_users
        location_name: '{{ "home" if is_state("binary_sensor.network_users", "on")
          else "not_home" }}'
      alias: Update Network Users Tracker Status
    alias: Update device trackers
    
    # .... other binary sensors used for determining presence can be listed here too!
  mode: single

I set this automation to fire every 1 minute, so that the current status is "known" to HA soon after it starts up, or config is reloaded. Maybe 1 minute is "too often", but it's just reading a value from the HA database, so, meh. Basically, I've learned that triggering automations based on state changes is flaky, and it's often more reliable to trigger some automations on a schedule and then check entity states in the condition or action.

Lastly, there needs to be a "person" in HA to associate this device tracker with. I created a generic "person" and associated the tracker with it. This can be done through Settings->People->Add Person. I called the fake person I made for this "Pamela", but the name doesn't matter.

The presence status of this person can now be used in automation and/or dashboard cards:

Home Assistant Card showing that someone is connected to the network, and HVAC is enabled by automation

Now, this obviously isn't perfect... One of the issues is that this can lead to the HVAC being left on if someone forgets a phone at home, for example. But that could have happened over the last year with the old implementation, and I've found that 1) it's rare, and 2) the savings from having a "smarter" HVAC outweighs the handful of situations where it's being wasteful. Also, this could be circumvented if some smart guest assigns a static IP outside any expected range I pass to the script above. But if they do that, then the joke is on them since they'll just slowly freeze as the HVAC is shut off because HA thinks no one is home. I do have some other presence detection things that help out too (like IR motion detectors), that can help a little though.

With all of that, HA can tell if someone is home if they're connected to a network, and have an IP from DHCP (or otherwise) that's being monitored. And HA is completely oblivious to any specific people. Again, that's mostly just a personal concern and probably not very existential for my setup, but maybe this will be helpful for folks who do use some external thing for managing HA data, and are concerned about how location status for household members is handled.

How bi­na­ry float­ing points can let us down

If you've played Mo­nop­oly, you'll know abuot the Bank Er­ror in Your Fa­vor card in the Com­mu­ni­ty Chest. Re­mem­ber this?

A bank er­ror in your fa­vor? Sweet! But what if the bank makes an er­ror in its fa­vor? Sure­ly that's just as pos­si­ble, right?

I'm here to tell you that if you're do­ing every­day fi­nan­cial cal­cu­la­tions—noth­ing fan­cy, but in­volv­ing mon­ey that you care about—then you might need to know that us­ing bi­na­ry float­ing point num­bers, then some­thing might be go­ing wrong. Let's see how bi­na­ry float­ing-point num­bers might yield bank er­rors in your fa­vor—or the bank's.

In a won­der­ful pa­per on dec­i­mal float­ing-point num­bers, Mike Col­ishaw gives an ex­am­ple.

Here's how you can re­pro­duce that in JavaScript:

(1.05 * 0.7).toPrecision(2); # 0.73

Some pro­gram­mers might not be aware of this, but many are. By point­ing this out I'm not try­ing to be a smar­ty­pants who knows some­thing you don't. For me, this ex­am­ple il­lus­trates just how com­mon this sort of er­ror might be.

For pro­gram­mers who are aware of the is­sue, one typ­i­cal ap­proache to deal­ing with it is this: Nev­er work with sub-units of a cur­ren­cy. (Some cur­ren­cies don't have this is­sue. If that's you and your prob­lem do­main, you can kick back and be glad that you don't need to en­gage in the fol­low­ing sorts of headaches.) For in­stance, when work­ing with US dol­lars of eu­ros, this ap­proach man­dates that one nev­er works with eu­ros and cents, but only with cents. In this set­ting, dol­lars ex­ist only as an ab­strac­tion on top of cents. As far as pos­si­ble, cal­cu­la­tions nev­er use floats. But if a float­ing-point num­ber threat­ens to come up, some form of round­ing is used.

An­oth­er aproach for a pro­gram­mer is to del­e­gate fi­nan­cial cal­cu­la­tions to an ex­ter­nal sys­tem, such as a re­la­tion­al data­base, that na­tive­ly sup­ports prop­er dec­i­mal cal­cu­la­tions. One dif­fi­cul­ty is that even if one del­e­gates these cal­cu­la­tions to an ex­ter­nal sys­tem, if one lets a float­ing-point val­ue flow int your pro­gram, even a val­ue that can be trust­ed, it may be­come taint­ed just by be­ing im­port­ed into a lan­guage that doesn't prop­er­ly sup­port dec­i­mals. If, for in­stance, the re­sult of a cal­cu­la­tion done in, say, Post­gres, is ex­act­ly 0.1, and that flows into your JavaScript pro­gram as a num­ber, it's pos­si­ble that you'll be deal­ing with a con­t­a­m­i­nat­ed val­ue. For in­stance:

(0.1).toPrecision(25) # 0.1000000000000000055511151

This ex­am­ple, ad­mit­ted­ly, re­quires quite a lot of dec­i­mals (19!) be­fore the ugly re­al­i­ty of the sit­u­a­tion rears its head. The re­al­i­ty is that 0.1 does not, and can­not, have an ex­act rep­re­sen­ta­tion in bi­na­ry. The ear­li­er ex­am­ple with the cost of a phone call is there to raise your aware­ness of the pos­si­bil­i­ty that one doesn't need to go 19 dec­i­mal places be­fore one starts to see some weird­ness show­ing up.

There are all sorts of ex­am­ples of this. It's ex­ceed­ing­ly rare for a dec­i­mal num­ber to have an ex­act rep­re­sen­ta­tion in bi­na­ry. Of the num­bers 0.1, 0.2, …, 0.9, only 0.5 can be ex­act­ly rep­re­sent­ed in bi­na­ry.

Next time you look at a bank state­ment, or a bill where some tax is cal­cu­lat­ed, I in­vite you to ask how that was cal­cu­lat­ed. Are they us­ing dec­i­mals, or floats? Is it cor­rect?

I'm work­ing on the dec­i­mal pro­pos­al for TC39 to try to work what it might be like to add prop­er dec­i­mal num­bers to JavaScript. There are a few very in­ter­est­ing de­grees of free­dom in the de­sign space (such as the pre­cise datatype to be used to rep­re­sent these kinds of num­ber), but I'm op­ti­mistic that a rea­son­able path for­ward ex­ists, that con­sen­sus be­tween JS pro­gram­mers and JS en­gine im­ple­men­tors can be found, and even­tu­al­ly im­ple­ment­ed. If you're in­ter­est­ed in these is­sues, check out the README in the pro­pos­al and get in touch!

Igalia started the Reforestation project in 2019. As one of the company’s Corporate Social Responsibility (CSR) efforts, the Reforestation project focuses on conserving and expanding native, old growth forests to capture, and long-term storing, carbon emissions. Partnering with Galnus, we have been working on reforestation of a 10.993 hectares area with 11,000 trees, which absorbs 66 tons of carbon dioxide each year.

Phase I – Rois

The first land where the project started was framed in the woods of the communal land of “San Miguel de Costa” in Rois. Rois is a municipality of northwestern Spain in the province of A Coruña in the autonomous community of Galicia. Environment in this land was highly altered by human action and the predominance of eucalyptus plantations left few examples of native forest.

After the agreement for land transfer signed on the 29th June 2019, the project started with creating maps and technical plans for the areas affected. Purposes of these plans are not only to build a cartographic base for management and accomplishment of the works but also to designate accurately these zones as “Reserved Areas” in future management plans.

Work carried out

• Clear and crush remaining eucalyptus and other exotic plants

Eucalyptus is an exotic and invasive species. It spread widely and had a negative impact on the environmental situation of this habitat. This work is to clear the existing growth and eliminate the regrowths of eucalyptus and other exotic plants before the start of planting work. In Q1 2020, 100% of exotic trees were cut as part of the Atlantic forest sponsorship. After that the focus was moved to eliminating the sprouts of eucalyptus grown from the seeds present in the soil in early years. Sprout and seeds elimination continues throughout the duration of the project.

• Acquire trees, shrubs and all other materials needed.

To improve forest structure and increase biodiversity for the affected area, the following native trees and shrubs were chosen and purchased in Q3 2019 –
– 1000 downy birch (Betula pubescens)
– 650 common oak (Quercus robur)
– 350 wild cherry (Prunus avium)
– 150 common holly (Ilex aquifolium)

A lot of other materials such as tree protectors, stakes etc. were also acquired at the time time.

• Planting

Planting started in Q4 2019, downy birches (Betula pubescens) and some common oaks (Quercus robur) were the first batch planted. This planting campaign continued in 2020 on the arrival of the rest of the trees.

• Care during the first year

The first year after planting is key to the future development and success of the project.

In Q1 2020, 100% of planted trees were already sprouting and alive. At this stage 100% of the restoration area has already been planted with at least 65% of the trees.

The first summer after planting is vital for the trees and shrubs to settle. During the first summer, trees adapt and take root in their new location. If they survive, their chances of developing correctly will increase exponentially and the following years they will be able to focus all their energy on their growth. The beginning of the summer 2020 was difficult due to hot and dry weather. However, the tree mortality rate remains within the expected range.

In Q4 2020, most of the planted trees are well-established and ready for spring sprouting. By Q1 2021 many of them have reached over 2 meters high.

The trees and shrubs settled happily after the first year of growth. This is the photo taken in Spring 2020

And see how they had grown in a year’s time –

• Wildlife studies

With plants sprouting all around the forest, insects, reptiles and birds start thriving too. The development of the ecosystem will provide for better results in future inventories and wildlife studies. In summer 2020 the wildlife studies started and first inventory was completed. In winter 2020 wildlife nest boxes were in the manufacturing process and some bat boxes were installed in Spring 2021.

• Improving biodiversity

In addition to ensuring the introduction of a wide variety of tree species to reforestation areas, the project has also put effort on enhancing biodiversity in existing forest areas. For example, One of the areas targeted by the project is one hectare of young secondary forest made up mainly of oaks (Quercus robur). In this forest we have been planting understory species such as holly (Ilex aquifolium), laurel (Laurus nobilis) or Atlantic pear (Pyrus cordata) to increase biodiversity and improve the structure and complexity of the forest.

Phase II – Eume and Rois

Phase II of the reforestation project set sail in winter 2020. While the Rois project had moved to a steady stage with most trees and shrubs planted and settled through the first year’s growth, this phase is to expand restoration of new forest area in Rois and to start a new project in “Fragas do Eume” natural park.

• Rois

In order to explain the progress in the Rois expansion project, the maps below distinguish three areas in different stages of development.
– Phase 1 (Green) – The green area is completely planted with native trees and free of eucalyptus sprouts.
– Phase 2 (Yellow) – Acacias have been eliminated in this area. The work to control this species and also the eucalyptus trees will continue.
– Phase 3 (Red) – The entire area is covered with dense plantations of eucalyptus and acacia. Work is in the preparation stage.

The following maps represent the progresses made between Spring to Fall in 2021.

Spring 2021:

Fall 2021:

• Eume

Unlike the Rois project, habitats in “Fragas do Eume” natural park are some of the largest and best preserved examples in the world of coastal Atlantic rainforest, where much of its biodiversity and original structure is still preserved. Unfortunately, most of these forests are young secondary forests under numerous threats such as the presence of eucalyptus and other invasive species.

Our project represents one of the largest actions in recent years for the elimination and control of exotic species and the restoration of native forest in the area, increasing native forest surface area, reconnecting fragmented forest patches and improving the landscape in the “Fragas do Eume Natural Park”.

In order to explain the progress in the Eume project, it also distinguishes three areas in different stages of development.

Phase 1 (Green) – All the environmental restoration works have been completed. Only maintenance tasks remain for the next few years.
phase 2 (Yellow) – This area is in the process of control and elimination of eucalyptus to start the restoration of native vegetation with native trees plantation works.
Phase 3 (Red) – Waiting for the loggers to cut the big eucalyptus trees in this area, in order to start the work associated with the project.

The following maps represent the progresses made between Spring to Fall in 2021.

Spring 2021:

Fall 2021:

Esmelle

In 2021 Igalia started working on a 0.538-hectare land owned by the Esmelle Valley Association. This land is borded by a road and a forest track on each side, and the Esmelle stream flows across it. This once mighty stream and its tributary springs are currently used to provide drinking water to the houses in the surrounding area. Unfortunately, in summer time, the flow of the stream could reduce drastically due to dry weather and exhausting uses. In addition, this area also suffers invasion of numerous eucalyptus trees and other exotic species. Recovering and restoring a good example of the Atlantic Forest will bring great benefit to this enormously altered and humanized case.

After work preparation and planning, major work were carried out in 2022 including –
– Elimination and control of eucalyptus and its sprouts.
– Clearing and ground preparation
– Tree seedling, plantation and protections (placement of tree stakes and protectors etc.)
– Maintenance and replacement of dead trees

This work will continue in the next two years. Main focus will be on reinforcement and enrichment of the plantation based on the availability of tree seedlings of certain species, and finishing all the remaining maintenance work.

What’s next?

Igalia sees Reforestation as a long term effort. While maintaining and developing the current projects, Igalia doesn’t stop looking for new candidates. In Q4 2022, Igalia started preparing a new Reforestation project – Galnus: O Courel.

This is the first time that Igalia considered developing an environmental compensation project in one of the most rural and least densely populated areas of Galicia. Igalia believes that the environmental, social and economic characteristics associated with this area, offers an opportunity to carry out
environmental restoration projects. If it goes as planned, major work will happen in the next two years. Something to look forward to!

Impacts

The Reforestation project is making contributions to our environment, and it has gone further. Here we’d like to share a picture of a 7-year-old boy’s work. This young student is from the community owning the Rois forest. He took advantage of a school newspaper project to communicate about our work in Rois forest.

Isn’t it a joy to see Iglaia’s Reforestation project is making impacts on our children, and our future?

The RISC-V instruction set architecture has seen huge excitement and growth in recent years (10B cores estimated to have shipped as of Dec 2022) and I've been keeping very busy with RISC-V related work at Igalia. I thought it would be fun to look beyond the cores I've been working with and to enumerate the SoCs that are available for direct purchase or in development boards that feature RISC-V cores programmable by the end user. I'm certain to be missing some SoCs or have some mistakes or missing information - any corrections very gratefully received at asb@muxup.com or @asbradbury. I'm focusing almost exclusively on the RISC-V aspects of each SoC - i.e. don't expect a detailed listing of other on-chip peripherals or accelerators.

A few thoughts

• It was absolutely astonishing how difficult it was to get basic information about the RISC-V specification implemented by many of these SoCs. In a number of cases, just a description of a "RV32 core at xxMHz" with further detective work being needed to find any information at all about even the standard instruction set extensions supported.
• The way I've focused on the details of individual cores does a bit of a disservice to those SoCs with compute clusters or interesting cache hierarchies. If I were to do this again (or if I revisit this in the future), I'd look to rectify that. There a whole bunch of other micro-architectural details it would be interesting to detail too.
  • I've picked up CoreMark numbers where available, but of course that's a very limited metric to compare cores. It's also not always clear which compiler was used, which extensions were targeted and so on. Plus when the figure is taken from an IP core product page, the numbers may refer to a newer version of the IP. Where there are multiple numbers floating about I've listed them all.
• There's a lot of chips here - but although I've likely missed some, not so many that it's impossible to enumerate. RISC-V is growing rapidly, so perhaps this will change in the next year or two.
• A large proportion of the SoC designs listed are based on proprietary core designs. The exceptions are the collection of SoCs based on the T-Head cores, the SiFive E31 and Kendryte K210 (Rocket-derived) and the GreenWaves GAP8/GAP9 (PULP-derived). As a long-term proponent of open source silicon I'd hope to see this change over time. Once a company has moved from a proprietary ISA to an open standard (RISC-V), there's a much easier transition path to switch from a proprietary IP core to one that's open source.

64-bit Linux-capable application processors

• StarFive JH7110
  • Core design:
    • 4 x RV64GC_Zba_Zbb SiFive U74 application cores, 1 x RV64IMAC SiFive S7 (this?) monitor core, and 1 x RV32IMFC SiFive E24 (ref, ref).
    • The U74 is a dual-issue in-order pipeline with 8 stages.
  • Key stats:
    • 1.5 GHz, fabbed on TSMC 28nm (ref).
    • StarFive report a CoreMark/MHz of 5.09.
  • Development board:
    • Available in the VisionFive 2 and Pine64 Star64 development boards.
• T-Head C910 ICE
  • Core design:
    • 2 x RV64GC T-Head C910 application cores and an additional T-Head C910 RV64GCV core (i.e., with the vector extension).
    • The C910 is a 3-issue out-of-order pipeline with 12 stages.
  • Key stats:
    • 1.2 GHz, fabbed on a 28nm process (ref).
  • Development board:
    • Available in the RVB-ICE.
• Allwinner D1-H (datasheet, user manual)
  • Core design:
    • 1 x RV64GC T-Head C906 application core. Additionally supports the unratified, v0.7.1 RISC-V vector specification (ref).
    • Single-issue in-order pipeline with 5 stages.
    • Verilog for the core is on GitHub under the Apache License (see discussion on what is included).
    • At least early versions of the chip incorrectly trapped on fence.tso. It's unclear if this has been fixed in later revisions.
  • Key stats:
    • 1GHz, taped out on a 22nm process node.
    • Reportedly 3.8 CoreMark/MHz.
  • Development board:
    • Available in a number of development boards (e.g. Allwinner Nezha, SiPeed Lichee RV, and more).
• StarFive JH7100
  • Core design:
    • 2 x RV64GC SiFive U74 application cores and 1 x RV32IMAFC SiFive E24.
    • The U74 is a dual-issue in-order pipeline with 8 stages.
  • Key stats:
    • 1.2GHz (as listed on StarFive's page but articles about the V1 board claimed 1.5GHz), presumably fabbed on TSMC 28nm (the JH7110 is).
    • The current U74 product page claims 5.75 CoreMark/MHz (previous reports suggested 4.9 CoreMark/MHz).
  • Development board:
    • Available on the VisionFive V1 development board.
• Kendryte K210 (datasheet)
  • Core design:
    • 2 x RV64GC application cores (reportedly implementations of the open-source Rocket core design).
    • If it's correct the K210 uses Rocket, it's a single-issue in-order pipeline with 5 stages.
    • Has a non-standard (older version of the privileged spec?) MMU (ref, so the nommu Linux port is typically used.
  • Key stats:
    • 400MHz, fabbed on TSMC 28nm.
  • Development board:
    • Available in development boards from SiPeed.
• MicroChip PolarFire SoC MPFSxxxT
  • Core design:
    • 4 x RV64GC SiFive U54 application cores, 1 x RV64IMAC SiFive E51 (now renamed to S51) monitor core.
    • The U54 is a single-issue, in-order pipeline with 5 stages.
  • Key stats:
    • 667 MHz (ref), fabbed on a 28nm node (ref).
    • Microchip report 3.125 CoreMark/MHz.
  • Development board:
    • Available in the 'Icicle' development board.
• SiFive FU740
  • Core design:
    • 4 x RV64GC SFive U74 application cores, 1 x RV64IMAC SiFive S71 monitor core.
    • It's hard to find details for the S71 core, the FU740 manual refers to it as an S7 while the HiFive Unmatched refers to it as the S71 - but neither have a page on SiFive's site. I'm told that the S71 has the same pipeline as the S76, just no support for the F and D extensions.
    • The U74 is a dual-issue in-order pipeline with 8 stages.
  • Key stats:
    • 1.2 GHz, fabbed on TSMC 28nm (ref).
    • The current U74 product page claims 5.75 CoreMark/MHz (previous reports suggested 4.9 CoreMark/MHz).
  • Development board:
    • Was available on the now-discontinued HiFive Unmatched development board.
• SiFive FU540
  • Core design:
    • 4 x RV64GC SiFive U54 application cores, 1 x RV64IMAC SiFive E51 (now renamed to S51) monitor core.
    • The U54 is a single-issue, in-order pipeline with 5 stages.
  • Key stats:
    • 1.5GHz, fabbed on TSMC 28nm (ref).
    • The current U54 product page claims 3.16 CoreMark/MHz but it was 2.75 in 2017.
  • Development board:
    • Was available on the now-discontinued HiFive Unleashed development board.
• Bouffalo Lab BL808
  • Core design:
    • 1 x RV64IMAFCV T-Head C906 application core, 1 x RV32IMAFCP T-Head E907, 1 x RV32E?? T-Head E902 (ref).
    • See notes elsewhere for more info on these T-Head core designs.
  • Key stats:
    • The C906 runs at 480 MHz, the E907 at 320 MHz and the E902 at 150 MHz.
  • Development board:
    • Available in the Ox64 from Pine64.
• Renesas RZ/Five
  • Core design:
    • 1 x RV64GC Andes AX45MP application core with additional Andes extensions (draft of the 'P' packed SIMD spec, Andes 'V5' extensions).
    • Unfortunately has a hardware bug where virtual addresses in a certain range are always treated as physical addresses.
    • Dual issue, in-order pipeline with 8 stages.
  • Key stats:
    • 1.0 GHz, 5.63 CoreMark/MHz (ref).
  • Development board:
    • Evaluation kit available.
• Kendryte K510
  • Core design:
    • 2 x RV64GC application cores and 1 x RV64GC core with DSP extensions. The Andes AX25MP appears to be used (ref).
  • Key stats:
    • 800 MHz.
  • Development board:
    • K510 CRB-KIT developer kit is available.
• (Upcoming) Intel-SiFive Horse Creek SoC
  • Core design:
    • 4 x "RV64GBC" SiFive P550 application cores (docs not yet available, but an overview is here). As the 'B' extension was broken up into smaller sub-extensions, this is perhaps RV64GC_Zba_Zbb like the SiFive U74.
    • 13 stage, 3 issue, out-of-order pipeline.
    • As the bit manipulation extension was split into a range of sub-extensions it's unclear exactly which of the 'B' family extensions will be supported.
  • Key stats:
    • 2.2 GHz, fabbed on Intel's '4' process node (ref).
  • Development board:
    • Will be available in the HiFive Pro P550 development board.
• (Upcoming) T-Head TH1520 (announcement)
  • Core design:
    • 4 x RV64GC T-Head C910 application cores, 1 x RV64GC T-Head C906, 1 x RV32IMC T-Head E902.
    • The C910 is a 3-issue out-of-order pipeline with 12 stages, the C906 is single-issue in-order with 5 stages, and the E902 is single-issue in-order with 2 stages.
    • Verilog for the cores is up on GitHub under the Apache license: C910, C906, E902. See discussion on what is included).
  • Key stats:
    • 2.4 GHz, fabbed on a 12nm process (ref).
  • Development board:
    • Will be available on the SiPeed Lichee Pi 4A.

Embedded / specialised SoCs (mostly 32-bit)

• SiFive FE310
  • Core design:
    • 1 x RV32IMAC SiFive E31 core.
    • Single-issue, in-order pipeline with 5 stages.
    • Derived from the open source Rocket core design (ref).
  • Key stats:
    • 320 MHz (ref), fabbed on TSMC 180nm (ref),
    • 2.73 CoreMark/MHz in the FE310-G002 datasheet, 3.17 CoreMark/MHz on the E31 product page.
  • Development board:
    • Available in development boards such as the HiFive 1 Rev B.
• GigaDevice GD32VF103 series
  • Core design:
    • 1 x RV32IMAC Nuclei Bumblebee N200 ("jointly developed by Nuclei System Technology and Andes Technology.")
    • No support for PMP (Physical Memory Protection), includes the 'ECLIC' interrupt controller derived from the CLIC design.
    • Single-issue, in-order pipeline with 2 stages.
  • Key stats:
    • 108 MHz. 360 CoreMark (ref), implying 3.33 CoreMark/MHz.
  • Development board:
    • Available in development boards such as the Sipeed Longan Nano.
• GreenWaves GAP8
  • Core design:
    • Fabric Controller (FC) core and compute cluster of 8 cores based on PULP. Implements RV32IMC with alongside additional custom extensions.
  • Key stats:
    • 175 MHz Fabric Controller, 250 MHz cluster. Fabbed on TSMC's 55nm process (ref).
    • 22.65 GOPS at 4.24mW/GOP (ref).
    • Shipped 150,000 units, composed of roughly 80% open source and 20% proprietary IP (ref).
  • Development board:
    • Available in the GAPuino.
• GreenWaves GAP9
  • Core design:
    • Fabric Controller (FC) and compute cluster of 9 cores. Extends the RV32IMC (plus extensions) GAP8 core design with additional custom extensions (ref).
  • Key stats:
    • 400 MHz Fabric Controller and computer cluster. Fabbed on Global Foundries 22nm FDX process (ref).
    • 150.8 GOPS at 0.33mW/GOP (ref).
  • Development board:
    • GAP9 evaluation kit listed on Greenwaves store but you must email to receive access to order it.
• Renesas RH850/U2B
  • Core design:
    • Features an NSITEXE DR1000C RISC-V parallel coprocessor, comprised of RV32I scalar processor units, a control core unit, and a vector processing unit based on the RISC-V vector extension.
  • Key stats:
    • 400 MHz. Fabbed on a 28nm process.
  • Development board:
    • None I can find.
• Renesas R9A02G020
  • Core design:
    • 1 x RV32IMC AndesCore N22 (additionally supporting the Andes 'Performance' and 'CoDense' instruction set extensions).
    • Single-issue in-order with a 2-stage pipeline.
  • Key stats:
    • 32 MHz.
  • Development board:
    • The R9A02G020-EVK motor control kit is available.
• Analog Devices MAX78000
  • Core design:
    • Features an RV32 RISC-V coprocessor of unknown (to me!) design and unknown ISA naming string.
  • Key stats:
    • 60 MHz (for the RISC-V co-processor), fabbed on a TSMC 40nm process (ref).
  • Development board:
    • MAX78000EVKIT and MAX78000FTHR.
• Espressif ESP32-C3 / ESP8685
  • Core design:
    • 1 x RV32IMC core of unknown design, single issue in-order 4-stage pipeline (ref).
  • Key stats:
    • 160 MHz, fabbed on a TSMC 40nm process (ref).
    • 2.55 CoreMark/MHz.
  • Development board:
    • Available in a number of formats including the ESP32-C3-DevKitM-1 and the M5Stack M5Stamp C3.
• Espressif ESP32-C2 / ESP8684
  • Core design:
    • 1 x RV32IMC core of unknown design, single issue in-order 4-stage pipeline (ref).
  • Key stats:
    • 160 MHz, fabbed on a TSMC 40nm process.
    • 2.55 CoreMark/MHz.
    • Die photo available in this article.
  • Development board:
    • Available in the ESP8684-DevKitM-1-H4.
• Espressif ESP32-C6
  • Core design:
    • 1 x RV32IMAC core of unknown design (four stage pipeline) and 1 x RV32IMAC core of unknown design (two stage pipeline) for low power operation (ref).
  • Key stats:
    • 160 MHz high performance (HP) core with 2.76 CoreMark/MHz, 20 MHz low power (LP) core.
  • Development board:
    • Available in the ESP32-C6-DevKitC-1-N8.
• HiSilicon Hi3861
  • Core design:
    • 1 x RV32IM core of unknown design, supporting additional non-standard compressed instruction set extensions (ref).
  • Key stats:
    • 160 MHz.
  • Development board:
    • A low-cost board is available advertising support for Harmony OS.
• Bouffalo Lab BL616/BL618
  • Core design:
    • 1 x RV32GC core of unknown design, also with support for the unratified 'P' packed SIMD extension (ref).
  • Key stats:
    • 320 MHz.
  • Development board:
    • Available in the Sipeed M0S module.
• Bouffalo Lab BL602/BL604
  • Core design:
    • 1 x RV32IMAFC SiFive E24 core (ref).
    • 3-stage pipeline.
  • Key stats:
    • 192 MHz, 3.1 CoreMark/MHz (ref).
  • Development board:
    • Available in the very low cost Pinecone evaluation board.
  • Other:
    • The BL702 appears to have the same core, so I haven't listed it separately.
• Bluetrum AB5301A
  • Core design:
    • 1 x RV32 core of uknown design and unknown ISA naming string (CNX Software tried and failed to get clarification.
  • Key stats:
    • 125 MHz.
  • Development board:
    • Available in the AB32VG1 development board.
• WCH CH583/CH582/CH581
  • Core design:
    • 1 x RV32IMAC QingKe V4a core, which also supports a "hardware prologue/epilogue" extension.
    • Single issue in-order pipeline with 2 stages.
    • Unlike the other QingKe V4 series core designs, the V4a doesn't support the custom 'extended instruction' (XW) instruction set extension.
  • Key stats:
    • 20MHz.
  • Development board:
    • An evaluation board is available.
• WCH CH32V307
  • Core design:
    • 1 x RV32IMAFC QingKe V4f core.
  • Key stats:
    • 144 MHz.
  • Development board:
    • An evaluation board is available.
• WVH CH32V208
  • Core design:
    • 1 x RV32IMAC QingKe V4c core with custom instruction set extensions ('XW' for sign-extended byte and half word operations).
  • Key info:
    • 144 MHz.
  • Development board:
    • Available here.
  • Other:
    • The CH32V203 is also available but I haven't listed it separately as it's not clear how the QingKe V4b core in that chip differs to the V4c in this one.
• WCH CH569 / WCH CH573 / WCH CH32V103
  • Core design:
    • 1 x RV32IMAC QinqKe V3a core.
  • Key stats:
    • 120 MHz (CH569), 20 MHz (CH573), 80 MHz (CH32V103).
  • Development board:
    • Evaluation boards available, e.g. this one for the CH32V103.
• WCH CH32V003
  • Core design:
    • 1 x RV32EC QingKe V2A with custom instruction set extensions ('XW' for sign-extended byte and half word operations).
  • Key stats:
    • 48 MHz.
    • Notably announced as costing less than 10 cents.
  • Development board:
    • Various boards available.
• PicoCom PC802
  • Core design:
    • Two clusters of 16 RV32IMAC Andes N25F cores (ref).
    • Single issue, in order pipeline with 5 stages.
    • Also see the 2021 RISC-V summit talk for more insight on the architecture.
  • Key stats:
    • Fabbed on TSMC 12nm process (ref).
  • Development board:
    • Several boards are available.
• CSM32RV20
  • Core design:
    • 1 x RV32IMAC core of unknown design.
  • Key stats:
    • 2.6 CoreMark/MHz.
  • Development board:
    • The product page suggests two boards are available.
• HPMicro HPM6750
  • Core design:
    • 2 x RV32IMAFDC cores (AndesCore D45) with an implementation of the draft 'P' packed SIMD spec (ref).
    • In-order dual-issue 8-stage pipeline.
  • Key stats:
    • 800 MHz
  • Development board:
    • Available in the HPM6750EVK as well as other variants.
• Renesas R9A06G150
  • Core design:
    • 1 x RV32IMAFC AndesCore D25F with additional vendor-specific instruction set extensions.
    • 5 stage pipeline, single issue in-order.
  • Key stats:
    • 100 MHz
  • Development board:
    • None available currently.