Leaving tomorrow, as I will be first at the Evince Hackfest

Time for a change! Almost 10 years ago I was starting to hack on a Blog engine with two friends, it was called Alinea and it powered this website for a long time. Back then hacking on your own Blog engine was the pre-requirement to host your blog :) But nowadays people just use Wordpress or similar platforms, if they still have a blog at all. Alinea fell into oblivion as I didn’t have time and motivation to maintain it.

Moving to Pelican was quite easy, since I’ve been writing content in ReST on the previous blog I only had to pull data from the database and hacked pelican_import.py a bit :)

Now that this website looks almost modern I’ll hopefully start to blog again, expect at least news about the WebKit work I still enjoy doing at Igalia.

For some time now I have decided to focus my work at Igalia on the graphics stack. As a result of this I had the chance to participate in a couple of very interesting projects like implementing Wayland support in WebKitGtk+ (a topic I have visited in this blog a number of times) and, lately, work on graphics drivers for Linux in the Mesa framework.

The graphics stack in Linux is complex and it is not always easy to find information and technical documentation that can aid beginners in their firsts steps. This is usually a very demanding domain, the brave individuals who decide to put their energy into it usually have their hands full hacking on the code and they don’t have that much room for documenting what they do in a way that is particularly accessible to newcomers.

As I mentioned above, I have been hacking on Mesa lately (particularly on the Intel i965 driver) and so far it as been a lot of fun, probably the most exciting work I have done at Igalia in all these years, but it is also certainly challenging, requiring me to learn a lot of new things and some times fairly complex stuff.

Getting involved in this is no easy endeavor, the learning curve is steep because the kind of work you do here is probably unlike anything you have done before: for starters it requires a decent understanding of OpenGL and capacity to understand OpenGL specifications and what they mean in the context of the driver, you also need to have a general understanding of how modern 3D-capable GPUs work and finally, you have to dig deeper and understand how the specific GPU that your driver targets works and what is the role that the driver needs to play to make that hardware work as intended. And that’s not all of it, a driver may need to support multiple generations of GPUs which sometimes can be significantly different from each other, requiring driver developers to write and merge multiple code paths that handle these differences. You can imagine the maintenance burden and extra complexity that comes from this.

Finally, we should also consider the fact that graphics drivers are among the most critical pieces of code you can probably have in a system, they need to be performant and stable for all supported hardware generations, which adds to the overall complexity.

All this stuff can be a bit overwhelming in the beginning for those who attempt to give their first steps in this world but I believe that this initial steep learning curve can be smoothed out by introducing some of the most important concepts in a way that is oriented specifically to new developers. The rest will still not be an easy task, it requires hard work, some passion, be willing to learn and a lot of attention to detail, but I think anyone passionate enough should be able to get into it with enough dedication.

I had to go through all this process myself lately, so I figured I am in a very good situation to try and address this problem myself, so that’s why I decided to write a series of posts to introduce people to the world of Mesa and 3D graphics drivers, with a focus on OpenGL and Intel GPUs, which is the area were I am currently developing my work. Although I’ll focus on Intel hardware I believe that many of the concepts that I will be introducing here are general enough so that they are useful also to people interested in other GPUs. I’ll try to be clear about when I am introducing general concepts and when I am discussing Intel specific stuff.

My next post, which will be the first in this series, will serve as an introduction to the Linux graphics stack and Linux graphics drivers. We will discuss what Mesa brings to the table exactly and what we mean when we talk about graphics drivers in Linux exactly. I think that should put us on the right track to start looking into the internals of Mesa.

So that’s it, if you are interested in learning more about Linux graphics and specifically Mesa and 3D graphics drivers, stay tuned! I’ll try my best to post regularly and often.

The first release candidates for LibreOffice 4.3.0 are already bouncing around the internet and, besides great new features like the ones I’ve been explaining in my latest posts, they come with a set of fixes to ease the life of screen reader users and developers alike. This is, once again, a part of the accessibility work we do at Igalia.

Back in April, when the Gran Canaria hackfest took place, I started working on the bug #71556. The problem was that typing on a document triggered a lot of unnecessary text-attributes-changed events. These events had a variety of origins, but most of them were caused by modifications in the internal text attribute rsid, used for change tracking and with no relevance per se for the user. The bug is not completely fixed, but addressing the problem with rsid attribute allowed us to get rid of the most annoying part of it; now LibreOffice only sends one unnecessary event when we type the first character in a new paragraph and not with every keystroke.

Bug #71558 is also related with the same kind of events; in this case, text-attributes-changed was not being triggered when a word became marked as misspelled. Actually, the spell-checking status is not internally treated as a text attribute and because of that there were no events indicating its change. The patch explicitly raises the event which lets the accessibility code check the status of the text attributes and find out the spelling mistake. While I was working on this issue, I also detected a weird behavior when checking the text attributes through the Python API; it resulted to be a bug in the bridge between AT-SPI and ATK, which I reported and fixed too.

A triaging session took me to bug #74681; the main issue reported there had already been fixed for a while and only small bits regarding missing accessible names in some buttons were missing. I fixed that allowing toolbar buttons to use their tooltip text as the accessible name if it is not explicitly set, and now the paragraph properties panel is fully accessible.

Finally, I retook the work I had been doing in relation with ATK roles at bug #39944 and detected wrong mappings for LibreOffice EDIT_BAR, EMBEDDED_OBJECT and HYPER_LINK accessible roles. I fixed them and opened a ticket in ATK bugzilla to create ATK roles for the five cases that were still registering custom roles. Once that ticket is managed, hopefully we will be able to close bug #75191 too, which is related with the deprecation of atk_role_register.

These fixes are added on top of the ones coded in February hackfest, making 4.3.0 the most accessible LibreOffice so far… Until the next version arrives, of course!

Even thought it has been a while since my last entry on this blog, I have been quite busy. During most of last year I brought my modest contributions into an awesome startup that you have probably heard of by now, helping them integrate GNOME technologies into their products. I was lucky to join their team at an early stage of the development and participate in key discussions.

But more on that project on future entries.

Today I want to talk about things that keep me busy these days, and are of course related to Web engines. Specifically, I want to talk about 2D rasterization and the process of putting pixels on the screen as fast as possible (aka, the 60 frames-per-second holy grail). I want to discuss an idea that has the potential to significantly increase the performance of page rendering by utilizing modern GPU capabilities, OpenGL, and a bit of help from Web engines’ magic.

This is a technical article, so if you are not very familiar with 2D rasterization, OpenGL or how Web engines draw stuff, I recommended you to take some time off and read about it. It is truly a world of wonders (and sometimes pain).

Instanced rendering

The core of the idea is based on instanced rendering. It is a fairly well known technique introduced by OpenGL 3.1 and OpenGL-ES 3.0 as extension GL_EXT_draw_instanced.

To draw geometry with OpenGL, one normally submits a primitive to the rendering pipeline. The primitive consists of a collection of vertices, and a number of attributes per each vertex. Traditionally, you could only submit one primitive at a time.

With instanced rendering, it is possible to send several “instances” of the same primitive (the same collection of vertices and attributes) on a single call. This dramatically reduces the overhead of pipeline state changes and gives the GPU driver a better chance at optimizing rendering of instances of a particular geometry.

Hence, it is generally a common practice for OpenGL applications to group rendering of similar geometry into batches, and submit them to the pipeline all at once as instances. This technique is commonly known as batching and merging.

Skia, the 2D rasterizer used by the Chromium and Android projects, and Cairo, a popular 2D rasterizer backing many projects such as GNOME and previous versions of Mozilla Firefox; both to some extent have support for some sort of instanced rendering in their respective GL backends.

Telling instances apart

Ok, it is possible to draw a bunch of primitives at once, but how can we make them look different? A way of customizing individual instances is necessary, otherwise they will all render on top of the previous one. Not very useful.

There are two ways of submitting per-instance information: one is by adding a “divisor” to the buffers containing vertex or attribute information (VBOs), which will tell the pipeline to use the divided chunks as per-instance information instead of per-vertex. glVertexAttribDivisor is used in this case.

The other way is to upload the per-instance information to a buffer texture (or any texture for that matter) and fetch the information of the corresponding vertex by sampling, using a new variable gl_InstanceID available in shader code, as the key. This variable will increase for each instance of the geometry being rendered (as oppose to per vertex, for which you have gl_VertexID).

So, a quick recap so far. We are able to draw several instances of the same geometry at once, very efficiently, and are able to upload arbitrary data to customize each of these instances at will.

But wait, there are caveats.

The ordering problem

So, lets say we can now group together all drawing operations that involve the same primitive (rectangle, line, circle, etc). What happens if we draw (say) a filled rectangle, then a circle on top, and then another rectangle on top of the circle?

Following the simple grouping rule, what will happen is that the two rectangles will be grouped together and drawn first in one call, then the circle. This will not render the expected result, since the circle will end up laying on top of the rectangle that was drawn after it.

This problem is commonly known as “ordering”, and it clearly breaks our otherwise super-performing batching and merging case.

So, in scenes that involve lots of geometry overlapping, the grouping is limited to contents that do not overlap, if we wanted to preserve the right order of operations.

In practice, it means that we first need to separate the content in layers, then group the same primitives within a single layer, and finally submit the batches from each layer in the right order.

But guess what? Browser engines already do exactly that. Engines build a layer tree (among several other trees) with the information contained in the HTML and CSS content (layout, styling, transformations, etc), where the content is separated in render nodes whose content do not normally overlap. The actual process is much more complicated than that, but this simplification is enough to illustrate the idea.

Now, what if?

First, for the sake of presenting an idea, lets ignore the 2D context of a canvas element by now. More on that later. Lets focus on most of the web sites out there.

If we look at the number of primitives typically used by the rendering of a page, they boil down to a handful. Essentially, they are:

• A rectangle: for almost all HTML elements, which are boxes. And character glyphs! which are normally rendered ahead of time and cached in a texture layout, then texture-mapped onto a rectangle. And images!, which are also texture-mapped onto rectangles.
• A thin line: for thin (<=1 pixel) borders, inset/outset effect, hr, etc. Thicker borders can be drawn as thin rectangles.
• A round corner: the quarter of a circle, filled or stroke, used to implement rounded rectangles (hello border-radius).
• A circle: for bulleted listings, radio-buttons, etc. Argueably, these can be rendered using unicode characters, so no need for specific geometry.

Lets stay with these. There are other cases that I will purposely ignore, like one seen in a rounded rectangle with different thickness in two consecutive borders.

Then we have, for each of these primitives, an evolutionary-like variety of background styles (imaged, colored, repeated, gradient, etc); transformations (rotation, translation, scaling, etc); border styles (again imaged, colored, with different thickness, etc), shadow and blurring effects, and so on.

With a working texture cache, we have a potentially good chance at aggressively grouping together drawing of these primitives, like rectangles for example, for all text glyphs, boxes and images.

So, what if we could submit to a smart shader all the information that describes and tells apart these grouped instances? Is it possible to efficiently pack and then re-interpret in a shader all the styling and transformation complexities of today's CSS-styled HTML elements?

A new approach

Existing 2D rasterizers used in Web engines (at least Skia and Cairo, whose source code is available to me) are general purpose drawing libraries. That means they should render deterministically for any kind of application, not only Web engines. Specifically, they need to avoid the ordering problem explained above, where the result of a set of overlapped drawing operations is different if you change their order.

There are several reasons why modern Web engines use general purpose 2D rasterizers (as opposed to rasterizers written specifically for the needs of Web content rendering). One clear reason is that they existed before (in the case of Cairo at least) as a generic 2D graphics library, and was later used for Web rendering. Other reason is that the implementation of the Canvas 2D spec requires a general purpose 2D API, because that's what it is. And there is a clear benefit in reusing your beautifully optimized Canvas 2D implementation to draw the rest of the Web contents. Also, these libraries evolved from a pixmap (image) backed rendering target, into libraries exploiting the hardware-acceleration of GPU cards. Both libraries now feature an OpenGL(ES) backend that is somehow forced to comply with the previously existing API and behaviors.

But that is sub-optimal for Web engines that simply want to draw non-overlapping content into layers, then draw the layers in order. And even though batching and merging do occur in the GL backends today, it is apparently far from optimal as we will see later.

So, if we completely ignore the ordering problem for the case of Web engines drawing already layered nodes onto an OpenGL based render target, we might be able to aggressively group together potentially all the operations that share the same primitive.

This is of course if, as mentioned above, we are able to describe the particularities of each instance of these primitives, hand them down to a smart shader for rendering, then do all that efficiently so that the performance gained in batching is not lost by uploading tons of instance information to the GPU or running heavy shader code.

It is unclear (to me) whether this is at all possible. That's is why this approach is just an idea yet lacking validation for the real world. But it is a research that could potentially boost performance of Web content rendering.

It shares some similarities with (and was partially inspired by) the way Android does font rendering.

A proof of concept

So, I was set up to write a proof of concept trying to validate or discard the idea as quickly as possible. The purpose is to write the minimum code that would allow meaningful comparison between this approach and exiting rasterizers (Skia being my first target for this), for specific use cases that are relevant to generic Web content rendering (not Canvas 2D).

My proof of concept is being developed at: https://github.com/elima/glr

So far, it just provides a few primitives: rectangle and rounded corner, allowing for 3 basic drawing operations: rectangle (filled or stroked), rounded rectangle (only filled) and character glyph (not text, just single characters).

Then each element drawn can be transformed (scaled, rotated and/or translated), laid-out on the canvas (top, left, width and height), and has a color or texture.

Anti-aliasing is achieved by multisampling with 8 samples per pixel. Character glyphs are not anti-aliased, that was too complex to put in a proof of concept and it is a problem already solved by others anyway. I used the simplest possible path to put a pre-cached glyph on the screen, and for that wrote a super naive texture cache, and used FreeType2 for rasterizing the glyphs.
The idea of including chars was to explore if text glyphs, which accounts for most of typical Web page's content, could be batched together with all others drawings that use a rectangle primitive.

Note should be taken that this proof-of-concept is not intended to become a new library. It is just a vehicle to validate an idea by providing the minimum implementation needed to test its limits. Eventually, all this would have to be implemented in existing libraries. I just happen to be very fluent at glib and C :), as to prototype fast.

Comparisons

Before we jump into FPS excitements, lets clarify that any comparison here should be taken with a grain of salt. 2D rasterizers are complex libraries that do a lot of non-trivial things like anti-aliasing, sub-pixel alignment, color space conversion, adaptation to the specifics of the underlying hardware/driver/GL-version combos, to name just a few.

Thus, any comparison should be put in the context of what code paths are being selected, what rendering operations are being grouped, and when and why they aren't; how many GL operations are submitted to the pipeline to render the same scene, etc.

I have included 3 initial examples that try to illustrate how batching and merging of "compatible" draws (sharing the same underlying primitive) improves performance when ordering is ignored, while at the same time each element can have its own color, layout and transformation. For each example, I have written a Skia counterpart that tries to render exactly the same, to the extent possible, for the sake of comparing.

The data below corresponds to runs in my laptop, which is a Thinkpad T440p running Debian GNU/Linux, has an integrated Intel(tm) GPU (4th gen), and the OpenGL driver is provided by Mesa 10.2.1.

I used apitrace to look at what GL commands are actually sent to the driver.

Lets start with the RectsAndText example. It basically draws a lot of alternating filled rectangles and character glyphs, each with its own color, transformation and layout. In the screencast below, both examples (Skia and glr) are running at the same time. This of course does not reflect real performance since both compete for GPU resources, but I decided to record it this way because the improvement is much better noticed. The frames-per-second decrease proportionally for both examples when run at the same time, so it remains relevant for comparison.

The window in the left corresponds to the Skia example, and the right to glr. The same goes for all screencasts below.

This video file is also available for download.

The difference is considerable. Skia performs at an average of 6-7 FPS while the new approach gives around 40 FPS. That’s a 5x-6x improvement, not bad. Also, notice that CPU usage is considerably higher in the case of Skia.

The interesting thing here is that in the case of glr, all elements are batched together (both rectangles and chars), so only one drawing operation is actually submitted to the pipeline, as you can see in the available apitrace dump. A trace for the corresponding Skia example is also available.

apitrace output for RectAndText glr example


apitrace output for RectAndText Skia example

The next example is Rects, which is similar but renders only rectangles, alternating between filled and stroked. The interesting bit is that in the case of glr, each style of rectangles is drawn onto one different layer, each layer operating on its own separate thread; demonstrating that parallelism is now possible during batching.

This video file is available for download.

In this example, the performance difference is even higher. glr is around 8x faster. Again, apitrace traces for the glr example and the Skia version are available. This time glr submits a total of 2 instanced drawing operations, one for filled rects and one for stroked.

apitrace output for Rects glr example


apitrace output for Rects Skia example

The last example draws several layers of non-overlapping rounded rectangles. As with previous examples, every element is given a unique layout, color and transformation. This example tries to illustrate that because batching operates only at layer level, the more layers you have the less you benefit from this technique. In this particular example, the gap is reduced considerably. In fact it looks like Skia is faster by a few FPS, but it is actually not true. When both examples are run together, Skia is faster, but if run separately, glr example is faster (though not much). I’m still figuring this out.

This video file is available for download.

And the traces for the glr example and the Skia example.

apitrace output for Layers glr example


apitrace output for Layers Skia example

If you are curious about the implementation, take a look at where most of the magic happens: GlrContext, GlrCanvas and GlrBatch objects, and the vertex shader. The rest of the code is mostly API and glue to provide a coherent way to use this approach. Specifically, an abstract concept of "layer" is introduced. The workflow goes this way:

• For initialization, a context, a rendering target and a canvas object are created. This is similar to how other 2D libraries work.
• In the rendering loop and for each frame, the canvas is first notified that a new frame will be rendered.
• Then any number of layer objects are created and attached to the canvas. The drawing API works against a layer (instead of a canvas), and will group and batch all the drawing operations in internal commands. When drawing to a layer finishes, the layer is notified that it is ready.
• Finally, the canvas is requested to finish the frame, right before swapping buffers. This call will wait for all the attached layers to finish (blocking if needed). Once all complete, the canvas will take the batched commands from each layer, in the order they were attached, and submit them to the pipeline for rendering.

One thing to remark is that layers are self-contained stateful objects, and can survive frames without needing to redraw.

Other benefits

One by-product derived from the fact that layers cache drawing operations in internal commands (which in turn use locally allocated buffers), is that layers now become data-parallel. This is a term rarely used in the context of OpenGL because as you probably know, the way its API is designed makes it a giant state machine, making any parallelization unpractical.

With this approach, layers can be drawn in separated threads (or fully moved to OpenCL), which can bring extra performance if there are several complex layers that need drawing at the same time.

Another potential extra benefit comes from the fact that the canvas renders to a target that is actually a framebuffer backed up by a multisample texture. This means we can use any previously rendered frame as a texture, the same way it currently works in both Chromium and Webkit, where layers are texture-mapped then composited into the final scene.

So, we have the flexibility that, if a particular layer is too complex or slow to draw, we can attach it alone to a canvas, render it, and use the texture as with the current model. But, if we are short on texture memory, it is possible to keep commands batched in layers and render them on every frame. This is kind of similar to what Chromium does, recording draw operations into an SkPicture and then re-playing back when needed.

Future work

This is an approach that needs validation for a number of real world use cases before it can be even considered for testing on a Web engine. It is key to explore how complex information (for example, multi-step gradient backgrounds, or complex border styling with rounded rectangles) can be passed to the shaders and rendered correctly and efficiently. Also, there are shadows and blurring effects, all parametrized to cover the most creative use cases, that also need verification against this model.

Basically, we need to understand the limits of the approach by trying to implement modern W3C specs, selecting the most complex features first.

Other important priorities are:

• Understand how much workload can be imposed on shaders side before the gained performance starts to degrade.
• Test on OpenGL-ES and constrained GPU on embedded ARM, to detect the minimum requirements.
• Figure out how to implement a mid-frame flushing mechanism when texture cache exhausts or command buffers get too large. This is not trivial, since to flush a layer (that is possibly running in a separate thread) it has to be blocked, then the canvas has to wait for all layers below it to finish and then execute their commands, then signal the blocked layer to continue.
• Try how scrolling would behave if previously batched layers are drawn for every frame, instead of using current scrolling techniques that rely on rolling big textures, or moving several tiles up and down. These techniques impose either great pressure on texture memory, or a lot of complexity on tile management (or both), specially in the context of new super-high resolution screens.

Conclusions and final words

I have tried to detail an idea that although not new, I believe has not been explored in full in the context of Web engines. It relies on two essential hypothesis:

• That it is possible to batch not only geometry, but the complex attributes of arbitrarily styled HTML elements, and render that geometry as instances using shader code.
• It is safe to ignore ordering of draw operations during rasterization phase, and leverage on Web engine’s layer tree to solve overlapping.

Modern GPUs and OpenGL APIs have great potential for optimizing 2D rasterization, but as it happens most of the times, there is no one solution to fit all. Instead, each particular application and use case requires a different set of strategies and trade-offs for optimum performance.

This approach, even if valid for a sufficient number of use cases is unlikely to go faster than existing approaches for all tests cases. Even less replace these approaches. This is pretty clear in the case of canvas 2D for example, which will continue to require a general purpose rasterizer. But if there is a sufficient number of use cases that would benefit from this approach to some degree, then maintaining one code path that enables it will already be a win.

Finally, I want to thank Samsung SRA for partially sponsoring the time I dedicated to pursue this idea, and also Igalia and igalians which are always there to back me up and help me move forward.

Now, is there anyone interested in helping me explore this idea further?

We continue introducing features that will be part of the 4.3.0 release of LibreOffice, which is coming soon. After having worked in the preservation of color in shapes, we worked on the different effects that can be applied to shapes and bitmaps.

There are three types of effects that are managed separately in the DrawingML specification.

General shape effects

Examples of these effects are inner or outer shadows, reflections, glow… They can be applied both to vectorial shapes or bitmaps, and several of them can be applied at the same time.

These effects are indicated with the a:effectLst tag inside the shape properties tag spPr. I won’t explain their specification in detail because you can find a very good description in this website, but you can get an idea by taking a look at the following example where three effects are applied: glow, inner shadow and reflection:

<a:effectLst>
  <a:glow rad="63500">
    <a:schemeClr val="accent2">
      <a:satMod val="175000" />
      <a:alpha val="40000" />
    </a:schemeClr>
  </a:glow>
  <a:innerShdw blurRad="63500" dist="50800"
  dir="2700000">
    <a:prstClr val="black">
      <a:alpha val="50000" />
    </a:prstClr>
  </a:innerShdw>
  <a:reflection blurRad="6350" stA="52000"
  endA="300" endPos="35000" dir="5400000"
  sy="-100000" algn="bl"
  rotWithShape="0" />
</a:effectLst>

Notice that some effects only have some attributes while others contain color specifications as child elements like the ones explained in the previous post.

3D effects

Shapes and bitmaps can be transformed into 3D objects and get lighting and camera modifications applied to them.

These effects are basically controlled by two children tags of spPr. One of them is a:scene3d and controls the camera and lighting, and the other one is a:sp3d which controls the transformation of the shape in a 3D object adding extrusion, bevels and a material effect to the surface. In the same website I linked before, you can read a description of scene3d and sp3d tags and their children. Find an example of their combined use below:

<a:scene3d>
  <a:camera prst="perspectiveRelaxedModerately"
  zoom="150000">
    <a:rot lat="19490639" lon="0"
    rev="12900001" />
  </a:camera>
  <a:lightRig rig="threePt" dir="t">
    <a:rot lat="0" lon="0" rev="4800000" />
  </a:lightRig>
</a:scene3d>
<a:sp3d z="488950" extrusionH="63500"
prstMaterial="metal">
  <a:bevelT w="165100" prst="coolSlant" />
  <a:extrusionClr>
    <a:schemeClr val="tx2" />
  </a:extrusionClr>
</a:sp3d>

Artistic effects

Effects from the last category act like the filters found in image manipulation programs (blur, grain or background removal among others) and that’s why they only can be applied to bitmaps. This is actually not a part of DrawingML spec but an extension over it.

There is an important difference with other filters; these ones come pre-calculated in the document. The bitmap linked by the DrawingML shape already comes with the effect, and the effect specification links a second bitmap that contains the original picture so the effect can be undone. This second bitmap is saved in the relatively new loss-less Windows Media Photo format.

Check the following example of a blip-filled shape; the actual filling comes from the file linked as rId6, while the effect definition is linked to rId7 which is a copy of the original image before the filter was applied:

<a:blip r:embed="rId6" cstate="print">
  <a:extLst>
    <a:ext uri="{BEBA8EAE-BF5A-486C-A8C5-ECC9F3942E4B}">
      <a14:imgProps xmlns:a14="http://schemas.microsoft.com/office/drawing/2010/main">
        <a14:imgLayer r:embed="rId7">
          <a14:imgEffect>
            <a14:artisticLightScreen trans="10000" gridSize="6" />
          </a14:imgEffect>
        </a14:imgLayer>
      </a14:imgProps>
    </a:ext>
  </a:extLst>
</a:blip>

These are the relations between the ids and the files contained in the document, as specified at document.xml.rels:

<Relationship Id="rId6"
  Type="http://schemas.openxmlformats.org/officeDocument/2006/relationships/image"
  Target="media/image2.png" />
<Relationship Id="rId7"
  Type="http://schemas.microsoft.com/office/2007/relationships/hdphoto"
  Target="media/hdphoto1.wdp" />

The funny thing of this approach is that LO was able to render these effects with no effort, although the program was not aware of the effect parameters or the original bitmap and these were being lost on save.

Preservation

We use again the grab bag technique to save all the tags and attributes related with the effects as a hidden property that will be used later in the export phase to re-build the effect definitions. In the case of artistic effects, we additionally need to make sure that the original bitmap is preserved; LibreOffice doesn’t support the Windows Media Photo format yet, but we can keep the raw stream of data and output it to a properly named file in the exported document. A small cache table is maintained by the exporter code to prevent that the same original file is saved more than once when two or more pictures apply effects to the same image.

We have finished with the improvements related to shapes and pictures, but there are a few interoperability features not yet mentioned which will be covered in a future post. Like the current and previous ones, they were developed by Igalia and sponsored by CloudOn.

Happy hacking!

In his last post my mate Sergio explained the different syntax to position elements inside a grid. Now is time to talk about the automatic placement feature, how it works and show some examples of its potential.

Auto-placement

The concept is defined in the specification:

Grid items that aren’t explicitly placed are automatically placed into an unoccupied space in the grid container.

So, let’s start with a simple example to show how it works. Imagine that you have the following 2×2 grid:

<div style="display: grid;">
    <div style="grid-row: 1; grid-column: 1 / span 2;">first</div>
    <div style="grid-row: 2; grid-column: 2;">second</div>
    <div>auto</div>
</div>

first item is using the 2 columns on the 1^st row, second item is placed on the 2^nd row 2^nd column. That leaves an empty space on the 2^nd row 1^st column, where the auto item is placed.

Simple example where the auto-positioned item is placed into the 2^nd row 1^st column

Of course you can do more complex things, like setting a specific position in one dimension (row or column) and left the other as auto, request more than one cell for the item, etc. as you can see in the following example of a 3×3 grid:

<div style="display: grid;">
    <div style="grid-column: span 2;">item 1</div>
    <div style="grid-column: 3;">item 2</div>
    <div style="grid-row: span 2;">item 3</div>
    <div style="grid-row: span 2; grid-column: span 2;">item 4</div>
    <div style="grid-row: 2;">item 5</div>
</div>

Example animation of how different auto-positioned items are placed in the grid

Let’s analyze why elements are positioned as you can see in the animation above:

• item 1: This is the first element placed, this item need 2 columns and it’s placed in the 1^st row 1^st-2^nd columns, as they’re empty.
• item 2: This item is only specifying the column, so it’s placed in the 1^st row 3^rd column, as it’s the first empty cell in the 3^rd column.
• item 3: In this case the items needs 2 rows and it’s placed in the 2^nd-3^rd rows and 1^st column.
• item 4: Here the example needs 2 rows and 2 columns, we’re lucky as there’re some empty cells so it’s placed in the 2^nd-3^rd rows and 2^nd-3^rd columns.
• item 5: This item needs to be placed in the 2^nd row, as there’s no empty cells in that row, it’ll be placed in a new column outside current grid, 2^nd row 4^th column. Imagine that this item is completely auto (we remove the “grid-row: 2;” style), then it’d be placed in a new row outside current grid, 4^th row 1^st column.
  Why in a new row and not in a new column? That’s managed by grid-auto-flow property that’ll be explained below.

grid-auto-flow property

Again from the spec:

The grid-auto-flow property controls the direction in which the search for unoccupied space takes place, and whether rows or columns are added as needed to accommodate the content.

The simpler (and easier to understand) values for this property are row (by default) and column. These values define the direction that the auto-placement algorithm should follow to place the auto-positioned items. For example if the value is row the algorithm will try to fill each row before jumping to the next one, adding new rows as necessary.

Also, this property allows to combine these values (row and column) with other keywords (that cannot not be used together):

• dense: By default auto-placement algorithm is sparse, that means that if placing an auto item has leave some empty cells in the grid (because of the item doesn’t fit) they won’t be used anymore. If dense is used, these holes will be used if smaller items come up later.
• stack: In this case the auto-placement algorithm looks for the first empty cell (like if it was adding a 1×1 item). Then it places all the auto-positioned items in that cell (independently of their size), stacked atop one another.

As usual it’s easier to understand with a HTML example:

<div style="display: grid;">
    <div style="grid-row: 1; grid-column: 2;">item 1</div>
    <div style="grid-column: span 2;">item 2</div>
    <div>item 3</div>
</div>

Example grid with different values for grid-auto-flow property

In the image you can check how the auto-positioned elements (item 2 and item 3) are placed depending on the different values of grid-auto-flow:

• sparse (grid-auto-flow: row;):
  • item 2: It looks for an empty area of 2 columns, so it skips cell in 1^st row 1^st column levaing it as an empty space. The item is placed in the 2^nd row 1^st-2^nd columns.
  • item 3: It looks for the next empty space, and it adds the item in the 3^rd row 1^st column. This will leave a hole in the 1^st row 1^st column.
• dense (grid-auto-flow: row dense;):
  • item 2: This is placed exactly the same than in the sparse case. The item is placed in the 2^nd row 1^st-2^nd columns.
  • item 3: It looks for an empty cell and it adds the item in the 1^st row and 1^st column as it’s still empty (item 2 didn’t use it before, because of it doesn’t fit there).
• stack (grid-auto-flow: stack row;):
  • item 2: It looks for the first empty cell, and it finds the 1^st row and 1^st column. Then all the auto-positioned items will be placed there. So item 2 is placed in 1^st row 1^st-2^nd column (even if it takes more than one cell, it’ll keep using as reference the first empty cell).
  • item 3: Again this is added in the 1^st row 1^st column over item 2.

Possibilities

As you can imagine this brings a lot of power to the CSS Grid Layout spec. You won’t need to always place your items explicitly in a grid, you can simply pass the responsibility to find the best position to the grid itself.

The typical example is to think in a regular form:

<form>
    <label>Name</label><input type="text" />
    <label>Mail</label><input type="text" />
    <label>Comments</label><textarea></textarea>
    <label>Accept policy</label><input type="checkbox" />
    <div id="buttons">
        <button>Accept</button>
        <button>Cancel</button>
    </div>
</form>

You could just apply the following CSS and you’d have a formatted form “for free”:

    form     { display: grid;           }
    label    { grid-column: 1;          }
    input    { grid-column: 2;          }
    #buttons { grid-column: 1 / span 2; }

The result will be something like the following picture:

Formatting regular form using grid auto-placement feature

Wrap-up

We’ve created a small demo (as part of our grid examples repository) that will allow you to play with the auto-placement feature and understand it better:

http://igalia.github.io/css-grid-layout/autoplacement.html

As you could read during the post, the auto-placement algorithm has been mentioned several times. This algorithm and the implementations details will be explained in a subsequent post.

If you’re wondering about current support in Blink and WebKit, the basis of auto-placement feature is already working on both engines. In Blink span support in auto-positioned items has recently landed and in WebKit the patch is ready to be integrated.
However, the new grid-auto-flow syntax (sparse, dense and stack) is still on the way (follow the Blink and WebKit bugs if you’re interested).
As you can see, we’ve been working hard on implementing the missing bits of this feature and we hope it would be fully supported soon in both Blink and WebKit.

Finally, we’d like to thank again Bloomberg for sponsoring our work around CSS Grid Layout.

Igalia and Bloomberg working together to build a better web

Several weeks ago, I was invited by INTECO to attend the seminar 'Secure Coding in C and C++'. The event took place at INTECO's Cyber-Security headquarters in León, Spain.

It was a great coincidence because Robert, the person teaching this seminar, and I are part of the teams in SEI and Igalia collaborating on browser security.

As you may know, the mission of the National Institute for Communication Technologies (INTECO), located in León (Spain)

This seminar was included in the Program on Excelence in Cibersecurity (PECS) in order to find and promote talent in Cibersecurity. The seminar focused on producing secure programs and designs using C and C++ languages. The seminar ran for four days providing a detailed explanation of common programming errors in C and C++.

Robert organized the material covering the following topics: string management, dynamic memory management, integral security, formatted output and file I/O. The lessons interleaved theory, practical exercises, mapping issues to CERT coding standards and debate around the flaws and vulnerabilities handled.

Robert is a person with long experience in Information Security. He is currently the Secure Coding Technical Manager in the CERT Program of Carnegie Mellon's Software Engineering Institute. I enjoyed a great time talking to him about the state of the art in Information Security, avoidable software defects and vulnerabilities.

At the end, I would like to thank to INTECO for hosting and organizing this event. They did a great job with the materials, infrastructure and organization. Congrats guys!

Those last weeks I was really busy here in Igalia. We were hacking in Chromium/Blink broadly, attending to BlinkOn 2, held our Assembly, enjoyed one of our summits and so on. On the top of all these things we started to collaborate with the Carnegie Mellon Software Engineering Institute (SEI) around browsers security too. Great news!

On the other hand, Robert commented on the SEI's blog about the importance of having in mind secure coding practices to prevent vulnerabilities while coding, and how the CERT Secure Coding Initiative at the SEI is supporting this approach with completed standards for C and Java. By the way, coding standards for C++, Perl and other languages are under development too.

CERT coding standards are valuable resources for the programmer taking care of Information Security. As Robert highlights, secure coding standards itemize coding errors that are the root causes of current software vulnerabilities, prioritizing them by severity, likelihood of exploitation, and remediation costs. Each rule in the standard includes examples of insecure code, as well as secure alternative implementations.

If you are interested about our collaboration with SEI and the research project to evaluate the costs of producing a CERT-conforming implementation of the Chromium browser you should not skip his post. It introduces the rest of lines and colleagues collaborating in this project too.

Lately I have been working on the WebKitGTK+ build bots that Igalia maintains.

Some of the improvements we achieved include:

• A faster machine for the x64 Debug bot.
• Splitting both the x64 Debug and Release bots in build-only and test-only.
• A performance bot!.

The most interesting one is the performance bot. You can check the numbers at http://perf.webkit.org. That numbers are not valid to compare the performance between the different WebKit ports (EFL/GTK+/Mac), because each one of these performance bots run on different hardware. The idea is to track the performance between revisions, and identify the commits that have a performance impact.

The performance bot we deployed is not very powerful (i5 661), but does the job. I took extra care to ensure that external factors don’t affect the performance results of the tests on the machine. This is a summary of that measures:

• I disabled cron and every other daemon.
• Gave the performance tests processes very high priority (nice -19, ionice -c1).
• Disabled Hyper-Threading: While the tests are run in sequential order, the javascript engine (JSC) or other parts of WebKit can be parallelized. HT is great if you want to squeeze the maximum performance of your machine, but it can become a problem when you care about consistent results related to performance (or you care about real time).
• By default the performance tests are run inside Xvfb, and this is not great if you are interested in the most realistic scenario possible. So I patched WebKit to allow running the tests on the native X display.

These are some details of the bot we deployed:

• The machine is an Intel Core i5 661 with 8GB of RAM.
• As explained, we run the tests on a real X server:
  

  $ glxinfo | grep "render.*:"
  direct rendering: Yes
  OpenGL renderer string: Mesa DRI Intel(R) Ironlake Desktop 
  

• The OS were the tests are run is Debian Jessie.
• The DE is GNOME 3: I disabled every power-save feature of GNOME (including the screensaver). Since the window of the browser that runs the tests is positioned at the top-left of the screen, I installed the extension Activities Configurator that allows to hide the top bar of activities, and to disable the hot upper-left corner.

If you want to run the performance tests on your machine, this is as easy as:

Tools/Scripts/run-perf-tests --platform=gtk

If you want to run them on your X display, simply export the variable USE_NATIVE_XDISPLAY before:

export USE_NATIVE_XDISPLAY=1

If you don’t want to run the full suite (takes no less than 2 hours), you can specify the test names (look on the PerformanceTests directory for the names).

For example to run the test DoYouEvenBench Speedometer (DoYouEvenBench was renamed to Speedometer):

Tools/Scripts/run-perf-tests --platform=gtk Speedometer/Full.html

Check the WebKit wiki for further information and keep and eye on http://perf.webkit.org: If you notice any performance regression, please let us know that by opening a bug.

This is the latest chapter of the series about interoperability features that will be part of LibreOffice 4.3, brought to you by Igalia and sponsored by CloudOn, like the previous ones.

Last week we explained our work with theme color preservation for fonts and paragraphs, and now we move to a very close topic: the preservation of theme colors and styles on shapes.

XML definitions

DrawingML includes new conventions to denominate colors, wider than the initial OOXML spec as used for paragraphs and fonts. Such color definitions can be used in the context of shape properties or style definitions to define fillings, line colors, gradients…

In first place, English color names (red, black, etc.) can be used besides RGB values and theme color names; and for any color, a set of transformations can be defined, similarly to the tint and shade properties in fonts and paragraph colors, but with more options. An important transformation is alpha to define the transparency of that color.

Find below one example of each color tag (for color names, RGB values or theme colors), with or without transformations:

<!-- color by name, with an alpha transform -->
<a:prstClr val="black">
  <a:alpha val="50000"/>
</a:prstClr>

<!-- color by RGB value, no transforms -->
<a:srgbClr val="FF0000"/>

<!-- theme color, with two transforms -->
<a:schemeClr val="accent1">
  <a:satMod val="175000"/>
  <a:alpha val="40000"/>
</a:schemeClr>

Shapes in DrawingML receive their attributes from two different sources: the style definitions and the own shape properties. The former act as a fallback of the latter; attributes from style definitions will only be applied if they are not overwritten by shape properties. For example, in case a shape contains both a filling definition among its properties and a filling style, the filling definition prevails.

The theme file defines a list of filling and line styles so they can be used by shapes. This is an example of an area filling style which adds a couple of color transformations:

<a:fillStyleLst>
  <a:solidFill>
    <a:schemeClr val="phClr">
      <a:tint val="15000"/>
      <a:satMod val="350000"/>
    </a:schemeClr>
  </a:solidFill>
  <!-- more fill style definitions -->
</a:fillStyleLst>

Notice it’s not using any particular color, phClr is an entry parameter of sorts for a color that will be indicated in the shape style definition. The style definition can add some transformations or even define a gradient fill based on that color.

A line style definition can be a bit more complex, because it has more format options: width, dash pattern, single or double… These options are available for shape properties too.

<a:lnStyleLst>
  <a:ln w="9525" cap="flat" cmpd="sng" algn="ctr">
    <a:solidFill>
      <a:schemeClr val="phClr"/>
    </a:solidFill>
    <a:prstDash val="solid"/>
  </a:ln>
  <!-- more line style definitions -->
</a:lnStyleLst>

This is how a shape from the document would be assigned to the first filling style and the second line style, specifying “accent1″ and “accent2″ colors as parameters:

<wps:style>
  <a:lnRef idx="2"> <!-- use the second element from fillStyleLst -->
    <a:schemeClr val="accent1"/>
  </a:lnRef>
  <a:fillRef idx="1"> <!-- use the first element from lnStyleLst -->
    <a:schemeClr val="accent2"/>
  </a:fillRef>
  ...
</wps:style>

Finally, this is how a shape can overwrite the style-defined attributes for its filling and line. The following XML chunk defines a solid filling for the area using “accent3″ color and a solid color for the shape line using “accent4″:

<wps:spPr> <!-- shape properties tag -->
  ...
  <a:solidFill> <!-- area filling -->
    <a:schemeClr val="accent3"/>
  </a:solidFill>
  <a:ln> <!-- line properties -->
    <a:solidFill>
      <a:schemeClr val="accent4"/>
    </a:solidFill>
  </a:ln>
  ...
</wps:spPr>

The importer code

The duty of the importer code is, in general, applying the style definitions, merging them with the shape properties and transforming the properties in a way that matches LibreOffice internal data model.

Applying the style definitions means getting the corresponding styles from the theme file and applying them to the specific colors indicated in the shape wps:style block. This results in a set of properties that has to be merged with the shape-specific ones, and this is done so the latter have priority, as explained before.

All properties must be transformed to match LibreOffice data model, and specially colors because of their complexity. Theme colors are translated to their RGB value checking the theme definition to match the color name with its value. As for color transformations, the importer applies all the transformations to the original color to get a final RGB value. The only exception is the alpha transformation, which is converted in a transparency value to be stored in the fill properties object (alpha and transparency are complementary; after homogenizing their units, alpha = 1 – transparency).

LibreOffice doesn’t have equivalent concepts to theme colors and color transformations other than alpha, and doesn’t have shape styles either. To prevent information loss, we will use a hidden property in the Shape object called the interop grab bag to fill it with any properties we need to rebuild this information on export:

• To preserve a theme color, we need to store its name and the complete list of transformations. We will also need the RGB value of the color with all the transformations applied, so we can compare it with the final color to know if the user has changed it. We do it both for line and filling colors.
• For preset and RGB colors, the loss of the transformations is not important so we don’t consider this special case.
• To preserve the style definitions, we store the idx attribute of each and their color parameter, with all transformations if necessary. We also store the RGB value of the color including transformations because we will need it too.

The exporter code

The normal behavior of the exporter is translating every LibreOffice shape property to DrawingML to write it to the document. In the case of colors, that means using a a:srgbClr tag to save it in RGB format and optionally adding an alpha transformation calculated from the shape transparency value.

The preservation of shape styles, area and line colors increases a bit the complexity. The shape grab bag must be checked for interoperability information saved in the import phase which should be saved back to the document under certain circumstances.

If there are any style definitions in the shape grab bag, they are always written back to the document; we need no additional verification because style definitions act as a fallback as we already explained. On the other hand, the case of shape properties has some complexity because we must perform several checks before deciding which area and line information we want to save:

• Does the final color match the original one? If it doesn’t, it means the user has changed it during edition, so the new one must be saved and the information in the grab bag discarded.
• If the original and final colors match, is there any theme color information in the grab bag? If there is, we must write it to the shape properties.
• If the original and final colors match and there is no theme color information in the grab bag, we must compare the color with the style color; if they match, it means the shape was using the style color so we must not write a shape property that would overwrite it.
• Otherwise, the shape was using a custom color different from the style color and must be written to the shape properties.

Bonus track: multiple color gradients

While I was working on this topic, I noticed that DrawingML allows to specify gradient fillings with any number of steps, unlike in previous MS Office documents (and LibreOffice ones) where you could only use two. This is an example of gradient with three steps:

<wps:spPr>
  ...
  <a:gradFill>
    <a:gsLst>
      <a:gs pos="0">
        <a:srgbClr val="ffff00" />
      </a:gs>
      <a:gs pos="50000">
        <a:srgbClr val="ffff33">
          <a:alpha val="20000" />
        </a:srgbClr>
      </a:gs>
      <a:gs pos="100000">
        <a:srgbClr val="ff0000" />
      </a:gs>
    </a:gsLst>
    <a:lin ang="5400000" />
  </a:gradFill>
  ...
</wps:spPr>

Every a:gs tag indicates a point in the gradient extension with the pos attribute, measured as a percentage, and the color of that point. The system should calculate the gradient between the colors of two consecutive points.

LibreOffice does an approximation of those gradients with more than two steps using only two colors when it imports the document. To prevent information loss, we use again the grab bag to store the complete gradient definition. In an analogous way, we store the original, approximated gradient information to be able to compare it with the gradient information in the moment of the save operation.

We can identify three situations in the exporter:

• The original and final gradients don’t match: it means the user has changed it during edition, so the new one must be saved and the information in the grab bag discarded.
• The original and final gradients match and there is a full gradient definition stored in the grab bag: we must write that gradient definition to the shape properties.
• The original and final gradients match and there isn’t any gradient definition stored in the grab bag: the gradient definition comes from the shape style and in that case we must not write the shape property that would, again, overwrite it.

One more thing or two

As you can see, this feature relies on the hidden interop grab bag property which contains more and more information as we increase its use to preserve unsupported properties. We hope to reduce its weight as those properties become natively supported. I would also like to comment that we have complemented our work with a set of unit tests that will help to detect regressions appearing in the future.

We haven’t yet finished with shapes; next post will be related to the preservation of different kinds of shape effects. It will be ready soon!

Last week, LibreOffice community branched out the version 4.3 in preparation of the next release in July. We had already introduced one of the new interoperability features that will be part of that release in the previous post in this series, and now we will continue explaining how OOXML theme colors will be preserved in LibreOffice.

Theme colors have a prominent place in the color palette in the latest Microsoft Office versions, as you can see:

They consist on a palette of ten colors, each one of them with five more variations. When a user changes the document theme, a new palette of colors will be loaded and any objects in the document that used one of those colors will be updated with the corresponding one from the new palette.

In the XML document, a theme color is identified by a name, and its variations are implemented with two attributes: shade to lighten the color and tint to darken it.

This is an example of theme color applied to the characters in a text run – a chunk of text with the same properties. In the run properties (rPr) we find the following tag (reference):

<w:rPr>
  <w:color w:val="E5B8B7" w:themeColor="accent2" w:themeTint="66"/>
</w:rPr>

This is the theme color named accent2 darkened a 66%. The w:val property indicates the RGB code of the final color, it’s ignored if w:themeColor is present.

The preservation of this tag is simple:

• On import, store w:themeColor, w:themeTint and w:themeShade properties as hidden properties in the text run.
• Store the original color too!
• On export, check if the original color has changed:
  • if it did, write only the new color in the w:val property.
  • if it didn’t, write the stored w:themeColor, w:themeTint and w:themeShade properties.

Now, an example of paragraph shade with theme colors (reference):

<w:pPr>
  <w:shd w:val="thinDiagStripe"
         w:color="215868" w:themeColor="accent5" w:themeShade="80"
         w:fill="DBE5F1" w:themeFill="accent1" w:themeFillTint="33" />
</w:pPr>

Paragraph shades can specify two colors, one for the background and another one for the pattern if present, with the type of pattern specified by the w:val attribute. The set of attributes w:color, w:themeColor, w:themeShade and w:themeTint work like in the case of font colors, specifying the RGB code, theme color name, shade and tint modifiers for the pattern; w:fill, w:themeFill w:themeFillTint and w:themeFillShade do the same for the background color.

The preservation of these properties works analogously; they are also stored as a hidden property of the text on import, and saved back on export in case the user hasn’t changed the fill color while editing the document with LibreOffice.

The w:shd tag can also be used in table or table cell properties with the same meaning. And speaking of tables, we also made sure that the table style property was preserved:

<w:tblPr>
  <w:tblStyle w:val="Tablaconcuadrcula" />
  ...
</w:tblPr>

We will continue next week with more shiny features brought to you by Igalia and CloudOn. Stay tuned!

I am a GNOME developer and I do not share Philip Van Hoof's views. He doesn't represent me.

Last  week I traveled to Zürich to attend to BlinkOn 2 at the office Google has there. This is the first time the event is done in Europe (previous one at US) and it was a very good opportunity to meet all the people working around Blink, Chromium and Skia.

Zürich

As Igalia has a long experience working on browsers technologies, it was represented by four igalians who are working on different areas: Manuel Rego on CSS, Xabier Rodríguez on multimedia and Eduardo Lima together with me on graphics and Skia.

The attendees of this conference were not only googlers, there were people from Opera, Samsung, Intel and much more companies interested on meeting each other and collaborate to improve Blink.

Google Zürich office

Conference

The talks were covering a lot of different areas of a browser: from memory leaks or image filters to security models to name a few of them. Worth to mention Manuel Rego’s talk which was about his latest work on CSS Grid Layout (more details in his blog).

BlinkOn 2

I have attended to the talks more related to graphics on the browser as this is one of the areas I am currently working on: “Rendering for Glass Time“, “Responsive images“, “Chasing Pixels“, “Filter effects“, “Storage modes for 2D canvas” and “60fps on mobile“.

I would like to highlight the talk given by Justin Novosad (Google) where he explained his proposal of different storage modes for 2D canvas: persistent memory, discardable memory and none. Each one with different use cases and consequences on memory usage with focus on mobile devices dealing with very large canvases and there was a discussion about how to provide a useful API for web developers.

I attended to more that the aforementioned talks: the always interesting “Memory leaks in Blink” report, “Web animations“, “Out-of-process iframes“, “Smooth to the touch: challenges in input on mobile“, the Opening talk and “Trace reading clinic” which was very useful to learn more about the tracing tools available in Chromium.

This conference was also the excuse to meet some Skia developers in person and talk about Igalia’s contributions and future work on this 2D graphics library.

I would like to thank Google for its great job organizing BlinkOn at its office in Zürich and also Igalia for sponsoring my trip.

Miscellanea

But as not everything would be work, I also met an old friend who is working at Google to give me an update of his last years  while drinking good coffee.

At the last day, just before returning home, I bought a very appreciated gift for my relatives… good Swiss chocolate! Yummy!

Zurich Google Office

Past week I’d been attending BlinkOn 2 at Google office in Zurich. I think that the best part of this kind of events is to have the chance to meet many well known people that you only see usually in the bug tracker and IRC.

It was nice to see other companies around too like Opera, Samsung, Intel, etc. We were four igalians there talking about the work we’re lately doing on Blink (check Sam’s post for more information).

CSS Grid Layout

As you might expect, this was my primary focus in the conference, as that’s what I’ve been working on during the last months.

Me during the CSS Grid Layout talk (photo by Calvaris)

On Tuesday, I was giving a talk about the work we’re doing on CSS Grid Layout sponsored by Bloomberg. It seems that people are interested in this feature and they liked the presentation, you can check the slides.

Julien Chaffraix (grid’s reviewer) was there and later we discussed about the approach to fix some issues that have been hanging around in the bugzilla for a while.

Selection

On Wednesday, I attended the selection talk given by Yoshifumi Inoue and Hayato Ito. They’re now focused on fixing the issues in Shadow DOM with selection.

Actually, they have some interesting ideas regarding how to avoid the problems with DOM ranges. Basically, they’ll try to use a start and end points for the selection instead of range itself, and then they’ll iterate over the composed tree in the right order.

On top of that, we talked about the issues that are present with other layout models, but we just concluded that selection is very complex and we should try to solve things step by step.

Thanks to the editing team for arranging the session.

Panel discussion in the main room (Tech Talk)

Other

Many other interesting talks were given during the event, like the 60fps talk by Nat Duca, the multi-column sessions by Opera guys, the Oilpan presentation by Mads Ager and many more. If you missed any of them you can check the slides in the BlinkOn 2 web page and see the videos of the talks in the main room whenever they’re available online.

Finally, I’d like to thank Igalia for sponsoring my trip and, of course, Google for the event (specially Max for the great organization and also Philip for arranging the hiking on Thursday which, despite of the foggy weather, was pretty nice).

Zurich view from Felsenegg

You already know I took part in the celebration of the Document Freedom Day speaking in an event in the University of Coruña. The nice people from GPUL had recorded the talks and now they are ready for everyone to enjoy, distribute or reuse under the terms of the Attribution-ShareAlike 4.0 International license.

Here are the talks:

Estándares abertos, a situación dende o punto de legal

• video
• slides

LibreOffice, o proxecto de referencia para a edición de documentos libres

• video
• slides

Sobre a importancia dos estándares e formatos libres

• video

This post is a recap to summarize all the work we’ve been doing in Igalia during the last months related to selection in CSS Regions.

Wrap-up

Back in October 2013 my mate Javi explained the different problems we detected on how selection worked in pages using CSS Regions. As part of this initial work, we were improving test coverage as it was explained in a previous post. Originally we were working both in WebKit and Blink, but as you probably know CSS Regions has been removed from Blink code during this year.

In parallel, we’ve been also working in optimizations over CSS Regions code. First providing some new selection performance tests and later implementing some performance patches in the part related with re-paint phase, which is very important during the selection process.

Making selection spec compliant

Our first solution was put aside as it had some rough edges and it was probably too ambitious as initial step.

Nonetheless in my last post we introduced a new idea to make selection in CSS Regions spec compliant. We called it the “Subtrees approach“. We’ve finally implemented it and the patch is already integrated upstream.

Example of selection in CSS Regions before and after the Subtrees approach patch

You can check the new behaviour using our test suite. As you could see now selection content always matches with the highlighted text in the web page.

Future

Somehow this is just the tip of the iceberg, a small step to make selection in CSS Regions work better. However, selection is still not completely natural from the user point of view in several cases. On top of that, selection in other layout models (flexbox, grid, multi-column, etc.) has very similar issues.

Our plan is to start a discussion inside WebKit and Blink communities about how selection works in the different layout models and what kind of changes can be done there to create a better user experience around selection interaction on the web (as Javi has introduced in his last post).

Finally, we’d like to publicly thank Adobe Web Platform team for giving us the chance to collaborate with them, specially Mihnea for all his support. Also, thanks to all the reviewers for their insightful feedback and precious time.

A few weeks ago, I had the opportunity to represent Igalia in the Webkit Contributors Meeting. It was hosted by Apple at their campus in Cupertino and, unlike what some Igalia’s fellows told me about, it wasn’t the huge event it used to be which gave me the chance to meet personally some of the well known hackers I interact with by IRC and bugzilla; that was definitively nice.

The usual unconference-like environment was something I liked a lot, specially how we made the session scheduling on the fly; it was funny to directly notice the interest of the audience on the talk I proposed. My involvement in Webkit during the last year has been improving and implementing CSS standards like CSS Regions and CSS Grid Layout, being precisely the former what my talk was about.

CSS Grid Layout

When I knew I was going to attend the meeting I saw the opportunity of spread out the work we are doing at Igalia with the support of Bloomberg on the implementation of the CSS Grid Layout standard, which we both consider very important for the Web because of the use cases it solves.

I had the feeling that the situation of this new feature could be considerably improved among the Webkit community and perhaps get the attention of more hackers and reviewers willing to collaborate and accelerate the implementation and, eventually, shipping it in future releases of the Safari browser. Considering that IE/Trident is already shipping it, Blink/Chromium has plans to do it as soon as possible and Mozilla is also willing to do it in the long term (the E.D is being written by members of those web engines), we at Igalia considered sensible to suggest the Webkit community to go in that direction.

Regarding the talk itself, I think it helped to increase the visibility of the CSS Grid Layout implementation in Webkit, so let’s see what this means for our work in the following months. If you are interested on the slides, you can check them out here. We talked about ways to enable the feature by default in the nightly builds, so it can be tested by Webkit hackers more easily, and also about the work to be done in the next months. I was happy to see this point being moved to the mailing list and finally become a patch landing. We have a web site with many examples and guidelines to activate the feature on different browsers:

http://igalia.github.io/css-grid-layout/

Selection

One of the goals to attend this meeting was to unblock the issue of Selection with CSS Regions, something that Igalia was involved on during some months last year in collaboration with Adobe. I discussed with David Hyatt the Sub-Trees Approach as a way of making Selection specification compliant when using CSS Regions, even though the issues from the end user perspective are still there. I consider this an important milestone because it puts the CSS Regions standard in a similar position than the rest of layout models affected by these Selection problems (CSS Grid Layout, MathML, Shadom DOM, Absolute Positioning and Multi-Column, among others).

Now a different challenge has to be addressed: how to provide a usable Selection for these layout models ? Even though it was not in the approved schedule, we managed to setup a meeting among different people interested on Selection. I have to say that it was a nice and interesting discussion, which these are the main conclusions I’d extract from:

• We need to go beyond the editing/selection specification, since the DOM nature of its definition is very limited for the new layout models.
• Emulating the iOS selection behavior was something most of the participants liked, so perhaps we should consider it in the future.
• There are several approaches we can follow, multi-range, sub-trees, region-as-containers, … all of them have benefits and cons and what is worst, some of them address only issues of specific layout models; the idea that there is not single solution for all the layout models was always present.
• Perhaps we should define different implementations of Selection for these specific features; something incremental since it’s very important to keep such an old and important feature like Selection as stable as it’s been all these years.

These ideas are not actually breaking news, since they were already mentioned in previous meetings, but I perceived a better mood now to implement more ambitious approaches to improve Selection. Some of the standards affected by these issues, like CSS Regions and CSS Grid Layout, are really important features so these technical challenges must be faced as soon as possible.

One of the action points agreed was to analyze case by case the issues each of these layout models have, defining very specific examples and test cases; we could later start a discussion on the best way to solve them. At Igalia we have been thinking about this for some time already, so we have created a web site and test cases repository to analyze the different issues present in the layout models we considered more interesting so far. We’d like to invite any hacker interested on this topic to contribute to the wiki and the test cases repository, even adding different layout models to be included in the analysis.

http://igalia.github.io/web-selection-examples/

Misc topics

I attended also some other talks and discussions interesting for the future of the Webkit project, which Igalia is quite committed to, specially as maintainers of the WebkitGtk+ port.

We participated in the discussion of “How to make WebKit more awesome” representing the WebkitGtk+ port community; it was a nice discussion with plenty of good ideas.

I attended the  CSS Regions and CSS Shapes talks, which were quite rich in terms of progress and roadmap announcements  and with the usual awesome demos. The CSS Regions talk presented by Andrei Abucur got quite much attention and it generated an interesting discussion about it’s future.

I attended the session about Subpixel Layout as well, something I was involved into some months ago. I found out some CSS Regions bugs related to the Subpixel Layout feature, which root cause was not enabling the SATURATED_LAYOUT_ARITHMETIC flag. Actually, we’ve recently enabled it in the WebkitGtk+ port with quite good results, although there are still a few regressions  I’m still working on.

Welcome to a new “GNOME 3.12 is out blog post”, somewhat late because I wanted to focus on 3.12.1 instead of the usual 3.12.0, and because I was away for several days due to Easter holidays.

Flowers and a mill at Keukenhof

As I said just after 3.10, Antía worked hard on adding keyboard navigation support to Evince, and Adrián provided an implementation of the tagged PDF specification for Poppler. The plan for the 3.12 cycle was to build upon their work in order to improve Evince’s accessibility support.

Thanks to the tagged-PDF implementation for Poppler, we were able to start experimenting with tagged PDF documents in Evince, and playing with all the cool things that tagged PDFs bring to the table. Finally we have available information about if we are in a paragraph, where a list starts, different levels of headings, and pretty much anything else one can put in an element tag. But while Adrián and Carlos García kept working on getting their patches pushed upstream (more than 15 patches were pushed during this cycle), Joanmarie Diggs and I realized that “only” a bare/plain implementation of this specification would be a hard animal to tame in order to be used by assistive technologies: Additional parsing and structuring will be needed in Poppler to properly implement ATK support in Evince.

Additionally, having working keyboard support in Evince made it finally possible to test real-world document accessibility with Orca (as opposed to just Accerciser). But in doing so, we found that the existing ATK support was incomplete or wrong in several places. So even the more basic PDF documents, those that should be also accessible without tagged PDF, were not properly accessible. Taking all this into account, we decided to focus on fixing the bugs in Evince’s core accessibility support as doing so would make all PDFs more accessible, but at the same time to continue working on the tagged-PDF support in order to start developing a concrete list of the improvements we will need added to Poppler.

So the main tasks on Evince during this cycle were:

• Reimplement AtkText
• Expose all document pages to the accessibility tools, not only the current one.
• Implementation of AtkDocument
• Some fixes to caret-navigation and hyperlink management

As a result of these changes:

• Several accessibility-triggered crashes have been eliminated
• Orca’s SayAll feature now works with Evince
• Prosody when reading documents with Orca has been improved
• The caret can be positioned and text selected via AT-SPI2

Some of this work was not quite in time for the 3.12.0 release, but has been included in 3.12.1. In addition, we are continuing to work on accessibility-related bug fixes which we anticipate will be included in 3.12.2.

As for what’s next: We encourage Orca users to give Evince a try and help us identify the bugs that remain in Evince’s core accessibility support. Anything that they find will be added to our high-priority TODO list. In the meantime, we will continue to work on enhancing Poppler’s tagged-PDF support and then exposing that structural information through Evince to assistive technologies.

Finally, I would like to thank the GNOME Foundation and the Friends of GNOME supporters for their contributions towards making a more accessible GNOME, as this work would not be possible without them.

This is the first post in a series that explains the work that we’ve been doing lately in LibreOffice at Igalia, which hopefully will be part of the future 4.3 release.

Document themes is one of the features of the Microsoft Office suite that doesn’t have support in LibreOffice yet. While it happens, which is a hard work that includes extending the ODF standard, we are focusing on the preservation of the theme information when a document coming from Office is edited in LibreOffice and saved back to OOXML.

First of all, we needed to preserve the theme files contained in the document, which are stored at /word/theme/ inside the document. LibreOffice was able to read and parse the theme definition file to assign colors and fonts to the document elements, but it was discarded after that and not preserved on export. Miguel and Andrés took care of that part of the job when working in the preservation of Smart-Art information and it’s already present in 4.2 release.

Then it was my turn to identify and preserve the theme-related attributes in the document, starting with font attributes. From the user point of view, there are two types of theme fonts in Word: Heading, named major internally, and Body, named minor. Besides this distinction, there are three types of fonts that users can manage separately: Latin, Asian and Complex Script (for Hebrew, Arabic, etc.). Users can set one specific font, or a major or minor theme font, for each of the three types.

What actual font this mess translates to depends on several things:

• Type of characters: the application decides if a portion of text is written in Latin, Asian or CS checking the range of characters it contains, and uses the font that the user set for that type.
• The document language: the theme file can define one font per language besides a fallback font per type:

        <a:minorFont>
          <!-- default fonts per type -->
          <a:latin typeface="Trebuchet MS" />
          <a:ea typeface="" />
          <a:cs typeface="" />
  
          <!-- language-specific fonts -->
          <a:font script="Hans" typeface="宋体" />
          <a:font script="Hebr" typeface="Arial" />
        </a:majorFont>
  

• The default language: the document settings file can define one default language for each font type:

  <w:themeFontLang w:val="en-US" w:eastAsia="zh-CN"
                   w:bidi="he-IL" />
  

How does it work altogether? With the above definition of the language settings, the default language for CS text is Hebrew (he-IL), and the theme defines the minor font Arial for that language. In the case of latin languages, it will be Trebuchet MS. This was a new behaviour in LibreOffice that we had to implement ourselves. Notice the mixed use of different naming conventions for languages and regions, which requires conversions.

My work on theme attributes comprises the preservation of font and shape colors too, but this post is already long enough; that will be part of the next chapter. Let me finish with a thank you to CloudOn for funding this development, and to Adam Fyne for his work detecting and triaging these theme preservation issues.

I’ve been working for a while already on the implementation of the CSS Grid Layout standard for the WebKit and Blink web engines. It’s a complex specification, indeed, like most of them, so I enjoyed a lot decrypting all the angles behind the language used to define the different CSS properties, their usage and limits, exceptions and so on.  It’s fair to start thanking the WebKit reviewers and Blink owners for their patient and support reviewing patches. It also worth mentioning that the E.D is still a live document with frequent changes and active discussions in the www-style mailing list, which is very active and supportive solving doubts and attending suggestions of the hackers working on the implementation.

Before continue reading, I’d strongly recommend reading the previous posts of my colleges Manuel and Sergio to understand the basic concepts of the CSS Grid Layout and its main features and advantages for the web.

I had the chance to land several patches in WebKit and Blink that improved the current implementation of the standard, both fixing bugs and adapting it to the latest syntax changes introduced in the spec, but perhaps the most noticeable improvements are, so far, the new grid-template and grid shorthands added recently.
 

The “grid-template” shorthand

 
Quoting the CSS Grid Layout specs:

The grid-template property is a shorthand for setting grid-template-columns, grid-template-rows, and grid-template-areas in a single declaration. It has several distinct syntax forms:

none | subgrid | <‘grid-template-columns’> / <‘grid-template-rows’> | [<’track-list’>/ ]? [<’line-names’>? <’string’> <’track-size’>?]+

It’s always easier if we have some examples at hand:

grid-template: auto 1fr auto / auto 1fr;
grid-template: 10px / "a" 15px;
grid-template: 10px / (head1) "a" 15px (tail1)
                      (head2) "b" 20px (tail2);
grid-template: (first) 10px repeat(2, (nav nav2) 15px) /       "a b c" 100px (nav)
                                                        (nav2) "d e f" 25px  (nav)
                                                        (nav2) "g h i" 25px  (last);

It’s important to notice that the subgrid functionality is under discussion to be postponed for the level 2 of the specification, hence  it was not implemented, for the time being,  in the shorthand either. This decision had the support of IE and Chromium browsers;   Mozilla partially agree on this, even though with some  doubts.

There was something special in the implementation of this shorthand property. Usually, the CSS property parsing methods are implemented straight forward, avoiding unnecessary or duplicated operations over the parsed value list. However, due to the ambiguity of the shorthand syntax, it’s not clear which form the expression belongs to until reaching the <string> clause. In order to reuse the <grid-template-{row, column}> parsing function, it was necessary to allow rewinding the parsedValue list in case of detecting the wrong form was being processed.

Another remarkable implementation detail was the change in the gridLineName parsing function, required to join the adjoining line names of the last and first columns (nav and nav2 in the example). See below the longhand equivalence of the last case in the previous example:

grid-template-columns: (first) 10px repeat(2, (nav nav2) 15px);
grid-template-rows: 100px (av nav2) 25px (nav nav2) auto (last):
grid-template-areas: "a b c" 
                     "d e f"
                     "g h i";

The “grid” shorthand

 
Quoting the CSS Grid Layout specs:

The grid property is a shorthand that sets all of the explicit grid properties (grid-template-rows, grid-template-columns, and grid-template-areas) as well as all the implicit grid properties (grid-auto-rows, grid-auto-columns, and grid-auto-flow) in a single declaration.

<‘grid-template’> | [<‘grid-auto-flow’> [<‘grid-auto-columns’>[/ <‘grid-auto-rows’>]?]?]

Even that the shorthand sets both implicit and explicit grid properties, it can be only specified either implicit or explicit grid properties; the missing properties will be set to the initial values. Now let’s see some examples:

grid: 15px / 10px;
grid: row 10px;
grid: column 10px / 20px;

The “grid” shorthand is the recommended mechanism even to define just the  the explicit shorthand, unless web authors are interested on cascade separately the impicit grid properties.
 

Current status and next steps

 
Both properties landed Blink trunk rencetly (revisions 170552 and 171143) and and they are waiting for the final review in WebKit, hopefully they will land soon. There are enough layout tests to cover the most common cases but perhaps some additional cases might be added in the future. As it was mentioned, there are certain ambiguities in both shorthands syntax and it’s also important to check out the www-style mailing list looking for changes that might require modifying the implementation, hence adding the proper test cases.

With the implemmentation of these two new shorthands, the properties implementation tasks are almost completed. We are working gonw on fixing bugs and implementing the alignment features. There is a quite important gap between the Blink and WebKit implementation, but we are working on porting patches as soon as possible, since we think it’s important to have both implementations synced.

I’ll attend the WebKit Contributors Meeting next week, so perhaps I could speed up the landing the patches for the shorthand properties. My main goal, though, will be to gather feedback from the WebKit community about the status of the CSS Grid Layout implementation, what features they miss the most, which bugs should have more priority and share with them our future plans at Igalia.

All this work was possbile thanks to the collaboration between Igalia and Bloomberg, We both are working hard to help and promote the wide adpoption of this standar, which will be shipped soon on IE and hopefully also in Chromimum. We are also following the efforts Mozila is doing, which give us the impresion that the interest of most of the browsers on this standar is quite high.

Igalia has opened internship positions for students, to work in tasks around WebKitGTK+. This is a great opportunity for you to become acquaintance with the WebKit project, GNOME, their communities, and also to learn about our company.

Many Igalians started their journey in the company by doing an internship during their study years and eventually came back to join us full-time, so if you are interested to join us at some point but are still in the middle of your studies, this is a great opportunity to get us to know you and have a wonderful summer.

More details in the official announcement. Applications are due on May 8th. Do not hesitate to apply!

Hi there, fellow readers. Today I’m starting a mini-series of posts to talk a little bit about the work I’ve been lately doing at Igalia around WebKit and Blink web engines. I’ve been involved in the implementation of a new standard called CSS Grid Layout in both engines. My mate rego has already talked about that, so take a look at his post if you need to know more about the basics. Read it? Great, let’s move on.

The new grid layout

I’d like to stress that grid layout is not (only) about aligning elements in columns and rows (after all you can already do it with tables or complicated layouts using positioned and/or floating blocks right?). What’s interesting is that a grid layout doesn’t have content structure. What does it mean? As the specs say:

unlike tables, grid layout doesn’t have content structure, and thus enables a wide variety of layouts not possible with tables. For example, the children of a grid container can position themselves such that they overlap and layer similar to positioned elements.

In addition, it provides a very fine grained control of the layout of the elements of the grid (generally with the help of media queries) based on device form factors, orientation, available space, etc…

Today, I’m going to focus specifically on how to position elements inside a grid, and in particular how to do that using named grid lines and named grid areas. Going too fast? No problem, let’s recap a bit.

Positioning grid items

In CSS grid layout a grid is defined by a set of intersecting horizontal and vertical lines which divide the space of the grid in areas where to place grid items. Those grid lines are numbered starting from number 1. Based on a declaration like:

	grid-template-rows: 50px 1fr 50px;
	grid-template-columns: 150px 1fr;

we would get the following

Grid lines defining a grid with 3 rows and 2 columns

So in order to position an item in a grid, you have the a very straightforward way of doing it which consist of specifying the the grid line’s start and end positions for either columns, rows or both. Given the above grid, and supposing that you want to place some item in the rightmost area on the bottom, you’d just need to specify it the following way:

	grid-row: 3 / 4;
	grid-column: 2 / 3;

Easy huh? Your item will span from column lines 2 to 3 and from row lines 3 to 4. There are tons of different ways to specify the position and the span for your grid items, so many that perhaps they deserve a separate blog post (just check this section of the specs if you need the details now).

Placing items using named grid lines

Despite being easy and even intuitive (at least for us developers), this way of placing grid items using grid line numbers might not be optimal for web authors. With large grids, knowing the exact grid line our item should span to could become an increasingly difficult task. So, why not give the grid lines a meaningful name? What about giving authors the possibility to assign a name (or multiple) to each grid line so they wouldn’t have to remember/compute the exact index of each grid line?

Naming grid lines is pretty easy. Authors just need to specify the names they want to assign to each line (yeah “names” in plural) in the grid definition by putting them into parentheses surrounding the definitions of the grid track breadths. Something like:

	grid-template-rows: (first start) 100px 40px (middle) 1fr 250px (last)

Based on this definition, the following declarations are totally equivalent:

	grid-row: first / middle;
	grid-row: 1 / middle;
	grid-row: first / 3;
	grid-row: 1 / 3;
	grid-row: start / middle;
	grid-row: start / 3;

I wouldn’t enter into discussing which one is the most descriptive or the best (there are some more), just choose the one you prefer.

Placing items using named areas

Although things like named grid lines ease the design task, authors don’t normally foresee the layout of a web page using a matrix of positions but instead they normally divide the available space in “areas” in their minds, we’re talking about areas like the footer, the side bar, the header, etc… So why don’t take advantage of that and allow the web authors to specify those grid areas?

For example, let’s imagine that we’re designing a website with a header on top, a footer in the bottom, a sidebar with links for example on your left and the important content on your right. We can easily (and very visually I’d say) define these four grid areas using the CSS Grid Layout properties. It’d be something like this:

	grid-template-areas: "  .     header  header"
	                     "sidebar content content"
	                     "sidebar content content"
	                     "  .     footer  footer";

There you go, a 4×3 matrix with 4 different named grid areas. Some remarks:

• The dot “.” means unnamed area
• “header” will occupy columns 2 and 3 in the first row
• “footer” will occupy columns 2 and 3 in the last row
• “sidebar” will span from 2nd to 3rd row in the first column
• “content” will fill columns 2 to 3 in rows 2 to 3

That’s all. Based on that definition, we could use the grid-area property to place our elements. We just need to specify the grid area where to place them and the grid will automatically position them based on their size and available free space.

Placing items using grid area’s implicit grid line names

Hmm, not enough for you? You said you still need something extra? So you want the simplicity of grid areas and the flexibility of named grid lines? No problem, the spec defines what’s called the implicit named grid lines generated by each defined grid area. What does it mean? Basically that for any grid area you define, let’s call it “A“, the engine will automatically generate “A-start” and “A-end” grid line names for columns and rows, so you could use them to position any grid item as if they were explicitly defined by you.

If I take the grid defined above with grid-template-areas it’s perfectly valid to position a grid item using the following declarations

	grid-column: header-start / 3;
	grid-row: 2 / sidebar-end;

which are equivalent to

	grid-column: 2 / 3;
	grid-row: 2 / 4;

As you might have wondered, you can mix all these three methods to position grid items together.

How is it implemented?

What I have described should be working fine in latest WebKit nightlies after I landed this. Regarding Blink, there are some missing bits already addressed here that will likely land soon. Obviously this is not a matter of a single change , we’re building all this on top of many other changes in WebKit and also in Blink.

In my next post I’ll talk about the implementation details and also about the differences between WebKit and Blink as we have followed slightly different paths to implement this. Stay tuned!

Acknowledgements

Many thanks for the nice reviews I got from both WebKit reviewers and Blink owners. Also I’d explicitly like to thank the awesome guys at Bloomberg for sponsoring this work.