Let's say you decide that you'd like to see what some other processes on your system are doing to a subtree of the file system. You don't want to have to change how those processes work -- you just want to see what files those processes create and delete.
One approach would be to just scan the file-system tree periodically, enumerating its contents. But when the file system tree is large and the change rate is low, that's not an optimal thing to do.
Fortunately, Linux provides an API to allow a process to receive notifications on file-system change events, called inotify. So you open up the inotify(7) manual page, and are greeted with this:
With careful programming, an application can use inotify to efficiently monitor and cache the state of a set of filesystem objects. However, robust applications should allow for the fact that bugs in the monitoring logic or races of the kind described below may leave the cache inconsistent with the filesystem state. It is probably wise to do some consistency checking, and rebuild the cache when inconsistencies are detected.
It's not exactly reassuring is it? I mean, "you had one job" and all.
Reading down a bit farther, I thought that with some "careful programming", I could get by. After a day of trying, I am now certain that it is impossible to build a correct recursive directory monitor with inotify, and I am not even sure that "good enough" solutions exist.
pitfall the first: buffer overflow
Fundamentally, inotify races the monitoring process with all other processes on the system. Events are delivered to the monitoring process via a fixed-size buffer that can overflow, and the monitoring process provides no back-pressure on the system's rate of filesystem modifications. With inotify, you have to be ready to lose events.
This I think is probably the easiest limitation to work around. The kernel can let you know when the buffer overflows, and you can tweak the buffer size. Still, it's a first indication that perfect is not possible.
pitfall the second: now you see it, now you don't
This one is the real kicker. Say you get an event that says that a file "frenemies.txt" has been created in the directory "/contacts/". You go to open the file -- but is it still there? By the time you get around to looking for it, it could have been deleted, or renamed, or maybe even created again or replaced! This is a TOCTTOU race, built-in to the inotify API. It is literally impossible to use inotify without this class of error.
The canonical solution to this kind of issue in the kernel is to use file descriptors instead. Instead of or possibly in addition to getting a name with the file change event, you get a descriptor to a (possibly-unlinked) open file, which you would then be responsible for closing. But that's not what inotify does. Oh well!
pitfall the third: race conditions between inotify instances
When you inotify a directory, you get change notifications for just that directory. If you want to get change notifications for subdirectories, you need to open more inotify instances and poll on them all. However now you have N2 problems: as poll and the like return an unordered set of readable file descriptors, each with their own ordering, you no longer have access to a linear order in which changes occurred.
It is impossible to build a recursive directory watcher that definitively says "ok, first /contacts/frenemies.txt was created, then /contacts was renamed to /peeps, ..." because you have no ordering between the different watches. You don't know that there was ever even a time that /contacts/frenemies.txt was an accessible file name; it could have been only ever openable as /peeps/frenemies.txt.
Of course, this is the most basic ordering problem. If you are building a monitoring tool that actually wants to open files -- good luck bubster! It literally cannot be correct. (It might work well enough, of course.)
reflections
As far as I am aware, inotify came out to address the needs of desktop search tools like the belated Beagle (11/10 good pupper just trying to get his pup on). Especially in the days of spinning metal, grovelling over the whole hard-drive was a real non-starter, especially if the search database should to be up-to-date.
But after looking into inotify, I start to see why someone at Google said that desktop search was in some ways harder than web search -- I mean we all struggle to find files on our own machines, even now, 15 years after the whole dnotify/inotify thing started. Part of it is that the given the choice between supporting reliable, fool-proof file system indexes on the one hand, and overclocking the IOPS benchmarks on the other, the kernel gave us inotify. I understand it, but inotify still sucks.
I dunno about you all but whenever I've had to document such an egregious uncorrectable failure mode as any of the ones in the inotify manual, I have rewritten the software instead. In that spirit, I hope that some day we shall send inotify to the pet cemetery, to rest in peace beside Beagle.
Hello, all! Today I'd like to share some work I have done recently as part of the Snabb user-space networking toolkit. Snabb is mainly about high-performance packet processing, but it also needs to communicate with management-oriented parts of network infrastructure. These communication needs are performed by a dedicated manager process, but that process has many things to do, and can't afford to make blocking operations.
Snabb is written in Lua, which doesn't have built-in facilities for concurrency. What we'd like is to have fibers. Fortunately, Lua's coroutines are powerful enough to implement fibers. Let's do that!
fibers in lua
First we need a scheduling facility. Here's the smallest possible scheduler: simply a queue of tasks and a function to run those tasks.
local task_queue = {}
function schedule_task(thunk)
table.insert(task_queue, thunk)
end
function run_tasks()
local queue = task_queue
task_queue = {}
for _,thunk in ipairs(queue) do thunk() end
end
For our purposes, a task is just a function that will be called with no arguments.
Now let's build fibers. This is easier than you might think!
local current_fiber = false
function spawn_fiber(fn)
local fiber = coroutine.create(fn)
schedule_task(function () resume_fiber(fiber) end)
end
function resume_fiber(fiber, ...)
current_fiber = fiber
local ok, err = coroutine.resume(fiber, ...)
current_fiber = nil
if not ok then
print('Error while running fiber: '..tostring(err))
end
end
function suspend_current_fiber(block, ...)
-- The block function should arrange to reschedule
-- the fiber when it becomes runnable.
block(current_fiber, ...)
return coroutine.yield()
end
Here, a fiber is simply a coroutine underneath. Suspending a fiber suspends the coroutine. Resuming a fiber runs the coroutine. If you're unfamiliar with coroutines, or coroutines in Lua, maybe have a look at the lua-users wiki page on the topic.
The difference between a fibers facility and just coroutines is that with fibers, you have a scheduler as well. Very much like Scheme's call-with-prompt, coroutines are one of those powerful language building blocks that should rarely be used directly; concurrent programming needs more structure than what Lua offers.
If you're following along, it's probably worth it here to think how you would implement yield based on these functions. A yield implementation should yield control to the scheduler, and resume the fiber on the next scheduler turn. The answer is here.
Even though Lua doesn't support multiple machine threads running concurrently, concurrency between fibers can still be fraught with bugs. Tony Hoare's Communicating Sequential Processes showed that we can avoid a class of these bugs by treating communication as a first-class concept.
Happily, the Concurrent ML project showed that it's possible to build these first-class communication facilities as a library, provided the language you are working in has threads of some kind, and fibers are enough. Last year I built a Concurrent ML library for Guile Scheme, and when in Snabb we had a similar need, I ported that code over to Lua. As it's a new take on the problem in a different language, I think I've been able to simplify things even more.
So let's take a crack at implementing Concurrent ML in Lua. In CML, the fundamental primitive for communication is the operation. An operation represents the potential for communication. For example, if you have a channel, it would have methods to return "get operations" and "put operations" on that channel. Actually receiving or sending a message on a channel occurs by performing those operations. One operation can be performed many times, or not at all.
Compared to a system like Go, for example, there are two main advantages of CML. The first is that CML allows non-deterministic choice between a number of potential operations in a generic way. For example, you can construct a operation that, when performed, will either get on one channel or wait for a condition variable to be signalled, whichever comes first. In Go, you can only select between operations on channels.
The other interesting part of CML is that operations are built from a uniform protocol, and so users can implement new kinds of operations. Compare again to Go where all you have are channels, and nothing else.
The CML operation protocol consists three related functions: try which attempts to directly complete an operation in a non-blocking way; block, which is called after a fiber has suspended, and which arranges to resume the fiber when the operation completes; and wrap, which is called on the result of a successfully performed operation.
In Lua, we can call this an implementation of an operation, and create it like this:
function new_op_impl(try, block, wrap)
return { try=try, block=block, wrap=wrap }
end
Now let's go ahead and write the guts of CML: the operation implementation. We'll represent an operation as a Lua object with two methods. The perform method will attempt to perform the operation, and return the resulting value. If the operation can complete immediately, the call to perform will return directly. Otherwise, perform will suspend the current fiber and arrange to continue only when the operation completes.
The wrap method "decorates" an operation, returning a new operation that, if and when it completes, will "wrap" the result of the completed operation with a function, by applying the function to the result. It's useful to distinguish the sub-operations of a non-deterministic choice from each other.
Here our new_op function will take an array of operation implementations and return an operation that, when performed, will synchronize on the first available operation. As you can see, it already has the equivalent of Go's select built in.
function new_op(impls)
local op = { impls=impls }
function op.perform()
for _,impl in ipairs(impls) do
local success, val = impl.try()
if success then return impl.wrap(val) end
end
local function block(fiber)
local suspension = new_suspension(fiber)
for _,impl in ipairs(impls) do
impl.block(suspension, impl.wrap)
end
end
local wrap, val = suspend_current_fiber(block)
return wrap(val)
end
function op.wrap(f)
local wrapped = {}
for _, impl in ipairs(impls) do
local function wrap(val)
return f(impl.wrap(val))
end
local impl = new_op_impl(impl.try, impl.block, wrap)
table.insert(wrapped, impl)
end
return new_op(wrapped)
end
return op
end
There's only one thing missing there, which is new_suspension. When you go to suspend a fiber because none of the operations that it's trying to do can complete directly (i.e. all of the try functions of its impls returned false), at that point the corresponding block functions will publish the fact that the fiber is waiting. However the fiber only waits until the first operation is ready; subsequent operations becoming ready should be ignored. The suspension is the object that manages this state.
function new_suspension(fiber)
local waiting = true
local suspension = {}
function suspension.waiting() return waiting end
function suspension.complete(wrap, val)
assert(waiting)
waiting = false
local function resume()
resume_fiber(fiber, wrap, val)
end
schedule_task(resume)
end
return suspension
end
As you can see, the suspension's complete method is also the bit that actually arranges to resume a suspended fiber.
Finally, just to round out the implementation, here's a function implementing non-deterministic choice from among a number of sub-operations:
function choice(...)
local impls = {}
for _, op in ipairs({...}) do
for _, impl in ipairs(op.impls) do
table.insert(impls, impl)
end
end
return new_op(impls)
end
on cml
OK, I'm sure this seems a bit abstract at this point. Let's implement something concrete in terms of these primitives: channels.
Channels expose two similar but different kinds of operations: put operations, which try to send a value, and get operations, which try to receive a value. If there's a sender already waiting to send when we go to perform a get_op, the operation continues directly, and we resume the sender; otherwise the receiver publishes its suspension to a queue. The put_op case is similar.
Finally we add some synchronous put and get convenience methods, in terms of their corresponding CML operations.
function new_channel()
local ch = {}
-- Queues of suspended fibers waiting to get or put values
-- via this channel.
local getq, putq = {}, {}
local function default_wrap(val) return val end
local function is_empty(q) return #q == 0 end
local function peek_front(q) return q[1] end
local function pop_front(q) return table.remove(q, 1) end
local function push_back(q, x) q[#q+1] = x end
-- Since a suspension could complete in multiple ways
-- because of non-deterministic choice, it could be that
-- suspensions on a channel's putq or getq are already
-- completed. This helper removes already-completed
-- suspensions.
local function remove_stale_entries(q)
local i = 1
while i <= #q do
if q[i].suspension.waiting() then
i = i + 1
else
table.remove(q, i)
end
end
end
-- Make an operation that if and when it completes will
-- rendezvous with a receiver fiber to send VAL over the
-- channel. Result of performing operation is nil.
function ch.put_op(val)
local function try()
remove_stale_entries(getq)
if is_empty(getq) then
return false, nil
else
local remote = pop_front(getq)
remote.suspension.complete(remote.wrap, val)
return true, nil
end
end
local function block(suspension, wrap)
remove_stale_entries(putq)
push_back(putq, {suspension=suspension, wrap=wrap, val=val})
end
return new_op({new_op_impl(try, block, default_wrap)})
end
-- Make an operation that if and when it completes will
-- rendezvous with a sender fiber to receive one value from
-- the channel. Result is the value received.
function ch.get_op()
local function try()
remove_stale_entries(putq)
if is_empty(putq) then
return false, nil
else
local remote = pop_front(putq)
remote.suspension.complete(remote.wrap)
return true, remote.val
end
end
local function block(suspension, wrap)
remove_stale_entries(getq)
push_back(getq, {suspension=suspension, wrap=wrap})
end
return new_op({new_op_impl(try, block, default_wrap)})
end
function ch.put(val) return ch.put_op(val).perform() end
function ch.get() return ch.get_op().perform() end
return ch
end
a wee example
You might be wondering what it's like to program with channels in Lua, so here's a little example that shows a prime sieve based on channels. It's not a great example of concurrency in that it's not an inherently concurrent problem, but it's cute to show computations in terms of infinite streams.
function prime_sieve(count)
local function sieve(p, rx)
local tx = new_channel()
spawn_fiber(function ()
while true do
local n = rx.get()
if n % p ~= 0 then tx.put(n) end
end
end)
return tx
end
local function integers_from(n)
local tx = new_channel()
spawn_fiber(function ()
while true do
tx.put(n)
n = n + 1
end
end)
return tx
end
local function primes()
local tx = new_channel()
spawn_fiber(function ()
local rx = integers_from(2)
while true do
local p = rx.get()
tx.put(p)
rx = sieve(p, rx)
end
end)
return tx
end
local done = false
spawn_fiber(function()
local rx = primes()
for i=1,count do print(rx.get()) end
done = true
end)
while not done do run_tasks() end
end
Here you also see an example of running the scheduler in the last line.
where next?
Let's put this into perspective: in a couple hundred lines of code, we've gone from minimal Lua to a language with lightweight multitasking, extensible CML-based operations, and CSP-style channels; truly a delight.
There are a number of possible ways to extend this code. One of them is to implement true multithreading, if the language you are working in supports that. In that case there are some small protocol modifications to take into account; see the notes on the Guile CML implementation and especially the Manticore Parallel CML project.
The implementation above is pleasantly small, but it could be faster with the choice of more specialized data structures. I think interested readers probably see a number of opportunities there.
In a library, you might want to avoid the global task_queue and implement nested or multiple independent schedulers, and of course in a parallel situation you'll want core-local schedulers as well.
The implementation above has no notion of time. What we did in the Snabb implementation of fibers was to implement a timer wheel, inspired by Juho Snellman's Ratas, and then add that timer wheel as a task source to Snabb's scheduler. In Snabb, every time the equivalent of run_tasks() is called, a scheduler asks its sources to schedule additional tasks. The timer wheel implementation schedules expired timers. It's straightforward to build CML timeout operations in terms of timers.
Additionally, your system probably has other external sources of communication, such as sockets. The trick to integrating sockets into fibers is to suspend the current fiber whenever an operation on a file descriptor would block, and arrange to resume it when the operation can proceed. Here's the implementation in Snabb.
The only difficult bit with getting nice nonblocking socket support is that you need to be able to suspend the calling thread when you see the EWOULDBLOCK condition, and for coroutines that is often only possible if you implemented the buffered I/O yourself. In Snabb that's what we did: we implemented a compatible replacement for Lua's built-in streams, in Lua. That lets us handle EWOULDBLOCK conditions in a flexible manner. Integrating epoll as a task source also lets us sleep when there are no runnable tasks.
Likewise in the Snabb context, we are also working on a TCP implementation. In that case you want to structure TCP endpoints as fibers, and arrange to suspend and resume them as appropriate, while also allowing timeouts. I think the scheduler and CML patterns are going to allow us to do that without much trouble. (Of course, the TCP implementation will give us lots of trouble!)
Additionally your system might want to communicate with fibers from other threads. It's entirely possible to implement CML on top of pthreads, and it's entirely possible as well to support communication between pthreads and fibers. If this is interesting to you, see Guile's implementation.
When I talked about fibers in an earlier article, I built them in terms of delimited continuations. Delimited continuations are fun and more expressive than coroutines, but it turns out that for fibers, all you need is the expressive power of coroutines -- multi-shot continuations aren't useful. Also I think the presentation might be more straightforward. So if all your language has is coroutines, that's still good enough.
There are many more kinds of standard CML operations; implementing those is also another next step. In particular, I have found semaphores and condition variables to be quite useful. Also, standard CML supports "guards", invoked when an operation is performed, and "nacks", invoked when an operation is definitively not performed because a choice selected some other operation. These can be layered on top; see the Parallel CML paper for notes on "primitive CML".
Also, the choice operator above is left-biased: it will prefer earlier impls over later ones. You might want to not always start with the first impl in the list.
The scheduler shown above is the simplest thing I could come up with. You may want to experiment with other scheduling algorithms, e.g. capability-based scheduling, or kill-safe abstractions. Do it!
Or, it could be you already have a scheduler, like some kind of main loop that's already there. Cool, you can use it directly -- all that fibers needs is some way to schedule functions to run.
godspeed
In summary, I think Concurrent ML should be better-known. Its simplicity and expressivity make it a valuable part of any concurrent system. Already in Snabb it helped us solve some longstanding gnarly issues by making the right solutions expressible.
As Adam Solove says, Concurrent ML is great, but it has a branding problem. Its ideas haven't penetrated the industrial concurrent programming world to the extent that they should. This article is another attempt to try to get the word out. Thanks to Adam for the observation that CML is really a protocol; I'm sure the concepts could be made even more clear, but at least this is a step forward.
All the code in this article is up on a gitlab snippet along with instructions for running the example program from the command line. Give it a go, and happy hacking with CML!
IPFIX is a standard for recording “flows”, which are packets with the same: source IP, destination IP, source port, destination port and protocol. It’s got a few uses but probably the most familiar is tracking usage for billing. The recording is done by something called a “probe” which gets the traffic in and builds a “flow table”, usually some form of hash-map which records a “flow” backed by additional data like how many bytes and packets have been transmitted. These flows are then exported by an “exporter”, usually just another process outside the performance critical “dataplane” (probe) to a collector which can store and display the information in a useful / pretty manner such as a web UI. IPFIX usually runs on a copy of the traffic to ensure not to hinder the traffic itself, which means we shouldn’t concern ourselves too much with what comes out at the end.
So, why am I telling you about IPFIX? My colleagues Asumu and Andy have already implemented this in Snabb, and it already exists as a plugin in VPP. What it does is not too tricky and thus it makes for a great example to try to re-implement as a method of learning VPP. We’re able to test performance and see how things work while hacking on something familiar. There are definitely more features it could have and that the stock VPP one probably does implement. The Snabb implementation and our VPP plugin are fairly bare-bones, but they do record some basic information and export it.
Measurements!
We’re going to use a server with two network cards connected in a pair. This allows us to attach a Snabb app we wrote to one end which will send packets and have those packets be received by another card by VPP. The network card works by having ring buffer with a read pointer and a write pointer. You advance the read pointer to access the next packets and the network card will advance the write buffer when writing new packets it receives. The card keeps track of how often it finds itself writing over the unread packets in a counter. You can see some of the counters, including the rx missed counter showing how many packets we’re dropping by being too slow. This is what it looks when you ask VPP to show those counters:
TenGigabitEthernet2/0/0 1 up TenGigabitEthernet2/0/0
Ethernet address 00:11:22:33:44:55
Intel 82599
carrier up full duplex speed 10000 mtu 9216
rx queues 1, rx desc 1024, tx queues 2, tx desc 1024
cpu socket 0
rx frames ok 2398280
rx bytes ok 143896800
rx missed 252816
extended stats:
rx good packets 2398280
rx good bytes 143896800
rx q0packets 2398280
rx q0bytes 143896800
rx size 64 packets 2658721
rx total packets 2658694
rx total bytes 159520296
rx l3 l4 xsum error 2658762
rx priority0 dropped 253924
local0 0 down local0
We need to find the no-drop rate (NDR), which is the highest speed where we can keep up with the load. When the card shows that we’re dropping packets, we know we’re going too fast and need to back off. We’re able to find this speed by performing a binary search. Bisecting the speed and testing testing if we’re dropping packets, if so then we need to go lower, bisecting the remaining speed, otherwise bisect the higher range. This is an example of our snabb app performing the binary search, looking for the NDR:
The output shows applying a specific load and the success or failed state is based on if the rx missed counter from VPP is showing packets dropped or not. If we apply a load and it fails then we attempt it three times. The output of the failed retries has been removed for brevity.
The sample data is a test data which includes a lot of flows, we have 6 different sample data sets which we test three times to be able to avarage the data. The sample datasets range from the smallest possible ethernet frame size, 64 bytes to the largest standard ethernet frame size of 1522 bytes.
Reaching peek performance
The initial results without any performance tweaking on our VPP plugin are:
The first thing that grabbed my interest was that we weren’t getting to full speed with larger packets, something that shouldn’t be too tricky. As it turns out we had added some ELOG statements, a light-weight method of logging in VPP. These are pretty useful for getting a good look inside your function so that you can figure out what happens and when. Turns out they’re not as lightweight as I’d have liked. Removing those allowed us to get better performance across the board, including reaching line-rate (10 Gigabit per second) for frame sizes of 200 and over.
Then my attention turned to the 64 and 100 frame sizes. I was wondering how long we were actually taking on average for each packet and how it compared to the stock VPP plugin. It turned out that we’re taking around 131 nanoseconds when the stock one takes around 67 nanoseconds. That’s a big difference, and for those who read my first networking post, you’ll remember I mentioned that you have around 67.2 nanoseconds per packet. We were definitely exceeding our budget. Taking a bit of a closer look, it’s taking us around 103 ns just to look up in our flow table whether a record exists or if it’s a new flow.
I decided to look at what the stock plugin does. VPP has a lot of different data-types built in, and they had decided to go down a different route than us. They’re using a vector where the hash of the flow key is the index and the value is an index into a pool. VPP pools are fast blocks of memory to fit fixed sized stuff into, you add things and get back an index and use that to fetch it. We’re using a bihash which is a bounded-index hash map, it’s pretty handy as we just have the key as the flow key and then backing it the flow entry containing for example: amount of packets, bytes, and the start and end times for the flow. We had a similar problem in Snabb where our hashing algorithm was slow for the large key. I thought maybe the same thing was occurring but in actuality, the stock VPP was using the same xxhash algorithm that bihashes use by default and that was getting good performance. Not that then.
The bi-hash works by having a vector of buckets, the amount of which are specified when you initialise it. When you add an item it selects the bucket based on the hash. Each bucket maintains it’s own cache able to fit a specific number of bi-hash entries (both key and value). The size of the cache is based a constant, defined for various key and value sized bi-hashes. The more buckets you have, the fewer keys per bucket and thus, more caches per bi-hash entries. The entries are stored on the heap in pages which the bucket manages. When an item isn’t in the cache of the bucket then, it performs a linear search over the allocated pages until it’s found. Here’s what the bucket data structure looks like:
We were only defining 1024 buckets which meant that we missing the cache a lot and falling back to slow linear searches. Increasing this value up to 32768, we start getting much better performance:
Now we seem to be outperforming both the stock VPP plugin and Snabb. The improvements over Snabb could be due to a number of things, however VPP does something rather interesting which may give it the edge. In VPP there is a loop which can handle two packets at once, it’s identical to the single packet loop but apparently operating on two packets at a time provides performance improvements. At a talk in FOSDEM they claimed it helped, and in some parts of VPP they have loops which handle 4 packets per iteration. I think that in the future exploring why we’re seeing the VPP plugin perform better compared to Snabb would be an interesting endeavour.
The Vulkan specification includes a number of optional features that drivers may or may not support, as described in chapter 30.1 Features. Application developers can query the driver for supported features via vkGetPhysicalDeviceFeatures() and then activate the subset they need in the pEnabledFeatures field of the VkDeviceCreateInfo structure passed at device creation time.
In the last few weeks I have been spending some time, together with my colleague Chema, adding support for one of these features in Anvil, the Intel Vulkan driver in Mesa, called shaderInt16, which we landed in Mesa master last week. This is an optional feature available since Vulkan 1.0. From the spec:
shaderInt16 specifies whether 16-bit integers (signed and unsigned) are supported in shader code. If this feature is not enabled, 16-bit integer types must not be used in shader code. This also specifies whether shader modules can declare the Int16 capability.
It is probably relevant to highlight that this Vulkan capability also requires the SPIR-V Int16 capability, which basically means that the driver’s SPIR-V compiler backend can actually digest SPIR-V shaders that declare and use 16-bit integers, and which is really the core of the functionality exposed by the Vulkan feature.
Ideally, shaderInt16 would increase the overall throughput of integer operations in shaders, leading to better performance when you don’t need a full 32-bit range. It may also provide better overall register usage since you need less register space to store your integer data during shader execution. It is important to remark, however, that not all hardware platforms (Intel or otherwise) may have native support for all possible types of 16-bit operations, and thus, some of them might still need to run in 32-bit (which requires injecting type conversion instructions in the shader code). For Intel platforms, this is the case for operations associated with integer division.
From the point of view of the driver, this is the first time that we generally exercise lower bit-size data types in the driver compiler backend, so if you find any bugs in the implementation, please file bug reports in bugzilla!
Speaking of shaderInt16, I think it is worth mentioning its interactions with other Vulkan functionality that we implemented in the past: the Vulkan 1.0 VK_KHR_16bit_storage extension (which has been promoted to core in Vulkan 1.1). From the spec:
The VK_KHR_16bit_storage extension allows use of 16-bit types in shader input and output interfaces, and push constant blocks. This extension introduces several new optional features which map to SPIR-V capabilities and allow access to 16-bit data in Block-decorated objects in the Uniform and the StorageBuffer storage classes, and objects in the PushConstant storage class. This extension allows 16-bit variables to be declared and used as user-defined shader inputs and outputs but does not change location assignment and component assignment rules.
While the shaderInt16 capability provides the means to operate with 16-bit integers inside a shader, the VK_KHR_16bit_storage extension provides developers with the means to also feed shaders with 16-bit integer (and also floating point) input data, such as Uniform/Storage Buffer Objects or Push Constants, from the applications side, plus, it also gives the opportunity for linked shader stages in a graphics pipeline to consume 16-bit shader inputs and produce 16-bit shader outputs.
VK_KHR_16bit_storage and shaderInt16 should be seen as two parts of a whole, each one addressing one part of a larger problem: VK_KHR_16bit_storage can help reduce memory bandwith for Uniform and Storage Buffer data accessed from the shaders and / or optimize Push Constant space, of which there are only a few bytes available, making it a precious shader resource, but without shaderInt16, shaders that are fed 16-bit input data are still required to convert this data to 32-bit internally for operation (and then back again to 16-bit for output if needed). Likewise, shaders that use shaderInt16 without VK_KHR_16bit_storage can only operate with 16-bit data that is generated inside the shader, which largely limits its usage. Both together, however, give you the complete functionality.
Conclusions
We are very happy to continue expanding the feature set supported in Anvil and we look forward to seeing application developers making good use of shaderInt16 in Vulkan to improve shader performance. As noted above, this is the first time that we fully enable the shader compiler backend to do general purpose operations on lower bit-size data types and there might be things that we can still improve or optimize. If you hit any issues with the implementation, please contact us and / or file bug reports so we can continue to improve the implementation.
Last weekend I attended Ubucon Europe 2018 in the city center of Gijón, Spain, at the Antiguo Instituto. This 3-day conference was full of talks, sometimes 3 or even 4 at the same time! The covered topics went from Ubuntu LoCo teams to Industrial Hardware… there was at least one talk interesting for all attendees! Please check the schedule if you want to know what happened at Ubucon ;-)
I would like to say thank you to all the organization team. They did a great job taking care of both the attendees and the speakers, they organized a very nice traditional espicha that introduced good cider and the Asturian gastronomy to the people coming from any part of the world. The conference was amazing and I am very pleased to have participated in this event. I recommend you to attend it next year… lots of fun and open-source!
Regarding my talk “Introduction to Mesa, an open-source graphics API”, there were 30 people attending to it. It was great to check that so many people were interested on a brief introduction about how Mesa works, how the community collaborate together and how anyone can become contributor to Mesa, even as a non-developer. I hope I can see them contributing to Mesa in the coming future!
Last week, I attended BlinkOn 9. I was very happy to
spend some time with my colleagues working on Chromium, including a new
developer who will join my team next week (to be announced soon!).
This edition had the usual format with presentations, brainstorming,
lightning talks and informal chats with Chromium developers. I attended several
interesting presentations on web platform standardization,
implementation and testing.
It was also great to talk to Googlers in order to coordinate on some of
Igalia’s projects such as
the collaboration with AMP or
MathML in Chromium.
A description of our non-hierarchical model and benefits it brings.
An overview of Igalia’s contribution to the Web Platform.
From the feedback I got, people appreciated the presentation and liked to get more
insight on Igalia. Unfortunately, I was not able to record
the talk due to technical issues. Of course, thirty minutes is a bit short to
develop all the ideas and reply properly to all the questions. But for those who
are interested here are more pointers:
About dependency on the customers, see the last
paragraph of “work groups” in Andy’s blog post “but that would be anarchy!” especially
treating customers as partners. As I said during the talk, as long as we
have enough customers we have some freedom to accept contracts that are more
interesting for our strategy and aligned with our values or negociate
improvements to existing contracts ; without worrying about unstability
and uncertainty.
About the meaning of “Igalia”, the simple answer is
“it does not mean anything”. If you join Igalia and get the opportunity to
learn about the company history, there is a more complete answer about how
the name was found…
Regarding founders of Igalia in 2001: Dape (who attended BlinkOn), Alex, Juanjo, Xavi, Berto and Chema are indeed still
working at Igalia and in general, very few people have left Igalia since its creation.
Finally we had two related tricky questions from Google employees:
How do you sync with the browser vendors’ own agenda?
What can Google (or any other browser vendor) could do to facilitate involvements of third-party contributors?
One could enumerate different situations but unfortunately there is not a
generic answer. In some cases collaboration worked very well and was
quite successful.
In other cases, things were more complicated and we had to “fight” to
convince browser vendors to keep some existing code or accept new features.
Communication is very important. We try to sync with browser vendors using video conferencing or by attending conferences, but some companies/teams are more or less inclined to reveal information (especially when strategic products
are involved). In general, I have the impression that the more the teams work
close to the Web Platform,
the more they are used to the democratic and open-source
culture and welcome third-party contributions.
Although the ideal is to work upstream, we have recently been developing skills
to manage separate forks and rebase them regularly against the main branch. This
is a good option to find a balance between the request of the customer to implement
features and the need of the browser vendors to focus on their own tasks.
Chromium for Wayland is a good
example of that approach.
Hence probably one way to help third-party contributors is to improve
communication. We had some issues with developers not even willing to talk to us
or not taking time to review or comment on our patches/CLs. If browser vendors
could indicate that they don’t like an approach as soon as possible or that
they won’t accept patches until some refactoring is complete, that would help
us a lot to discuss with clients, properly schedule our tasks and consider the
option of an experimental branch.
Another way to help third-party contributors would be to advertize more
that such contributions are actually possible. Indeed, many people think that
“everything is implemented by browser vendors” which can make difficult to
find clients for web platform development. When companies rant about Google
not implementing feature X, fixing bug Y or participating to standard Z, instead
of ignoring them or denying the importance of the request, it would probably be
more constructive to mention that they can actually pay consulting companies to
do that job. As I indicated in the talk, we recently had such successful
collaborations with Bloomberg,
Metrological or
AMP and we would be happy to find more!
There are probably more to reply to these questions, but that’s my quick
thought on the matter for now. I’ll try discussing with my
colleagues and see if we have more ideas to share.
gst-build is far superior than gst-uninstalled scripts for developing GStreamer, mainly because its meson and ninja usage. Nonetheless, to integrate external dependencies it is not as easy as in gst-uninstalled.
This guide aims to show how to integrate GStreamer-VAAPI dependencies, in this case with the Intel VA-API driver.
Dependencies
For now we will need meson master, since it this patch is required. The pull request is already merged but it is unreleased yet.
Installation
clone gst-build repository
$ git clone git://anongit.freedesktop.org/gstreamer/gst-build
$ cd gst-build
apply custom patch for gst-build
The patch will add the repositories for libva and intel-vaapi-driver.
$ wget https://people.igalia.com/vjaquez/gst-build-vaapi/0001-add-libva-and-intel-vaapi-driver-as-subprojects.patch
$ git am 0001-add-libva-and-intel-vaapi-driver-as-subprojects.patch
Running this command, all dependency repositories will be cloned, symbolic links created, and the build directory configured.
$ meson bulid
apply custom patches for libva, intel-vaapi-driver and gstreamer-vaapi
libva
This patch is required since the headers files uninstalled paths doesn’t match with the ones in the “include” directives.
$ cd libva
$ wget https://people.igalia.com/vjaquez/gst-build-vaapi/0001-build-add-headers-for-uninstalled-setup.patch
$ git am 0001-build-add-headers-for-uninstalled-setup.patch
$ cd -
The patch handles libva dependency as a subproject.
$ cd intel-vaapi-driver
$ wget https://people.igalia.com/vjaquez/gst-build-vaapi/0001-meson-support-libva-as-subproject.patch
$ git am 0001-meson-support-libva-as-subproject.patch
$ cd -
Note to myself: this patch must be split and merged in upstream.
$ cd gstreamer-vaapi
$ wget https://people.igalia.com/vjaquez/gst-build-vaapi/0001-build-meson-libva-gst-uninstall-friendly.patch
$ git am 0001-build-meson-libva-gst-uninstall-friendly.patch
$ cd -
For some time now I have been working on a personal project to render the well known Sponza model provided by Crytek using Vulkan. Here is a picture of the current (still a work-in-progress) result:
Sponza rendering
This screenshot was captured on my Intel Kabylake laptop, running on the Intel Mesa Vulkan driver (Anvil).
The following list includes the main features implemented in the demo:
Depth pre-pass
Forward and deferred rendering paths
Anisotropic filtering
Shadow mapping with Percentage-Closer Filtering
Bump mapping
Screen Space Ambient Occlusion (only on the deferred path)
Screen Space Reflections (only on the deferred path)
Tone mapping
Anti-aliasing (FXAA)
I have been thinking about writing post about this for some time, but given that there are multiple features involved I wasn’t sure how to scope it. Eventually I decided to write a “frame analysis” post where I describe, step by step, all the render passes involved in the production of the single frame capture showed at the top of the post. I always enjoyed reading this kind of articles so I figured it would be fun to write one myself and I hope others find it informative, if not entertaining.
To avoid making the post too dense I won’t go into too much detail while describing each render pass, so don’t expect me to go into the nitty-gritty of how I implemented Screen Space Ambient Occlussion for example. Instead I intend to give a high-level overview of how the various features implemented in the demo work together to create the final result. I will provide screenshots so that readers can appreciate the outputs of each step and verify how detail and quality build up over time as we include more features in the pipeline. Those who are more interested in the programming details of particular features can always have a look at the Vulkan source code (link available at the bottom of the article), look for specific tutorials available on the Internet or wait for me to write feature-specifc posts (I don’t make any promises though!).
If you’re interested in going through with this then grab a cup of coffe and get ready, it is going to be a long ride!
Step 0: Culling
This is the only step in this discussion that runs on the CPU, and while optional from the point of view of the result (it doesn’t affect the actual result of the rendering), it is relevant from a performance point of view. Prior to rendering anything, in every frame, we usually want to cull meshes that are not visible to the camera. This can greatly help performance, even on a relatively simple scene such as this. This is of course more noticeable when the camera is looking in a direction in which a significant amount of geometry is not visible to it, but in general, there are always parts of the scene that are not visible to the camera, so culling is usually going to give you a performance bonus.
In large, complex scenes with tons of objects we probably want to use more sophisticated culling methods such as Quadtrees, but in this case, since the number of meshes is not too high (the Sponza model is slightly shy of 400 meshes), we just go though all of them and cull them individually against the camera’s frustum, which determines the area of the 3D space that is visible to the camera.
The way culling works is simple: for each mesh we compute an axis-aligned bounding box and we test that box for intersection with the camera’s frustum. If we can determine that the box never intersects, then the mesh enclosed within it is not visible and we flag it as such. Later on, at rendering time (or rather, at command recording time, since the demo has been written in Vulkan) we just skip the meshes that have been flagged.
The algorithm is not perfect, since it is possible that an axis-aligned bounding box for a particular mesh is visible to the camera and yet no part of the mesh itself is visible, but it should not affect a lot of meshes and trying to improve this would incur in additional checks that could undermine the efficiency of the process anyway.
Since in this particular demo we only have static geometry we only need to run the culling pass when the camera moves around, since otherwise the list of visible meshes doesn’t change. If dynamic geometry were present, we would need to at least cull dynamic geometry on every frame even if the camera stayed static, since dynamic elements may step in (or out of) the viewing frustum at any moment.
Step 1: Depth pre-pass
This is an optional stage, but it can help performance significantly in many cases. The idea is the following: our GPU performance is usually going to be limited by the fragment shader, and very specially so as we target higher resolutions. In this context, without a depth pre-pass, we are very likely going to execute the fragment shader for fragments that will not end up in the screen because they are occluded by fragments produced by other geometry in the scene that will be rasterized to the same XY screen-space coordinates but with a smaller Z coordinate (closer to the camera). This wastes precious GPU resources.
One way to improve the situation is to sort our geometry by distance from the camera and render front to back. With this we can get fragments that are rasterized from background geometry quickly discarded by early depth tests before the fragment shader runs for them. Unfortunately, although this will certainly help (assuming we can spare the extra CPU work to keep our geometry sorted for every frame), it won’t eliminate all the instances of the problem in the general case.
Also, some times things are more complicated, as the shading cost of different pieces of geometry can be very different and we should also take this into account. For example, we can have a very large piece of geometry for which some pixels are very close to the camera while some others are very far away and that has a very expensive shader. If our renderer is doing front-to-back rendering without any other considerations it will likely render this geometry early (since parts of it are very close to the camera), which means that it will shade all or most of its very expensive fragments. However, if the renderer accounts for the relative cost of the shader execution it would probably postpone rendering it as much as possible, so by the time it actually renders it, it takes advantage of early fragment depth tests to avoid as many of its expensive fragment shader executions as possible.
Using a depth-prepass ensures that we only run our fragment shader for visible fragments, and only those, no matter the situation. The downside is that we have to execute a separate rendering pass where we render our geometry to the depth buffer so that we can identify the visible fragments. This pass is usually very fast though, since we don’t even need a fragment shader and we are only writing to a depth texture. The exception to this rule is geometry that has opacity information, such as opacity textures, in which case we need to run a cheap fragment shader to identify transparent pixels and discard them so they don’t hit the depth buffer. In the Sponza model we need to do that for the flowers or the vines on the columns for example.
Depth pre-pass output
The picture shows the output of the depth pre-pass. Darker colors mean smaller distance from the camera. That’s why the picture gets brighter as we move further away.
Now, the remaining passes will be able to use this information to limit their shading to fragments that, for a given XY screen-space position, match exactly the Z value stored in the depth buffer, effectively selecting only the fragments that will be visible in the screen. We do this by configuring the depth test to do an EQUAL test instead of the usual LESS test, which is what we use in the depth-prepass.
In this particular demo, running on my Intel GPU, the depth pre-pass is by far the cheapest of all the GPU passes and it definitely pays off in terms of overall performance output.
Step 2: Shadow map
In this demo we have single source of light produced by a directional light that simulates the sun. You can probably guess the direction of the light by checking out the picture at the top of this post and looking at the direction projected shadows.
I already covered how shadow mapping works in previous series of posts, so if you’re interested in the programming details I encourage you to read that. Anyway, the basic idea is that we want to capture the scene from the point of view of the light source (to be more precise, we want to capture the objects in the scene that can potentially produce shadows that are visible to our camera).
With that information, we will be able to inform out lighting pass so it can tell if a particular fragment is in the shadows (not visible from our light’s perspective) or in the light (visible from our light’s perspective) and shade it accordingly.
From a technical point of view, recording a shadow map is exactly the same as the depth-prepass: we basically do a depth-only rendering and capture the result in a depth texture. The main differences here are that we need to render from the point of view of the light instead of our camera’s and that this being a directional light, we need to use an orthographic projection and adjust it properly so we capture all relevant shadow casters around the camera.
Shadow map
In the image above we can see the shadow map generated for this frame. Again, the brighter the color, the further away the fragment is from the light source. The bright white area outside the atrium building represents the part of the scene that is empty and thus ends with the maximum depth, which is what we use to clear the shadow map before rendering to it.
In this case, we are using a 4096×4096 texture to store the shadow map image, much larger than our rendering target. This is because shadow mapping from directional lights needs a lot of precision to produce good results, otherwise we end up with very pixelated / blocky shadows, more artifacts and even missing shadows for small geometry. To illustrate this better here is the same rendering of the Sponza model from the top of this post, but using a 1024×1024 shadow map (floor reflections are disabled, but that is irrelevant to shadow mapping):
Sponza rendering with 1024×1024 shadow map
You can see how in the 1024×1024 version there are some missing shadows for the vines on the columns and generally blurrier shadows (when not also slightly distorted) everywhere else.
Step 3: GBuffer
In deferred rendering we capture various attributes of the fragments produced by rasterizing our geometry and write them to separate textures that we will use to inform the lighting pass later on (and possibly other passes).
What we do here is to render our geometry normally, like we did in our depth-prepass, but this time, as we explained before, we configure the depth test to only pass fragments that match the contents of the depth-buffer that we produced in the depth-prepass, so we only process fragments that we now will be visible on the screen.
Deferred rendering uses multiple render targets to capture each of these attributes to a different texture for each rasterized fragment that passes the depth test. In this particular demo our GBuffer captures:
Normal vector
Diffuse color
Specular color
Position of the fragment from the point of view of the light (for shadow mapping)
It is important to be very careful when defining what we store in the GBuffer: since we are rendering to multiple screen-sized textures, this pass has serious bandwidth requirements and therefore, we should use texture formats that give us the range and precision we need with the smallest pixel size requirements and avoid storing information that we can get or compute efficiently through other means. This is particularly relevant for integrated GPUs that don’t have dedicated video memory (such as my Intel GPU).
In the demo, I do lighting in view-space (that is the coordinate space used takes the camera as its origin), so I need to work with positions and vectors in this coordinate space. One of the parameters we need for lighting is surface normals, which are conveniently stored in the GBuffer, but we will also need to know the view-space position of the fragments in the screen. To avoid storing the latter in the GBuffer we take advantage of the fact that we can reconstruct the view-space position of any fragment on the screen from its depth (which is stored in the depth buffer we rendered during the depth-prepass) and the camera’s projection matrix. I might cover the process in more detail in another post, for now, what is important to remember is that we don’t need to worry about storing fragment positions in the GBuffer and that saves us some bandwidth, helping performance.
Let’s have a look at the various GBuffer textures we produce in this stage:
Normal vectors
GBuffer normal texture
Here we see the normalized normal vectors for each fragment in view-space. This means they are expressed in a coordinate space in which our camera is at the origin and the positive Z direction is opposite to the camera’s view vector. Therefore, we see that surfaces pointing to the right of our camera are red (positive X), those pointing up are green (positive Y) and those pointing opposite to the camera’s view direction are blue (positive Z).
It should be mentioned that some of these surfaces use normal maps for bump mapping. These normal maps are textures that provide per-fragment normal information instead of the usual vertex normals that come with the polygon meshes. This means that instead of computing per-fragment normals as a simple interpolation of the per-vertex normals across the polygon faces, which gives us a rather flat result, we use a texture to adjust the normal for each fragment in the surface, which enables the lighting pass to render more nuanced surfaces that seem to have a lot more volume and detail than they would have otherwise.
For comparison, here is the GBuffer normal texture without bump mapping enabled. The difference in surface detail should be obvious. Just look at the lion figure at the far end or the columns and and you will immediately notice the addditional detail added with bump mapping to the surface descriptions:
GBuffer normal texture (bump mapping disabled)
To make the impact of the bump mapping more obvious, here is a different shot of the final rendering focusing on the columns of the upper floor of the atrium, with and without bump mapping:
Bump mapping enabled
Bump mapping disabled
All the extra detail in the columns is the sole result of the bump mapping technique.
Diffuse color
GBuffer diffuse texture
Here we have the diffuse color of each fragment in the scene. This is basically how our scene would look like if we didn’t implement a lighting pass that considers how the light source interacts with the scene.
Naturally, we will use this information in the lighting pass to modulate the color output based on the light interaction with each fragment.
Specular color
GBuffer specular texture
This is similar to the diffuse texture, but here we are storing the color (and strength) used to compute specular reflections.
Similarly to normal textures, we use specular maps to obtain per-fragment specular colors and intensities. This allows us to simulate combinations of more complex materials in the same mesh by specifying different specular properties for each fragment.
For example, if we look at the cloths that hang from the upper floor of the atrium, we see that they are mostly black, meaning that they barely produce any specular reflection, as it is to be expected from textile materials. However, we also see that these same cloths have an embroidery that has specular reflection (showing up as a light gray color), which means these details in the texture have stronger specular reflections than its surrounding textile material:
Specular reflection on cloth embroidery
The image shows visible specular reflections in the yellow embroidery decorations of the cloth (on the bottom-left) that are not present in the textile segment (the blue region of the cloth).
Fragment positions from Light
GBuffer light-space position texture
Finally, we store fragment positions in the coordinate space of the light source so we can implement shadows in the lighting pass. This image may be less intuitive to interpret, since it is encoding space positions from the point of view of the sun rather than physical properties of the fragments. We will need to retrieve this information for each fragment during the lighting pass so that we can tell, together with the shadow map, which fragments are visible from the light source (and therefore are directly lit by the sun) and which are not (and therefore are in the shadows). Again, more detail on how that process works, step by step and including Vulkan source code in my series of posts on that topic.
Step 4: Screen Space Ambient Occlusion
With the information stored in the GBuffer we can now also run a screen-space ambient occlusion pass that we will use to improve our lighting pass later on.
The idea here, as I discussed in my lighting and shadows series, the Phong lighting model simplifies ambient lighting by making it constant across the scene. As a consequence of this, lighting in areas that are not directly lit by a light source look rather flat, as we can see in this image:
SSAO disabled
Screen-space Ambient Occlusion is a technique that gathers information about the amount of ambient light occlusion produced by nearby geometry as a way to better estimate the ambient light term of the lighting equations. We can then use that information in our lighting pass to modulate ambient light accordingly, which can greatly improve the sense of depth and volume in the scene, specially in areas that are not directly lit:
SSAO enabled
Comparing the images above should illustrate the benefits of the SSAO technique. For example, look at the folds in the blue curtains on the right side of the images, without SSAO, we barely see them because the lighting is too flat across all the pixels in the curtain. Similarly, thanks to SSAO we can create shadowed areas from ambient light alone, as we can see behind the cloths that hang from the upper floor of the atrium or behind the vines on the columns.
To produce this result, the output of the SSAO pass is a texture with ambient light intensity information that looks like this (after some blur post-processing to eliminate noise artifacts):
SSAO output texture
In that image, white tones represent strong light intensity and black tones represent low light intensity produced by occlusion from nearby geometry. In our lighting pass we will source from this texture to obtain per-fragment ambient occlusion information and modulate the ambient term accordingly, bringing the additional volume showcased in the image above to the final rendering.
Step 6: Lighting pass
Finally, we get to the lighting pass. Most of what we showcased above was preparation work for this.
The lighting pass mostly goes as I described in my lighting and shadows series, only that since we are doing deferred rendering we get our per-fragment lighting inputs by reading from the GBuffer textures instead of getting them from the vertex shader.
Basically, the process involves retrieving diffuse, ambient and specular color information from the GBuffer and use it as input for the lighting equations to produce the final color for each fragment. We also sample from the shadow map to decide which pixels are in the shadows, in which case we remove their diffuse and specular components, making them darker and producing shadows in the image as a result.
We also use the SSAO output to improve the ambient light term as described before, multipliying the ambient term of each fragment by the SSAO value we computed for it, reducing the strength of the ambient light for pixels that are surrounded by nearby geometry.
The lighting pass is also where we put bump mapping to use. Bump mapping provides more detailed information about surface normals, which the lighting pass uses to simulate more complex lighting interactions with mesh surfaces, producing significantly enhanced results, as I showcased earlier in this post.
After combining all this information, the lighting pass produces an output like this. Compare it with the GBuffer diffuse texture to see all the stuff that this pass is putting together:
Lighting pass output
Step 7: Tone mapping
After the lighting pass we run a number of post-processing passes, of which tone mapping is the first one. The idea behind tone mapping is this: normally, shader color outputs are limited to the range [0, 1], which puts a hard cap on our lighting calculations. Specifically, it means that when our light contributions to a particular pixel go beyond 1.0 in any color component, they get clamped, which can distort the resulting color in unrealistic ways, specially when this happens during intermediate lighting calculations (since the deviation from the physically correct color is then used as input to more computations, which then build on that error).
To work around this we do our lighting calculations in High Dynamic Range (HDR) which allows us to produce color values with components larger than 1.0, and then we run a tone mapping pass to re-map the result to the [0, 1] range when we are done with the lighting calculations and we are ready for display.
The nice thing about tone mapping is that it gives the developer control over how that mapping happens, allowing us to decide if we are interested in preserving more detail in the darker or brighter areas of the scene.
In this particular demo, I used HDR rendering to ramp up the intensity of the sun light beyond what I could have represented otherwise. Without tone mapping this would lead to unrealistic lighting in areas with strong light reflections, since would exceed the 1.0 per-color-component cap and lead to pure white colors as result, losing the color detail from the original textures. This effect can be observed in the following pictures if you look at the lit area of the floor. Notice how the tone-mapped picture better retains the detail of the floor texture while in the non tone-mapped version the floor seems to be over-exposed to light and large parts of it just become white as a result (shadow mapping has been disabled to better showcase the effects of tone-mapping on the floor):
Tone mapping disabled
Tone mapping enabled
Step 8: Screen Space Reflections (SSR)
The material used to render the floor is reflective, which means that we can see the reflections of the surrounding environment on it.
There are various ways to capture reflections, each with their own set of pros and cons. When I implemented my OpenGL terrain rendering demo I implemented water reflections using “Planar Reflections”, which produce very accurate results at the expense of requiring to re-render the scene with the camera facing in the same direction as the reflection. Although this can be done at a lower resolution, it is still quite expensive and cumbersome to setup (for example, you would need to run an additional culling pass), and you also need to consider that we need to do this for each planar surface you want to apply reflections on, so it doesn’t scale very well. In this demo, although it is not visible in the reference screenshot, I am capturing reflections from the floor sections of both stories of the atrium, so the Planar Reflections approach might have required me to render twice when fragments of both sections are visible (admittedly, not very often, but not impossible with the free camera).
So in this particular case I decided to experiment with a different technique that has become quite popular, despite its many shortcomings, because it is a lot faster: Screen Space Reflections.
As all screen-space techniques, the technique uses information already present in the screen to capture the reflection information, so we don’t have to render again from a different perspective. This leads to a number of limitations that can produce fairly visible artifacts, specially when there is dynamic geometry involved. Nevertheless, in my particular case I don’t have any dynamic geometry, at least not yet, so while the artifacts are there they are not quite as distracting. I won’t go into the details of the artifacts introduced with SSR here, but for those interested, here is a good discussion.
I should mention that my take on this is fairly basic and doesn’t implement relevant features such as the Hierarchical Z Buffer optimization (HZB) discussed here.
The technique has 3 steps: capturing reflections, applying roughness material properties and alpha blending:
Capturing reflections
I only implemented support for SSR in the deferred path, since like in the case of SSAO (and more generally all screen-space algorithms), deferred rendering is the best match since we are already capturing screen-space information in the GBuffer.
The first stage for this requires to have means to identify fragments that need reflection information. In our case, the floor fragments. What I did for this is to capture the reflectiveness of the material of each fragment in the screen during the GBuffer pass. This is a single floating-point component (in the 0-1 range). A value of 0 means that the material is not reflective and the SSR pass will just ignore it. A value of 1 means that the fragment is 100% reflective, so its color value will be solely the reflection color. Values in between allow us to control the strength of the reflection for each fragment with a reflective material in the scene.
One small note on the GBuffer storage: because this is a single floating-point value, we don’t necessarily need an extra attachment in the GBuffer (which would have some performance penalty), instead we can just put this in the alpha component of the diffuse color, since we were not using it (the Intel Mesa driver doesn’t support rendering to RGB textures yet, so since we are limited to RGBA we might as well put it to good use).
Besides capturing which fragments are reflective, we can also store another piece of information relevant to the reflection computations: the material’s roughness. This is another scalar value indicating how much blurring we want to apply to the resulting reflection: smooth metal-like surfaces can have very sharp reflections but with rougher materials that have not smooth surfaces we may want the reflections to look a bit blurry, to better represent these imperfections.
Besides the reflection and roughness information, to capture screen-space reflections we will need access to the output of the previous pass (tone mapping) from which we will retrieve the color information of our reflection points, the normals that we stored in the GBuffer (to compute reflection directions for each fragment in the floor sections) and the depth buffer (from the depth-prepass), so we can check for reflection collisions.
The technique goes like this: for each fragment that is reflective, we compute the direction of the reflection using its normal (from the GBuffer) and the view vector (from the camera and the fragment position). Once we have this direction, we execute a ray marching from the fragment position, in the direction of the reflection. For each point we generate, we take the screen-space X and Y coordinates and use them to retrieve the Z-buffer depth for that pixel in the scene. If the depth buffer value is smaller than our sample’s it means that we have moved past foreground geometry and we stop the process. If we got to this point, then we can do a binary search to pin-point the exact location where the collision with the foreground geometry happens, which will give us the screen-space X and Y coordinates of the reflection point. Once we have that we only need to sample the original scene (the output from the tone mapping pass) at that location to retrieve the reflection color.
As discussed earlier, the technique has numerous caveats, which we need to address in one way or another and maybe adapt to the characteristics of different scenes so we can obtain the best results in each case.
The output of this pass is a color texture where we store the reflection colors for each fragment that has a reflective material:
Reflection texture
Naturally, the image above only shows reflection data for the pixels in the floor, since those are the only ones with a reflective material attached. It is immediately obvious that some pixels lack reflection color though, this is due to the various limitations of the screen-space technique that are discussed in the blog post I linked above.
Because the reflections will be alpha-blended with the original image, we use the reflectiveness that we stored in the GBuffer as the base for the alpha component of the reflection color as well (there are other aspects that can contribute to the alpha component too, but I won’t go into that here), so the image above, although not visible in the screenshot, has a valid alpha channel.
Considering material roughness
Once we have captured the reflection image, the next step is to apply the material roughness settings. We can accomplish this with a simple box filter based on the roughness of each fragment: the larger the roughness, the larger the box filter we apply and the blurrier the reflection we get as a result. Because we store roughness for each fragment in the GBuffer, we can have multiple reflective materials with different roughness settings if we want. In this case, we just have one material for the floor though.
Alpha blending
Finally, we use alpha blending to incorporate the reflection onto the original image (the output from the tone mapping) ot incorporate the reflections to the final rendering:
SSR output
Step 9: Anti-aliasing (FXAA)
So far we have been neglecting anti-aliasing. Because we are doing deferred rendering Multi-Sample Anti-Aliasing (MSAA) is not an option: MSAA happens at rasterization time, which in a deferred renderer occurs before our lighting pass (specifically, when we generate the GBuffer), so it cannot account for the important effects that the lighting pass has on the resulting image, and therefore, on the eventual aliasing that we need to correct. This is why deferred renderers usually do anti-aliasing via post-processing.
In this demo I have implemented a well-known anti-aliasing post-processing pass known as Fast Approximate Anti Aliasing (FXAA). The technique attempts to identify strong contrast across neighboring pixels in the image to identify edges and then smooth them out using linear filtering. Here is the final result which matches the one I included as reference at the top of this post:
Anti-aliased output
The image above shows the results of the anti-aliasing pass. Compare that with the output of the SSR pass. You can see how this pass has effectively removed the jaggies observed in the cloths hanging from the upper floor for example.
Unlike MSAA, which acts on geometry edges only, FXAA works on all pixels, so it can also smooth out edges produced by shaders or textures. Whether that is something we want to do or not may depend on the scene. Here we can see this happening on the foreground column on the left, where some of the imperfections of the stone are slightly smoothed out by the FXAA pass.
Conclusions and source code
So that’s all, congratulations if you managed to read this far! In the past I have found articles that did frame analysis like this quite interesting so it’s been fun writing one myself and I only hope that this was interesting to someone else.
This demo has been implemented in Vulkan and includes a number of configurable parameters that can be used to tweak performance and quality. The work-in-progress source code is available here, but beware that I have only tested this on Intel, since that is the only hardware I have available, so you may find issues if you run this on other GPUs. If that happens, let me know in the comments and I might be able to provide fixes at some point.
I’ve devoted some of my time at Igalia to get a newer version of Chromium running on the GENIVI Development Platform (GDP).
Since the last update, there have been news regarding Wayland support in Chromium. My colleagues Antonio, Maksim and Frédéric have worked on a new Wayland backend following modern Chromium architecture. You can find more information in their own blogs and talks. I’m linking the most recent talk, from FOSDEM 2018.
Everyone can already try the new, Igalia-developed backend on their embedded devices using the meta-browser layer. I built it along with the GDP but discovered that it cannot run as it is, due to the lack of ivi-shell hooks in the new Chromium backend. This is going to be fixed in the mid-term, so I decided not to spend a lot of time researching this and chose a different solution for the current GDP release.
The LG SVL team recently announced the release of an updated Ozone Wayland backend for Chromium, based on the legacy implementation provided by Intel, as a part of the webOS OSE project. This is an update on the backend we were already running on the GDP, so it looked like a good idea to reuse their work.
I added the meta-lgsvl-browser layer to the GDP, which provides recipes for several Chromium flavors: chromium-lge is the one that builds upon the legacy Wayland backend and currently provides Chromium version 64.
The chromium-lge browser worked out-of-the-box on Raspberry Pi, but I faced some trouble with the other supported platforms. In the case of ARM64 platforms, we were finding a “relocation overflow” problem. This is something that my colleagues had already detected when trying the new Wayland backend on the R-Car gen. 3 platform, and it can be fixed by enabling compiler optimization flags for binary size.
In the case of Intel platforms, compilation failed due to a build-system assertion. It looks like Clang’s Control Flow Integrity feature is enabled by default on x64 Linux builds, but we aren’t using the Clang compiler. The solution consists just in disabling this feature, like the upstream meta-browser project was already doing.
The ongoing work is shared in this pull request. I hope to be able to make it for the next GDP release!
Last week I was in Berlin for the CSS Working Group (CSSWG) face-to-face meeting
representing Igalia, member of the CSSWG since last year.
Igalia has been working on the open web platform for many years,
where we help our customers with the implementation of different standards
on the open source web engines.
Inside the CSSWG we play the implementors role, providing valuable feedback
around the specifications we’re working on.
It was really nice to meet all the folks from the CSSWG there,
it’s amazing to be together with such a brilliant group of people in the same room.
And it’s lovely to see how easy is to talk with any of them, you all rock!
This is a brief post about my highlights from there, of course totally subjective and focused
on the topics I’m more interested.
CSS Grid Layout
We were discussing twoissues
of the current specification related to
the track sizing algorithm and its behavior in particular cases.
Some changes will be added in the specification to try to improve them
and we’ll need to update the implementations accordingly.
On top of that, we discussed about the Level 2 of the spec.
It’s already defined that this next level will include the following features:
The awaited subgrids feature:
There was the possibility of allowing subgrids in both axis (dual-axis) or only in one of them (per-axis),
note that the per-axis approach covers the dual-axis if you define the subgrid in both axis.
There are clear uses cases for the per-axis approach but the main doubt was
about how hard it’d be to implement.
Mats Palmgren from Mozilla posted a comment on the issue
explaining that he has just created a prototype for the feature following the per-axis idea,
so the CSSWG resolved to remove the dual-axis one from the spec.
And aspect-ratio controlled gutters:
Regarding this topic, the CSSWG decided to add a new ar unit.
We didn’t discuss anything more but we need to decide what we’ll do in the situations
where there’s no enough free space to fulfill the requested aspect-ratio,
should we ignore it or overflow in that case?
Just to add some context, the CSSWG test suites are now part of the web-platform-tests (WPT) repository.
This repository is being used by most browser vendors to share tests, including tests for new CSS features.
For example, at Igalia we’re currently using WPT test suites in all our developments.
The CSSWG uses the CSS Test Harness tool which has a build system
that adds some special requirements for the test suites.
One of them causes that we need to duplicate some files in the repository, which is not nice at all.
Several people in the CSSWG still rely on this tool mainly for two things:
Run manual tests and store their results:
Some CSS features like media queries or scrolling are hard to automate when writing tests,
so several specs have manual tests.
Probably WebDriver can help to automate this kind of tests, maybe not all though.
Extract status reports:
To verify that a spec fulfills the CR exit criteria, the current tooling has some nice reports,
it also provides info about the test coverage of the spec.
So we cannot get rid of the CSS Test Harness system at this point. We discussed about possible solutions
but none of them were really clear, also note that the lack of funding for this kind of work
makes it harder to move things forward.
I still believe the way to go would be to improve the WPT Dashboard (wpt.fyi)
so it can support the 2 features listed above.
If that’s the case maybe the specific CSS Test Harness stuff won’t be needed anymore,
thus the weird requirements for people working on the test suites will be gone,
and there would be a single tool for all the tests from the different working groups.
As a side note wpt.fyi needs some infrastructure improvements,
for example Microfost was not happy as Ahem font
(which is used a lot in CSS tests suites) is still not installed on the Windows virtual machines
that extract test results for wpt.fyi.
Floats, floats, floats
People are using floats to simulate CSS Shapes on browsers
that don’t have support yet.
That is causing that some special cases related to floats happen more frecuently,
and it’s hard to decide what’s the best thing to do on them.
The CSSWG was discussing what would be the best solution
when the non-floated content doesn’t fit in the space left by the floated elements.
The problem is quite complex to explain, but imagine the following picture where you have several floated elements.
An example of float layout
In this example there are a few floated elements restricting the area where the content can be painted,
if the browser needs to find the place to add a BFC (like a table)
it needs to decide where to place it avoiding overlapping any other floats.
There was a long discussion, and it seems the best choice would be that the browser tests all the options
and if there’s no overlapping then puts the table there (basically Option 1 in the linked illustration).
Still there are concerns about performance, so there’s still more work to be done here.
As a result of this discussion a new CSS Floats specification will be created
to describe the expected behavior in this kind of scenarios.
The members of the CSSWG were invited to the co-located TYPO Labs event.
I attended on Friday when Elika (fantasai), Myles and Rossen
gave a talk.
It was nice to see that CSS Grid Layout was mentioned in the first talk of the day, as an useful tool for typographers.
Variable fonts and Virtual Reality were clearly hot topics in several talks.
Elika (fantasai), Rossen and Myles in the CSSWG talk at TYPO Labs
It’s funny that the last time I was in Berlin was 10 years ago for a conference related to TYPO3,
totally unrelated but with a similar name. 😄
Other
Jihye Hong and Florian Rivoal were presenting the Spatial Navigation spec.
It seems LGE is quite interested in moving this forward,
but the feeling from the CSSWG discussion is that the spec still needs more work to get consolidated.
CSS Object Model (CSSOM) spec has a new and shiny editor,
Emilio Cobos is taking over it.
I’m sure this will help to unblock some of the oldissues
we’re waiting for before fixing some Grid Layout related stuff in Blink and WebKit.
Some pictures of Berlin
And that’s mostly all that I can remember now, I’m sure I’m missing many other important things.
It was a fantastic week and I even find some time for walking around Berlin as the weather was really pleasant.
Ubucon is a conference for users and developers of Ubuntu, one of the most popular GNU/Linux distributions in the world. The conference is full of talks covering very different topics, and optional activities for attendes like the traditional espicha.
I will attend as speaker to this conference with my talk “Introduction to Mesa, an open-source graphics API”. In this talk, I will give a brief introduction to Mesa, how it works and how to contribute to it. My talk will be on Sunday 29th April at 11:25am. Don’t miss it!
If you plan to attend the conference and you want to talk about open-source graphics drivers, Linux graphics stack, OpenGL/Vulkan or something alike, please drop me a line in my twitter.
My talk Upstream consultancy and Ceph RadosGW/S3 covered the context and value of the upstream contributions in the Ceph project, along with some examples of consulting and technical work that we carried out in Igalia ending with new features and improvements in the project.
If you could not attend and are interested in the topics we talked about, you can read more about the event here.
Some of the topics on the agenda that we discussed were hardware performance, reference architectures, open web-scale networking, operations, new use cases or storage products/services.
Outside of the agenda we also discussed the technical impact of BlueStore, the new storage backend in Luminous, the low-risk approaches to update clusters in production or how small and medium-sized companies are gradually adopting software-defined storage.
The latter allowed some of the attendees to follow the technical section of the talks in real time and experiment with configurations, protocols and tools throughout the event.
Finally thank all the people who participated in the organization and actively collaborated to make the event possible. See you at the next one!
Update 2018/05/14
The slides are now available through the Ceph community:
Early this year I was working on unprefixing the CSS Grid Layout gutter properties.
The properties were originally named grid-column-gap and grid-row-gap,
together with the grid-gap shorthand.
The CSS Working Group (CSSWG) decided to remove the grid- prefix
from these properties last summer, so they could be extended to be used
in other layout models like Flexbox.
I was not planning to write a blog post about this, but the task ended up
becoming something more than just renaming the properties,
so this post describes what it took to implement this.
Also people got quite excited about the possibility of animating grid gutters
when I announced that this was ready on Twitter.
So the theory seems pretty simply, we currently have 3 properties
with the grid- prefix and we want to remove it:
grid-column-gap becomes column-gap,
grid-row-gap becomes row-gap and
grid-gap becomes gap.
But column-gap is already an existent property,
defined by the Multicolumn spec,
which has been around for a long time.
So we cannot just create a new property, but we have to make it work
also for Grid Layout, and be sure that the syntax is equivalent.
Animatable properties
When I started to test Multicol column-gap I realized it was animatable,
however our implementations (Blink and WebKit)
of the Grid Layout gutter properties were not.
We’d need to make our properties animatable if we want to remove the prefixes.
Making the properties animatable is not complex at all, both Blink and WebKit
have everything ready to make this task easy for properties like the gutter ones
that represent lengths.
So I decided to do this as part of the unprefixing patch,
instead of something separated.
CSS Grid Layout gutters animation example (check it live)
Percentages
But there was something else, the Grid gutter properties
accept percentage values, however column-gap hadn’t that support yet.
So I added percentage support to column-gap for multicolumn,
as a preliminary patch for the unprefixing one.
There has been long discussions in the CSSWG about how to resolve percentages
on gutter properties.
The spec has recently changed so these properties
should be resolved to zero for content-based containers.
However my patch is not implementing that, as we don’t believe
there’s an easy way to support something like that in most of the web engines,
and Blink and WebKit are not exceptions.
Our patch follows what Microsoft Edge does in these cases,
and resolves the percentage gaps like it does for percentage widths or heights.
And the Firefox implementation that has just landed this week does the same.
CSS Multi-column percentage column-gap example (check it live)
I guess we’ll still have some extra discussions about this topic in the CSSWG,
but percentages themselves deserve their own blog post.
Implementation
Once all the previous problems got solved, I landed the patches
related to unprefixing the gutter properties in both Blink
and WebKit.
So you can use the unprefixed version since Chrome 66.0.3341.0
and Safari Technology Preview 50.
A simple Grid Layout example using the unprefixed gutter properties
Note that as specified in the spec, the previous prefixed properties
are still valid and will be kept as an alias to avoid breaking existent content.
Also it’s important to notice that now the gap shorthand applies
to Multicol containers too, as it sets the value of column-gap longhand
(together with row-gap which would be ignored by Multicol).
As usual in our last developments, we have been using
web-platform-tests repository
for all the tests related to this work.
As a result of this work we have now 16 new tests
that verify the support of these properties,
including tests for animations stuff too.
Running those tests on the different browsers, I realized there was
an inconsistency between css-align and css-multicol specifications.
Both specs define the column-gap property, but the computed value
was different.
I raised a CSSWG issue
that has been recently solved,
so that the computed value for column-gap: normal should still be normal.
This causes that the property won’t be animatable from normal to other values
as explained before.
This is the summary of the status of these tests in the main browser engines:
Blink and WebKit: They pass all the tests and follow last CSSWG resolution.
Edge: Unprefixed properties are available since version 41.
Percentage support is interoperable with Blink and WebKit.
The computed value of column-gap: normal is not normal there,
so this needs to get updated.
The task is completed and everything should be settled down at this point,
you can start using these unprefixed properties, and it seems that Firefox
will join the rest of browser by adding this support very soon.
Igalia and Bloomberg
working together to build a better web
Last, but not least, this is again part of the ongoing collaboration between
Igalia and
Bloomberg.
I don’t mind to repeat myself over and over, but it’s really worth
to highlight the support from Bloomberg in the CSS Grid Layout development,
they have been showing to the world that an external company can directly influence
in the new specifications from the standard bodies
and implementations by the browser vendors.
Thank you very much!
Finally and just as a heads-up, I’ll be in Berlin next week
for the CSSWG F2F meeting.
I’m sure we’ll have interesting conversations about CSS Grid Layout
and many other topics there.
As part of my work in the graphics team at Igalia, I’ve been helping with the effort to enable GL_ARB_gl_spirv for Intel’s i965 driver in Mesa. Most of the functionality of the extension is already working so in order to get more test coverage and discover unknown missing features we have been working to automatically convert some Piglit GLSL tests to use SPIR-V. Most of the GLSL tests are using a tool internal to Piglit called shader_runner. This tool largely simplifies the work needed to create a test by allowing it to be specified as a simple text file that just contains the required shaders and a list of commands to execute using them. The commands are very highlevel such as draw rect to submit a rectangle or probe rgba to check for a specific colour of a pixel in the framebuffer.
A number of times during this work I’ve encountered problems that look like they are general problems with Mesa’s SPIR-V compiler and aren’t specific to the GL_ARB_gl_spirv work. I wanted a way to easily confirm this but writing a minimal test case from scratch in Vulkan seemed like quite a lot of hassle. Therefore I decided it would be nice to have a tool like shader_runner that works with Vulkan. I made a start on implementing this and called it VkRunner.
Example
Here is an example shader test file for VkRunner:
[vertex shader]
#version 430
layout(location = 0) in vec3 pos;
void
main()
{
gl_Position = vec4(pos.xy * 2.0 - 1.0, 0.0, 1.0);
}
[fragment shader]
#version 430
layout(location = 0) out vec4 color_out;
layout(std140, push_constant) uniform block {
vec4 color_in;
};
void
main()
{
color_out = color_in;
}
[vertex data]
0/r32g32b32_sfloat
0.25 0.0775 0.0
0.145 0.3875 0.0
0.25 0.3175 0.0
0.25 0.0775 0.0
0.355 0.3875 0.0
0.25 0.3175 0.0
0.0775 0.195 0.0
0.4225 0.195 0.0
0.25 0.3175 0.0
[test]
# Clear the framebuffer to green
clear color 0.0 0.6 0.0 1.0
clear
# White rectangle in the topleft corner
uniform vec4 0 1.0 1.0 1.0 1.0
draw rect 0 0 0.5 0.5
# Green star in the topleft
uniform vec4 0 0.0 0.6 0.0 1.0
draw arrays TRIANGLE_LIST 0 9
# Verify a rectangle colour
relative probe rect rgb (0.5, 0.0, 0.5, 1.0) (0.0, 0.6, 0.0)
If this is run through VkRunner it will convert the shaders to SPIR-V on the fly by piping them through glslangValidator, create a pipeline and a framebuffer and then run the test commands. The framebuffer is just an offscreen buffer so VkRunner doesn’t use any window system extensions. However you can still see the result by passing the -i image.ppm option which cause it to write a PPM image of the final rendering.
The format is meant to be as close to shader_runner as possible but there are a few important differences. Vulkan can’t have uniforms in the default buffer and there is no way to access them by name. Instead you can use a push constant buffer and refer to the individual uniform using its byte offset in the buffer. VkRunner doesn’t yet support UBOs. The same goes for vertex attributes which are now specified using an explicit location rather than by name. In the vertex data section you can use names from VkFormat to specify the format with maximum flexibility, but it can also accept Piglit-style names for compatibilty.
Bonus features
VkRunner supports some extra features that shader_runner does not. Firstly you can specify a format for the framebuffer in the optional [require] section. For example you can make it use a floating-point framebuffer to be able to accurately probe results from functions.
If you want to use SPIR-V instead of GLSL in the source, you can specify the shader in a [fragment shader spirv] section. This is useful to test corner cases of the driver with tweaked shaders that glslang wouldn’t generate. You can get a base for the SPIR-V source by passing the -d option to VkRunner to get the disassembly of the generated SPIR-V. There is an example of this in the repo.
Status
Although it can already be used for a range of tests, VkRunner still has a lot of missing features compared to shader_runner. It does not support textures, UBOs and SSBOs. Take a look at the README for the complete documentation.
As I have been creating tests for problems that I encounter with Mesa I’ve been adding them to a branch called tests in the Github repo. I get the impression that Vulkan is somewhat under-tested compared to GL so it might be interesting to use these tests as a base to make a more complete Vulkan test suite. It could also potentially be merged into Piglit.
Last November I had a chance to dive into Intel Media SDK plugins in gst-plugins-bad. It was very good chance for me to learn how gstreamer elements are handling special memory with its own infrastructures like GstBufferPool and GstMemory. In this post I’m going to talk about the improvements of GStreamer Intel MSDK plugins in release 1.14, which is what I’ve been through since last November.
First of all, for those not familiar with Media SDK I’m going to explain what Media SDK is briefly.
Media SDK(Aka. MSDK) is the cross-platform API to access Intel’s hardware accelerated video encoder and decoder functions on Windows and Linux. You can get more information about MSDK here and here.
But on Linux so far, it’s hard to set up environment to make MSDK driver working.
If you want to set up development environment for MSDK and see what’s working on linux, you should follow the steps described in this page. But I would recommend you refer to the Victor’s post, which is very-well explained for this a little bit annoying stuff.
Additionally the driver in linux supports only Skylake as far as I know, which is very disappointing for users of other chipsets. I(and you probably) hope we can work on it without any specific patch/driver (since it is open source!) and any dependency of chipset. As far as I know, Intel has been considering this issue to be solved, so we’ll see what’s going to happen in the future.
Back to gstreamer, gstreamer plugins using MSDK landed in 2016.
At that time they were working fine with basic features for playback and encoding but there were several things to be improved, especially for performance.
Eventually, in the middle of last March, GStreamer 1.14 has been released including improvements of MSDK plugins, which is what I want to talk in this post.
So let me begin now.
Suuports bufferpool and starts using video memory.
This is a key feature that improves the preformance.
In MSDK, there are two types of memory supported in the driver.
One is “System Memory” and another is “Video Memory”. (There is one more type of memory, which is called “Opaque Memory” but won’t talk about it in this post)
System memory is a memory allocated on user space, which is normal.
System memory is being used in the plugins already, which is quite simple to use but not recommended since the performance is not good enough.
Video memory is a memory used by hardware acceleration device, also known as GPU, to hold frame and other types of video data.
For applications to use video memory, something specific on its platform should be implemented. On linux for example, we can use VA-API to handle video memory through MSDK.
And that’s included to the 1.14 release, which means we still need to implement something specific on Windows like this way.
To implement using video memory, I needed to implement GstMSDK(System/Video)Memory to generalize how to access and map the memory in the way of GStreamer. And GstMSDKBufferPool can allocate this type of memory and can be proposed to upstream and can be used in each MSDK element itself. There were lots of problems and argues during this work since the design of MSDK APIs and GStreamer infrastructure don’t match perfectly.
You can see discussion and patches here in this issue.
In addition, if using video memory on linux, we can use DMABuf by acquiring fd handle via VA-API at allocation time. Recently this has been done only for DMABuf export in this issue though it’s not included in 1.14 release.
Sharing context/session
For resource utilization, there needs to share MSDK session with each MSDK plugin in a pipeline.
A MSDK session maintains context for the use of any of decode,encode and convert(VPP) functions. Yes it’s just like a handle of the driver. One session can run exactly one of decode, encode and convert(VPP).
So let’s think about an usual transcoding pipeline. It should be like this, src - decoder - converter - encoder - sink. In this case we should use same MSDK session to decode, convert and encode. Otherwise we should copy the data from upstream to work with different session because sessions cannot share data, which should get much worse.
Also there’s one more thing. MSDK supports joining session. If application wants(or has) to use multiple sessions, it can join sessions to share data and we need to support it in the level of GStreamer as well.
All of these can be achieved by GstContext which provides a way of sharing not only between elements. You can see the patch in this issue, same as MSDK Bufferpool issue.
Adds vp8 and mpeg2 decoder.
Sree has added mpeg2/vc1 decoder and I have added vp8 decoder.
Supports a number of algorithms and tuning options in encoder
Encoders are exposing a number of rate control algorithms now and more encoder tuning options like trellis-quantiztion (h264), slice size control (h264), B-pyramid prediction(h264), MB-level bitrate control, frame partitioning and adaptive I/B frame insertion were added. The encoder now also handles force-key-unit events and can insert frame-packing SEIs for side-by-side and top-bottom stereoscopic 3D video.
All of this has been done by Sree and you can see the details in this issue
Capability of encoder’s sinkpad is improved by using VPP.
MSDK encoders had accepted only NV12 raw data since MSDK encoder supports only NV12 format. But other formats can be handled too if we convert them to NV12 by using VPP, which is also supported in the MSDK driver.
This has been done by slomo and I fixed a bug related to it. See this bug for more details.
You can find all of patches for MSDK plugins here.
As I said in the top of this post, all MSDK stuffs should be opened first and should support some of major Intel chipsets at least even if not all.
But now, the one thing that I can say is GStreamer MSDK plugins are being improved continuously.
We can see what’s happening in the near future.
Finally I want to say that Sree and Victor helped me a lot as a reviewer and an adviser with this work. I really appreciate it.
GStreamer VA-API is not a trivial piece of software. Even though, in my opinion it is a bit over-engineered, the complexity relies on its layered architecture: the user must troubleshoot in which layer is the failure.
So, bear in mind this architecture:
GStreamer VA-API is not a trivial piece of software. Even though, in my opinion it is a bit over-engineered, the complexity relies on its layered architecture: the user must troubleshoot in which layer is the failure.
So, bear in mind this architecture:
libva architecture
And the point of failure could be anywhere.
Drivers
libva is a library designed to load another library called driver or back-end. This driver is responsible to talk with the kernel, windowing platform, memory handling library, or any other piece of software or hardware that actually will do the video processing.
There are many drivers in the wild. As it is an API aiming to stateless video processing, and the industry is moving towards that way to process video, it is expected more drivers would appear in the future.
Nonetheless, not all the drivers have the same level of maturity, and some of them are abandon-ware. For this reason we decided in GStreamer VA-API, some time ago, to add a white list of functional drivers, basically, those developed by Mesa3D and this one from Intel. If you wish to disable that white-list, you can do it by setting an environment variable:
$ export GST_VAAPI_ALL_DRIVERS=1
Remember, if you set it, you are on your own, since we do not trust on the maturity of that driver yet.
Internal libvadriver version
Thus, there is an internal API between libva and the driver and it is versioned, meaning that the internal API version of the installed libva library must match with the internal API exposed by the driver. One of the causes that libva could not initialize a driver could be because the internal API version does not match.
Drivers path and driver name
By default there is a path where libva looks for drivers to load. That path is defined at compilation time. Following Debian’s file-system hierarchy standard (FHS) it should be set by distributions in /usr/lib/x86_64-linux-gnu/dri/. But the user can control this path with an environment variable:
The driver path, as a directory, might contain several drivers. libva will try to guess the correct one by querying the instantiated VA display (which could be either KMS/DRM, Wayland, Android or X11). If the user instantiates a VA display different of his running environment, the guess will be erroneous, the library loading will fail.
Although, there is a way for the user to set the driver’s name too. Again, by setting an environment variable:
$ export LIBVA_DRIVER_NAME=iHD
With this setting, libva will try to load iHD_drv_video.so (a new and experimental open source driver from Intel, targeted for MediaSDK —do not use it yet with GStreamer VAAPI—).
vainfo
vainfo is the diagnostic tool for VA-API. In a couple words, it will iterate on a list of VA displays, in try-and-error strategy, and try to initialize VA. In case of success, vainfo will report the driver signature, and it will query the driver for the available profiles and entry-points.
For example, my skylake board for development will report
Does this mean that VA-API processes video? No. It means that there is an usable VA display which could open a driver correctly and libva can extract symbols from it.
I would like to mention another tool, not official, but I like it a lot, since it extracts almost of the VA information available in the driver: vadumpcaps.c, written by Mark Thompson.
GStreamer VA-API registration
When GStreamer is launched, normally it will register all the available plugins and plugin features (elements, device providers, etc.). All that data is cache and keep until the cache file is deleted or the cache invalidated by some event.
At registration time, GStreamer VA-API will instantiate a DRM-based VA display, which works with no need of a real display (in other words, headless), and will query the driver for the profiles and entry-points tuples, in order to register only the available elements (encoders, decoders. sink, post-processor). If the DRM VA display fails, a list of VA displays will be tried.
In the case that libva could not load any driver, or the driver is not in the white-list, GStreamer VA-API will not register any element. Otherwise gst-inspect-1.0 will show the registered elements:
Beside the normal behavior, GStreamer VA-API will also invalidate GStreamer’s cache at every boot, or when any of the mentioned environment variables change.
Conclusion
A simple task list to review when GStreamer VA-API is not working at all is this:
#. Check your LIBVA_* environment variables
#. Verify that vainfo returns sensible information
#. Invalidate GStreamer’s cache (or just delete the file)
#. Check the output of gst-inspect-1.0 vaapi
And, if you decide to file a bug in bugzilla, please do not forget to attach the output of vainfo and the logs if the developer asks for them.
As you may already know, there is a new release of GStreamer, 1.14. In this blog post we will talk about the new features and improvements of GStreamer VA-API module, though you have a more comprehensive list of changes in the release notes.
Most of the topics explained along this blog post are already mentioned in the release notes, but a bit more detailed.
DMABuf usage
We have improved DMA-buf’s usage, mostly at downstream.
In the case of upstream, we just got rid a nasty hack which detected when to instantiate and use a buffer pool in sink pad with a dma-buf based allocator. This functionality has been already broken for a while, and that code was the wrong way to enabled it. The sharing of a dma-buf based buffer pool to upstream is going to be re-enabled after bug 792034 is merged.
For downstream, we have added the handling of memory:DMABuf caps feature. The purpose of this caps feature is to negotiate a media when the buffers are not map-able onto user space, because of digital rights or platform restrictions.
For example, currently intel-vaapi-driver doesn’t allow the mapping of its produced dma-buf descriptors. But, as we cannot know if a back-end produces or not map-able dma-buf descriptors, gstreamer-vaapi, when the allocator is instantiated, creates a dummy buffer and tries to map it, if it fails, memory:DMABuf caps feature is negotiated, otherwise, normal video caps are used.
VA-API usage
First of all, GStreamer VA-API has support now for libva-2.0, this means VA-API 1.10. We had to guard some deprecated symbols and the new ones. Nowadays most of distributions have upgraded to libva-2.0.
We have improved the initialization of the VA display internal structure (GstVaapiDisplay). Previously, if a X based display was instantiated, immediately it tried to grab the screen resolution. Obviously, this broke the usage of headless systems. We just delay the screen resolution check to when the VA display truly requires that information.
New API were added into VA, particularly for log handling. Now it is possible to redirect the log messages into a callback. Thus, we use it to redirect VA-API message into the GStreamer log mechanisms, uncluttering the console’s output.
Also, we have blacklisted, in autoconf and meson, libva version 0.99.0, because that version is used internally by the closed-source version of Intel MediaSDK, which is incompatible with official libva. By the way, there is a new open-source version of MediaSDK, but we will talk about it in a future blog post.
Application VA Display sharing
Normally, the object GstVaapiDisplay is shared among the pipeline through the GstContext mechanism. But this class is defined internally and it is not exposed to users since release 1.6. This posed a problem when an application wanted to create its own VA Display and share it with an embedded pipeline. The solution is a new context application message: gst.vaapi.app.Display, defined as a GstStructure with two fields: va-display with the application’s vaDisplay, and x11-display with the application’s X11 native display. In the future, a Wayland’s native handler will be processed too. Please note that this context message is only processed by vaapisink.
One precondition for this solution was the removal of the VA display cache mechanism, a lingered request from users, which, of course, we did.
Interoperability with appsink and similar
A hardware accelerated driver, as the Intel one, may have custom offsets and strides for specific memory regions. We use the GstVideoMeta to set this custom values. The problem comes when downstream does not handle this meta, for example, appsink. Then, the user expect the “normal” values for those variable, but in the case of GStreamer VA-API with a hardware based driver, when the user displays the frame, it is shown corrupted.
In order to fix this, we have to make a memory copy, from our custom VA-API images to an allocated system memory. Of course there is a big CPU penalty, but it is better than delivering wrong video frames. If the user wants a better performance, then they should seek for a different approach.
Resurrection of GstGLUploadTextureMeta for EGL renders
I know, GstGLUploadTextureMeta must die, right? I am convinced of it. But, Clutter video sink uses it, an it has a vast number of users, so we still have to support it.
Last release we had remove the support for EGL/Wayland in the last minute because we found a terrible bug just before the release. GLX support has always been there.
With Daniel van Vugt efforts, we resurrected the support for that meta in EGL. Though I expect the replacement of Clutter sink with glimagesink someday, soon.
vaapisink demoted in Wayland
vaapisink was demoted to marginal rank on Wayland because COGL cannot display YUV surfaces.
This means, by default, vaapisink won’t be auto-plugged when playing in Wayland.
The reason is because Mutter (aka GNOME) cannot display the frames processed by vaapisink in Wayland. Nonetheless, please note that in Weston, it works just fine.
Decoders
We have improved a little bit upstream renegotiation: if the new stream is compatible with the previous one, there is no need to reset the internal parser, with the exception of changes in codec-data.
low-latency property in H.264
A new property has added only to H.264 decoder: low-latency. Its purpose is for live streams that do not conform the H.264 specification (sadly there are many in the wild) and they need to twitch the spec implementation. This property force to push the frames in the decoded picture buffer as soon as possible.
base-only property in H.264
This is the result of the Google Summer of Code 2017, by Orestis Floros. When this property is enabled, all the MVC (Multiview Video Coding) or SVC (Scalable Video Coding) frames are dropped. This is useful if you want to reduce the processing time or if your VA-API driver does not support those kind of streams.
Encoders
In this release we have put a lot of effort in encoders.
Processing Regions of Interest
It is possible, for certain back-ends and profiles (for example, H.264 and H.265 encoders with Intel driver), to specify a set of regions of interest per frame, with a delta-qp per region. This mean that we would ask more quality in those regions.
In order to process regions of interest, upstream must add to the video frame, a list of GstVideoRegionOfInterestMeta. This list then is traversed by the encoder and it requests them if the VA-API profile, in the driver, supports it.
The common use-case for this feature is if you want to higher definition in regions with faces or text messages in the picture, for example.
New encoding properties
quality-level: For all the available encoders. This is number between 1 to 8, where a lower number means higher quality (and slower processing).
aud: This is for H.264 encoder only and it is available for certain drivers and platforms. When it is enabled, an AU delimiter is inserted for each encoded frame. This is useful for network streaming, and more particularly for Apple clients.
mbbrc: For H.264 only. Controls (auto/on/off) the macro-block bit-rate.
temporal-levels: For H.264 only. It specifies the number of temporal levels to include a the hierarchical frame prediction.
prediction-type: For H.264 only. It selects the reference picture selection mode.
The frames are encoded as different layers. A frame in a particular layer will use pictures in lower or same layer as references. This means decoder can drop frames in upper layer but still decode lower layer frames.
hierarchical-p: P frames, except in top layer, are reference frames. Base layer frames are I or B.
hierarchical-b: B frames , except in top most layer, are reference frames. All the base layer frames are I or P.
refs: Added for H.265 (it was already supported for H.264). It specifies the number of reference pictures.
qp-ip and qp-ib: For H.264 and H.265 encoders. They handle the QP (quality parameters) difference between the I and P frames, the the I and B frames respectively.
Set media profile via downstream caps
H.264 and H.265 encoders now can configure the desired media profile through the downstream caps.
Contributors
Many thanks to all the contributors and bug reporters.
1 Daniel van Vugt
46 Hyunjun Ko
1 Jan Schmidt
3 Julien Isorce
1 Matt Staples
2 Matteo Valdina
2 Matthew Waters
1 Michael Tretter
4 Nicolas Dufresne
9 Orestis Floros
1 Philippe Normand
4 Sebastian Dröge
24 Sreerenj Balachandran
1 Thibault Saunier
13 Tim-Philipp Müller
1 Tomas Rataj
2 U. Artie Eoff
1 VaL Doroshchuk
172 Víctor Manuel Jáquez Leal
3 XuGuangxin
2 Yi A Wang
Recently, my teammate Jessica wrote an excellent intro blog post about VPP. VPP is an open source user-space networking framework that we’re looking into at Igalia. I highly recommend reading Jessica’s post before this post to get aquainted with general VPP ideas.
In this blog post, I wanted to follow up on that blog post and talk a bit about some of the details of plugin construction in VPP and document some of the internals that took me some time to understand.
First, I’ll start off by talking about how to architect the flow of packets in and out of a plugin.
How and where do you add nodes in the graph?
VPP is based on a graph architecture, in which vectors of packets are transferred between nodes in a graph.
Here’s an illustration from a VPP presentation of what the graph might look like:
One thing that took me a while to understand, however, was the mechanics of how nodes in the graph are actually hooked together. I’ll try to explain some of that, starting with the types of nodes that are available.
Some of the nodes in VPP produce data from a network driver (perhaps backed by DPDK) or some other source for consumption. Other nodes are responsible for manipulating incoming data in the graph and possibly producing some output.
The former are called “input” nodes and are called on each main loop iteration. The latter are called “internal” nodes. The type of node is specified using the vlib_node_type_t type when declaring a node using VLIB_REGISTER_NODE.
There’s also another type of node that is useful, which is the “process” node. This exposes a thread-like behavior, in which the node’s callback routine can be suspended and reanimated based on events or a timer. This is useful for sending periodic notifications or polling some data that’s managed by another node.
Since input nodes are the start of the graph, they are responsible for generating packets from some source like a NIC or pcap file and injecting them into the rest of the graph.
For internal nodes, the packets have to come from somewhere else. There seem to be many ways to specify how an internal node gets its packets.
For example, it’s possible to tell VPP to direct all packets with a specific ethertype or IP protocol to a node that you write (see this wiki page for some of these options).
Another method, that’s convenient for use in external plugins, is using “feature arcs”. VPP comes with abstract concepts called “features” (which are distinct from actual nodes) that essentially form an overlay over the concrete graph of nodes.
These features can be enabled or disabled on a per-interface basis.
You can hook your own node into these feature arcs by using VNET_FEATURE_INIT:
1
2
3
4
5
6
7
8
9
/** * @brief Hook the sample plugin into the VPP graph hierarchy. */VNET_FEATURE_INIT(sample,static)={.arc_name="device-input",.node_name="sample",.runs_before=VNET_FEATURES("ethernet-input"),};
This code from the sample plugin is setting the sample node to run on the device-input arc. For a given arc like this, the VPP dispatcher will run the code for the start node on the arc and then run all of the features that are registered on that arc. This ordering is determined by the .runs_before clauses in those features.
As you can see, arcs have a start node. Inside the start node there is typically some code that checks if features are present on the node and, if the check succeeds, will designate the next node as one of those feature nodes. For example, this snippet from the DPDK plugin:
1
2
3
4
/* Do we have any driver RX features configured on the interface? */vnet_feature_start_device_input_x4(xd->vlib_sw_if_index,&next0,&next1,&next2,&next3,b0,b1,b2,b3);
From this snippet, you can see how the process of getting data through feature arcs is kicked off. In the next section, you’ll see how internal nodes in the middle of a feature arc can continue the chain.
BTW: the methods of hooking up your node into the graph I’ve described here aren’t exhaustive. For example, there’s also something called DPOs (“data path object”) that I haven’t covered at all. Maybe a topic for a future blog post (once I understand them!).
Specifying the next node
I went over a few methods for specifying how packets get to a node, but you may also wonder how packets get directed out of a node. There are also several ways to set that up too.
When declaring a VPP node using VLIB_REGISTER_NODE, you can provide the .next_nodes argument (along with .n_next_nodes) to specify an indexed list of next nodes in the graph. Then you can use the next node indices in the node callback to direct packets to one of the next nodes.
The sample plugin sets it up like this:
1
2
3
4
5
6
7
8
9
10
VLIB_REGISTER_NODE(sample_node)={/* ... elided for example ... */.n_next_nodes=SAMPLE_N_NEXT,/* edit / add dispositions here */.next_nodes={[SAMPLE_NEXT_INTERFACE_OUTPUT]="interface-output",},};
This declaration sets up the only next node as interface-output, which is the node that selects some hardware interface to transmit on.
Alternatively, it’s possible to programmatically fetch a next node index by name using vlib_get_node_by_name:
This excerpt is from the VPP IPFIX code that sends report packets periodically. This kind of approach seems more common in process nodes.
When using feature arcs, a common approach is to use vnet_feature_next to select a next node based on the feature mechanism:
1
2
3
/* Setup packet for next IP feature */vnet_feature_next(vnet_buffer(b0)->sw_if_index[VLIB_RX],&next0,b0);vnet_feature_next(vnet_buffer(b1)->sw_if_index[VLIB_RX],&next1,b1);
In this example, the next0 and next1 variables are being set by vnet_feature_next based on the appropriate next features in the feature arc.
(vnet_feature_next is a convenience function that uses vnet_get_config_data under the hood as described in this mailing list post if you want to know more about the mechanics)
Though even when using feature arcs, it doesn’t seem mandatory to use vnet_feature_next. The sample plugin does not, for example.
Buffer metadata
In addition to packets, nodes can also pass some metadata around to each other. Specifically, there is some “opaque” space reserved in the vlib_buffer_t structure that stores the packet data:
1
2
3
4
5
6
7
8
9
10
typedefstruct{/* ... elided ... */u32opaque[10];/**< Opaque data used by sub-graphs for their own purposes. See .../vnet/vnet/buffer.h/* ... */}vlib_buffer_t;
For networking purposes, the vnet library provides a data type vnet_buffer_opaque_t that is stored in the opaque field. This contains data useful to networking layers like Ethernet, IP, TCP, and so on.
The vnet data is unioned (that is, data for one layer may be overlaid on data for another) and it’s expected that graph nodes make sure not to stomp on data that needs to be set in their next nodes. You can see how the data intersects in this slide from an FD.io DevBoot.
One of the first fields stored in the opaque data is sw_if_index, which is set by the driver nodes initially. The field stores a 2-element array of input and output interface indices for a buffer.
One of the things that puzzled me for a bit was precisely how the sw_if_index field, which is very commonly read and set in VPP plugins, is used. Especially the TX half of the pair, which is more commonly manipulated. It looks like the field is often set to instruct VPP to choose a transmit interface for a packet.
For example, in the sample plugin code the index is set in the following way:
1
2
3
4
sw_if_index0=vnet_buffer(b0)->sw_if_index[VLIB_RX];/* Send pkt back out the RX interface */vnet_buffer(b0)->sw_if_index[VLIB_TX]=sw_if_index0;
What this is doing is getting the interface index where the packet was received and then setting the output index to be the same. This tells VPP to send off the packet to the same interface it was received from.
The receive index is set to 0, which means the local0 interface that is always configured. The use of ~0 means “don’t know yet”. The effect of that is (see the docs for ip4-lookup) that a lookup is done in the FIB attached to the receive interface, which is the local0 one in this case.
Some code inside VPP instead sets the transmit index to some user configurable fib_index to let the user specify a destination (often defaulting to ~0 if the user specified nothing):
Finally, I want to briefly talk about some additional resources for learning about VPP details.
There are numerous useful libraries that come with VPP, such as data structures (e.g., vectors and hash-tables), formatting, and logging. These are contained in source directories like vlib and vppinfra.
I found these slides on VPP libraries and its core infrastructure quite helpful. It’s from a workshop that was held in Paris. The slides give an overview of various programming components provided in VPP, and then you can look up the detailed APIs in the VPP docs.
There are also a number of other useful pages on the VPP wiki. If you prefer learning from videos, there are some good in-depth video tutorials on VPP from the official team too.
Since I’m a VPP newbie myself, you may want to carefully look at the source code and make sure what I’m saying is accurate. Please do let me know if you spot any inaccuracies in the post.
Future Ozone Wayland backend is still not ready for shipping. So we are announcing the release of an updated Ozone Wayland backend for Chromium, based on the implementation provided by Intel. It is rebased on top of latest stable Chromium release and you can find it in my team Github. Hope you will appreciate it.
Official Chromium on Linux desktop nowadays
Linux desktop is progressively migrating to use Wayland as the display server. It is the default option in Fedora, Ubuntu ~~and, more importantly, the next Ubuntu Long Term Support release will ship Gnome Shell Wayland display server by default~~ (P.S. since this post was originally written, Ubuntu has delayed the Wayland adoption for LTS).
As is, now, Chromium browser for Linux desktop support is based on X11. This means it will natively interact with an X server and with its XDG extensions for displaying the contents and receiving user events. But, as said, next generation of Linux desktop will be using Wayland display servers instead of X11. How is it working? Using XWayland server, a full X11 server built on top of Wayland protocol. Ok, but that has an impact on performance. Chromium needs to communicate and paint to X11 provided buffers, and then, those buffers need to be shared with Wayland display server. And the user events will need to be proxied from the Wayland display server through the XWayland server and X11 protocol. It requires more resources: more memory, CPU, and GPU. And it adds more latency to the communication.
Ozone
Chromium supports officially several platforms (Windows, Android, Linux desktop, iOS). But it provides abstractions for porting it to other platforms.
The set of abstractions is named Ozone (more info here). It allows to implement one or more platform components with the hooks for properly integrating with a platform that is in the set of officially supported targets. Among other things it provides abstractions for:
* Obtaining accelerated surfaces.
* Creating and obtaining windows to paint the contents.
* Interacting with the desktop cursor.
* Receiving user events.
* Interacting with the window manager.
Chromium and Wayland (2014-2016)
Even if Wayland was not used on Linux desktop, a bunch of embedded devices have been using Wayland for their display server for quite some time. LG has been shipping a full Wayland experience on the webOS TV products.
In the last 4 years, Intel has been providing an implementation of Ozone abstractions for Wayland. It was an amazing work that allowed running Chromium browser on top of a Wayland compositor. This backend has been the de facto standard for running Chromium browser on all these Wayland-enabled embedded devices.
But the development of this implementation has mostly stopped around Chromium 49 (though rebases on top of Chromium 51 and 53 have been provided).
Chromium and Wayland (2018+)
Since the end of 2016, Igalia has been involved on several initiatives to allow Chromium to run natively in Wayland. Even if this work is based on the original Ozone Wayland backend by Intel, it is mostly a rewrite and adaptation to the future graphics architecture in Chromium (Viz and Mus).
This is being developed in the Igalia GitHub, downstream, though it is expected to be landed upstream progressively. Hopefully, at some point in 2018, this new backend will be fully ready for shipping products with it. But we are still not there. ~~Some major missing parts are Wayland TextInput protocol and content shell support~~ (P.S. since this was written, both TextInput and content shell support are working now!).
Ok, so new Wayland backend is still not ready in some cases, and the old one is unmaintained. For that reason, LG is announcing the release of an updated legacy Ozone Wayland backend. It is essentially the original Intel backend, but ported to current Chromium stable.
Why? Because we want to provide a migration path to the future Ozone Wayland backend. And because we want to share this effort with other developers, willing to run Chromium in Wayland immediately, or that are still using the old backend and cannot immediately migrate to the new one.
WARNINGIf you are starting development for a product that is going to happen in 1-2 years… Very likely your best option is already migrating now to the new Ozone Wayland backend (and help with the missing bits). We will stop maintaining it ourselves once new Ozone Wayland backend lands upstream and covers all our needs.
What does this port include?
* Rebased on top of Chromium m60, m61, m62 and m63.
* Ported to GN.
* It already includes some changes to adapt to the new Ozone Wayland refactors.
For some time now I have been working on and off on a personal project with no other purpose than toying a bit with Vulkan and some rendering and shading techniques. Although I’ll probably write about that at some point, in this post I want to focus on Vulkan’s specialization constants and how they can provide a very visible performance boost when they are used properly, as I had the chance to verify while working on this project.
The concept behind specialization constants is very simple: they allow applications to set the value of a shader constant at run-time. At first sight, this might not look like much, but it can have very important implications for certain shaders. To showcase this, let’s take the following snippet from a fragment shader as a case study:
That is a snippet taken from a Screen Space Ambient Occlusion shader that I implemented in my project, a popular techinique used in a lot of games, so it represents a real case scenario. As we can see, the process involves a set of vector samples passed to the shader as a UBO that are processed for each fragment in a loop. We have made the maximum number of samples that the shader can use large enough to accomodate a high-quality scenario, but the actual number of samples used in a particular execution will be taken from a push constant uniform, so the application has the option to choose the quality / performance balance it wants to use.
While the code snippet may look trivial enough, let’s see how it interacts with the shader compiler:
The first obvious issue we find with this implementation is that it is preventing loop unrolling to happen because the actual number of samples to use is unknown at shader compile time. At most, the compiler could guess that it can’t be more than 64, but that number of iterations would still be too large for Mesa to unroll the loop in any case. If the application is configured to only use 24 or 32 samples (the value of our push constant uniform at run-time) then that number of iterations would be small enough that Mesa would unroll the loop if that number was known at shader compile time, so in that scenario we would be losing the optimization just because we are using a push constant uniform instead of a constant for the sake of flexibility.
The second issue, which might be less immediately obvious and yet is the most significant one, is the fact that if the shader compiler can tell that the size of the samples array is small enough, then it can promote the UBO array to a push constant. This means that each access to S.samples[i] turns from an expensive memory fetch to a direct register access for each sample. To put this in perspective, if we are rendering to a full HD target using 24 samples per fragment, it means that we would be saving ourselves from doing 1920x1080x24 memory reads per frame for a very visible performance gain. But again, we would be loosing this optimization because we decided to use a push constant uniform.
Vulkan’s specialization constants allow us to get back these performance optmizations without sacrificing the flexibility we implemented in the shader. To do this, the API provides mechanisms to specify the values of the constants at run-time, but before the shader is compiled.
Continuing with the shader snippet we showed above, here is how it can be rewritten to take advantage of specialization constants:
We are now informing the shader that we have a specialization constant NUM_SAMPLES, which represents the actual number of samples to use. By default (if the application doesn’t say otherwise), the specialization constant’s value is 64. However, now that we have a specialization constant in place, we can have the application set its value at run-time, like this:
The application code above sets up specialization constant information for shader consumption at run-time. This is done via an array of VkSpecializationMapEntry entries, each one determining where to fetch the constant value to use for each specialization constant declared in the shader for which we want to override its default value. In our case, we have a single specialization constant (with id 0), and we are taking its value (of integer type) from offset 0 of a buffer. In our case we only have one specialization constant, so our buffer is just the address of the variable holding the constant’s value (config.ssao.num_samples). When we create the Vulkan pipeline, we pass this specialization information using the pSpecializationInfo field of VkPipelineShaderStageCreateInfo. At that point, the driver will override the default value of the specialization constant with the value provided here before the shader code is optimized and native GPU code is generated, which allows the driver compiler backend to generate optimal code.
It is important to remark that specialization takes place when we create the pipeline, since that is the only moment at which Vulkan drivers compile shaders. This makes specialization constants particularly useful when we know the value we want to use ahead of starting the rendering loop, for example when we are applying quality settings to shaders. However, If the value of the constant changes frequently, specialization constants are not useful, since they require expensive shader re-compiles every time we want to change their value, and we want to avoid that as much as possible in our rendering loop. Nevertheless, it it is possible to compile the same shader with different constant values in different pipelines, so even if a value changes often, so long as we have a finite number of combinations, we can generate optimized pipelines for each one ahead of the start of the redendering loop and just swap pipelines as needed while rendering.
Conclusions
Specialization constants are a straight forward yet powerful way to gain control over how shader compilers optimize your code. In my particular pet project, applying specialization constants in a small number of shaders allowed me to benefit from loop unrolling and, most importantly, UBO promotion to push constants in the SSAO pass, obtaining performance improvements that ranged from 10% up to 20% depending on the configuration.
Finally, although the above covered specialization constants from the point of view of Vulkan, this is really a feature of the SPIR-V language, so it is also available in OpenGL with the GL_ARB_gl_spirv extension, which is core since OpenGL 4.6.
Multimedia applications based on GStreamer usually handle playback with the
playbin element. I recently added support for playbin3 in WebKit. This post
aims to document the changes needed on application side to support this new
generation flavour of playbin.
So, first of, why is it named playbin3 anyway? The GStreamer 0.10.x series had a
playbin element but a first rewrite (playbin2) made it obsolete in the GStreamer
1.x series. So playbin2 was renamed to playbin. That’s why a second rewrite is
nicknamed playbin3, I suppose :)
Why should you care about playbin3? Playbin3 (and the elements it’s using
internally: parsebin, decodebin3, uridecodebin3 among others) is the result of a
deep re-design of playbin2 (along with decodebin2 and uridecodebin) to better support:
gapless playback
audio cross-fading support (not yet implemented)
adaptive streaming
reduced CPU, memory and I/O resource usage
faster stream switching and full control over the stream selection process
This work was carried on mostly by Edward Hervey, he presented his work in
detail at 3 GStreamer conferences. If you want to learn more about this and the
internals of playbin3 make sure to watch his awesome presentations at the 2015
gst-conf, 2016 gst-conf and 2017 gst-conf.
Playbin3 was added in GStreamer 1.10. It is still considered experimental but in
my experience it works already very well. Just keep in mind you should use at
least the latest GStreamer 1.12 (or even the upcoming 1.14) release before
reporting any issue in Bugzilla. Playbin3 is not a drop-in replacement for
playbin, both elements share only a sub-set of GObject properties and signals.
However, if you don’t want to modify your application source code just yet, it’s
very easy to try playbin3 anyway:
$ USE_PLAYBIN3=1 my-playbin-based-app
Setting the USE_PLAYBIN environment variable enables a code path inside the
GStreamer playback plugin which swaps the playbin element for the playbin3
element. This trick provides a glance to the playbin3 element for the most lazy
people :) The problem is that depending on your use of playbin, you might get
runtime warnings, here’s an example with the Totem player:
$ USE_PLAYBIN3=1 totem ~/Videos/Agent327.mp4
(totem:22617): GLib-GObject-WARNING **: ../../../../gobject/gsignal.c:2523: signal 'video-changed' is invalid for instance '0x556db67f3170' of type 'GstPlayBin3'
(totem:22617): GLib-GObject-WARNING **: ../../../../gobject/gsignal.c:2523: signal 'audio-changed' is invalid for instance '0x556db67f3170' of type 'GstPlayBin3'
(totem:22617): GLib-GObject-WARNING **: ../../../../gobject/gsignal.c:2523: signal 'text-changed' is invalid for instance '0x556db67f3170' of type 'GstPlayBin3'
(totem:22617): GLib-GObject-WARNING **: ../../../../gobject/gsignal.c:2523: signal 'video-tags-changed' is invalid for instance '0x556db67f3170' of type 'GstPlayBin3'
(totem:22617): GLib-GObject-WARNING **: ../../../../gobject/gsignal.c:2523: signal 'audio-tags-changed' is invalid for instance '0x556db67f3170' of type 'GstPlayBin3'
(totem:22617): GLib-GObject-WARNING **: ../../../../gobject/gsignal.c:2523: signal 'text-tags-changed' is invalid for instance '0x556db67f3170' of type 'GstPlayBin3'
sys:1: Warning: g_object_get_is_valid_property: object class 'GstPlayBin3' has no property named 'n-audio'
sys:1: Warning: g_object_get_is_valid_property: object class 'GstPlayBin3' has no property named 'n-text'
sys:1: Warning: ../../../../gobject/gsignal.c:3492: signal name 'get-video-pad' is invalid for instance '0x556db67f3170' of type 'GstPlayBin3'
As mentioned previously, playbin and playbin3 don’t share the same set of
GObject properties and signals, so some changes in your application are required
in order to use playbin3.
If your application is based on the GstPlayer library then you should set the
GST_PLAYER_USE_PLAYBIN3 environment variable. GstPlayer already handles both
playbin and playbin3, so no changes needed in your application if you use GstPlayer!
Ok, so what if your application relies directly on playbin? Some
changes are needed! If you previously used playbin stream selection
properties and signals, you will now need to handle the GstStream and
GstStreamCollection APIs. Playbin3 will emit a stream collection
message on the bus, this is very nice because the collection
includes information (metadata!) about the streams (or tracks) the
media asset contains. In playbin this was handled with a bunch of
signals (audio-tags-changed, audio-changed, etc), properties (n-audio,
n-video, etc) and action signals (get-audio-tags, get-audio-pad, etc).
The new GstStream API provides a centralized and
non-playbin-specific access point for all these informations. To
select streams with playbin3 you now need to send a select_streams
event so that the demuxer can know exactly which streams should be
exposed to downstream elements. That means potentially improved
performance! Once playbin3 completed the stream selection it will emit
a streams selected message, the application should handle this
message and potentially update its internal state about the selected
streams. This is also the best moment to update your UI regarding the
selected streams (like audio track language, video track dimensions, etc).
Another small difference between playbin and playbin3 is about the source
element setup. In playbin there is a source read-only GObject property and a
source-setup GObject signal. In playbin3 only the latter is available, so your
application should rely on source-setup instead of the notify::source
GObject signal.
The gst-play-1.0 playback utility program already supports playbin3 so it
provides a good source of inspiration if you consider porting your application
to playbin3. As mentioned at the beginning of this post, WebKit also now
supports playbin3, however it needs to be enabled at build time using the CMake
-DUSE_GSTREAMER_PLAYBIN3=ON option. This feature is not part of the WebKitGTK+
2.20 series but should be shipped in 2.22. As a final note I wanted to
acknowledge my favorite worker-owned coop Igalia for allowing me to work on
this WebKit feature and also our friends over at Centricular for all the
quality work on playbin3.
Time for a change! Almost 10 years ago I was starting to hack on a
Blog engine with two friends, it was called Alinea and it powered
this website for a long time. Back then hacking on your own Blog
engine was the pre-requirement to host your blog :) But nowadays
people just use Wordpress or similar platforms, if they still have a
blog at all. Alinea fell into oblivion as I didn’t have time and
motivation to maintain it.
Moving to Pelican was quite easy, since I’ve been writing content in
ReST on the previous blog I only had to pull data from the database
and hacked pelican_import.py a bit :)
Now that this website looks almost modern I’ll hopefully start to blog
again, expect at least news about the WebKit work I still enjoy doing
at Igalia.
Back in January 2011 I wrote a GNOME-Shell extension allowing Gajim users to
carry on with their chats using the Empathy infrastructure and UI present in the
Shell. For some time the extension was also part of the official
gnome-shell-extensions module and then I had to move it to Github as a
standalone extension. Sadly I stopped using Gajim a few years ago and my
interest in maintaining this extension has decreased quite a lot.
I don’t know if this extension is actively used by anyone beyond the few bugs
reported in Github, so this is a call for help. If anyone still uses this
extension and wants it supported in future versions of GNOME-Shell, please send
me a mail so I can transfer ownership of the Github repository and see what I
can do for the extensions.gnome.org page as well.
(Huh, also. Hi blogosphere again! My last post was in 2014 it seems :))
Last week I attended the Web Engines Hackfest. The
event was sponsored by Igalia (also hosting the event), Adobe and Collabora.
As usual I spent most of the time working on the WebKitGTK+ GStreamer
backend and Sebastian Dröge kindly joined and helped out quite a
bit, make sure to read his post about the event!
We first worked on the WebAudio GStreamer backend, Sebastian cleaned
up various parts of the code, including the playback pipeline and the
source element we use to bridge the WebCore AudioBus with the playback
pipeline. On my side I finished the AudioSourceProvider patch that was
abandoned for a few months (years) in Bugzilla. It’s an interesting
feature to have so that web apps can use the WebAudio API with raw
audio coming from Media elements.
I also hacked on GstGL support for video rendering. It’s quite
interesting to be able to share the GL context of WebKit with
GStreamer! The patch is not ready yet for landing but thanks to
the reviews from Sebastian, Mathew Waters and Julien Isorce I’ll improve it
and hopefully commit it soon in WebKit ToT.
Sebastian also worked on Media Source Extensions support. We had a
very basic, non-working, backend that required… a rewrite, basically
:) I hope we will have this reworked backend soon in
trunk. Sebastian already has it working on Youtube!
The event was interesting in general, with discussions about rendering
engines, rendering and JavaScript.
Last blog post, I wrote about Snabb, a blazingly fast userspace networking framework. This blog post does stand alone, however I see networking from a Snabb mindset and there will be some comparisons between that and VPP, another userspace networking framework and the subject of this blog. I recommend folks read that first.
The What and Why of VPP
VPP is another framework that helps you write userspace networking applications using a kernel bypass, similar to Snabb. It came out of Cisco a few years after being open-sourced. Since then it’s made a name for itself, becoming quite popular for writing fast networking applications. As part of my work at Igalia I spent a few months with my colleague Asumu investigating and learning how to write plugins for it and seeing how it compares to Snabb, so that we can learn from it in the Snabb world as well as develop with it in addition.
The outlook of VPP is quite different from Snabb. The first thing is that it’s all in C, quite a difference when you’ve spent the last two years writing Lua. C is a pretty standard language for these sorts of things, however it’s a lot less flexible than more modern languages (such as Lua). I sometimes question whether its current popularity is still deserved. When you start VPP for the first time it configures itself as a router. All the additional functionality you want to provide is done via plugins you write and compile out of tree, and then load into VPP. The plugins then hook themselves into a graph of nodes, usually somewhere in the IP stack. Another difference and in my opinion one of the most compelling arguments for VPP is that you can use DPDK, a layer written by Intel (one of the larger network card makers) with a lot of the drivers. DPDK adds a node at the start called dpdk-input which feeds the rest of the graph with packets. As you might imagine with a full router it also has its own routing table populated by performing ARP or NDP requests (to get the address of hops ahead). It also provides ICMP facilities to, for example, respond to ping requests.
First thing to do is install it from the packages they provide. Once you’ve done that you can start it with your init system or directly from the vppcommand. You then access it via the command-line VPP provides. This is already somewhat of a divergence from Snabb where you compile the tree with the programs and apps you want to use present, and then run the main Snabb binary from the command line directly. The VPP shell is actually rather neat, it lets you query and configure most aspects of it. To give you a taste of it, this is configuring the network card and enabling the IPFIX plugin which comes with VPP:
vpp# set interface ip address TenGigabitEthernet2/0/0 10.10.1.2/24
vpp# set interface state TenGigabitEthernet2/0/0 up
vpp# flowprobe params record l3 active 120 passive 300
vpp# flowprobe feature add-del TenGigabitEthernet2/0/0 ip4
vpp# show int
Name Idx State Counter Count
TenGigabitEthernet2/0/0 1 up
local0 0 down drops 1
Let’s get cracking
I’m not intending this to be a tutorial, more for it to give you a taste of what it’s like working with VPP. Despite this, I hope you do find it useful if you’re hoping to hack on VPP yourself. Apologies if the blog seems a bit code heavy.
Whilst above I told you how the outlook was different, I want to reiterate. Snabb, for me, is like Lego. It has a bunch of building blocks (apps: ARP, NDP, firewalls, ICMP responders, etc.) that you put together in order to make exactly want you want, out of the bits you want. You only have to use the blocks you decide you want. To use some code from my previous blog post:
module(..., package.seeall)
local pcap = require("apps.pcap.pcap")
local Intel82599 = require("apps.intel_mp.intel_mp").Intel82599
function run(parameters)
-- Lua arrays are indexed from 1 not 0.
local pci_address = parameters[1]
local output_pcap = parameters[2]
-- Configure the "apps" we're going to use in the graph.
-- The "config" object is injected in when we're loaded by snabb.
local c = config.new()
config.app(c, "nic", Intel82599, {pciaddr = pci_address})
config.app(c, "pcap", pcap.PcapWriter, output_pcap)
-- Link up the apps into a graph.
config.link(c, "nic.output -> pcap.input")
-- Tell the snabb engine our configuration we've just made.
engine.configure(c)
-- Lets start the apps for 10 seconds!
engine.main({duration=10, report = {showlinks=true}})
end
This doesn’t have an IP stack, it grabs a bunch of packets from the Intel network card and dumps them into this Pcap (Packet capture) file. If you want more functionality like ARP or ICMP responding, you add those apps. I think both are valid approaches, though personally I have a strong preference for the way Snabb works. If you do need a fully setup router then of course, it’s much easier with VPP, you just start it and you’re ready to go. But a lot of the time, I don’t.
Getting to know VPP is, I think, one of its biggest drawbacks. There are the parts I’ve mentioned above about it being a complex router which can complicate things, and it being in C, which feels terse to someone who’s not worked much in it. There are also those parts where I feel that information needed to get off the ground is a bit hard to come by, something many projects struggle with, VPP being no exception. There is some information between the wiki, documentation, youtube channel and dev mailing list. Those are useful to look at and go through.
To get started writing your plugin there are a few parts you’ll need:
A graph node which takes in packets, does something and passes them on to the next node.
An interface to your plugin on the VPP command line so you can enable and disable it
Functionality to trace packets which come through (more on this later)
We started by using the sample plugin as a base. I think it works well, it has all the above and it’s pretty basic so it wasn’t too difficult to get up to speed with. Let’s look at some of the landmarks of the VPP plugin developer landscape:
This macro registers your plugin and provides some meta-data for it such as the function name, the trace function (again, more on this later), the next node and some error stuff. The other thing with the node is obviously the node function itself. Here is a shortened version of it with lots of comments to explain what’s going on:
always_inline uword
sample_node_fn (vlib_main_t * vm,
vlib_node_runtime_t * node,
vlib_frame_t * frame,
u8 is_ipv6)
{
u32 n_left_from, * from, * to_next;
sample_next_t next_index;
/* Grab a pointer to the first vector in the frame */
from = vlib_frame_vector_args (frame);
n_left_from = frame->n_vectors;
next_index = node->cached_next_index;
/* Iterate over the vectors */
while (n_left_from > 0)
{
u32 n_left_to_next;
vlib_get_next_frame (vm, node, next_index,
to_next, n_left_to_next);
/* There are usually two loops which look almost identical one takes
* two packets at a time (for additional speed) and the other loop does
* the *exact* same thing just for a single packet. There are also
* apparently some plugins which define some for 4 packets at once too.
*
* The advice given is to write the single packet loop (or read) and
* then write the worry about the multiple packet loops later. I've
* removed the body of the 2 packet loop to shorten the code, it just
* does what the single one does, you're not missing much.
*/
while (n_left_from >= 4 && n_left_to_next >= 2)
{
}
while (n_left_from > 0 && n_left_to_next > 0)
{
u32 bi0;
vlib_buffer_t * b0;
u32 next0 = SAMPLE_NEXT_INTERFACE_OUTPUT;
u32 sw_if_index0;
u8 tmp0[6];
ethernet_header_t *en0;
/* speculatively enqueue b0 to the current next frame */
bi0 = from[0];
to_next[0] = bi0;
from += 1;
to_next += 1;
n_left_from -= 1;
n_left_to_next -= 1;
/* Get the reference to the buffer */
b0 = vlib_get_buffer (vm, bi0);
en0 = vlib_buffer_get_current (b0); /* This is the ethernet header */
/* This is where you do whatever you'd like to with your packet */
/* ... */
/* Get the software index for the hardware */
sw_if_index0 = vnet_buffer(b0)->sw_if_index[VLIB_RX];
/* Send pkt back out the RX interface */
vnet_buffer(b0)->sw_if_index[VLIB_TX] = sw_if_index0;
/* Do we want to trace (used for debugging) */
if (PREDICT_FALSE((node->flags & VLIB_NODE_FLAG_TRACE)
&& (b0->flags & VLIB_BUFFER_IS_TRACED))) {
sample_trace_t *t =
vlib_add_trace (vm, node, b0, sizeof (*t));
}
/* verify speculative enqueue, maybe switch current next frame */
vlib_validate_buffer_enqueue_x1 (vm, node, next_index,
to_next, n_left_to_next,
bi0, next0);
}
vlib_put_next_frame (vm, node, next_index, n_left_to_next);
}
return frame->n_vectors;
}
I’ve tried to pare down the code to the important things as much as I can. I personally am not a huge fan of the code duplication which occurs due to the multiple loops for different amounts of packets in the vector, I think it makes the code a bit messy. It definitely goes against DRY (Don’t Repeat Yourself). I’ve not seen any statistics on the improvements nor had time to look into it myself yet, but I’ll take it as true that it’s worth it and go with it :-). The code definitely has more boilerplate than Lua. I think that’s the nature of C unfortunately.
Finally you need to hook the node into the graph, this is done with yet another macro, you choose the node and arc in the graph and it’ll put it into the graph for you:
In this case we’re having it run before ip4-lookup.
Tracing
Tracing is really useful when debugging both as an operator and a developer. You can enable a trace on a specific node for a certain amount of packets. The trace shows you which nodes they go through and usually the node provides extra information in the trace. Here’s an example of such a trace:
Packet 1
00:01:23:899371: dpdk-input
TenGigabitEthernet2/0/0 rx queue 0
buffer 0x1d3be2: current data 14, length 46, free-list 0, clone-count 0, totlen-nifb 0, trace 0x0
l4-cksum-computed l4-cksum-correct l2-hdr-offset 0 l3-hdr-offset 14
PKT MBUF: port 0, nb_segs 1, pkt_len 60
buf_len 2176, data_len 60, ol_flags 0x88, data_off 128, phys_addr 0x5b8ef900
packet_type 0x211 l2_len 0 l3_len 0 outer_l2_len 0 outer_l3_len 0
Packet Offload Flags
PKT_RX_L4_CKSUM_BAD (0x0008) L4 cksum of RX pkt. is not OK
PKT_RX_IP_CKSUM_GOOD (0x0080) IP cksum of RX pkt. is valid
Packet Types
RTE_PTYPE_L2_ETHER (0x0001) Ethernet packet
RTE_PTYPE_L3_IPV4 (0x0010) IPv4 packet without extension headers
RTE_PTYPE_L4_UDP (0x0200) UDP packet
IP4: 02:02:02:02:02:02 -> 90:e2:ba:a9:84:1c
UDP: 8.8.8.8 -> 193.5.1.176
tos 0x00, ttl 15, length 46, checksum 0xd8fa
fragment id 0x0000
UDP: 12345 -> 6144
length 26, checksum 0x0000
00:01:23:899404: ip4-input-no-checksum
UDP: 8.8.8.8 -> 193.5.1.176
tos 0x00, ttl 15, length 46, checksum 0xd8fa
fragment id 0x0000
UDP: 12345 -> 6144
length 26, checksum 0x0000
00:01:23:899430: ip4-lookup
fib 0 dpo-idx 13 flow hash: 0x00000000
UDP: 8.8.8.8 -> 193.5.1.176
tos 0x00, ttl 15, length 46, checksum 0xd8fa
fragment id 0x0000
UDP: 12345 -> 6144
length 26, checksum 0x0000
00:01:23:899434: ip4-load-balance
fib 0 dpo-idx 13 flow hash: 0x00000000
UDP: 8.8.8.8 -> 193.5.1.176
tos 0x00, ttl 15, length 46, checksum 0xd8fa
fragment id 0x0000
UDP: 12345 -> 6144
length 26, checksum 0x0000
00:01:23:899437: ip4-arp
UDP: 8.8.8.8 -> 193.5.1.176
tos 0x00, ttl 15, length 46, checksum 0xd8fa
fragment id 0x0000
UDP: 12345 -> 6144
length 26, checksum 0x0000
00:01:23:899441: error-drop
ip4-arp: address overflow drops
The format is a timestamp with the node name, with some extra information the plugin chooses to display. It lets me easily see this came in through dpdk-input on went through a few ip4 nodes until ip4-arp (which presumably sent out an ARP packet) and then it gets black-holed (because it doesn’t know where to send it). This is invaluable information when you want to see what’s going on, I can only imagine it’s great for operators too to debug their own setup / config.
VPP has a useful function called format and unformat. They work a bit like printf and scanf, however they can also take any struct or datatype with a format function and display (or parse) them. This means to display for example an ip4 address it’s just a matter of calling out to the function with the provided format_ip4_address. There are a whole slew of them which come with VPP and it’s trivial to write your own for your own data structures. The other thing to note when writing these tracing functions is that you need to remember to provide the data in your node to the trace function. It’s parsed after the fact as not to hinder performance. The struct we’re going to give to the trace looks like this:
sample_trace_t *t = vlib_add_trace (vm, node, b0, sizeof (*t));
/* These variables are defined in the node block posted above */
t->sw_if_index = sw_if_index0;
t->next_index = next0;
clib_memcpy (t->new_src_mac, en0->src_address,
sizeof (t->new_src_mac));
clib_memcpy (t->new_dst_mac, en0->dst_address,
sizeof (t->new_dst_mac));
Finally the thing we’ve been waiting for, the trace itself:
/* VPP actually comes with a format_mac_address, this is here to show you
* what a format functions look like
*/
static u8 *
format_mac_address (u8 * s, va_list * args)
{
u8 *a = va_arg (*args, u8 *);
return format (s, "%02x:%02x:%02x:%02x:%02x:%02x",
a[0], a[1], a[2], a[3], a[4], a[5]);
}
static u8 * format_sample_trace (u8 * s, va_list * args)
{
CLIB_UNUSED (vlib_main_t * vm) = va_arg (*args, vlib_main_t *);
CLIB_UNUSED (vlib_node_t * node) = va_arg (*args, vlib_node_t *);
sample_trace_t * t = va_arg (*args, sample_trace_t *);
s = format (s, "SAMPLE: sw_if_index %d, next index %d\n",
t->sw_if_index, t->next_index);
s = format (s, " new src %U -> new dst %U",
format_mac_address, t->new_src_mac,
format_mac_address, t->new_dst_mac);
return s;
}
Not bad, I don’t think. Whilst Snabb does make it very easy and has a lot of the same offerings to format and parse things, having for example ipv4:ntop, I appreciate how easy VPP makes doing things like this in C. Format and unformat are quite nice to work with. I love the tracing! Snabb doesn’t have a comparable tracing mechanism, it display a link report but nothing like the trace.
CLI Interface
We have our node, we can trace packets going through it and display some useful info specific to the plugin. We now want our plugin to work. To do that we need to tell VPP that we’re open for business and for it to send packets our way. Snabb doesn’t have a CLI similar to VPPs, partially because it works differently and there is less need for it. However I think the CLI comes into its own when you want to display lots of information and easily configure things on the fly. It also has a command from the shell you can use to execute commands, vppctl, allowing for scripting.
In VPP you use a macro to define a command, similar to the node definition above. This example provides the command name “sample macswap” in our case, a bit of help and the function itself. Here’s what it could look like:
static clib_error_t *
macswap_enable_disable_command_fn (vlib_main_t * vm,
unformat_input_t * input,
vlib_cli_command_t * cmd)
{
sample_main_t * sm = &sample_main;
u32 sw_if_index = ~0;
int enable_disable = 1;
/* Parse the command */
while (unformat_check_input (input) != UNFORMAT_END_OF_INPUT) {
if (unformat (input, "disable"))
enable_disable = 0;
else if (unformat (input, "%U", unformat_vnet_sw_interface,
sm->vnet_main, &sw_if_index));
else
break;
}
/* Display an error if we weren't provided with the interface name */
if (sw_if_index == ~0)
return clib_error_return (0, "Please specify an interface...");
/* This is what you call out to, in other to enable or disable the plugin */
vnet_feature_enable_disable ("device-input", "sample",
sw_if_index, enable_disable, 0, 0);
return 0;
}
VLIB_CLI_COMMAND (sr_content_command, static) = {
.path = "sample macswap",
.short_help =
"sample macswap <interface-name> [disable]",
.function = macswap_enable_disable_command_fn,
};
It’s quite simple. You see if the user is enabling or disabling the plugin (by checking for the presence of “disable” in their command), grab the interface name and then tell VPP to enable or disable your plugin on said interface.
Conclusion
VPP is powerful and fast, it shares a lot of similarities with Snabb, however they take very different approaches. I feel like the big differences are largely personal preferences: do you prefer C or Lua? Do you need an IP router? Are your network cards supported by Snabb or do you need to built them or use DPDK?
I think overall, during my excursion in VPP. I’ve enjoyed seeing the other side as it were. I do however, think I prefer working with Snabb. I find it faster to get up to speed with because things are simpler for simple projects. Lua also lets me have the power of C though it’s FFI, without subjecting me to the shortcomings C. I look forward however to hacking on both and maybe even seeing some of the ideas from the VPP world come into the Snabb world and the other-way round too!
At Igalia we are very proud of being a part of this: on the driver side, we have contributed the implementation of VK_KHR_16bit_storage, numerous bugfixes for issues raised by the Khronos Conformance Test Suite (CTS) and code reviews for some of the new Vulkan 1.1 features developed by Intel. On the CTS side, we have worked with other Khronos members in reviewing and testing additions to the test suite, identifying and providing fixes for issues in the tests as well as developing new tests.
Finally, I’d like to highlight the strong industry adoption of Vulkan: as stated in the Khronos press release, various other hardware vendors have already implemented conformant Vulkan 1.1 drivers, we are also seeing major 3D engines adopting and supporting Vulkan and AAA games that have already shipped with Vulkan-powered graphics. There is no doubt that this is only the beginning and that we will be seeing a lot more of Vulkan in the coming years, so look forward to it!
Vulkan and the Vulkan logo are registered trademarks of the Khronos Group Inc.
So far, I haven’t been involved with the extension development myself, but as I am looking forward to work on it in the future, I decided to familiarize myself with the use of SPIR-V in OpenGL. In this post, I will try to demonstrate with a short OpenGL example how to use this extension after I briefly explain what SPIR-V is, and why it is important to bring it to OpenGL.
With SPIR-V, the graphics program and the driver can avoid the overhead of parsing, compiling and linking the shaders. Also, it is easy to re-use shaders written in different shading languages. For example an OpenGL program for Linux can use an HLSL shader that was originally written for a Vulkan program for Windows, by loading its SPIR-V representation. The only requirement is that the OpenGL implementation and the driver support the ARB_gl_spirv extension.
OpenGL – SPIR-V example
So, here’s an example OpenGL program that loads SPIR-V shaders by making use of the OpenGL ARB_gl_spirv extension: https://github.com/hikiko/gl4
The example makes use of a vertex and a pixel shader written in GLSL 450. Some notes on it:
1- I had to write the shaders in a way compatible with the SPIR-V extension:
First of all, I had to forget about the traditional attribute locations, varyings and uniform locations. Each shader had to contain all the necessary information for linking. This means that I had to specify which are the input and output attributes/varyings at each shader stage and their locations/binding points inside the GLSL program. I’ve done this using the layout qualifier. I also placed the uniforms in Uniform Buffer Objects (UBO) as standalone uniforms are not supported when using SPIR-V 1. I couldn’t use UniformBlockBinding because again is not supported when using SPIR-V shaders (see ARB_gl_spirv : issues : 24 to learn why).
In the following tables you can see a side-by-side comparison of the traditional GLSL shaders I would use with older GLSL versions (left column) and the GLSL 450 shaders I used for this example (right column):
As you can see, in the modern GLSL version I’ve set the location for every attribute and varying as well as the binding number for the uniform buffer objects and the opaque types of the fragment shader (sampler2D) inside the shader. Also, the output varyings of the vertex stage (out variables) use the same locations with the equivalent input varyings of the fragment stage (in variables). There are also some minor changes in the language syntax: I can’t make use of the deprecated gl_FragColor and texture2D functions anymore.
2- GLSL to SPIR-V HOWTO:
First of all we need a GLSL to SPIR-V compiler.
On Linux, we can use the Khronos’s glslangValidator by checking out the code from this repository: https://github.com/KhronosGroup/glslang and installing it locally. Then, we can do something like:
for each stage (glslangValidator -h for more options). Note that -G is used to compile the shaders targeting the OpenGL platform. It should be avoided when the shaders will be ported to other platforms.
where sdr is our shader, buf is a buffer that contains the SPIR-V we loaded from the file and fsz the contents size (file size). Check out the load_shader function here: https://github.com/hikiko/gl4/blob/master/main.c (the case when SPIRV is defined).
We can then specialize the shaders using the function glSpecializeShaderARB that allows us to set the shader’s entry point (which is the function from which the execution begins) and the number of constants as well as the constants that will be used by this function. In our example the execution starts from the main function that is void, therefore we set "main", 0, 0 for each shader. Note that I load the glSpecializeShaderARB function at runtime because the linker couldn’t find it, you might not need to do this in your program.
Before we create the shader program it’s generally useful to perform some error checking:
In the code snippet above, I used the driver’s compiler (as we would use it if we had just compiled the GLSL code) to validate the shader’s SPIR-V representation.
Then, I created and used the shader program as usual (see load_program of main.c). And that’s it.
To run the example program you can clone it from here: https://github.com/hikiko/gl4, and supposing that you have installed the glslangValidator mentioned before you can run:
make
./test
inside the gl4/ directory.
[1]:Standalone uniforms with explicit locations can also be accepted but since this feature is not supported in other platforms (like Vulkan) the shaders that use it won’t be portable.
Are you tired of waiting for ages to build large C++ projects like WebKit? Slow headers are generally the problem. Your C++ source code file #includes a few headers, all those headers #include more, and those headers #include more, and more, and more, and since it’s C++ a bunch of these headers contain lots of complex templates to slow down things even more. Not fun.
It turns out that much of the time spent building large C++ projects is effectively spent parsing the same headers again and again, over, and over, and over, and over, and over….
There are three possible solutions to this problem:
Shred your CPU and buy a new one that’s twice as fast.
Use C++ modules: import instead of #include. This will soon become the best solution, but it’s not standardized yet. For WebKit’s purposes, we can’t use it until it works the same in MSVCC, Clang, and three-year-old versions of GCC. So it’ll be quite a while before we’re able to take advantage of modules.
Use unified builds (sometimes called unity builds).
WebKit has adopted unified builds. This work was done by Keith Miller, from Apple. Thanks, Keith! (If you’ve built WebKit before, you’ll probably want to say that again: thanks, Keith!)
For a release build of WebKitGTK+, on my desktop, our build times used to look like this:
real 62m49.535s
user 407m56.558s
sys 62m17.166s
That was taken using WebKitGTK+ 2.17.90; build times with any 2.18 release would be similar. Now, with trunk (or WebKitGTK+ 2.20, which will be very similar), our build times look like this:
real 33m36.435s
user 214m9.971s
sys 29m55.811s
Twice as fast.
The approach is pretty simple: instead of telling the compiler to build the original C++ source code files that developers see, we instead tell the compiler to build unified source files that look like this:
Since files are included only once per translation unit, we now have to parse the same headers only once for each unified source file, rather than for each individual original source file, and we get a dramatic build speedup. It’s pretty terrible, yet extremely effective.
Now, how many original C++ files should you #include in each unified source file? To get the fastest clean build time, you would want to #include all of your C++ source files in one, that way the compiler sees each header only once. (Meson can do this for you automatically!) But that causes two problems. First, you have to make sure none of the files throughout your entire codebase use conflicting variable names, since the static keyword and anonymous namespaces no longer work to restrict your definitions to a single file. That’s impractical in a large project like WebKit. Second, because there’s now only one file passed to the compiler, incremental builds now take as long as clean builds, which is not fun if you are a WebKit developer and actually need to make changes to it. Unifying more files together will always make incremental builds slower. After some experimentation, Apple determined that, for WebKit, the optimal number of files to include together is roughly eight. At this point, there’s not yet much negative impact on incremental builds, and past here there are diminishing returns in clean build improvement.
In WebKit’s implementation, the files to bundle together are computed automatically at build time using CMake black magic. Adding a new file to the build can change how the files are bundled together, potentially causing build errors in different files if there are symbol clashes. But this is usually easy to fix, because only files from the same directory are bundled together, so random unrelated files will never be built together. The bundles are always the same for everyone building the same version of WebKit, so you won’t see random build failures; only developers who are adding new files will ever have to deal with name conflicts.
To significantly reduce name conflicts, we now limit the scope of using statements. That is, stuff like this:
using namespace JavaScriptCore;
namespace WebCore {
//...
}
Has been changed to this:
namespace WebCore {
using namespace JavaScriptCore;
// ...
}
Some files need to be excluded due to unsolvable name clashes. For example, files that include X11 headers, which contain lots of unnamespaced symbols that conflict with WebCore symbols, don’t really have any chance. But only a few files should need to be excluded, so this does not have much impact on build time. We’ve also opted to not unify most of the GLib API layer, so that we can continue to use conventional GObject names in our implementation, but again, the impact of not unifying a few files is minimal.
We still have some room for further performance improvement, because some significant parts of the build are still not unified, including most of the WebKit layer on top. But I suspect developers who have to regularly build WebKit will already be quite pleased.
So, how do you write a shebang for a Python program? Let’s first set aside the python2/python3 issue and focus on whether to use env. Which of the following is correct?
#!/usr/bin/env python
#!/usr/bin/python
The first option seems to work in all environments, but it is banned in popular distros like Fedora (and I believe also Debian, but I can’t find a reference for this). Using env in shebangs is dangerous because it can result in system packages using non-system versions of python. python is used in so many places throughout modern systems, it’s not hard to see how using #!/usr/bin/env in an important package could badly bork users’ operating systems if they install a custom version of python in /usr/local. Don’t do this.
The second option is broken too, because it doesn’t work in BSD environments. E.g. in FreeBSD, python is installed in /usr/local/bin. So FreeBSD contributors have been upstreaming patches to convert #!/usr/bin/python shebangs to #!/usr/bin/env python. Meanwhile, Fedora has begun automatically rewriting #!/usr/bin/env python to #!/usr/bin/python, but with a warning that this is temporary and that use of #!/usr/bin/env python will eventually become a fatal error causing package builds to fail.
So obviously there’s no way to write a shebang that will work for both major Linux distros and major BSDs. #!/usr/bin/env python seems to work today, but it’s subtly very dangerous. Lovely. I don’t even know what to recommend to upstream projects.
Next problem: python2 versus python3. By now, we should all be well-aware of PEP 394. PEP 394 says you should never write a shebang like this:
#!/usr/bin/env python
#!/usr/bin/python
unless your python script is compatible with both python2 and python3, because you don’t know what version you’re getting. Your python script is almost certainly not compatible with both python2 and python3 (and if you think it is, it’s probably somehow broken, because I doubt you regularly test it with both). Instead, you should write the shebang like this:
#!/usr/bin/env python2
#!/usr/bin/python2
#!/usr/bin/env python3
#!/usr/bin/python3
This works as long as you only care about Linux and BSDs. It doesn’t work on macOS, which provides /usr/bin/python and /usr/bin/python2.7, but still no /usr/bin/python2 symlink, even though it’s now been six years since PEP 394. It’s hard to understate how frustrating this is.
So let’s say you are WebKit, and need to write a python script that will be truly cross-platform. How do you do it? WebKit’s scripts are only needed (a) during the build process or (b) by developers, so we get a pass on the first problem: using /usr/bin/env should be OK, because the scripts should never be installed as part of the OS. Using #!/usr/bin/env python — which is actually what we currently do — is unacceptable, because our scripts are python2 and that’s broken on Arch, and some of our developers use that. Using #!/usr/bin/env python2 would be dead on arrival, because that doesn’t work on macOS. Seems like the option that works for everyone is #!/usr/bin/env python2.7. Then we just have to hope that the Python community sticks to its promise to never release a python2.8 (which seems likely).
Some time ago I started a series of blog posts about IPv6 and network namespaces. The purpose of those posts was preparing the ground for covering a network function called B4 (Basic Bridging BroadBand).
The B4 network function is one of the main components of a lw4o6 architecture (RFC7596). The function runs within every CPEs (Customer’s Premises Equipment, essentially a home router) of a carrier’s network. This function takes care of two things: 1) NAPT the customer’s IPv4 traffic and 2) encapsulate it into IPv6. This is fundamental as lw4o6 proposes an IPv6-only network, which can still provide IPv4 services and connectivity. Besides lw4o6, the B4 function is also present in other architectures such as DS-Lite or MAP-E. In the case of lw4o6 the exact name of this function is lwB4. All these architectures rely on A+P mapping techniques and are managed by the Softwire WG.
The diagram below shows how a lw4o6 architecture works:
lw4o6 chart
Packets arriving the CPE from the customer (IPv4) are shown in red. Packets leaving the CPE to the carrier’s network are shown in blue (IPv6). The counterpart of a lwB4 function is the lwAFTR function, deployed at one of the border-routers of the carrrier’s network.
In the article ‘Dive into lw4o6’ I reviewed in detail how a lw4o6 architecture works. Please check out the article if you want to learn more.
At Igalia we implemented a high-performant lwAFTR network function. This network function has been part of Snabb since at least 2015, and has kept evolving and getting merged back to Snabb through new releases. We kindly thank Deutsche Telekom for their support financing this project, as well as Juniper networks, who also helped improving the status of Snabb’s lwAFTR.
While we were developing the lwAFTR network function we heavily tested it through a wide range of tests: end-to-end tests, performance tests, soak tests, etc. However, in some occassions we got to diagnose potential bugs in real deployments. To do that, we needed the other major component of a lw4o6 architecture: the B4 network function.
OpenWRT, the Linux-based OS powering many home routers, features a MAP network function to help deploying MAP-E architectures. This function can also be used to implement a B4 for DS-Lite or lw4o6. With the invaluable help of my colleagues Adrián and Carlos López I managed to setup an OpenWRT on a virtual machine with B4 enabled. However, I was not completely satisfied with the solution.
That led me to explore another solution very much inspired by an excelent blog post from Marcel Wiget: Lightweight 4over6 B4 Client in Linux Namespace. In this post Marcel describes how to build a B4 network function using standard Linux commands.
Basically, a B4 function does 2 things:
NAT44, which is possible to do it with iptables.
IPv4-in-IPv6 tunneling, which is possible to do it iproute2.
In addition, Marcel’s B4 network function is isolated into its own network namespace. That’s less of a headache than installing and configuring a virtual machine.
On the other hand, my deployment had a extra twist compared to a standard lw4o6 deployment. The lwAFTR I was trying to reach was somewhere out on the Internet, not within my ISP’s network. To make things worse, ISP providers in Spain are not rolling out IPv6 yet so I needed to use an IPv6 tunnel broker, more precisely Hurricane Electric (In the article ‘IPv6 tunnel’ I described how to set up such tunnel).
Basically my deployment looked like this:
lwB4-lwAFTR over Internet
After scratching my head during several days I came up with the following script: b4-to-aftr-over-inet.sh. I break it down below in pieces for better comprehension.
Warning: The script requires a Hurricane Electric tunnel up and running in order to work.
Our B4 would have the following provisioned data:
B4 IPv6: IPv6 address provided by Hurricane Electric.
B4 IPv4: 192.0.2.1
B4 port-range: 4096-8191
While the address of the AFTR is 2001:DB8::0001.
Given this settings our B4 is ready to match the following softwire in the lwAFTR’s binding table:
Definition of several constants. IPHT and IPNS stand for IP host and IP namespace. Our script will create a network namespace which requires a veth pair to communicate the namespace with the host. IPHT is an ULA address for the host side, while IPNS is an ULA address for the network namespace side. Likewise, IFHT and IFNS are the interface names for host and namespace sides respectively.
IFHE is the interface of the Hurricane Electric IPv6-in-IPv4 tunnel. We will use the IPv6 address of this interface as IPv6 source address of the B4.
# Reset everything
ip li del dev "${IFHT}" &>/dev/null
ip netns del "${NS}" &> /dev/null
Removes namespace and host-side interface if defined.
# Create a network namespace and enable loopback on it
ip netns add "${NS}"
ip netns exec "${NS}" ip li set dev lo up
# Create the veth pair and move one of the ends to the NS.
ip li add name "${IFHT}" type veth peer name "${IFNS}"
ip li set dev "${IFNS}" netns "${NS}"
# Configure interface ${IFHT} on the host
ip -6 addr add "${IPHT}/${CID}" dev "${IFHT}"
ip li set dev "${IFHT}" up
# Configure interface ${IFNS} on the network namespace.
ip netns exec "${NS}" ip -6 addr add "${IPNS}/${CID}" dev "${IFNS}"
ip netns exec "${NS}" ip li set dev "${IFNS}" up
The commands above set up the basics of the network namespace. A network namespace is created with two virtual-interface pairs (think of a patch cable). Each of the veth ends is assigned a private IPv6 address (ULA). One of the ends of the veth pair is moved into the network namespace while the other remains on the host side. In case of doubt, please check this other article I wrote about network namespaces.
# Create IPv4-in-IPv6 tunnel.
ip netns exec "${NS}" ip -6 tunnel add b4tun mode ipip6 local "${IPNS}" remote "${IPHT}" dev "${IFNS}"
ip netns exec "${NS}" ip addr add 10.0.0.1 dev b4tun
ip netns exec "${NS}" ip link set dev b4tun up
# All IPv4 packets go through the tunnel.
ip netns exec "${NS}" ip route add default dev b4tun
# Make ${IFNS} the default gw.
ip netns exec "${NS}" ip -6 route add default dev "${IFNS}"
From the B4 we will send IPv4 packets that will get encapsulated into IPv6. These packets will eventually leave the host via the Hurricane Electric tunnel. What we do here is to create an IPv4-in-IPv6 tunnel (ipip6) called b4tun. The tunnel has two ends: IPNS and IPHT. All IPv4 traffic started from the network namespace gets routed through b4tun, so it gets encapsulated. If the traffic if IPv6 native traffic it doesn’t need to get encapsulated, thus it’s simply forwarded to IFNS.
# Adjust MTU size.
ip netns exec "${NS}" ip li set mtu 1252 dev b4tun
ip netns exec "${NS}" ip li set mtu 1300 dev vpeer9
Since packets leaving the CPE get IPv6 encapsulated we need to make room for those extra bytes that will grow the packet size. Normally routing appliances have a default MTU size of 1500 bytes. That’s why we artificially reduce the MTU size of both b4tun and vpeer9 interfaces to a number lower than 1500. This technique is known as MSS (Maximum Segment Size) Clamping.
# NAT44.
ip netns exec "${NS}" iptables -t nat --flush
ip netns exec "${NS}" iptables -t nat -A POSTROUTING -p tcp -o b4tun -j SNAT --to $IP:$PORTRANGE
ip netns exec "${NS}" iptables -t nat -A POSTROUTING -p udp -o b4tun -j SNAT --to $IP:$PORTRANGE
ip netns exec "${NS}" iptables -t nat -A POSTROUTING -p icmp -o b4tun -j SNAT --to $IP:$PORTRANGE
Outgoing IPv4 packets leaving the B4 got their IPv4 source address and port sourced natted. The block of code above flushes the iptables’s NAT44 rules and creates new Source NAT rules for several protocols.
# Enable forwarding and IPv6 NAT
sysctl -w net.ipv6.conf.all.forwarding=1
ip6tables -t nat --flush
# Packets coming into the veth pair in the host side, change their destination address to AFTR.
ip6tables -t nat -A PREROUTING -i "${IFHT}" -j DNAT --to-destination "${AFTR_IPV6}"
# Outgoing packets change their source address to HE Client address (B4 address).
ip6tables -t nat -A POSTROUTING -o "${IFHE}" -j MASQUERADE
Outgoing packets leaving our host need to get their source address masqueraded to the IPv6 address assigned to the interface of the Hurricane Electric tunnel point. Likewise anything that comes into the host should seem to arrive from the lwAFTR, when actually its origin address is the IPv6 address of the other end of the Hurricane Electric tunnel. To overcome this problem I applied a NAT66 on source address and destination. Could this be done in a different way skipping the controversial NAT66? I’m not sure. I think the veth pairs need to get assigned private addresses so the only way to get the packets routed through the Internet is with a NAT66.
# Get into NS.
bash=/run/current-system/sw/bin/bash
ip netns exec ${NS} ${bash} --rcfile <(echo "PS1=\"${NS}> \"")
The last step gets us into the network namespace from which we will be able to run commands constrained into the environment created during the steps before.
I don’t know how much useful or reusable this script can be, but in hindsight coming up with this complex setting helped me learning several Linux networking tools. I think I could have never figured all this out without the help and support from my colleagues as well as the guidance from Marcel’s original script and blog post.
Ahoy, programming-language tinkerfolk! Today's rambling missive chews the gnarly bones of "inline caches", in general but also with particular respect to the Guile implementation of Scheme. First, a little intro.
inline what?
Inline caches are a language implementation technique used to accelerate polymorphic dispatch. Let's dive in to that.
By implementation technique, I mean that the technique applies to the language compiler and runtime, rather than to the semantics of the language itself. The effects on the language do exist though in an indirect way, in the sense that inline caches can make some operations faster and therefore more common. Eventually inline caches can affect what users expect out of a language and what kinds of programs they write.
But I'm getting ahead of myself. Polymorphic dispatch literally means "choosing based on multiple forms". Let's say your language has immutable strings -- like Java, Python, or Javascript. Let's say your language also has operator overloading, and that it uses + to concatenate strings. Well at that point you have a problem -- while you can specify a terse semantics of some core set of operations on strings (win!), you can't choose one representation of strings that will work well for all cases (lose!). If the user has a workload where they regularly build up strings by concatenating them, you will want to store strings as trees of substrings. On the other hand if they want to access characterscodepoints by index, then you want an array. But if the codepoints are all below 256, maybe you should represent them as bytes to save space, whereas maybe instead as 4-byte codepoints otherwise? Or maybe even UTF-8 with a codepoint index side table.
The right representation (form) of a string depends on the myriad ways that the string might be used. The string-append operation is polymorphic, in the sense that the precise code for the operator depends on the representation of the operands -- despite the fact that the meaning of string-append is monomorphic!
Anyway, that's the problem. Before inline caches came along, there were two solutions: callouts and open-coding. Both were bad in similar ways. A callout is where the compiler generates a call to a generic runtime routine. The runtime routine will be able to handle all the myriad forms and combination of forms of the operands. This works fine but can be a bit slow, as all callouts for a given operator (e.g. string-append) dispatch to a single routine for the whole program, so they don't get to optimize for any particular call site.
One tempting thing for compiler writers to do is to effectively inline the string-append operation into each of its call sites. This is "open-coding" (in the terminology of the early Lisp implementations like MACLISP). The advantage here is that maybe the compiler knows something about one or more of the operands, so it can eliminate some cases, effectively performing some compile-time specialization. But this is a limited technique; one could argue that the whole point of polymorphism is to allow for generic operations on generic data, so you rarely have compile-time invariants that can allow you to specialize. Open-coding of polymorphic operations instead leads to code bloat, as the string-append operation is just so many copies of the same thing.
Inline caches emerged to solve this problem. They trace their lineage back to Smalltalk 80, gained in complexity and power with Self and finally reached mass consciousness through Javascript. These languages all share the characteristic of being dynamically typed and object-oriented. When a user evaluates a statement like x = y.z, the language implementation needs to figure out where y.z is actually located. This location depends on the representation of y, which is rarely known at compile-time.
However for any given reference y.z in the source code, there is a finite set of concrete representations of y that will actually flow to that call site at run-time. Inline caches allow the language implementation to specialize the y.z access for its particular call site. For example, at some point in the evaluation of a program, y may be seen to have representation R1 or R2. For R1, the z property may be stored at offset 3 within the object's storage, and for R2 it might be at offset 4. The inline cache is a bit of specialized code that compares the type of the object being accessed against R1 , in that case returning the value at offset 3, otherwise R2 and offset r4, and otherwise falling back to a generic routine. If this isn't clear to you, Vyacheslav Egorov write a fine article describing and implementing the object representation optimizations enabled by inline caches.
Inline caches also serve as input data to later stages of an adaptive compiler, allowing the compiler to selectively inline (open-code) only those cases that are appropriate to values actually seen at any given call site.
but how?
The classic formulation of inline caches from Self and early V8 actually patched the code being executed. An inline cache might be allocated at address 0xcabba9e5 and the code emitted for its call-site would be jmp 0xcabba9e5. If the inline cache ended up bottoming out to the generic routine, a new inline cache would be generated that added an implementation appropriate to the newly seen "form" of the operands and the call-site. Let's say that new IC (inline cache) would have the address 0x900db334. Early versions of V8 would actually patch the machine code at the call-site to be jmp 0x900db334 instead of jmp 0xcabba6e5.
Patching machine code has a number of disadvantages, though. It inherently target-specific: you will need different strategies to patch x86-64 and armv7 machine code. It's also expensive: you have to flush the instruction cache after the patch, which slows you down. That is, of course, if you are allowed to patch executable code; on many systems that's impossible. Writable machine code is a potential vulnerability if the system may be vulnerable to remote code execution.
Perhaps worst of all, though, patching machine code is not thread-safe. In the case of early Javascript, this perhaps wasn't so important; but as JS implementations gained parallel garbage collectors and JS-level parallelism via "service workers", this becomes less acceptable.
For all of these reasons, the modern take on inline caches is to implement them as a memory location that can be atomically modified. The call site is just jmp *loc, as if it were a virtual method call. Modern CPUs have "branch target buffers" that predict the target of these indirect branches with very high accuracy so that the indirect jump does not become a pipeline stall. (What does this mean in the face of the Spectre v2 vulnerabilities? Sadly, God only knows at this point. Saddest panda.)
cry, the beloved country
I am interested in ICs in the context of the Guile implementation of Scheme, but first I will make a digression. Scheme is a very monomorphic language. Yet, this monomorphism is entirely cultural. It is in no way essential. Lack of ICs in implementations has actually fed back and encouraged this monomorphism.
Let us take as an example the case of property access. If you have a pair in Scheme and you want its first field, you do (car x). But if you have a vector, you do (vector-ref x 0).
What's the reason for this nonuniformity? You could have a generic ref procedure, which when invoked as (ref x 0) would return the field in x associated with 0. Or (ref x 'foo) to return the foo property of x. It would be more orthogonal in some ways, and it's completely valid Scheme.
We don't write Scheme programs this way, though. From what I can tell, it's for two reasons: one good, and one bad.
The good reason is that saying vector-ref means more to the reader. You know more about the complexity of the operation and what side effects it might have. When you call ref, who knows? Using concrete primitives allows for better program analysis and understanding.
The bad reason is that Scheme implementations, Guile included, tend to compile (car x) to much better code than (ref x 0). Scheme implementations in practice aren't well-equipped for polymorphic data access. In fact it is standard Scheme practice to abuse the "macro" facility to manually inline code so that that certain performance-sensitive operations get inlined into a closed graph of monomorphic operators with no callouts. To the extent that this is true, Scheme programmers, Scheme programs, and the Scheme language as a whole are all victims of their implementations. JavaScript, for example, does not have this problem -- to a small extent, maybe, yes, performance tweaks and tuning are always a thing but JavaScript implementations' ability to burn away polymorphism and abstraction results in an entirely different character in JS programs versus Scheme programs.
it gets worse
On the most basic level, Scheme is the call-by-value lambda calculus. It's well-studied, well-understood, and eminently flexible. However the way that the syntax maps to the semantics hides a constrictive monomorphism: that the "callee" of a call refer to a lambda expression.
Concretely, in an expression like (a b), in which a is not a macro, a must evaluate to the result of a lambda expression. Perhaps by reference (e.g. (define a (lambda (x) x))), perhaps directly; but a lambda nonetheless. But what if a is actually a vector? At that point the Scheme language standard would declare that to be an error.
The semantics of Clojure, though, would allow for ((vector 'a 'b 'c) 1) to evaluate to b. Why not in Scheme? There are the same good and bad reasons as with ref. Usually, the concerns of the language implementation dominate, regardless of those of the users who generally want to write terse code. Of course in some cases the implementation concerns should dominate, but not always. Here, Scheme could be more flexible if it wanted to.
what have you done for me lately
Although inline caches are not a miracle cure for performance overheads of polymorphic dispatch, they are a tool in the box. But what, precisely, can they do, both in general and for Scheme?
To my mind, they have five uses. If you can think of more, please let me know in the comments.
Firstly, they have the classic named property access optimizations as in JavaScript. These apply less to Scheme, as we don't have generic property access. Perhaps this is a deficiency of Scheme, but it's not exactly low-hanging fruit. Perhaps this would be more interesting if Guile had more generic protocols such as Racket's iteration.
Next, there are the arithmetic operators: addition, multiplication, and so on. Scheme's arithmetic is indeed polymorphic; the addition operator + can add any number of complex numbers, with a distinction between exact and inexact values. On a representation level, Guile has fixnums (small exact integers, no heap allocation), bignums (arbitrary-precision heap-allocated exact integers), fractions (exact ratios between integers), flonums (heap-allocated double-precision floating point numbers), and compnums (inexact complex numbers, internally a pair of doubles). Also in Guile, arithmetic operators are a "primitive generics", meaning that they can be extended to operate on new types at runtime via GOOPS.
The usual situation though is that any particular instance of an addition operator only sees fixnums. In that case, it makes sense to only emit code for fixnums, instead of the product of all possible numeric representations. This is a clear application where inline caches can be interesting to Guile.
Third, there is a very specific case related to dynamic linking. Did you know that most programs compiled for GNU/Linux and related systems have inline caches in them? It's a bit weird but the "Procedure Linkage Table" (PLT) segment in ELF binaries on Linux systems is set up in a way that when e.g. libfoo.so is loaded, the dynamic linker usually doesn't eagerly resolve all of the external routines that libfoo.so uses. The first time that libfoo.so calls frobulate, it ends up calling a procedure that looks up the location of the frobulate procedure, then patches the binary code in the PLT so that the next time frobulate is called, it dispatches directly. To dynamic language people it's the weirdest thing in the world that the C/C++/everything-static universe has at its cold, cold heart a hash table and a dynamic dispatch system that it doesn't expose to any kind of user for instrumenting or introspection -- any user that's not a malware author, of course.
But I digress! Guile can use ICs to lazily resolve runtime routines used by compiled Scheme code. But perhaps this isn't optimal, as the set of primitive runtime calls that Guile will embed in its output is finite, and so resolving these routines eagerly would probably be sufficient. Guile could use ICs for inter-module references as well, and these should indeed be resolved lazily; but I don't know, perhaps the current strategy of using a call-site cache for inter-module references is sufficient.
Fourthly (are you counting?), there is a general case of the former: when you see a call (a b) and you don't know what a is. If you put an inline cache in the call, instead of having to emit checks that a is a heap object and a procedure and then emit an indirect call to the procedure's code, you might be able to emit simply a check that a is the same as x, the only callee you ever saw at that site, and in that case you can emit a direct branch to the function's code instead of an indirect branch.
Here I think the argument is less strong. Modern CPUs are already very good at indirect jumps and well-predicted branches. The value of a devirtualization pass in compilers is that it makes the side effects of a virtual method call concrete, allowing for more optimizations; avoiding indirect branches is good but not necessary. On the other hand, Guile does have polymorphic callees (generic functions), and call ICs could help there. Ideally though we would need to extend the language to allow generic functions to feed back to their inline cache handlers.
Finally, ICs could allow for cheap tracepoints and breakpoints. If at every breakable location you included a jmp *loc, and the initial value of *loc was the next instruction, then you could patch individual locations with code to run there. The patched code would be responsible for saving and restoring machine state around the instrumentation.
Honestly I struggle a lot with the idea of debugging native code. GDB does the least-overhead, most-generic thing, which is patching code directly; but it runs from a separate process, and in Guile we need in-process portable debugging. The debugging use case is a clear area where you want adaptive optimization, so that you can emit debugging ceremony from the hottest code, knowing that you can fall back on some earlier tier. Perhaps Guile should bite the bullet and go this way too.
implementation plan
In Guile, monomorphic as it is in most things, probably only arithmetic is worth the trouble of inline caches, at least in the short term.
Another question is how much to specialize the inline caches to their call site. On the extreme side, each call site could have a custom calling convention: if the first operand is in register A and the second is in register B and they are expected to be fixnums, and the result goes in register C, and the continuation is the code at L, well then you generate an inline cache that specializes to all of that. No need to shuffle operands or results, no need to save the continuation (return location) on the stack.
The opposite would be to call ICs as if their were normal procedures: shuffle arguments into fixed operand registers, push a stack frame, and when the IC returns, shuffle the result into place.
Honestly I am looking mostly to the simple solution. I am concerned about code and heap bloat if I specify to every last detail of a call site. Also maximum speed comes with an adaptive optimizer, and in that case simple lower tiers are best.
sanity check
To compare these impressions, I took a look at V8's current source code to see where they use ICs in practice. When I worked on V8, the compiler was entirely different -- there were two tiers, and both of them generated native code. Inline caches were everywhere, and they were gnarly; every architecture had its own implementation. Now in V8 there are two tiers, not the same as the old ones, and the lowest one is a bytecode interpreter.
As an adaptive optimizer, V8 doesn't need breakpoint ICs. It can always deoptimize back to the interpreter. In actual practice, to debug at a source location, V8 will patch the bytecode to insert a "DebugBreak" instruction, which has its own support in the interpreter. V8 also supports optimized compilation of this operation. So, no ICs needed here.
Likewise for generic type feedback, V8 records types as data rather than in the classic formulation of inline caches as in Self. I think WebKit's JavaScriptCore uses a similar strategy.
V8 does use inline caches for property access (loads and stores). Besides that there is an inline cache used in calls which is just used to record callee counts, and not used for direct call optimization.
Surprisingly, V8 doesn't even seem to use inline caches for arithmetic (any more?). Fair enough, I guess, given that JavaScript's numbers aren't very polymorphic, and even with a system with fixnums and heap floats like V8, floating-point numbers are rare in cold code.
The dynamic linking and relocation points don't apply to V8 either, as it doesn't receive binary code from the internet; it always starts from source.
twilight of the inline cache
There was a time when inline caches were recommended to solve all your VM problems, but it would seem now that their heyday is past.
ICs are still a win if you have named property access on objects whose shape you don't know at compile-time. But improvements in CPU branch target buffers mean that it's no longer imperative to use ICs to avoid indirect branches (modulo Spectre v2), and creating direct branches via code-patching has gotten more expensive and tricky on today's targets with concurrency and deep cache hierarchies.
Besides that, the type feedback component of inline caches seems to be taken over by explicit data-driven call-site caches, rather than executable inline caches, and the highest-throughput tiers of an adaptive optimizer burn away inline caches anyway. The pressure on an inline cache infrastructure now is towards simplicity and ease of type and call-count profiling, leaving the speed component to those higher tiers.
In Guile the bounded polymorphism on arithmetic combined with the need for ahead-of-time compilation means that ICs are probably a code size and execution time win, but it will take some engineering to prevent the calling convention overhead from dominating cost.
Time to experiment, then -- I'll let y'all know how it goes. Thoughts and feedback welcome from the compilerati. Until then, happy hacking :)
I am on my way back from FOSDEM and thought I would share with yall some impressions from talks in the Networking devroom. I didn't get to go to all that many talks -- FOSDEM's hallway track is the hottest of them all -- but I did hit a select few. Thanks to Dave Neary at Red Hat for organizing the room.
The day started with a drum-beating talk that was very light on technical information.
Essentially Ray was arguing for an evolution of network function virtualization -- that instead of running VNFs on bare metal as was done in the days of yore, that people started to run them in virtual machines, and now they run them in containers -- what's next? Ray is saying that "cloud-native VNFs" are the next step.
Cloud-native VNFs to move from "greedy" VNFs that take charge of the cores that are available to them, to some kind of resource sharing. "Maybe users value flexibility over performance", says Ray. It's the Care Bears approach to networking: (resource) sharing is caring.
In practice he proposed two ways that VNFs can map to cores and cards.
One was in-process sharing, which if I understood him properly was actually as nodes running within a VPP process. Basically in this case VPP or DPDK is the scheduler and multiplexes two or more network functions in one process.
The other was letting Linux schedule separate processes. In networking, we don't usually do it this way: we run network functions on dedicated cores on which nothing else runs. Ray was suggesting that perhaps network functions could be more like "normal" Linux services. Ray doesn't know if Linux scheduling will work in practice. Also it might mean allowing DPDK to work with 4K pages instead of the 2M hugepages it currently requires. This obviously has the potential for more latency hazards and would need some tighter engineering, and ultimately would have fewer guarantees than the "greedy" approach.
Interesting side things I noticed:
All the diagrams show Kubernetes managing CPU node allocation and interface assignment. I guess in marketing diagrams, Kubernetes has completely replaced OpenStack.
One slide showed guest VNFs differentiated between "virtual network functions" and "socket-based applications", the latter ones being the legacy services that use kernel APIs. It's a useful terminology difference.
The talk identifies user-space networking with DPDK (only!).
Finally, I note that Conway's law is obviously reflected in the performance overheads: because there are organizational isolations between dev teams, vendors, and users, there are big technical barriers between them too. The least-overhead forms of resource sharing are also those with the highest technical consistency and integration (nodes in a single VPP instance).
This was a talk about getting good throughput from the NIC to userspace, but by using some kernel facilities. The idea is to get the kernel to set up the NIC and virtualize the transmit and receive ring buffers, but to let the NIC's DMA'd packets go directly to userspace.
The performance goal is 40Gbps for thousand-byte packets, or 25 Gbps for traffic with only the smallest packets (64 bytes). The fast path does "zero copy" on the packets if the hardware has the capability to steer the subset of traffic associated with the AF_XDP socket to that particular process.
The AF_XDP project builds on XDP, a newish thing where a little kind of bytecode can run on the kernel or possibly on the NIC. One of the bytecode commands (REDIRECT) causes packets to be forwarded to user-space instead of handled by the kernel's otherwise heavyweight networking stack. AF_XDP is the bridge between XDP on the kernel side and an interface to user-space using sockets (as opposed to e.g. AF_INET). The performance goal was to be within 10% or so of DPDK's raw user-space-only performance.
The benefits of AF_XDP over the current situation would be that you have just one device driver, in the kernel, rather than having to have one driver in the kernel (which you have to have anyway) and one in user-space (for speed). Also, with the kernel involved, there is a possibility for better isolation between different processes or containers, when compared with raw PCI access from user-space..
AF_XDP is what was previously known as AF_PACKET v4, and its numbers are looking somewhat OK. Though it's not upstream yet, it might be interesting to get a Snabb driver here.
I would note that kernel-userspace cooperation is a bit of a theme these days. There are other points of potential cooperation or common domain sharing, storage being an obvious one. However I heard more than once this weekend the kind of "I don't know, that area of the kernel has a different culture" sort of concern as that highlighted by Daniel Vetter in his recent LCA talk.
This talk is hard to summarize. Like the previous one, it's again about getting packets to userspace with some support from the kernel, but the speaker went really deep and I'm not quite sure what in the talk is new and what is known.
François-Frédéric is working on a new set of abstractions for relating the kernel and user-space. He works on OpenDataPlane (ODP), which is kinda like DPDK in some ways. ARM seems to be a big target for his work; that x86-64 is also a target goes without saying.
His problem statement was, how should we enable fast userland network I/O, without duplicating drivers?
François-Frédéric was a bit negative on AF_XDP because (he says) it is so focused on packets that it neglects other kinds of devices with similar needs, such as crypto accelerators. Apparently the challenge here is accelerating a single large IPsec tunnel -- because the cryptographic operations are serialized, you need good single-core performance, and making use of hardware accelerators seems necessary right now for even a single 10Gbps stream. (If you had many tunnels, you could parallelize, but that's not the case here.)
He was also a bit skeptical about standardizing on the "packet array I/O model" which AF_XDP and most NICS use. What he means here is that most current NICs move packets to and from main memory with the help of a "descriptor array" ring buffer that holds pointers to packets. A transmit array stores packets ready to transmit; a receive array stores maximum-sized packet buffers ready to be filled by the NIC. The packet data itself is somewhere else in memory; the descriptor only points to it. When a new packet is received, the NIC fills the corresponding packet buffer and then updates the "descriptor array" to point to the newly available packet. This requires at least two memory writes from the NIC to memory: at least one to write the packet data (one per 64 bytes of packet data), and one to update the DMA descriptor with the packet length and possible other metadata.
Although these writes go directly to cache, there's a limit to the number of DMA operations that can happen per second, and with 100Gbps cards, we can't afford to make one such transaction per packet.
François-Frédéric promoted an alternative I/O model for high-throughput use cases: the "tape I/O model", where packets are just written back-to-back in a uniform array of memory. Every so often a block of memory containing some number of packets is made available to user-space. This has the advantage of packing in more packets per memory block, as there's no wasted space between packets. This increases cache density and decreases DMA transaction count for transferring packet data, as we can use each 64-byte DMA write to its fullest. Additionally there's no side table of descriptors to update, saving a DMA write there.
Apparently the only cards currently capable of 100 Gbps traffic, the Chelsio and Netcope cards, use the "tape I/O model".
Incidentally, the DMA transfer limit isn't the only constraint. Something I hadn't fully appreciated before was memory write bandwidth. Before, I had thought that because the NIC would transfer in packet data directly to cache, that this wouldn't necessarily cause any write traffic to RAM. Apparently that's not the case. Later over drinks (thanks to Red Hat's networking group for organizing), François-Frédéric asserted that the DMA transfers would eventually use up DDR4 bandwidth as well.
A NIC-to-RAM DMA transaction will write one cache line (usually 64 bytes) to the socket's last-level cache. This write will evict whatever was there before. As far as I can tell, there are three cases of interest here. The best case is where the evicted cache line is from a previous DMA transfer to the same address. In that case it's modified in the cache and not yet flushed to main memory, and we can just update the cache instead of flushing to RAM. (Do I misunderstand the way caches work here? Do let me know.)
However if the evicted cache line is from some other address, we might have to flush to RAM if the cache line is dirty. That causes a memory write traffic. But if the cache line is clean, that means it was probably loaded as part of a memory read operation, and then that means we're evicting part of the network function's working set, which will later cause memory read traffic as the data gets loaded in again, and write traffic to flush out the DMA'd packet data cache line.
François-Frédéric simplified the whole thing to equate packet bandwidth with memory write bandwidth, that yes, the packet goes directly to cache but it is also written to RAM. I can't convince myself that that's the case for all packets, but I need to look more into this.
Of course the cache pressure and the memory traffic is worse if the packet data is less compact in memory; and worse still if there is any need to copy data. Ultimately, processing small packets at 100Gbps is still a huge challenge for user-space networking, and it's no wonder that there are only a couple devices on the market that can do it reliably, not that I've seen either of them operate first-hand :)
Talking with Snabb's Luke Gorrie later on, he thought that it could be that we can still stretch the packet array I/O model for a while, given that PCIe gen4 is coming soon, which will increase the DMA transaction rate. So that's a possibility to keep in mind.
At the same time, apparently there are some "coherent interconnects" coming too which will allow the NIC's memory to be mapped into the "normal" address space available to the CPU. In this model, instead of having the NIC transfer packets to the CPU, the NIC's memory will be directly addressable from the CPU, as if it were part of RAM. The latency to pull data in from the NIC to cache is expected to be slightly longer than a RAM access; for comparison, RAM access takes about 70 nanoseconds.
For a user-space networking workload, coherent interconnects don't change much. You still need to get the packet data into cache. True, you do avoid the writeback to main memory, as the packet is already in addressable memory before it's in cache. But, if it's possible to keep the packet on the NIC -- like maybe you are able to add some kind of inline classifier on the NIC that could directly shunt a packet towards an on-board IPSec accelerator -- in that case you could avoid a lot of memory transfer. That appears to be the driving factor for coherent interconnects.
At some point in François-Frédéric's talk, my brain just died. I didn't quite understand all the complexities that he was taking into account. Later, after he kindly took the time to dispell some more of my ignorance, I understand more of it, though not yet all :) The concrete "deliverable" of the talk was a model for kernel modules and user-space drivers that uses the paradigms he was promoting. It's a work in progress from Linaro's networking group, with some support from NIC vendors and CPU manufacturers.
This talk had the most magnificent beginning: a sort of "repent now ye sinners" sermon from Luke Gorrie, a seasoned veteran of software networking. Luke started by describing the path of righteousness leading to "driver heaven", a world in which all vendors have publically accessible datasheets which parsimoniously describe what you need to get packets flowing. In this blessed land it's easy to write drivers, and for that reason there are many of them. Developers choose a driver based on their needs, or they write one themselves if their needs are quite specific.
But there is another path, says Luke, that of "driver hell": a world of wickedness and proprietary datasheets, where even when you buy the hardware, you can't program it unless you're buying a hundred thousand units, and even then you are smitten with the cursed non-disclosure agreements. In this inferno, only a vendor is practically empowered to write drivers, but their poor driver developers are only incentivized to get the driver out the door deployed on all nine architectural circles of driver hell. So they include some kind of circle-of-hell abstraction layer, resulting in a hundred thousand lines of code like a tangled frozen beard. We all saw the abyss and repented.
Luke described the process that led to Mellanox releasing the specification for its ConnectX line of cards, something that was warmly appreciated by the entire audience, users and driver developers included. Wonderful stuff.
My Igalia colleague Asumu Takikawa took the last half of the presentation, showing some code for the driver for the Intel i210, i350, and 82599 cards. For more on that, I recommend his recent blog post on user-space driver development. It was truly a ray of sunshine in dark, dark Brussels.
This talk was a delightful introduction to VPP, but without all of the marketing; the sort of talk that makes FOSDEM worthwhile. Usually at more commercial, vendory events, you can't really get close to the technical people unless you have a vendor relationship: they are surrounded by a phalanx of salesfolk. But in FOSDEM it is clear that we are all comrades out on the open source networking front.
The speaker expressed great personal pleasure on having being able to work on open source software; his relief was palpable. A nice moment.
He also had some kind words about Snabb, too, saying at one point that "of course you can do it on snabb as well -- Snabb and VPP are quite similar in their approach to life". He trolled the horrible complexity diagrams of many "NFV" stacks whose components reflect the org charts that produce them more than the needs of the network functions in question (service chaining anyone?).
He did get to drop some numbers as well, which I found interesting. One is that recently they have been working on carrier-grade NAT, aiming for 6 terabits per second. Those are pretty big boxes and I hope they are getting paid appropriately for that :) For context he said that for a 4-unit server, these days you can build one that does a little less than a terabit per second. I assume that's with ten dual-port 40Gbps cards, and I would guess to power that you'd need around 40 cores or so, split between two sockets.
Finally, he finished with a long example on lightweight 4-over-6. Incidentally this is the same network function my group at Igalia has been building in Snabb over the last couple years, so it was interesting to see the comparison. I enjoyed his commentary that although all of these technologies (carrier-grade NAT, MAP, lightweight 4-over-6) have the ostensible goal of keeping IPv4 running, in reality "we're day by day making IPv4 work worse", mainly by breaking the assumption that just because you get traffic from port P on IP M, doesn't mean you can send traffic to M from another port or another protocol and have it reach the target.
All of these technologies also have problems with IPv4 fragmentation. Getting it right is possible but expensive. Instead, Ole mentions that he and a cross-vendor cabal of dataplane people have a "dark RFC" in the works to deprecate IPv4 fragmentation entirely :)
OK that's it. If I get around to writing up the couple of interesting Java talks I went to (I know right?) I'll let yall know. Happy hacking!
Khronos has recently announced the conformance program for OpenGL 4.6 and I am very happy to say that Intel has submitted successful conformance applications for various of its GPU models for the Mesa Linux driver. For specifics on the conformant hardware you can check the list of conformant OpenGL products at the Khronos webstite.
Being conformant on day one, which the Intel Mesa Vulkan driver also obtained back in the day, is a significant achievement. Besides Intel Mesa, only NVIDIA managed to do this, which I think speaks of the amount of work and effort that one needs to put to achieve it. The days where Linux implementations lagged behind are long gone, we should all celebrate this and acknowledge the important efforts that companies like Intel have put into making this a reality.
Over the last 8-9 months or so, I have been working together with some of my Igalian friends to keep the Intel drivers (for both OpenGL and Vulkan) conformant, so I am very proud that we have reached this milestone. Kudos to all my work mates who have worked with me on this, to our friends at Intel, who have been providing reviews for our patches, feedback and additional driver fixes, and to many other members in the Mesa community who have contributed to make this possible in one way or another.
Of course, OpenGL 4.6 conformance requires that we have an implementation of GL_ARB_gl_spirv, which allows OpenGL applications to consume SPIR-V shaders. If you have been following Igalia’s work, you have already seen some of my colleagues sending patches for this over the last months, but the feature is not completely upstreamed yet. We are working hard on this, but the scope of the implementation that we want for upstream is rather ambitious, since it involves to (finally) have a full shader linker in NIR. Getting that to be as complete as the current GLSL linker and in a shape that is good enough for review and upstreaming is going to take some time, but it is surely a worthwhile effort that will pay off in the future, so please look forward to it and be patient with us as we upstream more of it in the coming months.
It is also important to remark that OpenGL 4.6 conformance doesn’t just validate new features in OpenGL 4.6, it is a full conformance program for OpenGL drivers that includes OpenGL 4.6 functionality, and as such, it is a super set of the OpenGL 4.5 conformance. The OpenGL 4.6 CTS does, in fact, incorporate a whole lot of bugfixes and expanded coverage for OpenGL features that were already present in OpenGL 4.5 and prior.
What is the conformance process and why is it important?
It is a well known issue with standards that different implementations are not always consistent. This can happen for a number of reasons. For example, implementations have bugs which can make something work on one platform but not on another (which will then require applications to implement work arounds). Another reason for this is that some times implementators just have different interpretations of the standard.
The Khronos conformance program is intended to ensure that products that implement Khronos standards (such as OpenGL or Vulkan drivers) do what they are supposed to do and they do it consistently across implementations from the same or different vendors. This is achieved by producing an extensive test suite, the Conformance Test Suite (or CTS for short), which aims to verify that the semantics of the standard are properly implemented by as many vendors as possible.
Why is CTS different to other test suites available?
One reason is that CTS development includes Khronos members who are involved in the definition of the API specifications. This means there are people well versed in the spec language who can review and provide feedback to test developers to ensure that the tests are good.
Another reason is that before new tests go in, it is required that there are at least a number of implementation (from different vendors) that pass them. This means that various vendors have implemented the related specifications and these different implementations agree on the expected result, which is usually a good sign that the tests and the implementations are good (although this is not always enough!).
How does CTS and the Khronos conformance process help API implementators and users?
First, it makes it so that existing and new functionality covered in the API specifications is tested before granting the conformance status. This means that implementations have to run all these tests and pass them, producing the same results as other implementations, so as far as the test coverage goes, the implementations are correct and consistent, which is the whole point of this process: it wont’ matter if you’re running your application on Intel, NVIDIA, AMD or a different GPU vendor, if your application is correct, it should run the
same no matter the driver you are running on.
Now, this doesn’t mean that your application will run smoothly on all conformant platforms out of the box. Application developers still need to be aware that certain aspects or features in the specifications are optional, or that different hardware implementations may have different limits for certain things. Writing software that can run on multiple platforms is always a challenge and some of that will always need to be addressed on the application side, but at least the conformance process attempts to ensure that for applications that do their part of the work, things will work as intended.
There is another interesting implication of conformance that has to do with correct API specification. Designing APIs that can work across hardware from different vendors is a challenging process. With the CTS, Khronos has an opportunity to validate the specifications against actual implementations. In other words, the CTS allows Khronos to verify that vendors can implement the specifications as intended and revisit the specification if they can’t before releasing them. This ensures that API specifications are reasonable and a good match for existing hardware implementations.
Another benefit of CTS is that vendors working on any API implementation will always need some level of testing to verify their code during development. Without CTS, they would have to write their own tests (which would be biased towards their own interpretations of the spec anyway), but with CTS, they can leave that to Khronos and focus on the implementation instead, cutting down development times and sharing testing code with other vendors.
What about Piglit or other testing frameworks?
CTS doesn’t make Piglit obsolete or redundant at all. While CTS coverage will improve over the years it is nearly impossible to have 100% coverage, so having other testing frameworks around that can provide extra coverage is always good.
My experience working on the Mesa drivers is that it is surprisingly easy to break stuff, specially on OpenGL which has a lot of legacy stuff in it. I have seen way too many instances of patches that seemed correct and in fact fixed actual problems only to later see Piglit, CTS and/or dEQP report regressions on existing tests. The (not so surprising) thing is that many times, the regressions were reported on just one of these testing frameworks, which means they all provide some level of coverage that is not included in the others.
It is for this reason that the continuous integration system for Mesa provided by Intel runs all of these testing frameworks (and then some others). You just never get enough testing. And even then, some regressions slip into releases despite all the testing!
Also, for the case of Piglit in particular, I have to say that it is very easy to write new tests, specially shader runner tests, which is always a bonus. Writing tests for CTS or dEQP, for example, requires more work in general.
So what have I been doing exactly?
For the last 9 months or so, I have been working on ensuring that the Intel Mesa drivers for both Vulkan and OpenGL are conformant. If you have followed any of my work in Mesa over the last year or so, you have probably already guessed this, since most of the patches I have been sending to Mesa reference the conformance tests they fix.
To be more thorough, my work included:
Reviewing and testing patches submitted for inclusion in CTS that either fixed test bugs, extended coverage for existing features or added new tests for new API specifications. CTS is a fairly active project with numerous changes submitted for review pretty much every day, for OpenGL, OpenGL ES and Vulkan, so staying on top of things requires a significant dedication.
Ensuring that the Intel Mesa drivers passed all the CTS tests for both Vulkan and OpenGL. This requires to run the conformance tests, identifying test failures, identifying the cause for the failures and providing proper fixes. The fixes would go to CTS when the cause for the issue was a bogus test, to the driver, when it was a bug in our implementation or the fact that the driver was simply missing some functionality, or they could even go to the OpenGL or Vulkan specs, when the source of the problem was incomplete, ambiguous or incorrect spec language that was used to drive the test development. I have found instances of all these situations.
Where can I get the CTS code?
Good news, it is open source and available at GitHub.
This is a very important and welcomed change by Khronos. When I started helping Intel with OpenGL conformance, specifically for OpenGL 4.5, the CTS code was only available to specific Khronos members. Since then, Khronos has done a significant effort in working towards having an open source testing framework where anyone can contribute, so kudos to Khronos for doing this!
Going open source not only leverages larger collaboration and further development of the CTS. It also puts in the hands of API users a handful of small test samples that people can use to learn how some of the new Vulkan and OpenGL APIs released to the public are to be used, which is always nice.
What is next?
As I said above, CTS development is always ongoing, there is always testing coverage to expand for existing features, bugfixes to provide for existing tests, more tests that need to be adapted or changed to match corrections in in the spec language, new extensions and APIs that need testing coverage, etc.
And on the driver side, there are always new features to implement that come with their potential bugs that need to be fixed, occasional regressions that need to be addressed promptly, new bugs uncovered by new tests that need fixes, etc
So the fun never really stops
Final words
In this post, besides bringing the good news to everyone, I hope that I have made a good case for why the Khronos CTS is important for the industry and why we should care about it. I hope that I also managed to give a sense for the enormous amount of work that goes into making all of this possible, both on the side of Khronos and the side of the driver developer teams. I think all this effort means better drivers for everyone and I hope that we all, as users, come to appreciate it for that.
Finally, big thanks to Intel for sponsoring our work on Mesa and CTS, and also to Igalia, for having me work on this wonderful project.
OpenGL® and the oval logo are trademarks or registered trademarks of Silicon Graphics, Inc. in the United States and/or other countries worldwide. Additional license details are available on the SGI website.
As most of you know, I’ve been working at Igalia close to two years now. If you asked me what I do I’d probably say I hack on “networky stuff”, which is a bit opaque. I thought I’d take the time to tell you about the kind of things I work on.
A bit of background
Consider a data centre full of big expensive boxes that have some high speed network cards performing some specific function (e.g. firewalls, VoIP, etc.). These are all well and good but they cost a fortune and are just what they look like: black boxes. If you’d like to modify the functionality beyond what the vendor provides, you’d probably be out of luck. If the market changes and you need to provide other functionality, you’re stuck having to buy yet another big expensive box that only performs the one function, just like the old one.
You might be thinking, just buy regular server hardware and write some program to deal with those packets. Easy, right? Well… not so fast. Your run of the mill linux application written against the kernel is just too slow as it’s not designed to do these zippy networky things. It’s designed to be a general purpose operating system.
Instead of just relying on the kernel, you can tell linux to leave the network card alone. Linux allows you to poke around directly over PCI, so you can write a driver for the card outside the kernel. The driver will be part of your program allowing you to skip all the kernel faffing and get down to crunching those packets.
Great, right? Job done? Well, kind of…
I’m going to mainly talk about snabb for the rest of this blog post. It’s not the only game in town but I prefer it and it’s what I have the most experience with. Snabb is a framework which implements a bunch of drivers and other neat tools in Luajit (a lightning fast implementation of Lua). Imagine you’re a program using snabb, having all these packets coming at you at breakneck speed from your 10Gb card. You have about 67.2 nanoseconds per packet (my thanks to this helpful person for doing the work for me) to deal with it and move on to the next packet. That’s not much time, right? I’ll try to put it into context; here are some things you may wish to do whilst dealing with the packet:
L2 cache hit
4.3ns
L3 cache hit
7.9ns
Cache miss
32ns
Linux syscall[1]
87.77 ns or 41.85 ns
[1] – If CONFIG_AUDITSYSCALL is enabled it’s the higher one, if not it’s the lower one.
You can imagine that if you’re going full pelt, you don’t really have time to do much at all. Two cache misses and that’s your budget, one syscall and you’ve blown way past the budget. The solution to all this is to deal with a bunch of packets at once. Doing this allows you to go to RAM or whatever once for the whole set rather than doing it for each packet.
Dags att köra Snabb(t)
My colleague Diego has recently written a greatset of blogs which are more in-depth than this, so check those out if you’re interested. Snabb works by having a graph of “apps”; each app is usually responsible for doing one thing, for example: being a network driver, reading/writing pcap files, filtering, and so forth. The apps are then connected together to form a graph.
Lets see a real example of a very simple, but potentially useful app. We’re going to take packets in from an Intel 82599 10 Gigabit card and throw them into a pcap file (packet capture file). We’ll take the “output” of the driver app and connect it to the input of the pcap writing app. Here’s the code:
module(..., package.seeall)
local pcap = require("apps.pcap.pcap")
local Intel82599 = require("apps.intel_mp.intel_mp").Intel82599
function run(parameters)
-- Lua arrays are indexed from 1 not 0.
local pci_address = parameters[1]
local output_pcap = parameters[2]
-- Configure the "apps" we're going to use in the graph.
-- The "config" object is injected in when we're loaded by snabb.
local c = config.new()
config.app(c, "nic", Intel82599, {pciaddr = pci_address})
config.app(c, "pcap", pcap.PcapWriter, output_pcap)
-- Link up the apps into a graph.
config.link(c, "nic.output -> pcap.input")
-- Tell the snabb engine our configuration we've just made.
engine.configure(c)
-- Lets start the apps for 10 seconds!
engine.main({duration=10, report = {showlinks=true}})
end
Awesome!
Just to give you a taste of what is included in snabb out of the box (by no means an exhaustive list):
pcap reading / writing
PF filtering (like the openBSD firewall, more about speed and implementation from my colleagues Asumu and Andy)
Rate limiting
ARP and NDP
YANG (network configuration language for specifying, validating and creating parsers for configuration files)
…and the list continues, it also is pretty simple to create your own. Just define a Lua object which has a “pull” (input) and “push” (output) methods and do something with the packet in the middle. I mainly work on the LwAFTR which provides a solution to IPv4 exhaustion, this is done by wrapping the IPv4 packet up in a IPv6 at the user and unwrapping it at the ISP so they can give users a port range of an IPv4 address rather than the whole thing.
If you use macOS, the best way to use a recent development snapshot of WebKit is surely Safari Technology Preview. But until now, there’s been no good way to do so on Linux, short of running a development distribution like Fedora Rawhide.
Enter Epiphany Technology Preview. This is a nightly build of Epiphany, on top of the latest development release of WebKitGTK+, running on the GNOME master Flatpak runtime. The target audience is anyone who wants to assist with Epiphany development by testing the latest code and reporting bugs, so I’ve added the download link to Epiphany’s development page.
Since it uses Flatpak, there are no host dependencies asides from Flatpak itself, so it should work on any system that can run Flatpak. Thanks to the Flatpak sandbox, it’s far more secure than the version of Epiphany provided by your operating system. And of course, you enjoy automatic updates from GNOME Software or any software center that supports Flatpak.
Enjoy!
(P.S. If you want to use the latest stable version instead, with all the benefits provided by Flatpak, get that here.)
. If you wish to disable that white-list, you can do it by setting an environment variable:
driver version
