(these notes are being posted in two parts to make the length more manageable, part 2 is here)
The PipeWire community organised a hackfest in Nice, France, colocated with Embedded Recipes, the GStreamer hackfest, and a number of other events.
In attendance were members of the upstream community, as well as folks interested in PipeWire from Collabora, Red Hat, Qualcomm, Stream Unlimited, Texas Instruments, and Valve. In some cases these were the same person wearing upstream and professional hats, as some of us often do! :)
It was two days of fruitful and deep technical discussions, and lovely evenings hanging out in the Côte d’Azur. Shout out to George Kiagiadakis and Mark Filion for putting this together!
The topics were disparate and can be somewhat esoteric for folks who are not familiar with the Linux audio space. I will try to strike a balance between providing context and summarising the finer details we discussed. Please feel free to write in if I missed or can expand on anything.
Multistream nodes
A recurring topic for the last couple of years has been supporting multistream nodes. The PipeWire API currently offers a pw_stream interface that can offer a node with single input or output (closer to the PulseAudio API), and the pw_filter interface that provides a lower-level freeform API to individually manage ports on a node (closer to the JACK API).
The stream API while convenient, can be a bit unwieldy for realising concepts such as loopbacks and filters, because each set of inputs and outputs needs to be implemented as an individual node. If you’ve ever loaded the loopback module, for example, you would have noticed that there are two nodes created for each instance.
Wim has created a version of the API that allows a node to provide multiple streams, which allows us to keep the conveniences of the stream API, but more easily express ideas like the loopbacks, filters, etc. Each stream is effectively a group of ports on the node, and nodes can have an arbitrary number of input and output streams.
The code on the PipeWire side is ready. The primary idea is there will be a PortConfig param per stream, and this is where the format of the stream, and other metadata expressed on port groups (which is essentially what a stream is) will live.
We discussed what is needed in WirePlumber to make sure the linking logic adapts to this concept, and Julian will be implementing that in the coming weeks.
Settings
PipeWire has a generic metadata system based on the JACK API that is used for storing metadata (allowing you to attach a key/type/value, optionally attached to an object). This is also used by WirePlumber to provide its settings system (see wpctl settings), along with some key features such as a schema and persistence.
We discussed that it might be nicer to have the concept of settings as a first-class citizen, and possibly even standardise some settings for desktop wide usage (such as common processing elements). There was consensus that:
- A new settings interface (instead of extending metadata) would make sense
- The API should be asynchronous, and can fail
- A schema for valid settings and their types could be exposed as a well-known metadata key
- Implementors of the interface would perform validation
Security
We spoke about the current state of security for applications using PipeWire. For context, PipeWire has a fine-grained permissions model where each client can have selective access to what objects are visible to it, and what actions it may perform. There is also a less granular system, where a “manager” application can connect to the manager socket for full access. We broadly think about restricted security for sandboxed applications (primarily Flatpak).
One scenario is sandboxed PulseAudio applications getting full access via the pipewire-pulse server on the host. The discussion on this concluded that there is a way for pipewire-pulse to forward enough security-related information from sandboxed applications for us to apply sandbox restrictions to them, and we need to make that system work.
There was a discussion that it might be reasonable for our default policies to apply for all applications connecting to the regular PipeWire socket to be restricted (this does not prevent malicious applications from accessing the manager socket, but helps applications not do bad things erroneously).
This might be disruptive to introduce as a default change, so we might implement it via an opt-in setting so that there can be some broader testing and refinement of default permissions before flipping the switch for all users.
There are a number of mechanisms related to how security context properties are relayed, and how those properties are used by WirePlumber to determine permissions. We need to document and verify the expected behaviour here.
Flatpak and Portals
Relatedly there was a discussion about how things should fit in with Flatpak, and Sebastian Wick from the Flatpak team joined us briefly on the second day.
There was some discussion of making sure the PulseAudio socket is provided to the sandbox in a similar way to the PipeWire socket, such that some additional security properties can be assigned from the host in a way that the sandboxed client cannot override.
We agreed that we needed the ability for applications to specify with some granularity what permissions they require (via portals), and for us to grant only that (with user intervention, if needed). Broadly this is:
- Playback (optionally enumeration of sinks)
- Capture (optionally enumeration of sources)
- Default visibility of only the application’s own nodes
We also spoke about how we might want to associate PipeWire objects with applications. With Flatpak moving to using a cgroup for each application, this should become easier. We may also want to be able to have a way to associate a stream with a specific window (to, for example, share a window and its audio), which should be possible.
It was also noted that for some classes of applications, we may want a way for users to allow some of these permissions at install time (for example, a remote desktop application asking permission on every start can be annoying). This is already possible with Flatpak manifests (which are static, but we might need to add some more options here), and there is a potential entitlement system being discussed (for server-driven overrides to be distributed for malicious applications, for example).
Encapsulation and Collections
One topic that came up last year is the ability to encapsulate a group of nodes such that they appear as a single node to other applications in the system. This could be useful for:
- Collapsing all the output from an application so it appears to be providing a single stream
- Grouping all the filters for a sink or source node, and making it appear as a single node with all the processing hidden away
One piece to making such a system possible is to have a first-class notion of this group. Julian has an implementation of such an entity, called a “collection”. This is currently implemented on top of PipeWire metadata, but we agree that this is likely worth having an explicit PipeWire interface for.
Once that is in place, we discussed the possibility of having a smarter “proxy” node that can act as the interface that translates from the “outside” of the encapsulated region to the “inside”, so that format selection, volume changes, etc. can properly be proxied to the underlying device, for example.
Tooling improvements
It was noted that the tools we have (such as pw-top and pw-dot) can make it hard to get at some information, such as negotiated formats, rates, etc. They can also be “noisy” when we have a large number of filters and loopbacks.
While we did not have a concrete plan to tackle this, some of us have been playing with LLM-based tooling to generate some helper code for this sort of thing. At least my attempts have been too sloppy to share as yet, but it should be possible to get something useful with a structured approach.
That’s it for now. Watch this space for part 2!