From the Apple dev docs ( https://developer.apple.com/library/ios/#documentation/MusicAudio/Conceptual/CoreAudioOverview/WhatisCoreAudio/WhatisCoreAudio.html#//apple_ref/doc/uid/TP40003577-CH3-SW17 ) they do use CoreAudio, but it has some differences from OSX’s CA (unable to create custom codecs/audio units). I assume that adding a simple switch/case wouldn’t be feasible? I really end up going back to OSX and Windows: they have a single sound server that provides for all use cases, so clearly it is possible to design a system that works everywhere. Besides, if you can get it down to 2ms (or even 5 or 7, for that matter) I don’t think anyone is going to complain about that being too large:) Having better cooperation between JACK/PA would be fine as long as it is pretty transparent to the user (namely, install JACK with PA by default and enable JACK when required by an application). Obviously there would need to be more involved, but the goal of a “single” solution seems like it should be sought after.

Thanks for the response, Colin.

Best/Liam

Just a couple extra points on Arun’s reply.

It was my understanding that CoreAudio is not used on iOS? At least not the same incarnation as on the desktop space (please correct me if I’m wrong). I was always under the impression that CoreAudio served the Desktop and Pro uses cases well, but not so much the Mobile. PulseAudio on the other hand serves the Mobile and the Desktop use cases well but not the Pro.

As for the really, really low latency cases (i.e. Pro), I think PulseAudio’s protocol may ultimately reach an event horizon down at the 2ms like latencies… as we try and use SHM for data transfer between client and server, we pass around handles to blocks of memory that contain audio data rather than passing the audio data itself. For sufficiently large buffers this is obviously the most efficient approach but as soon as you get down to really small payloads, the metadata becomes bigger than the data itself (or at least approaches it) and thus the efficient savings that were there previously actually become overheads.

So with this in mind I think I have to agree with Arun and say that JACK and PA are tailored to mutually exclusive use cases.

With the DBus Device Reservation Protocol supported by PA, JACK can now gracefully ask for exclusive access to the device which at least eases the pain. We can also automatically load jack sink modules into PA to allow sound to carry on playing (assuming it’s patched in JACK) when this happens. So I think cooperation is the way to go here.

That said I fully support any work Arun or anyone else can do to reduce PA latency further for the cases that need it (e.g. games and voip), provided this does not damage the large latency benefits for e.g. media playback :)

Cheers

Col

Yikes, sorry about the using the wrong name. I didn’t even notice the name HAD changed, to be honest. Sorry:) I’m glad you think cgroups might help. I have been having a conversation elsewhere about hybrid kernels, and the topic of XNU’s scheduler came up. He thought that he’d read that in the 10.7 kernel Apple had added the the ability to place the scheduler in user space. Naturally I did a bit of research but I was unable to find any documentation to support his claim (he said he would try and find it, so hopefully I’m wrong and it is available b/c I am curious as to what they had in mind) but I did find that Apple groups threads into 4 categories: Normal, System high priority, kernel-mode only, and Real-time (this mode seems to use the same semantics as the new cfs scheduler, which is suggestive to me — here’s the url if you are interested: http://developer.apple.com/library/mac/#documentation/Darwin/Conceptual/KernelProgramming/scheduler/scheduler.html#//apple_ref/doc/uid/TP30000905-CH211-BEHJDFCA ) . This is pretty similar to the way Android works, and seems like an “easy” fit for Linux, in general. Currently, as I understand it, we use rtkit to boost priority when asked, and if allowed by system policy. So the burden would be on the app devs to support this (which they must do anyway to make use of rtkit). So, in short, the apps would tell need to tell us. I’m assuming, however, things aren’t as simple as I understand them. As for managing the cgroups, I’m not sure I see the problem. Assuming we already have a JACK (or whatever) bin, we can migrate a thread into a cgroup. I suppose you might make a slight tweak to rtkit to enable it to write to the mounted tasks file in /proc/cgroups/JACK/. This might be a bit of a security issue, however:) Thanks for the Indamixx2 link. The latency seemed excellent on the ivory app (I think that is what it was called). It was harder to tell on mixx, but that was simply b/c of the lack of sharp gestures the presenter was using, as opposed to the stabbing motions used when striking a key. So, it is at least nice to know PA can get low enough to be not noticeable (to me, at least). Out of curiosity, did you happen to measure latency when using just PA itself on the tablet? It seems a bit strange that PA simply residing on JACK would increase latency, especially if PA isn’t doing any routing. That is, I wonder how much latency was due to the apps themselves (the non-well optimized ones, that is), versus the presence of PA.

Best/Liam

I think you’re replying to me… not Arun…. so I’ll respond. :-)

Yes, I agree that CGroups is a possible fix for the RT-issue. ATM the complication is identifying which apps are JACK apps and should be brought into the audio/jack cgroup. I recall that cgroups are assigned on a per-process basis and there has to be a controller somewhere that sorts out the processes. I recall that Paul and Torben did some work to add this to the jack server… but I don’t know what the status of that is.

As for the MeeGo device, it was the Indamixx 2. Here’s an engadget video: http://www.engadget.com/2011/05/25/meego-conference-2011-sights-and-sounds-video/

The reason for PA+JACK was simple: MeeGo required PulseAudio to be MeeGo compliant. Also, it’s a decent alternative to the ALSA/JACK plugin so that non-JACK apps can use the JACK server. The objective was to run JACK all the time, so it was always PA-over-JACK. After I got it set up, I only messed with PA if there was a problem. Using JACK was the point of the device.

As for performance, yes we were able to meet our goals by using JACK and a tuned (and relatively un-patched) kernel. Because of PA and some of the heavy-weight shrink-wrap audio apps… the latency out of the box was set to 20 ms. It could do about 5-10 ms if you killed PA and used some well-written audio apps.

-gabriel

Best/Liam

1. It allows applications to share audio data. So you could chain a synth application into a standalone filter/wah and then into a multi-track recording app and then monitor out the speakers.

2. A fundamental part of JACK is that the applications must be written real-time safe in the audio callback, and the the callback runs with real-time priority (SCHED_FIFO).

1 is irrelevant on Android. Not only is it tedious to set up something like this on a mobile device… I get the impression that Google discourages this kind of interaction between applications.

Likewise, for #2: Android will ever allow an app to run with SCHED_FIFO. While the callback is being called, the application will have uninterrupted access to the CPU. For Pro-Audio this is filtered out by a discerning musician who chooses apps he can trust. On Android, someone can download a buggy app that totally locks up their phone. (Response: “Android sucks, bro!”)

Without SCHED_FIFO, there’s no point in using JACK. Using “nice” is not sufficient.

And then there’s power. JACK doesn’t care about power consumption. Period. It’s primary purpose is performance, and gets very unhappy if things aren’t performing. (Are you late coming back from a callback? That’s Game Over in JACK world. The audience just heard a pop over the public address and are not very impressed.)

FWIW, I’ve made PA run on top of JACK for a MeeGo device. (Requires Jack2 and dbus.) It works OK… but audio performance (not power) was the priority for that device. Also, patching full-screen audio apps into an audio patch wasn’t terribly fun (as opposed to laying out the windows on your desktop and editing the patch).

Best/Liam

So the short story is that there are very likely optimisations to PulseAudio that can get our latency down by quite a bit. I’m hoping to look at this soon, time permitting. That said, JACK (at least as far as the literature says) can do something in the order of 1.33ms. They manage to do this by setting up a dedicated SHM pipeline practically from the app to the hardware, AIUI. Given the PA architecture, this is a lot harder to do. So we can likely bring down our latency floor to something that should be “good enough” for a number of latency-sensitive application, but getting to what JACK can do is a lot harder.

Cheers.

In the interview you gave recently to Christian Schaller, you mentioned that you think that PA and JACK serve mutually exclusive purposes. Why is that the case? It seems as though you can have a sound server that serves both purposes (low power and latency) since Windows and Mac have done this (and PA was based off of CoreAudio, IIRC). Naively, I would think that unless PA has some inherent pathologies, you could set the policy such that it tries for maximum power savings (up to 2 sec buffer, IIRC, and depending on the hardware) but is overridden when a client requests low latency and is active. Since this is basically how things currently seem to work, there is obviously a hardware independent floor (meaning, aside from the drivers, and other systems) in how far down latency can be pushed with PA. Why is that?

Best/Liam

Audio should have low latency, even CPU is under full load by other program. Audio should not consume much power, even when serving low latency clients. more tests are needed to prove PA a solution for latency, quality control is important

maybe my expection is too high :)

If you’re up to it, do file a bug and we can try to figure out what’s happening (https://bugs.freedesktop.org/enter_bug.cgi?product=PulseAudio).

Also, as for hungry Android applications — there aren’t that many now, given the state of AudioFlinger latency.

1. CPU load and power consumption when pulseaudio is serving “low-latency” client(s). – VoIP software
2. worst latency encountered and their distribution when serving “low-latency” client(s). – VoIP software, Game and playing high quality movie
3. condition 2 + CPU under 100% loading. – Game and playing high quality movie

On my old PC, PA will use about 8% CPU when i play music, the CPU will be 0% if using Alsa directly. (my card supports hardware mixing…) –test 1

I try to conference with 5 people in Skype, PA uses up to 20% CPU (around 4% increase for each people, i guest its sound mixing algorithm is O(n)), and audio begin to glitches. I cannot imagine what i happen if i add a few more people –test 2

My Game simple have latency and glitches. The situation is even worse when i play games under virtual machine. — test 3

My problems are gone after purging PA.

I think the sound server must function well under “severe” environment, there are hundreds of “hungry” Android application waiting ahead.

“As for 100% CPU loading — even on the desktop, we grant real-time priority to the audio process, so the rest of your system is going to be pretty unresponsive before audio starts dropping.” –> For gaming, this is crazy