Arun Raghavan

Extremely pithy tagline here

Some of you might’ve noticed that there has been a bunch of work happening here at Collabora on making cool open source technologies such as GStreamer, Telepathy, Wayland and of course, PulseAudio available on Android.

Since my last blog post on this subject, I got some time to start looking at replacing AudioFlinger (recap: that’s Android’s native audio subsystem) with PulseAudio (recap: that’s the awesome Linux audio subsystem). This work can be broken up into 3 parts: playback, capture, and policy. The roles of playback and capture are obvious. For those who aren’t aware of system internals, the policy bits take care of audio routing, volumes, and other such things. For example, audio should play out of your headphones if they’re plugged in, off Bluetooth if you’ve got a headset paired, or the speakers if nothing’s plugged in. Also, depending on the device, the output volume might change based on the current output path.

I started by looking at solving the playback problem first. I’ve got the first 80% of this done (as we all know, the second 80% takes at least as long ;) ). This is done by replacing the native AudioTrack playback API with a simple wrapper that translates into the libpulse PulseAudio client API. There’s bits of the API that seem to be rarely used(loops and markers, primarily), and I’ve not gotten around to those yet. Basic playback works quite well, and here’s a video showing this. (Note: this and the next video will be served with yummy HTML5 goodness if you enabled the YouTube HTML5 beta).

(if the video doesn’t appear, you can watch it on YouTube)

Users of PulseAudio might have spotted that this now frees us up to do some fairly nifty things. One such thing is getting remote playback for free. For a long time now, there has been support for streaming audio between devices running PulseAudio. I wrote up a quick app to show this working on the Galaxy Nexus as well. Again, seeing this working is a lot more impressive than me describing it here, so here’s another video:

(if the video doesn’t appear, you can watch it on YouTube)

This is all clearly work in progress, but you can find the code for the AudioTrack wrapper as a patch for now. This will be a properly integrated tree that you can just pull and easily integrate into your Android build when it’s done. The PA Output Switcher app code is also available in a git repository.

I’m hoping to be able to continue hacking on the capture and policy bits. The latter, especially, promises to be involved, since there isn’t always a 1:1 mapping between AudioFlinger and PulseAudio concepts. Nothing insurmountable, though. :) Watch this space for more updates as I wade through the next bit.


Comments
Harvey Specter
Posted at 4:30 pm April 30, 2012
Robert Swain
Reply
Author

Very shiny stuff. Wouldn’t it be great if that kind of responsive music streaming were available in lots of devices and automagically discovered in some list in the app so you could easily upgrade from your phone / headphones to an AV receiver with decent speakers or wherever else.

Of course, it’s nice to have pulseaudio running and capable of doing all the Androidy stuff anyway. Nice work!

Harvey Specter
Posted at 7:46 pm April 30, 2012
Robert Piasek
Reply
Author

Very impressive! Keep up the good work.

Harvey Specter
Posted at 6:25 am May 1, 2012
liam
Reply
Author

Hi Arun,

Amazing work! Does this version of PA have the orc/arm optimizations? Have you done any latency tests yet? From your previous post, that was the single biggest revelation to me. Also, did you have to bring over rtkit or is that not needed until you work on the policy section?

Best/Liam

    Harvey Specter
    Posted at 10:44 am May 1, 2012
    Arun
    Reply
    Author

    Hey Liam,

    This vesion of PA does have the ARMv6 optimisations, but not Orc. In some experimentation, I found that the Orc work I did (which was for volumes) was actually less efficient than the hand-rolled armv6 code. This is because ARM offers a 32bit x 16bit multiplication primitive directly, which is what the entire x86 optimisation revolved around. There are still potential improvements to be made here, and s bit of hacking on Orc to have a generic 32×16 mul instruction would do the trick. Not been able to get to this, unfortunately.

    I didn’t have to bring over rtkit stuff yet. The PA side of things automatically gets RT prio via the old sched_setscheduler() mechanism. Android apps would be using the standard Android APIs to set their priority, anyway, so we might not even need rtkit.

    As for latency tests — my first focus has been to get the functionality going, so I haven’t done any stringent latency tests. There is a noticeable difference in latency in the dialler app between the stock image and my code, but this is hard to capture in audio/video.

    That said, here’s a very basic test that I ran: http://soundcloud.com/arunarunarun/sets/af-pa-comparison — I intend to do some testing and reporting with a more realisitc test case (like an app that generates sounds) before too long.

Harvey Specter
Posted at 9:25 am May 1, 2012
Gutz
Reply
Author

Hi Arun ,

Great progress so far! You mentioned the ability to control the policy for audio routing/volume , Does this involve in-depended application routing/volume control also ( Like in Vista/Win7/8 ) ?

I’ll keep watching your blog for updates!

    Harvey Specter
    Posted at 10:45 am May 1, 2012
    Arun
    Reply
    Author

    Hiya, this is something we’ve had in PulseAudio for a while now. Typically, though, this is not something that is used very often on phones and tablets, to keep the UI simple. So while it’s unlikely that you’ll be able to use this feature in a stock image, there would not be anything stopping someone from implementing this if they so desired.

Harvey Specter
Posted at 4:24 pm May 1, 2012
Alexy
Reply
Author

Good stuff Arun. Keep the good work going

Harvey Specter
Posted at 6:22 pm May 1, 2012
Craig
Reply
Author

Hi Arun,

This work is looking great! While I’m impressed with the ability to stream the audio from the handset via PA. Will the same be possible in the opposite direction? It would be great if I could carry the phone around, with the possibility of it outputting audio from the PC (more for having a portable jack to plug headphones in).

    Harvey Specter
    Posted at 9:27 am May 2, 2012
    Arun
    Reply
    Author

    Yep, since you’re running a PulseAudio server on both sides, anything you can do in one direction can almost certainly be done in the other direction.

    There might be issues to resolve such as preventing WiFi from sleeping on the phone when a client might connect, but these should be solvable.

      Harvey Specter
      Posted at 12:20 am July 4, 2013
      Freddie
      Reply
      Author

      That’s something that I’ve been wondering about … basically turning the phone into a wireless audio receiver for the XBMC box. I got used to using Bluetooth headphones for late-night movie watching, but they’ve since died and I can’t afford (or find any worth buying) new ones right now. Would be neat to be able to use my Android phone as a wireless audio receiver and be able to listen to the audio from the XBMC box via headphones on the Android device. :)

      Basically, replace Bluetooth with PA-over-WiFi.

      Unless someone knows of another way to do this with existing Android/Linux apps …

Harvey Specter
Posted at 8:07 pm May 2, 2012
Pedro Alves
Reply
Author

replacing AudioFlinger (recap: that’s Android’s native audio subsystem) with PulseAudio (recap: that’s the awesome Linux audio subsystem)

Android also uses the Linux kernel, and PulseAudio is not a kernel subsystem.

You meant:

“replacing AudioFlinger (recap: that’s Android’s native audio subsystem) with PulseAudio (recap: that’s the awesome GNU/Linux audio subsystem)”

    Harvey Specter
    Posted at 2:05 am May 3, 2012
    Jonathan Abbey
    Reply
    Author

    (recap: that’s the awesome GNU/Linux audio subsystem)

    Really? FSF is doing PulseAudio now?

    Harvey Specter
    Posted at 3:32 pm May 4, 2012
    Arun
    Reply
    Author

    I know what I meant, and my meaning is not unclear to anybody. Please keep the useless pedantary off my blog.

Harvey Specter
Posted at 8:23 pm May 3, 2012
Ian Meredith
Reply
Author

Hi Arun, this looks amazing, the decrease in audio latency is very significant. Sorry for some more amateur questions but anyway: As a beginner app dev at what stage will I be able to utilize PA in one of my applications? Eventually you will be looking to get PA into a release build of android is that right? Its just I’m so impressed by the results I’d love to implement PA as soon as I’m able to!

    Harvey Specter
    Posted at 9:25 pm May 4, 2012
    Arun
    Reply
    Author

    Hey Ian. For getting this out in usable shape, there’s still at least 2-3 months of work left to get a alpha-level build out. At some point in between, I’ll take a detour to write up some NDK support, so you could build PA apps using the NDK. Initially, this will only be useful to talk to PA servers on other machines, but eventually, hopefully we can get some traction on getting PA on more Android devices.

Harvey Specter
Posted at 6:50 am May 4, 2012
Yong Chang
Reply
Author

This is great!! I’ve been interestingly keep watching your blogs about this work since I’m trying to implement Bluetooth A2DP sink function in an Android device. Is it possible to make an Android device as an A2DP receiver (like headset) using Pulseaudio? I’ve tested it in Ubuntu, and your work makes me confident that it can be done in Android.

Best, Yong

    Harvey Specter
    Posted at 9:26 pm May 4, 2012
    Arun
    Reply
    Author

    Yi Yong, Thanks! :)

    Once all the basic ALSA bits are done, I would like to take a look at using the built-in copy of BlueZ that Android runs talking to PA. Once that’s done, what you’re saying should be very easy at the PA level.

    More system-level support for this (that is, support for doing this via the standard Android stack) is a bit harder, since the pieces are a bit more complicated, but this is still doable.

      Harvey Specter
      Posted at 12:47 am October 20, 2012
      Anthony Maciel
      Reply
      Author

      Have you had a chance to work on getting BlueZ and your port of PulseAudio communicating? I’d be very interested in getting A2DP working as well.

      And is there any way you could give a high-level overview of what might be required to get A2DP working with Android’s native AudioFlinger APIs? I know of a lot of people who would be interested in getting this functionality working, and I think that’d be the best way to do it in a cross-device friendly manner (unless PulseAudio becomes standard with Android). If it is straight-forward enough I plan on taking a stab at it, but I don’t have much experience with audio subsystems so if you could provide a starting point that’d save a ton of time.

      Thanks!

    Harvey Specter
    Posted at 8:39 pm May 28, 2012
    Jesse Burt
    Reply
    Author

    Arun,

    This is great…it’s one of the first things I thought of when I got my GN :) Yong, is your work concerning using an Ubuntu desktop as an A2DP sink public by any chance? This is something I’ve sought for a long time, and only had very limited success with.

    Thanks to both of you! Cheers, Jesse

      Harvey Specter
      Posted at 11:29 pm October 1, 2012
      Dmitry
      Reply
      Author

      i successfully implement it with a2dpd. not in ubuntu, but in SLAX – not too much differ anyway. source available on the net is buggy and needs to be fixed, but after fixing it worked really well for me.

Harvey Specter
Posted at 10:05 pm May 4, 2012
vitriolix
Reply
Author

Very exciting stuff. Nice work.

Harvey Specter
Posted at 11:23 pm May 4, 2012
Christoph
Reply
Author

Very impressive work.

Is it possible to have the current Android sound system stream sound to a linux computer? This functionality would be useful even without pulseaudio.

    Harvey Specter
    Posted at 9:56 am May 7, 2012
    Arun
    Reply
    Author

    If you want, say, to have all the audio from a single app streamed to a remote machine yes. There are likely apps to do this already, and if not, you could use PA in the NDK to build an app to do this (I’ll make my work usable from the NDK at some point in the not-too-distant future).

    If you want to stream audio from arbitrary apps, I don’t know of a simple way to do this other than the stuff that I am currently working on.

    Harvey Specter
    Posted at 9:35 am May 8, 2012
    skierpage
    Reply
    Author

    You can send audio over Bluetooth using Advanced Audio Distribution Profile, supported by most Android devices. Here’s a post about setting up Linux PulseAudio to receive it. There’s an apt-X codec for better audio quality supported by some Bluetooth implementations.

    Another way to stream audio is the complexity and interoperability roulette of DLNA. I had some luck sending audio from my Android phone using the subpar iMedia Share DLNA app to a PlayStation3 (Digital Media Player -> DM Receiver), but had lockups trying to browse the phone’s tracks from the PS3 (DM Server -> DM Controller?). There are DLNA feature packages for GStreamer and PulseAudio that supposedly can do similar things, but it wasn’t obvious to me how to configure Kubuntu for them so I stuck with the low-tech cable with 3.5mm jacks :-)

      Harvey Specter
      Posted at 1:26 am September 6, 2012
      Philip Mather
      Reply
      Author

      Skierpage or Arun, Any idea what it would take or how to get Apt-X on a Nexus? I have a couple of campatible devices but from everything I can find the GNEX doesn’t support it. My technical level is high enough that I can probably blunder my way through to coding at the NDK level (badly probably) but I’m concerned about the availablity of the codec seeing as it’s proprietary which is what I assume keeps it off the GNEX whilst it’s available on the S3? If I use PA is there a path to load Apt-X as a alternative codec somehow?

Harvey Specter
Posted at 4:49 am May 7, 2012
Frank Simon
Reply
Author

I’m working on an app project that requires the need to send 44.1khz 16 bit stereo audio that is live streaming audio content from a PC or Mac and then transmitted over a WAP that is standard WI Fi at 2.4 and or 5 ghz, a, b, g, or n, and then it must be received, decoded and reproduced in the earphones of the cell phone in less than 80 msec from the time of being received by the cell phone with high quality audio reproduction.

If Pulse Audio has the ability to possibly perform this task with concealed packet loss with as little compression and buffering as possible please contact me as soon as you can as your product may be able to used as a demonstration for a product we are developing in the United States. An opportunity exists for future development of a new and exciting product and service.

    Harvey Specter
    Posted at 9:40 am May 7, 2012
    Arun
    Reply
    Author

    Hi Frank. This depends a bit on the actual cell phone on the other side (do you control the stack?) and the various pieces in between on your stack as well as how stable the wifi connection is.

    There are various pieces that need some attention to get them to work better. For example, latency reporting for the tunnel sinks is a bit broken. And while we do have an RTP module for streaming remotely, there is no packet loss concealment as part of this.

    So to answer your question — if you have control over all the bits from start to finish, it is doable, but everything is not production-ready today.

    Harvey Specter
    Posted at 8:37 am May 9, 2012
    Frank Simon
    Reply
    Author

    Arun,

    Thank you for the quick response.

    We do have the possibility to control all the bits on at least one device as long as the codec works with standard Wi -Fi hardware. We will be serving on a VPN and can develop protocol to suit the application. Our Wi-Fi delivery is already suited for high density and we can dedicate gigabit AP’s as required.

    The delivery of the data via Wi Fi is under our complete control and we also have the ability to control certain functions of the handsets circuitry and software from a dynamic application controlled by our server in real time depending upon the ability of certain handsets.

    I believe that you would find this product and service possibly an exciting opportunity to develop Pulse Audio in supporting a newly granted and unique United States patent for delivering streaming live music content directly to cellphones at a concert in real time.

    Please contact me directly so we may discuss the possibilities. I look forward to speaking with you.

    Kind regards Frank Simon Sound at TheAudioAuction dot com

Harvey Specter
Posted at 2:33 pm May 8, 2012
Neeraj
Reply
Author

Great work Arun!..plz do post the latency results once get around to it..audioflinger latency is killing me right now..can’ wait to try dis out once the PA access via NDK is up..Awesome work again!

Harvey Specter
Posted at 10:07 pm May 18, 2012
Rodrigo Carvalho
Reply
Author

I hope this work be successful! Google should use it so Android could have some nice audio applications as iOS has…

I was voting for this issue (http://code.google.com/p/android/issues/detail?id=3434) and I saw a comment pointing to this link: http://forum.xda-developers.com/showthread.php?t=1621914. Have you already seen it? I hope it is helpful for you!

Thank you for the good work!

Harvey Specter
Posted at 3:44 am June 7, 2012
Josef
Reply
Author

Great work but it seems that the patches are down :/ Can someone (maybe Arun itself) please tell me where to find the patches or reupload them? Would be very sad if this great piece of work gets lost…

    Harvey Specter
    Posted at 3:38 pm June 7, 2012
    Josef
    Reply
    Author

    Ignore my post, they’re up again :)

Harvey Specter
Posted at 6:43 am June 14, 2012
eric
Reply
Author

Would love to see some ROMs integrate this. Hopefully it solves the audio lag problem. There are videos on YouTube demonstrating the lag via a Sonic The Hedgehog game in an emulator

Harvey Specter
Posted at 6:22 am June 15, 2012
Hansel
Reply
Author

Great app..and I’m using it for almost a month now.

Harvey Specter
Posted at 9:33 am June 16, 2012
Meynard
Reply
Author

It makes my phone sluggish maybe there are some glitches in ICS update so I switched to NewsAce Multimedia News Reader and find some great feature and good user interface it is the best app so far to replace Pulse and Taptu.

https://play.google.com/store/apps/details?id=com.tecace.newsace

Harvey Specter
Posted at 2:31 am July 8, 2012
The Android Love
Reply
Author

Thanks for this great post. Pulse Audio System is definitely going to bring in a fresh change.

Harvey Specter
Posted at 12:42 pm July 15, 2012
volcacius
Reply
Author

This is simply beautiful. Would it be possible to stream to a DLNA device coupling PA with a Rygel port? It would be such a great open airplay alternative.

Harvey Specter
Posted at 8:52 am July 17, 2012
Melanie
Reply
Author

You better switched to NewsAce it is the same with PULSE but it has a better user interface and features.

Market Link:https://play.google.com/store/apps/details?id=com.tecace.newsace

Harvey Specter
Posted at 10:56 am September 19, 2012
CommenterInNeed
Reply
Author

Have you possibly turned this into an apk? Does it have to be part of the rom to be used? I am no rom developper but what you have demonstrated is exactly what i need.

Thank you.

Harvey Specter
Posted at 5:00 am October 11, 2012
Dave Ferguson
Reply
Author

Has there been any collaboration between PulseAudio and the OPUS codec?

Harvey Specter
Posted at 1:43 pm April 2, 2013
William Smith
Reply
Author

Hi Arun,

Unfortunately, PulseAudio doesn’t build any longer with 4.2 AOSPs using the instructions you have posted.

Pulseaudio for Android would provide developers with exciting possibilities. Are you planning on continuing this work? This would be great.

    Harvey Specter
    Posted at 1:49 pm April 2, 2013
    Arun
    Reply
    Author

    I haven’t had much time to continue the work, but the basic build should still work (someone did get it running recently). The build is a bit quirky – if you have an error message you could share, I can try to help.

Harvey Specter
Posted at 4:23 pm April 18, 2013
Ramya
Reply
Author

Hi Arun,

I am trying to stream audio from my PC to OMAP4 based board running ICS on it using BT. I have enabled BT audio sink profile & able to connect it. But no audio.

So I tried the patch you have given on Android 4.1 ICS source. It gives some compilation error. ” Unable to locate format.h file. Tried googling, still no luck. ” How do I resolve this?

Any inputs? Thanks in advance.

Regards, Ramya.M

Harvey Specter
Posted at 3:54 am May 31, 2013
http://hackitch.
Reply
Author

Everyone loves what you guys tend to be up too. This kind of clever work and reporting! Keep up the superb works guys I’ve you guys to my personal blogroll.

Harvey Specter
Posted at 11:32 pm October 23, 2013
ubercoolguy
Reply
Author

hi arun i tried to install linux distro in android for this app with no use.. i want to play music badly when im chatting in skype or chatrooms from my android.. so pls i want an apk file which would have a checkbox to activate stereomix :/ like in windows or pulse audio in linux.. pls send me apk or tell me how to build it .. pls pls pls /\ i m in bad need of it..

Harvey Specter
Posted at 11:48 pm October 23, 2013
ubercoolguy
Reply
Author

course we can play it from mic.. but users r abusing me for the worst quality :'( like mic should catch the output from speakers not surroundings.. i donno the terminology and technology but hope u understand my problem.. pls understand my feelings.. tried linuxonandroid which runs a vm inside android but its too slow operaating and eating up the battery.,

Harvey Specter
Posted at 1:34 am November 19, 2014
paul
Reply
Author

Hello,

I’m trying to compile the app but it crashes on startup. I really want to use this for audio streaming :). Could u help me further with this, or send me a working compiled version? :)

Greets

Leave a Reply

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>