pulseaudio irc discussion, between João Paulo Rechi Vita and Lennart Poettering, August 2008

J: [...] I still didn't get how exactly a sink works, could you give an overview of the process ? I mean, when the sink_process_message is called, when a write is called, or point some place where I can learn by my self

L: "There is some documentation available in the wiki it's not completely up to date though the basic idea is that each sink or source is driven by its own thread that sleeps on the audio device's fd and then pushes or pulls data to/from the clients. Now the thing about this thread is designed to be lock-free.

The way how it communicates with the main event loop of PA is not via mutexes/semaphores and stuff but via messages pushed into lock-free queues the RT threads managed by the audio devices never block on those queues however the main event loop on the otherside of those queues may block there are two queues for each thread one from the single main event loop to the thread and one back in some cases more than a single sink or source is driven from a single RT thread the most prominent example here is the OSS driver

Because we have only a single fd for both recording and playback and the devices are syncrhonous they are driven from the same thread.

J: those queues are the pa_thread_mq, right?

L: Yes, pa_thread_mq stores a pair of queues:

pa_thread_mq->inq is from the main event loop to the specific thread
and pa_thread_mq->outq is from the thread back to the event loop
pa_thread_mq_xxx() can be used to set up the pair and hook them up with the main event loop

J: And pa_process_msg is the function that process the messages coming through these queues?

L: Yes, process_msg() is generally used to dispatch messages, on both sides of queues actually.

Every object that inherits from pa_msgobject can receive these messages (~signals), you send those msgs with pa_asncmsgq_send(), or _post().

_send() is synchronous (i.e. blocks until the function is received and handled on the receiving side)
_post() is asynchronous (i.e. it justs enqueues and returns)
_send() should thus only be used for communication from the main event loop to the thread but not the other way round

If no response is required for a messages, then _send() is highly recommended

J: and how do I know which kind of messages can come through the queue?

L: All kinds of messages can come through the queues. For example pa_sink, pa_source, pa_sink_input, pa_source_output can receive messages, and defined a few default ones. If you implement a pa_sink/.... you can add your own messages. If you check pulsecore/sink.h, you can see the messages understood by pa_sink in the type pa_sink_message_t there is a default implementation for pa_sink_process_msg() for each class usually if you subclass something you would handle what you want to handle and then call that generic handler to fully understand what is going on. In all this code you always be aware from which context a function is being called (i.e. if it is called from the main event loop or if it is called from one of the IO/RT event loops, or if it can be called from both for more recent code). I added small comments to many functions which explain the context because i got lost myself ;-)

J: I saw some of this comments, i think on sink.h

L: For some functionality of pa_sink/.... you can either implement it in the main thread, or in the RT thread: for those you can either overwrite process_msg, or you can fill in some other virtual function.

One example for this is the volcontrol stuff i.e. for a sink you can either implement set_volume/get_volume as function pointers in the pa_sink structure which would then be called form the main event loop or if you leave them as NULL you will get a message PA_SINK_MESSAGE_GET_VOLUME/PA_SINK_MESSAGE_SET_VOLUME in the RT thread instead.

This is useful because for some audio devices volume control is done in-line and for others vol control is out-of-line.

Since vol control is not time critical in any way it is usually a good idea to leav it out of the RT thread but sometimes the underlying API doesn't really allow that. Of course, as little as necessary should be run in the IO thread (oh, sometimes i call it "RT thread" and sometimes i called it "IO thread". those terms refer to the same...)

J: ok, i got it

L: The IO threads are supposed to be real-time threads if real-time scheduling is available. That's why i use those terms interchangingly.

Hmm, what else is there to say you should focus on module-esound-sink i guess it works very similar to your bt stuff i guess i.e. it is modelled around a single fd.

[...]

In the RT thread you should be running a pa_rtpoll() event loop, which is actually very similar to the main event loop but a bit more designed for RT stuff (though actually the pa_mainloop stuff should eventually be dropped in favour of pa_rtpoll entirely...)

J: that's how the esd sink works also, right?

L: Yes, all sinks use pa_rtpoll

So, where are you lost? any specific area? i can shed some light on?

J: This talk clarified some points to me

L: http://pulseaudio.org/wiki/ThreadingModel, that wiki doc stuff is still very much relevant, but unfortunately out of date.

J: I still not sure which messages should I take care of on sink_process_msg.

L: Usually you want to implement PA_SINK_MESSAGE_GET_LATENCY. It is necessary of getting the latency right.

J: I saw on esd_sink that it take cares of some messages about state changing and call pa_sink_process_msg

L: It uses PA_SINK_MESSAGE_SET_STATE to be notified about state changes of the sink (i.e. if we enter suspended mode), because it uses the time smoother framework and wants to stop the smoother as quickly as possible if we enter suspend mode.

J: the time smoother stuff I still didn't get, what states can a sink be on?

L: The states of a sink are these:

PA_SINK_INIT: the sink has been allocated, but has not yet been installed in the core with pa_sink_put()
PA_SINK_RUNNING: we are playing something
PA_SINK_SUSPENDED: the device has been temporarily suspended. usually this should cause the device file to be closed
PA_SINK_IDLE: very similar to PA_SINK_RUNNING but signifies that no non-paused stream is connected to the sink. Usually this should be handled like PA_SINK_RUNNING, but can be used as an optimization for some cases, since pa_sink_render() will always return silence in this state
PA_SINK_UNLINKED: the sink has been removed from the core and is being destructed

Since IDLE and RUNNING is so very similar you usually should check for both at the same time, which is why we have PA_SINK_IS_OPENED() which just takes the state and returns TRUE if the state is RUNNING or IDLE.

To tell the difference between "alive" and "dead" we also have PA_SINK_IS_LINKED() which can be used to check for everything that is not _INIT or _UNLINKED.

So much about the states. Again, usually you want to treat _RUNNING and _IDLE the same. But _SUSPEND needs special handling for most devices, but doesn't need to either.

The states for sources are very similar.

J: when a sink is suspended, does it still show up under volume manager and elsewhere?

L: Yes it does. As long as it is RUNNING, IDLE, SUSPENDED it is considered "alive" and can be used for everything. If it is INIT or UNLINKED, t is considered "dead" and doesn't show anywhere.

About the smoother: many audio devices don't really provide any good timing source (i.e. a2dp doesn't have any timing source at all afaik), which makes them difficult to use for uses where timing is crucial.

To remedy this i came up with the time smoothers they solve two problems:

they synthesize a kind of stable timing source for devices which haven't
it helps with estimate the deviations of timing sources from the local clock To elaborate a bit more about this:

For some devices you only have at very few points in time an only very vague idea of where the playback pointer currently is, for example: the esound sink.

Only after you filled up the TCP socket's buffer completely you have a minimal idea how much data is currently on its way to the other side. Otherwise the buffering of the TCP stack makes estimatations of how much data is still on its way completely impossible.

Hence, when we filled up the TCP socket completely, so that it doesn't take any further write requests for the moment, we tell the time smoother about it, and about how many bytes are currently in the TCP buffers, and then the time smoother will be able to help us telling the current playback time later on, when a client might be asking at arbitrary times.

I now use it at a lot of places actually. It is used in the alsa sink, for example, to deal with the fact that we need to estimate when the hw playback buffer will underrun, and we need that timing information in the time domain of the system clock, not of the sound card clock.

Also, on some sound cards asking for the timing info is very expensive. I use it on the client side too, so that whenever a client asks for the latency/playback time, i don't have to go to the server side and ask for the timing, but instead i can interpolate the timing from information i got a bit earlier.

So, you probably can consider the smoother kind of a black box: you fill in x and y when you have the data.

x being the local time, y being the estimate time of the source you want to interpolate/smoothen, and then you can ask by passing in another x later what the smooter thinks that y is now going to be on ther other timing source.

pa_smoother_put() pushes x/y into the smoother. pa_smoother_get() gets y for an x. And then there are a few more functions to deal with timing offsets and suspend/resuming of the timing source.

Internally, all it does is doing a spline interpolation between the data you fill in, and then use the calculate spline to estimate the values for the future (though actually it still does a little bit more, but that's just detail...)

J: And what about pa_smoother_set_time_offset()

L: Various time sources deviate in differnt ways, they can:

have a different start time. i.e. while one clock started on 1st jan 1970 as 0, another clock might have started when your compzter booted up and conisder that 0, or when an audio devcie started to play and consider that 0
have a different speed. i.e. when one clock thinks 1s passed a different clock things 1001ms passed
the data from one (or both) of the clocks is noisy. J: hum, so 1 is corrected with set_offset?

L: Yes, set_offset corrects the initial offset that is explained in 1). 2) and 3) are automatically corrected by the smoother (or at least it is tried to correct them automatically).

J: And how can I calculate this offset?

L: Depends if you just count bytes your write, then you would just call settime_offset(s, pa_rtclock_usec(), 0); when you start counting them.

If you have an explicit timing function (i.e. because your device exports a real clock), then you would just call that function when you start with your stuff, and pass its return value to settime_offset(), i.e. settime_offset(s, pa_rtclock_usec(), get_time_of_my_device());

You call that function only once when you start your stuff (if i remember)

The smoother is of course not perfect, and contains a lot of tweaks to make things work even in the worst conditions. But sometimes those tweaks might actually break things for you ;-) so if you experience problems, ping me!

[...]

J: one quick: what is the easiest way to test a new sink? do I have to hook some application to PA or is there any internal mechanism?

L: you can load module-sine i.e. write your own minimal default.pa that just loads your module and module-sine module-sink will create a single pa_sink_input on the default sink and just play a sine wave at 440hz

Oh, if you need to valgrind your stuff in the current git version you can now set $VALGRIND to 1 and run pa like that in a valgrind and it will disable the capabilities code.

We check for $VALGRIND only if you build without optimizations.

However, ./bootstrap.sh disables optimizations anyway. o you should be good, just wanted to warn about that ;-)

VALGRIND=1 libtool --mode=execute valgrind ./pulseaudio

That's how i run pa these days if i need to valgrind it.

J: Ok, thanks for all the explanations

L: You're welcome

jprvita goes back to coding

jprvita saves chat log for reference