Xlib and XCB
Alan Coopersmith
The two most popular questions about Xlib and XCB may be "What are they?" and "What's the difference?"
Most programming languages make it awkward for X applications to spit raw X protocol down the network and take apart the protocol coming back. Thus, X toolkits and applications are a lot easier to write if some library handles these jobs for them, providing an API that fits with the programming language and environment for connecting to the X server.
At the bottom level of the X client library stack are Xlib and XCB, two helper libraries (really sets of libraries) that provide API for talking to the X server. Xlib and XCB have different design goals, and were developed in different periods in the evolution of the X Window System.
Most application developers should call Xlib and XCB sparingly. Higher level toolkits provide more efficient programming models, and support features expected in modern applications, including support for complex internationalized input and output, accessibility, and integration with desktop environments. However, sometimes applications will find themselves needing to make calls to the raw underlying X11 libraries for operations not supported by toolkits. An application might need to make calls to X extension API's not covered in the current version of the toolkit's programming model. It is also common for drawing not to be wrapped by toolkit API's. [Yes? --po8]
The original C-language X11 API is libX11, often referred to as "Xlib". It was designed to look like a traditional library API, hiding the fact that calls result in protocol requests to a server. Calls that don't require a response from the X server are queued in a buffer to be sent as a batch of requests to the server. Those that require a response flush all the buffered requests and then block until the response is received.
Xlib's mix of synchronous and asynchronous behaviors causes some problems. Xlib's behaviour is often confusing to new programmers. Calls appear to work sometimes and not others, because it is not obvious which calls implicitly flush the buffer. The asynchronous nature of many calls makes it difficult to debug problems. When an error is reported, the stack trace shows the call that was being made when the error was received and processed, often many calls after the one that caused the error. Finally, Xlib's synchronous calls incur avoidable round-trip latency. This latency has a notable effect on application performance; in particular, startup times are often greatly increased.
After many years of experience with Xlib, and learning from it and other protocol interface libraries, a second attempt was made at defining a C language binding for X11: the "X11 C Binding" layer XCB. XCB makes the client-server nature of the protocol explicit in its design. The client is in charge of deciding when to flush the request buffer, when to read results and when to wait for the server to respond.
For instance, to lookup a window property, the Xlib code is a single function call:
XGetWindowProperty(dpy, win, atom, 0, 0, False, AnyPropertyType, &type_ret, &format_ret, &num_ret, &bytes_after, &prop_ret);
Xlib generates the request to the X server to retrieve the property and appends it to its buffer of requests. Since this is a request that requires a response, Xlib then flushes the buffer, sending the contents to the X server. Next, Xlib waits until the X server processes all the requests preceding the property retrieve request, and sends the property retrieve reply. Xlib then returns the reply to the client.
Xlib also provides convenience functions that wrap a property request. These convenience functions retrieve specific properties, knowing the details of each property and how to request and decode it. Examples include XGetWMName and XGetWMHints. Some of these functions could be written outside Xlib, but many use Xlib internals in non-trivial ways and are thus inseparable. [Yes? --po8]
XCB on the other hand, provides functions generated directly from the protocol descriptions in an "obvious" mechanistic way. XCB functions map directly onto the protocol, with separate functions to put requests into the outgoing buffer and to read results back from the X server asynchronously later. The XCB version of the above code is:
prop_cookie = xcb_get_property (dpy, False, win, atom, XCB_GET_PROPERTY_TYPE_ANY, 0, 0);
prop_reply = xcb_get_property_reply (dpy, prop_cookie, NULL);
The power of XCB is in allowing those two steps to have as much code as you want between them. The programmer decides when to wait for data, instead of being forced to wait for the data returned by a request at the time the request is issued.
Example: Converting xwininfo from Xlib to XCB
The program xwininfo is a command-line utility to print information about windows on an X server. It knows in advance, from the command line options, most of the data it needs to request information for each window from the server. Thus, xwininfo can make its requests all at once, and then wait for the results to start coming in. When using the -tree option to walk the window tree, xwininfo can request the data for all the children of the current window at once, batching even further. On a local connection on a single CPU server, this means less context switches between X client and server. On a multi-core/CPU server, the X server can process requests on one core while the client is handling the responses on another core as they become available, improving performance. On remote connections, the requests can be grouped into packets closer to the MTU size of the connection, instead of just sending whatever requests are in the buffer when a request is made that needs a response.
Version 1.1 of xwininfo was converted from Xlib to XCB by Alan Coopersmith. It was tested with a GNOME desktop session with a few clients. xwininfo was run as "xwininfo -root -all": this started xwininfo at the root of the window hierarchy and asked it to traverse the tree, requesting all the information available for each window along the way. In this sample session it found 114 windows. (In X, a window is simply a container for drawing output and receiving events. X windows are often regions of, or borders around, the "user windows"). When running locally on a four-core Intel Nehalem CPU, both versions ran so fast (0.05 seconds or less) that the difference in time was too small to accurately measure. To measure remote performance, "ssh -X" was used to tunnel an X11 connection from California to a computer in China, and from there back to the workstation in California, introducing a huge amount of latency. With this setup, the difference was dramatic between the two:
Xlib: 0.03u 0.02s 8:19.12 0.0%
xcb: 0.00u 0.00s 0:45.26 0.0%
Of course, xwininfo is an unusual X application in a few ways:
xwininfo runs through the requests as fast as it can and then exits, not waiting for user input (unless you use the mode where you click on a window to choose it, after which it runs through as normal). Once they are up and running, most X applications spend most of their time waiting for user input, so the overall runtime won't decrease as much by reducing the time spent communicating with the X server. However, application startups are typically dominated by round-trip times, and so proper use of XCB reduces the huge startup times of X applications running over high-latency (and even medium-latency) connections.
xwininfo uses only the core protocol and shape extension. It does not use the more complex extensions, such as Render or Xinput, that most modern applications use. Xinput, XKB, and GLX are especially problematic, as those have not yet been fully supported in an XCB release, though support has been worked on through some Google Summer of Code projects.
xwininfo is small enough that a complete reworking to use XCB in one shot was feasible. Most applications are much larger than this. XCB is aimed primarily at new code and at toolkits: it is designed specifically to interoperate with existing Xlib applications. Calls to Xlib and XCB can be mixed, so Xlib applications can be converted partially or incrementally if desired.
xwininfo used only raw Xlib, without any toolkit. Thus, it did not have to worry about which X library the toolkit used.
xwininfo uses only a few of the Xlib helper functions. This makes it more directly mappable to XCB. Applications that rely on Xlib's input method framework, compose key handling or character set conversion, for example, would be harder to port. Fortunately, modern toolkits handle most of this functionality in the toolkit layer anyway.
xwininfo did rely on Xlib helper functions for converting the window name property from other character sets---the XCB version currently only works for UTF-8 and Latin-1 window names. Since most modern toolkits use UTF-8, no one is likely to notice. Older applications with localized window names will fail, but there are few of these in use.
Mixing Xlib & XCB calls
As mentioned above, XCB provides a method for incremental conversion from Xlib to XCB. One can use libX11 to open the display and pass the Display pointer it returns to existing code, toolkits, and libraries. To call an XCB function, one can convert the Display pointer to an xcb_connection_t pointer for the same connection. This enables calling into Xlib and XCB from the same application.
Xlib and XCB compatibility was achieved by rebuilding libX11 as a layer on top of libxcb. Xlib and XCB share the same X server connection and pass control of it back and forth. That option was introduced in libX11 1.2, and is now always present (no longer optional) since the 2010 release of libX11 1.4.
Example: Converting xdpyinfo extension queries to XCB
xdpyinfo is another command-line tool in the standard X Window System toolset. Like xwininfo, xdpyinfo prints a lot of information about the X server. xdpyinfo calls many extensions, and few of its calls block waiting for a response from the server. If you add the "-queryExt" option, though, for every extension xdpyinfo calls XQueryExtension to print which request, event, and error ids are assigned to that extension in the currently running server. These ids are dynamically assigned, and vary depending on the set of extensions enabled in a given server build/configuration. Thus, the list of extension ids is critical information to have when debugging X error reports that reference them. Using "xdpyinfo -queryExt" is especially needed when reporting an X error message that comes from a custom error handler like the one in the gtk toolkit: such error handlers typically omit the extension information found in the default Xlib error handler, so the person reading the bug report will be unable to identify the extension in which the error was encountered.
The Xlib call XQueryExtension takes one extension name at a time, sends a request to the X server for the id codes for that extension, and waits for a response so it can return those ids to the caller. On the Xorg 1.7 server used as the test system for this conversion, there were 30 active X extensions, so that's 30 tiny packets sent to the X server, 30 times that the xdpyinfo client blocks in poll() waiting for a response, and 30 times that the X server goes through the client handling and request scheduling code before going back to block again on its own select() loop.
Note: XListExtensions can be used to get a list of available extensions that can be called with XQueryExtension.
A simple patch to xdpyinfo replaced just that loop of calls to XQueryExtension with two loops. The first loop called xcb_query_extension for each extension. When the entire batch of queries had been issued, a second loop called xcb_query_extension_reply to start collecting the batched replies. Gathering system call counts with "truss -c" showed the expected reduction in a number of system calls made by the xdpyinfo client:
| System call | Xlib | xcb |
|---|---|---|
| writev | 40 | 11 |
| poll | 80 | 22 |
| recv | 117 | 29 |
| total | 237 | 62 |
Over a TCP connection, the switch to XCB for this transaction reduced both the number of packets and (due to tcp packet header overhead) the overall amount of data:
| Xlib | xcb | |
|---|---|---|
| TCP packets | 93 | 35 |
| TCP bytes | 11554 | 7726 |
This sort of change is far more feasible than wholesale conversion to XCB for most applications. Find the hotspots where the application is waiting for data from the server and convert those. There are almost always opportunities in application startup code, when the application is gathering the information about the X server and session. Converting just those calls to more efficient sets of XCB calls can have major performance benefits. Earlier work by X developers reduced the latency of many applications by converting repeated calls to XInternAtom with a single call to fetch multiple atoms at once via XInternAtoms. XCB permits a generalization of this principle.
Extension libraries
Each new extension to the X11 protocol adds requests that clients can make to the X server. To allow client software to utilize these requests, most extensions offer API's built on top of Xlib or XCB. These API's use the library's connection marshalling to include their requests in the stream sent to the X server.
In the early X11 releases, many of the smaller and more common extensions were grouped into a common library, libXext. You will find several there today which are still in use, such as the MIT-SHM Shared Memory extension, the SHAPE extension for non-rectangular windows, and the SYNC extension for event synchronization. However, libXext also includes some API's for extensions no longer found in current Xorg server releases, such as App-Group and Low-Bandwidth X (LBX), as well as extensions many apps never use, such as DPMS for display power management. Since these extension API's cannot be removed from libXext without breaking any existing application which may be using them, the code is stuck in there.
Accordingly, new Xlib extension API's are no longer added to libXext. Instead a new library utilizing libX11 is created for each extension. Having a library per extension makes it easier to evolve the API for that extension, to deprecate an obsolete extension and to only link it into the clients that actually need it. Almost all modern extensions have their own Xlib API library---libXrender for the RENDER extension, libXcomposite for the COMPOSITE extension, and so on. A handful of extensions are so core to the protocol interaction that they are supported directly in libX11 itself, such as BigRequests, XC-MISC, and XKB.
When XCB added its API style to the mix, it followed the newer style and created a "libxcb"-prefixed library for each extension---libxcb-composite, libxcb-render, etc. Since XCB can generates the API code for an extension automatically from an XML description of the extension protocol, new extension API's are created by simply adding the extension description to the xcb-proto package and rebuilding. Unfortunately, some of the older extensions have complex protocols that are not easily described in the XML syntax. Work is ongoing to extend the syntax and code generator to handle these. The XKB & GLX protocols are current challenges.
API documentation
To write code using Xlib or XCB, you'll need to know the details of the library API. Xlib includes man pages for most functions, providing a good API reference. libXext includes man pages for some of the extension API's it includes, but not all of them. Man page coverage is even more spotty in the individual Xlib-based extension libraries.
There's also a more complete guide to the Xlib API, and specifications for many extension API's, in the X.Org online doc set at http://www.x.org/releases/current/doc/.
For extensions without Xlib-style API documentation, the calls are usually simple mappings to the protocol specifications provided in the above-linked doc set.
For XCB, the documentation relies even more heavily on the protocol specifications. The generated API is an exact mapping to the X protocol; it translates the C call data into X protocol encoded packets as straightforwardly as possible. The connection management and other functions in the XCB API are documented at http://xcb.freedesktop.org/XcbApi/. Work is in progress on adding support to XCB to generate Unix style reference man pages from the XML protocol descriptions as well, for developer convenience.
There is also a XCB tutorial, "Basic Graphics Programming With The XCB Library" at http://www.x.org/releases/current/doc/libxcb/tutorial/index.html.
Helping us improve our API documentation for either library stack is always appreciated. See the Documentation chapter later in this guide for more information.