Modularization Proposal

Paul Anderson, Alan Coopersmith, Egbert Eich, Adam Jackson, Kevin E. Martin and Keith Packard

Introduction

Traditionally, the X Window System source code has been comprised of many different components that are brought together into a single monolithic source tree. We propose to split the tree into logical modules that can be developed, built and maintained separately, but still fit together coherently into the larger source code base as they have in the monolithic tree.

This proposal provides rationale for modularization, attempts to address the known concerns people have raised, provides background on other modularization efforts, explains what the modularized source tree would look like, outlines the steps required to take this idea from concept to release, and suggests how future modular releases can be handled.

The pros and cons of modularization

The pros

• There are many open source developers that are intimidated by the large X code base. Working on smaller self-contained modules with well defined ABIs, is much less intimidating.
• Most other open source projects use autotools, so it's something that many people know. Imake is known in our community but it's a barrier to entry for most others.
• Lowering the barriers to entry will both attract new developers and help them to become productive members of the community more quickly. More developers means progress speeds up, and bugs are found and fixed quicker.
• You no longer have to build and install the entire tree just to work on one small part of it.
• Creating releases from modules is automated with the autotools, along with the ability to automatically verify that the release is correct.
• Bug fixes in one part of the tree can be made available to vendors in a more timely fashion. For example, a bug fix in the Radeon driver module could be checked in on a stable development branch and released as an update for vendors.
• Finer grained access control to the source code; each module should have its own maintainer and access controls. Global developers will still have universal access.
• It should be easier to use technology that's not under the X license. Right now, we have the xc/extras directory, because the code there is either not under the X license or maintained externally. These extras are often out of date and conflict with the installed versions on vendors' installations.

The cons

• The autotools are a new build technology, which throws away existing knowledge of the current imake build system. People that are not already familiar with the autotools will need to learn a new build system. That could slow some developers down for a while.
  • To address this issue, we need people who understand both imake and autotools well to write a transition guide for the benefit of all.
• Some platforms may not "come along" if there are no maintainers.
  • This should be treated as an opportunity for us to contact the old maintainers and see if they are interested joining the community, or for us to encourage to new maintainers to step forward to support platforms if the old maintainer is no longer interested or available.
• Some platforms may not "come along" if the current build maintainers don't have time to learn the new build system during the initial release.
  • The X.Org Foundation has been very good about having regular releases of our software and we are aggressively working to ensure that this continues. This regular release policy gives maintainers that don't have the time at present to complete the transition confidence that they will have opportunities in the near future to add their platforms to the new modular tree.
• Autodetection can add dependencies that you didn't intend. For example, a developer may have a library on their build system that isn't available on the target system.
  • Dependencies will be explicitly listed and documented. Configure options will be used to control which dependencies are enabled or disabled. These configure options will be standardized and documented in the transition guide.
• Autotools are very version-dependent, and some OSes might not have the most up-to-date versions available. For example, some OSes might use an OS-specific libtool, which might need to be updated to includes the latest patches to properly handle "so" versions (see below).
  • For developers who do not have the required autotools versions installed and don't want to overwrite their vendor-supported versions, it is possible to install them in out-of-the-way locations (e.g., not in $PATH) and still easily script the build process.
• Autotools are not very easy to understand (e.g., libtool).
  • Imake is also not very easy to understand for new developers. The autotools have the advantage that there are many sources for good information. The transition guide will also help. Many of the more complex tools (e.g., libtool) can be ignored almost all of the time as other tools (e.g., automake, in the case of libtool) manage the whole interaction.
• The various servers have different (and possibly incompatible) build options, which means that they may not be able to share the same build of the code. For example, one server might build DIX with XINPUT support, while another one might require that XINPUT be disabled.
  • This problem also affects the monolithic tree, and it is not easily solvable in either case without rebuilding the affected parts. This rebuilding task is simple with autotools.
• The configure scripts take a long time to complete; you have to run multiple "configures" from scratch, and it will take a while.
  • Configure caches can be used to speed up multiple configure runs, and jhbuild or scripts can be used to automate them.
• With the imake build system, you can build and test in place with the link directory inside the tree. Building self-contained user environments are more difficult with autotools.
  • jhbuild solves this since it runs a self-contained environment. Other solutions using chroot, LD_LIBRARY_PATH, etc. are also in use today with autotools.
• Legacy imake-only applications will need to be converted.
  • Imake is not going away and will still be needed for legacy applications maintained outside of our source tree. The transition guide that we use internally will be made available externally to help others become familiar with how the new autotool build system works so that they can convert their applications as time permits.

Background

During the past two years, several research projects have explored how to modularized and autotool the source tree. We can learn a great deal from these projects. Below is a description of the projects from two of those directly involved.

Keith Packard's experiences

The modularization work that Keith did was to split the sources up so that no distribution would have to break one of his packages apart in order to build their distribution -- i.e., no modules would need to be split across package boundaries. Distributing the software is then a matter collecting the pieces, rather than cutting things apart. To do that, he cut it pretty finely. He split it into three main chunks: xapps, xlibs, xserver. They are logically separate from a functionality standpoint. Also, he could control access to these separately.

In doing this, Keith found that the header files were done incorrectly for many extensions. He now has many small packages that only have the specification and the header for the extension. We can learn from the experience that Keith has had in this area for the libraries. The protocol headers are the foundation of the system. We have to get those working/structured properly for the rest to work.

Keith had to get a change made to 'libtool' (v1.5) so that the correct "so" versions could be specified. This is an absolute requirement for our product, but the change was made long enough ago that most people have it. There is an issue where certain OSes use a different libtool that may not have this change.

Keith did less work with the xapps, since he doesn't use many of them. Plus, autotooling an X application is simple; autotooling the libraries was a longer process.

Keith hasn't split the Xserver into pieces, since he treated it as just an application. However, for our modularization project, he pointed out that shipping individual drivers as separate modules is a fundamental goal, so we will need to do additional work along the lines of what was done with Debrix.

Adam Jackson's experiences with Debrix

Debrix was started by Daniel Stone as a second attempt at autotooling the Xorg server. The previous attempt was named "Xizzle" and floundered due to the DDX build-time dependence on various DIX config options discussed elsewhere. Daniel did most of the initial bootstrapping design, I imported most of the driver and input modules, and Jakub Stachowski contributed several autotooled extension modules.

The Debrix model split the server into four basic components:

• server core (cvs module name 'debrix')
• video drivers ('debrix-driver-ati' etc.)
• input drivers ('debrix-input-mouse' etc.)
• dynamically loadable extensions ('debrix-extension-dbe' etc.)

Some of the design decisions made in Debrix:

• The server component consisted of the xfree86 DDX, including the core dynamic modules like int10 and ddc.
• The video/input/extension modules were split off on the assumption that those would be the pieces users would be most likely to want to replace piecewise.
• To enable the non-server modules to build, the server component installed server API headers to $(PREFIX)/include/xorg. The "config.h" generated during the server build was installed as "debrix.h", so drivers could see the options used to build the server.

Some of the issues we encountered:

• Debrix was used as a staging area for the dlloader work that eventually landed in Xorg. We considered using libtool's 'ltdl' library instead, since some platforms don't have libdl but have their own APIs for dynamically loading modules, but this was quickly rejected. ltdl places some strange constraints on module usage and would require major ABI changes in basically every driver to accommodate ltdl's rather absurd symbol naming requirements. The executive decision was that platforms that didn't have libdl could implement it in 50 lines of API wrappers for much less pain.
• GLX support was hard to factor out. The server's GLcore software rasterizer requires a copy of the Mesa source to build, as well as some server-specific code. Jakub did have this working but it required telling the build process where to find a Mesa source tree via a configure option. This is unwieldy and will need to be addressed for 7.0.
• int10 support was initially through vm86 only. The x86emu version was added later, but x86emu lives under extras and thus might present a special case for the modular tree structure. Debrix simply imported it whole, but this is clearly wrong.
• No other DDXes besides the xfree86 DDX were attempted. The xnest, xwin, and xgl DDXes have been autotooled in Keith's xserver project. The darwin, dmx, sun, sunlynx, vfb, and xprint DDXes have not (to my knowledge) ever been autotooled. sun and sunlynx can probably be dropped at this point, but any lurking issues in the remaining imake-only DDXes are therefore still lurking.
• Docs and fonts are also logically separate, and were not attempted in Debrix.

The monolithic and modular source trees

This section outlines a proposal for what the modular source tree could look like. It provides details on each module's contents, the public interfaces, the dependencies between modules, and rationale. This and the next two sections are the heart of this proposal, and the parts that need to be accepted by the community before begin the modularization work.

It is important to note that the modular source tree is expected to contain the same code as the monolithic tree, but will be organized and built in a different way. For example, the code will be organized into logical modules. Autotools (http://sources.redhat.com/autobook/) will be used to build the modular sources individually, and with the help of jhbuild and other support tools, we will continue to be able to create completely self-contained builds.

The monolithic source tree will continue to exist as a self-contained source tree that can be built with imake, but it is expected that people will transition to the modular tree for future development (see the next section). We propose that both trees should co-exist side-by-side in the xorg CVS project hosted on freedesktop.org (see Figure 1 below).

From the Figure above, the monolithic source tree will continue in the xc hierarchy within the xorg project. Along side of it will be several separate modules: app, lib, proto, xserver, driver, font, doc, and util. Each of these modules will be detailed below. Not shown in Figure 1 are the extras, which will exist outside of the source tree and contain only those source tarballs required to build the tree, but for which xorg is not the canonical source repository.

The applications module: app

• Applications are relatively straight forward to autotool.
• Each app will be placed into a subdirectory of its own within the app module, and will not depend on any other apps being present within the checked-out module.
• Each app will have its own dependencies on specific libraries from the lib module being installed.

The libraries module: lib

• The libraries are one of the more difficult parts of the tree to modularize since they depend on both the API headers as well as the protocol headers, and other modules require them to be built and installed.
• The API headers and protocol are the public interface to our system.
• The API headers will be included with their respective libraries in this module.
• The protocol headers are required by both the lib and xserver modules and will be split into a proto module (see below).
• After these dependencies were worked out, the libraries become relatively straightforward to autotool.
• As with the other modules, each library will be placed into a separate subdirectory within the lib module and will not have any build-time source dependencies on other libs being present within the checked-out module.
• Library revisions have 3 component revisions, x.y.z, where: * x changes when the ABI is incompatible with the previous revision * y changes when the ABI is changed, but remains compatible * z changes when bugs are fixed that don't change the ABI
• Note that some platforms only have 1 or 2 component revisions for their libraries. On these systems, either the second component (y) could be bumped, or symbol versioning could be used by vendors who package the libraries.
• By default, all libraries are built as shared objects; however, with other build options, static, profiled and debug versions can be built.

The protocol module: proto

• As noted above, this module contains the protocol headers that are required to build the lib and xserver modules.
• In addition, the protocol specifications will be included in this module.

The X server module: xserver

• The X server is also one of the more difficult parts of the tree to modularize since it also has external dependencies, there is a strong desire among the vendors to separate the drivers into their own modules, and the different DDXs support different build options.
• The XFree86 drivers depend on the X server headers, and since we are splitting the drivers into a separate module, these headers will need to be installed before the driver module can built. These headers will installed in /usr/include/.../xserver as part of the normal install target. A special DDK install could also be added, if that proved useful.
• At some point we may want to investigate splitting each of the DDXs into separate modules, but not at this time. The current design does not preclude doing this research in the future.
• The xserver module depends on the proto module, and some of the DDXs depend on the lib module (e.g., Xnest, DMX).
• As noted above, some DDXs require different built options, which implies that not all DDXs will be able to use the same dix, os, mi, etc. compile. However, they can and should share the same source code. For example, the kdrive server does not support XINPUT whereas the Xorg server does, so dix (and other subdirs) will need to be built differently for each server. Note that these conflicts can be detected at configure time.
• The basic tree structure in the xserver module will look similar to the one used in the monolithic tree under xc/programs/Xserver. Notable exceptions include the drivers that will be moved to their own module (see below), and support programs that could be moved to the app module.

The XFree86 drivers module: driver

• Vendors have expressed a strong desire to be able to ship updates to individual drivers as bugs are fixed, so they have been split out into their own module.
• The drivers consist of both input and output (i.e., video) drivers.
• Each driver will be completely self-contained and placed in a separate subdir within the module, with the subdir name reflecting the driver interface and the device that it supports (e.g., xaa-mga, kaa-ati, xf86input-mouse, etc.).
• The driver module depends on the headers from the xserver module being installed.
• Vendors can choose to distribute the entire driver module as a whole, or they can subdivide this module into finer components if desired.

The font module: font

• This module contains the fonts traditionally distributed with the monolithic releases.
• Note that some fonts have more restrictive licenses and these should be fully documented in this module.
• The font module depends on the font tools that will be included in the app module be installed.

The documentation module: doc

• This module contains the documentation traditionally distributed with the monolithic releases that has not been moved to another module.
• The majority of documentation in xc/doc will be relocated to the module that they document. For example, the protocol specifications will be moved to the proto module and the library docs will be moved to the lib module.

The build utilities module: util

• This module contains the build tools and configuration files for legacy applications and libraries that have not been converted to use the autotools.
• Other utilities will also be placed here (e.g., makedepend, lndir and mkshadow).

The extras:

• This is not a module, but rather a separate cache of known working tarballs required to build various modules (e.g., Mesa, freetype).
• It is expected that the vendors will include known working and supported versions of these extras in their OS releases.
• By default, the modules described above will build using the installed versions, but can be overridden when necessary.
• These tarballs are made available as a convenience to those doing development on OSes that don't have the latest known working versions.

Transitioning from monolithic to modular

Now that we've seen what the modular source tree could look like in the section above, we can explain how to transition from our current monolithic tree to the proposed modular one. The transition can begin once this proposal has been finalized and accepted. Figure 2 shows the outline of how this transition will happen.

The modularization effort will begin by creating the top level modules. The order in which to create them is implicit in their dependencies (see previous section). During this development period, our goal is to share source files between both the monolithic and modular trees.

Several options were investigated for how to share source files during this development period including copying the ,v files as each module is created, creating symlinks within the CVS repository, and modifying the pre- and/or post-checkin scripts. Each of these options had significant downsides.

The solution we propose to share the source files is for developers working on modularization to use a script to populate their local modular trees with symlinks from a checked out version of the monolithic tree. During the initial development phase, the only files that will be checked directly into the modular tree -- and not in the monolithic tree -- are the ones related to the autotool build environment (e.g., configure.ac, Makefile.am). The symlink script will be updated as new modules are added or as files are moved around. If source file changes are required, then the developer working on modularization can make and test those changes in their local modular tree, and when the changes are ready to be checked in, they do so from the monolithic tree.

While this initial modular development occurs, development will continue as normal in the monolithic HEAD. Once the modular tree reaches a suitably stable point in all of the modules, both trees will be frozen for new features (Devel Freeze in Figure 2). We expected that this freeze will be announced several weeks before it takes place. At that time, the first release candidate (RC1) will be tagged, and we will begin to stabilize both trees for release.

During this stabilization phase, release testing will begin on both the modular and monolithic trees. The script that populates the modular tree with symlinks will continue to be used, and files will continue to be checked in from the monolithic tree. Additional release candidates will be tagged as bugs are fixed and the release stabilizes.

When the modular and monolithic trees reach the point where they could be released, the ,v files that have been symlinked will be copied within the CVS repository and a release candidate will be tagged (RCn in Figure 2). Additional testing will occur and any bugs caused by the copying of the ,v files will be fixed. Only major show stopper bug fixes will be accepted at that time, and they will need to be checked into both the monolithic and modular source trees.

When both releases are ready (Release Tag in Figure 2), the monolithic tree will be tagged with XORG-6_9_0 and the modular tree will be tagged with XORG-7_0_0. In addition, the 6.9 and 7.0 branches will created in their respective trees.

This plan was designed to ensure that the actual content of both the X11R6.9 and X11R7 releases are the same, so that no new features currently under development in the monolithic tree HEAD are left behind. Also, it allows those vendors who are not ready to move to a modular tree to have access to all of the latest features in the X11R6.9 release.

Future modular releases

Looking forward to the time after the initial release, modularization will change the way packages are added to the CVS tree and the release process. This section proposes how these tasks could be handled in the future.

Future X.Org Foundation releases can be constructed from an official list of the packages in the modular tree (along with specific package versions). This official release list should be maintained by the Architecture Working Group. The initial modular release will contain only what is included in the corresponding monolithic release.

After the initial release, we will open up the modules for new code and packages to be included. Each package will be placed in the most appropriate module. For example, applications can be added to the app module, libraries can be added to the lib module, etc. The maintainers will be responsible for freezing, stabilizing, and tagging for release their packages and can do so on their own schedules. This should help simply the official release process.

As new code or packages are added to the modular tree, the maintainers will need to request that their packages be added to the official release list, if they want them included in an official release. By default, their packages will not be included. A formal process for package maintainers to follow in order to have their code included in future official X.Org Foundation releases needs to be defined. This process is not something that we can work out in this proposal as it should be handled within the Architecture Working Group.

Official releases will be announced well in advance and with specific deadlines, so that both individual package maintainers whose packages are already included on the official release list can get their latest stable packages ready for inclusion, and package maintainers who want to have their packages included have the opportunity to discuss them with the Architecture Working Group before the feature freeze deadline.

Once the feature freeze deadline is reached, the release wranglers can determine which packages are included by looking at the official release list for the current release. Only packages within a module that are on the release list will be included in the release. However, if code within a package already on release list is not ready to ship (e.g., a new extension in the xserver module), then what is included in the official release becomes more complicated. The solution we propose is to require that unfinished development code live only on a branch until it is ready to be released. This solution will allow the new code to be developed on its own schedule and not hold up official releases. It will also help us to minimize the possibility of having to back out code during the release process.

During the stabilization phase of a release, inter-package testing and debugging can be done by installing the packages from the official package list. If problems are found during testing, bugs can be filed for the package maintainers to fix. Other details of how this process will happen should be worked out within the Release Wranglers.

Once the bugs have been resolved and the release wranglers feel the release is ready to ship, tarballs from the final packages on the official release list will be created and released.

• Updated 2005-03-31 in preparation to present to Architecture Working Group
• Updated 2005-04-11 and 2005-04-22 to remove inaccurate statements
• Updated 2005-04-29 to add anchors for external reference
• Updated 2005-05-13 to fix typo