The X11R6.9/X11R7 Release Testing Plan

The X11R6.9/X11R7 Release Testing Plan

Detailed test instructions

This section outlines the test procedure to follow when testing a release candidate. We need for everyone to test both the X11R6.9 (monolithic source tree) and the X11R7.0 (modular source tree). When you have completed a test run, please submit your test report to our results database so that we can track the progress of the release. We need these reports to determine if the code is ready for release or needs more work.

There are four main areas that we are interested in testing:

Build tests: Does the release build properly?
Install tests: Does the release install properly?
Conformance tests: Does the release conform to the specifications (i.e., does it pass the X Test Suite?)
Run tests: Does the release run the standard desktop and applications properly? Each of these are described below in detail.

Note that for any test(s) that fail, please file a bugzilla report in bugzilla and mark it as a blocker of the release bug.

Build tests

Monolithic build tests

For the monolithic source tree, each of the following build tests can be performed by copying the sample host.def file (or the alternate) to the xc/config/cf directory and the running make World >& World.LOG (or other such command as appropriate for your platform), and then checking the World.LOG file for any failures.

Build with empty host.def file (sample alternate)
Build with BuildServersOnly defined as YES (sample alternate)
Build with DoLoadableServer defined as NO (sample alternate) Note that some systems do not have a compatible version of Freetype2 installed on their system, so in addition to each build requirement above, defining HasFreetype2 as NO is permitted. Each alternate host.def file above have this define included.

Modular build tests

For the modular source tree, we ask that you build from CVS using the methods described in the Modular developer's guide. The build.sh script in CVS will allow you to build and install all of the module components in the appropriate order. By default, the build.sh script will exit if it fails to build or install any component.

If you would like to have the build.sh script warn you about build or install failures, then you can use the -n command-line flag to tell it to print error messages instead of exiting. For example, you could run build.sh -n /tmp/modular >& build.log and then once the script completes, you could grep '*****' build.log to see if there are any errors and which packages failed.

One additional test that further helps find problems is to have the build.sh script also run make distcheck on each package, which can be enabled with the -d command-line flag.

Install tests

Monlithic install tests

For the monolithic source tree, each of the following install tests can be performed by building the release (as described above using the sample or alternate host.def file provided), running make Install >& Install.LOG (or other such command as appropriate for your platform), and checking the Install.LOG output for any failures.

Build and install with no host.def file (sample alternate)
Build and install with: ProjectRoot defined to be something other than the default, and NothingOutsideProjectRoot defined as YES (sample alternate) Note that some systems do not have a compatible version of Freetype2 installed on their system, so in addition to each install requirement above, defining Has``Freetype2 as NO is permitted. Each alternate host.def file above have this define included.

Modular install tests

Since most of the modular source tree components have dependencies on other module components being built and installed, the build.sh script will both build and install all of the components in the appropriate dependency order. If the script completed without errors in the build step above, then the modular source tree both built and installed correctly.

Conformance tests

After installing the full release of either the monolithic, modular or both source trees, the conformance tests can be run using the X Test Suite (XTS). Note that XTS has been updated recently and the build instructions have changed from what they were for past releases.

Setting up the X Test Suite

We have prepared an XTS package for those that are running Linux on an x86 machine. You can download the prebuilt package here. Untar this package and cd into the xts5-Linux-i686 subdirectory.

For everyone else or if you have problems with the pre-built package, you will need to build XTS on your local system. To build it, download the mktestpkg.sh script from CVS and run it, which you can download here. This script will get the appropriate files, check out the latest version of XTS from CVS (Note when it asks you for a password, just hit enter). Then, change to the subdir that is created by the build script.

Now you should be ready to begin testing.

Note that an updated helper script (called xreg), which is used to run XTS, is already included in the test suite subdirectory.

Examples of how to use the xreg script

Here are some examples of how to use xreg to run the X test suite:

xreg -xtest -xvfb
- This runs xtest at all default depths using the Xvfb server.
- The default depths are 8, 15, 16, and 24+32.
- The "24+32" depth is one that uses a depth of 24 with a frame buffer bits per pixel of 32 (i.e., -depth 24 -fbbpp 32).
xreg -xtest -xorg -d 16
- This runs xtest at depth 16 using the Xorg server.
xreg -xtest -xvfb -d 15 -test XCopyArea
- This runs xtest at depth 15 using the Xvfb server, but it only runs the XCopy``Area test.
- Selecting individual tests is very useful to track down test failures.
xreg -xtest -xvfb -d 16 -xvfbwidth 1280 -xvfbheight 1024 -test XFillRectangles
- This runs xtest at depth 16 using the Xvfb server running at 1280x1024, but it only runs the XFill``Rectangles test. Notes on using xreg:
- The output from these test runs are stored in results subdirectory by default. You can change the default output dir using the -O xreg command-line option.
- The material below assumes that you have done a full install of the system to /usr/X11`R6. However, if you are using a differentProjectRoot(for the monolithic tree) or a different--prefix(for the modular tree), you can use the following command-line option to thexregscript to run from that alternate location:-projroot` path-to-your-project-root-or-prefix
- Running XTS at all bit depths can take a long time (6 hours on a P3 1GHz), so we suggest running them overnight.
- The files that are generated from an xreg run of xtest are:
  1. X-setup.DEPTH.DATE.TIME.output -- this file contains the output of the X server during the setup phase
  2. xts5.DEPTH.DATE.TIME.errors -- this file contains the list of errors found during the test run at depth DEPTH made on date DATE at time TIME.
  3. xts5.DEPTH.DATE.TIME.report -- this file contains the report of all tests run at depth DEPTH made on date DATE at time TIME.
  4. xts5.DEPTH.DATE.TIME.summary -- this file contains a summary of the errors found during the test run at depth DEPTH made on date DATE at time TIME. The summary file is only useful during full test runs (e.g., not when running individual tests).
  5. xts5.DEPTH.DATE.TIME.results -- this directory contains the journal from the tests run at depth DEPTH made on date DATE at time TIME as well as any error images generated.
- After running xtest, you can check to see if everything passed by looking at the summary/errors/report file(s) to see if there are any failures.
- As we get more test results in, we will be collecting a list of known failures. We can then update the xreg script to take those known errors into account in the summary file.
- The xreg script has only been tested on Linux systems. If there are problems with these scripts, please post patches to the xorg@lists.freedesktop.org mailing list.
- There are many other options to xreg (and it can be used to run other tests such as x11perf). Run xreg -help to see the usage message. After the XTS has been run with xreg, the report file lists the failures and a summary of the test run. Here is an example of the test report generated:

                           VSW5 SUMMARY RESULTS REPORT

Test suite version: 5.1.5
Specification version: Open Group Window Management (X11R5) document set
Test run by: guest
System: Linux t5 2.6.13-1.1576_FC5 #1 Sat Sep 24 15:23:34 EDT 2005 i686
Test run started: Monday November 07, 2005 11:52:31 PM
Test run ended:   Monday November 07, 2005 01:23:18 AM
Journal file: /tmp/xtest/xts5-Linux-i686/results/xts5.16bpp.20051107.204759.results/journal
TCC command line: tcc -e -s /tmp/xreg.tet_scen.26369 -x /tmp/xreg.tetexec.cfg.26369 -i /tmp/xtest/xts5-Linux-i686/results/xts5.16bpp.20051107.204759.results xts5 all
Report type: -d 1 -s 1


        CASES TESTS  PASS UNSUP UNTST NOTIU  WARN   FIP  FAIL UNRES  UNIN ABORT

Xproto    122   389   267     2   120     0     0     0     0     0     0     0
Xlib3     109   161   129     3    28     1     0     0     0     0     0     0
Xlib4      29   324   275    17    27     5     0     0     0     0     0     0
Xlib5      15    84    77     2     5     0     0     0     0     0     0     0
Xlib6       8    50    20     0    30     0     0     0     0     0     0     0
Xlib7      58   172   150     9    13     0     0     0     0     0     0     0
Xlib8      29   165   133    10    22     0     0     0     0     0     0     0
Xlib9      46  1472  1176    46    36   201     8     0     5     0     0     0
Xlib10     23    95    58     2    35     0     0     0     0     0     0     0
Xlib11     33   195    87    22    43    43     0     0     0     0     0     0
Xlib12     27   138   108     2    15    12     0     0     1     0     0     0
Xlib13     32   269   161     3   102     3     0     0     0     0     0     0
Xlib14     45    58    49     0     5     0     0     0     4     0     0     0
Xlib15     45   159   126     0    33     0     0     0     0     0     0     0
Xlib16     30   105    82     1    22     0     0     0     0     0     0     0
Xlib17     55   131   110     0    21     0     0     0     0     0     0     0
Xopen       8   127   125     2     0     0     0     0     0     0     0     0
Xt3        21    73    73     0     0     0     0     0     0     0     0     0
Xt4        33   192    94     0    98     0     0     0     0     0     0     0
Xt5        10    69    28     0    41     0     0     0     0     0     0     0
Xt6         7    71    71     0     0     0     0     0     0     0     0     0
Xt7        11   106    97     0     6     3     0     0     0     0     0     0
Xt8         7    43    35     0     4     0     0     0     4     0     0     0
Xt9        33   189   132     2    55     0     0     0     0     0     0     0
Xt10        8    17    16     0     1     0     0     0     0     0     0     0
Xt11       58   285   249     0    34     0     0     0     1     1     0     0
Xt12       22    67    56     0    11     0     0     0     0     0     0     0
Xt13       39   178   129     0    47     0     0     0     1     1     0     0
Xt14        2    18    18     0     0     0     0     0     0     0     0     0
Xt15        1     2     0     2     0     0     0     0     0     0     0     0
XtC        29   147    90     1    56     0     0     0     0     0     0     0
XtE         1     1     1     0     0     0     0     0     0     0     0     0

TOTAL     996  5552  4222   126   910   268     8     0    16     2     0     0

If the TOTAL line at the end of your report is similar to the one shown above, then we consider that your system has passed XTS. We are working to resolve the remaining failures and unresolved tests. From above, there are 16 failed and 2 unresolved tests. We will update the list above as we resolve more of these tests. If you are successful in fixing them, please let us know on the xorg@lists.freedesktop.org mailing list.

Actually running the conformance tests

For the purposes of the testing the monolithic and modular source trees for this release, we ask that you run one of the following scenarios:

For platforms based on a XFree86-style DDX, run XTS on Xorg either using the card driver for your video card or the dummy driver.
- For example: xreg -xtest -xorg
- This will run xtest at all the default depths using the Xorg server.
- Check the output of each report or summary file to make sure that all tests that are expected to pass do actually pass.
For all other platforms, Xvfb should be used. This step is optional for those who ran with the Xorg server above.
- For example: xreg -xtest -xvfb
- This will run xtest at all of the default depths using the Xvfb server.
- Check the output of each report or summary file to make sure that all tests that are expected to pass do actually pass. Additional notes:
The Xvfb server is special X server that uses a virtual framebuffer. It is normally built and installed with the full release. See the Xvfb(1) for more information about this server.
The dummy driver is a special driver available with the XFree86 DDX. To use the dummy driver, simply substitue it for your normal card driver in the Device section of your xorg.conf configuration file and comment out any card-specific options from that section. For example, if you normally uses an ATI driver, then you will have a Device section with Driver "ati" to let the X server know that you want it to load and use the ATI driver. It is not necessary to run both the card driver and the dummy driver for the conformance tests, but if you run into issues with the card driver not passing XTS, we recommend that you try running the dummy driver to see if the failures are resolved. This will help us narrow down any failures.

Run tests

After installing the full release and running the conformance tests, run the subset of tests listed below that applies to the platform being tested. We would like to get as much coverage as possible, so please run these tests on as many cards as possible.

Tests to run for each card:

X Test Suite (in case you haven't already run it)
x11perf (Note that you can use xreg to automatically run x11perf -- see the script for details)
rendertest (found in the xapps CVS repository on freedesktop.org)
Standard graphical environment (e.g., Gnome or KDE desktop)
GL tests: glxgears, gloss, quake3
Switch to/from VTs (on Linux) After running these tests, please enter them into our results database. This will allow us to track progress of the release. To see the current set of test results, go to the results summary page.