FTBFS

ARM specific package build failures

Overview

As of early 2011, armel is the Ubuntu architecture with most package build failures at any given time. This is due to lack of widespread hardware for developers to test on and to prevent the build issues stemming from differences from x86 and amd64 which most software in the archives is still mainly targeted at.

List of current issues

This page shows the currenly failed builds on all architectures.

http://qa.ubuntuwire.com/ftbfs/

Below is the list of issues filed on Ubuntu packages for ARM related porting issues, build failures, crashes and bugs in general. It is also the list being worked on on each Wednesday, during the Linaro ARM porting jams.

https://bugs.launchpad.net/ubuntu/+bugs?field.tag=arm-porting-queue&orderby=-id&start=0

There is (and should be) an overlap between the ARM package info on the above two pages.

Classes of issues

Build timeouts

These are due to a few packages requiring more physical RAM than the current ARM build server hardware has and time out due to excessive amounts of swapping. This issue will hopefully be solved during the Oneiric cycle by adding Pandaboards (1 GB RAM) to the build setup. There is no other solution so these packages should just be ignored for now.

The telltale sign of such packages is a line like this in the log of the failed build

"Build killed with signal 15 after 480 minutes of inactivity"

Many Haskell libraries are in this category as their profiling enabled builds require a lot of RAM. Some other examples include projects using very large C++ files with hairy include hierarchies.

https://bugs.launchpad.net/ubuntu/+bugs?field.tag=arm-build-timeout

Feel free to tag with arm-build-timeout other packages that fail like this on ARM. Sometimes the timeout is due to an actual bug in the packaging or the underlying kernel, but for ARM it is a sign of swapping, especially since it occurs in a compiler invocation step as seen by the last output.

Missing or incorrect platform specific code

Since the Ubuntu ARM port is a few years old already, most portability issues have been solved. Still some packages may remain that use assembly language that needs porting to ARM, or that have ARM assembly code but for an older architecture version than ARMv7 (which we build by default for) or have Thumb2 instruction set related failures. These all should be infrequent as upstreams are being increasingly aware of ARMv7/Thumb2.

A few examples are missing IT instruction before conditional execution in Thumb2 mode, using registers in asm code that are set aside by newer ABIs for specific purposes, use of deprecated instructions such as SWP, etc. Luckily the toolchain error messages are clear in these cases and the fix is straightforward if one knows a bit about ARM.

Some packages have configure checks written years ago for known platforms and thus do not detect ARM/armel/armv7. These should not be hard to fix, simply add ARM to their build-system.

OpenGL and Qt combination

Unlike x86 computers, the majority of consumer ARM devices are not equipped with GPUs that can hardware-accelarate OpenGL but instead support a subset of modern OpenGL called GLES - the latest most widely used version of which is GLES 2.0. This means that applications using GL need porting to GLES in order to perform acceptably on ARM.

Packages which are not ported fall back on the Mesa software renderer, which while not ideal, is not a package build failure so not interesting for this discussion.

Applications using only Qt for their UI work seamlessly on ARM, as here Qt is built to use its GLES 2.0 backend instead of OpenGL. However many applications use OpenGL directly along with Qt which causes conflicts on ARM due to the inclusion of both OpenGL and GLES headers, which have a considerable overlap.

Such applications need to be ported either to not use OpenGL (some uses are gratuitous or legacy and could be well covered via standard Qt API) or to provide a GLES port as well via conditional building or even runtime selection. This is not an easy task and is probably best done by upstreams.

The only other way to get around these failures is to go back to default Qt to OpenGL as it was in Ubuntu 10.10 and before. Gallium LLVMPipe rasterizer on ARM may prove to be good-enough sometime in the future.

C/C++ type definition differences

The Qt qreal type is float on ARM but double on other archs. If code uses double and qreal interchangeably it will work fine on x86 but break on ARM. Upstream app and library developers should fix the code. As Qt apps can rely a lot on generated code and metalanguages that are not plain standard C/C++, the fixes are not always a simple search and replace in C/C++ source code, but may require changes to build system or patches to the generating tools themselves.

Toolchain issues

As Linaro is very actively working on toolchain performance and generally making gcc better, regressions occur more frequently on ARM builds than on x86.

GCC internal errors

These are obvious errors as the package won't build and gcc asks you to file a bugreport with the ICE error. Usually a preprocessed source file that reproduces the issue is needed by toolchain developers to fix the issue.

More subtle errors

Sometimes packages FTBFS at the make test phase with segfaults or incorrect results. These can be due to bad code generation and can be verified by building with different versions of gcc and/or at different optimization levels. Safe bets are usually trying gcc-4.4 or gcc-4.5 as those

Linking errors

Unresolved symbols

When linking, library order matters, and some 'unresolved symbol' issues become apparent only on ARM. Can be fixed by reordering the libraries on the linker command line or adding needed libraries explicitly which on other arch may have been pulled in indirectly.

Multiarch libs not found

With the introduction of multiarch, some build systems which still hardcode or wrongly deduce that some libs are in /usr/lib instead of /usr/lib/${multiarch-dir} fail. These build systems need fixing upstream.

Missing APIs

Some APIs like ucontext are not available in ARM glibc so if a package uses this (deprecated) API, one needs to see if configurable alternatives are provided in the codebase.

Indirect failures

Some packages may need a certain versioned dependency which itself FTBFS so this too is prevented from building. Ex: ace is a large app which times out while building so anything depending on libace fails too.

Helping out

If you have ARM hardware to test on you can fix bugs as with any other package. Upstreaming and sending fixes to Debian is usually a good idea. Even if they do not yet have the issue (Ubuntu using newer toolchain, built for ARMv7, has different configuration and packaging choices) they may need it soon.

Bug triaging can be done even without trying any builds.

  • Close invalid bugs: Some bugs may get out of date if they are filed automatically on FTBFS but then are forgotten and not closed when a new build succeeds.
  • Check the issue with Debian/upstreams and forward upstream or link to upstream bugtracker patches
  • Tag them for easier retrieval: they all have the ftbfs and arm-porting-queue tags, but there can be other good ways to tag (arm-build-timeout , qt-opengl-arm, etc)

ARM/FTBFS (last edited 2011-07-13 19:57:34 by jani)