From Source to Executable
Now you know how source control works, and you’ve got a gigantic pile of source code sitting in a directory.
What do you do with it?
The process of turning source code into executable binary code can be extremely complicated. The more source code you have, the more complicated that process is. Almost every serious piece of software has its own build process, that every developer must follow — and woe be unto the developer who repeatedly breaks the build for everyone else.
In this chapter, you learn about how software is built. You learn about how the build process works in general, about some tools that you are likely to see, and you walk through a build for a FOSS project to see how it works in practice.
What is Building, Exactly?
There are many steps in the process of turning source code into a binary executable. Some examples of tasks that you might encounter during a typical build process:
- Linking object files and libraries. In the modern world, it’s crazy to write all of the code yourself. When you want to write output to the screen, you don’t write code that talks directly to the monitor; you use a library that handles input and output. When you want to play audio, you don’t handcode the waveforms yourself; you use audio codecs. When you compile the code, you almost always need to include libraries of one kind or another — which means you must know which libraries you need, and you must ensure that the libraries are where the compiler expects them to be, and that the libraries are all of the right version.
- Determining build order and dependencies. In complex software projects, it’s vital to keep track of dependencies. A change to code in a single library can have effects across your entire project, and might require some or all of your code to be recompiled — and often in a particular order. Keeping track of dozens of libraries, and references to those libraries by hundreds of source files, can be an ugly business.
- Testing the build results. It’s essential to know when you’ve introduced bugs sooner rather than later; new bugs are often easy to fix, and old bugs are often not so easy to fix. Also, it frequently happens that bugs, once fixed, creep back into code. Running basic tests on a project every time it’s built can be a good way to ensure that bugs get fixed and stay fixed.
- Packaging and/or Deploying. Sometimes you want to install the program you just compiled so that it can be run from anyhere on the system, and other programs or users can find it. Or sometimes you want to bundle it up into a format that allows anyone to take your executable and install it easily for themselves. You don’t want to do this for every build, but when you know that your build is good, one of the important final steps is to put the executable, and all documentation, in a central location.
Performing all of these tasks by hand would be time-consuming and difficult. Build automation tools allow the developer to handle all of these tasks automatically — and thus, to manage projects with a much higher degree of complexity.
Living With Complexity
Fair warning: sometimes code doesn’t compile. Sometimes you follow all the instructions, and it still doesn’t work. What then?
If this is your first experience dealing with a large codebase written by someone else, then welcome to the real world. As you run your first build, you may have very little idea of what’s actually going on. Don’t get discouraged. Have patience; you’ll soon begin to figure it all out.
You are about to walk through a software build process. The typical build command might return hundreds, or even thousands, of log entries, all of which scroll across the screen at lightning speed. You may understand all, some, or none of those entries.
That’s okay. Everyone starts somewhere.
Here are some points to keep in mind.
- Read instructions carefully. Almost every sizable FOSS project has a README or an INSTALL file that provides instructions for how to build and install the software. Read those instructions, and do your best to follow them to the letter. Understand that even the best instructions may leave out a step or two — and these are opportunities to improve the project.
- Don’t expect to understand every word. Very few developers understand every single word of every build log they encounter. Learning to distinguish between the important messages and the spurious messages takes time. Don’t be intimidated.
- Read logs carefully and thoughtfully. When you see an error, read back and think about what it could mean. Does the error say “couldn’t find something”? That probably means you didn’t install a library dependency properly. Was the error you found the only error in the log? In a 1000-line build log, the error at the end could be the result of another error dozens, or hundreds, of lines earlier. If your build doesn’t end happily, don’t panic. Relax and work your way through the problem.
- Google is your friend. If you don’t understand an error message, just Google it! Googling an error message can be a surprisingly effective method for determining what’s gone wrong. There’s a decent chance that someone before you has run into the same error, and someone has posted the solution to your problem on a message board or mailing list. Even if the answer isn’t obvious, there will frequently be clues. The Internet is a gigantic resource. Use it.
- Ask for help. If you’ve done your homework and still can’t figure out why your program isn’t building, get on the project’s mailing list, message board, or IRC channel, and ask for help. The more you’ve dug into the problem, and the more information you provide, the more likely it is that developers will help you figure out the problem.
Now it’s time to get on with it.
Building Freeciv: Watching GNU Autotools at Work
If you are working on a project that is written in C or C++, and that project is designed to run on Linux or UNIX, then it’s incredibly likely that you will be seeing GNU Autotools at work.
Developers use GNU Autotools to make sure that their users (that’s you) can compile and run a piece of software across a wide variety of hardware and software configurations.
You are now going to walk through the building of Freeciv. You checked out a local working repository of Freeciv in the last chapter, right? Now it’s time to turn all that code into a playable binary executable of the latest and awesomest version of Freeciv.
NOTE: Follow along with the sample build process, below. As you proceed through the build process, you may see many of the same errors; you may see completely different errors. No matter what happens, keep calm and carry on. Read the instructions. Don’t expect to understand every word. Read logs carefully and thoughtfully. Google is your friend. Ask for help.
Finding the Installation Instructions
Look for an INSTALL file in the top-level directory of the local repository. If you don’t find an INSTALL file, look for a README file. If you don’t find a README file, look for a script called configure and run it. And if you don’t find that, send a nice email to the maintainers of the project, and ask them if they could use some help with their installation instructions.
Oh, look, there’s an INSTALL file right there in the top level directory of the freeciv folder.
NOTE: the version of the INSTALL file referred to here is dated 22-Oct-2009. If it’s way out of date, then you’ll just have to buy the next edition of the textbook. Just kidding! Send in a patch, and we’ll fix it.
Every good INSTALL file tells you what the prerequisites are. If you find an INSTALL file that doesn’t, offer to fix it.
Freeciv does, though. Right there in the table of contents:
0. Prerequisites: 1. Prerequisites for the clients: 1a. Prerequisites for the Gtk+ client: 1b. Prerequisites for the SDL client: 1c. Prerequisites for the Xaw client:
There are two sets of requirements. One set of requirements is listed for general building with the following list:
- Unix (or similar).
- An ANSI C compiler
- A “make” program
- The programs from GNU gettext version 0.10.36 or better
- GNU autoconf, version 2.58 or better
- GNU automake, version 1.6 or better
Then another set of requirements is required for building the Freeciv clients, which can actually be compiled in different flavors, and each flavor has a different set of dependencies. You are just building the Gtk+ client, which means:
- “Glib” greater or equal to 2.4.0
- The “Atk” accessibility library
- The “Pango” text layout and rendering library
- The “Gtk+” widget library greater or equal to 2.4.0
That’s a lot of stuff. How do you get all that stuff?
This is where Linux distributions shine. All modern Linux distributions have package management programs that allow for easy location and installation of FOSS software. In this example case, presume the Fedora distribution, but Ubuntu uses similar commands and works in largely the same way.
First, make sure that you have a C compiler. In fact, use the compiler that Freeciv recommends in the INSTALL file: gcc.
[gregdek@ip-10-242-118-147 freeciv]$ rpm -q gcc package gcc is not installed
Hm! You have your first problem.
RPM is a program that maintains a list of software packages installed on the system. With this command, you asked, “tell me if you have gcc installed”. And RPM said “nope, sorry.” Which means you turn to RPM’s big brother, yum. (In the Ubuntu/Debian world, these commands would be dpkg and apt-get, respectively.)
Next you ask yum to install gcc:
[gregdek@ip-10-242-118-147 freeciv]$ yum install gcc Loaded plugins: fastestmirror You need to be root to perform this command.
Oh, right. If you’re going to work with Linux, you need to know when you need to be regular user, and when you need to be root. When you’re adding new software to the system, to be accessed by other programs and potentially other users, you need to be root. (You can also use the su -c command, which allows you to masquerade as root.)
[root@ip-10-242-118-147 ~]# yum install gcc Loaded plugins: fastestmirror Determining fastest mirrors * updates-newkey: kdeforge.unl.edu * fedora: kdeforge.unl.edu * updates: kdeforge.unl.edu updates-newkey | 2.3 kB 00:00 fedora | 2.1 kB 00:00 updates | 2.6 kB 00:00 Setting up Install Process Parsing package install arguments Resolving Dependencies --> Running transaction check ---> Package gcc.i386 0:4.1.2-33 set to be updated --> Processing Dependency: glibc-devel >= 2.2.90-12 for package: gcc --> Running transaction check ---> Package glibc-devel.i386 0:2.7-2 set to be updated --> Finished Dependency Resolution Dependencies Resolved ================================================================================ Package Arch Version Repository Size ================================================================================ Installing: gcc i386 4.1.2-33 fedora 5.2 M Installing for dependencies: glibc-devel i386 2.7-2 fedora 2.0 M Transaction Summary ================================================================================ Install 2 Package(s) Update 0 Package(s) Remove 0 Package(s) Total download size: 7.2 M Is this ok [y/N]:
Lots of interesting information here! And you already start to see how software fits together. The programs yum and rpm work together to make sure that when you choose to install gcc, you also get everything that gcc needs to be useful — in this case, the header and object files necessary for developing programs that use the standard C libraries. You asked for one software package, but now you get two.
Is this ok [y/N]: y Downloading Packages: (1/2): glibc-devel-2.7-2.i386.rpm | 2.0 MB 00:00 (2/2): gcc-4.1.2-33.i386.rpm | 5.2 MB 00:00 -------------------------------------------------------------------------------- Total 6.4 MB/s | 7.2 MB 00:01 ============================== Entering rpm code =============================== Running rpm_check_debug Running Transaction Test Finished Transaction Test Transaction Test Succeeded Running Transaction Installing : glibc-devel 1/2 Installing : gcc 2/2 =============================== Leaving rpm code =============================== Installed: gcc.i386 0:4.1.2-33 Dependency Installed: glibc-devel.i386 0:2.7-2 Complete!
All right, that’s one dependency down. Which also means that you can build anything that needs GCC in the future, so that’s useful.
Now you need a make program. They recommend gmake, so see if it’s installed.
[root@ip-10-242-118-147 ~]# rpm -q gmake package gmake is not installed
All right, install it.
[root@ip-10-242-118-147 ~]# yum install gmake Loaded plugins: fastestmirror Loading mirror speeds from cached hostfile * updates-newkey: kdeforge.unl.edu * fedora: kdeforge.unl.edu * updates: kdeforge.unl.edu Setting up Install Process Parsing package install arguments No package gmake available. Nothing to do
Wait, what’s this? It appears that gmake isn’t installed, and isn’t available? What’s going on?
Well, upon reading the INSTALL instructions more closely, there is this nugget:
You can check if you have GNU make installed on your system by typing: % make -v [and if this doesn't work, try "gmake -v"] The output should include "GNU Make" somewhere.
Get in the habit of reading instructions.
[root@ip-10-242-118-147 ~]# make -v GNU Make 3.81 Copyright (C) 2006 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. This program built for i386-redhat-linux-gnu
All right, you’ve got make. How about GNU gettext?
[root@ip-10-242-118-147 ~]# rpm -q gettext gettext-0.16.1-12.fc8
Very good. Autoconf?
[root@ip-10-242-118-147 ~]# rpm -q autoconf package autoconf is not installed [root@ip-10-242-118-147 ~]# yum install -y autoconf (snip lots of yum output) Installed: autoconf.noarch 0:2.61-9.fc8 Dependency Installed: imake.i386 0:1.0.2-5.fc8 Complete!
All set. Automake?
[root@ip-10-242-118-147 ~]# yum install -y automake (snip lots of yum output) Installed: automake.noarch 0:1.10-6 Complete!
Note that this time you didn’t even bother to see if the RPM was installed first; you just installed it, because if automake had already been installed, yum would have let us know:
[root@ip-10-242-118-147 ~]# yum install -y automake Loaded plugins: fastestmirror Loading mirror speeds from cached hostfile * updates-newkey: kdeforge.unl.edu * fedora: kdeforge.unl.edu * updates: kdeforge.unl.edu Setting up Install Process Parsing package install arguments Package automake-1.10-6.noarch already installed and latest version Nothing to do
That’s half of the prerequisites. Time for the other half.
[root@ip-10-242-118-147 ~]# yum install pkg-config (snip) No package pkg-config available. Nothing to do [root@ip-10-242-118-147 ~]# yum install pkgconfig (snip) Package 1:pkgconfig-0.22-4.fc8.i386 already installed and latest version Nothing to do
Okay, so you’ve got pkg-config installed, even though they seem to be calling it pkgconfig for some reason, which was discovered through a lucky guess — noting that in the INSTALL file, even though they call it pkg-config, the file in question is called pkgconfig-0.14.0.tar.gz. These kinds of little inconsistencies are maddening, and you will find them everywhere in your software life, so get used to them.
Now for Glib:
[root@ip-10-242-118-147 ~]# yum install Glib (snip) No package Glib available. * Maybe you meant: glib Nothing to do
Oh, maybe you did mean glib. Thanks!
[root@ip-10-242-118-147 ~]# yum install glib (snip) Installed: glib.i386 1:1.2.10-28.fc8 Complete!
This is kind of slow, isn’t it? Can you specify multiple packages to be installed at once? Indeed you can.
[root@ip-10-242-118-147 ~]# yum install atk pango gtk+ (snip) Package atk-1.20.0-1.fc8.i386 already installed and latest version (snip) Installed: gtk+.i386 1:1.2.10-59.fc8 pango.i386 0:1.18.4-1.fc8 Dependency Installed: cairo.i386 0:1.4.14-1.fc8 libX11.i386 0:1.1.3-4.fc8 libXext.i386 0:1.0.1-4.fc8 libXft.i386 0:2.1.12-3.fc8 libXi.i386 0:1.1.3-1.fc8 libXrender.i386 0:0.9.4-1.fc8 xorg-x11-filesystem.noarch 0:7.1-2.fc6 Complete!
Or so it would appear — but appearances can be deceiving.
Once you have all of the prerequisites installed, the next step is to run the configure script.
The configure script, in this case, is generated by the GNU Autotools, a set of tools that examine lots and lots (and lots and lots) of variables about your system. It checks your compiler, it checks your hardware, it checks all kinds of stuff, and as a result of all of these checks (literally hundreds), it builds a makefile that the compiler uses to build the binary executable.
Simple, right? Give it a try. But first, read the instructions:
2. Generating Makefiles ======================= This section contains two parts, one for generating makefiles from svn versions and one for generating makefiles from release versions. 2a. Generating the Makefile for svn versions: ============================================= This step is only needed for svn versions. To create the makefile just type % ./autogen.sh This will create the configure script and will run it. All parameters of autogen.sh are passed to configure. Read the next section about the parameters which can be passed to configure.
All right, seems simple enough. Run it and see what happens:
[gregdek@ip-10-242-118-147 freeciv]$ ./autogen.sh + checking for autoconf >= 2.58 ... found 2.61, ok. + checking for autoheader >= 2.58 ... found 2.61, ok. + checking for automake >= 1.6 ... found 1.10, ok. + checking for aclocal >= 1.6 ... found 1.10, ok. + checking for libtoolize >= 1.4.3 ... You must have libtoolize installed to compile freeciv. Download the appropriate package for your distribution, or get the source tarball at ftp://ftp.gnu.org/pub/gnu/libtool/ + checking for xgettext >= 0.10.36 ... found 0.16.1, ok. + checking for msgfmt >= 0.10.36 ... found 0.16.1, ok.
Oops. Looks like the script found a missing dependency that’s not documented! Fortunately, GNU Autotools found it. Install libtoolize — or libtool, which is it? Anyway, it’s probably one of them:
[root@ip-10-242-118-147 FREECIV]# yum install libtoolize (snip) No package libtoolize available. Nothing to do [root@ip-10-242-118-147 FREECIV]# yum install libtool (snip) Installed: libtool.i386 0:1.5.24-3.fc8 Complete!
All right, try that again.
[gregdek@ip-10-242-118-147 freeciv]$ ./autogen.sh + checking for autoconf >= 2.58 ... found 2.61, ok. + checking for autoheader >= 2.58 ... found 2.61, ok. + checking for automake >= 1.6 ... found 1.10, ok. + checking for aclocal >= 1.6 ... found 1.10, ok. + checking for libtoolize >= 1.4.3 ... found 1.5.24, ok. + checking for xgettext >= 0.10.36 ... found 0.16.1, ok. + checking for msgfmt >= 0.10.36 ... found 0.16.1, ok. + running aclocal ... + running autoheader ... + running autoconf ... + running libtoolize ... Putting files in AC_CONFIG_AUX_DIR, `bootstrap'. + running automake ... configure.ac:63: installing `bootstrap/missing' configure.ac:63: installing `bootstrap/install-sh' ai/Makefile.am: installing `bootstrap/depcomp' common/Makefile.am:97: `%'-style pattern rules are a GNU make extension utility/Makefile.am:60: `%'-style pattern rules are a GNU make extension + removing config.cache ... + running configure ... I am going to run ./configure with no arguments - if you wish to pass any to it, please specify them on the ./autogen.sh command line.
OK, so far so good! It’s successfully created a configure script, and now it’s running that script. Fingers crossed…
checking build system type... i686-pc-linux-gnu checking host system type... i686-pc-linux-gnu checking for a BSD-compatible install... /usr/bin/install -c checking whether build environment is sane... yes checking for a thread-safe mkdir -p... /bin/mkdir -p checking for gawk... gawk checking whether make sets $(MAKE)... yes checking for style of include used by make... GNU checking for gcc... gcc checking for C compiler default output file name... a.out checking whether the C compiler works... yes checking whether we are cross compiling... no checking for suffix of executables... checking for suffix of object files... o checking whether we are using the GNU C compiler... yes checking whether gcc accepts -g... yes checking for gcc option to accept ISO C89... none needed checking dependency style of gcc... gcc3 checking how to run the C preprocessor... gcc -E checking for grep that handles long lines and -e... /bin/grep checking for egrep... /bin/grep -E checking for ANSI C header files... yes checking for sys/types.h... yes checking for sys/stat.h... yes checking for stdlib.h... yes checking for string.h... yes checking for memory.h... yes checking for strings.h... yes checking for inttypes.h... yes checking for stdint.h... yes checking for unistd.h... yes checking for gethostbyname2... yes checking for inet_pton... yes checking for inet_ntop... yes checking for getnameinfo... yes checking for AF_INET6... yes checking for a sed that does not truncate output... /bin/sed checking for gawk... (cached) gawk checking for gcc... (cached) gcc checking whether we are using the GNU C compiler... (cached) yes checking whether gcc accepts -g... (cached) yes checking for gcc option to accept ISO C89... (cached) none needed checking dependency style of gcc... (cached) gcc3 checking how to run the C preprocessor... gcc -E checking for g++... no checking for c++... no checking for gpp... no checking for aCC... no checking for CC... no checking for cxx... no checking for cc++... no checking for cl.exe... no checking for FCC... no checking for KCC... no checking for RCC... no checking for xlC_r... no checking for xlC... no checking whether we are using the GNU C++ compiler... no checking whether g++ accepts -g... no checking dependency style of g++... none checking whether the C++ compiler works... no configure: error: no acceptable C++ compiler found in $PATH See `config.log' for more details.
Wait… what? But… but… it’s written in C! Why do you need a C++ compiler for C?
Annoyed, Google “freeciv c++”, looking for clues. (Remember: Google is your friend.) And on the first page of search results, there is this gem of wisdom:
“Freeciv is a free turn-based multiplayer strategy game, in which each player becomes the leader … Mostly written in C.”
Mostly written in C. Following a few links, discover that at least part of it is written in C++. Sigh. So, install the GNU C++ compiler — Googling “gnu c++ fedora” tells us that the package name is likely called gcc-c++.
[root@ip-10-242-118-147 ~]# yum install gcc-c++ (snip) Installed: gcc-c++.i386 0:4.1.2-33 Dependency Installed: libstdc++-devel.i386 0:4.1.2-33 Complete!
[gregdek@ip-10-242-118-147 freeciv]$ ./autogen.sh + checking for autoconf >= 2.58 ... found 2.61, ok. + checking for autoheader >= 2.58 ... found 2.61, ok. + checking for automake >= 1.6 ... found 1.10, ok. (snip to where we broke the last time...) checking for g++... g++ checking whether we are using the GNU C++ compiler... yes (snip to where we fail again...) checking for gzgets in -lz... no configure: error: Could not find zlib library. configure failed
So take stock at this point of where you are.
Freeciv is a good program. It runs on many platforms. There are dozens of active developers working on pieces of the codebase at any given time. And yet, in the space of 15 minutes, you have already found a handful of places where the documentation could be improved — and you haven’t even made it to a first successful build!
This is what complex software is like. Getting it 100% right is incredibly hard, and there are always little places to improve.
Anyway, back to business. Missing zlib? Install zlib.
[root@ip-10-242-118-147 ~]# yum install zlib (snip) Package zlib-1.2.3-14.fc8.i386 already installed and latest version Nothing to do
Many FOSS projects split their libraries in two. They provide runtime libraries, and they provide development libraries. In fact, if you read the INSTALL document all the way down to section 10:
To compile freeciv on a debian system you need the following packages: Common requirements: gcc libc6-dev libreadline4-dev zlib1g-dev xlib6g-dev
Well, this isn’t a Debian system, it’s a Fedora system — but the principle is the same. You need development libraries. Googling “fedora zlib” returns, in the top link, a reference to zlib-devel. So try that:
[root@ip-10-242-118-147 ~]# yum install zlib-devel (snip) Installed: zlib-devel.i386 0:1.2.3-14.fc8 Complete!
Try, try again.
[gregdek@ip-10-242-118-147 freeciv]$ ./autogen.sh (snip to the next error...) configure: error: could not guess which client to compile configure failed
It clearly says in the INSTALL file (you did read the INSTALL file very closely, didn’t you?) that “Gtk+” is the default client. So what guess is required?
Going back a little farther up in the log, though, you see the problem:
configure: checking for which client to compile:... checking for pkg-config... /usr/bin/pkg-config checking for GTK+ - version >= 2.4.0... no *** Could not run GTK+ test program, checking why... *** The test program failed to compile or link. See the file config.log for the *** exact error that occured. This usually means GTK+ is incorrectly installed. checking for sdl-config... no checking for SDL - version >= 1.1.4... no *** The sdl-config script installed by SDL could not be found *** If SDL was installed in PREFIX, make sure PREFIX/bin is in *** your path, or set the SDL_CONFIG environment variable to the *** full path to sdl-config. checking for X... no checking whether Xfuncproto was supplied... no, found: FUNCPROTO=15 NARROWPROTO checking for Xfuncproto control definition FUNCPROTO... yes: 15 checking for Xfuncproto control definition NARROWPROTO... yes: 1 checking pkg-config is at least version 0.9.0... yes checking for PNG... no checking for png_read_image in -lpng12... no checking for png_read_image in -lpng... no checking png.h usability... no checking png.h presence... no checking for png.h... no checking extra paths for Xpm... library no, include no checking for XOpenDisplay in X library -lX11... no checking will compile gui-ftwl... no checking will compile gui-beos... no configure: error: could not guess which client to compile configure failed
It seems to want a GTK+ version of greater than 2.4.0. Isn’t that what you installed?
[gregdek@ip-10-242-118-147 freeciv]$ rpm -q gtk+ gtk+-1.2.10-59.fc8
Hm! Stuck? Once again, Google is your friend. Google for “Fedora gtk+” and the very first link is to a discussion on a mailing list, where someone asks, “is there a gtk+2?” And a helpful Fedora community member says, “yes, but it’s called gtk2”.
So try that.
[root@ip-10-242-118-147 ~]# yum install gtk2 (snip) Installed: gtk2.i386 0:2.12.8-2.fc8 Dependency Installed: cups-libs.i386 1:1.3.9-2.fc8 libXcomposite.i386 0:0.4.0-3.fc8 libXcursor.i386 0:1.1.9-1.fc8 libXfixes.i386 0:4.0.3-2.fc8 libXinerama.i386 0:1.0.2-3.fc8 libXrandr.i386 0:1.2.2-1.fc8 Complete!
OK, that’s a lot of extra stuff, but if gtk2 needs it, then gtk2 needs it.
Try, try again.
(snip) checking for GTK+ - version >= 2.4.0... no *** Could not run GTK+ test program, checking why... *** The test program failed to compile or link. See the file config.log for the *** exact error that occured. This usually means GTK+ is incorrectly installed. (snip) configure: error: could not guess which client to compile configure failed
Same exact error. Maybe there’s a gtk2-devel?
[root@ip-10-242-118-147 ~]# yum install gtk2-devel (snip) Installed: gtk2-devel.i386 0:2.12.8-2.fc8 Dependency Installed: atk-devel.i386 0:1.20.0-1.fc8 cairo-devel.i386 0:1.4.14-1.fc8 docbook-dtds.noarch 0:1.0-33.fc8 docbook-style-dsssl.noarch 0:1.79-4.1 docbook-style-xsl.noarch 0:1.73.2-5.fc8 docbook-utils.noarch 0:0.6.14-11.fc8 fontconfig-devel.i386 0:2.4.2-5.fc8 freetype-devel.i386 0:2.3.5-5.fc8 gc.i386 0:7.0-6.fc8 glib2-devel.i386 0:2.14.6-2.fc8 gtk-doc.noarch 0:1.9-4.fc8 libX11-devel.i386 0:1.1.3-4.fc8 libXau-devel.i386 0:1.0.3-3.fc8 libXcursor-devel.i386 0:1.1.9-1.fc8 libXdamage.i386 0:1.1.1-3.fc8 libXdmcp-devel.i386 0:1.0.2-4.fc8 libXext-devel.i386 0:1.0.1-4.fc8 libXfixes-devel.i386 0:4.0.3-2.fc8 libXft-devel.i386 0:2.1.12-3.fc8 libXi-devel.i386 0:1.1.3-1.fc8 libXinerama-devel.i386 0:1.0.2-3.fc8 libXrandr-devel.i386 0:1.2.2-1.fc8 libXrender-devel.i386 0:0.9.4-1.fc8 libXxf86vm.i386 0:1.0.1-4.fc8 libpng-devel.i386 2:1.2.33-1.fc8 libxcb-devel.i386 0:1.1-1.1.fc8 mesa-libGL.i386 0:7.0.2-3.fc8 mesa-libGL-devel.i386 0:7.0.2-3.fc8 openjade.i386 0:1.3.2-30.fc8 opensp.i386 0:1.5.2-6.fc8 pango-devel.i386 0:1.18.4-1.fc8 perl-SGMLSpm.noarch 0:1.03ii-16.2.1 rarian.i386 0:0.6.0-4.fc8 rarian-compat.i386 0:0.6.0-4.fc8 sgml-common.noarch 0:0.6.3-21.fc8 w3m.i386 0:0.5.2-5.fc8 xml-common.noarch 0:0.6.3-21.fc8 xorg-x11-proto-devel.noarch 0:7.3-3.fc8 Complete!
There is, indeed — and it brings in a ton of dependencies! Including -devel versions of other dependencies you thought you’d satisfied, such as atk-devel and cairo-devel.
Imagine if you had to sort through all of these dependencies yourself, by hand. With all of the effort you’ve already gone through to identify and satisfy a relatively small number of dependencies, imagine the work that would be required. As imperfect as this process clearly is, it could be orders of magnitude worse.
Try, try again.
****************** Configuration Summary ****************** Build freeciv client: yes Debugging support: some Client frontends: Gtk-2.0: yes SDL: no Xaw: no Win32: no FTWL: no Stub: no Build freeciv server: yes Debugging support: some A uth database support: no Now type 'make' to compile freeciv.
It worked! It worked it worked it worked it worked!
You are, at long last, ready to make.
Look, once again, at the excellent instructions:
If all has gone well previous to this point, then compiling Freeciv should be as easy as typing "make" (or preferably, "gmake"). If you have problems, read the file BUGS, and follow the advice carefully. If the problem is with "gettext", please read the Native Language Support section, below, for possible work-arounds. After compilation, the important results are: - The "client/freeciv-<GUI>" and "server/freeciv-server" binaries. - The "data/" directory, which contains the graphics and scenarios. - The "po/" directory, which contains the localization files. - The "civ" and "ser" scripts. It's perfectly feasible to play Freeciv in this directory, without installing it. If you do this, the "civ" and "ser" scripts may be useful, although they are not as necessary as they used to be. See the README file for more information.
Do as it says. In fact, try a new trick:
[gregdek@ip-10-242-118-147 freeciv]$ make 1>/tmp/freeciv-make.out 2>/tmp/freeciv-make.err &  1517 [gregdek@ip-10-242-118-147 freeciv]$
Compiling an entire software project can take quite a while, and generates a prodigious amount of data — and watching a bazillion lines of code fly by is only fun for the first few seconds. So this old-school UNIX command lets you capture all that output to look at it later. The make command is the make command, of course. The 1>/tmp/freeciv-make.out option tells the job to put the standard output into /tmp/freeciv-make.out, and 2>/tmp/freeciv-make.err tells the job to put error messages into /tmp/freeciv-make.err, and & tells the job to run in the background, so that you can be free to do other things. When the job completes, it tells you so.
As a truly ridiculous amount of computation takes place, wander off for a cup of coffee, and take this time to engage in serious business. When you come back, hit enter, and see on the screen:
+ Done make > /tmp/freeciv-make.out 2> /tmp/freeciv-make.err
Ah, very good. Now have a look at the log files. How long are they? Here’s the clever wc (word count) command, with the -l parameter to show the number of lines instead of number of words:
[gregdek@ip-10-242-118-147 freeciv]$ wc -l /tmp/freeciv-make.out 1148 /tmp/freeciv-make.out [gregdek@ip-10-242-118-147 freeciv]$ wc -l /tmp/freeciv-make.err 551 /tmp/freeciv-make.err [gregdek@ip-10-242-118-147 freeciv]$
Whoa! 551 errors? Take a look at that.
[gregdek@ip-10-242-118-147 freeciv]$ less /tmp/freeciv-make.err
The less command is a tool that allows you to scroll through a text file, search the file for text, and so on. The command man less gives you the manual page for the less command. For now, the page-up and page-down keys can take you through the file. Here are some sample lines from that error file:
packets_gen.c: In function 'receive_packet_spaceship_info_100': packets_gen.c:23371: warning: dereferencing type-punned pointer will break strict-aliasing rules packets_gen.c: In function 'send_packet_spaceship_info_100': packets_gen.c:23560: warning: dereferencing type-punned pointer will break strict-aliasing rules packets_gen.c: In function 'receive_packet_ruleset_unit_100': packets_gen.c:23830: warning: dereferencing type-punned pointer will break strict-aliasing rules packets_gen.c: In function 'send_packet_ruleset_unit_100': packets_gen.c:24178: warning: dereferencing type-punned pointer will break strict-aliasing rules packets_gen.c: In function 'receive_packet_ruleset_game_100': packets_gen.c:24743: warning: dereferencing type-punned pointer will break strict-aliasing rules packets_gen.c: In function 'send_packet_ruleset_game_100': packets_gen.c:24824: warning: dereferencing type-punned pointer will break strict-aliasing rules
Hmm, lots and lots of warnings. In fact, paging up and down through the whole file quickly reveals that the error file is full of nothing but warnings. Not ideal, and maybe something that someone should fix (patches welcome), but generally, warnings don’t prevent a program from compiling.
Next look at the regular output of the make. Actually, use the tail command, which just shows the end of the file (the last 10 lines by default):
[gregdek@ip-10-242-118-147 freeciv]$ tail /tmp/freeciv-make.out gcc -DHAVE_CONFIG_H -I. -I.. -I../server -I../utility -I../common -I../ai -I../common/aicore -I../server/generator - I../client -I../client/include -DLOCALEDIR="\"/usr/local/share/locale\"" -DDEFAULT_DATA_PATH="\".:data:~/.freeciv/2.3:/usr/local/share/freeciv\"" - DDEFAULT_SAVES_PATH="\"\"" -DDEFAULT_SCENARIO_PATH="\".:data/scenario:~/.freeciv/scenarios:/usr/local/share/freeciv/scenario\"" -Wall - Wpointer-arith -Wcast-align -Wmissing-prototypes -Wmissing-declarations -g -O2 -MT civmanual.o -MD -MP -MF .deps/civmanual.Tpo -c -o civmanual.o civmanual.c mv -f .deps/civmanual.Tpo .deps/civmanual.Po /bin/sh ../libtool --preserve-dup-deps --tag=CC --mode=link gcc -Wall -Wpointer-arith -Wcast-align -Wmissing-prototypes -Wmissing- declarations -g -O2 -o civmanual civmanual.o ../server/libfreeciv-srv.la ../client/helpdata.lo ../server/scripting/libscripting.la ../dependencies/lua-5.1/src/liblua.a ../dependencies/toluaxx/src/lib/libtolua.a ../server/generator/libgenerator.la ../common/libfreeciv.la -lm -lz mkdir .libs gcc -Wall -Wpointer-arith -Wcast-align -Wmissing-prototypes -Wmissing-declarations -g -O2 -o civmanual civmanual.o ../client/helpdata.o ../server/.libs/libfreeciv-srv.a ../server/scripting/.libs/libscripting.a ../dependencies/lua-5.1/src/liblua.a ../dependencies/toluaxx/src/lib/libtolua.a ../server/generator/.libs/libgenerator.a ../common/.libs/libfreeciv.a -lm -lz make: Leaving directory `/home/gregdek/FREECIV/freeciv/manual' make: Entering directory `/home/gregdek/FREECIV/freeciv' make: Nothing to be done for `all-am'. make: Leaving directory `/home/gregdek/FREECIV/freeciv' make: Leaving directory `/home/gregdek/FREECIV/freeciv' [gregdek@ip-10-242-118-147 freeciv]$
That’s what make looks like when it succeeds. Be sure that you successfully generated the important bits, as you recall from the INSTALL guide:
After compilation, the important results are: - The "client/freeciv-<GUI>" and "server/freeciv-server" binaries. - The "data/" directory, which contains the graphics and scenarios. - The "po/" directory, which contains the localization files. - The "civ" and "ser" scripts.
See if you find these by using the ls command.
[gregdek@ip-10-242-118-147 freeciv]$ ls client/freeciv-* client/freeciv-gtk2 [gregdek@ip-10-242-118-147 freeciv]$ ls server/freeciv-server server/freeciv-server [gregdek@ip-10-242-118-147 freeciv]$ ls data/ Freeciv civclient.dsc freeciv-server.icns isotrident Freeciv.in civclient.dsc.in freeciv-server.png isotrident.tilespec Makefile civserver.dsc freeciv.rc misc Makefile.am civserver.dsc.in freeciv.rc-2.0 nation Makefile.in civserver.room graphics scenario amplio civserver.room.in gtk_menus.xml stdsounds amplio.tilespec default helpdata.txt stdsounds.soundspec buildings default.serv hex2t themes civ1 flags hex2t.tilespec trident civ1.serv fonts icons trident.tilespec civ2 freeciv-client.icns isophex wonders civ2.serv freeciv-client.png isophex.tilespec [gregdek@ip-10-242-118-147 freeciv]$ ls po/ ChangeLog cs.gmo es.gmo he.po nb.po ro.po Makefile cs.po es.po hu.gmo nl.gmo ru.gmo Makefile.in da.gmo et.gmo hu.po nl.po ru.po Makefile.in.in da.po et.po it.gmo no.gmo statistics.rb POTFILES de.gmo et.po.sig it.po no.po sv.gmo POTFILES.in de.po fa.gmo ja.gmo pl.gmo sv.po POTFILES.skip el.gmo fa.po ja.po pl.po tr.gmo ar.gmo el.po fi.gmo ko.gmo pt.gmo tr.po ar.po en_GB.gmo fi.po ko.po pt.po uk.gmo ca.gmo en_GB.po fr.gmo lt.gmo pt_BR.gmo uk.po ca.po eo.gmo fr.po lt.po pt_BR.po zh_CN.gmo check_po.pl eo.po he.gmo nb.gmo ro.gmo zh_CN.po [gregdek@ip-10-242-118-147 freeciv]$ ls civ civ [gregdek@ip-10-242-118-147 freeciv]$ ls ser ser [gregdek@ip-10-242-118-147 freeciv]$
That certainly looks like everything was generated — but the proof is in the pudding. Does the code run? Run the ser script, which starts the freeciv server, just to make sure:
[gregdek@ip-10-242-118-147 freeciv]$ ./ser This is the server for Freeciv version 2.2.99-dev You can learn a lot about Freeciv at http://www.freeciv.org/ 2: Loading rulesets 2: AI*1 has been added as Easy level AI-controlled player. 2: AI*2 has been added as Easy level AI-controlled player. 2: AI*3 has been added as Easy level AI-controlled player. 2: AI*4 has been added as Easy level AI-controlled player. 2: AI*5 has been added as Easy level AI-controlled player. 2: Now accepting new client connections. For introductory help, type 'help'. > quit Goodbye. [gregdek@ip-10-242-118-147 freeciv]$
It works! Of course, the only real way to tell if it works is to play a very long game of Freeciv, but that is an exercise left to the reader.
Review: What Just Happened?
Now that you’ve successfully transformed code into gold, step back and take a look at what’s going on. The process you just walked through is virtually identical to a large number of FOSS projects, so you’re likely to see it often.
Where do all of these complicated configuration files come from in the first place, and why?
The goal of good build tools is to allow the developer to describe the project as simply as possible. There’s a lot of complexity, but it’s important to isolate that complexity.
Developers of an Autotools-based project seek to describe the build process of the entire project in two files:
- Makefile.am, which (basically) describes what needs to be built;
- configure.in (or configure.ac), which (basically) describes how it needs to be built.
These descriptions start out simple, but as the project grows in complexity, the descriptions of the project naturally grow as well.
Given these configuration files, you ran through the Autotools suite. The purpose of the Autotools suite is to build a good configure script, which in turn builds a good Makefile, which in turn builds good executable binaries.
+--------------+ | configure.in | | Makefile.am | +--------------+ | | Run autogen.sh | (a wrapper for the Autogen tools) v +--------------+ | configure | +--------------+ | | ./configure | (repeat as necessary) | v +--------------+ | Makefile | +--------------+ | | make | v +--------------+ | Lots of | | successfully | | compiled | | code | +--------------+
Note: one of the things you did not do was to run the make install command, which would install Freeciv as an “official” binary that could be run by anyone on the system. Since this is a development version, keep it in your own working directory, where you can happily break it as much as you want.
Again: very few people know every detail about Autotools, or about any build tools. Experienced developers know as much as they need to know to get the job done, and they know where to look for help when they’re stuck.
Exercise – Building Your Developer Workstation
By now, you have probably chosen a FOSS project that interests you. Maybe you’ve been running that code from a pre-built binary that you downloaded, but now it’s time to build the executables from scratch.
Go check out the project’s codebase from its SCM. Walk through the entire build process. In some cases, this might be pretty simple; it some cases, it might be quite complicated. Use all of the tools available to you: install instructions, mailing lists, Google, IRC, etc. Build the code, and then run the code; that’s your goal.
As you go through this build process, blog your process in a manner similar to how you walked through your build of Freeciv in this chapter.
If the build was easy, proceed to the next exercise: create two separate builds in two separate directories — the latest stable release, and the release from HEAD. FOSS developers frequently do this to compare behaviors between versions.
John Calcote wrote an excellent guide to the GNU Autotools. If you find yourself working with a project that uses these tools, Calcote’s work is a great online reference for beginners.
The much longer, but more definitive, work on GNU Autotools is GNU Autoconf, Automake, and Libtool (also known as “The Goat Book”) by Gary V. Vaughan, Ben Elliston, Tom Tromey and Ian Lance Taylor. Interestingly, it was a collaborative work written online by people who had never met in person.
The Mozilla project uses GNU Autotools for all of their projects, and their build documentation is available online. It’s a great exercise for those who want experience with building complex software designed to run on multiple platforms.
For those who want to start their own project and want it to be Autotools-friendly in the first place, here’s a great tutorial.
GNU Autotools is very common, and this book had to start somewhere with an exploration of build automation — but there are lots of other excellent build tools. In the world of Java software, Ant and Maven are extremely popular choices. Here’s a fairly comprehensive list of popular build automation tools.
Building software can be frustrating, and a lot of FOSS projects do a lot of things wrong. To be clear: most proprietary software projects probably do a lot of things wrong too, but the failures of FOSS projects are visible for all to see — and thus, to learn from. Tom ‘spot’ Callaway has compiled a list of things that projects do wrong, in a piece entitled How to tell if a FLOSS project is doomed to FAIL.