Getting the Code

Introduction

TOSThis is a conversation that you never want to hear.

“Oh no! The frobnitz is broken!”

“But it was working last week.”

“I don’t know what happened – was it the changes I made to the gorblewhonker?”

“Wait, I was putting that function in the gorblewhonker. I told you after class on Thursday, remember?”

“Shoot. Do you have last week’s version with the working frobnitz? What’s the difference?”

“Maybe. I don’t know. Does this mean we have to redo all the improvements we’ve made to the blooglebox since then?”

“Argh.”

There are tools that allow you to avoid these kinds of conversations.

Have you ever created a folder that looked something like this?

mycode-1.py
mycode-2.py
mycode-2-with-rachel's-changes.py
mycode-working.py
mycode-LATEST.py

If so, you have used version control. According to Wikipedia version control “is the management of changes to documents, programs, and other information stored as computer files.”

A system that manages version control for software development is called a source code management system, or an SCM for short. In this chapter, you will learn the basic use of source control management.

A Brief Introduction to FOSS Source Control Management Tools

The FOSS world has developed many excellent SCMs to choose from. Each have their strengths and weaknesses, and choosing which SCM to use for a new project is always a popular subject for debate.

You may want to start your own project someday, and you will have to choose an SCM. Right now, though, you will be working with existing projects, which means the SCM has been chosen for you. The following five SCMs are very popular, and you’re likely to see them often:

Eventually you may use all of these SCMs, or a different SCM entirely. Each SCM has some unique characteristics — but most basic version control concepts are common to all SCMs.

Since these concepts are new, the focus will be on learning one SCM: Subversion.

Exercise – Install Subversion on Your System

Install Subversion on your system. Subversion clients exists for all platforms; search the Internet for instructions. Ask your classmates, or ask on IRC, if you need help.

Getting Help With Subversion

(Adapted from Version Control with Subversion under Creative Commons Attribution License v2.0.)

Before reading on, here is the most important command you ever need when using Subversion: svn help. The Subversion command-line client is self-documenting — at any time, a quick svn help SUBCOMMAND describes the syntax, options, and behavior of the subcommand.

svn help update
update (up): Bring changes from the repository into the working copy.
usage: update [PATH...]

If no revision is given, bring working copy up-to-date with HEAD rev.
Else synchronize working copy to revision given by -r.

For each updated item a line will start with a character reporting the
action taken.  These characters have the following meaning:
...

Getting Started: checkout

(Adapted from Version Control with Subversion under Creative Commons Attribution License v2.0.)

Most of the time, you start using a Subversion repository by doing a checkout of your project. Checking out a repository creates a working copy of it on your local machine. This copy contains the HEAD (latest revision) of the Subversion repository that you specify on the command line:

$ svn checkout http://svn.collab.net/repos/svn/trunk
A    trunk/Makefile.in
A    trunk/ac-helpers
A    trunk/ac-helpers/install.sh
A    trunk/ac-helpers/install-sh
A    trunk/build.conf

Checked out revision 8810.

Although the above example checks out the trunk directory, you can just as easily check out any deep subdirectory of a repository by specifying the subdirectory in the checkout URL:

$ svn checkout \
http://svn.collab.net/repos/svn/trunk/subversion/tests/cmdline/
A    cmdline/revert_tests.py
A    cmdline/diff_tests.py
A    cmdline/autoprop_tests.py
A    cmdline/xmltests
A    cmdline/xmltests/svn-test.sh

Checked out revision 8810.

Since Subversion uses a copy-modify-merge model, you can start right in making changes to the files and directories in your working copy. Your working copy is just like any other collection of files and directories on your system. You can edit and change them, move them around, you can even delete the entire working copy and forget about it.

While you can certainly check out a working copy with the URL of the repository as the only argument, you can also specify a directory after your repository URL. This places your working copy in the new directory that you name. For example:

$ svn checkout http://svn.collab.net/repos/svn/trunk subv
A    subv/Makefile.in
A    subv/ac-helpers
A    subv/ac-helpers/install.sh
A    subv/ac-helpers/install-sh
A    subv/build.conf

Checked out revision 8810.

That places your working copy in a directory named subv instead of a directory named trunk as we did previously. The directory subv is created if it doesn’t already exist.

Exercise – Initial Checkout of the Sample Codebase – FIX

Create a local checkout of the sample codebase in the TOS repository. 

The Basic Subversion Work Cycle

(Adapted from Version Control with Subversion under Creative Commons Attribution License v2.0.)

Subversion has numerous features, options, and bells and whistles, but on a day-to-day basis, odds are that you only use a few of them. In this section we run through the most common things you might find yourself doing with Subversion in the course of a day’s work.

The typical work cycle looks like this:

Update your working copy

  • svn update

Make changes

  • svn add
  • svn delete
  • svn copy
  • svn move

Examine your changes

  • svn status
  • svn diff

Possibly undo some changes

  • svn revert

Resolve Conflicts (Merge Others’ Changes)

  • svn update
  • svn resolved

Commit your changes

  • svn commit

Update Your Working Copy

When working on a project with a team, you want to update your working copy to receive any changes made since your last update by other developers on the project. Use svn update to bring your working copy into sync with the latest revision in the repository.

$ svn update
U  foo.c
U  bar.c
Updated to revision 2.

In this case, someone else checked in modifications to both foo.c and bar.c since the last time you updated, and Subversion has updated your working copy to include those changes.

When the server sends changes to your working copy via svn update, a letter code is displayed next to each item to let you know what actions Subversion performed to bring your working copy up-to-date. We cover the meaning of these letters shortly.

Exercise – Get Updates From the Sample Repository

Update your working copy of the TOS repo. Has anything changed?

Make Changes to Your Working Copy

Now you can get to work and make changes in your working copy. It’s usually most convenient to decide on a discrete change (or set of changes) to make, such as writing a new feature, fixing a bug, etc. The Subversion commands that you use here are svn add, svn delete, svn copy, svn move, and svn mkdir. However, if you are merely editing files that are already in Subversion, you may not need to use any of these commands until you commit.

There are two kinds of changes you can make to your working copy: file changes and tree changes. You don’t need to tell Subversion that you intend to change a file; just make your changes using your text editor, word processor, graphics program, or whatever tool you would normally use. Subversion automatically detects which files have been changed, and in addition handles binary files just as easily as it handles text files — and just as efficiently too. For tree changes, you can ask Subversion to mark files and directories for scheduled removal, addition, copying, or moving. These changes may take place immediately in your working copy, but no additions or removals happen in the repository until you commit them.

Here is an overview of the five Subversion subcommands that you’ll use most often to make tree changes.

svn add foo

Schedule file, directory, or symbolic link foo to be added to the repository. When you next commit, foo becomes a child of its parent directory. Note that if foo is a directory, everything underneath foo is scheduled for addition. If you only want to add foo itself, pass the --non-recursive (-N) option.

svn delete foo

Schedule file, directory, or symbolic link foo to be deleted from the repository. If foo is a file or link, it is immediately deleted from your working copy. If foo is a directory, it is not deleted, but Subversion schedules it for deletion. When you commit your changes, foo is entirely removed from your working copy and the repository.

svn copy foo bar

Create a new item bar as a duplicate of foo and automatically schedule bar for addition. When bar is added to the repository on the next commit, its copy history is recorded (as having originally come from foo). The svn copy command does not create intermediate directories.

svn move foo bar

This command is exactly the same as running svn copy foo bar; svn delete foo. That is, bar is scheduled for addition as a copy of foo, and foo is scheduled for removal. The svn move command does not create intermediate directories.

svn mkdir blort

This command is exactly the same as running mkdir blort; svn add blort. That is, a new directory named blort is created and scheduled for addition.

Exercise – Create a Biography File and Add It to the Local Repository

Using other biography files as examples, create a biography file of yourself in the bio/ directory and add it to your local repository. Also add a link to that file in the index.html file in the root directory.

Examine Your Changes

Subversion has been optimized to help you with this task, and is able to do many things without communicating with the repository. In particular, your working copy contains a hidden cached pristine copy of each version controlled file within the .svn area. Because of this, Subversion can quickly show you how your working files have changed, or even allow you to undo your changes without contacting the repository.

svn status

To get an overview of your changes, use the svn status command. You may use svn status more than any other Subversion command.

If you run svn status at the top of your working copy with no arguments, it detects all file and tree changes you’ve made. Below are a few examples of the most common status codes that svn status can return. (Note that the text following # is not actually printed by svn status.)

A       stuff/loot/bloo.h   # file is scheduled for addition
C       stuff/loot/lump.c   # file has textual conflicts from an update
D       stuff/fish.c        # file is scheduled for deletion
M       bar.c               # the content in bar.c has local modifications

In this output format svn status prints six columns of characters, followed by several whitespace characters, followed by a file or directory name. The first column tells the status of a file or directory and/or its contents. The codes we listed are:

A item

The file, directory, or symbolic link item has been scheduled for addition into the repository.

C item

The file item is in a state of conflict. That is, changes received from the server during an update overlap with local changes that you have in your working copy. You must resolve this conflict before committing your changes to the repository.

D item

The file, directory, or symbolic link item has been scheduled for deletion from the repository.

M item

The contents of the file item have been modified.

If you pass a specific path to svn status, you get information about that item alone:

$ svn status stuff/fish.c
D      stuff/fish.c

The svn status command also has a --verbose (-v) option, which shows you the status of every item in your working copy, even if it has not been changed:

$ svn status -v
M     44    23    sally     README
44    30    sally     INSTALL
M     44    20    harry     bar.c
44    18    ira       stuff
44    35    harry     stuff/trout.c
D     44    19    ira       stuff/fish.c
44    21    sally     stuff/things
A     0     ?     ?        stuff/things/bloo.h
44    36    harry     stuff/things/gloo.c

This is the long form output of svn status. The letters in the first column mean the same as before, but the second column shows the working-revision of the item. The third and fourth columns show the revision in which the item last changed, and who changed it.

None of the prior invocations to svn status contact the repository — instead, they compare the metadata in the .svn directory with the working copy. Finally, there is the --show-updates (-u) option, which contacts the repository and adds information about things that are out-of-date:

$ svn status -u -v
M       *        44        23    sally     README
M      44        20    harry     bar.c
*      44        35    harry     stuff/trout.c
D      44        19    ira       stuff/fish.c
A       0         ?     ?        stuff/things/bloo.h
Status against revision:   46

Notice the two asterisks: if you were to run svn update at this point, you would receive changes to README and trout.c. This tells you some very useful information — you need to update and get the server changes on README before you commit, or the repository will reject your commit for being out-of-date. (More on this subject later.)

The svn status command can display much more information about the files and directories in your working copy than we’ve shown here — for an exhaustive description of svn status and its output, see svn status.

svn diff

Another way to examine your changes is with the svn diff command. You can find out exactly how you’ve modified things by running svn diff with no arguments, which prints out file changes in unified diff format:

$ svn diff
Index: bar.c
===================================================================
--- bar.c (revision 3)
+++ bar.c (working copy)
@@ -1,7 +1,12 @@
+#include <sys/types.h>
+#include <sys/stat.h>
+#include <unistd.h>
+
+#include <stdio.h>

int main(void) {
-  printf("Sixty-four slices of American Cheese...\n");
+  printf("Sixty-five slices of American Cheese...\n");
return 0;
}

Index: README
===================================================================
--- README (revision 3)
+++ README (working copy)
@@ -193,3 +193,4 @@
+Note to self:  pick up laundry.

Index: stuff/fish.c
===================================================================
--- stuff/fish.c (revision 1)
+++ stuff/fish.c (working copy)
-Welcome to the file known as 'fish'.
-Information on fish will be here soon.

Index: stuff/things/bloo.h
===================================================================
--- stuff/things/bloo.h (revision 8)
+++ stuff/things/bloo.h (working copy)
+Here is a new file to describe
+things about bloo.

The svn diff command produces this output by comparing your working files against the cached pristine copies within the .svn area. Files scheduled for addition are displayed as all added text, and files scheduled for deletion are displayed as all deleted text.

Output is displayed in unified diff format. That is, removed lines are prefaced with – and added lines are prefaced with +. The svn diff command also prints filename and offset information useful to the patch program, so you can generate patches by redirecting the diff output to a file:

svn diff > patchfile

You could, for example, email the patch file to another developer for review or testing prior to commit.

Subversion uses its internal diff engine, which produces unified diff format, by default. If you want diff output in a different format, specify an external diff program using --diff-cmd and pass any flags you’d like to it using the --extensions (-x) option. For example, to see local differences in file foo.c in context output format while ignoring case differences, you might run svn diff --diff-cmd /usr/bin/diff --extensions '-i' foo.c.

Undoing Working Changes

Suppose while viewing the output of svn diff you determine that all the changes you made to a particular file are mistakes. Maybe you shouldn’t have changed the file at all, or perhaps it would be easier to make different changes starting from scratch.

This is a perfect opportunity to use svn revert:

$ svn revert README
Reverted 'README'
 

Subversion reverts the file to its pre-modified state by overwriting it with the cached pristine copy from the .svn area. But also note that svn revert can undo any scheduled operations — for example, you might decide that you don’t want to add a new file after all:

$ svn status foo
? foo

$ svn add foo
A foo

$ svn revert foo
Reverted 'foo'

$ svn status foo
? foo

Note: svn revert ITEM has exactly the same effect as deleting ITEM from your working copy and then running svn update -r BASE ITEM. However, if you’re reverting a file, svn revert has one very noticeable difference — it doesn’t have to communicate with the repository to restore your file. Which is very useful if you’re working disconnected from the coffeeshop.

Or perhaps you mistakenly removed a file from version control:

$ svn status README
README

$ svn delete README
D README

$ svn revert README
Reverted 'README'

$ svn status README
README

Resolve Conflicts (Merging Changes of Others)

We’ve already seen how svn status -u can predict conflicts. Suppose you run svn update and some interesting things occur:

$ svn update
U  INSTALL
G  README
C  bar.c
Updated to revision 46.

The U and G codes are no cause for concern; those files cleanly absorbed changes from the repository. The files marked with U contained no local changes but were Updated with changes from the repository. The G stands for merGed, which means that the file had local changes to begin with, but the changes coming from the repository didn’t overlap with the local changes.

But the C stands for conflict. This means that the changes from the server overlapped with your own, and now you have to manually choose between them.

Whenever a conflict occurs, three things typically occur to assist you in noticing and resolving that conflict:

  • Subversion prints a C during the update, and remembers that the file is in a state of conflict.
  • If Subversion considers the file to be mergeable, it places conflict markers — special strings of text which delimit the sides of the conflict—into the file to visibly demonstrate the overlapping areas. (Subversion uses the svn:mime-type property to decide if a file is capable of contextual, line-based merging.)
  • For every conflicted file, Subversion places three extra unversioned files in your working copy:
    • filename.mine — this is your file as it existed in your working copy before you updated your working copy — that is, without conflict markers. This file has only your latest changes in it. (If Subversion considers the file to be unmergeable, then the .mine file isn’t created, since it would be identical to the working file.)
    • filename.rOLDREV — this is the file that was the BASE revision before you updated your working copy. That is, the file that you checked out before you made your latest edits. OLDREV is the revision number of the file in your .svn directory.
    • filename.rNEWREV — this is the file that your Subversion client just received from the server when you updated your working copy. This file corresponds to the HEAD revision of the repository. NEWREV is the revision number of the repositry HEAD.

For example, Sally makes changes to the file sandwich.txt in the repository. Harry has just changed the file in his working copy and checked it in. Sally updates her working copy before checking in and she gets a conflict:

$ svn update
C  sandwich.txt
Updated to revision 2.
$ ls -1
sandwich.txt
sandwich.txt.mine
sandwich.txt.r1
sandwich.txt.r2

At this point, Subversion does not allow Sally to commit the file sandwich.txt until the three temporary files are removed:

$ svn commit -m "Add a few more things"
svn: Commit failed (details follow):
svn: Aborting commit: '/home/sally/svn-work/sandwich.txt' remains in conflict

To resolve a conflict do one of three things:

  • Merge the conflicted text by hand (by examining and editing the conflict markers within the file).
  • Copy one of the temporary files on top of the working file.
  • Run svn revert FILENAME to throw away all of the local changes.

Once the conflict is resolved, let Subversion know by running svn resolved. This removes the three temporary files and Subversion no longer considers the file to be in a state of conflict.

$ svn resolved sandwich.txt
Resolved conflicted state of 'sandwich.txt'

Merging conflicts by hand can be quite intimidating the first time you attempt it, but with a little practice, it can become as easy as falling off a bike.

Here’s an example. Due to a miscommunication, you and Sally, your collaborator, both edit the file sandwich.txt at the same time. Sally commits her changes, and when you go to update your working copy, you get a conflict and you’re going to have to edit sandwich.txt to resolve the conflicts. First, let’s take a look at the file:

$ cat sandwich.txt
Top piece of bread
Mayonnaise
Lettuce
Tomato
Provolone
<<<<<<< .mine
Salami
Mortadella
Prosciutto
=======
Sauerkraut
Grilled Chicken
>>>>>>> .r2
Creole Mustard
Bottom piece of bread

The strings of less-than signs, equal signs, and greater-than signs are conflict markers, and are not part of the actual data in conflict. You generally want to ensure that those are removed from the file before your next commit. The text between the first two sets of markers is composed of the changes you made in the conflicting area:

<<<<<<< .mine
Salami
Mortadella
Prosciutto
=======

The text between the second and third sets of conflict markers is the text from Sally’s commit:

=======
Sauerkraut
Grilled Chicken
>>>>>>> .r2
 

Usually you won’t want to just delete the conflict markers and Sally’s changes — she’s going to be awfully surprised when the sandwich arrives and it’s not what she wanted. So this is where you pick up the phone or walk across the office and explain to Sally that you can’t get sauerkraut from an Italian deli. Once you’ve agreed on the changes to check in, edit your file and remove the conflict markers.

Top piece of bread
Mayonnaise
Lettuce
Tomato
Provolone
Salami
Mortadella
Prosciutto
Creole Mustard
Bottom piece of bread

Now run svn resolved, and you’re ready to commit your changes:

$ svn resolved sandwich.txt
$ svn commit -m "Go ahead and use my sandwich, discarding Sally's edits."

Note that svn resolved, unlike most of the other commands we deal with in this chapter, requires an argument. In any case, you want to be careful and only run svn resolved when you’re certain that you’ve fixed the conflict in your file — once the temporary files are removed, Subversion lets you commit the file even if it still contains conflict markers.

If you ever get confused while editing the conflicted file, you can always consult the three files that Subversion creates for you in your working copy — including your file as it was before you updated. You can even use a third-party interactive merging tool to examine those three files.

Copying a File Onto Your Working File

If you get a conflict and decide that you want to throw out your changes, you can merely copy one of the temporary files created by Subversion over the file in your working copy:

$ svn update
C  sandwich.txt
Updated to revision 2.
$ ls sandwich.*
sandwich.txt  sandwich.txt.mine  sandwich.txt.r2  sandwich.txt.r1
$ cp sandwich.txt.r2 sandwich.txt
$ svn resolved sandwich.txt

If you get a conflict, and upon examination decide that you want to throw out your changes and start your edits again, just revert your changes:

$ svn revert sandwich.txt
Reverted 'sandwich.txt'
$ ls sandwich.*
sandwich.txt

Note that when you revert a conflicted file, you don’t have to run svn resolved.

Commit Your Changes

Finally! Your edits are finished, you’ve merged all changes from the server, and you’re ready to commit your changes to the repository.

The svn commit command sends all of your changes to the repository. When you commit a change, you need to supply a log message, describing your change and why you made it, if relevant. Your log message is attached to the new revision you create. If your log message is brief, you may wish to supply it on the command line using the --message (or -m) option:

$ svn commit -m "Corrected number of cheese slices."
Sending sandwich.txt
Transmitting file data .
Committed revision 3.

However, if you’ve been composing your log message as you work, you may want to tell Subversion to get the message from a file by passing the filename with the --file (-F) option:

$ svn commit -F logmsg
Sending sandwich.txt
Transmitting file data .
Committed revision 4.

If you fail to specify either the --message or --file option, then Subversion automatically starts your system default editor for composing a log message.

If you’re in your editor writing a commit message and decide that you want to cancel your commit, you can just quit your editor without saving changes. If you’ve already saved your commit message, simply delete the text, save again, then abort.

$ svn commit
Waiting for Emacs...Done

Log message unchanged or not specified
a)bort, c)ontinue, e)dit
a
$

The repository doesn’t know or care if your changes make any sense as a whole; it only checks to make sure that nobody else has changed any of the same files that you did when you weren’t looking. If somebody has done that, the entire commit fails with a message informing you that one or more of your files is out-of-date:

$ svn commit -m "Add another rule"
Sending rules.txt
svn: Commit failed (details follow):
svn: Your file or directory 'sandwich.txt' is probably out-of-date

(The exact wording of this error message depends on the network protocol and server you’re using, but the idea is the same in all cases.)

At this point, you need to run svn update, deal with any merges or conflicts that result, and attempt your commit again.

That covers the basic work cycle for using Subversion. There are many other features in Subversion that you can use to manage your repository and working copy, but most of your day-to-day use of Subversion involves only the commands discussed in this chapter.

Exercise – Commit Code to the Repository

Commit your local changes to the repository. The account information for writing to the repository can be found in Appendix A.

Blog about the process. Did your commit work the first time? If not, why not? Were there conflicts? What did you do to resolve them?

Supplemental Reading

Much of this chapter was based on the excellent book, Version Control with Subversion by Ben Collins-Sussman, Brian W. Fitzpatrick and C. Michael Pilato.

There are important differences between a client-server SCM, such as Subversion or CVS, and a distributed SCM, such as Mercurial or Git. In the client-server model, developers use a shared single repository; in the distributed model, each developer works directly with their own local repository, and changes are shared between repositories as a separate step.

Subversion is a great SCM to learn for those who want to make the jump to distributed SCMs. Here are two excellent guides:

Supplemental Exercises

Freeciv is one of the most popular FOSS games, and it’s all hosted in a Subversion repository. Go to the Freeciv developer site and download the code from both HEAD and the latest release tag. Download them into separate directories. Do your best to figure out what the current differences are, and summarize those differences in a blog post. (Note: this might take a while.)