scord

About

scord reduces the disk space used by a subversion working copy with large and/or many files; ignoring other subversion overhead, scord reduces the size of a working copy to just the size of the working files, instead of 2x their size. Subversion users may find the disk savings useful for large source code or media repositories. scord stands for "Subversion Check Out, Reduced Disk" and is under the MIT license. scord currently only reduces disk space usage for Subversion ≤1.6.

More about

A subversion working copy (without scord) uses 2x the disk space of the working files because it contains two copies of each file; the second is a pristine version used by subversion to compute diffs and report status without contacting the repository [SVN Book]. While this saves bandwidth (sometimes significantly, eg large working copies!) and permits offline operation, it is unfortunate when most working versions are unmodified (rendering the pristine versions unnecessary). The subversion developers and other users have long understood this drawback, but difficult issues have and continue to postpone solutions (see Why not ... for the why).

scord solves this issue by interposing between filesystem access and the filesystem itself, where it detects when the working version becomes modified or is reverted and is able to respond by recreating or eliminating the pristine version. scord is thus able to store explicit pristine versions for only locally modified files.

To use scord you must be able to trust it with your local modifications; scord's internal file updates use the same techniques to safeguard against crashes as subversion uses for working copy updates.

Use

scord serves a filesystem in which you keep your subversion working copy(ies). A scord filesystem is no different from any other filesystem; scord merely detects disk space reduction opportunities in subversion working copies. To start using scord:

$ mkdir .scord_photos/ photos/
# scord will expose the filesystem in photos/ and store scord data in .scord_photos/:
$ scord .scord_photos/ photos/
$ svn checkout <URL> photos/
$ svn info photos/
Path: ...

# see the disk savings:
$ du -sh .scord_photos/ photos/
20G    .scord_photos/
40G    photos/

# stop scord, on Linux, via fusermount or kill -SIGTERM:
$ fusermount -u photos/
# (or on Mac OS X, by unmounting the file system or using kill -SIGTERM)

You may also add startscord to your startup programs and create a ~/.startscordrc to automatically start scord for each scord checkout. StartScord.app is a Mac OS X frontend to startscord. See the startscord man page for details.

Automatic scord Startup

scord includes three methods for automatic startup:

startscord (Linux and Mac OS X)

On Linux machines and remotely-accessed Mac OS X machines you may add startscord to your shell startup (e.g., run startscord from your ~/.bash_profile) and create a ~/.startscordrc to automatically start scord for each scord checkout when you log in. See the startscord man page for details.

StartScord.app (Mac OS X)

You may add StartScord.app to your user's startup programs and create a ~/.startscordrc to automatically start scord for each scord checkout when you log in. See the startscord man page for details. StartScord.app is a Mac OS X session frontend to startscord.

Automount (Mac OS X)

You may also start a scord mount on demand by editing /etc/fstab: Add the following line to /etc/fstab, creating the file if it does not already exist, “dummy:/Users/bob/.scord_photos /Users/bob/photos scord allow_other 0 0”. And then ask automount to reread /etc/fstab:

$ sudo automount -c

You should then see the directory in mount's output:

$ mount
...
map -static on /Users/bob/photos (autofs, automounted)

automount support requires MacFUSE >= 2.0.

Download

Current release: 1.1.2 (RSS Feed of Release Announcements):

• Source (Linux and Mac OS X): scord-1.1.2.tar.gz
• Mac OS X Installer (Intel): Scord-1.1.2.dmg

Development (in subversion): https://svn.code.sf.net/p/scord/code/trunk/scord (Browse)

Software requirements

• Linux (2.6 tested) or Mac OS X
• FUSE:
  • Linux: FUSE ≥2.5 (≥2.6 is better)
  • Mac OS X: MacFUSE ≥1.7
• Subversion 0.7–1.6 (but not ≥1.7 at this time)

Documentation

• Man pages: scord(1), scordstatus(1), and startscord(1)
• Mailing list: scord-list (user and devel discussions)
• SourceForge project page

Drawbacks

While scord can save substantial disk space for a working copy, it also (currently) includes the following drawbacks:

• scord can negatively impact working copy speed:
  • it must do maintenance work behind the scenes — comparing, copying, and deleting files as they are modified
  • it is implemented as a user program (instead of a kernel module), which increases the time a filesystem operation takes and increases filesystem cache usage
  However, scord can also increase working copy speed; for example, with scord a svn update or checkout need write only half as much to disk.
• scord does not preserve all file metadata:
  • scord assumes files are owned by the scord owner and group
  • scord assumes the permissions of pristine file versions are the working sibling's minus write access
  • for shared files, the working and pristine file share their change, modification, and access times
  • scord does not preserve extended attributes
  • only processes with the same user id as the scord daemon may access the daemon's filesystem
• scord places restrictions on filesystem usage:
  • removing an open file does not immediately remove the file; it is temporarily renamed and then removed upon close (FUSE artifact)
  • scord does not support hard links
• scord complicates disk space usage:
  Because scord's disk space optimizations depend on whether a file is modified, the disk space needed by filesystem operations is less obvious with scord. For example, appending a byte to a 1GB file may require 1GB of free disk space.

The first three of these four drawbacks are primarily implementation limitations; scord development assistance is welcome!

Why not modify subversion/why create scord?

There were two reasons to create scord instead of modify subversion:

• A rewrite of the working copy API appears to be a required step to enhance subversion to reduce working copy disk usage, and even a solid design for this large change has remained elusive.

  Subversion developers documented the pristine version storage tradeoff at least as far back as 2001. The tradeoff persists because all discussed (subversion!) solutions appear to require a working copy API rewrite (breaking compatibility) and because, while several solutions have been discussed, no single solution has shown through as the obvious end solution. Many of these solutions include benefits scord does not even try to obtain — so there is most definitely reason to continue pursual! — but scord achieves its goal now and with complete compatibility (...on the comparatively few platforms it supports).

  Someday, it's theoretically possible to improve subversion such that it does not rely on the existence of cached, pristine files in .svn/text-base/. It's a big change, though. ...

  Source: Ben Collins-Sussman: Subversion issue 525, 2001-10-11

  
  

  Related subversion issues and discussions (and if you know of other useful information, I would be most interested):

  • Subversion issue entries:
    • Issue 525: allow working copies without text-base/ (Created 2001-10)
    • Issue 908: Store text-base compressed (Created 2002-09)
  • Subversion mailing list discussions:
    • Eliminating the text-base penalty (2002-09) (presentation and discussion of two approaches and their usage)
    • text-base penalty: A proposed solution (2002-12) (showstoppers for moving all SCO source from SCCS to subversion, general discussion)
    • Optional | Compressed text-base storage (2005-06) (Google SoC project proposal discussion regarding pristine compression)
    • Optional text base design discussion? (2006-10) (disk space usage is still a problem)
    • Another working copy library (2007-01, 2007-02, 2007-03) (discussions on whether to address in subversion 1.x or 2.x, how to work in the new api/break compatibility)
    • A text-base penalty solution (without a working copy rewrite) (2007-05) (initial scord message, problem statement)

• scord detects file edits itself (no explicit, prior user notification).

  Whereas several subversion design ideas (and many other SCMs, eg Perforce and BitKeeper) require the user to inform the SCM client before editing a file, scord detects file changes itself as they are made. This simplifies normal user interaction and also easily supports mass file regeneration (eg by the build system). Unlike SCMs with explicit edit commands, scord does not give subversion the ability to find all modified files without crawling the entire working copy (but, perhaps scord could!).

scord compared with existing solutions for subversion

An existing solution to obtain smaller working copies is to export the working copy and manually track local and repository changes. scord achieves the same disk savings (ignoring other subversion overhead) while maintaining the features of a subversion working copy.

The subversion-related SVK lists its disk space savings as one of the two reasons to use it instead of subversion [WhySVK]. scord-housed subversion working copies use even less disk space than SVK working copies (which store the entire repository's file and history set) and scord does not come with a change in development model. At the same time, SVK provides an entire, local repository mirror.

fsvs is an alternative subversion client targeted at backing up directory trees. As a client replacement, fsvs uses its own working copy metadata format and stores only the working version of each file. As such, fsvs provides many benefits similar to and beyond scord's, at the cost of replacing the subversion client and breaking compatibility with subversion working copy tools.

People

Chris Frost