File: [local] / www / Attic / ctm.html (download) (as text)
Revision 1.26, Fri Feb 13 07:49:52 2004 UTC (20 years, 3 months ago) by david
Branch: MAIN
Changes since 1.25: +4 -4 lines
uppercase HTML -> lowercase HTML
ok xsa@
|
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"
"http://www.w3.org/TR/html4/loose.dtd">
<html>
<head>
<title>CTM</title>
<link rev="made" href="mailto:www@openbsd.org">
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<meta name="resource-type" content="document">
<meta name="description" content="the OpenBSD/CTM page">
<meta name="keywords" content="openbsd,ctm">
<meta name="distribution" content="global">
<meta name="copyright" content="This document copyright 1996-2004 by OpenBSD.">
</head>
<body bgcolor="#FFFFFF" text="#000000" link="#23238E">
<center>
<img alt="CTM for OpenBSD" src="images/bsdctm.gif" width="401" height="126">
</center>
<h3><font color="#0000e0">What is CTM?</font></h3>
<p>
CTM is a system which was designed by Poul-Henning Kamp for making
changes to a source tree available on a regular basis by email.
This is a good way to stay up to date with the current source tree or
CVS repository if you have a slow Internet connection, for instance via modem.
<p>
OpenBSD CTM distributes all modules of the OpenBSD CVS repository
plus checked out versions of several of these modules.
CTM uses very low bandwidth for distributing the changes to the tree,
and compresses them using <tt><b>gzip -9</b></tt>. In contrast to
<a href="cvsup.html">CVSup</a>, CTM is written in plain C language
and therefore available for every platform supported by OpenBSD.
CTM implements a so called push system, while all other methods to
get access to the OpenBSD CVS repository are pull systems.
<p>
For <b>any</b> problems, suggestions, reports and questions regarding CTM,
contact the CTM maintainer
<a href="mailto:hgw@d1906.inka.de">Hans Günter Weigand</a>.
<small>(Please do not ask other OpenBSD people for CTM support,
as they are busy doing other things. Thank you.)</small>
<h3><font color="#0000e0">How does it work?</font></h3>
<p>
The basic idea of CTM is that you subscribe to a special mailing list
for a particular CVS module or source tree. Each day you receive email messages
containing all the changes to that tree during that period.
One such set of differences is called a <b>delta</b>.
Actually, the CTM server makes new deltas every twelve hours.
<p>
You begin with a base set, against which later deltas are patched. To
begin using CTM you download the latest base set and all deltas
generated after that. Once you have processed those you can process
later deltas you receive from the mailing list. The base sets are split
into pieces of 1400kB each for easier downloading over bad lines.
<p>
Base sets are generated once every 50 relative deltas, so you never
have to grab more than 50 deltas to catch up.
<p>
There are currently two kinds of base set files. For quite a while we
were making <tt><b>tar.gz</b></tt> files manually, but we are
back to real CTM base deltas. The latter contain
<a href="http://www.openbsd.org/cgi-bin/man.cgi?query=md5&sektion=1">md5(1)</a>
checksums and are generated automatically, thus are a bit safer. You
can distinguish these two kinds of base sets by their filename. An
example for a <tt><b>tar.gz</b></tt> file is
<tt><b>OpenBSD-cvs.1450.tar.gz</b></tt>, whereas a CTM base delta
would be named <tt><b>OpenBSD-cvs.1500A.gz</b></tt>. See below for
detailed instructions.
<h3><font color="#0000e0">How much bandwidth does CTM use?</font></h3>
<p>
You obtain the base set and all relative deltas up to the recent one
via ftp from the sites listed below. Sizes are:
<ul>
<li>Approximately 215MB for OpenBSD-cvs.
<li>Approximately 40MB for OpenBSD-cvs-x11.
<li>Approximately 65MB for OpenBSD-cvs-xf4.
<li>Approximately 83MB for OpenBSD-src.
<li>Approximately 36MB for OpenBSD-src-x11.
<li>Approximately 4MB for OpenBSD-ports.
<br>Keep in mind that these are compressed sizes. The uncompressed
files are about five times as large.
<li>Relative deltas vary from 10kB to 500kB, and occasionally a couple of
megabytes.
<li>CTM always splits deltas into 100kB mail messages which are reassembled
automatically.
<li>If a delta is larger than 5MB, it is not sent via mail. Instead,
you will receive a notification to get the delta by ftp. Larger
CTM updates are rare, though, usually occurring only after a major
import of something like binutils, Perl, gcc, XFree86, etc., or applying a
tag to the CVS repository.
</ul>
<h3><font color="#0000e0">How do I use CTM?</font></h3>
<ul>
<li>Subscribe to the CTM mailing list for the source set you want.
<p>
There are several mailing lists related to CTM. There is a list for
updates to the CVS repository, a list for the source tree, a list
for the X11 part of both, and one for the ports tree, plus
a list for log messages from the CTM delta generation runs.
<p>
The following commands will subscribe you to the appropriate lists:
<p>
For the CVS repository (CVSROOT, ports, src and www modules,
but not X11 and XF4):
<pre>
echo subscribe OpenBSD-cvs | mail majordomo@OpenBSD.org
</pre>
For the X11 module of the CVS repository:
<pre>
echo subscribe OpenBSD-cvs-x11 | mail majordomo@OpenBSD.org
</pre>
For the XF4 module of the CVS repository:
<pre>
echo subscribe OpenBSD-cvs-xf4 | mail majordomo@OpenBSD.org
</pre>
For a checked out copy of the src module:
<pre>
echo subscribe OpenBSD-src | mail majordomo@OpenBSD.org
</pre>
For a checked out copy of the X11 module:
<pre>
echo subscribe OpenBSD-src-x11 | mail majordomo@OpenBSD.org
</pre>
For a checked out copy of the ports module:
<pre>
echo subscribe OpenBSD-ports-ctm | mail majordomo@OpenBSD.org
</pre>
For the CTM log list:
<pre>
echo subscribe ctm-log | mail majordomo@OpenBSD.org
</pre>
<p>
<li>Install the CTM program from a package or from the ports
collection. You'll find the port at
<tt><b>/usr/ports/devel/ctm</b></tt>.
</ul>
<b>Note:</b> The examples below just cover OpenBSD-cvs for clarity. The
other source sets are handled in the same fashion.
<p><ul>
<li>Download the base set from an FTP site. You'll find it in the
subdirectory <tt><b>base/OpenBSD-cvs</b></tt>. Get all files in
that directory. Once you have them all you can create the base set
file and unpack it with the following commands, depending on the type
of file you got:
<pre>
cat split/OpenBSD-cvs.1500A.gz.* > OpenBSD-cvs.1500A.gz
cd target
ctm -v -v -v .../OpenBSD-cvs.1500A.gz
</pre>
<p>
or
<pre>
cat split/OpenBSD-cvs.1450.tar.gz.* > OpenBSD-cvs.1450.tar.gz
cd target
tar -xzvf .../OpenBSD-cvs.1450.tar.gz
</pre>
<p>
where <tt><b>split</b></tt> is the directory in which you have
placed the files you downloaded and <tt><b>target</b></tt> is the
place you have chosen for unpacking the source set.
<p> It is not necessary to keep these base set files around, if
you're sure you've got a complete tree. But you can keep the base set
file(s) for backup purposes. However, be sure to note the number of
the base set before continuing.
<p>
<li>You now need to get the deltas which have been generated since the
base set. Go to one of the FTP sites listed below into the subdirectory
<tt><b>OpenBSD-cvs</b></tt> and download every file whose number is
<em>larger</em> than the number of the base set you used. Then apply
these deltas:
<pre>
cd target
ctm -v -v -v deltas/OpenBSD-cvs.*
</pre>
<p>
where <tt><b>target</b></tt> is the directory where your source set
tree is, and <tt><b>deltas</b></tt> is where you have stored the
deltas.
<p>
<li>Wait until you get your first delta from the mailing list.
<p>
<li>Unpack, assemble, and apply the CTM delta using <tt><b>ctm_rmail</b></tt>.
<p>
<pre>
ctm_rmail -p /tmp -d deltas -b target folder
</pre>
<p>
where <tt><b>folder</b></tt> is the mail folder containing the delta
mail. This will decode the CTM delta and save it in the
<tt><b>deltas</b></tt> directory. The delta will be a file of the
form <tt><b>OpenBSD-cvs.<var>XXXX</var>.gz</b></tt>, where <var>XXXX</var>
is the number of the delta. Then the delta will be applied to the
directory tree located at <tt><b>target</b></tt>, the place where you
unpacked the base set.
<p>
Repeat this last step every time you receive a delta from the
mailing list. If you like, you can automate this step using procmail,
maildrop, or any similar program from the
<a href="ports.html">ports collection</a>.
</ul>
<p>
Here is the list of FTP servers carrying CTM deltas:
<ul>
<li>The primary FTP server can be reached at:
<p>
<a href="ftp://openbsd.rug.ac.be:/pub/OpenBSD-ctm/">
ftp://openbsd.rug.ac.be:/pub/OpenBSD-ctm/
</a>.
<p>
This server is automatically updated by the CTM delta generation process.
Base sets are
available as big single files in the <tt><b>base</b></tt> directory.
The split files are in <tt><b>base-split</b></tt>.
</ul>
<ul>
<li><a href="ftp://ctm.se.OpenBSD.org:/pub/OpenBSD-ctm/">
ftp://ctm.se.OpenBSD.org:/pub/OpenBSD-ctm/
</a>
<p>
This server is updated by CTM deltas received from the mailing lists. It
additionally has the split base deltas in the <tt><b>base</b></tt>
subdirectory, mirrored from
<a href="ftp://openbsd.rug.ac.be:/pub/OpenBSD-ctm/">
ftp://openbsd.rug.ac.be:/pub/OpenBSD-ctm/
</a>.
</ul>
<ul>
<li><a href="ftp://ctm.OpenBSD.org:/pub/OpenBSD-ctm/">
ftp://ctm.OpenBSD.org:/pub/OpenBSD-ctm/
</a>
<p>
is identical to
<p>
<li><a href="ftp://ctm.ca.OpenBSD.org:/pub/OpenBSD-ctm/">
ftp://ctm.ca.OpenBSD.org:/pub/OpenBSD-ctm/
</a>
<p>
and is a mirror of
<a href="ftp://ctm.se.OpenBSD.org:/pub/OpenBSD-ctm/">
ftp://ctm.se.OpenBSD.org
</a>
.
</ul>
<p>
<b>NOTE:</b> <a href="mailto:hgw@d1906.inka.de">I'm</a> very interested
in finding other sites around the world that would like to
mirror these deltas.
<p>
<h3><font color="#0000e0">Should I choose the source tree or the CVS repository?</font></h3>
<p>
This depends on your diskspace, among other factors. Since CTM
doesn't deal very well with files which have been modified locally,
the better choice is surely the <a href="anoncvs.html">CVS</a>
repository. You can check out your working copy of the source tree
from your CVS repository then, can track a stable branch,
and keep your local modifications in
your source tree. CVS is smart enough to merge in your local changes.
You can also use the full range of CVS commands except commands like
<tt><b>cvs commit</b></tt> and <tt><b>cvs tag</b></tt>.
<p>
The drawback, however, is the amount of diskspace it takes. A
checked-out source tree takes about 370MB or so, but if you have your
own CVS repository you need more than 1Gb for it, <b>plus</b> another
370MB for the checked out tree.
This doesn't count the 100MB or more (depending on the architecture)
you will need for generated files during a build, i.e. compiling the sources.
<p>
If you just get the source tree, you will need to deal with local
modifications. One way to do this is to use the <b>union</b> filesystem,
(see <a href="http://www.openbsd.org/cgi-bin/man.cgi?query=mount_union&sektion=8">mount_union(8)</a>)
although there have been reports that the <b>union</b> filesystem can be
unstable if both the upper and lower layers reside on the same physical
filesystem. If you place your upper and lower layers on separate
filesystems you should have no problems. A sample of this is as follows:
<p>
Suppose your CTM-updated tree is in <tt><b>/usr/src-ctm</b></tt>, and
your real source tree, the one you make your modifications to and do
your builds from, is in <tt><b>/usr/src</b></tt>.
<tt><b>/usr/src</b></tt> should be initially an empty directory. The
following command will set up the union mount:
<pre>
mount -t union -o -b /usr/src-ctm /usr/src
</pre>
<p>
Modifications made to files in <tt><b>/usr/src</b></tt> will become a
file contained within <tt><b>/usr/src</b></tt>, hiding the one in
<tt><b>src-ctm</b></tt>. If changes are made via CTM to the
underlying <tt><b>src-ctm</b></tt> tree, those changes will <b>not</b>
be seen if there is a file in the upper layer hiding it.
<p>
You should periodically unmount the
union and search for files which are local to the union filesystem.
<pre>
umount /usr/src
find /usr/src -type f
</pre>
<p>
The commands <tt><b>ls -W</b></tt> and <tt><b>rm -W</b></tt> will be
useful too, as objects named "whiteouts" in <tt><b>/usr/src</b></tt>
will also hide files in <tt><b>src-ctm</b></tt>.
<p>
If you get the CVS repository, you can use the <tt><b>cvs checkout</b></tt>
command to check out a source tree from it. Each time you update
with CTM you can use the <tt><b>cvs update</b></tt> command to
update your source repository.
<p>
To get an initial src repository:
<pre>
cd /usr
cvs -qd YOUR_CVS_REPOSITORY checkout -P src
</pre>
<p>
and after each CTM update:
<pre>
cd /usr/src
cvs -q update -Pd
</pre>
<h3><font color="#0000e0">How stable is CTM for OpenBSD?</font></h3>
CTM has been in use for over five years for OpenBSD, and for FreeBSD
for an even longer period of time. CTM is very reliable and stable.
<h3><font color="#0000e0">What does the future hold for CTM?</font></h3>
There are numerous (small) changes and optimizations possible.
Among them:
<ul>
<li> CTM never sends a delta bigger than 5MB via mail. This could be
changed in two ways: The maximum delta size could be increased,
or a queue could be installed, from which only a certain number
of messages are mailed out daily. E. g., if a delta is 6MB in
size, it is split into 96 mail messages. These are put into a
queue. On the first day, the first 50 messages are sent, on the
second day the remaining 46, plus the first 4 messages of the
following delta.
<li> The CTM software could be rewritten, or at least fixed in a
better manner. Its C implementation is not an example of good
programming. That would also be a nice opportunity for adding
new features, like signing CTM deltas with PGP.
</ul>
Mail <a href="mailto:hgw@d1906.inka.de">me</a> your opinions and wishes!
<h3><font color="#0000e0">More information</font></h3>
If you want to learn more about CTM, a good place to start with is the
FreeBSD Handbook:
<ul>
<li> <a href="http://www.FreeBSD.org/handbook/ctm.html">
http://www.FreeBSD.org/handbook/ctm.html
</a>
</ul>
There are also man pages for all of the CTM utilities.
<h3><font color="#0000e0">Summary:</font></h3>
<p>
<ul>
<li> Mailing lists:
<ul>
<li><tt>echo subscribe OpenBSD-cvs | mail majordomo@OpenBSD.org</tt>
<li><tt>echo subscribe OpenBSD-cvs-x11 | mail majordomo@OpenBSD.org</tt>
<li><tt>echo subscribe OpenBSD-cvs-xf4 | mail majordomo@OpenBSD.org</tt>
<li><tt>echo subscribe OpenBSD-src | mail majordomo@OpenBSD.org</tt>
<li><tt>echo subscribe OpenBSD-src-x11 | mail majordomo@OpenBSD.org</tt>
<li><tt>echo subscribe OpenBSD-ports-ctm | mail majordomo@OpenBSD.org</tt>
<li><tt>echo subscribe ctm-log | mail majordomo@OpenBSD.org</tt>
</ul>
<p>
<li> FTP (for base sets and relative deltas):
<ul>
<li> <a href="ftp://openbsd.rug.ac.be:/pub/OpenBSD-ctm/">
ftp://openbsd.rug.ac.be:/pub/OpenBSD-ctm/</a>
<li> <a href="ftp://ctm.OpenBSD.org:/pub/OpenBSD-ctm/">
ftp://ctm.OpenBSD.org:/pub/OpenBSD-ctm/</a>
<li> <a href="ftp://ctm.ca.OpenBSD.org:/pub/OpenBSD-ctm/">
ftp://ctm.ca.OpenBSD.org:/pub/OpenBSD-ctm/</a>
<li> <a href="ftp://ctm.se.OpenBSD.org:/pub/OpenBSD-ctm/">
ftp://ctm.se.OpenBSD.org:/pub/OpenBSD-ctm/</a>
</ul>
</ul>
<p>
Important notes and announcements about CTM will be posted to the
announce@OpenBSD.org mailing list.
<p>
<h3><font color="#0000e0">Acknowledgements</font></h3>
<ul>
<li> Poul-Henning Kamp (phk@FreeBSD.org) for making CTM possible and helping
getting the delta generation running
<li> Theo de Raadt (deraadt@theos.com) for making OpenBSD possible and
initially providing the resources needed for creating the CTM
deltas on cvs.OpenBSD.org (now they are generated elsewhere), and
for the nameserver magic
<li> Todd C. Miller (millert@OpenBSD.org) for setting up and
maintaining the CTM mailinglists
<li> Bob Beck (beck@OpenBSD.org), Artur Grabowski (art@OpenBSD.org),
Magnus Holmberg (mho@OpenBSD.org) and
Wolfram Schneider (wosch@FreeBSD.org) for setting up the ftp space
for the base sets and deltas on ctm.*OpenBSD.org
<li> Thomas Graichen (graichen@OpenBSD.org) for starting CTM for
OpenBSD and maintaining it all those years
<li> Wim Vandeputte (wvdputte@OpenBSD.org) for hosting CTM now
<li> ... and all the others who contributed indirectly
</ul>
<p>
OpenBSD/CTM logo designed for the OpenBSD Project by
<a href="mailto:flipk@openbsd.org">Phillip F Knaack</a>.
<p>
<hr>
<a href="index.html"><img height="24" width="24" src="back.gif" border="0" alt="OpenBSD" /></a>
<a href="mailto:www@openbsd.org">www@openbsd.org</a>
<br>
<small>$OpenBSD: ctm.html,v 1.26 2004/02/13 07:49:52 david Exp $</small>
</body>
</html>
![[BACK]](/icons/back.gif){kind=icon}
Return to ctm.html CVS log ![[TXT]](/icons/text.gif){kind=icon}
![[DIR]](/icons/dir.gif){kind=icon}
Up to [local] / www