www/ctm.html - diff

Return to ctm.html CVS log

Up to [local] / www

Diff for /www/Attic/ctm.html between version 1.13 and 1.14

version 1.13, 2001/01/01 22:01:21

version 1.14, 2001/02/24 23:37:06

Line 9

</center>

<P>

Ctm is a system which was designed by Poul-Henning Kamp for making

CTM is a system which was designed by Poul-Henning Kamp for making

changes to a source tree available on a daily basis by email.

This is a good way to stay up to date with the current source- or

This is a good way to stay up to date with the current source tree or

CVS tree if you have a bad Internet connection, for instance via modem.

CVS repository if you have a slow Internet connection, for instance via modem.

<P>

Ctm uses very low bandwidth for distributing the changes to the tree,

CTM uses very low bandwidth for distributing the changes to the tree,

and compresses them using gzip -9. In contrast to <a

and compresses them using <tt><b>gzip -9</b></tt>. In contrast to

href="cvsup.html">CVSup</a>, ctm is written in plain C and therefore

<a href="cvsup.html">CVSup</a>, CTM is written in plain C language

available for every platform supported by OpenBSD.

and therefore available for every platform supported by OpenBSD.

<p>

For <b>any</b> problems, suggestions, reports and questions regarding CTM

contact the CTM maintainer

<a href="mailto:hgw@d1906.inka.de">Hans Günter Weigand</a>.

<small>(Please do not ask other OpenBSD people for CTM support,

as they are busy doing other things. Thank you.)</small>

<P>

The basic idea of ctm is that you subscribe to a special mailing list

The basic idea of CTM is that you subscribe to a special mailing list

for a particular source tree. Each day you receive an email message

containing all the changes to that tree during that period.

One set of differences is called a "delta".

One such set of differences is called a <b>delta</b>.

<P>

You begin with a base set, against which later deltas are patched. To

begin using ctm you download the latest base set and all deltas

begin using CTM you download the latest base set and all deltas

generated after that. Once you have processed those you can process

later deltas you receive from the mailing list. The base sets are split

into pieces of 1400kB each for easier downloading over bad lines, and

into pieces of 1400kB each for easier downloading over bad lines.

you can even put them onto floppies this way.

<P>

Base sets are generated once every 50 relative deltas, so you never

have to grab more than 50 deltas to catch up.

<P>

There are currently two kinds of base delta. For quite a while we

There are currently two kinds of base set files. For quite a while we

were making <tt><b>tar.gz</b></tt> files manually, but we are

switching back to real ctm base deltas. The latter contain md5

switching back to real CTM base deltas. The latter contain MD5

checksums and are generated automatically, thus are a bit safer. You

can distinguish these two kinds of base delta by their filename. An

can distinguish these two kinds of base set by their filename. An

example for a <tt><b>tar.gz</b></tt> file is

<tt><b>OpenBSD-cvs.1450.tar.gz</b></tt>, whereas a ctm base delta

<tt><b>OpenBSD-cvs.1450.tar.gz</b></tt>, whereas a CTM base delta

would be named <tt><b>OpenBSD-cvs.1500A.gz</b></tt>. See below for

detailed instructions.

<H3>How much bandwidth does ctm use?</H3>

<H3>How much bandwidth does CTM use?</H3>

<P>

You obtain the base set and all relative deltas up to the recent one

Line 66

Line 72

<li>Approximately 65MB for OpenBSD-src.

<li>Approximately 35MB for OpenBSD-src-x11.

<li>Approximately 2MB for OpenBSD-ports.

<li>Keep in mind that these are compressed sizes. The uncompressed

<br>Keep in mind that these are compressed sizes. The uncompressed

files are about five times as large.

<li>Relative deltas vary from 10kB to 100kB, and occasionally a couple of

<li>Relative deltas vary from 10kB to 500kB, and occasionally a couple of

megabytes.

<li>ctm always splits deltas into 100kB messages which are reassembled

<li>CTM always splits deltas into 100kB mail messages which are reassembled

automatically.

<li>If a delta is larger than 5MB, it is not sent via mail. Instead, you

<li>If a delta is larger than 5MB, it is not sent via mail. Instead,

will receive a notification to get the delta by ftp.

you will receive a notification to get the delta by ftp. Larger

CTM updates are rare, though, usually occurring only after a major

import of something like binutils, perl, gcc, etc, or applying a

CVS tag to the CVS repository.

</ul>

<P>

Larger ctm updates are rare, though, usually occurring only after a major

import of something like binutils, perl, gcc, etc.

<UL>

<LI>Subscribe to the ctm mailing list for the source set you want.

<LI>Subscribe to the CTM mailing list for the source set you want.

<p>

There are several mailing lists related to ctm. There is a list for

There are several mailing lists related to CTM. There is a list for

updates to the CVS tree, a list for the source tree, a list

updates to the CVS repository, a list for the source tree, a list

for the X11 part of both, and one for the ports tree, plus

a list for log messages from the ctm delta generation runs.

a list for log messages from the CTM delta generation runs.

<p>

The following commands will subscribe you to the appropriate lists:

<p>

For the CVS tree (without the X11 part):

For the CVS repository (without the X11 part):

<pre>

echo subscribe OpenBSD-cvs | mail majordomo@OpenBSD.org

</pre>

For the X11 part of the CVS tree:

For the X11 part of the CVS repository:

<pre>

echo subscribe OpenBSD-cvs-x11 | mail majordomo@OpenBSD.org

</pre>

Line 112

Line 117

<pre>

echo subscribe OpenBSD-ports-ctm | mail majordomo@OpenBSD.org

</pre>

For the ctm log list:

For the CTM log list:

<pre>

echo subscribe ctm-log | mail majordomo@OpenBSD.org

</pre>

<p>

<li>Install the ctm program from a package or from the ports

<li>Install the CTM program from a package or from the ports

collection. You'll find the port at

<tt><b>/usr/ports/devel/ctm</b></tt>.

Line 131

Line 136

<li>Download the base set from an FTP site. You'll find it in the

subdirectory <tt><b>base/OpenBSD-cvs</b></tt>. Get all files in

that directory. Once you have them all you can create the base set

file and unpack it with the following commands, depending on the kind

file and unpack it with the following commands, depending on the type

of base delta you got:

of file you got:

<pre>

cat split/OpenBSD-cvs.1500A.gz.* > OpenBSD-cvs.1500A.gz

Line 151

Line 156

<p>

where <tt><b>split</b></tt> is the directory in which you have

placed the files you downloaded and <tt><b>target</b></tt> is the

place you have chosen for the source set.

place you have chosen for unpacking the source set.

<p>

<p> It is not necessary to keep these base set files around, if

It is not necessary to keep these files around, if you're sure you've

you're sure you've got a complete tree. But you can keep the base set

got a complete tree. But you can keep the base set file(s) for backup

file(s) for backup purposes. However, be sure to note the number of

purposes. However, be sure to note the number of the base set before

the base set before continuing.

continuing.

<p>

<li>You now need to get the deltas which have been generated since the

Line 179

Line 183

<p>

<LI>Wait until you get your first delta from the mailing list.

<p>

<LI>Unpack, assemble, and apply the ctm delta using ctm_rmail.

<LI>Unpack, assemble, and apply the CTM delta using <tt><b>ctm_rmail</b></tt>.

<p>

<pre>

Line 188

Line 192

<p>

where <tt><b>folder</b></tt> is the mail folder containing the delta

mail. This will decode the ctm delta and save it in the

mail. This will decode the CTM delta and save it in the

<tt><b>deltas</b></tt> directory. The delta will be a file of the

form <tt><b>OpenBSD-cvs.<var>XXXX</var>.gz</b></tt>, where <var>XXXX</var>

is the number of the delta. Then the delta will be applied to the

directory tree located at <tt><b>target</b></tt>, the place where you

unpacked the base delta.

unpacked the base set.

<p>

Repeat this last step every time you receive a delta from the

Line 202

Line 206

</ul>

<p>

The addresses for the deltas are:

The addresses of the FTP servers carrying CTM deltas are:

<ul>

Line 222

Line 226

</a>

</ul>

<ul>

<li>An experimental server can be reached at

ftp://openbsd.rug.ac.be:/ftp/pub/OpenBSD-ctm/

</a>.

This server has a slightly different directory layout. Base sets are

available as big single files in the <tt><b>base</b></tt> directory.

The split files are in <tt><b>base-split</b></tt>.

</ul>

<p>

<b>NOTE:</b> I'm very interested in finding other sites around the world

<b>NOTE:</b> <a href="mailto:hgw@d1906.inka.de">I'm</a> very interested

that would like to mirror these deltas.

in finding other sites around the world that would like to

mirror these deltas.

<p>

<h3>Should I choose the source or CVS tree?</h3>

<h3>Should I choose the source tree or the CVS repository?</h3>

<P>

This depends on your diskspace, among other factors. Since ctm

This depends on your diskspace, among other factors. Since CTM

doesn't deal with files which have been modified locally, the "better"

doesn't deal very well with files which have been modified locally,

choice is probably the CVS tree. You then can check out your working

the "better" choice is probably the CVS repository. You can check out

source tree from your CVS tree, and keep your local modifications in

your working copy of the source tree from your CVS repository then,

your source tree, because cvs is smart enough to merge in your local

and keep your local modifications in your source tree. CVS is

changes.

smart enough to merge in your local changes. You can also use the

full range of CVS commands except <tt><b>cvs commit</b></tt>.

<p>

The drawback, however, is the amount of diskspace it takes. A checked-out

The drawback, however, is the amount of diskspace it takes. A

source tree takes about 350MB or so, but if you have your own cvs tree you

checked-out source tree takes about 350MB or so, but if you have your

need about 1Gb for it, <b>plus</b> another 350MB for the checked out tree.

own CVS repository you need more than 1Gb for it, <b>plus</b> another

350MB for the checked out tree.

<p>

This doesn't count the 100MB or more (depending on the architecture)

you will need to do a build, i.e. compile the sources.

Line 255

Line 270

you should have no problems. A sample of this is as follows:

<p>

Suppose your ctm-updated tree is in <tt><b>/usr/src-ctm</b></tt>, and

Suppose your CTM-updated tree is in <tt><b>/usr/src-ctm</b></tt>, and

your real source tree, the one you make your modifications to and do

your builds from, is in <tt><b>/usr/src</b></tt>.

<tt><b>/usr/src</b></tt> should be initially an empty directory. The

Line 266

Line 281

</pre>

<p>

Modifications made to files in /usr/src will become a file contained within

Modifications made to files in <tt><b>/usr/src</b></tt> will become a

/usr/src, hiding the one in src-ctm. If changes are made via ctm to the

file contained within <tt><b>/usr/src</b></tt>, hiding the one in

underlying src-ctm tree, those changes will <b>not</b> be seen if there is

<tt><b>src-ctm</b></tt>. If changes are made via CTM to the

a file in the upper layer hiding it.

underlying <tt><b>src-ctm</b></tt> tree, those changes will <b>not</b>

be seen if there is a file in the upper layer hiding it.

<p>

You should periodically unmount the

Line 282

Line 298

<p>

The commands <tt><b>ls -W</b></tt> and <tt><b>rm -W</b></tt> will be

useful too, as objects named "whiteouts" in /usr/src will also hide

useful too, as objects named "whiteouts" in <tt><b>/usr/src</b></tt> will also hide

files in src-ctm.

files in <tt><b>src-ctm</b></tt>.

<p>

If you get the CVS tree, you can use the <tt><b>cvs checkout</b></tt>

If you get the CVS repository, you can use the <tt><b>cvs checkout</b></tt>

command to check out a source tree from it, and each time you update

command to check out a source tree from it. Each time you update

with ctm you can use the <tt><b>cvs update</b></tt> command to

with CTM you can use the <tt><b>cvs update</b></tt> command to

update your source tree.

update your source repository.

<p>

To get an initial src tree:

To get an initial src repository:

<pre>

cd /usr

cvs -qd YOUR_CVS_TREE checkout -A src

cvs -qd YOUR_CVS_REPOSITORY checkout -A src

</pre>

<p>

and after each ctm update:

and after each CTM update:

<pre>

cd /usr/src

cvs -q update -PAd

</pre>

<H3>How stable is ctm for OpenBSD?</H3>

<H3>How stable is CTM for OpenBSD?</H3>

Ctm has been in use for over four years for OpenBSD, and for FreeBSD

CTM has been in use for over four years for OpenBSD, and for FreeBSD

for an even longer period of time. Ctm is very reliable and stable.

for an even longer period of time. CTM is very reliable and stable.

<h3>What does the future hold for CTM?</h3>

There are numerous (small) changes and optimizations possible.

Among them are:

<ul>

<li> CTM never sents a delta bigger than 5MB via mail. This could be

changed in two ways: The maximum delta size could be increased,

or a queue could be installed, from which only a certain number

of messages are mailed out daily. I. e., if a delta is 6MB in

size, it is split into 96 mail messages. These are put into a

queue. On the first day, the first 50 messages are sent, on the

second day the remaining 46, plus the first 4 messages of the

following delta.

<li> CTM could be run two times per day.

</ul>

<H3>More information</H3>

If you want to learn more about ctm, a good place to start with is the

If you want to learn more about CTM, a good place to start with is the

FreeBSD Handbook:

<ul>

Line 320

Line 355

</a>

</ul>

There are also man pages for all of the ctm utilities.

There are also man pages for all of the CTM utilities.

<p>

<h3>Summary:</h3>

Summary:

<p>

Line 339

Line 373

<li><tt>echo subscribe ctm-log | mail majordomo@OpenBSD.org</tt>

</ul>

<li> FTP (for base and relative deltas):

<li> FTP (for base sets and relative deltas):

<ul>

Line 352

Line 386

</ul>

<p>

Important notes and announcements about ctm will be posted to:

Important notes and announcements about CTM will be posted to:

<ul>

<li> misc@OpenBSD.org

<li> announce@OpenBSD.org

Line 363

Line 397

<H3>Acknowledgements</H3>

<ul>

<li> Poul-Henning Kamp (phk@FreeBSD.org) for making ctm possible and helping

<li> Poul-Henning Kamp (phk@FreeBSD.org) for making CTM possible and helping

getting the delta generation running

<li> Theo de Raadt (deraadt@theos.com) for making OpenBSD possible and

initially providing the resources needed for creating the ctm

initially providing the resources needed for creating the CTM

deltas on cvs.OpenBSD.org (now they are generated elsewhere), and

for the nameserver magic

<li> Todd C. Miller (millert@OpenBSD.org) for setting up and

maintaining the ctm mailinglists

maintaining the CTM mailinglists

<li> Bob Beck (beck@OpenBSD.org), Artur Grabowski (art@OpenBSD.org),

Magnus Holmberg (mho@OpenBSD.org) and

Wolfram Schneider (wosch@FreeBSD.org) for setting up the ftp space

for the bases and deltas on ctm.*OpenBSD.org

for the base sets and deltas on ctm.*OpenBSD.org

<li> Thomas Graichen (graichen@OpenBSD.org) for starting ctm for

<li> Thomas Graichen (graichen@OpenBSD.org) for starting CTM for

OpenBSD and maintaining it all those years

<li> Wim Vandeputte (wvdeputte@OpenBSD.org) for hosting ctm now

<li> Wim Vandeputte (wvdputte@OpenBSD.org) for hosting CTM now

<li> ... and all the others who contributed indirectly

</ul>

<p>

For any problems, suggestions, reports and questions regarding ctm

contact the ctm maintainer

<a href="mailto:hgw@d1906.inka.de">Hans Günter Weigand</a>.

<p>

OpenBSD/CTM logo designed for the OpenBSD Project by

<a href="mailto:flipk@openbsd.org">Phillip F Knaack</a>.