Annotation of www/ctm.html, Revision 1.21
1.1 graichen 1: <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2//EN">
2: <HTML>
3: <HEAD>
4: <TITLE>CTM</TITLE>
5: </HEAD>
1.5 flipk 6: <BODY BGCOLOR="#FFFFFF" TEXT="#000000" LINK="#23238E">
1.1 graichen 7:
1.5 flipk 8: <center>
1.13 naddy 9: <img alt="CTM for OpenBSD" src="images/bsdctm.gif" width="401" height="126">
1.5 flipk 10: </center>
1.1 graichen 11:
1.19 naddy 12: <H3><font color="#0000e0">What is CTM?</font></H3>
1.1 graichen 13:
14: <P>
1.14 naddy 15: CTM is a system which was designed by Poul-Henning Kamp for making
1.19 naddy 16: changes to a source tree available on a regular basis by email.
1.14 naddy 17: This is a good way to stay up to date with the current source tree or
18: CVS repository if you have a slow Internet connection, for instance via modem.
1.1 graichen 19:
1.5 flipk 20: <P>
1.19 naddy 21: OpenBSD CTM distributes all modules of the OpenBSD CVS repository
1.21 ! pvalchev 22: plus checked out versions of several of these modules.
1.14 naddy 23: CTM uses very low bandwidth for distributing the changes to the tree,
24: and compresses them using <tt><b>gzip -9</b></tt>. In contrast to
25: <a href="cvsup.html">CVSup</a>, CTM is written in plain C language
26: and therefore available for every platform supported by OpenBSD.
1.19 naddy 27: CTM implements a so called push system, while all other methods to
28: get access to the OpenBSD CVS repository are pull systems.
1.14 naddy 29:
30: <p>
1.16 horacio 31: For <b>any</b> problems, suggestions, reports and questions regarding CTM,
1.14 naddy 32: contact the CTM maintainer
33: <a href="mailto:hgw@d1906.inka.de">Hans Günter Weigand</a>.
34: <small>(Please do not ask other OpenBSD people for CTM support,
35: as they are busy doing other things. Thank you.)</small>
1.1 graichen 36:
1.19 naddy 37: <H3><font color="#0000e0">How does it work?</font></H3>
1.1 graichen 38:
1.5 flipk 39: <P>
1.14 naddy 40: The basic idea of CTM is that you subscribe to a special mailing list
1.19 naddy 41: for a particular CVS module or source tree. Each day you receive email messages
1.5 flipk 42: containing all the changes to that tree during that period.
1.14 naddy 43: One such set of differences is called a <b>delta</b>.
1.15 naddy 44: Actually, the CTM server makes new deltas every twelve hours.
1.1 graichen 45:
1.5 flipk 46: <P>
1.13 naddy 47: You begin with a base set, against which later deltas are patched. To
1.14 naddy 48: begin using CTM you download the latest base set and all deltas
1.13 naddy 49: generated after that. Once you have processed those you can process
50: later deltas you receive from the mailing list. The base sets are split
1.14 naddy 51: into pieces of 1400kB each for easier downloading over bad lines.
1.1 graichen 52:
1.5 flipk 53: <P>
1.13 naddy 54: Base sets are generated once every 50 relative deltas, so you never
55: have to grab more than 50 deltas to catch up.
56:
57: <P>
1.14 naddy 58: There are currently two kinds of base set files. For quite a while we
1.13 naddy 59: were making <tt><b>tar.gz</b></tt> files manually, but we are
1.19 naddy 60: back to real CTM base deltas. The latter contain
1.16 horacio 61: <a href="http://www.openbsd.org/cgi-bin/man.cgi?query=md5&sektion=1">md5(1)</a>
1.13 naddy 62: checksums and are generated automatically, thus are a bit safer. You
1.16 horacio 63: can distinguish these two kinds of base sets by their filename. An
1.13 naddy 64: example for a <tt><b>tar.gz</b></tt> file is
1.14 naddy 65: <tt><b>OpenBSD-cvs.1450.tar.gz</b></tt>, whereas a CTM base delta
1.13 naddy 66: would be named <tt><b>OpenBSD-cvs.1500A.gz</b></tt>. See below for
67: detailed instructions.
1.1 graichen 68:
1.19 naddy 69: <H3><font color="#0000e0">How much bandwidth does CTM use?</font></H3>
1.1 graichen 70:
1.5 flipk 71: <P>
1.13 naddy 72: You obtain the base set and all relative deltas up to the recent one
73: via ftp from the sites listed below. Sizes are:
1.5 flipk 74:
75: <ul>
1.19 naddy 76: <li>Approximately 215MB for OpenBSD-cvs.
77: <li>Approximately 40MB for OpenBSD-cvs-x11.
78: <li>Approximately 65MB for OpenBSD-cvs-xf4.
79: <li>Approximately 83MB for OpenBSD-src.
80: <li>Approximately 36MB for OpenBSD-src-x11.
81: <li>Approximately 4MB for OpenBSD-ports.
1.14 naddy 82: <br>Keep in mind that these are compressed sizes. The uncompressed
1.13 naddy 83: files are about five times as large.
1.14 naddy 84: <li>Relative deltas vary from 10kB to 500kB, and occasionally a couple of
1.13 naddy 85: megabytes.
1.14 naddy 86: <li>CTM always splits deltas into 100kB mail messages which are reassembled
1.13 naddy 87: automatically.
1.14 naddy 88: <li>If a delta is larger than 5MB, it is not sent via mail. Instead,
89: you will receive a notification to get the delta by ftp. Larger
90: CTM updates are rare, though, usually occurring only after a major
1.19 naddy 91: import of something like binutils, Perl, gcc, XFree86, etc., or applying a
92: tag to the CVS repository.
1.5 flipk 93: </ul>
1.1 graichen 94:
1.19 naddy 95: <H3><font color="#0000e0">How do I use CTM?</font></H3>
1.1 graichen 96:
97: <UL>
1.14 naddy 98: <LI>Subscribe to the CTM mailing list for the source set you want.
1.5 flipk 99: <p>
1.14 naddy 100: There are several mailing lists related to CTM. There is a list for
101: updates to the CVS repository, a list for the source tree, a list
1.13 naddy 102: for the X11 part of both, and one for the ports tree, plus
1.14 naddy 103: a list for log messages from the CTM delta generation runs.
1.5 flipk 104: <p>
105: The following commands will subscribe you to the appropriate lists:
1.13 naddy 106: <p>
1.19 naddy 107: For the CVS repository (CVSROOT, ports, src and www modules,
108: but not X11 and XF4):
1.13 naddy 109: <pre>
110: echo subscribe OpenBSD-cvs | mail majordomo@OpenBSD.org
111: </pre>
1.15 naddy 112: For the X11 module of the CVS repository:
1.13 naddy 113: <pre>
114: echo subscribe OpenBSD-cvs-x11 | mail majordomo@OpenBSD.org
115: </pre>
1.19 naddy 116: For the XF4 module of the CVS repository:
117: <pre>
118: echo subscribe OpenBSD-cvs-xf4 | mail majordomo@OpenBSD.org
119: </pre>
120: For a checked out copy of the src module:
1.13 naddy 121: <pre>
122: echo subscribe OpenBSD-src | mail majordomo@OpenBSD.org
123: </pre>
1.19 naddy 124: For a checked out copy of the X11 module:
1.13 naddy 125: <pre>
126: echo subscribe OpenBSD-src-x11 | mail majordomo@OpenBSD.org
127: </pre>
1.19 naddy 128: For a checked out copy of the ports module:
1.13 naddy 129: <pre>
130: echo subscribe OpenBSD-ports-ctm | mail majordomo@OpenBSD.org
131: </pre>
1.14 naddy 132: For the CTM log list:
1.13 naddy 133: <pre>
134: echo subscribe ctm-log | mail majordomo@OpenBSD.org
135: </pre>
136:
137: <p>
1.14 naddy 138: <li>Install the CTM program from a package or from the ports
1.13 naddy 139: collection. You'll find the port at
140: <tt><b>/usr/ports/devel/ctm</b></tt>.
141:
142: </ul>
143: <b>Note:</b> The examples below just cover OpenBSD-cvs for clarity. The
144: other source sets are handled in the same fashion.
145:
146: <p><ul>
147:
148: <li>Download the base set from an FTP site. You'll find it in the
149: subdirectory <tt><b>base/OpenBSD-cvs</b></tt>. Get all files in
150: that directory. Once you have them all you can create the base set
1.14 naddy 151: file and unpack it with the following commands, depending on the type
152: of file you got:
1.13 naddy 153:
154: <pre>
155: cat split/OpenBSD-cvs.1500A.gz.* > OpenBSD-cvs.1500A.gz
156: cd target
157: ctm -v -v -v .../OpenBSD-cvs.1500A.gz
158: </pre>
159: <p>
160: or
161:
162: <pre>
1.19 naddy 163: cat split/OpenBSD-cvs.1450.tar.gz.* > OpenBSD-cvs.1450.tar.gz
1.13 naddy 164: cd target
1.19 naddy 165: tar -xzvf .../OpenBSD-cvs.1450.tar.gz
1.13 naddy 166: </pre>
167:
168: <p>
169: where <tt><b>split</b></tt> is the directory in which you have
170: placed the files you downloaded and <tt><b>target</b></tt> is the
1.14 naddy 171: place you have chosen for unpacking the source set.
1.5 flipk 172:
1.14 naddy 173: <p> It is not necessary to keep these base set files around, if
174: you're sure you've got a complete tree. But you can keep the base set
175: file(s) for backup purposes. However, be sure to note the number of
176: the base set before continuing.
1.5 flipk 177:
178: <p>
1.13 naddy 179: <li>You now need to get the deltas which have been generated since the
1.19 naddy 180: base set. Go to one of the FTP sites listed below into the subdirectory
1.13 naddy 181: <tt><b>OpenBSD-cvs</b></tt> and download every file whose number is
182: <em>larger</em> than the number of the base set you used. Then apply
183: these deltas:
1.5 flipk 184:
1.13 naddy 185: <pre>
186: cd target
187: ctm -v -v -v deltas/OpenBSD-cvs.*
188: </pre>
1.5 flipk 189:
1.13 naddy 190: <p>
191: where <tt><b>target</b></tt> is the directory where your source set
192: tree is, and <tt><b>deltas</b></tt> is where you have stored the
193: deltas.
1.5 flipk 194:
1.13 naddy 195: <p>
196: <LI>Wait until you get your first delta from the mailing list.
197: <p>
1.14 naddy 198: <LI>Unpack, assemble, and apply the CTM delta using <tt><b>ctm_rmail</b></tt>.
1.13 naddy 199: <p>
200:
201: <pre>
202: ctm_rmail -p /tmp -d deltas -b target folder
203: </pre>
1.5 flipk 204:
205: <p>
1.13 naddy 206: where <tt><b>folder</b></tt> is the mail folder containing the delta
1.14 naddy 207: mail. This will decode the CTM delta and save it in the
1.13 naddy 208: <tt><b>deltas</b></tt> directory. The delta will be a file of the
209: form <tt><b>OpenBSD-cvs.<var>XXXX</var>.gz</b></tt>, where <var>XXXX</var>
210: is the number of the delta. Then the delta will be applied to the
211: directory tree located at <tt><b>target</b></tt>, the place where you
1.14 naddy 212: unpacked the base set.
1.5 flipk 213:
214: <p>
1.13 naddy 215: Repeat this last step every time you receive a delta from the
1.19 naddy 216: mailing list. If you like, you can automate this step using procmail,
217: maildrop, or any similar program from the
218: <a href="ports.html">ports collection</a>.
1.13 naddy 219: </ul>
1.5 flipk 220:
221: <p>
1.19 naddy 222: Here is the list of FTP servers carrying CTM deltas:
1.10 deraadt 223:
224: <ul>
1.15 naddy 225: <li>The primary FTP server can be reached at:
226: <p>
1.17 jufi 227: <a href="ftp://openbsd.rug.ac.be:/pub/OpenBSD-ctm/">
228: ftp://openbsd.rug.ac.be:/pub/OpenBSD-ctm/
1.15 naddy 229: </a>.
230: <p>
231: This server is automatically updated by the CTM delta generation process.
232: Base sets are
233: available as big single files in the <tt><b>base</b></tt> directory.
234: The split files are in <tt><b>base-split</b></tt>.
1.10 deraadt 235: </ul>
236:
237: <ul>
238: <li><a href="ftp://ctm.se.OpenBSD.org:/pub/OpenBSD-ctm/">
239: ftp://ctm.se.OpenBSD.org:/pub/OpenBSD-ctm/
1.5 flipk 240: </a>
1.15 naddy 241: <p>
242: This server is updated by CTM deltas received from the mailing lists. It
243: additionally has the split base deltas in the <tt><b>base</b></tt>
1.19 naddy 244: subdirectory, mirrored from
245: <a href="ftp://openbsd.rug.ac.be:/pub/OpenBSD-ctm/">
246: ftp://openbsd.rug.ac.be:/pub/OpenBSD-ctm/
247: </a>.
1.5 flipk 248: </ul>
249:
1.14 naddy 250: <ul>
1.15 naddy 251: <li><a href="ftp://ctm.OpenBSD.org:/pub/OpenBSD-ctm/">
252: ftp://ctm.OpenBSD.org:/pub/OpenBSD-ctm/
253: </a>
254: <p>
255: is identical to
256: <p>
257: <li><a href="ftp://ctm.ca.OpenBSD.org:/pub/OpenBSD-ctm/">
258: ftp://ctm.ca.OpenBSD.org:/pub/OpenBSD-ctm/
259: </a>
260: <p>
261: and is a mirror of
262: <a href="ftp://ctm.se.OpenBSD.org:/pub/OpenBSD-ctm/">
263: ftp://ctm.se.OpenBSD.org
264: </a>
265: .
1.14 naddy 266: </ul>
267:
1.5 flipk 268: <p>
1.14 naddy 269: <b>NOTE:</b> <a href="mailto:hgw@d1906.inka.de">I'm</a> very interested
270: in finding other sites around the world that would like to
271: mirror these deltas.
1.5 flipk 272: <p>
1.1 graichen 273:
1.19 naddy 274: <h3><font color="#0000e0">Should I choose the source tree or the CVS repository?</font></h3>
1.1 graichen 275:
1.5 flipk 276: <P>
1.14 naddy 277: This depends on your diskspace, among other factors. Since CTM
278: doesn't deal very well with files which have been modified locally,
1.19 naddy 279: the better choice is surely the <a href="anoncvs.html">CVS</a>
1.16 horacio 280: repository. You can check out your working copy of the source tree
1.19 naddy 281: from your CVS repository then, can track a stable branch,
282: and keep your local modifications in
1.16 horacio 283: your source tree. CVS is smart enough to merge in your local changes.
1.19 naddy 284: You can also use the full range of CVS commands except commands like
285: <tt><b>cvs commit</b></tt> and <tt><b>cvs tag</b></tt>.
1.5 flipk 286:
287: <p>
1.14 naddy 288: The drawback, however, is the amount of diskspace it takes. A
1.19 naddy 289: checked-out source tree takes about 370MB or so, but if you have your
1.14 naddy 290: own CVS repository you need more than 1Gb for it, <b>plus</b> another
1.19 naddy 291: 370MB for the checked out tree.
1.13 naddy 292: This doesn't count the 100MB or more (depending on the architecture)
1.19 naddy 293: you will need for generated files during a build, i.e. compiling the sources.
1.5 flipk 294:
295: <p>
296: If you just get the source tree, you will need to deal with local
297: modifications. One way to do this is to use the <b>union</b> filesystem,
1.16 horacio 298: (see <a href="http://www.openbsd.org/cgi-bin/man.cgi?query=mount_union&sektion=8">mount_union(8)</a>)
299: although there have been reports that the <b>union</b> filesystem can be
300: unstable if both the upper and lower layers reside on the same physical
301: filesystem. If you place your upper and lower layers on separate
302: filesystems you should have no problems. A sample of this is as follows:
1.5 flipk 303:
304: <p>
1.14 naddy 305: Suppose your CTM-updated tree is in <tt><b>/usr/src-ctm</b></tt>, and
1.13 naddy 306: your real source tree, the one you make your modifications to and do
307: your builds from, is in <tt><b>/usr/src</b></tt>.
308: <tt><b>/usr/src</b></tt> should be initially an empty directory. The
309: following command will set up the union mount:
310:
311: <pre>
312: mount -t union -o -b /usr/src-ctm /usr/src
313: </pre>
314:
315: <p>
1.14 naddy 316: Modifications made to files in <tt><b>/usr/src</b></tt> will become a
317: file contained within <tt><b>/usr/src</b></tt>, hiding the one in
318: <tt><b>src-ctm</b></tt>. If changes are made via CTM to the
319: underlying <tt><b>src-ctm</b></tt> tree, those changes will <b>not</b>
320: be seen if there is a file in the upper layer hiding it.
1.5 flipk 321:
322: <p>
323: You should periodically unmount the
324: union and search for files which are local to the union filesystem.
1.13 naddy 325:
326: <pre>
327: umount /usr/src
328: find /usr/src -type f
329: </pre>
330:
331: <p>
332: The commands <tt><b>ls -W</b></tt> and <tt><b>rm -W</b></tt> will be
1.19 naddy 333: useful too, as objects named "whiteouts" in <tt><b>/usr/src</b></tt>
334: will also hide files in <tt><b>src-ctm</b></tt>.
1.13 naddy 335:
336: <p>
1.14 naddy 337: If you get the CVS repository, you can use the <tt><b>cvs checkout</b></tt>
338: command to check out a source tree from it. Each time you update
339: with CTM you can use the <tt><b>cvs update</b></tt> command to
340: update your source repository.
1.13 naddy 341:
342: <p>
1.14 naddy 343: To get an initial src repository:
1.13 naddy 344: <pre>
345: cd /usr
1.20 chris 346: cvs -qd YOUR_CVS_REPOSITORY checkout -P src
1.13 naddy 347: </pre>
348: <p>
1.14 naddy 349: and after each CTM update:
1.13 naddy 350: <pre>
351: cd /usr/src
1.20 chris 352: cvs -q update -Pd
1.13 naddy 353: </pre>
1.5 flipk 354:
1.19 naddy 355: <H3><font color="#0000e0">How stable is CTM for OpenBSD?</font></H3>
1.5 flipk 356:
1.19 naddy 357: CTM has been in use for over five years for OpenBSD, and for FreeBSD
1.14 naddy 358: for an even longer period of time. CTM is very reliable and stable.
359:
1.19 naddy 360: <h3><font color="#0000e0">What does the future hold for CTM?</font></h3>
1.14 naddy 361:
362: There are numerous (small) changes and optimizations possible.
1.15 naddy 363: Among them:
1.14 naddy 364:
365: <ul>
1.15 naddy 366: <li> CTM never sends a delta bigger than 5MB via mail. This could be
1.14 naddy 367: changed in two ways: The maximum delta size could be increased,
368: or a queue could be installed, from which only a certain number
1.15 naddy 369: of messages are mailed out daily. E. g., if a delta is 6MB in
1.14 naddy 370: size, it is split into 96 mail messages. These are put into a
371: queue. On the first day, the first 50 messages are sent, on the
372: second day the remaining 46, plus the first 4 messages of the
373: following delta.
1.15 naddy 374: <li> The CTM software could be rewritten, or at least fixed in a
1.16 horacio 375: better manner. Its C implemtation is not an example of good
1.15 naddy 376: programming. That would also be a nice opportunity for adding
377: new features, like signing CTM deltas with PGP.
1.14 naddy 378: </ul>
1.5 flipk 379:
1.19 naddy 380: Mail <a href="mailto:hgw@d1906.inka.de">me</a> your opinions and wishes!
381:
382: <H3><font color="#0000e0">More information</font></H3>
1.5 flipk 383:
1.14 naddy 384: If you want to learn more about CTM, a good place to start with is the
1.13 naddy 385: FreeBSD Handbook:
1.5 flipk 386:
387: <ul>
388: <li> <a href="http://www.FreeBSD.org/handbook/ctm.html">
389: http://www.FreeBSD.org/handbook/ctm.html
390: </a>
391: </ul>
392:
1.14 naddy 393: There are also man pages for all of the CTM utilities.
1.5 flipk 394:
1.19 naddy 395: <h3><font color="#0000e0">Summary:</font></h3>
1.5 flipk 396:
397: <p>
398: <ul>
399: <li> Mailing lists:
400:
401: <ul>
1.13 naddy 402: <li><tt>echo subscribe OpenBSD-cvs | mail majordomo@OpenBSD.org</tt>
403: <li><tt>echo subscribe OpenBSD-cvs-x11 | mail majordomo@OpenBSD.org</tt>
1.19 naddy 404: <li><tt>echo subscribe OpenBSD-cvs-xf4 | mail majordomo@OpenBSD.org</tt>
1.13 naddy 405: <li><tt>echo subscribe OpenBSD-src | mail majordomo@OpenBSD.org</tt>
406: <li><tt>echo subscribe OpenBSD-src-x11 | mail majordomo@OpenBSD.org</tt>
407: <li><tt>echo subscribe OpenBSD-ports-ctm | mail majordomo@OpenBSD.org</tt>
408: <li><tt>echo subscribe ctm-log | mail majordomo@OpenBSD.org</tt>
1.5 flipk 409: </ul>
1.15 naddy 410: <p>
1.14 naddy 411: <li> FTP (for base sets and relative deltas):
1.5 flipk 412:
413: <ul>
1.18 jufi 414: <li> <a href="ftp://openbsd.rug.ac.be:/pub/OpenBSD-ctm/">
415: ftp://openbsd.rug.ac.be:/pub/OpenBSD-ctm/</a>
416: <li> <a href="ftp://ctm.OpenBSD.org:/pub/OpenBSD-ctm/">
417: ftp://ctm.OpenBSD.org:/pub/OpenBSD-ctm/</a>
418: <li> <a href="ftp://ctm.ca.OpenBSD.org:/pub/OpenBSD-ctm/">
419: ftp://ctm.ca.OpenBSD.org:/pub/OpenBSD-ctm/</a>
420: <li> <a href="ftp://ctm.se.OpenBSD.org:/pub/OpenBSD-ctm/">
421: ftp://ctm.se.OpenBSD.org:/pub/OpenBSD-ctm/</a>
1.5 flipk 422: </ul>
423: </ul>
424:
425: <p>
1.19 naddy 426: Important notes and announcements about CTM will be posted to the
427: announce@OpenBSD.org mailing list.
1.5 flipk 428:
429: <p>
430:
1.19 naddy 431: <H3><font color="#0000e0">Acknowledgements</font></H3>
1.5 flipk 432:
1.13 naddy 433: <ul>
1.14 naddy 434: <li> Poul-Henning Kamp (phk@FreeBSD.org) for making CTM possible and helping
1.13 naddy 435: getting the delta generation running
1.5 flipk 436:
437: <li> Theo de Raadt (deraadt@theos.com) for making OpenBSD possible and
1.14 naddy 438: initially providing the resources needed for creating the CTM
1.13 naddy 439: deltas on cvs.OpenBSD.org (now they are generated elsewhere), and
1.10 deraadt 440: for the nameserver magic
441:
442: <li> Todd C. Miller (millert@OpenBSD.org) for setting up and
1.14 naddy 443: maintaining the CTM mailinglists
1.10 deraadt 444:
445: <li> Bob Beck (beck@OpenBSD.org), Artur Grabowski (art@OpenBSD.org),
446: Magnus Holmberg (mho@OpenBSD.org) and
447: Wolfram Schneider (wosch@FreeBSD.org) for setting up the ftp space
1.14 naddy 448: for the base sets and deltas on ctm.*OpenBSD.org
1.1 graichen 449:
1.14 naddy 450: <li> Thomas Graichen (graichen@OpenBSD.org) for starting CTM for
1.13 naddy 451: OpenBSD and maintaining it all those years
1.1 graichen 452:
1.14 naddy 453: <li> Wim Vandeputte (wvdputte@OpenBSD.org) for hosting CTM now
1.1 graichen 454:
1.13 naddy 455: <li> ... and all the others who contributed indirectly
456: </ul>
1.1 graichen 457:
458:
1.13 naddy 459: <p>
1.5 flipk 460: OpenBSD/CTM logo designed for the OpenBSD Project by
461: <a href="mailto:flipk@openbsd.org">Phillip F Knaack</a>.
1.12 jufi 462: <p>
463: <hr>
464: <a href="index.html"><img height="24" width="24" src="back.gif" border="0" alt="OpenBSD" /></a>
465: <a href="mailto:www@openbsd.org">www@openbsd.org</a>
1.13 naddy 466: <br>
1.21 ! pvalchev 467: <small>$OpenBSD: ctm.html,v 1.20 2002/02/02 05:14:48 chris Exp $</small>
1.1 graichen 468:
1.12 jufi 469: </body>
470: </html>