Annotation of www/ctm.html, Revision 1.18
1.1 graichen 1: <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2//EN">
2: <HTML>
3: <HEAD>
4: <TITLE>CTM</TITLE>
5: </HEAD>
1.5 flipk 6: <BODY BGCOLOR="#FFFFFF" TEXT="#000000" LINK="#23238E">
1.1 graichen 7:
1.5 flipk 8: <center>
1.13 naddy 9: <img alt="CTM for OpenBSD" src="images/bsdctm.gif" width="401" height="126">
1.5 flipk 10: </center>
1.1 graichen 11:
1.14 naddy 12: <H3>What is CTM?</H3>
1.1 graichen 13:
14: <P>
1.14 naddy 15: CTM is a system which was designed by Poul-Henning Kamp for making
1.5 flipk 16: changes to a source tree available on a daily basis by email.
1.14 naddy 17: This is a good way to stay up to date with the current source tree or
18: CVS repository if you have a slow Internet connection, for instance via modem.
1.1 graichen 19:
1.5 flipk 20: <P>
1.14 naddy 21: CTM uses very low bandwidth for distributing the changes to the tree,
22: and compresses them using <tt><b>gzip -9</b></tt>. In contrast to
23: <a href="cvsup.html">CVSup</a>, CTM is written in plain C language
24: and therefore available for every platform supported by OpenBSD.
25:
26: <p>
1.16 horacio 27: For <b>any</b> problems, suggestions, reports and questions regarding CTM,
1.14 naddy 28: contact the CTM maintainer
29: <a href="mailto:hgw@d1906.inka.de">Hans Günter Weigand</a>.
30: <small>(Please do not ask other OpenBSD people for CTM support,
31: as they are busy doing other things. Thank you.)</small>
1.1 graichen 32:
1.5 flipk 33: <H3>How does it work?</H3>
1.1 graichen 34:
1.5 flipk 35: <P>
1.14 naddy 36: The basic idea of CTM is that you subscribe to a special mailing list
1.5 flipk 37: for a particular source tree. Each day you receive an email message
38: containing all the changes to that tree during that period.
1.14 naddy 39: One such set of differences is called a <b>delta</b>.
1.15 naddy 40: Actually, the CTM server makes new deltas every twelve hours.
1.1 graichen 41:
1.5 flipk 42: <P>
1.13 naddy 43: You begin with a base set, against which later deltas are patched. To
1.14 naddy 44: begin using CTM you download the latest base set and all deltas
1.13 naddy 45: generated after that. Once you have processed those you can process
46: later deltas you receive from the mailing list. The base sets are split
1.14 naddy 47: into pieces of 1400kB each for easier downloading over bad lines.
1.1 graichen 48:
1.5 flipk 49: <P>
1.13 naddy 50: Base sets are generated once every 50 relative deltas, so you never
51: have to grab more than 50 deltas to catch up.
52:
53: <P>
1.14 naddy 54: There are currently two kinds of base set files. For quite a while we
1.13 naddy 55: were making <tt><b>tar.gz</b></tt> files manually, but we are
1.16 horacio 56: switching back to real CTM base deltas. The latter contain
57: <a href="http://www.openbsd.org/cgi-bin/man.cgi?query=md5&sektion=1">md5(1)</a>
1.13 naddy 58: checksums and are generated automatically, thus are a bit safer. You
1.16 horacio 59: can distinguish these two kinds of base sets by their filename. An
1.13 naddy 60: example for a <tt><b>tar.gz</b></tt> file is
1.14 naddy 61: <tt><b>OpenBSD-cvs.1450.tar.gz</b></tt>, whereas a CTM base delta
1.13 naddy 62: would be named <tt><b>OpenBSD-cvs.1500A.gz</b></tt>. See below for
63: detailed instructions.
1.1 graichen 64:
1.14 naddy 65: <H3>How much bandwidth does CTM use?</H3>
1.1 graichen 66:
1.5 flipk 67: <P>
1.13 naddy 68: You obtain the base set and all relative deltas up to the recent one
69: via ftp from the sites listed below. Sizes are:
1.5 flipk 70:
71: <ul>
1.13 naddy 72: <li>Approximately 115MB for OpenBSD-cvs.
73: <li>Approximately 37MB for OpenBSD-cvs-x11.
1.15 naddy 74: <li>Approximately 56MB for OpenBSD-cvs-xf4.
1.13 naddy 75: <li>Approximately 65MB for OpenBSD-src.
76: <li>Approximately 35MB for OpenBSD-src-x11.
77: <li>Approximately 2MB for OpenBSD-ports.
1.14 naddy 78: <br>Keep in mind that these are compressed sizes. The uncompressed
1.13 naddy 79: files are about five times as large.
1.14 naddy 80: <li>Relative deltas vary from 10kB to 500kB, and occasionally a couple of
1.13 naddy 81: megabytes.
1.14 naddy 82: <li>CTM always splits deltas into 100kB mail messages which are reassembled
1.13 naddy 83: automatically.
1.14 naddy 84: <li>If a delta is larger than 5MB, it is not sent via mail. Instead,
85: you will receive a notification to get the delta by ftp. Larger
86: CTM updates are rare, though, usually occurring only after a major
87: import of something like binutils, perl, gcc, etc, or applying a
88: CVS tag to the CVS repository.
1.5 flipk 89: </ul>
1.1 graichen 90:
1.14 naddy 91: <H3>How do I use CTM?</H3>
1.1 graichen 92:
93: <UL>
1.14 naddy 94: <LI>Subscribe to the CTM mailing list for the source set you want.
1.5 flipk 95: <p>
1.14 naddy 96: There are several mailing lists related to CTM. There is a list for
97: updates to the CVS repository, a list for the source tree, a list
1.13 naddy 98: for the X11 part of both, and one for the ports tree, plus
1.14 naddy 99: a list for log messages from the CTM delta generation runs.
1.5 flipk 100: <p>
101: The following commands will subscribe you to the appropriate lists:
1.13 naddy 102: <p>
1.15 naddy 103: For the CVS repository (without the X11 and XF4 modules):
1.13 naddy 104: <pre>
105: echo subscribe OpenBSD-cvs | mail majordomo@OpenBSD.org
106: </pre>
1.15 naddy 107: For the X11 module of the CVS repository:
1.13 naddy 108: <pre>
109: echo subscribe OpenBSD-cvs-x11 | mail majordomo@OpenBSD.org
110: </pre>
1.15 naddy 111: For the source tree (without the X11 and XF4 modules):
1.13 naddy 112: <pre>
113: echo subscribe OpenBSD-src | mail majordomo@OpenBSD.org
114: </pre>
1.15 naddy 115: For the X11 module of the source tree:
1.13 naddy 116: <pre>
117: echo subscribe OpenBSD-src-x11 | mail majordomo@OpenBSD.org
118: </pre>
119: For the ports tree:
120: <pre>
121: echo subscribe OpenBSD-ports-ctm | mail majordomo@OpenBSD.org
122: </pre>
1.14 naddy 123: For the CTM log list:
1.13 naddy 124: <pre>
125: echo subscribe ctm-log | mail majordomo@OpenBSD.org
126: </pre>
127:
1.15 naddy 128: There is no mailing list for the XF4 module, yet. It is available
129: via FTP only.
130:
1.13 naddy 131: <p>
1.14 naddy 132: <li>Install the CTM program from a package or from the ports
1.13 naddy 133: collection. You'll find the port at
134: <tt><b>/usr/ports/devel/ctm</b></tt>.
135:
136: </ul>
137: <b>Note:</b> The examples below just cover OpenBSD-cvs for clarity. The
138: other source sets are handled in the same fashion.
139:
140: <p><ul>
141:
142: <li>Download the base set from an FTP site. You'll find it in the
143: subdirectory <tt><b>base/OpenBSD-cvs</b></tt>. Get all files in
144: that directory. Once you have them all you can create the base set
1.14 naddy 145: file and unpack it with the following commands, depending on the type
146: of file you got:
1.13 naddy 147:
148: <pre>
149: cat split/OpenBSD-cvs.1500A.gz.* > OpenBSD-cvs.1500A.gz
150: cd target
151: ctm -v -v -v .../OpenBSD-cvs.1500A.gz
152: </pre>
153: <p>
154: or
155:
156: <pre>
157: cat split/OpenBSD-cvs.1500.tar.gz.* > OpenBSD-cvs.1500.tar.gz
158: cd target
159: tar -xzvf .../OpenBSD-cvs.1500.tar.gz
160: </pre>
161:
162: <p>
163: where <tt><b>split</b></tt> is the directory in which you have
164: placed the files you downloaded and <tt><b>target</b></tt> is the
1.14 naddy 165: place you have chosen for unpacking the source set.
1.5 flipk 166:
1.14 naddy 167: <p> It is not necessary to keep these base set files around, if
168: you're sure you've got a complete tree. But you can keep the base set
169: file(s) for backup purposes. However, be sure to note the number of
170: the base set before continuing.
1.5 flipk 171:
172: <p>
1.13 naddy 173: <li>You now need to get the deltas which have been generated since the
174: base set. Go to the FTP site in the subdirectory
175: <tt><b>OpenBSD-cvs</b></tt> and download every file whose number is
176: <em>larger</em> than the number of the base set you used. Then apply
177: these deltas:
1.5 flipk 178:
1.13 naddy 179: <pre>
180: cd target
181: ctm -v -v -v deltas/OpenBSD-cvs.*
182: </pre>
1.5 flipk 183:
1.13 naddy 184: <p>
185: where <tt><b>target</b></tt> is the directory where your source set
186: tree is, and <tt><b>deltas</b></tt> is where you have stored the
187: deltas.
1.5 flipk 188:
1.13 naddy 189: <p>
190: <LI>Wait until you get your first delta from the mailing list.
191: <p>
1.14 naddy 192: <LI>Unpack, assemble, and apply the CTM delta using <tt><b>ctm_rmail</b></tt>.
1.13 naddy 193: <p>
194:
195: <pre>
196: ctm_rmail -p /tmp -d deltas -b target folder
197: </pre>
1.5 flipk 198:
199: <p>
1.13 naddy 200: where <tt><b>folder</b></tt> is the mail folder containing the delta
1.14 naddy 201: mail. This will decode the CTM delta and save it in the
1.13 naddy 202: <tt><b>deltas</b></tt> directory. The delta will be a file of the
203: form <tt><b>OpenBSD-cvs.<var>XXXX</var>.gz</b></tt>, where <var>XXXX</var>
204: is the number of the delta. Then the delta will be applied to the
205: directory tree located at <tt><b>target</b></tt>, the place where you
1.14 naddy 206: unpacked the base set.
1.5 flipk 207:
208: <p>
1.13 naddy 209: Repeat this last step every time you receive a delta from the
1.16 horacio 210: mailing list. If you like, you can automate this step using procmail
1.13 naddy 211: or any similar program.
212: </ul>
1.5 flipk 213:
214: <p>
1.14 naddy 215: The addresses of the FTP servers carrying CTM deltas are:
1.10 deraadt 216:
217: <ul>
1.15 naddy 218: <li>The primary FTP server can be reached at:
219: <p>
1.17 jufi 220: <a href="ftp://openbsd.rug.ac.be:/pub/OpenBSD-ctm/">
221: ftp://openbsd.rug.ac.be:/pub/OpenBSD-ctm/
1.15 naddy 222: </a>.
223: <p>
224: This server is automatically updated by the CTM delta generation process.
225: Base sets are
226: available as big single files in the <tt><b>base</b></tt> directory.
227: The split files are in <tt><b>base-split</b></tt>.
1.10 deraadt 228: </ul>
229:
230: <ul>
231: <li><a href="ftp://ctm.se.OpenBSD.org:/pub/OpenBSD-ctm/">
232: ftp://ctm.se.OpenBSD.org:/pub/OpenBSD-ctm/
1.5 flipk 233: </a>
1.15 naddy 234: <p>
235: This server is updated by CTM deltas received from the mailing lists. It
236: additionally has the split base deltas in the <tt><b>base</b></tt>
237: subdirectory.
1.5 flipk 238: </ul>
239:
1.14 naddy 240: <ul>
1.15 naddy 241: <li><a href="ftp://ctm.OpenBSD.org:/pub/OpenBSD-ctm/">
242: ftp://ctm.OpenBSD.org:/pub/OpenBSD-ctm/
243: </a>
244: <p>
245: is identical to
246: <p>
247: <li><a href="ftp://ctm.ca.OpenBSD.org:/pub/OpenBSD-ctm/">
248: ftp://ctm.ca.OpenBSD.org:/pub/OpenBSD-ctm/
249: </a>
250: <p>
251: and is a mirror of
252: <a href="ftp://ctm.se.OpenBSD.org:/pub/OpenBSD-ctm/">
253: ftp://ctm.se.OpenBSD.org
254: </a>
255: .
1.14 naddy 256: </ul>
257:
1.5 flipk 258: <p>
1.14 naddy 259: <b>NOTE:</b> <a href="mailto:hgw@d1906.inka.de">I'm</a> very interested
260: in finding other sites around the world that would like to
261: mirror these deltas.
1.5 flipk 262: <p>
1.1 graichen 263:
1.14 naddy 264: <h3>Should I choose the source tree or the CVS repository?</h3>
1.1 graichen 265:
1.5 flipk 266: <P>
1.14 naddy 267: This depends on your diskspace, among other factors. Since CTM
268: doesn't deal very well with files which have been modified locally,
1.16 horacio 269: the "better" choice is probably the <a href="anoncvs.html">CVS</a>
270: repository. You can check out your working copy of the source tree
271: from your CVS repository then, and keep your local modifications in
272: your source tree. CVS is smart enough to merge in your local changes.
273: You can also use the full range of CVS commands except <tt><b>cvs
274: commit</b></tt>.
1.5 flipk 275:
276: <p>
1.14 naddy 277: The drawback, however, is the amount of diskspace it takes. A
278: checked-out source tree takes about 350MB or so, but if you have your
279: own CVS repository you need more than 1Gb for it, <b>plus</b> another
280: 350MB for the checked out tree.
1.13 naddy 281: This doesn't count the 100MB or more (depending on the architecture)
282: you will need to do a build, i.e. compile the sources.
1.5 flipk 283:
284: <p>
285: If you just get the source tree, you will need to deal with local
286: modifications. One way to do this is to use the <b>union</b> filesystem,
1.16 horacio 287: (see <a href="http://www.openbsd.org/cgi-bin/man.cgi?query=mount_union&sektion=8">mount_union(8)</a>)
288: although there have been reports that the <b>union</b> filesystem can be
289: unstable if both the upper and lower layers reside on the same physical
290: filesystem. If you place your upper and lower layers on separate
291: filesystems you should have no problems. A sample of this is as follows:
1.5 flipk 292:
293: <p>
1.14 naddy 294: Suppose your CTM-updated tree is in <tt><b>/usr/src-ctm</b></tt>, and
1.13 naddy 295: your real source tree, the one you make your modifications to and do
296: your builds from, is in <tt><b>/usr/src</b></tt>.
297: <tt><b>/usr/src</b></tt> should be initially an empty directory. The
298: following command will set up the union mount:
299:
300: <pre>
301: mount -t union -o -b /usr/src-ctm /usr/src
302: </pre>
303:
304: <p>
1.14 naddy 305: Modifications made to files in <tt><b>/usr/src</b></tt> will become a
306: file contained within <tt><b>/usr/src</b></tt>, hiding the one in
307: <tt><b>src-ctm</b></tt>. If changes are made via CTM to the
308: underlying <tt><b>src-ctm</b></tt> tree, those changes will <b>not</b>
309: be seen if there is a file in the upper layer hiding it.
1.5 flipk 310:
311: <p>
312: You should periodically unmount the
313: union and search for files which are local to the union filesystem.
1.13 naddy 314:
315: <pre>
316: umount /usr/src
317: find /usr/src -type f
318: </pre>
319:
320: <p>
321: The commands <tt><b>ls -W</b></tt> and <tt><b>rm -W</b></tt> will be
1.14 naddy 322: useful too, as objects named "whiteouts" in <tt><b>/usr/src</b></tt> will also hide
323: files in <tt><b>src-ctm</b></tt>.
1.13 naddy 324:
325: <p>
1.14 naddy 326: If you get the CVS repository, you can use the <tt><b>cvs checkout</b></tt>
327: command to check out a source tree from it. Each time you update
328: with CTM you can use the <tt><b>cvs update</b></tt> command to
329: update your source repository.
1.13 naddy 330:
331: <p>
1.14 naddy 332: To get an initial src repository:
1.13 naddy 333: <pre>
334: cd /usr
1.14 naddy 335: cvs -qd YOUR_CVS_REPOSITORY checkout -A src
1.13 naddy 336: </pre>
337: <p>
1.14 naddy 338: and after each CTM update:
1.13 naddy 339: <pre>
340: cd /usr/src
341: cvs -q update -PAd
342: </pre>
1.5 flipk 343:
1.14 naddy 344: <H3>How stable is CTM for OpenBSD?</H3>
1.5 flipk 345:
1.14 naddy 346: CTM has been in use for over four years for OpenBSD, and for FreeBSD
347: for an even longer period of time. CTM is very reliable and stable.
348:
349: <h3>What does the future hold for CTM?</h3>
350:
351: There are numerous (small) changes and optimizations possible.
1.15 naddy 352: Among them:
1.14 naddy 353:
354: <ul>
1.15 naddy 355: <li> CTM never sends a delta bigger than 5MB via mail. This could be
1.14 naddy 356: changed in two ways: The maximum delta size could be increased,
357: or a queue could be installed, from which only a certain number
1.15 naddy 358: of messages are mailed out daily. E. g., if a delta is 6MB in
1.14 naddy 359: size, it is split into 96 mail messages. These are put into a
360: queue. On the first day, the first 50 messages are sent, on the
361: second day the remaining 46, plus the first 4 messages of the
362: following delta.
1.15 naddy 363: <li> The CTM software could be rewritten, or at least fixed in a
1.16 horacio 364: better manner. Its C implemtation is not an example of good
1.15 naddy 365: programming. That would also be a nice opportunity for adding
366: new features, like signing CTM deltas with PGP.
1.14 naddy 367: </ul>
1.5 flipk 368:
369: <H3>More information</H3>
370:
1.14 naddy 371: If you want to learn more about CTM, a good place to start with is the
1.13 naddy 372: FreeBSD Handbook:
1.5 flipk 373:
374: <ul>
375: <li> <a href="http://www.FreeBSD.org/handbook/ctm.html">
376: http://www.FreeBSD.org/handbook/ctm.html
377: </a>
378: </ul>
379:
1.14 naddy 380: There are also man pages for all of the CTM utilities.
1.5 flipk 381:
1.14 naddy 382: <h3>Summary:</h3>
1.5 flipk 383:
384: <p>
385:
386: <ul>
387: <li> Mailing lists:
388:
389: <ul>
1.13 naddy 390: <li><tt>echo subscribe OpenBSD-cvs | mail majordomo@OpenBSD.org</tt>
391: <li><tt>echo subscribe OpenBSD-cvs-x11 | mail majordomo@OpenBSD.org</tt>
392: <li><tt>echo subscribe OpenBSD-src | mail majordomo@OpenBSD.org</tt>
393: <li><tt>echo subscribe OpenBSD-src-x11 | mail majordomo@OpenBSD.org</tt>
394: <li><tt>echo subscribe OpenBSD-ports-ctm | mail majordomo@OpenBSD.org</tt>
395: <li><tt>echo subscribe ctm-log | mail majordomo@OpenBSD.org</tt>
1.5 flipk 396: </ul>
1.15 naddy 397: <p>
1.14 naddy 398: <li> FTP (for base sets and relative deltas):
1.5 flipk 399:
400: <ul>
1.18 ! jufi 401: <li> <a href="ftp://openbsd.rug.ac.be:/pub/OpenBSD-ctm/">
! 402: ftp://openbsd.rug.ac.be:/pub/OpenBSD-ctm/</a>
! 403: <li> <a href="ftp://ctm.OpenBSD.org:/pub/OpenBSD-ctm/">
! 404: ftp://ctm.OpenBSD.org:/pub/OpenBSD-ctm/</a>
! 405: <li> <a href="ftp://ctm.ca.OpenBSD.org:/pub/OpenBSD-ctm/">
! 406: ftp://ctm.ca.OpenBSD.org:/pub/OpenBSD-ctm/</a>
! 407: <li> <a href="ftp://ctm.se.OpenBSD.org:/pub/OpenBSD-ctm/">
! 408: ftp://ctm.se.OpenBSD.org:/pub/OpenBSD-ctm/</a>
1.5 flipk 409: </ul>
410: </ul>
411:
412: <p>
1.14 naddy 413: Important notes and announcements about CTM will be posted to:
1.5 flipk 414: <ul>
1.15 naddy 415: <li> announce@OpenBSD.org
1.5 flipk 416: <li> misc@OpenBSD.org
417: </ul>
418:
419: <p>
420:
421: <H3>Acknowledgements</H3>
422:
1.13 naddy 423: <ul>
1.14 naddy 424: <li> Poul-Henning Kamp (phk@FreeBSD.org) for making CTM possible and helping
1.13 naddy 425: getting the delta generation running
1.5 flipk 426:
427: <li> Theo de Raadt (deraadt@theos.com) for making OpenBSD possible and
1.14 naddy 428: initially providing the resources needed for creating the CTM
1.13 naddy 429: deltas on cvs.OpenBSD.org (now they are generated elsewhere), and
1.10 deraadt 430: for the nameserver magic
431:
432: <li> Todd C. Miller (millert@OpenBSD.org) for setting up and
1.14 naddy 433: maintaining the CTM mailinglists
1.10 deraadt 434:
435: <li> Bob Beck (beck@OpenBSD.org), Artur Grabowski (art@OpenBSD.org),
436: Magnus Holmberg (mho@OpenBSD.org) and
437: Wolfram Schneider (wosch@FreeBSD.org) for setting up the ftp space
1.14 naddy 438: for the base sets and deltas on ctm.*OpenBSD.org
1.1 graichen 439:
1.14 naddy 440: <li> Thomas Graichen (graichen@OpenBSD.org) for starting CTM for
1.13 naddy 441: OpenBSD and maintaining it all those years
1.1 graichen 442:
1.14 naddy 443: <li> Wim Vandeputte (wvdputte@OpenBSD.org) for hosting CTM now
1.1 graichen 444:
1.13 naddy 445: <li> ... and all the others who contributed indirectly
446: </ul>
1.1 graichen 447:
448:
1.13 naddy 449: <p>
1.5 flipk 450: OpenBSD/CTM logo designed for the OpenBSD Project by
451: <a href="mailto:flipk@openbsd.org">Phillip F Knaack</a>.
1.12 jufi 452: <p>
453: <hr>
454: <a href="index.html"><img height="24" width="24" src="back.gif" border="0" alt="OpenBSD" /></a>
455: <a href="mailto:www@openbsd.org">www@openbsd.org</a>
1.13 naddy 456: <br>
1.18 ! jufi 457: <small>$OpenBSD: ctm.html,v 1.17 2001/09/30 17:39:11 jufi Exp $</small>
1.1 graichen 458:
1.12 jufi 459: </body>
460: </html>
![[BACK]](/icons/back.gif){kind=icon}
Return to ctm.html CVS log ![[TXT]](/icons/text.gif){kind=icon}
![[DIR]](/icons/dir.gif){kind=icon}
Up to [local] / www