Annotation of www/ctm.html, Revision 1.15
1.1 graichen 1: <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2//EN">
2: <HTML>
3: <HEAD>
4: <TITLE>CTM</TITLE>
5: </HEAD>
1.5 flipk 6: <BODY BGCOLOR="#FFFFFF" TEXT="#000000" LINK="#23238E">
1.1 graichen 7:
1.5 flipk 8: <center>
1.13 naddy 9: <img alt="CTM for OpenBSD" src="images/bsdctm.gif" width="401" height="126">
1.5 flipk 10: </center>
1.1 graichen 11:
1.14 naddy 12: <H3>What is CTM?</H3>
1.1 graichen 13:
14: <P>
1.14 naddy 15: CTM is a system which was designed by Poul-Henning Kamp for making
1.5 flipk 16: changes to a source tree available on a daily basis by email.
1.14 naddy 17: This is a good way to stay up to date with the current source tree or
18: CVS repository if you have a slow Internet connection, for instance via modem.
1.1 graichen 19:
1.5 flipk 20: <P>
1.14 naddy 21: CTM uses very low bandwidth for distributing the changes to the tree,
22: and compresses them using <tt><b>gzip -9</b></tt>. In contrast to
23: <a href="cvsup.html">CVSup</a>, CTM is written in plain C language
24: and therefore available for every platform supported by OpenBSD.
25:
26: <p>
27: For <b>any</b> problems, suggestions, reports and questions regarding CTM
28: contact the CTM maintainer
29: <a href="mailto:hgw@d1906.inka.de">Hans Günter Weigand</a>.
30: <small>(Please do not ask other OpenBSD people for CTM support,
31: as they are busy doing other things. Thank you.)</small>
1.1 graichen 32:
1.5 flipk 33: <H3>How does it work?</H3>
1.1 graichen 34:
1.5 flipk 35: <P>
1.14 naddy 36: The basic idea of CTM is that you subscribe to a special mailing list
1.5 flipk 37: for a particular source tree. Each day you receive an email message
38: containing all the changes to that tree during that period.
1.14 naddy 39: One such set of differences is called a <b>delta</b>.
1.15 ! naddy 40: Actually, the CTM server makes new deltas every twelve hours.
1.1 graichen 41:
1.5 flipk 42: <P>
1.13 naddy 43: You begin with a base set, against which later deltas are patched. To
1.14 naddy 44: begin using CTM you download the latest base set and all deltas
1.13 naddy 45: generated after that. Once you have processed those you can process
46: later deltas you receive from the mailing list. The base sets are split
1.14 naddy 47: into pieces of 1400kB each for easier downloading over bad lines.
1.1 graichen 48:
1.5 flipk 49: <P>
1.13 naddy 50: Base sets are generated once every 50 relative deltas, so you never
51: have to grab more than 50 deltas to catch up.
52:
53: <P>
1.14 naddy 54: There are currently two kinds of base set files. For quite a while we
1.13 naddy 55: were making <tt><b>tar.gz</b></tt> files manually, but we are
1.14 naddy 56: switching back to real CTM base deltas. The latter contain MD5
1.13 naddy 57: checksums and are generated automatically, thus are a bit safer. You
1.14 naddy 58: can distinguish these two kinds of base set by their filename. An
1.13 naddy 59: example for a <tt><b>tar.gz</b></tt> file is
1.14 naddy 60: <tt><b>OpenBSD-cvs.1450.tar.gz</b></tt>, whereas a CTM base delta
1.13 naddy 61: would be named <tt><b>OpenBSD-cvs.1500A.gz</b></tt>. See below for
62: detailed instructions.
1.1 graichen 63:
1.14 naddy 64: <H3>How much bandwidth does CTM use?</H3>
1.1 graichen 65:
1.5 flipk 66: <P>
1.13 naddy 67: You obtain the base set and all relative deltas up to the recent one
68: via ftp from the sites listed below. Sizes are:
1.5 flipk 69:
70: <ul>
1.13 naddy 71: <li>Approximately 115MB for OpenBSD-cvs.
72: <li>Approximately 37MB for OpenBSD-cvs-x11.
1.15 ! naddy 73: <li>Approximately 56MB for OpenBSD-cvs-xf4.
1.13 naddy 74: <li>Approximately 65MB for OpenBSD-src.
75: <li>Approximately 35MB for OpenBSD-src-x11.
76: <li>Approximately 2MB for OpenBSD-ports.
1.14 naddy 77: <br>Keep in mind that these are compressed sizes. The uncompressed
1.13 naddy 78: files are about five times as large.
1.14 naddy 79: <li>Relative deltas vary from 10kB to 500kB, and occasionally a couple of
1.13 naddy 80: megabytes.
1.14 naddy 81: <li>CTM always splits deltas into 100kB mail messages which are reassembled
1.13 naddy 82: automatically.
1.14 naddy 83: <li>If a delta is larger than 5MB, it is not sent via mail. Instead,
84: you will receive a notification to get the delta by ftp. Larger
85: CTM updates are rare, though, usually occurring only after a major
86: import of something like binutils, perl, gcc, etc, or applying a
87: CVS tag to the CVS repository.
1.5 flipk 88: </ul>
1.1 graichen 89:
1.14 naddy 90: <H3>How do I use CTM?</H3>
1.1 graichen 91:
92: <UL>
1.14 naddy 93: <LI>Subscribe to the CTM mailing list for the source set you want.
1.5 flipk 94: <p>
1.14 naddy 95: There are several mailing lists related to CTM. There is a list for
96: updates to the CVS repository, a list for the source tree, a list
1.13 naddy 97: for the X11 part of both, and one for the ports tree, plus
1.14 naddy 98: a list for log messages from the CTM delta generation runs.
1.5 flipk 99: <p>
100: The following commands will subscribe you to the appropriate lists:
1.13 naddy 101: <p>
1.15 ! naddy 102: For the CVS repository (without the X11 and XF4 modules):
1.13 naddy 103: <pre>
104: echo subscribe OpenBSD-cvs | mail majordomo@OpenBSD.org
105: </pre>
1.15 ! naddy 106: For the X11 module of the CVS repository:
1.13 naddy 107: <pre>
108: echo subscribe OpenBSD-cvs-x11 | mail majordomo@OpenBSD.org
109: </pre>
1.15 ! naddy 110: For the source tree (without the X11 and XF4 modules):
1.13 naddy 111: <pre>
112: echo subscribe OpenBSD-src | mail majordomo@OpenBSD.org
113: </pre>
1.15 ! naddy 114: For the X11 module of the source tree:
1.13 naddy 115: <pre>
116: echo subscribe OpenBSD-src-x11 | mail majordomo@OpenBSD.org
117: </pre>
118: For the ports tree:
119: <pre>
120: echo subscribe OpenBSD-ports-ctm | mail majordomo@OpenBSD.org
121: </pre>
1.14 naddy 122: For the CTM log list:
1.13 naddy 123: <pre>
124: echo subscribe ctm-log | mail majordomo@OpenBSD.org
125: </pre>
126:
1.15 ! naddy 127: There is no mailing list for the XF4 module, yet. It is available
! 128: via FTP only.
! 129:
1.13 naddy 130: <p>
1.14 naddy 131: <li>Install the CTM program from a package or from the ports
1.13 naddy 132: collection. You'll find the port at
133: <tt><b>/usr/ports/devel/ctm</b></tt>.
134:
135: </ul>
136: <b>Note:</b> The examples below just cover OpenBSD-cvs for clarity. The
137: other source sets are handled in the same fashion.
138:
139: <p><ul>
140:
141: <li>Download the base set from an FTP site. You'll find it in the
142: subdirectory <tt><b>base/OpenBSD-cvs</b></tt>. Get all files in
143: that directory. Once you have them all you can create the base set
1.14 naddy 144: file and unpack it with the following commands, depending on the type
145: of file you got:
1.13 naddy 146:
147: <pre>
148: cat split/OpenBSD-cvs.1500A.gz.* > OpenBSD-cvs.1500A.gz
149: cd target
150: ctm -v -v -v .../OpenBSD-cvs.1500A.gz
151: </pre>
152: <p>
153: or
154:
155: <pre>
156: cat split/OpenBSD-cvs.1500.tar.gz.* > OpenBSD-cvs.1500.tar.gz
157: cd target
158: tar -xzvf .../OpenBSD-cvs.1500.tar.gz
159: </pre>
160:
161: <p>
162: where <tt><b>split</b></tt> is the directory in which you have
163: placed the files you downloaded and <tt><b>target</b></tt> is the
1.14 naddy 164: place you have chosen for unpacking the source set.
1.5 flipk 165:
1.14 naddy 166: <p> It is not necessary to keep these base set files around, if
167: you're sure you've got a complete tree. But you can keep the base set
168: file(s) for backup purposes. However, be sure to note the number of
169: the base set before continuing.
1.5 flipk 170:
171: <p>
1.13 naddy 172: <li>You now need to get the deltas which have been generated since the
173: base set. Go to the FTP site in the subdirectory
174: <tt><b>OpenBSD-cvs</b></tt> and download every file whose number is
175: <em>larger</em> than the number of the base set you used. Then apply
176: these deltas:
1.5 flipk 177:
1.13 naddy 178: <pre>
179: cd target
180: ctm -v -v -v deltas/OpenBSD-cvs.*
181: </pre>
1.5 flipk 182:
1.13 naddy 183: <p>
184: where <tt><b>target</b></tt> is the directory where your source set
185: tree is, and <tt><b>deltas</b></tt> is where you have stored the
186: deltas.
1.5 flipk 187:
1.13 naddy 188: <p>
189: <LI>Wait until you get your first delta from the mailing list.
190: <p>
1.14 naddy 191: <LI>Unpack, assemble, and apply the CTM delta using <tt><b>ctm_rmail</b></tt>.
1.13 naddy 192: <p>
193:
194: <pre>
195: ctm_rmail -p /tmp -d deltas -b target folder
196: </pre>
1.5 flipk 197:
198: <p>
1.13 naddy 199: where <tt><b>folder</b></tt> is the mail folder containing the delta
1.14 naddy 200: mail. This will decode the CTM delta and save it in the
1.13 naddy 201: <tt><b>deltas</b></tt> directory. The delta will be a file of the
202: form <tt><b>OpenBSD-cvs.<var>XXXX</var>.gz</b></tt>, where <var>XXXX</var>
203: is the number of the delta. Then the delta will be applied to the
204: directory tree located at <tt><b>target</b></tt>, the place where you
1.14 naddy 205: unpacked the base set.
1.5 flipk 206:
207: <p>
1.13 naddy 208: Repeat this last step every time you receive a delta from the
209: mailing list. If you like you can automate this step using procmail
210: or any similar program.
211: </ul>
1.5 flipk 212:
213: <p>
1.14 naddy 214: The addresses of the FTP servers carrying CTM deltas are:
1.10 deraadt 215:
216: <ul>
1.15 ! naddy 217: <li>The primary FTP server can be reached at:
! 218: <p>
! 219: <a href="ftp://openbsd.rug.ac.be:/ftp/pub/OpenBSD-ctm/">
! 220: ftp://openbsd.rug.ac.be:/ftp/pub/OpenBSD-ctm/
! 221: </a>.
! 222: <p>
! 223: This server is automatically updated by the CTM delta generation process.
! 224: Base sets are
! 225: available as big single files in the <tt><b>base</b></tt> directory.
! 226: The split files are in <tt><b>base-split</b></tt>.
1.10 deraadt 227: </ul>
228:
229: <ul>
230: <li><a href="ftp://ctm.se.OpenBSD.org:/pub/OpenBSD-ctm/">
231: ftp://ctm.se.OpenBSD.org:/pub/OpenBSD-ctm/
1.5 flipk 232: </a>
1.15 ! naddy 233: <p>
! 234: This server is updated by CTM deltas received from the mailing lists. It
! 235: additionally has the split base deltas in the <tt><b>base</b></tt>
! 236: subdirectory.
1.5 flipk 237: </ul>
238:
1.14 naddy 239: <ul>
1.15 ! naddy 240: <li><a href="ftp://ctm.OpenBSD.org:/pub/OpenBSD-ctm/">
! 241: ftp://ctm.OpenBSD.org:/pub/OpenBSD-ctm/
! 242: </a>
! 243: <p>
! 244: is identical to
! 245: <p>
! 246: <li><a href="ftp://ctm.ca.OpenBSD.org:/pub/OpenBSD-ctm/">
! 247: ftp://ctm.ca.OpenBSD.org:/pub/OpenBSD-ctm/
! 248: </a>
! 249: <p>
! 250: and is a mirror of
! 251: <a href="ftp://ctm.se.OpenBSD.org:/pub/OpenBSD-ctm/">
! 252: ftp://ctm.se.OpenBSD.org
! 253: </a>
! 254: .
1.14 naddy 255: </ul>
256:
1.5 flipk 257: <p>
1.14 naddy 258: <b>NOTE:</b> <a href="mailto:hgw@d1906.inka.de">I'm</a> very interested
259: in finding other sites around the world that would like to
260: mirror these deltas.
1.5 flipk 261: <p>
1.1 graichen 262:
1.14 naddy 263: <h3>Should I choose the source tree or the CVS repository?</h3>
1.1 graichen 264:
1.5 flipk 265: <P>
1.14 naddy 266: This depends on your diskspace, among other factors. Since CTM
267: doesn't deal very well with files which have been modified locally,
268: the "better" choice is probably the CVS repository. You can check out
269: your working copy of the source tree from your CVS repository then,
270: and keep your local modifications in your source tree. CVS is
271: smart enough to merge in your local changes. You can also use the
272: full range of CVS commands except <tt><b>cvs commit</b></tt>.
1.5 flipk 273:
274: <p>
1.14 naddy 275: The drawback, however, is the amount of diskspace it takes. A
276: checked-out source tree takes about 350MB or so, but if you have your
277: own CVS repository you need more than 1Gb for it, <b>plus</b> another
278: 350MB for the checked out tree.
1.13 naddy 279: This doesn't count the 100MB or more (depending on the architecture)
280: you will need to do a build, i.e. compile the sources.
1.5 flipk 281:
282: <p>
283: If you just get the source tree, you will need to deal with local
284: modifications. One way to do this is to use the <b>union</b> filesystem,
285: although there have been reports that the <b>union</b> filesystem can be
286: unstable if both the upper and lower layers reside on the same physical
1.6 todd 287: filesystem. If you place your upper and lower layers on separate filesystems
1.5 flipk 288: you should have no problems. A sample of this is as follows:
289:
290: <p>
1.14 naddy 291: Suppose your CTM-updated tree is in <tt><b>/usr/src-ctm</b></tt>, and
1.13 naddy 292: your real source tree, the one you make your modifications to and do
293: your builds from, is in <tt><b>/usr/src</b></tt>.
294: <tt><b>/usr/src</b></tt> should be initially an empty directory. The
295: following command will set up the union mount:
296:
297: <pre>
298: mount -t union -o -b /usr/src-ctm /usr/src
299: </pre>
300:
301: <p>
1.14 naddy 302: Modifications made to files in <tt><b>/usr/src</b></tt> will become a
303: file contained within <tt><b>/usr/src</b></tt>, hiding the one in
304: <tt><b>src-ctm</b></tt>. If changes are made via CTM to the
305: underlying <tt><b>src-ctm</b></tt> tree, those changes will <b>not</b>
306: be seen if there is a file in the upper layer hiding it.
1.5 flipk 307:
308: <p>
309: You should periodically unmount the
310: union and search for files which are local to the union filesystem.
1.13 naddy 311:
312: <pre>
313: umount /usr/src
314: find /usr/src -type f
315: </pre>
316:
317: <p>
318: The commands <tt><b>ls -W</b></tt> and <tt><b>rm -W</b></tt> will be
1.14 naddy 319: useful too, as objects named "whiteouts" in <tt><b>/usr/src</b></tt> will also hide
320: files in <tt><b>src-ctm</b></tt>.
1.13 naddy 321:
322: <p>
1.14 naddy 323: If you get the CVS repository, you can use the <tt><b>cvs checkout</b></tt>
324: command to check out a source tree from it. Each time you update
325: with CTM you can use the <tt><b>cvs update</b></tt> command to
326: update your source repository.
1.13 naddy 327:
328: <p>
1.14 naddy 329: To get an initial src repository:
1.13 naddy 330: <pre>
331: cd /usr
1.14 naddy 332: cvs -qd YOUR_CVS_REPOSITORY checkout -A src
1.13 naddy 333: </pre>
334: <p>
1.14 naddy 335: and after each CTM update:
1.13 naddy 336: <pre>
337: cd /usr/src
338: cvs -q update -PAd
339: </pre>
1.5 flipk 340:
1.14 naddy 341: <H3>How stable is CTM for OpenBSD?</H3>
1.5 flipk 342:
1.14 naddy 343: CTM has been in use for over four years for OpenBSD, and for FreeBSD
344: for an even longer period of time. CTM is very reliable and stable.
345:
346: <h3>What does the future hold for CTM?</h3>
347:
348: There are numerous (small) changes and optimizations possible.
1.15 ! naddy 349: Among them:
1.14 naddy 350:
351: <ul>
1.15 ! naddy 352: <li> CTM never sends a delta bigger than 5MB via mail. This could be
1.14 naddy 353: changed in two ways: The maximum delta size could be increased,
354: or a queue could be installed, from which only a certain number
1.15 ! naddy 355: of messages are mailed out daily. E. g., if a delta is 6MB in
1.14 naddy 356: size, it is split into 96 mail messages. These are put into a
357: queue. On the first day, the first 50 messages are sent, on the
358: second day the remaining 46, plus the first 4 messages of the
359: following delta.
1.15 ! naddy 360: <li> The CTM software could be rewritten, or at least fixed in a
! 361: better manner. It's C implemtation is not an example of good
! 362: programming. That would also be a nice opportunity for adding
! 363: new features, like signing CTM deltas with PGP.
1.14 naddy 364: </ul>
1.5 flipk 365:
366: <H3>More information</H3>
367:
1.14 naddy 368: If you want to learn more about CTM, a good place to start with is the
1.13 naddy 369: FreeBSD Handbook:
1.5 flipk 370:
371: <ul>
372: <li> <a href="http://www.FreeBSD.org/handbook/ctm.html">
373: http://www.FreeBSD.org/handbook/ctm.html
374: </a>
375: </ul>
376:
1.14 naddy 377: There are also man pages for all of the CTM utilities.
1.5 flipk 378:
1.14 naddy 379: <h3>Summary:</h3>
1.5 flipk 380:
381: <p>
382:
383: <ul>
384: <li> Mailing lists:
385:
386: <ul>
1.13 naddy 387: <li><tt>echo subscribe OpenBSD-cvs | mail majordomo@OpenBSD.org</tt>
388: <li><tt>echo subscribe OpenBSD-cvs-x11 | mail majordomo@OpenBSD.org</tt>
389: <li><tt>echo subscribe OpenBSD-src | mail majordomo@OpenBSD.org</tt>
390: <li><tt>echo subscribe OpenBSD-src-x11 | mail majordomo@OpenBSD.org</tt>
391: <li><tt>echo subscribe OpenBSD-ports-ctm | mail majordomo@OpenBSD.org</tt>
392: <li><tt>echo subscribe ctm-log | mail majordomo@OpenBSD.org</tt>
1.5 flipk 393: </ul>
1.15 ! naddy 394: <p>
1.14 naddy 395: <li> FTP (for base sets and relative deltas):
1.5 flipk 396:
397: <ul>
1.15 ! naddy 398: <li> <a href="ftp://openbsd.rug.ac.be:/ftp/pub/OpenBSD-ctm/">
! 399: ftp://openbsd.rug.ac.be:/ftp/pub/OpenBSD-ctm/</a>
1.10 deraadt 400: <li> <a href="ftp://ctm.OpenBSD.org:/pub/OpenBSD-ctm">
1.13 naddy 401: ftp://ctm.OpenBSD.org:/pub/OpenBSD-ctm</a>
1.10 deraadt 402: <li> <a href="ftp://ctm.ca.OpenBSD.org:/pub/OpenBSD-ctm">
1.13 naddy 403: ftp://ctm.ca.OpenBSD.org:/pub/OpenBSD-ctm</a>
1.10 deraadt 404: <li> <a href="ftp://ctm.se.OpenBSD.org:/pub/OpenBSD-ctm">
1.13 naddy 405: ftp://ctm.se.OpenBSD.org:/pub/OpenBSD-ctm</a>
1.5 flipk 406: </ul>
407: </ul>
408:
409: <p>
1.14 naddy 410: Important notes and announcements about CTM will be posted to:
1.5 flipk 411: <ul>
1.15 ! naddy 412: <li> announce@OpenBSD.org
1.5 flipk 413: <li> misc@OpenBSD.org
414: </ul>
415:
416: <p>
417:
418: <H3>Acknowledgements</H3>
419:
1.13 naddy 420: <ul>
1.14 naddy 421: <li> Poul-Henning Kamp (phk@FreeBSD.org) for making CTM possible and helping
1.13 naddy 422: getting the delta generation running
1.5 flipk 423:
424: <li> Theo de Raadt (deraadt@theos.com) for making OpenBSD possible and
1.14 naddy 425: initially providing the resources needed for creating the CTM
1.13 naddy 426: deltas on cvs.OpenBSD.org (now they are generated elsewhere), and
1.10 deraadt 427: for the nameserver magic
428:
429: <li> Todd C. Miller (millert@OpenBSD.org) for setting up and
1.14 naddy 430: maintaining the CTM mailinglists
1.10 deraadt 431:
432: <li> Bob Beck (beck@OpenBSD.org), Artur Grabowski (art@OpenBSD.org),
433: Magnus Holmberg (mho@OpenBSD.org) and
434: Wolfram Schneider (wosch@FreeBSD.org) for setting up the ftp space
1.14 naddy 435: for the base sets and deltas on ctm.*OpenBSD.org
1.1 graichen 436:
1.14 naddy 437: <li> Thomas Graichen (graichen@OpenBSD.org) for starting CTM for
1.13 naddy 438: OpenBSD and maintaining it all those years
1.1 graichen 439:
1.14 naddy 440: <li> Wim Vandeputte (wvdputte@OpenBSD.org) for hosting CTM now
1.1 graichen 441:
1.13 naddy 442: <li> ... and all the others who contributed indirectly
443: </ul>
1.1 graichen 444:
445:
1.13 naddy 446: <p>
1.5 flipk 447: OpenBSD/CTM logo designed for the OpenBSD Project by
448: <a href="mailto:flipk@openbsd.org">Phillip F Knaack</a>.
1.12 jufi 449: <p>
450: <hr>
451: <a href="index.html"><img height="24" width="24" src="back.gif" border="0" alt="OpenBSD" /></a>
452: <a href="mailto:www@openbsd.org">www@openbsd.org</a>
1.13 naddy 453: <br>
1.15 ! naddy 454: <small>$OpenBSD: ctm.html,v 1.14 2001/02/24 23:37:06 naddy Exp $</small>
1.1 graichen 455:
1.12 jufi 456: </body>
457: </html>
![[BACK]](/icons/back.gif){kind=icon}
Return to ctm.html CVS log ![[TXT]](/icons/text.gif){kind=icon}
![[DIR]](/icons/dir.gif){kind=icon}
Up to [local] / www