Annotation of src/usr.bin/rsync/rsync.5, Revision 1.1
1.1 ! benno 1: .\" $OpenBSD$
! 2: .\"
! 3: .\" Copyright (c) 2019 Kristaps Dzonsons <kristaps@bsd.lv>
! 4: .\"
! 5: .\" Permission to use, copy, modify, and distribute this software for any
! 6: .\" purpose with or without fee is hereby granted, provided that the above
! 7: .\" copyright notice and this permission notice appear in all copies.
! 8: .\"
! 9: .\" THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
! 10: .\" WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
! 11: .\" MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
! 12: .\" ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
! 13: .\" WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
! 14: .\" ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
! 15: .\" OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
! 16: .\"
! 17: .Dd $Mdocdate$
! 18: .Dt RSYNC 5
! 19: .Os
! 20: .Sh NAME
! 21: .Nm rsync
! 22: .Nd rsync wire protocol
! 23: .Sh DESCRIPTION
! 24: The
! 25: .Nm
! 26: protocol described in this relates to the BSD-licensed
! 27: .Xr openrsync 1 ,
! 28: a re-implementation of the GPL-licensed reference utility
! 29: .Xr rsync 1 .
! 30: It is compatible with version 27 of the reference.
! 31: .Pp
! 32: In this document, the
! 33: .Qq client process
! 34: refers to the utility as run on the operator's local computer.
! 35: The
! 36: .Qq server process
! 37: is run either on the local or remote computer, depending upon the
! 38: command-line given file locations.
! 39: .Pp
! 40: There are a number of options in the protocol that are dictated by command-line
! 41: flags.
! 42: These will be noted as
! 43: .Fl n
! 44: for dry-run,
! 45: .Fl l
! 46: for links,
! 47: .Fl r
! 48: for recursion,
! 49: .Fl v
! 50: for verbose, and
! 51: .Fl -delete
! 52: for deletion (before).
! 53: .Ss Data types
! 54: The binary protocol encodes all data in little-endian format.
! 55: Integers are signed 32-bit, shorts are signed 16-bit, bytes are unsigned
! 56: 8-bit.
! 57: A long is variable-length.
! 58: For values less than the maximum integer, the value is transmitted and
! 59: read as a 32-bit integer.
! 60: For values greater, the value is transmitted first as a maximum integer,
! 61: then a 64-bit signed integer.
! 62: .Pp
! 63: There are three types of checksums: long (slow), short (fast), and
! 64: whole-file.
! 65: The fast checksum is a derivative of Adler-32.
! 66: The slow checksum is MD4,
! 67: made over the checksum seed first (serialised in little-endian format),
! 68: then the data.
! 69: The whole-file applies MD4 to the file first, then the checksum seed at
! 70: the end (also serialised in little-endian format).
! 71: .Ss Multiplexing
! 72: Most
! 73: .Nm
! 74: transmissions are wrapped in a multiplexing envelope protocol.
! 75: It is composed as follows:
! 76: .Pp
! 77: .Bl -enum -compact
! 78: .It
! 79: envelope header (4 bytes)
! 80: .It
! 81: envelope payload (arbitrary length)
! 82: .El
! 83: .Pp
! 84: The first byte of the envelope header consists of a tag.
! 85: If the tag is 7, the payload is normal data.
! 86: Otherwise, the payload is out-of-band server messages.
! 87: If the tag is 1, it is an error on the sender's part and must trigger an
! 88: exit.
! 89: This limits message payloads to 24 bit integer size,
! 90: .Li 0x0fffffff .
! 91: .Pp
! 92: The only data not using this envelope are the initial handshake between
! 93: client and server.
! 94: .Ss File list
! 95: A central part of the protocol is the file list, which is generated by
! 96: the sender.
! 97: It consists of all files that must be sent to the receiver, either
! 98: explicitly as given or recursively generated.
! 99: .Pp
! 100: The file list itself consists of filenames and attributes (mode, time,
! 101: size, etc.).
! 102: Filenames must be relative to the destination root and not be absolute
! 103: or contain backtracking.
! 104: So if a file is given to the sender as
! 105: .Pa ../../foo/bar ,
! 106: it must be sent as
! 107: .Pa foo/bar .
! 108: .Pp
! 109: The file list should be cleaned of inappropriate files prior to sending.
! 110: For example, if
! 111: .Fl l
! 112: is not specified, symbolic links may be omitted.
! 113: Directory entries without
! 114: .Fl r
! 115: may also be omitted.
! 116: Duplicates may be omitted.
! 117: .Pp
! 118: The receiver
! 119: .Em must not
! 120: assume that the file list is clean.
! 121: It should not omit inappropriate files from the file list (which would
! 122: affect the indexing), but may omit them during processing.
! 123: .Pp
! 124: Prior to be sent from sender to receiver, and upon being received, the
! 125: file list must be lexicographically sorted such as with
! 126: .Xr strcmp 3 .
! 127: Subsequent references to the file are by index in the sorted list.
! 128: .Ss Client process
! 129: The client can operate in sender or receiver mode depending upon the
! 130: command-line source and destination.
! 131: .Pp
! 132: If the destination directory (sink) is remote, the client is in sender
! 133: mode: the client will push its data to the server.
! 134: If the source file is remote, it is in receiver mode: the server pushes
! 135: to the client.
! 136: If neither are remote, the client operates in sender mode.
! 137: These are all mutually exclusive.
! 138: .Pp
! 139: When the client starts, regardless its mode, it first handshakes the
! 140: server.
! 141: This exchange is
! 142: .Em not
! 143: multiplexed.
! 144: .Pp
! 145: .Bl -enum -compact
! 146: .It
! 147: send local version (integer)
! 148: .It
! 149: receive remote version (integer)
! 150: .It
! 151: receive random seed (integer)
! 152: .El
! 153: .Pp
! 154: Following this, the client multiplexes when reading from the server.
! 155: Transmissions sent from client to server are not multiplexed.
! 156: It then enters the
! 157: .Sx Update exchange
! 158: protocol.
! 159: .Ss Server process
! 160: The server can operate in sender or receiver mode depending upon how the
! 161: client starts the server.
! 162: This may be directly from the parent process (when invoked for local
! 163: files) or indirectly via a remote shell.
! 164: .Pp
! 165: When in sender mode, the server pushes data to the client.
! 166: (This is equivalent to receiver mode for the client.)
! 167: In receiver, the opposite is true.
! 168: .Pp
! 169: When the server starts, regardless the mode, it first handshakes the
! 170: client.
! 171: This exchange is
! 172: .Em not
! 173: multiplexed.
! 174: .Pp
! 175: .Bl -enum -compact
! 176: .It
! 177: send local version (integer)
! 178: .It
! 179: receive remote version (integer)
! 180: .It
! 181: send random seed (integer)
! 182: .El
! 183: .Pp
! 184: Following this, the server multiplexes when writing to the client.
! 185: (Transmissions received from the client are not multiplexed.)
! 186: It then enters the
! 187: .Sx Update exchange
! 188: protocol.
! 189: .Ss Update exchange
! 190: When the client or server is in sender mode, it begins by conditionally
! 191: sending the exclusion list.
! 192: At this time, this is always empty.
! 193: .Pp
! 194: .Bl -enum -compact
! 195: .It
! 196: if
! 197: .Fl -delete
! 198: and the client, exclusion list zero (integer)
! 199: .El
! 200: .Pp
! 201: It then sends the
! 202: .Sx File list .
! 203: Prior to being sent, the file list should be lexicographically sorted.
! 204: .Pp
! 205: .Bl -enum -compact
! 206: .It
! 207: status byte (integer)
! 208: .It
! 209: inherited filename length (optional, byte)
! 210: .It
! 211: filename length (integer or byte)
! 212: .It
! 213: file (byte array)
! 214: .It
! 215: file length (long)
! 216: .It
! 217: file modification time (optional, time_t, integer)
! 218: .It
! 219: file mode (optional, mode_t, integer)
! 220: .It
! 221: if a symbolic link and
! 222: .Fl l ,
! 223: the link target's length (integer)
! 224: .It
! 225: if a symbolic link and
! 226: .Fl l ,
! 227: the link target (byte array)
! 228: .El
! 229: .Pp
! 230: The status byte may consist of the following bits and determines which
! 231: of the optional fields are transmitted.
! 232: .Pp
! 233: .Bl -tag -compact -width Ds
! 234: .It 0x02
! 235: Do not send the file mode: it is a repeat of the last file's mode.
! 236: .It 0x20
! 237: Inherit some of the prior file name.
! 238: Enables the inherited filename length transmission.
! 239: .It 0x40
! 240: Use full integer length for file name.
! 241: Otherwise, use only the byte length.
! 242: .It 0x80
! 243: Do not send the file modification time: it is a repeat of the last
! 244: file's.
! 245: .El
! 246: .Pp
! 247: If the status byte is zero, the file-list has terminated.
! 248: The sender then sends any IO error values, which for
! 249: .Xr openrsync 1
! 250: is always zero.
! 251: .Pp
! 252: .Bl -enum -compact
! 253: .It
! 254: constant zero (integer)
! 255: .El
! 256: .Pp
! 257: The server sender then reads the exclusion list, which is always zero.
! 258: .Pp
! 259: .Bl -enum -compact
! 260: .It
! 261: if server, constant zero (integer)
! 262: .El
! 263: .Pp
! 264: Following that, the sender receives data regarding the receiver's copy
! 265: of the file list contents.
! 266: This data is not ordered in any way.
! 267: Each of these requests starts as follows:
! 268: .Pp
! 269: .Bl -enum -compact
! 270: .It
! 271: file index or -1 to signal a change of phase (integer)
! 272: .El
! 273: .Pp
! 274: The phase starts in phase 1, then proceeds to phase 2, and phase 3
! 275: signals an end of transmission (no subsequent blocks).
! 276: If a phase change occurs, the sender must write back the -1 constant
! 277: integer value and increment its phase state.
! 278: .Pp
! 279: Blocks are read as follows:
! 280: .Pp
! 281: .Bl -enum -compact
! 282: .It
! 283: block index (integer)
! 284: .El
! 285: .Pp
! 286: In
! 287: .Pq Fl n
! 288: mode, the sender may immediately write back the index (integer) to skip
! 289: the following.
! 290: .Pp
! 291: .Bl -enum -compact
! 292: .It
! 293: number of blocks (integer)
! 294: .It
! 295: block length in the file (integer)
! 296: .It
! 297: long checksum length (integer)
! 298: .It
! 299: terminal (remainder) block length (integer)
! 300: .El
! 301: .Pp
! 302: And for each block:
! 303: .Pp
! 304: .Bl -enum -compact
! 305: .It
! 306: short checksum (integer)
! 307: .It
! 308: long checksum (bytes of checksum length)
! 309: .El
! 310: .Pp
! 311: The client then compares the two files, block by block, and updates the
! 312: server with mismatches as follows.
! 313: .Pp
! 314: .Bl -enum -compact
! 315: .It
! 316: file index (integer)
! 317: .It
! 318: number of blocks (integer)
! 319: .It
! 320: block length (integer)
! 321: .It
! 322: long checksum length (integer)
! 323: .It
! 324: remainder block length (integer)
! 325: .El
! 326: .Pp
! 327: Then for each block:
! 328: .Pp
! 329: .Bl -enum -compact
! 330: .It
! 331: data chunk size (integer)
! 332: .It
! 333: data chunk (bytes)
! 334: .It
! 335: block index subsequent to chunk or zero for finished (integer)
! 336: .El
! 337: .Pp
! 338: Following this sequence, the sender sends the followng:
! 339: .Pp
! 340: .Bl -enum -compact
! 341: .It
! 342: whole-file long checksum (16 bytes)
! 343: .El
! 344: .Pp
! 345: The sender then either handles the next queued file or, if the receiver
! 346: has written a phase change, the phase change step.
! 347: .Pp
! 348: If the sender is the server and
! 349: .Fl v
! 350: has been specified, the sender must send statistics.
! 351: .Pp
! 352: .Bl -enum -compact
! 353: .It
! 354: total bytes read (long)
! 355: .It
! 356: total bytes written (long)
! 357: .It
! 358: total size of files (long)
! 359: .El
! 360: .Pp
! 361: Finally, the sender must read a final constant-value integer.
! 362: .Pp
! 363: .Bl -enum -compact
! 364: .It
! 365: end-of-sequence -1 value (integer)
! 366: .El
! 367: .Pp
! 368: If in receiver mode, the inverse above (write instead of read, read
! 369: instead of write) is performed.
! 370: .Pp
! 371: The receiver begins by conditionally writing, then reading, the
! 372: exclusion list count, which is always zero.
! 373: .Pp
! 374: .Bl -enum -compact
! 375: .It
! 376: if client, send zero (integer)
! 377: .It
! 378: if receiver and
! 379: .Fl -delete ,
! 380: read zero (integer)
! 381: .El
! 382: .Pp
! 383: The receiver then proceeds with reading the
! 384: .Sx File list
! 385: as already
! 386: defined.
! 387: Following the list, the receiver reads the IO error, which must be zero.
! 388: .Pp
! 389: .Bl -enum -compact
! 390: .It
! 391: constant zero (integer)
! 392: .El
! 393: .Pp
! 394: The receiver must then sort the file names lexicographically.
! 395: .Pp
! 396: If there are no files in the file list at this time, the receiver must
! 397: exit prior to sending per-file data.
! 398: It then proceeds with the file blocks.
! 399: .Pp
! 400: For file blocks, the receiver must look at each file that is not up to
! 401: date, defined by having the same file size and timestamp, and send it to
! 402: the server.
! 403: Symbolic links and directory entries are never sent to the server.
! 404: .Pp
! 405: After the second phase has completed and prior to writing the
! 406: end-of-data signal, the client receiver reads statistics.
! 407: This is only performed with
! 408: .Pq Fl v .
! 409: .Pp
! 410: .Bl -enum -compact
! 411: .It
! 412: total bytes read (long)
! 413: .It
! 414: total bytes written (long)
! 415: .It
! 416: total size of files (long)
! 417: .El
! 418: .Pp
! 419: Finally, the receiver must send the constant end-of-sequence marker.
! 420: .Pp
! 421: .Bl -enum -compact
! 422: .It
! 423: end-of-sequence -1 value (integer)
! 424: .El
! 425: .Ss Sender and receiver asynchrony
! 426: The sender and receiver need not work in lockstep.
! 427: The receiver may send file update requests as quickly as it parses them,
! 428: and respond to the sender's update notices on demand.
! 429: Similarly, the sender may read as many update requests as it can, and
! 430: service them in any order it wishes.
! 431: .Pp
! 432: The sender and receiver synchronise state only at the end of phase.
! 433: .Pp
! 434: The reference
! 435: .Xr rsync 1
! 436: takes advantage of this with a two-process receiver, one for sending
! 437: update requests (the generator) and another for receiving.
! 438: .Xr openrsync 1
! 439: uses an event-loop model instead.
! 440: .\" .Sh CONTEXT
! 441: .\" For section 9 functions only.
! 442: .\" .Sh RETURN VALUES
! 443: .\" For sections 2, 3, and 9 function return values only.
! 444: .\" .Sh ENVIRONMENT
! 445: .\" For sections 1, 6, 7, and 8 only.
! 446: .\" .Sh FILES
! 447: .\" .Sh EXIT STATUS
! 448: .\" For sections 1, 6, and 8 only.
! 449: .\" .Sh EXAMPLES
! 450: .\" .Sh DIAGNOSTICS
! 451: .\" For sections 1, 4, 6, 7, 8, and 9 printf/stderr messages only.
! 452: .\" .Sh ERRORS
! 453: .\" For sections 2, 3, 4, and 9 errno settings only.
! 454: .Sh SEE ALSO
! 455: .Xr openrsync 1 ,
! 456: .Xr rsync 1 ,
! 457: .Xr rsyncd 5
! 458: .\" .Sh STANDARDS
! 459: .\" .Sh HISTORY
! 460: .\" .Sh AUTHORS
! 461: .\" .Sh CAVEATS
! 462: .Sh BUGS
! 463: Time values are sent as 32-bit integers.
! 464: .Pp
! 465: When in server mode
! 466: .Em and
! 467: when communicating to a client with a newer protocol (>27), the phase
! 468: change integer (-1) acknowledgement must be sent twice by the sender.
! 469: The is probably a bug in the reference implementation.
![[BACK]](/icons/back.gif){kind=icon}
Return to rsync.5 CVS log ![[TXT]](/icons/text.gif){kind=icon}
![[DIR]](/icons/dir.gif){kind=icon}
Up to [local] / src / usr.bin / rsync