File: [local] / www / papers / bsdcan11-mandoc-openbsd.html (download) (as text)
Revision 1.3, Tue Jul 3 00:29:49 2012 UTC (11 years, 11 months ago) by schwarze
Branch: MAIN
CVS Tags: HEAD
Changes since 1.2: +3 -3 lines
This purely english page was of course intended to be US-ASCII,
so properly encode the four accented characters that accidentally
crept in; patch from Ian Ropers at gmail dot com, thanks.
|
<HTML>
<BODY>
<H1><A HREF="http://www.bsdcan.org/2011/schedule/events/230.en.html">Mandoc
in OpenBSD</A></H1>
<H2>Training a foal to replace a venerable workhorse</H2>
<H3><A HREF="http://www.bsdcan.org/2011/schedule/speakers/146.en.html">Ingo
Schwarze</A><BR>
<schwarze@openbsd.org></H3>
<H4><A HREF="http://www.bsdcan.org/2011/">BSDCan 2011</A><BR>
Ottawa, May 13</H4>
<P>Return to <A HREF="http://www.openbsd.org/papers/">OpenBSD
Presentations & Papers</A></P>
<p><a href="http://www.flickr.com/photos/tomkoadam/4778126822/"><img
src="http://farm5.static.flickr.com/4115/4778126822_555b453a1e.jpg"></a></p>
<p>Csikó - Foal. - Photo: Adam Tomkó @flickr (CC)</p>
<HR>
<P>Ingo Schwarze: Mandoc in OpenBSD - page 2: INTRO I -
BSDCan 2011, May 13, Ottawa</P>
<H1>What for?</H1>
<H2>Formatting man(1) pages.</H2>
<P>→ See also the companion talk by Kristaps Dzonsons,
"The roff tradition and mandoc".</P>
<PRE>
$ mandoc /usr/src/share/man/man7/mdoc.7
$ man mdoc
</PRE>
<H3>Both commands produce:</H3>
<PRE>
$ man mdoc<BR>
MDOC(7) OpenBSD Reference Manual MDOC(7)
NAME
mdoc - mdoc language reference
DESCRIPTION
The mdoc language is used to format BSD UNIX manuals. This reference
document describes its syntax, structure, and usage. The reference
implementation is mandoc(1); the COMPATIBILITY section describes
compatibility with other troff -mdoc implementations.
</PRE>
<H3>Make documentation
<A HREF="http://xkcd.com/912/">easily accessible</A>:</H3>
<UL>
<LI>Have it at as few places as possible,
and as much as possible at one single place.
<LI>In OpenBSD, almost all documentation is manuals,
and the little that doesn't fit there is all in the FAQ.
</UL>
<H3>Thus, manuals are very important in OpenBSD.</H3>
<HR>
<P>Ingo Schwarze: Mandoc in OpenBSD - page 3: INTRO II -
BSDCan 2011, May 13, Ottawa</P>
<H1>What from?</H1>
<P>→ See also Kristaps' talk.</P>
<H2>mdoc(7) or man(7) source files - plus roff(7), tbl(7), ...</H2>
<PRE>
$ less /usr/src/share/man/man7/mdoc.7
.Dd $Mdocdate: June 18 2011 $ \" rcs(1) style Mdocdate ID
.Dt MDOC 7 \" prologue: meta information
.Os
.Sh NAME
.Nm mdoc \" one word document title
.Nd mdoc language reference \" one line document description
.Sh DESCRIPTION \" section structure
The \" free text
.Nm mdoc \" logical markup
language is used to format
.Bx
.Ux
manuals.
This reference document describes its syntax, structure, and
usage.
The reference implementation is
.Xr mandoc 1 ; \" cross references to other manuals
the
.Sx COMPATIBILITY \" cross references inside one manual
section describes compatibility with other troff \-mdoc implementations.
.Pp
An
.Nm
document follows simple rules: lines beginning with the control
character
.Sq \. \" block enclosures
are parsed for macros.
Other lines are interpreted within the scope of
prior macros:
.Bd -literal -offset indent \" display blocks
\&.Sh Macro lines change control state. \" escaping
Other lines are interpreted within the current state.
.Ed
</PRE>
<HR>
<P>Ingo Schwarze: Mandoc in OpenBSD - page 4: INTRO III -
BSDCan 2011, May 13, Ottawa</P>
<H1>Table of contents</H1>
<UL>
<LI>OpenBSD 4.6 - the foal's pasture
<LI>OpenBSD 4.7 - serious training
<LI>OpenBSD 4.8 - drawing the cart
<LI>OpenBSD 4.9 - removing groff
<LI>Recurring topics
<LI>Conclusions
<LI>Future directions
</UL>
<HR>
<P>Ingo Schwarze: Mandoc in OpenBSD - page 5: Apr to Dec 2009 -
BSDCan 2011, May 13, Ottawa</P>
<H1>Grazing the foal's pasture</H1>
<H2>Half a year free of serious responsibilities.</H2>
<UL>
<LI>Imported mandoc into the OpenBSD tree.
<LI>Collected initial experience using mandoc.
<LI>Main occupation during this time: Fixing bugs.
<LI>Slowly started setting up a regression suite.
<LI>Established the contact to everybody involved.
</UL>
<H4>Related timeline:</H4>
2008 Nov 22: first commit to mdocml.bsd.lv by kristaps@<BR>
2009 Mar 27: first direct commit by schwarze@ to OpenBSD (not mandoc)<BR>
2009 Apr 06: mandoc imported into OpenBSD by kristaps@<BR>
2009 Apr 15: first help from another OpenBsd developer (miod@)<BR>
2009 May 23: schwarze@ first talks to jmc@ about mandoc<BR>
2009 May 31: at c2k9 in Edmonton, schwarze@ talks to deraadt@<BR>
2009 Jun 09: kristaps@ agrees to work closely together<BR>
2009 Jun 14: merge to OpenBSD started by schwarze@<BR>
2009 Jun 15: first patches merged back from OpenBSD to bsd.lv<BR>
2009 Jun 21: mandoc usable in OpenBSD and in sync with bsd.lv<BR>
2009 Jun 23: bugfixing in OpenBSD started by schwarze@<BR>
2009 Jul 05: OpenBSD 4.6 release rolled without mandoc<BR>
2009 Jul 12: joerg@ sends his first patch from NetBSD<BR>
2009 Jul 18: uqs@ sends his first patch from FreeBSD<BR>
2009 Oct 27: start src/regress/usr.bin/mandoc<BR>
2010 Jan 02: start of systematic integration<BR>
<HR>
<P>Ingo Schwarze: Mandoc in OpenBSD - page 6: TRAIN I -
Jan and Feb 2010 - BSDCan 2011, May 13, Ottawa</P>
<H1>See the world!</H1>
<H2>Let mandoc build the whole tree.</H2>
<H3>Systematically investigate all fatal issues.</H3>
<UL>
<LI>Ignore all non-fatal issues for now.
<LI>Fix the easy problems in mandoc, right away.
<LI>Work around hard ones by changing manuals in the tree.
</UL>
<H3>Most workarounds got fixed later:</H3>
<UL>
<LI>block opened, but not closed, and vice versa
<LI>bad nesting of (full) blocks
<LI>free-form text in context requiring macros
<LI>blank lines outside literal context
<LI>low-level requests not (yet) implemented in mandoc
<LI>wrong number of arguments
<LI>...
</UL>
<H4>Related timeline:</H4>
2010 Jan 02: first patches to mdoc(7) manuals to fix the build with mandoc<BR>
<EM>"Fine. Even if mandoc goes nowhere, it has found some bugs. ;)"</EM>
<STRONG>jmc@</STRONG><BR>
2010 Feb 17: first patch to a man(7) manual in order to fix the build<BR>
2010 Feb 20: found first manual bug caused by DocBook<BR>
2010 Feb 24: first non-fatal manual fix found by -Tlint<BR>
2010 Feb 25: tree now builds with mandoc<BR>
2010 Mar 18: OpenBSD 4.7 release rolled without mandoc<BR>
<HR>
<P>Ingo Schwarze: Mandoc in OpenBSD - page 7: TRAIN II - Feb 2010 -
BSDCan 2011, May 13, Ottawa</P>
<H1>Block nesting</H1>
<H2>An example of nice mandoc design</H2>
<P>→ See also Kristaps' talk.</P>
<H3>Example manual snippet using nested blocks:</H3>
<PRE>
$ mandoc chgrp.1
SYNOPSIS
chgrp [-fh] [-R [-H | -L | -P]] group file ...
</PRE>
<H3>The corresponding mdoc(7) source code:</H3>
<PRE>
$ less chgrp.1
[...]
.Sh SYNOPSIS
.Nm chgrp
.Op Fl fh
.Oo
.Fl R
.Op Fl H | L | P
.Oc
.Ar group
.Ar
[...]
</PRE>
<H3>Mandoc respresents this by this syntax tree:</H3>
<PRE>
$ mandoc -Ttree chgrp.1 # much simplified
Sh (block) SYNOPSIS
Nm (block) chgrp
Op (block)
Fl (elem) fh
Oo (block)
Fl (elem) R
Op (block)
Fl (elem) H
(text) |
Fl (elem) L
(text) |
Fl (elem) P
Ar (elem) group
Ar (elem) file ...
</PRE>
<H3>Traditional roff/mdoc design: no block structure!</H3>
<H4>low level: roff requests provide</H4>
<UL>
<LI>physical formatting
<LI>registers to store data
<LI>macro expansion
</UL>
<H4>high level: mdoc macros</H4>
<UL>
<LI>call physical formatting
<LI>set registers to keep state
<LI>have no syntax tree
</UL>
<HR>
<P>Ingo Schwarze: Mandoc in OpenBSD - page 8: TRAIN III - Feb to Jun 2010 -
BSDCan 2011, May 13, Ottawa</P>
<H1>Blood, sweat, and tears</H1>
<H2>Badly nested blocks and the Xo macro</H2>
<H3>Explicit blocks, badly nested:</H3>
<PRE>
.Ao ao
.Bo bo
.No ac Ac
.No bc Bc
<ao [bo ac> bc]
</PRE>
<H3>Implicit block broken by explicit block:</H3>
<PRE>
.Aq aq Bo bo eol
.No bc Bc
<aq [bo eol> bc]
</PRE>
<H3>The most important case in practice: .It Xo</H3>
<PRE>
$ less find.1
[...]
.It Xo
.Ic -exec Ar utility
.Op argument ...
.No ;
.Xc
-exec utility [argument ...] ;
</PRE>
<H3>Our first thought: deprecate this abomination!</H3>
<UL>
<LI>use: .It Ic -exec Ar utility Oo argument ... Oc No ;
<LI>but lots of manuals use ".It Xo" - historical argument limit
<LI>no way to change all manuals, everywhere
<LI>or even the habits of people writing manuals...
<LI>reluctant implementation:
<UL>
<LI>Ao block contains the *whole* Bo block
<LI>Bo block contains an Ao body-end element
</UL>
</UL>
<H3>All mdoc(7) manuals build at this point!</H3>
<H4>Related timeline:</H4>
2010 Feb 23 remove .Oo .Xo .Oc .Xc mis-nesting from manuals (questionable)<BR>
2010 Feb 26 support .It Xo (good)<BR>
2010 Jun 29 support badly nested blocks in general (even better)<BR>
<HR>
<P>Ingo Schwarze: Mandoc in OpenBSD - page 9: TRAIN IV - Mar 2010 -
BSDCan 2011, May 13, Ottawa</P>
<H1>Hey, aren't you cheating?</H1>
<H2>Explicit pod2man(1) preamble support.</H2>
<H3>Example code from the standard pod2man preamble:</H3>
<PRE>
$ less perl.1
[...]
.de Sp \" Vertical space (when we can't use .PP)
.if t .sp .5v
.if n .sp
..
.de Vb \" Begin verbatim text
.ft CW
.nf
.ne \\$1
..
.ds C+ C\v'-.1v'\h'-1p'\s-2+\h'-1p'+\s0\v'.1v'\h'-1p'
</PRE>
<H3>This uses low-level roff requests:</H3>
<UL>
<LI>.ds user defined strings
<LI>.de macro definitions
<LI>.if conditionals
<LI>and some more...
</UL>
<H3>No chance to get all that implemented quickly.</H3>
<H4>Quick and dirty solution:</H4>
<UL>
<LI>Hardcode the strings (always the same - it's a preamble).
<LI>Implement the macros as if they were man(7) macros.
<LI>Parse and ignore the rest.
</UL>
<H3>Now all man(7) manuals build as well!</H3>
<P>No fixes in man(7) manuals, just workarounds in mandoc.</P>
<H4>Related timeline:</H4>
2010 Mar 01 implement pod2man(1) pseudo-macros<BR>
2010 Sep 20 mandoc can now handle the standard pod2man preamble<BR>
2010 Nov 28 remove the pod2man(1) pseudo-macros<BR>
<HR>
<P>Ingo Schwarze: Mandoc in OpenBSD - page 10: TRAIN V - conclusions -
BSDCan 2011, May 13, Ottawa</P>
<H1>Don't be lazy.</H1>
<H2>What went wrong while getting the tree to build?</H2>
<UL>
<LI>We should have started several months earlier.
<LI>At first, mandoc reported way too many fatal errors.
<LI>Too few mandoc bugs were fixed in this phase.
<LI>Too many workarounds committed to manuals instead.
</UL>
<H3>Lessons learnt:</H3>
<UL>
<LI>From the start, avoid throwing fatal errors when possible.
<LI>By all means, report errors, but try hard to keep going.
<LI>Try real-world usage as early as possible and use the feedback.
<LI>Move quickly into production, don't delay for polishing.
<LI>Code will only really mature when used.
</UL>
<p><a href="http://www.flickr.com/photos/kristavandervoorden/4737488285/"><img
src="http://farm5.static.flickr.com/4134/4737488285_1048b87c8a.jpg"></a></p>
<p>New Forest Foal. -
Photo: Krista van der Voorden <Kris*V*@flickr> (CC)</p>
<HR>
<P>Ingo Schwarze: Mandoc in OpenBSD - page 11: DRAW I - Apr 2010 -
BSDCan 2011, May 13, Ottawa</P>
<H1>What are you waiting for?</H1>
<H2>Switch over the tree as soon as possible.</H2>
<UL>
<LI>Ready for production just before the 4.7 release.
<LI>Switch over the build right after the release,
to have as much time as possible to fix fallout.
<LI>Bug reports from real-world users come in now.
<UL>
<LI>Give priority to these bug reports.
<LI>All serious fallout fixed very quickly - within a few days.
</UL>
<LI>Lots of time for polishing before the first release.
</UL>
<H4>Related timeline:</H4>
2010 Mar 01: mandoc ready to build the tree<BR>
2010 Mar 18: OpenBSD 4.7 release rolled without mandoc<BR>
2010 Mar 20: Xenocara can now build with mandoc as well<BR>
2010 Apr 02: link mandoc to the OpenBSD build<BR>
2010 Apr 03: switch base build to mandoc, excepting tbl pages<BR>
2010 Apr 03: fix all fatal issues where mandoc kills ports builds<BR>
2010 Apr 04: first port maintainer explicitely switches to mandoc<BR>
2010 Apr 04: fix first mandoc bug that was found by ports usage<BR>
2010 Apr 05: espie@ implements USE_GROFF framework and groff-1.15 port<BR>
2010 Apr 07: first major merge from bsd.lv after the switch<BR>
2010 Apr 08: first mandoc bugfix found in ports propagates upstream<BR>
2010 Aug 12: OpenBSD 4.8 release rolled with mandoc<BR>
<p><a href="http://www.flickr.com/photos/tiny_packages/5045038219/"><img
src="http://farm5.static.flickr.com/4084/5045038219_9f577115dd.jpg"></a></p>
<p>Photo: tiny_package @flickr (CC)</p>
<HR>
<P>Ingo Schwarze: Mandoc in OpenBSD - page 12: DRAW II - Apr and May 2010 -
BSDCan 2011, May 13, Ottawa</P>
<H1>If ifs & ands were docs & mans...</H1>
<H2>No way around some low-level roff requests.</H2>
<UL>
<LI>Original design:
<UL>
<LI>Forget about the complete bottom layer.
<LI>Only implement high-level mdoc(7) & man(7) macros.
</UL>
<LI>Didn't work out.
<UL>
<LI>Real-world manuals use some low-level requests.
<LI>Realizing that was a slow, painful process.
<LI>We reluctantly edged in some low-level roff support.
<LI>Complexity was kept to the bare minimum.
</UL>
<LI>Both steps have proven to be just right:
<UL>
<LI>Starting from the high level gave a clean design.
<LI>Roff reluctance prevented getting off track.
<LI>Surprise: Rather little impact in the end.
</UL>
</UL>
<H4>Related timeline:</H4>
2010 Apr 25: implement roff conditional instructions, man(7) only<BR>
2010 May 15: mandoc roff library started by kristaps@<BR>
2010 May 19: the roff library replaces my preliminary support for conditionals<BR>
2010 Jul 03: rudimentary implementation of user-defined strings<BR>
2010 Sep 22: interesting commit message: no hope for .de<BR>
2010 Nov 25: implement the .de (define macro) roff instruction<BR>
<HR>
<P>Ingo Schwarze: Mandoc in OpenBSD - page 13: DRAW III - May 2010 -
BSDCan 2011, May 13, Ottawa</P>
<H1>Desperation lead to success:</H1>
<H2>How an afterthought improved the design.</H2>
<H3>Original mandoc main program:</H3>
<UL>
<LI>main loop to read input files
<LI>push line after line into the parser backends
<LI>parsers look for high-level macros, e.g. mdoc(7)
<LI>call the formatting frontend on the resulting syntax tree
</UL>
<H3>Modified mandoc main program:</H3>
<UL>
<LI>main loop to read input files
<LI><STRONG>call the roff preprocessor on each line</STRONG>
<LI>push line after line into the parser backends
<LI>parsers look for high-level macros, e.g. mdoc(7)
<LI>call the formatting frontend on the resulting syntax tree
</UL>
<H3>Paradigmatic switch:</H3>
<H4>roff: expand high-level macros into low-level requests</H4>
<UL>
<LI>then pass the low-level requests into the formatters
<LI>all structural info lost long before the main parser
</UL>
<H4>mandoc: first handle low-level requests (preprocessor)</H4>
<UL>
<LI>then pass the high-level code to the parsers
<LI>structural info kept even when intermixed with low-level roff
</UL>
<P>→ See Kristaps' talk for applications</P>
<HR>
<P>Ingo Schwarze: Mandoc in OpenBSD - page 14: DRAW IV - May 2010 -
BSDCan 2011, May 13, Ottawa</P>
<H1>.if .ds .de</H1>
<H2>Examples of roff macros we need.</H2>
<UL>
<LI>ignore:
<UL>
<LI>adjustment: .ad .ta
<LI>spacing: .ne .ns
<LI>hyphenation: .hy .nh
<LI>character formatting: .ps .tr
</UL>
<LI>error out:
<UL>
<LI>input line traps: .it
</UL>
<LI>partial implementation:
<UL>
<LI>conditionals: .if .ie .el
<LI>register definitions: .nr
</UL>
<LI>complete implementation:
<UL>
<LI>string definitions: .ds
<LI>macro definitions: .de .rm
<LI>source file inclusion: .so
</UL>
</UL>
<p><a href="http://www.flickr.com/photos/tambako/3578468294/"><img
src="http://farm4.static.flickr.com/3385/3578468294_a22342f725.jpg"></a></p>
<p>Very young zebra. - Photo: Tambako @flickr (CC)</p>
<HR>
<P>Ingo Schwarze: Mandoc in OpenBSD - page 15: DRAW V - Jun and Jul 2010 -
BSDCan 2011, May 13, Ottawa</P>
<H1>Invented here.<BR>
<H2>Home-grown non-standard features even in OpenBSD.</H2>
<H3>Almost all related to SYNOPSIS formatting:</H3>
<UL>
<LI>Roff register to control SYNOPSIS mode.
<LI>Nm indentation in SYNOPSIS mode.
<LI>Word keeps in SYNOPSIS mode.
</UL>
<P>Quite some work to get this right,
for features of no theoretical importance
that are not even documented.</P>
<P>Similar stuff would probably come up
in other systems, when switching them over.</P>
<H4>Related timeline:</H4>
2010 Jun 25 schwarze@ at the c2k10 hackathon in Edmonton<BR>
2010 Jun 26 basic implementation of .Bk/.Ek<BR>
2010 Jun 27 full .nr nS support, unbreaking the kernel manuals<BR>
2010 Jul 01 improve .Nm indentation in the SYNOPSIS<BR>
<HR>
<P>Ingo Schwarze: Mandoc in OpenBSD - page 16: DRAW VI - Summer 2010 -
BSDCan 2011, May 13, Ottawa</P>
<H1>Count your beans.</H1>
<H2>Automatic comparisons as part of the build system.</H2>
<H3>Iterative process, several cycles:</H3>
<UL>
<LI>Run comparisons on a small part of the tree.
<LI>Identify parsing and formatting issues.
<LI>Prioritize by frequency and severity.
<LI>Fix sufficiently few that you don't lose too much time.
<LI>Fix sufficiently many to significantly reduce the noise.
<LI>Start over with a larger part of the tree.
</UL>
<H3>In parallel, tweak groff to reduce noise in comparisons:</H3>
<UL>
<LI>no adjustment
<LI>no hyphenation
<LI>OpenBSD headers
<LI>no special handling of Pa in FILES
</UL>
<H3>A text file in CVS was the ideal bug tracker:</H3>
<UL>
<LI>Keep the number of known issues low.
<LI>Don't let the complexity of the backlog outgrow the complexity of the code.
</UL>
<H4>Related timeline:</H4>
2010 Aug 15 systematic bug hunting in /bin and /sbin<BR>
2011 Jan 23 systematic bug hunting in /usr/bin<BR>
<HR>
<P>Ingo Schwarze: Mandoc in OpenBSD - page 17: DIGRESSIONS -
BSDCan 2011, May 13, Ottawa</P>
<H1>All work and no play...</H1>
<H2>Nice features along the way</H2>
<H3>Critical points:</H3>
<UL>
<LI>Reliable parsing.
<LI>Correct ASCII rendering.
</UL>
<H3>However, fancy output is fun.</H3>
<UL>
<LI>We took the time now and again.
<LI>Almost all projects went well: e.g. HTML, XHTML, PostScript, PDF
<LI>One was overdone and a dead end: hyphenation
</UL>
<H4>Related timeline:</H4>
2009 Oct 21 HTML and XHTML output modes by kristaps@<BR>
2009 Oct 24 hyphenation patch by schwarze@ (still unused)<BR>
2010 Mar 01 proper inter-sentence spacing for mdoc(7)<BR>
2010 Mar 05 at the end of lines, split words at existing hyphens<BR>
2010 Apr 22 proper tab handling<BR>
2010 Jun 10 PostScript output mode started by kristaps@<BR>
2010 Jul 31 PDF output mode completed by kristaps@<BR>
2010 Sep 09 use mandoc instead of groff to build PostScript manuals<BR>
<p><a href="http://www.flickr.com/photos/gazzat/3495392530/"><img
src="http://farm4.static.flickr.com/3317/3495392530_2004bef6c7.jpg"></a></p>
<p>Foals Just Wanna Have Fun. -
Photo: Gary Tanner <gazzat@flickr> (CC)</p>
<HR>
<P>Ingo Schwarze: Mandoc in OpenBSD - page 18: REMOVE I - early Oct 2010 -
BSDCan 2011, May 13, Ottawa</P>
<H1>Only a dozen pages...</H1>
<H2>... are using tbl, the last obstacle to groff removal.</H2>
<UL>
<LI>We long shunned the task:
<UL>
<LI>Even though basic code was long available,
<LI>these were very few pages,
<LI>and it was unclear how to do it best;
<LI>we nearly waited too long for the release.
</UL>
<LI>Finally, i deliberately chose the minimal route:
<UL>
<LI>hook the code directly into the high-level parsers
<LI>working integration within a single weekend
<LI>immediately rely on it for production
</UL>
<LI>Kristaps implemented a better way later:
<UL>
<LI>parse tbl block macros on the roff level
<LI>call the parser for the content from the main program
just like in the case of mdoc(7) and man(7)
</UL>
</UL>
<H4>Related timeline:</H4>
2010 May: stand-alone implementation of tbl started by kristaps@<BR>
2010 Aug 12: OpenBSD 4.8 release rolled with mandoc<BR>
2010 Oct 15: import tbl parser and renderer written by kristaps@<BR>
2010 Oct 17: build tbl(1) pages with mandoc(1), not groff<BR>
2010 Oct 18: disconnect groff from the base build<BR>
2010 Oct 18: "I absolutely don't intend to merge tbl into mandoc" kristaps@<BR>
2011 Jan 04: clean tbl integration by kristaps, remove mine<BR>
<HR>
<P>Ingo Schwarze: Mandoc in OpenBSD - page 19: REMOVE II - late Oct 2010 -
BSDCan 2011, May 13, Ottawa</P>
<H1>Not scared by one-way streets!</H1>
<H2>Groff emigrating to ports land.</H2>
<UL>
<LI>Groff and ports tree already prepared by espie@.
<UL>
<LI>Use a groff build dependency and install formatted manuals.
</UL>
<LI>X11 manuals got briefly broken:
<UL>
<LI>Support for file inclusion (.so) arrived one week late.
</UL>
<LI>Set up instructions for checking ports.
<LI>Start moving over ports to mandoc case by case.
<UL>
<LI>Installing source manuals.
</UL>
<LI>Bug reports from porters start flowing in.
<UL>
<LI>Again, give priority to fixing real-world issues.
</UL>
</UL>
<H4>Related timeline:</H4>
2010 Aug 12: OpenBSD 4.8 release rolled with mandoc<BR>
2010 Oct 18: disconnect groff from the base build<BR>
2010 Oct 19: switch default /etc/man.conf to mandoc<BR>
2010 Oct 23: schwarze@ at p2k10 hackathon in Budapest<BR>
2010 Oct 26: support .so (low-level roff "switch source file")<BR>
2010 Oct 27: OpenBSD ports FAQ section about mandoc and groff<BR>
2010 Oct 29: landry@ performs the first major USE_GROFF removal<BR>
2010 Oct 29: millert@ removes colcrt(1), checknr(1), soelim(1)<BR>
2011 Feb 07: use mandoc in Xenocara Imake builds<BR>
2011 Mar 02: OpenBSD 4.9 release rolled without groff<BR>
2011 Mar 12: cvs rm groff<BR>
<p><a href="http://www.flickr.com/photos/garycycles7/5658207855/"><img
src="http://farm6.static.flickr.com/5144/5658207855_416d19a4b5.jpg"></a></p>
<p>Cart. - Photo: garycycles7 @flickr (CC)</p>
<HR>
<P>Ingo Schwarze: Mandoc in OpenBSD - page 20: CONCLUSIONS I -
BSDCan 2011, May 13, Ottawa</P>
<H1>Done!</H1>
<H2>Reached the stage: It just works.</H2>
<H3>To get there:</H3>
<UL>
<LI>Re-implemented a small family of languages.
<LI>Turned the whole paradigm upside down.
<LI>Remained compatible where it matters.
<LI>Flouted compatibility that would just hinder.
</UL>
<H3>Other systems can now do the same, if they want.</H3>
<UL>
<LI>... without much risk.
<LI>... without having to fear major surprises.
<LI>... getting support from us in case of need.
</UL>
<H3>Just mail us!</H3>
<HR>
<P>Ingo Schwarze: Mandoc in OpenBSD - page 21: RECURRING I -
BSDCan 2011, May 13, Ottawa</P>
<H1>Show what's relevant, not more.</H1>
<H2>Why it is critical to get errors and warnings right.</H2>
<UL>
<LI>An error message is missing or too generic:
<UL>
<LI>Users have a hard time to fix their errors.
</UL>
<LI>A warning message is missing:
<UL>
<LI>Users don't even notice their dangerous idioms.
</UL>
<LI>A few warnings too many, or too prominent:
<UL>
<LI>Users get annoyed and switch off all warnings.
</UL>
<LI>More than one or two knobs:
<UL>
<LI>Users don't remember and don't use them.
</UL>
<LI>Too few and too many can happen all at once!
<UL>
<LI>(It did with mandoc, and it had too many knobs - at first.)
</UL>
</UL>
<H4>Related timeline:</H4>
2009 Jul 12: fewer knobs: remove -Wsyntax -Wcompat<BR>
2010 May 13: fewer knobs: remove -fno-ign-chars<BR>
2010 May 23: unified error and warning system by kristaps@<BR>
2010 Aug 19: simple, consistent user interface for error handling<BR>
2010 Oct 24: do not throw fatal errors when there is no need to<BR>
2010 Oct 26: downgrade nearly 20 errors to warnings<BR>
2011 Jan 16: downgrade yet another bunch of fatal errors<BR>
2011 Jan 22: check argument count validation for all in_line() macros<BR>
<HR>
<P>Ingo Schwarze: Mandoc in OpenBSD - page 22: RECURRING II -
BSDCan 2011, May 13, Ottawa</P>
<H1>Bogue déjà vue:</H1>
<H2>Collecting regression tests.</H2>
<UL>
<LI>Slow start in 2009:
<UL>
<LI>Many new bugs, few regressions.
<LI>Automatic tests not a priority.
<LI>However, start collecting anyway!
</UL>
<LI>The code matures...
<UL>
<LI>More tests show up naturally.
<LI>They slowly become more important.
<LI>Periodic sorting and checking in.
</UL>
<LI>Current state:
<UL>
<LI>When doing major merges,
<LI>the suite regularly catches issues.
<LI>The continuous effort starts to pay off.
</UL>
</UL>
<H4>Related timeline:</H4>
2009 Oct 27: start src/regress/usr.bin/mandoc<BR>
2010 Jun 30: major update of the mandoc test suite<BR>
2010 Jul 01: enable mandoc regression tests; ok phessler@<BR>
2010 Dec 04: major additions to the regression suite<BR>
2011 Feb 05: commit many regression tests found in my trees<BR>
<p><a href="http://www.flickr.com/photos/jimmy-jay/4672901414/"><img
src="http://farm5.static.flickr.com/4021/4672901414_d738b718fe.jpg"></a></p>
<p>Horse Fly Portrait. - Photo: Jeff Burcher <VonShawn@flickr> (CC)</p>
<HR>
<P>Ingo Schwarze: Mandoc in OpenBSD - page 23: RECURRING III -
BSDCan 2011, May 13, Ottawa</P>
<H1>Bad patches triggering good ones</H1>
<H2>Preliminary code put in and ripped out again.</H2>
<P>That may seem inefficient,
but actually it's a perfectly sane approach:</P>
<UL>
<LI>The first implementation explores the feature.
<LI>The second implementation gets it right.
<LI>Just don't let the first one sprawl until it can't be ripped out any more.
</UL>
<H4>Related timeline:</H4>
<P>This approach got used in at least five cases:</P>
2010 Mar 01 - May 14: end of sentence detection<BR>
2010 Apr 25 - May 19: roff conditionals<BR>
2010 Mar 01 - Sep 20: pod2man preamble<BR>
2010 Jun 16 - Jun 27: roff registers<BR>
2010 Oct 15 - Jan 04: tbl integration<BR>
<HR>
<P>Ingo Schwarze: Mandoc in OpenBSD - page 24: CONCLUSIONS II -
BSDCan 2011, May 13, Ottawa</P>
<H1>Newly designed.</H1>
<H2>Clean implementation of dirty languages.</H2>
<UL>
<LI>Started coding with the nice high-level stuff.
<UL>
<LI>That gave us a very clean overall design.
</UL>
<LI>Edged in low-level ugliness later, where required.
<UL>
<LI>Even while the tool was already in production!
</UL>
<LI>Only possible since we kept it small and simple.
<UL>
<LI>Otherwise, such stunts would break the design.
</UL>
</UL>
<H3>Shun complexity!</H3>
<P>This is a third reason why simplicity is key:<BR>
1. correctness and 2. security are well known.</P>
<H4>But flexibility is important as well!</H4>
<HR>
<P>Ingo Schwarze: Mandoc in OpenBSD - page 25: CONCLUSIONS III -
BSDCan 2011, May 13, Ottawa</P>
<H1>Move fast!</H1>
<H2>How a replacement project can succeed.</H2>
<UL>
<LI>Quickly put your work to real-world use.
<UL>
<LI>Do not let it rot in a corner of the OS.
</UL>
<LI>Keep moving fast, do not fear change.
<UL>
<LI>But don't trap your users with incompatible changes.
</UL>
</UL>
<H3>And keep the balance:</H3>
<UL>
<LI>No need to do it when it makes no difference.
<LI>You won't find users when you break behaviour.
<LI>You won't find bugs when you don't have users.
</UL>
<HR>
<P>Ingo Schwarze: Mandoc in OpenBSD - page 26: FUTURE I - Mar and Apr 2011 -
BSDCan 2011, May 13, Ottawa</P>
<H1>The future is already past.</H1>
<H2>When i proposed this talk, groff-1.20 was a plan.</H2>
<H3>We now provide a groff-1.21 port.</H3>
<UL>
<LI>Advantage of the move to ports even for groff lovers:
<LI>It is much easier to update ports than base.
<LI>And the GPL v3 doesn't hurt so much in the ports tree.
<LI>NetBSD and FreeBSD are stuck with groff-1.19.2 which was released in 2005.
</UL>
<H3>According to the ChangeLog, groff-1.21 has:</H3>
<UL>
<LI>Unicode, wide characters, Japanese localization.
<LI>XHTML output (crappy, of course, but at least something).
<LI>Many tbl(1) improvements, including MathML output.
<LI>Better performance, bugfixes, features, and much more...
</UL>
<H4>Related timeline:</H4>
2011 Mar 19: update ports groff from 1.15 to 1.21<BR>
2011 Mar 20: rudimentary eqn support by kristaps@<BR>
2011 Apr 24: tweak mandoc to conform to newest groff habits<BR>
2011 Apr 26: fixed groff-1.21 invocation for Imake ports<BR>
<HR>
<P>Ingo Schwarze: Mandoc in OpenBSD - page 27: FUTURE II -
BSDCan 2011, May 13, Ottawa</P>
<H1>Not done...</H1>
<H2>Possible future directions.</H2>
<UL>
<LI>Infrastructure:
<UL>
<LI>Install manual sources, not preformatted manuals.
<LI>Rewrite apropos(1) and makewhatis(8) to use mandoc.
<LI>Use that to replace man.cgi on the websites.
</UL>
<LI>Functionality:
<UL>
<LI>Write pod2mdoc(1) in Perl.
<LI>Implement -mdoc -Tman:
<UL>
<LI>millert@ wants to convert sudo(1) manuals to mdoc(7)
<LI>but some systems neither want mdoc(7) nor mandoc(1)
</UL>
<LI>Implement -man -Tmdoc:
<UL>
<LI>to help with migration to modern tools.
<LI>much more difficult to implement this direction.
</UL>
</UL>
<LI>And of course:
<UL>
<LI>bugfixing, missing functionality, documentation, ...
</UL>
</UL>
<H4>Related timeline:</H4>
2011 Jul 02: p2k11 - worldwide main hackathon in Edmonton<BR>
2011 Sep 16: s2k11 - European general hackathon in Ljubljana<BR>
<HR>
<P>Ingo Schwarze: Mandoc in OpenBSD - page 28 -
BSDCan 2011, May 13, Ottawa</P>
<H1>Thanks!</H1>
<UL>
<LI>Kristaps@ Dzonsons (bsd.lv)
<UL>
<LI>for writing mandoc.
</UL>
<LI>Joerg@ Sonnenberger (NetBSD)
<UL>
<LI>for important code contributions.
<LI>for hosting an excellent mandoc hackathon at BEC.de.
<LI>for lots of testing in the NetBSD tree.
</UL>
<LI>Jason McIntyre (jmc@)
<UL>
<LI>for excellently and tirelessly maintaining our manuals.
<LI>for helping with countless bug reports.
<LI>for discussing countless questions regarding mdoc(7).
</UL>
<LI>Theo de Raadt (deraadt@)
<UL>
<LI>for inviting Kristaps and getting mandoc imported.
(Otherwise, i might have missed the project.)
<LI>for ongoing encouragement, in particular
to make OpenBSD developers and users our guinea pigs.
(None complained, they seemed to enjoy it.)
</UL>
<LI>Marc Espie (espie@)
<UL>
<LI>for getting ports integration done.
<LI>for quite some help in various respects.
</UL>
<LI>Ulrich Spoerlein (uqs@ FreeBSD)
<UL>
<LI>for lots of bug reports.
<LI>for a couple of patches and discussions.
<LI>for lots of testing in the FreeBSD tree.
</UL>
<LI>Matthieu@ Herrb, Todd@ Fries, Miod@ Vallat
<UL>
<LI>for help with Xenocara integration.
</UL>
<LI>Peter Hessler (phessler@)
<UL>
<LI>for help with the regression infrastructure.
<LI>for portability testing.
</UL>
<LI>Igor Sobrado (sobrado@)
<UL>
<LI>for lots of bug reports and discussions.
</UL>
<LI>Christian Weisgerber (naddy@)
<UL>
<LI>for very thorough testing.
</UL>
</UL>
<H4>For important bug reports and discussions:</H4>
<UL>
<LI>Stuart Henderson (sthen@)
<LI>Landry Breuil (landry@)
<LI>Todd Miller (millert@)
<LI>Brad@ Smith (comstyle.com)
<LI>Will Backman
<LI>Thomas Klausner (wiz@ NetBSD)
<LI>Sascha Wildner (DragonFly BSD)
<LI>Werner Lemberg (GNU)
</UL>
<P>David Coppa - Edd Barrett - Giovanni Bechis - Gleydson Soares -
Ian Darwin - Janne Johansson - Jasper Lievisse Adriaanse - Kurt Miller -
Matthew Dempsky - Matthias Kilian - Nicholas Marriott - Paul Irofti -
Philipp Guenther - Remi Pointel - Ted Unangst - Thordur Bjornsson
(all OpenBSD)</P>
<P>Anthony J. Bentley - Chris Bennett - Frantisek Holop - Igor Zinovik -
James Jerkins - Maxim Belooussov - Mikolaj Kucharski - Nicolas Joly -
Pascal Stumpf - Ryan Flannery - Tim van der Molen - Tristan Le Guern -
Yuri Pankov</P>
<P>... and probably some i have forgotten (sorry for that)</P>
<H4>For images under CC licenses:</H4>
<A HREF=http://www.flickr.com/photos/tomkoadam/4778126822/>http://www.flickr.com/photos/tomkoadam/4778126822/</A><BR>
<A HREF=http://www.flickr.com/photos/kristavandervoorden/4737488285/>http://www.flickr.com/photos/kristavandervoorden/4737488285/</A><BR>
<A HREF=http://www.flickr.com/photos/tiny_packages/5045038219/>http://www.flickr.com/photos/tiny_packages/5045038219/</A><BR>
<A HREF=http://www.flickr.com/photos/tambako/3578468294/>http://www.flickr.com/photos/tambako/3578468294/</A><BR>
<A HREF=http://www.flickr.com/photos/gazzat/3495392530/>http://www.flickr.com/photos/gazzat/3495392530/</A><BR>
<A HREF=http://www.flickr.com/photos/garycycles7/5658207855/>http://www.flickr.com/photos/garycycles7/5658207855/</A><BR>
<A HREF=http://www.flickr.com/photos/jimmy-jay/4672901414/>http://www.flickr.com/photos/jimmy-jay/4672901414/</A><BR>
<A HREF=http://www.flickr.com/photos/66176388@N00/3436935367/>http://www.flickr.com/photos/66176388@N00/3436935367/</A><BR>
<p><a href="http://www.flickr.com/photos/66176388@N00/3436935367/"><img
src="http://farm4.static.flickr.com/3312/3436935367_f4ee02b7e0.jpg"></a></p>
<p>A Youngster on the Quantocks. -
Photo: Mark Robinson <me'nthedogs'@flickr> (CC)</p>
</BODY>
</HTML>
![[BACK]](/icons/back.gif){kind=icon}
Return to bsdcan11-mandoc-openbsd.html CVS log ![[TXT]](/icons/text.gif){kind=icon}
![[DIR]](/icons/dir.gif){kind=icon}
Up to [local] / www / papers