diff options
Diffstat (limited to 'plugins/Updater/bzip2-1.0.3/manual.xml')
| -rw-r--r-- | plugins/Updater/bzip2-1.0.3/manual.xml | 2964 |
1 files changed, 2964 insertions, 0 deletions
diff --git a/plugins/Updater/bzip2-1.0.3/manual.xml b/plugins/Updater/bzip2-1.0.3/manual.xml new file mode 100644 index 0000000000..a7fbcb3375 --- /dev/null +++ b/plugins/Updater/bzip2-1.0.3/manual.xml @@ -0,0 +1,2964 @@ +<?xml version="1.0"?> <!-- -*- sgml -*- --> +<!DOCTYPE book PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN" + "http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd"[ + +<!-- various strings, dates etc. common to all docs --> +<!ENTITY % common-ents SYSTEM "entities.xml"> %common-ents; +]> + +<book lang="en" id="userman" xreflabel="bzip2 Manual"> + + <bookinfo> + <title>bzip2 and libbzip2, version 1.0.6</title> + <subtitle>A program and library for data compression</subtitle> + <copyright> + <year>&bz-lifespan;</year> + <holder>Julian Seward</holder> + </copyright> + <releaseinfo>Version &bz-version; of &bz-date;</releaseinfo> + + <authorgroup> + <author> + <firstname>Julian</firstname> + <surname>Seward</surname> + <affiliation> + <orgname>&bz-url;</orgname> + </affiliation> + </author> + </authorgroup> + + <legalnotice> + + <para>This program, <computeroutput>bzip2</computeroutput>, the + associated library <computeroutput>libbzip2</computeroutput>, and + all documentation, are copyright © &bz-lifespan; Julian Seward. + All rights reserved.</para> + + <para>Redistribution and use in source and binary forms, with + or without modification, are permitted provided that the + following conditions are met:</para> + + <itemizedlist mark='bullet'> + + <listitem><para>Redistributions of source code must retain the + above copyright notice, this list of conditions and the + following disclaimer.</para></listitem> + + <listitem><para>The origin of this software must not be + misrepresented; you must not claim that you wrote the original + software. If you use this software in a product, an + acknowledgment in the product documentation would be + appreciated but is not required.</para></listitem> + + <listitem><para>Altered source versions must be plainly marked + as such, and must not be misrepresented as being the original + software.</para></listitem> + + <listitem><para>The name of the author may not be used to + endorse or promote products derived from this software without + specific prior written permission.</para></listitem> + + </itemizedlist> + + <para>THIS SOFTWARE IS PROVIDED BY THE AUTHOR "AS IS" AND ANY + EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, + THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A + PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE + AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, + EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED + TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND + ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING + IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF + THE POSSIBILITY OF SUCH DAMAGE.</para> + + <para>PATENTS: To the best of my knowledge, + <computeroutput>bzip2</computeroutput> and + <computeroutput>libbzip2</computeroutput> do not use any patented + algorithms. However, I do not have the resources to carry + out a patent search. Therefore I cannot give any guarantee of + the above statement. + </para> + +</legalnotice> + +</bookinfo> + + + +<chapter id="intro" xreflabel="Introduction"> +<title>Introduction</title> + +<para><computeroutput>bzip2</computeroutput> compresses files +using the Burrows-Wheeler block-sorting text compression +algorithm, and Huffman coding. Compression is generally +considerably better than that achieved by more conventional +LZ77/LZ78-based compressors, and approaches the performance of +the PPM family of statistical compressors.</para> + +<para><computeroutput>bzip2</computeroutput> is built on top of +<computeroutput>libbzip2</computeroutput>, a flexible library for +handling compressed data in the +<computeroutput>bzip2</computeroutput> format. This manual +describes both how to use the program and how to work with the +library interface. Most of the manual is devoted to this +library, not the program, which is good news if your interest is +only in the program.</para> + +<itemizedlist mark='bullet'> + + <listitem><para><xref linkend="using"/> describes how to use + <computeroutput>bzip2</computeroutput>; this is the only part + you need to read if you just want to know how to operate the + program.</para></listitem> + + <listitem><para><xref linkend="libprog"/> describes the + programming interfaces in detail, and</para></listitem> + + <listitem><para><xref linkend="misc"/> records some + miscellaneous notes which I thought ought to be recorded + somewhere.</para></listitem> + +</itemizedlist> + +</chapter> + + +<chapter id="using" xreflabel="How to use bzip2"> +<title>How to use bzip2</title> + +<para>This chapter contains a copy of the +<computeroutput>bzip2</computeroutput> man page, and nothing +else.</para> + +<sect1 id="name" xreflabel="NAME"> +<title>NAME</title> + +<itemizedlist mark='bullet'> + + <listitem><para><computeroutput>bzip2</computeroutput>, + <computeroutput>bunzip2</computeroutput> - a block-sorting file + compressor, v1.0.6</para></listitem> + + <listitem><para><computeroutput>bzcat</computeroutput> - + decompresses files to stdout</para></listitem> + + <listitem><para><computeroutput>bzip2recover</computeroutput> - + recovers data from damaged bzip2 files</para></listitem> + +</itemizedlist> + +</sect1> + + +<sect1 id="synopsis" xreflabel="SYNOPSIS"> +<title>SYNOPSIS</title> + +<itemizedlist mark='bullet'> + + <listitem><para><computeroutput>bzip2</computeroutput> [ + -cdfkqstvzVL123456789 ] [ filenames ... ]</para></listitem> + + <listitem><para><computeroutput>bunzip2</computeroutput> [ + -fkvsVL ] [ filenames ... ]</para></listitem> + + <listitem><para><computeroutput>bzcat</computeroutput> [ -s ] [ + filenames ... ]</para></listitem> + + <listitem><para><computeroutput>bzip2recover</computeroutput> + filename</para></listitem> + +</itemizedlist> + +</sect1> + + +<sect1 id="description" xreflabel="DESCRIPTION"> +<title>DESCRIPTION</title> + +<para><computeroutput>bzip2</computeroutput> compresses files +using the Burrows-Wheeler block sorting text compression +algorithm, and Huffman coding. Compression is generally +considerably better than that achieved by more conventional +LZ77/LZ78-based compressors, and approaches the performance of +the PPM family of statistical compressors.</para> + +<para>The command-line options are deliberately very similar to +those of GNU <computeroutput>gzip</computeroutput>, but they are +not identical.</para> + +<para><computeroutput>bzip2</computeroutput> expects a list of +file names to accompany the command-line flags. Each file is +replaced by a compressed version of itself, with the name +<computeroutput>original_name.bz2</computeroutput>. Each +compressed file has the same modification date, permissions, and, +when possible, ownership as the corresponding original, so that +these properties can be correctly restored at decompression time. +File name handling is naive in the sense that there is no +mechanism for preserving original file names, permissions, +ownerships or dates in filesystems which lack these concepts, or +have serious file name length restrictions, such as +MS-DOS.</para> + +<para><computeroutput>bzip2</computeroutput> and +<computeroutput>bunzip2</computeroutput> will by default not +overwrite existing files. If you want this to happen, specify +the <computeroutput>-f</computeroutput> flag.</para> + +<para>If no file names are specified, +<computeroutput>bzip2</computeroutput> compresses from standard +input to standard output. In this case, +<computeroutput>bzip2</computeroutput> will decline to write +compressed output to a terminal, as this would be entirely +incomprehensible and therefore pointless.</para> + +<para><computeroutput>bunzip2</computeroutput> (or +<computeroutput>bzip2 -d</computeroutput>) decompresses all +specified files. Files which were not created by +<computeroutput>bzip2</computeroutput> will be detected and +ignored, and a warning issued. +<computeroutput>bzip2</computeroutput> attempts to guess the +filename for the decompressed file from that of the compressed +file as follows:</para> + +<itemizedlist mark='bullet'> + + <listitem><para><computeroutput>filename.bz2 </computeroutput> + becomes + <computeroutput>filename</computeroutput></para></listitem> + + <listitem><para><computeroutput>filename.bz </computeroutput> + becomes + <computeroutput>filename</computeroutput></para></listitem> + + <listitem><para><computeroutput>filename.tbz2</computeroutput> + becomes + <computeroutput>filename.tar</computeroutput></para></listitem> + + <listitem><para><computeroutput>filename.tbz </computeroutput> + becomes + <computeroutput>filename.tar</computeroutput></para></listitem> + + <listitem><para><computeroutput>anyothername </computeroutput> + becomes + <computeroutput>anyothername.out</computeroutput></para></listitem> + +</itemizedlist> + +<para>If the file does not end in one of the recognised endings, +<computeroutput>.bz2</computeroutput>, +<computeroutput>.bz</computeroutput>, +<computeroutput>.tbz2</computeroutput> or +<computeroutput>.tbz</computeroutput>, +<computeroutput>bzip2</computeroutput> complains that it cannot +guess the name of the original file, and uses the original name +with <computeroutput>.out</computeroutput> appended.</para> + +<para>As with compression, supplying no filenames causes +decompression from standard input to standard output.</para> + +<para><computeroutput>bunzip2</computeroutput> will correctly +decompress a file which is the concatenation of two or more +compressed files. The result is the concatenation of the +corresponding uncompressed files. Integrity testing +(<computeroutput>-t</computeroutput>) of concatenated compressed +files is also supported.</para> + +<para>You can also compress or decompress files to the standard +output by giving the <computeroutput>-c</computeroutput> flag. +Multiple files may be compressed and decompressed like this. The +resulting outputs are fed sequentially to stdout. Compression of +multiple files in this manner generates a stream containing +multiple compressed file representations. Such a stream can be +decompressed correctly only by +<computeroutput>bzip2</computeroutput> version 0.9.0 or later. +Earlier versions of <computeroutput>bzip2</computeroutput> will +stop after decompressing the first file in the stream.</para> + +<para><computeroutput>bzcat</computeroutput> (or +<computeroutput>bzip2 -dc</computeroutput>) decompresses all +specified files to the standard output.</para> + +<para><computeroutput>bzip2</computeroutput> will read arguments +from the environment variables +<computeroutput>BZIP2</computeroutput> and +<computeroutput>BZIP</computeroutput>, in that order, and will +process them before any arguments read from the command line. +This gives a convenient way to supply default arguments.</para> + +<para>Compression is always performed, even if the compressed +file is slightly larger than the original. Files of less than +about one hundred bytes tend to get larger, since the compression +mechanism has a constant overhead in the region of 50 bytes. +Random data (including the output of most file compressors) is +coded at about 8.05 bits per byte, giving an expansion of around +0.5%.</para> + +<para>As a self-check for your protection, +<computeroutput>bzip2</computeroutput> uses 32-bit CRCs to make +sure that the decompressed version of a file is identical to the +original. This guards against corruption of the compressed data, +and against undetected bugs in +<computeroutput>bzip2</computeroutput> (hopefully very unlikely). +The chances of data corruption going undetected is microscopic, +about one chance in four billion for each file processed. Be +aware, though, that the check occurs upon decompression, so it +can only tell you that something is wrong. It can't help you +recover the original uncompressed data. You can use +<computeroutput>bzip2recover</computeroutput> to try to recover +data from damaged files.</para> + +<para>Return values: 0 for a normal exit, 1 for environmental +problems (file not found, invalid flags, I/O errors, etc.), 2 +to indicate a corrupt compressed file, 3 for an internal +consistency error (eg, bug) which caused +<computeroutput>bzip2</computeroutput> to panic.</para> + +</sect1> + + +<sect1 id="options" xreflabel="OPTIONS"> +<title>OPTIONS</title> + +<variablelist> + + <varlistentry> + <term><computeroutput>-c --stdout</computeroutput></term> + <listitem><para>Compress or decompress to standard + output.</para></listitem> + </varlistentry> + + <varlistentry> + <term><computeroutput>-d --decompress</computeroutput></term> + <listitem><para>Force decompression. + <computeroutput>bzip2</computeroutput>, + <computeroutput>bunzip2</computeroutput> and + <computeroutput>bzcat</computeroutput> are really the same + program, and the decision about what actions to take is done on + the basis of which name is used. This flag overrides that + mechanism, and forces bzip2 to decompress.</para></listitem> + </varlistentry> + + <varlistentry> + <term><computeroutput>-z --compress</computeroutput></term> + <listitem><para>The complement to + <computeroutput>-d</computeroutput>: forces compression, + regardless of the invokation name.</para></listitem> + </varlistentry> + + <varlistentry> + <term><computeroutput>-t --test</computeroutput></term> + <listitem><para>Check integrity of the specified file(s), but + don't decompress them. This really performs a trial + decompression and throws away the result.</para></listitem> + </varlistentry> + + <varlistentry> + <term><computeroutput>-f --force</computeroutput></term> + <listitem><para>Force overwrite of output files. Normally, + <computeroutput>bzip2</computeroutput> will not overwrite + existing output files. Also forces + <computeroutput>bzip2</computeroutput> to break hard links to + files, which it otherwise wouldn't do.</para> + <para><computeroutput>bzip2</computeroutput> normally declines + to decompress files which don't have the correct magic header + bytes. If forced (<computeroutput>-f</computeroutput>), + however, it will pass such files through unmodified. This is + how GNU <computeroutput>gzip</computeroutput> behaves.</para> + </listitem> + </varlistentry> + + <varlistentry> + <term><computeroutput>-k --keep</computeroutput></term> + <listitem><para>Keep (don't delete) input files during + compression or decompression.</para></listitem> + </varlistentry> + + <varlistentry> + <term><computeroutput>-s --small</computeroutput></term> + <listitem><para>Reduce memory usage, for compression, + decompression and testing. Files are decompressed and tested + using a modified algorithm which only requires 2.5 bytes per + block byte. This means any file can be decompressed in 2300k + of memory, albeit at about half the normal speed.</para> + <para>During compression, <computeroutput>-s</computeroutput> + selects a block size of 200k, which limits memory use to around + the same figure, at the expense of your compression ratio. In + short, if your machine is low on memory (8 megabytes or less), + use <computeroutput>-s</computeroutput> for everything. See + <xref linkend="memory-management"/> below.</para></listitem> + </varlistentry> + + <varlistentry> + <term><computeroutput>-q --quiet</computeroutput></term> + <listitem><para>Suppress non-essential warning messages. + Messages pertaining to I/O errors and other critical events + will not be suppressed.</para></listitem> + </varlistentry> + + <varlistentry> + <term><computeroutput>-v --verbose</computeroutput></term> + <listitem><para>Verbose mode -- show the compression ratio for + each file processed. Further + <computeroutput>-v</computeroutput>'s increase the verbosity + level, spewing out lots of information which is primarily of + interest for diagnostic purposes.</para></listitem> + </varlistentry> + + <varlistentry> + <term><computeroutput>-L --license -V --version</computeroutput></term> + <listitem><para>Display the software version, license terms and + conditions.</para></listitem> + </varlistentry> + + <varlistentry> + <term><computeroutput>-1</computeroutput> (or + <computeroutput>--fast</computeroutput>) to + <computeroutput>-9</computeroutput> (or + <computeroutput>-best</computeroutput>)</term> + <listitem><para>Set the block size to 100 k, 200 k ... 900 k + when compressing. Has no effect when decompressing. See <xref + linkend="memory-management" /> below. The + <computeroutput>--fast</computeroutput> and + <computeroutput>--best</computeroutput> aliases are primarily + for GNU <computeroutput>gzip</computeroutput> compatibility. + In particular, <computeroutput>--fast</computeroutput> doesn't + make things significantly faster. And + <computeroutput>--best</computeroutput> merely selects the + default behaviour.</para></listitem> + </varlistentry> + + <varlistentry> + <term><computeroutput>--</computeroutput></term> + <listitem><para>Treats all subsequent arguments as file names, + even if they start with a dash. This is so you can handle + files with names beginning with a dash, for example: + <computeroutput>bzip2 -- + -myfilename</computeroutput>.</para></listitem> + </varlistentry> + + <varlistentry> + <term><computeroutput>--repetitive-fast</computeroutput></term> + <term><computeroutput>--repetitive-best</computeroutput></term> + <listitem><para>These flags are redundant in versions 0.9.5 and + above. They provided some coarse control over the behaviour of + the sorting algorithm in earlier versions, which was sometimes + useful. 0.9.5 and above have an improved algorithm which + renders these flags irrelevant.</para></listitem> + </varlistentry> + +</variablelist> + +</sect1> + + +<sect1 id="memory-management" xreflabel="MEMORY MANAGEMENT"> +<title>MEMORY MANAGEMENT</title> + +<para><computeroutput>bzip2</computeroutput> compresses large +files in blocks. The block size affects both the compression +ratio achieved, and the amount of memory needed for compression +and decompression. The flags <computeroutput>-1</computeroutput> +through <computeroutput>-9</computeroutput> specify the block +size to be 100,000 bytes through 900,000 bytes (the default) +respectively. At decompression time, the block size used for +compression is read from the header of the compressed file, and +<computeroutput>bunzip2</computeroutput> then allocates itself +just enough memory to decompress the file. Since block sizes are +stored in compressed files, it follows that the flags +<computeroutput>-1</computeroutput> to +<computeroutput>-9</computeroutput> are irrelevant to and so +ignored during decompression.</para> + +<para>Compression and decompression requirements, in bytes, can be +estimated as:</para> +<programlisting> +Compression: 400k + ( 8 x block size ) + +Decompression: 100k + ( 4 x block size ), or + 100k + ( 2.5 x block size ) +</programlisting> + +<para>Larger block sizes give rapidly diminishing marginal +returns. Most of the compression comes from the first two or +three hundred k of block size, a fact worth bearing in mind when +using <computeroutput>bzip2</computeroutput> on small machines. +It is also important to appreciate that the decompression memory +requirement is set at compression time by the choice of block +size.</para> + +<para>For files compressed with the default 900k block size, +<computeroutput>bunzip2</computeroutput> will require about 3700 +kbytes to decompress. To support decompression of any file on a +4 megabyte machine, <computeroutput>bunzip2</computeroutput> has +an option to decompress using approximately half this amount of +memory, about 2300 kbytes. Decompression speed is also halved, +so you should use this option only where necessary. The relevant +flag is <computeroutput>-s</computeroutput>.</para> + +<para>In general, try and use the largest block size memory +constraints allow, since that maximises the compression achieved. +Compression and decompression speed are virtually unaffected by +block size.</para> + +<para>Another significant point applies to files which fit in a +single block -- that means most files you'd encounter using a +large block size. The amount of real memory touched is +proportional to the size of the file, since the file is smaller +than a block. For example, compressing a file 20,000 bytes long +with the flag <computeroutput>-9</computeroutput> will cause the +compressor to allocate around 7600k of memory, but only touch +400k + 20000 * 8 = 560 kbytes of it. Similarly, the decompressor +will allocate 3700k but only touch 100k + 20000 * 4 = 180 +kbytes.</para> + +<para>Here is a table which summarises the maximum memory usage +for different block sizes. Also recorded is the total compressed +size for 14 files of the Calgary Text Compression Corpus +totalling 3,141,622 bytes. This column gives some feel for how +compression varies with block size. These figures tend to +understate the advantage of larger block sizes for larger files, +since the Corpus is dominated by smaller files.</para> + +<programlisting> + Compress Decompress Decompress Corpus +Flag usage usage -s usage Size + + -1 1200k 500k 350k 914704 + -2 2000k 900k 600k 877703 + -3 2800k 1300k 850k 860338 + -4 3600k 1700k 1100k 846899 + -5 4400k 2100k 1350k 845160 + -6 5200k 2500k 1600k 838626 + -7 6100k 2900k 1850k 834096 + -8 6800k 3300k 2100k 828642 + -9 7600k 3700k 2350k 828642 +</programlisting> + +</sect1> + + +<sect1 id="recovering" xreflabel="RECOVERING DATA FROM DAMAGED FILES"> +<title>RECOVERING DATA FROM DAMAGED FILES</title> + +<para><computeroutput>bzip2</computeroutput> compresses files in +blocks, usually 900kbytes long. Each block is handled +independently. If a media or transmission error causes a +multi-block <computeroutput>.bz2</computeroutput> file to become +damaged, it may be possible to recover data from the undamaged +blocks in the file.</para> + +<para>The compressed representation of each block is delimited by +a 48-bit pattern, which makes it possible to find the block +boundaries with reasonable certainty. Each block also carries +its own 32-bit CRC, so damaged blocks can be distinguished from +undamaged ones.</para> + +<para><computeroutput>bzip2recover</computeroutput> is a simple +program whose purpose is to search for blocks in +<computeroutput>.bz2</computeroutput> files, and write each block +out into its own <computeroutput>.bz2</computeroutput> file. You +can then use <computeroutput>bzip2 -t</computeroutput> to test +the integrity of the resulting files, and decompress those which +are undamaged.</para> + +<para><computeroutput>bzip2recover</computeroutput> takes a +single argument, the name of the damaged file, and writes a +number of files <computeroutput>rec0001file.bz2</computeroutput>, +<computeroutput>rec0002file.bz2</computeroutput>, etc, containing +the extracted blocks. The output filenames are designed so that +the use of wildcards in subsequent processing -- for example, +<computeroutput>bzip2 -dc rec*file.bz2 > +recovered_data</computeroutput> -- lists the files in the correct +order.</para> + +<para><computeroutput>bzip2recover</computeroutput> should be of +most use dealing with large <computeroutput>.bz2</computeroutput> +files, as these will contain many blocks. It is clearly futile +to use it on damaged single-block files, since a damaged block +cannot be recovered. If you wish to minimise any potential data +loss through media or transmission errors, you might consider +compressing with a smaller block size.</para> + +</sect1> + + +<sect1 id="performance" xreflabel="PERFORMANCE NOTES"> +<title>PERFORMANCE NOTES</title> + +<para>The sorting phase of compression gathers together similar +strings in the file. Because of this, files containing very long +runs of repeated symbols, like "aabaabaabaab ..." (repeated +several hundred times) may compress more slowly than normal. +Versions 0.9.5 and above fare much better than previous versions +in this respect. The ratio between worst-case and average-case +compression time is in the region of 10:1. For previous +versions, this figure was more like 100:1. You can use the +<computeroutput>-vvvv</computeroutput> option to monitor progress +in great detail, if you want.</para> + +<para>Decompression speed is unaffected by these +phenomena.</para> + +<para><computeroutput>bzip2</computeroutput> usually allocates +several megabytes of memory to operate in, and then charges all +over it in a fairly random fashion. This means that performance, +both for compressing and decompressing, is largely determined by +the speed at which your machine can service cache misses. +Because of this, small changes to the code to reduce the miss +rate have been observed to give disproportionately large +performance improvements. I imagine +<computeroutput>bzip2</computeroutput> will perform best on +machines with very large caches.</para> + +</sect1> + + + +<sect1 id="caveats" xreflabel="CAVEATS"> +<title>CAVEATS</title> + +<para>I/O error messages are not as helpful as they could be. +<computeroutput>bzip2</computeroutput> tries hard to detect I/O +errors and exit cleanly, but the details of what the problem is +sometimes seem rather misleading.</para> + +<para>This manual page pertains to version &bz-version; of +<computeroutput>bzip2</computeroutput>. Compressed data created by +this version is entirely forwards and backwards compatible with the +previous public releases, versions 0.1pl2, 0.9.0 and 0.9.5, 1.0.0, +1.0.1, 1.0.2 and 1.0.3, but with the following exception: 0.9.0 and +above can correctly decompress multiple concatenated compressed files. +0.1pl2 cannot do this; it will stop after decompressing just the first +file in the stream.</para> + +<para><computeroutput>bzip2recover</computeroutput> versions +prior to 1.0.2 used 32-bit integers to represent bit positions in +compressed files, so it could not handle compressed files more +than 512 megabytes long. Versions 1.0.2 and above use 64-bit ints +on some platforms which support them (GNU supported targets, and +Windows). To establish whether or not +<computeroutput>bzip2recover</computeroutput> was built with such +a limitation, run it without arguments. In any event you can +build yourself an unlimited version if you can recompile it with +<computeroutput>MaybeUInt64</computeroutput> set to be an +unsigned 64-bit integer.</para> + +</sect1> + + + +<sect1 id="author" xreflabel="AUTHOR"> +<title>AUTHOR</title> + +<para>Julian Seward, +<computeroutput>&bz-email;</computeroutput></para> + +<para>The ideas embodied in +<computeroutput>bzip2</computeroutput> are due to (at least) the +following people: Michael Burrows and David Wheeler (for the +block sorting transformation), David Wheeler (again, for the +Huffman coder), Peter Fenwick (for the structured coding model in +the original <computeroutput>bzip</computeroutput>, and many +refinements), and Alistair Moffat, Radford Neal and Ian Witten +(for the arithmetic coder in the original +<computeroutput>bzip</computeroutput>). I am much indebted for +their help, support and advice. See the manual in the source +distribution for pointers to sources of documentation. Christian +von Roques encouraged me to look for faster sorting algorithms, +so as to speed up compression. Bela Lubkin encouraged me to +improve the worst-case compression performance. +Donna Robinson XMLised the documentation. +Many people sent +patches, helped with portability problems, lent machines, gave +advice and were generally helpful.</para> + +</sect1> + +</chapter> + + + +<chapter id="libprog" xreflabel="Programming with libbzip2"> +<title> +Programming with <computeroutput>libbzip2</computeroutput> +</title> + +<para>This chapter describes the programming interface to +<computeroutput>libbzip2</computeroutput>.</para> + +<para>For general background information, particularly about +memory use and performance aspects, you'd be well advised to read +<xref linkend="using"/> as well.</para> + + +<sect1 id="top-level" xreflabel="Top-level structure"> +<title>Top-level structure</title> + +<para><computeroutput>libbzip2</computeroutput> is a flexible +library for compressing and decompressing data in the +<computeroutput>bzip2</computeroutput> data format. Although +packaged as a single entity, it helps to regard the library as +three separate parts: the low level interface, and the high level +interface, and some utility functions.</para> + +<para>The structure of +<computeroutput>libbzip2</computeroutput>'s interfaces is similar +to that of Jean-loup Gailly's and Mark Adler's excellent +<computeroutput>zlib</computeroutput> library.</para> + +<para>All externally visible symbols have names beginning +<computeroutput>BZ2_</computeroutput>. This is new in version +1.0. The intention is to minimise pollution of the namespaces of +library clients.</para> + +<para>To use any part of the library, you need to +<computeroutput>#include <bzlib.h></computeroutput> +into your sources.</para> + + + +<sect2 id="ll-summary" xreflabel="Low-level summary"> +<title>Low-level summary</title> + +<para>This interface provides services for compressing and +decompressing data in memory. There's no provision for dealing +with files, streams or any other I/O mechanisms, just straight +memory-to-memory work. In fact, this part of the library can be +compiled without inclusion of +<computeroutput>stdio.h</computeroutput>, which may be helpful +for embedded applications.</para> + +<para>The low-level part of the library has no global variables +and is therefore thread-safe.</para> + +<para>Six routines make up the low level interface: +<computeroutput>BZ2_bzCompressInit</computeroutput>, +<computeroutput>BZ2_bzCompress</computeroutput>, and +<computeroutput>BZ2_bzCompressEnd</computeroutput> for +compression, and a corresponding trio +<computeroutput>BZ2_bzDecompressInit</computeroutput>, +<computeroutput>BZ2_bzDecompress</computeroutput> and +<computeroutput>BZ2_bzDecompressEnd</computeroutput> for +decompression. The <computeroutput>*Init</computeroutput> +functions allocate memory for compression/decompression and do +other initialisations, whilst the +<computeroutput>*End</computeroutput> functions close down +operations and release memory.</para> + +<para>The real work is done by +<computeroutput>BZ2_bzCompress</computeroutput> and +<computeroutput>BZ2_bzDecompress</computeroutput>. These +compress and decompress data from a user-supplied input buffer to +a user-supplied output buffer. These buffers can be any size; +arbitrary quantities of data are handled by making repeated calls +to these functions. This is a flexible mechanism allowing a +consumer-pull style of activity, or producer-push, or a mixture +of both.</para> + +</sect2> + + +<sect2 id="hl-summary" xreflabel="High-level summary"> +<title>High-level summary</title> + +<para>This interface provides some handy wrappers around the +low-level interface to facilitate reading and writing +<computeroutput>bzip2</computeroutput> format files +(<computeroutput>.bz2</computeroutput> files). The routines +provide hooks to facilitate reading files in which the +<computeroutput>bzip2</computeroutput> data stream is embedded +within some larger-scale file structure, or where there are +multiple <computeroutput>bzip2</computeroutput> data streams +concatenated end-to-end.</para> + +<para>For reading files, +<computeroutput>BZ2_bzReadOpen</computeroutput>, +<computeroutput>BZ2_bzRead</computeroutput>, +<computeroutput>BZ2_bzReadClose</computeroutput> and +<computeroutput>BZ2_bzReadGetUnused</computeroutput> are +supplied. For writing files, +<computeroutput>BZ2_bzWriteOpen</computeroutput>, +<computeroutput>BZ2_bzWrite</computeroutput> and +<computeroutput>BZ2_bzWriteFinish</computeroutput> are +available.</para> + +<para>As with the low-level library, no global variables are used +so the library is per se thread-safe. However, if I/O errors +occur whilst reading or writing the underlying compressed files, +you may have to consult <computeroutput>errno</computeroutput> to +determine the cause of the error. In that case, you'd need a C +library which correctly supports +<computeroutput>errno</computeroutput> in a multithreaded +environment.</para> + +<para>To make the library a little simpler and more portable, +<computeroutput>BZ2_bzReadOpen</computeroutput> and +<computeroutput>BZ2_bzWriteOpen</computeroutput> require you to +pass them file handles (<computeroutput>FILE*</computeroutput>s) +which have previously been opened for reading or writing +respectively. That avoids portability problems associated with +file operations and file attributes, whilst not being much of an +imposition on the programmer.</para> + +</sect2> + + +<sect2 id="util-fns-summary" xreflabel="Utility functions summary"> +<title>Utility functions summary</title> + +<para>For very simple needs, +<computeroutput>BZ2_bzBuffToBuffCompress</computeroutput> and +<computeroutput>BZ2_bzBuffToBuffDecompress</computeroutput> are +provided. These compress data in memory from one buffer to +another buffer in a single function call. You should assess +whether these functions fulfill your memory-to-memory +compression/decompression requirements before investing effort in +understanding the more general but more complex low-level +interface.</para> + +<para>Yoshioka Tsuneo +(<computeroutput>tsuneo@rr.iij4u.or.jp</computeroutput>) has +contributed some functions to give better +<computeroutput>zlib</computeroutput> compatibility. These +functions are <computeroutput>BZ2_bzopen</computeroutput>, +<computeroutput>BZ2_bzread</computeroutput>, +<computeroutput>BZ2_bzwrite</computeroutput>, +<computeroutput>BZ2_bzflush</computeroutput>, +<computeroutput>BZ2_bzclose</computeroutput>, +<computeroutput>BZ2_bzerror</computeroutput> and +<computeroutput>BZ2_bzlibVersion</computeroutput>. You may find +these functions more convenient for simple file reading and +writing, than those in the high-level interface. These functions +are not (yet) officially part of the library, and are minimally +documented here. If they break, you get to keep all the pieces. +I hope to document them properly when time permits.</para> + +<para>Yoshioka also contributed modifications to allow the +library to be built as a Windows DLL.</para> + +</sect2> + +</sect1> + + +<sect1 id="err-handling" xreflabel="Error handling"> +<title>Error handling</title> + +<para>The library is designed to recover cleanly in all +situations, including the worst-case situation of decompressing +random data. I'm not 100% sure that it can always do this, so +you might want to add a signal handler to catch segmentation +violations during decompression if you are feeling especially +paranoid. I would be interested in hearing more about the +robustness of the library to corrupted compressed data.</para> + +<para>Version 1.0.3 more robust in this respect than any +previous version. Investigations with Valgrind (a tool for detecting +problems with memory management) indicate +that, at least for the few files I tested, all single-bit errors +in the decompressed data are caught properly, with no +segmentation faults, no uses of uninitialised data, no out of +range reads or writes, and no infinite looping in the decompressor. +So it's certainly pretty robust, although +I wouldn't claim it to be totally bombproof.</para> + +<para>The file <computeroutput>bzlib.h</computeroutput> contains +all definitions needed to use the library. In particular, you +should definitely not include +<computeroutput>bzlib_private.h</computeroutput>.</para> + +<para>In <computeroutput>bzlib.h</computeroutput>, the various +return values are defined. The following list is not intended as +an exhaustive description of the circumstances in which a given +value may be returned -- those descriptions are given later. +Rather, it is intended to convey the rough meaning of each return +value. The first five actions are normal and not intended to +denote an error situation.</para> + +<variablelist> + + <varlistentry> + <term><computeroutput>BZ_OK</computeroutput></term> + <listitem><para>The requested action was completed + successfully.</para></listitem> + </varlistentry> + + <varlistentry> + <term><computeroutput>BZ_RUN_OK, BZ_FLUSH_OK, + BZ_FINISH_OK</computeroutput></term> + <listitem><para>In + <computeroutput>BZ2_bzCompress</computeroutput>, the requested + flush/finish/nothing-special action was completed + successfully.</para></listitem> + </varlistentry> + + <varlistentry> + <term><computeroutput>BZ_STREAM_END</computeroutput></term> + <listitem><para>Compression of data was completed, or the + logical stream end was detected during + decompression.</para></listitem> + </varlistentry> + +</variablelist> + +<para>The following return values indicate an error of some +kind.</para> + +<variablelist> + + <varlistentry> + <term><computeroutput>BZ_CONFIG_ERROR</computeroutput></term> + <listitem><para>Indicates that the library has been improperly + compiled on your platform -- a major configuration error. + Specifically, it means that + <computeroutput>sizeof(char)</computeroutput>, + <computeroutput>sizeof(short)</computeroutput> and + <computeroutput>sizeof(int)</computeroutput> are not 1, 2 and + 4 respectively, as they should be. Note that the library + should still work properly on 64-bit platforms which follow + the LP64 programming model -- that is, where + <computeroutput>sizeof(long)</computeroutput> and + <computeroutput>sizeof(void*)</computeroutput> are 8. Under + LP64, <computeroutput>sizeof(int)</computeroutput> is still 4, + so <computeroutput>libbzip2</computeroutput>, which doesn't + use the <computeroutput>long</computeroutput> type, is + OK.</para></listitem> + </varlistentry> + + <varlistentry> + <term><computeroutput>BZ_SEQUENCE_ERROR</computeroutput></term> + <listitem><para>When using the library, it is important to call + the functions in the correct sequence and with data structures + (buffers etc) in the correct states. + <computeroutput>libbzip2</computeroutput> checks as much as it + can to ensure this is happening, and returns + <computeroutput>BZ_SEQUENCE_ERROR</computeroutput> if not. + Code which complies precisely with the function semantics, as + detailed below, should never receive this value; such an event + denotes buggy code which you should + investigate.</para></listitem> + </varlistentry> + + <varlistentry> + <term><computeroutput>BZ_PARAM_ERROR</computeroutput></term> + <listitem><para>Returned when a parameter to a function call is + out of range or otherwise manifestly incorrect. As with + <computeroutput>BZ_SEQUENCE_ERROR</computeroutput>, this + denotes a bug in the client code. The distinction between + <computeroutput>BZ_PARAM_ERROR</computeroutput> and + <computeroutput>BZ_SEQUENCE_ERROR</computeroutput> is a bit + hazy, but still worth making.</para></listitem> + </varlistentry> + + <varlistentry> + <term><computeroutput>BZ_MEM_ERROR</computeroutput></term> + <listitem><para>Returned when a request to allocate memory + failed. Note that the quantity of memory needed to decompress + a stream cannot be determined until the stream's header has + been read. So + <computeroutput>BZ2_bzDecompress</computeroutput> and + <computeroutput>BZ2_bzRead</computeroutput> may return + <computeroutput>BZ_MEM_ERROR</computeroutput> even though some + of the compressed data has been read. The same is not true + for compression; once + <computeroutput>BZ2_bzCompressInit</computeroutput> or + <computeroutput>BZ2_bzWriteOpen</computeroutput> have + successfully completed, + <computeroutput>BZ_MEM_ERROR</computeroutput> cannot + occur.</para></listitem> + </varlistentry> + + <varlistentry> + <term><computeroutput>BZ_DATA_ERROR</computeroutput></term> + <listitem><para>Returned when a data integrity error is + detected during decompression. Most importantly, this means + when stored and computed CRCs for the data do not match. This + value is also returned upon detection of any other anomaly in + the compressed data.</para></listitem> + </varlistentry> + + <varlistentry> + <term><computeroutput>BZ_DATA_ERROR_MAGIC</computeroutput></term> + <listitem><para>As a special case of + <computeroutput>BZ_DATA_ERROR</computeroutput>, it is + sometimes useful to know when the compressed stream does not + start with the correct magic bytes (<computeroutput>'B' 'Z' + 'h'</computeroutput>).</para></listitem> + </varlistentry> + + <varlistentry> + <term><computeroutput>BZ_IO_ERROR</computeroutput></term> + <listitem><para>Returned by + <computeroutput>BZ2_bzRead</computeroutput> and + <computeroutput>BZ2_bzWrite</computeroutput> when there is an + error reading or writing in the compressed file, and by + <computeroutput>BZ2_bzReadOpen</computeroutput> and + <computeroutput>BZ2_bzWriteOpen</computeroutput> for attempts + to use a file for which the error indicator (viz, + <computeroutput>ferror(f)</computeroutput>) is set. On + receipt of <computeroutput>BZ_IO_ERROR</computeroutput>, the + caller should consult <computeroutput>errno</computeroutput> + and/or <computeroutput>perror</computeroutput> to acquire + operating-system specific information about the + problem.</para></listitem> + </varlistentry> + + <varlistentry> + <term><computeroutput>BZ_UNEXPECTED_EOF</computeroutput></term> + <listitem><para>Returned by + <computeroutput>BZ2_bzRead</computeroutput> when the + compressed file finishes before the logical end of stream is + detected.</para></listitem> + </varlistentry> + + <varlistentry> + <term><computeroutput>BZ_OUTBUFF_FULL</computeroutput></term> + <listitem><para>Returned by + <computeroutput>BZ2_bzBuffToBuffCompress</computeroutput> and + <computeroutput>BZ2_bzBuffToBuffDecompress</computeroutput> to + indicate that the output data will not fit into the output + buffer provided.</para></listitem> + </varlistentry> + +</variablelist> + +</sect1> + + + +<sect1 id="low-level" xreflabel=">Low-level interface"> +<title>Low-level interface</title> + + +<sect2 id="bzcompress-init" xreflabel="BZ2_bzCompressInit"> +<title>BZ2_bzCompressInit</title> + +<programlisting> +typedef struct { + char *next_in; + unsigned int avail_in; + unsigned int total_in_lo32; + unsigned int total_in_hi32; + + char *next_out; + unsigned int avail_out; + unsigned int total_out_lo32; + unsigned int total_out_hi32; + + void *state; + + void *(*bzalloc)(void *,int,int); + void (*bzfree)(void *,void *); + void *opaque; +} bz_stream; + +int BZ2_bzCompressInit ( bz_stream *strm, + int blockSize100k, + int verbosity, + int workFactor ); +</programlisting> + +<para>Prepares for compression. The +<computeroutput>bz_stream</computeroutput> structure holds all +data pertaining to the compression activity. A +<computeroutput>bz_stream</computeroutput> structure should be +allocated and initialised prior to the call. The fields of +<computeroutput>bz_stream</computeroutput> comprise the entirety +of the user-visible data. <computeroutput>state</computeroutput> +is a pointer to the private data structures required for +compression.</para> + +<para>Custom memory allocators are supported, via fields +<computeroutput>bzalloc</computeroutput>, +<computeroutput>bzfree</computeroutput>, and +<computeroutput>opaque</computeroutput>. The value +<computeroutput>opaque</computeroutput> is passed to as the first +argument to all calls to <computeroutput>bzalloc</computeroutput> +and <computeroutput>bzfree</computeroutput>, but is otherwise +ignored by the library. The call <computeroutput>bzalloc ( +opaque, n, m )</computeroutput> is expected to return a pointer +<computeroutput>p</computeroutput> to <computeroutput>n * +m</computeroutput> bytes of memory, and <computeroutput>bzfree ( +opaque, p )</computeroutput> should free that memory.</para> + +<para>If you don't want to use a custom memory allocator, set +<computeroutput>bzalloc</computeroutput>, +<computeroutput>bzfree</computeroutput> and +<computeroutput>opaque</computeroutput> to +<computeroutput>NULL</computeroutput>, and the library will then +use the standard <computeroutput>malloc</computeroutput> / +<computeroutput>free</computeroutput> routines.</para> + +<para>Before calling +<computeroutput>BZ2_bzCompressInit</computeroutput>, fields +<computeroutput>bzalloc</computeroutput>, +<computeroutput>bzfree</computeroutput> and +<computeroutput>opaque</computeroutput> should be filled +appropriately, as just described. Upon return, the internal +state will have been allocated and initialised, and +<computeroutput>total_in_lo32</computeroutput>, +<computeroutput>total_in_hi32</computeroutput>, +<computeroutput>total_out_lo32</computeroutput> and +<computeroutput>total_out_hi32</computeroutput> will have been +set to zero. These four fields are used by the library to inform +the caller of the total amount of data passed into and out of the +library, respectively. You should not try to change them. As of +version 1.0, 64-bit counts are maintained, even on 32-bit +platforms, using the <computeroutput>_hi32</computeroutput> +fields to store the upper 32 bits of the count. So, for example, +the total amount of data in is <computeroutput>(total_in_hi32 +<< 32) + total_in_lo32</computeroutput>.</para> + +<para>Parameter <computeroutput>blockSize100k</computeroutput> +specifies the block size to be used for compression. It should +be a value between 1 and 9 inclusive, and the actual block size +used is 100000 x this figure. 9 gives the best compression but +takes most memory.</para> + +<para>Parameter <computeroutput>verbosity</computeroutput> should +be set to a number between 0 and 4 inclusive. 0 is silent, and +greater numbers give increasingly verbose monitoring/debugging +output. If the library has been compiled with +<computeroutput>-DBZ_NO_STDIO</computeroutput>, no such output +will appear for any verbosity setting.</para> + +<para>Parameter <computeroutput>workFactor</computeroutput> +controls how the compression phase behaves when presented with +worst case, highly repetitive, input data. If compression runs +into difficulties caused by repetitive data, the library switches +from the standard sorting algorithm to a fallback algorithm. The +fallback is slower than the standard algorithm by perhaps a +factor of three, but always behaves reasonably, no matter how bad +the input.</para> + +<para>Lower values of <computeroutput>workFactor</computeroutput> +reduce the amount of effort the standard algorithm will expend +before resorting to the fallback. You should set this parameter +carefully; too low, and many inputs will be handled by the +fallback algorithm and so compress rather slowly, too high, and +your average-to-worst case compression times can become very +large. The default value of 30 gives reasonable behaviour over a +wide range of circumstances.</para> + +<para>Allowable values range from 0 to 250 inclusive. 0 is a +special case, equivalent to using the default value of 30.</para> + +<para>Note that the compressed output generated is the same +regardless of whether or not the fallback algorithm is +used.</para> + +<para>Be aware also that this parameter may disappear entirely in +future versions of the library. In principle it should be +possible to devise a good way to automatically choose which +algorithm to use. Such a mechanism would render the parameter +obsolete.</para> + +<para>Possible return values:</para> + +<programlisting> +BZ_CONFIG_ERROR + if the library has been mis-compiled +BZ_PARAM_ERROR + if strm is NULL + or blockSize < 1 or blockSize > 9 + or verbosity < 0 or verbosity > 4 + or workFactor < 0 or workFactor > 250 +BZ_MEM_ERROR + if not enough memory is available +BZ_OK + otherwise +</programlisting> + +<para>Allowable next actions:</para> + +<programlisting> +BZ2_bzCompress + if BZ_OK is returned + no specific action needed in case of error +</programlisting> + +</sect2> + + +<sect2 id="bzCompress" xreflabel="BZ2_bzCompress"> +<title>BZ2_bzCompress</title> + +<programlisting> +int BZ2_bzCompress ( bz_stream *strm, int action ); +</programlisting> + +<para>Provides more input and/or output buffer space for the +library. The caller maintains input and output buffers, and +calls <computeroutput>BZ2_bzCompress</computeroutput> to transfer +data between them.</para> + +<para>Before each call to +<computeroutput>BZ2_bzCompress</computeroutput>, +<computeroutput>next_in</computeroutput> should point at the data +to be compressed, and <computeroutput>avail_in</computeroutput> +should indicate how many bytes the library may read. +<computeroutput>BZ2_bzCompress</computeroutput> updates +<computeroutput>next_in</computeroutput>, +<computeroutput>avail_in</computeroutput> and +<computeroutput>total_in</computeroutput> to reflect the number +of bytes it has read.</para> + +<para>Similarly, <computeroutput>next_out</computeroutput> should +point to a buffer in which the compressed data is to be placed, +with <computeroutput>avail_out</computeroutput> indicating how +much output space is available. +<computeroutput>BZ2_bzCompress</computeroutput> updates +<computeroutput>next_out</computeroutput>, +<computeroutput>avail_out</computeroutput> and +<computeroutput>total_out</computeroutput> to reflect the number +of bytes output.</para> + +<para>You may provide and remove as little or as much data as you +like on each call of +<computeroutput>BZ2_bzCompress</computeroutput>. In the limit, +it is acceptable to supply and remove data one byte at a time, +although this would be terribly inefficient. You should always +ensure that at least one byte of output space is available at +each call.</para> + +<para>A second purpose of +<computeroutput>BZ2_bzCompress</computeroutput> is to request a +change of mode of the compressed stream.</para> + +<para>Conceptually, a compressed stream can be in one of four +states: IDLE, RUNNING, FLUSHING and FINISHING. Before +initialisation +(<computeroutput>BZ2_bzCompressInit</computeroutput>) and after +termination (<computeroutput>BZ2_bzCompressEnd</computeroutput>), +a stream is regarded as IDLE.</para> + +<para>Upon initialisation +(<computeroutput>BZ2_bzCompressInit</computeroutput>), the stream +is placed in the RUNNING state. Subsequent calls to +<computeroutput>BZ2_bzCompress</computeroutput> should pass +<computeroutput>BZ_RUN</computeroutput> as the requested action; +other actions are illegal and will result in +<computeroutput>BZ_SEQUENCE_ERROR</computeroutput>.</para> + +<para>At some point, the calling program will have provided all +the input data it wants to. It will then want to finish up -- in +effect, asking the library to process any data it might have +buffered internally. In this state, +<computeroutput>BZ2_bzCompress</computeroutput> will no longer +attempt to read data from +<computeroutput>next_in</computeroutput>, but it will want to +write data to <computeroutput>next_out</computeroutput>. Because +the output buffer supplied by the user can be arbitrarily small, +the finishing-up operation cannot necessarily be done with a +single call of +<computeroutput>BZ2_bzCompress</computeroutput>.</para> + +<para>Instead, the calling program passes +<computeroutput>BZ_FINISH</computeroutput> as an action to +<computeroutput>BZ2_bzCompress</computeroutput>. This changes +the stream's state to FINISHING. Any remaining input (ie, +<computeroutput>next_in[0 .. avail_in-1]</computeroutput>) is +compressed and transferred to the output buffer. To do this, +<computeroutput>BZ2_bzCompress</computeroutput> must be called +repeatedly until all the output has been consumed. At that +point, <computeroutput>BZ2_bzCompress</computeroutput> returns +<computeroutput>BZ_STREAM_END</computeroutput>, and the stream's +state is set back to IDLE. +<computeroutput>BZ2_bzCompressEnd</computeroutput> should then be +called.</para> + +<para>Just to make sure the calling program does not cheat, the +library makes a note of <computeroutput>avail_in</computeroutput> +at the time of the first call to +<computeroutput>BZ2_bzCompress</computeroutput> which has +<computeroutput>BZ_FINISH</computeroutput> as an action (ie, at +the time the program has announced its intention to not supply +any more input). By comparing this value with that of +<computeroutput>avail_in</computeroutput> over subsequent calls +to <computeroutput>BZ2_bzCompress</computeroutput>, the library +can detect any attempts to slip in more data to compress. Any +calls for which this is detected will return +<computeroutput>BZ_SEQUENCE_ERROR</computeroutput>. This +indicates a programming mistake which should be corrected.</para> + +<para>Instead of asking to finish, the calling program may ask +<computeroutput>BZ2_bzCompress</computeroutput> to take all the +remaining input, compress it and terminate the current +(Burrows-Wheeler) compression block. This could be useful for +error control purposes. The mechanism is analogous to that for +finishing: call <computeroutput>BZ2_bzCompress</computeroutput> +with an action of <computeroutput>BZ_FLUSH</computeroutput>, +remove output data, and persist with the +<computeroutput>BZ_FLUSH</computeroutput> action until the value +<computeroutput>BZ_RUN</computeroutput> is returned. As with +finishing, <computeroutput>BZ2_bzCompress</computeroutput> +detects any attempt to provide more input data once the flush has +begun.</para> + +<para>Once the flush is complete, the stream returns to the +normal RUNNING state.</para> + +<para>This all sounds pretty complex, but isn't really. Here's a +table which shows which actions are allowable in each state, what +action will be taken, what the next state is, and what the +non-error return values are. Note that you can't explicitly ask +what state the stream is in, but nor do you need to -- it can be +inferred from the values returned by +<computeroutput>BZ2_bzCompress</computeroutput>.</para> + +<programlisting> +IDLE/any + Illegal. IDLE state only exists after BZ2_bzCompressEnd or + before BZ2_bzCompressInit. + Return value = BZ_SEQUENCE_ERROR + +RUNNING/BZ_RUN + Compress from next_in to next_out as much as possible. + Next state = RUNNING + Return value = BZ_RUN_OK + +RUNNING/BZ_FLUSH + Remember current value of next_in. Compress from next_in + to next_out as much as possible, but do not accept any more input. + Next state = FLUSHING + Return value = BZ_FLUSH_OK + +RUNNING/BZ_FINISH + Remember current value of next_in. Compress from next_in + to next_out as much as possible, but do not accept any more input. + Next state = FINISHING + Return value = BZ_FINISH_OK + +FLUSHING/BZ_FLUSH + Compress from next_in to next_out as much as possible, + but do not accept any more input. + If all the existing input has been used up and all compressed + output has been removed + Next state = RUNNING; Return value = BZ_RUN_OK + else + Next state = FLUSHING; Return value = BZ_FLUSH_OK + +FLUSHING/other + Illegal. + Return value = BZ_SEQUENCE_ERROR + +FINISHING/BZ_FINISH + Compress from next_in to next_out as much as possible, + but to not accept any more input. + If all the existing input has been used up and all compressed + output has been removed + Next state = IDLE; Return value = BZ_STREAM_END + else + Next state = FINISHING; Return value = BZ_FINISH_OK + +FINISHING/other + Illegal. + Return value = BZ_SEQUENCE_ERROR +</programlisting> + + +<para>That still looks complicated? Well, fair enough. The +usual sequence of calls for compressing a load of data is:</para> + +<orderedlist> + + <listitem><para>Get started with + <computeroutput>BZ2_bzCompressInit</computeroutput>.</para></listitem> + + <listitem><para>Shovel data in and shlurp out its compressed form + using zero or more calls of + <computeroutput>BZ2_bzCompress</computeroutput> with action = + <computeroutput>BZ_RUN</computeroutput>.</para></listitem> + + <listitem><para>Finish up. Repeatedly call + <computeroutput>BZ2_bzCompress</computeroutput> with action = + <computeroutput>BZ_FINISH</computeroutput>, copying out the + compressed output, until + <computeroutput>BZ_STREAM_END</computeroutput> is + returned.</para></listitem> <listitem><para>Close up and go home. Call + <computeroutput>BZ2_bzCompressEnd</computeroutput>.</para></listitem> + +</orderedlist> + +<para>If the data you want to compress fits into your input +buffer all at once, you can skip the calls of +<computeroutput>BZ2_bzCompress ( ..., BZ_RUN )</computeroutput> +and just do the <computeroutput>BZ2_bzCompress ( ..., BZ_FINISH +)</computeroutput> calls.</para> + +<para>All required memory is allocated by +<computeroutput>BZ2_bzCompressInit</computeroutput>. The +compression library can accept any data at all (obviously). So +you shouldn't get any error return values from the +<computeroutput>BZ2_bzCompress</computeroutput> calls. If you +do, they will be +<computeroutput>BZ_SEQUENCE_ERROR</computeroutput>, and indicate +a bug in your programming.</para> + +<para>Trivial other possible return values:</para> + +<programlisting> +BZ_PARAM_ERROR + if strm is NULL, or strm->s is NULL +</programlisting> + +</sect2> + + +<sect2 id="bzCompress-end" xreflabel="BZ2_bzCompressEnd"> +<title>BZ2_bzCompressEnd</title> + +<programlisting> +int BZ2_bzCompressEnd ( bz_stream *strm ); +</programlisting> + +<para>Releases all memory associated with a compression +stream.</para> + +<para>Possible return values:</para> + +<programlisting> +BZ_PARAM_ERROR if strm is NULL or strm->s is NULL +BZ_OK otherwise +</programlisting> + +</sect2> + + +<sect2 id="bzDecompress-init" xreflabel="BZ2_bzDecompressInit"> +<title>BZ2_bzDecompressInit</title> + +<programlisting> +int BZ2_bzDecompressInit ( bz_stream *strm, int verbosity, int small ); +</programlisting> + +<para>Prepares for decompression. As with +<computeroutput>BZ2_bzCompressInit</computeroutput>, a +<computeroutput>bz_stream</computeroutput> record should be +allocated and initialised before the call. Fields +<computeroutput>bzalloc</computeroutput>, +<computeroutput>bzfree</computeroutput> and +<computeroutput>opaque</computeroutput> should be set if a custom +memory allocator is required, or made +<computeroutput>NULL</computeroutput> for the normal +<computeroutput>malloc</computeroutput> / +<computeroutput>free</computeroutput> routines. Upon return, the +internal state will have been initialised, and +<computeroutput>total_in</computeroutput> and +<computeroutput>total_out</computeroutput> will be zero.</para> + +<para>For the meaning of parameter +<computeroutput>verbosity</computeroutput>, see +<computeroutput>BZ2_bzCompressInit</computeroutput>.</para> + +<para>If <computeroutput>small</computeroutput> is nonzero, the +library will use an alternative decompression algorithm which +uses less memory but at the cost of decompressing more slowly +(roughly speaking, half the speed, but the maximum memory +requirement drops to around 2300k). See <xref linkend="using"/> +for more information on memory management.</para> + +<para>Note that the amount of memory needed to decompress a +stream cannot be determined until the stream's header has been +read, so even if +<computeroutput>BZ2_bzDecompressInit</computeroutput> succeeds, a +subsequent <computeroutput>BZ2_bzDecompress</computeroutput> +could fail with +<computeroutput>BZ_MEM_ERROR</computeroutput>.</para> + +<para>Possible return values:</para> + +<programlisting> +BZ_CONFIG_ERROR + if the library has been mis-compiled +BZ_PARAM_ERROR + if ( small != 0 && small != 1 ) + or (verbosity <; 0 || verbosity > 4) +BZ_MEM_ERROR + if insufficient memory is available +</programlisting> + +<para>Allowable next actions:</para> + +<programlisting> +BZ2_bzDecompress + if BZ_OK was returned + no specific action required in case of error +</programlisting> + +</sect2> + + +<sect2 id="bzDecompress" xreflabel="BZ2_bzDecompress"> +<title>BZ2_bzDecompress</title> + +<programlisting> +int BZ2_bzDecompress ( bz_stream *strm ); +</programlisting> + +<para>Provides more input and/out output buffer space for the +library. The caller maintains input and output buffers, and uses +<computeroutput>BZ2_bzDecompress</computeroutput> to transfer +data between them.</para> + +<para>Before each call to +<computeroutput>BZ2_bzDecompress</computeroutput>, +<computeroutput>next_in</computeroutput> should point at the +compressed data, and <computeroutput>avail_in</computeroutput> +should indicate how many bytes the library may read. +<computeroutput>BZ2_bzDecompress</computeroutput> updates +<computeroutput>next_in</computeroutput>, +<computeroutput>avail_in</computeroutput> and +<computeroutput>total_in</computeroutput> to reflect the number +of bytes it has read.</para> + +<para>Similarly, <computeroutput>next_out</computeroutput> should +point to a buffer in which the uncompressed output is to be +placed, with <computeroutput>avail_out</computeroutput> +indicating how much output space is available. +<computeroutput>BZ2_bzCompress</computeroutput> updates +<computeroutput>next_out</computeroutput>, +<computeroutput>avail_out</computeroutput> and +<computeroutput>total_out</computeroutput> to reflect the number +of bytes output.</para> + +<para>You may provide and remove as little or as much data as you +like on each call of +<computeroutput>BZ2_bzDecompress</computeroutput>. In the limit, +it is acceptable to supply and remove data one byte at a time, +although this would be terribly inefficient. You should always +ensure that at least one byte of output space is available at +each call.</para> + +<para>Use of <computeroutput>BZ2_bzDecompress</computeroutput> is +simpler than +<computeroutput>BZ2_bzCompress</computeroutput>.</para> + +<para>You should provide input and remove output as described +above, and repeatedly call +<computeroutput>BZ2_bzDecompress</computeroutput> until +<computeroutput>BZ_STREAM_END</computeroutput> is returned. +Appearance of <computeroutput>BZ_STREAM_END</computeroutput> +denotes that <computeroutput>BZ2_bzDecompress</computeroutput> +has detected the logical end of the compressed stream. +<computeroutput>BZ2_bzDecompress</computeroutput> will not +produce <computeroutput>BZ_STREAM_END</computeroutput> until all +output data has been placed into the output buffer, so once +<computeroutput>BZ_STREAM_END</computeroutput> appears, you are +guaranteed to have available all the decompressed output, and +<computeroutput>BZ2_bzDecompressEnd</computeroutput> can safely +be called.</para> + +<para>If case of an error return value, you should call +<computeroutput>BZ2_bzDecompressEnd</computeroutput> to clean up +and release memory.</para> + +<para>Possible return values:</para> + +<programlisting> +BZ_PARAM_ERROR + if strm is NULL or strm->s is NULL + or strm->avail_out < 1 +BZ_DATA_ERROR + if a data integrity error is detected in the compressed stream +BZ_DATA_ERROR_MAGIC + if the compressed stream doesn't begin with the right magic bytes +BZ_MEM_ERROR + if there wasn't enough memory available +BZ_STREAM_END + if the logical end of the data stream was detected and all + output in has been consumed, eg s-->avail_out > 0 +BZ_OK + otherwise +</programlisting> + +<para>Allowable next actions:</para> + +<programlisting> +BZ2_bzDecompress + if BZ_OK was returned +BZ2_bzDecompressEnd + otherwise +</programlisting> + +</sect2> + + +<sect2 id="bzDecompress-end" xreflabel="BZ2_bzDecompressEnd"> +<title>BZ2_bzDecompressEnd</title> + +<programlisting> +int BZ2_bzDecompressEnd ( bz_stream *strm ); +</programlisting> + +<para>Releases all memory associated with a decompression +stream.</para> + +<para>Possible return values:</para> + +<programlisting> +BZ_PARAM_ERROR + if strm is NULL or strm->s is NULL +BZ_OK + otherwise +</programlisting> + +<para>Allowable next actions:</para> + +<programlisting> + None. +</programlisting> + +</sect2> + +</sect1> + + +<sect1 id="hl-interface" xreflabel="High-level interface"> +<title>High-level interface</title> + +<para>This interface provides functions for reading and writing +<computeroutput>bzip2</computeroutput> format files. First, some +general points.</para> + +<itemizedlist mark='bullet'> + + <listitem><para>All of the functions take an + <computeroutput>int*</computeroutput> first argument, + <computeroutput>bzerror</computeroutput>. After each call, + <computeroutput>bzerror</computeroutput> should be consulted + first to determine the outcome of the call. If + <computeroutput>bzerror</computeroutput> is + <computeroutput>BZ_OK</computeroutput>, the call completed + successfully, and only then should the return value of the + function (if any) be consulted. If + <computeroutput>bzerror</computeroutput> is + <computeroutput>BZ_IO_ERROR</computeroutput>, there was an + error reading/writing the underlying compressed file, and you + should then consult <computeroutput>errno</computeroutput> / + <computeroutput>perror</computeroutput> to determine the cause + of the difficulty. <computeroutput>bzerror</computeroutput> + may also be set to various other values; precise details are + given on a per-function basis below.</para></listitem> + + <listitem><para>If <computeroutput>bzerror</computeroutput> indicates + an error (ie, anything except + <computeroutput>BZ_OK</computeroutput> and + <computeroutput>BZ_STREAM_END</computeroutput>), you should + immediately call + <computeroutput>BZ2_bzReadClose</computeroutput> (or + <computeroutput>BZ2_bzWriteClose</computeroutput>, depending on + whether you are attempting to read or to write) to free up all + resources associated with the stream. Once an error has been + indicated, behaviour of all calls except + <computeroutput>BZ2_bzReadClose</computeroutput> + (<computeroutput>BZ2_bzWriteClose</computeroutput>) is + undefined. The implication is that (1) + <computeroutput>bzerror</computeroutput> should be checked + after each call, and (2) if + <computeroutput>bzerror</computeroutput> indicates an error, + <computeroutput>BZ2_bzReadClose</computeroutput> + (<computeroutput>BZ2_bzWriteClose</computeroutput>) should then + be called to clean up.</para></listitem> + + <listitem><para>The <computeroutput>FILE*</computeroutput> arguments + passed to <computeroutput>BZ2_bzReadOpen</computeroutput> / + <computeroutput>BZ2_bzWriteOpen</computeroutput> should be set + to binary mode. Most Unix systems will do this by default, but + other platforms, including Windows and Mac, will not. If you + omit this, you may encounter problems when moving code to new + platforms.</para></listitem> + + <listitem><para>Memory allocation requests are handled by + <computeroutput>malloc</computeroutput> / + <computeroutput>free</computeroutput>. At present there is no + facility for user-defined memory allocators in the file I/O + functions (could easily be added, though).</para></listitem> + +</itemizedlist> + + + +<sect2 id="bzreadopen" xreflabel="BZ2_bzReadOpen"> +<title>BZ2_bzReadOpen</title> + +<programlisting> +typedef void BZFILE; + +BZFILE *BZ2_bzReadOpen( int *bzerror, FILE *f, + int verbosity, int small, + void *unused, int nUnused ); +</programlisting> + +<para>Prepare to read compressed data from file handle +<computeroutput>f</computeroutput>. +<computeroutput>f</computeroutput> should refer to a file which +has been opened for reading, and for which the error indicator +(<computeroutput>ferror(f)</computeroutput>)is not set. If +<computeroutput>small</computeroutput> is 1, the library will try +to decompress using less memory, at the expense of speed.</para> + +<para>For reasons explained below, +<computeroutput>BZ2_bzRead</computeroutput> will decompress the +<computeroutput>nUnused</computeroutput> bytes starting at +<computeroutput>unused</computeroutput>, before starting to read +from the file <computeroutput>f</computeroutput>. At most +<computeroutput>BZ_MAX_UNUSED</computeroutput> bytes may be +supplied like this. If this facility is not required, you should +pass <computeroutput>NULL</computeroutput> and +<computeroutput>0</computeroutput> for +<computeroutput>unused</computeroutput> and +n<computeroutput>Unused</computeroutput> respectively.</para> + +<para>For the meaning of parameters +<computeroutput>small</computeroutput> and +<computeroutput>verbosity</computeroutput>, see +<computeroutput>BZ2_bzDecompressInit</computeroutput>.</para> + +<para>The amount of memory needed to decompress a file cannot be +determined until the file's header has been read. So it is +possible that <computeroutput>BZ2_bzReadOpen</computeroutput> +returns <computeroutput>BZ_OK</computeroutput> but a subsequent +call of <computeroutput>BZ2_bzRead</computeroutput> will return +<computeroutput>BZ_MEM_ERROR</computeroutput>.</para> + +<para>Possible assignments to +<computeroutput>bzerror</computeroutput>:</para> + +<programlisting> +BZ_CONFIG_ERROR + if the library has been mis-compiled +BZ_PARAM_ERROR + if f is NULL + or small is neither 0 nor 1 + or ( unused == NULL && nUnused != 0 ) + or ( unused != NULL && !(0 <= nUnused <= BZ_MAX_UNUSED) ) +BZ_IO_ERROR + if ferror(f) is nonzero +BZ_MEM_ERROR + if insufficient memory is available +BZ_OK + otherwise. +</programlisting> + +<para>Possible return values:</para> + +<programlisting> +Pointer to an abstract BZFILE + if bzerror is BZ_OK +NULL + otherwise +</programlisting> + +<para>Allowable next actions:</para> + +<programlisting> +BZ2_bzRead + if bzerror is BZ_OK +BZ2_bzClose + otherwise +</programlisting> + +</sect2> + + +<sect2 id="bzread" xreflabel="BZ2_bzRead"> +<title>BZ2_bzRead</title> + +<programlisting> +int BZ2_bzRead ( int *bzerror, BZFILE *b, void *buf, int len ); +</programlisting> + +<para>Reads up to <computeroutput>len</computeroutput> +(uncompressed) bytes from the compressed file +<computeroutput>b</computeroutput> into the buffer +<computeroutput>buf</computeroutput>. If the read was +successful, <computeroutput>bzerror</computeroutput> is set to +<computeroutput>BZ_OK</computeroutput> and the number of bytes +read is returned. If the logical end-of-stream was detected, +<computeroutput>bzerror</computeroutput> will be set to +<computeroutput>BZ_STREAM_END</computeroutput>, and the number of +bytes read is returned. All other +<computeroutput>bzerror</computeroutput> values denote an +error.</para> + +<para><computeroutput>BZ2_bzRead</computeroutput> will supply +<computeroutput>len</computeroutput> bytes, unless the logical +stream end is detected or an error occurs. Because of this, it +is possible to detect the stream end by observing when the number +of bytes returned is less than the number requested. +Nevertheless, this is regarded as inadvisable; you should instead +check <computeroutput>bzerror</computeroutput> after every call +and watch out for +<computeroutput>BZ_STREAM_END</computeroutput>.</para> + +<para>Internally, <computeroutput>BZ2_bzRead</computeroutput> +copies data from the compressed file in chunks of size +<computeroutput>BZ_MAX_UNUSED</computeroutput> bytes before +decompressing it. If the file contains more bytes than strictly +needed to reach the logical end-of-stream, +<computeroutput>BZ2_bzRead</computeroutput> will almost certainly +read some of the trailing data before signalling +<computeroutput>BZ_SEQUENCE_END</computeroutput>. To collect the +read but unused data once +<computeroutput>BZ_SEQUENCE_END</computeroutput> has appeared, +call <computeroutput>BZ2_bzReadGetUnused</computeroutput> +immediately before +<computeroutput>BZ2_bzReadClose</computeroutput>.</para> + +<para>Possible assignments to +<computeroutput>bzerror</computeroutput>:</para> + +<programlisting> +BZ_PARAM_ERROR + if b is NULL or buf is NULL or len < 0 +BZ_SEQUENCE_ERROR + if b was opened with BZ2_bzWriteOpen +BZ_IO_ERROR + if there is an error reading from the compressed file +BZ_UNEXPECTED_EOF + if the compressed file ended before + the logical end-of-stream was detected +BZ_DATA_ERROR + if a data integrity error was detected in the compressed stream +BZ_DATA_ERROR_MAGIC + if the stream does not begin with the requisite header bytes + (ie, is not a bzip2 data file). This is really + a special case of BZ_DATA_ERROR. +BZ_MEM_ERROR + if insufficient memory was available +BZ_STREAM_END + if the logical end of stream was detected. +BZ_OK + otherwise. +</programlisting> + +<para>Possible return values:</para> + +<programlisting> +number of bytes read + if bzerror is BZ_OK or BZ_STREAM_END +undefined + otherwise +</programlisting> + +<para>Allowable next actions:</para> + +<programlisting> +collect data from buf, then BZ2_bzRead or BZ2_bzReadClose + if bzerror is BZ_OK +collect data from buf, then BZ2_bzReadClose or BZ2_bzReadGetUnused + if bzerror is BZ_SEQUENCE_END +BZ2_bzReadClose + otherwise +</programlisting> + +</sect2> + + +<sect2 id="bzreadgetunused" xreflabel="BZ2_bzReadGetUnused"> +<title>BZ2_bzReadGetUnused</title> + +<programlisting> +void BZ2_bzReadGetUnused( int* bzerror, BZFILE *b, + void** unused, int* nUnused ); +</programlisting> + +<para>Returns data which was read from the compressed file but +was not needed to get to the logical end-of-stream. +<computeroutput>*unused</computeroutput> is set to the address of +the data, and <computeroutput>*nUnused</computeroutput> to the +number of bytes. <computeroutput>*nUnused</computeroutput> will +be set to a value between <computeroutput>0</computeroutput> and +<computeroutput>BZ_MAX_UNUSED</computeroutput> inclusive.</para> + +<para>This function may only be called once +<computeroutput>BZ2_bzRead</computeroutput> has signalled +<computeroutput>BZ_STREAM_END</computeroutput> but before +<computeroutput>BZ2_bzReadClose</computeroutput>.</para> + +<para>Possible assignments to +<computeroutput>bzerror</computeroutput>:</para> + +<programlisting> +BZ_PARAM_ERROR + if b is NULL + or unused is NULL or nUnused is NULL +BZ_SEQUENCE_ERROR + if BZ_STREAM_END has not been signalled + or if b was opened with BZ2_bzWriteOpen +BZ_OK + otherwise +</programlisting> + +<para>Allowable next actions:</para> + +<programlisting> +BZ2_bzReadClose +</programlisting> + +</sect2> + + +<sect2 id="bzreadclose" xreflabel="BZ2_bzReadClose"> +<title>BZ2_bzReadClose</title> + +<programlisting> +void BZ2_bzReadClose ( int *bzerror, BZFILE *b ); +</programlisting> + +<para>Releases all memory pertaining to the compressed file +<computeroutput>b</computeroutput>. +<computeroutput>BZ2_bzReadClose</computeroutput> does not call +<computeroutput>fclose</computeroutput> on the underlying file +handle, so you should do that yourself if appropriate. +<computeroutput>BZ2_bzReadClose</computeroutput> should be called +to clean up after all error situations.</para> + +<para>Possible assignments to +<computeroutput>bzerror</computeroutput>:</para> + +<programlisting> +BZ_SEQUENCE_ERROR + if b was opened with BZ2_bzOpenWrite +BZ_OK + otherwise +</programlisting> + +<para>Allowable next actions:</para> + +<programlisting> +none +</programlisting> + +</sect2> + + +<sect2 id="bzwriteopen" xreflabel="BZ2_bzWriteOpen"> +<title>BZ2_bzWriteOpen</title> + +<programlisting> +BZFILE *BZ2_bzWriteOpen( int *bzerror, FILE *f, + int blockSize100k, int verbosity, + int workFactor ); +</programlisting> + +<para>Prepare to write compressed data to file handle +<computeroutput>f</computeroutput>. +<computeroutput>f</computeroutput> should refer to a file which +has been opened for writing, and for which the error indicator +(<computeroutput>ferror(f)</computeroutput>)is not set.</para> + +<para>For the meaning of parameters +<computeroutput>blockSize100k</computeroutput>, +<computeroutput>verbosity</computeroutput> and +<computeroutput>workFactor</computeroutput>, see +<computeroutput>BZ2_bzCompressInit</computeroutput>.</para> + +<para>All required memory is allocated at this stage, so if the +call completes successfully, +<computeroutput>BZ_MEM_ERROR</computeroutput> cannot be signalled +by a subsequent call to +<computeroutput>BZ2_bzWrite</computeroutput>.</para> + +<para>Possible assignments to +<computeroutput>bzerror</computeroutput>:</para> + +<programlisting> +BZ_CONFIG_ERROR + if the library has been mis-compiled +BZ_PARAM_ERROR + if f is NULL + or blockSize100k < 1 or blockSize100k > 9 +BZ_IO_ERROR + if ferror(f) is nonzero +BZ_MEM_ERROR + if insufficient memory is available +BZ_OK + otherwise +</programlisting> + +<para>Possible return values:</para> + +<programlisting> +Pointer to an abstract BZFILE + if bzerror is BZ_OK +NULL + otherwise +</programlisting> + +<para>Allowable next actions:</para> + +<programlisting> +BZ2_bzWrite + if bzerror is BZ_OK + (you could go directly to BZ2_bzWriteClose, but this would be pretty pointless) +BZ2_bzWriteClose + otherwise +</programlisting> + +</sect2> + + +<sect2 id="bzwrite" xreflabel="BZ2_bzWrite"> +<title>BZ2_bzWrite</title> + +<programlisting> +void BZ2_bzWrite ( int *bzerror, BZFILE *b, void *buf, int len ); +</programlisting> + +<para>Absorbs <computeroutput>len</computeroutput> bytes from the +buffer <computeroutput>buf</computeroutput>, eventually to be +compressed and written to the file.</para> + +<para>Possible assignments to +<computeroutput>bzerror</computeroutput>:</para> + +<programlisting> +BZ_PARAM_ERROR + if b is NULL or buf is NULL or len < 0 +BZ_SEQUENCE_ERROR + if b was opened with BZ2_bzReadOpen +BZ_IO_ERROR + if there is an error writing the compressed file. +BZ_OK + otherwise +</programlisting> + +</sect2> + + +<sect2 id="bzwriteclose" xreflabel="BZ2_bzWriteClose"> +<title>BZ2_bzWriteClose</title> + +<programlisting> +void BZ2_bzWriteClose( int *bzerror, BZFILE* f, + int abandon, + unsigned int* nbytes_in, + unsigned int* nbytes_out ); + +void BZ2_bzWriteClose64( int *bzerror, BZFILE* f, + int abandon, + unsigned int* nbytes_in_lo32, + unsigned int* nbytes_in_hi32, + unsigned int* nbytes_out_lo32, + unsigned int* nbytes_out_hi32 ); +</programlisting> + +<para>Compresses and flushes to the compressed file all data so +far supplied by <computeroutput>BZ2_bzWrite</computeroutput>. +The logical end-of-stream markers are also written, so subsequent +calls to <computeroutput>BZ2_bzWrite</computeroutput> are +illegal. All memory associated with the compressed file +<computeroutput>b</computeroutput> is released. +<computeroutput>fflush</computeroutput> is called on the +compressed file, but it is not +<computeroutput>fclose</computeroutput>'d.</para> + +<para>If <computeroutput>BZ2_bzWriteClose</computeroutput> is +called to clean up after an error, the only action is to release +the memory. The library records the error codes issued by +previous calls, so this situation will be detected automatically. +There is no attempt to complete the compression operation, nor to +<computeroutput>fflush</computeroutput> the compressed file. You +can force this behaviour to happen even in the case of no error, +by passing a nonzero value to +<computeroutput>abandon</computeroutput>.</para> + +<para>If <computeroutput>nbytes_in</computeroutput> is non-null, +<computeroutput>*nbytes_in</computeroutput> will be set to be the +total volume of uncompressed data handled. Similarly, +<computeroutput>nbytes_out</computeroutput> will be set to the +total volume of compressed data written. For compatibility with +older versions of the library, +<computeroutput>BZ2_bzWriteClose</computeroutput> only yields the +lower 32 bits of these counts. Use +<computeroutput>BZ2_bzWriteClose64</computeroutput> if you want +the full 64 bit counts. These two functions are otherwise +absolutely identical.</para> + +<para>Possible assignments to +<computeroutput>bzerror</computeroutput>:</para> + +<programlisting> +BZ_SEQUENCE_ERROR + if b was opened with BZ2_bzReadOpen +BZ_IO_ERROR + if there is an error writing the compressed file +BZ_OK + otherwise +</programlisting> + +</sect2> + + +<sect2 id="embed" xreflabel="Handling embedded compressed data streams"> +<title>Handling embedded compressed data streams</title> + +<para>The high-level library facilitates use of +<computeroutput>bzip2</computeroutput> data streams which form +some part of a surrounding, larger data stream.</para> + +<itemizedlist mark='bullet'> + + <listitem><para>For writing, the library takes an open file handle, + writes compressed data to it, + <computeroutput>fflush</computeroutput>es it but does not + <computeroutput>fclose</computeroutput> it. The calling + application can write its own data before and after the + compressed data stream, using that same file handle.</para></listitem> + + <listitem><para>Reading is more complex, and the facilities are not as + general as they could be since generality is hard to reconcile + with efficiency. <computeroutput>BZ2_bzRead</computeroutput> + reads from the compressed file in blocks of size + <computeroutput>BZ_MAX_UNUSED</computeroutput> bytes, and in + doing so probably will overshoot the logical end of compressed + stream. To recover this data once decompression has ended, + call <computeroutput>BZ2_bzReadGetUnused</computeroutput> after + the last call of <computeroutput>BZ2_bzRead</computeroutput> + (the one returning + <computeroutput>BZ_STREAM_END</computeroutput>) but before + calling + <computeroutput>BZ2_bzReadClose</computeroutput>.</para></listitem> + +</itemizedlist> + +<para>This mechanism makes it easy to decompress multiple +<computeroutput>bzip2</computeroutput> streams placed end-to-end. +As the end of one stream, when +<computeroutput>BZ2_bzRead</computeroutput> returns +<computeroutput>BZ_STREAM_END</computeroutput>, call +<computeroutput>BZ2_bzReadGetUnused</computeroutput> to collect +the unused data (copy it into your own buffer somewhere). That +data forms the start of the next compressed stream. To start +uncompressing that next stream, call +<computeroutput>BZ2_bzReadOpen</computeroutput> again, feeding in +the unused data via the <computeroutput>unused</computeroutput> / +<computeroutput>nUnused</computeroutput> parameters. Keep doing +this until <computeroutput>BZ_STREAM_END</computeroutput> return +coincides with the physical end of file +(<computeroutput>feof(f)</computeroutput>). In this situation +<computeroutput>BZ2_bzReadGetUnused</computeroutput> will of +course return no data.</para> + +<para>This should give some feel for how the high-level interface +can be used. If you require extra flexibility, you'll have to +bite the bullet and get to grips with the low-level +interface.</para> + +</sect2> + + +<sect2 id="std-rdwr" xreflabel="Standard file-reading/writing code"> +<title>Standard file-reading/writing code</title> + +<para>Here's how you'd write data to a compressed file:</para> + +<programlisting> +FILE* f; +BZFILE* b; +int nBuf; +char buf[ /* whatever size you like */ ]; +int bzerror; +int nWritten; + +f = fopen ( "myfile.bz2", "w" ); +if ( !f ) { + /* handle error */ +} +b = BZ2_bzWriteOpen( &bzerror, f, 9 ); +if (bzerror != BZ_OK) { + BZ2_bzWriteClose ( b ); + /* handle error */ +} + +while ( /* condition */ ) { + /* get data to write into buf, and set nBuf appropriately */ + nWritten = BZ2_bzWrite ( &bzerror, b, buf, nBuf ); + if (bzerror == BZ_IO_ERROR) { + BZ2_bzWriteClose ( &bzerror, b ); + /* handle error */ + } +} + +BZ2_bzWriteClose( &bzerror, b ); +if (bzerror == BZ_IO_ERROR) { + /* handle error */ +} +</programlisting> + +<para>And to read from a compressed file:</para> + +<programlisting> +FILE* f; +BZFILE* b; +int nBuf; +char buf[ /* whatever size you like */ ]; +int bzerror; +int nWritten; + +f = fopen ( "myfile.bz2", "r" ); +if ( !f ) { + /* handle error */ +} +b = BZ2_bzReadOpen ( &bzerror, f, 0, NULL, 0 ); +if ( bzerror != BZ_OK ) { + BZ2_bzReadClose ( &bzerror, b ); + /* handle error */ +} + +bzerror = BZ_OK; +while ( bzerror == BZ_OK && /* arbitrary other conditions */) { + nBuf = BZ2_bzRead ( &bzerror, b, buf, /* size of buf */ ); + if ( bzerror == BZ_OK ) { + /* do something with buf[0 .. nBuf-1] */ + } +} +if ( bzerror != BZ_STREAM_END ) { + BZ2_bzReadClose ( &bzerror, b ); + /* handle error */ +} else { + BZ2_bzReadClose ( &bzerror, b ); +} +</programlisting> + +</sect2> + +</sect1> + + +<sect1 id="util-fns" xreflabel="Utility functions"> +<title>Utility functions</title> + + +<sect2 id="bzbufftobuffcompress" xreflabel="BZ2_bzBuffToBuffCompress"> +<title>BZ2_bzBuffToBuffCompress</title> + +<programlisting> +int BZ2_bzBuffToBuffCompress( char* dest, + unsigned int* destLen, + char* source, + unsigned int sourceLen, + int blockSize100k, + int verbosity, + int workFactor ); +</programlisting> + +<para>Attempts to compress the data in <computeroutput>source[0 +.. sourceLen-1]</computeroutput> into the destination buffer, +<computeroutput>dest[0 .. *destLen-1]</computeroutput>. If the +destination buffer is big enough, +<computeroutput>*destLen</computeroutput> is set to the size of +the compressed data, and <computeroutput>BZ_OK</computeroutput> +is returned. If the compressed data won't fit, +<computeroutput>*destLen</computeroutput> is unchanged, and +<computeroutput>BZ_OUTBUFF_FULL</computeroutput> is +returned.</para> + +<para>Compression in this manner is a one-shot event, done with a +single call to this function. The resulting compressed data is a +complete <computeroutput>bzip2</computeroutput> format data +stream. There is no mechanism for making additional calls to +provide extra input data. If you want that kind of mechanism, +use the low-level interface.</para> + +<para>For the meaning of parameters +<computeroutput>blockSize100k</computeroutput>, +<computeroutput>verbosity</computeroutput> and +<computeroutput>workFactor</computeroutput>, see +<computeroutput>BZ2_bzCompressInit</computeroutput>.</para> + +<para>To guarantee that the compressed data will fit in its +buffer, allocate an output buffer of size 1% larger than the +uncompressed data, plus six hundred extra bytes.</para> + +<para><computeroutput>BZ2_bzBuffToBuffDecompress</computeroutput> +will not write data at or beyond +<computeroutput>dest[*destLen]</computeroutput>, even in case of +buffer overflow.</para> + +<para>Possible return values:</para> + +<programlisting> +BZ_CONFIG_ERROR + if the library has been mis-compiled +BZ_PARAM_ERROR + if dest is NULL or destLen is NULL + or blockSize100k < 1 or blockSize100k > 9 + or verbosity < 0 or verbosity > 4 + or workFactor < 0 or workFactor > 250 +BZ_MEM_ERROR + if insufficient memory is available +BZ_OUTBUFF_FULL + if the size of the compressed data exceeds *destLen +BZ_OK + otherwise +</programlisting> + +</sect2> + + +<sect2 id="bzbufftobuffdecompress" xreflabel="BZ2_bzBuffToBuffDecompress"> +<title>BZ2_bzBuffToBuffDecompress</title> + +<programlisting> +int BZ2_bzBuffToBuffDecompress( char* dest, + unsigned int* destLen, + char* source, + unsigned int sourceLen, + int small, + int verbosity ); +</programlisting> + +<para>Attempts to decompress the data in <computeroutput>source[0 +.. sourceLen-1]</computeroutput> into the destination buffer, +<computeroutput>dest[0 .. *destLen-1]</computeroutput>. If the +destination buffer is big enough, +<computeroutput>*destLen</computeroutput> is set to the size of +the uncompressed data, and <computeroutput>BZ_OK</computeroutput> +is returned. If the compressed data won't fit, +<computeroutput>*destLen</computeroutput> is unchanged, and +<computeroutput>BZ_OUTBUFF_FULL</computeroutput> is +returned.</para> + +<para><computeroutput>source</computeroutput> is assumed to hold +a complete <computeroutput>bzip2</computeroutput> format data +stream. +<computeroutput>BZ2_bzBuffToBuffDecompress</computeroutput> tries +to decompress the entirety of the stream into the output +buffer.</para> + +<para>For the meaning of parameters +<computeroutput>small</computeroutput> and +<computeroutput>verbosity</computeroutput>, see +<computeroutput>BZ2_bzDecompressInit</computeroutput>.</para> + +<para>Because the compression ratio of the compressed data cannot +be known in advance, there is no easy way to guarantee that the +output buffer will be big enough. You may of course make +arrangements in your code to record the size of the uncompressed +data, but such a mechanism is beyond the scope of this +library.</para> + +<para><computeroutput>BZ2_bzBuffToBuffDecompress</computeroutput> +will not write data at or beyond +<computeroutput>dest[*destLen]</computeroutput>, even in case of +buffer overflow.</para> + +<para>Possible return values:</para> + +<programlisting> +BZ_CONFIG_ERROR + if the library has been mis-compiled +BZ_PARAM_ERROR + if dest is NULL or destLen is NULL + or small != 0 && small != 1 + or verbosity < 0 or verbosity > 4 +BZ_MEM_ERROR + if insufficient memory is available +BZ_OUTBUFF_FULL + if the size of the compressed data exceeds *destLen +BZ_DATA_ERROR + if a data integrity error was detected in the compressed data +BZ_DATA_ERROR_MAGIC + if the compressed data doesn't begin with the right magic bytes +BZ_UNEXPECTED_EOF + if the compressed data ends unexpectedly +BZ_OK + otherwise +</programlisting> + +</sect2> + +</sect1> + + +<sect1 id="zlib-compat" xreflabel="zlib compatibility functions"> +<title>zlib compatibility functions</title> + +<para>Yoshioka Tsuneo has contributed some functions to give +better <computeroutput>zlib</computeroutput> compatibility. +These functions are <computeroutput>BZ2_bzopen</computeroutput>, +<computeroutput>BZ2_bzread</computeroutput>, +<computeroutput>BZ2_bzwrite</computeroutput>, +<computeroutput>BZ2_bzflush</computeroutput>, +<computeroutput>BZ2_bzclose</computeroutput>, +<computeroutput>BZ2_bzerror</computeroutput> and +<computeroutput>BZ2_bzlibVersion</computeroutput>. These +functions are not (yet) officially part of the library. If they +break, you get to keep all the pieces. Nevertheless, I think +they work ok.</para> + +<programlisting> +typedef void BZFILE; + +const char * BZ2_bzlibVersion ( void ); +</programlisting> + +<para>Returns a string indicating the library version.</para> + +<programlisting> +BZFILE * BZ2_bzopen ( const char *path, const char *mode ); +BZFILE * BZ2_bzdopen ( int fd, const char *mode ); +</programlisting> + +<para>Opens a <computeroutput>.bz2</computeroutput> file for +reading or writing, using either its name or a pre-existing file +descriptor. Analogous to <computeroutput>fopen</computeroutput> +and <computeroutput>fdopen</computeroutput>.</para> + +<programlisting> +int BZ2_bzread ( BZFILE* b, void* buf, int len ); +int BZ2_bzwrite ( BZFILE* b, void* buf, int len ); +</programlisting> + +<para>Reads/writes data from/to a previously opened +<computeroutput>BZFILE</computeroutput>. Analogous to +<computeroutput>fread</computeroutput> and +<computeroutput>fwrite</computeroutput>.</para> + +<programlisting> +int BZ2_bzflush ( BZFILE* b ); +void BZ2_bzclose ( BZFILE* b ); +</programlisting> + +<para>Flushes/closes a <computeroutput>BZFILE</computeroutput>. +<computeroutput>BZ2_bzflush</computeroutput> doesn't actually do +anything. Analogous to <computeroutput>fflush</computeroutput> +and <computeroutput>fclose</computeroutput>.</para> + +<programlisting> +const char * BZ2_bzerror ( BZFILE *b, int *errnum ) +</programlisting> + +<para>Returns a string describing the more recent error status of +<computeroutput>b</computeroutput>, and also sets +<computeroutput>*errnum</computeroutput> to its numerical +value.</para> + +</sect1> + + +<sect1 id="stdio-free" + xreflabel="Using the library in a stdio-free environment"> +<title>Using the library in a stdio-free environment</title> + + +<sect2 id="stdio-bye" xreflabel="Getting rid of stdio"> +<title>Getting rid of stdio</title> + +<para>In a deeply embedded application, you might want to use +just the memory-to-memory functions. You can do this +conveniently by compiling the library with preprocessor symbol +<computeroutput>BZ_NO_STDIO</computeroutput> defined. Doing this +gives you a library containing only the following eight +functions:</para> + +<para><computeroutput>BZ2_bzCompressInit</computeroutput>, +<computeroutput>BZ2_bzCompress</computeroutput>, +<computeroutput>BZ2_bzCompressEnd</computeroutput> +<computeroutput>BZ2_bzDecompressInit</computeroutput>, +<computeroutput>BZ2_bzDecompress</computeroutput>, +<computeroutput>BZ2_bzDecompressEnd</computeroutput> +<computeroutput>BZ2_bzBuffToBuffCompress</computeroutput>, +<computeroutput>BZ2_bzBuffToBuffDecompress</computeroutput></para> + +<para>When compiled like this, all functions will ignore +<computeroutput>verbosity</computeroutput> settings.</para> + +</sect2> + + +<sect2 id="critical-error" xreflabel="Critical error handling"> +<title>Critical error handling</title> + +<para><computeroutput>libbzip2</computeroutput> contains a number +of internal assertion checks which should, needless to say, never +be activated. Nevertheless, if an assertion should fail, +behaviour depends on whether or not the library was compiled with +<computeroutput>BZ_NO_STDIO</computeroutput> set.</para> + +<para>For a normal compile, an assertion failure yields the +message:</para> + +<blockquote> +<para>bzip2/libbzip2: internal error number N.</para> +<para>This is a bug in bzip2/libbzip2, &bz-version; of &bz-date;. +Please report it to me at: &bz-email;. If this happened +when you were using some program which uses libbzip2 as a +component, you should also report this bug to the author(s) +of that program. Please make an effort to report this bug; +timely and accurate bug reports eventually lead to higher +quality software. Thanks. Julian Seward, &bz-date;. +</para></blockquote> + +<para>where <computeroutput>N</computeroutput> is some error code +number. If <computeroutput>N == 1007</computeroutput>, it also +prints some extra text advising the reader that unreliable memory +is often associated with internal error 1007. (This is a +frequently-observed-phenomenon with versions 1.0.0/1.0.1).</para> + +<para><computeroutput>exit(3)</computeroutput> is then +called.</para> + +<para>For a <computeroutput>stdio</computeroutput>-free library, +assertion failures result in a call to a function declared +as:</para> + +<programlisting> +extern void bz_internal_error ( int errcode ); +</programlisting> + +<para>The relevant code is passed as a parameter. You should +supply such a function.</para> + +<para>In either case, once an assertion failure has occurred, any +<computeroutput>bz_stream</computeroutput> records involved can +be regarded as invalid. You should not attempt to resume normal +operation with them.</para> + +<para>You may, of course, change critical error handling to suit +your needs. As I said above, critical errors indicate bugs in +the library and should not occur. All "normal" error situations +are indicated via error return codes from functions, and can be +recovered from.</para> + +</sect2> + +</sect1> + + +<sect1 id="win-dll" xreflabel="Making a Windows DLL"> +<title>Making a Windows DLL</title> + +<para>Everything related to Windows has been contributed by +Yoshioka Tsuneo +(<computeroutput>tsuneo@rr.iij4u.or.jp</computeroutput>), so +you should send your queries to him (but perhaps Cc: me, +<computeroutput>&bz-email;</computeroutput>).</para> + +<para>My vague understanding of what to do is: using Visual C++ +5.0, open the project file +<computeroutput>libbz2.dsp</computeroutput>, and build. That's +all.</para> + +<para>If you can't open the project file for some reason, make a +new one, naming these files: +<computeroutput>blocksort.c</computeroutput>, +<computeroutput>bzlib.c</computeroutput>, +<computeroutput>compress.c</computeroutput>, +<computeroutput>crctable.c</computeroutput>, +<computeroutput>decompress.c</computeroutput>, +<computeroutput>huffman.c</computeroutput>, +<computeroutput>randtable.c</computeroutput> and +<computeroutput>libbz2.def</computeroutput>. You will also need +to name the header files <computeroutput>bzlib.h</computeroutput> +and <computeroutput>bzlib_private.h</computeroutput>.</para> + +<para>If you don't use VC++, you may need to define the +proprocessor symbol +<computeroutput>_WIN32</computeroutput>.</para> + +<para>Finally, <computeroutput>dlltest.c</computeroutput> is a +sample program using the DLL. It has a project file, +<computeroutput>dlltest.dsp</computeroutput>.</para> + +<para>If you just want a makefile for Visual C, have a look at +<computeroutput>makefile.msc</computeroutput>.</para> + +<para>Be aware that if you compile +<computeroutput>bzip2</computeroutput> itself on Win32, you must +set <computeroutput>BZ_UNIX</computeroutput> to 0 and +<computeroutput>BZ_LCCWIN32</computeroutput> to 1, in the file +<computeroutput>bzip2.c</computeroutput>, before compiling. +Otherwise the resulting binary won't work correctly.</para> + +<para>I haven't tried any of this stuff myself, but it all looks +plausible.</para> + +</sect1> + +</chapter> + + + +<chapter id="misc" xreflabel="Miscellanea"> +<title>Miscellanea</title> + +<para>These are just some random thoughts of mine. Your mileage +may vary.</para> + + +<sect1 id="limits" xreflabel="Limitations of the compressed file format"> +<title>Limitations of the compressed file format</title> + +<para><computeroutput>bzip2-1.0.X</computeroutput>, +<computeroutput>0.9.5</computeroutput> and +<computeroutput>0.9.0</computeroutput> use exactly the same file +format as the original version, +<computeroutput>bzip2-0.1</computeroutput>. This decision was +made in the interests of stability. Creating yet another +incompatible compressed file format would create further +confusion and disruption for users.</para> + +<para>Nevertheless, this is not a painless decision. Development +work since the release of +<computeroutput>bzip2-0.1</computeroutput> in August 1997 has +shown complexities in the file format which slow down +decompression and, in retrospect, are unnecessary. These +are:</para> + +<itemizedlist mark='bullet'> + + <listitem><para>The run-length encoder, which is the first of the + compression transformations, is entirely irrelevant. The + original purpose was to protect the sorting algorithm from the + very worst case input: a string of repeated symbols. But + algorithm steps Q6a and Q6b in the original Burrows-Wheeler + technical report (SRC-124) show how repeats can be handled + without difficulty in block sorting.</para></listitem> + + <listitem><para>The randomisation mechanism doesn't really need to be + there. Udi Manber and Gene Myers published a suffix array + construction algorithm a few years back, which can be employed + to sort any block, no matter how repetitive, in O(N log N) + time. Subsequent work by Kunihiko Sadakane has produced a + derivative O(N (log N)^2) algorithm which usually outperforms + the Manber-Myers algorithm.</para> + + <para>I could have changed to Sadakane's algorithm, but I find + it to be slower than <computeroutput>bzip2</computeroutput>'s + existing algorithm for most inputs, and the randomisation + mechanism protects adequately against bad cases. I didn't + think it was a good tradeoff to make. Partly this is due to + the fact that I was not flooded with email complaints about + <computeroutput>bzip2-0.1</computeroutput>'s performance on + repetitive data, so perhaps it isn't a problem for real + inputs.</para> + + <para>Probably the best long-term solution, and the one I have + incorporated into 0.9.5 and above, is to use the existing + sorting algorithm initially, and fall back to a O(N (log N)^2) + algorithm if the standard algorithm gets into + difficulties.</para></listitem> + + <listitem><para>The compressed file format was never designed to be + handled by a library, and I have had to jump though some hoops + to produce an efficient implementation of decompression. It's + a bit hairy. Try passing + <computeroutput>decompress.c</computeroutput> through the C + preprocessor and you'll see what I mean. Much of this + complexity could have been avoided if the compressed size of + each block of data was recorded in the data stream.</para></listitem> + + <listitem><para>An Adler-32 checksum, rather than a CRC32 checksum, + would be faster to compute.</para></listitem> + +</itemizedlist> + +<para>It would be fair to say that the +<computeroutput>bzip2</computeroutput> format was frozen before I +properly and fully understood the performance consequences of +doing so.</para> + +<para>Improvements which I was able to incorporate into 0.9.0, +despite using the same file format, are:</para> + +<itemizedlist mark='bullet'> + + <listitem><para>Single array implementation of the inverse BWT. This + significantly speeds up decompression, presumably because it + reduces the number of cache misses.</para></listitem> + + <listitem><para>Faster inverse MTF transform for large MTF values. + The new implementation is based on the notion of sliding blocks + of values.</para></listitem> + + <listitem><para><computeroutput>bzip2-0.9.0</computeroutput> now reads + and writes files with <computeroutput>fread</computeroutput> + and <computeroutput>fwrite</computeroutput>; version 0.1 used + <computeroutput>putc</computeroutput> and + <computeroutput>getc</computeroutput>. Duh! Well, you live + and learn.</para></listitem> + +</itemizedlist> + +<para>Further ahead, it would be nice to be able to do random +access into files. This will require some careful design of +compressed file formats.</para> + +</sect1> + + +<sect1 id="port-issues" xreflabel="Portability issues"> +<title>Portability issues</title> + +<para>After some consideration, I have decided not to use GNU +<computeroutput>autoconf</computeroutput> to configure 0.9.5 or +1.0.</para> + +<para><computeroutput>autoconf</computeroutput>, admirable and +wonderful though it is, mainly assists with portability problems +between Unix-like platforms. But +<computeroutput>bzip2</computeroutput> doesn't have much in the +way of portability problems on Unix; most of the difficulties +appear when porting to the Mac, or to Microsoft's operating +systems. <computeroutput>autoconf</computeroutput> doesn't help +in those cases, and brings in a whole load of new +complexity.</para> + +<para>Most people should be able to compile the library and +program under Unix straight out-of-the-box, so to speak, +especially if you have a version of GNU C available.</para> + +<para>There are a couple of +<computeroutput>__inline__</computeroutput> directives in the +code. GNU C (<computeroutput>gcc</computeroutput>) should be +able to handle them. If you're not using GNU C, your C compiler +shouldn't see them at all. If your compiler does, for some +reason, see them and doesn't like them, just +<computeroutput>#define</computeroutput> +<computeroutput>__inline__</computeroutput> to be +<computeroutput>/* */</computeroutput>. One easy way to do this +is to compile with the flag +<computeroutput>-D__inline__=</computeroutput>, which should be +understood by most Unix compilers.</para> + +<para>If you still have difficulties, try compiling with the +macro <computeroutput>BZ_STRICT_ANSI</computeroutput> defined. +This should enable you to build the library in a strictly ANSI +compliant environment. Building the program itself like this is +dangerous and not supported, since you remove +<computeroutput>bzip2</computeroutput>'s checks against +compressing directories, symbolic links, devices, and other +not-really-a-file entities. This could cause filesystem +corruption!</para> + +<para>One other thing: if you create a +<computeroutput>bzip2</computeroutput> binary for public distribution, +please consider linking it statically (<computeroutput>gcc +-static</computeroutput>). This avoids all sorts of library-version +issues that others may encounter later on.</para> + +<para>If you build <computeroutput>bzip2</computeroutput> on +Win32, you must set <computeroutput>BZ_UNIX</computeroutput> to 0 +and <computeroutput>BZ_LCCWIN32</computeroutput> to 1, in the +file <computeroutput>bzip2.c</computeroutput>, before compiling. +Otherwise the resulting binary won't work correctly.</para> + +</sect1> + + +<sect1 id="bugs" xreflabel="Reporting bugs"> +<title>Reporting bugs</title> + +<para>I tried pretty hard to make sure +<computeroutput>bzip2</computeroutput> is bug free, both by +design and by testing. Hopefully you'll never need to read this +section for real.</para> + +<para>Nevertheless, if <computeroutput>bzip2</computeroutput> dies +with a segmentation fault, a bus error or an internal assertion +failure, it will ask you to email me a bug report. Experience from +years of feedback of bzip2 users indicates that almost all these +problems can be traced to either compiler bugs or hardware +problems.</para> + +<itemizedlist mark='bullet'> + + <listitem><para>Recompile the program with no optimisation, and + see if it works. And/or try a different compiler. I heard all + sorts of stories about various flavours of GNU C (and other + compilers) generating bad code for + <computeroutput>bzip2</computeroutput>, and I've run across two + such examples myself.</para> + + <para>2.7.X versions of GNU C are known to generate bad code + from time to time, at high optimisation levels. If you get + problems, try using the flags + <computeroutput>-O2</computeroutput> + <computeroutput>-fomit-frame-pointer</computeroutput> + <computeroutput>-fno-strength-reduce</computeroutput>. You + should specifically <emphasis>not</emphasis> use + <computeroutput>-funroll-loops</computeroutput>.</para> + + <para>You may notice that the Makefile runs six tests as part + of the build process. If the program passes all of these, it's + a pretty good (but not 100%) indication that the compiler has + done its job correctly.</para></listitem> + + <listitem><para>If <computeroutput>bzip2</computeroutput> + crashes randomly, and the crashes are not repeatable, you may + have a flaky memory subsystem. + <computeroutput>bzip2</computeroutput> really hammers your + memory hierarchy, and if it's a bit marginal, you may get these + problems. Ditto if your disk or I/O subsystem is slowly + failing. Yup, this really does happen.</para> + + <para>Try using a different machine of the same type, and see + if you can repeat the problem.</para></listitem> + + <listitem><para>This isn't really a bug, but ... If + <computeroutput>bzip2</computeroutput> tells you your file is + corrupted on decompression, and you obtained the file via FTP, + there is a possibility that you forgot to tell FTP to do a + binary mode transfer. That absolutely will cause the file to + be non-decompressible. You'll have to transfer it + again.</para></listitem> + +</itemizedlist> + +<para>If you've incorporated +<computeroutput>libbzip2</computeroutput> into your own program +and are getting problems, please, please, please, check that the +parameters you are passing in calls to the library, are correct, +and in accordance with what the documentation says is allowable. +I have tried to make the library robust against such problems, +but I'm sure I haven't succeeded.</para> + +<para>Finally, if the above comments don't help, you'll have to +send me a bug report. Now, it's just amazing how many people +will send me a bug report saying something like:</para> + +<programlisting> +bzip2 crashed with segmentation fault on my machine +</programlisting> + +<para>and absolutely nothing else. Needless to say, a such a +report is <emphasis>totally, utterly, completely and +comprehensively 100% useless; a waste of your time, my time, and +net bandwidth</emphasis>. With no details at all, there's no way +I can possibly begin to figure out what the problem is.</para> + +<para>The rules of the game are: facts, facts, facts. Don't omit +them because "oh, they won't be relevant". At the bare +minimum:</para> + +<programlisting> +Machine type. Operating system version. +Exact version of bzip2 (do bzip2 -V). +Exact version of the compiler used. +Flags passed to the compiler. +</programlisting> + +<para>However, the most important single thing that will help me +is the file that you were trying to compress or decompress at the +time the problem happened. Without that, my ability to do +anything more than speculate about the cause, is limited.</para> + +</sect1> + + +<sect1 id="package" xreflabel="Did you get the right package?"> +<title>Did you get the right package?</title> + +<para><computeroutput>bzip2</computeroutput> is a resource hog. +It soaks up large amounts of CPU cycles and memory. Also, it +gives very large latencies. In the worst case, you can feed many +megabytes of uncompressed data into the library before getting +any compressed output, so this probably rules out applications +requiring interactive behaviour.</para> + +<para>These aren't faults of my implementation, I hope, but more +an intrinsic property of the Burrows-Wheeler transform +(unfortunately). Maybe this isn't what you want.</para> + +<para>If you want a compressor and/or library which is faster, +uses less memory but gets pretty good compression, and has +minimal latency, consider Jean-loup Gailly's and Mark Adler's +work, <computeroutput>zlib-1.2.1</computeroutput> and +<computeroutput>gzip-1.2.4</computeroutput>. Look for them at +<ulink url="http://www.zlib.org">http://www.zlib.org</ulink> and +<ulink url="http://www.gzip.org">http://www.gzip.org</ulink> +respectively.</para> + +<para>For something faster and lighter still, you might try Markus F +X J Oberhumer's <computeroutput>LZO</computeroutput> real-time +compression/decompression library, at +<ulink url="http://www.oberhumer.com/opensource">http://www.oberhumer.com/opensource</ulink>.</para> + +</sect1> + + + +<sect1 id="reading" xreflabel="Further Reading"> +<title>Further Reading</title> + +<para><computeroutput>bzip2</computeroutput> is not research +work, in the sense that it doesn't present any new ideas. +Rather, it's an engineering exercise based on existing +ideas.</para> + +<para>Four documents describe essentially all the ideas behind +<computeroutput>bzip2</computeroutput>:</para> + +<literallayout>Michael Burrows and D. J. Wheeler: + "A block-sorting lossless data compression algorithm" + 10th May 1994. + Digital SRC Research Report 124. + ftp://ftp.digital.com/pub/DEC/SRC/research-reports/SRC-124.ps.gz + If you have trouble finding it, try searching at the + New Zealand Digital Library, http://www.nzdl.org. + +Daniel S. Hirschberg and Debra A. LeLewer + "Efficient Decoding of Prefix Codes" + Communications of the ACM, April 1990, Vol 33, Number 4. + You might be able to get an electronic copy of this + from the ACM Digital Library. + +David J. Wheeler + Program bred3.c and accompanying document bred3.ps. + This contains the idea behind the multi-table Huffman coding scheme. + ftp://ftp.cl.cam.ac.uk/users/djw3/ + +Jon L. Bentley and Robert Sedgewick + "Fast Algorithms for Sorting and Searching Strings" + Available from Sedgewick's web page, + www.cs.princeton.edu/~rs +</literallayout> + +<para>The following paper gives valuable additional insights into +the algorithm, but is not immediately the basis of any code used +in bzip2.</para> + +<literallayout>Peter Fenwick: + Block Sorting Text Compression + Proceedings of the 19th Australasian Computer Science Conference, + Melbourne, Australia. Jan 31 - Feb 2, 1996. + ftp://ftp.cs.auckland.ac.nz/pub/peter-f/ACSC96paper.ps</literallayout> + +<para>Kunihiko Sadakane's sorting algorithm, mentioned above, is +available from:</para> + +<literallayout>http://naomi.is.s.u-tokyo.ac.jp/~sada/papers/Sada98b.ps.gz +</literallayout> + +<para>The Manber-Myers suffix array construction algorithm is +described in a paper available from:</para> + +<literallayout>http://www.cs.arizona.edu/people/gene/PAPERS/suffix.ps +</literallayout> + +<para>Finally, the following papers document some +investigations I made into the performance of sorting +and decompression algorithms:</para> + +<literallayout>Julian Seward + On the Performance of BWT Sorting Algorithms + Proceedings of the IEEE Data Compression Conference 2000 + Snowbird, Utah. 28-30 March 2000. + +Julian Seward + Space-time Tradeoffs in the Inverse B-W Transform + Proceedings of the IEEE Data Compression Conference 2001 + Snowbird, Utah. 27-29 March 2001. +</literallayout> + +</sect1> + +</chapter> + +</book> |