1.. SPDX-License-Identifier: 0BSD
 2
 3============================
 4XZ data compression in Linux
 5============================
 6
 7Introduction
 8============
 9
10XZ is a general purpose data compression format with high compression
11ratio. The XZ decompressor in Linux is called XZ Embedded. It supports
12the LZMA2 filter and optionally also Branch/Call/Jump (BCJ) filters
13for executable code. CRC32 is supported for integrity checking.
14
15See the `XZ Embedded`_ home page for the latest version which includes
16a few optional extra features that aren't required in the Linux kernel
17and information about using the code outside the Linux kernel.
18
19For userspace, `XZ Utils`_ provide a zlib-like compression library
20and a gzip-like command line tool.
21
22.. _XZ Embedded: https://tukaani.org/xz/embedded.html
23.. _XZ Utils: https://tukaani.org/xz/
 
 
24
25XZ related components in the kernel
26===================================
27
28The xz_dec module provides XZ decompressor with single-call (buffer
29to buffer) and multi-call (stateful) APIs in include/linux/xz.h.
 
 
 
 
 
 
 
 
30
31For decompressing the kernel image, initramfs, and initrd, there
32is a wrapper function in lib/decompress_unxz.c. Its API is the
33same as in other decompress_*.c files, which is defined in
34include/linux/decompress/generic.h.
35
36For kernel makefiles, three commands are provided for use with
37``$(call if_changed)``. They require the xz tool from XZ Utils.
38
39- ``$(call if_changed,xzkern)`` is for compressing the kernel image.
40  It runs the script scripts/xz_wrap.sh which uses arch-optimized
41  options and a big LZMA2 dictionary.
42
43- ``$(call if_changed,xzkern_with_size)`` is like ``xzkern`` above but
44  this also appends a four-byte trailer containing the uncompressed size
45  of the file. The trailer is needed by the boot code on some archs.
46
47- Other things can be compressed with ``$(call if_needed,xzmisc)``
48  which will use no BCJ filter and 1 MiB LZMA2 dictionary.
49
50Notes on compression options
51============================
52
53Since the XZ Embedded supports only streams with CRC32 or no integrity
54check, make sure that you don't use some other integrity check type
55when encoding files that are supposed to be decoded by the kernel.
56With liblzma from XZ Utils, you need to use either ``LZMA_CHECK_CRC32``
57or ``LZMA_CHECK_NONE`` when encoding. With the ``xz`` command line tool,
58use ``--check=crc32`` or ``--check=none`` to override the default
59``--check=crc64``.
60
61Using CRC32 is strongly recommended unless there is some other layer
62which will verify the integrity of the uncompressed data anyway.
63Double checking the integrity would probably be waste of CPU cycles.
64Note that the headers will always have a CRC32 which will be validated
65by the decoder; you can only change the integrity check type (or
66disable it) for the actual uncompressed data.
67
68In userspace, LZMA2 is typically used with dictionary sizes of several
69megabytes. The decoder needs to have the dictionary in RAM:
70
71- In multi-call mode the dictionary is allocated as part of the
72  decoder state. The reasonable maximum dictionary size for in-kernel
73  use will depend on the target hardware: a few megabytes is fine for
74  desktop systems while 64 KiB to 1 MiB might be more appropriate on
75  some embedded systems.
76
77- In single-call mode the output buffer is used as the dictionary
78  buffer. That is, the size of the dictionary doesn't affect the
79  decompressor memory usage at all. Only the base data structures
80  are allocated which take a little less than 30 KiB of memory.
81  For the best compression, the dictionary should be at least
82  as big as the uncompressed data. A notable example of single-call
83  mode is decompressing the kernel itself (except on PowerPC).
84
85The compression presets in XZ Utils may not be optimal when creating
86files for the kernel, so don't hesitate to use custom settings to,
87for example, set the dictionary size. Also, xz may produce a smaller
88file in single-threaded mode so setting that explicitly is recommended.
89Example::
90
91    xz --threads=1 --check=crc32 --lzma2=dict=512KiB inputfile
92
93xz_dec API
94==========
95
96This is available with ``#include <linux/xz.h>``.
97
98.. kernel-doc:: include/linux/xz.h
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

 
 
  1============================
  2XZ data compression in Linux
  3============================
  4
  5Introduction
  6============
  7
  8XZ is a general purpose data compression format with high compression
  9ratio and relatively fast decompression. The primary compression
 10algorithm (filter) is LZMA2. Additional filters can be used to improve
 11compression ratio even further. E.g. Branch/Call/Jump (BCJ) filters
 12improve compression ratio of executable data.
 13
 14The XZ decompressor in Linux is called XZ Embedded. It supports
 15the LZMA2 filter and optionally also BCJ filters. CRC32 is supported
 16for integrity checking. The home page of XZ Embedded is at
 17<https://tukaani.org/xz/embedded.html>, where you can find the
 18latest version and also information about using the code outside
 19the Linux kernel.
 20
 21For userspace, XZ Utils provide a zlib-like compression library
 22and a gzip-like command line tool. XZ Utils can be downloaded from
 23<https://tukaani.org/xz/>.
 24
 25XZ related components in the kernel
 26===================================
 27
 28The xz_dec module provides XZ decompressor with single-call (buffer
 29to buffer) and multi-call (stateful) APIs. The usage of the xz_dec
 30module is documented in include/linux/xz.h.
 31
 32The xz_dec_test module is for testing xz_dec. xz_dec_test is not
 33useful unless you are hacking the XZ decompressor. xz_dec_test
 34allocates a char device major dynamically to which one can write
 35.xz files from userspace. The decompressed output is thrown away.
 36Keep an eye on dmesg to see diagnostics printed by xz_dec_test.
 37See the xz_dec_test source code for the details.
 38
 39For decompressing the kernel image, initramfs, and initrd, there
 40is a wrapper function in lib/decompress_unxz.c. Its API is the
 41same as in other decompress_*.c files, which is defined in
 42include/linux/decompress/generic.h.
 43
 44scripts/xz_wrap.sh is a wrapper for the xz command line tool found
 45from XZ Utils. The wrapper sets compression options to values suitable
 46for compressing the kernel image.
 47
 48For kernel makefiles, two commands are provided for use with
 49$(call if_needed). The kernel image should be compressed with
 50$(call if_needed,xzkern) which will use a BCJ filter and a big LZMA2
 51dictionary. It will also append a four-byte trailer containing the
 52uncompressed size of the file, which is needed by the boot code.
 53Other things should be compressed with $(call if_needed,xzmisc)
 54which will use no BCJ filter and 1 MiB LZMA2 dictionary.
 
 
 55
 56Notes on compression options
 57============================
 58
 59Since the XZ Embedded supports only streams with no integrity check or
 60CRC32, make sure that you don't use some other integrity check type
 61when encoding files that are supposed to be decoded by the kernel. With
 62liblzma, you need to use either LZMA_CHECK_NONE or LZMA_CHECK_CRC32
 63when encoding. With the xz command line tool, use --check=none or
 64--check=crc32.
 
 65
 66Using CRC32 is strongly recommended unless there is some other layer
 67which will verify the integrity of the uncompressed data anyway.
 68Double checking the integrity would probably be waste of CPU cycles.
 69Note that the headers will always have a CRC32 which will be validated
 70by the decoder; you can only change the integrity check type (or
 71disable it) for the actual uncompressed data.
 72
 73In userspace, LZMA2 is typically used with dictionary sizes of several
 74megabytes. The decoder needs to have the dictionary in RAM, thus big
 75dictionaries cannot be used for files that are intended to be decoded
 76by the kernel. 1 MiB is probably the maximum reasonable dictionary
 77size for in-kernel use (maybe more is OK for initramfs). The presets
 78in XZ Utils may not be optimal when creating files for the kernel,
 79so don't hesitate to use custom settings. Example::
 80
 81	xz --check=crc32 --lzma2=dict=512KiB inputfile
 82
 83An exception to above dictionary size limitation is when the decoder
 84is used in single-call mode. Decompressing the kernel itself is an
 85example of this situation. In single-call mode, the memory usage
 86doesn't depend on the dictionary size, and it is perfectly fine to
 87use a big dictionary: for maximum compression, the dictionary should
 88be at least as big as the uncompressed data itself.
 
 
 
 
 
 
 
 
 89
 90Future plans
 91============
 
 
 92
 93Creating a limited XZ encoder may be considered if people think it is
 94useful. LZMA2 is slower to compress than e.g. Deflate or LZO even at
 95the fastest settings, so it isn't clear if LZMA2 encoder is wanted
 96into the kernel.
 97
 98Support for limited random-access reading is planned for the
 99decompression code. I don't know if it could have any use in the
100kernel, but I know that it would be useful in some embedded projects
101outside the Linux kernel.
102
103Conformance to the .xz file format specification
104================================================
105
106There are a couple of corner cases where things have been simplified
107at expense of detecting errors as early as possible. These should not
108matter in practice all, since they don't cause security issues. But
109it is good to know this if testing the code e.g. with the test files
110from XZ Utils.
111
112Reporting bugs
113==============
114
115Before reporting a bug, please check that it's not fixed already
116at upstream. See <https://tukaani.org/xz/embedded.html> to get the
117latest code.
118
119Report bugs to <lasse.collin@tukaani.org> or visit #tukaani on
120Freenode and talk to Larhzu. I don't actively read LKML or other
121kernel-related mailing lists, so if there's something I should know,
122you should email to me personally or use IRC.
123
124Don't bother Igor Pavlov with questions about the XZ implementation
125in the kernel or about XZ Utils. While these two implementations
126include essential code that is directly based on Igor Pavlov's code,
127these implementations aren't maintained nor supported by him.