Piece format V4

Advanced reference

For developers who read or write GSplit pieces without GSplit.

GSplit 4 writes every piece set in the V4 format described here. A V4 piece is the slice of the original file with a self-describing trailer after it. The trailer carries the set identity, the position of the slice in the original file, checksums, the original name and the file metadata, so that any single piece identifies its set and a reader can check each piece on its own.

Purpose and design goals

Self-describing pieces. Any piece of a set gives the set id, the piece number, the original name, the original size and the naming mask. A joiner can start from any piece and find the others.
Detected damage. A CRC-32 protects the trailer body, another protects the data of each piece, and a SHA-256 protects the whole file. A damaged piece is named when it is read.
No guessing. The trailer ends with a fixed 32-byte locator that gives the length and the checksum of the body. A reader never has to guess where the trailer starts.
Same bytes everywhere. Integers are little-endian and written field by field, never as a memory dump of a record. Times are UTC.
Extensible. Optional extensions in a type, length, value list can be added; readers skip the types they do not know.
Repeatable sizes. The trailer has the same length for every piece of a set, so the size of a piece on disk is known before it is written.

GUnite 3 does not read V4. To join pieces made by GSplit 4, use GUnite 4, gunitec, or GSplit 4. The Self-Uniting app always carries GUnite 4. GSplit 4 never writes the older formats; it reads them, see Legacy piece formats.

Conventions

Type	Meaning
`u8`, `u16`, `u32`	Unsigned integer of 1, 2 or 4 bytes, little-endian.
`i16`, `i64`	Signed two's-complement integer of 2 or 8 bytes, little-endian.
`string`	`u16` length in bytes, then that many UTF-8 bytes. No terminator. Length 0 is the empty string. At most 65535 bytes.
CRC-32	The standard CRC-32 (IEEE 802.3, the one of zlib, PNG and ZIP): reflected polynomial `EDB88320`, initial value `FFFFFFFF`, final inversion. The CRC-32 of the ASCII text `123456789` is `CBF43926`.
SHA-256	The standard SHA-256, stored as its 32 raw bytes.

Piece layout

Every piece has the same shape, from the start of the file to its end.

prefix    prefixLength bytes   optional: byte order mark and/or repeated header lines
data      dataLength bytes     the slice of the original file
suffix    suffixLength bytes   optional: footer lines (text mode)
body      bodyLength bytes     the trailer body, protected by a CRC-32
locator   32 bytes             fixed size, at the very end of the file

The size of a piece file is prefixLength + dataLength + suffixLength + bodyLength + 32. A reader copies exactly dataLength bytes found prefixLength bytes after the start of the file; the prefix and the suffix are skipped. The body is the same length for every piece of a set (same strings, same extensions).

The prefix and the suffix exist for text splitting by lines: GSplit can repeat a header line at the start of each piece and add a footer line at the end. They are not part of the original file. In the first piece, a header typed by the user is its prefix; a byte order mark that belongs to the source file is data, not prefix.

Locator

The locator is the last 32 bytes of the file. Offsets are counted from the end of the file.

Offset from the end	Size	Field	Value
-32	8	`magic`	`47 53 50 4C 49 54 34 00`, the text `GSPLIT4` followed by a zero byte.
-24	2	`formatVersion`	`u16`, 4.
-22	2	`locatorFlags`	`u16`, reserved. Written 0.
-20	4	`bodyLength`	`u32`, length of the body in bytes.
-16	4	`bodyCRC32`	`u32`, CRC-32 of the `bodyLength` bytes of the body.
-12	12	`reserved`	Written as zero bytes.

The body is the bodyLength bytes that end right before the locator.

Body: fixed part

The fixed part is 148 bytes. Offsets are counted from the first byte of the body.

Offset	Size	Field	Type	Meaning
0	16	`setId`	16 bytes	UUID version 4 of the piece set, in RFC 4122 byte order. The same in every piece of the set.
16	4	`pieceIndex`	`u32`	Number of this piece, starting at 1. Never 0.
20	4	`pieceCount`	`u32`	Number of pieces in the set, or 0 when unknown. See Which values go where.
24	4	`flags`	`u32`	Bit field, see Flags.
28	8	`originalOffset`	`i64`	Offset of this piece's data in the original file. Not negative.
36	8	`dataLength`	`i64`	Number of data bytes in this piece. Not negative.
44	4	`prefixLength`	`u32`	Bytes before the data.
48	4	`suffixLength`	`u32`	Bytes between the data and the body.
52	4	`pieceCRC32`	`u32`	CRC-32 of the `dataLength` data bytes of this piece only.
56	4	`runningCRC32`	`u32`	Reserved. Written 0, ignored by readers.
60	8	`originalSize`	`i64`	Size of the original file in bytes. Not negative.
68	4	`fileCRC32`	`u32`	Reserved. Written 0, ignored by readers.
72	32	`fileSHA256`	32 bytes	SHA-256 of the whole original file. Valid only when the `hasFileSha256` flag is set.
104	8	`mtimeSeconds`	`i64`	Modification time of the original: seconds since 1970-01-01 00:00:00 UTC.
112	4	`mtimeNanos`	`u32`	Nanoseconds to add, 0 to 999999999.
116	8	`birthSeconds`	`i64`	Creation time of the original, in seconds since the Unix epoch, UTC. Valid only when `hasBirthTime` is set.
124	4	`birthNanos`	`u32`	Nanoseconds to add, 0 to 999999999.
128	8	`splitTimeSeconds`	`i64`	Time at which the set was made, in seconds since the Unix epoch, UTC.
136	2	`utcOffsetMinutes`	`i16`	Offset from UTC of the source machine at split time, in minutes, east positive. Informative: it lets a reader show the times as the sender saw them.
138	2	`reserved`	`u16`	Written 0.
140	4	`winAttributes`	`u32`	Windows file attributes (`FILE_ATTRIBUTE_*`). Valid only when `hasWinAttr` is set.
144	4	`unixMode`	`u32`	Unix `st_mode` of the original. Valid only when `hasUnixMode` is set.

Flags

Bit	Value	Name	Meaning
0	`0001`	`lastPiece`	This is the last piece of the set.
1	`0002`	`spannedSet`	The set was made disk by disk (spanned set).
2	`0004`	`textMode`	Text mode: the set was made by lines, and a prefix or a suffix may be present.
3	`0008`	`hasFileSha256`	`fileSHA256` is valid.
4	`0010`	reserved	Written 0, ignored when read.
5	`0020`	`hasBirthTime`	`birthSeconds` and `birthNanos` are valid.
6	`0040`	`hasUnixMode`	`unixMode` is valid.
7	`0080`	`hasWinAttr`	`winAttributes` is valid.

All other bits are reserved: write 0, ignore when reading.

Body: strings

Right after the fixed part come six strings, in this order. Each one is a string as defined in Conventions.

Order	Field	Meaning	Writer limit
1	`originalName`	Base name of the original file.	Never shortened. A name longer than 65535 bytes is refused.
2	`mask`	Naming mask of the pieces; it must contain `{num}`.	1024 characters
3	`title`	Title typed by the user.	255 characters
4	`author`	Author typed by the user.	255 characters
5	`description`	Description typed by the user.	4096 characters
6	`setLabel`	Free text label of the set, see Set id and label.	255 characters

A writer that exceeds a limit cuts the text on a UTF-8 character boundary, never inside a character. Limits are counted in characters; the 65535-byte maximum is counted in bytes and is the only limit a reader enforces.

originalName is a base name, never a path. A reader treats it as untrusted text: it keeps only what follows the last / or \, replaces characters the file system refuses, and falls back to the name united (with a warning) for an empty name, ., .., a name with control characters, or a Windows device name. The mask gets the same treatment and falls back to disk{num}.gsd when it holds neither {num} nor {oru}.

Body: extensions

After the strings come the extensions:

u16  count
count times:
    u16  type
    u32  length
    length bytes of value

A reader skips the types it does not know. A body may also carry bytes after the last extension; a reader ignores them, which leaves room for later minor versions.

Type	Name	Value
1	`creator`	UTF-8 text naming the program, for example `GSplit 4.0.0 win64`.
2	`textInfo`	`u16` length of the encoding name, the encoding name in UTF-8 (for example `utf-8`, `utf-16le`, `windows-1252`), then `u8` end-of-line style: 0 unknown, 1 LF, 2 CRLF, 3 CR. Informative.
3	`pieceSHA256`	32 bytes, SHA-256 of the `data` of the piece. Reserved: GSplit 4.0 does not write it.
4	`comment`	UTF-8 free text. Reserved.

Set id and label

setId is a random UUID version 4 chosen when the set is made: the bits of the version nibble are 0100 and the variant bits 10. All pieces of a set carry the same setId. A piece 2 to n belongs to the set when its setId equals the one of piece 1 and its pieceIndex is the expected number. Name, title and mask are not part of the identity, so two sets with the same title are never mixed.

The user can type a "Set unique ID" in GSplit (the command line option --id, the pieceSetID of a project file). The text is handled as follows:

Text typed	`setId`	`setLabel`
A UUID, with or without braces, in any letter case (also 32 hex digits without dashes).	That UUID.	Empty.
Empty, or `none`.	A new random UUID.	Empty.
Any other text.	A new random UUID.	The text, unchanged.

When a program shows a set id as text, it writes 8-4-4-4-12 lowercase hexadecimal digits between braces: {00112233-4455-6677-8899-aabbccddeeff}.

Times

All times in the body are UTC, as a number of seconds since 1970-01-01 00:00:00 UTC (negative before 1970) plus nanoseconds. Windows and Linux file times both convert to this without loss. utcOffsetMinutes is only for display.

A reader refuses a body whose times it cannot turn into a date: seconds outside -62104060800 to 253370764800 (years 2 to 9998), nanoseconds of 1000000000 or more, or an offset beyond 1440 minutes either way.

On joining, the modification time is restored on every system. The creation time and the attributes are restored on Windows, and the permission bits of unixMode (mask 0777, never set-uid) on Unix. Owners, extended attributes and alternate streams are not stored.

Which values go where

Piece 1 is written last. It holds the final pieceCount and the fileSHA256, with the hasFileSha256 flag set. The SHA-256 of the original is computed while the data is read, and is always stored.
Pieces 2 to n are written as the split goes, with pieceCount 0 and fileSHA256 zero (flag clear). On a set that is not spanned, a final pass then rewrites the body and the locator of pieces 2 to n with the final count and the SHA-256, without changing their length, so that every piece carries everything.
Spanned sets (disk by disk) have no final pass: pieceCount may stay 0, the lastPiece flag on the last piece is authoritative, and only piece 1 holds the SHA-256. If piece 1 of a spanned set is damaged, the hash of the result cannot be checked.
lastPiece is set on the last piece, also when its data exactly fills it.
Interrupted final pass. The pieces may then disagree about the count. A reader takes the largest non-zero count it sees. If every piece says 0, it follows the naming mask upward until a file is missing; the last one found must carry lastPiece.
No CRC-32 of the whole file. pieceCRC32 names a damaged piece as soon as it is read, and fileSHA256 checks the whole result; a CRC-32 of the file would repeat the second. runningCRC32, fileCRC32 and flag bit 4 keep their place in the body, written 0 and ignored by readers.
Text mode. prefixLength and suffixLength are how a reader skips repeated header lines, byte order marks and footers.

How a reader finds and checks the trailer

Take the last 32 bytes of the file. A file shorter than 32 bytes is not a V4 piece.
Compare the magic and check that formatVersion is 4. Check that bodyLength is at most the file size minus 32, and at most 16 MiB (16777216). If any test fails, the file is not a V4 piece: try the older formats.
Read the bodyLength bytes before the locator and compute their CRC-32. If it differs from bodyCRC32, the file is a V4 piece with damaged metadata. Report it as damaged; do not fall back to another format.
Decode the body. Reject it when it is shorter than 148 bytes, when a string or an extension ends past the end of the body, when pieceIndex is 0, when a length or offset is negative, or when a time is out of range (see Times).
Check that the file is at least prefixLength + dataLength + suffixLength + bodyLength + 32 bytes. The data are the dataLength bytes at prefixLength from the start.
Compute the CRC-32 of the data and compare it with pieceCRC32.
Check the membership of each piece: same setId as piece 1 and the expected pieceIndex.

To join, write each piece's data at originalOffset of the output. A piece that is missing or refused can then leave zero bytes at its place, and the other pieces keep their own position and their own check. When the output is complete, compute its SHA-256 and compare it with fileSHA256 of piece 1 (or of any piece carrying hasFileSha256). Before writing anything, check every piece at once so that all the problems are reported together.

Any piece can identify the set: originalName, originalSize and mask are in every body. To find piece 1 from another piece, expand the mask with the number 1.

How a writer computes the checksums

Choose setId (see Set id and label) and build the strings and extensions once. They are the same for every piece, so the body has the same length for every piece and the overhead per piece is bodyLength + 32 plus any prefix and suffix.
For each piece, write the prefix, the data and the suffix. Compute pieceCRC32 on the data bytes alone, with the standard CRC-32, starting afresh for each piece.
Compute the SHA-256 of the original file as its data is read.
Fill the per-piece fields: pieceIndex, originalOffset, dataLength, prefixLength, suffixLength, pieceCRC32, and lastPiece when nothing remains.
Encode the body field by field, little-endian, then compute bodyCRC32 as the CRC-32 of the body bytes and write the 32-byte locator.
Write piece 1 last with pieceCount, fileSHA256 and the hasFileSha256 flag. On a set that is not spanned, rewrite pieces 2 to n in place with the same two values.

Worked example

This is the body used by the byte-exact test of the GSplit sources. It describes piece 2 of 3 of a file a.bin of 123456789 bytes: 5 bytes of prefix, 1000 bytes of data that start at offset 0x1122334455667788, and one creator extension GS4. It sets the reserved flag bit 4 and fills the two reserved CRC fields to show that a reader ignores them. Fields are written one per line; the body is all of them in order, without the comments.

00112233445566778899aabbccddeeff   setId
02000000                           pieceIndex 2
03000000                           pieceCount 3
11000000                           flags 0x11: lastPiece and the reserved bit 4
8877665544332211                   originalOffset 0x1122334455667788
e803000000000000                   dataLength 1000
05000000                           prefixLength 5
00000000                           suffixLength 0
2639f4cb                           pieceCRC32 0xCBF43926
efbeadde                           runningCRC32 0xDEADBEEF (reserved)
15cd5b0700000000                   originalSize 123456789
04030201                           fileCRC32 0x01020304 (reserved)
000102030405060708090a0b0c0d0e0f
101112131415161718191a1b1c1d1e1f   fileSHA256
00f1536500000000                   mtimeSeconds 1700000000
e7030000                           mtimeNanos 999
0000000000000000                   birthSeconds 0
00000000                           birthNanos 0
01f1536500000000                   splitTimeSeconds 1700000001
88ff                               utcOffsetMinutes -120
0000                               reserved
20000000                           winAttributes 0x20
a4810000                           unixMode 0x81A4
0500 612e62696e                    originalName "a.bin"
0600 647b6e756d7d                  mask "d{num}"
0000 0000 0000 0000                title, author, description, setLabel (empty)
0100                               1 extension
0100 03000000 475334               type 1 creator, 3 bytes "GS4"

The body is 182 bytes. Its CRC-32 is 0xBE61C02D, so the 32 locator bytes are:

4753504c49543400   magic "GSPLIT4" and a zero byte
0400               formatVersion 4
0000               locatorFlags
b6000000           bodyLength 182
2dc061be           bodyCRC32 0xBE61C02D
000000000000000000000000   reserved

The CRC-32 of the ASCII text 123456789 is 0xCBF43926; this is the test value for a CRC-32 routine.

Files written next to the pieces

GSplit can write three small text files in the folder of the pieces. They are not part of the piece format, and no reader needs them to join a V4 set, but other tools can use them. Each is named after the original file name; when the user chose to rename existing files, GSplit adds the same (k) prefix to them as to the pieces.

SHA-256 manifest

<original name>.sha256 uses the format of sha256sum: one line per piece in piece order, then one line for the original file when its hash is known. A line is the 64 lowercase hexadecimal digits of the SHA-256, two spaces, the file name, and a line feed (LF). The text is UTF-8 without byte order mark. The hash of a piece is that of the whole piece file as finally written, trailer included. A piece whose hash is unknown has no line.

<64 hex digits>  report.pdf.001
<64 hex digits>  report.pdf.002
<64 hex digits>  report.pdf

A reader of this manifest accepts a line of at least 67 characters: 64 hexadecimal digits, a space, a space or * (the binary mark of sha256sum), then the name. It ignores a leading byte order mark, accepts LF or CRLF line ends, ignores a ./ before a name, and compares names without case on Windows. It does not load a manifest larger than 16 MiB.

CRC-32 file

<original name>.crc is a text file in the format of Total Commander. It holds the standard CRC-32 of the original file, which a V4 trailer does not store:

filename=report.pdf
size=1048576
crc32=1A2B3C4D

Lines end with CR LF. crc32 is eight uppercase hexadecimal digits. A reader looks for the keys filename, size and crc32 without case, trims their values, and accepts a $ before the hexadecimal digits.

Join scripts

<original name>.join.bat (Windows, CR LF) and <original name>.join.sh (Linux, LF, #!/bin/sh) rebuild the file from the pieces. For pieces without metadata they concatenate the pieces and print or check the SHA-256 against the manifest; for pieces with metadata they call gunitec. See GUnite and Command line: gunitec.