LTFS consistency and Index snapshots

An LTFS Volume must be in a consistent state when the volume is exchanged with another LTFS system. The LTFS Format Specification defines consistent state as:

“A volume is consistent when both partitions are complete and the last Index Construct in the Index Partition has a back pointer to the last Index Construct in the Data Partition.”

where a complete parition is:

“An LTFS partition that consists of an LTFS Label Construct and a Content Area, where the last construct in the Content Area is an Index Construct.”

The LTFS Format Specification additionally includes the following conformance statement:

“Recorded media claiming conformance to this format shall be in a consistent state when interchanged or stored.”

In practice this conformance statement means that when an LTFS Volume is ejected from an LTO drive the volume must be consistent. Software implementing support for the LTFS format can expect a consistent LTFS cartridge when the cartridge is loaded and reject inconsistent cartridges.

Logical layout of a consistent LTFS volume showing historical Indexes with back-pointers and multiple files written and edited in-place.

The LTFS software writes an updated Index to the end of the Data Partition (DP). The Indexes stored in the DP are interleaved with the file content that is also stored in the DP. LTFS maintains each Index written to the DP for three core reasons:

• data safety,
• performance, and
• snapshots/rollback.

Data safety
To protect the data stored on the media it is vital that LTFS commit an updated Index to the media soon after the file content is written. Without the updated Index, new file content cannot be accessed. Additionally, maintaining the current Index in the DP along with the same logical Index stored in the Index Partition (IP) provides a redundant copy of the Index on the media. These redundant copies mean that in the event of catastrophic failure there is a second chance to read a valid current Index.

Performance
In normal LTFS operation the tape head is positioned in the DP to allow rapid access the file content storage area to service read and write requests. When a new Index needs to be written to the media the LTFS software will write the new Index to the DP and continue servicing any concurrent write operations. By writing the Index to the DP in-line with the file content the LTFS software does not incur a penalty for switching to the other partition and seeking back to the Beginning of Tape (BOT).

Snapshots and rollback
The Indexes written to the DP are likely to be followed by file content during write operations. LTO data tape is sequential media which makes it impossible to delete these Indexes without destroying any file content data that is written after the Index. Maintaining these Indexes provides the ability to inspect and potentially revert the LTFS Volume to any earlier time for which an Index exists on the media. For example, if you work with an LTFS Volume by adding, updating, and deleting files. If you later decide that these changes are not wanted then you can either mount the LTFS Volume read-only using an earlier Index to show and access an earlier version of the volume, or you can permanently revert the LTFS Volume to the an earlier state using a rollback operation.

The Index written to the IP is typically written during unmount processing of the LTFS file-system. During unmount the following operations are performed in order:

1. all file content written to the volume but still in memory is flushed to the DP,
2. an updated Index is written to the DP,
3. LTO cartridge memory is written with new MAM parameters,
4. the tape head is moved to the IP,
5. any file content destined for the IP is written overwriting the old Index,
6. a replica of the current Index is written to the IP overwriting the old Index,
7. LTO cartridge memory is written with new MAM parameters, and
8. media is rewound then optionally ejected.

Note that it is safe to overwrite the old Index on the IP because the LTFS software has successfully written the current Index at the end of the DP in operation #2. The details of how MAM parameters are used and the conditions in which file content may be written to the IP are described in a later blog post.