The Linear Tape File System (LTFS) relies on support for partitioning was introduced in LTO generation 5. Partitioning a LTO5 cartridge divides the media in two separate data storage areas known as “partitions”. Each partition can be written to without impacting data stored in the other partition on the media.
A data tape formatted for use with LTFS has two partitions, an Index Partition (IP) and a Data Partition (DP). The Index Partition has a relatively small capacity of 37.5GB. The Data Partition comprises the remaining 1.43TB of available capacity on the media.
LTFS writes an Index that holds all the file and folder meta-data for the LTFS Volume to the Index Partition. The LTFS Format Specification defines that a consistent LTFS Volume must, along with other properties, have the most-recent Index written to the end of both the IP and the DP. In the diagram below, “Index2” is the most-recent Index. The specification defines that an LTFS Volume must be consistent when it is exchanged with another system. In practice, this means that any compliant implementation of the specification that operates on an LTFS Volume must ensure that the media is consistent when the media is ejected.
At mount-time the LTFS software reads the current Index from the IP and builds an in-memory structure representing all of the folders and files stored on the media. This structure contains meta-data such as file timestamps, file permissions, file name, file size, etc. The structure also contains the location on the DP for each data extent that holds part of the file content.
When a user or an application traverses a mounted Linear Tape File System the LTFS software can return filenames, folders, timestamps, file sizes, and other meta-data from the in-memory index structure. When the user double-clicks on a file to open it, or an application reads from a file the LTFS software causes the tape drive to seek to the start of the relevant data extents and reads the data from the tape media.
LTO5 tape drives are able to stream data for reading and writing at 140MB/s. This speed is about 40% higher than the best hard-drives which top out at around 100MB/s. Of course, since LTO data tape is sequential access media there is a high seek time for data tape. Worst case seek time is moving from one end of the physical tape to the other end. An end-to-end seek will take roughly 90 seconds. So the average seek time is around 45 seconds.
Logical layout of a LTFS Volume on LTO5 media. Beginning of Tape (BOT) at the left edge of diagram, End of Tape (EOT) at the right edge. Diagram is not to scale. BOT to EOT is 846 meters (~2775 feet) and tape width top-to-bottom is 1.27 centimeters (1/2 inch).
LTO5 data tape writes data in a sequence of wraps on the physical media. Each wrap consists of one track of data written in a forward direction along the length of the tape plus another track of data written in a reverse direction along the length of the tape. There are 80 such wraps on LTO5 media. To fully traverse all of the stored data on a full LTO 5 cartridge requires traversing the length of the tape one-hundred and sixty times. (Eighty times in the forward direction interleaved with eighty times in the reverse direction.)
As a result of the wrap-based data layout a seek to the beginning of a randomly selected file is often quite fast. Typically a seek to the beginning of a file will only require a short tape movement along the length of the tape along with a lateral head movement that is perpendicular to the axis of tape movement. These lateral head moves are performed in a few seconds.
Writes to a file stored in LTFS are performed by seeking to the end of the data and writing the file content to the media. If the file already exists in the LTFS volume then only the “over-written” areas of the file are written at the end of the DP. The extent list for such files that are modified “in-place” (on tape media) is updated to insert the new extents into the appropriate offsets in the extent list for the file.
Pingback: sync() behavior in LTFS | Michael Richmond
Pingback: Can you use LTFS on LTO4 or earlier media? | Michael Richmond
Hello,
I’m new in LTFS knowledge, and I’ve a question about using Hards disks instead of cartridges… It seems funny to will to do this with a FS dedicated to tapes… But my purpose would be for storing data which intrinsically sequential (I’m in video security domain). I need to store continuously videos on many disks as fast as possible, and as secure as possible. LTFS seems done for this, and considering hard disks as piece of tape could be valuable…
I would like to have your thoughts about this
regards
Xavier
Hello Xavier,
LTFS cannot be directly applied to a hard-drive without a substantial re-write of the LTFS software implementation. It is not clear why anyone would invest the development effort for this re-write because the LTFS Format would result in substantial performance loss when used on current hard drive technology. (This performance loss is when compared to traditional file-systems and is due to LTFS not taking advantage of the direct random addressing that is not provided by tape media.)
If you need to store streaming video on hard drives as fast as possible, I suggest that you investigate RAID-based hard drive arrays or RAID configurations of flash storage. Depending on the typical video file-size and speed versus storage utilization you might get some additional benefit by tweaking the block-size used by a traditional hard-drive focused filesystem.
Although the user-experience of video data is intrinsically sequential, many video codecs and container formats are decidedly not completely sequential in structure. For example, some video codecs store a frame index and/or a color-map at the end of the video file. The location of this index or color-map requires the end of the file to be read first during any playback operation.
Thanks Michael for this detailed answer!
We already actually use up to now our own RAID ZFS-based solution for storing video… I only thought that LTFS would better match the sequential nature of video storing… but it seemed I was wrong 😉
best regards
xavier
Pingback: Filesystem recovery examples with ltfsck - Michael RichmondMichael Richmond
Hi Michael,
I think I know the answer, hopefully it’s still worth asking: what happens when a .dmg is written to LTFS? Can Snow Leopard/Lion logically mount the .dmg, and then allow random access directly to a file within the disk image? Or more correctly, can the OS mount the .dmg without reading the entire image from tape? I suspect the answer is “no” because current LTFS implementations use a MacFUSE abstraction layer. If/when LTFS support is integrated into the OS, I suspect there might be a mechanism for logically mounting .dmg from tape without reading the entire image file.
The reason I ask is that tapeless media for video has a very specific file/folder structure, so we have advised customers to use .dmg as a foolproof way to lock down and preserve the structure when backing up to tape or disk.
Thanks for your help!
Patrick Gough
Patrick,
I understand your concern over locking down the specific file/folder structure. LTFS does not add additional files to the filesystem, nor does LTFS manipulate the folder structure of the filesystem. The only files and folders stored by LTFS are the files and folders put there by the user. I know of at least one company who have deployed products based on LTFS that copy tapeless media folders straight to LTFS without any problem. Specifically, this company is copying the content of P2 cards directly to a folder in an LTFS Volume. The only constraint that the LTFS software places on the files/folders is a handful of characters that are disallowed by LTFS because the characters lead to cross-platform compatibility issues. The disallowed characters are: null; slash; colon; double quotation mark; asterisk; question mark; less than sign; greater than sign; backslash; vertical line; and control codes U+009, U+00A, and U+00D.
With regard to storing the .dmg files and allowing random access, I expect that this will work. I do not recall whether I tested this explicit use-case for LTFS.
The MacFUSE framework allows individual byte-level access to files and does not cause any additional reads other than those requested by applications accessing the filesystem. However, the Mac OS X default behavior of verifying the .dmg file prior to mounting the disk image is likely to be a full read of the fisk image. You can skip the verify step by clicking the ⊗ button in the disk image mount dialog box. Otherwise, the disk image verification can be disabled at the system level by issuing the following terminal command:
I recommend that you only do this if you fully understand the implications of disabling the default behavior.
With regard to integration of LTFS into the OS, I assume you are referring to an in-kernel implementation of LTFS. In my opinion an in-kernel implementation of LTFS is very unlikely. The over-head incurred by use of FUSE and FUSE-like frameworks is negligible based on our testing of LTFS on Mac OS X, Linux, and Windows. Additionally, LTFS uses XML data structures for the filesystem meta-data. Parsing XML data in kernel-space is not a pleasant idea from a technical perspective. Given that the current user-space implementation does not suffer from performance problems or other limitations due to the user-space implementation, I do not personally see what value would be achieved by an in-kernel implementation. Particularly given the engineering complexity of performing such an implementation.
Regards,
Michael Richmond
hey Michael,
we just bought an LTO drive in order to archive our company work. most of the material is video files and final cut projects, what would be the best way to archive these onto the LTO tapes. i have been told that using the ltfs system will not accept my files but the person who told me this does not seem to understand a lot about this issue. is that true?
i am not a technical person but i try to understand how it works to get the best solution for my company.
one more thing, the new tapes are 1.5 tb – 3 tb, can we really compress video tapes?
i apologize if my question is very basic.
The LTFS filesystem will natively store any files that are copied to LTFS formatted tapes. The LTFS software package includes formatting tools that will allow you to format LTO5 data tapes to the LTFS format. The constraints imposed when copying a file to an LTFS tape is a) the file must be smaller than the available free space on the LTFS volume, and b) the filename should not contain characters that are illegal or disallowed in the LTFS filesystem.
The illegal characters are ‘/’ (forward slash), ‘:’ (colon), and null. The disallowed characters are ‘*’ (asterisk), ‘?’ (question mark), ‘”‘ (double quotation mark), ” (greater than sign, ‘\’ (backslash), ‘|’ (vertical line), and the control codes U+0009, U+000A, U+000D. The illegal characters are often unusable in filenames on any filesystem. The disallowed characters are characters that we identified during LTFS development as causing problems for cross-platform compatibility of LTFS volumes. We made the decision to block use of these characters in our software to avoid problems when using LTFS volumes on different platforms.
LTFS5 tapes support 1.5TB of raw, uncompressed storage. After formatting with LTFS there is about 1.47TB of available storage on the tape. Average video files do not gain much benefit from compression because the video data is very dense in common file formats. I have seen video get some benefit from compression, but that benefit is on the order of ~5-10%.
Some tape vendors choose to market their tapes as LTO5 with 1.5TB of storage. Other tape vendors choose to quote the “1.5TB – 3.0TB” of storage where the high number assumes the data is 2:1 compressible.
Hi Michael,
I am trying to make some diagrams about LTO5 and LTFS and got a little confused about the math. You state that each wrap consists of 2 tracks (one in each direction) and there are 80 wraps on the tape, which means that there are 160 tracks per tape. Yet wiki says that there are 1280 tracks per tape, which means that every wrap should consist of 16 tracks… What’s the catch?
And another question: In your “Partitioning in LTO5 and LTFS” article you state that the data capacity of the wrap is 37.5 GB. If you multiply 37.5 with 80 you get roughly 3TB, when it should be 1.5 TB according to specs… How is that possible?
Thanks and regards,
In my post I am using the word “track” in a loose sense to indicate data that is written in one direction. According to Wikipedia, “When the first data band is filled (they are actually filled in 3, 1, 0, 2 order across the tape), the head assembly is moved to the second data band and a new set of wraps is written. The total number of tracks on the tape is (4 data bands) × (11–20 wraps per band) × (8 or 16 tracks per wrap).” Taking those numbers, 4 data bands * 20 wraps per band = 80 wraps per tape.
My statement about the wrap size of 37.5GB is in error. The correct statement is that the Index Partition and the guard wrap decrease available storage by roughly 37.5GB. Since the Index Partition occupies a single wrap and the guard wrap occupies another wrap. The correct math would be 37.5GB divided by 2 wraps = ~18.75GB per wrap. These numbers will vary in real use due to media quality and data compression.
Hi Michael, you seem to have an intricate knowledge of the LTFS filesystem.
I have been trying to write an LTFS tape/volume of Avid MXF files (Mac OSX 10.6.4)
For performance I consolidate the media files onto an internal HDD before writing to the LTFS volume.
I was initially successful with small test copies but now when I copy 900+ MB to the volume I am getting a “disk full” error when I know there is at least another 400MB free on the volume.
I have tried both command-line and Finder copies, both get the same result.
I have eliminated any illegal characters.
It’s almost like the filesystem is running out of inodes (if it was a disk filesystem).
The files are not unusually small, and it’s only about 6000 files.
I am now creating a .spareseimage file on the consolidating HDD and plan on writing that to LTFS but I shouldn’t have to go to those lengths to create a tape.
Thanks
Matt
Sorry Michael, I’ve just been reading more of your site – you are indeed THE GURU of LTFS.
Thanks Matt. As dev lead and architect for the Linux, OS X, and Windows releases; author of the OS X implementation; and a primary author of the LTFS Format Specification, I know a fair bit about LTFS inside and out.
Did you identify the cause of your copy problem? I know that we tested LTFS for OS X for well over 6000 files on a tape. However you are using OS X 10.6.4. At the time I left IBM, MacFUSE did not officially support Snow Leopard and I have seen some data corruption issues if MacFUSE is used on 10.6.x. As such, LTFS was not supported on OS X versions more recent than 10.5.x.
I haven’t found anything out of spec, I’m using the HP LTFS binaries and they specify 10.6 (32bit mode) as a compatible OS with MacFUSE 2.0.3,2.
They also specify 10.6 (64bit mode) might work with MacFUSE 2.2 from Tuxera.
I know you wrote the IBM implementation – is your software hard coded to only work with IBM drives?
I worked directly with HP during the LTFS development. I know that the HP version of LTFS for OS X is a copy of the LTFS codebase I developed at IBM and released as open-source with the addition of code to replace the tape operations code with code specific for the HP drive. That is, the filesystem operations and interaction with MacFUSE remain the same between both IBM and HP versions.
I have worked with the MacFUSE 2.2 code from Tuxera and have still seen intermittent problems with that version of MacFUSE.
At the time that I left the LTFS project, the IBM LTFS software was not hard-coded to only work with IBM drives. However, the IBM codebase defaulted to using the IBM tape backend code.
The LTFS code is implemented in a number of layers with the lowest layer communicating with the tape drive. This lowest layer is known as the “backend”, or “tape backend”. In the IBM version of LTFS, by default the tape backend is the backend that is written specifically for IBM LTO drives.
HP implemented their own backend that is written specifically for the HP LTO drives.
As you are probably aware, the MacFUSE landscape was rather messy a few years ago with the MacFUSE codebase effectively abandoned. The work by Tuxera and more recently “FUSE for OS X” has revived the MacFUSE landscape. I am not aware of anything that would preclude the use of these more recent FUSE versions with LTFS although there may be API changes that would need to be patched.
Hello Michael!
I recently started to take up with LTFS. I read the specification and other docs, but i didn’t get the answer whether is it possible to read the index files (the xml structure) directly, on any platform. Sorry if my question is very basic.
Thanks
Bela
The index data structures are internal to the filesystem and not directly accessible on any platform. This would be the equivalent of reading the inodes directly on a unix filesystem.
LTFS is implemented as a library so the library may be used to parse and access the LTFS index data structures.
Hi, Michael.
IBM released Version 1.2.5 in March, which includes the support of OS X 10.7.
Now it runs on
– Redhat Enterprise Linux 5.5, 5.6, 5.7, and 6.1
– SUSE Linux Enterprise Server 11 SP1
– Mac OS X 10.5.8 (32-bit)
– Mac OS X 10.6.8 (32-bit)
– Mac OS X 10.7.2 (32-bit and 64-bit)
– Windows 7 SP1 (32-bit and 64-bit)
– Windows Server 2008 R2 SP1 (64-bit)
Michael,
First, thanks for all of your work on LTFS. From the vast research I did before settling on an archiving solution it became clear that LTFS was the only viable non-proprietary option besides maybe TAR. I recently picked up an HP 3000 series sas lto 5 drive and have it running on an atto sas card under osx 10.6.7… I know you worked for IBM and did mainly software, but maybe you can answer these more hardware questions I have.
Basically, I read a post while researching from a good source that certain brands of lto-5 media could cause drive issues if the media was used primarily for archiving because the tape was too brittle or hard. This person recommended HP tape stock only. Unfortunately I could only find sony tape stock so thats what I am testing with.
After getting LTFS installed and macfuse (64bit) I formatted a tape. I noticed in the diagnostics log window (HP gui) a lot of entries that seemed like errors (expected some received this etc). The tape formatted and i could mount it so I didnt think much of it. When I tried to copy files however they transfer went terribly slow, like 3MB/s and i noticed tons of entries coming into the log similar to the ones above. I canceled the copy and actually reformatted the tape. I noticed this time it gave no errors and when I copied files they copied at the speed of the input device (60MB/s) also with no errors. Eventually I decided to reformat the tape to test again and I saw the same errors as the first time. The HP software started to seem buggy and unresponsive so after an unmount I rebooted. This time the format worked perfectly and files are copying at the right speed.
I guess my question is: Does this seem more like a problem with Tuxeras macfuse or some other software issue? Or does this seem like a media issue? Have you seen anything like this before? I can probably dig out logs if that helps. Any information you might have would be greatly appreciated.
Thanks much,
-m
If you are running LTFS with the log settings turned up to debug or trace levels you are likely to see significant performance decrease.
I have not used the Tuxeras MacFuse or HP drives so I cannot comment specifically about those parts of your stack. To eliminate your drive from the equation you could use the file backend for LTFS. This would allow you to perform some tests to (hopefully) establish consistent speeds.
Another really quick follow up… Im noticing a significant speed drop (60 down to 40MB/S) when archiving directories that contain say 10,000 files that are 8MB each. This is compared to large multi-GB files. Is this an inherent limitation of LTFS (or LTO-5)? Or could there be a different issue here?
Thanks,
M
Manu,
The speed drop could be explained by LTFS writing index structures periodically during your copy. This would depend on the options that you are using when mounting the tape. LTFS supports multiple options that control when an index is written to the tape. One option allows a time-based setting so that an index is written every X minutes if there are changes in the index. Another option configures the filesystem to write an index each time a file is closed. A third option configures LTFS to write an index only when the tape is unmounted.
For raw speed, writing an index only when the tape is unmounted will ensure that 100% of the potential tape speed is used to write file data. But this mode has the risk that if your computer crashes before unmounting the tape then the content written to LTFS will be lost because the index is only stored in memory until it is written to tape.
I think the default is to write an index each time a file is closed. But for archiving a large number of small files then this means that there will potentially be a large number of index updates to the tape.
I have a question, how many MB are there in a full wrap in a LTO5 media? I mean, how many data is written in a forward direction and in a reverse direction along the length of the tape?
The answer is provided in a different blog post:
There is approximately 37.5GB in each wrap. I say approximately because media errors (such as due to old, worn, or marginal media) can cause less data to be written to a wrap. This figure is also for uncompressed data. If hardware compression is used then potentially much more data may be stored per wrap.
Hi Michael
I want write a Program that Read/Write files on tape with LTFS Format in .Net
It is logical for me to implement LTFS in my program ?
And Compliance with laws on the LTFS I can do to implement it?
thanks
If you are only interested in reading and writing files to an LTFS tape then all you need to implement is normal file routines. That is, open a file (specifying a path to the LTFS volume), then write data to the file, and close the file. These are exactly the same steps as you would use to write a file to a hard drive.
Similarly, reading a file from LTFS involves the same steps you would used to read a file stored on a hard drive. This is the purpose of implementing LTFS as a filesystem.
Thank you very much for LTFS posts Michael, very informative.
Eric
Mac OS X 10.6.8
IBM TS2250 Tape Drive
ATTO ExpressSAS H680
LTFS version 1.2.5 w/ OSXFUSE 2.5.2
I hope I’m not necro-ing, but this thread has been the best source of information I have found so hopefully asking here is ok.
Since the 2:0 compression ratio written on the Cartridge is just an estimate, and Windows explorer reports the size of the drive as the native, how do you access it? This may seem like a dense question, but do you just select say 2.3TB of data and drag it onto your LTFS drive and hope it fits? This seems kind of counter to good practice. It also seems like, at least with my deployment in Windows 7, to not want to do that. I formatted the drive with compression enabled, but if I go into device manager and look at it, it shows my data as being written “Natively” and the size of the data from source to the tape is identical so I would kind of agree. Am I missing something? is there something I need to do to force compression? is there a better way to predict and or organize how my data will fit?
Sheldon,
I’m not sure that I completely follow your question.
Windows has built-in support for storage systems that use a different amount of space on the media than is occupied by file being stored. A common example of this is sparse files. The LTFS implementation hooks into the standard Windows kernel-level APIs to support this difference in sizes.
It is not clear from your question whether the data you are writing is likely to be compressible data. In all Windows explorer views it is mandatory for the filesystem to report the “complete” file size rather than displaying the space physically occupied on the media. In this text I mean that the “complete file size” is the file size before any benefit from compression. I refer to the “on media size” as the space consumed on the media by the data for that file.
I do not have a Windows machine available to check the “Natively” message that you mention.