LTFS Podcasts

Podcast

In many ways the Linear Tape File System (LTFS) is my baby. I was the architect of the development work and a primary author of the LTFS Format Specification. I have spent a good deal of time advocating for LTFS including my posts about LTFS on this blog.

Last year I was offered an opportunity to bring more of the LTFS story to light. Jay Livens (Director, Product & Solutions Marketing, Iron Mountain) asked me if I would record some podcasts with him about the development of LTFS. Over the course of a few months we recorded 3 interviews with each session focused on a different aspect of the LTFS development; the Creation, the Technology, and the Future.

The podcasts that came out of these interviews were published today on the Iron Mountain website.

Posted in architecture, LTFS | Leave a comment

stealth startup ⥤ StackStorm

A couple of months ago I left my position at SanDisk to join a hush-hush stealth startup. The experience has alternated between feeling weird, and the most normal thing in the world. Weird when friends and former colleagues ask what I’m working on and I feel like a shifty agent from a three-letter agency. “We work on, you know making things better.” Normal because this is the valley and stealth is just part of the industry.

I’ve always felt that this was the right move for me and so far it’s been great.

Today we are announcing the company to the world…

StackStorm Logo

StackStorm is an emerging leader of the third wave of operations automation. Built from the ground up with DevOps in mind, StackStorm’s vision is a world of self-driving data centers that learn over time how to better operate themselves. StackStorm solutions leverage existing configuration management and monitoring solutions to deliver automation that safely ties together today’s loosely coupled, heterogeneous, ever changing cloud infrastructures.

ic-linkedin ic-facebook

Posted in michael | Leave a comment

Bridging ExpressCard to PCIe for LTO

In the mid-1990’s I worked as a photoshop jockey for the Sydney Morning Herald and other Fairfax Media newspapers. At that time we would regularly send a photographer into the field with a large Pelican case that held a digital field kit.

The kit consisted of  a maxed out Powerbook 180 (Greyscale 9.8″ screen, 14MB RAM, 33MHz M68030 CPU, 56.7kbps modem) and a Nikon LS-1000 film scanner with associated cables and power supplies. The kit allowed a photographer to shoot film, process the film at a local mini-lab in the field, scan the images, and transfer them to the newspaper office. In practice this allowed comparatively rather rapid delivery of photos into the editorial process.

During my work on LTFS I messed around with a bunch of side projects on the periphery of LTFS. In one of these projects tried to devise a portable system that would support LTFS. My thinking was largely modeled on my earlier experience with the Powerbook-based field kit. I wanted to be able to give someone a portable box that could be used with a laptop and a digital camera that could be used as a videographer field unit. My idea was something that could be carried to any location, would be easy to set up, and allowed the user to dump content onto LTFS tapes and FedEx the tapes back to home base.

As a demonstration unit I assembled an enclosure that held a LTO5 drive, a SAS HBA, and a PCIe to ExpressCard bridge. This enclosure ended up having the footprint of a 15″ Powerbook and was a little taller than a LTO half-high drive. I showed the demo unit at several trade shows and other public events we did for LTFS.

LTO/LTFS portable drive at NAB 2010LTO/LTFS portable drive at NAB 2010

While refining this portable LTFS box I played with a few different ExpressCard->PCIe bridges and some early thunderbolt->PCIe bridges. A recent question on the Mac Enterprise mailing list asked:

“One route is to go with a TB to PCI chassis then SCSI card to LTO but what little I have found says this is not reliable. Ditto TB to SAS then to SAS LTO. Anyone have any experience with setups like this and with what software?

Tape is pretty much a given. This is a situation where people need to be able to go back in time and pick up a file or folder state from 3 months ago.”

My work building this LTFS portable drive provided me with somewhat relevant experience. My answer to the mailing list may be useful for a broader audience so I’ve included it below:

My experience using both thunderbolt->PCIe and ExpressCard->PCIe bridges has largely been successful. I used the bridges with both SAS and Fibre Channel HBAs to connect LTO4 and LTO5 drives to a few different OS X machines (and some Dell and Lenovo systems). In all cases I found the ExpressCard/Thunderbolt->PCIe bridges reliable.

The few problems I encountered were always traced back to the HBAs and/or LTO drive not supporting or recognizing various sleep state events. (Or OS X not reinitializing the bus after waking up from sleep which amounts to the same thing.)

Fortunately working around this sleep issue is trivial. Just disable system sleep. Allow the monitor to go to sleep if you like, but ensure that the OS X system does not sleep.

If you are going with LTO tape in the scenario you briefly describe, you will need to work out how you want to handle the file meta-data. (Filename, timestamps, folders, permissions, etc.) Traditional tape systems maintain separate parallel databases to hold the metadata. This works, but if the database gets corrupted then all of your tapes suddenly hold garbage.

A more recent approach that is possible with LTO5 and LTO6 is to use LTFS. LTFS is a POSIX-compliant filesystem that uses LTO tape as the primary filesystem datastore. This means the file meta-data is stored on the tape right alongside the file content. As a result, the tape is self-contained and can be mounted exactly like a USB drive can be mounted. Data can be added to the LTFS volume by dragging and dropping files. Restoring data is also a drag and drop operation.

Alternatively, since LTFS is POSIX compliant applications can seek to any byte offset within a file in LTFS. In comparison, traditional systems require a parallel database, or a packaging format like “tar”. Packaging formats typically require an extensive tape traverse during restore. Whereas LTFS supports a direct seek to the data being accessed. LTFS also provides file-system versioning baked into the filesystem. The file-system versioning might be of use in your scenario.

The implementation of LTFS for OS X, Linux, and Windows is free for single tape-drive operation.

Posted in data storage, demo, file-system, hardware, LTFS, open source, os x | Leave a comment

Atari 2600 hardware mod

This week, much of the gaming world focused the release of GTA V from Rockstar Games. I found some spare time to get an old Atari system cleaned up and working by installing an atari 2600 composite output mod.

After 20+ years in a closet our well loved Atari 2600 was a little worse for wear. The console has accumulated grime and the reset switch is no-longer springy. Besides cleaning off the dust and replacing a faulty reset switch I wanted to improve the video output.

The stock Atari 2600 produces an RF modulated video/audio signal as output. On old TVs the RF signal connects to the antenna-in jack. On modern TVs it is often easier to hook up a composite video input. Eliminating the RF modulation should also improve the picture quality.

A few different hardware hacks that produce composite out from an Atari are described on various websites. The output board I selected is one of the better composite output circuit designs.

Parts list

Original console:

While dismantling I found the remains of a previous home-made repair – an elastic band to add springiness to the reset switch. The original spring seems to have gone missing over the years. While the console was dismantled I replaced the reset switch with a new old-stock part.

After installation:
atari 2600 composite output board installed

The video mod board is in the lower right with the rainbow ribbon cable going from the motherboard (inside the thick aluminium RF shield box) to the video mod then a second ribbon cable running up the right side to the composite and s-video output jacks. A couple of pieces of kapton tape hold the video mod board securely in place.

New ports:

Results

Photograph of Pole Position game on Atari 2600 heavy-sixer console before hardware modification. (Photograph of LCD TV screen.)
Pole Position on Atari 2600 before modification

Photograph of Pole Position game on Atari 2600 heavy-sixer console after hardware modification.
Pole Position on atari 2600 composite output after installing video mod board

Overall the mod is a resounding success. RF noise is eliminated, image colors have better saturation, and the console can be connected to standard composite input jacks thereby eliminating the need to re-tune the TV. Installing the video mod involved bypassing the RF modulator and removing the original coax cable.

Basic testing with Pole Position and Pac-Mac during reassembly showed normal operation. I’ll update this post when I’ve tested with other games.

Posted in atari 2600, gaming, hardware | Leave a comment

Create a FreeBSD 9.0 machine image under Linux

This is a guide to building a virtual machine image for KVM that runs FreeBSD with virtio drivers installed.

Requirements:

Notes:

Commands and to be performed under Linux are shown in green.
Commands and file edits to be performed under the FreeBSD guest OS are shown in purple.

Create empty disk image (10GB):

  •  On your Linux host, run the command:
dd if=/dev/null of=freebsd-9_0.raw bs=1024 seek=10485760

Boot FreeBSD installer and install:

  • On your Linux host, run the command:
sudo kvm -vnc 0.0.0.0:54 -drive file=freebsd-9_0.raw,if=ide -net nic -net user -cdrom FreeBSD-9.0-RELEASE-amd64-disc1.iso
  • Connect to the guest OS using a VNC client pointed to port 5954. (5900 + 54 from command above.)
  • Step through the FreeBSD installer:
    • allocate the whole virtual drive to FreeBSD.
    • Configure networking to use IPv4 and DHCP.
  • Reboot the FreeBSD guest.

Install Virt IO drivers:

  • Download the binary package from http://people.freebsd.org/~kuriyama/virtio/
  • Install the package by running the command “pkg_add virtio-kmod-9-0.239473.tbz” in the guest OS.
  • Edit /boot/loader.conf to add the following lines:
virtio_load="YES"
virtio_pci_load="YES"
virtio_blk_load="YES"
if_vtnet_load="YES"
  • Edit /etc/fstab in the guest OS to change the device identifiers to “/dev/vtbd*“. (The following example shows the original device entries commented out followed by the virtio entries.)
# Device Mountpoint FStype Options Dump Pass#
#/dev/ada0p2 / ufs rw 1 1
#/dev/ada0p3 none swap sw 0 0
 /dev/vtbd0p2 / ufs rw 1 1
 /dev/vtbd0p3 none swap sw 0 0
  • Edit /etc/rc.conf in the guest OS to add the line shown in bold. (This line configures FreeBSD to use DHCP when bringing up the “vtnet0” network interface.):
hostname="freebsd-90"
ifconfig_re0="DHCP"
 ifconfig_vtnet0="DHCP"
sshd_enable="YES"
# Set dumpdev to "AUTO" to enable crash dumps, "NO" to disable
dumpdev="NO"
  • Power off the guest.
  • Relaunch using virtio for the virtual hard drive and virtual network adapter using the following KVM command:
sudo kvm -vnc 0.0.0.0:54 -drive file=freebsd-9_0.raw,if=virtio -net nic,model=virtio -net user
  • Verify correct hard drive and network operation in the guest OS.
  • If the hard drive and network appear to be working correctly then power off the FreeBSD guest.

VirtIO in FreeBSD setup resources:

Posted in operating systems, virtualization | Leave a comment

Changing the size of an image in Photoshop CS6

An image file such as a JPEG or PNG file contains two types of data. The image essence and metadata.

The image essence is what is usually thought of as the image, i.e. it is colours and the position of those colors in the picture. When an image file is viewed, the image essence is unpacked into a two-dimensional array of color values in memory. Typically each pixel is represented as a four-value tuple: (red, green, blue, alpha). Each element in the tuple can have a value from 0 to 255 and the “alpha” value indicates the level of transparency for the pixel.

The metadata can hold information such as a description of the photo, copyright information, camera details, and other fields. Additionally, the metadata contains information about the dimensions of the image. The dimensions are represented in two different formats. These formats are referred to as the “Pixel Dimensions” and the “Document Size” in Photoshop.

The “Pixel Dimensions” is the size of the image measured in pixels in the X and Y directions. These pixel-based dimensions allow the image essence to be mapped from a linear array (such as a file) to a two-dimensional array in memory. The Pixel Dimensions size determines the size of the image when viewed in the majority of software because software is typically written to show one image pixel for every screen pixel.

The “Document Size” dimensions format consists of three values: (width, height, resolution). The width and height are be represented in Photoshop as linear units (mm, inches, etc) or as relative units (percentage of original). The resolution value is measured in pixels/inch (or pixels/cm). The Document Size dimensions are typically most important when printing an image. If the pixels/inch value matches the resolution of your printer (dots/inch) then the image on the printout will be width by height in size. However, if the resolution of your printer is twice that of your image you are likely to get a printed image that is half the height and half the width of the “Document Size” values. The reason is that that without any specific request for scaling, the printer will map one image pixel to one dot of output.

To resize an image in photoshop, open the image and select the “Image Size” tool from the “Image menu. In Photoshop CS6 this produces the dialog shown below:

Photoshop CS6 Image Size Dialog

Changing the Pixel Dimensions will change the size of the image as measured in pixels. The checkboxes at the bottom of this dialog control which numeric values are locked to the same relationship as in the original image.

Resolution values specific to the output or viewing device. Typical values for various devices are:

    • old monitor: 72 pixels/inch (common old publishing assumption)
    • average monitor: ~120 pixels/inch
    • low-end printer: 300 dots/inch
    • typical printer: 600 dots/inch
    • Retina laptop: 220 pixels/inch

As with most Photoshop dialogs, holding down the option-key will change the “Cancel” button to a “Reset” button. The reset button will revert the settings in the dialog back to the original values from the image.

The Image Size dialog is best used to change the size of an image without adding or removing any image content[1. Other posts describe the related “Canvas Tool” (for adding to images without scaling) and the “Cropping Tool” (to remove parts of the image.)]. That is, the Image Size tool is used to scale an image up or down.

Posted in photoshop | Leave a comment

And the Emmy goes to…    LTFS

IBM_LTO5_media

IBM LTO5 data tape media.


OMFG! My project won an Emmy!

The Academy of Television Arts & Sciences has announced the recipients of the 63rd Primetime Emmy® Engineering Awards. This year, one of four Engineering Emmys will be awarded one of which goes to IBM for the LTFS technology.

I was responsible for the architecture, format specification, and development lead for the Linear Tape File System (LTFS) from the start of the product development. During my work on LTFS I was instrumental in producing LTFS Single Drive Edition for Linux, OS X and Windows in 2010 (all platforms released within 12 months of starting the project); producing LTFS Library Edition in 2011; and laying plans for future LTFS products up to my last day at IBM.

omgbbqwtf!! Over.

“I’d like to thank the Academy, …”

Posted in award, emmy, ibm, LTFS, Uncategorized | 7 Comments

joining Nimbula

After popping the IBM stack frame off my call stack the next step is to push a new Nimbula frame onto the stack. Today is the end of my first week working at a startup and, as expected, the experience is dramatically different from working at IBM.

I will need to come up to speed before I can talk intelligently about what Nimbula is doing and my role in the company. Right now, after a week I can only comment on the simplicity of the on-boarding process.

Continue reading

Posted in michael, nimbula | Leave a comment

leaving IBM

Today was my last day at IBM Almaden. I’ve been walking into the same building 5 days a week for about 9 years now. It’ll be weird to not do the same next week.

In my time here, I have shipped 4 different LTFS releases, numerous LTFS fixpacks, 3 LTFS Format Specifications, the first Information Archive appliance release, several internal projects, along with multiple papers and patents. I’ve worked with some extremely talented people and have helped an industry change the way they store and manage their data.

I will be taking some personal time before starting in my new role. I will continue posting my brain dumps about LTFS to capture as much information as I can share publicly. Over time I expect to post more about the new space in which I will be working.

Posted in blog, ibm, michael, Uncategorized | 1 Comment

LTFS Format Specification and Open-Source

In an earlier post I mentioned that prior to the introduction of LTFS an application typically relied on proprietary APIs to read and write data tape. These proprietary APIs typically have used proprietary data formats on the tape media. These proprietary APIs and data formats lock an application developer to a specific layer of software which mediates access to the data tape storage.

With LTFS we took a very different approach. The LTFS filesystem-based approach to accessing data stored on tape brings significant benefits to the tape market. For example, application developers no longer need to learn specialized APIs, and users get a fully self-describing storage medium that largely behaves the same as more common storage media such as hard-disk drives and memory sticks.

The benefits afforded by LTFS are arguably sufficient to support releasing LTFS as proprietary software that uses a closed data format on the tape media. Rather than treating LTFS as yet another proprietary on-tape format, the LTFS team at IBM elected to release the LTFS Single Drive Edition (LTFS-SDE) software as open-source[1. Open-source release of LTFS-SDE is available for Linux and Mac OS X. The LTFS-SDE implementation for MS Windows is freely available as a binary release. The MS Windows implementation could not be released as open-source due to licensing terms on necessary third-party components]. The LTFS open-source software release is licensed under the LGPL license to protect the LTFS format while providing application developers the ability to dynamically link with LTFS binary code without risking their own intellectual property. Each fix-pack released for LTFS-SDE has also been released under the LGPL.

In addition to the open-source software release, the LTFS team wrote the LTFS Format Specification document and released this specification to the public. This Format Specification describes the on-media data structures in sufficient detail for independent developers to read and write LTFS tapes that are compatible with other LTFS implementations.

During development of the LTFS software and the LTFS Format Specification the IBM team built a working relationship with developers at both HP and Quantum through the LTO Consortium. These relationships afforded an opportunity to inform HP and Quantum about the up-coming LTFS technology and allowed for some pre-release co-ordination between these three LTO 5 vendors. These on-going relationships have provided the opportunity to conduct inter-operablity testing between all three vendors. We are fortunate to enjoy continuing discussions between vendors on the direction and development of the LTFS format.

This communication, the open-specification, and the open-source approach to software distribution is motivated by a desire for LTFS to become an industry-wide standard. During LTFS development we recognized that if LTFS was a proprietary release then we would be adding to the fragmentation and incompatibility of data tape use. Instead, a release of LTFS as an open implementation with an open specification would create an environment that may lead to broad industry adoption. Broad adoption along with the ease of use benefits may result in expanded use of tape media for data storage. Expansion of the tape market would be good for the industry and all of us involved in the data tape business.

Posted in data storage, LTFS, open source, specification | 4 Comments