The Big Event
The other day, I was working with some very old financial archive files on my Mac. I was curious, so I double-clicked one .sit file and nothing at all happened. So I dragged the archive to Stuff Expander 12, the latest version I had, and got this error message:
The Error that got me thinking
Annoyed, but not alarmed, I did some investigating. I tried expanding some other .sit files and they “unstuffed” just fine. So it appeared that I may have had a file that:
- was corrupted
- was missing a needed resource fork (from Mac OS 9 and early Mac OS X days)
- had also been encrypted, maybe by another app
The Big Questions
That got me curious about how many StuffIt archives I had accumulated over the years. A search with PathFinder showed that I had 2,882 .sit files and 44 .sitx files. And then, of course, my next thought was, mindful of the Snow Leopard upgrade I just did on some fairly new Macs, “What if a day comes, because of technical developments, that I can no longer open those archives? Will the current version of StuffIt Expander work all the way back to the beginning of StuffIt technology? What tools might I need to keep current? I decided to find out.
Tracking Down the StuffIt Developer
My next step was to figure out the chronology of the company, Aladdin, and the technology. Recall that Raymond Lau developed StuffIt in 1987 and Aladdin Software was formed to market it. StuffIt quickly became the de facto standard for compressing and archiving files in Mac OS, especially in 1990s when we had relatively limited and expensive disk storage. StuffIt Expander a simple utility to expand archives, was a standard third party add-on for all Macs up until Mac OS X 10.4, Tiger. Its big brother, the full featured StuffIt Deluxe was widely used to create and encrypt archives that could easily be sent via e-mail.
Eventually, Aladdin became Allume, and in 2005, Allume was purchased by Smith Micro Software. I contacted Matthew Covington, Senior Director of Product Development at Smith Micro in Watsonville, CA where the StuffIt team is. Not only did he provide some suggestions on what the problem with my archive file was, but also brought me up to date on the StuffIt technology. Here’s what I learned from Mr. Covington:
Recovering Old Knowledge for Data Migration
1. “The current version of StuffIt Expander, which is StuffIt 2009, is able to open archives created by any prior version of StuffIt. This covers the previous history of StuffIt archives ( i.e. the .sit file format) as well as the newer generation of StuffIt X archives (.sitx file format). Note that Expander is able to open a whole slew of other file formats in addition to the ones that we’ve created — for example the Expander can open zip archives, including encrypted zip archives and zip archives created with newer extensions to the Zip format that WinZip and others have created in recent years that the Finder cannot.”
In summary, the current StuffIt Expander should be all you’ll ever need to open any StuffIt archive ever created. StuffIt Expander 2009 can be downloaded from Smith Micro, free. Cool. One problem solved. Regarding my stubborn archive, Mr. Covington wrote:
2. “… does the error result from a bug in StuffIt, or does the error result from some kind of corruption to the archive file itself? The bug in Expander hypothesis can easily be tested — I’d recommend that you download the free Expander 2009 update and see if that has any better luck on your files. If neither StuffIt 12 or StuffIt 2009 can open the archives, then the problem is most likely with the file itself.”
I replaced StuffIt 12 with StuffIt 2009, but the archive still wouldn’t open.
Mr. Covington continued: “In some cases, older versions of the StuffIt file format were dependent on information stored in the resource fork of the file, so if the resource fork has been lost, such files won’t open.”
I checked the file with this command in the terminal:
ls -l Finance.sit/..namedfork/rsrc
It showed a 338 byte resource fork on a 36 MB file. So evidently that wasn’t the problem either.
“There are also issues with archive size. If the .sit archive is above 4 GB in size, it is likely to be corrupt…. One troubleshooting tip that does require StuffIt Deluxe, is that in some cases the corruption in the file can be worked around by browsing the contents of the archive in the Archive Manager application that ships as part of the StuffIt Deluxe application.
“In some cases, while Expander is unable to process the complete archive due to corruption, Archive Manager can allow you to pull individual files out of the archive without having to access the corrupt area of the file. We’ve tried to put a lot of the older troubleshooting methods for opening corrupt archives into StuffIt Expander, however, so in many cases we’re able to skip over corruption in the file automatically to extract as much as possible. A failure right at the beginning of the process, which is what I’m guessing you see, indicates a much more serious problem with the file.”
I was sent an NFR copy of StuffIt Deluxe 2009 to see if the Archive manager would open the reluctant archive. That also failed.
Next, I was interested to find out what files got instaleld by StuffIt Deluxe. I got the answer:
3. “There are a number of components installed outside the applications folder with StuffIt Deluxe. In the case of Deluxe, the engine is installed in common location where multiple applications can access a single instance of the library. We also install a Spotlight plugin and a few other items that end up living outside the Applications folder. When you run the installer, you should see an inventory run by.*
“At this point we’re aware of no bugs or other issues with StuffIt Deluxe 2009 and Snow Leopard. Magic Menu (which loads in the Finder menu bar) and the Spotlight plugin both appear to load and function correctly. Needless to say we’ve had plenty of time to test for compatibility through access to Apple’s developer seeds.”
My last gasp attempt was to note that the file type and creator of the apparently corrupted file has been changed. Type = “enc2”; Creator = “crp2”. In fact, it should have been Type = “SIT5”; Creator = “SIT!” That suggested that I may have supplied some after-the-fact encryption on the file that I forgot about over the last 9 years. It may have been with a Classic utility, which I can no longer run, or perhaps a Mac OS 9 native encryption utility. Then the file was either corrupted or I’ll have to find the utility that encrypted it before StuffIt Archive Manager will work.
Lessons and References for Data Migration
Lesson #1 in data migration: Use standard file formats, keep durable, handy records on any methods used for compression or encryption and associated passwords. Keep track of and update tools used to archive, compress or encrypt files at each stage of OS and hardware migration. In other words, don’t upgrade if you can’t carry your tools forward — or deal with the files first to make sure they can carry forward.
— Chronology Of StuffIt
Here is the progression of the StuffIt Archive format over time. It might prove helpful in understanding the evolution of various archives you may encounter.
- 1993 — Introduction of the StuffIt 3 archive format (used .sit extension) — Better compression compared to original StuffIt archive format.
- 1998 — Introduction of the StuffIt 5 archive format (used .sit extension) — Again, improved compression over prior version. This version of StuffIt also removed some of the dependencies on resource fork data to enable better cross platform support sending files between Mac and PC.
- 2002 — Introduction of the StuffIt X file format. (used .sitx extension) — Better compression, strong encryption up to 512-bit, increased the number of files allowed in a single archive, increased the maximum allowable archive size.
- 2005 — An updated to the StuffIt X file format added compression of JPEG images –- able to compress JPEGs by an additional 20 to 30 percent without further reductions to image quality.
- 2007 – An update to the StuffIt X file format added compression of the Microsoft Office 2007 file format (introduced in Office 2008 on the Mac). These new document formats (.docx, .xlsx, .pptx) are essentially compressed with Zip compression by the Microsoft applications, so the Finder’s zip compression is unable to make them smaller. StuffIt introduced a new technique that allowed Office 2007/8 documents to be further compressed, and for inline JPEGs stored in these documents to be compressed using the JPEG compression method introduced in 2005.
- 2008 – An update to the StuffIt X file format added support for specialized compression of 24-bit images – TIFFs, PNGs, Bitmaps, etc. Also added was improved compression of PDF and MP3 files.
— Progression of File Type and Creator
StuffIt 1.5.1 Extension: .sit Type: SIT! Creator: SIT! StuffIt 2 (released with StuffIt Deluxe 3.0) Extension: .sit Type: SITD Creator: SIT! StuffIt 5 (released with StuffIt Deluxe 5.0) Extension: .sit Type: SIT5 Creator: SIT! StuffIt X 7.0 (Released with StuffIt Deluxe 7.0 – this is the newer StuffIt X file format. Extension: .sitx Type: SITX Creator: SIT
This information could come in handy if, somehow, the type or creator of an archive has been altered or corrupted. It might be possible to change that file data and achieve a successful unstuff. There are a few apps plus command line tools and developer tools to change the file type and creator, but that’s a subject for another article.
Stupid Installer Tricks
Note that the StuffIt Expander uses an installer. I asked Mr. Covington about that, and he said: “Expander is completely self contained. The StuffIt Engine is included in the application package. The fact we’re using an installer at all is really down to customer feedback. We went to a drag and drop install at one point in time, but in the end a lot of the new users we were getting (and this is especially true for users who switched to the Mac from Windows) were just more comfortable with running an installer.”
That’s an astounding bit of information, and Mr. Covington gave his permission to share that tidbit from a developer’s standpoint. Of course, in the Mac community, we prize drag and drop installs and expect all the guts to be inside the package. However, many Windows users are likely not familiar with the package concept, that is, that all Mac OS X apps are really directories that act like a single file. Not knowing that — and how the Mac OS X file system is set up — they, perhaps, prefer the psychological comfort of an installer.
Lesson #2 in data migration: Become aware of conventions and protocols in various OSes that could cause misunderstandings about the structure of the data and the correct use of software tools.
End Of File
What I’m going to do next is to take a look at my .sit and .sitx archives to see if any of them are critical and really should exit their archival status. Just for the peace of mind. However, knowing that I have the tools I need to open any StuffIt archive means that I probably don’t need to unstuff all 2,882 files. For now.
Modern Internet technologies and consumer terabyte drives make it less and less imperative to compress files or collections of files except in special circumstances, mostly for transmission. However, if you do elect to compress files and then add the complication of encryption, it’s wise to keep your tools up to date and understand how they work.
Here’s another tip for home users: designate a standard family password for personal, encrypted .dmg, .zip, and .sit(x) files that your spouse and children know. Doing all these things will insure that your valuable personal data, compressed so long ago, can migrate successfully into the future.
______
* Install Log:
Removed /Library/Frameworks/StuffIt.framework was not found. /Library/Frameworks/StuffItCore.framework was not found. /Library/Frameworks/StuffItSupport.framework was not found. /Library/Contextual Menu Items/StuffItCM.plugin was not found. /Library/PreferencePanes/StuffIt AVR.prefpane was not found. /Library/Spotlight/StuffIt.mdimporter was not found. /usr/local/bin/stuff was not found. /usr/local/bin/unstuff was not found. /Library/QuickLook/StuffIt.qlgenerator was not found. /Users/john/Library/Frameworks/StuffIt.framework was not found. /Users/john/Library/Frameworks/StuffItCore.framework was not found. /Users/john/Library/Frameworks/StuffItSupport.framework was not found. /Users/john/Library/Contextual Menu Items/StuffItCM.plugin was not found. /Users/john/Library/PreferencePanes/StuffIt AVR.prefpane was not found. /Users/john/Library/Spotlight/StuffIt.mdimporter was not found. /Users/john/usr/local/bin/stuff was not found. /Users/john/usr/local/bin/unstuff was not found. /Users/john/Library/QuickLook/StuffIt.qlgenerator was not found. Installed StuffIt.framework successfully installed. StuffItCore.framework successfully installed. StuffItCM.plugin successfully installed. stuff successfully installed. unstuff successfully installed. StuffIt.qlgenerator successfully installed.