The Importance of Data Integrity

As you will have probably seen on this website and in posts on newsgroups and message boards by SPS members, floppy disks are getting old. The media that holds all those beloved games was never designed to last a lifetime. Ten to thirty years is one quote, and according to that article, more than ten years is only usual with special care. To compound this, we are starting to see a few game disks from 1985 that have completely lost their magnetism. We now need to be very careful about dumping and verifying games, because there are a worrying amount of bad ones as has been seen already.

Dealing with magnetic media so long after some of the games were first produced is always going to be a big problem. We are seeing more and more people are finding errors appearing on their disks, and there are probably many more do not even know their disks are bad, even if their games appear to work.

Analyser Integrity Checks

Unlike anything that we know of, our analyser checks the encoding and checksums contained on a disk to the smallest detail. Therefore only tracks that have no encoding or checksum errors (defined by the track format) score 100% correctness.

The analyser collects the best possible sample combination of blocks from a track, therefore some games that are otherwise unreadable can be restored to a good a state since the analyser can decide (since it can check), what should have been read.

Reverse engineering a disk that contains this “hidden” integrity information is the only proper way to ensure the data is correct. As we work with disk formats, we can see any extra information and use it to help us verify the dump that one stage further. This will benefit everybody when we all finally know for sure that a disk image is 100% good.

This is not the end of the story since a good track is not necessarily a genuine one.

Other Solutions

Other software may indicate a track is 100% correct while our analyser detects errors. Why? Basically because it checks everything and it does not assume anything. Obviously, the fact that we get more information with our dumping tools helps here. Many of these errors are recoverable so long as you know to look for them. However they just go completely unnoticed reading casually or if the reading software assumes things it should not.

On the other hand, sometimes our analyser says a track is 100% correct, when other software says there are errors. Why? It is because the disks are read in special and low level ways, reading problems that occur when a disk is read in the normal way are not an issue when dumped using our tools. It is important to note that reading problems are fairly common, and are different to actual hard errors from corruption or Bit Rot, which cannot be recovered without very expensive equipment, and perhaps impossible even then, depending on the extent of the damage.

Not even the loading routines of the software is good enough. Some games do not perform checksums or any other integrity checks, even though they contain integrity information on the disks, it remains unused. This was usually for duplication and verification purposes in the original mastering system. Therefore, integrity checks based solely on looking at the loader from the game may end up reading bad data without noticing it. This is rather unsurprising since the game itself is subject to the same problem.

The best thing you can do in this situation is to play-test the games all the way through, taking every possible game path. For most games, this is practically impossible let alone time consuming, and even for the few where it might be possible, can you really be 100% sure? Probably not.

If you know of any other software that does actually do something similar to ours, then feel free to let us know. There was/is probably commercial based “data recovery” solutions that cost the earth to do this kind of stuff, but there is probably nothing “mainstream” and almost certainly nothing using just software.

Summary

Data integrity and data authenticity are very important to us and so we give these things extra special attention. We could have started imaging games a full two years before we did without taking these things into account, but it would not have been preservation, and it would have been pointless.

Our technology lets us see if a disk has been corrupted, fallen victim to Bit Rot, or even been modified. Knowing that the disk data has been damaged or modified is incredibly useful, because it means we can really see if a disk is factory-new or not.

Related Links