The wait is over!

22 December 2002

Okay, lets get this bit out of the way first...

Complete save framework now completed, with very strict checking of any possible problems. This includes problems that may be encountered only by the decoder (the image loader, see “Decoder” below) so it does not need this kind of (very complex) mechanism being already taken care of by the encoder.

Complete “IPF” saver completed. IPF is the release image file type, it stands for “Interchangeable Preservation Format”, and is called such for some very good reasons:

• Although we currently focus on the Amiga, it will not always be this way. The image format was designed to be used for anything, and as such, it is possible for any platform to have its disks supported. In fact, some of the track formats used for Amiga copy protection were PC/Atari anyway.
• It is not just for disks: There are many types of digital media that can be contained by the image format. You name it, it is likely it can be supported. Uses include: ROM images, embedded information from “dongle” protection (like those used in Robocop 3) and maybe one day even CDROM’s.

So what does that leave us with? Well, because of above, we cannot mention the platform or media type so the decision was made to call it exactly what it is, and what it is used for: The Interchangeable Preservation Format.

Everything in place.

Complete track decoder/loader capable of detecting any kind of problems with the image.

Encoder now identifies the version of the image creation software.

Two way conversion of detected density types between the release and dump images. Release images look identical to dumps on the track map, though a release is much “cleaner”, since it has already been processed. Release images can be analysed just like real dumps; this is a very accurate test of the resulting images. Re-analysed images (apart from gap info and unique image data) are exact replicas of a real dump.

The analyser defaults to generating missing data, so all features such as graphs are available and actually look frighteningly similar to real dumps, apart from the fact that all the missing data is generated from the defined description of the disk.

From the varied density types, Copylock Amiga and the newer version of Copylock is generated at the moment. Any others, including things like Copylock ST, will be available when a release happens containing it. Some examples of future types can be seen at the end of this WIP.

The first version of the IPF API (the plugin and associated tools) was completed on 1st December 2002. A historic day for us! As you may have seen from the page, the IPF image support plugin is now available. This can be used to enable IPF support in emulators, and other tools. If you intend to download and use the provided package(s), you will need to pay close attention (and agree to) the license. Though close inspection is more relevant to developers looking to include IPF support in their tools.

The API (for use by anything wishing to access IPF images) provides certain functionality for generating data on the fly, and these decoding options can be combined, as follows.

• Create word size aligned track data
• Re-align tracks to the index sync as if they were written originally that way without altering the release image. This is only really useful for mastering.
• Generate cell density samples for variable density tracks
• Generate density for automatically sized cells
• Generate density for unformatted cells
• Generate unformatted data
• Generate unformatted data, that changes each disk revolution

The “track re-aligning as index synced data” is automatically disabled when a variable density track is encountered, since the cell density map needs to be bit exact to the (altered) data positions otherwise it may not pass protection checks.

The density map is generated according to data found on the disk, and as such it faithfully reproduces each cell originally written to the disk.

As you probably guessed by now, support for IPF images has been added to WinUAE using the plugin. Other ports of UAE should follow, and only wait as we port the plugin to different platforms. [Update: other ports of UAE with IPF support are now available]

On the 8th December 2002 we spent some hours with mailing with Toni and the result is a WinUAE that works perfectly with Copylock images! This is a milestone achievement because it means that UAE now supports disk protections with variable cell density. Apparently after the previous few batches of mails with Toni on improving UAE‘s disk controller code, he went ahead and started implementing support for cell density on the assumption that we might actually do something one day. Well done Toni.

“A service announcement follows...”

As you may be aware from previous postings on this site, we will not give any images to people unless we know they are legally entitled to them. So, unless you are a copyright owner, a contributor of the game, or you run a website that can prove it has the permission to host it, please do not ask us for images of games!

Apart from the moral standpoint, which carries a lot of weight with us as software developers, we are just not prepared to invite the legal hassle. If you own the game, then fine, your disk is likely to go bad one day - so dump it for the retro community! We then know you own the game (since we know when a game has been mastered with a commercial duplicator, and when it has not) and you are “future proofed” against any mishaps with your disks.

If you own a game and cannot dump it for whatever reason, we apologise in advance for being unable to provide you with a copy. Hopefully you can find it on one of the various legal games sites. Otherwise you are out of luck until we as a community can persuade copyright owners that they can get some great PR by releasing old games to the community.

If you do see IPF files for download somewhere that are not actually legally distributable, then that is just not our problem. We cannot be held responsible for what people do with their disk images, just like we are not responsible for somebody who images a disk in any other format and sticks it on the net. If you own the game, then great! Download a preservable form of it! Otherwise, you may be breaking the law in your country.

“Service announcement ends.”

All preserved games will be catalogued on this website. You can use the search section to see what has been done, and the images that are dumped and thus available for possible preservation in the future.

One of the things we thought about when we started getting close to releasing disk images, is how to summarise all the information about a game into an easily processable form. There are other projects that have made great strides in this field, like . However, currently our needs are slightly different.

We are preserving manuals, boxes, disk labels, audio cassettes, stickers, posters and basically anything else that came with a game. This stuff is pure art! But how do we catalogue all of this?

FIME to be moved

Due to a current lack of a universally agreed game preservation metadata standard, we have developed our own way of describing preservation artefacts. If at some point in the future an agreed standard comes along, we will almost certainly move to it. For now though, we have the ““gameinfo.xml““. This is a file that will be present in all archives of preserved games from us. It is defined in . Apart from preservation information, it is possible to use these meta files for things like emulator frontends.

As an example, here the metadata file for first game preserved Wizard Warz, for the Commodore Amiga. You can view it in a text editor and some web browsers (click on it and see if yours is supported). The XML Schema for this metadata scheme is here and here. There is also documentation describing this scheme.

Note that IPF image files were designed to be properly identifiable (not just based on CRCs) and use release and revision numbers embedded in the image. It is much cleaner this way.

Information is likely to be added and removed from this file as we get new versions of the game, a better scan of the manual, etc. For this reason, the latest version of all info files will probably be available on the site that can be downloaded at leisure.

• Aufschwung Ost
• Benefactor
• Cyberblast
• Dragonflight
• Hunter: Support for the dual-boot. However, the Amiga version does not have the Atari boot block EDC written, this may prevent the running of the dual-format disk on an ST. Probably a deliberate incompatibility to prevent cross platform use, well, unless the Atari ST runs bootblocks with a bad checksum... We will investigate that in due course.
• Magic Marble
• Nightdawn
• Prime Mover
• Pro Tennis Tour
• Probe: on Supremacy and various Sega conversions
• reDOS V2: North Sea Inferno, which has MFM encoding errors.
• reDOS V5: FATE Gates of Dawn.
• Shuffle
• SmartDOS: On Rise of the Robots. Apart from the fact that the EDC is strangely calculated on the output stream (which may not match the readable data due to unrelated bits changing, bit shifting etc), the EDC does not cover the last 10 bytes of the track data. The data is also full of MFM encoding errors that the loader expects to be readable. This one is certainly a high contender for one of the worst disk formats ever. It is clearly anything but “smart”.
• Tracker
• Troddlers
• Wayne Gretzky Hockey
• Yo Joe
• Zeppelin/Gonzo (slightly modified) format on Wipeout

Added support for Aunt Arctic Adventure to THBB, The Bitmap Brothers format. Strange that this games does not seem to have been written by The Bitmap’s, at least none of the developers seem to match on . So perhaps the “floppy specialist” (quote from the credits) nicked the loader from Speedball...?

• Sega: It is Probe.

• If “mark” bytes values are not set (set to any mark) and their weight is 0 (decoding score unaffected) the raw input stream data is replicated on the output stream. This is required to reproduce the encoding errors present on Rise of the Robots.
• Automatic detection of modified/copied tracks. Should be around 99% accurate, which is much easier than finding it out manually. Manual checks are still done to cover the remaining 1%, and are usually caused by duplication on old mastering equipment.
• If it is possible that a track was not mastered commercially it is indicated by a ‘+’ sign.
• Analyser info window contains summary information (see Analyser Reporting below) and the image type loaded, “Dump”, “Release”, “Mix” or [invalid].
• Various other enhancements including detecting and correcting possible problems caused by gaps.

• Amiga platform dependency eliminated.
• NTSC/PAL settings removed as it was tied to Amiga written (modified / copy) tracks. The machine type is automatically detected on suspected modified tracks.
• Platform (Amiga, Atari, PC, etc.) can be set by system configuration that gives hints for the analysation process.

• Useless track density filter. It turned out that it was never actually useful.

The analyser now spits out a report file on each dump analysed. This helps us to identify any problem disks quicker, and it may help to generate more of the database content automatically (the idea being that anyone can go and look at the state of a dump from the website).

This information contains:

• Dump identifier
• Number of tracks dumped
• Number of tracks bad
• Number of tracks modified
• Number of noise (definitely unformatted) tracks
• Number of tracks recorded (tracks written in any way)
• Number of tracks unknown
• Number of tracks ignored (not selected as part of the game for various reasons, such as containing traceback information or containing junk such as the case with recycled disks).

“Recording type” detection using the track data. Currently only “MFM” and “Noise” (unformatted) types are supported. Remember though that this is just Analyser support for release images, this does not effect the dumping tool which has not changed since it was first used by contributors over a year ago.

Unlike normal formats, recording types cannot be selected manually, the analyser internally processes them before format analysation takes place. In previous WIPs, this is referred to as the physical layer.

If a track is definitely 100% unformatted data, further processing is cancelled. In order to avoid mistakes only 100% noise tracks are cancelled, and the detector is very sensitive to any kind of possible data pattern. This may result in less than 100% noise tracks, especially towards the outer rim of a disk, or if the disk is “recycled” (older unsold/bad copy of game wiped and rewritten by duplicator).

In order to avoid confusion noise tracks not scoring 100%, or tracks with unknown MFM recording etc, do not count as bad, but “noise” and “unknown” respectively. “[MFM recording][score]” means the track is definitely MFM written as [score] or 100% percentage, but no formats yet described match the data to the slightest extent. “Protec” tracks contain such data. Basically this means that there is no way for the analyser to match the data to a described format in any way, the only thing it knows is that there is some recorded data there.

The data node (the actual data decoded using the descriptor) is always [unknown] if the recording type is not detected by the analyser or does not match known formats, since there is no way to describe what the data could possibly mean.

Found a very interesting density graph on this one! We have not seen anything like this before... Track 0.1 has 6 (!!!) different densities.

Another interesting density graph on track 0.1 on Cavitas.