Graphics file formats FAQ

Who cares about graphics file formats?

Well, programmers do mostly. But end-users (that is, non-programmers) do as well.

The typical end-user only cares about storing their graphics information using a format that most graphics programs and filters can read.

End-users are typically not concerned with the internal arrangement of the data within the graphics file itself. They only want the format to do its job by representing their data correctly in a permanent form.

Programmers, on the other hand, are that rare breed of human that just can't leave information well enough alone. They need to know how every byte is arranged to see if someone knows something that they don't (and often snicker contentedly to themselves when they find that it is really they that know more). Programmers will then use this information to write code that may never see the light of distribution, but nevertheless, they will have had fun and gained enlightenment from writing it.

It doesn't matter which of these two types of people you are. If you have even the slightest interest in graphics file formats then you may be counted as one who cares.

What is raster, vector, metafile, PDL, VRML, and so forth?

These terms are used to classify the type of data a graphics file contains. Raster files (also called bitmapped files) contain graphics information described as pixels, such as photographic images. Vector files contain data described as mathematical equations and are typically used to store line art and CAD information. Metafiles are formats that may contain either raster or vector graphics data. Page Description Languages (PDL) are used to describe the layout of a printed page of graphics and text.

Animation formats are usually collections of raster data that is displayed in a sequence. Multi-dimensional object formats store graphics data as a collection of objects (data and the code that manipulates it) that may be rendered (displayed) in a variety of perspectives. Virtual Reality Modeling Language (VRML) is a 3D, object-oriented language used for describing "virtual worlds" networked via the Internet and hyperlinked within the World Wide Web. Multimedia file formats are capable of storing any of the previously mentioned types of data, including sound and video information.

Can graphics files be infected with a virus?

For most types of graphics file formats currently available the answer is "no". A virus (or worm, Trojan horse, and so forth) is fundamentally a collection of code (that is, a program) that contains instructions which are executed by a CPU. Most graphics files, however, contain only static data and no executable code. The code that reads, writes, and displays graphics data is found in translation and display programs and not in the graphics files themselves. If reading or writing a graphics file caused a system malfunction is it most likely the fault of the program reading the file and not of the graphics file data itself.

With the introduction of multimedia we have seen new formats appear, and modifications to older formats made, that allow executable instructions to be stored within a file format. These instructions are used to direct multimedia applications to play sounds or music, prompt the user for information, or display other graphics and video information. And such multimedia display programs may perform these functions by interfacing with their environment via an API, or by direct interaction with the operating system. One might also imagine a truly object-oriented graphics file as containing the code required to read, write, and display itself.

Once again, any catastrophes that result from using these multimedia application is most like the result of unfound bugs in the software and not some sinister instructions in the graphics file data. Such "logic bombs" are typically exorcised through the use of testing using a wide variety of different image files for test cases.

If you have a virus scanning program that indicates a specific graphics file is infected by virus, then it is very possible that the file coincidentally contains a byte pattern that the scanning programming recognizes as a key byte signature identifying a virus. Contact the author (or even read the documentation!) of the virus scanning program to discuss the probability of the mis-identification of a clean file as being infected by a virus. Save the graphics file, as the author will most likely wish to examine it as well.

If you suspect a graphics file to be at the heart of a virus problem you are experiencing, then also consider the possibility that the graphics file's transport mechanism (floppy disk, tape or shell archive file, compressed archive file, and so forth) might be the original source of the virus and not the graphics file itself.

Can graphics files be encrypted?

Of course you can encrypt a graphics file. After all, most encryption algorithms don't care about the intellectual content of a file. All they chew on is a series of byte values. Therefore, most any encryption program that works on ordinary text files will work on graphics files as well.

Why would you want to encrypt a graphics file? Mostly to control who can view its contents. You can invent a proprietary file format and that might slow a file format hack down for, say, five or ten minutes. You could add a proprietary data compression scheme, possibly a twisted variation of an already public algorithm. But there are so many people out there with nothing better to do than hack at unknown data formats that your data would probably be exposed in little time. But suppose we top off all this effort by encrypting the graphics file itself as we would an ordinary text file. Would your data then be safe?

Realize that an encrypted graphics file still might not be very secure. For every data encryption algorithm there exists at least one method of getting around it, although it may take hundreds of computers and many years to fully employ and execute that method!

For example, one of the more popular methods used to encrypt data is the Vernam or XOR cipher. This cipher Exclusive ORs the plain-text data with a single, random, fixed-length key. The longer the key the harder it is to break the cipher. A totally random key the length of your data is impossible to break. Shorter and less-random keys are easier to break.

XOR is very simple and fast, which is a must for a graphics file translators/viewers that must decrypt a file on the fly. A problem, however, is that most graphics files contain fixed size headers which vary only slightly in content from file to file. If you knew the approximate contents of the header of an encrypted file you could XOR a "decrypted" header with the encrypted file and possibly produce the key used to encrypt the file. A short key might be very easily discovered in this way.

If you wish to use a public key/private key encryption method, then storing the public key in the file format header (usually as a 4-byte field) and only encrypting the image data would be the way to go. The SMPTE DPX file format supports such an encryption feature.

If you really need to make the contents of a graphics file secure, then I'd suggest not only using some form of data encryption, but also create an unconventional and proprietary file format and do not publish its format specification.

How can I convert the XXX format to the YYY format?

With a file conversion program, of course! Without a doubt one of the most frequently asked categories of questions on comp.graphics.misc is how to convert one format to another. In every case the answer is some type of conversion program or filter, but which one?

Section IV of the FAQ is an attempt to list every known graphics file display and conversion program and application. Although far from complete, this list may contain the program you need. Go to the subsection of the particular operating system you are using and scan through Imports: and Exports: formats listed and see if the formats you needs to use are there.

In some cases the information in a listing may make the conversion capabilities of a program a bit misleading. For example, a program that can import a vector .DWG file and export a raster .BMP file may not necessarily be able to perform a .DWG->.BMP (vector->raster) conversion (AutoCAD R12 can, BTW). And just because a program can both import and export TIFF files doesn't mean it's capable of a TIFF(CMYK)->TIFF(RGB) conversion (as Adobe Photoshop can do). As always, read the ocumentation, contact and ask the author of the program, or find a user of the program and ask them.

Do I really need the specification of the format I'm using?

It depends upon the results you are trying to obtain. If you have code that supports the XXX format and you find it easy (and legal) to integrate that code into your program, then you may be tempted to do so. But realize that your program will support the XXX format in just the same way as the previous program did. In other words, your program will now work the same, but it will really be no better.

By obtaining the format specification you can make an attempt to fully support all of the features and capabilities a graphics or multimedia file format has to offer. If you use pre-written code that only supports a small subset of the format's features then you are not doing justice to the format and cheating your users out of functionality they might need.

Always strive to create the best programs possible within reason of time and money. Obtain the specs, look at code, and talk to programmers who have worked with the format before. You might gain some insight and save yourself some hair-pulling by supporting a feature that someone didn't think to include in the original requirements for your program.

How can I tell if a graphics file is corrupt?

The easiest way is to display the file and decide if what you see on the screen or the printer is correct. This method is not fool-proof, however, because not all information stored in a graphics file is used for displaying the data it contains. Textual comments, alternate color maps, and unused fields in the header might be munged and go undetected.

A frequent source of corruption occurs when 8-bit graphics data is transported via a 7-bit communications channel. The 8th bit of each byte is cleared (set to zero) and you are left with garbage. ASCII-mode file transfers may also translate carriage returns (0Dh) to line feeds (0Ah), or to CR/LF pairs depending upon if the file is being transferred to a Unix (LF-only), Macintosh (CR-only), or MS-DOS (CR/LF) system.

The PNG file format supports an elegant solution to the quick detection of this type of corruption. The first character of every PNG file is the 8-bit value 89h. If this value is read as 09h, the 8th bit has been zeroed and you know the file is corrupt.

Most graphics files do not contain any real, built-in error detection features. The standard way to check for corruption of any type of data file is to perform some sort of error-detection scheme on the file. Such schemes commonly used are Checksum calculations and the Cyclic Redundancy Check (CRC). These algorithms are commonly used in the world of synchronous serial communications for detecting errors in data streams.

If you only wanted to provide error detection for the graphical data contained in a file, but not the header, then a 2- or 4-byte field in the header could be used to store the CRC-16 or CRC-32 value of the data. But what good is pure data if the header is possibly corrupt?

If we calculate the CRC value of the entire file and then store that calculated value in the header we will have just "corrupted" the file! You could initialize the CRC field with zeros, calculate the value, store the value, and specify that the entire file need be read into memory and the CRC value field set to all zeros before the CRC calculation is made.

File formats that segment their data into blocks or chunks would be able to perform a CRC on each section individually (another feature found in the PNG file format). Each section would store the CRC value as the last 2 or 4 bytes of the block and the CRC value field would never be read for the purpose of the CRC calculation. This method makes it easier to find the location of the error(s) in a file. If the CRC error occured in an unnecessary block of data, the file might still be useful anyway. This block-style CRC checking also saves the reader from performing a time-consuming CRC calculation an entire, possibly very large, graphics file.

But all this can be quite a pain. Can't we avoid modifying a file and just store the CRC value externally to the file? Maybe using some sort of encapsulating "wrapper"?

If you want to make sure a graphics file (or any file for that matter) has been transported through a communications channel without sustaining any corruption, first store it using a file archiving program that supports error checking of the files contained in the archive.(Several good error-checking file archiving programs include PKZIP, gzip, and zoo. The ar and tar Unix archiving programs do not support error checking). When the graphics file is stored, the archival program calculates the CRC value of the file. If the CRC value does not match the file's calculated CRC after it is unarchived after transport, you know that the file has been corrupted.

Note: make sure you turn compression OFF when archiving many types of graphics files. File archival programs use compression by default and will attempt to make your compressed data even smaller (which usually results in larger data, unless the archiver is smart enough to detect the negative compression and not attempt to compress the file). ASCII-based files (such as PostScript and DXF) and some RLE-encoded files (such as PCX) will be compressed, while other formats supporting more advanced data compression methods (such as JPEG and LZW) will surely grow in size.

How can I make transparent and interlaced GIFs for a Web page?

Transparent GIFs are used to eliminate the typical rectangular borders associated with most bitmapped images that appear on a Web page. The creator of a GIF image may set certain pixels within the bitmap to a color desiganted within the image file as "transparent". When this GIF image is displayed by a Web browser the transparent pixels take on the color of the user's display. This is identical to the blue screen effect found in chromakeying.

GIF89a files are made transparent by the use of graphics file editing software (GIF87a files do not support transparency and must first be converted to the GIF89a format). Such software will set the transparency color flag and the transparent color index value of a Graphics Control Extension block within the GIF89a file. Any pixel set to the specified transparent color index value will take on the background color of the display device when displayed.

Interlaced GIFs are used to give the user a idea of that an image looks like before all of the bitmapped data has been received. Non-interlaced images paint in a linear fashion from the top to the bottom of the display. Over a slow link it make take several minutes for an image to paint. When 50% of the bitmapped data is received only the top half of the image is displayed. The contents bottom half is still a mystery to the user.

Interlaced GIFs paint quickly over the entire display first as a very low resolution image. Only about 12.5% of the bitmap is displayed in this first pass. The GIF image is then repainted in three more passes, with each pass supplying more resolution until all of the data is received (12.5%, 25%, and 50% respectively). A user can usually get a good idea of what the entire image is when only 30-50% of the bitmapped data has been received. This is very useful for knowing when to abort an image being viewd via a slow link.

Interlacing is supported by both the GIF87a and GIF89a formats. Graphics file editing software that supports interlaced GIF should not only be able to display such GIF files, but also convert non-interlaced GIFs to the interlaced format (and back again as well). This involves reading in the GIF bitmap and writing it back out using the GIF 4-pass interlace scheme. In a non-interlaced GIF the pixel lines are displayed in consecutive order. For example, the lines of a 16-line non-interlaced GIF file are stored as 0, 1, 2, 3, 4, 5...15. The lines of the same 16-line bitmap in an interlaced GIF would be stored as 0, 8, 4, 12, 2, 6, 10, 14, 1, 3, 5, 7, 9, 11, 13, 15.

Graphics file format software packages for Unix which handles both trasparent and interlaced GIFs include NETPBM and giftool. For the Macintosh: Transparency, Graphic Converter, Imagery, and clip2gif

The Visioneering image manipulation page will allow you to convert your GIFs to transparent and interlaced without having special software on your system. Your GIF files will be read, converted, and written using the Web. Visioneering's page is at:

http://www.vrl.com/Imaging/

More detailed information on images used in Web pages can be found in the FAQ for the newsgroup comp.infosystems.www.authoring.images found at:

http://www.island.net/help/faq/www_faq/ http://www.io.org/faq/www/index.html

A collection of links to a number of Web and FTP resources that store information on transparent and interlaced GIFs for Unix, Macintosh, and MS-DOS/Windows, including executable programs and tutorials, may be found at:

http://members.aol.com/htmlguru/transparent_images.html http://dragon.jpl.nasa.gov/~adam/transparent.html ftp://csi.jpl.nasa.gov/pub/graphics/transparent.faq

How do I combine still images to make animations?

You might have a collection of imaes and drawings stored in a variety of still-image formats (TIFF, BMP, IFF, and so forth) and want to combine them into an animation or video file format that wil allow you to play them back.

Have a look at the following Web page:

http://www.prism.uvsq.fr/public/wos/multimedia/

Is it "Tag" or "Tagged" Image File Format?

Revision 5.0 of TIFF specification specifically states the acronym "TIFF" is "Tag Image File Format". The majority of people, however, intuitively say "Tagged" rather than "Tag". Interestingly enough, the TIFF 6.0 specification does not spell out the acronym TIFF.

Whaddya mean there's no "Targa" file format?

The popular "Targa" file format is really the "TGA format". "Targa" is the name of the Truevision graphics display adapter which first used the TGA format. Specifically, Revision 1.0 of TGA is designated the "Original TGA format" and Revision 2.0 is the "New TGA Format".

Choosy programmers choose "gif" or "jif"?

The pronunciation of "GIF" is specified in the GIF specification to be "jif", as in "jiffy", rather then "gif", which most people seem to prefer. This does seem strange because the "G" is from the word "Graphics" and not "Jraphics".

Why are there so many ".PIC" and ".IMG" formats?

Because people with very little imagination are allowed to choose file extensions for graphics files, that's why.

But seriously, there does seem to be a proliferation of file formats with the file extension ".PIC" (for "picture") and ".IMG" (for "image"). Other popular extensions (in both upper and lower case) are ".RGB", ".RAW", ".ASC", and ".MAP".

My advise to you is never assume the format of a data file based only on its file extension. The name and the extension of any file are completely arbitrary and therefore could be anything. This is why the most graphics file conversion and display programs attempt to recognize graphics files based on their internal structure and not their file name or extension.

Where can I get the spec for the GIF24 format?

A GIF24 standard file format has never been officially introduced or released to the public. The original effort by CompuServe and others, to create a 24-bit revision of the GIF format was never completed. The problems create by Unisys' LZW patent restrictions and the subsequent disdainment of GIF by many developers is probably mostly to blame.

It has been said that CompuServe abandoned GIF24 in favor of PNG format, who developers hope that one day will completely replace GIF. But it is not evident that CompuServe contributes in any remarkable way to the ongoing development of PNG.

Is there an uncompressed GIF format?

Realizing that the heart of the GIF patent controversy is the LZW data compression algorithm itself, you may ask if there is a raw or uncompressed version of GIF that can be read and written without using the LZW alogrithm. Officially, the answer is no.

The GIF specification does not defined a way to store uncompressed bitmap data. All bitmap data stored in a GIF file is compressed using the LZW algorithm. If you did write a program that stored uncompressed data using the GIF format, no other GIF reader would be able to decode the GIF files it created.

So is there a way to modify the compressed data in a GIF file so it is no longer in a format described by the LZW patent, but still readable by GIF decoders? They answer to this is yes!

When a GIF file is compressed, an initial LZW code table is created based on the bit-depth of the raw image data being LZW-encoded. For example, a bitmap with 4-bit pixels will be encoded with an LZW code table initially containing 18 entries: 16 color indicies ranging from 00000 to 01111, a clear code (10000), and a end-of-data code (10001).

As LZW encoding proceedes, color codes from the data are used to form new table entries, and its the formation of these new entries that is the heart of LZW encoding. If an encoder only used the initial table and did not create any new table entry codes, then all of the resulting encoded data will be codes representing the indicies of the colors stored the in the GIF file's active color table.

This process is explained in a post made to comp.graphics.misc by Dr. Tom Lane on 05 Dec 1996:

...the idea is to emit only single-symbol string codes, plus a Clear code every so often to keep the decoder from jacking up the code width. In this mode your encoder is simply packing N-bit pixel values into N+1-bit fields and keeping count; nothing patentable there. Note that the data is not merely not compressed, it's *expanded*: you need 9 bits per pixel for an 8-bit GIF. I wouldn't care to use this trick for low-depth data. The worst case is for 1-bit (black and white) data; not only do you need 2 bits/pixel, but every other symbol has to be a Clear to keep the code width down to 2 bits ... net result, 4:1 expansion.

Because this encoder ends up storing N+1 bits for every N bits of data, plus a clear code every 2^N-2 codes, an 8-bit "non-compressed" GIF image will be 1/8th larger than the same bitmap stored as an LZW-compressed GIF.

Tom explained this a few days later:

Note, however, that you have to insert "clear" codes often enough to prevent the decoder from ratcheting up the symbol width, or else keep track of what the current symbol width should be. It's been a while since I looked at this in detail, but I think you need a clear every 2^N-2 codes, where N is the underlying data depth, if you want the symbol width to stay at N+1 bits.