NWN Data Files

원출처 : www.torlack.com/index.html?topics=nwndata_itp 

What is an ITP File?

(Note: ITP files are more correctly known as the GFF and BioWare's documentation can be found here.) nwn.bioware.com/developers/gff.html 

An ITP file can be thought of as a hierarchical collection variables.

That cleared it up, didn't it.

In an ITP file, you have a series of what can be considered variable assignments such as 'PlayerName = "Francis"'. There can practically be an infinite number of variable assignments in one file. Also, ITP files allow you to nest variable assignments much the way a programmer might nest structures.

ITP files are very common in NWN. The format is used for a wide collection of files such as; ITP, BIC, DLG, GIT, GFF, FAC, UTI, UTC, UTT, UTS, UTE, UTD, UTP, UTM, and UTW. All of these different files share the same common file format. So in reality calling the file format ITP is inaccurate, but it is the name we all know best.

Data Stored in an ITP File
ITP files contain seven sections of data. The sections are as follows:

File Header - Gives us enough information to begin decoding the remaining sections.
Entry Table - Provides us with our collections of variables and their nesting.
Element Table - Provides the information about all the variables contained within an entry.
Variable Names Table - List of all the different variables names.
Variable Data Section - Depending on the element in question, this data block can contain a assortment of data.
Multiple Element Map (MultiMap) - Used to list which elements are contained within an entry when that entry contains more than one element.
List Section - Provides a method for an element to have one or more entries as children.
The File Header
Offset Type Description

Offset	Type	Description
0x0000	char [4]	4 byte signature.
0x0004	char [4]	4 byte version.
0x0008	UINT32	Offset from start of the file to the first entry
0x000C	UINT32	Number of entries in the file
0x0010	UINT32	Offset from start of the file to the first element
0x0014	UINT32	Number of elements in the file
0x0018	UINT32	Offset from start of the file to the first variable name
0x001C	UINT32	Number of names in the variable name block
0x0020	UINT32	Offset from start of the file to the first variable data
0x0024	UINT32	Number of bytes in the variable data block
0x0028	UINT32	Offset from start of the file to the first multimap
0x002C	UINT32	Number of bytes in the multimap table
0x0030	UINT32	Offset from start of the file to the first list
0x0034	UINT32	Number of bytes in the list table
0x0038	Total length of the structure

All offsets in the file header are from the start of the file. The file header is always at offset zero. It seems that the offset to the first entry is always 0x0038. But please note that for entries and elements, we are given the number of entries or elements. For the four remaining data sections, the total section length is given in bytes.

The Entity Table
The entity table is an array of structures of the following form:

Offset	Type	Description
0x0000	UINT32	Entity code (Use unknown)
0x0004	UINT32	Element index or MultiMap offset
0x0008	UINT32	Number of elements in this entry
0x000C	Total length of the structure

The first entry in the entry table is the root of whole hierarchy. From that one entry, all other entries and elements can be accessed in a logical form.

Entries only serve one function, to contain one or more elements. Depending on the number of elements contained in an entry, accessing the elements can either be easy or require indirect referencing. When an entry contains only one element, and thus the value at offset 0x0008 is one, then the value at offset 0x0004 is the index in the element table of the only child. When an entry contains more than one element, then the value at offset 0x0004 contains a byte offset into the multimap table. This offset points to an array of element numbers stored as UINT32 values.

Here is an example of how you might iterate through the list of elements for an entity. This code assumes that the whole file is resident in memory in a contiguous buffer.

char *pFileData = AddressOfTheITPDataInMemory;
Header *pHeader = (Header *) pFileData;
Entry *pEntryTable = (Entry *) &pFileData [pHeader ->EntryOffset];
Element *pElementTable = (Element *) &pFileData [pHeader ->ElementOffset];

Entry *pEntry = &pEntryTable [0]; // For this example, use root entry

if (pEntry ->Count == 1) // We have one element
{
Element *pElement = &pElementTable [pEntry ->Offset];
// Do something with the element
}
else //more than one
{
UINT32 *pMMap = (UINT32 *) &pFileData [pHeader ->MultiMapOffset + pEntry ->Offset];
for (int i = 0; i < pEntry ->Count; i++)
{
Element *pElement = &pElementTable [pMMap [i]];
// Do something with the element
}
}

There is one trick you can use to reduce duplicated code. I use this trick in my software.

char *pFileData = AddressOfTheITPDataInMemory;
Header *pHeader = (Header *) pFileData;
Entry *pEntryTable = (Entry *) &pFileData [pHeader ->EntryOffset];
Element *pElementTable = (Element *) &pFileData [pHeader ->ElementOffset];

Entry *pEntry = &pEntryTable [0]; // For this example, use root entry

UINT32 *pMMap;
if (pEntry ->Count == 1)
pMMap = &pEntry ->Offset;
else
pMMap = (UINT32 *) &pFileData [pHeader ->MultiMapOffset + pEntry ->Offset];

for (int i = 0; i < pEntry ->Count; i++)
{
Element *pElement = &pElementTable [pMMap [i]];
// Do something with the element
}

The Element Table
The element table is an array of structures of the following form:

Offset	Type	Description
0x0000	UINT32	Variable type (0-15)
0x0004	UINT32	Variable name index
0x0008	VARIES	Data for the variable
0x000C	Total length of the structure

As you can see, the element structure is very basic. From the structure we know the name of the variable, the type of the variable, and the assigned value of the variable. The only difficult part is extracting the value from the structure.

There are 16 know variable types supported by the ITP format. They are as follows:

Value	Type	Access	Data
0	UINT8	Direct	Unsigned byte
1	INT8	Direct	Signed byte
2	UINT16	Direct	Unsigned word
3	INT16	Direct	Signed word
4	UINT32	Direct	Unsigned longword
5	INT32	Direct	Signed longword
6	UINT64	Indirect	Unsigned quadword
7	INT64	Indirect	Signed quadword
8	FLOAT	Indirect	Four byte floating point value
9	DOUBLE	Indirect	Eight byte floating point value
10	STRING	Indirect	Counted string
11	RESREF	Indirect	Counted resource name
12	STRREF	Complex	Multilingual capable string
13	DATREF	Complex	Counted binary data
14	CAPREF	Complex	List of child elements
15	LIST	Complex	List of child entries

For the simple data types, UINT8, INT8, UINT16, INT16, UINT32, INT32 and FLOAT, the value is just stored directly into offset 0x0008. Since NWN is for Intel systems, all the data values are stored in little endian format. Thus, for UINT8 and INT8, the value is a single byte stored literally at offset 0x0008 in the element structure. A UINT16 or INT16 takes up the first two bytes. The remaining three basic data types take up all four bytes. Access to the different values of the element can be done with this structure.

struct Element
{
UINT32 Type;
UINT32 NameIndex;
union
{
UINT8 ui8;
INT8 si8;
UINT16 ui16;
INT16 si16;
UINT32 ui32;
INT32 si32;
FLOAT flt;
UINT32 Offset;
};
};

Note: The "Offset" member can be used to access the indirect and complex data stored in the variable data section..

In the case of UINT64, INT64 and DOUBLE, the value stored at offset 0x0008 is a byte offset into the variable data region. Starting at that address would be stored the value in question.

char *pFileData = AddressOfTheITPDataInMemory;
Header *pHeader = (Header *) pFileData;
Element *pElementTable = (Element *) &pFileData [pHeader ->ElementOffset];

Element *pElement = &pElementTable [0]; // For this example, use first element

if (pElement ->Type == 6)
{
UINT64 ul64 = *((UINT64 *) &pFileData [
pHeader ->VarDataOffset + pElement ->Offset]);
}
else if (pElement ->Type == 7)
{
INT64 l64 = *((INT64 *) &pFileData [
pHeader ->VarDataOffset + pElement ->Offset]);
}
else if (pElement ->Type == 9)
{
double d = *((double *) &pFileData [
pHeader ->VarDataOffset + pElement ->Offset]);
}

The STRING and RESREF types are two more indirect types but require a bit more work to access. For both the STRING and RESREF the actual data is stored in the variable data section starting at the given offset. They both being with the length of string. Following the length, are the actual bytes of the string. The string is not NULL terminated. However, STRING and RESREF do differ. In the case of STRING, the string length is stored as a UINT32 value. In the case of RESREF, the string length is stored as a UINT8 value.

char *pFileData = AddressOfTheITPDataInMemory;
Header *pHeader = (Header *) pFileData;
Element *pElementTable = (Element *) &pFileData [pHeader ->ElementOffset];

Element *pElement = &pElementTable [0]; // For this example, use first element

if (pElement ->Type == 10) //STRING
{
UINT32 Length = *((UINT32 *) &pFileData [
pHeader ->VarDataOffset + pElement ->Offset]);
char *String = (char *) &pFileData [
pHeader ->VarDataOffset + pElement ->Offset +
sizeof (UINT32)]
}
else if (pElement ->Type == 11) //RESREF
{
UINT8 Length = *((UINT8 *) &pFileData [
pHeader ->VarDataOffset + pElement ->Offset]);
char *String = (char *) &pFileData [
pHeader ->VarDataOffset + pElement ->Offset +
sizeof (UINT8)]
}

The STRREF has two uses. First, it references a string in the dialog.tlk file. Second, it provides localized (human language specific) version of the string. The offset in the element points to a variable length structure that begins with the following 12 bytes of data.

Offset	Type	Description
0x0000	UINT32	Number of bytes included in the STRREF not including this count.
0x0004	INT32	ID of the string in the in the dialog.tlk file.  If none, then -1.
0x0008	UINT32	Number of language specific strings.  This value is usually zero.
0x000C	Total length of the structure

If the number of language specific strings is zero, then the length of the total STRREF is only 12 bytes. The text of the string can be retrieved from dialog.tlk. The value stored as the size of the STRREF will be 8 since the 4 bytes in the count aren't included. However, if the number of language specific strings is not zero, then there will be the given number of the following structure following the initial STRREF structure.

Offset	Type	Description
0x0000	UINT32	Language of the string (values unknown)
0x0004	UINT32	Number of bytes in the following string
0x0008	CHAR []	Variable length string
VARIES	Total length of the structure will always be 8 plus the length of the string

The DATREF associates a block of arbitrary binary data with the element in question. At the offset in the variable data section given in the element is a UINT32 that has the number of bytes in the DATREF. Following the UINT32 is the binary data.

The CAPREF element type allows for an element to contain an array of other elements by referencing a single entry. The offset in the element structure specifies the entry number. All the elements that entry references can be considered as the children of the element.

The LIST element type allows for an element to contain an array of other entries. For a LIST type, the offset in the element structure specifies an offset into the list table. This is a byte offset even though the table itself is just a series of UINT32 values. At the first UINT32 pointed to by the offset will be the number of entries referenced by the element. Following this count will be the actual entry numbers.

char *pFileData = AddressOfTheITPDataInMemory;
Header *pHeader = (Header *) pFileData;
Element *pElementTable = (Element *) &pFileData [pHeader ->ElementOffset];

Element *pElement = &pElementTable [0]; // For this example, use first element

if (pElement ->Type == 15) //LIST
{
UINT32 *List = *((UINT32 *) &pFileData [
pHeader ->ListDataOffset + pElement ->Offset]);
UINT32 EntryCount = List [0];
for (int i = 0; i < (int) EntryCount; i++)
{
UINT32 EntryIndex = List [i + 1];
// do something with Entry
}
}

The Variable Name Table
The variable name table is a series of variable names stored in 16 byte long character strings. Care should be taken when processing the variable name strings. If the string is less than 16 characters in length, it will be NULL terminated. However, if the variable name is 16 characters long, then it will not. The easiest thing to do is copy the string into a 17 byte character string and then set the 17th character to NULL. Thus, if the string is 16 characters long, the 17th character will force a NULL termination.

The Remaining Tables
All other data tables are purely subservient to the entry and element tables. See the section on the elements to see how this information is interpreted.

Credits

Even though I was able to decode much of the ITP format soon after the game was released, I want to thank Logxen, Seg Falt and Chanteur for the information they published about the file format. Their information filled in some very important missing elements such as CAPREF, DATREF and elements of STRREF.