Gnutella Generic Extension Protocol (GGEP)

Change History

Background

The Gnutella 0.4 Protocol is a bare bones protocol for sharing files. Over time, servent implementers have targeted different sections of the protocol for extensions. An example of this is the Bear Share Trailer that was added to the Query Hit.

Today, proposals exist to pack yet more data into the existing protocol. Unfortunately, many of these were designed for a single purpose and will close off future extensions.

This document describes a standardized format for the creation of arbitrary new extensions. This new standard allows for :

• The inclusion of either text or binary data.
• Servent vendors to create extensions at will.
• Multiple extensions to be output within responses
• Unambiguous parsing

Connection Handshake Headers

Servents that have support the forwarding of all packets that contain GGEP extensions (whether or not they can process them), must include a new header in the Gnutella 0.6 connection handshake indicating this support.  This will allow other servents to know what types of packets this servent can accept.  The format of this header is

GGEP : majorversion'.'minorversion

Format

Length Encoding

The length field uses an encoding technique that ensures that 0x0 is never the value of any byte.  Steps were also taken to ensure that the encoding is compact. In this technique, a length field is the concatenation of length chunks.  The format of each length chunk (which contains 6 bits of length info) is described in bit level below:

Format

76543210			
MLxxxxxx
			

M = 1 if there is another length chunk in the sequence, else 0

L = 1 if this is the last length chunk in the sequence, else 0

xxxxxx = 6 bits of data.

01aaaaaa ==> aaaaaa (2^6 values = 0-63)

10bbbbbb 01aaaaaa ==> bbbbbbaaaaaa (2^12 values = 0-4095)

10ccccccc 10bbbbbb 01aaaaaa ==> ccccccbbbbbbaaaaaa (2^18 values = 0-262143)

Boundary Cases

• 0 = 01 000000b = 0x40
• 63 = 01 111111b = 0x7f
• 64 = 10 000001  01 000000b = 0x8140
• 4095 = 10 111111  01 111111b = 0xbf7f
• 4096 = 10 000001  10 000000  01 000000b = 0x818040
• 262143 = 10 111111  10 111111  01 111111b = 0xbfbf7f

As you see, when the bits are concatenated, the number is in big endian format.

Non-Error Checking Parsing Pseudo Code

int length = 0;
byte b;
do
{
   b = *extensionbuf++;
   length = (length << 6) | (b&0x3f);
} while (0x40 != b & 0x40);		
		

GGEP Block

GGEP Extension Prefix
GGEP Extension Header 0
GGEP Extension Data 0
.
.
.
GGEP Extension Header N
GGEP Extension Data N

Extension Block

Extension blocks may contain an arbitrary number of GGEP blocks packed one against another.  Although this behavior is allowed, developers are encouraged to merge multiple GGEP blocks into a single GGEP block.  If a newer extension format is created (either a new version of GGEP or another format altogether), they will appear AFTER the last GGEP block of an earlier version.

GGEP Block 0
.
.
.
GGEP Block N

GGEP Extension Prefix

Byte Positions	Name	Comments
0	Magic	This is a magic number is used to help distinguish GGEP extensions from legacy data which may exist.  It must be set to the value 0xC3.

GGEP Extension Header

GGEP Extension Header Flags

Bit Positions	Name	Comments
7	Last Extension	When set, this is the last extension in the GGEP block.
6	Encoding	The value contained in this field dictates the type of encoding which should be applied to the extension data (after possible compression).
5	Compression	The value contained in this field dictates the type of compression that should be applied to the extension data.
4	Reserved	This field is currently reserved for future use.  It must be set to 0.
3-0	ID Len	Value 1-15 can be stored here.  Since this will not be zero, it ensures this byte will not be 0x0.

Encoding Types

Compression Types

Usage

Ping Message

The Clip2 document states that ping messages have no payloads.  Given this definition, existing servent vendors drop connections that issue pings containing payloads.  As such, developers are suggested to allow widespread distribution of GGEP enabled servents before releasing extensions for the ping message.  Similarly, they should only forward ping messages containing GGEP extensions to other servents who have indicated their support of GGEP via the handshake header. 

Servents are instructed to forward all ping messages containing GGEP blocks they do not understand regardless of ping/pong reduction schemes.

The payload of the ping message is now defined to be:

Extension Block

Pong Message

Being that there are not currently extensions to the pong message and the last field has a fixed length, it is easy to extend this message to include GGEP.  That said, since the current pong message currently has a fixed length, existing servents may drop connections if they receive pongs containing extensions.  To this end, developers are suggested to only include GGEP blocks in response to ping messages containing GGEP blocks, as that will guarantee that the pathway is GGEP enabled.

The payload of the pong message is now defined to be:

Port

IP Address

Num Files Shared

Num KBytes Shared

Extension Block

Query Message

The Query message is redefined in a way as not to break Gnutella 0.4 compatible servents.

QHD Private Data

Until now, servent vendors have been left to define the format of this opaque field.  Ones that are able to write to this field, only read from it if they recognize their vendor code.  To this end, we must first indicate to all clients that the private data section contains a GGEP block, so they know to crack open the field.  To do this, we use an open data bit 5 of the Flags and Flag2 fields (the format of this field is defined in the Clip2 document).  Remember that the ability to crack the GGEP block does not mean that one is able to understand the extensions contained within.

For compatibility with a couple of existing servents that already use this field, it is necessary to search for the GGEP block by looking for the first occurance of the GGEP magic byte. 

Private Data Format for servents that already use the private area and are trying to retain compatibility with older versions of their code  This will be phased out over time:

Private Data Format for new servents:

Note that the code necessary to find the GGEP block in both formats is identical.

Query Hit Result

Servent vendors must be careful to ensure that 0x0 does not appear in any extension data placed embedded into the Query Hit Result.  One does this by using any of the available encoding options.

File Index

File Size

File Name

0x0

Extension Block

Push Message

Being that there are not currently extensions to the push message and the last field has a fixed length, it is easy to extend this message to include GGEP.  That said, since the current push message currently has a fixed length, it is possible that old servents will validate against that length, and throw out push messages that include GGEP extensions.  To this end, servents should only send push messages containing extension blocks to other servents that have indicated GGEP support via the connection handshake.

The payload of the push message is now defined to be:

Servent ID

File Index

IP Address

Port

Extension Block

Notes/Issues

Existing Standards

• The HUGE 0.93 specification should be modified to transmit data inside a GGEP extension.
• Bearshare sends encrypted data in the QHD private data field.  It is possible that this data could start with bytes that would pass for a valid Extension Prefix and an Extension Header.  Given the wide distribution of the Bearshare client and its usage of the field, changes to that servent may take a while to distribute to the masses.  Servent vendors implementing GGEP should ensure they gracefully handle this case.

Implementation Notes.

• Extension implementers are urged to publish the ID, format, and expected data size for their extensions in the GDF database called "GGEP Extensions."
• When encoding an extension, be sure the apply the encoding AFTER the optional compression step.  Likewise, when decoding the extension, do so BEFORE decompressing the data.
• One should only compress data if doing so will make a material difference in the resulting packet size.
• Extensions within a block should be processed in the order in which they appear.
• Please note that most gnutella clients will drop messages, and possibly connections if the message size is larger than a certain threshold (which varies according to message type).  Please pay attention to these limits when creating and bundling new extensions.
• Details about the COBS encoding scheme may be found at http://www.acm.org/sigcomm/sigcomm97/papers/p062.pdf 
• Details about the Deflate compression scheme may be found at http://www.faqs.org/rfcs/rfc1951.html and http://www.gzip.org/zlib/

Appendix A - Creating Extension IDs

The Extension ID field in the GGEP header is a binary field consisting of between 1 and 15 bytes.  It cannot contain the byte 0x0, and one must be able to compare IDs with a simple binary comparison.  Asisde from those rules, GGEP does not mandate any particular format, but does encourage the creation of short IDs that are free from conflicts.  One should also note that Extension IDs are meant to be consumed by machines.  To this end, the following techniques are recommended.  If one has a strong need to create an alternative format, be sure to avoid conflicts with the following schemes.

GDF Registered Extensions

Any Extension ID of less than 4 bytes must be stored in the appropriate GDF database.  Any Extension ID of less than 3 bytes must be also be approved by the GDF.  The format of the extension data must also be registered.

VendorID Extensions

This simple technique allows for the creation of ExtensionIDs based upon uses the following format VendorID|'.'|BinaryData VendorID for a Gnutella servent is a 4 byte value that has been registered in the GDF Peer Codes database.  In the QueryHit Descriptor, this case is case insensitive.  With ExtensionIDs, the case matters, as one must be able to perform a binary comparison on the ID.  This means an ExtensionID of "SWAP.1" and "swap.1" are different, but both "belong" the vendor who ones the code "SWAP."

Appendix B - Peaceful Co-Existence

It is unfortunately the case that GGEP extensions will co-exist with legacy extensions for quite some time.  In some cases, both may exist in the same space allocated for extensions.  The below algorithm for interpreting extension fields will help sort out such this co-existence.

1. Peek 1 byte, if is GGEP magic goto 2, else goto 6
2. Read GGEP magic byte
3. Read and process a single GGEP extension (extension headers, extension data)
4. If the extension flag do not have the Last_Extension bit set, goto 3
5. Goto 7
6. Read until end of extension field, or 0x1c, whichever comes first.  It will be up to you to determine which legacy extension exists in this space.
7. If no bytes left, quit
8. Peek 1 byte, if 0x1c or 0x0, advance 1 byte
9. Goto 1