GTP: Generic Transfer Protocol

Written By: Seairth Jacobs
Document Status: Draft : Revision 4.2
Last Updated: July 23, 2002

Preface to the Preface

The first thing I should point out is that this is a "work in progress". It is, by no means complete. Further, it is possible that some of this is subject to change. As I get feedback, this document will be improved upon. (For instance, I am looking for a better name that GTP). Ultimately, I hope it will lay the foundation for a new and simple transfer protocol that can be used for the next generation of networked applications. Happy reading...

Note: Wherever I show markup, I use ONX. Even without reading the ONX specs, it should be fairly easy to understand the examples given. Also, I believe that the same concepts could be applied to XML or another markup language or syntax just as effectively.

Note: I use the formal ContainerNode and ValueNode format within this document only to increase readability. As with any ONX infoblock, this is not a requirement. All sample ONX infoblocks below could be represented in the compact format. In an actual implementation, the compact format would be preferable to the formal format to reduce network overhead and processing time.

Preface

RPC (Remote Procedure Calls) has become a hot topic lately. There is a good reason for this: as web technologies such as XML evolve, there has been a shift in the focus of application design away from traditional monolithic applications towards distributed comonents. With an increased emphasis on making the interfaces of these components public and standardized, RPC has been reintroduced as the glue that allows all of these components to work together. However, many of the current RPC implementations are based on traditional RPC techniques (and are sometime even piggybacked on other existing protocol such as HTTP) and do not necessarily address the needs of emerging technologies.

What is needed is a new RPC model that meets the needs of both existing and emerging technologies while keeping it both simple enough to use and flexible enough to meet needs that haven't been thought of yet. A single general-purpose RPC model would provide only a standardized basic foundation for RPC applications of all sorts. An RPC application based on this foundation would be able to focus on its specific needs instead of the underlying RPC mechanisms. Further, a new class of applications would be able to be developed which would understand the RPC mechanisms and act upon them without necessarily needing to know anything about the application is layered on top of it.

With all of these thoughts in mind, I started working on a new "RPC" model to meet the needs I just mentioned. The reason that "RPC" here is quoted is that the model I am working on is very generic and tagging it as an RPC implementation may be too restrictive a view in some peoples minds. On the other hand, RPC is a popular buzzword right now and may also help to bring attention to a concept like this. In the end, however, I have chosen to continue to refer to GTP as a general-purpose RPC model to avoid additional confusion.

And now, the Generic Transfer Protocol...

Verbs

Every RPC must contain a verb. This verb defines the action that should be taken. In data manipulation, there are five basic possible actions: ADD, EDIT, DELETE, GET, and PUT. All other actions can be defined as a combination of these basic verbs. Each of the verbs have a simple, yet well-defined meaning:

• ADD : This verb adds a resource.
• EDIT : This verb alters a resource.
• DELETE : This verb removes a resource.
• GET : Unlike the first three verbs, this verb does not explicitly alter a resource. Its only purpose is to retrieve a resource. This verb is idempotent.
• PUT : Like the GET verb, this verb does not explicitly alter any data. Its only purpose is to submit data. However, this verbs is not idempotent.

PUT Verb

Normally, the first four verbs would be sufficient to handle all forms of data manipulation. However, the fifth verb (PUT) is also a part of all data operations, but in a more implicit manner. For instances, suppose a GET request is sent from a client application to a server. The server would respond with an answer of some sort. This answer could be the data asked for in the GET request, or it could be an error. In either event, the server must send back a response to the client. For now, think of this return operation as a PUT.

Another example might be when you want to implement a function that looks like: OldValue = UpdateData(NewValue). In this case, the server may use the PUT verb to send back the old data.

Yet another example is when the PUT is used by the client to send data to the server. For instance, think of the HTTP POST operation. Here, HTTP is taking some data and sending it to the server. The server takes the data and does whatever it wants with it (the client has no control over the server's internal actions). The server then generates a response and sends it back to the client.

This brings up the question of what distinguishes a GET from a PUT. A GET is a request for something to be sent back. The GET should not cause the requested resource to be altered in any way. Usually, the GET request contains little to no data beyond what is necessary to identify what the request is asking for. A PUT, on the other hand, is for sending data without specifically saying what action should be taken by the recipient. As a result, a PUT may have no more effect on a resource than a GET or could cause the resource to be altered like an EDIT, ADD, DETELE, or any composite of those verbs. A PUT may or may not get a response back depending on the appliction implementation (see below).

Composite Actions

By combining these verbs into different sequence (along with a bit of conditional testing in most cases), more complex actions can be taken. In other words, the above five verbs are the basic building blocks on which all other "verbs" can be defined.

For instance, suppose you want to do a sequence like "lock, replace, unlock". You could perform these actions with the following pseudo-code:

IF SUCCESS(ADD LockOnResource)
    
    Resource = GET ResourceID
    
    IF SUCCESS(Resource)
        IF FAIL(EDIT ResourceID + Resource)
            ERROR "Failed To EDIT Resource"
        ENDIF
    ELSE
        IF FAIL(ADD ResourceID + Resource)
            ERROR "Failed To ADD Resource"
        ENDIF
    ENDIF

    DELETE LockOnResource
ELSE
    ERROR "Failed To LOCK LockOnResource"
ENDIF

This simple example shows how the verbs could be used to create a more complex sequence of actions. If were were to look at the actual messages being sent back and forth, it might look like this (assuming the remote resource exists):

Nouns

With the above verbs, any action can be performed given a sufficiently descriptive noun or set of nouns. In this case, a noun refers to the target of the verb. The noun could be anything from a specific piece of data to a highly abstract concept. While GTP is limited to five verbs, there is no limit to what the nouns can be. However, all possible nouns fall into a limited number of categories. These categories will look familiar:

• Where : noun describing the location of the target.
• What : noun describing the contents of the source.
• When : noun describing the timing of the verb in relation to the target.
• Who : noun describing the owner of the verb and the noun(s) sent.

The two remaining "categories" are not directly related to nouns, but are included here for thoroughness. They are:

• How : this is actually the verb that describes the action to take.
• Why : this isn't necessarily a noun as far as GTP is concerned. Instead, the "why" is the purpose behind the RPC. This drives the design and implemenation requirements as well as the limitations on all of the other categories. This is examined in more detail below in "Applications".

Where Nouns

The Where noun identifies the target of the verb. The most common example of a Where noun is the host name and path in a URL. An example of this would be "www.seairth.com/web/rpc/". Another recognizable Where noun would be an e-mail address such as "seairth@seairth.com".

What Nouns

The What noun identifies the contents or subject of the verb. For instance, in an HTTP GET request, the What noun might be the requested filename, such as "gtp.html". In an HTTP POST request, the What noun is the data being sent in the content section. When sending an e-mail, the What noun would be considered the contents of the message (Subject, Body, etc).

Note: Commonly, the filename in the URL for a POST operation is the name of the application that will process the data and return a response. In this case, it is more appropriate to think of the filename as part of the Where noun.

When Nouns

The When noun identifies the timing relationship of the verb to the target. Most commonly, the implicit value for this noun is "now". However, with the increase in distributed programming, it is foreseeable that some applications may require more explicit control over the timing of the verb acting upon the target. Frankly, I cannot think of a good example for this, but I am sure there are many out there right now.

Who Nouns

The Who noun identfies the owner or originator of the verb and its related nouns. In HTTP, many would consider this to be an implicit "anonymous" person. However, there is more that describes the originator, such as the User-Agent, the Accepts list, etc. In their entirety, these nouns define the originator and are therefor all part of the Who noun group. Another example would be the "From" header in an SMTP message, as well as the "x-agent" header and anything else that is required to describe the originator to the recipient.

Applications

GTP does nothing by itself. It is not capable of transfering files, sending e-mail, or even giving the time of day. This is the job of applications. Applications use GTP as the underlying mechanism for transfering messages in a standardized format. Applications then lay an additional application-specific layer on top of GTP according to its needs or specifications.

GTP itself doesn't care what the application actually does. Its only concern is ensuring that certain parts of the application message conform to a generalized format that a GTP-enabled application will be capable of processing (though not necessarily understanding). However, for an application to have an better understanding of a GTP message, it is important to have an easily identifiable moniker that indicates what specification the message conforms to. For instance, a GTP application that transfers hypertext might have an application moniker of "HypertextGTP". By including this in the GTP message, it would be possible for each GTP-enabled application to look at the message and determine if it understood all of the message or not. In this manner, an application that looked for the "MailGTP" moniker would not recognize the "HypertextGTP" moniker and might reject it or pass it on to another GTP-enabled application for further inspection.

The actual value of the moniker could be anything. For instance, a simple word or phrase like "HypertextGTP" could be used. On the other hand, a URL like "http://www.seairth.com/web/apps/HypertextGTP.html" might decrease the possibility of ambiguity. However, I do not suggest using http URLs unless they are resolvable (point to an actual resource). If you want to to have a descriptive moniker and not have it be resolvable, I suggest looking into URNs or PublicIDs. Otherwise, the moniker is not required to be globally unique (though this is strongly recommended), but it should be unique within the problem domain that it is being used. The only requirement is that any application that understands a given specification must use the moniker associated to that specification.

Putting It All Together

Now that we have verbs and nouns to work with, the next thing we need is a standard format to convey them with. The simplest approach might be to use the preceding specification to help construct the format. As a result, we will start with the most basic part of the format: the GTP wrapper.

:onx{
    :gtp{
        
    }gtp
}onx

This identifies the ONX infoblock as being a GTP message. The verb(s) and noun(s) for the message will all be inside of this ContainerNode.

The next level of the message is a bit more complicated. However, to give us somewhere to start, here is a possible example of the format:

:onx{
    :gtp{
        :type["moniker of the specification goes here"]
        :to{
            where noun(s) go here
        }to
        :from{
            who noun(s) go here
        }from
        :when{
            when noun(s) go here
        }when
        :Action{
            what nouns(s) go here
        }Action
    }gtp
}onx

In the above format, only the type[] ValueNode and Action{} ContainerNode are required. The to{}, from{}, and when{} ContainerNodes can optionally be used based on the application requirements. For the ContainerNode labeled "Action", the actual name of the node can be any of the verbs: add{}, edit{}, delete{}, get{}, or put{}. Also, there can be more than one Action ContainerNode within the same GTP message. A few simple examples of this format might be:

:onx{
    :gtp{
        :type["HypertextGTP"]
        :to{ :Host["www.seairth.com"]}
        :from{
            :User-Agent["GTP Browser" "0" "1"]
            :Accept["*/*"]
        }from
        :get{ :Resource["/web/rpc/gtp.html"]}
    }gtp
}onx

:onx{
    :gtp{
        :type["MailGTP"]
        :to{ :Address["seairth@seairth.com"]}
        :from{ :Address["info@seairth.com"]}
        :put{
            :Message["Check this out..."]
            :Attachment{
                :MIME-Type["text/html"]
                :Value["http://www.seairth.com/web/gtp.html"]
            }Attachment
        }put
    }gtp
}onx

:onx{
    :gtp{
        :type["FileGTP"]
        :to[ :Host["ftp.seairth.com"]}
        :from{ :User["seairth" "mypwd"]}
        :get{ :file["/path/file.ext"]}
        :delete{ :file["/path/file2.ext"]}
        :add{ :file["/path/file3.ext" "contents of the file"]}
    }gtp
}onx

As can be seen, the actual contents of the ContainerNodes were different based on the application. However, the underlying structure of GTP is identical in both cases.

Node Order

One of the challenges in constructing an effective format or structure for a GTP message is the difficulty in determining which piece of information should come before or after other information. The reason for this is that different applications will be more focused on different things. For instance, an application that forwards messages may first want to examine the type[] ValueNode. Then if it is a message that it can forward, it might want to look at the to{} ContainerNode next, while ignoring all other parts of the message. On the other hand, an application that is responsible for authenticating the sender of a message might want to look at the type[] ValueNode followed by the from{} ContainerNode. It may even want to look at the name of the Action{} ContainerNode next to determine if the sender has sufficient rights to perform the given action. As can be seen, the best order of the information varies based on the applications function. And since the same message may be examined by more than one application, it is not necessarily possible to arrange the information on a specification-by-specification basis.

Since it is not possible to please all parties, the order of the nodes have been restricted to help improve general processing times. First, the type[] ValueNode must always be first. After that, the suggested order of the next three fields is to{}, from{}, and when{}, but may come in any order. Finally, the Action{} ContainerNode(s) must come last. When there are more than one Action{} within a given GTP message, they must be processed in the order they are received.

Implementing Communication Models

There are many communication models available when working with distributed applications. These models have two attributes that, together, define the possible communications models between two paries:

• Connection method : The connection method indicates how two parties communicate with each other. This is a range from "highly disconnected" to "highly connected". On the "Highly Disconnected" end, one party may send data to another party without ever knowing if the data arerived or not. One example of near this level of connectivity would be UDP. On the "Highly Connected" end, the two parties are fully aware of their connection status to one another at all times. TCP is an example closer to the "Highly Connected" end of the spectrum.
• Relationship : This is can be viewed as the amount of control one party has over the other party. This is a range from "full control" to "no control". Each party falls somewhere along this range in relationship to another party. If you were to look at an application accessing a database server, the application might have some level of control over the server, while the server has little or no control at all over the application. On the other hand, if you looked at a simple web browser and server serving up static pages, the client would have little control over the server, but the server has some control over the client (what HTML is sent back to be displayed). In the middle, you might have some sort of peer-to-peer application like a chat program in which each party has the same amount of control over the other.

With GTP, the possible combinations of communication models is very flexible. GTP itself does not put any restrictions or requirements on the underlying connection method used. In Internet applications, GTP could be used equally well over UDP or TCP. Also, GTP works well regardless of the relationship model used in the application. In fact, there is nothing in GTP that says that a sent message should be followed by a recieved message. An application could just send GTP messages into the ether if it wanted to.

More Examples

To help see how GTP might work in some more cases, here is an example or two:

Example 1

Converter Server: suppose you need a general purpose server that will do conversions for you of whatever type (that's supported anyhow). Since this may be a pay service (say per conversion in this case), we need to "pay" first. One thing to note about the following example is the use of a generic host name "gtp.seairth.com". The reason for this is that the host could be a generic server that handles GTP messages. It would then look at the "TYPE" ValueNode to determine which server module would be responsible for processing the request. The message sequence looks like:

GET Authorize Message (1)

:onx{
    :gtp{
        :type["ConversionsBySeairthCom"]
        :to{ :Host["gtp.seairth.com"]}
        :get{
            :ConversionTokens{
                :Count["2"]
                :Account{ :ID["SEA123"] :Password["mypwd"]}
            }ConversionTokens
        }get
    }gtp
}onx

PUT Authorized Message (2)

:onx{
    :gtp{
        :type["ConversionsBySeairthCom"]
        :from{ :Host["gtp.seairth.com"]}
        :put{ :ConversionTokens["123456" "123462"]}
    }gtp
}onx

GET Conversion Message (3)

:onx{
    :gtp{
        :type["ConversionsBySeairthCom"]
        :to{ :Host["gtp.seairth.com"]}
        :get{
            :Conversion{
                :Token["123456"]
                :Currency{ :Value["100.00"] :From["USD"] :To["CAD"]}
            }Conversion
            :Conversion{
                :Token["123462"]
                :Weight{ :Value["34"] :From["kg"] :To["lbs"]}
            }Conversion
        }get
    }GTP
}ONX

PUT Conversion Message (4)

:onx{
    :gtp{
        :type["ConversionsBySeairthCom"]
        :from{ :Host["gtp.seairth.com"]}
        :put{
            :Conversion{
                :Token["123456"]
                :Currency: :Value["159.31"]}
            }Conversion
            :Conversion{
                :Token["123462"]
                :Weight{ :Value["74.96"]}
            }Conversion
        }put
    }gtp
}onx

Example 2

Chat program: In this scenario, the chat client connects to a chat server, gets the list of other users, and starts sending messages to one user in particular. The message sequence looks like:

ADD User Message (1)

:onx{
    :gtp{
        :type["ChatGTP"]
        :to{ :Host["chat.seairth.com"]}
        :add{
            :Session{:User["seairth" "mypwd"]}
        }add
    }gtp
}onx

PUT Success Message (2)

:onx{
    :gtp{
        :type["ChatGTP"]
        :from{ :Host["chat.seairth.com"]}
        :put{ :Session{ :ID["123"]}}
    }gtp
}onx

GET UserList Message (3)

:onx{
    :gtp{
        :type["ChatGTP"]
        :to{ :Host["chat.seairth.com"]}
        :from{ :Session["123"]}
        :get{
            :UserList{
                :Filter{ :User{ :ID["Lynn"]}}
            }UserList
        }get
    }gtp
}onx

PUT UserList Message (4)

:onx{
    gtp{
        :type["ChatGTP"]
        :to{ :Session["123"]}
        :from{ :Host["chat.seairth.com"]}
        :put{
            :UserList{
                :User["Lynn" "456"]
            }UserList
        }put
    }gtp
}onx

PUT User Message (5)

:onx{
    :gtp{
        :type["ChatGTP"]
        :to{ :Host["chat.seairth.com"] :Session["456"]}
        :from{ :Session["123"]}
        :put{ :Message["Hey. It's me. Do you have time to chat right now?"]}
    }gtp
}onx

PUT User Message (6)

:onx{
    :gtp{
        :type["ChatGTP"]
        :to{ :Session["456"]}
        :from{ :Host["chat.seairth.com"] :Session["123"]}
        :put{ :Message["Hey. It's me. Do you have time to chat right now?"]}
    }gtp
}onx

PUT User Message (7)

:onx{
    :gtp{
        :type["ChatGTP"]
        :to{ :Host["chat.seairth.com"] :Session["123"]}
        :from{ :Session["456"]}
        :put{ :Message["Not right now. Maybe in an hour or so..."]}
    }gtp
}onx

PUT User Message (8)

:onx{
    :gtp{
        :type["ChatGTP"]
        :to{ :Session["123"]}
        :from{ :Host["chat.seairth.com"] :Session["456"]}
        :put{ :Message["Not right now. Maybe in an hour or so..."]}
    }gtp
}onx

DELETE User Message (9)

:onx{
    :gtp{
        :type["ChatGTP"]
        :to{ :Host["chat.seairth.com"]}
        :from{ :Session["123"]}
        :delete{ :Session{ :ID["123"]}}
    }gtp
}onx

Closing Thoughts

While this specification may not be complete, I feel that it is a good start. It provides for a good foundation on which new (and maybe even modified version of existing) applications can be placed. Or, at the very least, it provides some good food for thought (I hope). Having said that... any thoughts? Corrections? Nits to pick? Let me know. I would love to get some feedback and improve this document (which I am sure it needs).