Designing a RESTful Web Application

This blog entry is my attempt to get all the concepts of RESTful web service design straight.

By Ryan McGreal

Posted June 21, 2010 in Blog (Last Updated December 23, 2014)

1	Introduction
2	Representational State Transfer
3	Hypertext Transfer Protocol
	3.1	HTTP Requests
	3.2	HTTP Responses
4	Resources
5	HTTP Methods
	5.1	GET Method
	5.2	POST Method
	5.3	PUT Method
	5.4	DELETE Method
	5.5	Idempotence
	5.6	PUT vs. POST
6	Conceiving the Web Service: A Resource/Method Table
7	HTTP Response Data Formats
	7.1	XML
	7.2	JSON
	7.3	YAML
8	REST Best Practices

1 Introduction ↑

I'm working on a couple of projects that involve building a web service, and I decided early on that because of our business constraints - having to communicate with a variety of different systems of varying levels of sophistication - it made sense to keep the web service as simple and accessible as possible.

That pointed me toward designing a RESTful web service that transmits data in a simple format over straight HTTP. After all, just about any programming language imaginable can make an HTTP request. I also decided to go with JSON for the data format, in part because I've been experimenting lately with CouchDB and appreciate both the simplicity and flexibility of JSON and the fact that you can find a JSON parser for any language.

This blog entry is my attempt to get all the concepts of RESTful web service design straight. There's a good chance that some of this information is wrong; and if you notice something, please let me know about it. I'll investigate your argument and update the essay as applicable.

With that in mind, here we go.

2 Representational State Transfer ↑

Representational State Transfer, or REST, is a model for designing networked software systems based around clients and servers. In a RESTful system, a client makes a request for a resource on a server, and the server issues a response that includes a representation of the resource.

The concept was formalized in 2000 by Roy Fielding, one of the architects of Hypertext Transfer Protocol (HTTP), about more which below, in his doctoral dissertation. It is not surprising, then, that REST and HTTP mesh very smoothly.

A RESTful client-server system is stateless, meaning each request against the server contains all the information the server needs to process it; and cacheable, in that the server can specify whether and for how long resource representations can be cached either locally on the client or on intermediate servers between the client and the server.

3 Hypertext Transfer Protocol ↑

Hypertext Transfer Protocol (HTTP) is a stateless protocol based on a client requesting a resource across a network and the server providing a response. As such, an HTTP transaction entails a request and a response. The request goes from the client to the server, and the response goes from the server back to the client.

3.1 HTTP Requests ↑

An HTTP request has three parts:

The request line, which includes the HTTP method (or "verb"), the uniform resource identifier (URI), and the HTTP version. E.g.

GET /articles/1/ HTTP/1.1
One or more optional HTTP headers, which are key/value pairs that characterize the data being requested and/or provided.
An optional message body, which is data being sent from the client to the server as part of the request.

3.2 HTTP Responses ↑

An HTTP response also has three parts:

The HTTP status code, indicating the status of the requested URI, e.g.

HTTP/1.1 200 OK
One or more optional HTTP headers, which are key/value pairs that characterize the data being provided.
An optional message body, which is the data being returned to the client in response to the request.

4 Resources ↑

HTTP deals in resources. Each URI points to a resource on the server. Think of URIs as nouns, not verbs, with one URI for each resource.

For example, if you want to add the ability to create an article, it might be tempting to create a URI called /create_article. This is wrong, because it conflates the object (the resource) and the action (creation). Instead, it makes more sense to have a resource called /articles and a method that lets you create articles.

5 HTTP Methods ↑

HTTP defines several methods, or "verbs", to execute on a resource: HEAD, GET, POST, PUT, DELETE, TRACE, OPTIONS, CONNECT, and PATCH. However, the following four are most commonly used in web services:

5.1 GET Method ↑

To retrieve a resource, issue an HTTP GET request. GET requests are idempotent (see below), which means making a GET request multiple times does not cause any change in the resource that is requested.

GET requests do not include a message body, but GET responses usually do.

5.2 POST Method ↑

To submit data to be processed, issue an HTTP POST request. POST requests require a message body, i.e. the data to be processed.

For example, if there is a resource called /articles and you want to add a new article, issue a POST request to /articles with the content. The server will create a new subsidiary URI under /articles - for example, /articles/9001 - and assign that URI to the content you sent with your POST request.

Important note: POST requests are not idempotent (see below), meaning multiple POST requests will create multiple resources with unique identifiers.

5.3 PUT Method ↑

To place content at an existing resource, issue an HTTP PUT request. For example, if there is a URI /articles/9001 and you want to replace the content served at that URI, issue a PUT request to that URI with the new content.

Important note: PUT requests are idempotent, i.e. issuing 1 or 5 or 50 identical PUT requests will have the same effect on the resource. PUT requests must include a message body (the resource to be placed at the URL).

5.4 DELETE Method ↑

To remove a resource (and remove its accompanying URI), issue an HTTP DELETE request. DELETE requests should be idempotent, i.e. issuing 1 or 5 or 50 identical DELETE requests will delete exactly one resource. DELETE requests do not require a message body.

5.5 Idempotence ↑

This funny-looking word is crucial to designing an effective web service. A request is idempotent if issuing it more than once does not change the resource state beyond issuing it just once. Read that again if you have to.

For example, a DELETE request is idempotent if the first request deletes a resource at a URI, and the second request does nothing because the resource at that URI is already deleted.

For another example, a PUT request is idempotent if the first request updates a resource at a URI, and the second request updates the same resource in the same way at the same URI.

A request is not idempotent if issuing it more than once does change the resource. For example, a POST request to add a comment to a document is not idempotent, if issuing the POST request twice adds the comment twice (so that the document contains two identical comments with separate URLs).

5.6 PUT vs. POST ↑

My original understanding of HTTP methods was that you would use PUT to create a resource and POST to update it. This seems in keeping with common sense, but it breaks down when you apply the all-important filter of idempotence.

An interesting discussion on Stack Overflow tackles this issue, but what convinced me to change my mental model was this observation:

POST creates a child resource, so POST to /items creates a resources that lives under the /items resource. Eg. /items/1.

PUT is for creating or updating something with a known URL.

This collision between the inclination to regard PUT as creating and POST as updating is a significant source of confusion about how to well-design a RESTful system, and deserves more attention.

Furthermore, it's tempting to assume that the HTTP verbs line up precisely with the SQL CRUD verbs, but while they're superficially similar, they're not identical. Treating them as such leads to this kind of gotcha.

It's important to keep the logic of HTTP methods separate from the logic of SQL queries, and to develop specialized appropriate logic between the two domains that ensures the data processing on the server produces responses that satisfy the requirements of the HTTP methods (particularly in respect to idempotence).

6 Conceiving the Web Service: A Resource/Method Table ↑

At a conceptual level, a RESTful web service API is a matrix of resources and methods that exposes the functionality of the service to third party applications. Below is an example of what that matrix might look like.

Again, note well that actions are not mapped to URIs. A resource is an object, a noun, and the action inheres to the HTTP Verb, not to the URI. As a result, the same resource URI can serve different responses (corresponding with different actions) depending on the HTTP Verb.

REST Resource/Action Matrix
Request				Server Action	Response	Idempotent
Resource	Parameters	Method	Data	Server Action	Response	Idempotent
/articles		GET		makes a list of articles	returns a list of articles	Yes
/articles		POST	article details	creates a new article	returns confirmation and id	No
/articles	/id	GET		gets article details	returns article details	Yes
/articles	/id	PUT	new article details	updates article details	returns confirmation and updated article details	Yes
/articles	/id	DELETE		deletes an article	returns confirmation of deleted article	Yes

This is the RESTful way to organize a web service: URIs are objects and HTTP Verbs are actions performed on those objects.

7 HTTP Response Data Formats ↑

Every web server is also a RESTful web service, accepting GET and POST requests to particular resources, and then performing actions and serving data in response.

A conventional web server delivers its data in HTML format (text/html), with related Javascript (text/javascript), CSS (text/css) and image files. HTML is an excellent format for marking up textual data for human use, but it has very limited expressive power for structuring data beyond simple documents.

The most common formats used to transmit structured data across HTTP are XML and JSON, with an honourable mention for YAML.

7.1 XML ↑

XML, or eXtensible Markup Language, is a markup (i.e. tag) based syntax based on SGML for formatting structured, text-based data. XML is a format in which to create domain specific markup languages that define particular data structures.

For example, RSS and Atom are XML language standards defined to structure documents published to websites so that the documents can be 'syndicated' to feed readers and third party sites for display.

Likewise, the default underlying structure of Microsoft Office documents since Office 2007 is an XML language called OOXML.

XML structures data by defining elements, properties, data types and allowable nesting rules in an XML schema called a Document Type Definition, or DTD.

A given XML document specifies which DTD schema should define its structure with a Document Type Declaration, or DOCTYPE.

A given XML document can reference multiple DTDs by using namespaces.

Here is a sample XML file containing contact information about a person, taken from Wikipedia.




    John
    Smith
    25
    
        21 2nd Street
        New York
        NY
        10021
    
    212 555-1234
    646 555-4567

XML must be:

Well-formed - elements are properly nested, tags are properly closed and match case, special characters are escaped, and Unicode characters are encoded; and
Valid - its elements and attributes match the rules defined in the DTD.

An XML Language called XSLT can be used to map XML documents into other markup languages, e.g. HTML.

XML is in wide use in applications that transfer structured data over the internet.

7.2 JSON ↑

JSON, or "JavaScript Object Notation", is a lightweight data format introduced in 2001 by Douglas Crockford.

Pronounced "Jason", JSON is based on JavaScript object literal notation, a syntax for creating objects in JavaScript by literally describing their properties and methods. Here is the JSON equivalent to the XML code in the previous section.

{
    "firstName": "John",
    "lastName": "Smith",
    "age": 25,
    "address": {
        "streetAddress": "21 2nd Street",
        "city": "New York",
        "state": "NY",
        "postalCode": "10021"
    },
    "phoneNumber": [
        { "type": "home", "number": "212 555-1234" },
        { "type": "fax", "number": "646 555-4567" }
    ]
 }

While the XML above contained 367 characters (not including indentation), the equivalent JSON contains only 272 characters - only three-quarters as large.

JSON supports lists (ordered sets of values) and dictionaries (unordered collections of key/value pairs) with arbitrary nesting and various data types: number, string, boolean, list, *object and null.

Like XML, JSON is also in wide use in applications that transfer structured data over the internet.

A major advantage over XML is that the syntax is much simpler and less verbose, which makes it lighter across networks as well as more human-readable.

Mature JSON parsers are available for a wide range of programming languages in addition to JavaScript. Python, for example, includes a json parser as part of its standard library (as of version 2.6; earlier versions can use the third-party simplejson library).

A decent JSON parser converts an object back and forth between the programming language's native data types and their JSON equivalents. That way, an application can receive a JSON object, convert it into a native object, process it natively, and then convert the final result back to JSON to be dispatched elsewhere.

7.3 YAML ↑

YAML, pronounced to rhyme with "camel", is a recursive acronym meaning "YAML Ain't Markup Language". Whereas JSON is a more minimal data format than XML, YAML takes minimalism to an extreme, eschewing quotation marks, brackets and curly braces altogether in favour of significant indentation and line breaks.

The YAML equivalent to the XML and JSON contact examples above would be:

firstName: John
lastName: Smith
age: 25
address: 
    streetAddress: 21 2nd Street
    city: New York
    state: NY
    postalCode: 10021
phoneNumber: 
    - type: home
      number: 212 555-1234
    - type: fax
      number: 646 555-4567

That works out to just 239 characters - including the significant white space.

8 REST Best Practices ↑

A RESTful web API ought to be discoverable by its users. One crucial way to do that is by making sure that each resource in your API includes URLs to subsidiary URLs.

Here is an example, using JSON:

{
"articles": 
    [
        {
            "title": "My First Baguette",
            "description": "After a month of reading about how to make baguettes, I finally took the plunge today.",
            "date_published": "2011-07-11",
            "url": "http://quandyfactory.com/blog/81/my_first_baguette"
        },
        {
            "title": "Tenn. Passes Controversial Lawnmower Theft Bill",
            "description": "The lawnmowing industry has successfully lobbied the Tennessee State Government to pass a groundbreaking law making it a criminal offence to loan your lawnmower to a neighbour.",
            "date_published": "2011-06-02",
            "url": "http://quandyfactory.com/blog/78/tenn_passes_controversial_lawnmower_theft_bill"
        }
    ]
}

When you do this, you make it easy for API users to discover your API structure and functionality without having to keep referring to obscure documentation.

Special thanks to Adrian Duyzer for reading a draft of this essay and setting me straight on the respective roles of PUT and POST.