DukePersonServer

All related code and materials for preparation of Fall 2019 ECE564.

Features

• Pure Swift back-end setup with Kitura Framework
• First personal Server Side Swift and Ubuntu Swift attempt
• WebUI API interface based on OpenAPI Specification
• WebUI database management tool based on pgweb
• Front-end design to preview all prettified DukePerson entries and their json response on the fly!
• Full supports for REST URI HTTP Methods GET/POST/PUT/DELETE
• File upload via encoded json or multipart/form-data
• Basic user authentication control
• Improved Markdown capability based on Kitura-Markdown
• Better HTTP status code handling and user friendly 404 page.
• More to find in details!

Example

Above video demonstrates the front-end design for preview of entries. In order to make students debug easier, and to have a quick preview of either their uploaded entry is correct or not, this web interface is provided. More features can be found in the section Usage down below.

In order to have a better understanding of the APIs, the table structure in the database, and the response json format, please visit links below within DukeBlue WiFi:

1. WebClient Demo
2. OpenAPI
3. pgweb Admin Page

Also, a set of self-explanatory sample code snippets are provided under /ECE564_2019_Playground_APIdemos directory. Please refer to those playground snippets to learn how to make HTTP requests to the server APIs.

Please note: some of the APIs requires authentication or password due to certain reasons. After the semester is over, more would be disclosed in this document.

Usage

In addition to the primary function of accessing the APIs with code, the service also provides basic web graphical user interfaces to facilitate debugging and data management.

OpenAPI WebUI

The back-end service embedded in the Kitura OpenAPI package to provide a user-friendly and nice looking interface to test with all the Codable Routes available.

For example, if you want to test with the GET method of /entries_anonym API, instead of making a call with Postman or directly access the URL, you can now test it with one click here in the web page. By clicking Try it out in the corner of the API, you can see the standard request and response formats, so that you can design your struct and code accordingly.

Also, with the help of this feature, you do not need to construct your own request json to test with any POST and PUT APIs. Instead, you can just change the default template json as you wish, then call the API with correct method just in the page. It will give you the status code and response body to let you know if your request succeeded.

pgweb database admin tool

Since Kitura framework does not come with a WebUI database management tool, in contrast to Django Admin, I decide to add a third-party PostgreSQL web interface to visualize the tables and relations.

This tool is lightweight and well polished, and come with an easy-to-deploy release version. After logging into the database with PostgreSQL's credentials, you could view all the basic information of the tables, including constraints, table structure and filtered results.

Besides, pgweb allows DBAdmin to execute raw SQL statements from web page. While it does not provide any click-editing feature like Django Admin, it is still handy to construct SQL snippets to make adjustments to the tables.

Moreover, pgweb also displays the ongoing connections and previous activities, which makes it easier to track the status of the database for DukePersonServer.

API calls with Swift functions

The main feature for DukePersonServer is to create a set of standardized RESTful APIs for the students in ECE564 course. REST APIs are well supported by all the network functionalities of Swift's Foundation framework and is praised as one of the better ways for lightweight back-end communications. More will be covered about the designs and specs of APIs in next section.

Designs and highlights

Web Client template engine

As I'm not well versed in front-end programming, I chose to adapt a more mature stylesheet design from an open source project. This CSS stylesheet layout entries as cards in the page, and provides text labels and image box to exhibit basic information of an entry.

The original design does not support dynamic templating, API calls and authentication.

First, I added the capability of dynamic template generation with Stencil template engine. This is a widespread template engine written in Swift, and provides similar syntax to Liquid template engine of Jekyll. The iteration of entries as below.

By providing the iterable array in the back-end and associate it with Stencil, the web page can be generated at run time dynamically, and display the modifications in database when the page refreshes.

{% for entry in entries %}
<article class="card" style="background-color: #{{ entry.backgroundColorCode }};">
    <div class="top-content-box">
        <p>{{ entry.firstname }} {{ entry.lastname }} - {{ entry.role }} - {{ entry.netid }} <br>{{ entry.displayDate }} - {{ entry.displayTime }}</p>
        <div style="float:right">
            <input id={{ entry.id }} class="delete-button" type="submit" value="📝" onClick="getEntry(this.id)">
            <input id={{ entry.id }} class="delete-button" type="submit" value="&#10005;" onClick="deleteEntry(this.id)">
        </div>
    </div>
    <img class="add-new-image" src={{ entry.avatar }}  title="Posted by {{ entry.user }}">
</article>
{% endfor %}

Web Client Ajax API callers

Then I followed W3CSchool tutorials to implement those JavaScript/JQuery functions for each button in a very primitive way. Below is an example of how I make a GET /entries_anonym/{id} API call with JavaScript.

function getEntry(id) {
    var url = '/entries_anonym/' + id;
    var idanchor = '#' + id;
    $.ajax({
        url: url,
        type: 'GET',
        contentType: 'application/json',
        statusCode: {
            200: function(response) {
            },
            201: function(response) {
            },
            401: function(response) {
                alert('Please login to make changes.');
            },
            403: function(response) {
                alert('403: This is not your post.');
            },
            404: function(response) {
                alert('404 Not found!');
            }
        },
        success: function (data, status) {
            $("#myDialog").html(
                $('<pre>').html(
                    syntaxHighlightJSON(data)
                )
            )
            showDialog()
        }
    });
}

As you can tell, I hard-coded error messages and response actions in the function. After the call retrieves data from the API, it display it as a prettified json string in a JQuery dialog, which is more usable than printing in console log or display in an alert.

Also, the web client supports deleting entries from the page. By clicking the delete button of a card, a DELETE API call is made and the entries is delete from the database, while the front-end part just remove that element instantly.

Thanks to the stylesheet, the layout of the page is responsive and can adapt to both desktop and mobile browsers.

Markdown render engine

In addition to the Stencil template engine, I also embedded the Markdown render engine. Markdown is widely supported by modern websites, and currently you are reading a GitLab flavored markdown document! It is so helpful to draft styled web pages with simple syntax, and to avoid displaying raw-text web pages which looks like 1990's.

The engine follows CMark, and does not support many handy features that are conventionally included in de-facto standards such as GitHub flavored Markdown or so.

I've ported the markdown stylesheet from GitHub, and right now the page looks exactly the same as GitHub styles. Cool!

PostgreSQL database

As opposed to Django ORM's unified model-defined table structure and relations, Kitura ORM still preserves the differences between a table and a model. Thanks to Kitura developers' efforts to bridge some of the functionalities of tables into model definition, it is easier to create and maintain the table within the framework.

When DukePersonServer launches, it first establish a connection to the local PostgreSQL database with default pooling options. Then it tries to sync the table from database. If the table does not exists, then it will create the table according to the models defined for the table.

Limitations remains for PostgreSQL DB, and I took several detours to mitigate those issues. First is that column names in PQDB must be lowercased. By naming all the model attribute names in a snake case fashion, the problem is avoided. Another issue is the ORM does not support array of objects as its field. Hence I serialized the array of Hobbies and Languages into a json string and store it in the database. When decoding from API response, the constructor of DukePersonEntry would then de-serialize those fields to struct attributes.

Swift metrics

I also included the Swift Metrics package for monitoring purposes. It works, but is not really useful at least for now, since the service is so lightweight and does not consume much resources.

OpenAPI WebUI

As mentioned above, the WebUI has great advantage of testing APIs in situ and getting rid of desktop softwares such as Postman. By programming and providing all the response body as application/json format, the APIs increases human readability of entries, and make it easier to debug.

Besides, the WebUI also displays all the exposed data models on the page. For example, students will need to communicate with the server with a DukePersonEntry struct, and its structure and attribute's optionality is displayed on the page. Furthermore, those "inner" models will not be exposed, as far as they are not involved in the HTTP requests procedure. That means, the designer can change the table and model persistence structures without causing troubles with user APIs, which is a good separation between View and Model layers.

Finally, OpenAPI combines all the needed information together: status code, response data, host URL, HTTP method, parameter, etc. Users and students can have a unified look of the whole lifespan of API calls.

APIs

DukePersonServer service provides a complete set of methods to make CRUD operations to DukePersonEntrys table, and provides 2 different file upload APIs to upload DukePerson's avatar image. Below will dive into details of API design.

PUT capability of POST

The commonsense definitions of POST and PUT methods are as below. However, sometimes people from previous ages of Internet also expect a simplified version of REST with only GET/POST methods. Therefore, I added the update/replace functionalities to POST method, so that updating an entry can either be through POST or PUT.

The function would first check if an entry already exists in the table. If it exists, it will user the update method; otherwise it will create a new entry in the table. When updating an entry, the original UUID will be preserved.

let complete =  { (entry: UserDukePersonEntry?, error: RequestError?) -> Void in
    guard let entry = entry else {
        return completion(nil, error)
    }
    completion(DukePersonEntry(entry), error)  // return an entry consistent with database uuid
}
let netidQuery = DukePersonQueryParams(netid: entry.netid)
// since netid is set as unique in the database, no worries for collision here.
UserDukePersonEntry.findAll(using: Database.default, matching: netidQuery) { entries, error in
    guard let entries = entries else {
        return completion(nil, error)
    }
    if entries.count == 0 {  // no same netid in database, add new entry to it
        var savedEntry = entry
        savedEntry.id = UUID().uuidString
        UserDukePersonEntry(savedEntry, for: GlobalAnonymousUser).save(using: Database.default, complete)
    }
    else if entries.count == 1{  // existing entry in database, update it
        let existedentry = entries[0]
        guard let existedid = existedentry.id else {
            return completion(nil, error)
        }
        var savedEntry = entry
        savedEntry.id = existedid
        UserDukePersonEntry(savedEntry, for: GlobalAnonymousUser).update(id: existedid, complete)
    }
    else {  // database itself has error for preserving more than 2 same netid
        return completion(nil, RequestError.internalServerError)  // the default error for violating uniqueness.
    }
}

Authenticated and Anonymous APIs

At first glance of the requirements, I thought about a fully anonymous API design. But after tried with the prototype, I decided to provide both authenticated and anonymous APIs, and only allow authenticated access when it involves critical operations such as delete entry, delete user or clear a table.

For example, when I want to delete a user, the user must first provide his correct credentials. Otherwise, a 403 Forbidden will be sent as response.

func deleteUser(user: UserAuth, id: String, completion: @escaping (RequestError?) -> Void) {
    if user.id != id {
        completion(RequestError.forbidden)
        return
    }
    DukePersonEntry.deleteAll(for: user) { error in
        if let error = error {
            return completion(error)
        }
        return UserAuth.delete(id: id, completion)
    }
}

Anonymous APIs are implemented for simplicity, as beginners may mess up with the request headers. For example, when an anonymous user want to get one entry with id from the database, instead of enforcing him to provide credentials, the server can just return the result.

func getOneEntryAnonymously(id: String, completion: @escaping (DukePersonEntry?, RequestError?) -> Void) {
    let complete = { (entry: UserDukePersonEntry?, error: RequestError?) -> Void in
        guard let entry = entry else {
            return completion(nil, error)
        }
        return completion(DukePersonEntry(entry), nil)
    }
    UserDukePersonEntry.find(id: id, complete)
}

There are arguable pros and cons between authenticated and anonymous calls, and I prefer authenticated versions, since it could draw a clearer line between each user and their permissions. The biggest drawback is they introduce more complexity, and make students harder to code their requests. I'd like to let Prof. make this one a bonus feature for those arduous students to figure out how to get authenticated.

File Upload

APIs/Routes in Kitura can be divided into Codable Routes and Raw Routes.

Kitura uses the standard design of routing for many back-end frameworks. With the addition of Codable protocol introduced in Swift 4, it is even simpler to create an API as Codable, and let Kitura handle more things instead of using Raw Routes.

Raw route literally stands for the process where the developer handles parsing and decoding requests, makes transactions to database and constructs response all manually. In this way, I use the primitive method of file uploading: POST multipart/form-data.

The code is long and verbose, and I won't paste it here. Basically it first parse every part from the multipart body data, then do a checksum to check the file integrity and then save the file to disk, as well as making transactions to the database. One thing to notice is that the response text should be defined in a better way. This is included in the potential improvements list for the project.

Codable route, on the other hand, is much shorter and clearer. The process is to received a json that contains a base64 encoded string, which represents the image file entity. Decoding is something as below.

guard let file = Data(base64Encoded: file_entity_string) else {
    return completion(nil, RequestError(.internalServerError, body: CustomError.base64DecodeError))
}

After receiving the file with both methods, the code will first check if it should be an update or a creation. Also, it would force update the avatar field of a DukePersonEntry, so that the person will always have the latest avatar URL in his entry.

Finally, I also coded a delete all API to erase all the files from database and disk, to clean it up.

Custom error code and message

I plan to add more to this feature later on, and right now it is a struct with multiple struct variables that provides pre-defined response messages. It looks like the snippet below.

struct CustomError : Model {
    var statusCode: Int
    var errorMessage: String
    // example error definition
    static var base64DecodeError : CustomError { return self.init(statusCode: 501, errorMessage: "Error: base64 string conversion back failed!") }
}

Progress notes

Please refer to 👉my blog posts👈(Chinese) to see the deployment notes of this service.

Potential improvements

More to check at here.

1. Back-end sanity check
   If the student POST a mal-formatted JSON, say gender: "M" or gender: "True", the back-end should perform some sanity check before putting that into the database. A list of possible fallbacks should be discussed, for gender/degree/department/role

2. Better error messages
   Current messages only provide default describing string of the error. I'd like to enumerate all possible errors and give specified messages for each one, so it would be easier for students to debug their program.

3. Image cleaning process
   Currently the deleteAll method only delete file paths in the database. To clean those dangling files, one way is to delete them when a new file with same name but different extension is uploaded. The other way is to recursively delete all the files under avatar directory when the method is called.

Contributing

Contributions are very welcome 🙌.

According to the discussion with Prof. Telford, a weekly based template project might be created, so that students could refer to the template project after they submit their homework, to ensure they can obtain credits even if they cannot fulfill the requirements of previous step. Also, the progress can be ensured with everyone in class with same context.

You could make merge requests for more test-case snippets in playground or add features or make potential improvements mentioned above as a good start to learn about this project!

Installation

Manual

• For the server part, for macOS (not tested):

1. Install PostgreSQL database distribution release on your computer.
2. Build the project directly from Xcode
3. Test locally at http://localhost:5640

• For Ubuntu 16.04:

1. Setup Ubuntu Swift and PostgreSQL environment
2. Clone or download the project
3. Under its root directory, swift build and it will automatically download all dependencies and compile. After build succeeded, run with swift run
4. swift package clean to clear previous build
5. HTTP server and proxy with Apache, Nginx or other tools

Credits

• The template for this description is taken from SwiftKit.

• The stylesheets and frontend designs are adapted from Kitura_EmojiServer

• Referred tutorials and posts are included in my blog post1 and post2 (Chinese).

License

DukePersonServer
Copyright © Ting Chen 2019
All rights reserved. 

last revision: 190704