New S3 transport (Amazon) in FastReport VCL

In this article, we will explore the new S3 transport. Functionally, S3 differs significantly from what most people are used to working with, so at the beginning of the article, we will take a detailed look at its main features.

What is it?

S3 is not a file storage like Dropbox, Google Drive, Mega, and others. S3 is object storage. And first of all, we need to understand the difference between these two concepts.

File storage has the usual hierarchical (tree) structure. That is, familiar directories (folders) in which other directories and files are stored.

Object storage is relatively new (the oldest are now about 20 years old). They differ slightly in their implementations, so most of the features will be common, but there will be some that only apply to S3.

All object storages are "flat," meaning they do not have a hierarchical structure, although they can "imitate" it.

The first entity you will encounter when starting to work with S3 is Bucket. The closest analogy is “Map” in many programming languages, where data is requested by key. A bucket can only store files within itself, but not other buckets.

Let's imagine the bucket more clearly.

Using the key, you can get the metadata corresponding to it and interpret it yourself as you require. Much more interesting is the imitation of the hierarchical folder structure in such a system.

Certain "folders" are the same objects as files. They are simply colored differently for visual clarity. However, they have one distinguishing feature—a slash (\) at the end of the key. Often, these "folders" do not contain any data (although they can).

To view the contents of the "folder" folder1, we need to filter by the key prefix "folder1\". But what is more interesting is deleting "folders." If we try to delete the "folder" folder1 in the Web GUI, some services will allow it, but others will give us an error saying "cannot delete the non-empty folder."

If we delete a single record with the key "folder1\" through the API, most GUIs will stop rendering that specific "folder." We will also be unable to access it, but, in reality, the files inside that "folder" will still exist. Once we create a "folder" named folder1 again, it will already contain the forgotten files. The key has very few restrictions and, for example, it can be something like "////." These will be "folders" with empty names.

Why is object storage better than file storage?

1. Object storage is significantly cheaper. Each object is not tied to a specific device, so object storage is easily expanded to the required volume. You can rent a petabyte (about 1,000,000 gigabytes) worth of object storage quite easily.

2. Object storage has a very fast search. But naturally, object storage is best used for unstructured (non-tree) data.

New authentication system

All previous transports worked with the OAuth 2 protocol. S3, in theory, can work with OAuth 2 as well, but it has its built-in authentication system in its API. OAuth 2, using the Client ID and Secret ID, obtained a token that lived for a certain amount of time and could sign any requests.

The internal authentication of S3, using the same Client ID and Secret ID, generates a new disposable token for each operation using SHA256HMAC based on about 30 variables (including token creation time, required operation, and so on). The server only verifies the correct generation of the token, and if the token received by the server matches the one generated by the server itself, the operation is performed. After that, the token cannot be used anymore.

AWS S3

Although the transport is named "TfrxS3IOTransport," clearly referencing AWS S3 (Amazon Simple Storage Service), its API has been copied by many object storage systems (which can be loosely referred to as S3-like), so this transport allows you to work with not only Amazon's service.

Adding transport to the application and its configuration 

At the very beginning, you need to add the appropriate component from the “FastReport VCL Internet transports ” tab.

You can use the context menu on a component. It will establish the connection directly from the development environment. Click on the "Edit connection" submenu.

S3 connection dialog

When you need to open or save to a file using transport, the following standard authorization dialog will be available to you (if there was no authorization previously).

Let's look at all the unique fields of this form:

Server URL—URL used in the API. To find the URL, you need to read the documentation; for AWS S3, it should be “s3.amazonaws.com” by default.

Client ID—public key for authorization (often called Access Key).

Secret ID—secret key for authorization (often called Secret Access Key).

Region—you need to read the documentation for possible region values. Sometimes, there are unusual requirements: for example, in AWS S3, you can only request a list of buckets with the “us-east-1” region.

Bucket—in order not to manually select a bucket each time, you can specify it in this field; sometimes buckets are characterized by a specific region.

Bucket style—There are 2 types of specifying a bucket: path style (aws.s3.com/bucket) and virtual hosted style (bucket.aws.s3.com. "Path style" is considered outdated because, through a specific DNS, there can be multiple requests to different buckets, which can create additional load. Some services support both types, some support only one, and some can be configured for different situations. You can read about this at the following link.

If the connection is successful, you will see the standard FastReport VCL file browser (the screenshot below shows a list of files inside the bucket).

This completes the connection setup successfully. Now you know how to connect to S3 in FastReport VCL.

It remains only to note the last nuances of our implementation. Our team has not implemented creating and deleting buckets. So far, buckets are visually no different from folders (in our first version of the GUI), as it would be too risky. Also, deleting a folder along with its contents has not yet been implemented. Also, the feature of uploading large files in parts is missing (recommended for files larger than 100 megabytes).

This transport has many configuration details and nuances, but it can be an optimal alternative to file storage in certain cases.