File Service

The File Service is a web service that can be used to store and retrieve files.

Basic concepts

Immutability

Files stored by the File Service are immutable. You may DELETE a file, but a file can never change. To update a file you will need to upload a new version and refer to that one instead.

Parameters

How files are stored is controlled by three parameters: space, owner and path.

Space

The space parameter determines the kind of storage being used.

The tmp space is intended for files with a limited life-span, typically no more than a few days.

The content space is intended for files that should be stored permanently.

Owner

The owner parameter is used to group files.

In the tmp space the owner would typically be a user ID.

In the content space the owner would be the main alias of the content it belongs to (the implicit contentid/ namespace being skipped).

May contain any character that is a allowed in a URI path segment. May not contain a / character.

Path

Uniquely identifies the file within a space and host. May contain any character that is a allowed in a URI path, including /.

Storage

The file data is stored in Amazon S3, or a compatible web service such as minio.

See Service Configuration for example configuration.

N.b. The S3 bucket should be configured to expire objects with a key that begins with tmp after a few days.

Web Service API

File operations

These are the file operations supported by the File Service.

By default the file service is deployed at http://ace-file-service:8082/ inside Docker, and this is used in the examples below.

When a file is uploaded, a URI is generated to identify the file. This URI should be used when fetching the file from the file service. Note that while scheme, host and path are normally used to generate the URI, you must use the returned URI as it may rewrite some or all of the parameters.

Upload file

To upload a file, make a POST request to a sub path of the service containing the desired scheme, optionally specifying host and path. The relative URL format is /file/$scheme[/$host/$path]. If not specified, the host and path parameters will be generated.

NOTE: The file name parameter supplied to the file upload endpoint will not be the actual file name of the file in the file storage. See below for more information.

Example: uploading a file using only scheme

$ curl -i -s -X POST \
    -H "Content-Type: text/plain" \
    -H "Accept: application/json" \
    -H "X-Auth-Token: $TOKEN" \
    --data-binary @lorem.txt \
    "http://ace-file-service:8082/file/tmp"

HTTP/1.1 201 Created
Date: Tue, 13 Aug 2019 06:51:49 GMT
Ace-Response-Origin: FileService (HvKMcLajsvMy)
Ace-Api-Version: 1.9.1
Location: http://ace-file-service:8082/file/tmp/admin/11538b64-450e-4aa9-b1a9-ee33c573a834
Content-Type: application/json
Content-Length: 166

{
  "URI":"tmp:admin/11538b64-450e-4aa9-b1a9-ee33c573a834",
  "length":93,
  "creationTime":1565679109000,
  "modifiedTime":1565679109000,
  "accessTime":-1,
  "mimeType":"text/plain"
}

Example: uploading a file using scheme, host and path

$ curl -i -s -X POST \
    -H "Content-Type: text/plain" \
    -H "Accept: application/json" \
    -H "X-Auth-Token: $TOKEN" \
    --data-binary @lorem.txt \
    "http://ace-file-service:8082/file/tmp/admin/file.txt"

HTTP/1.1 201 Created
Date: Tue, 13 Aug 2019 06:53:04 GMT
Ace-Response-Origin: FileService (HvKMcLajsvMy)
Ace-Api-Version: 1.9.1
Location: http://ace-file-service:8082/file/tmp/admin/01199a12-837f-42ef-a5a8-707bf3d0968d
Content-Type: application/json
Content-Length: 166

{
  "URI":"tmp:admin/01199a12-837f-42ef-a5a8-707bf3d0968d",
  "length":93,
  "creationTime":1565679184000,
  "modifiedTime":1565679184000,
  "accessTime":-1,
  "mimeType":"text/plain"
}

Note the Location header and the URI property of the response JSON. The uploaded file automatically receives a globally unique file object ID even if a file path was specified in the file upload endpoint URL.

Also note that the URI response JSON property value can not be used directly with the ACE file APIs. Clients should either use the Location header or build the path using the scheme, host and path components of the URI response property.

Maximum file size

The File Service will only accept uploads of files with a maximum size less than the configured maximum size. File uploads exceeding the maximum file size will be rejected with an HTTP 413 Payload Too Large response. Please see the File Service Configuration for example configuration.

Chunked encoding

The File Service support uploading files using chunked transfer encoding which can be useful when streaming data where the total size is unknown beforehand. Chunked uploads still needs to fit within the configured maximum file upload size and will be rejected whenever the data stream exceeds that value.

Example:

$ curl -i -s -X POST \
    -H "Content-Type: text/plain" \
    -H "Accept: application/json" \
    -H "X-Auth-Token: $TOKEN" \
    -H "Transfer-Encoding: chunked" \
    --data-binary @lorem.txt \
    "http://ace-file-service:8082/file/tmp/admin/file.txt"

HTTP/1.1 201 Created
Date: Tue, 13 Aug 2019 06:53:04 GMT
Ace-Response-Origin: FileService (HvKMcLajsvMy)
Ace-Api-Version: 1.9.1
Location: http://ace-file-service:8082/file/tmp/admin/01199a12-837f-42ef-a5a8-707bf3d0968d
Content-Type: application/json
Content-Length: 166

{
  "URI":"tmp:admin/01199a12-837f-42ef-a5a8-707bf3d0968d",
  "length":93,
  "creationTime":1565679184000,
  "modifiedTime":1565679184000,
  "accessTime":-1,
  "mimeType":"text/plain"
}

Download a file

To download a file, make a GET request to the Location returned from the previous POST.

$ curl -i \
    -H "X-Auth-Token: $TOKEN" \
    "http://ace-file-service:8082/file/tmp/sysadmin/90a45ff9-79f3-47b9-87cd-6b8399169957"

HTTP/1.1 200 OK
Server: Apache-Coyote/1.1
Cache-Control: private, no-transform, max-age=31536000
Content-Type: text/plain
Content-Length: 395

Lorem ipsum dolor sit amet, consectetur adipiscing elit.
Mauris dignissim interdum sem a sollicitudin. Mauris orci nibh,
suscipit a justo nec, facilisis blandit arcu. Suspendisse potenti.

NOTE: The value of the Content-Type HTTP header returned when retrieving a file from the File Service will be based on the actual file data as determined by Apache Tika.

Delete a file

To remove a file from the File Service, issue a DELETE request to the Location returned from the previous POST.

$ curl -i -X DELETE \
    -H "X-Auth-Token: $TOKEN" \
    "http://ace-file-service:8082/file/tmp/sysadmin/90a45ff9-79f3-47b9-87cd-6b8399169957"

HTTP/1.1 200 OK
Server: Apache-Coyote/1.1
Content-Length: 0
Date: Thu, 06 Mar 2014 12:00:50 GMT

A subsequent GET requests for this resource will no longer succeed.

File info

The file info returned when you uploaded the file can be retrieved again by making a GET request to /info/scheme/host/path

$ curl -s \
    -H "Accept: application/json" \
    -H "X-Auth-Token: $TOKEN" \
    "http://ace-file-service:8082/file/info/tmp/sysadmin/90a45ff9-79f3-47b9-87cd-6b8399169957"

{
    "URI" : "tmp://sysadmin/90a45ff9-79f3-47b9-87cd-6b8399169957",
    "length" : "395",
    "accessTime" : "-1",
    "creationTime" : "1394107854000",
    "mimeType" : "text/plain",
    "modifiedTime" : "1394107854000"
}

Authentication

You must be authenticated to use the File Service.

File Delivery Service

The File Delivery Service provides a readonly way to access files from the File Service, with an added content awareness.

By default the file delivery service is deployed at http://ace-file-delivery-service:8088/ inside Docker, and this is used in the examples below.

The file service is designed without an internal cache, so a HTTP cache like Varnish is recommended. See Notes on reverse proxy for image delivery for a configuration example.

Retrieving files

There are two ways to retrieve a file from a content from the file delivery service:

http://ace-file-delivery-service:8088/version/c:MTY4M2ZlNDktMjNkYy00:MTBmMTc5/path/to/file.txt

http://ace-file-delivery-service:8088/alias/namespace/myAlias/path/to/file.txt

Content types

The File Delivery Service uses Apache Tika to determine the content type / mime type of a file being delivered. Both the file data and the file extension of the file being retrieved will be used. The determined mime type will be returned as part of the Content-Type HTTP header of the file response.

Variants

The file delivery service looks at the file aspect of the given content, containing a map from paths within the content to paths in the file service. This enables projects to override that mapping, e.g. for removing the mappings of some files that should not be made available through the File Delivery Service.

The variant config has the alias of aceFileDelivery.variant, and the default mapping behaviour is that the main image file (denoted by the aceImage aspect) is not available, but all other files are. There is of course a possibility to adjust the mappings to other behaviours.