Transfer Post Processing

After a new transfer has been uploaded and before the recipient is notified, a number of potentially long-running tasks are executed on the transfer and its files. This article explains the chosen architecture.

From Reservation To Notification

Creating a new transfer consists of three stages:

1. Creating a reservation, which acts as a draft transfer. It contains all data of the future transfer, except for the payload (which often takes long to upload).
2. Uploading a number of files into that reservation. This stage can take a long time, and uploading of files can (and should) be split into segments.
3. Confirming the reservation. This essentially close it and creates the transfers for the recipients.

When the sender confirms a reservation, several tasks are performed synchronously within the confirm-request. Most importantly, the server checks if all files have been uploaded as promised, and if the sender's contract has any problems with the data. At this point, the reservation-confirmation-request may get rejected, allowing the sender to re-upload files, etc and submit the confirmation again. If the checks were successful, the sender may receive an upload-success-eMail. Afterwards, longer-running post-processing is triggered, and the sender's request is completed.

Post processing of a transfer includes scanning files for malware and calculating hashsums. The time required for this can take an indeterminable time. As a result, the process is implemented very much asynchronously. Only once all necessary tasks are completed, is the new-transfer eMail being sent to the recipient.

Post Processing Flow

Since post processing can take an indeterminable time, dependent on the number and size of files as well as the availability of storage backends and their individual computing capacity/load, the implementation follows ideas from functional programming:

• Where possible, break the work up into tasks without side effects and data-dependency.

When scanning a file for malware and calculating its hashsum, the operations do not affect each other.

• Distribute these tasks across all available resources and allow them to be executed in parallel with the available concurrency of the cluster.
• Map-and-reduce the results of functions into the governing entity.

A transfer is considered infected, if at least one of its files is infected.

Rather than executing tasks sequentially and in a controlled fashion (by a governing thread which waits for the execution of each task it triggers), the tasks are orchestrated by a state engine. Events trigger a transfer's progression through the state engine. Depending on the current state of a transfer and its objects, a stage may be entered or exited, and subsequent tasks may be triggered. The results of these tasks will again trigger the execution of the state engine. Eventually a transfer exists the state engine with processing status completed.

State Engine

Execution of the state engine is safe at any time. For the duration of an execution for a transfer, the state engine obtains a distributed cluster-wide lock for that transfer, to avoid side effects. This provides cluster-wide thread safety. The lock is released at the end.

Each transfer runs through the following states:

1. newTransfer
2. malwareScan
3. hashCalculation
4. notification
5. completed

Each stage has entry- and exit-requirements that determine if it is necessary, already completed or already entered.

For example, the malware-scanning stage is necessary if either the sender or the recipient has the feature contractually available and enabled. The stage is entered by requesting each file to scanned exactly once. The stage is exited only after all scan operations have returned (with specific results or an exception). The stage is exited exactly once, at which point the sender of the transfer may be notified of detected malware.