Transaction Log

Purpose

Record information on events around transfers and users in a customer-readable format, for the purpose of auditing.

Details

The transaction log feature ensures that events occurring on a storagehost are recorded for long-term storage. The recorded data is customer-readable for easy postprocessing.

Recorded data

The recorded events contain metadata about the transaction:

  • unique identifier, UUID Version 1 Format
  • timestamp, in UTC
  • storagehost name (FQDN)
  • event type
  • principal: the email address of the authenticated user who initiated the transaction
  • client-ip and port
  • originator: user-agent, or local component
  • additionally, depending on the type of event:
    • for objects: objectId, filename, mime-type, filesize, chunk-start, chunk-end, hashsum, malware string, reservationId, recipientId (if possible)
    • for reservations & transfers: reservationId, recipientId, joinId, is-in-drive, sender-email, receiver-email, folderId, folder name, drive type, parentFolderId, list of objectIds, expiration timestamp, is-password-protected, subject (in cleartext), message (as SHA256 hash), mailboxId, mailbox title
    • for folders: folderId, folder name, drive type, parentFolderId
    • for folder-access: folderId, folder name, account, access privilege
    • for email notifications: recipient email, SMTP message id
    • for admin-right changes: email address of changed user, id of changed role
    • for mailbox changes: mailbox ID, mailbox title
    • for roles: roleId, role name, assigned user
    • for groups: groupId, group name, group scope

A record will always show data after it has been changed. For example, when moving a folder, the record will show the new parentFolderId. Determining the previous parent folder (the source) requires searching earlier records.

These records are not anonymised.

Event Types

The following type of events are recorded in the transaction log:

Users, accounts

  • account_created
  • account_registered
  • account_removed
  • account_invalidated
  • profile_updated
  • account_login
  • account_password_reset_requested
  • account_password_changed
  • individual_admin_right_granted
  • individual_admin_right_revoked
  • role_admin_right_granted
  • role_admin_right_revoked

Reservations

  • reservation_created
  • reservation_cancelled

Transfers

  • object_upload_started
  • object_upload_ended
  • object_hashsum_calculated
  • object_malware_scanned
  • object_download_started
  • object_download_ended
  • object_preview_fetched
  • transfer_zip_download_started
  • transfer_zip_download_ended
  • transfer_created (this covers all use cases, including forward transfer, copy to drive, upload to drive, etc)
  • transfer_extended
  • transfer_expired
  • transfer_moved (from one drive folder to another)
  • transfer_expiration_scheduled
  • transfer_expiration_cleared

Drive

  • folder_created
  • folder_removed
  • folder_moved
  • folder_access_granted
  • folder_access_removed

Mailboxes

  • mailbox_created
  • mailbox_updated
  • mailbox_removed

Roles

  • role_created
  • role_updated
  • role_removed
  • role_assigned
  • role_deassigned

Groups

  • group_created
  • group_updated
  • group_removed

Notifications

  • transfer_notification_email
  • transfer_bounce_email
  • upload_confirmation_email
  • transfer_delivery_email
  • transfer_expiration_warning_email
  • transfer_malware_warning_email
  • transfer_blocked_warning_email
  • mailbox_quota_warning_email
  • drive_cleanup_warning_email
  • folder_activity_email
  • failed_invitation_email
  • drive_root_folder_email
  • active_users_email

Features, Server configuration

  • adminunit_created
  • adminunit_updated
  • adminunit_activated
  • adminunit_deactivated
  • storagehost_created
  • storagehost_removed
  • storagehost_activated
  • transaction_log_enabled
  • transaction_log_disabled
  • maintenance_access_enabled
  • maintenance_access_disabled
  • malware_protection_enabled
  • malware_protection_disabled
  • block_after_download_enabled
  • block_after_download_disabled
  • force_transfer_password_enabled
  • force_transfer_password_disabled
  • extended_password_criteria_enabled
  • extended_password_criteria_disabled
  • recipient_authentication_enabled
  • recipient_authentication_disabled

Data storage, customer access

If the feature is enabled, the transaction log entry is created by background processes on the server. The data is stored in the TeamBeam drive in a root-folder called Transaction Log. If such a folder does not exist, it will be created automatically. The system user of the adminunit is set as the owner of the folder. Further management of all data in this folder is within the responsibility of the administrators of the adminunit. This includes in particular: deleting old data, setting up notifications and granting/revoking access.

Data structure

The transaction log entry is recorded in CSV format. The transaction log is recorded in daily intervals, and objects are being added to the transfer as they become available. Empty logs are not written. Objects are recorded with mimetype text/csv. The filename of objects identifies the version of the data structure.

  • Team-Drive root folder Transaction Log
    • folder 2016 (year)
      • transfer 2016-06 (month)
        • object 20160617.v1.csv (day)
        • object 20160618.v1.csv
        • object 20160624.v1.csv
        • object 20160624.v2.csv (version changed)

Every transaction log object has a header-row, detailing the columns. Field delimiter is comma. Values containing commas are enclosed in double quotes. Character encoding is UTF-8. Each transaction log entry is added as a single row. Rows are terminated by newline. Rows are ordered by time.

The developer portal provides documentation on the data structure. Changes to the data structure are tracked over time and identified by the version number in the object filename.

Example file:

Syslog output

If configured, transaction log entries are sent in JSON encoding via Syslog protocol to a (remote) receiver. Transaction logging must be activated. Messages are sent via UDP to the configured destination.

Example output:

<142>Apr  3 11:23:07 applicationserver {"targetObjectId":"fe9ktkpq14",
"targetFilename":"2017-04-03.v1.csv","uuid":"25ef1d50-184f-11e7-ba63-08002720b5b7",
"type":"account_login","timestamp":1491211387291,"storagehost":"example.teambeam.net",
"principal":"bob@example.test","ip":"192.0.2.168","port":48767,
"userAgent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.133 Safari/537.36",
"component":null,"objectId":null,"filename":null,"mimeType":null,
"filesize":null,"chunkStart":null,"chunkEnd":null,"hashsum":null,
"algorithm":null,"malwareStatus":null,"malwareSignature":null,"sender":null,
"recipient":null,"reservationId":null,"recipientId":null,"joinId":null,
"destinationType":null,"priority":null,"isInDrive":null,"objectIdList":null,
"expirationTimestamp":null,"isPasswordProtected":null,"subject":null,
"messageHash":null,"folderIdx":null,"folderName":null,"driveType":null,
"parentFolderIdx":null,"mailboxIdx":null,"mailboxTitle":null,
"accountName":null,"accessType":null,"roleIdx":null,"roleName":null,
"smtpMessageId":null}

Configuration

  • Scope: adminunit
  • Privileges: admin
  • Default: disabled

TODO: Describe how it is configured and where; include screenshot or example values

  • adminunit configuration flag: transactionLogEnabled

Appserver

The feature must be enabled on application-server level.

  • Config file: /etc/skp/server.properties
  • Property: feature.transactionLogEnabled
  • Default: false
  • Reloadable: yes

If the feature is disabled, the events may still be processed for statistical purposes, but will not be recorded into any adminunit's transaction log.

Syslog output

The feature must be enabled on application-server level.

  • Config file: /etc/skp/server.logback.xml
  • Property: customer.level, default: OFF, allowed values: OFF,INFO
  • Property: customer.hostname, default: localhost
  • Property: customer.port, default: 514
  • Property: customer.facility, default: LOCAL1, allowed values as per syslog protocol

The logback configuration is activated automatically within 10 seconds, without server restart or reload.

Dependencies

The feature requires that the contract enables the drive and grants sufficient drive space.

Conflicts

none