File Structure

Hublink uses file name and file size to determine if a file needs to be uploaded (or re-uploaded) to the cloud. Some common issues to be aware of:

• Overwriting files byte-for-byte: If a file is overwritten byte-for-byte, the file size will not change, and Hublink will not upload the file again.
• Large files: If a file is large, the upload time will be proportional to the file size, which may cause delays in data transfer and impacts battery life.

Files in Amazon S3 are organized according to the upload_path and optional append_path settings. The resulting file structure is:

bucket_name/
├── upload_path/               # From hublink.upload_path
│   ├── append_path/           # From hublink.append_path reference
│   │   └── filename.csv       # Actual files
│   └── another_path/
└── ...

Path Construction

The append_path field supports multiple nested JSON values separated by forward slashes. The path is constructed as follows:

1. Starts with upload_path
2. Appends each value specified in append_path if it exists and is not empty
3. Skips any missing or empty values
4. Sanitizes the path to ensure it’s valid for S3 storage:
   • Allows alphanumeric characters (a-z, A-Z, 0-9)
   • Allows hyphen (-), underscore (_), plus (+), period (.)
   • Removes duplicate slashes
   • Removes trailing slashes

Path Construction Examples

upload_path: "/FED" with append_path: "subject:id"

bucket/
└── FED/
    └── mouse001/
        └── data.csv

upload_path: "/FED" with append_path: "subject:id/experimenter:name"

bucket/
└── FED/
    └── mouse001/
        └── john_doe/
            └── data.csv

Note

If a value referenced in append_path is missing or empty, that segment will be skipped. For example, if experimenter:name is missing in the path subject:id/experimenter:name, the final path would be /FED/mouse001 instead of /FED/mouse001/john_doe.