Internet Archive
At a glance
Section titled “At a glance”| Capability | Support |
|---|---|
| rclone support tier | Tier 3: Supported |
| Checksums | MD5, SHA-1, CRC-32 |
| Modification times | Kept, to 1 ns |
| Case-insensitive names | No |
| Duplicate names allowed | No |
| Server-side copy | Yes |
| Server-side move and rename | No |
| Folder move and rename | No: each file is moved in turn |
| Free space reporting | Yes |
| Empty folders | No: folders exist only while they hold files |
| Change notifications | No |
| Uploads of unknown size | No |
| Public links | Yes |
The Internet Archive backend utilizes Items on archive.org
Refer to IAS3 API documentation for the API this backend uses.
command.) You may put subdirectories in too, e.g. remote:item/path/to/dir.
Unlike S3, listing up all items uploaded by you isn’t supported.
Once you have made a remote, you can use it like this:
Make a new item
rclone mkdir remote:itemList the contents of a item
rclone ls remote:itemSync /home/local/directory to the remote item, deleting any excess
files in the item.
rclone sync --interactive /home/local/directory remote:itemBecause of Internet Archive’s architecture, it enqueues write operations (and extra post-processings) in a per-item queue. You can check item’s queue at https://catalogd.archive.org/history/item-name-here. Because of that, all uploads/deletes will not show up immediately and takes some time to be available. The per-item queue is enqueued to an another queue, Item Deriver Queue. You can check the status of Item Deriver Queue here. This queue has a limit, and it may block you from uploading, or even deleting. You should avoid uploading a lot of small files for better behavior.
You can optionally wait for the server’s processing to finish, by setting
non-zero value to wait_archive key. By making it wait, rclone can do normal
file comparison. Make sure to set a large enough value (e.g. 30m0s for smaller
files) as it can take a long time depending on server’s queue.
About metadata
Section titled “About metadata”This backend supports setting, updating and reading metadata of each file. The metadata will appear as file metadata on Internet Archive. However, some fields are reserved by both Internet Archive and rclone.
The following are reserved by Internet Archive:
namesourcesizemd5crc32sha1formatold_versionviruschecksummation
Trying to set values to these keys is ignored with a warning.
Only setting mtime is an exception. Doing so make it the identical
behavior as setting ModTime.
rclone reserves all the keys starting with rclone-. Setting value for
these keys will give you warnings, but values are set according to request.
If there are multiple values for a key, only the first one is returned. This is a limitation of rclone, that supports one value per one key. It can be triggered when you did a server-side copy.
Reading metadata will also provide custom (non-standard nor reserved) ones.
Filtering auto generated files
Section titled “Filtering auto generated files”The Internet Archive automatically creates metadata files after
upload. These can cause problems when doing an rclone sync as rclone
will try, and fail, to delete them. These metadata files are not
changeable, as they are created by the Internet Archive automatically.
These auto-created files can be excluded from the sync using metadata filtering.
rclone sync ... --metadata-exclude "source=metadata" --metadata-exclude "format=Metadata"Which excludes from the sync any files which have the
source=metadata or format=Metadata flags which are added to
Internet Archive auto-created files.
Options
Section titled “Options”Standard options
Section titled “Standard options”The wizard asks for these when you add the drive.
access_key_id
Section titled “access_key_id”IAS3 Access Key.
Leave blank for anonymous access. You can find one here: https://archive.org/account/s3.php
Type: string
secret_access_key
Section titled “secret_access_key”IAS3 Secret Key (password).
Leave blank for anonymous access.
Type: string
item_derive
Section titled “item_derive”Whether to trigger derive on the IA item or not. If set to false, the item will not be derived by IA upon upload. The derive process produces a number of secondary files from an upload to make an upload more usable on the web. Setting this to false is useful for uploading files that are already in a format that IA can display or reduce burden on IA’s infrastructure.
Type: bool · Default: true
Advanced options
Section titled “Advanced options”The wizard shows these under Advanced options. Most drives work without changing any of them.
endpoint
Section titled “endpoint”IAS3 Endpoint.
Leave blank for default value.
Type: string · Default: https://s3.us.archive.org
front_endpoint
Section titled “front_endpoint”Host of InternetArchive Frontend.
Leave blank for default value.
Type: string · Default: https://archive.org
item_metadata
Section titled “item_metadata”Metadata to be set on the IA item, this is different from file-level metadata that can be set using –metadata-set. Format is key=value and the ‘x-archive-meta-’ prefix is automatically added.
Type: stringArray · Default: []
disable_checksum
Section titled “disable_checksum”Don’t ask the server to test against MD5 checksum calculated by rclone. Normally rclone will calculate the MD5 checksum of the input before uploading it so it can ask the server to check the object against checksum. This is great for data integrity checking but can cause long delays for large files to start uploading.
Type: bool · Default: true
wait_archive
Section titled “wait_archive”Timeout for waiting the server’s processing tasks (specifically archive and book_op) to finish. Only enable if you need to be guaranteed to be reflected after write operations. 0 to disable waiting. No errors to be thrown in case of timeout.
Type: Duration · Default: 0s
encoding
Section titled “encoding”The encoding for the backend.
See the encoding section in the overview for more info.
Type: Encoding · Default: Slash,LtGt,CrLf,Del,Ctl,InvalidUtf8,Dot
description
Section titled “description”Description of the remote.
Type: string