Microsoft Azure Blob Storage

At a glance

Capability	Support
rclone support tier	Tier 1: Core
Checksums	MD5
Modification times	Kept, to 1 ns
Case-insensitive names	No
Duplicate names allowed	No
Server-side copy	Yes
Server-side move and rename	No
Folder move and rename	No: each file is moved in turn
Free space reporting	No
Empty folders	No: folders exist only while they hold files
Change notifications	No
Uploads of unknown size	Yes
Public links	No

From rclone’s documentation

The rest of this page is adapted from rclone’s documentation for this backend (rclone v1.75.2, MIT licence). It is written for rclone’s command line: where it mentions a flag such as --azureblob-example, set the option example in the wizard’s options instead. Commands such as rclone copy aren’t part of CloudMount.

command.) You may put subdirectories in too, e.g. remote:container/path/to/dir.

Setup notes

In CloudMount you add this drive from the Add drive screen, and the wizard asks for the options listed under Options. The notes below come from rclone’s setup guide.

--fast-list

This remote supports --fast-list which allows you to use fewer transactions in exchange for more memory. See the rclone docs for more details.

Modification times and hashes

The modification time is stored as metadata on the object with the mtime key. It is stored using RFC3339 Format time with nanosecond precision. The metadata is supplied during directory listings so there is no performance overhead to using it.

If you wish to use the Azure standard LastModified time stored on the object as the modified time, then use the --use-server-modtime flag. Note that rclone can’t set LastModified, so using the --update flag when syncing is recommended if using --use-server-modtime.

MD5 hashes are stored with blobs. However blobs that were uploaded in chunks only have an MD5 if the source remote was capable of MD5 hashes, e.g. the local disk.

Metadata and tags

Rclone can map arbitrary metadata to Azure Blob headers, user metadata, and tags when --metadata is enabled (or when using --metadata-set / --metadata-mapper).

• Headers: Set these keys in metadata to map to the corresponding blob headers:
  • cache-control, content-disposition, content-encoding, content-language, content-type.
• User metadata: Any other non-reserved keys are written as user metadata (keys are normalized to lowercase). Keys starting with x-ms- are reserved and are not stored as user metadata.
• Tags: Provide x-ms-tags as a comma-separated list of key=value pairs, e.g. x-ms-tags=env=dev,team=sync. These are applied as blob tags on upload and on server-side copies. Whitespace around keys/values is ignored.
• Modtime override: Provide mtime in RFC3339/RFC3339Nano format to override the stored modtime persisted in user metadata. If mtime cannot be parsed, rclone logs a debug message and ignores the override.

Notes:

• Rclone ignores reserved x-ms-* keys (except x-ms-tags) for user metadata.

Performance

When uploading large files, increasing the value of --azureblob-upload-concurrency will increase performance at the cost of using more memory. The default of 16 is set quite conservatively to use less memory. It maybe be necessary raise it to 64 or higher to fully utilize a 1 GBit/s link with a single file transfer.

Restricted filename characters

In addition to the default restricted characters set the following characters are also replaced:

Character	Value	Replacement
/	0x2F	／
\	0x5C	＼

File names can also not end with the following characters. These only get replaced if they are the last character in the name:

Character	Value	Replacement
.	0x2E	．

Invalid UTF-8 bytes will also be replaced, as they can’t be used in JSON strings.

Authentication

There are a number of ways of supplying credentials for Azure Blob Storage. Rclone tries them in the order of the sections below.

Env Auth

If the env_auth config parameter is true then rclone will pull credentials from the environment or runtime.

It tries these authentication methods in this order:

1. Environment Variables
2. Managed Service Identity Credentials
3. Azure CLI credentials (as used by the az tool)

These are described in the following sections

Env Auth: 1. Environment Variables

If env_auth is set and environment variables are present rclone authenticates a service principal with a secret or certificate, or a user with a password, depending on which environment variable are set. It reads configuration from these variables, in the following order:

1. Service principal with client secret
   • AZURE_TENANT_ID: ID of the service principal’s tenant. Also called its “directory” ID.
   • AZURE_CLIENT_ID: the service principal’s client ID
   • AZURE_CLIENT_SECRET: one of the service principal’s client secrets
2. Service principal with certificate
   • AZURE_TENANT_ID: ID of the service principal’s tenant. Also called its “directory” ID.
   • AZURE_CLIENT_ID: the service principal’s client ID
   • AZURE_CLIENT_CERTIFICATE_PATH: path to a PEM or PKCS12 certificate file including the private key.
   • AZURE_CLIENT_CERTIFICATE_PASSWORD: (optional) password for the certificate file.
   • AZURE_CLIENT_SEND_CERTIFICATE_CHAIN: (optional) Specifies whether an authentication request will include an x5c header to support subject name / issuer based authentication. When set to “true” or “1”, authentication requests include the x5c header.
3. User with username and password
   • AZURE_TENANT_ID: (optional) tenant to authenticate in. Defaults to “organizations”.
   • AZURE_CLIENT_ID: client ID of the application the user will authenticate to
   • AZURE_USERNAME: a username (usually an email address)
   • AZURE_PASSWORD: the user’s password
4. Workload Identity
   • AZURE_TENANT_ID: Tenant to authenticate in
   • AZURE_CLIENT_ID: Client ID of the application the user will authenticate to
   • AZURE_FEDERATED_TOKEN_FILE: Path to projected service account token file
   • AZURE_AUTHORITY_HOST: Authority of an Azure Active Directory endpoint (default: login.microsoftonline.com).

Env Auth: 2. Managed Service Identity Credentials

When using Managed Service Identity if the VM(SS) on which this program is running has a system-assigned identity, it will be used by default. If the resource has no system-assigned but exactly one user-assigned identity, the user-assigned identity will be used by default.

If the resource has multiple user-assigned identities you will need to unset env_auth and set use_msi instead. See the use_msi section.

If you are operating in disconnected clouds, or private clouds such as Azure Stack you may want to set disable_instance_discovery = true. This determines whether rclone requests Microsoft Entra instance metadata from https://login.microsoft.com/ before authenticating. Setting this to true will skip this request, making you responsible for ensuring the configured authority is valid and trustworthy.

Env Auth: 3. Azure CLI credentials (as used by the az tool)

Credentials created with the az tool can be picked up using env_auth.

For example if you were to login with a service principal like this:

az login --service-principal -u XXX -p XXX --tenant XXX

Then you could access rclone resources like this:

rclone lsf :azureblob,env_auth,account=ACCOUNT:CONTAINER

Or

rclone lsf --azureblob-env-auth --azureblob-account=ACCOUNT :azureblob:CONTAINER

Which is analogous to using the az tool:

az storage blob list --container-name CONTAINER --account-name ACCOUNT --auth-mode login

Account and Shared Key

This is the most straight forward and least flexible way. Just fill in the account and key lines and leave the rest blank.

SAS URL

This can be an account level SAS URL or container level SAS URL.

To use it leave account and key blank and fill in sas_url.

An account level SAS URL or container level SAS URL can be obtained from the Azure portal or the Azure Storage Explorer. To get a container level SAS URL right click on a container in the Azure Blob explorer in the Azure portal.

If you use a container level SAS URL, rclone operations are permitted only on a particular container, e.g.

rclone ls azureblob:container

You can also list the single container from the root. This will only show the container specified by the SAS URL.

$ rclone lsd azureblob:
container/

Note that you can’t see or access any other containers - this will fail

rclone ls azureblob:othercontainer

Container level SAS URLs are useful for temporarily allowing third parties access to a single container or putting credentials into an untrusted environment such as a CI build server.

Service principal with client secret

If these variables are set, rclone will authenticate with a service principal with a client secret.

• tenant: ID of the service principal’s tenant. Also called its “directory” ID.
• client_id: the service principal’s client ID
• client_secret: one of the service principal’s client secrets

The credentials can also be placed in a file using the service_principal_file configuration option.

Service principal with certificate

If these variables are set, rclone will authenticate with a service principal with certificate.

• tenant: ID of the service principal’s tenant. Also called its “directory” ID.
• client_id: the service principal’s client ID
• client_certificate_path: path to a PEM or PKCS12 certificate file including the private key.
• client_certificate_password: (optional) password for the certificate file.
• client_send_certificate_chain: (optional) Specifies whether an authentication request will include an x5c header to support subject name / issuer based authentication. When set to “true” or “1”, authentication requests include the x5c header.

NB client_certificate_password must be obscured - see rclone obscure.

User with username and password

If these variables are set, rclone will authenticate with username and password.

• tenant: (optional) tenant to authenticate in. Defaults to “organizations”.
• client_id: client ID of the application the user will authenticate to
• username: a username (usually an email address)
• password: the user’s password

Microsoft doesn’t recommend this kind of authentication, because it’s less secure than other authentication flows. This method is not interactive, so it isn’t compatible with any form of multi-factor authentication, and the application must already have user or admin consent. This credential can only authenticate work and school accounts; it can’t authenticate Microsoft accounts.

NB password must be obscured - see rclone obscure.

Managed Service Identity Credentials

If use_msi is set then managed service identity credentials are used. This authentication only works when running in an Azure service. env_auth needs to be unset to use this.

However if you have multiple user identities to choose from these must be explicitly specified using exactly one of the msi_object_id, msi_client_id, or msi_mi_res_id parameters.

If none of msi_object_id, msi_client_id, or msi_mi_res_id is set, this is equivalent to using env_auth.

Federated Identity Credentials

If these variables are set, rclone will authenticate with federated identity.

• tenant: tenant ID of the storage
• client_id: client ID of the application the user will authenticate to storage
• msi_client_id: managed identity client ID of the application the user will authenticate to

By default “api://AzureADTokenExchange” is used as scope for token retrieval over MSI. This token is then exchanged for actual storage token using ‘tenant’ and ‘client_id’.

Azure CLI tool az

If use_az is set, use Azure CLI tool az as the sole means of authentication.

Setting this can be useful if you wish to use the az CLI on a host with a System Managed Identity that you do not want to use.

Don’t set env_auth at the same time.

Anonymous

If you want to access resources with public anonymous access then set account only. You can do this without making an rclone config:

rclone lsf :azureblob,account=ACCOUNT:CONTAINER

Options

Standard options

The wizard asks for these when you add the drive.

account

Azure Storage Account Name.

Set this to the Azure Storage Account Name in use.

Leave blank to use SAS URL or Emulator, otherwise it needs to be set.

If this is blank and if env_auth is set it will be read from the environment variable AZURE_STORAGE_ACCOUNT_NAME if possible.

Type: string

env_auth

Read credentials from runtime (environment variables, CLI or MSI).

See the authentication docs for full info.

Type: bool · Default: false

key

Storage Account Shared Key.

Leave blank to use SAS URL or Emulator.

Type: string

sas_url

SAS URL for container level access only.

Leave blank if using account/key or Emulator.

Type: string

connection_string

Storage Connection String.

Connection string for the storage. Leave blank if using other auth methods.

Type: string

tenant

ID of the service principal’s tenant. Also called its directory ID.

Set this if using

• Service principal with client secret
• Service principal with certificate
• User with username and password

Type: string

client_id

The ID of the client in use.

Set this if using

• Service principal with client secret
• Service principal with certificate
• User with username and password

Type: string

client_secret

One of the service principal’s client secrets

Set this if using

• Service principal with client secret

Type: string

client_certificate_path

Path to a PEM or PKCS12 certificate file including the private key.

Set this if using

• Service principal with certificate

Type: string

client_certificate_password

Password for the certificate file (optional).

Optionally set this if using

• Service principal with certificate

And the certificate has a password.

NB Input to this must be obscured - see rclone obscure.

Type: string

Advanced options

The wizard shows these under Advanced options. Most drives work without changing any of them.

client_send_certificate_chain

Send the certificate chain when using certificate auth.

Specifies whether an authentication request will include an x5c header to support subject name / issuer based authentication. When set to true, authentication requests include the x5c header.

Optionally set this if using

• Service principal with certificate

Type: bool · Default: false

username

User name (usually an email address)

Set this if using

• User with username and password

Type: string

password

The user’s password

Set this if using

• User with username and password

NB Input to this must be obscured - see rclone obscure.

Type: string

service_principal_file

Path to file containing credentials for use with a service principal.

Leave blank normally. Needed only if you want to use a service principal instead of interactive login.

$ az ad sp create-for-rbac --name "<name>" \
  --role "Storage Blob Data Owner" \
  --scopes "/subscriptions/<subscription>/resourceGroups/<resource-group>/providers/Microsoft.Storage/storageAccounts/<storage-account>/blobServices/default/containers/<container>" \
  > azure-principal.json

See “Create an Azure service principal” and “Assign an Azure role for access to blob data” pages for more details.

It may be more convenient to put the credentials directly into the rclone config file under the client_id, tenant and client_secret keys instead of setting service_principal_file.

Type: string

disable_instance_discovery

Skip requesting Microsoft Entra instance metadata

This should be set true only by applications authenticating in disconnected clouds, or private clouds such as Azure Stack.

It determines whether rclone requests Microsoft Entra instance metadata from https://login.microsoft.com/ before authenticating.

Setting this to true will skip this request, making you responsible for ensuring the configured authority is valid and trustworthy.

Type: bool · Default: false

use_msi

Use a managed service identity to authenticate (only works in Azure).

When true, use a managed service identity to authenticate to Azure Storage instead of a SAS token or account key.

If the VM(SS) on which this program is running has a system-assigned identity, it will be used by default. If the resource has no system-assigned but exactly one user-assigned identity, the user-assigned identity will be used by default. If the resource has multiple user-assigned identities, the identity to use must be explicitly specified using exactly one of the msi_object_id, msi_client_id, or msi_mi_res_id parameters.

Type: bool · Default: false

msi_object_id

Object ID of the user-assigned MSI to use, if any.

Leave blank if msi_client_id or msi_mi_res_id specified.

Type: string

msi_client_id

Object ID of the user-assigned MSI to use, if any.

Leave blank if msi_object_id or msi_mi_res_id specified.

Type: string

msi_mi_res_id

Azure resource ID of the user-assigned MSI to use, if any.

Leave blank if msi_client_id or msi_object_id specified.

Type: string

use_emulator

Uses local storage emulator if provided as ‘true’.

Leave blank if using real azure storage endpoint.

Type: bool · Default: false

use_az

Use Azure CLI tool az for authentication

Set to use the Azure CLI tool az as the sole means of authentication.

Setting this can be useful if you wish to use the az CLI on a host with a System Managed Identity that you do not want to use.

Don’t set env_auth at the same time.

Type: bool · Default: false

endpoint

Endpoint for the service.

Leave blank normally.

Type: string

upload_cutoff

Cutoff for switching to chunked upload (<= 256 MiB) (deprecated).

Type: string

chunk_size

Upload chunk size.

Note that this is stored in memory and there may be up to “–transfers” * “–azureblob-upload-concurrency” chunks stored at once in memory.

Type: SizeSuffix · Default: 4Mi

upload_concurrency

Concurrency for multipart uploads.

This is the number of chunks of the same file that are uploaded concurrently.

If you are uploading small numbers of large files over high-speed links and these uploads do not fully utilize your bandwidth, then increasing this may help to speed up the transfers.

In tests, upload speed increases almost linearly with upload concurrency. For example to fill a gigabit pipe it may be necessary to raise this to 64. Note that this will use more memory.

Note that chunks are stored in memory and there may be up to “–transfers” * “–azureblob-upload-concurrency” chunks stored at once in memory.

Type: int · Default: 16

copy_cutoff

Cutoff for switching to multipart copy.

Any files larger than this that need to be server-side copied will be copied in chunks of chunk_size using the put block list API.

Files smaller than this limit will be copied with the Copy Blob API.

Type: SizeSuffix · Default: 8Mi

copy_concurrency

Concurrency for multipart copy.

This is the number of chunks of the same file that are copied concurrently.

These chunks are not buffered in memory and Microsoft recommends setting this value to greater than 1000 in the azcopy documentation.

https://learn.microsoft.com/en-us/azure/storage/common/storage-use-azcopy-optimize#increase-concurrency

In tests, copy speed increases almost linearly with copy concurrency.

Type: int · Default: 512

copy_total_concurrency

Global concurrency limit for multipart copy chunks.

This limits the total number of multipart copy chunks running at once across all files.

Set to 0 to disable this limiter.

Type: int · Default: 0

use_copy_blob

Whether to use the Copy Blob API when copying to the same storage account.

If true (the default) then rclone will use the Copy Blob API for copies to the same storage account even when the size is above the copy_cutoff.

Rclone assumes that the same storage account means the same config and does not check for the same storage account in different configs.

There should be no need to change this value.

Type: bool · Default: true

list_chunk

Size of blob list.

This sets the number of blobs requested in each listing chunk. Default is the maximum, 5000. “List blobs” requests are permitted 2 minutes per megabyte to complete. If an operation is taking longer than 2 minutes per megabyte on average, it will time out ( source ). This can be used to limit the number of blobs items to return, to avoid the time out.

Type: int · Default: 5000

use_arrow_list

Use the Apache Arrow listing format.

If set, directory listings are fetched using the ListBlobs Apache Arrow response format instead of XML. Arrow responses are smaller and much cheaper to parse, making listings of large containers several times faster. Combine with “list_parallelism” for the biggest gains.

“Blob Listing with Apache Arrow” is in public preview at Microsoft and is only supported on flat namespace accounts. On accounts with a hierarchical namespace (ADLS Gen2), or where the feature is otherwise unavailable, the server returns XML and the listing transparently falls back to the normal XML path (logged at debug level).

Type: bool · Default: false

list_parallelism

Number of parallel shards to list a directory with.

If set greater than 1, the blob name keyspace of each directory is split into ranges which are listed concurrently using the Arrow startFrom/endBefore range parameters. This can dramatically speed up listing containers with millions of objects, for both recursive (ListR) and single directory listings. Speed keeps improving up to a parallelism of around 30.

This has no effect unless “use_arrow_list” is also set, as Arrow is the only listing path that supports server-side name ranges. If the account does not support range listing (e.g. it has a hierarchical namespace) the listing falls back to sequential. The default of 0 (or

1. lists sequentially.

Type: int · Default: 0

access_tier

Access tier of blob: hot, cool, cold or archive.

Archived blobs can be restored by setting access tier to hot, cool or cold. Leave blank if you intend to use default access tier, which is set at account level

If there is no “access tier” specified, rclone doesn’t apply any tier. rclone performs “Set Tier” operation on blobs while uploading, if objects are not modified, specifying “access tier” to new one will have no effect. If blobs are in “archive tier” at remote, trying to perform data transfer operations from remote will not be allowed. User should first restore by tiering blob to “Hot”, “Cool” or “Cold”.

Type: string

archive_tier_delete

Delete archive tier blobs before overwriting.

Archive tier blobs cannot be updated. So without this flag, if you attempt to update an archive tier blob, then rclone will produce the error:

can't update archive tier blob without --azureblob-archive-tier-delete

With this flag set then before rclone attempts to overwrite an archive tier blob, it will delete the existing blob before uploading its replacement. This has the potential for data loss if the upload fails (unlike updating a normal blob) and also may cost more since deleting archive tier blobs early may be chargable.

Type: bool · Default: false

disable_checksum

Don’t store MD5 checksum with object metadata.

Normally rclone will calculate the MD5 checksum of the input before uploading it so it can add it to metadata on the object. This is great for data integrity checking but can cause long delays for large files to start uploading.

Type: bool · Default: false

encoding

The encoding for the backend.

See the encoding section in the overview for more info.

Type: Encoding · Default: Slash,BackSlash,Del,Ctl,RightPeriod,InvalidUtf8

public_access

Public access level of a container: blob or container.

Type: string

Choices:

• (empty): The container and its blobs can be accessed only with an authorized request. It’s a default value.
• blob: Blob data within this container can be read via anonymous request.
• container: Allow full public read access for container and blob data.

directory_markers

Upload an empty object with a trailing slash when a new directory is created

Empty folders are unsupported for bucket based remotes, this option creates an empty object ending with “/”, to persist the folder.

This object also has the metadata “hdi_isfolder = true” to conform to the Microsoft standard.

Type: bool · Default: false

no_check_container

If set, don’t attempt to check the container exists or create it.

This can be useful when trying to minimise the number of transactions rclone does if you know the container exists already.

Type: bool · Default: false

no_head_object

If set, do not do HEAD before GET when getting objects.

Type: bool · Default: false

delete_snapshots

Set to specify how to deal with snapshots on blob deletion.

Type: string

Choices:

• (empty): By default, the delete operation fails if a blob has snapshots
• include: Specify ‘include’ to remove the root blob and all its snapshots
• only: Specify ‘only’ to remove only the snapshots but keep the root blob.

decompress

If set this will decompress gzip encoded objects.

It is possible to upload objects to Azure Blob Storage with “Content-Encoding: gzip” set. Normally rclone will download these files as compressed objects.

If this flag is set then rclone will decompress these files with “Content-Encoding: gzip” as they are received. This means that rclone can’t check the size and hash but the file contents will be decompressed.

Type: bool · Default: false

description

Description of the remote.

Type: string

Custom upload headers

You can set custom upload headers with the --header-upload flag.

• Cache-Control
• Content-Disposition
• Content-Encoding
• Content-Language
• Content-Type
• X-MS-Tags

Eg --header-upload "Content-Type: text/potato" or --header-upload "X-MS-Tags: foo=bar".

Limitations

MD5 sums are only uploaded with chunked files if the source has an MD5 sum. This will always be the case for a local to azure copy.

rclone about is not supported by the Microsoft Azure Blob storage backend. Backends without this capability cannot determine free space for an rclone mount or use policy mfs (most free space) as a member of an rclone union remote.

See List of backends that do not support rclone about and rclone about.

Azure Storage Emulator Support

You can run rclone with the storage emulator (usually azurite).

To do this, just set up a new remote with rclone config following the instructions in the introduction and set use_emulator in the advanced settings as true. You do not need to provide a default account name nor an account key. But you can override them in the account and key options. (Prior to v1.61 they were hard coded to azurite’s devstoreaccount1.)

Also, if you want to access a storage emulator instance running on a different machine, you can override the endpoint parameter in the advanced settings, setting it to http(s)://<host>:<port>/devstoreaccount1 (e.g. http://10.254.2.5:10000/devstoreaccount1).