Object storage

Object storage on Upsun adds an S3-compatible bucket to your application. You enable it by setting an object-storage size on the app’s resources. Upsun provisions the bucket and connects it to your app. Object storage is integrated with the rest of the platform. When you back up an environment, the bucket’s contents are included in the backup, and when you branch an environment, the bucket is cloned along with the environment’s other data. You manage it together with the rest of your project rather than as a separate service.

Object storage is currently a prerelease feature. This feature requires a minimum usage of 0.5TB and incurs additional costs that will be billed separately. As this is a prerelease feature, the pricing is being finalized. Please reach out to your account manager to get more information about pricing as well as to request the functionality activation. Please provide a project ID of the project you want it to be enabled for along with the request.To get your project ID, run upsun projects and copy the corresponding ID.

When to use object storage

Use object storage when your app needs to store large or growing volumes of unstructured data (uploads, media, backups, generated artifacts) and you want to reach it through the S3 API. For small amounts of file data that a single instance reads and writes through the filesystem, a persistent mount is usually a better fit.

Enable object storage

Set an object-storage size on an app. The size is in MB, in the app:size format. Run upsun resources:set with the --object-storage option:

Terminal

upsun resources:set --object-storage <APP_NAME>:<SIZE_IN_MB>

For example, to give the myapp app 512 GB (524288 MB) of object storage:

Terminal

upsun resources:set --object-storage myapp:524288

To view the current allocation, run upsun resources:get. When an app has object storage, the output includes an Object storage (MB) column.

Sizing rules

Object storage is allocated in fixed steps, within a minimum and a maximum. The size you request must be a multiple of the step and fall within these bounds:

Rule	Value
Step	524288 MB (512 GB)
Minimum	524288 MB (512 GB)
Maximum	10485760 MB (10 TB)

Relationship reference

Enabling object storage adds a relationship named object-storage to your app. This relationship name is reserved. If your configuration already declares a relationship called object-storage, the deployment fails and you must rename the relationship to proceed. The relationship provides the connection details for an S3-compatible endpoint. Upsun automatically generates service environment variables for it within your application container, in the $<RELATIONSHIP-NAME>_<SERVICE-PROPERTY> format. The prefix follows the relationship name, so a container that reaches the bucket through a relationship named bucket, as in Share a bucket across containers, gets BUCKET_ACCESS_KEY rather than OBJECT_STORAGE_ACCESS_KEY. Here is example information available through the service environment variables themselves, or through the PLATFORM_RELATIONSHIPS environment variable.

Service environment variables
PLATFORM_RELATIONSHIPS environment variable

You can obtain the complete list of available service environment variables in your app container by running upsun ssh env.Note that the information about the relationship can change when an app is redeployed or restarted. So your apps should only rely on the service environment variables directly rather than hard coding any values.

For some advanced use cases, you can use the PLATFORM_RELATIONSHIPS environment variable. The structure of the PLATFORM_RELATIONSHIPS environment variable can be obtained by running upsun relationships in your terminal:Here is an example of how to gather that information in a .environment file:

.environment

# Decode the built-in credentials object variable.
export RELATIONSHIPS_JSON="$(echo "$PLATFORM_RELATIONSHIPS" | base64 --decode)"

# Set environment variables for individual credentials.
export AWS_ACCESS_KEY_ID="$(echo "$RELATIONSHIPS_JSON" | jq -r '.["object-storage"][0].access_key')"
export AWS_SECRET_ACCESS_KEY="$(echo "$RELATIONSHIPS_JSON" | jq -r '.["object-storage"][0].secret_key')"
export AWS_DEFAULT_REGION="$(echo "$RELATIONSHIPS_JSON" | jq -r '.["object-storage"][0].region')"

# Some SDKs read AWS_REGION instead of AWS_DEFAULT_REGION.
export AWS_REGION="$AWS_DEFAULT_REGION"

# Read the bucket name rather than assuming it.
export S3_BUCKET="$(echo "$RELATIONSHIPS_JSON" | jq -r '.["object-storage"][0].buckets | keys[0]')"

The object-storage key has to match the relationship name. A container that reaches the bucket through a relationship named bucket uses .["bucket"][0] instead. A key that matches no relationship exports the literal string null rather than failing, so requests are signed with invalid credentials.

Connect from your app

The endpoint URL is http://<HOST>:<PORT>, built from the host and port values in the relationship. The endpoint is reachable over your project’s internal network only and isn’t exposed publicly. A single bucket is created per app, named after the app, and the relationship lists it in buckets, an object keyed by bucket name. So an app named myapp reaches its bucket at http://<HOST>:<PORT>/myapp/<KEY>. Take the name from the buckets keys rather than assuming it, so the same code keeps working in a container that reaches another app’s bucket through a shared relationship. When configuring your S3 client, note two requirements:

Use path-style addressing. The bucket name goes in the URL path (http://<HOST>:<PORT>/myapp/object.txt), not the hostname. Disable virtual-hosted-style addressing.
Use the credentials from the relationship. The endpoint authenticates every request, so configure your client with the access_key, secret_key, and region values from the relationship. Empty or arbitrary credentials are rejected.

Example (Python, boto3)

import base64
import json
import os

import boto3

relationships = json.loads(base64.b64decode(os.environ["PLATFORM_RELATIONSHIPS"]))
endpoint = relationships["object-storage"][0]

s3 = boto3.client(
    "s3",
    endpoint_url=f"http://{endpoint['host']}:{endpoint['port']}",
    aws_access_key_id=endpoint["access_key"],
    aws_secret_access_key=endpoint["secret_key"],
    region_name=endpoint["region"],
    config=boto3.session.Config(s3={"addressing_style": "path"}),
)

bucket = list(endpoint["buckets"].keys())[0]

s3.put_object(Bucket=bucket, Key="hello.txt", Body=b"Hello, World!")

Other containers can use an app’s bucket through a relationship:

A worker that doesn’t declare its own relationships block inherits the parent app’s relationships, including object-storage, with no extra configuration.
A worker with its own relationships block, another application, or a task can reach an application’s bucket by targeting that app’s object-storage endpoint:

.upsun/config.yaml

applications:
  myapp:
    # ...
    workers:
      queue:
        relationships:
          bucket: 'myapp:object-storage'

The relationship name on the left (bucket here) is yours to choose. Targeting an app that doesn’t have object storage configured fails the deployment.

Resize and remove

To resize, pass a new value that is a valid multiple of the step:

Terminal

upsun resources:set --object-storage myapp:1048576

Set the size to 0 to remove object storage from the app:

Terminal

upsun resources:set --object-storage myapp:0

Setting the size to 0, or removing the app, deletes the bucket and all of its contents on the next deployment. There’s no soft-delete or retention period. Back up any data you need before removing.

When you branch an environment and clone its data, the object-storage size is inherited along with the data, so the new environment keeps its bucket.

Limitations

Each app gets a single bucket, named after the app. Multiple buckets per app aren’t supported.
A bucket belongs to one app. Other containers can share it through a relationship, but buckets can’t be reassigned between apps.
The endpoint is internal to the project and isn’t exposed publicly.
The following S3 features aren’t implemented: bucket versioning, object locking, server-side encryption (SSE), access control lists (ACLs), bucket policies, and lifecycle policies. Requests for them return a “not implemented” error.
Concurrent writes to the same object are last-write-wins with no object locking.

Inspect the bucket from an SSH session

You can explore your bucket directly from a running container using the AWS CLI.

Run S3 commands

The endpoint authenticates every request, so pass the credentials from the relationship. Inside the container they’re available as service environment variables, along with the host and port. Pointed at a custom endpoint with --endpoint-url, the AWS CLI uses path-style addressing on its own, so no extra configuration is needed.

Terminal

AWS_ACCESS_KEY_ID="$OBJECT_STORAGE_ACCESS_KEY" \
AWS_SECRET_ACCESS_KEY="$OBJECT_STORAGE_SECRET_KEY" \
AWS_DEFAULT_REGION="$OBJECT_STORAGE_REGION" \
aws s3 --endpoint-url "http://$OBJECT_STORAGE_HOST:$OBJECT_STORAGE_PORT" ls s3://<APP_NAME>/

Other standard S3 commands work the same way, with the same credentials and --endpoint-url flag:

Goal	Command
List objects	`ls s3://<APP_NAME>/`
Download a file	`cp s3://<APP_NAME>/path/to/file ./local-file`
Upload a file	`cp ./local-file s3://<APP_NAME>/path/to/file`
Sync a directory	`sync ./local-dir s3://<APP_NAME>/prefix/`
Delete an object	`rm s3://<APP_NAME>/path/to/file`

Get Started

Core Concepts

Configure Apps & Services

Build, Deploy & Run

Manage Projects

Manage Environments

Manage Resources

Observability & Metrics

Integrations

Troubleshooting

Resources

When to use object storage

Enable object storage

Sizing rules

Relationship reference

Connect from your app

Example (Python, boto3)

Resize and remove

Limitations

Inspect the bucket from an SSH session

Run S3 commands

​When to use object storage

​Enable object storage

​Sizing rules

​Relationship reference

​Connect from your app

​Example (Python, boto3)

​Share a bucket across containers

​Resize and remove

​Limitations

​Inspect the bucket from an SSH session

​Run S3 commands

When to use object storage

Enable object storage

Sizing rules

Relationship reference

Connect from your app

Example (Python, boto3)

Share a bucket across containers

Resize and remove

Limitations

Inspect the bucket from an SSH session

Run S3 commands