Elasticsearch and Enterprise Search (search service)

Upsun offers two Elasticsearch-based search services:

Elastic Enterprise Search: current Premium service, recommended for all new projects.
Elasticsearch (end of life): versions up to 7.10, no longer maintained.

Elastic Enterprise Search (current)

Elastic Enterprise Search is available as a Premium service. It replaces the legacy Elasticsearch service, which is no longer receiving security updates. For full feature documentation, see the Elastic Enterprise Search docs. To add this service, contact Sales.

Enterprise Search supported versions

You can select the major and minor version. Patch versions are applied periodically for bug fixes and the like. When you deploy your app, you always get the latest available patches.

9.3
8.19

Enterprise Search deprecated versions

No deprecated versions.

Enterprise Search retired versions

The following versions have been retired and are no longer available. If your project uses a retired version, you must update to a supported version of Elastic Enterprise search, available as a premium service.

7.17

Elasticsearch (end of life)

Replaced by Elastic Enterprise Search

Elasticsearch changed its license in version 7.11 and is no longer open source. Instead, Upsun offers Elastic Enterprise Search as a premium service. Contact Sales to add it to your project.

Versions up to 7.10 remain available but receive no further security updates from upstream. For full reference documentation, see the Elasticsearch docs. If you don’t need the Enterprise Search feature set, OpenSearch is a free, actively maintained alternative. To switch, follow the same procedure as for upgrading.

Elasticsearch supported versions

You can select the major and minor version. Patch versions are applied periodically for bug fixes and the like. When you deploy your app, you always get the latest available patches.

7.10

Elasticsearch deprecated versions

No deprecated versions.

Elasticsearch retired versions

The following versions have been retired and are no longer available. If your project uses a retired version, you must update to a supported version of Elasticsearch or a supported version of Enterprise Search, available as a premium service.

Usage example

1. Configure the service

To define the service, use the elasticsearch type: For Elastic Enterprise Search, use the elasticsearch-enterprise type and specify a supported version. Changing the name of the service replaces it with a new service and all existing data is lost. Back up your data before changing the service.

2. Define the relationship

To define the relationship, use the following configuration:

Using default endpoints
Using explicit endpoints

You can define SERVICE_NAME as you like, so long as it’s unique between all defined services and matches in both the application and services configuration.The example above uses default endpoint configuration for relationships. That is, it uses default endpoints behind the scenes, providing a relationship (the network address a service is accessible from) that is identical to the name of that service.Depending on your needs, instead of default endpoint configuration, you can use explicit endpoint configuration.With the above definition, the application container now has access to the service through the relationship SERVICE_NAME and its corresponding service environment variables.

You can define SERVICE_NAME and <RELATIONSHIP_NAME> as you like, so long as it’s unique between all defined services and relationships and matches in both the application and services configuration.The example above uses explicit endpoint configuration for relationships.Depending on your needs, instead of explicit endpoint configuration, you can use default endpoint configuration.With the above definition, the application container now has access to the service through the relationship <RELATIONSHIP_NAME> and its corresponding service environment variables.

Example configuration

Using default endpoints
Using explicit endpoints

For Elastic Enterprise Search, use the elasticsearch-enterprise type and specify a supported version.

Use in app

To use the configured service in your app, add a configuration file similar to the following to your project. Configuration for Elastic Enterprise Search can differ. Specify elasticsearch-enterprise as the service type. This configuration defines a single application (myapp). myapp has access to the elasticsearch service through the corresponding service environment variables (as per default endpoint configuration for relationships). From this, myapp can retrieve access credentials to the service through the relationship environment variable.

myapp/.environment

# Set environment variables for individual credentials.
# See: /docs/development/variables#service-environment-variables
export ELASTIC_SCHEME="${ELASTICSEARCH_SCHEME}"
export ELASTIC_HOST="${ELASTICSEARCH_HOST}"
export ELASTIC_PORT="${ELASTICSEARCH_PORT}"

# Surface more common Elasticsearch connection string variables for use in app.
export ELASTIC_USERNAME="${ELASTICSEARCH_USERNAME}"
export ELASTIC_PASSWORD="${ELASTICSEARCH_PASSWORD}"
export ELASTIC_HOSTS=["$ELASTIC_SCHEME://$ELASTIC_HOST:$ELASTIC_PORT"]

The .environment file in the myapp directory is automatically sourced by Upsun into the runtime environment, making ELASTIC_HOSTS available to connect to the service. ELASTIC_HOSTS, and all Upsun-service environment variables like ELASTICSEARCH_HOST, are environment-dependent. Unlike the build produced for a given commit, they can’t be reused across environments and only allow your app to connect to a single service instance on a single environment. A file similar to this is generated automatically for you when using the upsun ify command to migrate a codebase to Upsun.

When you create an index on Elasticsearch, don’t specify the number_of_shards or number_of_replicas settings in your Elasticsearch API call. These values are set automatically based on available resources.

Relationship reference

For each service defined through a relationship to your application, Upsun automatically generates corresponding environment variables within your application container, in the $<RELATIONSHIP-NAME>_<SERVICE-PROPERTY> format. Here is example information available through the service environment variables themselves, or through the PLATFORM_RELATIONSHIPS environment variable.

Service environment variables
`PLATFORM_RELATIONSHIPS` environment variable

You can obtain the complete list of available service environment variables in your app container by running upsun ssh env.The information about the relationship can change when an app is redeployed or restarted or the relationship is changed. Your apps should only rely on the service environment variables directly rather than hard-coding any values.

For some advanced use cases, you can use the PLATFORM_RELATIONSHIPS environment variable. The structure of the PLATFORM_RELATIONSHIPS environment variable can be obtained by running upsun relationships in your terminal:Here is an example of how to gather PLATFORM_RELATIONSHIPS environment variable information in a .environment file:

.environment

# Decode the built-in credentials object variable.
export RELATIONSHIPS_JSON="$(echo "$PLATFORM_RELATIONSHIPS" | base64 --decode)"

# Set environment variables for individual credentials.
export APP_ELASTICSEARCH_HOST=="$(echo "$RELATIONSHIPS_JSON" | jq -r '.elasticsearch[0].host')"

For Elastic Enterprise Search, use elasticsearch-enterprise as the service type.

Authentication

This section applies to the Elasticsearch (end of life) service only. By default, Elasticsearch (end of life) has no authentication. No username or password is required to connect to it. Starting with Elasticsearch 7.2, you can optionally enable HTTP Basic authentication. To do so, include the following in your .upsun/config.yaml configuration: For Elastic Enterprise Search, use the elasticsearch-enterprise type and and specify a supported version. That enables mandatory HTTP Basic auth on all requests. The credentials are available in any relationships that point at that service, in the username and password properties. You can obtain the complete list of available service environment variables in your app container by running upsun ssh env. Note that the information about the relationship can change when an app is redeployed or restarted or the relationship is changed. So your apps should only rely on the service environment variables directly rather than hard coding any values. This functionality is generally not required if Elasticsearch isn’t exposed on its own public HTTP route. However, certain applications may require it, or it lets you safely expose Elasticsearch directly to the web. To do so, add a route to .upsun/routes.yaml that has elasticsearch:elasticsearch as its upstream (where elasticsearch is whatever you named the service). For example:

Plugins

Elasticsearch offers a number of plugins. To enable them, list them under the configuration.plugins key in your .upsun/config.yaml file: For Elastic Enterprise Search, use the elasticsearch-enterprise type and specify a supported version. In this example, the analysis-icu plugin is enabled. If there is a publicly available plugin you need that isn’t listed here, contact support.

Available plugins

This is the complete list of official Elasticsearch plugins that can be enabled:

Plugin	Description	2.4	5.x	6.x	7.x	8.x
`analysis-icu`	Support ICU Unicode text analysis	*	*	*	*	*
`analysis-nori`	Integrates Lucene Nori analysis module into Elasticsearch			*	*	*
`analysis-kuromoji`	Japanese language support	*	*	*	*	*
`analysis-smartcn`	Smart Chinese Analysis Plugins	*	*	*	*	*
`analysis-stempel`	Stempel Polish Analysis Plugin	*	*	*	*	*
`analysis-phonetic`	Phonetic analysis	*	*	*	*	*
`analysis-ukrainian`	Ukrainian language support		*	*	*	*
`cloud-aws`	AWS Cloud plugin, allows storing indices on AWS S3	*
`delete-by-query`	Support for deleting documents matching a given query	*
`discovery-multicast`	Ability to form a cluster using TCP/IP multicast messages	*
`ingest-attachment`	Extract file attachments in common formats (such as PPT, XLS, and PDF)		*	*	*	*
`ingest-user-agent`	Extracts details from the user agent string a browser sends with its web requests		*	*
`lang-javascript`	JavaScript language plugin, allows the use of JavaScript in Elasticsearch scripts		*
`lang-python`	Python language plugin, allows the use of Python in Elasticsearch scripts	*	*
`mapper-annotated-text`	Adds support for text fields with markup used to inject annotation tokens into the index			*	*	*
`mapper-attachments`	Mapper attachments plugin for indexing common file types	*	*
`mapper-murmur3`	Murmur3 mapper plugin for computing hashes at index-time	*	*	*	*	*
`mapper-size`	Size mapper plugin, enables the `_size` meta field	*	*	*	*	*
`repository-s3`	Support for using S3 as a repository for Snapshot/Restore		*	*	*	*
`transport-nio`	Support for NIO transport				*	*

Plugin removal

Removing plugins previously added in your .upsun/config.yaml file doesn’t automatically uninstall them from your Elasticsearch instances. This is deliberate, as removing a plugin can result in data loss or corruption of existing data that relied on that plugin. Removing a plugin usually requires reindexing. To permanently remove a previously enabled plugin, upgrade the service to create a new instance of Elasticsearch and migrate to it. In most cases this isn’t necessary, as an unused plugin has no appreciable impact on the server.

Upgrading

The Elasticsearch data format sometimes changes between versions in incompatible ways. Elasticsearch doesn’t include a data upgrade mechanism as it’s expected that all indexes can be regenerated from stable data if needed. To upgrade (or downgrade) Elasticsearch, use a new service from scratch. There are two approaches.

Destructive upgrade

In your .upsun/config.yaml file, change the version and name of your Elasticsearch service. Also update the reference to the renamed service in the application’s relationship block. When you push the change to Upsun, the old service is deleted and a new one with the new name is created with no data. You can then have your application reindex data as appropriate. This approach has the downsides of temporarily having an empty Elasticsearch instance, which your application might or might not handle gracefully, and needing to rebuild your index afterward. Depending on the size of your data that could take a while.

Transitional upgrade

With a transitional approach, you temporarily have two Elasticsearch services. Add a second Elasticsearch service with the new version, a new name, and give it a new relationship in .upsun/config.yaml. You can optionally run in that configuration for a while to allow your application to populate indexes in the new service as well. Once you’re ready to switch over, remove the old Elasticsearch service and relationship. You can optionally give the new Elasticsearch service the old relationship name if that’s easier for your app to handle. Your application is now using the new Elasticsearch service. This approach has the benefit of never being without a working Elasticsearch instance. On the downside, it requires two running Elasticsearch servers temporarily, each of which consumes resources and needs adequate disk space. Depending on the size of your data, this can require significant space.

Exporting data

Elasticsearch data is stored in on-disk indexes. The recommended way to export it is using the Snapshot and Restore API over an SSH tunnel.

Open an SSH tunnel to your Elasticsearch service:

Terminal

upsun tunnel:single --relationship <RELATIONSHIP_NAME>

Terminal

curl -X PUT "http://127.0.0.1:9200/_snapshot/my_backup" \
  -H 'Content-Type: application/json' \
  -d '{
    "type": "fs",
    "settings": {
      "location": "/tmp/es-snapshots"
    }
  }'

Trigger a snapshot:

Terminal

curl -X PUT "http://127.0.0.1:9200/_snapshot/my_backup/snapshot_1?wait_for_completion=true"

Retrieve the snapshot files from the service container using the CLI:

Terminal

upsun ssh -- tar -czf /tmp/es-snapshot.tar.gz /tmp/es-snapshots
upsun scp remote:/tmp/es-snapshot.tar.gz ./es-snapshot.tar.gz

Alternatively, you can re-index all your data from the primary data source (for example your application database) instead of using snapshots. If authentication is enabled on your Elasticsearch service, add the -u user:password flag to the curl commands.

​Elastic Enterprise Search (current)

​Enterprise Search supported versions

​Enterprise Search deprecated versions

​Enterprise Search retired versions

​Elasticsearch (end of life)

Replaced by Elastic Enterprise Search

​Elasticsearch supported versions

​Elasticsearch deprecated versions

​Elasticsearch retired versions

​Usage example

​1. Configure the service

​2. Define the relationship

​Example configuration

​Use in app

​Relationship reference

​Authentication

​Plugins

​Available plugins

​Plugin removal

​Upgrading

​Destructive upgrade

​Transitional upgrade

​Exporting data

Elastic Enterprise Search (current)

Enterprise Search supported versions

Enterprise Search deprecated versions

Enterprise Search retired versions

Elasticsearch (end of life)

Elasticsearch supported versions

Elasticsearch deprecated versions

Elasticsearch retired versions

Usage example

1. Configure the service

2. Define the relationship

Example configuration

Use in app

Relationship reference

Authentication

Plugins

Available plugins

Plugin removal

Upgrading

Destructive upgrade

Transitional upgrade

Exporting data