> ## Documentation Index
> Fetch the complete documentation index at: https://developer.upsun.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Sanitizing databases: PostgreSQL and Django

> Sanitize PostgreSQL data in preview environments for Django apps.

export const VariableBlock = ({name}) => {
  return <var spellCheck={false} title={`Replace '${name}' with your own data`}>{name}</var>;
};

Databases of live websites often contain personally identifiable information (PII)
such as full names, mailing addresses, and phone numbers.
To ensure people reviewing code changes can't access information they shouldn't,
sanitize your databases of any PII that they may contain.

This example goes through the process for a PostgreSQL database using Django.

## Before you begin

You need:

* A project with a [PostgreSQL database](/docs/add-services/postgresql).
* A command interface installed:
  * If doing it manually, the [Upsun CLI](/cli).
  * Otherwise, make sure `pqsl` is installed in your environment.

This guide is about sanitizing PostgreSQL databases.

This guide doesn't address:

* Sanitizing NoSQL Databases (such as [MongoDB](/docs/add-services/mongodb))
* Input validation and input sanitization, which both help prevent security vulnerabilities

## Sanitize the database

Make sure that you only sanitize preview environments and **never** the production environment.
Otherwise you may lose most or even all of the relevant data stored in your database.

First, take a [database dump](/docs/add-services/postgresql#exporting-data) of your preview environment.
This is just a safety precaution.
Production data isn't altered.
To get a database dump, run the following command:

`upsun db:dump -e DEVELOPMENT_ENVIRONMENT_NAME`.

<Tabs>
  <Tab title="Manually">
    Assumptions:

    * `users` is the table where all of your PII is stored in the `staging` development database.
    * `staging` is an exact copy of your production database.

    1. Connect to the `staging` database by running `upsun sql -e staging`.

    2. Display all fields from your `users` table, to select which ones need to be redacted. Run the following query:

       ```sql theme={null}
       main=> SELECT * FROM users;
       ```

       You see output like the following:

       ```sql theme={null}
       id   |                user_email               |     display_name
       -----+-----------------------------------------+-----------------------
       3501 | daniel02@yourcompany.com                | Jason Brown
       3502 | ismith@kim.com                          | Sandra Griffin
       3503 | olee@coleman-rodriguez.com              | Miss Christine Morgan
       ```

    3. Change the fields where PII is contained with the [`UPDATE` statement](https://mariadb.com/kb/en/update/).
       For example, to change the display name of users with an email address not in your company's domain
       to a random value, run the following query:

       ```sql theme={null}
       UPDATE users
       SET display_name==substring(md5(display_name||'$PLATFORM_PROJECT_ENTROPY') for 8);
       WHERE email NOT LIKE '%@yourcompany%'
       ```

       Adapt and run that query for all fields that you need to sanitize.
       If you modify fields that you shouldn't alter, [you can restore them](/docs/environments/restore) from the dump you took in step 1.

       You can create a script to automate the sanitization process to be run automatically on each new deployment.
       Once you have a working script, add your script to sanitize the database to [a `deploy` hook](/docs/configure-apps/hooks/hooks-comparison#deploy-hook):

       ```yaml theme={null}
       applications:
           myapp:

               # ...

               hooks:
                   deploy: |

                       # ...

                       cd /app/public
                       if [ "$PLATFORM_ENVIRONMENT_TYPE" = production ]; then
                           # Do whatever you want on the production site.
                       else
                           # The sanitization of the database should happen here (since it's non-production)
                           sanitize_the_database.sh
                       fi
       ```
  </Tab>

  <Tab title="Using a script with Django and `psql`">
    Assumptions:

    * `users` is the table where all of your PII is stored in the `staging` development database.
    * `database` is the relationship name for the PostgreSQL service.

    Set up a script by following these steps:

    1. Retrieve service credentials from the [service environment variables](/docs/development/variables#service-environment-variables) to use the `psql` command interface.
       Export these values to a [`.environment` file](/docs/development/variables/set-variables#set-variables-via-script)
       or include them directly in the sanitization script.

       ```bash .environment theme={null}
       # Pull credentials from the service environment variables.
       export DB_USER="${DATABASE_USERNAME}"
       export DB_HOST="${DATABASE_HOST}"
       export DB_PORT="${DATABASE_PORT}"
       export DB_PASS="${DATABASE_PASSWORD}"
       ```

    2. Create an executable sanitizing script by running the following command:

       ```bash theme={null}
       touch sanitize.sh && chmod +x sanitize.sh
       ```

    3. Make the script sanitize environments with an [environment type](/docs/administration/users#environment-type-roles)
       other than `production`.

       The following example runs only in preview environments
       and sanitizes the `display_name` and `email` columns of the `users` table.
       Adjust the details to fit your data.

       ```bash sanitize.sh theme={null}
       #!/usr/bin/env bash

       if [ "$PLATFORM_ENVIRONMENT_TYPE" != production ]; then
           # Sanitize data
           PGPASSWORD=$DB_PASS psql -c "UPDATE users SET display_name=substring(md5(display_name||'$PLATFORM_PROJECT_ENTROPY') for 8);" -U $DB_USER -h $DB_HOST -p $DB_PORT
           PGPASSWORD=$DB_PASS psql -c "UPDATE users SET email=substring(md5(email||'$PLATFORM_PROJECT_ENTROPY') for 8);" -U $DB_USER -h $DB_HOST -p $DB_PORT
       fi
       ```

       To sanitize only on the initial deploy and not all future deploys,
       on sanitization create a file on a [mount](/docs/configure-apps/image-properties/mounts).
       Then add a check for the file as in the following example:

       ```bash sanitize.sh theme={null}
       #!/usr/bin/env bash

       if [ "$PLATFORM_ENVIRONMENT_TYPE" != production ] && [ ! -f <MOUNT_PATH>/is_sanitized ]; then
           # Sanitize data
           touch <MOUNT_PATH>/is_sanitized
       fi
       ```

    4. Update the deploy hook to run your script on each deploy.

       ```yaml .upsun/config.yaml theme={null}
       applications:
         myapp:
           hooks:
             build: ...
             deploy: |
               python manage.py migrate
               bash sanitize.sh
       ```

    5. Commit your changes by running the following command:

       ```bash theme={null}
       git add .environment sanitize.sh .upsun/config.yaml&& git commit -m "Add sanitization."
       ```

       Push the changes to `staging` and verify that environment's database was sanitized.
       Once merged to production, all data from future preview environments are sanitized on environment creation.
  </Tab>
</Tabs>

## What's next

You learned how to remove sensitive data from a database.

To replace sensitive data with other meaningful data, you can add a `faker` to the process.
A `faker` is a program that generates fake data that looks real.
Having meaningful PII-free data allows you to keep your current Q\&A, external reviews, and other processes.
To add a faker, adapt your sanitizing queries to replace each value that contains PII with a new value generated by the faker.

You might also want to make sure that you [implement input validation](https://cheatsheetseries.owasp.org/cheatsheets/Input_Validation_Cheat_Sheet.html#goals-of-input-validation).

If your database contains a lot of data, consider using the [`REINDEX` statement](https://www.postgresql.org/docs/current/sql-reindex.html) to help improve performance.