Override or Customize SystemD Environment Settings

There are many reasons to need to change default environment values in the systemd service files. For example, the default PGDATA location.

Systemd service files (also called "systemd unit files") detail what commands should be used to start and stop services as well as environment variables that should be set for those services to run, much like how system users can export environment variables to be used in their login session in their shell profile files (e.g. .bash_profile or .zsh_profile). The service files will contain default values for those environment variables, such as PGDATA=/var/lib/pgsql/16/data for PostgreSQL 16's service file which needs to be changed if you have set up your server's data directory at a different location.

The systemd service files installed by packages are placed somewhere under /usr/lib/systemd. When changes are needed for the environment variables set in a given service file it can be tempting to edit the value in that file directly, but later package updates can overwrite these files, destroying your changes and breaking the service on your system, so this should not be done.

Instead, systemd itself provides a means of a service's definition via files under /etc/systemd. Below we show an example for changing the default PGDATA directory for PostgreSQL 16 systemd service file, but it is valid for EPAS services as well, by using edb-as-XX.service services instead.

With PostgreSQL 16, the default value for the environment variable PGDATA is set in the file /usr/lib/systemd/system/postgresql-16.service and is typically set to the /var/lib/pgsql/16/data as seen by this line in the file's [Service] section:

Environment=PGDATA=/var/lib/pgsql/16/data/

1. To edit the PostgreSQL service information we could create and edit the needed file under /etc/systemd, but systemd will handle that for us with the systemctl edit {service name} command. Running this will open the file appropriate (creating it if needed) with the default system editor (that can be overridden by setting $SYSTEMD_EDITOR):

systemctl edit postgresql-16.service

2. Override the PGDATA environment variable by adding the following text and replacing /data/pgsql/data with your desired location. If the database places its PID file in its data directory we need to tell the service where that is so we'll set that here, too:

[Service]
Environment=PGDATA=/data/pgsql/data
PIDFile=/data/pgsql/data/postmaster.pid

3. Saving will create the file /etc/systemd/system/postgresql-16.service.d/override.conf.

4. The systemd daemon automatically reloads itself if the systemctl edit {service name} command was used. But if the user edited files directly inside the /etc/systemd directory, they will now need to be reloaded so that new values take effect:

systemctl daemon-reload

5. You can check the current service definition with systemctl cat {service name}, the overrides you've just set will be at the end:

systemctl cat postgresql-16.service

6. If the PGDATA setting is correct, then the database can be started at this point with systemctl start {service name}:

systemctl start postgresql-16.service

For the Postgres and EPAS systemd service definitions, we should usually not need to change anything beyond that the data directory path. Editing other pieces of the service definition can have negative and surprising effects, and should be done only following EDB recommendations. There are several known trouble points:

1. Avoid changing the data directory location of EPAS to be the same as the default for community PostgreSQL (i.e. do not change /var/lib/edb/<major_version/data to /var/lib/pgsql/<major_version>/data) and vice versa. The defaults for each are intentionally separate to allow for easier migrations. Using the same directory will lead to unnecessary permissions issues as one operating system user creates the directory and the other attempts to write to it, leading to race conditions and intermittent boot failures.

2. Avoid using remote LDAP user/groups for services for User and Group values in services. Such users are meant for human users, not services. Using them for services leads to race conditions on boot, which cause all services (not only database services) to not start automatically on boot. Adding Requires=network-online.target to the database unit only guarantees some network is available, not that the remote LDAP server is reachable and responsive, so race conditions still might cause your database service to not start on boot. Moreover, any failover might also be impacted, in case the failure (to which the failover is reacting to) also affects the connectivity to the LDAP server, meaning the service might not start at the failover. Error messages associated with this are in the form "Failed at step GROUP" and "Failed at step USER", reported directly by systemd.

3. Avoid using listen_addresses=<IP> in your database configuration (i.e, prefer listen_addresses=* and instead filtering the accesses with pg_hba.conf and firewalls), because even with Requires=network-online.target there is no guarantee that that specific IP would have been assigned to an interface (especially in a server with multiple interfaces, IPs and log boots), which means a race condition exists and could mean the database service wouldn't start on boot. Moreover, the IP can be removed at any moment, which means even a service restart when the instance was up is not guaranteed to work.

4. If remote volumes are being used, then please ask EDB for help when setting the _netdev flag for each mount point and adjusting the unit boot order with systemd.path afterwards. Failure to set them properly leads to race conditions on boot, which might cause the service to not start or start with fewer data directories (especially tablespaces), leading to posterior crashes.