Migrate Be Production instance to new VPS

[pacodelaluna]

Checklist based on general guide https://github.com/openfoodfoundation/ofn-install/wiki/Migrating-a-Production-Server

Each server may be slightly different, so make sure to update the list as needed.

1. Setting up the new server

• Check old server config for any additional services to be aware of. Document any necessary steps for migration. Eg:
  • ls /etc/nginx/sites-enabled
  • systemctl --state=running
• Hosting: provision new server with current supported version of Ubuntu
• DNS: drop TTL of records pointing to old IP address (eg root and www).

config

• Review host_vars/x/config.yml, clean up if needed
• Review group_vars/x.yml, clean up if needed
• Review ofn-secrets:be_prod/secrets.yml, clean up if needed
  • Take note of third party services, and if they need updating (eg update IP whitelist for new IP)

setup

Enable passthrough on current server to allow new server to generate a certificate:

• ansible-playbook playbooks/letsencrypt_proxy.yml -l be_prod -e "proxy_target=<new_ip>"

Then setup new server. Ensure you have the correct secrets (current secrets are usually fine).
ansible-playbook -l be_prod2 -e "@../ofn-secrets/be_prod/secrets.yml" playbooks/

• setup.yml
• provision.yml
• deploy.yml
• db_integrations (Permit DB access for n8n, Metabase)

initial migration

• Ensure sidekiq is disabled, to avoid creating subscription orders when data is loaded:
  sudo systemctl stop sidekiq && sudo systemctl disable sidekiq
• Setup direct ssh access for ofn-admin and openfoodnetwork as per guide

ansible-playbook -l be_prod -e rsync_to=be_prod2 playbooks/

• db_transfer.yml &&
• transfer_assets.yml

Make sure to clear cache so that instance settings are applied:
cd ~/apps/openfoodnetwork/current; bin/rails runner -e production "Rails.cache.clear"

2. Testing

Use a hosts file entry to test new server using the domain name

• test reboot
• send test mail (/admin/mail_methods/edit).
• terms of service file: /admin/terms_of_service_files
• shop catalogue display correctly, with images, add to cart, begin checkout, login.
  note: check cookies if login won't work
• Check integrations
  • Payments (check Stripe connect status /admin/stripe_connect_settings/edit)
  • New Relic
  • Bugsnag

3. Migration

preparation

• Reset database on new server, to avoid any migration issues due to being out of sync
  bin/rake db:reset (You will need to confirm. Make sure you're on the new server!)
• Update ansible_host IP in inventory/hosts and ensure provision works (this will update host in .env.production).
  ansible-playbook playbooks/provision.yml -l be_prod
• ansible-playbook playbooks/deploy.yml -l be_prod -e "git_version=vX.Y.Z" matching version with current prod (you can check this at the bottom of /admin/ dashboard)
• old server: make a tiny data change to verify later (eg add . in meta description /admin/general_settings/edit)

switchover: old server

• 🚧 ansible-playbook playbooks/maintenance_mode.yml -l be_prod
• ansible-playbook -l be_prod -e rsync_to=be_prod2 playbooks/db_transfer.yml &&
• ansible-playbook -l be_prod -e rsync_to=be_prod2 playbooks/transfer_assets.yml
• Transfer /var/lib/redis-jobs/dump.rdb to new server (see guide
• sudo systemctl stop postgresql (ensure other integrations no longer touch it)

switchover: new server

• sudo systemctl restart puma; sudo systemctl start sidekiq redis-jobs
• cd ~/apps/openfoodnetwork/current; bin/rails runner -e production "Rails.cache.clear" (or migrate redis-cache/dump.rdb also)
• Use a hosts file entry to test that there are no alarm bells, eg:
  • tiny data change is present. undo it.
  • shopfront and checkout looks good
  • upload a product image
  • ~/apps/openfoodnetwork/current/log/production.log and sidekiq.log
• ⏭️ Update DNS to point to new server
• get confirmation from local team
• make sure the entries in ofn-install are up to date: set the new IP address and remove any temporary entry made for the migration
• Update documentation:
  • https://github.com/openfoodfoundation/ofn-install/wiki/Current-servers
  • This migration guide if necessary

4. Cleanup (after 48hrs)

• check server access logs to verify no traffic
• shut down the old VPS
• delete old VPS (or rename for future deletion)
• check Metabase sync if required: https://data.openfoodnetwork.org/admin/databases/
• check n8n
• check backups are functioning

Rollback plan

• If an error occurs after DNS change, users may have interacted with the new server (eg made payments).
  • if serious, consider putting new server into maintenance mode to avoid further changes, while investigating
  • otherwise seek to resolve issue in-place.
  • avoid changing DNS back to old server, because it could be missing any new payments.

[pacodelaluna]

ofn-admin@be-prod:~$ ls /etc/nginx/sites-enabled
000_redirect_www.conf  default.conf  ofn_443.conf  ofn_80.conf

ofn-admin@be-prod:~$ systemctl --state=running
  UNIT                              LOAD   ACTIVE SUB     DESCRIPTION                                                  
  proc-sys-fs-binfmt_misc.automount loaded active running Arbitrary Executable File Formats File System Automount Point
  init.scope                        loaded active running System and Service Manager                                   
  session-54039.scope               loaded active running Session 54039 of user ofn-admin                              
  accounts-daemon.service           loaded active running Accounts Service                                             
  atd.service                       loaded active running Deferred execution scheduler                                 
  atop.service                      loaded active running Atop advanced performance monitor                            
  atopacct.service                  loaded active running Atop process accounting daemon                               
  cron.service                      loaded active running Regular background program processing daemon                 
  dbus.service                      loaded active running D-Bus System Message Bus                                     
  fail2ban.service                  loaded active running Fail2Ban Service                                             
  fwupd.service                     loaded active running Firmware update daemon                                       
  getty@tty1.service                loaded active running Getty on tty1                                                
  irqbalance.service                loaded active running irqbalance daemon                                            
  ModemManager.service              loaded active running Modem Manager                                                
  multipathd.service                loaded active running Device-Mapper Multipath Device Controller                    
  networkd-dispatcher.service       loaded active running Dispatcher daemon for systemd-networkd                       
  newrelic-infra.service            loaded active running New Relic Infrastructure Agent                               
  nginx.service                     loaded active running nginx - high performance web server                          
  polkit.service                    loaded active running Authorization Manager                                        
  postgresql@12-main.service        loaded active running PostgreSQL Cluster 12-main                                   
  puma.service                      loaded active running Puma HTTP Server                                             
  qemu-guest-agent.service          loaded active running QEMU Guest Agent                                             
  redis-cache.service               loaded active running Advanced key-value store                                     
  redis-jobs.service                loaded active running Advanced key-value store                                     
  redis-server.service              loaded active running Advanced key-value store                                     
  rsyslog.service                   loaded active running System Logging Service                                       
  serial-getty@ttyS0.service        loaded active running Serial Getty on ttyS0                                        
  sidekiq.service                   loaded active running sidekiq                                                      
  snapd.service                     loaded active running Snap Daemon                                                  
  ssh.service                       loaded active running OpenBSD Secure Shell server                                  
  systemd-journald.service          loaded active running Journal Service                                              
  systemd-logind.service            loaded active running Login Service                                                
  systemd-networkd.service          loaded active running Network Service                                              
  systemd-resolved.service          loaded active running Network Name Resolution                                      
  systemd-timesyncd.service         loaded active running Network Time Synchronization                                 
  systemd-udevd.service             loaded active running udev Kernel Device Manager                                   
  udisks2.service                   loaded active running Disk Manager                                                 
  unattended-upgrades.service       loaded active running Unattended Upgrades Shutdown                                 
  upower.service                    loaded active running Daemon for power management                                  
  user@1000.service                 loaded active running User Manager for UID 1000                                    
  dbus.socket                       loaded active running D-Bus System Message Bus Socket                              
  multipathd.socket                 loaded active running multipathd control socket                                    
  snapd.socket                      loaded active running Socket activation for snappy daemon                          
  syslog.socket                     loaded active running Syslog Socket                                                
  systemd-journald-audit.socket     loaded active running Journal Audit Socket                                         
  systemd-journald-dev-log.socket   loaded active running Journal Socket (/dev/log)                                    
  systemd-journald.socket           loaded active running Journal Socket                                               
  systemd-networkd.socket           loaded active running Network Service Netlink Socket                               
  systemd-udevd-control.socket      loaded active running udev Control Socket                                          
  systemd-udevd-kernel.socket       loaded active running udev Kernel Socket                                           

LOAD   = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB    = The low-level unit activation state, values depend on unit type.

50 loaded units listed.

[pacodelaluna]

Ok, I am not sure why but I having issues with the Nginx service on the live Belgium prod instance.
I have been able to load the playbook, but I was failing to run the provision one on the new server. I tried to run back the letsencrypt_proxy playbook and I faced an error with the Nginx setup now, related to the Brotli module. Actually, the Nginx was updated to 1.28.0, but not the Brotli module. Nginx was not able to restart, I had to remove the Brotli configuration for now.

-- The job identifier is 438295.
Apr 09 19:28:41 be-prod nginx[2870689]: nginx: [emerg] module "/etc/nginx/modules/ngx_http_brotli_filter_module.so" version 1026002 in>
Apr 09 19:28:41 be-prod systemd[1]: nginx.service: Control process exited, code=exited, status=1/FAILURE
-- Subject: Unit process exited
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
-- 
-- An ExecStart= process belonging to unit nginx.service has exited.
-- 
-- The process' exit code is 'exited' and its exit status is 1.
Apr 09 19:28:41 be-prod systemd[1]: nginx.service: Failed with result 'exit-code'.

[dacook]

I believe we use Brotli because it has better compression than gzip, which results in less network transfer costs and energy use.

There's an ansible role that used to build it, but now it's included more recent ubuntu distributions, so I'm guessing that was skipped:

    - role: brotli_nginx
      when: ansible_distribution_major_version <= '20'

I don't know what's wrong sorry, but it should be fine not to use brotli.

[dacook]

Hi Paco, have you only done the items that are checked above? Some of the preparation steps will be essential for the later steps.

For example, if you are planning to use the db_transfer.yml playbook , you will first need to:

Setup direct ssh access for ofn-admin and openfoodnetwork as per guide

Alternatively, you can of course manually download and upload the data, it will just take longer.

Also, you will need a valid SSL certificate for the prod domain. Are you planning to simply generate it after switching DNS?

[pacodelaluna]

Hi David,

I just managed the migration during the last hour. The server was cut for around 30 minutes, the Coopcircuits team was aware of it.
For the database transfer, I did it manually as I was not able to run the playbook from my local machine, maybe I missed something.
For the SSL certificates, I also started by copying them, and then I verified it was working good after the DNS switch. Renewal should work fine.
About Redis, I check that with redis-cli, and I couldn't find data at the time of migration, so I skipped it.
I did basic testing (update, checkout, file upload, email, Stripe), no issues, I will of course keep an eye on it tomorrow morning.
I did a PR to update the new IP address for Ansible.
I will update the documentation tomorrow.

Please tell me if you have any question or if I missed anything.

[dacook]

Congrats Paco!
It seems unusual to me that there would be no redis data. It exports the live data to file when you shutdown the redis process.
This is included in the migration guide because the data can contain scheduled jobs (for example sending emails, processing subscriptions). But perhaps there were no scheduled jobs at this time 🤷 .

[sigmundpetersen]

Hi @pacodelaluna can this be closed?

[pacodelaluna]

@sigmundpetersen Yes! I have decommissioned the old VPS on IsHosting, the backups are managed and the doc has been updated.