ACE persistent data backup and restoration
Backup
Couchbase is the primary source for backing up ACE persistent data. To make (and restore) Couchbase backups we recommend you to use the Couchbase backup tools.
Note that search access to restored data can take a while, since indexing the data can take a long time. Therefore, we recommend backing up Solr indexes along with Couchbase data.
NOTE: Solr backups should always be made prior to Couchbase backups since otherwise you risk ending up with restored search data that doesn't match the Couchbase data.
Restoration
In case of data loss from Couchbase, we recommend the following steps in order to restore the system to its latest backup state:
-
Shut down all running ACE and project services.
-
Shut down all backend services (Solr, Couchbase, Kafka, ZooKeeper).
-
Investigate and try to fix the issues(s) that forced you to do a data restoration.
-
Start Couchbase and, if needed, recreate the cluster.
-
Restore the Couchbase data from backup (see Enterprise Backup Tutorial).
-
Start Solr. If Solr data is lost, restore from backup.
-
Start Kafka and ZooKeeper.
-
Start the ACE services, followed by the project services.
Deploying a new version of ACE in a Docker swarm
To update the ACE version in a Docker swarm, first pull (update) the new Docker images on each of the swarm nodes. Then update the swarm service stack using the following command on one of the swarm manager:
docker stack deploy -f docker-compose-swarm-upgrade-ace.yml
There may be special cases where a specific order of deployment is necessary. In such cases, the ACE release notes will include any needed information.
How the services behave during upgrade can be customised in the compose-file under each service config (see Update Config).
To do a rolling upgrade of a specific swarm service, see Apply rolling updates to a service.
Making changes to service configuration in a Docker swarm
docker stack deploy -f docker-compose-swarm.yml should apply changes to configuration.
Docker configuration best practices
Avoid using Docker secrets for configuration in general since secrets cannot be easily modified and configuration may need to be changed in a running Docker swarm service stack.
Troubleshooting common problems
| Problem | Solution |
|---|---|
| New content not having any effect on site / application. | Make sure that the content is published on the view used by the site / application. |
| Unable to find newly created content, even when searching in CoDe. | Verify that the content is available on an indexed view in the aceIndexing variant. |
| None of the other solutions help. | Check the logs for the ace-indexer and kafka-connect to verify that they are alive and well. If not, restart the kafka-connect containers, do a full reindex, and verify that everything works now. |
General maintenance
Monitoring
It is always a good idea to keep track of metrics and monitor logs for anomalies.
How to pre-populate ACE caches
If you are running a web site on ACE, for instance using ace-web, then you should use a cache warmer before opening for external load.
Third party maintenance guidelines
Couchbase
Each night there should be a new full or incremental backup of the Couchbase data.
Be sure to monitor the disk space usage of Couchbase. As far as we know, it's not possible for Couchbase to recover from a state where it has run out of disk space.