Following our previous blog post about Neo4j and Travis-CI this post covers how to use a restored Redis DB as data source for testing within a NodeJS app.
Main requirement: Use a restore of a Redis AOF File as a data source for NodeJS unit/component/integration tests.
Overview
That is the final file ... maybe it is easier installing Redis "by hand" than using the pre-configured Travis-CI services and maybe there are other ways to configure these services (before its initial upstart) by e.g. a config.yaml as stated here for some other database types ... Nevertheless this is one possible solution:
dist: trusty
sudo: required
language: node_js
node_js:
- "6"
services:
- redis-server
before_script:
- npm install
- cp ./config/travis/dnb_tst2_app6_s1_wb_cert_key.pem ~
- cp ./config/travis/dnb_tst2_app6_s1_wb_cert_pub.pem ~
- cp ./config/travis/data.tar ~
- sudo service redis-server stop
- sleep 10
- sudo cat /etc/redis/redis.conf
- sudo sed -i 's|appendonly no|appendonly yes|g' /etc/redis/redis.conf
- sudo sed -i "s|# requirepass foobared|requirepass ${DNB_ENV_APP_S2_DB_PASS}|g" /etc/redis/redis.conf
- sudo cat /etc/redis/redis.conf
- sudo ls -al /var/lib/redis
- sudo find /var/lib/redis -mindepth 1 -delete
- sudo tar -xvf ~/data.tar -C /var/lib/redis --strip 1
- sudo chown redis:redis /var/lib/redis/appendonly.aof
- sudo chmod 660 /var/lib/redis/appendonly.aof
- sudo ls -al /var/lib/redis
- sudo service redis-server start
- sleep 20
script:
- sudo cat /var/log/redis/redis-server.log
- swagger project start &
- sleep 10
- swagger project test
The whole can be found on Github.
Step by step
We use the following base image:
dist: trusty
sudo: required
And then declare the service utility to use:
services:
- redis-server
In the before_script stage we then copy the data.tar backup file to the travis user home directory:
before_script:
...
- cp ./config/travis/data.tar ~
Travis-CI services start at very first of the build cycle with the default Travis-CI config. As a consequence one needs to stop the Redis service and sleep a little before doing a custom configuration:
- sudo service redis-server stop
- sleep 10
After having stopped Redis we can start reconfiguring it ... First we edit the config file redis.conf so that Redis will use appendonly for data persistence (you might as well drop this setting and leave it with the RDB mode ... it is even faster at startup ... depends on the backup file you want to restore), then enable password authentication via requirepass, and the print the edited redis.conf file:
- sudo sed -i 's|appendonly no|appendonly yes|g' /etc/redis/redis.conf
- sudo sed -i "s|# requirepass foobared|requirepass ${DNB_ENV_APP_S2_DB_PASS}|g" /etc/redis/redis.conf
- sudo cat /etc/redis/redis.conf
Next we take care of the data ... The pre-configured data path is /var/lib/redis which we delete completely, then tar the backup file into it, and adjust file permissions and ownership afterwards. Just in case we list the contents to be able to compare the pre-configured with our customized settings:
- sudo ls -al /var/lib/redis
- sudo find /var/lib/redis -mindepth 1 -delete
- sudo tar -xvf ~/data.tar -C /var/lib/redis --strip 1
- sudo chown redis:redis /var/lib/redis/appendonly.aof
- sudo chmod 660 /var/lib/redis/appendonly.aof
- sudo ls -al /var/lib/redis
After that there are only two more steps in the before_script stage ... start the redis-server service and then wait a little to make sure that Redis is up in the next build stage:
- sudo service redis-server start
- sleep 20
Almost done ... in the script build stage we look into redis-server.log to check the correct start-up and then we can start NodeJS (here with the swagger commands), sleep again a little, and then test:
script:
- sudo cat /var/log/redis/redis-server.log
- swagger project start &
- sleep 10
- swagger project test
Have fun testing.
Further Information
Github Repo:
https://github.com/daten-und-bass/postcode-geopos-api/
https://github.com/daten-und-bass/postcode-geopos-api/blob/master/.travis.yml
Travis-CI:
https://docs.travis-ci.com/user/database-setup/
Redis Data Persistence Options:
https://redis.io/topics/persistence
Neo4j and Travis-CI:
https://daten-und-bass.io/blog/travis-ci-testing-for-nodejs-and-neo4j-with-a-restored-db-backup/