Travis CI Testing for NodeJS and Redis with a Restored AOF File

Following our previous blog post about Neo4j and Travis-CI this post covers how to use a restored Redis DB as data source for testing within a NodeJS app.

Main requirement: Use a restore of a Redis AOF File as a data source for NodeJS unit/component/integration tests.

Overview

That is the final file ... maybe it is easier installing Redis "by hand" than using the pre-configured Travis-CI services and maybe there are other ways to configure these services (before its initial upstart) by e.g. a config.yaml as stated here for some other database types ... Nevertheless this is one possible solution:

dist: trusty
sudo: required

language: node_js
node_js:
  - "6"

services:
  - redis-server

before_script:
  - npm install
  - cp ./config/travis/dnb_tst2_app6_s1_wb_cert_key.pem ~
  - cp ./config/travis/dnb_tst2_app6_s1_wb_cert_pub.pem ~
  - cp ./config/travis/data.tar ~
  - sudo service redis-server stop
  - sleep 10
  - sudo cat /etc/redis/redis.conf
  - sudo sed -i 's|appendonly no|appendonly yes|g' /etc/redis/redis.conf
  - sudo sed -i "s|# requirepass foobared|requirepass ${DNB_ENV_APP_S2_DB_PASS}|g" /etc/redis/redis.conf
  - sudo cat /etc/redis/redis.conf
  - sudo ls -al  /var/lib/redis
  - sudo find /var/lib/redis -mindepth 1 -delete
  - sudo tar -xvf ~/data.tar -C /var/lib/redis --strip 1 
  - sudo chown redis:redis /var/lib/redis/appendonly.aof
  - sudo chmod 660 /var/lib/redis/appendonly.aof 
  - sudo ls -al  /var/lib/redis
  - sudo service redis-server start
  - sleep 20

script:
  - sudo cat /var/log/redis/redis-server.log
  - swagger project start &
  - sleep 10
  - swagger project test

The whole can be found on Github.

Step by step

We use the following base image:

dist: trusty
sudo: required

And then declare the service utility to use:

services:
  - redis-server

In the before_script stage we then copy the data.tar backup file to the travis user home directory:

before_script:
  ...
  - cp ./config/travis/data.tar ~

Travis-CI services start at very first of the build cycle with the default Travis-CI config. As a consequence one needs to stop the Redis service and sleep a little before doing a custom configuration:

  - sudo service redis-server stop
  - sleep 10

After having stopped Redis we can start reconfiguring it ... First we edit the config file redis.conf so that Redis will use appendonly for data persistence (you might as well drop this setting and leave it with the RDB mode ... it is even faster at startup ... depends on the backup file you want to restore), then enable password authentication via requirepass, and the print the edited redis.conf file:

  - sudo sed -i 's|appendonly no|appendonly yes|g' /etc/redis/redis.conf
  - sudo sed -i "s|# requirepass foobared|requirepass ${DNB_ENV_APP_S2_DB_PASS}|g" /etc/redis/redis.conf
  - sudo cat /etc/redis/redis.conf

Next we take care of the data ... The pre-configured data path is /var/lib/redis which we delete completely, then tar the backup file into it, and adjust file permissions and ownership afterwards. Just in case we list the contents to be able to compare the pre-configured with our customized settings:

  - sudo ls -al  /var/lib/redis
  - sudo find /var/lib/redis -mindepth 1 -delete
  - sudo tar -xvf ~/data.tar -C /var/lib/redis --strip 1 
  - sudo chown redis:redis /var/lib/redis/appendonly.aof
  - sudo chmod 660 /var/lib/redis/appendonly.aof 
  - sudo ls -al  /var/lib/redis

After that there are only two more steps in the before_script stage ... start the redis-server service and then wait a little to make sure that Redis is up in the next build stage:

  - sudo service redis-server start
  - sleep 20

Almost done ... in the script build stage we look into redis-server.log to check the correct start-up and then we can start NodeJS (here with the swagger commands), sleep again a little, and then test:

script:
  - sudo cat /var/log/redis/redis-server.log
  - swagger project start &
  - sleep 10
  - swagger project test

Have fun testing.

Further Information

Github Repo:

https://github.com/daten-und-bass/postcode-geopos-api/

https://github.com/daten-und-bass/postcode-geopos-api/blob/master/.travis.yml

Travis-CI:

https://docs.travis-ci.com/user/database-setup/

Redis Data Persistence Options:

https://redis.io/topics/persistence

Neo4j and Travis-CI:

https://daten-und-bass.io/blog/travis-ci-testing-for-nodejs-and-neo4j-with-a-restored-db-backup/