Travis CI Testing for NodeJS and Neo4j with a Restored DB Backup

by arne

After struggling for quite a while with how to integrate a Neo4j database into automated testing with Travis CI (using the Travis services utility) and - surprisingly - not finding too much information about it we thought it is worth sharing how we managed to use a restored Neo4j backup in unit tests for NodeJS.

Main requirement: Use a restore of a Neo4j graph.db as a data source for NodeJS unit/component/integration tests.

Overview

First off, here is the final result ... maybe this is not the only way and maybe even this is not the best way. If you know e.g. how to pass a config.yaml to Neo4j as mentioned here for other databases please drop us a hint.

dist: trusty
sudo: required

language: node_js
node_js:
  - "6"

services:
  - neo4j
  - redis-server

before_script:
  - npm install
  - cp ./config/travis/app3_tst2_s1_wb_key.pem ~
  - cp ./config/travis/app3_tst2_s1_wb_pub.pem ~
  - cp ./config/travis/pki1_tst2_ca_pub.pem ~
  - cp ./config/travis/data.tar ~
  - sudo service neo4j stop
  - sleep 20
  - sudo rm /var/lib/neo4j/certificates/neo4j.key
  - sudo rm /var/lib/neo4j/certificates/neo4j.cert
  - sudo cp ./config/travis/app3_tst2_s2_db_key.pem /var/lib/neo4j/certificates/neo4j.key
  - sudo cp ./config/travis/app3_tst2_s2_db_pub.pem /var/lib/neo4j/certificates/neo4j.cert
  - sudo rm -rd /var/lib/neo4j/data/databases
  - (cd /var/lib/neo4j/data && sudo tar xvf ~/data.tar --strip 1)
  - sudo rm /var/lib/neo4j/data/databases/graph.db/store_lock
  - sudo chown -R neo4j:neo4j /var/lib/neo4j/data
  - sudo chown neo4j:adm /var/lib/neo4j/data/databases
  - sudo chown neo4j:adm /var/lib/neo4j/data/dbms
  - echo 'dbms.allow_format_migration=true' | sudo tee --append /etc/neo4j/neo4j.conf
  - sudo sed -i 's|dbms.security.auth_enabled=false|dbms.security.auth_enabled=true|g' /etc/neo4j/neo4j.conf
  - cat /etc/neo4j/neo4j.conf
  - sudo service neo4j start
  - sleep 30

script:
  - curl http://neo4j:"${DNB_TEST_PASS}"@localhost:7474/db/data/labels
  - cat /var/log/neo4j/neo4j.log 
  - swagger project start &
  - sleep 20
  - swagger project test

Update: Meanwhile base paths and directory names have slightly changed ... e.g.

/var/lib/neo4j/data now is /usr/local/neo4j-3.2.1/data (also file ownership now is completely travis:travis) and /etc/neo4j/neo4j.conf now also is in /usr/local/neo4j-3.2.1/conf/neo4j.conf.

The whole, up-to-date file can be found here. The basic steps however have remained the same.

Step by step

We use the following base image:

dist: trusty
sudo: required

And then declare the service utility to use:

services:
  - neo4j

The easy part is copying the data.tar file to the travis home directory:

  - cp ./config/travis/data.tar ~

Travis services start at very first of the build cycle and there is no way (we know of) to configure such a service before its initial upstart - except the above mentioned config.yaml for some other database types. Especially there is no equivalent to before_script such as e.g. before_service.

As a consequence one needs to stop the Neo4j service and sleep a little before doing the actual configuration:

  - sudo service neo4j stop
  - sleep 20

(After each start of the Neo4j service the Travis build log prints helpful details on e.g. paths, limits, and configuration. This can be used for the further steps.)

After having stopped Neo4j we can start reconfiguring it. First we use our own (self-signed) certificates so that we are in control of the CA file which is used in requests from NodeJS to Neo4J via the npm request module:

  - sudo rm /var/lib/neo4j/certificates/neo4j.key
  - sudo rm /var/lib/neo4j/certificates/neo4j.cert
  - sudo cp ./config/travis/app3_tst2_s2_db_key.pem /var/lib/neo4j/certificates/neo4j.key
  - sudo cp ./config/travis/app3_tst2_s2_db_pub.pem /var/lib/neo4j/certificates/neo4j.cert

(This probably unneccessary if you are using a Bolt-based driver for Neo4j instead of the REST API.)

Then we start with the database itself. First we remove the databases directory completely (including the current graph.db) and then, in a sub-shell, untar the db backup in the data directory. Lastly we remove the lock file:

  - sudo rm -rd /var/lib/neo4j/data/databases
  - (cd /var/lib/neo4j/data && sudo tar xvf ~/data.tar --strip 1)
  - sudo rm /var/lib/neo4j/data/databases/graph.db/store_lock

(The last step is only necessary if your backup really contains a store_lock.)

As the backup came from a Neo4j instance running on the official docker image we now need to adjust file permissions. The Travis CI service runs as neo4j whereas the Neo4j official docker image does not have a User directive in its Dockerfile. The folders databases and dbms belong to the group adm, all other files and directories beneath are neo4j:neo4j:

  - sudo chown -R neo4j:neo4j /var/lib/neo4j/data
  - sudo chown neo4j:adm /var/lib/neo4j/data/databases
  - sudo chown neo4j:adm /var/lib/neo4j/data/dbms

(Again: For something like that use the information printed to the Travis build log after a service started and do a ls -al <dir> or cat <file>... it saves a lot of time ... and guess work. ;-> )

Then we edit the Neo4j config file and allow format migrations (depends on your backup if it needs this) and then enable authentication which is disabled in the Travis CI service by default:

  - echo 'dbms.allow_format_migration=true' | sudo tee --append /etc/neo4j/neo4j.conf
  - sudo sed -i 's|dbms.security.auth_enabled=false|dbms.security.auth_enabled=true|g' /etc/neo4j/neo4j.conf

(Our restored data already contains an auth file in the dbms directory with correct credentials for the user neo4j.)

Lastly we cat the neo4j.conf file and start the service. As Neo4j takes some time to start we add some buffer and sleep for some seconds:

  - cat /etc/neo4j/neo4j.conf
  - sudo service neo4j start
  - sleep 30

Then in the script step (you might as well do it before) we check if Neo4j is available and if the correct labels are returned. Just in case we also do a cat on Neo4j's log file in /var/log/neo4j/neo4j.log

  - curl http://neo4j:"${DNB_TEST_PASS}"@localhost:7474/db/data/labels
  - cat /var/log/neo4j/neo4j.log

If everything (;->) went well (as already mentioned: it took some time ... and nerves) we can start NodeJS (here with the swagger commands), sleep again a little and then test:

  - swagger project start &
  - sleep 20
  - swagger project test

Have fun graph testing!

Further Information

Github repo:

https://github.com/daten-und-bass/fliXnet

https://github.com/daten-und-bass/fliXnet/blob/master/.travis.yml

Travis:

https://docs.travis-ci.com/user/database-setup/

Backup:

https://daten-und-bass.io/blog/backup-neo4j-community-edition-plain-linux-and-docker/