Neo4j Scaling

Prerequisites:

Insights >= v10.5
Insights supports Neo4j (4.4.4) database for this functionality.
Install neo4j streams plugin in all database servers (i.e. master & replicas)
Configure & set up Kafka server.
Only Admin can access this functionality.

Steps to set up Kafka:

Step 1: Download Kafka.

Follow below steps & update the properties:

tar -xzf kafka_2.13-3.6.0.tgz

vi kafka_2.13-3.6.0/config/server.properties

#Properties

listeners=PLAINTEXT://0.0.0.0:9092

advertised.linsteners=PLAINTEXT://[Kafka_Server_IP]:9092

log.dirs=/tmp/kafka-logs

log.retention.hours=5

zookeeper.connect=[Kafka_Server_IP]:2181

Step 2: Start the zookeeper.

bin/zookeeper-server-start.sh config/zookeeper.properties

Step 3: Start the Kafka.

bin/kafka-server-start.sh config/server.properties

Step 4: To monitor Kafka, download the Kafka-UI image from docker hub.

docker run -it -p 8080:8080 -e DYNAMIC_CONFIG_ENABLED=true --net=host provectuslabs/kafka-ui

For more info click here.

Keep the Master & Replica neo4j credentials same.

Steps to install neo4j streams plugin:

Step 1: Download the neo4j streams plugin.

Step 2: Create a streams.conf file inside $NEO4J_HOME/conf and add the below properties.

Kafka Properties

kafka.bootstrap.servers=[Kafka_Server_IP]:9092
kafka.acks=1
kafka.retries=2
kafka.batch.size=0
kafka.buffer.memory=33554432
kafka.reindex.batch.size=1000
kafka.session.timeout.ms=25000
kafka.connection.timeout.ms=20000
kafka.replication=1
kafka.linger.ms=1
kafka.transactional.id=
kafka.topic.discovery.polling.interval=300000
kafka.streams.log.compaction.strategy=delete

For Master Neo4j Server-

streams.source.topic.nodes.<TOPIC_NAME>=DATA{*}
streams.source.topic.relationships.<TOPIC_NAME>=BRANCH_HAS_COMMITS{*};BRANCH_HAS_PULL_REQUESTS{*};FILE_HAS_COMMITS{*}
streams.source.topic.relationships. insights-topic.key_strategy=all
streams.source.enabled=true
streams.sink.enabled=false

DATA{*} includes other Insights labels (i.e. JIRA, GITHUB, SONAR, JENKINS etc.)

Mention all the relationships which you want to capture.

For Replica Neo4j Server-

kafka.group.id=replica1
streams.source.enabled=false
streams.sink.enabled=true
streams.sink.topic.cdc.schema=insights-topic
streams.sink.poll.interval=300

In Neo4j - CREATE INDEX on :DATA(uuid);

For more info click here.

App Server: On which Insights App is running.
Master Server: On which master Neo4j is running.
Replica Server: New server to scale master Neo4j.

Steps to set up SSH:

App Server:

Note: User should have root level access.

# Install netcat
sudo yum install -y nc
su – [user_name]  
cd /home/[user_name]/.ssh
# IF THE USER DOESN'T HAVE PUBLIC_KEY (id_rsa.pub). CREATE IT USING BELOW keygen COMMAND ELSE IGNORE. 
ssh-keygen -t rsa
cd .ssh/
# CREATE AN App_Server_key FILE & PASTE THE PUBLIC_KEY (id_rsa.pub)
App_Server_key >> authorized_keys

App_Server_key file is critical & can be used further in below steps.

Replica Server:

# Install netcat
sudo yum install -y nc
# CREATE A REPLICA_USER
sudo useradd replica_user
# CREATE A REPLICA_GROUP
sudo groupadd replica_group       
usermod -aG replica_group replica_user
su – replica_user
cd/home/[replica_user]
# TO CREATE REPLICA_USER PUBLIC_KEY (id_rsa.pub)
ssh-keygen -t rsa
cd .ssh/
# CREATE A Replica1_Server_key FILE & PASTE PUBLIC_KEY (id_rsa.pub)
cat [App_Server_Key] >> authorized_keys  (Copy App Server key)
chmod 600 authorized_keys              
exit

Replica1_Server_key file is critical & can be used further in below steps.

# PROVIDING ONLY SPECIFIC PERMISSIONS TO SPECIFIC DIRECTORY & FILE.
sudo chgrp -R replica_group /opt/NEO4J_HOME/neo4j-Insights/data
chmod -R g+rw /opt/NEO4J_HOME/neo4j-Insights/data
sudo chgrp replica_group /opt/NEO4J_HOME/neo4j-Insights/conf/streams.conf
chmod g+rw /opt/NEO4J_HOME/neo4j-Insights/conf/streams.conf

visudo 
Cmnd_Alias CUSTOM_SERVICES = /sbin/service Neo4j stop, /sbin/service Neo4j start, /sbin/service Neo4j status
replica_user ALL=NOPASSWD:CUSTOM_SERVICES

Be cautious while doing below steps. This may lead to login issue.

Don’t forget to add master_user (user which you use to login) in AllowUsers.

sudo vi /etc/ssh/sshd_config  
AllowUsers  root   ec2-user   replica_user  <master_user>
sudo systemctl restart sshd

Master Server:

# Install netcat
sudo yum install -y nc
# CREATE A REPLICA_USER
sudo useradd replica_user
# CREATE A REPLICA_GROUP
sudo groupadd replica_group
usermod -aG replica_group replica_user
su - replica_user
cd /home/[replica_user]/.ssh
# TO CREATE REPLICA_USER PUBLIC_KEY (id_rsa.pub)
ssh-keygen -t rsa        
cd .ssh/
# CREATE A Replica1_Server_key FILE & PASTE PUBLIC_KEY (id_rsa.pub)
cat [Replica1_Server_key] >> authorized_keys  (Copy Replica Server key)
cat [App_Server_Key] >> authorized_keys  (Copy App Server key)
chmod 600 authorized_keys              
exit

# PROVIDING ONLY SPECIFIC PERMISSIONS TO SPECIFIC DIRECTORY & FILE.
sudo chgrp replica_group /opt/NEO4J_HOME/neo4j-Insights/data
chmod g+r /opt/NEO4J_HOME/neo4j-Insights/data
sudo chgrp replica_group /opt/NEO4J_HOME/neo4j-Insights/conf/streams.conf
chmod g+rw /opt/NEO4J_HOME/neo4j-Insights/conf/streams.conf

visudo
Cmnd_Alias CUSTOM_SERVICES = /sbin/service Neo4j stop, /sbin/service Neo4j start, /sbin/service Neo4j status
replica_user ALL=NOPASSWD:CUSTOM_SERVICES

Be cautious while doing below steps. This may lead to login issue.

Don’t forget to add master_user (user which you use to login) in AllowUsers.

sudo vi /etc/ssh/sshd_config  
AllowUsers   root   ec2-user   replica_user  <master_user>
sudo systemctl restart sshd

Steps to configure Neo4j Scaling:

Make sure Replica server is set up & neo4j is up & running on it.
Stop Neo4j on Master server & Insights Engine on App server.

Step 1: Navigate to Neo4j Scaling screen click on the configure button to configure source & replica details as shown below.

Step 2: After configuring source & replica details, click on the save button to save the configuration. Wait for few minutes & refresh the front screen, replica details will get displayed.

Step 3: To see the status logs, click on the details button. It will display resync status logs as shown below.

Step 4: Once you get “Replica resynced successfully” message in resync status logs.

Restart the Neo4j on Master server & Insights Engine on App server.

Step 5: To remove a replica, select that replica & click on the delete button. You can view the delete status log inside the additional details screen as shown below.

Make sure Replica server is set up & neo4j is up & running on it.
Stop Neo4j on Master server & Insights Engine on App server.

Step 6: To resync all replicas, click on the resync button. It will resync all replicas.