...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
Prerequisites:
Insights >= v10.5
Insights supports Neo4j (4.4.4) database for this functionality.
Install neo4j streams plugin in all database servers (i.e. master & replicas)
Configure & set upKafka server.
Only Admin can access this functionality.
...
| Expand |
|---|
| title | Follow below steps & update the properties: |
|---|
|
tar -xzf kafka_2.13-3.6.0.tgz vi kafka_2.13-3.6.0/config/server.properties #Properties listeners=PLAINTEXT://0.0.0.0:9092 advertised.linsteners=PLAINTEXT://[Kafka_Server_IP]:9092 log.dirs=/tmp/kafka-logs log.retention.hours=5 zookeeper.connect=[Kafka_Server_IP]:2181 |
Step 2: Start the zookeeper.
...
Step 1: Download the neo4j streams plugin.
Step 2: Create a streams.conf file inside $NEO4J_HOME/conf and add the below properties.
| Expand |
|---|
|
| Code Block |
|---|
| kafka.bootstrap.servers=[Kafka_Server_IP]:9092
kafka.acks=1
kafka.retries=2
kafka.batch.size=0
kafka.buffer.memory=33554432
kafka.reindex.batch.size=1000
kafka.session.timeout.ms=25000
kafka.connection.timeout.ms=20000
kafka.replication=1
kafka.linger.ms=1
kafka.transactional.id=
kafka.topic.discovery.polling.interval=300000
kafka.streams.log.compaction.strategy=delete |
For Master Neo4j Server-
| Code Block |
|---|
streams.source.topic.nodes.<TOPIC_NAME>=DATA{*}
streams.source.topic.relationships.<TOPIC_NAME>=BRANCH_HAS_COMMITS{*};BRANCH_HAS_PULL_REQUESTS{*};FILE_HAS_COMMITS{*}
streams.source.topic.relationships. insights-topic.key_strategy=all
streams.source.enabled=true
streams.sink.enabled=false |
| Info |
|---|
DATA{*} includes other Insights labels (i.e. JIRA, GITHUB, SONAR, JENKINS etc.) |
| Info |
|---|
Mention all the relationships which you want to capture. |
For Replica Neo4j Server-
| Code Block |
|---|
| kafka.group.id=replica1
streams.source.enabled=false
streams.sink.enabled=true
streams.sink.topic.cdc.schema=insights-topic
streams.sink.poll.interval=300 |
|
...
| Expand |
|---|
|
| Panel |
|---|
| panelIconId | atlassian-note |
|---|
| panelIcon | :note: |
|---|
| bgColor | #FFEBE6 |
|---|
| Note: User should have root level access. |
| Code Block |
|---|
| # Install netcat
sudo yum install -y nc
su – [user_name]
cd /home/[user_name]/.ssh
# IF THE USER DOESN'T HAVE PUBLIC_KEY (id_rsa.pub). CREATE IT USING BELOW keygen COMMAND ELSE IGNORE.
ssh-keygen -t rsa
cd .ssh/
# CREATE AN App_Server_key FILE & PASTE THE PUBLIC_KEY (id_rsa.pub)
App_Server_key >> authorized_keys |
| Info |
|---|
App_Server_key file is critical & can be used further in below steps. |
|
| Expand |
|---|
|
| Code Block |
|---|
| # Install netcat
sudo yum install -y nc
# CREATE A REPLICA_USER
sudo useradd replica_user
# CREATE A REPLICA_GROUP
sudo groupadd replica_group
usermod -aG replica_group replica_user
su – replica_user
cd/home/[replica_user]
# TO CREATE REPLICA_USER PUBLIC_KEY (id_rsa.pub)
ssh-keygen -t rsa
cd .ssh/
# CREATE A Replica1_Server_key FILE & PASTE PUBLIC_KEY (id_rsa.pub)
cat [App_Server_Key] >> authorized_keys (Copy App Server key)
chmod 600 authorized_keys
exit |
| Info |
|---|
Replica1_Server_key file is critical & can be used further in below steps. |
| Code Block |
|---|
# PROVIDING ONLY SPECIFIC PERMISSIONS TO SPECIFIC DIRECTORY & FILE.
sudo chgrp -R replica_group /opt/NEO4J_HOME/neo4j-Insights/data
chmod -R g+rw /opt/NEO4J_HOME/neo4j-Insights/data
sudo chgrp replica_group /opt/NEO4J_HOME/neo4j-Insights/conf/streams.conf
chmod g+rw /opt/NEO4J_HOME/neo4j-Insights/conf/streams.conf
visudo
Cmnd_Alias CUSTOM_SERVICES = /sbin/service Neo4j stop, /sbin/service Neo4j start, /sbin/service Neo4j status
replica_user ALL=NOPASSWD:CUSTOM_SERVICES |
| Panel |
|---|
| panelIconId | atlassian-note |
|---|
| panelIcon | :note: |
|---|
| bgColor | #FFBDAD |
|---|
| Be cautious while doing below steps. This may lead to login issue. Don’t forget to add master_user (user which you use to login) in AllowUsers. |
| Code Block |
|---|
| sudo vi /etc/ssh/sshd_config
AllowUsers root ec2-user replica_user <master_user>
sudo systemctl restart sshd |
|
...
Steps to configure Neo4j Scaling:
Step 1: Navigate to Neo4j Scaling screen click on theconfigurebuttonto configure source & replica details as shown below.
...
Step 3: To see the status logs, click on the details button. It will display resync status logs as shown below.
...
Step 4: Once you get “Replica resynced successfully” message in resync status logs.
| Tip |
|---|
Restart the Neo4j on Master server & Insights Engine on App server. |
Step 5: To remove a replica, select that replica & click on the delete button. You can view the delete status log inside the additional details screen as shown below.
...