Network Mapping

net2httpnmap

Its main function is get Netbox Aggregates and send to hi-worker.

net2httpnmap process all aggregates and deposit the results in a defined topic kafka inside Django-admin/Kafka/kafka_topic_nmap.

Send aggregates to nmap-api and process the results

$ ./manage.py net2httpnmap -t aggregates -m nmap \
    --nmap-name hi-worker

Explanation:

./manage.py = Script of administration Django.

net2httpnmap = Module nmap.

-t aggregates = reference to the Aggregates in netbox.

-m = type of mapping in nmap, recommended nmap.

--nmap-name = Nmap name defined in the console of administration of Django, in "Hostinventory - Nmaps". Indicates the api to use.

Kafka Consumers

Its main function is get the messages of the Kafka queue, and insert/update the messages in netbox.

pipeline-ip

Get aggregates of nmap Kafka topic and:

  • insert/update in netbox.
  • consume the api nmap with the ip and determine if the ip is a windows, linux or network device.
  • Put the results in the different kafka queue
$ ./manage.py pipeline-ip  --kafka-name kafka-conn \
    --nmap-name hi-worker --netbox-name netbox --async true

Explanation:

./manage.py = Script of administration of Django.

pipeline-ip = Module kafka.

--kafka-name = Kafka name defined in the console of administration of Django, in "Hostinventory - Nmaps". Indicates the connection to Kafka server.

--nmap-name = Nmap name defined in the console of administration of Django, in "Hostinventory - Nmaps". Indicates the api to use.

--netbox-name = Name of netbox defined in the console of administration of Django, in "Netbox - Netbox conns". Indicates the connection to Netbox.

consumer

Get the ips sent from the pipeline module of the different kafka queues and send the ip's to the APIs. Then put the data in a new kakfa queue.

$ ./manage.py consumer  --kafka-name kafka-conn \
    --nmap-name hi-worker --topic-name goLe-ssh

Explanation:

./manage.py = Script of administration of Django.

consumer = Module module that interacts with the API's.

--kafka-name = Kafka name defined in the console of administration of Django, in "Hostinventory - Nmaps". Indicates the connection to Kafka server.

--nmap-name = Nmap name defined in the console of administration of Django, in "Hostinventory - Nmaps". Indicates the api to use.

--topic-name = The kafka queue that will listen to the data sent from the pipeline-ip module, there are 3 queues:

  • goLe-ssh #devices that respond to port 22
  • goLe-win #devices that respond to port 135
  • goLe-telnet #devices that respond to port 23

merge_data

This module receives the data sent by the module consumer to be merged with the netbox data. Finally the data is sent to another kakfa queue.

./manage.py merge_data --kafka-name kafka-conn --netbox-name netbox

Explanation:

./manage.py = Script of administration of Django.

merge_data = Module that merged the data with netbox.

--kafka-name = Kafka name defined in the console of administration of Django, in "Hostinventory - Nmaps". Indicates the connection to Kafka server.

--netbox-name = Name of netbox defined in the console of administration of Django, in "Netbox - Netbox conns". Indicates the connection to Netbox.

Netbox, goLe applications inside Redis keys

You can sync devices and VMs information from Netbox to redis database. It's is used to match application and VMS/devices with goLe applications.

goLe-hi-manager connect application with netbox VMs/devices by the custom field app_code.

Update Netbox devices on Redis:

Redis keys: netbox_devices and netbox_app_code

curl http://gole-hi-manager-url/service/update-redis/devices/

Upate Netbox virtual machines on Redis

Redis keys: netbox_vms and netbox_app_code

curl http://gole-hi-manager-url/service/update-redis/vms/

Update all by the manage command:

./manage.py netbox2redis
System check identified some issues:

Create redis keys - first time