Elasticsearch

The Role of Elasticsearch in DeepFinder

Elasticsearch is a NoSQL database introduced to leverage faster data search and aggregation performance than PostgreSQL. In DeepFinder, it is primarily used for the following purposes:
* Detection Log Storage: A storage for detection logs to enable fast searching.
* Dashboard and Reports: Data aggregation for dashboard visualizations and report generation.

Causes and Impact of Elasticsearch Process Stoppage

The Elasticsearch process can stop mainly for the two reasons below:
* Exceeding disk usage threshold: When disk usage is 80% or higher.
* Insufficient memory: Lack of available JVM memory due to excessive index accumulation.

Therefore, it is recommended to store only the most frequently searched data, from around the last 3 months, in Elasticsearch and to periodically delete older data.

Info

Periodic log cleanup (deletion) can be configured in the [Environment Management] > [Environment Settings] > [Log Retention Period] tab.
Old logs will be automatically deleted according to the set 'Deletion Date' and 'Deletion Time'.

Since Elasticsearch is used for supplementary functions like fast search and statistics, even if the process stops, it does not affect core security functions such as real-time threat detection and blocking.
All of DeepFinder's detection logs are stored simultaneously in both Elasticsearch and the PostgreSQL DB. In case of an Elasticsearch failure, you can view the logs stored in PostgreSQL by changing the search target at the top of the [Detection Logs] screen from 'Elastic' to 'DB'.

Elasticsearch Recovery Procedure

1) Restart Elasticsearch
There are cases where the Manager UI shows Elasticsearch as down, but the process is actually running on the server. This could be a 'Hanging' state where Elasticsearch is unresponsive. Therefore, restart the service using the command below.

[root@i ~]# service dfserver-elastic restart
useradd: user 'elastic' already exists
Stopping elasticsearch:                                        [  OK  ]
Starting elasticsearch:                                        [  OK  ]

[root@i ~]# ps -ef |grep elastic
elastic   899      1 99 10:57 ?        00:00:16 /usr/bin/java -Xms3g -Xmx3g -XX:+UseConcMarkSweepGC ...
elastic   896    899  0 10:57 ?        00:00:00 /usr/local/dfserver/elasticsearch/modules/x-pack-ml ...

2) Check the Status
Immediately after restarting the service, a Connection refused error may occur. Wait a moment for the process to fully start, then check the status with the command below.

[root@i ~]# curl localhost:9200/_cat/health
curl: (7) Failed to connect to localhost port 9200: Connection refused

[root@i ~]# curl localhost:9200/_cat/health
1684375627 02:07:07 elasticsearch red 1 1 575 575 0 4 1821 5 144.3ms 24.0%

3) View and Delete Indices
If the status is shown as red or yellow, it may be due to an accumulation of indices. Check the list and size of indices with the command below.
If excessively large indices are found, use the DELETE method to delete old indices.
(Note: To delete all indices, use the command curl -XDELETE localhost:9200/*.)

[root@i ~]# curl localhost:9200/_cat/indices?v
health status index                   uuid                     pri rep docs.count docs.deleted store.size pri.store.size
yellow open   pslist-2023-04-08       Fwl4Db5YTV-vZCKZGwisvw   5   1       4790            0       12.1mb         12.1mb
yellow open   pfmdata-2023-04-25      Uui9AviiSyeCg8AhxBpxew   5   1        778            0      579.6kb        579.6kb
...

[root@i ~]# curl -XDELETE localhost:9200/pslist-2023-04* {"acknowledged":true}

4) Check Disk Space
If disk usage is still over 80% after deleting indices, it may be necessary to clean up other data, such as PostgreSQL DB logs.
In this case, please contact the DeepFinder technical support team for assistance.