Ci-dessous, les différences entre deux révisions de la page.
wiki:monitoring:elk:elasticsearch [2017/03/06 11:26] theagentsmith |
wiki:monitoring:elk:elasticsearch [2020/06/27 18:16] |
||
---|---|---|---|
Ligne 1: | Ligne 1: | ||
- | ====== Cluster Elasticsearch ====== | ||
- | Extrait du site web d' | ||
- | |||
- | > Elasticsearch is a flexible and powerful open source, distributed, | ||
- | |||
- | Nous utilisons actuellement la version 5.1.1 d' | ||
- | |||
- | ====== Utilisation ====== | ||
- | |||
- | La methode native et directe d' | ||
- | |||
- | exemple : http:// | ||
- | |||
- | Pour des requetes très lourdes il est aussi possible d' | ||
- | |||
- | ====== Mise en place du cluster ====== | ||
- | |||
- | ===== Définition des noeuds ===== | ||
- | |||
- | Notre cluster sera composé de quatre (K)VM avec 5 Go de RAM, 32 Go de disque dur et 2 coeurs + 2 sockets chacune. | ||
- | |||
- | ^ Nom | Luffy | Dhalsim | ||
- | ^ hostname | ||
- | ^ IP | 192.168.102.227 | 192.168.102.228 | 192.168.102.229 | 192.168.102.231 | | ||
- | |||
- | Pour faciliter le parametrage, | ||
- | |||
- | ===== Installation d' | ||
- | |||
- | On utilise les dépôts ElasticSearch afin de simplifier les mises à jour. | ||
- | |||
- | <file bash install-sources.sh> | ||
- | export http_proxy=http:// | ||
- | wget -qO - http:// | ||
- | sudo echo "deb http:// | ||
- | </ | ||
- | |||
- | <file bash install-es.sh> | ||
- | sudo apt update | ||
- | sudo apt install openjdk-8-jre elasticsearch | ||
- | </ | ||
- | |||
- | ===== Configuration ===== | ||
- | Elle se trouve majoritairement dans le fichier ''/ | ||
- | |||
- | ==== ElasticSearch ==== | ||
- | |||
- | < | ||
- | # ======================== Elasticsearch Configuration ========================= | ||
- | # | ||
- | # NOTE: Elasticsearch comes with reasonable defaults for most settings. | ||
- | # | ||
- | # | ||
- | # | ||
- | # The primary way of configuring a node is via this file. This template lists | ||
- | # the most important settings you may want to configure for a production cluster. | ||
- | # | ||
- | # Please see the documentation for further information on configuration options: | ||
- | # < | ||
- | # | ||
- | # ---------------------------------- Cluster ----------------------------------- | ||
- | # | ||
- | # Use a descriptive name for your cluster: | ||
- | # | ||
- | cluster.name: | ||
- | # | ||
- | # ------------------------------------ Node ------------------------------------ | ||
- | # | ||
- | # Use a descriptive name for the node: | ||
- | # | ||
- | node.name: es-luffy | ||
- | # | ||
- | # Add custom attributes to the node: | ||
- | # | ||
- | # | ||
- | # | ||
- | # ----------------------------------- Paths ------------------------------------ | ||
- | # | ||
- | # Path to directory where to store the data (separate multiple locations by comma): | ||
- | # | ||
- | path.data: / | ||
- | # | ||
- | # Path to log files: | ||
- | # | ||
- | path.logs: / | ||
- | # | ||
- | # ----------------------------------- Memory ----------------------------------- | ||
- | # | ||
- | # Lock the memory on startup: | ||
- | # | ||
- | # | ||
- | # | ||
- | # Make sure that the heap size is set to about half the memory available | ||
- | # on the system and that the owner of the process is allowed to use this | ||
- | # limit. | ||
- | # | ||
- | # Elasticsearch performs poorly when the system is swapping the memory. | ||
- | # | ||
- | # ---------------------------------- Network ----------------------------------- | ||
- | # | ||
- | # Set the bind address to a specific IP (IPv4 or IPv6): | ||
- | # | ||
- | network.host: | ||
- | # | ||
- | # Set a custom port for HTTP: | ||
- | # | ||
- | http.port: 9200 | ||
- | # | ||
- | # For more information, | ||
- | # < | ||
- | # | ||
- | # --------------------------------- Discovery ---------------------------------- | ||
- | # | ||
- | # Pass an initial list of hosts to perform discovery when new node is started: | ||
- | # The default list of hosts is [" | ||
- | # | ||
- | discovery.zen.ping.unicast.hosts: | ||
- | # | ||
- | # Prevent the "split brain" by configuring the majority of nodes (total number of nodes / 2 + 1): | ||
- | # | ||
- | discovery.zen.minimum_master_nodes: | ||
- | # | ||
- | # For more information, | ||
- | # < | ||
- | # | ||
- | # ---------------------------------- Gateway ----------------------------------- | ||
- | # | ||
- | # Block initial recovery after a full cluster restart until N nodes are started: | ||
- | # | ||
- | gateway.expected_nodes: | ||
- | gateway.expected_master_nodes: | ||
- | gateway.expected_data_nodes: | ||
- | gateway.recover_after_time: | ||
- | gateway.recover_after_nodes: | ||
- | # | ||
- | # For more information, | ||
- | # < | ||
- | # | ||
- | # ---------------------------------- Various ----------------------------------- | ||
- | # | ||
- | # Require explicit names when deleting indices: | ||
- | # | ||
- | # | ||
- | </ | ||
- | |||
- | ==== JVM ==== | ||
- | |||
- | Editez le fichier ''/ | ||
- | < | ||
- | ## JVM configuration | ||
- | |||
- | ################################################################ | ||
- | ## IMPORTANT: JVM heap size | ||
- | ################################################################ | ||
- | ## | ||
- | ## You should always set the min and max JVM heap | ||
- | ## size to the same value. For example, to set | ||
- | ## the heap to 4 GB, set: | ||
- | ## | ||
- | ## -Xms4g | ||
- | ## -Xmx4g | ||
- | ## | ||
- | ## See https:// | ||
- | ## for more information | ||
- | ## | ||
- | ################################################################ | ||
- | |||
- | # Xms represents the initial size of total heap space | ||
- | # Xmx represents the maximum size of total heap space | ||
- | |||
- | -Xms3g | ||
- | -Xmx3g | ||
- | |||
- | ################################################################ | ||
- | ## Expert settings | ||
- | ################################################################ | ||
- | ## | ||
- | ## All settings below this section are considered | ||
- | ## expert settings. Don't tamper with them unless | ||
- | ## you understand what you are doing | ||
- | ## | ||
- | ################################################################ | ||
- | |||
- | ## GC configuration | ||
- | -XX: | ||
- | -XX: | ||
- | -XX: | ||
- | |||
- | ## optimizations | ||
- | |||
- | # disable calls to System#gc | ||
- | -XX: | ||
- | |||
- | # pre-touch memory pages used by the JVM during initialization | ||
- | -XX: | ||
- | |||
- | ## basic | ||
- | |||
- | # force the server VM (remove on 32-bit client JVMs) | ||
- | -server | ||
- | |||
- | # explicitly set the stack size (reduce to 320k on 32-bit client JVMs) | ||
- | -Xss1m | ||
- | |||
- | # set to headless, just in case | ||
- | -Djava.awt.headless=true | ||
- | |||
- | # ensure UTF-8 encoding by default (e.g. filenames) | ||
- | -Dfile.encoding=UTF-8 | ||
- | |||
- | # use our provided JNA always versus the system one | ||
- | -Djna.nosys=true | ||
- | |||
- | # use old-style file permissions on JDK9 | ||
- | -Djdk.io.permissionsUseCanonicalPath=true | ||
- | |||
- | # flags to keep Netty from being unsafe | ||
- | -Dio.netty.noUnsafe=true | ||
- | -Dio.netty.noKeySetOptimization=true | ||
- | |||
- | # log4j 2 | ||
- | -Dlog4j.shutdownHookEnabled=false | ||
- | -Dlog4j2.disable.jmx=true | ||
- | -Dlog4j.skipJansi=true | ||
- | |||
- | ## heap dumps | ||
- | |||
- | # generate a heap dump when an allocation from the Java heap fails | ||
- | # heap dumps are created in the working directory of the JVM | ||
- | -XX: | ||
- | |||
- | # specify an alternative path for heap dumps | ||
- | # ensure the directory exists and has sufficient space | ||
- | # | ||
- | |||
- | ## GC logging | ||
- | |||
- | # | ||
- | # | ||
- | # | ||
- | # | ||
- | # | ||
- | # | ||
- | |||
- | # log GC status to a file with time stamps | ||
- | # ensure the directory exists | ||
- | # | ||
- | |||
- | # Elasticsearch 5.0.0 will throw an exception on unquoted field names in JSON. | ||
- | # If documents were already indexed with unquoted fields in a previous version | ||
- | # of Elasticsearch, | ||
- | # | ||
- | # WARNING: This option will be removed in Elasticsearch 6.0.0 and is provided | ||
- | # only for migration purposes. | ||
- | # | ||
- | </ | ||
- | |||
- | ==== Logging ==== | ||
- | |||
- | < | ||
- | status = error | ||
- | |||
- | # log action execution errors for easier debugging | ||
- | logger.action.name = org.elasticsearch.action | ||
- | logger.action.level = debug | ||
- | |||
- | appender.console.type = Console | ||
- | appender.console.name = console | ||
- | appender.console.layout.type = PatternLayout | ||
- | appender.console.layout.pattern = [%d{ISO8601}][%-5p][%-25c{1.}] %marker%m%n | ||
- | |||
- | appender.rolling.type = RollingFile | ||
- | appender.rolling.name = rolling | ||
- | appender.rolling.fileName = ${sys: | ||
- | appender.rolling.layout.type = PatternLayout | ||
- | appender.rolling.layout.pattern = [%d{ISO8601}][%-5p][%-25c{1.}] %marker%.-10000m%n | ||
- | appender.rolling.filePattern = ${sys: | ||
- | appender.rolling.policies.type = Policies | ||
- | appender.rolling.policies.time.type = TimeBasedTriggeringPolicy | ||
- | appender.rolling.policies.time.interval = 1 | ||
- | appender.rolling.policies.time.modulate = true | ||
- | |||
- | rootLogger.level = info | ||
- | rootLogger.appenderRef.console.ref = console | ||
- | rootLogger.appenderRef.rolling.ref = rolling | ||
- | |||
- | appender.deprecation_rolling.type = RollingFile | ||
- | appender.deprecation_rolling.name = deprecation_rolling | ||
- | appender.deprecation_rolling.fileName = ${sys: | ||
- | appender.deprecation_rolling.layout.type = PatternLayout | ||
- | appender.deprecation_rolling.layout.pattern = [%d{ISO8601}][%-5p][%-25c{1.}] %marker%.-10000m%n | ||
- | appender.deprecation_rolling.filePattern = ${sys: | ||
- | appender.deprecation_rolling.policies.type = Policies | ||
- | appender.deprecation_rolling.policies.size.type = SizeBasedTriggeringPolicy | ||
- | appender.deprecation_rolling.policies.size.size = 1GB | ||
- | appender.deprecation_rolling.strategy.type = DefaultRolloverStrategy | ||
- | appender.deprecation_rolling.strategy.max = 4 | ||
- | |||
- | logger.deprecation.name = org.elasticsearch.deprecation | ||
- | logger.deprecation.level = warn | ||
- | logger.deprecation.appenderRef.deprecation_rolling.ref = deprecation_rolling | ||
- | logger.deprecation.additivity = false | ||
- | |||
- | appender.index_search_slowlog_rolling.type = RollingFile | ||
- | appender.index_search_slowlog_rolling.name = index_search_slowlog_rolling | ||
- | appender.index_search_slowlog_rolling.fileName = ${sys: | ||
- | appender.index_search_slowlog_rolling.layout.type = PatternLayout | ||
- | appender.index_search_slowlog_rolling.layout.pattern = [%d{ISO8601}][%-5p][%-25c] %marker%.-10000m%n | ||
- | appender.index_search_slowlog_rolling.filePattern = ${sys: | ||
- | appender.index_search_slowlog_rolling.policies.type = Policies | ||
- | appender.index_search_slowlog_rolling.policies.time.type = TimeBasedTriggeringPolicy | ||
- | appender.index_search_slowlog_rolling.policies.time.interval = 1 | ||
- | appender.index_search_slowlog_rolling.policies.time.modulate = true | ||
- | |||
- | logger.index_search_slowlog_rolling.name = index.search.slowlog | ||
- | logger.index_search_slowlog_rolling.level = trace | ||
- | logger.index_search_slowlog_rolling.appenderRef.index_search_slowlog_rolling.ref = index_search_slowlog_rolling | ||
- | logger.index_search_slowlog_rolling.additivity = false | ||
- | |||
- | appender.index_indexing_slowlog_rolling.type = RollingFile | ||
- | appender.index_indexing_slowlog_rolling.name = index_indexing_slowlog_rolling | ||
- | appender.index_indexing_slowlog_rolling.fileName = ${sys: | ||
- | appender.index_indexing_slowlog_rolling.layout.type = PatternLayout | ||
- | appender.index_indexing_slowlog_rolling.layout.pattern = [%d{ISO8601}][%-5p][%-25c] %marker%.-10000m%n | ||
- | appender.index_indexing_slowlog_rolling.filePattern = ${sys: | ||
- | appender.index_indexing_slowlog_rolling.policies.type = Policies | ||
- | appender.index_indexing_slowlog_rolling.policies.time.type = TimeBasedTriggeringPolicy | ||
- | appender.index_indexing_slowlog_rolling.policies.time.interval = 1 | ||
- | appender.index_indexing_slowlog_rolling.policies.time.modulate = true | ||
- | |||
- | logger.index_indexing_slowlog.name = index.indexing.slowlog.index | ||
- | logger.index_indexing_slowlog.level = trace | ||
- | logger.index_indexing_slowlog.appenderRef.index_indexing_slowlog_rolling.ref = index_indexing_slowlog_rolling | ||
- | logger.index_indexing_slowlog.additivity = false | ||
- | </ | ||
- | |||
- | ===== Configuration de l'OS ===== | ||
- | |||
- | Elasticsearch est bien plus performant lorsqu' | ||
- | |||
- | On crée le fichier ''/ | ||
- | < | ||
- | elasticsearch soft nofile 65535 | ||
- | elasticsearch hard nofile 65535 | ||
- | elasticsearch - memlock unlimited | ||
- | </ | ||
- | |||
- | Il est à noter qu' | ||
- | |||
- | ===== Entretien du cluster avec Curator ===== | ||
- | |||
- | Elasticsearch est assez bête dans sa gestion des données: il part du principe qu'il aura suffisamment d' | ||
- | |||
- | Pour retirer un index "à la main", par exemple si on a fait une bourde de configuration : API REST : https:// | ||
- | |||
- | Sinon, on utilise curator, depuis sa dernière version il utilise des fichiers de conf : | ||
- | |||
- | close30.yml | ||
- | < | ||
- | --- | ||
- | actions: | ||
- | 1: | ||
- | action: close | ||
- | description: | ||
- | Close indices older than 30 days (based on index name), for logstash- | ||
- | prefixed indices. | ||
- | options: | ||
- | delete_aliases: | ||
- | timeout_override: | ||
- | continue_if_exception: | ||
- | disable_action: | ||
- | filters: | ||
- | - filtertype: pattern | ||
- | kind: prefix | ||
- | value: logstash- | ||
- | exclude: | ||
- | - filtertype: age | ||
- | source: name | ||
- | direction: older | ||
- | timestring: ' | ||
- | unit: days | ||
- | unit_count: 30 | ||
- | exclude: | ||
- | </ | ||
- | |||
- | delete60.yml | ||
- | < | ||
- | --- | ||
- | actions: | ||
- | 1: | ||
- | action: delete_indices | ||
- | description: | ||
- | Delete indices older than 60 days (based on index name), for logstash- | ||
- | prefixed indices. Ignore the error if the filter does not result in an | ||
- | actionable list of indices (ignore_empty_list) and exit cleanly. | ||
- | options: | ||
- | ignore_empty_list: | ||
- | timeout_override: | ||
- | continue_if_exception: | ||
- | disable_action: | ||
- | filters: | ||
- | - filtertype: pattern | ||
- | kind: prefix | ||
- | value: logstash- | ||
- | exclude: | ||
- | - filtertype: age | ||
- | source: name | ||
- | direction: older | ||
- | timestring: ' | ||
- | unit: days | ||
- | unit_count: 60 | ||
- | exclude: | ||
- | </ | ||
- | |||
- | open60.yml | ||
- | < | ||
- | --- | ||
- | actions: | ||
- | 1: | ||
- | action: open | ||
- | description: | ||
- | Open indices older than 30 days but younger than 60 days (based on index | ||
- | name), for logstash- prefixed indices. | ||
- | options: | ||
- | timeout_override: | ||
- | continue_if_exception: | ||
- | disable_action: | ||
- | filters: | ||
- | - filtertype: pattern | ||
- | kind: prefix | ||
- | value: logstash- | ||
- | exclude: | ||
- | - filtertype: age | ||
- | source: name | ||
- | direction: older | ||
- | timestring: ' | ||
- | unit: days | ||
- | unit_count: 30 | ||
- | exclude: | ||
- | - filtertype: age | ||
- | source: name | ||
- | direction: younger | ||
- | timestring: ' | ||
- | unit: days | ||
- | unit_count: 60 | ||
- | exclude: | ||
- | </ | ||
- | |||
- | On lance les scripts Close30 et Delete60 dans cron.daily/ | ||
- | |||
- | < | ||
- | curator --config / | ||
- | curator --config / | ||
- | </ | ||
- | |||
- | On lancera à la main Open60 : | ||
- | |||
- | < | ||
- | curator --config / | ||
- | </ | ||
- | |||
- | ====== Monitoring ====== | ||
- | |||
- | En dehors du monitoring traditionnel du service **elasticsearch** (vérifier que le processus tourne | ||
- | et qu'il écoute sur le bon port), nous avons utilisé des **UserParameters** dans la configuration de | ||
- | l' | ||
- | |||
- | < | ||
- | |||
- | UserParameter=nb_node, | ||
- | UserParameter=nb_data_node, | ||
- | UserParameter=active_shard, | ||
- | UserParameter=active_primary_shard, | ||
- | UserParameter=unasigned_shards, | ||
- | UserParameter=relocating_shards, | ||
- | UserParameter=initializing_shards, | ||
- | UserParameter=delayed_unassigned_shards, | ||
- | UserParameter=number_of_pending_tasks, | ||
- | UserParameter=task_max_waiting_in_queue_millis, | ||
- | UserParameter=document_count, | ||
- | UserParameter=heap_use_percent, | ||
- | UserParameter=file_desc_percent, | ||
- | UserParameter=ram_percent, | ||
- | UserParameter=index_total, | ||
- | UserParameter=indexing.delete_total, | ||
- | UserParameter=search.query_total, | ||
- | |||
- | </ | ||
- | |||
- | Cela nous permet également d' | ||
- | sur notre cluster. |