Ci-dessous, les différences entre deux révisions de la page.
Les deux révisions précédentes Révision précédente Prochaine révision | Révision précédente | ||
wiki:monitoring:elk:elasticsearch [2017/03/06 11:25] theagentsmith |
wiki:monitoring:elk:elasticsearch [2017/07/10 15:23] varens [Monitoring] |
||
---|---|---|---|
Ligne 1: | Ligne 1: | ||
+ | ====== Cluster Elasticsearch ====== | ||
+ | Extrait du site web d' | ||
+ | |||
+ | > Elasticsearch is a flexible and powerful open source, distributed, | ||
+ | |||
+ | Nous utilisons actuellement la version 5.1.1 d' | ||
+ | |||
+ | ====== Utilisation ====== | ||
+ | |||
+ | La methode native et directe d' | ||
+ | |||
+ | exemple : http:// | ||
+ | |||
+ | Pour des requetes très lourdes il est aussi possible d' | ||
+ | |||
+ | ====== Mise en place du cluster ====== | ||
+ | |||
+ | ===== Définition des noeuds ===== | ||
+ | |||
+ | Notre cluster sera composé de quatre (K)VM avec 5 Go de RAM, 32 Go de disque dur et 2 coeurs + 2 sockets chacune. | ||
+ | |||
+ | ^ Nom | Luffy | Dhalsim | ||
+ | ^ hostname | ||
+ | ^ IP | 192.168.102.227 | 192.168.102.228 | 192.168.102.229 | 192.168.102.231 | | ||
+ | |||
+ | Pour faciliter le parametrage, | ||
+ | |||
+ | ===== Installation d' | ||
+ | |||
+ | On utilise les dépôts ElasticSearch afin de simplifier les mises à jour. | ||
+ | |||
+ | <file bash install-sources.sh> | ||
+ | export http_proxy=http:// | ||
+ | wget -qO - http:// | ||
+ | sudo echo "deb http:// | ||
+ | </ | ||
+ | |||
+ | <file bash install-es.sh> | ||
+ | sudo apt update | ||
+ | sudo apt install openjdk-8-jre elasticsearch | ||
+ | </ | ||
+ | |||
+ | ===== Configuration ===== | ||
+ | Elle se trouve majoritairement dans le fichier ''/ | ||
+ | |||
+ | ==== ElasticSearch ==== | ||
+ | |||
+ | < | ||
+ | # ======================== Elasticsearch Configuration ========================= | ||
+ | # | ||
+ | # NOTE: Elasticsearch comes with reasonable defaults for most settings. | ||
+ | # | ||
+ | # | ||
+ | # | ||
+ | # The primary way of configuring a node is via this file. This template lists | ||
+ | # the most important settings you may want to configure for a production cluster. | ||
+ | # | ||
+ | # Please see the documentation for further information on configuration options: | ||
+ | # < | ||
+ | # | ||
+ | # ---------------------------------- Cluster ----------------------------------- | ||
+ | # | ||
+ | # Use a descriptive name for your cluster: | ||
+ | # | ||
+ | cluster.name: | ||
+ | # | ||
+ | # ------------------------------------ Node ------------------------------------ | ||
+ | # | ||
+ | # Use a descriptive name for the node: | ||
+ | # | ||
+ | node.name: es-luffy | ||
+ | # | ||
+ | # Add custom attributes to the node: | ||
+ | # | ||
+ | # | ||
+ | # | ||
+ | # ----------------------------------- Paths ------------------------------------ | ||
+ | # | ||
+ | # Path to directory where to store the data (separate multiple locations by comma): | ||
+ | # | ||
+ | path.data: / | ||
+ | # | ||
+ | # Path to log files: | ||
+ | # | ||
+ | path.logs: / | ||
+ | # | ||
+ | # ----------------------------------- Memory ----------------------------------- | ||
+ | # | ||
+ | # Lock the memory on startup: | ||
+ | # | ||
+ | # | ||
+ | # | ||
+ | # Make sure that the heap size is set to about half the memory available | ||
+ | # on the system and that the owner of the process is allowed to use this | ||
+ | # limit. | ||
+ | # | ||
+ | # Elasticsearch performs poorly when the system is swapping the memory. | ||
+ | # | ||
+ | # ---------------------------------- Network ----------------------------------- | ||
+ | # | ||
+ | # Set the bind address to a specific IP (IPv4 or IPv6): | ||
+ | # | ||
+ | network.host: | ||
+ | # | ||
+ | # Set a custom port for HTTP: | ||
+ | # | ||
+ | http.port: 9200 | ||
+ | # | ||
+ | # For more information, | ||
+ | # < | ||
+ | # | ||
+ | # --------------------------------- Discovery ---------------------------------- | ||
+ | # | ||
+ | # Pass an initial list of hosts to perform discovery when new node is started: | ||
+ | # The default list of hosts is [" | ||
+ | # | ||
+ | discovery.zen.ping.unicast.hosts: | ||
+ | # | ||
+ | # Prevent the "split brain" by configuring the majority of nodes (total number of nodes / 2 + 1): | ||
+ | # | ||
+ | discovery.zen.minimum_master_nodes: | ||
+ | # | ||
+ | # For more information, | ||
+ | # < | ||
+ | # | ||
+ | # ---------------------------------- Gateway ----------------------------------- | ||
+ | # | ||
+ | # Block initial recovery after a full cluster restart until N nodes are started: | ||
+ | # | ||
+ | gateway.expected_nodes: | ||
+ | gateway.expected_master_nodes: | ||
+ | gateway.expected_data_nodes: | ||
+ | gateway.recover_after_time: | ||
+ | gateway.recover_after_nodes: | ||
+ | # | ||
+ | # For more information, | ||
+ | # < | ||
+ | # | ||
+ | # ---------------------------------- Various ----------------------------------- | ||
+ | # | ||
+ | # Require explicit names when deleting indices: | ||
+ | # | ||
+ | # | ||
+ | </ | ||
+ | |||
+ | ==== JVM ==== | ||
+ | |||
+ | Editez le fichier ''/ | ||
+ | < | ||
+ | ## JVM configuration | ||
+ | |||
+ | ################################################################ | ||
+ | ## IMPORTANT: JVM heap size | ||
+ | ################################################################ | ||
+ | ## | ||
+ | ## You should always set the min and max JVM heap | ||
+ | ## size to the same value. For example, to set | ||
+ | ## the heap to 4 GB, set: | ||
+ | ## | ||
+ | ## -Xms4g | ||
+ | ## -Xmx4g | ||
+ | ## | ||
+ | ## See https:// | ||
+ | ## for more information | ||
+ | ## | ||
+ | ################################################################ | ||
+ | |||
+ | # Xms represents the initial size of total heap space | ||
+ | # Xmx represents the maximum size of total heap space | ||
+ | |||
+ | -Xms3g | ||
+ | -Xmx3g | ||
+ | |||
+ | ################################################################ | ||
+ | ## Expert settings | ||
+ | ################################################################ | ||
+ | ## | ||
+ | ## All settings below this section are considered | ||
+ | ## expert settings. Don't tamper with them unless | ||
+ | ## you understand what you are doing | ||
+ | ## | ||
+ | ################################################################ | ||
+ | |||
+ | ## GC configuration | ||
+ | -XX: | ||
+ | -XX: | ||
+ | -XX: | ||
+ | |||
+ | ## optimizations | ||
+ | |||
+ | # disable calls to System#gc | ||
+ | -XX: | ||
+ | |||
+ | # pre-touch memory pages used by the JVM during initialization | ||
+ | -XX: | ||
+ | |||
+ | ## basic | ||
+ | |||
+ | # force the server VM (remove on 32-bit client JVMs) | ||
+ | -server | ||
+ | |||
+ | # explicitly set the stack size (reduce to 320k on 32-bit client JVMs) | ||
+ | -Xss1m | ||
+ | |||
+ | # set to headless, just in case | ||
+ | -Djava.awt.headless=true | ||
+ | |||
+ | # ensure UTF-8 encoding by default (e.g. filenames) | ||
+ | -Dfile.encoding=UTF-8 | ||
+ | |||
+ | # use our provided JNA always versus the system one | ||
+ | -Djna.nosys=true | ||
+ | |||
+ | # use old-style file permissions on JDK9 | ||
+ | -Djdk.io.permissionsUseCanonicalPath=true | ||
+ | |||
+ | # flags to keep Netty from being unsafe | ||
+ | -Dio.netty.noUnsafe=true | ||
+ | -Dio.netty.noKeySetOptimization=true | ||
+ | |||
+ | # log4j 2 | ||
+ | -Dlog4j.shutdownHookEnabled=false | ||
+ | -Dlog4j2.disable.jmx=true | ||
+ | -Dlog4j.skipJansi=true | ||
+ | |||
+ | ## heap dumps | ||
+ | |||
+ | # generate a heap dump when an allocation from the Java heap fails | ||
+ | # heap dumps are created in the working directory of the JVM | ||
+ | -XX: | ||
+ | |||
+ | # specify an alternative path for heap dumps | ||
+ | # ensure the directory exists and has sufficient space | ||
+ | # | ||
+ | |||
+ | ## GC logging | ||
+ | |||
+ | # | ||
+ | # | ||
+ | # | ||
+ | # | ||
+ | # | ||
+ | # | ||
+ | |||
+ | # log GC status to a file with time stamps | ||
+ | # ensure the directory exists | ||
+ | # | ||
+ | |||
+ | # Elasticsearch 5.0.0 will throw an exception on unquoted field names in JSON. | ||
+ | # If documents were already indexed with unquoted fields in a previous version | ||
+ | # of Elasticsearch, | ||
+ | # | ||
+ | # WARNING: This option will be removed in Elasticsearch 6.0.0 and is provided | ||
+ | # only for migration purposes. | ||
+ | # | ||
+ | </ | ||
+ | |||
+ | ==== Logging ==== | ||
+ | |||
+ | < | ||
+ | status = error | ||
+ | |||
+ | # log action execution errors for easier debugging | ||
+ | logger.action.name = org.elasticsearch.action | ||
+ | logger.action.level = debug | ||
+ | |||
+ | appender.console.type = Console | ||
+ | appender.console.name = console | ||
+ | appender.console.layout.type = PatternLayout | ||
+ | appender.console.layout.pattern = [%d{ISO8601}][%-5p][%-25c{1.}] %marker%m%n | ||
+ | |||
+ | appender.rolling.type = RollingFile | ||
+ | appender.rolling.name = rolling | ||
+ | appender.rolling.fileName = ${sys: | ||
+ | appender.rolling.layout.type = PatternLayout | ||
+ | appender.rolling.layout.pattern = [%d{ISO8601}][%-5p][%-25c{1.}] %marker%.-10000m%n | ||
+ | appender.rolling.filePattern = ${sys: | ||
+ | appender.rolling.policies.type = Policies | ||
+ | appender.rolling.policies.time.type = TimeBasedTriggeringPolicy | ||
+ | appender.rolling.policies.time.interval = 1 | ||
+ | appender.rolling.policies.time.modulate = true | ||
+ | |||
+ | rootLogger.level = info | ||
+ | rootLogger.appenderRef.console.ref = console | ||
+ | rootLogger.appenderRef.rolling.ref = rolling | ||
+ | |||
+ | appender.deprecation_rolling.type = RollingFile | ||
+ | appender.deprecation_rolling.name = deprecation_rolling | ||
+ | appender.deprecation_rolling.fileName = ${sys: | ||
+ | appender.deprecation_rolling.layout.type = PatternLayout | ||
+ | appender.deprecation_rolling.layout.pattern = [%d{ISO8601}][%-5p][%-25c{1.}] %marker%.-10000m%n | ||
+ | appender.deprecation_rolling.filePattern = ${sys: | ||
+ | appender.deprecation_rolling.policies.type = Policies | ||
+ | appender.deprecation_rolling.policies.size.type = SizeBasedTriggeringPolicy | ||
+ | appender.deprecation_rolling.policies.size.size = 1GB | ||
+ | appender.deprecation_rolling.strategy.type = DefaultRolloverStrategy | ||
+ | appender.deprecation_rolling.strategy.max = 4 | ||
+ | |||
+ | logger.deprecation.name = org.elasticsearch.deprecation | ||
+ | logger.deprecation.level = warn | ||
+ | logger.deprecation.appenderRef.deprecation_rolling.ref = deprecation_rolling | ||
+ | logger.deprecation.additivity = false | ||
+ | |||
+ | appender.index_search_slowlog_rolling.type = RollingFile | ||
+ | appender.index_search_slowlog_rolling.name = index_search_slowlog_rolling | ||
+ | appender.index_search_slowlog_rolling.fileName = ${sys: | ||
+ | appender.index_search_slowlog_rolling.layout.type = PatternLayout | ||
+ | appender.index_search_slowlog_rolling.layout.pattern = [%d{ISO8601}][%-5p][%-25c] %marker%.-10000m%n | ||
+ | appender.index_search_slowlog_rolling.filePattern = ${sys: | ||
+ | appender.index_search_slowlog_rolling.policies.type = Policies | ||
+ | appender.index_search_slowlog_rolling.policies.time.type = TimeBasedTriggeringPolicy | ||
+ | appender.index_search_slowlog_rolling.policies.time.interval = 1 | ||
+ | appender.index_search_slowlog_rolling.policies.time.modulate = true | ||
+ | |||
+ | logger.index_search_slowlog_rolling.name = index.search.slowlog | ||
+ | logger.index_search_slowlog_rolling.level = trace | ||
+ | logger.index_search_slowlog_rolling.appenderRef.index_search_slowlog_rolling.ref = index_search_slowlog_rolling | ||
+ | logger.index_search_slowlog_rolling.additivity = false | ||
+ | |||
+ | appender.index_indexing_slowlog_rolling.type = RollingFile | ||
+ | appender.index_indexing_slowlog_rolling.name = index_indexing_slowlog_rolling | ||
+ | appender.index_indexing_slowlog_rolling.fileName = ${sys: | ||
+ | appender.index_indexing_slowlog_rolling.layout.type = PatternLayout | ||
+ | appender.index_indexing_slowlog_rolling.layout.pattern = [%d{ISO8601}][%-5p][%-25c] %marker%.-10000m%n | ||
+ | appender.index_indexing_slowlog_rolling.filePattern = ${sys: | ||
+ | appender.index_indexing_slowlog_rolling.policies.type = Policies | ||
+ | appender.index_indexing_slowlog_rolling.policies.time.type = TimeBasedTriggeringPolicy | ||
+ | appender.index_indexing_slowlog_rolling.policies.time.interval = 1 | ||
+ | appender.index_indexing_slowlog_rolling.policies.time.modulate = true | ||
+ | |||
+ | logger.index_indexing_slowlog.name = index.indexing.slowlog.index | ||
+ | logger.index_indexing_slowlog.level = trace | ||
+ | logger.index_indexing_slowlog.appenderRef.index_indexing_slowlog_rolling.ref = index_indexing_slowlog_rolling | ||
+ | logger.index_indexing_slowlog.additivity = false | ||
+ | </ | ||
+ | |||
+ | ===== Configuration de l'OS ===== | ||
+ | |||
+ | Elasticsearch est bien plus performant lorsqu' | ||
+ | |||
+ | On crée le fichier ''/ | ||
+ | < | ||
+ | elasticsearch soft nofile 65535 | ||
+ | elasticsearch hard nofile 65535 | ||
+ | elasticsearch - memlock unlimited | ||
+ | </ | ||
+ | |||
+ | Il est à noter qu' | ||
+ | |||
+ | ===== Entretien du cluster avec Curator ===== | ||
+ | |||
+ | Elasticsearch est assez bête dans sa gestion des données: il part du principe qu'il aura suffisamment d' | ||
+ | |||
+ | Pour retirer un index "à la main", par exemple si on a fait une bourde de configuration : API REST : https:// | ||
+ | |||
+ | Sinon, on utilise curator, depuis sa dernière version il utilise des fichiers de conf : | ||
+ | |||
+ | close30.yml | ||
+ | < | ||
+ | --- | ||
+ | actions: | ||
+ | 1: | ||
+ | action: close | ||
+ | description: | ||
+ | Close indices older than 30 days (based on index name), for logstash- | ||
+ | prefixed indices. | ||
+ | options: | ||
+ | delete_aliases: | ||
+ | timeout_override: | ||
+ | continue_if_exception: | ||
+ | disable_action: | ||
+ | filters: | ||
+ | - filtertype: pattern | ||
+ | kind: prefix | ||
+ | value: logstash- | ||
+ | exclude: | ||
+ | - filtertype: age | ||
+ | source: name | ||
+ | direction: older | ||
+ | timestring: ' | ||
+ | unit: days | ||
+ | unit_count: 30 | ||
+ | exclude: | ||
+ | </ | ||
+ | |||
+ | delete60.yml | ||
+ | < | ||
+ | --- | ||
+ | actions: | ||
+ | 1: | ||
+ | action: delete_indices | ||
+ | description: | ||
+ | Delete indices older than 60 days (based on index name), for logstash- | ||
+ | prefixed indices. Ignore the error if the filter does not result in an | ||
+ | actionable list of indices (ignore_empty_list) and exit cleanly. | ||
+ | options: | ||
+ | ignore_empty_list: | ||
+ | timeout_override: | ||
+ | continue_if_exception: | ||
+ | disable_action: | ||
+ | filters: | ||
+ | - filtertype: pattern | ||
+ | kind: prefix | ||
+ | value: logstash- | ||
+ | exclude: | ||
+ | - filtertype: age | ||
+ | source: name | ||
+ | direction: older | ||
+ | timestring: ' | ||
+ | unit: days | ||
+ | unit_count: 60 | ||
+ | exclude: | ||
+ | </ | ||
+ | |||
+ | open60.yml | ||
+ | < | ||
+ | --- | ||
+ | actions: | ||
+ | 1: | ||
+ | action: open | ||
+ | description: | ||
+ | Open indices older than 30 days but younger than 60 days (based on index | ||
+ | name), for logstash- prefixed indices. | ||
+ | options: | ||
+ | timeout_override: | ||
+ | continue_if_exception: | ||
+ | disable_action: | ||
+ | filters: | ||
+ | - filtertype: pattern | ||
+ | kind: prefix | ||
+ | value: logstash- | ||
+ | exclude: | ||
+ | - filtertype: age | ||
+ | source: name | ||
+ | direction: older | ||
+ | timestring: ' | ||
+ | unit: days | ||
+ | unit_count: 30 | ||
+ | exclude: | ||
+ | - filtertype: age | ||
+ | source: name | ||
+ | direction: younger | ||
+ | timestring: ' | ||
+ | unit: days | ||
+ | unit_count: 60 | ||
+ | exclude: | ||
+ | </ | ||
+ | |||
+ | On lance les scripts Close30 et Delete60 dans cron.daily/ | ||
+ | |||
+ | < | ||
+ | curator --config / | ||
+ | curator --config / | ||
+ | </ | ||
+ | |||
+ | On lancera à la main Open60 : | ||
+ | |||
+ | < | ||
+ | curator --config / | ||
+ | </ | ||
+ | |||
+ | ====== Monitoring ====== | ||
+ | |||
+ | En dehors du monitoring traditionnel du service **elasticsearch** (vérifier que le processus tourne | ||
+ | et qu'il écoute sur le bon port), nous avons utilisé des **UserParameters** dans la configuration de | ||
+ | l' | ||
+ | |||
+ | < | ||
+ | |||
+ | UserParameter=nb_node, | ||
+ | UserParameter=nb_data_node, | ||
+ | UserParameter=active_shard, | ||
+ | UserParameter=active_primary_shard, | ||
+ | UserParameter=unasigned_shards, | ||
+ | UserParameter=relocating_shards, | ||
+ | UserParameter=initializing_shards, | ||
+ | UserParameter=delayed_unassigned_shards, | ||
+ | UserParameter=number_of_pending_tasks, | ||
+ | UserParameter=task_max_waiting_in_queue_millis, | ||
+ | UserParameter=document_count, | ||
+ | UserParameter=heap_use_percent, | ||
+ | UserParameter=file_desc_percent, | ||
+ | UserParameter=ram_percent, | ||
+ | UserParameter=index_total, | ||
+ | UserParameter=indexing.delete_total, | ||
+ | UserParameter=search.query_total, | ||
+ | |||
+ | </ | ||
+ | |||
+ | Cela nous permet également d' | ||
+ | sur notre cluster. |