Service discovery

Service discovery extracts all the potentially useful information from different sources, converts it to the configurations and exports them to the different destinations.

Pipeline

The service discovery pipeline has four jobs:

1. discovery

   Dynamically discovers monitoring targets by collecting events from kubernetes API server. Currently it collects POD and SERVICE events.

2. tag

   Dynamically add tags to discovered monitoring targets. Based on the POD and SERVICE fields and using patterns on them, one or more tags are attached to the monitoring targets.

3. build

   Dynamically creates data collection configurations for the monitored targets, using templates.

4. export

   Dynamically exports data collection configurations to allow netdata data collection plugins to use them. Data collection jobs in netdata are created and destroyed as needed.

Routing in a job and between jobs based on tags and selector.

Pipeline configuration:

name: <name>
discovery: <discovery_config>
tag: <tag_config>
build: <build_config>
export: <export_config>

Tags and selectors

Tag, build and export jobs have selector, the pipeline routes a target/config to the job only if its tags matches job selectors.

Both tags and selector are just lists of words.

A word must match the regex ^[a-zA-Z][a-zA-Z0-9=_.]*$.

Tags special cases:

• -word: the word will be removed on tags merging.

Selectors special cases:

• !word: should not contain the word.
• word|word|word: should contain any word.

Discovery

Discovery job dynamically discovers targets using one of the supported service-discovery mechanisms.

Supported mechanisms:

• kubernetes

Discovery configuration:

k8s:
  - <kubernetes_discovery_config>

Kubernetes

Kubernetes discoverer retrieves targets from Kubernetes' REST API. It always stays synchronized with the cluster state.

Kubernetes discovery configuration options:

# Mandatory. Tags to add to all discovered targets.
tags: <tags>

# Mandatory. The Kubernetes role of entities that should be discovered.
role: <role>

# Optional. Discover only targets that exist on the same node as service-discovery.
# This option works only for 'pod' role and it requires MY_NODE_NAME env variable to be set.
local_mode: <boolean>

# Optional. If omitted, all namespaces are used.
namespaces:
  - <namespace>

One of the following role types can be configured to discover targets:

• pod
• service

Pod Role

The pod role discovers all pods and exposes their containers as targets. For each declared port of a container, it generates single target. If there is no declared port it generates one target with empty Port, PortName and PortProtocol fields.

Available pod target fields:

Name	Type	Value
TUID	string	Namespace_Name_ContName_PortProtocol_Port
Address	string	PodIP:Port
Namespace	string	pod.metadata.namespace
Name	string	pod.metadata.name
Annotations	map[string]string	pod.metadata.annotations
Labels	map[string]string	pod.metadata.labels
NodeName	string	pod.spec.nodeName
PodIP	string	pod.status.podIP
ContName	string	pod.spec.containers.name
Image	string	pod.spec.containers.image
Env	map[string]string	pod.spec.containers.env + pod.spec.containers.envFrom
Port	string	pod.spec.containers.ports.containerPort
PortName	string	pod.spec.containers.ports.name
PortProtocol	string	pod.spec.containers.ports.protocol

Service Role

The service role discovers a target for each service port for each service.

Available service target fields:

Name	Type	Value
TUID	string	Namespace_Name_PortProtocol_Port
Address	string	Name.Namespace.svc:Port
Namespace	string	svc.metadata.namespace
Name	string	svc.metadata.name
Annotations	map[string]string	svc.metadata.annotations
Labels	map[string]string	svc.metadata.labels
Port	string	pod.spec.containers.ports.containerPort
PortName	string	pod.spec.containers.ports.name
PortProtocol	string	pod.spec.containers.ports.protocol
ClusterIP	string	svc.spec.clusterIP
ExternalName	string	svc.spec.externalName
Type	string	svc.spec.ports.type

Tag

Tag job tags targets discovered by discovery job. Its purpose is service identification.

Configuration is a list of tag rules:

- <tag_rule_config>

Tag rule configuration options:

# Mandatory. Routes targets to this tag rule with tags matching this selector.
selector: <selector>

# Mandatory. Tags to merge with the target tags if at least on of the match rules matches.
tags: <tags>

# Mandatory. Match rules, at least one should be defined. 
match:
  # Optional. Routes targets to this match rule with tags matching this selector.
  - selector: <selector>

    # Mandatory. Tags to merge with the target tags if this rule expression evaluates to true.
    tags: <tags>

    # Mandatory. Match expression.
    expr: <expression>

Match expression evaluation result should be true or false.

Expression syntax is go-template.

Available functions

• go-template built-in functions.
• sprig functions.
• custom functions.

Custom functions:

• glob reports whether arg1 matches the shell file name pattern.
• re reports whether arg1 contains any match of the regular expression pattern.

All these functions accepts two or more arguments, returning in effect:

func(arg1, arg2) || func(arg1, arg3) || func(arg1, arg4) ...

Build

Build job creates configurations from targets.

Configuration is a list of build rules:

- <build_rule_config>

Build rule configuration options:

# Mandatory. Routes targets to this rule with tags matching this selector.
selector: <selector>

# Mandatory. Tags to add to all built by this rule configurations.
tags: <tags>

# Mandatory. Apply rules, at least one should be defined. 
apply:
  # Mandatory. Routes targets to this apply rule with tags matching this selector.
  - selector: <selector>

    # Optional. Tags to add to configurations built by this apply rule.
    tags: <tags>

    # Mandatory. Configuration template.
    template: <template>

Template syntax is go-template.

Available functions

• go-template built-in functions.
• sprig functions.
• custom functions.

Custom functions:

• glob reports whether arg1 matches the shell file name pattern.
• re reports whether arg1 contains any match of the regular expression pattern.

All these functions accepts two or more arguments, returning in effect:

func(arg1, arg2) || func(arg1, arg3) || func(arg1, arg4) ...

Export

Export job exports configurations built by build job.

Supported exporters:

• file

Export configuration:

file:
  - <file_exporter_config>

File

File exporter writes configurations to a specific file.

# Mandatory. Routes configurations to this exporter with tags matching this selector.
selector: <selector>

# Mandatory. Absolute path to a file.
filename: <filename>

Troubleshooting

Service-discovery has debug mode and special stdout exporter which is enabled only when it's running from the terminal.

CLI:

Usage:
  sd [OPTION]...

Application Options:
      --config-file= Configuration file path
      --config-map=  Configuration ConfigMap (name:key)
  -d, --debug        Debug mode

Help Options:
  -h, --help         Show this help message