A new plugin inputs.opensearch with the RemoteStats API

[Mmuzaf]

Use Case

Hello Team,

Currently, the following inputs plugins are available for use with the OpenSearch service: inputs.elasticsearch, inputs.elasticsearch_query, inputs.opensearch_query. It is possible to use the elasticsearch plugins with the OpenSearch service with some caveats, as the APIs are aligned (new just need to replace appropriate strings).

While the OpenSearch service is being developed, a new API comes into play that we are also interested in scraping for the running nodes:

• Segment Replication API

GET /_cat/segment_replication
GET /_cat/segment_replication/{index}

• Remote Store Stats API

GET _remotestore/stats/<index_name>
GET _remotestore/stats/<index_name>/<shard_id>

I briefly checked Elasticsearch for the corresponding metrics API and it seems that there is no 1-to-1 match for the metrics we need. So I propose to add a new opensearch plugin to inputs, so we can query new metrics.

I have a draft of changes to demonstrate the possible solution of integrating new metrics to the elasticsearch plugin, please let me know if you are iterested in introducing a new inputs plugin and I need to submit a PR>

Configuration Example:

[[inputs.elasticsearch]]
tagexclude = []
servers = [ "http://127.0.0.1:9200",]
local = true
ccr_stats = true
cluster_health = true
cluster_health_level = "indices"
cluster_stats = true

[processors]
[[processors.strings]]
namepass = [ "elasticsearch_*",]
[[processors.strings.replace]]
measurement = "*"
old = "elasticsearch_"
new = "opensearch_"

Expected behavior

A new opensearch plugin is added to handle OpenSearch API specifics.

Actual behavior

Not able to scrape new opensearch metrics, need to replace elasticsearch_ with opensearch_.

https://opensearch.org/docs/latest/api-reference/cat/cat-segment-replication/
https://opensearch.org/docs/latest/tuning-your-cluster/availability-and-recovery/remote-store/remote-store-stats-api/

Additional info

No response

[Mmuzaf]

Hi @zak-pawel, I'd appreciate it if you could share your thoughts on this proposal.

[zak-pawel]

Hey @Mmuzaf
Perhaps you mentioned me because of my recent contributions to the files in the inputs.elasticsearch, inputs.elasticsearch_query, and inputs.opensearch_query plugins. However, they were mainly focused on fixing new issues found by linters.

@srebhan, maybe you can help?

[Mmuzaf]

If I'm not mistaken, this is probably another reason to create a dedicated input plugin for OpenSearch. I found another stats we use -- cross-cluster replication statistics, and the API between Elasticsearch and OpenSearch is also different.

Elasticsearch:
https://www.elastic.co/docs/api/doc/elasticsearch/v8/operation/operation-ccr-follow-stats

GET /{index}/_ccr/stats

Opensearch:
https://opensearch.org/docs/latest/tuning-your-cluster/replication-plugin/api/#get-follower-cluster-stats

GET /_plugins/_replication/follower_stats
GET /_plugins/_replication/leader_stats

[srebhan]

@Mmuzaf if the API really differs, I'm not against adding a new inputs.opensearch plugin. However, I want to see this new plugin being based on a solid and maintained library and not based on HTTP API calls in the plugin! If that's what you had in mind, I'm happy to review a PR... ;-)

[Mmuzaf]

@srebhan Thanks for the answer. Am I understanding correctly that we need to use the official os go client the same way we do for the inputs.opensearch_query plugin to query the stats as a replacement for the http calls?