Hi, you will find great install guide on the official website. Also you need to decide if you will use high availability mode or just master <-> client setup. Install icinga on some VM and then setup it to be the master, then you can start installing it on clients. https://icinga.com/docs/icinga-2/latest/doc/02-installation/
I've been able to make this one work before. I've had fantastic luck with Digital Ocean guides. Never actually used DO as a service for anything, but they do seem to offer really good guides on stuff.
We wrote meerkat as a pure icinga API client. It doesn't support multiple icinga backends, but to make it similar to thruk I spose it could. https://icinga.com/blog/2020/12/25/sol1-releases-meerkat-next-generation-dashboards-for-icinga2/ If you can't restructure your monitoring layout you could probably deploy a standalone icinga2 instance each DC and an umbrella one that handles notifications and dashboards, purely checking the API. Kinda silly though. I would encourage you to consider netbox as well. It's a whole different idea, moving your doco to a dcim and then integrating it to icinga using director. Then stuff gets monitored as it's documented.
Use joins to add in other objects. Then you can reference it in your API filter.
https://icinga.com/docs/icinga-2/latest/doc/12-icinga2-api/#icinga2-api-config-objects-query-joins
perl snippet from a script I have to acknowledge all service problems on a host that might give you a sense of how it works.
my $service_filter = "service.state != ServiceOK && service.downtime_depth == 0.0 && service.acknowledgement == 0.0 && regex(\"$host\", host.name) ";
my $client = REST::Client->new();
$client->setHost($self->{'host'});
$client->addHeader("Accept", "application/json");
$client->addHeader("Authorization", "Basic " . encode_base64($self->{'userpass'}));
my %json_data = (
type => "Service",
filter => $service_filter,
joins => [ 'host.name', 'host.address' ],
author => $user,
comment =>$comment,
notify => 1,
);
my $data = encode_json(\%json_data);
$client->POST("/v1/actions/acknowledge-problem", $data);
You could add in another filter based on the service.name or something to get really specific on your acknowledgement.
https://icinga.com/docs/icinga2/latest/doc/06-distributed-monitoring/#endpoints
All endpoints in the same zone work as high-availability setup. For example, if you have two nodes in the master zone, they will load-balance the check execution.
Your user is not an admin and in fact has no privileges so you're only seeing limited options
If you're using database authentication you can insert an admin user with the instructions here
It's more common to enable a feature to forward the data to a specific service, the influxdb writer feature is an example of that and it what I use. A grafana instance using the influx data gives you a powerful tool. If you really want to do your own analysis you can read the perfdata files yourself as documented here: https://icinga.com/docs/icinga2/latest/doc/14-features/#writing-performance-data-files