> ## Documentation Index
> Fetch the complete documentation index at: https://docs.bagofwords.com/llms.txt
> Use this file to discover all available pages before exploring further.
# Observability and monitoring
> Connect Splunk, Elasticsearch, OpenSearch, Jaeger, Zabbix, Prometheus, AppDynamics, AWS CloudWatch and AWS Cost Explorer so the agent can investigate logs, traces, metrics and spend
These connectors let the agent investigate operational data — logs, search indices, distributed traces, monitoring metrics, and cloud spend — alongside your business data. None of them speak SQL natively, so Bag of words maps each platform onto a catalog the agent can reason about and issues queries in the platform's own language.
| Connector | What the catalog looks like | Query surface | Edition |
| :- | :- | :- | :- |
| [Splunk](#splunk) | One table per `index::sourcetype` | SPL | Beta |
| [Elasticsearch](#elasticsearch) | One object per index or index pattern | Query DSL, SQL, ES\|QL | Beta |
| [OpenSearch](#opensearch) | One object per index or index pattern | Query DSL, SQL | Beta |
| [Jaeger](#jaeger) | Fixed tables: services, operations, spans, dependencies | Parameterized trace search | Beta |
| [Zabbix](#zabbix) | Fixed tables: hosts, items, triggers, problems, events, history, trends | JSON-RPC query specs | Enterprise |
| [Prometheus](#prometheus) | One table per metric, with its labels as columns | PromQL | Beta |
| [AWS CloudWatch](#aws-cloudwatch) | `log_group::` and `metric::` tables on one connection | Logs Insights, metric queries | Beta |
| [AWS Cost Explorer](#aws-cost-explorer) | Cost and usage tables | Cost Explorer API calls | Beta |
| [AppDynamics](#appdynamics) | Fixed tables: applications, tiers, nodes, backends, business transactions, service flows, metrics, events, health-rule violations | JSON query specs | Beta |
***
## Splunk
The Splunk connector is available in every edition, and is currently in **beta**.
Bag of words talks to the Splunk REST API on the management port (`https://:8089`). There is no SQL endpoint — queries are **SPL** run as oneshot search jobs.
Tables are `index::sourcetype` pairs, enumerated with one cheap `tstats` call that reads tsidx metadata rather than raw events, so discovery cost does not grow with the number of sourcetypes. Columns cost a real search, so fields are sampled only for the top sourcetypes by volume; the rest stay thin and the agent samples their fields on demand.
### Fields
| Field | Required | Default | Notes |
| :- | :- | :- | :- |
| `host` | Yes | — | Splunk host, e.g. `splunk.acme.com`, or a full management URL such as `https://splunk.acme.com:8089`. |
| `port` | No | `8089` | REST/management port. Ignored when `host` is a full URL. |
| `verify_ssl` | No | `true` | Verify the server's TLS certificate. Disable only for self-signed certificates. |
| `discovery_window_days` | No | `7` | Default lookback applied to schema discovery and to searches that omit a time range. |
| `max_sampled_sourcetypes` | No | `50` | How many sourcetypes, ranked by event volume, get their fields sampled during indexing. Keeps reindexing cheap. |
### Authentication modes
Both modes support **system** and **user** scope.
**Authentication Token — `token`** (the default)
| Field | Required | Notes |
| :- | :- | :- |
| `api_token` | Yes | A Splunk authentication token (**Settings → Tokens**). Recommended, and the way to connect to Splunk Cloud. |
**Username / Password — `userpass`**
| Field | Required | Notes |
| :- | :- | :- |
| `username` | Yes | A Splunk user with search access to the target indexes. |
| `password` | Yes | That user's password. |
***
## Elasticsearch
The Elasticsearch connector is in **beta**.
Indices, aliases and data streams map to catalog objects, and the index **mapping** is the schema — discovery is a single bulk `GET /_mapping`, with no document sampling. Daily or rolling time-series indices (`logs-app-2026.07.10`, `…07.09`, …) are collapsed into one `logs-app-*` object holding the union of their fields, so the catalog stays a handful of patterns instead of one entry per day.
Queries use the native query DSL wrapped in a JSON envelope, with `POST /_sql` and ES|QL (`POST /_query`, Elasticsearch 8.11+) available as escape hatches.
### Fields
| Field | Required | Default | Notes |
| :- | :- | :- | :- |
| `host` | Yes | — | Host, e.g. `localhost`, or a full URL such as `https://es.example.com:9200`. |
| `port` | No | `9200` | REST port. Ignored when `host` is a full URL. |
| `secure` | No | `true` | Connect over HTTPS. Elasticsearch 8.x uses TLS by default. Ignored when `host` is a full URL. |
| `verify_certs` | No | `true` | Disable only for clusters using self-signed demo certificates. |
| `index_pattern` | No | — | Comma-separated index names or globs to expose, e.g. `logs-*,metrics-*`. Default: all non-system indices. |
### Authentication modes
**API Key — `apikey`** (the default; scopes: system and user)
| Field | Required | Notes |
| :- | :- | :- |
| `api_key` | Yes | An Elasticsearch API key. Paste either the encoded key or the raw `id:api_key` pair (**Stack Management → API keys**). Recommended for 8.x. |
**Username / Password — `userpass`** (scopes: system and user)
| Field | Required | Notes |
| :- | :- | :- |
| `user` | Yes | Username for HTTP basic auth, e.g. `elastic` or a role user with read access. |
| `password` | Yes | That user's password. |
**No Authentication — `none`** (scope: system, no fields). For clusters with security disabled or gated purely at the network layer.
***
## OpenSearch
The OpenSearch connector is in **beta**.
OpenSearch behaves like Elasticsearch: indices map to catalog objects, the mapping is the schema (one `GET /_mapping`, no document sampling), and time-series indices collapse into patterns. Queries are the native query DSL, with an SQL escape hatch via the bundled plugin (`/_plugins/_sql`).
### Fields
| Field | Required | Default | Notes |
| :- | :- | :- | :- |
| `host` | Yes | — | Host, e.g. `localhost`, or a full URL such as `https://search.example.com:9200`. |
| `port` | No | `9200` | REST port. Ignored when `host` is a full URL. |
| `secure` | No | `false` | Connect over HTTPS. Ignored when `host` is a full URL. |
| `verify_certs` | No | `true` | Disable only for clusters using self-signed demo certificates. |
| `index_pattern` | No | — | Comma-separated index names or globs to expose, e.g. `logs-*,orders`. Default: all non-system indices. |
### Authentication modes
**Username / Password — `userpass`** (the default; scopes: system and user)
| Field | Required | Notes |
| :- | :- | :- |
| `user` | Yes | Username for HTTP basic auth (the OpenSearch security plugin). |
| `password` | Yes | That user's password. |
**No Authentication — `none`** (scope: system, no fields). For clusters with security disabled or network-gated.
***
## Jaeger
The Jaeger connector is in **beta**.
Jaeger has no query language — it is a parameterized trace search — so instead of discovering tables Bag of words presents a fixed catalog that mirrors the Query API: `services`, `operations`, `spans` and `dependencies`. Span rows are flattened to identity, topology, timing and status columns, plus the span's own tags.
### Fields
| Field | Required | Default | Notes |
| :- | :- | :- | :- |
| `base_url` | Yes | — | Jaeger Query URL including scheme and port, e.g. `http://jaeger:16686`. |
| `verify_ssl` | No | `true` | Verify the server's TLS certificate. Disable only for self-signed certs on internal hosts. |
| `default_lookback` | No | `1h` | Search window used when a query omits one, e.g. `1h`, `6h`, `2d`. |
| `default_limit` | No | `20` | Default maximum number of traces returned by a span search. |
### Authentication modes
**No Auth (network-gated) — `none`** (the default; scope: system, no fields). Jaeger Query typically has no auth of its own.
**Username / Password (Basic) — `basic`** (scopes: system and user)
| Field | Required | Notes |
| :- | :- | :- |
| `username` | Yes | For HTTP basic auth — typically a reverse proxy in front of Jaeger Query. |
| `password` | Yes | That user's password. |
**Bearer Token — `bearer`** (scopes: system and user)
| Field | Required | Notes |
| :- | :- | :- |
| `token` | Yes | Sent as `Authorization: Bearer `. Used when Jaeger Query sits behind an auth proxy or gateway. |
***
## Zabbix
The Zabbix connector is part of the **Enterprise Edition**.
Bag of words talks to the Zabbix JSON-RPC 2.0 API (the `/api_jsonrpc.php` endpoint is appended to your frontend URL automatically). Queries are JSON specs that map one-to-one onto Zabbix API methods — `host.get`, `item.get`, `problem.get`, `history.get`, and so on.
The catalog is a fixed set of virtual tables — hosts, items, triggers, problems, events, history, trends — because Zabbix's data model is stable. The `items` table is enriched from a live `item.get` so the agent sees the value types actually present.
### Fields
| Field | Required | Default | Notes |
| :- | :- | :- | :- |
| `url` | Yes | — | Your Zabbix frontend URL, e.g. `https://zabbix.acme.com`. |
| `verify_ssl` | No | `true` | Verify the server's TLS certificate. Disable only for self-signed certificates. |
| `history_window_days` | No | `7` | Default lookback applied when querying metric history or trends without an explicit time range. |
### Authentication modes
Both modes support **system** and **user** scope.
**API Token — `token`** (the default)
| Field | Required | Notes |
| :- | :- | :- |
| `api_token` | Yes | A Zabbix API token (**Users → API tokens**). Recommended for Zabbix 5.4+, and the way to connect in SSO environments. |
**Username / Password — `userpass`**
| Field | Required | Notes |
| :- | :- | :- |
| `username` | Yes | A Zabbix user with read access to the monitored hosts. Used on older installs without API tokens. |
| `password` | Yes | That user's password. |
Zabbix's SAML/OIDC SSO governs the **frontend** only — the API never accepts an external identity provider's token. An SSO user should mint a personal API token in the UI and use the token mode.
***
## Prometheus
The Prometheus connector is in **beta**.
Bag of words talks to the Prometheus HTTP API and queries in **PromQL**. Each metric becomes a table whose columns are that metric's labels, so the agent can filter and group without knowing the label set in advance.
Metric **type** matters for correct queries — a counter read without `rate()` returns a meaningless monotonic total — so each metric carries its type from `/api/v1/metadata`. Histogram and summary components are typed from their base metric: `_bucket` resolves to a histogram (use `histogram_quantile`), `_sum` and `_count` to counters.
Schema discovery is bounded to a recent window rather than the whole retention period, so a long-lived instance does not materialize every series it has ever seen. Widen it with `discovery_lookback_hours` only if a metric is scraped less often than the window.
The connector works against any Prometheus-compatible API, including Thanos, Cortex and Grafana Mimir — set `org_id` for the multi-tenant header those back-ends expect.
### Fields
| Field | Required | Default | Notes |
| :- | :- | :- | :- |
| `base_url` | Yes | — | Prometheus server URL including scheme and port, e.g. `http://prometheus:9090`. |
| `verify_ssl` | No | `true` | Verify the server's TLS certificate. Disable only for self-signed certs on internal hosts. |
| `org_id` | No | — | Sent as `X-Scope-OrgID` for multi-tenant back-ends (Thanos, Cortex, Grafana Mimir). |
| `metric_prefix` | No | — | Bound discovery on a large instance, e.g. `node_` or `http_`. Leave blank to index all metrics. |
| `discovery_lookback_hours` | No | `1` | How far back discovery looks for a metric's labels. Raise it only if a metric is scraped less often than that and its labels are being missed. |
| `timeout` | No | `30` | Per-request timeout in seconds for the Prometheus HTTP API. Raise it for a slow or heavily loaded instance. |
### Authentication modes
**No Auth (network-gated) — `none`** (the default; scope: system, no fields). For an instance reachable only over a VPN or internal network.
**Username / Password (Basic) — `basic`** (scopes: system and user)
| Field | Required | Notes |
| :- | :- | :- |
| `username` | Yes | For HTTP basic auth — typically a reverse proxy in front of Prometheus. |
| `password` | Yes | That user's password. |
**Bearer Token — `bearer`** (scopes: system and user)
| Field | Required | Notes |
| :- | :- | :- |
| `token` | Yes | Sent as `Authorization: Bearer `. Used by most hosted or managed Prometheus offerings. |
***
## AWS CloudWatch
The AWS CloudWatch connector is in **beta**.
Logs and metrics on **one** connection, because an investigation crosses both halves — the function erroring in Logs is the one whose duration lives in Metrics. Splitting them across two connectors would put them on separate connections with no way for the agent to correlate.
Tables are namespaced by prefix, the way Splunk namespaces `index::sourcetype`:
```text theme={null}
log_group::/aws/lambda/checkout @timestamp, @message, @logStream
+ JSON keys sampled from recent events
metric::AWS/EC2/CPUUtilization + timestamp + value
```
Log discovery is schema-on-read: CloudWatch has no field catalog, so a log group's columns come from sampling recent events and parsing JSON out of `@message`. Only the largest groups by stored bytes are sampled; the rest stay thin and the agent inspects them on demand. Metrics are grouped by namespace and metric name, with their dimension names unioned — the API returns one entry per dimension-set variant.
Logs Insights bills on bytes **scanned**, and a `limit` clause does not reduce that — the time window is the only real cost control. Keep `discovery_window_hours` tight and set `log_group_prefix` on a busy account.
### Fields
| Field | Required | Default | Notes |
| :- | :- | :- | :- |
| `region` | Yes | — | AWS region to read CloudWatch from, e.g. `eu-west-1`. Log groups and metrics are per-region. |
| `log_group_prefix` | No | — | Only discover log groups whose name starts with this, e.g. `/aws/lambda/`. Strongly recommended on a busy account. |
| `metric_namespaces` | No | — | Comma-separated namespaces to index as tables, e.g. `AWS/EC2,AWS/RDS`. Leave blank to index all. |
| `discovery_window_hours` | No | `24` | How far back to sample log events when inferring a log group's columns. Larger windows find more fields and scan more bytes. |
| `max_sampled_log_groups` | No | `25` | Cap on how many log groups, ranked by stored bytes, get their fields sampled during indexing. |
### Authentication modes
All three modes are **system** scope.
**AWS Access Key — `aws_keys`** (the default)
| Field | Required | Notes |
| :- | :- | :- |
| `access_key` | Yes | AWS access key id. |
| `secret_key` | Yes | AWS secret access key. |
| `session_token` | No | For temporary credentials. |
**AWS Assume Role (STS) — `aws_role`**
| Field | Required | Notes |
| :- | :- | :- |
| `role_arn` | Yes | ARN of the IAM role to assume for CloudWatch access. |
| `access_key` | No | Used to assume the role. Leave blank to assume it from the instance profile or IRSA. |
| `secret_key` | No | Secret for the key above. |
**AWS Default Chain — `aws_default`** (no fields). Resolves credentials the way boto3 normally does — environment variables, shared config, EC2 instance profile, or EKS IRSA. This is the mode for a deployment running inside AWS, where no secret should be stored at all.
***
## AWS Cost Explorer
The AWS Cost Explorer connector is in **beta**.
Bag of words queries the AWS **Cost Explorer** API (`ce`) so the agent can analyze spend and usage over time — by service, account, tag, or any other Cost Explorer dimension.
### Fields
| Field | Required | Default | Notes |
| :- | :- | :- | :- |
| `region_name` | Yes | — | AWS region used for the Cost Explorer client, e.g. `us-east-1`. |
### Authentication
**AWS Keys — `key`** (scopes: system and user)
| Field | Required | Notes |
| :- | :- | :- |
| `access_key` | Yes | AWS access key id. |
| `secret_key` | Yes | AWS secret access key. |
The credential needs permission to call the Cost Explorer read APIs (`ce:GetCostAndUsage` and related). Because the mode is also available at user scope, individual users can supply their own keys instead of sharing the admin's — per-user credentials on a database-style connection require an Enterprise license. See [Authentication and access](/data-sources/authentication).
***
## AppDynamics
The AppDynamics connector is in **beta**.
Bag of words talks to the Cisco / Splunk AppDynamics **Controller REST API** (`https:///controller/rest/...`). On-premises controllers are the target deployment — validated against the 21.x API surface, which has been stable since the 4.x era — and SaaS controllers serve the same API.
There is no free-form query language. Queries are JSON specs that map onto a fixed catalog of virtual tables: the topology half (`applications`, `tiers`, `nodes`, `backends`, `business_transactions`, `service_flows`) and the time-bounded activity half (metric data, events, health-rule violations, snapshots). A query returns at most 50,000 rows, and activity tables default to a 60-minute window when the spec gives no time range.
### The service map
AppDynamics has no supported flow-map endpoint — the UI is driven by the unsupported session-auth `/controller/restui/*` API, which this connector deliberately avoids. Instead, the `service_flows` table derives the edge list by enumerating metric **names** (not data) under `Overall Application Performance||External Calls|`, where each child name encodes an outbound call such as `Call-HTTP to Discovered backend call "PaymentsGateway"`. That is one cheap metrics-browse call per tier with no metric data fetched, so it stays well under Controller rate limits.
### Fields
| Field | Required | Default | Notes |
| :- | :- | :- | :- |
| `controller_url` | Yes | — | Controller URL, e.g. `https://appd.acme.com:8090` (on-prem) or `https://acme.saas.appdynamics.com`. |
| `account_name` | No | `customer1` | The controller account. On-premises single-tenant controllers use `customer1`. |
| `verify_ssl` | No | `true` | Verify the controller's TLS certificate. Disable only for self-signed or private-CA certs on trusted networks. |
### Authentication modes
Both modes support **system** and **user** scope.
**Username + Password — `userpass`** (the default)
| Field | Required | Notes |
| :- | :- | :- |
| `username` | Yes | The Controller expects the login qualified with the account name (`user@account`). The `account_name` above is appended automatically unless the username already contains `@`. |
| `password` | Yes | That user's password. |
This is the mode for locked-down on-prem controllers that cannot issue an API client.
**API Client (OAuth) — `api_client`**
| Field | Required | Notes |
| :- | :- | :- |
| `client_name` | Yes | Name of an API Client created under **Settings → Administration → API Clients**. |
| `client_secret` | Yes | That API Client's secret. |
The connector exchanges these for OAuth client-credentials tokens against `/controller/api/oauth/access_token`. Tokens are short-lived (5 minutes by default) and are cached and refreshed automatically. Prefer this mode when you want an auditable service identity rather than a named user's credentials.
This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.