.. _root_cause_analysis_guide: Root cause analysis (RCA) ========================= What is it? ----------- **Root cause analysis** (RCA) gathers the monitoring context of a Host, Service, Hostgroup or Realm and uses the :ref:`ia_assistant` to produce a diagnostic report. The **Root Cause Analysis** section of **Observability** stores the analyses of the selected Realm so that users can review them later. ---- What is it for? --------------- RCA helps investigate an incident without checking each data source separately. Host and Service analyses correlate the available evidence around an incident. Realm analyses provide an overview of affected Hosts, explain what happened and identify possible developments supported by the evidence. Stored analyses let users compare diagnoses without repeating an analysis. ---- How does it work? ----------------- An analysis runs only when a user requests it for a Host, Service, Hostgroup or Realm. It does not run periodically. Information sources ^^^^^^^^^^^^^^^^^^^ WOCU gathers the applicable monitoring states, dependencies, events and anomalies and passes them to the configured assistant. Available DNS and IP address management data can provide network context for Host and Service analyses; inaccessible sources are omitted. The Host is the correlation point for a Service analysis, with priority given to the selected Service. For a Realm analysis, WOCU gathers counters, current problems, down or unreachable Hosts grouped by their down parent Host, events from the last six hours and anomalies from the last 24 hours. It inspects up to 50 Hosts and indicates when there are more. Unreadable sources are omitted. Realm counters come from WOCU rather than estimates generated by the model. The assistant uses the Realm's own agent when provisioned, otherwise the existing global agent. See :ref:`conf_observability_script` for configuration. Processing and stored results ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ WOCU validates results before displaying them. Sensitive information is masked in requests and responses, including streamed responses and stored conversation output. Structured results reject unexpected fields. An answer that still contains detected secrets or raw data after one repair attempt is reported as failed. Masking covers known patterns and does not guarantee that every possible secret is recognized. Each analysis is stored with its target, requester, dates, status and result. The request modals send one POST with ``wait: true`` and display a spinner until the response contains the result. They do not poll the analysis detail. On completion, the spinner is replaced by the report. The modal does not close automatically, either while waiting or after displaying the result. Users can open the analyses table instead of waiting in the modal. Closing the modal or navigating to the table ends the browser's wait without deleting an existing stored analysis. A background maintenance task marks analyses that remain pending or running beyond the configured threshold as failed. A late result cannot replace an analysis already closed by that task. See :ref:`rca_api` for the API contract. Analysed period ^^^^^^^^^^^^^^^ The current time shown to the assistant is the next 5-minute boundary after the request. Requests made close in time therefore receive the same reference time. The reference time ends the period of a Realm or Hostgroup analysis. For a Host or a Service analysis, the monitoring events are read around the last state change of the object, or around its last check when there is no last state change, from 60 minutes before to 15 minutes after that instant. The reference time is used only when the request carries neither. Reuse of a recent analysis ^^^^^^^^^^^^^^^^^^^^^^^^^^ When a user requests an analysis they already requested a short time ago, WOCU returns that analysis instead of starting another one. This keeps the answer stable and avoids a new assistant run. It applies to every way of requesting an analysis: the background request, the request with ``wait: true`` and the synchronous endpoint. Two requests are the same analysis when they share the requesting user, the Realm, the target (Host, Service, Realm or Hostgroup), the analysed object (the Host, the Host and Service, the Hostgroup or the Realm itself) and the connector. Other users, other objects and other targets always start their own analysis. For a Host or a Service analysis the incident instant must also be the same. It is the ``last_state_change`` of the request, or its ``last_check`` when there is no ``last_state_change``; two requests without any instant also match. A new incident of the same Host or Service, with a different instant, therefore starts a new analysis even inside the reuse window. Realm and Hostgroup analyses have no incident instant and keep the rule above. An analysis is reused when it was requested within the reuse window and is **Pending**, **Running** or **Completed**. A **Failed** analysis, including one closed by the maintenance task, is never reused, so the next request starts a new one. A reused analysis that is still pending or running is not run again. The background request and the request with ``wait: true`` answer with it in progress, and its result is read from the analysis detail once it completes. The synchronous endpoint waits for that analysis to finish and answers with its outcome; if it has not finished within the time the endpoint allows for an analysis (``WOCU_KIBANA_CHAT_TIMEOUT``, 90 seconds by default), it answers with a ``504`` error and the analysis keeps running. It can be read later from the analysis list or requested again. Every response to a request includes ``"reused": true`` when an earlier analysis was returned and ``"reused": false`` when a new one was started. The status code and the rest of the response do not change. The window is set with the ``OBSERVABILITY_RCA_REUSE_MINUTES`` environment variable. The default is 15 minutes; ``0`` disables the reuse and every request starts a new analysis. Dependencies and duration ^^^^^^^^^^^^^^^^^^^^^^^^^ RCA requires Observability for the Realm, the AI assistant credentials and inference connector, and the applicable data sources. An analysis can take a few minutes. The waiting notice is available in English and Spanish and tells users that the result can also be viewed in the Root Cause Analysis table. ---- How do you use it? ------------------ Analyse a Realm ^^^^^^^^^^^^^^^ #. Select a Realm with Observability enabled. #. In the breadcrumb, click **Realm RCA**, identified by a magnifying-glass icon. Its tooltip is **Realm root cause analysis**. #. Wait in the modal. A spinner remains visible while the analysis runs; the report appears when it finishes. The modal stays open until you close it or navigate to the table. #. Alternatively, click **View root cause analyses** to open **Operation > Observability > Root Cause Analysis** in the same Realm, with **Type** set to **Realm**. Analyse a Host or Service ^^^^^^^^^^^^^^^^^^^^^^^^^ #. Open **Operation > Assets > Hosts** and select a Host to open its detail view. For a Service, open its detail from the Host or from **Operation > Assets > Services**. #. Click **Host RCA** or **Service RCA**, respectively, beside the actions selector. Both buttons use a magnifying-glass icon and the same link style as **Realm RCA**; their text is underlined on hover. The existing **Root Cause Analysis** action menu entry also starts the same workflow (:ref:`action_rca`). #. Wait for the spinner to be replaced by the report, or click **View root cause analyses** to open the table. A Host request sets **Type** to **Host** and filters by the Host name. A Service request sets **Type** to **Service** and filters by both the Host name and Service description. Analyse a Hostgroup ^^^^^^^^^^^^^^^^^^^ #. Open **Operation > Assets > Host Groups** and select a group to open its detail view. #. Click **Host group RCA** in the header. It uses the same magnifying-glass icon, link style, hover underline and light/dark themes as the Host and Service RCA buttons. #. Wait for the spinner to be replaced by the report. The request uses the group's source Realm and sends ``target: "hostgroup"``, ``hostgroup_name`` and ``wait: true`` to ``ai_assistant_root_cause_analysis``. Only one POST is made; the modal does not poll the detail endpoint or close automatically. #. Alternatively, click **View root cause analyses**. The table opens in the source Realm with **Type** set to **Host group** and **Host group** set to the selected group's name. The button requires the same ``ai-assistant-root-cause-analysis`` action permission as Host and Service, with category ``livestatus``. There is no separate Hostgroup RCA permission. It is hidden when that action is unavailable. These shortcuts are shown for assets belonging to a Realm with Observability. In a multirealm view, availability and the requested analysis use the asset's source Realm. Host, Service and Hostgroup shortcuts are hidden without the existing RCA action permission and disabled when the RCA endpoint is unavailable. Navigate from the modal to the table ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ The **View root cause analyses** button becomes available after the request is accepted, provided the list endpoint is available; users do not need to wait for completion. Clicking it closes the modal and opens the table with the filters described above. The background analysis continues. These navigation filters replace previously saved filters and clear the saved text search so that unrelated criteria do not hide the requested analysis. Column preferences are preserved. Users can then change or clear the filters normally. The button opens the filtered list, not a single analysis detail. Review stored analyses ^^^^^^^^^^^^^^^^^^^^^^ #. Open **Operation > Observability > Root Cause Analysis** in the Realm. #. Search, sort or apply filters to find an analysis. When arriving from a request modal, the corresponding Host, Service or Realm filters are already applied. #. Select a row or its summary to open its detail in the side panel. Host and Service links continue to open the corresponding asset views. #. Read the status and report. The subtitle identifies the Host or Service; for a Realm analysis it shows **Realm: **. #. Close the panel to continue reviewing the table. Reopening a row retrieves its current detail. An open table detail panel does not automatically refresh when the analysis changes. ---- Available options ----------------- .. list-table:: :header-rows: 1 :widths: 28 72 * - Option - Description * - **Realm RCA** - Starts an analysis of the selected Realm from the breadcrumb. * - **Host RCA** / **Service RCA** - Starts the existing RCA action from an asset detail view, using the same action permission as the action menu entry. * - **Host group RCA** - Starts an analysis for the selected Hostgroup from its detail view, using the existing Host and Service RCA action permission. * - **View root cause analyses** - Opens the analyses table in the source Realm, filtered by the request target and, as applicable, the Host name, Service description or Hostgroup name. Available while waiting for the response and after completion. * - Search - Searches stored analyses on the server, together with the applied filters and pagination. * - **Type** - Filters by **Host**, **Service**, **Realm** or **Host group**. Multiple targets can be selected. * - **Analysis status** - Filters by **Pending**, **Running**, **Completed** or **Failed**. Multiple statuses can be selected. * - **Root cause found** - Filters completed results by **Yes** or **No**. An unfinished or failed analysis must not be interpreted as a confirmed negative finding. * - **Host** - Filters by a case-insensitive fragment of the Host name: for example, ``wocu`` matches ``wocu-devel``. Empty values are ignored. * - **Service** - Filters by the exact Service description. Empty values are ignored. * - **Host group** - Filters by the exact Hostgroup name. Empty values are ignored. * - Date range - Filters by the request date. There is no date restriction by default. * - Apply, clear and refresh - Apply the selected filters, clear them or retrieve the latest table data. Changing filters does not start another analysis. * - Columns and pagination - Control visible columns, order, width and the current page. Search, filters and table preferences are stored in the browser per Realm. The default order shows the most recent request first. * - CSV and JSON export - Downloads all records matching the applied filters, search and supported ordering, rather than only the visible page. Availability follows the existing CSV and JSON action permissions. Exports include list metadata and summaries, not the full detail report. The default columns are **Requested at**, **Type**, **Host**, **Service**, **Trigger state**, **Analysis status**, **Root cause found**, **Summary** and **Requested by**. The table has no create, edit or delete actions; use the RCA entry points described above to request another analysis. ---- Result ------ A completed analysis displays a readable report in the request modal and remains stored in the Realm's Root Cause Analysis table. Closing the modal does not delete it. * **Host or Service:** summary, root cause found, timeline, causal chain, impact and recommended actions. * **Realm:** summary, available Host and Service counters, what happened, possible root causes and what may happen next. Missing counters are not interpreted as zero. Empty report sections can be omitted. * **Detail panel:** stored report and any recorded error, with the summary used when there is no report. The report uses the requester's language; interface labels use the viewer's language. Modals, the detail panel and shortcuts support light and dark themes. A completed analysis can state that the evidence is insufficient to identify a root cause. **Root cause found** distinguishes positive findings, negative findings and analyses without a finished result. Confidence indicates whether a cause is confirmed, probable or possible; possible developments use the same probability scale. An empty outlook is not a prediction. If the request or analysis fails, the modal displays an error instead of continuing to show the spinner. A Realm submission returning HTTP 408 or 504 displays **The backend timed out while running the analysis. Please try again.** Once submission has succeeded, the table button remains available even if retrieving or completing the analysis later fails. Users can compare stored analyses and export the list. Request a new analysis when an updated diagnosis is needed. RCA supports investigation and does not replace the operator's verification. ---- Considerations and limitations ------------------------------ Permissions and scope ^^^^^^^^^^^^^^^^^^^^^ * The history table requires the **Root Cause Analysis** widget and list endpoint. Reading a stored detail requires the detail endpoint. Request modals use the synchronous endpoint ``ai_assistant_root_cause_analysis`` and do not fetch the detail endpoint. History access alone does not grant the Host, Service or Hostgroup RCA action. * Host, Service and Hostgroup shortcuts require the existing ``ai-assistant-root-cause-analysis`` action permission. Removing it hides the shortcut even when a shared endpoint remains available. * UI visibility checks do not add target-level authorization to the server. Requests without the required permission receive ``403``. * Authorized Realm users can read its analyses and see their requesters. * Realm requests and the history table operate at Realm level. Host, Service and Hostgroup requests from multirealm asset views use the asset's source Realm. Analysis endpoints return ``404`` for a multirealm. Agent requirements ^^^^^^^^^^^^^^^^^^ * Observability and the AI assistant backend, credentials and inference connector must be configured and available. See :ref:`conf_observability_realm` and :ref:`conf_observability_script`. * Administrators provision the Realm's own agent using the configuration script. Without provisioning, AI requests fail with a configuration error. They authenticate with the API key of the Realm's Elasticsearch backend. * Inference connectors must be available in the Realm Space; preconfigured connectors are available in all Spaces. If none is available, the assistant reports it instead of using the global Space. * Legacy global conversations remain stored for accounting but are hidden from the Realm assistant. Provisioning details are in the technical appendix. Duration and failures ^^^^^^^^^^^^^^^^^^^^^ * An analysis can take a few minutes. The request modal waits for its stored status and has no automatic close timer. * If a worker fails, an analysis can remain pending or running until the maintenance task marks it as failed after the configured threshold. A new analysis can always be requested. * AI assistant access and consumption limits apply to the requester. They are checked when the analysis runs, not when it is created, and appear as a failed analysis with the corresponding limit message. * Available evidence depends on configured integrations and data coverage. Missing sources can leave the report incomplete. A stored record does not imply that the analysis completed successfully; check its status and error. * Sensitive-information masking covers known patterns and cannot guarantee recognition of every possible secret. * RCA suggests diagnostic checks; it does not establish certainty or apply corrective changes to the monitored infrastructure. ---- Related information ------------------- * :ref:`observability_module`: Observability configuration and capabilities. * :ref:`conf_observability_realm`: Realm backend and credentials. * :ref:`conf_observability_script`: automated Realm Space and agent setup. * :ref:`ia_assistant`: conversations and further investigation. * :ref:`action_rca`: Host and Service root cause analysis action. * :ref:`hostsinventory` and :ref:`services-inventory-assets`: asset entry points. * :ref:`rca_realm_agent`: Realm agent provisioning and credential management. * :ref:`rca_agent_consumption`: agent consumption collection permissions. * :ref:`rca_api`: request modes, statuses, result schemas and examples. * :doc:`Observability `: module configuration and existing assistant workflows. ---- Technical appendix: Root Cause Analysis API ------------------------------------------- Request flow, storage and masking ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Sensitive information is masked in requests and responses, including streamed responses and stored conversation output. Structured results reject unexpected fields. An answer that still contains detected secrets or raw data after one repair attempt is reported as failed. Masking covers known patterns and does not guarantee that every possible secret is recognized. All request modals use the synchronous endpoint ``ai_assistant_root_cause_analysis`` and display a spinner until the single POST returns its result. They replace the spinner with the report on completion, show errors on failure and remain open until the user closes them or navigates to the table. Closing a modal stops the browser's wait without deleting an existing stored analysis. When the request reuses an analysis that is still in progress, the POST waits for it; if it does not finish within the time the endpoint allows, the response is a timeout error and the analysis keeps running. See :ref:`rca_api` for the complete API contract. Analyses are stored with their target, requester, dates, status and result. A background maintenance task marks analyses that remain pending or running beyond the configured stale threshold as failed. A late result cannot replace an analysis already closed by that task. .. _rca_realm_agent: Per-realm AI agent ^^^^^^^^^^^^^^^^^^ The ``configure_kibana_realm_spaces`` command described in :ref:`conf_observability_script` provisions the AI agent of the Realm in its Kibana Space, in addition to the Space, role, data views and ML job: - Eight **query tools**: a detail tool and a density tool for each authorized source (metrics, monitoring events, syslog and traps). - The **agent** ``wocu-``, with permanent instructions rendered from ``observability/prompts/agent_instructions.md`` and the eight tools above. Running the command again creates missing tools and agents, updates definitions that changed and leaves matching definitions untouched. It deletes obsolete Space tools whose id starts with ``wocu..``; other tools are preserved. Chat, Ask AI, Root Cause Analysis and consumption collection authenticate with ``realm.elasticsearch_backend.api_key``. The command does not create, store, rotate or invalidate a separate AI API key. Configure the credential on the Realm's Elasticsearch backend. Runtime requests use that key. Provisioning requires ``ELASTICSEARCH_MASTER_API_KEY``; the command fails when it is unset. The provisioning credential needs the permissions to manage the Space, roles, data views, ML jobs, tools and agent. No dedicated service-user credentials are required to create an agent key. The Realm must have a provisioned agent and Space. Missing provisioning returns an explicit configuration error; there is no fallback to the global agent. Legacy global conversations stay stored for accounting but cannot be listed, opened or continued through the Realm assistant. Kibana connectors belong to a Space. The inference connectors used by the AI assistant must be available in the Realm Space; preconfigured connectors are available in every Space. If no connector is available there, the assistant reports it instead of falling back to the global Space. .. Important:: The instructions Markdown does not replace permissions: index and field access follows the configured backend API key's permissions. Tools select their sources and fields and mask recognized secrets before returning data. WOCU also redacts sensitive information on the server, in the text it sends to the AI assistant and in the text it returns. Passwords, tokens, API keys, cookies, private keys, SNMP communities, credentials inside URLs and connection strings, and the addresses of the realm backend are replaced by ``[OCULTO]`` in the monitoring context and plugin outputs before the prompt is built, and in chat answers (also while streaming), Ask AI answers, root cause analyses and the stored conversation history. A root cause analysis that still contains a secret or raw data (code blocks, JSON, log lines, long encoded strings) after one repair turn is reported as failed instead of shown. This redaction only covers the patterns above; monitored hosts and services are not hidden by it. Two layers mask secrets and both are best-effort. First, the agent tools mask secrets inside the queries themselves: the free-text columns they read (event messages, trap texts, log messages) go through ``REPLACE`` expressions in the ES|QL query, so a matching secret is excluded before it reaches the model. Second, the server-side redaction above covers the prompts and the answers. Neither restricts the permissions of the backend API key. The ambiguous words ``pass``, ``auth`` and ``session`` are only redacted after ``=`` or when the value looks like a credential (one token of 8 or more characters with a digit or a symbol), so ordinary text such as ``session: closed`` is left intact. .. Note:: The mapping of the ES9 ``monitoring-events-*`` index is not defined in any repository, so the fields selected by the query tools follow the WOCU readers. Confirm it on a cluster before relying on it. Traps use the field ``MessageText``, which the ES9 fluentd writer (``fluentd/trap.conf``) keeps; ``trap_message`` is removed there. The tools of the agent work **per host**: each one needs the ``host`` of a monitored host, so the realm-level root cause analysis uses them only for the hosts named in the WOCU context and takes the realm-wide picture (overview, problems, events and anomalies) from WOCU itself. .. _rca_agent_consumption: Agent consumption collection ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ The periodic AI assistant consumption task calls ``POST /api/agent_builder/agents/{agent_id}/consumption``, available from **Stack 9.4** and in **Serverless**. Although this operation reads usage, Kibana authorizes it through ``agentBuilder:manageAgents``, which requires ``feature_agentBuilder.all`` on the credential used for collection. For a provisioned Realm agent, collection uses the Elasticsearch backend API key in the agent's Space, using the same credential as chat and RCA. In Stack 9.3 and deployments without Agent Builder, collection is skipped automatically. .. _rca_api: Root Cause Analysis API ^^^^^^^^^^^^^^^^^^^^^^^ This section is the contract between the backend and the user interface of the root cause analysis (RCA). Every analysis is stored with the realm, the host or service it refers to, the state that triggered it, the user who requested it and its outcome. All the authorized users of the realm can read it. An analysis only runs when a user requests it. Analyses are of three kinds of target: a host, a service of a host, or the whole realm. Only realms are supported. A multirealm answers ``404`` on every endpoint of this section. UI wiring ~~~~~~~~~ * The Observability tab of the realm has the section and widget ``root-cause-analysis`` (ui migration 0178). It is the table of analyses with the detail panel. The user sees the section when the widget is granted to its profile. * The endpoints are published to the frontend in the permissions map, under these keys (the URL name with ``-`` replaced by ``_``): ============================================ ======== ================================================== Key Method Use ============================================ ======== ================================================== ``observability_root_cause_analyses`` ``GET`` List the analyses of the realm. ``observability_root_cause_analyses`` ``POST`` Request a new analysis (host, service or realm). ``observability_root_cause_analysis_detail`` ``GET`` Read one analysis (poll it until it finishes). ``ai_assistant_root_cause_analysis`` ``POST`` Legacy synchronous endpoint for existing clients. ============================================ ======== ================================================== * The URLs of the map keep the placeholders ````, ```` and, for the detail, ````, percent-encoded as ``%3Crealm_type%3E``, ``%3Crealm_name%3E`` and ``%3Crca_id%3E``. The client replaces them: the realm type (``realm``) and name of the current realm, and the ``id`` of the analysis. The client never builds these paths by hand. * Permissions are granted per URL name, so one grant covers the ``GET`` and the ``POST`` of ``observability_root_cause_analyses``. The two new URLs are granted by three owners: the ``ai-assistant`` widget (ui migration 0177), and the widget ``root-cause-analysis`` (the section) and the host and service row action ``ai-assistant-root-cause-analysis`` (ui migration 0178). A user who holds any of them can create, poll and read the analysis. The legacy URL is granted by that action and by the ``ai-assistant`` widget. A user without the grant gets ``403``. * The Realm breadcrumb trigger is shown only for an Observability Realm and requires ``ai_assistant_root_cause_analysis``. The Host and Service shortcuts also require the existing ``ai-assistant-root-cause-analysis`` action and are disabled when that action is not granted. The history table requires the ``root-cause-analysis`` widget and its list endpoint. These visibility checks do not add server-side authorization per target. * The requests must be made by the user interface (AJAX, with the header ``X-Requested-With: XMLHttpRequest``) from a logged-in session; otherwise the backend answers ``400`` or a redirect to the login. * An error response is always ``{"error": ""}``, with a text already in the language of the user (except ``Invalid JSON body``). Create an analysis ~~~~~~~~~~~~~~~~~~ ``POST`` on ``observability_root_cause_analyses`` with a JSON body. By default, or with ``"wait": false``, the response is ``202`` and the analysis runs in the background:: {"id": 12, "status": "pending"} To obtain the result in the same request, include ``"wait": true``. This mode runs the stored analysis directly, without queuing a background task, and returns HTTP ``200`` with the full detail shape described below, including ``result``, ``message``, ``status`` and ``error_message``. A failed analysis also returns its stored detail with ``status: "failed"`` and a safe, localized ``error_message``; submission errors still use their usual HTTP status. No additional detail request is required. ``wait`` must be a JSON boolean; strings, numbers and null are rejected with ``400`` before an analysis is created. It controls execution only and is not stored in the analysis request. Body of a **host** analysis (``target`` may be left out: it defaults to ``host``): * ``host_name`` (required): the name of the host in WOCU, not its display name. * Optional: ``host_display_name``, ``host_address``, ``state`` (the Nagios state number of the host: 0 up, 1 down, 2 unreachable), ``trigger_state`` (the label of the state as the user sees it, at most 64 characters), ``plugin_output`` (text of the last check, at most 10000 characters), ``business_impact``, ``last_state_change`` and ``last_check`` (epoch seconds), ``connector_id``. When ``host_display_name`` is not sent, WOCU reads it from monitoring with one query limited to a single host and column and to 15 seconds, and keeps the technical name if it cannot be read. Body of a **service** analysis: the same fields with ``"target": "service"`` and, in addition, ``service_description`` (required) and the optional ``service_display_name``. The ``state`` is then the service state number (0 OK, 1 warning, 2 critical, 3 unknown). The correlation is made on the host of the service. Body of a **realm** analysis: ``{"target": "realm"}``, with an optional ``connector_id``. Any other field, such as a host or service field, is ignored. The analysis uses the same limits of the AI assistant as the others. Body of a **hostgroup** analysis: ``target`` is ``hostgroup`` and ``hostgroup_name`` is the required WOCU group name, not a display label. The group must exist in the source Realm. ``connector_id`` is optional; Host and Service fields are discarded. The request modal sends: .. code-block:: json {"target": "hostgroup", "hostgroup_name": "QA-HOSTGROUP", "wait": true} All the text fields are trimmed and cut to 255 characters (``plugin_output`` to 10000). ``state``, ``last_check`` and ``last_state_change`` are kept as they are only when they are plain numbers; any other value is converted to text and cut to 255 characters. A ``state``, ``last_check`` or ``last_state_change`` that is not a finite number (``NaN`` or infinity) is refused with ``400`` (``'' must be a finite number``). Any other key of the body is dropped. The synchronous endpoint (``ai_assistant_root_cause_analysis``) shares this body and accepts the targets ``host``, ``service``, ``realm`` and ``hostgroup``. It waits for the analysis to finish and returns the result in the same response. Its response shape is unchanged. The request modals use this endpoint. Errors of the creation (``error`` text localized): .. list-table:: :header-rows: 1 :widths: 10 90 * - Status - Cause * - 400 - Body that is not valid JSON (``Invalid JSON body``) or not an object, ``target`` that is not ``host``, ``service`` or ``realm``, missing ``host_name``, missing ``service_description`` for a service, ``state``, ``last_check`` or ``last_state_change`` that is not a finite number. * - 403 - The user lacks the endpoint or the realm. * - 404 - The realm does not exist, has no Observability, or it is a multirealm. * - 503 - The analysis could not be queued. The body is ``{"error": "...", "id": 12}`` and the stored row is already ``failed``, so it can be read with the detail endpoint; request a new analysis later. The limits of the AI assistant (monthly tokens of the realm or the user, or the assistant disabled for the user) are not checked when the analysis is created, but when it runs: they appear as a failed analysis (see below). Asynchronous flow ~~~~~~~~~~~~~~~~~ 1. ``POST`` returns ``202`` with ``id`` and ``status: "pending"``. 2. The client polls ``GET`` on ``observability_root_cause_analysis_detail`` every few seconds, until ``status`` is ``completed`` or ``failed``. 3. Statuses: ``pending`` (stored and waiting for a worker), ``running`` (a worker is analysing it), ``completed`` (finished: ``result`` and ``message`` are available) and ``failed`` (it could not be completed: read ``error_message``). ``error_message`` is empty unless the analysis failed. When it failed, it is a text localized in the language of the user who requested it, safe to show as it is: it never carries technical details. Most failures say that the analysis could not be completed and that the user should try again in a few minutes. If the limits of the AI assistant stopped it (assistant disabled for the user, monthly limit of the realm or of the user), the error message is the message of that limit instead. The task of the analysis has a soft limit of 900 s and a hard limit of 960 s. This asynchronous API flow is separate from the synchronous endpoint that the request modals use. A row no longer stays in ``pending`` or ``running`` forever if the worker died: a periodic task (``AutomatedStaleRootCauseAnalysisReaper``, every 5 minutes) marks the analyses that are still unfinished after ``WOCU_RCA_STALE_AFTER_SECONDS`` (default 1200 s, 20 minutes) as ``failed``, with the generic ``error_message``. A new analysis can always be requested. List analyses ~~~~~~~~~~~~~ ``GET`` on ``observability_root_cause_analyses``. All the authorized users of the realm see every analysis of the realm, with the user who requested each one in ``requested_by``. Query parameters (all optional): .. list-table:: :header-rows: 1 :widths: 25 75 * - Parameter - Meaning * - ``limit`` - Page size: default 20, maximum 100 (a larger value is reduced to 100); an integer greater than 0. An empty value, ``none`` or ``null`` is not an integer and answers ``400``. * - ``offset`` - Rows skipped: default 0, not negative. An empty value, ``none`` or ``null`` answers ``400``. * - ``ordering`` - ``created_at`` (default ``-created_at``), ``finished_at``, ``host_name``, ``service_description``, ``status`` or ``root_cause_found``; a ``-`` prefix is descending. Rows with an equal value are ordered by ``id`` in the same direction, so the pages are stable. * - ``search`` - Case-insensitive text found in ``host_name``, ``host_display_name``, ``service_description``, ``service_display_name`` or ``summary``. * - ``target`` - ``host``, ``service``, ``realm`` or ``hostgroup``; several values separated by commas are accepted. * - ``status`` - ``pending``, ``running``, ``completed`` or ``failed``; several values separated by commas are accepted. * - ``root_cause_found`` - ``true`` or ``false``. * - ``host_name`` - Case-insensitive match on any part of the WOCU host name, so ``wocu`` finds ``wocu-devel``. An empty value is ignored, as if the parameter was not sent. * - ``service_description`` - Exact match. An empty value is ignored. * - ``hostgroup_name`` - Exact match on the Hostgroup name. An empty value is ignored. * - ``time_from`` - Created at or after: epoch seconds or ISO 8601. * - ``time_to`` - Created at or before: epoch seconds or ISO 8601. An invalid value (unknown choice or ordering, non-integer or empty ``limit`` or ``offset``, bad date, ``time_from`` later than ``time_to``) answers ``400`` with ``{"error": "..."}``. An empty ``search`` is ignored. The response is:: {"total": 57, "count": 20, "limit": 20, "offset": 0, "time_from": null, "time_to": null, "results": [ ... ]} * ``total``: the number of analyses that match the filters, in every page. * ``count``: the number of rows of this page (``len(results)``). * ``limit`` and ``offset``: the values applied. * ``time_from`` and ``time_to``: the window of the request in epoch milliseconds, or ``null`` when it was not sent. * ``results``: the rows. A row has the fields of the detail, without ``result`` and ``message``: read the detail of an analysis to get them. The ``host_name`` filter matches by case-insensitive substring, so ``host_name=web-1`` also matches ``web-10``: ``limit=1`` can return the row of another host. To find the **latest analysis of one exact host**, ask for ``?target=host&host_name=&ordering=-created_at``, page through the results (newest first, with ``offset`` and ``limit``) and keep the first row whose ``host_name`` equals the host exactly. For the latest analysis of a **service**, do the same with ``?target=service&host_name=&service_description=&ordering=-created_at`` and compare the ``host_name`` of each row. Add ``&status=completed`` for the latest one that has a result. Without ``target``, the filter by ``host_name`` also returns the analyses of the services of that host. If no row has the exact ``host_name``, the object has no analysis yet. Analysis detail ~~~~~~~~~~~~~~~ ``GET`` on ``observability_root_cause_analysis_detail``. An analysis of another realm, an id that does not exist or an id that is not a number answers ``404``. The fields (the list rows carry all of them but ``result`` and ``message``): ========================= ============================================================ Field Content ========================= ============================================================ ``id`` Integer id of the analysis. ``target`` ``host``, ``service``, ``realm`` or ``hostgroup``. ``host_name`` WOCU name of the host; empty for a realm or hostgroup analysis. ``host_display_name`` Visible name of the host. The technical name when none was sent and monitoring could not provide it; empty for a realm or hostgroup analysis. ``service_description`` Service of a service analysis, else empty. ``service_display_name`` Visible name of the service, else empty. ``hostgroup_name`` WOCU name of the hostgroup of a hostgroup analysis, else empty. ``trigger_state`` Label of the state that triggered it, as the user saw it (for example ``Down``); empty for a realm or hostgroup analysis. ``anchor_at`` ISO 8601 date of the last state change of the object, else of its last check; ``null`` when unknown. It identifies the incident for the reuse of a host or service analysis. ``status`` ``pending``, ``running``, ``completed`` or ``failed``. ``root_cause_found`` ``true`` / ``false`` once completed; ``null`` before. For a realm analysis it is ``true`` when ``root_causes`` is not empty. ``summary`` Text of ``result.summary`` (empty until completed). ``error_message`` See the asynchronous flow; empty unless failed. ``requested_by`` ``{"username", "full_name"}`` of the user who requested it, or ``null`` if that user was deleted. ``conversation_id`` Id of the AI assistant conversation of the run, so the user can continue the investigation in the assistant; ``null`` when none was created. ``created_at`` ISO 8601: the request. ``started_at`` ISO 8601: a worker took it; ``null`` while pending. ``finished_at`` ISO 8601: completed or failed; ``null`` until then. ``result`` Detail only. The structured result (below), ``null`` unless completed. ``message`` Detail only. The result rendered as markdown, in the language of the requester, with the dates in the time zone of the project. Request modals and the current detail panel render it as Markdown. The list uses ``summary`` and does not include the full report. Empty unless completed. ========================= ============================================================ All the dates of the response (``anchor_at``, ``created_at``, ``started_at``, ``finished_at``) are ISO 8601 strings with an explicit UTC offset (``2026-11-05T09:12:03.512344+00:00``), because the project stores them in UTC (``USE_TZ = True``; ``TIME_ZONE`` is ``Europe/Madrid``). The client converts them to the zone of the user. The dates inside ``result`` end in ``Z`` (UTC). The only place where the backend writes dates in the project zone is the ``message`` markdown. Result of a host or service analysis ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ``result.kind`` is ``"host"`` for host and service analyses, and ``result.schema_version`` is ``1``. Both are added by the server; the assistant never writes them. Branch on ``kind`` to pick the layout and on ``schema_version`` to know the shape: a change that is not backwards compatible raises the version. A client that does not recognise the ``kind`` or the ``schema_version`` of a result must not read the other fields of ``result``. It shows the ``summary`` of the analysis as the headline and the ``message`` of the detail, which is the markdown the server renders from the stored result, in the language of the requester. * ``kind``, ``schema_version``. * ``summary`` (string): one or two sentences: what happens and, at root, why. * ``root_cause``: ``found`` (boolean), ``description`` (string, what was checked when not found), ``confidence`` (see the enumerations), ``started_at`` (ISO 8601 UTC of the earliest causal event, or ``null`` when ``found`` is false). * ``timeline``: at most 10 entries, oldest first, possibly empty. Each: ``at`` (ISO 8601 UTC), ``source``, ``description``, ``role``. * ``causal_chain``: at most 5 links ordered from cause to effect, possibly empty. Each: ``description``, ``confidence``. * ``impact`` (string, possibly empty). * ``recommended_actions``: at most 3 strings, from the quickest to the most expensive, possibly empty. * ``analysis_window``: ``{from, to}`` (ISO 8601 UTC) of the analysed range, or ``null``. ``causal_chain``, ``impact``, ``recommended_actions`` and ``analysis_window`` are always present in the stored result (empty or ``null`` if the assistant left them out); ``summary``, ``root_cause`` and ``timeline`` always have data. Result of a realm analysis ~~~~~~~~~~~~~~~~~~~~~~~~~~ ``result.kind`` is ``"realm"`` and ``result.schema_version`` is ``1``. A hostgroup analysis returns this same structure, about the member hosts of the hostgroup: its ``overview`` holds only the ``hosts`` counters of those members. * ``kind``, ``schema_version``. * ``overview``: the counters measured by WOCU when the analysis ran. The assistant never writes them: WOCU adds them after validating the answer. **Any sub-object may be missing** (a source that could not be read is left out), and ``overview`` itself can be ``{}``. Do not assume them. * ``hosts``: ``up``, ``down``, ``unreachable``, ``pending`` (integers). * ``services``: ``ok``, ``warning``, ``critical``, ``unknown``, ``pending`` (integers). * ``problems``: integer, the active problems (hosts down or unreachable plus services in warning, critical or unknown). It is only present when the hosts and the services counters are both present. * ``summary`` (string): two or three sentences that answer the three questions. * ``what_happened``: at most 10 entries, oldest first. Each: ``at`` (ISO 8601 UTC), ``description``, ``role``. * ``root_causes``: at most 5, the most important first, possibly empty. Each: ``description``, ``affected_hosts`` (integer, 0 if unknown), ``confidence``. * ``outlook``: what may happen next; at most 5 entries, possibly empty (when there is not enough evidence). It is built from the anomalies detected in the last 24 hours and only exists on standard realms with an Elasticsearch 9 observability backend. Each: ``description``, ``likelihood``, ``affected`` (free text naming the hosts or services concerned, an empty string if unknown). * ``recommended_actions``: at most 3 strings, possibly empty. * ``analysis_window``: ``{from, to}`` or ``null``. WOCU gathers the context before asking the assistant: the counters, the current problems, the hosts down or unreachable grouped by the parent host that is down (at most 50 hosts are inspected, and the analysis says so when there are more), the events of the last 6 hours and the anomalies of the last 24 hours. A source that cannot be read is left out. For a hostgroup analysis, the hosts down or unreachable are the members of the hostgroup. When a member is blocked by a parent host outside the hostgroup, WOCU also reads the state of that parent and of the hosts above it (up to 3 levels and 50 hosts, in one read per level) so the analysis can name the parent that is down as the cause. The counters still cover the member hosts only, and the analysis says when that read was cut or could not be done. Realm-level button ~~~~~~~~~~~~~~~~~~ The **Realm RCA** breadcrumb button sends ``{"target": "realm", "wait": true}`` to ``ai_assistant_root_cause_analysis``. It makes one POST, displays a spinner while waiting and renders the returned ``message`` (or ``summary``) on completion. Failures display ``error_message``. There are no repeated detail requests or automatic closing. Host, Service and Hostgroup modals use the same endpoint and waiting behavior. The **View root cause analyses** button opens the Realm table with ``?target=realm``. Host and Service modals navigate with ``target=host`` or ``target=service`` and ``host_name``; Service requests also include ``service_description``. Hostgroup requests use ``target=hostgroup`` and ``hostgroup_name``. Values are URL-encoded. The table applies these query parameters to its existing filters, clears saved search and other filters, keeps column preferences and starts from the first page. Users can then change or clear the filters normally. Navigation closes the modal and ends the browser's wait without deleting an existing stored analysis. The rendered Realm result includes these answers when data is available: * **How many hosts went down**: ``result.overview.hosts.down``, and ``result.overview.hosts.unreachable`` for the hosts that cannot be reached because a parent is down. If ``overview.hosts`` is missing, show no number (never zero). ``root_causes[].affected_hosts`` tells how many hosts each cause affects. * **What happened**: ``result.what_happened`` (with ``result.summary`` as the headline). * **What may happen next**: ``result.outlook``. The current Markdown renderer omits this section when the list is empty; an empty list does not provide a prediction. The analyses of the realm are found with ``target=realm`` in the list. Enumerations ~~~~~~~~~~~~ The backend never translates these values: the ``source``, ``role``, ``confidence``, ``likelihood``, ``status``, ``target`` and ``kind`` values are stable codes and the frontend translates them for display. The free texts (``summary``, ``description``, ``impact``, ``recommended_actions``, ``affected``, ``error_message``) are already written in the language of the user who requested the analysis. ====================== =========================== =========================================== Field Value Meaning ====================== =========================== =========================================== ``kind`` ``host`` Result of a host or service analysis. ``kind`` ``realm`` Result of a realm or hostgroup analysis. ``target`` ``host`` Analysis of a host. ``target`` ``service`` Analysis of a service (``result.kind`` is ``host``). ``target`` ``realm`` Analysis of the whole realm. ``target`` ``hostgroup`` Analysis of a hostgroup (``result.kind`` is ``realm``). ``status`` ``pending`` Waiting for a worker. ``status`` ``running`` Being analysed. ``status`` ``completed`` Finished with a ``result``. ``status`` ``failed`` Finished without a ``result``. ``source`` ``metrics`` Time series of the host (CPU, memory, disk, network). ``source`` ``logs`` System and application logs. ``source`` ``monitoring`` State changes and checks of WOCU. ``source`` ``network_alerts`` SNMP traps received from devices. ``source`` ``wocu`` What WOCU itself knows (hierarchy, status, host groups, IPAM). ``role`` ``root_cause`` The first causal event. ``role`` ``consequence`` An event that follows from the root cause. ``role`` ``context`` Relevant event that is not causal. ``confidence`` ``confirmed`` Backed by data that was read. ``confidence`` ``probable`` Reasoned inference on partial evidence. ``confidence`` ``possible`` Conjecture without direct evidence. ``likelihood`` ``confirmed`` Already happening or certain. ``likelihood`` ``probable`` Reasonably expected. ``likelihood`` ``possible`` Conceivable, without direct evidence. ====================== =========================== =========================================== ``source`` is only used in the timeline of a host or service result. ``role`` is used in ``timeline`` and ``what_happened``. ``confidence`` is used in ``root_cause``, ``causal_chain`` and ``root_causes``, and ``likelihood`` in ``outlook``. All the free texts are plain text: they carry no markdown or HTML and must be rendered as text (escaped). They never name the platform, the data sources, fields or queries. Only ``message`` is markdown. Examples ~~~~~~~~ Detail of a completed host analysis (``GET .../observability/root-cause-analyses/12``): .. code-block:: text { "id": 12, "target": "host", "host_name": "srv-web01", "host_display_name": "Web server 01", "service_description": "", "service_display_name": "", "trigger_state": "Down", "anchor_at": "2026-11-05T08:48:57+00:00", "status": "completed", "root_cause_found": true, "summary": "The web server is down because its upstream router lost power.", "error_message": "", "requested_by": {"username": "ana", "full_name": "Ana Gil"}, "conversation_id": 45, "created_at": "2026-11-05T09:12:03.512344+00:00", "started_at": "2026-11-05T09:12:04.101200+00:00", "finished_at": "2026-11-05T09:12:41.870015+00:00", "result": { "kind": "host", "schema_version": 1, "summary": "The web server is down because its upstream router lost power.", "root_cause": { "found": true, "description": "The upstream router stopped answering.", "confidence": "confirmed", "started_at": "2026-11-05T08:40:00Z" }, "timeline": [ { "at": "2026-11-05T08:40:00Z", "source": "network_alerts", "description": "Link down alert from the upstream router.", "role": "root_cause" }, { "at": "2026-11-05T08:48:57Z", "source": "monitoring", "description": "The host stopped answering its checks.", "role": "consequence" } ], "causal_chain": [ {"description": "The router lost power.", "confidence": "confirmed"}, {"description": "The host became unreachable.", "confidence": "probable"} ], "impact": "Three services are unavailable.", "recommended_actions": ["Check the power of the upstream router."], "analysis_window": { "from": "2026-11-05T07:48:57Z", "to": "2026-11-05T09:03:57Z" } }, "message": "## Summary\n..." } Detail of a completed realm analysis: .. code-block:: json { "id": 13, "target": "realm", "host_name": "", "host_display_name": "", "service_description": "", "service_display_name": "", "trigger_state": "", "anchor_at": null, "status": "completed", "root_cause_found": true, "summary": "Fifteen hosts are down because the core switch failed.", "error_message": "", "requested_by": {"username": "ana", "full_name": "Ana Gil"}, "conversation_id": 46, "created_at": "2026-11-05T09:20:00.100000+00:00", "started_at": "2026-11-05T09:20:00.900000+00:00", "finished_at": "2026-11-05T09:20:38.400000+00:00", "result": { "kind": "realm", "schema_version": 1, "overview": { "hosts": {"up": 240, "down": 3, "unreachable": 12, "pending": 1}, "services": {"ok": 900, "warning": 20, "critical": 7, "unknown": 2, "pending": 0}, "problems": 44 }, "summary": "Fifteen hosts are down because the core switch failed. The storage cabinet may fill up next.", "what_happened": [ { "at": "2026-11-05T08:40:00Z", "description": "The core switch stopped answering.", "role": "root_cause" }, { "at": "2026-11-05T08:42:00Z", "description": "The hosts behind it became unreachable.", "role": "consequence" } ], "root_causes": [ { "description": "Core switch failure.", "affected_hosts": 15, "confidence": "probable" } ], "outlook": [ { "description": "Disk usage keeps growing and may fill the volume.", "likelihood": "possible", "affected": "storage-01" } ], "recommended_actions": ["Restore the core switch."], "analysis_window": { "from": "2026-11-05T03:20:00Z", "to": "2026-11-05T09:20:00Z" } }, "message": "## Summary\n..." } List response (``GET .../observability/root-cause-analyses?limit=2``), where the second row failed: .. code-block:: json { "total": 57, "count": 2, "limit": 2, "offset": 0, "time_from": null, "time_to": null, "results": [ { "id": 13, "target": "realm", "host_name": "", "host_display_name": "", "service_description": "", "service_display_name": "", "trigger_state": "", "anchor_at": null, "status": "completed", "root_cause_found": true, "summary": "Fifteen hosts are down because the core switch failed.", "error_message": "", "requested_by": {"username": "ana", "full_name": "Ana Gil"}, "conversation_id": 46, "created_at": "2026-11-05T09:20:00.100000+00:00", "started_at": "2026-11-05T09:20:00.900000+00:00", "finished_at": "2026-11-05T09:20:38.400000+00:00" }, { "id": 11, "target": "service", "host_name": "srv-web01", "host_display_name": "Web server 01", "service_description": "load", "service_display_name": "CPU load", "trigger_state": "Critical", "anchor_at": "2026-11-05T07:55:10+00:00", "status": "failed", "root_cause_found": null, "summary": "", "error_message": "The analysis could not be completed. Try again in a few minutes.", "requested_by": {"username": "ana", "full_name": "Ana Gil"}, "conversation_id": null, "created_at": "2026-11-05T08:00:00.000000+00:00", "started_at": "2026-11-05T08:00:01.000000+00:00", "finished_at": "2026-11-05T08:00:30.000000+00:00" } ] } The legacy synchronous endpoint ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ``ai_assistant_root_cause_analysis`` (``POST``) is used by the Realm, Host, Service and Hostgroup request modals and by existing synchronous clients. It waits for the analysis, stores it as the others and answers ``{"id", "conversation_id", "kibana_conversation_id", "title", "status", "result", "message"}``, where ``status`` is ``completed`` or ``failed`` and, when it failed, ``result`` is ``null`` and ``message`` the user-facing error. A request that reused an analysis still in progress and did not see it finish in time is answered with ``504`` and an ``error`` text instead, like an analysis that runs out of time (see *Reuse of a recent analysis*). A completed ``result`` has the same structure, with ``kind`` and ``schema_version``. The errors of the AI assistant limits are answered with their own status (``403``, ``429``). The endpoint accepts Host, Service, Realm and Hostgroup targets. The limits of the AI assistant apply to the user who requested the analysis.