Monitoring¶

For the simplest cases the OCC command provides talk:monitor:calls to get the list of the conversations that have an active call as well as their participant count, and talk:monitor:room to get more details for a specific conversation, like the number of attendees, how many are active and how many are in the call.

For a more comprehensive monitoring please see below.

High Performance Backend¶

The High Performance Backend is composed by the signaling server, which is used to send messages to the participants in a conversation (messages as in the IP address of the other end to establish a connection or events about changes in the settings of a conversation to update it, not only chat messages) and that all clients in a conversation connect to, and Janus, the WebRTC gateway, which is used to exchange the actual media during calls (audio, video and screen shares).

Signaling server¶

The signaling server exposes several metrics that can be queried with Prometheus: https://nextcloud-spreed-signaling.readthedocs.io/en/latest/prometheus-metrics/

Note that the endpoint can be queried only from the IPs allowed in the configuration (which, by default, only allows 127.0.0.1). Therefore, if Prometheus is not running in the same machine as the signaling server you would need to explicitly allow its IP address in stats->allowed_ips in the signaling server configuration.

The signaling server metrics also provide some stats related to the media, like the current number of publishers and subscribers (that is, participants sending and receiving audio/video/screen shares). Starting with version 2.1.0 of the signaling server those metrics also include the current bandwidth, total number of slow link events, total number of lost media packets... In previous versions that detailed information had to be directly queried from Janus instead.

In order to provide the WebRTC metrics through the signaling server some settings need to be adjusted in Janus configuration, although this should be automatically done when installing the High Performance Backend from the repository, which is only available to customers.

Janus¶

Janus provides an admin API that can be used to get information about the different sessions in use. This is a poll-based protocol; it also provides an event system to be notified when something changes (which is what the signaling server uses internally to get that information).

Unfortunately Janus does not provide a Prometheus endpoint, but the events can be sent using HEPipe.js to a HOMER instance with a Prometheus endpoint. However, Janus is not aware in any way of signaling sessions or Talk conversations (each publisher has its own Janus videoroom, and the handlers of publishers and subscribers are specific to Janus and unrelated to the signaling sessions), so there is no easy way to correlate the events with them.

The Janus handlers and the signaling sessions could be correlated by parsing the signaling logs, but even in that case it could be tricky because the Janus handles and signaling sessions are split in two different messages (for example, in the case of subscribers, first _Attached subscriber to room ROOM_ID of publisher PUBLISHER_ID in plugin PLUGIN_ID in session SESSION_ID as HANDLE_ID is printed and then _Subscribing STREAM_TYPE from PUBLISHER_ID as INTERNAL_ID in session SUBSCRIBER_ID is printed, but other messages can be printed in between and prevent linking both messages). Due to that, it may not be possible to get a finer grained picture with the values segregated by specific calls and instead they would need to be aggregated as total values.

Besides using version 2.1.0 or later of the signaling server as described above, if the values to be monitored are just the upload and download bandwidth used by calls it would be much easier to simply monitor the network interfaces using the node exporter from Prometheus. Although the network would be used by other processes as well (for example, the signaling server or the signaling proxy) most of the bandwidth would be typically consumed by the media in the calls (assuming of course that the machine is used only for the High Performance Backend).

TURN server¶

coturn provides a Prometheus endpoint since version 4.5.2, with improvements in later versions (like being able to configure the listening port since 4.6.0). Please keep in mind that Prometheus support needs to be enabled at compile time, so it may or may not be available if you are using a prebuilt package. You can verify if coturn has Prometheus support enabled by running turnserver --help 2>&1 | grep -- --prometheus and checking if the option is listed.

Note, however, that even in the latest coturn version Prometheus metrics are updated only at the end of a session. That is, they are not updated in real time while the media is being relayed by coturn, but only once the connection is closed. Some time ago work was started to update the stats in real time, but the feature is still being developed.

Recording server¶

The recording server provides a Prometheus endpoint that exposes a few metrics about the recordings.

Hardware resources¶

Independently of the metrics that might be provided by some of the tools described above, it is recommended to monitor the hardware resources in all the servers using a specific tool for that, like the node exporter from Prometheus. In some cases it might be interesting to monitor specific processes, for example with the process exporter from Prometheus community, to know for example the CPU usage of the signaling server and Janus independently one of the other if both are running in the same machine.