Troubleshooting Memory Consumption¶

Bugs like memory leaks can exhaust the available memory of a Nextcloud server process and make it crash hard. This document describes possible ways and tools to get insights of the problem.

PHP memprof Extension¶

Programs that crash with memory exhaustion can be analyzed with the memprof extension. Please see the maintainer's Github readme for installation instructions.

Warning: When the extension is enabled it doesn't add any overhead by default, but profiling can be enabled by anyone. Therefore we advice against enabling the extension permanently in production. A nonpriviledged user could start profiling with a HTTP request and cause additional server load, and in the worst case a denial of service attack.

Profiling can be enabled in two ways

1. Using a environment variable. Useful to enable for CLI commands, cron or globally.
2. Using a URL or body parameter of a HTTP request. Useful for failing HTTP request.

CLI Commands and Cron Jobs¶

You can create a memory profile of a failing occ job like this:

# Base usage for occ when the extension is not enabled globally (recommended)
MEMPROF_PROFILE=dump_on_limit php -dextension=memprof.so -f occ …
# Base usage for occ when the extension is enabled globally
MEMPROF_PROFILE=dump_on_limit php -f occ …

# Example: user info
MEMPROF_PROFILE=dump_on_limit php -dextension=memprof.so -f occ user:info admin

# Example: cron
MEMPROF_PROFILE=dump_on_limit php -dextension=memprof.so -f cron.php

Here's an example for an occ invocation that we let fail intentionally by giving it way too little memory:

MEMPROF_PROFILE=dump_on_limit php -d memory_limit=5M -f occ user:info admin

This will fail with

PHP Fatal error:  Allowed memory size of 5242880 bytes exhausted (tried to allocate 4096 bytes) (memprof dumped to /tmp/memprof.callgrind.1831474847411406) in /var/www/nextcloud/3rdparty/composer/autoload_static.php on line 3803

The memprof.callgrind.* file can be opened for inspection with kcachegrind, or sent to Nextcloud support for assisted analysis.

Web Requests¶

Web requests are a bit harder to trigger. The easiest way is to trigger them via cURL:

curl https://cloud.example.com/login -d MEMPROF_PROFILE=dump_on_limit

The profile will be written to /tmp. Web servers running with systemd might use a scoped subdirectory of /tmp.

You can also monitor HTTP requests in the developer tools of Firefox, wait for the HTTP 500 status of the request that runs out of memory, and right-click to Copy Value > Copy as cURL. Append -d MEMPROF_PROFILE=dump_on_limit to send the memprof trigger along the request.

PHP Xdebug Extension¶

The Xdebug extension can create a profile of a PHP process for analysis. Please see the official website for installation instructions.

Xdebug has a noticeable performance overhead. We advice to only enable it selectively.

CLI Commands and Cron Jobs¶

You can enable the Xdebug extension dynamically and switch the mode to profiling:

php -d zend_extension=xdebug.so -d xdebug.mode=profile occ user:info admin

A file called cachegrind.out.*.gz will be written to /tmp. It can be opened for inspection with kcachegrind, or sent to Nextcloud support for assisted analysis.

All Web Requests¶

For web requests you have to enable Xdebug via php.ini:

zend_extension=xdebug.so
xdebug.mode=profile
xdebug.start_with_request=yes

Every request will be profiled. The file creation timestamp should help you identify the relevant one.

Specific Web Requests¶

As a more targeted alternative you can start Xdebug only for certain requests:

zend_extension=xdebug.so
xdebug.mode=profile
xdebug.start_with_request=trigger
xdebug.trigger_value=nextcloudsupport

Arbitrary requests will no longer be profiled. Profiling only happens when you send the trigger value as URL or body parameter:

curl https://cloud.example.com/login -d XDEBUG_TRIGGER=nextcloudsupport

A file called cachegrind.out.*.gz will be written to /tmp. It can be opened for inspection with kcachegrind, or sent to Nextcloud support for assisted analysis.