In Linux, why does Insight run out of memory for no good reason?

Insight is a multi-threaded application. Using multiple threads lets a program run more efficiently and also takes advantage of multiple CPUs and CPU cores. When Insight starts, it prepares a pool of "worker threads", ready to perform calculations and collect results.

In the Linux OS, threads and processes are handled very similarly. There is a default setting that limits the total number a user can have at any one time. The default setting is often very low and not appropriate for a single-user workstation.

Check your limits

These commands should all be run from a terminal window.

  • To find the maximum number of threads and processes a single user can have at any one time: ulimit -a
    • Several pieces of information are returned, look for: max user processes
    • For a single-user workstation or virtual machine, this value can be set to a much larger number (e.g. 100000) without causing issues.
  • To find the current number of running threads -- this number will vary depending on the number of programs and activity on the machine:
    1. ps -elfT | wc -l OR
    2. cat/proc/loadavg
      1. Look at the fourth column (it includes a '/' character).
      • This field shows: "currently executing processes" / "total processes"
      • e.g. 0.72 0.58 0.48 2/867 19334 -> 2 executing processes / 867 total
  • To find the the total limit across all users for the machine:
    • sysctl -a | grep kernel.pid_max

Update the limit

These commands require administrator/root permissions. Be careful when making changes to system files or have your IT administrator do it for you.

  • Set the process limit to a new value. The limit will return to the default when the machine is restarted.
    • ulimit -u 10000

Updating the limit permanently depends on the version of Linux you are using.

 

Note: You must completely log out of the account and log back in for these changes to take effect.

 

If you are using PAM, these settings can be configured in:

  • Configuration file: /etc/security/limits.conf
  • These lines set the limit for all users:
*          soft    nproc     16384
*          hard    nproc     16384

If you are using a RedHat variant, you can try:

  • Configuration file: /etc/security/limits.d/90-nproc.conf
  • These lines set the limit for all users:
*          soft    nproc     16384
*          hard    nproc     16384