Home » Service wont start due to No space left on device

Service wont start due to No space left on device

  • by

When starting cPanel via command-line, the following is shown:

# /scripts/restartsrv_cpsrvd
Waiting for “cpsrvd” to start ……info [restartsrv_cpsrvd] systemd failed to start the service “cpanel” (The “/usr/bin/systemctl restart cpanel.service --no-ask-password” command (process 27232) reported error number 1 when it ended.): Job for cpanel.service failed because a timeout was exceeded. See "systemctl status cpanel.service" and "journalctl -xe" for details.

…Waiting for cpsrvd,cpaneld,webmaild,whostmgrd to shutdown ... not running.
info [restartsrv_cpsrvd] systemd failed to start the service “cpanel” (The “/usr/bin/systemctl restart cpanel.service --no-ask-password” command (process 27387) reported error number 1 when it ended.): Job for cpanel.service failed because a timeout was exceeded. See "systemctl status cpanel.service" and "journalctl -xe" for details.

…failed.

Cpanel::Exception::Services::StartError
Service Status
 undefined status from Cpanel::ServiceManager::Services::Cpsrvd

Service Error
 (XID xxxxx) The “cpsrvd” service failed to start.

Startup Log
 Jan 01 00:00:00 server.hostname.com restartsrv_cpsrvd[27391]: /usr/local/cpanel/libexec/cpsrvd-dormant: Waiting to bind to /usr/local/cpanel/var/cpwrapd.sock (No space left on device)....
 Jan 01 00:00:00 server.hostname.com restartsrv_cpsrvd[27391]: /usr/local/cpanel/libexec/cpsrvd-dormant: Waiting to bind to /usr/local/cpanel/var/cpwrapd.sock (No space left on device)....
 Jan 01 00:00:00 server.hostname.com restartsrv_cpsrvd[27391]: /usr/local/cpanel/libexec/cpsrvd-dormant: Waiting to bind to /usr/local/cpanel/var/cpwrapd.sock (No space left on device)....
 Jan 01 00:00:00 server.hostname.com restartsrv_cpsrvd[27391]: /usr/local/cpanel/libexec/cpsrvd-dormant: Waiting to bind to /usr/local/cpanel/var/cpwrapd.sock (No space left on device)....
 Jan 01 00:00:00 server.hostname.com restartsrv_cpsrvd[27391]: /usr/local/cpanel/libexec/cpsrvd-dormant: Waiting to bind to /usr/local/cpanel/var/cpwrapd.sock (No space left on device)....
 Jan 01 00:00:00 server.hostname.com restartsrv_cpsrvd[27391]: /usr/local/cpanel/libexec/cpsrvd-dormant: Waiting to bind to /usr/local/cpanel/var/cpwrapd.sock (No space left on device)....
 Jan 01 00:00:00 server.hostname.com systemd[1]: cpanel.service start operation timed out. Terminating.
 Jan 01 00:00:00 server.hostname.com systemd[1]: Failed to start cPanel services.
 Jan 01 00:00:00 server.hostname.com systemd[1]: Unit cpanel.service entered failed state.
 Jan 01 00:00:00 server.hostname.com systemd[1]: cpanel.service failed.

cpsrvd has failed. Contact your system administrator if the service does not automagically recover.

Check disk usage

This is due to no disk space being available on the device cPanel was installed on. You can review available storage with the df command.

# df -h
Filesystem      Size  Used Avail Use% Mounted on
devtmpfs        3.8G     0  3.8G   0% /dev
tmpfs           3.9G     0  3.9G   0% /dev/shm
tmpfs           3.9G  250M  3.6G   7% /run
tmpfs           3.9G     0  3.9G   0% /sys/fs/cgroup
/dev/vda1        50G   50G   20K 100% /
/dev/loop0      1.1G  6.2M  975M   1% /tmp
tmpfs           783M     0  783M   0% /run/user/0

Check inodes usage

Additionally, if your storage capacity is not at 100%, cPanel still may be unable to write files due to your system having too many files. This can be seen based on the number of inodes used on your system.

# df -ih
Filesystem Inodes IUsed IFree IUse% Mounted on
devtmpfs     122K   378  122K    1% /dev
tmpfs        124K     1  124K    1% /dev/shm
tmpfs        124K   757  124K    1% /run
tmpfs        124K     7  124K    1% /sys/fs/cgroup
/dev/vda1    8.2M  347K  7.9M    5% /
/dev/loop0   126K    43  125K    1% /tmp
tmpfs        124K     1  124K    1% /run/user/0

What is an inode?

Inodes or Index Nodes, are fixed-length table entries in a Linux file system, each of which holds information about one specific file (regular files, folders, hard links, etc). An inode stores basic information about a regular file, directory, or other file system objects. Every inode in Linux has a unique number associated with it which can be seen with this command:

ls -i $FILENAME 

For example:

ls -i /etc/passwd
232394 /etc/passwd

One way to run out of space in a filesystem is to use up all your inodes. If many small files are present, this can cause the pool of inodes to be consumed prematurely. Even if you might still have enough free space on your disk, you won’t be able to create new files, because the file system is incapable of allocating an inode number to the new file.

The important information to remember about inodes is that they are always limited in number and once exhausted you cannot simply make more. The only way to increase the number of inodes would be to resize the disk.

For some file systems, inodes are pre-allocated at the time a file system is created. However, for some more modern file systems inodes are created dynamically as they are needed. Either way, their number is always limited regardless of how they are allocated.

The number and the location of all the available inodes for a given disk are saved in a special record inside a file system called a superblock. The number of available/used/free inodes can be seen by running this command against the disk: (Run df -h to see available disks)

df -i /dev/(device)

What to do when the inode usage (IUse%) has exceeded the recommended value?

You might face situations where a process fails due to "insufficient available inodes" or some other similar error messages. The only way to circumvent this issue would be to either resize the disk or to free up inodes by removing unwanted files from the file system. Here we look at the latter option:

First, you need to identify what directories contain more files (AKA inodes):

for i in $DIR*; do echo $i; find $i -maxdepth 12 | wc -l; done

so I suggest you upgrade immediately your server or delete your temporary files 🙂

Tags:

Leave a Reply

Your email address will not be published. Required fields are marked *

4 − 3 =