Yes, it’s required. We should try to get slurmdbd running, then slurmctld, then slurm. To see the reason a daemon failed to start you can use e.g.:
journalctl -u slurmdbd --since boot
Note that our installer deploys some configuration files for all these. In our configuration, slurmdbd requires a MariaDB database (which the installer normally sets up).
Assuming everything was installed properly, MariaDB doesn’t like system crashes and power failures, so if you had one you’ll need to type in a couple of commands to repair the database.
● mariadb.service - MariaDB database server
Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; vendor preset: disabled)
Active: active (running) since Sat 2017-12-23 10:40:46 +08; 1 weeks 4 days ago
But slurmdbd is not:
`Job for slurmdbd.service failed because a timeout was exceeded. See "systemctl status slurmdbd.service" and "journalctl -xe" for details.`
-- Logs begin at Thu 2017-12-21 18:34:07 +08, end at Wed 2018-01-03 19:43:55 +08. --
Jan 03 17:18:48 ESSCGeo-Sen2Agri systemd[1]: Starting Slurm DBD accounting daemon...
Jan 03 17:20:18 ESSCGeo-Sen2Agri systemd[1]: slurmdbd.service start operation timed out. Terminating.
Jan 03 17:20:18 ESSCGeo-Sen2Agri systemd[1]: Failed to start Slurm DBD accounting daemon.
Jan 03 17:20:18 ESSCGeo-Sen2Agri systemd[1]: Unit slurmdbd.service entered failed state.
Jan 03 17:20:18 ESSCGeo-Sen2Agri systemd[1]: slurmdbd.service failed.
Jan 03 17:22:35 ESSCGeo-Sen2Agri systemd[1]: Starting Slurm DBD accounting daemon...
Jan 03 17:24:05 ESSCGeo-Sen2Agri systemd[1]: slurmdbd.service start operation timed out. Terminating.
Jan 03 17:24:05 ESSCGeo-Sen2Agri systemd[1]: Failed to start Slurm DBD accounting daemon.
Jan 03 17:24:05 ESSCGeo-Sen2Agri systemd[1]: Unit slurmdbd.service entered failed state.
Jan 03 17:24:05 ESSCGeo-Sen2Agri systemd[1]: slurmdbd.service failed.
Jan 03 19:38:25 ESSCGeo-Sen2Agri systemd[1]: Starting Slurm DBD accounting daemon...
Jan 03 19:39:55 ESSCGeo-Sen2Agri systemd[1]: slurmdbd.service start operation timed out. Terminating.
Jan 03 19:39:55 ESSCGeo-Sen2Agri systemd[1]: Failed to start Slurm DBD accounting daemon.
Jan 03 19:39:55 ESSCGeo-Sen2Agri systemd[1]: Unit slurmdbd.service entered failed state.
Jan 03 19:39:55 ESSCGeo-Sen2Agri systemd[1]: slurmdbd.service failed.
[2018-01-03T17:19:58.779] error: mysql_real_connect failed: 1045 Access denied for user 'slurm'@'localhost' (using password: YES)
[2018-01-03T17:19:58.779] error: The database must be up when starting the MYSQL plugin. Trying again in 5 seconds.
[2018-01-03T17:20:03.780] error: mysql_real_connect failed: 1045 Access denied for user 'slurm'@'localhost' (using password: YES)
[2018-01-03T17:20:03.780] error: The database must be up when starting the MYSQL plugin. Trying again in 5 seconds.
[2018-01-03T17:20:08.780] error: mysql_real_connect failed: 1045 Access denied for user 'slurm'@'localhost' (using password: YES)
[2018-01-03T17:20:08.780] error: The database must be up when starting the MYSQL plugin. Trying again in 5 seconds.
[2018-01-03T17:20:13.781] error: mysql_real_connect failed: 1045 Access denied for user 'slurm'@'localhost' (using password: YES)
[2018-01-03T17:20:13.781] error: The database must be up when starting the MYSQL plugin. Trying again in 5 seconds.
[2018-01-03T17:22:35.924] error: mysql_real_connect failed: 1045 Access denied for user 'slurm'@'localhost' (using password: YES)
[2018-01-03T17:22:35.924] error: The database must be up when starting the MYSQL plugin. Trying again in 5 seconds.
[2018-01-03T17:22:40.925] error: mysql_real_connect failed: 1045 Access denied for user 'slurm'@'localhost' (using password: YES)
This is only a snippet of the error, but the errors are all the same.
That means that the SLURM database is not set up. You previously mentioned doing a manual installation, that could be the cause.
The install script runs the following (as root):
yum -y install mariadb-server mariadb
systemctl start mariadb
systemctl enable mariadb
mysql_secure_installation # it answers with 'y' on everything, but doesn't change the root password
mysql -u root -p -e "create database slurm_acct_db;create user slurm@localhost;set password for slurm@localhost = password('sen2agri');grant usage on *.* to slurm;grant all privileges on slurm_acct_db.* to slurm;flush privileges;" # this will ask for the root password, which is empty
But again, there are a lot of other steps involved in the SLURM (and system) setup, which is why I strongly discourage doing them manually.
Hi,
It seems that I had the same output for the log.
[2018-01-03T19:39:30.601] error: mysql_real_connect failed: 1045 Access denied for user 'slurm'@'localhost' (using password: YES)
[2018-01-03T19:39:30.601] error: The database must be up when starting the MYSQL plugin. Trying again in 5 seconds.
[2018-01-03T19:39:35.602] error: mysql_real_connect failed: 1045 Access denied for user 'slurm'@'localhost' (using password: YES)
[2018-01-03T19:39:35.602] error: The database must be up when starting the MYSQL plugin. Trying again in 5 seconds.
[2018-01-03T19:39:40.602] error: mysql_real_connect failed: 1045 Access denied for user 'slurm'@'localhost' (using password: YES)
I already reset the root password of mysql to a new one. Does this have something to do with the error?
mysql -u root -p
create database slurm_acct_db;
create user slurm@localhost;
set password for slurm@localhost = password('sen2agri');
grant usage on *.* to slurm;
grant all privileges on slurm_acct_db.* to slurm;
flush privileges;
\q
If you get an error saying that the database or user already exist, you can skip to the next line.
