recover CactiEZ
公司网络设备监控用得是一台kvm的虚机,上面装的是CactiEZ,结果机房搬家,匆忙中这台虚机没有halt,直接关掉了实体机。麻烦就来了。
搬家完毕后kvm启动,然后登陆web界面的时候死活登录不上去,很郁闷啊。看看httpd的error.log,首先是一堆时区错误,先改掉:
编辑/etc/php.ini
date.timezone = Asia/Shanghai
时区不报错了,报告申请内存错
PHP Fatal error: Allowed memory size of 1073741824 bytes exhausted
见鬼了,把内存增加到1G,2G,3G,4G,8G,top看进程都是逐渐被吃光,干脆不限制:
memory_limit = -1
问题依旧,没办法,注意到top看进程的时候,httpd一直在吃内存,而mysql的cpu占用率达到60%。
于是进去mysql,看看是什么进程在跑:
mysql> show processlist;
+----+-----------+-----------+-------+---------+------+----------------+-----------------------------------------------------------------+
| Id | User | Host | db | Command | Time | State | Info |
+----+-----------+-----------+-------+---------+------+----------------+-----------------------------------------------------------------+
| 37 | cactiuser | localhost | cacti | Query | 0 | Opening tables | select value from `cacti`.`settings` where name='log_verbosity' |
| 43 | root | localhost | NULL | Query | 0 | NULL | show processlist |
+----+-----------+-----------+-------+---------+------+----------------+-----------------------------------------------------------------+
2 rows in set (0.00 sec)
执行上面的sql语句,结果报错:
mysql> select value from `cacti`.`settings` where name='log_verbosity';
ERROR 145 (HY000): Table './cacti/settings' is marked as crashed and should be repaired
真相大白,肯定是暴力关机导致mysql的库有部分损坏,得修复一下吧:
cd /var/lib/mysql/cacti
for i in `ls -1 *.MYI`;do myisamchk -r $i;done
终于修好了,好在CactiEZ这个集成环境用得是MyISAM,没用到innodb,否则就修不好了。