Cannot execute the statement. DBRM is read only

3 posts / 0 new
Last post
gao1738
gao1738's picture
Offline
Last seen: 1 month 3 weeks ago
Joined: Jan 2 2014
Junior Boarder

Posts: 10

dennis gao
Cannot execute the statement. DBRM is read only

Hi all,

I install the infinidb with multiple+seperate mode.

I test the failover of infinidb during user operation.
I do the following steps:
1. I shutdown the OAM PM (pm1)
2. then the hot standby PM (pm2) becomes the new OAM PM
3. I do a long update, which will take more than 2 mins
4. during the update, I start the orignial OAM PM (pm1)
5. after about 30 seconds, the pm1 is added in to system as the standby OAM

Now, I am not able to execute any modification sql.

mysql> insert into tt values (2,'2014-2-11',0,1,1,1,1,1,1,1,1,1,1);
ERROR 168 (HY000): Cannot execute the statement. DBRM is read only!

I execute the viewtablelock:
./bin/viewtablelock
There is 1 table lock

Table LockID Process PID Session Txn CreationTime State DBRoots
test.loadtest 1 DMLProc 29387 5 23 Sun Jan 12 17:23:15 2014 LOADING 1,2

I try to clear it:
./bin/cleartablelock 1
BRM error: DBRM is in READ-ONLY mode
Table lock 1 is not cleared.

I restart the whole system, but it is still locked...

The process status of infinidb is correct:
Process Module Status Last Status Change Process ID
------------------ ------ --------------- ------------------------ ----------
ProcessMonitor um1 ACTIVE Sun Jan 12 16:14:54 2014 10474
ServerMonitor um1 ACTIVE Sun Jan 12 17:38:53 2014 708
DBRMWorkerNode um1 ACTIVE Sun Jan 12 17:38:57 2014 768
ExeMgr um1 ACTIVE Sun Jan 12 17:39:14 2014 1982
DDLProc um1 ACTIVE Sun Jan 12 17:39:18 2014 2072
DMLProc um1 ACTIVE Sun Jan 12 17:39:25 2014 2110
mysqld um1 ACTIVE Sun Jan 12 17:39:06 2014 673

ProcessMonitor pm1 ACTIVE Sun Jan 12 17:24:57 2014 16465
ProcessManager pm1 HOT_STANDBY Sun Jan 12 17:38:42 2014 14628
DBRMControllerNode pm1 COLD_STANDBY Sun Jan 12 17:38:59 2014
ServerMonitor pm1 ACTIVE Sun Jan 12 17:39:05 2014 14937
DBRMWorkerNode pm1 ACTIVE Sun Jan 12 17:39:04 2014 14982
DecomSvr pm1 ACTIVE Sun Jan 12 17:39:08 2014 15001
PrimProc pm1 ACTIVE Sun Jan 12 17:39:11 2014 15013
WriteEngineServ

gao1738
gao1738's picture
Offline
Last seen: 1 month 3 weeks ago
Joined: Jan 2 2014
Junior Boarder

Posts: 10

dennis gao
Re: Cannot execute the statement. DBRM is read only

More infomation:

I found following error in the crit.log

Jan 13 10:01:14 host-10-0-32-13 DMLProc[3184]: 14.262335 |0|0|0| C 20 CAL0002: problem with rollback transaction 23and DBRM is setting to readonly and table lock is not released: Error encountered while opening version buffer file oid:dbroot = 0:1 and error message:Input/output error (WriteEngine returns error 1)
Jan 13 10:01:14 host-10-0-32-13 DMLProc[3184]: 14.424883 |0|0|0| C 20 CAL0002: problem with rollback transaction 23and DBRM is setting to readonly and table lock is not released: Can't read DBRM isReadWrite [ DBRM is in READ-ONLY mode]
Jan 13 10:04:19 host-10-0-32-13 controllernode[3821]: 19.381782 |0|0|0| C 29 CAL0000: DBRM: getAllTableLocks(): network error
Jan 13 10:08:44 host-10-0-32-13 DMLProc[6328]: 44.528911 |0|0|0| C 20 CAL0002: problem with rollback transaction 23and DBRM is setting to readonly and table lock is not released: Error encountered while opening version buffer file oid:dbroot = 0:1 and error message:Input/output error (WriteEngine returns error 1)
Jan 13 10:08:44 host-10-0-32-13 DMLProc[6328]: 44.679478 |0|0|0| C 20 CAL0002: problem with rollback transaction 23and DBRM is setting to readonly and table lock is not released: Can't read DBRM isReadWrite [ DBRM is in READ-ONLY mode]
Jan 13 10:27:16 host-10-0-32-13 joblist[8929]: 16.554357 |0|0|0| C 05 CAL0000: distributedenginecomm.cpp @ 370 DEC: lost connection to 10.0.32.16
Jan 13 10:27:16 host-10-0-32-13 joblist[8929]: 16.604827 |0|0|0| C 05 CAL0000: distributedenginecomm.cpp @ 370 DEC: lost connection to 10.0.32.18

gao1738
gao1738's picture
Offline
Last seen: 1 month 3 weeks ago
Joined: Jan 2 2014
Junior Boarder

Posts: 10

dennis gao
Re: Cannot execute the statement. DBRM is read only

Hi,all

I release the lock by updating dbrm/BRM_saves_current according to post:
http://infinidb.org/forum/11-general-infinidb/2894-infinidb-not-starting

But I occur new problems:
It can insert and select right now, but when I do a update, I got the following error:

mysql> update loadtest set val_1 = val_1 +1 where id < 30000;
ERROR 122 (HY000): CAL0002: Update Failed: The File does not exist.
mysql> update loadtest set val_1 = val_1 +1 where id < 30000;
ERROR 122 (HY000): Lost connection to DMLProc
mysql> update loadtest set val_1 = val_1 +1 where id < 30000;
ERROR 122 (HY000): CAL0002: Update Failed: a BRM VB copy write.
mysql> update loadtest set val_1 = val_1 +1 where id < 30000;
ERROR 122 (HY000): CAL0002: Update Failed: a BRM VB copy write.

How to fix...

best regards,

dennis