...
So the objective is to get rid of these "idle in transaction" connections. There are 2 solutions, the fist first of which is just a workaround:
...
2) Find and fix the places in the code where transactions are not safely completed. This is surprisingly hard, because although the application framework initialises and closes transactions at start and finish, there are a variety of things that can happen causing the transaction to fail. Principally, exceptions that occur at the database level can have unfortunate consequences for the state of the transaction, so this need needs to be looked at. In addition, any situation which causes a Context object to go out of scope (which can happen if additional context objects are created in the course of the processing) can leave a hanging transaction.
...
There is no obvious way of tracking down all these situations, but we can offer a query which will allow us to generate a "signature" for the requests that cause the problems. This is done using the following SQL:
Code Blockpanel |
---|
SELECT relation, transaction, pid, mode, granted, relname
pg_locks
JOIN pg_stat_user_tables
pg_locks.relation = pg_stat_user_tables.relid
pg_locks.pid=' []'; |
This will give us a list of table names that a given process (identified by its pid) has locked and not released. This will hopefully allow us to identify which requests are causing the problem. Therefore, if you run this query, you should be able to see if this is your own code (accessing, for example, your tables, or a set of tables that one of your operations uses) or whether it is a DSpace bug. You should post this signature to the mailing list if you can't debug it yourself.
For example, this is an output from the above query:
...
relation |
---|
...
transaction | pid | mode | granted | relname | |
---|---|---|---|---|---|
18034 |
| 4909 | AccessShareLock | t | group2groupcache |
18113 |
| 4909 | AccessShareLock | t | metadatavalue |
18046 |
| 4909 | AccessShareLock | t | item |
18008 |
| 4909 | AccessShareLock | t | eperson |
18248 |
| 4909 | AccessShareLock | t | handle |
18235 |
| 4909 | AccessShareLock | t | epersongroup2eperson |
So process 4909 has AccessShareLocks on 6 tables, and these 6 tables should help us identify where to look for the bug. In this case it is particularly unclear, but a process which uses groups, epersons, items handles and metadata is probably a deposit process.
...
which produces something like:
...
datid |
---|
...
datname |
---|
...
procpid |
---|
...
usesysid |
---|
...
usename |
---|
...
current_query |
---|
...
query_start | |
---|---|
17142 |
...
dspace |
...
1107 |
...
1 |
...
dsprd |
...
<IDLE> |
...
2007-06-11 11:21:36.147423+01 | |
17142 |
...
dspace |
...
1401 |
...
1 |
...
dsprd |
...
<IDLE> |
...
2007-06-11 11:22:41.730301+01 | |
17142 |
...
dspace |
...
25163 |
...
1 |
...
dsprd |
...
<IDLE> |
...
2007-06-11 11:21:36.097556+01 | |
17142 |
...
dspace |
...
27973 |
...
1 |
...
dsprd |
...
update history set creation_date = '...' |
...
2007-06-11 11:25:39.687193+01 | |
17142 |
...
dspace |
...
916 |
...
1 |
...
dsprd |
...
<IDLE> |
...
2007-06-11 11:21:32.759784+01 |
Combining queued queries with locks
...
Code Block |
---|
/usr/bin/test `/usr/bin/pgrep -f 'idle in transaction' | \ /usr/bin/wc \-l ` \-gt 20 && /usr/bin/pkill \-o \-f 'idle in transaction' |
In English: every minute, if there are more than 20 "idle in
transaction" Postgres processes, it kills the oldest one.
...
Code Block |
---|
/bin/test `/usr/local/bin/pgrep -f '127.0.0.1' | \ /usr/bin/wc \-l ` \-gt 20 && /usr/local/bin/pkill \-n \-f '127.0.0.1' |