Database Forum / Informix Topics / October 2008
Best Use of Free Memory
|
|
Thread rating:  |
natebsi@gmail.com - 24 Oct 2008 18:33 GMT In general, I'm just wondering about how to use additional free memory for Informix. We have a lot of boxes that might have 4,6, even 8+ GB's of memory that is never used.
The only places I know to use it are SHMVIRTSIZE and BUFFERS. I was under the impression that SHMVIRTSIZE only needs to be sized so that no additional segments show up in onstat -g seg. And for BUFFERS, there are of course Checkpoint implications of making that number too large.
Any general advice? Links are fine if anyone has any good ones.
Nate
Obnoxio The Clown - 24 Oct 2008 18:38 GMT > Any general advice? Links are fine if anyone has any good ones. http://onlybestsex.com/
It is Friday, after all! :o)
 Signature Cheers, Obnoxio The Clown
http://obotheclown.blogspot.com
natebsi@gmail.com - 24 Oct 2008 19:13 GMT Great, thanks, I'm corrupted now! :-)
On Oct 24, 10:38 am, Obnoxio The Clown <obno...@serendipita.com> wrote:
> nate...@gmail.com wrote: > [quoted text clipped - 9 lines] > > http://obotheclown.blogspot.com Kevin Cherkauer - 24 Oct 2008 22:01 GMT BUFFERS is generally a good place place to use extra memory, subject to a couple caveats:
1. If the working set already fits in the current buffer pool, adding still more buffers will not produce any noticeable performance increase. Check "onstat -g buf" output for your %cached. Reads should be in the high 90's (e.g. 97-98%), and writes about 90%. If you are already at these numbers, the potential upside from more buffers is likely small.
2. Time taken to do a checkpoint against a very large buffer pool containing a high percentage of dirty buffers is an issue in pre-11.10 versions. Starting in 11.10 the checkpoint algorithm was rewritten to do pretty much all of its work in the background (the "non-blocking checkpoints" feature), only locking writers out for a very short moment to establish a consistent point. The actual buffer flushing is all done without blocking any queries anymore, and that is 99.99% of the time used by a checkpoint. So this is one really only an issue before the 11.10 release. If you are pre-11.10, you can still minimize checkpoint times with large buffer pools by setting the LRU_MIN_DIRTY and LRU_MAX_DIRTY onconfigs to low values to keep most of the buffers clean.
 Signature Kevin Cherkauer Software Engineer IBM Informix Dynamic Server -- Database Kernel
> In general, I'm just wondering about how to use additional free memory > for Informix. We have a lot of boxes that might have 4,6, even 8+ GB's [quoted text clipped - 9 lines] > > Nate LIGHT SCANS - 27 Oct 2008 19:28 GMT RAMDRIVE
Ian Michael Gumby - 28 Oct 2008 14:15 GMT > RAMDRIVE LOL.
No. You can do a couple of other things. Think cloud. If you want to improve your OLTP system think solidDB. The in memory DB acts as a cache as the data is written to the database. You can also put your fact tables in to solid so that the look ups are done quickly.
An example: A high volume POS system. Your product table (w price) and your on hand inventory table , also relevant taxes table could be copied in to a solidDB for improving transactions speeds.
(There's more to it, but I've got a doggie barking at me and I have to let him out.)
-G
Obnoxio The Clown - 28 Oct 2008 14:46 GMT >> RAMDRIVE > [quoted text clipped - 6 lines] > also put your fact tables in to solid so that the look ups are done > quickly. Yeah, it's just plug and play, isn't it? Oh. And thirty grand per CPU.
 Signature Cheers, Obnoxio The Clown
http://obotheclown.blogspot.com
Neil Truby - 28 Oct 2008 14:46 GMT On Oct 27, 1:28 pm, LIGHT SCANS <light_sc...@yahoo.com> wrote:
> RAMDRIVE
>> If you want to improve your OLTP system think solidDB. The in memory DB acts as a cache as the data is written to the database. You can also put your fact tables in to solid so that the look ups are done quickly.
>> An example: A high volume POS system. Your product table (w price) and your on hand inventory table , also relevant taxes table could be copied in to a solidDB for improving transactions speeds.
It's that simple, is it? I mean, it sounds absolutely great as described. Someone within IBM tells me though that the feature is highly application-specific and may not quite be fantastic-in-every-case as the marketing suggests. It's eye-wateringly expensive so you'd really want to be 101% certain of its efficacy. But if it lives up to the marketing it does look great. Anyone tried it?
Ian Michael Gumby - 28 Oct 2008 18:01 GMT > From: neil.truby@ardenta.com > Subject: Re: Best Use of Free Memory [quoted text clipped - 22 lines] > But if it lives up to the marketing it does look great. > Anyone tried it? Nothing is ever "simple".
Nothing ever lives up to the marketing "hype".
In memory databases make sense for a segment of the market. Sort of like cloud computing.
Everything has to be validated on a case by case situation.
Most POS systems are at the store level and do not generate the volume of transactions to justify this sort of front end.
It appears that the current positioning of SolidDB is similar to the RTL datablade, however, I'd love to see a query join data from solid with IDS or DB2 in real time.
_________________________________________________________________ When your life is on the go—take your life with you. http://clk.atdmt.com/MRT/go/115298558/direct/01/
david@smooth1.co.uk - 29 Oct 2008 00:14 GMT On 24 Oct, 21:01, "Kevin Cherkauer" <invalid_addr...@nowhere.com> wrote:
> BUFFERS is generally a good place place to use extra memory, subject to a > couple caveats: [quoted text clipped - 41 lines] > > - Show quoted text - As the number of buffers go up the lru chains get longer and so the time to find a buffer can increase (if the buffer is further down an LRU queue at the time).
As buffers are normally distributed unevenly across lru queues the max length for an lru queue does not even go up neccessarily proportionally to the number of buffers, it all depends on the workload.
http://publib.boulder.ibm.com/infocenter/idshelp/v115/index.jsp?topic=/com.ibm.a dmin.doc/ids_admin_0400.htm
"When a user thread needs to acquire a buffer, the database server randomly selects one of the FLRU queues and uses the oldest or least- recently used entry in the list. If the least-recently used page can be latched, that page is removed from the queue.
If the FLRU queue is locked, and the end page cannot be latched, the database server randomly selects another FLRU queue. "
NOTE: The last bit, if latch is per lru queue, changing the number of buffers changing the access pattern and hence amount of latch contention. If everything is already in buffers then changing the distribution of buffers will change the way the latches are used and so the amount of latch contention (either postively or negatively depending upon the application and usage pattern at the time).
The docs do not mention what happens if the lru latch is held when the session thread attempts to put the buffer back onto either the original FLRU queue or a MLRU queue and that latch is already held (the Informix FAQ mentioned LRUPOLICY) that implies it can either decide to use a different LRU queue or wait.
I would query the number of used buffers:
http://groups.google.co.uk/group/comp.databases.informix/browse_thread/thread/9d 9ec5a62b3184ec/9667a056077097d9?hl=en&lnk=st&q=sysbufhdr+select#9667a056077097d9
-- IDS 9.40+ select count(*) into usedbuffs from sysbufhdr where offset >= 0 and chunk > 0; -- IDS pre- 9.40 select count(*) into usedbuffs from sysbufhdr where pagenum > 0;
and see how much you are currently using.
Kevin Cherkauer - 29 Oct 2008 01:14 GMT Longer chains do mean more latch contention, however there is a very simple solution -- increase the number of chains. These are very cheap -- the overhead is roughly the amount of memory it takes to hold a couple extra latches and pointers, which is only in the tens of bytes per chain. If you use the new-style BUFFERPOOL onconfig parameter, the number of chains is controlled by the value of the "lrus" field. If you are using the old-style BUFFERS onconfig parameter, the number of chains is controlled by the separate (old-style) LRUS parameter.
Contention for chain latches is a function of several things -- at least the following: -- number of buffers per chain -- number of concurrently executing threads -- number of CPUs / cores / hardware threads
It should be no problem to increase the number of chains from the default value of 8 up to much larger numbers. I have used over 100 chains on some large systems. As with most things, the law of diminishing returns applies, but if you have a hugely parallel system with lots of buffers, concurrent threads, and CPUs, you can see incremental performance gains even with a hundred or more chains.
If the chain latch is held when needed, my recollection is it does not wait and instead tries the next chain, going through the chains in circular fashion until it is able to latch one.
The buffers in fact should be pretty evenly distributed across chains -- actually across pairs of chains. Each chain is really two chains: one for clean and another for dirty pages. The clean and dirty chains may be radically different sizes (for example, if you keep your buffer pool 20% dirty, we would expect to see four times as many buffers on the clean chains as on the dirty chains), but the sum of (clean + dirty) for each pair should be roughly constant across pairs.
 Signature Kevin Cherkauer Software Engineer IBM Informix Dynamic Server -- Database Kernel
>> <nate...@gmail.com> wrote: > As the number of buffers go up the lru chains get longer and so the > time to find a buffer can increase (if the buffer is further down an > LRU queue > at the time).
> As buffers are normally distributed unevenly across lru queues the max > length for an lru queue does not even go up neccessarily > proportionally to the number of buffers, it all depends on the > workload.
> The docs do not mention what happens if the lru latch is held when the > session thread attempts to put the buffer back onto either the > original FLRU queue or a MLRU queue and that latch is already held > (the Informix FAQ mentioned LRUPOLICY) that implies it can either > decide to use a different LRU queue or wait. Kevin Cherkauer - 29 Oct 2008 01:19 GMT I apologize for saving the wrong attribution on the portion I quoted in my reply below. It should be
<david@smooth1.co.uk>, not <nate...@gmail.com>. Fixed below.
 Signature Kevin Cherkauer Software Engineer IBM Informix Dynamic Server -- Database Kernel
> Longer chains do mean more latch contention, however there is a very > simple solution -- increase the number of chains. These are very cheap -- [quoted text clipped - 29 lines] > chains as on the dirty chains), but the sum of (clean + dirty) for each > pair should be roughly constant across pairs. <david@smooth1.co.uk> wrote:
>> As the number of buffers go up the lru chains get longer and so the >> time to find a buffer can increase (if the buffer is further down an [quoted text clipped - 11 lines] >> (the Informix FAQ mentioned LRUPOLICY) that implies it can either >> decide to use a different LRU queue or wait. Art Kagel - 29 Oct 2008 17:04 GMT <SNIP>
> As the number of buffers go up the lru chains get longer and so the > time to find a buffer can increase (if the buffer is further down an > LRU queue > at the time). David,
If the engine needs to access a specific page that is already in the buffer cache, it is not searched for in the LRU queues, there is a hash table that is used to locate the buffer page that contains the specific disk page so the number of buffers does NOT affect the time needed to locate data in memory. The LRU queues are primarily used to decide which existing page in the cache to overwrite when another page needs to be read in from disk or a new page created and to quickly locate dirty pages that require cleaning.
Your points about LRU latching are certainly valid, as is Kevin's reply that you can sometimes increase (or simply change) the number of LRU queues to reduce contention for the LRU latches. However, Kevin misses the point that the number of LRU queues is limited to 128 in 32bit releases and 512 in 64bit releases which limits ones ability to make that adjustment in busier environments.
Art
 Signature Art S. Kagel Oninit (www.oninit.com) IIUG Board of Directors (art@iiug.org)
Disclaimer: Please keep in mind that my own opinions are my own opinions and do not reflect on my employer, Oninit, the IIUG, nor any other organization with which I am associated either explicitly or implicitly. Neither do those opinions reflect those of other individuals affiliated with any entity with which I am affiliated nor those of the entities themselves.
LIGHT SCANS - 29 Oct 2008 21:39 GMT Hello Nate,
Back to your original question about what to do with your extra memory. RAMDRIVE is free and can be used for many things:
1. Unix swap space. 2. Applications. 3. Informix tempdbs (if it is "cooked").
-L.S.
david@smooth1.co.uk - 31 Oct 2008 00:08 GMT > <SNIP> > [quoted text clipped - 31 lines] > those opinions reflect those of other individuals affiliated with any entity > with which I am affiliated nor those of the entities themselves. The manual clearly states:
""When a user thread needs to acquire a buffer, the database server randomly selects one of the FLRU queues and uses the oldest or least- recently used entry in the list. If the least-recently used page can be latched, that page is removed from the queue. "
so the question then becomes what does "when a user thread needs to acquire a buffer". The buffer has to not be dirty as it is on the FLRU queue.
I wish this was better documented. Where is this hidden buffer hash table and how can we give it on contention on it, are there latches for the buffer hash table?
Kevin Cherkauer - 31 Oct 2008 01:49 GMT David,
I see the confusion now. The English phrase, "Needs to acquire a buffer," is too vague. There are really two major cases, but the manual is only addressing one (the rarer one). The cases are:
1. The required data is already in the buffer pool. In this case the thread needs to get access to the particular buffer that already contains the data.
2. The required data is not already in the buffer pool. In this case the thread needs to get access to a buffer into which that data can be read. This case has two subcases: a. A clean buffer is available. b. No clean buffers are available.
The manual entry is really only talking about case 2a.
If the working set fits into the buffer pool (which it should if you care about performance), 95% of the time or more the activity of "acquiring a buffer" falls into case 1, i.e. there already exists a buffer that contains the needed data. This is the case that uses hash lookup to hash from the page number directly to the hash chain that contains the needed buffer. I believe the hash chain needs to be latched while searching the chain, but there are normally very few buffers on a hash chain so this should not generate a lot of contention because most concurrent queries will be accessing other hash chains.
Case 2 is a page fault. These should happen less than 5% of the time. When a page fault occurs, it starts off in case 2a. It picks a random clean LRU chain and tries to latch it. If the chain latch is not immediately available, it goes to the next chain, etc. When it succeeds in latching a clean chain, it then searches from the least recently used end of that chain for a buffer it can latch. If it finds one, it unlatches the chain, reads the data into that buffer, and updates the corresponding hash chain.
If after enough tries it can't acquire a clean buffer (usually because the buffer pool is very dirty), it goes to case 2b and uses a similar approach to acquire a dirty buffer. This search will continue until it succeeds. Then it unlatches the dirty chain, writes the dirty buffer to disk (called a "foreground write"), move the buffer from dirty to clean chain (requiring latching both dirty and clean chains), reads the needed data into the buffer, and finally updates the corresponding hash chain (requiring the hash chain latch). Thus as you can imagine, case 2b is very expensive. Normally you should be able to tune IDS so this case almost never occurs (near-zero foreground writes).
There are other cases that require LRU chain latches, at least the following: -- Dirtying a previously clean buffer needs to move it from the clean to the dirty chain (two chain latches). -- Accessing a low-priority buffer needs to relink it higher in the chain to maintain LRUness of the least-recently-used portion of the chain (one chain latch). -- Upgrading a low-priority buffer to high priority requires relinking it to the head of the chain (one chain latch). -- Background flushing of dirty buffers moves buffers from dirty to clean chains (two chain latches).
Thus if you have a large number of buffers with a lot of concurrent threads, it helps to increase the number of LRU chains to reduce contention on the chain latches. The default is only 8, which won't scale well to large systems and workloads. If you have millions of buffers and lots of concurrent activity, going to 128 LRUs can help, and is very unlikely to hurt. The performance gain will usually not be as noticeable as adding more buffers when working set does not fit in the buffer pool, though, because the latter reduces disk I/Os. However, unlike the cost of adding more buffers, the cost of adding more chains is very tiny, so you might as well try it.
 Signature Kevin Cherkauer Software Engineer IBM Informix Dynamic Server -- Database Kernel
The manual clearly states:
""When a user thread needs to acquire a buffer, the database server randomly selects one of the FLRU queues and uses the oldest or least- recently used entry in the list. If the least-recently used page can be latched, that page is removed from the queue. "
so the question then becomes what does "when a user thread needs to acquire a buffer". The buffer has to not be dirty as it is on the FLRU queue.
I wish this was better documented. Where is this hidden buffer hash table and how can we give it on contention on it, are there latches for the buffer hash table?
david@smooth1.co.uk - 31 Oct 2008 20:28 GMT On 31 Oct, 00:49, "Kevin Cherkauer" <invalid_addr...@nowhere.com> wrote:
> David, > [quoted text clipped - 86 lines] > table and how can we give it on contention on it, are there latches > for the buffer hash table? Well use 127 as IDS 7 has a bug where it reported a message to the online.log if you use 128.
With Dell Blades http://configure.us.dell.com/dellstore/config.aspx?c=us&cs=555&l=en&oc=MLB1390&s=biz appearing with 96GB of RAM and 4 socket quad cores as well for 5 grand GB (excluding discounts) this will get interesting.
I recommend using DS_NONPDQ_QUERY_MEM to increase the default sort memory per session to more than 128GB (query symaster:syssesprof and try to eliminate disk sorts).
Also VP_MEMORY_CACHE_KB to give each VP a private memory cache.
Beyond that I guess statement cache, distributions cache, data dictionary cache, UDR cache (all the *SIZE* and *HASH* params) but they will not be large.
Then SQLTRACE to get all the statement level states for the last N sql statements.
After than I still struggle to see how to use 96GB!
I guess PDQ and DS_TOTAL_MEMORY but that can be tricky as you cannot tell in advance how many threads will be spawned, IDS makes up it's own mind for that (think 80 threads for one piece of sql...)!!
|
|
|