Understanding vm.nr_hugepages, vm.hugetlb_shm_group, kernel.shmall, kernel.shmmax, kernel.shmmni
Setting up new database server and want to make sure that all the OS limits are set properly. It require little bit of calculation to be done. There are confusing bits what to divide by page size and what to divide by huge page size.
Hoping that below explanation should help in the understanding and setting them correctly.
Getting Current values for all the variables
cat /etc/sysctl.conf | egrep '(shm|huge)' | sort
Hugepage size used on the machine
grep -i "Hugepagesize" /proc/meminfo
vm.nr_hugepages = Number of hugepages available for a Unix Group. Make sure they are more than kernel.shmall.
Note: Make sure that the HugePages are many enough to cover all your database SGAs
vm.hugetlb_shm_group = Group ID of the Unix group which can use the hugepages defined under vm.nr_hugepages
kernel.shmall = Total number of pages available for all databases to place their SGAs.
Note: it is number of pages and not number of hugepages (Command is $getconf PAGE_SIZE )
Note: it is number of pages. It is not the size.
Note: pga_aggregate_target is not part of this. pga_aggregate_target is taken from OS RAM.
kernel.shmmax = SHMMAX is the maximum size of a single shared memory segment set in bytes.
Ideally, we would like SGA_TARGET to fit in one shared memory segment at startup by having
SGA_TARGET < SHMMAX
If SGA_TARGET > SHMMAX , then Oracle will try to use contiguous multi-segment to use to fit the SGA_TARGET.
If it is not able to do so, then it will use non-contiguous multi-segment allocation and in this
Oracle has to grab the free memory segments fragmented between used spaces.
Note: It has nothing to do with the total SGA for all databases. The database will start even if this value
is smaller than SGA
kernel.shmmni = It can vary for different OS.
Default use 4096
General Notes:
Make sure that the RAM and HugePages have enough space to cover all your database SGAs
Make sure the total SGA is less than the installed RAM and re-calculate HugePages.
Make sure that the HugePages are enough to cover all your database SGAs
Real world problem:
My server has 90G RAM. I will run 2 DBs on this server.
DB01 Requirements
-----------------
SGA = 12G
PGA = 4G
DB02 Requirements
-----------------
SGA = 7G
PGA = 10G
How can I setup the follwoing parameters?
vm.nr_hugepages, vm.hugetlb_shm_group, kernel.shmall, kernel.shmmax
Solution:
Value for kernel.shmmni
Setting the value to 4096 as that is recommended for redhat version I am using.
kernel.shmmni = 4096
Calcualte kernel.shmall
Find Total SGA required which is 12 + 7 = 19G. It will be used to calculate kernel.shmall (Remember this value is numer of pages).
Page side we are using is 4k. So we will need 19G / 4k = Total number of pages required to hold SGA of both databases.
In this case it will be (19 * 1024 * 1024) / 4 = 4980736 pages
We can set kernel.shmall = 4980736 (Note: this is minimum value, you should add little more if you want to add more database later on.
If you want to create another DB then there will be a need to change this parameter and restart of the server will be required. )
Calcualte kernel.shmmax
SHMMAX is the maximum size of a single shared memory segment set in bytes.
The biggest SGA out of two databases is 12G.
In this case it will be 12 * 1024 * 1024 = 12582912
kernel.shmmax = 12582912
Calcualte vm.nr_hugepages
Looking at shmall in our case we have allocated 19G so we have to calculate number of pages required to fit in for 19G.
In our gase the huge page size is 2048k
In this case it will be (19 * 1024 * 1024) / 2048 = 9728
But 9728 may not fit evenly on 19G so I will allocate 512 more hugepages for this to happen smoohtly. Hence end result for the parameter will be 9728 + 512 = 10240
vm.nr_hugepages = 10240
Calcualte vm.hugetlb_shm_group
This is the group ID of the oracle user. Just issue the id oracle on shell prompt and you will get the GID.
$ id oracle
Setting up new database server and want to make sure that all the OS limits are set properly. It require little bit of calculation to be done. There are confusing bits what to divide by page size and what to divide by huge page size.
Hoping that below explanation should help in the understanding and setting them correctly.
Getting Current values for all the variables
cat /etc/sysctl.conf | egrep '(shm|huge)' | sort
Hugepage size used on the machine
grep -i "Hugepagesize" /proc/meminfo
vm.nr_hugepages = Number of hugepages available for a Unix Group. Make sure they are more than kernel.shmall.
Note: Make sure that the HugePages are many enough to cover all your database SGAs
vm.hugetlb_shm_group = Group ID of the Unix group which can use the hugepages defined under vm.nr_hugepages
kernel.shmall = Total number of pages available for all databases to place their SGAs.
Note: it is number of pages and not number of hugepages (Command is $getconf PAGE_SIZE )
Note: it is number of pages. It is not the size.
Note: pga_aggregate_target is not part of this. pga_aggregate_target is taken from OS RAM.
kernel.shmmax = SHMMAX is the maximum size of a single shared memory segment set in bytes.
Ideally, we would like SGA_TARGET to fit in one shared memory segment at startup by having
SGA_TARGET < SHMMAX
If SGA_TARGET > SHMMAX , then Oracle will try to use contiguous multi-segment to use to fit the SGA_TARGET.
If it is not able to do so, then it will use non-contiguous multi-segment allocation and in this
Oracle has to grab the free memory segments fragmented between used spaces.
Note: It has nothing to do with the total SGA for all databases. The database will start even if this value
is smaller than SGA
kernel.shmmni = It can vary for different OS.
Default use 4096
General Notes:
Make sure that the RAM and HugePages have enough space to cover all your database SGAs
Make sure the total SGA is less than the installed RAM and re-calculate HugePages.
Make sure that the HugePages are enough to cover all your database SGAs
Real world problem:
My server has 90G RAM. I will run 2 DBs on this server.
DB01 Requirements
-----------------
SGA = 12G
PGA = 4G
DB02 Requirements
-----------------
SGA = 7G
PGA = 10G
How can I setup the follwoing parameters?
vm.nr_hugepages, vm.hugetlb_shm_group, kernel.shmall, kernel.shmmax
Solution:
Value for kernel.shmmni
Setting the value to 4096 as that is recommended for redhat version I am using.
kernel.shmmni = 4096
Calcualte kernel.shmall
Find Total SGA required which is 12 + 7 = 19G. It will be used to calculate kernel.shmall (Remember this value is numer of pages).
Page side we are using is 4k. So we will need 19G / 4k = Total number of pages required to hold SGA of both databases.
In this case it will be (19 * 1024 * 1024) / 4 = 4980736 pages
We can set kernel.shmall = 4980736 (Note: this is minimum value, you should add little more if you want to add more database later on.
If you want to create another DB then there will be a need to change this parameter and restart of the server will be required. )
Calcualte kernel.shmmax
SHMMAX is the maximum size of a single shared memory segment set in bytes.
The biggest SGA out of two databases is 12G.
In this case it will be 12 * 1024 * 1024 = 12582912
kernel.shmmax = 12582912
Calcualte vm.nr_hugepages
Looking at shmall in our case we have allocated 19G so we have to calculate number of pages required to fit in for 19G.
In our gase the huge page size is 2048k
In this case it will be (19 * 1024 * 1024) / 2048 = 9728
But 9728 may not fit evenly on 19G so I will allocate 512 more hugepages for this to happen smoohtly. Hence end result for the parameter will be 9728 + 512 = 10240
vm.nr_hugepages = 10240
Calcualte vm.hugetlb_shm_group
This is the group ID of the oracle user. Just issue the id oracle on shell prompt and you will get the GID.
$ id oracle
