Hacking uBuntu to Improve Performance
Wednesday, April 13, 2011
Tunning Kernel Parameters
Many of the tunable performance items can be configured directly by the kernel. The command sysctl is used to view current kernel settings and adjust them. For example, to display all available parameters (in a sorted list), use:
sudo sysctl -a | sort | more
Note: There are a few tunable parameters that can only be accessed by root. Without sudo, you can still view most of the kernel parameters.
Each of the kernel parameters are in a field = value format. For example, the parameter kernel.threads-max = 16379 sets the maximum number of concurrent processes to 16,379. This is smaller than the maximum number of unique PIDs (65,536). Lowering the number of PIDs can improve performance on systems with slow CPUs or little RAM since it reduces the number of simultaneous tasks. On high-performance computers with dual processors, this value can be large. As an example, my 350 MHz iMac is set to 2,048, my dual-processor 200 MHz PC is set to 1024, and my 2.8 GHz dual processor PC is set to 16,379.
Tip: The kernel configures the default number of threads based on the available resources. Installing the same Ubuntu version on different hardware may set a different value. If you need an identical system (for testing, critical deployment, or sensitive compatibility), be sure to explicitly set this value.
There are two ways to adjust the kernel parameters. First, you can do it on the command line. For example, sudo sysctl -w kernel.threads-max=16000. This change takes effect immediately but is not permanent; if you reboot, this change will be lost. The other way to make a kernel change is to add the parameter to the /etc/sysctl.conf file. Adding the line kernel.threads-max=16000 will make the change take effect on the next reboot. Usually when tuning, you first use sysctl –w. If you like the change, then you can add it to /etc/sysctl.conf. Using sysctl –w first allows you to test modifications. In the event that everything breaks, you can always reboot to recover before committing the changes to /etc/sysctl.conf.
Computing Swap
The biggest improvement you can make to any computer runnning Ubuntu is to add RAM. In general, the speed impact from adding RAM is bigger than the impact from a faster processor. For example, increasing the RAM from 128 MB to 256 MB can turn the installation from a day job to an hour. Increasing to 1 GB of RAM shrinks the installation to minutes.
There is an old rule-of-thumb about the amount of swap space. The conventional wisdom says that you should have twice as much swap as RAM. A computer with 256 MB of RAM should start with 512 MB of swap. Although this is a good idea for memory limited systems, it isn't practical for high-end home user systems. If you have 1 GB of RAM, then you probably will never need swap space—and you are very unlikely to need 2 GB of swap unless you are planning on doing video editing or audio composition.
There is a limit to the amount of usable RAM. A 32-bit PC architecture like the Intel Pentium i686 family can only access 4 GB of RAM. This is a hardware limitation. Assuming you can insert 16 GB of RAM into the computer, only the first 4 GB will be addressable. The second limitation comes from the Linux 2.6 kernel used by Ubuntu's Dapper Drake (as well as other Linux variants). The kernel can only access 1 GB of RAM. Any remaining RAM is unused. For this reason, it is currently not worth investing in more than 1 GB of RAM. Also, since the total virtual memory (RAM + swap) is limited to 4 GB, you should not need to allocate more than 3 GB of swap on a 1 GB RAM system.
Note: If you compile the kernel from scratch, there is a HIGHMEM flag that can be set to access up to 4 GB of RAM on a 32-bit architecture. Unfortunately, there is a reported performance hit since most hardware drivers cannot access the high memory. If you set this flag, then do not bother with any swap space—the total amount of memory (RAM + swap) cannot be larger than 4 GB.
Modifying Shared Memory
Some sections of virtual memory can be earmarked for use by multiple applications. Shared memory allows different programs to communicate quickly and share large volumes of information. Applications such as X-Windows, the Gnome Desktop, Nautilus, X-session manager, Gnome Panel, Trashcan applet, and Firefox all use shared memory. If programs cannot allocate or access the shared memory that they need, then programs will fail to start.
The inter-process communication status command, ipcs, displays the current shared memory allocations as well as the PIDs that created and last accessed the memory.
Tip: There are many other flags for ipcs. For example, ipcs -m –t shows the last time the shared memory was accessed, and ipcs -m –c shows access permissions. In addition, ipcs can show semaphore and message queue allocations.
Programs can allocate shared memory in two different ways: temporary and permanent. A temporary allocation means the memory remains shared until all applications release the memory handle. When no applications remain attached (the ipcs nattch field), the memory is freed. In contrast, permanent allocations can remain even when no programs are currently using it. This allows programs to save state in a shared communication buffer.
Sometimes shared memory becomes abandoned. To forcefully free abandoned memory, use ipcrm. You will need to specify the shared memory ID found from using ipcs (the shmid column).
More often than not, you will be more concerned with allocating more shared memory rather than freeing abandoned segments. For example, databases and high-performance web servers work better when there is more shared memory available. The sysctl command shows the current shared memory allocations:
$ sysctl kernel | grep shm
kernel.shmmni = 4096
kernel.shmall = 2097152
kernel.shmmax = 33554432
In this example, there is a total of 33,554,432 bytes (32 MB) of shared memory available. A single application can allocate up to 2 MB (2,097,152 bytes), and the minimal allocation unit is 4096 bytes. These sizes are plenty for most day-to-day usage, but if you plan to run a database such as Oracle, MySQL, or PostgreSQL, then you will almost certainly need to increase these values.
Changing Per User Settings
Beyond the kernel settings are parameters for configuring users. The ulimit command is built into the bash shell and provides limits for specific application. Running ulimit –a shows the current settings.
These setting show, for example, that a single user-shell can have a maximum of 1,024 open files and core dumps are disabled (size 0). Developers will probably want to enable core dumps, with ulimit -c 100 (for 100 blocks). If you want to increase the number of open files to 2,048, use ulimit -n 2048.
The user cannot change some limits. For example, if you want to increase the number of open files to 2,048 then you would use ulimit -n 2048, except only root can increase this value. In contrast, the user can always lower the value.
Note: Some values have an upper limit defined by the kernel. For example, although root can increase the number of open file handles, this cannot be increased beyond the value set by the kernel parameter fs.file-max (sysctl fs.file-max).
On a multi-user system, the administrator may want to limit the number of processes any single user can have running. This can be done by adding a ulimit statement to the /etc/bash.bashrc script:
# Only change the maximum number of processes for users, not root.
if [ `/usr/bin/id -u` -ne 0 ] ; then
ulimit -u 2048 2>/dev/null # ignore errors if user set it lower
else
ulimit -u unlimited # root can run anything
fi
Speeding Up Boot Time
The default init scripts found in /etc/init.d/ and the /etc/rc*.d/ directories are good for most systems, but they may not be needed on your specific system. If you do not need a service then you can disable it. This can reduce the boot time for your system. In some cases, it can also speed up the overall running speed by freeing resources. Use sysv-rc-conf (sudo apt-get install sysv-rc-conf) to change the enable/disable settings. Some of the services to consider disabling include:
anacron—As mentioned earlier, this subsystem periodically runs processes. You may want to disable it and move any critical services to cron.
atd and cron—By default, there are not at or cron jobs scheduled. If you do not need these services, then they can be disabled. Personally, I would always leave them enabled since they take relatively few resources.
apmd—This service handles power management and is intended for older systems that do not support the ACPI interface. It only monitors the battery. If you have a newer laptop (or are not using a laptop), then you probably do not need this service enabled.
acpid—The acpid service monitors battery levels and special laptop buttons such as screen brightness, volume control, and wireless on/off. Although intended for laptops, it can also support some desktop computers that use special keys on the keyboard (for example, a www button to start the browser). If you are not using a laptop and do not have special buttons on your keyboard, then you probably do not need this service.
bluez-utiles—This provides support for BlueTooth devices. If you don't have any, then this can be disabled.
dns-clean, ppp, and pppd-dns—These services are used for dynamic, dial-up connections. If you do not use dialup, then these can be disabled.
hdparm—This system is used to tune disk drive performance. It is not essential and, unless configured, does not do anything. The configuration file is /etc/hdparm.conf and it is not enabled by default.
hplip—This provides Linux support for the HP Linux Image and Printing system. If you do not need it, then it can be disabled. Without this, you can still print using the lpr and CUPS systems.
mdadm, mdadm-raid, and lvm—file system support for RAID (mdadm and mdadm-raid) and Logical Volume groups (lvm). If you do not use either, then these can be disabled.
nfs-common, nfs-kernel-server, and portmap—reused by NFS—present if you installed NFS support. If you do not need NFS all the time, then you can disable these and only start the services when you need them:
sudo /etc/init.d/portmap start
sudo /etc/init.d/nfs-common start
sudo /etc/init.d/nfs-kernel-server start
pcmcia and pcmciautils—provide support for PCMCIA devices on laptops. If you do not have any PCMCIA slots on your computer, then you do not need these services.
powernowd and powernowd.early—services are used to control variable-speed CPUs. Newer computers and laptops should have these enabled, but older systems (for example, my dual-processor 200 MHz PC) do not need it.
readahead and readahead-desktop—services are used to preload libraries so some applications will initially start faster. In a tradeoff for speed, these services slow down the initial boot time of the system and consume virtual memory with preloaded libraries. If you have limited RAM, then you should consider disabling these services.
rsync— a replacement for the remote copy (rcp) command. Few people need this—sed to synchronize files between computers.
vbesave—services monitors the Video BIOS real-time configuration. This is an ACPI function and is usually used on laptops when switching between the laptop display and an external display. If your computer does not support APCI or does not switch between displays, then you do not need this service.
Tip: There is a System>Admin>Services applet for enabling and disabling some services. However, this applet only knows of a few services; it does not list every available service. The sysv-rc-conf command recognizes far more services and offers more management options.
The sysv-rc-conf command shows most of the system services. However, it does not show all of them. If the service's name ends with .sh, contains .dpkg-, or is named rc or rcS, then it is treated as a non-modifiable system service. To change these services, you will need to manually modify the /etc/init.d/ and /etc/rc*.d/ directory contents.
Leave It On!
Although there are many services that you probably do not need, there are a few that are essential. You should not turn off these essential services unless you really know what you are doing:
dbus—s messaging services.
gdm— the Gnome Desktop. Only disable this if you do not want a graphical desktop.
klogd— the kernel log daemon. Removing it disables system logging.
makedev and udev—create all device nodes.
module-init-tools—kernel modules specified in /etc/modules.
networking and loopback—start and stop the network. Disabling removes the network configuration at boot.
procps.sh—kernel tuning parameters added to /etc/sysctl.conf are processed by this service.
urandom—eds the real random number generator that is used by most cryptographic system. You should leave it enabled.
As a rule of thumb, if you do not know what it is, then leave it on. Also, if the service only runs in single-user mode (rcS) that it is usually smart to not change it. Single user mode is where you should go when everything fails in order to repair the system.
Summary
Ubuntu is designed for the average computer user. As a result, there are some running processes that may not be needed and some resources that are not used optimally. Using a variety of commands, you can see what is running, what resources are available, and what resources are being used. The system can be tuned by adjusting kernel parameters, shell parameters, and settings for specific applications.
Many of the tunable performance items can be configured directly by the kernel. The command sysctl is used to view current kernel settings and adjust them. For example, to display all available parameters (in a sorted list), use:
sudo sysctl -a | sort | more
Note: There are a few tunable parameters that can only be accessed by root. Without sudo, you can still view most of the kernel parameters.
Each of the kernel parameters are in a field = value format. For example, the parameter kernel.threads-max = 16379 sets the maximum number of concurrent processes to 16,379. This is smaller than the maximum number of unique PIDs (65,536). Lowering the number of PIDs can improve performance on systems with slow CPUs or little RAM since it reduces the number of simultaneous tasks. On high-performance computers with dual processors, this value can be large. As an example, my 350 MHz iMac is set to 2,048, my dual-processor 200 MHz PC is set to 1024, and my 2.8 GHz dual processor PC is set to 16,379.
Tip: The kernel configures the default number of threads based on the available resources. Installing the same Ubuntu version on different hardware may set a different value. If you need an identical system (for testing, critical deployment, or sensitive compatibility), be sure to explicitly set this value.
There are two ways to adjust the kernel parameters. First, you can do it on the command line. For example, sudo sysctl -w kernel.threads-max=16000. This change takes effect immediately but is not permanent; if you reboot, this change will be lost. The other way to make a kernel change is to add the parameter to the /etc/sysctl.conf file. Adding the line kernel.threads-max=16000 will make the change take effect on the next reboot. Usually when tuning, you first use sysctl –w. If you like the change, then you can add it to /etc/sysctl.conf. Using sysctl –w first allows you to test modifications. In the event that everything breaks, you can always reboot to recover before committing the changes to /etc/sysctl.conf.
Computing Swap
The biggest improvement you can make to any computer runnning Ubuntu is to add RAM. In general, the speed impact from adding RAM is bigger than the impact from a faster processor. For example, increasing the RAM from 128 MB to 256 MB can turn the installation from a day job to an hour. Increasing to 1 GB of RAM shrinks the installation to minutes.
There is an old rule-of-thumb about the amount of swap space. The conventional wisdom says that you should have twice as much swap as RAM. A computer with 256 MB of RAM should start with 512 MB of swap. Although this is a good idea for memory limited systems, it isn't practical for high-end home user systems. If you have 1 GB of RAM, then you probably will never need swap space—and you are very unlikely to need 2 GB of swap unless you are planning on doing video editing or audio composition.
There is a limit to the amount of usable RAM. A 32-bit PC architecture like the Intel Pentium i686 family can only access 4 GB of RAM. This is a hardware limitation. Assuming you can insert 16 GB of RAM into the computer, only the first 4 GB will be addressable. The second limitation comes from the Linux 2.6 kernel used by Ubuntu's Dapper Drake (as well as other Linux variants). The kernel can only access 1 GB of RAM. Any remaining RAM is unused. For this reason, it is currently not worth investing in more than 1 GB of RAM. Also, since the total virtual memory (RAM + swap) is limited to 4 GB, you should not need to allocate more than 3 GB of swap on a 1 GB RAM system.
Note: If you compile the kernel from scratch, there is a HIGHMEM flag that can be set to access up to 4 GB of RAM on a 32-bit architecture. Unfortunately, there is a reported performance hit since most hardware drivers cannot access the high memory. If you set this flag, then do not bother with any swap space—the total amount of memory (RAM + swap) cannot be larger than 4 GB.
Modifying Shared Memory
Some sections of virtual memory can be earmarked for use by multiple applications. Shared memory allows different programs to communicate quickly and share large volumes of information. Applications such as X-Windows, the Gnome Desktop, Nautilus, X-session manager, Gnome Panel, Trashcan applet, and Firefox all use shared memory. If programs cannot allocate or access the shared memory that they need, then programs will fail to start.
The inter-process communication status command, ipcs, displays the current shared memory allocations as well as the PIDs that created and last accessed the memory.
Tip: There are many other flags for ipcs. For example, ipcs -m –t shows the last time the shared memory was accessed, and ipcs -m –c shows access permissions. In addition, ipcs can show semaphore and message queue allocations.
Programs can allocate shared memory in two different ways: temporary and permanent. A temporary allocation means the memory remains shared until all applications release the memory handle. When no applications remain attached (the ipcs nattch field), the memory is freed. In contrast, permanent allocations can remain even when no programs are currently using it. This allows programs to save state in a shared communication buffer.
Sometimes shared memory becomes abandoned. To forcefully free abandoned memory, use ipcrm. You will need to specify the shared memory ID found from using ipcs (the shmid column).
More often than not, you will be more concerned with allocating more shared memory rather than freeing abandoned segments. For example, databases and high-performance web servers work better when there is more shared memory available. The sysctl command shows the current shared memory allocations:
$ sysctl kernel | grep shm
kernel.shmmni = 4096
kernel.shmall = 2097152
kernel.shmmax = 33554432
In this example, there is a total of 33,554,432 bytes (32 MB) of shared memory available. A single application can allocate up to 2 MB (2,097,152 bytes), and the minimal allocation unit is 4096 bytes. These sizes are plenty for most day-to-day usage, but if you plan to run a database such as Oracle, MySQL, or PostgreSQL, then you will almost certainly need to increase these values.
Changing Per User Settings
Beyond the kernel settings are parameters for configuring users. The ulimit command is built into the bash shell and provides limits for specific application. Running ulimit –a shows the current settings.
These setting show, for example, that a single user-shell can have a maximum of 1,024 open files and core dumps are disabled (size 0). Developers will probably want to enable core dumps, with ulimit -c 100 (for 100 blocks). If you want to increase the number of open files to 2,048, use ulimit -n 2048.
The user cannot change some limits. For example, if you want to increase the number of open files to 2,048 then you would use ulimit -n 2048, except only root can increase this value. In contrast, the user can always lower the value.
Note: Some values have an upper limit defined by the kernel. For example, although root can increase the number of open file handles, this cannot be increased beyond the value set by the kernel parameter fs.file-max (sysctl fs.file-max).
On a multi-user system, the administrator may want to limit the number of processes any single user can have running. This can be done by adding a ulimit statement to the /etc/bash.bashrc script:
# Only change the maximum number of processes for users, not root.
if [ `/usr/bin/id -u` -ne 0 ] ; then
ulimit -u 2048 2>/dev/null # ignore errors if user set it lower
else
ulimit -u unlimited # root can run anything
fi
Speeding Up Boot Time
The default init scripts found in /etc/init.d/ and the /etc/rc*.d/ directories are good for most systems, but they may not be needed on your specific system. If you do not need a service then you can disable it. This can reduce the boot time for your system. In some cases, it can also speed up the overall running speed by freeing resources. Use sysv-rc-conf (sudo apt-get install sysv-rc-conf) to change the enable/disable settings. Some of the services to consider disabling include:
anacron—As mentioned earlier, this subsystem periodically runs processes. You may want to disable it and move any critical services to cron.
atd and cron—By default, there are not at or cron jobs scheduled. If you do not need these services, then they can be disabled. Personally, I would always leave them enabled since they take relatively few resources.
apmd—This service handles power management and is intended for older systems that do not support the ACPI interface. It only monitors the battery. If you have a newer laptop (or are not using a laptop), then you probably do not need this service enabled.
acpid—The acpid service monitors battery levels and special laptop buttons such as screen brightness, volume control, and wireless on/off. Although intended for laptops, it can also support some desktop computers that use special keys on the keyboard (for example, a www button to start the browser). If you are not using a laptop and do not have special buttons on your keyboard, then you probably do not need this service.
bluez-utiles—This provides support for BlueTooth devices. If you don't have any, then this can be disabled.
dns-clean, ppp, and pppd-dns—These services are used for dynamic, dial-up connections. If you do not use dialup, then these can be disabled.
hdparm—This system is used to tune disk drive performance. It is not essential and, unless configured, does not do anything. The configuration file is /etc/hdparm.conf and it is not enabled by default.
hplip—This provides Linux support for the HP Linux Image and Printing system. If you do not need it, then it can be disabled. Without this, you can still print using the lpr and CUPS systems.
mdadm, mdadm-raid, and lvm—file system support for RAID (mdadm and mdadm-raid) and Logical Volume groups (lvm). If you do not use either, then these can be disabled.
nfs-common, nfs-kernel-server, and portmap—reused by NFS—present if you installed NFS support. If you do not need NFS all the time, then you can disable these and only start the services when you need them:
sudo /etc/init.d/portmap start
sudo /etc/init.d/nfs-common start
sudo /etc/init.d/nfs-kernel-server start
pcmcia and pcmciautils—provide support for PCMCIA devices on laptops. If you do not have any PCMCIA slots on your computer, then you do not need these services.
powernowd and powernowd.early—services are used to control variable-speed CPUs. Newer computers and laptops should have these enabled, but older systems (for example, my dual-processor 200 MHz PC) do not need it.
readahead and readahead-desktop—services are used to preload libraries so some applications will initially start faster. In a tradeoff for speed, these services slow down the initial boot time of the system and consume virtual memory with preloaded libraries. If you have limited RAM, then you should consider disabling these services.
rsync— a replacement for the remote copy (rcp) command. Few people need this—sed to synchronize files between computers.
vbesave—services monitors the Video BIOS real-time configuration. This is an ACPI function and is usually used on laptops when switching between the laptop display and an external display. If your computer does not support APCI or does not switch between displays, then you do not need this service.
Tip: There is a System>Admin>Services applet for enabling and disabling some services. However, this applet only knows of a few services; it does not list every available service. The sysv-rc-conf command recognizes far more services and offers more management options.
The sysv-rc-conf command shows most of the system services. However, it does not show all of them. If the service's name ends with .sh, contains .dpkg-, or is named rc or rcS, then it is treated as a non-modifiable system service. To change these services, you will need to manually modify the /etc/init.d/ and /etc/rc*.d/ directory contents.
Leave It On!
Although there are many services that you probably do not need, there are a few that are essential. You should not turn off these essential services unless you really know what you are doing:
dbus—s messaging services.
gdm— the Gnome Desktop. Only disable this if you do not want a graphical desktop.
klogd— the kernel log daemon. Removing it disables system logging.
makedev and udev—create all device nodes.
module-init-tools—kernel modules specified in /etc/modules.
networking and loopback—start and stop the network. Disabling removes the network configuration at boot.
procps.sh—kernel tuning parameters added to /etc/sysctl.conf are processed by this service.
urandom—eds the real random number generator that is used by most cryptographic system. You should leave it enabled.
As a rule of thumb, if you do not know what it is, then leave it on. Also, if the service only runs in single-user mode (rcS) that it is usually smart to not change it. Single user mode is where you should go when everything fails in order to repair the system.
Summary
Ubuntu is designed for the average computer user. As a result, there are some running processes that may not be needed and some resources that are not used optimally. Using a variety of commands, you can see what is running, what resources are available, and what resources are being used. The system can be tuned by adjusting kernel parameters, shell parameters, and settings for specific applications.
