Overview of Unix/Linux System Tuning

Make sure you collect performance data on production servers.  Turn your collected performance data into useful reports.  If things aren't as they should be (the baseline and system requirements determine this):

1. Define the problem
2. Determine the cause(s)
3. Decide what to do about it (state specific performance goals)
4. Formulate a plan (a change order, which gets scheduled)
5. Monitor the result

Remember: If it ain't broke, don't fix!

You can change some parameters with echo and /proc/sys/* (or for Solaris you can also use rctladm).

Modern systems are almost completely self-tuning.  Change tunable parameters only very reluctantly!  Changing any parameters usually does more harm than good.  Usually the default kernel parameters are fine, and changing them makes performace much worse (or breaks something).  If you wish to tweak the tunable parameters do a Google search first.  A (somewhat dated) starting point is Server Oriented System Tuning Info.  A modern Linux performance tuning guide can be found at access.redhat.com/.../Red_Hat_Enterprise_Linux-7-Performance_Tuning_Guide-en-US.pdf.

Performance is determined by the following:

• CPU — priorities, scheduling, scheduler and other kernel parameters, number of CPUs
  Use these tools to see information to recognize a CPU shortage:
  vmstat interval [count]  colums: r = # of waiting processes that could be running, ca = # of context switches, us = % usertime, sy = % system time, id = % idle time.
  sar (system accounting).
  uptime (5, 10, 15 minute load averages).
  procinfo [-a] (all of the above and more.
• Memory — per process limits, swap space, some parameters
  Use these tools to see information to recognize a RAM shortage:
  vmstat interval [count]  colums: free = amount of avail RAM, so = # of pages swapped.
  free (check amount free, calculate %free, say with a script).
  swapon -s (shows amount of swap space used).

  (An excellent discussion of the Linux kernel's memory parameters you can examine and/or change can be found at www.westnet.com/~gsmith/content/linux-pdflush.htm.)

• Disks — number of controllers, hardware RAID, LVM design, file placement, I/O parameters such as DMA and number buffers
  Monitor disk, controller, and filesystem stats, to recognize poor disk layouts, controller/bus bottlenecks, spot explosive log files and disk hogs:
  i2cdump, iostat, hdparm, and smartctl.
  Monitor disk space with: df, quota, and du.
• Network — Memory config, number of NICs and their config, traffic shaping and other network parameters, network infrastructure (external to the host) upgrades
  Use to spot bandwidth and network hardware problems:
  ip -s link show  or  ifconfig  (monitor your network interfaces).
  netstat, ping, nc (netcat), traceroute, lsof, and nmap  (monitor overall network activity)
• Applications (servers) — Many servers have status options and/or separate status commands, or you can examine their log and configuration files.
  Examples of services to monitor include: httpd, DNS (named), FTP, SSH, DB, mail, etc.
  Load balancing (clusters) may be needed.

Useful tools: ps, top, w, /proc, signals (kill -l, man 7 signal), kill, killall, nice, renice, lsof, fuser.

Schedule non-time sensitive jobs via cron.

Process resource limits can be controlled with:

   ulimit [-a] [-H]

Where -a = show all limits and -H means show hard limits.

This limits are typically set in the login scripts (or from PAM).