Apache2 - High CPU Load

kar-bear

New Member
Hallo,

seit ca. 2 Monaten habe ich auf meinem Dedicated Server (Strato, 4 GB Ram, 2 x 2,2 GHz Dual-Core AMD Opteron(tm) Processor 1214 HE mit openSUSE 11.1 (x86_64), Plesk 9.2.3, Apache/2.2.10, mod_ssl/2.2.10, OpenSSL/0.9.8h, PHP/5.2.11 with Suhosin-Patch, mod_python/3.3.1, Python/2.6, mod_jk/1.2.26, mod_perl/2.0.4 Perl/v5.10.0) Probleme mit den httpd2-prefork Prozessen und einer CPU load von bis zu 50.00. Außerdem frisst sich der Ram immer mehr voll.
Bei einer CPU load von 8.00 fängt es dann langsam an das der Webseitenaufruf unertäglich langsam wird, bis es dann gar nicht mehr geht und sich der Apache neu startet. Nach einem Apache Neustart geht die CPU load wieder auf normale Werte unter 1.00 zurück, um sich dann, mit immer mehr httpd2-prefork Prozessen die teilweise 50% CPU und 3.1% Ram beanspruchen, wieder hochzuschrauben. Ich starte nun den Apache ständig neu wenn die CPU load über 10.00 geht.
Ich habe schon versucht einige Optimierungen an der Apache server-tuning.conf probiert (KeepAlive On/Off, MaxClients hoch und runter gesetzt, MaxRequestsPerChild hoch und runter gesetzt), habe unötige Apache Module entfernt und gestern Mysql optimiert. Die Mysql Optimierung hat auch schon Erfolge gebracht (die CPU load steigt nicht höher wie 10.00) aber ich bin weit entfernt von einer Auslastung (CPU load unter 1.00, ganz selten mal bis zu 5.00) die bei diesem Server 1 Monat lang normal war. Ich betreibe nur 2 Webseiten, auf dem Server, mit ungefähr 15 PHP/Mysql Scripten die zusammen auf ca. 2000 Besucher am Tag kommen. Eine der beiden Webseiten ist eine Videoseite mit Serverseitiger Videoumwandlung (ffmpeg).
Ende Juni 2009 habe ich diesen Server bezogen. Voher war ich 1 1/2 Jahre mit denselben 2 Webseiten auf einem anderen Strato Server mit nur 2 GB Ram und einem schwächeren Prozessor und hatte nie Probleme mit zu hoher Auslastung. Die Besucher der beiden Webseiten sind in den letzen Monaten auch nur mäßig gestiegen. Dort sehe ich also nicht die Ursache.
Zu dem Zeitpunkt als ich dieses Problem bemerkte wurden keine kürzlichen Veränderungen vorgenommen, außer ein Plesk Update auf Version 9.2.
Ich habe den Server auch schon komplett neu installiert, ohne Erfolg.
Außerdem fragte ich Strato, ob ein Hardwarefehler vorliegen könnte, was diese mit Nein beantworteten.
Nun einge Auszüge:

Top Ausgabe vor der Mysql Optimierung
Code:
top - 13:02:19 up 11:21,  1 user,  load average: 40.76, 32.53, 18.66
Tasks: 161 total,  41 running, 120 sleeping,   0 stopped,   0 zombie
Cpu(s): 97.3%us,  2.3%sy,  0.0%ni,  0.0%id,  0.0%wa,  0.0%hi,  0.3%si,  0.0%st
Mem:   4028796k total,  3329844k used,   698952k free,    49756k buffers
Swap:  2104496k total,    31480k used,  2073016k free,   828624k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 3037 mysql     20   0  283m  35m 5496 S   26  0.9  57:25.72 mysqld
10258 wwwrun    20   0  547m 132m 6488 R    5  3.4   3:00.99 httpd2-prefork
12211 wwwrun    20   0  538m 123m 6308 R    5  3.1   2:44.49 httpd2-prefork
12218 wwwrun    20   0  531m 116m 6304 R    5  3.0   2:39.61 httpd2-prefork
12389 wwwrun    20   0  517m 101m 6128 R    5  2.6   0:57.77 httpd2-prefork
12678 wwwrun    20   0  465m  50m 6388 R    5  1.3   0:22.72 httpd2-prefork
12764 wwwrun    20   0  491m  76m 6000 R    5  1.9   0:39.39 httpd2-prefork
12775 wwwrun    20   0  497m  80m 4792 R    5  2.1   0:45.64 httpd2-prefork
12947 wwwrun    20   0  480m  65m 5580 R    5  1.7   0:31.80 httpd2-prefork
13032 wwwrun    20   0  464m  49m 6104 R    5  1.2   0:19.62 httpd2-prefork
13158 wwwrun    20   0  456m  40m 5380 R    5  1.0   0:14.82 httpd2-prefork
13306 wwwrun    20   0  454m  38m 5452 R    5  1.0   0:12.42 httpd2-prefork
13545 wwwrun    20   0  441m  26m 6144 R    5  0.7   0:04.60 httpd2-prefork
12622 wwwrun    20   0  526m 110m 5396 R    4  2.8   1:03.90 httpd2-prefork
12685 wwwrun    20   0  503m  88m 6296 R    4  2.2   0:48.75 httpd2-prefork
12696 wwwrun    20   0  499m  83m 6028 R    4  2.1   0:43.23 httpd2-prefork
12726 wwwrun    20   0  509m  93m 5356 R    4  2.4   0:53.53 httpd2-prefork
12731 wwwrun    20   0  505m  89m 5500 R    4  2.3   0:48.73 httpd2-prefork
12802 wwwrun    20   0  484m  69m 5984 R    4  1.8   0:35.22 httpd2-prefork
11494 wwwrun    20   0  461m  46m 6556 R    4  1.2   1:54.36 httpd2-prefork
11876 wwwrun    20   0  519m 103m 6456 R    4  2.6   1:03.72 httpd2-prefork
12384 wwwrun    20   0  482m  66m 5564 R    4  1.7   2:10.10 httpd2-prefork
12471 wwwrun    20   0  548m 133m 6016 R    4  3.4   1:20.02 httpd2-prefork
12725 wwwrun    20   0  491m  75m 5432 R    4  1.9   0:39.67 httpd2-prefork
12727 wwwrun    20   0  472m  55m 6496 R    4  1.4   0:24.26 httpd2-prefork
12765 wwwrun    20   0  458m  41m 6472 R    4  1.1   0:14.86 httpd2-prefork
12766 wwwrun    20   0  464m  49m 6364 R    4  1.3   0:19.88 httpd2-prefork
12780 wwwrun    20   0  476m  61m 6340 R    4  1.6   0:28.86 httpd2-prefork
12945 wwwrun    20   0  473m  57m 6044 R    4  1.5   0:24.44 httpd2-prefork
12965 wwwrun    20   0  473m  58m 6132 R    4  1.5   0:27.34 httpd2-prefork
12971 wwwrun    20   0  476m  60m 5416 R    4  1.5   0:29.90 httpd2-prefork
12972 wwwrun    20   0  478m  63m 6180 R    4  1.6   0:28.42 httpd2-prefork
13030 wwwrun    20   0  473m  58m 5484 R    4  1.5   0:26.74 httpd2-prefork
13037 wwwrun    20   0  468m  53m 6296 R    4  1.4   0:21.34 httpd2-prefork
13064 wwwrun    20   0  467m  51m 6056 R    4  1.3   0:21.08 httpd2-prefork
13110 wwwrun    20   0  450m  35m 6168 R    4  0.9   0:09.56 httpd2-prefork
13154 wwwrun    20   0  438m  23m 6284 R    4  0.6   0:02.32 httpd2-prefork
13389 wwwrun    20   0  445m  30m 6100 R    4  0.8   0:05.16 httpd2-prefork
13642 wwwrun    20   0  436m  19m 4660 R    4  0.5   0:01.84 httpd2-prefork
13651 wwwrun    20   0  436m  20m 5480 R    4  0.5   0:01.02 httpd2-prefork
13604 wwwrun    20   0  437m  21m 6084 S    2  0.5   0:00.46 httpd2-prefork
13726 wwwrun    20   0  432m  18m 3568 S    1  0.5   0:00.04 httpd2-prefork
13135 wwwrun    20   0  437m  22m 6040 S    1  0.6   0:01.76 httpd2-prefork
13551 wwwrun    20   0  432m  19m 4156 S    1  0.5   0:02.00 httpd2-prefork
13594 wwwrun    20   0  437m  21m 5552 S    1  0.5   0:00.34 httpd2-prefork
13663 wwwrun    20   0  436m  20m 5436 S    1  0.5   0:00.82 httpd2-prefork
13717 wwwrun    20   0  432m  19m 3788 S    1  0.5   0:00.06 httpd2-prefork
    1 root      20   0   864  248  204 S    0  0.0   0:01.10 init
    2 root      15  -5     0    0    0 S    0  0.0   0:00.00 kthreadd
    3 root      RT  -5     0    0    0 S    0  0.0   0:00.00 migration/0
    4 root      15  -5     0    0    0 S    0  0.0   0:00.08 ksoftirqd/0
    5 root      RT  -5     0    0    0 S    0  0.0   0:00.00 migration/1
    6 root      15  -5     0    0    0 S    0  0.0   0:00.10 ksoftirqd/1

Top Ausgabe von heute nach der Mysql Optimierung
Code:
top - 15:52:09 up 2 days, 22:21,  1 user,  load average: 8.37, 4.95, 2.70
Tasks: 111 total,  10 running, 100 sleeping,   0 stopped,   1 zombie
Cpu(s): 99.3%us,  0.7%sy,  0.0%ni,  0.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   4019640k total,  3874648k used,   144992k free,    96084k buffers
Swap:  2104496k total,    14920k used,  2089576k free,  2983392k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
16403 wwwrun    20   0  527m 114m 4644 R   26  2.9   1:01.48 httpd2-prefork
16454 wwwrun    20   0  474m  62m 5068 R   25  1.6   0:28.20 httpd2-prefork
16457 wwwrun    20   0  483m  73m 7888 R   25  1.9   0:32.33 httpd2-prefork
16402 wwwrun    20   0  530m 117m 4588 R   23  3.0   1:01.84 httpd2-prefork
16608 wwwrun    20   0  465m  53m 5068 R   23  1.4   0:22.28 httpd2-prefork
16665 wwwrun    20   0  432m  19m 4728 R   23  0.5   0:02.04 httpd2-prefork
15996 wwwrun    20   0  546m 143m  14m R   23  3.7   2:27.99 httpd2-prefork
16423 wwwrun    20   0  500m  91m 8300 R   23  2.3   0:42.77 httpd2-prefork
16345 wwwrun    20   0  428m  17m 5480 S    6  0.4   1:20.36 httpd2-prefork
    1 root      20   0  1064   88   52 S    0  0.0   0:28.20 init
    2 root      15  -5     0    0    0 S    0  0.0   0:00.00 kthreadd
    3 root      RT  -5     0    0    0 S    0  0.0   0:00.08 migration/0
    4 root      15  -5     0    0    0 S    0  0.0   0:00.28 ksoftirqd/0
    5 root      RT  -5     0    0    0 S    0  0.0   0:00.10 migration/1
    6 root      15  -5     0    0    0 S    0  0.0   0:00.36 ksoftirqd/1
    7 root      15  -5     0    0    0 S    0  0.0   0:10.52 events/0
    8 root      15  -5     0    0    0 S    0  0.0   0:20.72 events/1
    9 root      15  -5     0    0    0 S    0  0.0   0:00.00 khelper
   10 root      15  -5     0    0    0 S    0  0.0   0:00.00 kintegrityd/0
   11 root      15  -5     0    0    0 S    0  0.0   0:00.00 kintegrityd/1
   12 root      15  -5     0    0    0 S    0  0.0   0:00.52 kblockd/0
   13 root      15  -5     0    0    0 S    0  0.0   0:06.62 kblockd/1
   14 root      15  -5     0    0    0 S    0  0.0   0:00.00 kacpid
   15 root      15  -5     0    0    0 S    0  0.0   0:00.00 kacpi_notify
   16 root      15  -5     0    0    0 S    0  0.0   0:00.00 cqueue
   17 root      15  -5     0    0    0 S    0  0.0   0:00.00 kseriod
   18 root      15  -5     0    0    0 S    0  0.0   0:52.77 kondemand/0
   19 root      15  -5     0    0    0 S    0  0.0   0:00.00 kondemand/1
   22 root      15  -5     0    0    0 S    0  0.0   2:53.51 kswapd0
   23 root      15  -5     0    0    0 S    0  0.0   0:00.00 aio/0
   24 root      15  -5     0    0    0 S    0  0.0   0:00.00 aio/1
   25 root      15  -5     0    0    0 S    0  0.0   0:00.00 kpsmoused
  194 root      15  -5     0    0    0 S    0  0.0   0:00.00 ksuspend_usbd
  196 root      15  -5     0    0    0 S    0  0.0   0:00.00 khubd
  225 root      15  -5     0    0    0 S    0  0.0   0:00.00 ata/0
  226 root      15  -5     0    0    0 S    0  0.0   0:00.00 ata/1
  230 root      15  -5     0    0    0 S    0  0.0   0:00.00 ata_aux
  242 root      15  -5     0    0    0 S    0  0.0   0:00.00 scsi_eh_0
  243 root      15  -5     0    0    0 S    0  0.0   0:00.00 scsi_eh_1
  246 root      15  -5     0    0    0 S    0  0.0   0:00.00 scsi_eh_2
  255 root      15  -5     0    0    0 S    0  0.0   0:00.00 scsi_eh_3
  546 root      15  -5     0    0    0 S    0  0.0   2:46.15 md1_raid1
  578 root      15  -5     0    0    0 S    0  0.0   1:34.79 kjournald
  642 root      16  -4 16788  236  168 S    0  0.0   0:00.16 udevd
 1101 root      15  -5     0    0    0 S    0  0.0   0:00.00 scsi_eh_4
 1102 root      15  -5     0    0    0 S    0  0.0   0:00.00 scsi_eh_5
 1233 root      15  -5     0    0    0 S    0  0.0   0:00.00 kauditd
 1259 root      15  -5     0    0    0 S    0  0.0   0:00.00 kstriped
 1264 root      15  -5     0    0    0 S    0  0.0   0:00.00 md0_raid1
 1707 root      20   0  4092  500  180 S    0  0.0   0:04.22 klogd
 1710 root      20   0 19892 1732  616 S    0  0.0   0:37.72 syslog-ng
 1711 messageb  20   0 21132  516  388 S    0  0.0   0:18.70 dbus-daemon
 2226 haldaemo  20   0 33104 1148  624 S    0  0.0   0:28.24 hald

Meine aktuelle server-tuning.conf
Code:
##
## Server-Pool Size Regulation (MPM specific)
##

# the MPM (multiprocessing module) is not a dynamically loadable module in the
# sense of other modules. It is a compile time decision which one is used. We
# provide different apache2 MPM packages, containing different httpd2 binaries
# compiled with the available MPMs. See APACHE_MPM in /etc/sysconfig/apache2.

# prefork MPM
<IfModule prefork.c>
	# number of server processes to start
	# http://httpd.apache.org/docs/2.2/mod/mpm_common.html#startservers
	StartServers         5
	# minimum number of server processes which are kept spare
	# http://httpd.apache.org/docs/2.2/mod/prefork.html#minspareservers
	MinSpareServers      5
	# maximum number of server processes which are kept spare
	# http://httpd.apache.org/docs/2.2/mod/prefork.html#maxspareservers
	MaxSpareServers     10
	# highest possible MaxClients setting for the lifetime of the Apache process.
	# http://httpd.apache.org/docs/2.2/mod/mpm_common.html#serverlimit
	ServerLimit        100
	# maximum number of server processes allowed to start
	# http://httpd.apache.org/docs/2.2/mod/mpm_common.html#maxclients
	MaxClients         100
	# maximum number of requests a server process serves
	# http://httpd.apache.org/docs/2.2/mod/mpm_common.html#maxrequestsperchild
	MaxRequestsPerChild  10000
</IfModule>

# worker MPM
<IfModule worker.c>
	# initial number of server processes to start
	# http://httpd.apache.org/docs/2.2/mod/mpm_common.html#startservers
	StartServers         3
	# minimum number of worker threads which are kept spare
	# http://httpd.apache.org/docs/2.2/mod/mpm_common.html#minsparethreads
	MinSpareThreads     25
	# maximum number of worker threads which are kept spare
	# http://httpd.apache.org/docs/2.2/mod/mpm_common.html#maxsparethreads
	MaxSpareThreads     75
	# upper limit on the configurable number of threads per child process
	# http://httpd.apache.org/docs/2.2/mod/mpm_common.html#threadlimit
	ThreadLimit         64
	# maximum number of simultaneous client connections
	# http://httpd.apache.org/docs/2.2/mod/mpm_common.html#maxclients
	MaxClients         150
	# number of worker threads created by each child process
	# http://httpd.apache.org/docs/2.2/mod/mpm_common.html#threadsperchild
	ThreadsPerChild     25
	# maximum number of requests a server process serves
	# http://httpd.apache.org/docs/2.2/mod/mpm_common.html#maxrequestsperchild
	MaxRequestsPerChild  10000
</IfModule>


#
# KeepAlive: Whether or not to allow persistent connections (more than
# one request per connection). Set to "Off" to deactivate.
#
KeepAlive On

#
# MaxKeepAliveRequests: The maximum number of requests to allow
# during a persistent connection. Set to 0 to allow an unlimited amount.
# We recommend you leave this number high, for maximum performance.
#
MaxKeepAliveRequests 100

#
# KeepAliveTimeout: Number of seconds to wait for the next request from the
# same client on the same connection.
#
KeepAliveTimeout 2

#
# EnableMMAP: Control whether memory-mapping is used to deliver
# files (assuming that the underlying OS supports it).
# The default is on; turn this off if you serve from NFS-mounted
# filesystems.  On some systems, turning it off (regardless of
# filesystem) can improve performance; for details, please see
# http://httpd.apache.org/docs-2.2/mod/core.html#enablemmap
#
#EnableMMAP off

#
# EnableSendfile: Control whether the sendfile kernel support is
# used  to deliver files (assuming that the OS supports it).
# The default is on; turn this off if you serve from NFS-mounted
# filesystems.  Please see
# http://httpd.apache.org/docs-2.2/mod/core.html#enablesendfile
#
#EnableSendfile off

<IfModule mod_setenvif.c>
	#
	# The following directives modify normal HTTP response behavior to
	# handle known problems with browser implementations.
	#
	BrowserMatch "Mozilla/2" nokeepalive
	BrowserMatch "MSIE 4\.0b2;" nokeepalive downgrade-1.0 force-response-1.0
	BrowserMatch "RealPlayer 4\.0" force-response-1.0
	BrowserMatch "Java/1\.0" force-response-1.0
	BrowserMatch "JDK/1\.0" force-response-1.0

	#
	# The following directive disables redirects on non-GET requests for
	# a directory that does not include the trailing slash.  This fixes a
	# problem with Microsoft WebFolders which does not appropriately handle
	# redirects for folders with DAV methods.
	# Same deal with Apple's DAV filesystem and Gnome VFS support for DAV.
	#
	BrowserMatch "Microsoft Data Access Internet Publishing Provider" redirect-carefully
	BrowserMatch "^WebDrive" redirect-carefully
	BrowserMatch "^WebDAVFS/1.[012]" redirect-carefully
	BrowserMatch "^gnome-vfs" redirect-carefully
</IfModule>

Apache error log Auszug während einer high CPU load Phase
Code:
*** glibc detected *** /usr/sbin/httpd2-prefork: double free or corruption (!prev): 0x00007f9a6c448930 ***
======= Backtrace: =========
/lib64/libc.so.6[0x7f9a6991c108]
/lib64/libc.so.6(cfree+0x76)[0x7f9a6991dc66]
/usr/lib64/apache2/mod_php5.so(php_module_shutdown+0x87)[0x7f9a664d89e7]
/usr/lib64/apache2/mod_php5.so(php_module_shutdown_wrapper+0x9)[0x7f9a664d8a39]
/usr/lib64/apache2/mod_php5.so[0x7f9a665940a1]
/usr/lib64/libapr-1.so.0(apr_pool_destroy+0x87)[0x7f9a6a4800a7]
/usr/sbin/httpd2-prefork[0x7f9a6b510a3e]
/usr/sbin/httpd2-prefork[0x7f9a6b5110db]
/lib64/libpthread.so.0[0x7f9a69e13a90]
/lib64/libc.so.6(strlen+0xa5)[0x7f9a699245f5]
/lib64/libc.so.6(__strdup+0x16)[0x7f9a69924296]
/usr/lib64/apache2/mod_php5.so[0x7f9a664dafca]
/usr/lib64/apache2/mod_php5.so(zend_error+0x5cb)[0x7f9a6651ef5b]
/usr/lib64/apache2/mod_php5.so[0x7f9a66557850]
/usr/lib64/apache2/mod_php5.so[0x7f9a66577a02]
/usr/lib64/apache2/mod_php5.so(execute+0x1a4)[0x7f9a66542904]
/usr/lib64/php5/extensions/suhosin.so[0x7f9a5aaf3036]
/usr/lib64/apache2/mod_php5.so[0x7f9a6655941e]
/usr/lib64/apache2/mod_php5.so(execute+0x1a4)[0x7f9a66542904]
/usr/lib64/php5/extensions/suhosin.so[0x7f9a5aaf3036]
/usr/lib64/apache2/mod_php5.so[0x7f9a6655941e]
/usr/lib64/apache2/mod_php5.so(execute+0x1a4)[0x7f9a66542904]
/usr/lib64/php5/extensions/suhosin.so[0x7f9a5aaf3036]
/usr/lib64/apache2/mod_php5.so(zend_execute_scripts+0x168)[0x7f9a6651e798]...
...7fff57662000-7fff5768a000 rw-p 7ffffffd7000 00:00 0                      [stack]
7fff57706000-7fff57707000 r-xp 7fff57706000 00:00 0                      [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
[Thu Oct 22 17:02:01 2009] [warn] child process 20610 still did not exit, sending a SIGTERM
[Thu Oct 22 17:02:01 2009] [warn] child process 18626 still did not exit, sending a SIGTERM
[Thu Oct 22 17:02:01 2009] [warn] child process 20379 still did not exit, sending a SIGTERM
[Thu Oct 22 17:02:01 2009] [warn] child process 20296 still did not exit, sending a SIGTERM
[Thu Oct 22 17:02:01 2009] [warn] child process 19858 still did not exit, sending a SIGTERM
[Thu Oct 22 17:02:03 2009] [warn] child process 20610 still did not exit, sending a SIGTERM
[Thu Oct 22 17:02:03 2009] [warn] child process 18626 still did not exit, sending a SIGTERM
[Thu Oct 22 17:02:03 2009] [warn] child process 20379 still did not exit, sending a SIGTERM
[Thu Oct 22 17:02:03 2009] [warn] child process 20296 still did not exit, sending a SIGTERM
[Thu Oct 22 17:02:05 2009] [error] child process 20610 still did not exit, sending a SIGKILL
[Thu Oct 22 17:02:05 2009] [error] child process 18626 still did not exit, sending a SIGKILL
[Thu Oct 22 17:02:05 2009] [error] child process 20379 still did not exit, sending a SIGKILL
[Thu Oct 22 17:02:05 2009] [error] child process 20296 still did not exit, sending a SIGKILL
[Thu Oct 22 17:02:06 2009] [notice] caught SIGTERM, shutting down

Ich weiss nicht was ich noch machen kann. Bitte gebt mir einen Rat.
Vielen Dank! :)
 
Back
Top