VServer schmiert ab / BUG: soft lockup - CPU#1 stuck for 26s!

d3p · Dec 9, 2015

Netcup VServer schmiert ab / BUG: soft lockup - CPU#1 stuck for 26s!

Hallo,

ich bin vor kurzem von einem dedizierten Rootserver auf einen virtuellen gewechselt, und habe seitdem immer wieder Aussetzer bzw. Abstürze.

Die Virtualisierungstechnik ist KVM.

Im Syslog findet sich folgendes:

Code:

Dec  9 12:34:32 server1 kernel: [726172.400038] hrtimer: interrupt took 19235578 ns
Dec  9 12:34:32 server1 kernel: [726148.000014] INFO: rcu_sched detected stalls on CPUs/tasks: { 0} (detected by 2, t=16983 jiffies)
Dec  9 12:34:32 server1 kernel: [726148.000014] sending NMI to all CPUs:
Dec  9 12:34:32 server1 kernel: [726148.000014] NMI backtrace for cpu 2
Dec  9 12:34:32 server1 kernel: [726148.000014] CPU 2 
Dec  9 12:34:32 server1 kernel: [726148.000014] Modules linked in: nfnetlink_log nfnetlink nfsd nfs nfs_acl auth_rpcgss fscache lockd sunrpc loop snd_pcm snd_page_alloc snd_timer snd soundcore i2c_piix4 i2c_core psmouse joydev serio_raw pcspkr evdev crc32c_intel ghash_clmulni_intel processor virtio_balloon thermal_sys button container aesni_intel aes_x86_64 aes_generic cryptd ext4 crc16 jbd2 mbcache usbhid hid sg sr_mod cdrom ata_generic virtio_blk floppy e1000 uhci_hcd ehci_hcd ata_piix libata scsi_mod virtio_pci virtio_ring virtio usbcore usb_common [last unloaded: scsi_wait_scan]
Dec  9 12:34:32 server1 kernel: [726148.000014] 
Dec  9 12:34:32 server1 kernel: [726148.000014] Pid: 14, comm: kworker/2:0 Not tainted 3.2.0-4-amd64 #1 Debian 3.2.68-1+deb7u6 Red Hat KVM
Dec  9 12:34:32 server1 kernel: [726148.000014] RIP: 0010:[<ffffffff8102b3e5>]  [<ffffffff8102b3e5>] native_write_msr_safe+0x6/0x9
Dec  9 12:34:32 server1 kernel: [726148.000014] RSP: 0018:ffff88033fd03dc0  EFLAGS: 00000046
Dec  9 12:34:32 server1 kernel: [726148.000014] RAX: 0000000000000400 RBX: 0000000000000400 RCX: 0000000000000830
Dec  9 12:34:32 server1 kernel: [726148.000014] RDX: 0000000000000002 RSI: 0000000000000400 RDI: 0000000000000830
Dec  9 12:34:32 server1 kernel: [726148.000014] RBP: 000000000000cc42 R08: 0000000000000000 R09: ffffffff8168f1e0
Dec  9 12:34:32 server1 kernel: [726148.000014] R10: 0000000000000000 R11: 0000000034a882d1 R12: 0000000000000002
Dec  9 12:34:32 server1 kernel: [726148.000014] R13: 0000000000000400 R14: 0000000000000002 R15: 0000000000080000
Dec  9 12:34:32 server1 kernel: [726148.000014] FS:  0000000000000000(0000) GS:ffff88033fd00000(0000) knlGS:0000000000000000
Dec  9 12:34:32 server1 kernel: [726148.000014] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Dec  9 12:34:32 server1 kernel: [726148.000014] CR2: 00007f5b7bb88000 CR3: 0000000331946000 CR4: 00000000001406e0
Dec  9 12:34:32 server1 kernel: [726148.000014] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Dec  9 12:34:32 server1 kernel: [726148.000014] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Dec  9 12:34:32 server1 kernel: [726148.000014] Process kworker/2:0 (pid: 14, threadinfo ffff8803328e4000, task ffff8803328c4740)
Dec  9 12:34:32 server1 kernel: [726148.000014] Stack:
Dec  9 12:34:32 server1 kernel: [726148.000014]  ffffffff81027884 0000000000000096 ffffffff8102795c 0000000000000002
Dec  9 12:34:32 server1 kernel: [726148.000014]  ffffffff8168f1e0 ffff88033fd03e38 0000000000002710 ffffffff81622280
Dec  9 12:34:32 server1 kernel: [726148.000014]  ffffffff81622380 0000000000000001 ffffffff81624380 0000000000000000
Dec  9 12:34:32 server1 kernel: [726148.000014] Call Trace:
Dec  9 12:34:32 server1 kernel: [726148.000014]  <IRQ> 
Dec  9 12:34:32 server1 kernel: [726148.000014]  [<ffffffff81027884>] ? native_x2apic_icr_write+0x12/0x16
Dec  9 12:34:32 server1 kernel: [726148.000014]  [<ffffffff8102795c>] ? __x2apic_send_IPI_mask+0x7b/0xb7
Dec  9 12:34:32 server1 kernel: [726148.000014]  [<ffffffff8102514e>] ? arch_trigger_all_cpu_backtrace+0x4d/0x7b
Dec  9 12:34:32 server1 kernel: [726148.000014]  [<ffffffff81096648>] ? __rcu_pending+0x21a/0x358
Dec  9 12:34:32 server1 kernel: [726148.000014]  [<ffffffff81011a6d>] ? arch_local_irq_save+0x5/0x13
Dec  9 12:34:32 server1 kernel: [726148.000014]  [<ffffffff8106c560>] ? tick_nohz_handler+0xd0/0xd0
Dec  9 12:34:32 server1 kernel: [726148.000014]  [<ffffffff81096ab3>] ? rcu_check_callbacks+0x90/0xcc
Dec  9 12:34:32 server1 kernel: [726148.000014]  [<ffffffff81052dde>] ? update_process_times+0x31/0x63
Dec  9 12:34:32 server1 kernel: [726148.000014]  [<ffffffff8106c5ca>] ? tick_sched_timer+0x6a/0x90
Dec  9 12:34:32 server1 kernel: [726148.000014]  [<ffffffff8106275a>] ? __run_hrtimer+0xac/0x135
Dec  9 12:34:32 server1 kernel: [726148.000014]  [<ffffffff81062e44>] ? hrtimer_interrupt+0xd7/0x1b1
Dec  9 12:34:32 server1 kernel: [726148.000014]  [<ffffffff813584ec>] ? call_softirq+0x1c/0x30
Dec  9 12:34:32 server1 kernel: [726148.000014]  [<ffffffff81024757>] ? smp_apic_timer_interrupt+0x6f/0x82
Dec  9 12:34:32 server1 kernel: [726148.000014]  [<ffffffff81356d5e>] ? apic_timer_interrupt+0x6e/0x80
Dec  9 12:34:32 server1 kernel: [726148.000014]  <EOI> 
Dec  9 12:34:32 server1 kernel: [726148.000014]  [<ffffffff810713ce>] ? arch_local_irq_restore+0x2/0x8
Dec  9 12:34:32 server1 kernel: [726148.000014]  [<ffffffff81351267>] ? _raw_spin_unlock_irqrestore+0xe/0xf
Dec  9 12:34:32 server1 kernel: [726148.000014]  [<ffffffffa00c0192>] ? e1000_update_stats+0x926/0x931 [e1000]
Dec  9 12:34:32 server1 kernel: [726148.000014]  [<ffffffff8103665c>] ? should_resched+0x5/0x23
Dec  9 12:34:32 server1 kernel: [726148.000014]  [<ffffffffa00c04dc>] ? e1000_watchdog+0x33f/0x4ac [e1000]
Dec  9 12:34:32 server1 kernel: [726148.000014]  [<ffffffff8105b717>] ? process_one_work+0x161/0x269
Dec  9 12:34:32 server1 kernel: [726148.000014]  [<ffffffff8105c6e0>] ? worker_thread+0xc2/0x145
Dec  9 12:34:32 server1 kernel: [726148.000014]  [<ffffffff8105c61e>] ? manage_workers.isra.25+0x15b/0x15b
Dec  9 12:34:32 server1 kernel: [726148.000014]  [<ffffffff8105f829>] ? kthread+0x76/0x7e
Dec  9 12:34:32 server1 kernel: [726148.000014]  [<ffffffff813583f4>] ? kernel_thread_helper+0x4/0x10
Dec  9 12:34:32 server1 kernel: [726148.000014]  [<ffffffff8105f7b3>] ? kthread_worker_fn+0x139/0x139
Dec  9 12:34:32 server1 kernel: [726148.000014]  [<ffffffff813583f0>] ? gs_change+0x13/0x13
Dec  9 12:34:32 server1 kernel: [726148.000014] Code: 89 d0 89 0f 48 c1 e0 20 48 09 f0 c3 89 f9 0f 32 45 31 c0 89 c7 48 89 d0 44 89 06 48 c1 e0 20 89 f9 48 09 c8 c3 89 f0 89 f9 0f 30 <31> c0 c3 89 f9 0f 33 89 c7 48 89 d0 48 c1 e0 20 89 f9 48 09 c8 
Dec  9 12:34:32 server1 kernel: [726148.000014] Call Trace:
Dec  9 12:34:32 server1 kernel: [726148.000014]  <IRQ>  [<ffffffff81027884>] ? native_x2apic_icr_write+0x12/0x16
Dec  9 12:34:32 server1 kernel: [726148.000014]  [<ffffffff8102795c>] ? __x2apic_send_IPI_mask+0x7b/0xb7
Dec  9 12:34:32 server1 kernel: [726148.000014]  [<ffffffff8102514e>] ? arch_trigger_all_cpu_backtrace+0x4d/0x7b
Dec  9 12:34:32 server1 kernel: [726148.000014]  [<ffffffff81096648>] ? __rcu_pending+0x21a/0x358
Dec  9 12:34:32 server1 kernel: [726148.000014]  [<ffffffff81011a6d>] ? arch_local_irq_save+0x5/0x13
Dec  9 12:34:32 server1 kernel: [726148.000014]  [<ffffffff8106c560>] ? tick_nohz_handler+0xd0/0xd0
Dec  9 12:34:32 server1 kernel: [726148.000014]  [<ffffffff81096ab3>] ? rcu_check_callbacks+0x90/0xcc
Dec  9 12:34:32 server1 kernel: [726148.000014]  [<ffffffff81052dde>] ? update_process_times+0x31/0x63
Dec  9 12:34:32 server1 kernel: [726148.000014]  [<ffffffff8106c5ca>] ? tick_sched_timer+0x6a/0x90
Dec  9 12:34:32 server1 kernel: [726148.000014]  [<ffffffff8106275a>] ? __run_hrtimer+0xac/0x135
Dec  9 12:34:32 server1 kernel: [726148.000014]  [<ffffffff81062e44>] ? hrtimer_interrupt+0xd7/0x1b1
Dec  9 12:34:32 server1 kernel: [726148.000014]  [<ffffffff813584ec>] ? call_softirq+0x1c/0x30
Dec  9 12:34:32 server1 kernel: [726148.000014]  [<ffffffff81024757>] ? smp_apic_timer_interrupt+0x6f/0x82
Dec  9 12:34:32 server1 kernel: [726148.000014]  [<ffffffff81356d5e>] ? apic_timer_interrupt+0x6e/0x80
Dec  9 12:34:32 server1 kernel: [726148.000014]  <EOI>  [<ffffffff810713ce>] ? arch_local_irq_restore+0x2/0x8
Dec  9 12:34:32 server1 kernel: [726148.000014]  [<ffffffff81351267>] ? _raw_spin_unlock_irqrestore+0xe/0xf
Dec  9 12:34:32 server1 kernel: [726148.000014]  [<ffffffffa00c0192>] ? e1000_update_stats+0x926/0x931 [e1000]
Dec  9 12:34:32 server1 kernel: [726148.000014]  [<ffffffff8103665c>] ? should_resched+0x5/0x23
Dec  9 12:34:32 server1 kernel: [726148.000014]  [<ffffffffa00c04dc>] ? e1000_watchdog+0x33f/0x4ac [e1000]
Dec  9 12:34:32 server1 kernel: [726148.000014]  [<ffffffff8105b717>] ? process_one_work+0x161/0x269
Dec  9 12:34:32 server1 kernel: [726148.000014]  [<ffffffff8105c6e0>] ? worker_thread+0xc2/0x145
Dec  9 12:34:32 server1 kernel: [726148.000014]  [<ffffffff8105c61e>] ? manage_workers.isra.25+0x15b/0x15b
Dec  9 12:34:32 server1 kernel: [726148.000014]  [<ffffffff8105f829>] ? kthread+0x76/0x7e
Dec  9 12:34:32 server1 kernel: [726148.000014]  [<ffffffff813583f4>] ? kernel_thread_helper+0x4/0x10
Dec  9 12:34:32 server1 kernel: [726148.000014]  [<ffffffff8105f7b3>] ? kthread_worker_fn+0x139/0x139
Dec  9 12:34:32 server1 kernel: [726148.000014]  [<ffffffff813583f0>] ? gs_change+0x13/0x13
Dec  9 12:34:32 server1 kernel: [726212.901000] NMI backtrace for cpu 0
Dec  9 12:34:32 server1 kernel: [726212.901000] CPU 0 
Dec  9 12:34:32 server1 kernel: [726212.901000] Modules linked in: nfnetlink_log nfnetlink nfsd nfs nfs_acl auth_rpcgss fscache lockd sunrpc loop snd_pcm snd_page_alloc snd_timer snd soundcore i2c_piix4 i2c_core psmouse joydev serio_raw pcspkr evdev crc32c_intel ghash_clmulni_intel processor virtio_balloon thermal_sys button container aesni_intel aes_x86_64 aes_generic cryptd ext4 crc16 jbd2 mbcache usbhid hid sg sr_mod cdrom ata_generic virtio_blk floppy e1000 uhci_hcd ehci_hcd ata_piix libata scsi_mod virtio_pci virtio_ring virtio usbcore usb_common [last unloaded: scsi_wait_scan]
Dec  9 12:34:32 server1 kernel: [726212.901000] 
Dec  9 12:34:32 server1 kernel: [726212.901000] Pid: 0, comm: swapper/0 Not tainted 3.2.0-4-amd64 #1 Debian 3.2.68-1+deb7u6 Red Hat KVM
Dec  9 12:34:32 server1 kernel: [726212.901000] RIP: 0010:[<ffffffffa00be800>]  [<ffffffffa00be800>] e1000_alloc_rx_buffers+0x27b/0x28a [e1000]
Dec  9 12:34:32 server1 kernel: [726212.901000] RSP: 0018:ffff88033fc03ca0  EFLAGS: 00010286
Dec  9 12:34:32 server1 kernel: [726212.901000] RAX: ffffc900025c2818 RBX: ffff88032fa50740 RCX: 00000000bcc7a000
Dec  9 12:34:32 server1 kernel: [726212.901000] RDX: 00000000000005f2 RSI: 00000000bcc7a000 RDI: ffff88032fa50740
Dec  9 12:34:32 server1 kernel: [726212.901000] RBP: ffffc900025b0ca0 R08: 0000000000000000 R09: 00000000000005f2
Dec  9 12:34:32 server1 kernel: [726212.901000] R10: ffff8801da22b9c0 R11: ffff880332a24090 R12: ffff88015fb692c0
Dec  9 12:34:32 server1 kernel: [726212.901000] R13: 00000000000000ed R14: ffff88032f8a7780 R15: 0000000000000000
Dec  9 12:34:32 server1 kernel: [726212.901000] FS:  0000000000000000(0000) GS:ffff88033fc00000(0000) knlGS:0000000000000000
Dec  9 12:34:32 server1 kernel: [726212.901000] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Dec  9 12:34:32 server1 kernel: [726212.901000] CR2: 00007f21535ba000 CR3: 000000020fd05000 CR4: 00000000001406f0
Dec  9 12:34:32 server1 kernel: [726212.901000] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Dec  9 12:34:32 server1 kernel: [726212.901000] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Dec  9 12:34:32 server1 kernel: [726212.901000] Process swapper/0 (pid: 0, threadinfo ffffffff81600000, task ffffffff8160d020)
Dec  9 12:34:32 server1 kernel: [726212.901000] Stack:
Dec  9 12:34:32 server1 kernel: [726212.901000]  ffff88033fc03cd0 00000000bcc7a000 00000000000005f2 ffffffff00000000
Dec  9 12:34:32 server1 kernel: [726212.901000]  000005f229977bc0 ffff880332a24090 ffff88032fa50000 00000000000005f2
Dec  9 12:34:32 server1 kernel: [726212.901000]  ffff88003743fee0 ffff88032fa50740 ffff880329977bc0 ffff88032f8a7780
Dec  9 12:34:32 server1 kernel: [726212.901000] Call Trace:
Dec  9 12:34:32 server1 kernel: [726212.901000]  <IRQ> 
Dec  9 12:34:32 server1 kernel: [726212.901000]  [<ffffffffa00beb25>] ? e1000_clean_rx_irq+0x316/0x39e [e1000]
Dec  9 12:34:32 server1 kernel: [726212.901000]  [<ffffffffa00be433>] ? e1000_clean+0x2a9/0x3fb [e1000]
Dec  9 12:34:32 server1 kernel: [726212.901000]  [<ffffffff8102b2d2>] ? kvm_clock_read+0x17/0x1a
Dec  9 12:34:32 server1 kernel: [726212.901000]  [<ffffffff8103b0c7>] ? check_preempt_curr+0x36/0x5f
Dec  9 12:34:32 server1 kernel: [726212.901000]  [<ffffffff810713ed>] ? arch_local_irq_save+0x11/0x17
Dec  9 12:34:32 server1 kernel: [726212.901000]  [<ffffffff81291d23>] ? net_rx_action+0xa1/0x1aa
Dec  9 12:34:32 server1 kernel: [726212.901000]  [<ffffffff81245fe3>] ? add_interrupt_randomness+0x38/0x155
Dec  9 12:34:32 server1 kernel: [726212.901000]  [<ffffffff8104c48e>] ? __do_softirq+0xb9/0x177
Dec  9 12:34:32 server1 kernel: [726212.901000]  [<ffffffff813584ec>] ? call_softirq+0x1c/0x30
Dec  9 12:34:32 server1 kernel: [726212.901000]  [<ffffffff8100fa91>] ? do_softirq+0x3c/0x7b
Dec  9 12:34:32 server1 kernel: [726212.901000]  [<ffffffff8104c6f6>] ? irq_exit+0x3c/0x99
Dec  9 12:34:32 server1 kernel: [726212.901000]  [<ffffffff8100f66a>] ? do_IRQ+0x82/0x98
Dec  9 12:34:32 server1 kernel: [726212.901000]  [<ffffffff813514ee>] ? common_interrupt+0x6e/0x6e
Dec  9 12:34:32 server1 kernel: [726212.901000]  <EOI> 
Dec  9 12:34:32 server1 kernel: [726212.901000]  [<ffffffff8102b36c>] ? native_safe_halt+0x2/0x3
Dec  9 12:34:32 server1 kernel: [726212.901000]  [<ffffffff81014938>] ? default_idle+0x47/0x7f
Dec  9 12:34:32 server1 kernel: [726212.901000]  [<ffffffff8100d24c>] ? cpu_idle+0xaf/0xf2
Dec  9 12:34:32 server1 kernel: [726212.901000]  [<ffffffff816aab3b>] ? start_kernel+0x3bd/0x3c8
Dec  9 12:34:32 server1 kernel: [726212.901000]  [<ffffffff816aa140>] ? early_idt_handlers+0x140/0x140
Dec  9 12:34:32 server1 kernel: [726212.901000]  [<ffffffff816aa3c4>] ? x86_64_start_kernel+0x104/0x111
Dec  9 12:34:32 server1 kernel: [726212.901000] Code: fe ff ff 45 39 6e 18 74 22 45 85 ed 45 89 6e 18 75 04 45 8b 6e 14 41 ff cd 0f ae f8 41 0f b7 46 36 48 03 83 58 03 00 00 44 89 28 <48> 83 c4 48 5b 5d 41 5c 41 5d 41 5e 41 5f c3 41 57 48 8d 87 58 
Dec  9 12:34:32 server1 kernel: [726212.901000] Call Trace:
Dec  9 12:34:32 server1 kernel: [726212.901000]  <IRQ>  [<ffffffffa00beb25>] ? e1000_clean_rx_irq+0x316/0x39e [e1000]
Dec  9 12:34:32 server1 kernel: [726212.901000]  [<ffffffffa00be433>] ? e1000_clean+0x2a9/0x3fb [e1000]
Dec  9 12:34:32 server1 kernel: [726212.901000]  [<ffffffff8102b2d2>] ? kvm_clock_read+0x17/0x1a
Dec  9 12:34:32 server1 kernel: [726212.901000]  [<ffffffff8103b0c7>] ? check_preempt_curr+0x36/0x5f
Dec  9 12:34:32 server1 kernel: [726212.901000]  [<ffffffff810713ed>] ? arch_local_irq_save+0x11/0x17
Dec  9 12:34:32 server1 kernel: [726212.901000]  [<ffffffff81291d23>] ? net_rx_action+0xa1/0x1aa
Dec  9 12:34:32 server1 kernel: [726212.901000]  [<ffffffff81245fe3>] ? add_interrupt_randomness+0x38/0x155
Dec  9 12:34:32 server1 kernel: [726212.901000]  [<ffffffff8104c48e>] ? __do_softirq+0xb9/0x177
Dec  9 12:34:32 server1 kernel: [726212.901000]  [<ffffffff813584ec>] ? call_softirq+0x1c/0x30
Dec  9 12:34:32 server1 kernel: [726212.901000]  [<ffffffff8100fa91>] ? do_softirq+0x3c/0x7b
Dec  9 12:34:32 server1 kernel: [726212.901000]  [<ffffffff8104c6f6>] ? irq_exit+0x3c/0x99
Dec  9 12:34:32 server1 kernel: [726212.901000]  [<ffffffff8100f66a>] ? do_IRQ+0x82/0x98
Dec  9 12:34:32 server1 kernel: [726212.901000]  [<ffffffff813514ee>] ? common_interrupt+0x6e/0x6e
Dec  9 12:34:32 server1 kernel: [726212.901000]  <EOI>  [<ffffffff8102b36c>] ? native_safe_halt+0x2/0x3
Dec  9 12:34:32 server1 kernel: [726212.901000]  [<ffffffff81014938>] ? default_idle+0x47/0x7f
Dec  9 12:34:32 server1 kernel: [726212.901000]  [<ffffffff8100d24c>] ? cpu_idle+0xaf/0xf2
Dec  9 12:34:32 server1 kernel: [726212.901000]  [<ffffffff816aab3b>] ? start_kernel+0x3bd/0x3c8
Dec  9 12:34:32 server1 kernel: [726212.901000]  [<ffffffff816aa140>] ? early_idt_handlers+0x140/0x140
Dec  9 12:34:32 server1 kernel: [726212.901000]  [<ffffffff816aa3c4>] ? x86_64_start_kernel+0x104/0x111
Dec  9 12:34:32 server1 kernel: [726212.900571] NMI backtrace for cpu 1
Dec  9 12:34:32 server1 kernel: [726212.900571] CPU 1 
Dec  9 12:34:32 server1 kernel: [726212.900571] Modules linked in: nfnetlink_log nfnetlink nfsd nfs nfs_acl auth_rpcgss fscache lockd sunrpc loop snd_pcm snd_page_alloc snd_timer snd soundcore i2c_piix4 i2c_core psmouse joydev serio_raw pcspkr evdev crc32c_intel ghash_clmulni_intel processor virtio_balloon thermal_sys button container aesni_intel aes_x86_64 aes_generic cryptd ext4 crc16 jbd2 mbcache usbhid hid sg sr_mod cdrom ata_generic virtio_blk floppy e1000 uhci_hcd ehci_hcd ata_piix libata scsi_mod virtio_pci virtio_ring virtio usbcore usb_common [last unloaded: scsi_wait_scan]
Dec  9 12:34:32 server1 kernel: [726212.900571] 
Dec  9 12:34:32 server1 kernel: [726212.900571] Pid: 2471, comm: nginx Not tainted 3.2.0-4-amd64 #1 Debian 3.2.68-1+deb7u6 Red Hat KVM
Dec  9 12:34:32 server1 kernel: [726212.900571] RIP: 0010:[<ffffffffa00be109>]  [<ffffffffa00be109>] e1000_xmit_frame+0xa9a/0xb1b [e1000]
Dec  9 12:34:32 server1 kernel: [726212.900571] RSP: 0018:ffff88032f50b988  EFLAGS: 00010286
Dec  9 12:34:32 server1 kernel: [726212.900571] RAX: ffffc900025c3818 RBX: ffff88003768fc70 RCX: ffff88003743e430
Dec  9 12:34:32 server1 kernel: [726212.900571] RDX: 0000000000000044 RSI: 0000000022100000 RDI: 000000002210001f
Dec  9 12:34:32 server1 kernel: [726212.900571] RBP: ffff88032f8a77c0 R08: 0000000000000000 R09: 0000000025e2adc7
Dec  9 12:34:32 server1 kernel: [726212.900571] R10: 0000000000000000 R11: ffff88032fa50000 R12: ffff88032fa50000
Dec  9 12:34:32 server1 kernel: [726212.900571] R13: ffffc900025aac90 R14: 0000000000000000 R15: 0000000000000043
Dec  9 12:34:32 server1 kernel: [726212.900571] FS:  00007f4adf9f7700(0000) GS:ffff88033fc80000(0000) knlGS:0000000000000000
Dec  9 12:34:32 server1 kernel: [726212.900571] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Dec  9 12:34:32 server1 kernel: [726212.900571] CR2: 00007f2153410000 CR3: 00000002cf5b7000 CR4: 00000000001406e0
Dec  9 12:34:32 server1 kernel: [726212.900571] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Dec  9 12:34:32 server1 kernel: [726212.900571] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Dec  9 12:34:32 server1 kernel: [726212.900571] Process nginx (pid: 2471, threadinfo ffff88032f50a000, task ffff88032f139100)
Dec  9 12:34:32 server1 kernel: [726212.900571] Stack:
Dec  9 12:34:32 server1 kernel: [726212.900571]  ffff880059adeab8 ffff88032fa50000 0000000000000dc7 0000100000000009
Dec  9 12:34:32 server1 kernel: [726212.900571]  ffff88030000001f ffff88032fa50740 0000000000000001 0000000000000001
Dec  9 12:34:32 server1 kernel: [726212.900571]  0000004100000002 ffff880332a24000 ffff880332a24090 0000000000000001
Dec  9 12:34:32 server1 kernel: [726212.900571] Call Trace:
Dec  9 12:34:32 server1 kernel: [726212.900571]  [<ffffffff81292b85>] ? dev_hard_start_xmit+0x3d2/0x518
Dec  9 12:34:32 server1 kernel: [726212.900571]  [<ffffffff812a920f>] ? sch_direct_xmit+0x61/0x135
Dec  9 12:34:32 server1 kernel: [726212.900571]  [<ffffffff81292f8d>] ? dev_queue_xmit+0x2c2/0x46b
Dec  9 12:34:32 server1 kernel: [726212.900571]  [<ffffffff8102bb5c>] ? pvclock_clocksource_read+0x42/0xb2
Dec  9 12:34:32 server1 kernel: [726212.900571]  [<ffffffff812bbca6>] ? ip_finish_output2+0x1bc/0x1f9
Dec  9 12:34:32 server1 kernel: [726212.900571]  [<ffffffff812bd06d>] ? ip_queue_xmit+0x28e/0x2b9
Dec  9 12:34:32 server1 kernel: [726212.900571]  [<ffffffff812ce3a1>] ? tcp_transmit_skb+0x6e6/0x719
Dec  9 12:34:32 server1 kernel: [726212.900571]  [<ffffffff812cee36>] ? tcp_write_xmit+0x6a0/0x7a9
Dec  9 12:34:32 server1 kernel: [726212.900571]  [<ffffffff810eba4e>] ? virt_to_head_page+0x6/0x29
Dec  9 12:34:32 server1 kernel: [726212.900571]  [<ffffffff810ebcd7>] ? virt_to_cache+0x7/0x23
Dec  9 12:34:32 server1 kernel: [726212.900571]  [<ffffffff810ebd09>] ? ksize+0x16/0x1c
Dec  9 12:34:32 server1 kernel: [726212.900571]  [<ffffffff8134fe9c>] ? _cond_resched+0x7/0x1c
Dec  9 12:34:32 server1 kernel: [726212.900571]  [<ffffffff812cef80>] ? __tcp_push_pending_frames+0x18/0x44
Dec  9 12:34:32 server1 kernel: [726212.900571]  [<ffffffff812c476b>] ? tcp_sendmsg+0x64d/0x72d
Dec  9 12:34:32 server1 kernel: [726212.900571]  [<ffffffff81280812>] ? sock_aio_write+0x121/0x135
Dec  9 12:34:32 server1 kernel: [726212.900571]  [<ffffffff810fae8c>] ? do_sync_write+0xb4/0xec
Dec  9 12:34:32 server1 kernel: [726212.900571]  [<ffffffff8102bb5c>] ? pvclock_clocksource_read+0x42/0xb2
Dec  9 12:34:32 server1 kernel: [726212.900571]  [<ffffffff81165799>] ? security_file_permission+0x16/0x2d
Dec  9 12:34:32 server1 kernel: [726212.900571]  [<ffffffff810fb584>] ? vfs_write+0xa9/0xe9
Dec  9 12:34:32 server1 kernel: [726212.900571]  [<ffffffff810fb75a>] ? sys_write+0x45/0x6b
Dec  9 12:34:32 server1 kernel: [726212.900571]  [<ffffffff813562b2>] ? system_call_fastpath+0x16/0x1b
Dec  9 12:34:32 server1 kernel: [726212.900571] Code: 41 0f 44 d0 41 ff ca 45 85 d2 75 ca 41 8b 84 24 8c 09 00 00 09 41 08 0f ae f8 0f b7 45 2a 89 55 18 49 03 84 24 98 0a 00 00 89 10 <8b> 45 1c 8b 4d 18 31 d2 39 c8 77 03 8b 55 14 ff c8 45 31 ed 29 
Dec  9 12:34:32 server1 kernel: [726212.900571] Call Trace:
Dec  9 12:34:32 server1 kernel: [726212.900571]  [<ffffffff81292b85>] ? dev_hard_start_xmit+0x3d2/0x518
Dec  9 12:34:32 server1 kernel: [726212.900571]  [<ffffffff812a920f>] ? sch_direct_xmit+0x61/0x135
Dec  9 12:34:32 server1 kernel: [726212.900571]  [<ffffffff81292f8d>] ? dev_queue_xmit+0x2c2/0x46b
Dec  9 12:34:32 server1 kernel: [726212.900571]  [<ffffffff8102bb5c>] ? pvclock_clocksource_read+0x42/0xb2
Dec  9 12:34:32 server1 kernel: [726212.900571]  [<ffffffff812bbca6>] ? ip_finish_output2+0x1bc/0x1f9
Dec  9 12:34:32 server1 kernel: [726212.900571]  [<ffffffff812bd06d>] ? ip_queue_xmit+0x28e/0x2b9
Dec  9 12:34:32 server1 kernel: [726212.900571]  [<ffffffff812ce3a1>] ? tcp_transmit_skb+0x6e6/0x719
Dec  9 12:34:32 server1 kernel: [726212.900571]  [<ffffffff812cee36>] ? tcp_write_xmit+0x6a0/0x7a9
Dec  9 12:34:32 server1 kernel: [726212.900571]  [<ffffffff810eba4e>] ? virt_to_head_page+0x6/0x29
Dec  9 12:34:32 server1 kernel: [726212.900571]  [<ffffffff810ebcd7>] ? virt_to_cache+0x7/0x23
Dec  9 12:34:32 server1 kernel: [726212.900571]  [<ffffffff810ebd09>] ? ksize+0x16/0x1c
Dec  9 12:34:32 server1 kernel: [726212.900571]  [<ffffffff8134fe9c>] ? _cond_resched+0x7/0x1c
Dec  9 12:34:32 server1 kernel: [726212.900571]  [<ffffffff812cef80>] ? __tcp_push_pending_frames+0x18/0x44
Dec  9 12:34:32 server1 kernel: [726212.900571]  [<ffffffff812c476b>] ? tcp_sendmsg+0x64d/0x72d
Dec  9 12:34:32 server1 kernel: [726212.900571]  [<ffffffff81280812>] ? sock_aio_write+0x121/0x135
Dec  9 12:34:32 server1 kernel: [726212.900571]  [<ffffffff810fae8c>] ? do_sync_write+0xb4/0xec
Dec  9 12:34:32 server1 kernel: [726212.900571]  [<ffffffff8102bb5c>] ? pvclock_clocksource_read+0x42/0xb2
Dec  9 12:34:32 server1 kernel: [726212.900571]  [<ffffffff81165799>] ? security_file_permission+0x16/0x2d
Dec  9 12:34:32 server1 kernel: [726212.900571]  [<ffffffff810fb584>] ? vfs_write+0xa9/0xe9
Dec  9 12:34:32 server1 kernel: [726212.900571]  [<ffffffff810fb75a>] ? sys_write+0x45/0x6b
Dec  9 12:34:32 server1 kernel: [726212.900571]  [<ffffffff813562b2>] ? system_call_fastpath+0x16/0x1b
Dec  9 12:34:32 server1 kernel: [726212.912006] NMI backtrace for cpu 3
Dec  9 12:34:32 server1 kernel: [726212.912006] CPU 3 
Dec  9 12:34:32 server1 kernel: [726212.912006] Modules linked in: nfnetlink_log nfnetlink nfsd nfs nfs_acl auth_rpcgss fscache lockd sunrpc loop snd_pcm snd_page_alloc snd_timer snd soundcore i2c_piix4 i2c_core psmouse joydev serio_raw pcspkr evdev crc32c_intel ghash_clmulni_intel processor virtio_balloon thermal_sys button container aesni_intel aes_x86_64 aes_generic cryptd ext4 crc16 jbd2 mbcache usbhid hid sg sr_mod cdrom ata_generic virtio_blk floppy e1000 uhci_hcd ehci_hcd ata_piix libata scsi_mod virtio_pci virtio_ring virtio usbcore usb_common [last unloaded: scsi_wait_scan]
Dec  9 12:34:32 server1 kernel: [726212.912006] 
Dec  9 12:34:32 server1 kernel: [726212.912006] Pid: 0, comm: swapper/3 Not tainted 3.2.0-4-amd64 #1 Debian 3.2.68-1+deb7u6 Red Hat KVM
Dec  9 12:34:32 server1 kernel: [726212.912006] RIP: 0010:[<ffffffff8102b36c>]  [<ffffffff8102b36c>] native_safe_halt+0x2/0x3
Dec  9 12:34:32 server1 kernel: [726212.912006] RSP: 0018:ffff88033291fef0  EFLAGS: 00000246
Dec  9 12:34:32 server1 kernel: [726212.912006] RAX: 0000000000000000 RBX: ffff88033291e000 RCX: 0000000000000000
Dec  9 12:34:32 server1 kernel: [726212.912006] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff88033291e000
Dec  9 12:34:32 server1 kernel: [726212.912006] RBP: ffffffff8168f1e0 R08: 0000000000000000 R09: ffffffff8168f1e0
Dec  9 12:34:32 server1 kernel: [726212.912006] R10: 0000000000000601 R11: ffff88033291e000 R12: 0000000000000000
Dec  9 12:34:32 server1 kernel: [726212.912006] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
Dec  9 12:34:32 server1 kernel: [726212.912006] FS:  0000000000000000(0000) GS:ffff88033fd80000(0000) knlGS:0000000000000000
Dec  9 12:34:32 server1 kernel: [726212.912006] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Dec  9 12:34:32 server1 kernel: [726212.912006] CR2: 00007f2145e73008 CR3: 00000002dd877000 CR4: 00000000001406e0
Dec  9 12:34:32 server1 kernel: [726212.912006] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Dec  9 12:34:32 server1 kernel: [726212.912006] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Dec  9 12:34:32 server1 kernel: [726212.912006] Process swapper/3 (pid: 0, threadinfo ffff88033291e000, task ffff8803328fe100)
Dec  9 12:34:32 server1 kernel: [726212.912006] Stack:
Dec  9 12:34:32 server1 kernel: [726212.912006]  ffffffff81014938 ffffffff8168f1e0 ffffffff8100d24c 0000000000000000
Dec  9 12:34:32 server1 kernel: [726212.912006]  bd9e6a7c2cb84c47 00000000b1c7b947 0000000000000003 0000000000000000
Dec  9 12:34:32 server1 kernel: [726212.912006]  ffffffff8134328a 0000000000000000 b93446556384bb8b 0000000000000000
Dec  9 12:34:32 server1 kernel: [726212.912006] Call Trace:
Dec  9 12:34:32 server1 kernel: [726212.912006]  [<ffffffff81014938>] ? default_idle+0x47/0x7f
Dec  9 12:34:32 server1 kernel: [726212.912006]  [<ffffffff8100d24c>] ? cpu_idle+0xaf/0xf2
Dec  9 12:34:32 server1 kernel: [726212.912006]  [<ffffffff8134328a>] ? start_secondary+0x1da/0x1e0
Dec  9 12:34:32 server1 kernel: [726212.912006] Code: ee 48 c7 c7 1f 89 4c 81 31 c0 e8 7e f5 31 00 48 83 c4 18 89 d8 5b 5d 41 5c 41 5d c3 90 90 90 9c 58 c3 57 9d c3 fa c3 fb c3 fb f4 <c3> f4 c3 0f 1f 80 00 00 00 00 c3 0f 1f 80 00 00 00 00 c3 0f 06 
Dec  9 12:34:32 server1 kernel: [726212.912006] Call Trace:
Dec  9 12:34:32 server1 kernel: [726212.912006]  [<ffffffff81014938>] ? default_idle+0x47/0x7f
Dec  9 12:34:32 server1 kernel: [726212.912006]  [<ffffffff8100d24c>] ? cpu_idle+0xaf/0xf2
Dec  9 12:34:32 server1 kernel: [726212.912006]  [<ffffffff8134328a>] ? start_secondary+0x1da/0x1e0
Dec  9 12:35:12 server1 kernel: [726222.800034] ------------[ cut here ]------------
Dec  9 12:35:12 server1 kernel: [726222.800049] WARNING: at /build/linux-Tvajqd/linux-3.2.68/net/sched/sch_generic.c:256 dev_watchdog+0xf2/0x151()
Dec  9 12:35:12 server1 kernel: [726222.800051] Hardware name: KVM
Dec  9 12:35:12 server1 kernel: [726222.800053] NETDEV WATCHDOG: eth0 (e1000): transmit queue 0 timed out
Dec  9 12:35:12 server1 kernel: [726222.800054] Modules linked in: nfnetlink_log nfnetlink nfsd nfs nfs_acl auth_rpcgss fscache lockd sunrpc loop snd_pcm snd_page_alloc snd_timer snd soundcore i2c_piix4 i2c_core psmouse joydev serio_raw pcspkr evdev crc32c_intel ghash_clmulni_intel processor virtio_balloon thermal_sys button container aesni_intel aes_x86_64 aes_generic cryptd ext4 crc16 jbd2 mbcache usbhid hid sg sr_mod cdrom ata_generic virtio_blk floppy e1000 uhci_hcd ehci_hcd ata_piix libata scsi_mod virtio_pci virtio_ring virtio usbcore usb_common [last unloaded: scsi_wait_scan]
Dec  9 12:35:12 server1 kernel: [726222.800101] Pid: 0, comm: swapper/2 Not tainted 3.2.0-4-amd64 #1 Debian 3.2.68-1+deb7u6
Dec  9 12:35:12 server1 kernel: [726222.800102] Call Trace:
Dec  9 12:35:12 server1 kernel: [726222.800104]  <IRQ>  [<ffffffff81046de1>] ? warn_slowpath_common+0x78/0x8c
Dec  9 12:35:12 server1 kernel: [726222.800113]  [<ffffffff81046e8d>] ? warn_slowpath_fmt+0x45/0x4a
Dec  9 12:35:12 server1 kernel: [726222.800121]  [<ffffffff812a8d51>] ? netif_tx_lock+0x40/0x75
Dec  9 12:35:12 server1 kernel: [726222.800124]  [<ffffffff812a8ec1>] ? dev_watchdog+0xf2/0x151
Dec  9 12:35:12 server1 kernel: [726222.800127]  [<ffffffff81052618>] ? run_timer_softirq+0x19a/0x261
Dec  9 12:35:12 server1 kernel: [726222.800129]  [<ffffffff812a8dcf>] ? netif_tx_unlock+0x49/0x49
Dec  9 12:35:12 server1 kernel: [726222.800133]  [<ffffffff81066439>] ? timekeeping_get_ns+0xd/0x2a
Dec  9 12:35:12 server1 kernel: [726222.800136]  [<ffffffff8104c48e>] ? __do_softirq+0xb9/0x177
Dec  9 12:35:12 server1 kernel: [726222.800141]  [<ffffffff813584ec>] ? call_softirq+0x1c/0x30
Dec  9 12:35:12 server1 kernel: [726222.800147]  [<ffffffff8100fa91>] ? do_softirq+0x3c/0x7b
Dec  9 12:35:12 server1 kernel: [726222.800149]  [<ffffffff8104c6f6>] ? irq_exit+0x3c/0x99
Dec  9 12:35:12 server1 kernel: [726222.800154]  [<ffffffff8102475c>] ? smp_apic_timer_interrupt+0x74/0x82
Dec  9 12:35:12 server1 kernel: [726222.800169]  [<ffffffff81356d5e>] ? apic_timer_interrupt+0x6e/0x80
Dec  9 12:35:12 server1 kernel: [726222.800171]  <EOI>  [<ffffffff81096b8d>] ? rcu_needs_cpu+0x50/0x1bb
Dec  9 12:35:12 server1 kernel: [726222.800178]  [<ffffffff8102b36c>] ? native_safe_halt+0x2/0x3
Dec  9 12:35:12 server1 kernel: [726222.800182]  [<ffffffff81014938>] ? default_idle+0x47/0x7f
Dec  9 12:35:12 server1 kernel: [726222.800185]  [<ffffffff8100d24c>] ? cpu_idle+0xaf/0xf2
Dec  9 12:35:12 server1 kernel: [726222.800190]  [<ffffffff8134328a>] ? start_secondary+0x1da/0x1e0
Dec  9 12:35:12 server1 kernel: [726222.800192] ---[ end trace 877820c1f31d5380 ]---
Dec  9 12:35:40 server1 kernel: [726280.565671] BUG: soft lockup - CPU#1 stuck for 26s! [mysqld:4141]
Dec  9 12:35:40 server1 kernel: [726280.566088] Modules linked in: nfnetlink_log nfnetlink nfsd nfs nfs_acl auth_rpcgss fscache lockd sunrpc loop snd_pcm snd_page_alloc snd_timer snd soundcore i2c_piix4 i2c_core psmouse joydev serio_raw pcspkr evdev crc32c_intel ghash_clmulni_intel processor virtio_balloon thermal_sys button container aesni_intel aes_x86_64 aes_generic cryptd ext4 crc16 jbd2 mbcache usbhid hid sg sr_mod cdrom ata_generic virtio_blk floppy e1000 uhci_hcd ehci_hcd ata_piix libata scsi_mod virtio_pci virtio_ring virtio usbcore usb_common [last unloaded: scsi_wait_scan]
Dec  9 12:35:40 server1 kernel: [726280.566142] CPU 1 
Dec  9 12:35:40 server1 kernel: [726280.566143] Modules linked in: nfnetlink_log nfnetlink nfsd nfs nfs_acl auth_rpcgss fscache lockd sunrpc loop snd_pcm snd_page_alloc snd_timer snd soundcore i2c_piix4 i2c_core psmouse joydev serio_raw pcspkr evdev crc32c_intel ghash_clmulni_intel processor virtio_balloon thermal_sys button container aesni_intel aes_x86_64 aes_generic cryptd ext4 crc16 jbd2 mbcache usbhid hid sg sr_mod cdrom ata_generic virtio_blk floppy e1000 uhci_hcd ehci_hcd ata_piix libata scsi_mod virtio_pci virtio_ring virtio usbcore usb_common [last unloaded: scsi_wait_scan]
Dec  9 12:35:40 server1 kernel: [726280.566173] 
Dec  9 12:35:40 server1 kernel: [726280.566177] Pid: 4141, comm: mysqld Tainted: G        W    3.2.0-4-amd64 #1 Debian 3.2.68-1+deb7u6 Red Hat KVM
Dec  9 12:35:40 server1 kernel: [726280.566181] RIP: 0010:[<ffffffff813515f6>]  [<ffffffff813515f6>] retint_careful+0xe/0x32
Dec  9 12:35:40 server1 kernel: [726280.566193] RSP: 0000:ffff880329751f88  EFLAGS: 00000203
Dec  9 12:35:40 server1 kernel: [726280.566195] RAX: 00007fc0c5e74000 RBX: ffffffff81066439 RCX: ffff880329750000
Dec  9 12:35:40 server1 kernel: [726280.566197] RDX: 0000000000000008 RSI: ffff880329751f78 RDI: 000000000000fe2e
Dec  9 12:35:40 server1 kernel: [726280.566199] RBP: 00007fc0c5e74000 R08: ffffffff814052a0 R09: 0000000000000000
Dec  9 12:35:40 server1 kernel: [726280.566201] R10: 0000000000000000 R11: 00007fc0d2205e38 R12: ffff88032f1a4080
Dec  9 12:35:40 server1 kernel: [726280.566203] R13: ffff88015fbeb400 R14: ffff88032f6cb1c0 R15: 00000000000127c0
Dec  9 12:35:40 server1 kernel: [726280.566206] FS:  00007fc0bea41700(0000) GS:ffff88033fc80000(0000) knlGS:0000000000000000
Dec  9 12:35:40 server1 kernel: [726280.566208] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Dec  9 12:35:40 server1 kernel: [726280.566210] CR2: 00007f2153410000 CR3: 000000031c0f8000 CR4: 00000000001406e0
Dec  9 12:35:40 server1 kernel: [726280.566215] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Dec  9 12:35:40 server1 kernel: [726280.566217] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Dec  9 12:35:40 server1 kernel: [726280.566220] Process mysqld (pid: 4141, threadinfo ffff880329750000, task ffff88032f6cb1c0)
Dec  9 12:35:40 server1 kernel: [726280.566222] Stack:
Dec  9 12:35:40 server1 kernel: [726280.566503]  00007fc0c5e74000 0000000000000000 0000000000000000 0000000000000000
Dec  9 12:35:40 server1 kernel: [726280.566508]  a4a46aee384fff9f 00007fc0c5e75000 ddc05329c75d11a3 ddc05329c75d11a3
Dec  9 12:35:40 server1 kernel: [726280.566512]  00007fc0c5e77ff8 ffffffffffffff02 00007fc0d1aae9a0 0000000000000033
Dec  9 12:35:40 server1 kernel: [726280.566516] Call Trace:
Dec  9 12:35:40 server1 kernel: [726280.569621] Code: 00 00 ff ff 5f 65 48 0b 04 25 08 a0 00 00 0f 01 f8 0f 1f 00 48 89 c4 58 e9 9f ff ff ff 0f ba e2 03 73 2c fb 0f 1f 80 00 00 00 00 <57> e8 48 ea ff ff 5f 65 48 8b 0c 25 c8 b6 00 00 48 81 e9 d8 1f 
Dec  9 12:35:40 server1 kernel: [726280.569621] Call Trace:
Dec  9 12:37:56 server1 kernel: [726416.567402] BUG: soft lockup - CPU#2 stuck for 33s! [cron:2299]
Dec  9 12:37:56 server1 kernel: [726381.850950] ------------[ cut here ]------------
Dec  9 12:37:56 server1 kernel: [726381.850950] WARNING: at /build/linux-Tvajqd/linux-3.2.68/kernel/watchdog.c:241 watchdog_overflow_callback+0x93/0x9e()
Dec  9 12:37:56 server1 kernel: [726381.850950] Hardware name: KVM
Dec  9 12:37:56 server1 kernel: [726381.850950] Watchdog detected hard LOCKUP on cpu 1
Dec  9 12:37:56 server1 kernel: [726381.850950] Modules linked in: nfnetlink_log nfnetlink nfsd nfs nfs_acl auth_rpcgss fscache lockd sunrpc loop snd_pcm snd_page_alloc snd_timer snd soundcore i2c_piix4 i2c_core psmouse joydev serio_raw pcspkr evdev crc32c_intel ghash_clmulni_intel processor virtio_balloon thermal_sys button container aesni_intel aes_x86_64 aes_generic cryptd ext4 crc16 jbd2 mbcache usbhid hid sg sr_mod cdrom ata_generic virtio_blk floppy e1000 uhci_hcd ehci_hcd ata_piix libata scsi_mod virtio_pci virtio_ring virtio usbcore usb_common [last unloaded: scsi_wait_scan]
Dec  9 12:37:56 server1 kernel: [726381.850950] Pid: 160, comm: kworker/1:1 Tainted: G        W    3.2.0-4-amd64 #1 Debian 3.2.68-1+deb7u6
Dec  9 12:37:56 server1 kernel: [726381.850950] Call Trace:
Dec  9 12:37:56 server1 kernel: [726381.850950]  <NMI>  [<ffffffff81046de1>] ? warn_slowpath_common+0x78/0x8c
Dec  9 12:37:56 server1 kernel: [726381.850950]  [<ffffffff81046e8d>] ? warn_slowpath_fmt+0x45/0x4a
Dec  9 12:37:56 server1 kernel: [726381.850950]  [<ffffffff81013e64>] ? sched_clock+0x5/0x8
Dec  9 12:37:56 server1 kernel: [726381.850950]  [<ffffffff81090c78>] ? watchdog_overflow_callback+0x93/0x9e
Dec  9 12:37:56 server1 kernel: [726381.850950]  [<ffffffff81090be5>] ? touch_nmi_watchdog+0x60/0x60
Dec  9 12:37:56 server1 kernel: [726381.850950]  [<ffffffff810b0dd1>] ? __perf_event_overflow+0xf9/0x17b
Dec  9 12:37:56 server1 kernel: [726381.850950]  [<ffffffff810641b2>] ? local_clock+0x24/0x2b
Dec  9 12:37:56 server1 kernel: [726381.850950]  [<ffffffff810af21f>] ? perf_event_update_userpage+0x9/0x9d
Dec  9 12:37:56 server1 kernel: [726381.850950]  [<ffffffff8101a3e3>] ? x86_perf_event_set_period+0x101/0x10b
Dec  9 12:37:56 server1 kernel: [726381.850950]  [<ffffffff8101d7c6>] ? intel_pmu_handle_irq+0x222/0x283
Dec  9 12:37:56 server1 kernel: [726381.850950]  [<ffffffff8100fd13>] ? dump_trace+0x243/0x252
Dec  9 12:37:56 server1 kernel: [726381.850950]  [<ffffffff81352222>] ? nmi_handle.isra.0+0x41/0x64
Dec  9 12:37:56 server1 kernel: [726381.850950]  [<ffffffff813522c2>] ? do_nmi+0x7d/0x258
Dec  9 12:37:56 server1 kernel: [726381.850950]  [<ffffffff81351ae0>] ? nmi+0x20/0x30
Dec  9 12:37:56 server1 kernel: [726381.850950]  [<ffffffff81071422>] ? do_raw_spin_lock+0x15/0x1b
Dec  9 12:37:56 server1 kernel: [726381.850950]  <<EOE>>  [<ffffffff810ec17a>] ? drain_array+0x4b/0xc9
Dec  9 12:37:56 server1 kernel: [726381.850950]  [<ffffffff810ec3b1>] ? cache_reap+0xc8/0x1a2
Dec  9 12:37:56 server1 kernel: [726381.850950]  [<ffffffff8105b717>] ? process_one_work+0x161/0x269
Dec  9 12:37:56 server1 kernel: [726381.850950]  [<ffffffff8105c6e0>] ? worker_thread+0xc2/0x145
Dec  9 12:37:56 server1 kernel: [726381.850950]  [<ffffffff8105c61e>] ? manage_workers.isra.25+0x15b/0x15b
Dec  9 12:37:56 server1 kernel: [726381.850950]  [<ffffffff8105f829>] ? kthread+0x76/0x7e
Dec  9 12:37:56 server1 kernel: [726381.850950]  [<ffffffff813583f4>] ? kernel_thread_helper+0x4/0x10
Dec  9 12:37:56 server1 kernel: [726381.850950]  [<ffffffff8105f7b3>] ? kthread_worker_fn+0x139/0x139
Dec  9 12:37:56 server1 kernel: [726381.850950]  [<ffffffff813583f0>] ? gs_change+0x13/0x13
Dec  9 12:37:56 server1 kernel: [726381.850950] ---[ end trace 877820c1f31d5381 ]---
Dec  9 12:37:56 server1 kernel: [726416.571367] Modules linked in: nfnetlink_log nfnetlink nfsd nfs nfs_acl auth_rpcgss fscache lockd sunrpc loop snd_pcm snd_page_alloc snd_timer snd soundcore i2c_piix4 i2c_core psmouse joydev serio_raw pcspkr evdev crc32c_intel ghash_clmulni_intel processor virtio_balloon thermal_sys button container aesni_intel aes_x86_64 aes_generic cryptd ext4 crc16 jbd2 mbcache usbhid hid sg sr_mod cdrom ata_generic virtio_blk floppy e1000 uhci_hcd ehci_hcd ata_piix libata scsi_mod virtio_pci virtio_ring virtio usbcore usb_common [last unloaded: scsi_wait_scan]
Dec  9 12:37:56 server1 kernel: [726416.571367] CPU 2 
Dec  9 12:37:56 server1 kernel: [726416.571367] Modules linked in: nfnetlink_log nfnetlink nfsd nfs nfs_acl auth_rpcgss fscache lockd sunrpc loop snd_pcm snd_page_alloc snd_timer snd soundcore i2c_piix4 i2c_core psmouse joydev serio_raw pcspkr evdev crc32c_intel ghash_clmulni_intel processor virtio_balloon thermal_sys button container aesni_intel aes_x86_64 aes_generic cryptd ext4 crc16 jbd2 mbcache usbhid hid sg sr_mod cdrom ata_generic virtio_blk floppy e1000 uhci_hcd ehci_hcd ata_piix libata scsi_mod virtio_pci virtio_ring virtio usbcore usb_common [last unloaded: scsi_wait_scan]
Dec  9 12:37:56 server1 kernel: [726416.571367] 
Dec  9 12:37:56 server1 kernel: [726416.571367] Pid: 2299, comm: cron Tainted: G        W    3.2.0-4-amd64 #1 Debian 3.2.68-1+deb7u6 Red Hat KVM
Dec  9 12:37:56 server1 kernel: [726416.571367] RIP: 0033:[<00007fa6fc429478>]  [<00007fa6fc429478>] 0x7fa6fc429477
Dec  9 12:37:56 server1 kernel: [726416.571367] RSP: 002b:00007fff4e3a6868  EFLAGS: 00010217
Dec  9 12:37:56 server1 kernel: [726416.571367] RAX: 0000000000000000 RBX: 000000000000fe2e RCX: 00007fa6fc429470
Dec  9 12:37:56 server1 kernel: [726416.571367] RDX: 0000000000000000 RSI: 00007fff4e3a6a10 RDI: 00007fff4e3a6a10
Dec  9 12:37:56 server1 kernel: [726416.571367] RBP: 00007fff4e3a6910 R08: 00007fff4e3a6870 R09: 0000000000000000
Dec  9 12:37:56 server1 kernel: [726416.571367] R10: 0000000000000008 R11: 0000000000000246 R12: 0000000000000000
Dec  9 12:37:56 server1 kernel: [726416.571367] R13: 0000000000000000 R14: 00007fff4e3a6b70 R15: 00000000ffffffff
Dec  9 12:37:56 server1 kernel: [726416.571367] FS:  00007fa6fcd4b7a0(0000) GS:ffff88033fd00000(0000) knlGS:0000000000000000
Dec  9 12:37:56 server1 kernel: [726416.571367] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Dec  9 12:37:56 server1 kernel: [726416.571367] CR2: 00007f5b7bb88000 CR3: 0000000330cb5000 CR4: 00000000001406e0
Dec  9 12:37:56 server1 kernel: [726416.571367] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Dec  9 12:37:56 server1 kernel: [726416.571367] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Dec  9 12:37:56 server1 kernel: [726416.571367] Process cron (pid: 2299, threadinfo ffff880330ccc000, task ffff88032fa410c0)
Dec  9 12:37:56 server1 kernel: [726416.571367] 
Dec  9 12:37:56 server1 kernel: [726416.571367] Call Trace:
Dec  9 12:38:04 server1 kernel: [726424.620114] INFO: task kworker/2:0:14 blocked for more than 120 seconds.
Dec  9 12:38:04 server1 kernel: [726424.625127] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Dec  9 12:38:04 server1 kernel: [726424.625864] kworker/2:0     D ffff88033fd127c0     0    14      2 0x00000000
Dec  9 12:38:04 server1 kernel: [726424.625874]  ffff8803328c4740 0000000000000046 0000000000000000 ffff8803328f0780
Dec  9 12:38:04 server1 kernel: [726424.625878]  00000000000127c0 ffff8803328e5fd8 ffff8803328e5fd8 ffff8803328c4740
Dec  9 12:38:04 server1 kernel: [726424.625880]  0000000000000002 0000000181041d01 0000000000000000 7fffffffffffffff
Dec  9 12:38:04 server1 kernel: [726424.625883] Call Trace:
Dec  9 12:38:04 server1 kernel: [726424.625893]  [<ffffffff8135030b>] ? schedule_timeout+0x2c/0xdb
Dec  9 12:38:04 server1 kernel: [726424.625899]  [<ffffffff81038139>] ? set_next_entity+0x32/0x55
Dec  9 12:38:04 server1 kernel: [726424.625905]  [<ffffffff8100d02f>] ? load_TLS+0x7/0xa
Dec  9 12:38:04 server1 kernel: [726424.625908]  [<ffffffff8100d66c>] ? __switch_to+0x101/0x265
Dec  9 12:38:04 server1 kernel: [726424.625910]  [<ffffffff8134ff51>] ? wait_for_common+0xa0/0x119
Dec  9 12:38:04 server1 kernel: [726424.625913]  [<ffffffff8103f751>] ? try_to_wake_up+0x197/0x197
Dec  9 12:38:04 server1 kernel: [726424.625917]  [<ffffffff8105ae16>] ? wait_on_work+0xe6/0x11c
Dec  9 12:38:04 server1 kernel: [726424.625919]  [<ffffffff8105a450>] ? worker_set_flags+0x8f/0x8f
Dec  9 12:38:04 server1 kernel: [726424.625922]  [<ffffffff8105ba76>] ? __cancel_work_timer+0xb2/0xf4
Dec  9 12:38:04 server1 kernel: [726424.626089]  [<ffffffffa00bc889>] ? e1000_down_and_stop+0x2f/0x48 [e1000]
Dec  9 12:38:04 server1 kernel: [726424.626094]  [<ffffffffa00bf1da>] ? e1000_down+0x108/0x15e [e1000]
Dec  9 12:38:04 server1 kernel: [726424.626098]  [<ffffffffa00c1373>] ? e1000_reset_task+0x53/0x6c [e1000]
Dec  9 12:38:04 server1 kernel: [726424.626100]  [<ffffffff8105b717>] ? process_one_work+0x161/0x269
Dec  9 12:38:04 server1 kernel: [726424.626103]  [<ffffffff8105c6e0>] ? worker_thread+0xc2/0x145
Dec  9 12:38:04 server1 kernel: [726424.626105]  [<ffffffff8105c61e>] ? manage_workers.isra.25+0x15b/0x15b
Dec  9 12:38:04 server1 kernel: [726424.626108]  [<ffffffff8105f829>] ? kthread+0x76/0x7e
Dec  9 12:38:04 server1 kernel: [726424.626112]  [<ffffffff813583f4>] ? kernel_thread_helper+0x4/0x10
Dec  9 12:38:04 server1 kernel: [726424.626115]  [<ffffffff8105f7b3>] ? kthread_worker_fn+0x139/0x139
Dec  9 12:38:04 server1 kernel: [726424.626117]  [<ffffffff813583f0>] ? gs_change+0x13/0x13
Dec  9 12:38:04 server1 kernel: [726424.626158] INFO: task kworker/2:2:171 blocked for more than 120 seconds.
Dec  9 12:38:04 server1 kernel: [726424.626416] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Dec  9 12:38:04 server1 kernel: [726424.627023] kworker/2:2     D ffff88033fd127c0     0   171      2 0x00000000
Dec  9 12:38:04 server1 kernel: [726424.627031]  ffff88032f6cb8c0 0000000000000046 00000000000127c0 ffff880330e13100
Dec  9 12:38:04 server1 kernel: [726424.627034]  00000000000127c0 ffff88032fa53fd8 ffff88032fa53fd8 ffff88032f6cb8c0
Dec  9 12:38:04 server1 kernel: [726424.627037]  ffffffff81351267 ffffffff81042c7b ffff88032f6cb8c0 ffff88032fa51000
Dec  9 12:38:04 server1 kernel: [726424.627039] Call Trace:
Dec  9 12:38:04 server1 kernel: [726424.627044]  [<ffffffff81351267>] ? _raw_spin_unlock_irqrestore+0xe/0xf
Dec  9 12:38:04 server1 kernel: [726424.627048]  [<ffffffff81042c7b>] ? set_cpus_allowed_ptr+0xdd/0xec
Dec  9 12:38:04 server1 kernel: [726424.627050]  [<ffffffff8135079c>] ? __mutex_lock_common.isra.5+0xff/0x164
Dec  9 12:38:04 server1 kernel: [726424.627052]  [<ffffffff8135068a>] ? mutex_lock+0x1a/0x2d
Dec  9 12:38:04 server1 kernel: [726424.627055]  [<ffffffff81037240>] ? test_tsk_need_resched+0xa/0x13
Dec  9 12:38:04 server1 kernel: [726424.627058]  [<ffffffff8103b0e3>] ? check_preempt_curr+0x52/0x5f
Dec  9 12:38:04 server1 kernel: [726424.627066]  [<ffffffffa00c01e5>] ? e1000_watchdog+0x48/0x4ac [e1000]
Dec  9 12:38:04 server1 kernel: [726424.627069]  [<ffffffff8105b717>] ? process_one_work+0x161/0x269
Dec  9 12:38:04 server1 kernel: [726424.627073]  [<ffffffff81059ff6>] ? need_to_create_worker+0x9/0x1c
Dec  9 12:38:04 server1 kernel: [726424.627075]  [<ffffffff8105c6e0>] ? worker_thread+0xc2/0x145
Dec  9 12:38:04 server1 kernel: [726424.627077]  [<ffffffff8105c61e>] ? manage_workers.isra.25+0x15b/0x15b
Dec  9 12:38:04 server1 kernel: [726424.627080]  [<ffffffff8105f829>] ? kthread+0x76/0x7e
Dec  9 12:38:04 server1 kernel: [726424.627082]  [<ffffffff813583f4>] ? kernel_thread_helper+0x4/0x10
Dec  9 12:38:04 server1 kernel: [726424.627085]  [<ffffffff8105f7b3>] ? kthread_worker_fn+0x139/0x139
Dec  9 12:38:04 server1 kernel: [726424.627089]  [<ffffffff813583f0>] ? gs_change+0x13/0x13
Dec  9 12:39:01 server1 /USR/SBIN/CRON[23643]: (root) CMD (  [ -x /usr/lib/php5/sessionclean ] && /usr/lib/php5/sessionclean)
Dec  9 12:40:01 server1 /USR/SBIN/CRON[23666]: (root) CMD (/usr/local/maldetect/maldet --mkpubpaths >> /dev/null 2>&1)
Dec  9 12:40:04 server1 kernel: [726544.624119] INFO: task kworker/2:0:14 blocked for more than 120 seconds.
Dec  9 12:40:04 server1 kernel: [726544.627939] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Dec  9 12:40:04 server1 kernel: [726544.628526] kworker/2:0     D ffff88033fd127c0     0    14      2 0x00000000
Dec  9 12:40:04 server1 kernel: [726544.628538]  ffff8803328c4740 0000000000000046 0000000000000000 ffff8803328f0780
Dec  9 12:40:04 server1 kernel: [726544.628542]  00000000000127c0 ffff8803328e5fd8 ffff8803328e5fd8 ffff8803328c4740
Dec  9 12:40:04 server1 kernel: [726544.628545]  0000000000000002 0000000181041d01 0000000000000000 7fffffffffffffff
Dec  9 12:40:04 server1 kernel: [726544.628548] Call Trace:
Dec  9 12:40:04 server1 kernel: [726544.628578]  [<ffffffff8135030b>] ? schedule_timeout+0x2c/0xdb
Dec  9 12:40:04 server1 kernel: [726544.628591]  [<ffffffff81038139>] ? set_next_entity+0x32/0x55
Dec  9 12:40:04 server1 kernel: [726544.628598]  [<ffffffff8100d02f>] ? load_TLS+0x7/0xa
Dec  9 12:40:04 server1 kernel: [726544.628601]  [<ffffffff8100d66c>] ? __switch_to+0x101/0x265
Dec  9 12:40:04 server1 kernel: [726544.628603]  [<ffffffff8134ff51>] ? wait_for_common+0xa0/0x119
Dec  9 12:40:04 server1 kernel: [726544.628609]  [<ffffffff8103f751>] ? try_to_wake_up+0x197/0x197
Dec  9 12:40:04 server1 kernel: [726544.628618]  [<ffffffff8105ae16>] ? wait_on_work+0xe6/0x11c
Dec  9 12:40:04 server1 kernel: [726544.628620]  [<ffffffff8105a450>] ? worker_set_flags+0x8f/0x8f
Dec  9 12:40:04 server1 kernel: [726544.628623]  [<ffffffff8105ba76>] ? __cancel_work_timer+0xb2/0xf4
Dec  9 12:40:04 server1 kernel: [726544.628645]  [<ffffffffa00bc889>] ? e1000_down_and_stop+0x2f/0x48 [e1000]
Dec  9 12:40:04 server1 kernel: [726544.628649]  [<ffffffffa00bf1da>] ? e1000_down+0x108/0x15e [e1000]
Dec  9 12:40:04 server1 kernel: [726544.628653]  [<ffffffffa00c1373>] ? e1000_reset_task+0x53/0x6c [e1000]
Dec  9 12:40:04 server1 kernel: [726544.628655]  [<ffffffff8105b717>] ? process_one_work+0x161/0x269
Dec  9 12:40:04 server1 kernel: [726544.628658]  [<ffffffff8105c6e0>] ? worker_thread+0xc2/0x145
Dec  9 12:40:04 server1 kernel: [726544.628660]  [<ffffffff8105c61e>] ? manage_workers.isra.25+0x15b/0x15b
Dec  9 12:40:04 server1 kernel: [726544.628664]  [<ffffffff8105f829>] ? kthread+0x76/0x7e
Dec  9 12:40:04 server1 kernel: [726544.628671]  [<ffffffff813583f4>] ? kernel_thread_helper+0x4/0x10
Dec  9 12:40:04 server1 kernel: [726544.628674]  [<ffffffff8105f7b3>] ? kthread_worker_fn+0x139/0x139
Dec  9 12:40:04 server1 kernel: [726544.628676]  [<ffffffff813583f0>] ? gs_change+0x13/0x13
Dec  9 12:40:04 server1 kernel: [726544.628687] INFO: task kworker/2:2:171 blocked for more than 120 seconds.
Dec  9 12:40:04 server1 kernel: [726544.628971] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Dec  9 12:40:04 server1 kernel: [726544.629430] kworker/2:2     D ffff88033fd127c0     0   171      2 0x00000000
Dec  9 12:40:04 server1 kernel: [726544.629434]  ffff88032f6cb8c0 0000000000000046 00000000000127c0 ffff880330e13100
Dec  9 12:40:04 server1 kernel: [726544.629436]  00000000000127c0 ffff88032fa53fd8 ffff88032fa53fd8 ffff88032f6cb8c0
Dec  9 12:40:04 server1 kernel: [726544.629439]  ffffffff81351267 ffffffff81042c7b ffff88032f6cb8c0 ffff88032fa51000
Dec  9 12:40:04 server1 kernel: [726544.629441] Call Trace:
Dec  9 12:40:04 server1 kernel: [726544.629445]  [<ffffffff81351267>] ? _raw_spin_unlock_irqrestore+0xe/0xf
Dec  9 12:40:04 server1 kernel: [726544.629449]  [<ffffffff81042c7b>] ? set_cpus_allowed_ptr+0xdd/0xec
Dec  9 12:40:04 server1 kernel: [726544.629451]  [<ffffffff8135079c>] ? __mutex_lock_common.isra.5+0xff/0x164
Dec  9 12:40:04 server1 kernel: [726544.629454]  [<ffffffff8135068a>] ? mutex_lock+0x1a/0x2d
Dec  9 12:40:04 server1 kernel: [726544.629456]  [<ffffffff81037240>] ? test_tsk_need_resched+0xa/0x13
Dec  9 12:40:04 server1 kernel: [726544.629459]  [<ffffffff8103b0e3>] ? check_preempt_curr+0x52/0x5f
Dec  9 12:40:04 server1 kernel: [726544.629464]  [<ffffffffa00c01e5>] ? e1000_watchdog+0x48/0x4ac [e1000]
Dec  9 12:40:04 server1 kernel: [726544.629467]  [<ffffffff8105b717>] ? process_one_work+0x161/0x269
Dec  9 12:40:04 server1 kernel: [726544.629470]  [<ffffffff81059ff6>] ? need_to_create_worker+0x9/0x1c
Dec  9 12:40:04 server1 kernel: [726544.629472]  [<ffffffff8105c6e0>] ? worker_thread+0xc2/0x145
Dec  9 12:40:04 server1 kernel: [726544.629474]  [<ffffffff8105c61e>] ? manage_workers.isra.25+0x15b/0x15b
Dec  9 12:40:04 server1 kernel: [726544.629476]  [<ffffffff8105f829>] ? kthread+0x76/0x7e
Dec  9 12:40:04 server1 kernel: [726544.629479]  [<ffffffff813583f4>] ? kernel_thread_helper+0x4/0x10
Dec  9 12:40:04 server1 kernel: [726544.629482]  [<ffffffff8105f7b3>] ? kthread_worker_fn+0x139/0x139
Dec  9 12:40:04 server1 kernel: [726544.629484]  [<ffffffff813583f0>] ? gs_change+0x13/0x13
Dec  9 12:42:04 server1 kernel: [726664.628090] INFO: task kworker/2:0:14 blocked for more than 120 seconds.
Dec  9 12:42:04 server1 kernel: [726664.631165] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Dec  9 12:42:04 server1 kernel: [726664.631612] kworker/2:0     D ffff88033fd127c0     0    14      2 0x00000000
Dec  9 12:42:04 server1 kernel: [726664.631628]  ffff8803328c4740 0000000000000046 0000000000000000 ffff8803328f0780
Dec  9 12:42:04 server1 kernel: [726664.631632]  00000000000127c0 ffff8803328e5fd8 ffff8803328e5fd8 ffff8803328c4740
Dec  9 12:42:04 server1 kernel: [726664.631634]  0000000000000002 0000000181041d01 0000000000000000 7fffffffffffffff
Dec  9 12:42:04 server1 kernel: [726664.631639] Call Trace:
Dec  9 12:42:04 server1 kernel: [726664.631668]  [<ffffffff8135030b>] ? schedule_timeout+0x2c/0xdb
Dec  9 12:42:04 server1 kernel: [726664.631693]  [<ffffffff81038139>] ? set_next_entity+0x32/0x55
Dec  9 12:42:04 server1 kernel: [726664.631701]  [<ffffffff8100d02f>] ? load_TLS+0x7/0xa
Dec  9 12:42:04 server1 kernel: [726664.631704]  [<ffffffff8100d66c>] ? __switch_to+0x101/0x265
Dec  9 12:42:04 server1 kernel: [726664.631706]  [<ffffffff8134ff51>] ? wait_for_common+0xa0/0x119
Dec  9 12:42:04 server1 kernel: [726664.631712]  [<ffffffff8103f751>] ? try_to_wake_up+0x197/0x197
Dec  9 12:42:04 server1 kernel: [726664.631720]  [<ffffffff8105ae16>] ? wait_on_work+0xe6/0x11c
Dec  9 12:42:04 server1 kernel: [726664.631722]  [<ffffffff8105a450>] ? worker_set_flags+0x8f/0x8f
Dec  9 12:42:04 server1 kernel: [726664.631725]  [<ffffffff8105ba76>] ? __cancel_work_timer+0xb2/0xf4
Dec  9 12:42:04 server1 kernel: [726664.631856]  [<ffffffffa00bc889>] ? e1000_down_and_stop+0x2f/0x48 [e1000]
Dec  9 12:42:04 server1 kernel: [726664.631860]  [<ffffffffa00bf1da>] ? e1000_down+0x108/0x15e [e1000]
Dec  9 12:42:04 server1 kernel: [726664.631864]  [<ffffffffa00c1373>] ? e1000_reset_task+0x53/0x6c [e1000]
Dec  9 12:42:04 server1 kernel: [726664.631866]  [<ffffffff8105b717>] ? process_one_work+0x161/0x269
Dec  9 12:42:04 server1 kernel: [726664.631869]  [<ffffffff8105c6e0>] ? worker_thread+0xc2/0x145
Dec  9 12:42:04 server1 kernel: [726664.631871]  [<ffffffff8105c61e>] ? manage_workers.isra.25+0x15b/0x15b
Dec  9 12:42:04 server1 kernel: [726664.631874]  [<ffffffff8105f829>] ? kthread+0x76/0x7e
Dec  9 12:42:04 server1 kernel: [726664.631881]  [<ffffffff813583f4>] ? kernel_thread_helper+0x4/0x10
Dec  9 12:42:04 server1 kernel: [726664.631884]  [<ffffffff8105f7b3>] ? kthread_worker_fn+0x139/0x139
Dec  9 12:42:04 server1 kernel: [726664.631886]  [<ffffffff813583f0>] ? gs_change+0x13/0x13
Dec  9 12:42:04 server1 kernel: [726664.631921] INFO: task kworker/2:2:171 blocked for more than 120 seconds.
Dec  9 12:42:04 server1 kernel: [726664.632248] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Dec  9 12:42:04 server1 kernel: [726664.632651] kworker/2:2     D ffff88033fd127c0     0   171      2 0x00000000
Dec  9 12:42:04 server1 kernel: [726664.632659]  ffff88032f6cb8c0 0000000000000046 00000000000127c0 ffff880330e13100
Dec  9 12:42:04 server1 kernel: [726664.632662]  00000000000127c0 ffff88032fa53fd8 ffff88032fa53fd8 ffff88032f6cb8c0
Dec  9 12:42:04 server1 kernel: [726664.632664]  ffffffff81351267 ffffffff81042c7b ffff88032f6cb8c0 ffff88032fa51000
Dec  9 12:42:04 server1 kernel: [726664.632667] Call Trace:
Dec  9 12:42:04 server1 kernel: [726664.632671]  [<ffffffff81351267>] ? _raw_spin_unlock_irqrestore+0xe/0xf
Dec  9 12:42:04 server1 kernel: [726664.632675]  [<ffffffff81042c7b>] ? set_cpus_allowed_ptr+0xdd/0xec
Dec  9 12:42:04 server1 kernel: [726664.632677]  [<ffffffff8135079c>] ? __mutex_lock_common.isra.5+0xff/0x164
Dec  9 12:42:04 server1 kernel: [726664.632679]  [<ffffffff8135068a>] ? mutex_lock+0x1a/0x2d
Dec  9 12:42:04 server1 kernel: [726664.632682]  [<ffffffff81037240>] ? test_tsk_need_resched+0xa/0x13
Dec  9 12:42:04 server1 kernel: [726664.632685]  [<ffffffff8103b0e3>] ? check_preempt_curr+0x52/0x5f
Dec  9 12:42:04 server1 kernel: [726664.632690]  [<ffffffffa00c01e5>] ? e1000_watchdog+0x48/0x4ac [e1000]
Dec  9 12:42:04 server1 kernel: [726664.632693]  [<ffffffff8105b717>] ? process_one_work+0x161/0x269
Dec  9 12:42:04 server1 kernel: [726664.632695]  [<ffffffff81059ff6>] ? need_to_create_worker+0x9/0x1c
Dec  9 12:42:04 server1 kernel: [726664.632697]  [<ffffffff8105c6e0>] ? worker_thread+0xc2/0x145
Dec  9 12:42:04 server1 kernel: [726664.632699]  [<ffffffff8105c61e>] ? manage_workers.isra.25+0x15b/0x15b
Dec  9 12:42:04 server1 kernel: [726664.632702]  [<ffffffff8105f829>] ? kthread+0x76/0x7e
Dec  9 12:42:04 server1 kernel: [726664.632705]  [<ffffffff813583f4>] ? kernel_thread_helper+0x4/0x10
Dec  9 12:42:04 server1 kernel: [726664.632707]  [<ffffffff8105f7b3>] ? kthread_worker_fn+0x139/0x139
Dec  9 12:42:04 server1 kernel: [726664.632710]  [<ffffffff813583f0>] ? gs_change+0x13/0x13
Dec  9 12:43:28 server1 shutdown[23719]: shutting down for system reboot
Dec  9 12:43:28 server1 init: Switching to runlevel: 6
Dec  9 12:43:29 server1 vnstatd[2046]: SIGTERM received, exiting.
Dec  9 12:43:30 server1 mysqld: 151209 12:43:30 [Note] /usr/sbin/mysqld: Normal shutdown
Dec  9 12:43:30 server1 mysqld: 
Dec  9 12:43:30 server1 mysqld: 151209 12:43:30 [Note] Event Scheduler: Purging the queue. 0 events
Dec  9 12:43:30 server1 mysqld: 151209 12:43:30  InnoDB: Starting shutdown...
Dec  9 12:43:33 server1 mysqld: 151209 12:43:33  InnoDB: Shutdown completed; log sequence number 3625647798
Dec  9 12:43:33 server1 mysqld: 151209 12:43:33 [Note] /usr/sbin/mysqld: Shutdown complete
Dec  9 12:43:33 server1 mysqld: 
Dec  9 12:43:33 server1 mysqld: 
Dec  9 12:43:33 server1 mysqld: Memory status:
Dec  9 12:43:33 server1 mysqld: Non-mmapped space allocated from system: 10186752
Dec  9 12:43:33 server1 mysqld: Number of free chunks:#011#011#011 67
Dec  9 12:43:33 server1 mysqld: Number of fastbin blocks:#011#011 0
Dec  9 12:43:33 server1 mysqld: Number of mmapped regions:#011#011 17
Dec  9 12:43:33 server1 mysqld: Space in mmapped regions:#011#011 49676288
Dec  9 12:43:33 server1 mysqld: Maximum total allocated space:#011#011 0
Dec  9 12:43:33 server1 mysqld: Space available in freed fastbin blocks: 0
Dec  9 12:43:33 server1 mysqld: Total allocated space:#011#011#011 8657696
Dec  9 12:43:33 server1 mysqld: Total free space:#011#011#011 1529056
Dec  9 12:43:33 server1 mysqld: Top-most, releasable space:#011#011 548544
Dec  9 12:43:33 server1 mysqld: Estimated memory (with thread stack):    59863040
Dec  9 12:43:33 server1 mysqld: 
Dec  9 12:43:33 server1 mysqld: 
Dec  9 12:43:33 server1 mysqld: 
Dec  9 12:43:33 server1 mysqld: Events status:
Dec  9 12:43:33 server1 mysqld: LLA = Last Locked At  LUA = Last Unlocked At
Dec  9 12:43:33 server1 mysqld: WOC = Waiting On Condition  DL = Data Locked
Dec  9 12:43:33 server1 mysqld: 
Dec  9 12:43:33 server1 mysqld: Event scheduler status:
Dec  9 12:43:33 server1 mysqld: State      : INITIALIZED
Dec  9 12:43:33 server1 mysqld: Thread id  : 0
Dec  9 12:43:33 server1 mysqld: LLA        : n/a:0
Dec  9 12:43:33 server1 mysqld: LUA        : n/a:0
Dec  9 12:43:33 server1 mysqld: WOC        : NO
Dec  9 12:43:33 server1 mysqld: Workers    : 0
Dec  9 12:43:33 server1 mysqld: Executed   : 0
Dec  9 12:43:33 server1 mysqld: Data locked: NO
Dec  9 12:43:33 server1 mysqld: 
Dec  9 12:43:33 server1 mysqld: Event queue status:
Dec  9 12:43:33 server1 mysqld: Element count   : 0
Dec  9 12:43:33 server1 mysqld: Data locked     : NO
Dec  9 12:43:33 server1 mysqld: Attempting lock : NO
Dec  9 12:43:33 server1 mysqld: LLA             : drop_schema_events:382
Dec  9 12:43:33 server1 mysqld: LUA             : drop_schema_events:384
Dec  9 12:43:33 server1 mysqld: WOC             : NO
Dec  9 12:43:33 server1 mysqld: Next activation : never
Dec  9 12:43:33 server1 mysqld_safe: mysqld from pid file /var/run/mysqld/mysqld.pid ended
Dec  9 12:43:34 server1 acpid: exiting

Für mich sieht es so aus , als wenn mein MySQL-Server dieses Problem hervorruft:

Code:

 BUG: soft lockup - CPU#1 stuck for 26s! [mysqld:4141]

Hatte jemand mal ein ähnliches Problem in Zusammenhang mit MySQL und einem KVM-Server?

IP-Projects.de · Dec 9, 2015

Das ist kein Kernel Bug. Einmal ein Kernel Update einspielen am besten auch auf dem Host.

d3p · Dec 9, 2015

IP-Projects.de said:
Das ist kein Kernel Bug. Einmal ein Kernel Update einspielen am besten auch auf dem Host.

Huh?
Ich vermute, dass das "k" bei "kein" zu viel ist, oder?

Mein Kernel ist schon aktuell.

Code:

Linux server1 3.2.0-4-amd64 #1 SMP Debian 3.2.68-1+deb7u6 x86_64 GNU/Linux

Mal schauen, was mein Hoster dazu sagt.

E: Gibt es hierzu eventuell einen Bugreport, den ich meinem Hoster "zuschieben" kann?

Danke!

IP-Projects.de · Dec 9, 2015

d3p said:
Huh?
Ich vermute, dass das "k" bei "kein" zu viel ist, oder?

Mein Kernel ist schon aktuell.

Code:

Linux server1 3.2.0-4-amd64 #1 SMP Debian 3.2.68-1+deb7u6 x86_64 GNU/Linux

Mal schauen, was mein Hoster dazu sagt.

E: Gibt es hierzu eventuell einen Bugreport, den ich meinem Hoster "zuschieben" kann?

Danke!

Das ist Ein nicht Kein

... Du findest dazu einiges bei Google. https://www.google.de/search?q=soft...&oe=utf-8&gws_rd=cr&ei=2jRoVofPMYeNsAGwu4awBA

Ich hatte ein vergleichbares Problem letztens bei einem Proxmox Host. Dort war das Problem bei einem 2.6er Kernel.

d3p · Dec 9, 2015

IP-Projects.de said:
Ich hatte ein vergleichbares Problem letztens bei einem Proxmox Host. Dort war das Problem bei einem 2.6er Kernel.

Auch bei Vollvirtualisierung?

Mein Hoster sagt, dass es sehr unwahrscheinlich ist, dass es mit dem Host zusammenhängt, da:
1. es sich um eine Vollvirtualisierung handelt
2. man ansonsten gehäuft Kundenbeschwerden erhalten hätte.

Auf meine Frage bezüglich einer Migration auf einen anderen Host, kam nur die Antwort, dass dies mit "einem hohen Arbeitsaufwand
verbunden" wäre.

Ärgerlich.

Ich probier mal einen anderen Kernel aus.

maltris · Dec 9, 2015

Ich probier mal einen anderen Kernel aus.

ODER

einen anderen Provider.

d3p · Dec 9, 2015

maltris said:
ODER

einen anderen Provider.

Ich bin erst vor kurzem umgezogen, und möchte eigentlich vermeiden, dass ich erneut alles installieren und umkonfigurieren muss.

Was ich nun gemacht habe:
- anderen Kernel (3.16) aus Backports installiert.
Werde nachher schauen, dass ich das System neustarte, und vom neuen Kernel boote.
- MySQL-Server auf 5.6 aktualisiert

Denke nicht, dass MySQL etwas damit zu tun hat, aber eine Aktualisierung kann sicherlich nicht schaden.

IP-Projects.de · Dec 10, 2015

Du kannst auch gerne den 4.2.6 von uns einmal ausprobieren.

http://mirror.ip-projects.de/kernel/

Dank der Vollvirtualisierung sollte dieser auch in einer KVM VM einwandfrei funktionieren. Wenn du die GS Variante nimmst, hat dieser sogar 1000 hz

http://mirror.ip-projects.de/kernel/linux-image-4.2.6-gs_426_amd64.deb

das könnte dir noch ein Quantum Performance bringen.

d3p · Dec 10, 2015

Und wieder abgestürzt....

IP-Projects.de said:
Du kannst auch gerne den 4.2.6 von uns einmal ausprobieren.

Danke für den Tipp.
Ich hab nun den 4.2.6 installiert und von diesem gestartet.

Mal schauen, wie lange das nun gut geht.

d3p · Dec 10, 2015

Bitte entschuldigt diesen "Doppelpost" / "Push"

Auch mit dem Kernel (4.2.3) von IP-Projects schmiert mir der VServer in unregelmäßigen Abständen ab.

Hat sonst noch jemand eine Idee, an was das liegen kann?

Code:

[Thu Dec 10 17:27:44 2015] INFO: rcu_sched detected stalls on CPUs/tasks: { 0} (detected by 1, t=6640 jiffies, g=521522, c=521521, q=100)
[Thu Dec 10 17:27:44 2015] Task dump for CPU 0:
[Thu Dec 10 17:27:44 2015] swapper/0       R  running task        0     0      0 0x00000008
[Thu Dec 10 17:27:44 2015]  0000000000000000 0000000000000000 0000000000000000 0000000000000000
[Thu Dec 10 17:27:44 2015]  ffffffff81a40578 0000000000000000 0000000000000001 ffffffffffffffc4
[Thu Dec 10 17:27:44 2015]  ffffffff8103e86f 0000000000000010 0000000000000246 ffffffff81a03f10
[Thu Dec 10 17:27:44 2015] Call Trace:
[Thu Dec 10 17:27:44 2015]  [<ffffffff8103e86f>] ? native_safe_halt+0x2/0x3
[Thu Dec 10 17:27:44 2015]  [<ffffffff81014fcd>] ? default_idle+0x17/0x26
[Thu Dec 10 17:27:44 2015]  [<ffffffff81014fb6>] ? trace_cpu_idle_rcuidle+0x47/0x47
[Thu Dec 10 17:27:44 2015]  [<ffffffff81076501>] ? cpu_startup_entry+0x147/0x23e
[Thu Dec 10 17:27:44 2015]  [<ffffffff81af7efa>] ? start_kernel+0x434/0x43f
[Thu Dec 10 17:27:44 2015]  [<ffffffff81af78c0>] ? set_init_arg+0x52/0x52
[Thu Dec 10 17:27:44 2015]  [<ffffffff81af7117>] ? early_idt_handler_array+0x117/0x120
[Thu Dec 10 17:27:44 2015]  [<ffffffff81af7117>] ? early_idt_handler_array+0x117/0x120
[Thu Dec 10 17:27:44 2015]  [<ffffffff81af75e6>] ? x86_64_start_kernel+0x12e/0x13b
[Thu Dec 10 17:28:17 2015] INFO: rcu_sched detected stalls on CPUs/tasks: { 0} (detected by 1, t=7886 jiffies, g=521557, c=521556, q=416)
[Thu Dec 10 17:28:17 2015] Task dump for CPU 0:
[Thu Dec 10 17:28:17 2015] php5-fpm        R  running task        0  4831   3400 0x00000008
[Thu Dec 10 17:28:17 2015]  ffff8800370e8c40 ffff88032ef28b40 ffffffff810431fc ffff880300000000
[Thu Dec 10 17:28:17 2015]  00000000811407a2 ffff88032ef28ba8 0000000000000002 000000000356df90
[Thu Dec 10 17:28:17 2015]  0000000000000000 000000000242c2c0 0000000003831f88 0000000000000048
[Thu Dec 10 17:28:17 2015] Call Trace:
[Thu Dec 10 17:28:17 2015]  [<ffffffff810431fc>] ? __do_page_fault+0x3b7/0x412
[Thu Dec 10 17:28:17 2015]  [<ffffffff815467c8>] ? async_page_fault+0x28/0x30
[Thu Dec 10 17:28:17 2015] rcu_sched kthread starved for 7735 jiffies! g521557 c521556 f0x0
[Thu Dec 10 17:28:36 2015] INFO: rcu_sched self-detected stall on CPU {
[Thu Dec 10 17:29:06 2015] INFO: rcu_sched detected stalls on CPUs/tasks: { 1} (detected by 3, t=7756 jiffies, g=521599, c=521598, q=849)
[Thu Dec 10 17:29:06 2015] Task dump for CPU 1:
[Thu Dec 10 17:29:06 2015] php5-fpm        R  running task        0  4844   3400 0x00000008
[Thu Dec 10 17:29:06 2015]  ffff880330b1b100 ffff88032e528b40 ffffffff81043113 ffff880300000000
[Thu Dec 10 17:29:06 2015]  00000000ffffffe0 ffff88032e528ba8 0000000000000002 0000000002bbf53a
[Thu Dec 10 17:29:06 2015]  0000000000000000 0000000000f586e0 000000000377fea0 00007ffcefd9e5bc
[Thu Dec 10 17:29:06 2015] Call Trace:
[Thu Dec 10 17:29:06 2015]  [<ffffffff81043113>] ? __do_page_fault+0x2ce/0x412
[Thu Dec 10 17:29:06 2015]  [<ffffffff815467c8>] ? async_page_fault+0x28/0x30
[Thu Dec 10 17:29:06 2015] rcu_sched kthread starved for 7756 jiffies! g521599 c521598 f0x0
[Thu Dec 10 17:28:36 2015]  1}  (t=7756 jiffies g=521599 c=521598 q=889)
[Thu Dec 10 17:28:36 2015] Task dump for CPU 1:
[Thu Dec 10 17:28:36 2015] php5-fpm        R  running task        0  4844   3400 0x00000008
[Thu Dec 10 17:28:36 2015]  ffffffff81a42480 0000000000000001 ffffffff8108b0c4 ffff88033fc96cc0
[Thu Dec 10 17:28:36 2015]  ffffffff81a42480 ffff88033fc96cc0 ffff88033fc83798 0000000000000000
[Thu Dec 10 17:28:36 2015]  ffffffff8108d428 0000000000000286 ffffffff81544543 ffff88032f108620
[Thu Dec 10 17:28:36 2015] Call Trace:
[Thu Dec 10 17:28:36 2015]  <IRQ>  [<ffffffff8108b0c4>] ? rcu_dump_cpu_stacks+0x55/0x87
[Thu Dec 10 17:28:36 2015]  [<ffffffff8108d428>] ? rcu_check_callbacks+0x1e0/0x58b
[Thu Dec 10 17:28:36 2015]  [<ffffffff81544543>] ? _raw_spin_unlock_irqrestore+0xb/0xc
[Thu Dec 10 17:28:36 2015]  [<ffffffff8109bdd0>] ? tick_nohz_handler+0x78/0x78
[Thu Dec 10 17:28:36 2015]  [<ffffffff8109084d>] ? update_process_times+0x24/0x47
[Thu Dec 10 17:28:36 2015]  [<ffffffff8109b77a>] ? tick_sched_handle+0x32/0x3d
[Thu Dec 10 17:28:36 2015]  [<ffffffff8109bdff>] ? tick_sched_timer+0x2f/0x55
[Thu Dec 10 17:28:36 2015]  [<ffffffff81090f61>] ? __hrtimer_run_queues+0xcf/0x1a4
[Thu Dec 10 17:28:36 2015]  [<ffffffff810913db>] ? hrtimer_interrupt+0x62/0x106
[Thu Dec 10 17:28:36 2015]  [<ffffffff81035d14>] ? smp_apic_timer_interrupt+0x25/0x34
[Thu Dec 10 17:28:36 2015]  [<ffffffff8154571b>] ? apic_timer_interrupt+0x6b/0x70
[Thu Dec 10 17:28:36 2015]  [<ffffffff814a8239>] ? find_exception+0x27/0x35
[Thu Dec 10 17:28:36 2015]  [<ffffffff814a822b>] ? find_exception+0x19/0x35
[Thu Dec 10 17:28:36 2015]  [<ffffffff814a9299>] ? __ip_route_output_key+0x42a/0x60c
[Thu Dec 10 17:28:36 2015]  [<ffffffff814a948d>] ? ip_route_output_flow+0x12/0x3d
[Thu Dec 10 17:28:36 2015]  [<ffffffff814b4a08>] ? inet_csk_route_req+0xe6/0x129
[Thu Dec 10 17:28:36 2015]  [<ffffffff814c5bd5>] ? tcp_v4_route_req+0x12/0x26
[Thu Dec 10 17:28:36 2015]  [<ffffffff814bc9fe>] ? tcp_conn_request+0x3f7/0x653
[Thu Dec 10 17:28:36 2015]  [<ffffffff81075e38>] ? __wake_up_sync_key+0x40/0x58
[Thu Dec 10 17:28:36 2015]  [<ffffffff8146d5ef>] ? sk_wake_async+0x19/0x3c
[Thu Dec 10 17:28:36 2015]  [<ffffffff814c00aa>] ? tcp_rcv_state_process+0x63/0x9dd
[Thu Dec 10 17:28:36 2015]  [<ffffffff814b3578>] ? __inet_lookup_established+0x3c/0xf3
[Thu Dec 10 17:28:36 2015]  [<ffffffff814b5364>] ? inet_csk_search_req+0x4a/0xb0
[Thu Dec 10 17:28:36 2015]  [<ffffffff814c7e80>] ? tcp_v4_do_rcv+0x219/0x26a
[Thu Dec 10 17:28:36 2015]  [<ffffffff814c8baa>] ? tcp_v4_rcv+0x4f3/0x768
[Thu Dec 10 17:28:36 2015]  [<ffffffff814ac1a8>] ? ip_local_deliver_finish+0xcc/0x155
[Thu Dec 10 17:28:36 2015]  [<ffffffff814ac365>] ? ip_local_deliver+0x4d/0xa5
[Thu Dec 10 17:28:36 2015]  [<ffffffff814abe3e>] ? ip_rcv_finish+0x7b/0x2c9
[Thu Dec 10 17:28:36 2015]  [<ffffffff814ac63d>] ? ip_rcv+0x280/0x317
[Thu Dec 10 17:28:36 2015]  [<ffffffff8147cd16>] ? deliver_ptype_list_skb+0x3b/0x74
[Thu Dec 10 17:28:36 2015]  [<ffffffff81470c99>] ? skb_gso_transport_seglen+0x69/0x69
[Thu Dec 10 17:28:36 2015]  [<ffffffff8147f24f>] ? __netif_receive_skb_core+0x4ba/0x540
[Thu Dec 10 17:28:36 2015]  [<ffffffff8103e639>] ? kvm_clock_read+0x1b/0x1d
[Thu Dec 10 17:28:36 2015]  [<ffffffff810944c4>] ? timekeeping_get_ns+0xa/0x2e
[Thu Dec 10 17:28:36 2015]  [<ffffffff81094af4>] ? ktime_get_with_offset+0x4b/0x63
[Thu Dec 10 17:28:36 2015]  [<ffffffff8147f4b1>] ? netif_receive_skb_internal+0x4e/0x7f
[Thu Dec 10 17:28:36 2015]  [<ffffffff8147fe65>] ? napi_gro_receive+0x5a/0xb7
[Thu Dec 10 17:28:36 2015]  [<ffffffffa0078f24>] ? e1000_clean_rx_irq+0x2b4/0x3d7 [e1000]
[Thu Dec 10 17:28:36 2015]  [<ffffffffa00795fb>] ? e1000_clean+0x596/0x711 [e1000]
[Thu Dec 10 17:28:36 2015]  [<ffffffff8153efde>] ? __slab_free+0x18b/0x28b
[Thu Dec 10 17:28:36 2015]  [<ffffffff8147faae>] ? net_rx_action+0x11a/0x2da
[Thu Dec 10 17:28:36 2015]  [<ffffffff8138d919>] ? paravirt_read_tsc+0x5/0x8
[Thu Dec 10 17:28:36 2015]  [<ffffffff8138ffe7>] ? add_interrupt_randomness+0x30/0x17c
[Thu Dec 10 17:28:36 2015]  [<ffffffff8104fa41>] ? __do_softirq+0xd6/0x22d
[Thu Dec 10 17:28:36 2015]  [<ffffffff810382d5>] ? ioapic_ack_level+0x59/0xef
[Thu Dec 10 17:28:36 2015]  [<ffffffff8104fcf5>] ? irq_exit+0x38/0x7c
[Thu Dec 10 17:28:36 2015]  [<ffffffff8100ec5c>] ? do_IRQ+0x9b/0xb1
[Thu Dec 10 17:28:36 2015]  [<ffffffff8154542b>] ? common_interrupt+0x6b/0x6b
[Thu Dec 10 17:28:36 2015]  <EOI> 
[Thu Dec 10 17:29:06 2015] INFO: rcu_sched detected stalls on CPUs/tasks: { 0} (detected by 1, t=21990 jiffies, g=521642, c=521641, q=382)
[Thu Dec 10 17:30:34 2015] Task dump for CPU 0:
[Thu Dec 10 17:30:34 2015] mysqld          R  running task        0  4861   2407 0x00000008
[Thu Dec 10 17:30:34 2015]  ffff88033fc0f740 ffff8800b86f7a60 0000000000000001 0000000000000286
[Thu Dec 10 17:30:34 2015]  ffffffff81544543 ffffffff81090e71 0000000000000000 0000000000000286
[Thu Dec 10 17:30:34 2015]  ffff8800b86f7a60 0000000000000000 0000000005f5e100 ffffffff81090e88
[Thu Dec 10 17:30:34 2015] Call Trace:
[Thu Dec 10 17:30:34 2015]  [<ffffffff81544543>] ? _raw_spin_unlock_irqrestore+0xb/0xc
[Thu Dec 10 17:30:34 2015]  [<ffffffff81090e71>] ? hrtimer_try_to_cancel+0x88/0x93
[Thu Dec 10 17:30:34 2015]  [<ffffffff81090e88>] ? hrtimer_cancel+0xc/0x16
[Thu Dec 10 17:30:34 2015]  [<ffffffff81544128>] ? schedule_hrtimeout_range_clock+0x99/0xbf
[Thu Dec 10 17:30:34 2015]  [<ffffffff815446df>] ? _raw_spin_lock_irqsave+0x17/0x3b
[Thu Dec 10 17:30:34 2015]  [<ffffffff81075eac>] ? remove_wait_queue+0xf/0x48
[Thu Dec 10 17:30:34 2015]  [<ffffffff8112ac2b>] ? fput+0xd/0x81
[Thu Dec 10 17:30:34 2015]  [<ffffffff81137de6>] ? poll_freewait+0x37/0x85
[Thu Dec 10 17:30:34 2015]  [<ffffffff81138ccb>] ? do_sys_poll+0x388/0x40a
[Thu Dec 10 17:30:34 2015]  [<ffffffff81122ead>] ? __kmalloc_node_track_caller+0x11d/0x134
[Thu Dec 10 17:30:34 2015]  [<ffffffff814f610a>] ? sk_wake_async+0x19/0x3c
[Thu Dec 10 17:30:34 2015]  [<ffffffff814f6273>] ? unix_write_space+0x60/0x62
[Thu Dec 10 17:30:34 2015]  [<ffffffff8146faa4>] ? sock_wfree+0x35/0x4f
[Thu Dec 10 17:30:34 2015]  [<ffffffff814f712a>] ? unix_destruct_scm+0x41/0x48
[Thu Dec 10 17:30:34 2015]  [<ffffffff814f6fdd>] ? scm_destroy+0xc/0x25
[Thu Dec 10 17:30:34 2015]  [<ffffffff814f7029>] ? scm_recv.isra.27+0x33/0xf3
[Thu Dec 10 17:30:34 2015]  [<ffffffff81121742>] ? kmem_cache_free+0x32/0xd7
[Thu Dec 10 17:30:34 2015]  [<ffffffff814f830c>] ? unix_stream_read_generic+0x5df/0x606
[Thu Dec 10 17:30:34 2015]  [<ffffffff814f83d7>] ? unix_stream_recvmsg+0x3a/0x3f
[Thu Dec 10 17:30:34 2015]  [<ffffffff814f6fa5>] ? unix_seq_start+0x42/0x42
[Thu Dec 10 17:30:34 2015]  [<ffffffff8112ac2b>] ? fput+0xd/0x81
[Thu Dec 10 17:30:34 2015]  [<ffffffff8146be39>] ? SYSC_recvfrom+0xd6/0xe8
[Thu Dec 10 17:30:34 2015]  [<ffffffff8103e639>] ? kvm_clock_read+0x1b/0x1d
[Thu Dec 10 17:30:34 2015]  [<ffffffff81094b4a>] ? ktime_get_ts64+0x3e/0x7d
[Thu Dec 10 17:30:34 2015]  [<ffffffff81138ddd>] ? SyS_poll+0x48/0xb1
[Thu Dec 10 17:30:34 2015]  [<ffffffff815448ee>] ? entry_SYSCALL_64_fastpath+0x12/0x71
[Thu Dec 10 17:30:34 2015] rcu_sched kthread starved for 21990 jiffies! g521642 c521641 f0x0
[Thu Dec 10 17:31:18 2015] INFO: rcu_sched detected stalls on CPUs/tasks: { 0} (detected by 1, t=10641 jiffies, g=521692, c=521691, q=628)
[Thu Dec 10 17:31:18 2015] Task dump for CPU 0:
[Thu Dec 10 17:31:18 2015] ksoftirqd/0     R  running task        0     3      2 0x00000000
[Thu Dec 10 17:31:18 2015]  000000000000002f ffffffff81542306 000000010022ae12 0420804000000002
[Thu Dec 10 17:31:18 2015]  ffff880331584000 ffff880333002110 ffff880331561880 ffffffff81a388c0
[Thu Dec 10 17:31:18 2015]  ffffffff81064d4e 0000000000000000 0000000000000000 ffffffff8104fbae
[Thu Dec 10 17:31:18 2015] Call Trace:
[Thu Dec 10 17:31:18 2015]  [<ffffffff81542306>] ? __schedule+0x409/0x4f1
[Thu Dec 10 17:31:18 2015]  [<ffffffff81064d4e>] ? smpboot_unregister_percpu_thread+0x5a/0x5a
[Thu Dec 10 17:31:18 2015]  [<ffffffff8104fbae>] ? run_ksoftirqd+0x16/0x32
[Thu Dec 10 17:31:18 2015]  [<ffffffff81064e74>] ? smpboot_thread_fn+0x126/0x13e
[Thu Dec 10 17:31:18 2015]  [<ffffffff810629fd>] ? kthread+0x99/0xa1
[Thu Dec 10 17:31:18 2015]  [<ffffffff81062964>] ? kthread_parkme+0x16/0x16
[Thu Dec 10 17:31:18 2015]  [<ffffffff81544cdf>] ? ret_from_fork+0x3f/0x70
[Thu Dec 10 17:31:18 2015]  [<ffffffff81062964>] ? kthread_parkme+0x16/0x16
[Thu Dec 10 17:31:18 2015] rcu_sched kthread starved for 10562 jiffies! g521692 c521691 f0x0
[Thu Dec 10 17:31:19 2015] INFO: rcu_sched detected stalls on CPUs/tasks: { 0} (detected by 1, t=10023 jiffies, g=521735, c=521734, q=484)
[Thu Dec 10 17:31:19 2015] Task dump for CPU 0:
[Thu Dec 10 17:31:19 2015] php5-fpm        R  running task        0  4818   3400 0x00000008
[Thu Dec 10 17:31:19 2015]  0000000000000001 ffffffff81542306 ffffffff811280e7 ffff8800370e8000
[Thu Dec 10 17:31:19 2015]  ffffff9c811407a2 ffff8800ad9d4000 000000000242c2c0 0000000002ebc998
[Thu Dec 10 17:31:19 2015]  ffff8800ad9d4000 000000000242c2c0 0000000002ebc998 ffffffff815424e3
[Thu Dec 10 17:31:19 2015] Call Trace:
[Thu Dec 10 17:31:19 2015]  [<ffffffff81542306>] ? __schedule+0x409/0x4f1
[Thu Dec 10 17:31:19 2015]  [<ffffffff811280e7>] ? SYSC_faccessat+0x175/0x186
[Thu Dec 10 17:31:19 2015]  [<ffffffff815424e3>] ? schedule+0x6b/0x78
[Thu Dec 10 17:31:19 2015]  [<ffffffff8154552c>] ? retint_careful+0x14/0x32
[Thu Dec 10 17:31:19 2015] rcu_sched kthread starved for 9927 jiffies! g521735 c521734 f0x0
[Thu Dec 10 17:31:58 2015] TCP: request_sock_TCP: Possible SYN flooding on port 80. Sending cookies.  Check SNMP counters.
[Thu Dec 10 17:31:59 2015] INFO: rcu_sched detected stalls on CPUs/tasks: {} (detected by 1, t=7426 jiffies, g=521777, c=521776, q=910)
[Thu Dec 10 17:31:59 2015] All QSes seen, last rcu_sched kthread activity 7425 (4297257804-4297250379), jiffies_till_next_fqs=1, root ->qsmask 0x0
[Thu Dec 10 17:31:59 2015] mysqld          R  running task        0  4812   2407 0x00000000
[Thu Dec 10 17:31:59 2015]  ffffffff81a42480 ffff88033fc96cc0 ffffffff8108d6c1 0000000000000000
[Thu Dec 10 17:31:59 2015]  ffff88033fc96180 00007f214f063130 000000000000038e 000000000007f631
[Thu Dec 10 17:31:59 2015]  0000000000000082 ffff880322650c40 ffffffff81ad5f70 0000000000000001
[Thu Dec 10 17:31:59 2015] Call Trace:
[Thu Dec 10 17:31:59 2015]  <IRQ>  [<ffffffff8108d6c1>] ? rcu_check_callbacks+0x479/0x58b
[Thu Dec 10 17:31:59 2015]  [<ffffffff8109bdd0>] ? tick_nohz_handler+0x78/0x78
[Thu Dec 10 17:31:59 2015]  [<ffffffff8109084d>] ? update_process_times+0x24/0x47
[Thu Dec 10 17:31:59 2015]  [<ffffffff8109b77a>] ? tick_sched_handle+0x32/0x3d
[Thu Dec 10 17:31:59 2015]  [<ffffffff8109bdff>] ? tick_sched_timer+0x2f/0x55
[Thu Dec 10 17:31:59 2015]  [<ffffffff81090f61>] ? __hrtimer_run_queues+0xcf/0x1a4
[Thu Dec 10 17:31:59 2015]  [<ffffffff810913db>] ? hrtimer_interrupt+0x62/0x106
[Thu Dec 10 17:31:59 2015]  [<ffffffff81035d14>] ? smp_apic_timer_interrupt+0x25/0x34
[Thu Dec 10 17:31:59 2015]  [<ffffffff8154571b>] ? apic_timer_interrupt+0x6b/0x70
[Thu Dec 10 17:31:59 2015]  <EOI> 
[Thu Dec 10 17:31:59 2015] rcu_sched kthread starved for 7425 jiffies! g521777 c521776 f0x2
[Thu Dec 10 17:32:31 2015] INFO: rcu_sched self-detected stall on CPU { 1}  (t=9184 jiffies g=521816 c=521815 q=1241)
[Thu Dec 10 17:32:31 2015] rcu_sched kthread starved for 9184 jiffies! g521816 c521815 f0x0
[Thu Dec 10 17:32:31 2015] Task dump for CPU 1:
[Thu Dec 10 17:32:31 2015] php5-fpm        R  running task        0  4845   3400 0x00000008
[Thu Dec 10 17:32:31 2015]  ffffffff81a42480 0000000000000001 ffffffff8108b0c4 ffff88033fc96cc0
[Thu Dec 10 17:32:31 2015]  ffffffff81a42480 ffff88033fc96cc0 ffff88033fc83848 0000000000000000
[Thu Dec 10 17:32:31 2015]  ffffffff8108d428 0000000000177fbe 00000003300c9cfa ffff88032f108620
[Thu Dec 10 17:32:31 2015] Call Trace:
[Thu Dec 10 17:32:31 2015]  <IRQ>  [<ffffffff8108b0c4>] ? rcu_dump_cpu_stacks+0x55/0x87
[Thu Dec 10 17:32:31 2015]  [<ffffffff8108d428>] ? rcu_check_callbacks+0x1e0/0x58b
[Thu Dec 10 17:32:31 2015]  [<ffffffff8109bdd0>] ? tick_nohz_handler+0x78/0x78
[Thu Dec 10 17:32:31 2015]  [<ffffffff8109084d>] ? update_process_times+0x24/0x47
[Thu Dec 10 17:32:31 2015]  [<ffffffff8109b77a>] ? tick_sched_handle+0x32/0x3d
[Thu Dec 10 17:32:31 2015]  [<ffffffff8109bdff>] ? tick_sched_timer+0x2f/0x55
[Thu Dec 10 17:32:31 2015]  [<ffffffff81090f61>] ? __hrtimer_run_queues+0xcf/0x1a4
[Thu Dec 10 17:32:31 2015]  [<ffffffff810913db>] ? hrtimer_interrupt+0x62/0x106
[Thu Dec 10 17:32:31 2015]  [<ffffffff81035d14>] ? smp_apic_timer_interrupt+0x25/0x34
[Thu Dec 10 17:32:31 2015]  [<ffffffff8154571b>] ? apic_timer_interrupt+0x6b/0x70
[Thu Dec 10 17:32:31 2015]  [<ffffffff8153ec5b>] ? __slab_alloc+0x379/0x3c0
[Thu Dec 10 17:32:31 2015]  [<ffffffff814a8239>] ? find_exception+0x27/0x35
[Thu Dec 10 17:32:31 2015]  [<ffffffff814a822b>] ? find_exception+0x19/0x35
[Thu Dec 10 17:32:31 2015]  [<ffffffff814a9299>] ? __ip_route_output_key+0x42a/0x60c
[Thu Dec 10 17:32:31 2015]  [<ffffffff814a948d>] ? ip_route_output_flow+0x12/0x3d
[Thu Dec 10 17:32:31 2015]  [<ffffffff814b5c78>] ? inet_csk_route_child_sock+0x119/0x161
[Thu Dec 10 17:32:31 2015]  [<ffffffff814c69b5>] ? tcp_v4_syn_recv_sock+0x11a/0x24b
[Thu Dec 10 17:32:31 2015]  [<ffffffff814c9849>] ? tcp_check_req+0x299/0x36b
[Thu Dec 10 17:32:31 2015]  [<ffffffff814b5364>] ? inet_csk_search_req+0x4a/0xb0
[Thu Dec 10 17:32:31 2015]  [<ffffffff814c7d70>] ? tcp_v4_do_rcv+0x109/0x26a
[Thu Dec 10 17:32:31 2015]  [<ffffffff814c8baa>] ? tcp_v4_rcv+0x4f3/0x768
[Thu Dec 10 17:32:31 2015]  [<ffffffff814ac1a8>] ? ip_local_deliver_finish+0xcc/0x155
[Thu Dec 10 17:32:31 2015]  [<ffffffff814ac365>] ? ip_local_deliver+0x4d/0xa5
[Thu Dec 10 17:32:31 2015]  [<ffffffff814abe3e>] ? ip_rcv_finish+0x7b/0x2c9
[Thu Dec 10 17:32:31 2015]  [<ffffffff814ac63d>] ? ip_rcv+0x280/0x317
[Thu Dec 10 17:32:31 2015]  [<ffffffff8147cd16>] ? deliver_ptype_list_skb+0x3b/0x74
[Thu Dec 10 17:32:31 2015]  [<ffffffff8147f24f>] ? __netif_receive_skb_core+0x4ba/0x540
[Thu Dec 10 17:32:31 2015]  [<ffffffff8103e639>] ? kvm_clock_read+0x1b/0x1d
[Thu Dec 10 17:32:31 2015]  [<ffffffff810944c4>] ? timekeeping_get_ns+0xa/0x2e
[Thu Dec 10 17:32:31 2015]  [<ffffffff81094af4>] ? ktime_get_with_offset+0x4b/0x63
[Thu Dec 10 17:32:31 2015]  [<ffffffff8147f4b1>] ? netif_receive_skb_internal+0x4e/0x7f
[Thu Dec 10 17:32:31 2015]  [<ffffffff8147fe65>] ? napi_gro_receive+0x5a/0xb7
[Thu Dec 10 17:32:31 2015]  [<ffffffffa0078f24>] ? e1000_clean_rx_irq+0x2b4/0x3d7 [e1000]
[Thu Dec 10 17:32:31 2015]  [<ffffffffa00795fb>] ? e1000_clean+0x596/0x711 [e1000]
[Thu Dec 10 17:32:31 2015]  [<ffffffff814c26e7>] ? tcp_transmit_skb+0x75f/0x790
[Thu Dec 10 17:32:31 2015]  [<ffffffff81013e7d>] ? paravirt_sched_clock+0x5/0x8
[Thu Dec 10 17:32:31 2015]  [<ffffffff8106d79d>] ? sched_clock_local+0xd/0x6f
[Thu Dec 10 17:32:31 2015]  [<ffffffff814c5038>] ? tcp_delack_timer_handler+0x17f/0x17f
[Thu Dec 10 17:32:31 2015]  [<ffffffff8147faae>] ? net_rx_action+0x11a/0x2da
[Thu Dec 10 17:32:31 2015]  [<ffffffff8138d919>] ? paravirt_read_tsc+0x5/0x8
[Thu Dec 10 17:32:31 2015]  [<ffffffff8138ffe7>] ? add_interrupt_randomness+0x30/0x17c
[Thu Dec 10 17:32:31 2015]  [<ffffffff8104fa41>] ? __do_softirq+0xd6/0x22d
[Thu Dec 10 17:32:31 2015]  [<ffffffff810382d5>] ? ioapic_ack_level+0x59/0xef
[Thu Dec 10 17:32:31 2015]  [<ffffffff8104fcf5>] ? irq_exit+0x38/0x7c
[Thu Dec 10 17:32:31 2015]  [<ffffffff8100ec5c>] ? do_IRQ+0x9b/0xb1
[Thu Dec 10 17:32:31 2015]  [<ffffffff8154542b>] ? common_interrupt+0x6b/0x6b
[Thu Dec 10 17:32:31 2015]  <EOI> 
[Thu Dec 10 17:33:07 2015] INFO: rcu_sched detected stalls on CPUs/tasks: { 0} (detected by 1, t=9487 jiffies, g=521857, c=521856, q=1283)
[Thu Dec 10 17:33:07 2015] Task dump for CPU 0:
[Thu Dec 10 17:33:07 2015] mysqld          R  running task        0  4865   2407 0x00000008
[Thu Dec 10 17:33:07 2015]  0000000000004000 00000000021ac000 ffffffff8112a473 ffff88003703a4c0
[Thu Dec 10 17:33:07 2015]  00007f2147ce4000 ffff88032ec60400 00007f2100000001 0000000000000001
[Thu Dec 10 17:33:07 2015]  00000000021ac000 ffff88032ec60400 00007f2100000001 ffffffff815424e3
[Thu Dec 10 17:33:07 2015] Call Trace:
[Thu Dec 10 17:33:07 2015]  [<ffffffff8112a473>] ? SyS_pread64+0x61/0x87
[Thu Dec 10 17:33:07 2015]  [<ffffffff815424e3>] ? schedule+0x6b/0x78
[Thu Dec 10 17:33:07 2015]  [<ffffffff815448ee>] ? entry_SYSCALL_64_fastpath+0x12/0x71
[Thu Dec 10 17:33:07 2015] rcu_sched kthread starved for 9484 jiffies! g521857 c521856 f0x0

Deleted member 15972 · Dec 10, 2015

Ich wüsste ehrlich gesagt nicht, wie das in Deinen Verantwortungsbereich fallen soll.
Solche Kernelcrashs werden durch Probleme mit Hardware oder Treiber verursacht.

IMO (vielleicht fällt jemandem etwas anderes ein) ist das Sache des Hosters, denn dort *muss* irgend etwas schief laufen.
Wobei ich gerne wüsste, welcher Hoster das ist, der einen sehr hohen Arbeitsaufwand für eine Migration hätte (falls Du mir den Namen per PM schicken möchtest, wäre das sehr lieb).

remote_mind · Dec 10, 2015

d3p said:

Code:

[Thu Dec 10 17:28:36 2015] INFO: rcu_sched self-detected stall on CPU {
[Thu Dec 10 17:29:06 2015] INFO: rcu_sched detected stalls on CPUs/tasks: { 1} (detected by 3, t=7756 jiffies, g=521599, c=521598, q=849)
[Thu Dec 10 17:29:06 2015] Task dump for CPU 1:
[Thu Dec 10 17:29:06 2015] php5-fpm        R  running task        0  4844   3400 0x00000008
[Thu Dec 10 17:29:06 2015]  ffff880330b1b100 ffff88032e528b40 ffffffff81043113 ffff880300000000
[Thu Dec 10 17:29:06 2015]  00000000ffffffe0 ffff88032e528ba8 0000000000000002 0000000002bbf53a
[Thu Dec 10 17:29:06 2015]  0000000000000000 0000000000f586e0 000000000377fea0 00007ffcefd9e5bc
[Thu Dec 10 17:29:06 2015] Call Trace:
[Thu Dec 10 17:29:06 2015]  [<ffffffff81043113>] ? __do_page_fault+0x2ce/0x412
[Thu Dec 10 17:29:06 2015]  [<ffffffff815467c8>] ? async_page_fault+0x28/0x30
[Thu Dec 10 17:29:06 2015] rcu_sched kthread starved for 7756 jiffies! g521599 c521598 f0x0
[Thu Dec 10 17:28:36 2015]  1}  (t=7756 jiffies g=521599 c=521598 q=889)
[Thu Dec 10 17:28:36 2015] Task dump for CPU 1:
[Thu Dec 10 17:28:36 2015] php5-fpm        R  running task        0  4844   3400 0x00000008
[Thu Dec 10 17:28:36 2015]  ffffffff81a42480 0000000000000001 ffffffff8108b0c4 ffff88033fc96cc0
[Thu Dec 10 17:28:36 2015]  ffffffff81a42480 ffff88033fc96cc0 ffff88033fc83798 0000000000000000
[Thu Dec 10 17:28:36 2015]  ffffffff8108d428 0000000000000286 ffffffff81544543 ffff88032f108620

Ist das ein copy/paste Fehler oder läuft die Uhr hier ein Stück rückwärts?
Oder der stall-Warn wird von einem zweiten Stall-Warn 'unterbrochen'?

Nils

d3p · Dec 10, 2015

Orebor said:
I(falls Du mir den Namen per PM schicken möchtest, wäre das sehr lieb).

Ich mach's einfach öffentlich.

Es handelt sich hierbei um den Hoster "Netcup".

Sicherlich verstehe ich, dass es mehrere Komponenten im Hintergrund gibt (Hostsystem, Kundendatenbank etc.) die miteinander verknüpft sind und so eventuell eine Migration auf eine andere Maschine erschweren, aber gleich so abzublocken ist leider sehr schade.

Wie gesagt, die eingesetzte Software ist nahezu identisch, wie bei meinem alten Hetzner-Server, auf dem das Projekt vor dem Umzug (problemlos) lief.

remote_mind said:
Ist das ein copy/paste Fehler oder läuft die Uhr hier ein Stück rückwärts?
Oder der stall-Warn wird von einem zweiten Stall-Warn 'unterbrochen'?

Die Uhr läuft korrekt, und es handelt sich auch um keinen copy/paste-Fehler.

Deleted member 15972 · Dec 10, 2015

Das merkwürdige Log *könnte* durch eine Racecondition verursacht worden sein.
Wie ist denn die übliche Auslastung der Systemressourcen?
Gibt es evtl. eine hohe steal time?
Da ich mir noch vorstellen *könnte*, dass das an Overcommitement liegt.

Fairerweise muss ich allerdings dazu sagen, dass ich das bei Netcup bisher nicht erlebt hätte, also Hosts die massiv überbucht wären o.ä.
Vielleicht kann sich auch Felix dazu äußern, der ist hier ja sehr aktiv.

d3p · Dec 10, 2015

Orebor said:
Wie ist denn die übliche Auslastung der Systemressourcen?
Gibt es evtl. eine hohe steal time?

Nicht sehr hoch.

Ich hab 4x dedizierte CPU-Kerne zugewiesen und 12GB RAM.

Code:

top - 18:59:46 up  4:04,  2 users,  load average: 0.37, 0.35, 0.38
Tasks: 164 total,   2 running, 162 sleeping,   0 stopped,   0 zombie
%Cpu(s):  7.1 us,  0.8 sy,  0.0 ni, 91.4 id,  0.3 wa,  0.0 hi,  0.2 si,  0.3 st
KiB Mem:  12309520 total,  3269228 used,  9040292 free,   166296 buffers
KiB Swap:  1492988 total,        0 used,  1492988 free,  1939484 cached

Ich hab derzeit kein Monitoring (Munin / Zabbix / etc.) installiert, deswegen kann ich bezüglich der steal time keine Aussage machen.

remote_mind · Dec 10, 2015

Achso, auffällig wäre noch das in vielen der Meldungen etwas auf die Netwerkkarte hinweist 'e1000'. Evtl. mal probiert die Offloading-Funktionen im Treiber abzuschalten?

Nils

d3p · Dec 10, 2015

remote_mind said:
Achso, auffällig wäre noch das in vielen der Meldungen etwas auf die Netwerkkarte hinweist 'e1000'. Evtl. mal probiert die Offloading-Funktionen im Treiber abzuschalten?

Nein, noch nicht.
Meinst du den TSO?

Netcup empfiehlt diesen Treiber zwar, ich kann aber auch "virtio" oder "rtl8139"
auswählen.

d3p · Dec 10, 2015

Sorry für den Push.
Auch das brachte keine Besserung.

Ich wechsel jetzt mal den Treiber.

Code:

[Thu Dec 10 21:07:54 2015] INFO: rcu_sched detected stalls on CPUs/tasks: { 3} (detected by 0, t=5252 jiffies, g=1310969, c=1310968, q=2690)
[Thu Dec 10 21:07:54 2015] Task dump for CPU 3:
[Thu Dec 10 21:07:54 2015] mysqld          R  running task        0  2840   2407 0x00000000
[Thu Dec 10 21:07:54 2015]  ffff880331564980 ffffffff81542306 000000012f31bf58 ffff880331618c40
[Thu Dec 10 21:07:54 2015]  0000000000030001 ffff88032f31c000 00007f2137324b00 000000010111e4a4
[Thu Dec 10 21:07:54 2015]  ffff88032f31c000 00007f2137324b00 000000010111e4a4 ffffffff815424e3
[Thu Dec 10 21:07:54 2015] Call Trace:
[Thu Dec 10 21:07:54 2015]  [<ffffffff81542306>] ? __schedule+0x409/0x4f1
[Thu Dec 10 21:07:54 2015]  [<ffffffff815424e3>] ? schedule+0x6b/0x78
[Thu Dec 10 21:07:54 2015]  [<ffffffff8154552c>] ? retint_careful+0x14/0x32
[Thu Dec 10 21:07:54 2015] rcu_sched kthread starved for 5252 jiffies! g1310969 c1310968 f0x0
[Thu Dec 10 21:08:30 2015] INFO: rcu_sched detected stalls on CPUs/tasks: {} (detected by 0, t=6393 jiffies, g=1311002, c=1311001, q=974)
[Thu Dec 10 21:08:30 2015] All QSes seen, last rcu_sched kthread activity 6392 (4300498254-4300491862), jiffies_till_next_fqs=1, root ->qsmask 0x0
[Thu Dec 10 21:08:30 2015] mysqld          R  running task        0  7025   2407 0x00000008
[Thu Dec 10 21:08:30 2015]  ffffffff81a42480 ffff88033fc16cc0 ffffffff8108d6c1 0000000000000000
[Thu Dec 10 21:08:30 2015]  0000000000001155 0000000000000004 00000000000003ce 000000000014011a
[Thu Dec 10 21:08:30 2015]  0000000000000082 ffff8800370e8000 ffffffff81ad5f70 0000000000000000
[Thu Dec 10 21:08:30 2015] Call Trace:
[Thu Dec 10 21:08:30 2015]  <IRQ>  [<ffffffff8108d6c1>] ? rcu_check_callbacks+0x479/0x58b
[Thu Dec 10 21:08:30 2015]  [<ffffffff8109bdd0>] ? tick_nohz_handler+0x78/0x78
[Thu Dec 10 21:08:30 2015]  [<ffffffff8109084d>] ? update_process_times+0x24/0x47
[Thu Dec 10 21:08:30 2015]  [<ffffffff8109b77a>] ? tick_sched_handle+0x32/0x3d
[Thu Dec 10 21:08:30 2015]  [<ffffffff8109bdff>] ? tick_sched_timer+0x2f/0x55
[Thu Dec 10 21:08:30 2015]  [<ffffffff81090f61>] ? __hrtimer_run_queues+0xcf/0x1a4
[Thu Dec 10 21:08:30 2015]  [<ffffffff810913db>] ? hrtimer_interrupt+0x62/0x106
[Thu Dec 10 21:08:30 2015]  [<ffffffff81035d14>] ? smp_apic_timer_interrupt+0x25/0x34
[Thu Dec 10 21:08:30 2015]  [<ffffffff8154571b>] ? apic_timer_interrupt+0x6b/0x70
[Thu Dec 10 21:08:30 2015]  [<ffffffff8108eb33>] ? arch_local_irq_enable+0x7/0x8
[Thu Dec 10 21:08:30 2015]  [<ffffffff81090121>] ? run_timer_softirq+0x1db/0x1ea
[Thu Dec 10 21:08:30 2015]  [<ffffffff8104fa41>] ? __do_softirq+0xd6/0x22d
[Thu Dec 10 21:08:30 2015]  [<ffffffff81094a60>] ? ktime_get+0x37/0x46
[Thu Dec 10 21:08:30 2015]  [<ffffffff81047d48>] ? do_flush_tlb_all+0x36/0x36
[Thu Dec 10 21:08:30 2015]  [<ffffffff8104fcf5>] ? irq_exit+0x38/0x7c
[Thu Dec 10 21:08:30 2015]  [<ffffffff81035d19>] ? smp_apic_timer_interrupt+0x2a/0x34
[Thu Dec 10 21:08:30 2015]  [<ffffffff8154571b>] ? apic_timer_interrupt+0x6b/0x70
[Thu Dec 10 21:08:30 2015]  <EOI>  [<ffffffff8109f704>] ? csd_lock_wait+0x4/0xc
[Thu Dec 10 21:08:30 2015]  [<ffffffff8109fab3>] ? smp_call_function_single+0xb9/0xd7
[Thu Dec 10 21:08:30 2015]  [<ffffffff81047d48>] ? do_flush_tlb_all+0x36/0x36
[Thu Dec 10 21:08:30 2015]  [<ffffffff81047e05>] ? native_flush_tlb_others+0x29/0x2e
[Thu Dec 10 21:08:30 2015]  [<ffffffff812d543e>] ? cpumask_next+0x16/0x18
[Thu Dec 10 21:08:30 2015]  [<ffffffff81047b2d>] ? flush_tlb_others+0x5/0x8
[Thu Dec 10 21:08:30 2015]  [<ffffffff81047f30>] ? flush_tlb_mm_range+0xcb/0xd4
[Thu Dec 10 21:08:30 2015]  [<ffffffff81102a17>] ? tlb_flush_mmu_tlbonly+0x2b/0x68
[Thu Dec 10 21:08:30 2015]  [<ffffffff81102e18>] ? tlb_flush_mmu+0x9/0x12
[Thu Dec 10 21:08:30 2015]  [<ffffffff81102e2c>] ? tlb_finish_mmu+0xb/0x2f
[Thu Dec 10 21:08:30 2015]  [<ffffffff81104043>] ? zap_page_range+0x90/0xa2
[Thu Dec 10 21:08:30 2015]  [<ffffffff810eeac8>] ? get_page+0x9/0x21
[Thu Dec 10 21:08:30 2015]  [<ffffffff811294e9>] ? __vfs_write+0x7e/0xa6
[Thu Dec 10 21:08:30 2015]  [<ffffffff8110766b>] ? find_vma+0x11/0x5c
[Thu Dec 10 21:08:30 2015]  [<ffffffff811111c7>] ? SyS_madvise+0x42c/0x681
[Thu Dec 10 21:08:30 2015]  [<ffffffff8112b201>] ? __sb_end_write+0x24/0x52
[Thu Dec 10 21:08:30 2015]  [<ffffffff8112ac2b>] ? fput+0xd/0x81
[Thu Dec 10 21:08:30 2015]  [<ffffffff815448ee>] ? entry_SYSCALL_64_fastpath+0x12/0x71
[Thu Dec 10 21:08:30 2015] rcu_sched kthread starved for 6392 jiffies! g1311002 c1311001 f0x2
[Thu Dec 10 21:10:04 2015] INFO: rcu_sched detected stalls on CPUs/tasks: {} (detected by 0, t=23364 jiffies, g=1311032, c=1311031, q=1484)
[Thu Dec 10 21:10:04 2015] All QSes seen, last rcu_sched kthread activity 23337 (4300521704-4300498367), jiffies_till_next_fqs=1, root ->qsmask 0x0
[Thu Dec 10 21:10:04 2015] mysqld          R  running task        0  2840   2407 0x00000008
[Thu Dec 10 21:10:04 2015]  ffffffff81a42480 ffff88033fc16cc0 ffffffff8108d6c1 0000000000000000
[Thu Dec 10 21:10:04 2015]  0000000000000000 000000000000bb01 00000000000005cc 0000000000140138
[Thu Dec 10 21:10:04 2015]  0000000000000082 ffff8800372b55c0 ffffffff81ad5f70 0000000000000000
[Thu Dec 10 21:10:04 2015] Call Trace:
[Thu Dec 10 21:10:04 2015]  <IRQ>  [<ffffffff8108d6c1>] ? rcu_check_callbacks+0x479/0x58b
[Thu Dec 10 21:10:04 2015]  [<ffffffff8109bdd0>] ? tick_nohz_handler+0x78/0x78
[Thu Dec 10 21:10:04 2015]  [<ffffffff8109084d>] ? update_process_times+0x24/0x47
[Thu Dec 10 21:10:04 2015]  [<ffffffff8109b77a>] ? tick_sched_handle+0x32/0x3d
[Thu Dec 10 21:10:04 2015]  [<ffffffff8109bdff>] ? tick_sched_timer+0x2f/0x55
[Thu Dec 10 21:10:04 2015]  [<ffffffff81090f61>] ? __hrtimer_run_queues+0xcf/0x1a4
[Thu Dec 10 21:10:04 2015]  [<ffffffff810913db>] ? hrtimer_interrupt+0x62/0x106
[Thu Dec 10 21:10:04 2015]  [<ffffffff81035d14>] ? smp_apic_timer_interrupt+0x25/0x34
[Thu Dec 10 21:10:04 2015]  [<ffffffff8154571b>] ? apic_timer_interrupt+0x6b/0x70
[Thu Dec 10 21:10:04 2015]  <EOI>  [<ffffffff81545526>] ? retint_careful+0xe/0x32
[Thu Dec 10 21:10:04 2015] rcu_sched kthread starved for 23337 jiffies! g1311032 c1311031 f0x2
[Thu Dec 10 21:10:11 2015] INFO: rcu_sched detected stalls on CPUs/tasks: { 3} (detected by 0, t=10228 jiffies, g=1311033, c=1311032, q=1664)
[Thu Dec 10 21:10:11 2015] Task dump for CPU 3:
[Thu Dec 10 21:10:11 2015] mysqld          R  running task        0  2836   2407 0x00000008
[Thu Dec 10 21:10:11 2015]  ffff8803315d8000 ffffffff81542306 ffffffff81544543 ffff8803315d0c40
[Thu Dec 10 21:10:11 2015]  ffff88032f3a8c40 ffff8800b85c0000 ffff8800b85bfd28 ffff8800b85bfcc0
[Thu Dec 10 21:10:11 2015]  ffff8800b85c0000 ffff8800b85bfd28 ffff8800b85bfcc0 ffffffff81090dfb
[Thu Dec 10 21:10:11 2015] Call Trace:
[Thu Dec 10 21:10:11 2015]  [<ffffffff81542306>] ? __schedule+0x409/0x4f1
[Thu Dec 10 21:10:11 2015]  [<ffffffff81544543>] ? _raw_spin_unlock_irqrestore+0xb/0xc
[Thu Dec 10 21:10:11 2015]  [<ffffffff81090dfb>] ? hrtimer_try_to_cancel+0x12/0x93
[Thu Dec 10 21:10:11 2015]  [<ffffffff8109cefe>] ? futex_wait_queue_me+0x80/0xbe
[Thu Dec 10 21:10:11 2015]  [<ffffffff81090e88>] ? hrtimer_cancel+0xc/0x16
[Thu Dec 10 21:10:11 2015]  [<ffffffff8109d8d5>] ? futex_wait+0x1cc/0x1e7
[Thu Dec 10 21:10:11 2015]  [<ffffffff8109d0c6>] ? get_futex_key+0x78/0x1cc
[Thu Dec 10 21:10:11 2015]  [<ffffffff81090a52>] ? enqueue_hrtimer+0x69/0x69
[Thu Dec 10 21:10:11 2015]  [<ffffffff8109ced3>] ? futex_wait_queue_me+0x55/0xbe
[Thu Dec 10 21:10:11 2015]  [<ffffffff8109d328>] ? futex_wake+0x10e/0x126
[Thu Dec 10 21:10:11 2015]  [<ffffffff8109ea87>] ? do_futex+0xd6/0x7fa
[Thu Dec 10 21:10:11 2015]  [<ffffffff8106ec29>] ? update_entity_load_avg+0x1ab/0x318
[Thu Dec 10 21:10:11 2015]  [<ffffffff8106edb7>] ? set_next_entity+0x21/0x43
[Thu Dec 10 21:10:11 2015]  [<ffffffff8107198a>] ? pick_next_task_fair+0x138/0x31b
[Thu Dec 10 21:10:11 2015]  [<ffffffff8100c01d>] ? paravirt_write_msr+0xb/0xe
[Thu Dec 10 21:10:11 2015]  [<ffffffff8100c889>] ? __switch_to+0x3b3/0x41a
[Thu Dec 10 21:10:11 2015]  [<ffffffff8108e886>] ? SyS_time+0xb/0x2f
[Thu Dec 10 21:10:11 2015]  [<ffffffff810038d0>] ? emulate_vsyscall+0x228/0x2ef
[Thu Dec 10 21:10:11 2015]  [<ffffffff81042cfa>] ? __bad_area_nosemaphore+0xaa/0x1eb
[Thu Dec 10 21:10:11 2015]  [<ffffffff815467c8>] ? async_page_fault+0x28/0x30
[Thu Dec 10 21:10:11 2015] rcu_sched kthread starved for 8797 jiffies! g1311033 c1311032 f0x0
[Thu Dec 10 21:12:07 2015] INFO: rcu_sched self-detected stall on CPU
[Thu Dec 10 21:12:07 2015] INFO: rcu_sched detected stalls on CPUs/tasks: { 0} (detected by 3, t=9787 jiffies, g=1311041, c=1311040, q=2169)
[Thu Dec 10 21:12:07 2015] Task dump for CPU 0:
[Thu Dec 10 21:12:07 2015] mysqld          R  running task        0  2840   2407 0x00000008
[Thu Dec 10 21:12:07 2015]  000000000000bb3d 0000000000000005 0000000000000005 ffffffff81c25000
[Thu Dec 10 21:12:07 2015]  0000000000000046 ffff88032f318000 0000000000000008 ffff88032f31bf78
[Thu Dec 10 21:12:07 2015]  000000000000fe2e ffffffffffffff10 ffffffff81545526 0000000000000010
[Thu Dec 10 21:12:07 2015] Call Trace:
[Thu Dec 10 21:12:07 2015]  [<ffffffff81545526>] ? retint_careful+0xe/0x32
[Thu Dec 10 21:12:07 2015]  [<ffffffff815448ee>] ? entry_SYSCALL_64_fastpath+0x12/0x71
[Thu Dec 10 21:12:07 2015] rcu_sched kthread starved for 9787 jiffies! g1311041 c1311040 f0x0
[Thu Dec 10 21:12:07 2015]  { 0}  (t=9787 jiffies g=1311041 c=1311040 q=2206)
[Thu Dec 10 21:12:07 2015] rcu_sched kthread starved for 9787 jiffies! g1311041 c1311040 f0x2
[Thu Dec 10 21:12:07 2015] Task dump for CPU 0:
[Thu Dec 10 21:12:07 2015] mysqld          R  running task        0  2840   2407 0x00000008
[Thu Dec 10 21:12:07 2015]  ffffffff81a42480 0000000000000000 ffffffff8108b0c4 ffff88033fc16cc0
[Thu Dec 10 21:12:07 2015]  ffffffff81a42480 ffff88033fc16cc0 ffff88032f31bea8 0000000000000000
[Thu Dec 10 21:12:07 2015]  ffffffff8108d428 ffffffff81544543 ffff88033fc03ea0 000000000000bb3d
[Thu Dec 10 21:12:07 2015] Call Trace:
[Thu Dec 10 21:12:07 2015]  <IRQ>  [<ffffffff8108b0c4>] ? rcu_dump_cpu_stacks+0x55/0x87
[Thu Dec 10 21:12:07 2015]  [<ffffffff8108d428>] ? rcu_check_callbacks+0x1e0/0x58b
[Thu Dec 10 21:12:07 2015]  [<ffffffff81544543>] ? _raw_spin_unlock_irqrestore+0xb/0xc
[Thu Dec 10 21:12:07 2015]  [<ffffffff8109bdd0>] ? tick_nohz_handler+0x78/0x78
[Thu Dec 10 21:12:07 2015]  [<ffffffff8109084d>] ? update_process_times+0x24/0x47
[Thu Dec 10 21:12:07 2015]  [<ffffffff8109b77a>] ? tick_sched_handle+0x32/0x3d
[Thu Dec 10 21:12:07 2015]  [<ffffffff8109bdff>] ? tick_sched_timer+0x2f/0x55
[Thu Dec 10 21:12:07 2015]  [<ffffffff81090f61>] ? __hrtimer_run_queues+0xcf/0x1a4
[Thu Dec 10 21:12:07 2015]  [<ffffffff810913db>] ? hrtimer_interrupt+0x62/0x106
[Thu Dec 10 21:12:07 2015]  [<ffffffff81035d14>] ? smp_apic_timer_interrupt+0x25/0x34
[Thu Dec 10 21:12:07 2015]  [<ffffffff8154571b>] ? apic_timer_interrupt+0x6b/0x70
[Thu Dec 10 21:12:07 2015]  <EOI>  [<ffffffff81545526>] ? retint_careful+0xe/0x32
[Thu Dec 10 21:12:07 2015]  [<ffffffff815448ee>] ? entry_SYSCALL_64_fastpath+0x12/0x71
[Thu Dec 10 21:12:07 2015] INFO: rcu_sched self-detected stall on CPU { 0}  (t=20219 jiffies g=1311056 c=1311055 q=1207)
[Thu Dec 10 21:12:07 2015] rcu_sched kthread starved for 20219 jiffies! g1311056 c1311055 f0x0
[Thu Dec 10 21:13:28 2015] INFO: rcu_sched detected stalls on CPUs/tasks: { 0} (detected by 2, t=20219 jiffies, g=1311056, c=1311055, q=1207)
[Thu Dec 10 21:13:28 2015] Task dump for CPU 0:
[Thu Dec 10 21:13:28 2015] mysqld          R  running task        0  7207   2407 0x00000008
[Thu Dec 10 21:13:28 2015]  0000000000000180 0000000000000180 ffffffff81c25000 0000000000000046
[Thu Dec 10 21:13:28 2015]  ffff88027a1a4000 0000000000000008 ffff88027a1a7f78 000000000000fe2e
[Thu Dec 10 21:13:28 2015]  ffffffffffffff10 ffffffff81545526 0000000000000010 0000000000000203
[Thu Dec 10 21:13:28 2015] Call Trace:
[Thu Dec 10 21:13:28 2015]  [<ffffffff81545526>] ? retint_careful+0xe/0x32
[Thu Dec 10 21:13:28 2015] rcu_sched kthread starved for 20219 jiffies! g1311056 c1311055 f0x0
[Thu Dec 10 21:13:28 2015] Task dump for CPU 0:
[Thu Dec 10 21:13:28 2015] mysqld          R  running task        0  7207   2407 0x00000008
[Thu Dec 10 21:13:28 2015]  ffffffff81a42480 0000000000000000 ffffffff8108b0c4 ffff88033fc16cc0
[Thu Dec 10 21:13:28 2015]  ffffffff81a42480 ffff88033fc16cc0 ffff88027a1a7ea8 0000000000000000
[Thu Dec 10 21:13:28 2015]  ffffffff8108d428 ffff88032f3e5628 0000000000000000 000000000000bd03
[Thu Dec 10 21:13:28 2015] Call Trace:
[Thu Dec 10 21:13:28 2015]  <IRQ>  [<ffffffff8108b0c4>] ? rcu_dump_cpu_stacks+0x55/0x87
[Thu Dec 10 21:13:28 2015]  [<ffffffff8108d428>] ? rcu_check_callbacks+0x1e0/0x58b
[Thu Dec 10 21:13:28 2015]  [<ffffffff8109bdd0>] ? tick_nohz_handler+0x78/0x78
[Thu Dec 10 21:13:28 2015]  [<ffffffff8109084d>] ? update_process_times+0x24/0x47
[Thu Dec 10 21:13:28 2015]  [<ffffffff8109b77a>] ? tick_sched_handle+0x32/0x3d
[Thu Dec 10 21:13:28 2015]  [<ffffffff8109bdff>] ? tick_sched_timer+0x2f/0x55
[Thu Dec 10 21:13:28 2015]  [<ffffffff81090f61>] ? __hrtimer_run_queues+0xcf/0x1a4
[Thu Dec 10 21:13:28 2015]  [<ffffffff810913db>] ? hrtimer_interrupt+0x62/0x106
[Thu Dec 10 21:13:28 2015]  [<ffffffff81035d14>] ? smp_apic_timer_interrupt+0x25/0x34
[Thu Dec 10 21:13:28 2015]  [<ffffffff8154571b>] ? apic_timer_interrupt+0x6b/0x70
[Thu Dec 10 21:13:28 2015]  <EOI>  [<ffffffff81545526>] ? retint_careful+0xe/0x32
[Thu Dec 10 21:15:34 2015] INFO: rcu_sched detected stalls on CPUs/tasks: { 1} (detected by 0, t=7957 jiffies, g=1316514, c=1316513, q=9)
[Thu Dec 10 21:15:34 2015] Task dump for CPU 1:
[Thu Dec 10 21:15:34 2015] php5-fpm        R  running task        0  7143   3400 0x00000008
[Thu Dec 10 21:15:34 2015]  ffff880036c6aa80 ffff880331663220 0000000000000000 0000000800000000
[Thu Dec 10 21:15:34 2015]  ffff8800adbb7d98 ffff8800adbb7d98 ffffffff81140f17 ffff880331663220
[Thu Dec 10 21:15:34 2015]  0000000000000001 ffffffff81140f4c 8080808080808080 ffff8800adbb7d98
[Thu Dec 10 21:15:34 2015] Call Trace:
[Thu Dec 10 21:15:34 2015]  [<ffffffff81140f17>] ? __legitimize_mnt+0x26/0x4d
[Thu Dec 10 21:15:34 2015]  [<ffffffff81140f4c>] ? legitimize_mnt+0xe/0x27
[Thu Dec 10 21:15:34 2015]  [<ffffffff8113184b>] ? unlazy_walk+0x6d/0x124
[Thu Dec 10 21:15:34 2015]  [<ffffffff81131050>] ? path_put+0xd/0x16
[Thu Dec 10 21:15:34 2015]  [<ffffffff8104fc16>] ? __local_bh_enable_ip+0x16/0x75
[Thu Dec 10 21:15:34 2015]  [<ffffffff814ce4d4>] ? inet_compute_pseudo+0x32/0x32
[Thu Dec 10 21:15:34 2015]  [<ffffffff8146d866>] ? sock_i_uid+0x31/0x37
[Thu Dec 10 21:15:34 2015]  [<ffffffff8104fbf6>] ? do_softirq+0x2c/0x36
[Thu Dec 10 21:15:34 2015]  [<ffffffff8104fc5f>] ? __local_bh_enable_ip+0x5f/0x75
[Thu Dec 10 21:15:34 2015]  [<ffffffff814cf92e>] ? udp_lib_get_port+0x2cc/0x2e1
[Thu Dec 10 21:15:34 2015]  [<ffffffff814ce4d4>] ? inet_compute_pseudo+0x32/0x32
[Thu Dec 10 21:15:34 2015]  [<ffffffff8146da28>] ? sk_prot_alloc+0xa7/0xe4
[Thu Dec 10 21:15:34 2015]  [<ffffffff814d8859>] ? inet_autobind+0x1f/0x4b
[Thu Dec 10 21:15:34 2015]  [<ffffffff814d88d6>] ? inet_dgram_connect+0x51/0x6f
[Thu Dec 10 21:15:34 2015]  [<ffffffff8146cb49>] ? SYSC_connect+0x6d/0x8f
[Thu Dec 10 21:15:34 2015]  [<ffffffff8146b5d5>] ? sock_alloc_file+0xb7/0x10a
[Thu Dec 10 21:15:34 2015]  [<ffffffff8146b845>] ? SYSC_socket+0x5e/0x8e
[Thu Dec 10 21:15:34 2015]  [<ffffffff815448ee>] ? entry_SYSCALL_64_fastpath+0x12/0x71
[Thu Dec 10 21:15:34 2015] rcu_sched kthread starved for 7926 jiffies! g1316514 c1316513 f0x0
[Thu Dec 10 21:16:12 2015] INFO: rcu_sched detected stalls on CPUs/tasks: { 3} (detected by 1, t=7894 jiffies, g=1316970, c=1316969, q=66)
[Thu Dec 10 21:16:12 2015] Task dump for CPU 3:
[Thu Dec 10 21:16:12 2015] nginx           R  running task        0  2400   2383 0x00000008
[Thu Dec 10 21:16:12 2015]  0000000000008145 000000000184db73 00007fadf67956a8 ffffffff814d9275
[Thu Dec 10 21:16:12 2015]  000000000184db73 0000000000008145 0000000000000000 000000002ee9be80
[Thu Dec 10 21:16:12 2015]  ffff88032ee9be80 ffff88032ee9bf20 ffff88032ee9bf20 ffffffff8146a252
[Thu Dec 10 21:16:12 2015] Call Trace:
[Thu Dec 10 21:16:12 2015]  [<ffffffff814d9275>] ? inet_recvmsg+0x4e/0x62
[Thu Dec 10 21:16:12 2015]  [<ffffffff8146a252>] ? sock_read_iter+0x63/0x83
[Thu Dec 10 21:16:12 2015]  [<ffffffff8112958c>] ? __vfs_read+0x7b/0x9e
[Thu Dec 10 21:16:12 2015]  [<ffffffff8112a2b6>] ? vfs_read+0x70/0xc2
[Thu Dec 10 21:16:12 2015]  [<ffffffff810038d0>] ? emulate_vsyscall+0x228/0x2ef
[Thu Dec 10 21:16:12 2015]  [<ffffffff8112a359>] ? SyS_read+0x51/0x85
[Thu Dec 10 21:16:12 2015]  [<ffffffff815448ee>] ? entry_SYSCALL_64_fastpath+0x12/0x71
[Thu Dec 10 21:16:12 2015] rcu_sched kthread starved for 7511 jiffies! g1316970 c1316969 f0x0
[Thu Dec 10 21:16:52 2015] INFO: rcu_sched detected stalls on CPUs/tasks: {} (detected by 1, t=9082 jiffies, g=1317309, c=1317308, q=678)
[Thu Dec 10 21:17:22 2015] All QSes seen, last rcu_sched kthread activity 8876 (4300631178-4300622302), jiffies_till_next_fqs=1, root ->qsmask 0x0
[Thu Dec 10 21:17:22 2015] php5-fpm        R  running task        0  7243   3400 0x00000008
[Thu Dec 10 21:17:22 2015]  ffffffff81a42480 ffff88033fc96cc0 ffffffff8108d6c1 0000000000000000
[Thu Dec 10 21:17:22 2015]  ffff880277f44f8c 0000000000000001 00000000000002a6 00000000001419bd
[Thu Dec 10 21:17:22 2015]  0000000000000082 ffff880277f44980 ffffffff81ad5f70 0000000000000000
[Thu Dec 10 21:17:22 2015] Call Trace:
[Thu Dec 10 21:17:22 2015]  <IRQ>  [<ffffffff8108d6c1>] ? rcu_check_callbacks+0x479/0x58b
[Thu Dec 10 21:17:22 2015]  [<ffffffff8109bdd0>] ? tick_nohz_handler+0x78/0x78
[Thu Dec 10 21:17:22 2015]  [<ffffffff8109084d>] ? update_process_times+0x24/0x47
[Thu Dec 10 21:17:22 2015]  [<ffffffff8109b77a>] ? tick_sched_handle+0x32/0x3d
[Thu Dec 10 21:17:22 2015]  [<ffffffff8109bdff>] ? tick_sched_timer+0x2f/0x55
[Thu Dec 10 21:17:22 2015]  [<ffffffff81090f61>] ? __hrtimer_run_queues+0xcf/0x1a4
[Thu Dec 10 21:17:22 2015]  [<ffffffff810913db>] ? hrtimer_interrupt+0x62/0x106
[Thu Dec 10 21:17:22 2015]  [<ffffffff81035d14>] ? smp_apic_timer_interrupt+0x25/0x34
[Thu Dec 10 21:17:22 2015]  [<ffffffff8154571b>] ? apic_timer_interrupt+0x6b/0x70
[Thu Dec 10 21:17:22 2015]  <EOI>  [<ffffffff8104fc16>] ? __local_bh_enable_ip+0x16/0x75
[Thu Dec 10 21:17:22 2015]  [<ffffffff814ce4d4>] ? inet_compute_pseudo+0x32/0x32
[Thu Dec 10 21:17:22 2015]  [<ffffffff8146d866>] ? sock_i_uid+0x31/0x37
[Thu Dec 10 21:17:22 2015]  [<ffffffff814cf79c>] ? udp_lib_get_port+0x13a/0x2e1
[Thu Dec 10 21:17:22 2015]  [<ffffffff814cf753>] ? udp_lib_get_port+0xf1/0x2e1
[Thu Dec 10 21:17:22 2015]  [<ffffffff814ce4d4>] ? inet_compute_pseudo+0x32/0x32
[Thu Dec 10 21:17:22 2015]  [<ffffffff8146da28>] ? sk_prot_alloc+0xa7/0xe4
[Thu Dec 10 21:17:22 2015]  [<ffffffff814d8859>] ? inet_autobind+0x1f/0x4b
[Thu Dec 10 21:17:22 2015]  [<ffffffff814d88d6>] ? inet_dgram_connect+0x51/0x6f
[Thu Dec 10 21:17:22 2015]  [<ffffffff8146cb49>] ? SYSC_connect+0x6d/0x8f
[Thu Dec 10 21:17:22 2015]  [<ffffffff8146b5d5>] ? sock_alloc_file+0xb7/0x10a
[Thu Dec 10 21:17:22 2015]  [<ffffffff8146b845>] ? SYSC_socket+0x5e/0x8e
[Thu Dec 10 21:17:22 2015]  [<ffffffff815448ee>] ? entry_SYSCALL_64_fastpath+0x12/0x71
[Thu Dec 10 21:17:22 2015] rcu_sched kthread starved for 8876 jiffies! g1317309 c1317308 f0x2
[Thu Dec 10 21:17:47 2015] INFO: rcu_sched detected stalls on CPUs/tasks: {} (detected by 2, t=6200 jiffies, g=1317329, c=1317328, q=2110)
[Thu Dec 10 21:17:47 2015] All QSes seen, last rcu_sched kthread activity 6199 (4300637439-4300631240), jiffies_till_next_fqs=1, root ->qsmask 0x0
[Thu Dec 10 21:17:47 2015] nginx           R  running task        0  2401   2383 0x00000000
[Thu Dec 10 21:17:47 2015]  ffffffff81a42480 ffff88033fd16cc0 ffffffff8108d6c1 0000000000000000
[Thu Dec 10 21:17:47 2015]  0000000000000086 ffffffff81069aba 000000000000083e 00000000001419d1
[Thu Dec 10 21:17:47 2015]  0000000000000082 ffff88032f3ad5c0 ffffffff81ad5f70 0000000000000000
[Thu Dec 10 21:17:47 2015] Call Trace:
[Thu Dec 10 21:17:47 2015]  <IRQ>  [<ffffffff8108d6c1>] ? rcu_check_callbacks+0x479/0x58b
[Thu Dec 10 21:17:47 2015]  [<ffffffff81069aba>] ? sched_ttwu_pending+0x3c/0x55
[Thu Dec 10 21:17:47 2015]  [<ffffffff8109bdd0>] ? tick_nohz_handler+0x78/0x78
[Thu Dec 10 21:17:47 2015]  [<ffffffff8109084d>] ? update_process_times+0x24/0x47
[Thu Dec 10 21:17:47 2015]  [<ffffffff8109b77a>] ? tick_sched_handle+0x32/0x3d
[Thu Dec 10 21:17:47 2015]  [<ffffffff8109bdff>] ? tick_sched_timer+0x2f/0x55
[Thu Dec 10 21:17:47 2015]  [<ffffffff81090f61>] ? __hrtimer_run_queues+0xcf/0x1a4
[Thu Dec 10 21:17:47 2015]  [<ffffffff810913db>] ? hrtimer_interrupt+0x62/0x106
[Thu Dec 10 21:17:47 2015]  [<ffffffff81035d14>] ? smp_apic_timer_interrupt+0x25/0x34
[Thu Dec 10 21:17:47 2015]  [<ffffffff8154571b>] ? apic_timer_interrupt+0x6b/0x70
[Thu Dec 10 21:17:47 2015]  <EOI>  [<ffffffff8106584b>] ? arch_local_irq_enable+0x7/0x8
[Thu Dec 10 21:17:47 2015]  [<ffffffff81068282>] ? finish_task_switch+0x9d/0x141
[Thu Dec 10 21:17:47 2015]  [<ffffffff81542306>] ? __schedule+0x409/0x4f1
[Thu Dec 10 21:17:47 2015]  [<ffffffff815424e3>] ? schedule+0x6b/0x78
[Thu Dec 10 21:17:47 2015]  [<ffffffff8154411e>] ? schedule_hrtimeout_range_clock+0x8f/0xbf
[Thu Dec 10 21:17:47 2015]  [<ffffffff81090a52>] ? enqueue_hrtimer+0x69/0x69
[Thu Dec 10 21:17:47 2015]  [<ffffffff810912a5>] ? hrtimer_start_expires+0x14/0x17
[Thu Dec 10 21:17:47 2015]  [<ffffffff81158bc8>] ? ep_poll+0x12a/0x263
[Thu Dec 10 21:17:47 2015]  [<ffffffff81121791>] ? kmem_cache_free+0x81/0xd7
[Thu Dec 10 21:17:47 2015]  [<ffffffff81069f30>] ? try_to_wake_up+0x1af/0x1af
[Thu Dec 10 21:17:47 2015]  [<ffffffff81158e9a>] ? SYSC_epoll_wait+0x83/0xb3
[Thu Dec 10 21:17:47 2015]  [<ffffffff815448ee>] ? entry_SYSCALL_64_fastpath+0x12/0x71
[Thu Dec 10 21:17:47 2015] rcu_sched kthread starved for 6199 jiffies! g1317329 c1317328 f0x2
[Thu Dec 10 21:17:48 2015] INFO: rcu_sched self-detected stall on CPU { 3}  (t=8378 jiffies g=1317357 c=1317356 q=2792)
[Thu Dec 10 21:17:48 2015] INFO: rcu_sched detected stalls on CPUs/tasks: { 3} (detected by 1, t=8378 jiffies, g=1317357, c=1317356, q=2792)
[Thu Dec 10 21:17:48 2015] Task dump for CPU 3:
[Thu Dec 10 21:17:48 2015] mysqld          R  running task        0  2840   2407 0x00000008
[Thu Dec 10 21:17:48 2015]  0000000000000000 0000000000000000 ffffffff81c25000 0000000000000046
[Thu Dec 10 21:17:48 2015]  ffff88032f318000 0000000000000008 ffff88032f31bf78 000000000000fe2e
[Thu Dec 10 21:17:48 2015]  ffffffffffffff10 ffffffff81545526 0000000000000010 0000000000000203
[Thu Dec 10 21:17:48 2015] Call Trace:
[Thu Dec 10 21:17:48 2015]  [<ffffffff81545526>] ? retint_careful+0xe/0x32
[Thu Dec 10 21:17:48 2015] rcu_sched kthread starved for 8377 jiffies! g1317357 c1317356 f0x0
[Thu Dec 10 21:17:48 2015] rcu_sched kthread starved for 8377 jiffies! g1317357 c1317356 f0x2
[Thu Dec 10 21:17:48 2015] Task dump for CPU 3:
[Thu Dec 10 21:17:48 2015] mysqld          R  running task        0  2840   2407 0x00000008
[Thu Dec 10 21:17:48 2015]  ffffffff81a42480 0000000000000003 ffffffff8108b0c4 ffff88033fd96cc0
[Thu Dec 10 21:17:48 2015]  ffffffff81a42480 ffff88033fd96cc0 ffff88032f31bea8 0000000000000000
[Thu Dec 10 21:17:48 2015]  ffffffff8108d428 ffffffff81a42480 0000000180400038 000000000000b7ca
[Thu Dec 10 21:17:48 2015] Call Trace:
[Thu Dec 10 21:17:48 2015]  <IRQ>  [<ffffffff8108b0c4>] ? rcu_dump_cpu_stacks+0x55/0x87
[Thu Dec 10 21:17:48 2015]  [<ffffffff8108d428>] ? rcu_check_callbacks+0x1e0/0x58b
[Thu Dec 10 21:17:48 2015]  [<ffffffff8109bdd0>] ? tick_nohz_handler+0x78/0x78
[Thu Dec 10 21:17:48 2015]  [<ffffffff8109084d>] ? update_process_times+0x24/0x47
[Thu Dec 10 21:17:48 2015]  [<ffffffff8109b77a>] ? tick_sched_handle+0x32/0x3d
[Thu Dec 10 21:17:48 2015]  [<ffffffff8109bdff>] ? tick_sched_timer+0x2f/0x55
[Thu Dec 10 21:17:48 2015]  [<ffffffff81090f61>] ? __hrtimer_run_queues+0xcf/0x1a4
[Thu Dec 10 21:17:48 2015]  [<ffffffff810913db>] ? hrtimer_interrupt+0x62/0x106
[Thu Dec 10 21:17:48 2015]  [<ffffffff81035d14>] ? smp_apic_timer_interrupt+0x25/0x34
[Thu Dec 10 21:17:48 2015]  [<ffffffff8154571b>] ? apic_timer_interrupt+0x6b/0x70
[Thu Dec 10 21:17:48 2015]  <EOI>  [<ffffffff81545526>] ? retint_careful+0xe/0x32
[Thu Dec 10 21:18:21 2015] INFO: rcu_sched detected stalls on CPUs/tasks: {} (detected by 3, t=7022 jiffies, g=1317364, c=1317363, q=700)
[Thu Dec 10 21:18:21 2015] All QSes seen, last rcu_sched kthread activity 7017 (4300652972-4300645955), jiffies_till_next_fqs=1, root ->qsmask 0x0
[Thu Dec 10 21:18:21 2015] mysqld          R  running task        0  2840   2407 0x00000008
[Thu Dec 10 21:18:21 2015]  ffffffff81a42480 ffff88033fd96cc0 ffffffff8108d6c1 0000000000000000
[Thu Dec 10 21:18:21 2015]  000002008106e98c 000000000000ba3c 00000000000002bc 00000000001419f4
[Thu Dec 10 21:18:21 2015]  0000000000000082 ffff8800372b55c0 ffffffff81ad5f70 0000000000000000
[Thu Dec 10 21:18:21 2015] Call Trace:
[Thu Dec 10 21:18:21 2015]  <IRQ>  [<ffffffff8108d6c1>] ? rcu_check_callbacks+0x479/0x58b
[Thu Dec 10 21:18:21 2015]  [<ffffffff8109bdd0>] ? tick_nohz_handler+0x78/0x78
[Thu Dec 10 21:18:21 2015]  [<ffffffff8109084d>] ? update_process_times+0x24/0x47
[Thu Dec 10 21:18:21 2015]  [<ffffffff8109b77a>] ? tick_sched_handle+0x32/0x3d
[Thu Dec 10 21:18:21 2015]  [<ffffffff8109bdff>] ? tick_sched_timer+0x2f/0x55
[Thu Dec 10 21:18:21 2015]  [<ffffffff81090f61>] ? __hrtimer_run_queues+0xcf/0x1a4
[Thu Dec 10 21:18:21 2015]  [<ffffffff810913db>] ? hrtimer_interrupt+0x62/0x106
[Thu Dec 10 21:18:21 2015]  [<ffffffff81035d14>] ? smp_apic_timer_interrupt+0x25/0x34
[Thu Dec 10 21:18:21 2015]  [<ffffffff8154571b>] ? apic_timer_interrupt+0x6b/0x70
[Thu Dec 10 21:18:21 2015]  <EOI>  [<ffffffff81545526>] ? retint_careful+0xe/0x32
[Thu Dec 10 21:18:21 2015] rcu_sched kthread starved for 7017 jiffies! g1317364 c1317363 f0x2
[Thu Dec 10 21:18:49 2015] INFO: rcu_sched detected stalls on CPUs/tasks: { 1} (detected by 3, t=5658 jiffies, g=1317385, c=1317384, q=1463)
[Thu Dec 10 21:18:49 2015] Task dump for CPU 1:
[Thu Dec 10 21:18:49 2015] mysqld          R  running task        0  7312   2407 0x00000008
[Thu Dec 10 21:18:49 2015]  0000000000004000 0000000001704000 ffffffff8112a473 ffff88032ecf3d40
[Thu Dec 10 21:18:49 2015]  00007f2142a68000 ffff88032ec60400 00007f2100000001 0000000003569e40
[Thu Dec 10 21:18:49 2015]  0000000001704000 ffff88032ec60400 00007f2100000001 00007f213febb380
[Thu Dec 10 21:18:49 2015] Call Trace:
[Thu Dec 10 21:18:49 2015]  [<ffffffff8112a473>] ? SyS_pread64+0x61/0x87
[Thu Dec 10 21:18:49 2015]  [<ffffffff815448ee>] ? entry_SYSCALL_64_fastpath+0x12/0x71
[Thu Dec 10 21:18:49 2015] rcu_sched kthread starved for 5643 jiffies! g1317385 c1317384 f0x0
[Thu Dec 10 21:19:34 2015] INFO: rcu_sched self-detected stall on CPU { 3}  (t=5250 jiffies g=1317423 c=1317422 q=488)
[Thu Dec 10 21:19:34 2015] rcu_sched kthread starved for 5190 jiffies! g1317423 c1317422 f0x0
[Thu Dec 10 21:19:34 2015] Task dump for CPU 2:
[Thu Dec 10 21:19:34 2015] mysqld          R  running task        0  7344   2407 0x00000008
[Thu Dec 10 21:19:34 2015]  0000000000000000 0000000000000001 000000013a1a4be0 ffff88032f11bd40
[Thu Dec 10 21:19:34 2015]  0000000000000000 ffff8803305b4000 00007f2128000020 00007f21280007b8
[Thu Dec 10 21:19:34 2015]  00007f2128068920 00007f2128000020 0000000000003fc0 0000000000000410
[Thu Dec 10 21:19:34 2015] Call Trace:
[Thu Dec 10 21:19:34 2015]  [<ffffffff815448ee>] ? entry_SYSCALL_64_fastpath+0x12/0x71
[Thu Dec 10 21:19:34 2015] Task dump for CPU 3:
[Thu Dec 10 21:19:34 2015] mysqld          R  running task        0  7361   2407 0x00000008
[Thu Dec 10 21:19:34 2015]  ffffffff81a42480 0000000000000003 ffffffff8108b0c4 ffff88033fd96cc0
[Thu Dec 10 21:19:34 2015]  ffffffff81a42480 ffff88033fd96cc0 ffff88032fe8bc18 0000000000000000
[Thu Dec 10 21:19:34 2015]  ffffffff8108d428 ffff88033fd961e8 000000008106e98c 0000000000000096
[Thu Dec 10 21:19:34 2015] Call Trace:
[Thu Dec 10 21:19:34 2015]  <IRQ>  [<ffffffff8108b0c4>] ? rcu_dump_cpu_stacks+0x55/0x87
[Thu Dec 10 21:19:34 2015]  [<ffffffff8108d428>] ? rcu_check_callbacks+0x1e0/0x58b
[Thu Dec 10 21:19:34 2015]  [<ffffffff81047d48>] ? do_flush_tlb_all+0x36/0x36
[Thu Dec 10 21:19:34 2015]  [<ffffffff8109bdd0>] ? tick_nohz_handler+0x78/0x78
[Thu Dec 10 21:19:34 2015]  [<ffffffff8109084d>] ? update_process_times+0x24/0x47
[Thu Dec 10 21:19:34 2015]  [<ffffffff8109b77a>] ? tick_sched_handle+0x32/0x3d
[Thu Dec 10 21:19:34 2015]  [<ffffffff8109bdff>] ? tick_sched_timer+0x2f/0x55
[Thu Dec 10 21:19:34 2015]  [<ffffffff81090f61>] ? __hrtimer_run_queues+0xcf/0x1a4
[Thu Dec 10 21:19:34 2015]  [<ffffffff81047d48>] ? do_flush_tlb_all+0x36/0x36
[Thu Dec 10 21:19:34 2015]  [<ffffffff810913db>] ? hrtimer_interrupt+0x62/0x106
[Thu Dec 10 21:19:34 2015]  [<ffffffff81035d14>] ? smp_apic_timer_interrupt+0x25/0x34
[Thu Dec 10 21:19:34 2015]  [<ffffffff8154571b>] ? apic_timer_interrupt+0x6b/0x70
[Thu Dec 10 21:19:34 2015]  <EOI>  [<ffffffff8109f704>] ? csd_lock_wait+0x4/0xc
[Thu Dec 10 21:19:34 2015]  [<ffffffff8109fab3>] ? smp_call_function_single+0xb9/0xd7
[Thu Dec 10 21:19:34 2015]  [<ffffffff81047d48>] ? do_flush_tlb_all+0x36/0x36
[Thu Dec 10 21:19:34 2015]  [<ffffffff81047e05>] ? native_flush_tlb_others+0x29/0x2e
[Thu Dec 10 21:19:34 2015]  [<ffffffff812d543e>] ? cpumask_next+0x16/0x18
[Thu Dec 10 21:19:34 2015]  [<ffffffff81047b2d>] ? flush_tlb_others+0x5/0x8
[Thu Dec 10 21:19:34 2015]  [<ffffffff81047f30>] ? flush_tlb_mm_range+0xcb/0xd4
[Thu Dec 10 21:19:34 2015]  [<ffffffff81102a17>] ? tlb_flush_mmu_tlbonly+0x2b/0x68
[Thu Dec 10 21:19:34 2015]  [<ffffffff81102e18>] ? tlb_flush_mmu+0x9/0x12
[Thu Dec 10 21:19:34 2015]  [<ffffffff81102e2c>] ? tlb_finish_mmu+0xb/0x2f
[Thu Dec 10 21:19:34 2015]  [<ffffffff81104043>] ? zap_page_range+0x90/0xa2
[Thu Dec 10 21:19:34 2015]  [<ffffffff810eeac8>] ? get_page+0x9/0x21
[Thu Dec 10 21:19:34 2015]  [<ffffffff81101fab>] ? set_pte_at+0x5/0x8
[Thu Dec 10 21:19:34 2015]  [<ffffffff811049c5>] ? handle_mm_fault+0x715/0xb3d
[Thu Dec 10 21:19:34 2015]  [<ffffffff8146be39>] ? SYSC_recvfrom+0xd6/0xe8
[Thu Dec 10 21:19:34 2015]  [<ffffffff8110766b>] ? find_vma+0x11/0x5c
[Thu Dec 10 21:19:34 2015]  [<ffffffff811111c7>] ? SyS_madvise+0x42c/0x681
[Thu Dec 10 21:19:34 2015]  [<ffffffff8103e639>] ? kvm_clock_read+0x1b/0x1d
[Thu Dec 10 21:19:34 2015]  [<ffffffff8108e886>] ? SyS_time+0xb/0x2f
[Thu Dec 10 21:19:34 2015]  [<ffffffff815448ee>] ? entry_SYSCALL_64_fastpath+0x12/0x71

IP-Projects.de · Dec 11, 2015

Wir hatten so ein Problem auch einmal bei einem Server der massiv unter SYN Flood stand. Dabei wurde durch die SYN Pakete ein Overhead an der Netzwerkkarte provuziert was zeitweise zu Paketloss und zu den genannten Fehlerbild von dir führte. Auch die Kernel BUG Meldung war hier nahezu identisch, nur mit einem anderen Service. Leider konnte ich die Art der Attacke nicht weiter analysieren, da der Kunde kurzerhand Cloudflair davor geschaltet hat. Auffällig war aber, dass ausschließlich ein CPU Core belastet wurde.

Kannst du einmal schauen, wie viele Anfragen aktuell auf den Server lauten:

netstat -n

sollte dir hier eine Liste der Anfragen und zugehörigen Ports liefern.

d3p · Dec 11, 2015

IP-Projects.de said:
Kannst du einmal schauen, wie viele Anfragen aktuell auf den Server lauten:

netstat -n

sollte dir hier eine Liste der Anfragen und zugehörigen Ports liefern.

Port 80: 136
Port 443: 165

Mit dem neuen NIC-Treiber läufts derzeit stabil.

Ist es dann aber immer noch ein lokales Problem meines VServers, oder ist da dennoch eig. der Hoster "schuld" ?

VServer schmiert ab / BUG: soft lockup - CPU#1 stuck for 26s!

Member

Active Member

Member

Active Member

Member

Member

Member

Active Member

Member

Member

Deleted member 15972

Guest

New Member

Member

Deleted member 15972

Guest

Member

New Member

Member

Member

Active Member

Member

We value your privacy