Asus P6T Deluxe v2 Network Problems With Linux

The Problem

Under certain conditions, the LAN controller (i.e. NIC) on an Asus P6T Deluxe v2 motherboard may start sending strange packets after the computer is shut down.  This constant stream of packets from the NIC may cause all traffic through your switch to halt.  The cause of the strange packets appears to be triggered by a bug in the 'sky2' network driver in Linux.  See the information below to learn more about what conditions trigger this behavior and possible workarounds that can be used until the bug is fixed.

NOTE: This problem does not apply to Windows users.

Background

I recently assembled a computer using the Asus P6T Deluxe v2 motherboard and ran into an issue with the dual gigabit LAN controllers built-in to the motherboard.  While I was at work last week, I remotely connected to my new system through SSH and then shut it down after I had retrieved some information I needed (I usually like to shut down my systems while I'm not at home just to conserve energy).  I was also connected to another sytstem at home and about fifteen minutes later, after shutting down the Asus P6T-based system, I lost all network connectivity to the systems on my home network.  Without going into too many details, I eventually found out that the LAN controllers on the P6T motherboard can sometimes go into a weird state (after the machine is shut down) where they repeatedly send packets that look like the following (output is from 'tcpdump'):

21:57:39.787385 00:26:18:46:34:67 > 01:80:c2:00:00:01, ethertype Unknown (0x8808), length 60:
        0x0000:  0001 ffff 0000 0000 0000 0000 0000 0000  ................
        0x0010:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0020:  0000 0000 0000 0000 0000 0000 0000       ..............

The P6T LAN controller sends about 50 of these packets per second and after a while this causes all network traffic going through my switch to cease.  As you might imagine, it is especially annoying if this happens while I am at work and no longer have a way of accessing any of my systems until I get home.

At the time I discovered this, I called Asus technical support to see if this was a known issue or if I could possibly have a faulty motherboard.  Calling them was a mistake.  All I got was a condescending tone from the other side of the line and a person telling me that it was impossible for the NIC to send packets while the system was shut down.  He wouldn't believe what I was telling him.  That conversation ended pretty quickly once I realized I wouldn't be getting any support from Asus on this issue.  Before I hung up though they did mention that someone with a Maximus II Formula motherboard had called in with a similar issue a week earlier.  I later found out that the Maximus motherboard uses the exact same LAN controller as the P6T motherboard, a Marvell 88E8056 chip.  I would suspect that any motherboard that uses this Marvell controller would experience the same network problems.

After the lack of support from Asus, I decided to dig into the issue myself and after a lot of testing I found out how to reproduce the problem 100% of the time.  Before I explain how to reproduce the behavior/bug, here is a summary of the hardware and software I have tested with:

• Motherboard: Asus P6T Deluxe v2
• Dual gigabit NICs using the Marvell 88E8056 chip.
• Other motherboards using the Marvell LAN controller may also be affected.
  
• Switch: Netgear GS108T Gigabit Switch
• I'm not sure if network activity halts when other gigabit switches are used.
  
• OS's: Express Gate (Splashtop Linux), Ubuntu 9.04, Ubuntu 9.10 (alpha)
  

Testing and Observations

The table below summarizes each of the scenarios I tested to narrow down the problem.  The first four columns in the table show the variables that were changed for each test:

• OS / Distro - I tested on Windows, Express Gate (Splashtop), Ubuntu 9.04, and Ubuntu 9.10.
• Kernel Version - I tested with the standard installed kernel on all distributions except Ubuntu 9.04 where I decided to test with a vanilla non-patched kernel from kernel.org in addition to the standard Ubuntu kernel.
• Switch Port Configuration - I used a managed switch and performed tests where the switch port connected to the P6T NIC was configured for either 10, 100, or 1000 Mb/s (and it did make a difference as you'll see in the table).
• WOL enabled before shutdown? - I recorded whether or not wake-on-lan was configured before the system was shut down (e.g on linux wake-on-lan can be turned on with a command similar to 'ethtool -s eth0 wol g').

 The last four columns record my observations:

• Ethernet Link Reset During Shutdown? 
• During some tests, the led link lights on the switch would indicate that the link was momentarily shut down and then reestablished at the time the system was powered down.
• On other tests, the led link lights on the switch would indicate that the link was never reset during shutdown (as if the link was still operating in the same mode as it was when the system was turned on).
• In other tests, power to the NIC was shut off and the link was never reestablished (as expected if WOL was not enabled).
  
• Ethernet speed after shutdown - During the tests where the ethernet link to the switch was reset after the sytem was shut down, I recorded the speed of the reestablished ethernet link.
• WOL works? - Shows whether or not wake-on-LAN worked after the system was shut down.  For some tests this column is not applicable because WOL was never enabled/configured.
• Strange packet bug encountered? - Shows whether or not I was able to see the strange packet sending behavior.
• NOTE: In cases where the packet sending bug was observed, I first had to send around eight broadcast packets on the network to trigger the problem where the P6T NIC started sending the strange packets.  I could send any type of broadcast packet.  For example, the problem could be triggered by sending either broadcast pings (e.g 'ping -b 192.168.1.255') or wake-on-LAN magic packets (they didn't have to be specifically directed to the P6T NIC MAC address).
• NOTE: In each case where I encountered the packet sending bug, traffic immediately resumed through the switch after after unplugging the cable from the offending network port.

Evironment / Settings				Observations
OS/Distro	Kernel Version	Switch Port Configuration (Mb/s)	WOL enabled before shutdown?	Ethernet Link Reset During Shutdown?	Ethernet speed after shutdown	WOL works?	Strange packet bug encountered?
Windows 7 RTM	n/a	10	yes	yes	10	yes	no
Windows 7 RTM	n/a	100	yes	yes	10	yes	no
Windows 7 RTM	n/a	1000	yes	yes	10	yes	no
Express Gate	Unknown	10	no	no	10	n/a (WOL not enabled)	no
Express Gate	Unknown	100	no	no	100	n/a (WOL not enabled)	no
Express Gate	Unknown	1000	no	no	1000	n/a (WOL not enabled)	yes
Ubuntu 9.04	2.6.28, 2.6.31	10	yes	yes	10	no	no
Ubuntu 9.04	2.6.28, 2.6.31	100	yes	yes	100	no	no
Ubuntu 9.04	2.6.28, 2.6.31	1000	yes	yes	100	no	no
Ubuntu 9.04	2.6.28, 2.6.31	10	no	no	n/a (NIC gets powered down)	n/a (WOL not enabled)	no
Ubuntu 9.04	2.6.28, 2.6.31	100	no	no	n/a (NIC gets powered down)	n/a (WOL not enabled)	no
Ubuntu 9.04	2.6.28, 2.6.31	1000	no	no	n/a (NIC gets powered down)	n/a (WOL not enabled)	no
Ubuntu 9.10	2.6.31	10	yes	yes	10	no	no
Ubuntu 9.10	2.6.31	100	yes	yes	10	no	no
Ubuntu 9.10	2.6.31	1000	yes	yes	100	no	no
Ubuntu 9.10	2.6.31	10	no	no	10	n/a (WOL not enabled)	no
Ubuntu 9.10	2.6.31	100	no	no	100	n/a (WOL not enabled)	no
Ubuntu 9.10	2.6.31	1000	no	no	1000	n/a (WOL not enabled)	yes

A couple of things to note from the table:

• On Windows, where WOL worked and the bug was never encountered, the ethernet link was always reestablished at 10 Mb/s when the system was shut down no matter what speed of the link was when the system was powered on.
• On Linux, if WOL was enabled, the ethernet link would be reestablished at either 10 or 100 Mb/s when the sytem was shut down (depending on the speed of the link while the system was running).
• On Linux, when WOL was not enabled and the system was affected by the bug, the ethernet link was never reset when the system was shut down and it remained operating at the same speed as when the system was running.  It was as if the NIC was still operating in the same mode as when the sytem was turned on instead of going into a special mode that listened for WOL packets.
  

Summary

The strange packet sending bug was encountered if all of the following conditions were met:

• The system was powered down from one of the following distros
• Ubuntu 9.10
• Express Gate (i.e. Splashtop) Linux built-in on the motherboard
• Status of other distros is unknown.  I suspect that this problem may not be kernel-version specific but due to a patch that some distros apply to the sky2 driver...but I am not certain.
• A Gigabit switch was used AND the ethernet link between the switch and LAN controller was established at 1000 Mb/s before the system was shut down
  
• The sky2 network driver was not configured for wake-on-lan before the system was shut down
  

Possible Workarounds:

• Enable wake-on-lan before the system is shut down (wake-on-lan will not work, but at least the NIC won't start sending the strange packets)
• Use a Linux distro or kernel that does not exhibit this problem
  

Anyhow, hopefully this helps somebody that has been experiencing the same network problems and wasn't sure why it was happening (I was worried for a little bit that it might be a faulty motherboard or a failing switch).  If this problem gets fixed in the sky2 driver on Linux, I'll update this post to reflect the status.