Sunday, September 13, 2009

Asus P6T Deluxe v2 Network Problems With Linux

Asus P6T Deluxe v2
The Problem

Under certain conditions, the LAN controller (i.e. NIC) on an Asus P6T Deluxe v2 motherboard may start sending strange packets after the computer is shut down.  This constant stream of packets from the NIC may cause all traffic through your switch to halt.  The cause of the strange packets appears to be triggered by a bug in the 'sky2' network driver in Linux.  See the information below to learn more about what conditions trigger this behavior and possible workarounds that can be used until the bug is fixed.

NOTE: This problem does not apply to Windows users.

Background

I recently assembled a computer using the Asus P6T Deluxe v2 motherboard and ran into an issue with the dual gigabit LAN controllers built-in to the motherboard.  While I was at work last week, I remotely connected to my new system through SSH and then shut it down after I had retrieved some information I needed (I usually like to shut down my systems while I'm not at home just to conserve energy).  I was also connected to another sytstem at home and about fifteen minutes later, after shutting down the Asus P6T-based system, I lost all network connectivity to the systems on my home network.  Without going into too many details, I eventually found out that the LAN controllers on the P6T motherboard can sometimes go into a weird state (after the machine is shut down) where they repeatedly send packets that look like the following (output is from 'tcpdump'):

21:57:39.787385 00:26:18:46:34:67 > 01:80:c2:00:00:01, ethertype Unknown (0x8808), length 60:
        0x0000:  0001 ffff 0000 0000 0000 0000 0000 0000  ................
        0x0010:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0020:  0000 0000 0000 0000 0000 0000 0000       ..............

The P6T LAN controller sends about 50 of these packets per second and after a while this causes all network traffic going through my switch to cease.  As you might imagine, it is especially annoying if this happens while I am at work and no longer have a way of accessing any of my systems until I get home.

At the time I discovered this, I called Asus technical support to see if this was a known issue or if I could possibly have a faulty motherboard.  Calling them was a mistake.  All I got was a condescending tone from the other side of the line and a person telling me that it was impossible for the NIC to send packets while the system was shut down.  He wouldn't believe what I was telling him.  That conversation ended pretty quickly once I realized I wouldn't be getting any support from Asus on this issue.  Before I hung up though they did mention that someone with a Maximus II Formula motherboard had called in with a similar issue a week earlier.  I later found out that the Maximus motherboard uses the exact same LAN controller as the P6T motherboard, a Marvell 88E8056 chip.  I would suspect that any motherboard that uses this Marvell controller would experience the same network problems.

After the lack of support from Asus, I decided to dig into the issue myself and after a lot of testing I found out how to reproduce the problem 100% of the time.  Before I explain how to reproduce the behavior/bug, here is a summary of the hardware and software I have tested with:
  • Motherboard: Asus P6T Deluxe v2
    • Dual gigabit NICs using the Marvell 88E8056 chip.
    • Other motherboards using the Marvell LAN controller may also be affected.
  • Switch: Netgear GS108T Gigabit Switch
    • I'm not sure if network activity halts when other gigabit switches are used.
  • OS's: Express Gate (Splashtop Linux), Ubuntu 9.04, Ubuntu 9.10 (alpha)
Testing and Observations

The table below summarizes each of the scenarios I tested to narrow down the problem.  The first four columns in the table show the variables that were changed for each test:
  • OS / Distro - I tested on Windows, Express Gate (Splashtop), Ubuntu 9.04, and Ubuntu 9.10.
  • Kernel Version - I tested with the standard installed kernel on all distributions except Ubuntu 9.04 where I decided to test with a vanilla non-patched kernel from kernel.org in addition to the standard Ubuntu kernel.
  • Switch Port Configuration - I used a managed switch and performed tests where the switch port connected to the P6T NIC was configured for either 10, 100, or 1000 Mb/s (and it did make a difference as you'll see in the table).
  • WOL enabled before shutdown? - I recorded whether or not wake-on-lan was configured before the system was shut down (e.g on linux wake-on-lan can be turned on with a command similar to 'ethtool -s eth0 wol g').
 The last four columns record my observations:
  • Ethernet Link Reset During Shutdown? 
    • During some tests, the led link lights on the switch would indicate that the link was momentarily shut down and then reestablished at the time the system was powered down.
    • On other tests, the led link lights on the switch would indicate that the link was never reset during shutdown (as if the link was still operating in the same mode as it was when the system was turned on).
    • In other tests, power to the NIC was shut off and the link was never reestablished (as expected if WOL was not enabled).
  • Ethernet speed after shutdown - During the tests where the ethernet link to the switch was reset after the sytem was shut down, I recorded the speed of the reestablished ethernet link.
  • WOL works? - Shows whether or not wake-on-LAN worked after the system was shut down.  For some tests this column is not applicable because WOL was never enabled/configured.
  • Strange packet bug encountered? - Shows whether or not I was able to see the strange packet sending behavior.
    • NOTE: In cases where the packet sending bug was observed, I first had to send around eight broadcast packets on the network to trigger the problem where the P6T NIC started sending the strange packets.  I could send any type of broadcast packet.  For example, the problem could be triggered by sending either broadcast pings (e.g 'ping -b 192.168.1.255') or wake-on-LAN magic packets (they didn't have to be specifically directed to the P6T NIC MAC address).
    • NOTE: In each case where I encountered the packet sending bug, traffic immediately resumed through the switch after after unplugging the cable from the offending network port.

Evironment / Settings
Observations
 OS/Distro Kernel
Version

Switch Port
Configuration
(Mb/s)
WOL
enabled
before
shutdown?
Ethernet Link
Reset During
Shutdown?
Ethernet
speed after
shutdown
WOL
works?
Strange
packet bug
encountered?
Windows 7 RTM n/a 10 yes yes 10 yes no
Windows 7 RTM n/a 100 yes yes 10 yes no
Windows 7 RTM n/a 1000 yes yes 10 yes no
Express Gate Unknown 10 no no 10 n/a
(WOL not
enabled)
no
Express Gate Unknown 100 no no 100 n/a
(WOL not
enabled)
no
Express Gate Unknown 1000 no no 1000 n/a
(WOL not
enabled)
yes
Ubuntu 9.04 2.6.28,
2.6.31
10 yes yes 10 no no
Ubuntu 9.04 2.6.28,
2.6.31
100 yes yes 100 no no
Ubuntu 9.04 2.6.28,
2.6.31
1000 yes yes 100 no no
Ubuntu 9.04 2.6.28,
2.6.31
10 no no n/a
(NIC gets
powered
down)
n/a
(WOL not
enabled)
no
Ubuntu 9.04 2.6.28,
2.6.31
100 no no n/a
(NIC gets
powered
down)
n/a
(WOL not
enabled)
no
Ubuntu 9.04 2.6.28,
2.6.31
1000 no
no
n/a
(NIC gets
powered
down)
n/a
(WOL not
enabled)
no
Ubuntu 9.10 2.6.31 10 yes yes 10 no no
Ubuntu 9.10 2.6.31 100 yes yes 10 no no
Ubuntu 9.10 2.6.31 1000 yes yes 100 no no
Ubuntu 9.10 2.6.31 10 no no 10 n/a
(WOL not
enabled)
no
Ubuntu 9.10 2.6.31 100 no no 100 n/a
(WOL not
enabled)
no
Ubuntu 9.10 2.6.31 1000 no no 1000 n/a
(WOL not
enabled)
yes

A couple of things to note from the table:
  • On Windows, where WOL worked and the bug was never encountered, the ethernet link was always reestablished at 10 Mb/s when the system was shut down no matter what speed of the link was when the system was powered on.
    • On Linux, if WOL was enabled, the ethernet link would be reestablished at either 10 or 100 Mb/s when the sytem was shut down (depending on the speed of the link while the system was running).
  • On Linux, when WOL was not enabled and the system was affected by the bug, the ethernet link was never reset when the system was shut down and it remained operating at the same speed as when the system was running.  It was as if the NIC was still operating in the same mode as when the sytem was turned on instead of going into a special mode that listened for WOL packets.
Summary

The strange packet sending bug was encountered if all of the following conditions were met:
  • The system was powered down from one of the following distros
    • Ubuntu 9.10
    • Express Gate (i.e. Splashtop) Linux built-in on the motherboard
    • Status of other distros is unknown.  I suspect that this problem may not be kernel-version specific but due to a patch that some distros apply to the sky2 driver...but I am not certain.
  • A Gigabit switch was used AND the ethernet link between the switch and LAN controller was established at 1000 Mb/s before the system was shut down
  • The sky2 network driver was not configured for wake-on-lan before the system was shut down
Possible Workarounds:
  • Enable wake-on-lan before the system is shut down (wake-on-lan will not work, but at least the NIC won't start sending the strange packets)
  • Use a Linux distro or kernel that does not exhibit this problem
Anyhow, hopefully this helps somebody that has been experiencing the same network problems and wasn't sure why it was happening (I was worried for a little bit that it might be a faulty motherboard or a failing switch).  If this problem gets fixed in the sky2 driver on Linux, I'll update this post to reflect the status.

Friday, September 11, 2009

Wake-On-LAN With Windows 7 and Asus P6T Deluxe v2 Motherboard

Article Last Updated: 3/3/2010

These instructions describe the steps required to get wake-on-lan (WOL) working with an Asus P6T Deluxe v2 motherboard and Windows 7 RTM.  Even if you don't have this specific motherboard, the instructions should be sufficient for learning how to enable WOL on most Windows systems. Also, I'll assume that if you found these instructions you already know what WOL is so I'll skip the explanation.

NOTE: For Linux users, WOL isn't currently working with the NICs on this motherboard.  These NICs use the 'sky2' network driver which has apparently had a history of problems.  I've tested on Ubuntu 9.04 and 9.10 with both standard Ubuntu kernels and vanilla 2.6.31 kernels from kernel.org and WOL just doesn't work.  I've also found other issues with these 'sky2' driver on Linux and will make another blog post on that subject soon.

To get WOL working on the Asus P6T Deluxe v2 motherboard, you will need to perform the following steps:
  1. Enable wake-on-lan in the systems BIOS setup utility
  2. Install the most recent network driver available through Windows Update
  3. Make sure the network driver in Windows 7 is configured properly for wake-on-lan
Each of these steps is shown in greater detail below.  Also, in case it isn't clear, since WOL needs to be enabled through the operating system as well as through the BIOS it will only work after you power down from the OS that you enabled the settings in.  For example, if you have a dual boot system with both Linux and Windows (and have only enabled WOL settings through Windows), WOL will only work after the computer has been shut down from Windows.

BIOS Setup

Enter the BIOS setup before the system boots and go to the Power tab --> APM Configuration screen and make sure that Power On By PCIE Devices is enabled.

The NICs in this motherboard are connected through the PCI-Express bus so Power on by PCIE Devices is the only option you need to enable (i.e. Power On by PCI Devices is not needed).  For other systems, you may need to consult your computer or motherboard documentation for information on what settings you need to enable for wake-on-lan support.

Install Most Recent Network Drivers from Asus

Update (3/3/2010) - It is no longer necessary to download the network driver from the Asus website. The Marvell driver that is available through Windows Update provides the desired functionality, so just make sure Windows is up-to-date.

On Windows 7 RTM (i.e. the final retail version), the network drivers that are automatically added by the OS during installation do not work with wake-on-lan.  I had to install the latest drivers from the Asus download site (http://support.asus.com/download) before WOL started working.  In the future, Microsoft may provide a more up-to-date driver for these network cards but as of right now (9/11/2009) it is necessary to download the more recent driver from Asus.

Enable Wake-On-LAN in the Network Driver Settings

Open the driver configuration window for the NIC by opening the system Control Panel --> opening the Device Manager --> expanding Network adapters, then right-clicking on the Marvell Yukon Gigabit Ethernet Controller and selecting properties.  This will bring up the driver configuration window seen below. Go to the Advanced tab and make sure that Wake From Shutdown is set to On.


Also, make sure that Wake-Up Capabilities is set to Magic Packet (you don't need Magic Packet and Pattern Match unless you know what it is).


Next, go to the Power Management tab and make sure your settings match the following (they should all be enabled by default):

That is it.  You should now be able to power off your machine and wake it with your preferred WOL utility/software.