TIPS: Poor Network Performance in Windows under Virtual Machine

TCP Chimney Offload, Receive Side Scaling, and Network Direct Memory Access

Sounds super awesome huh?!? But what is it?

I have around 20 virtual-machines, running under VMware Workstation, they allow me to test BulletProof FTP Server for Windows under all the different versions, editions and flavors of Windows that now exist. It’s exhausting actually, mostly because I need to perform Windows Updates every few months.

The host for all these virtual-machines, as well as my development, runs on a custom built ASUS Rampage III with Intel Core i7 970 with 6/12 physical/logical cores, 24 GB DDR2 RAM, storage is via ARC-1880i in RAID0+1 of 4x Samsung SSD. Why does any of that matter? Because, I’m trying to underscore that it’s a blazing fast computer.

But I noticed some network-speed problems, specially under Windows 2008 R1 and R2. Part of the issue was that “vmware tools” on the machines were not updated (do this asap, it will run much better), but Windows Update was taking almost an hour; just to see if there were updates. I dig some digging and found a bunch of articles, linked below. If you are having network-performance issues, please look to see if you are affected by this feature…

NOTE: TCP Chimney Offload does not appear to be a feature that is unique to Windows 2008, as I’ve found it set to “automatic” in default installs of Windows 7, 8, Server 2008; “disabled” in Vista.

HOWTO: Is the MAGIC on?

c:\] netsh int tcp show global
Querying active state...

TCP Global Parameters
----------------------------------------------
Receive-Side Scaling State : enabled
Chimney Offload State : automatic
NetDMA State : enabled
Direct Cache Acess (DCA) : disabled
Receive Window Auto-Tuning Level : normal
Add-On Congestion Control Provider : none
ECN Capability : disabled
RFC 1323 Timestamps : disabled
** The above autotuninglevel setting is the result of Windows Scaling heuristics
overriding any local/policy configuration on at least one profile.

HOWTO: Turn Off the MAGIC

c:\] netsh interface tcp set global rss=disabled chimney=disabled autotuninglevel=disabled
Ok.

Information about the TCP Chimney Offload, Receive Side Scaling, and Network Direct Memory Access features in Windows Server 2008
http://support.microsoft.com/kb/951037

Poor network performance on Windows 2008 Server virtual machine (1009517)
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1009517

Windows Server 2008 R2 Windows updates very slow download time
http://social.technet.microsoft.com/Forums/windowsserver/en-US/ca80c34c-a4e5-4027-880a-9646aa80b352/windows-server-2008-r2-windows-updates-very-slow-download-time

NETSH to disable advanced features
http://social.technet.microsoft.com/Forums/exchange/en-US/968afa2d-e326-49f2-b6a8-1f264e82beea/netsh-to-disable-advanced-features?forum=exchange2010

The effect of TCP Chimney offload on viewing network traffic
http://blogs.technet.com/b/networking/archive/2008/11/14/the-effect-of-tcp-chimney-offload-on-viewing-network-traffic.aspx

KNOWN PROBLEM: Slow Startup after Several Weeks of Heavy Usage

KNOWN PROBLEM: Slow Startup after Several Weeks of Heavy Usage

In order to provide information on Server Monitor -> Files Uploaded/Downloaded, the software writes entries into secondary files stored in the Storage-Location directory. These files contain one-line per file uploaded/downloaded and as the usage of the program increases, these files get bigger and bigger. Start-Up of the program will be effected as these files increase in size and the time to load the file into memory increases (and the footprint in memory increases).

Server Monitor -> Files Downloaded/Uploaded

Server Monitor -> Files Downloaded/Uploaded

To resolve this problem, it’s necessary to manually archive these files to another location; this will “reset” the statistics found in Server Monitor -> Files Uploaded/Downloaded.

You can manually archive these files:
1) Closing BulletProof FTP Server and ensure the SERVICE is not current running/active
2) Use My Computer or a Command-Prompt and navigate to the Storage-Location.
3) Locate the files named below and move them to some archive directory of your choosing (or delete them):

bpftpserver-common-downloads.ini
bpftpserver-common-uploads.ini

4) Start BulletProof FTP Server

NOTE: In a later version, this will be resolved with a re-organization and re-write of this function in order to save the information in a way that doesn’t affect performance.

HOWTO: Using Internet Explorer to connect to a FTP-Server

Configuring Internet Explorer to be able to connect to BulletProof FTP Server (or any FTP Server) is very easy to do and requires the toggling of two (2) options from:

Windows Start Button -> Control Panel -> Network & Internet -> Internet Options, click the “Advanced” tab and check “Enable FTP Folder” and “Use Passive FTP”.

Below are screen-shots for accessing these features in Windows. Please click the small-image to view the full-image.

KNOWN PROBLEMS: BulletProof FTP Server 2011

Jan 31, 2012: As of today there are 3 (THREE) known problems in BulletProof FTP Server 2011:

Windows System-Service under Vista, 7, Server 2008R2:
In order to run BulletProof FTP Server 2011 as a Windows System-Service, you will need to disable User-Access-Control (UAC):
SERVICE: User-Access Control (UAC) prevents running Windows System-Service


Access-Violation when starting the GUI under Vista, 7, Server 2008, Server 2008R2:
Microsoft has released a patch called KB2533623 which will deny the program the ability to load a “virtual” DLL. However, this breaks the BulletProof FTP Server 2011 and needs to be uninstalled:
Crash on Program Start: Windows Update KB2533623


FTP-User can login, but can’t see any directory-listing:
aka “Unable to retrieve directory-listing”
aka “Internet Explorer cannot display the webpage”
aka “Oops! Google Chrome could not connect to FTP.YOURHOST.COM”

this also appears in the Log-Watch for BulletProof FTP Server 2011 as
2012-01-31 13:12:33 – mmc [000005] [12.13.14.15] – PASV
2012-01-31 13:12:33 – mmc [000005] [12.13.14.15] – 227 Entering Passive Mode (12,13,14,15,156,119)
2012-01-31 13:12:34 – mmc [000005] [12.13.14.15] – INFO: user disconnected gracefully. (00:00:01)

REASON: Stateful-Packet-Inspection (SPI) and Passive-Mode:
The first thing to do when setting up BulletProof FTP Server 2011 is to configure it to operate behind your NAT/Firewall (HOWTO).

This is called Passive-Mode (PASV) support and is a must for most installations. However many NAT/Firewalls have a “feature” called Stateful-Packet-Inspection or SPI (INFO) that sniffs the network-traffic for FTP packets and mangles the information in the packets to dynamically open ports on your firewall. Sadly, it almost always fails to correctly change the entire packet. In order to “help” SPI, it’s recommend to change the Static IP Address to the Listening IP address for your machine running BPFTP Server.

In order to better “cooperate” with SPI, it’s recommend that you try changing the Static IP Address under Management -> Settings -> NAT/Firewall to the Listening IP on the left side of the main-screen under Server Info.

SCREENSHOT: Management -> Settings -> NAT/Firewall

TIPS: How to serve your Network-Drives via FTP

BulletProof FTP Server has the ability to serve files on your Mapped Network Drive via FTP. But you will need to add some special parameters to make sure that the credentials for the mapped drive are saved and the connection is toggled as persistent.

This is best done via the command-line (CLI) using the NET.EXE USE command.

Take the following example:

c:> %SystemRoot%\system32\net.exe use h: \\myserver\myshare /user:myusername /persistent:yes

In this example we are mapping drive-letter “H:” to a computer named “myserver” with a share-name of “myshare”, where the credentials needed to login to the machine are specified as “myusername” (if you were part of a domain, it would be “mydomain\myuser). The “/persistent:yes” instructs Windows to reconnect the drive at login.

Full Docs on the NET.EXE USE command:
http://www.microsoft.com/resources/documentation/windows/xp/all/proddocs/en-us/net_use.mspx?mfr=true

Microsoft Windows, by default, disconnect network-drives after X minutes of inactivity. Remapping the drive is done quickly and usually without any interruption. However if you have any problems, it’s recommended that you increase the default 600 seconds (10 minutes) to a larger number:
Mapped Drive Connection to Network Share May Be Lost

Crash on Program Start: Windows Update KB2533623


Faulting application bpftpserver-2011.exe, version 2011.1.0.71, time stamp 0x4e8f8103, faulting module unknown, version 0.0.0.0, time stamp 0x00000000, exception code 0xc000001d, fault offset 0x03094c99, process id 0x818, application start time 0x01cc9567a0e85aad

We’ve experienced some random crashes, at start-up in the program that have been found to be related to a patch from Microsoft Update with Microsoft Security Advisory: Insecure library loading could allow remote code execution (KB 2533623).

Technically, this patch prevents the loading of external DLL’s without a fully-qualified-path to the DLL. Unfortunately, we utilize a virtualized DLL that is loaded at program start that does not load a DLL-file on the system, so it can fail when KB2533623 is applied to this system.


Is KB2533623 installed on my installation of Windows?

Windows GUI: You can list all of the updates for your installation of Windows from the following location:

Windows Start -> Control Panel -> Programs -> View Installed Updates

Alternatively, Internet Explorer can take you there with the following:
http://go.microsoft.com/fwlink/?LinkID=3326

Windows CLI: Alternatively you can execute the following at a command-line:

C:\Users\myusername> wmic qfe list | find "2533623"
http://support.microsoft.com/?kbid=2533623 MYCOMPUTERNAME Update KB2533623 MYCOMPUTERNAME\myusername 7/15/2011


How do I uninstall KB2533623?

Windows GUI: You can uninstall individual updates from Microsoft by loading the following:

Right-Click and select "Uninstall"

Windows CLI: If you are using Windows 2008 R2, you can then uninstall it from the command line using the WUSA.EXE command. (If you aren’t using Windows 2008 R2, you’ll need to use the above GUI method).

C:\Users\myusername> wusa /uninstall /kb:2533623

Frequently Asked Questions (FAQ)

Q: We are running an older version of BulletProof FTP Server. If we upgrade to your latest version, will our configuration/user list/etc. transfer from our old version?
Q: Can I backup the Settings, Users and Groups?
Q: Can BulletProof FTP Server run as a Windows System-Service?
Q: My customers can’t login to BPFTP Server!
Q: My customers can login, but they can’t get a directory-listing!
Q: Do I get a discount for upgrading or renewing my previous order?
Q: I purchased BulletProof FTP Server a few years ago and I lost my key. Can you resend it?


 

Q: We are running an older version of BulletProof FTP Server. If we upgrade to your latest version, will our configuration/user list/etc. transfer from our old version?

A: YES! Absolutely. Everything is stored in INI files (settings, users, groups, etc), when the current version of the software is installed; it will go out and look for these files in the default locations for previous versions. If anything is found, the program will prompt you and ask if it’s ok to migrate these files to the current version.

FAQ: Can I Upgrade and retail all my Users, Groups and Settings?

 

Q: Can I backup the Settings, Users and Groups?

A: YES! You just need to make sure the program is not running when you backup the files. For a list of the default storage-locations for these files, please see the URL below:
http://blog.builtbp.com/2011/12/faq-where-does-bpftp-server-store-the-users-groups-and-settings/

 

Q: Can BulletProof FTP Server run as a Windows System-Service?

A: YES! The latest version includes very tight support for running BulletProof FTP Server as a Windows System-Service. Please see this guide:
HOWTO: Windows System-Service
Windows System Service: Logon As Service

 

Q: My customers can’t login to BPFTP Server!
Q: My customers can login, but they can’t get a directory-listing!

A: The machine is most likely firewalled. This means there’s a physical-device which is acting as a firewall (aka NAT/Firewall/Router/WiFi,etc) and/or Windows Firewall and/or (some other) Internet Security that is protecting the computer.

In order to run BulletProof FTP Server in a secure environment, you *MUST* configure PASSIVE-MODE and open at least two tcp/ip ports to allow full functionality. These two ports are called:
Control-Port (default tcp/ip 21) provides the ability for the ftp-client to login and issue commands
Data-Ports provide a conduit for the ftp-client to connect to the ftp-server for file-transfers, directory-listings, etc

HOWTO: Windows Firewall and BPFTP Server 2011
HOWTO: Setting up NAT/Passive/Firewall Support
FTP Server Firewall Ports and Tester

 

Q: Do I get a discount for upgrading or renewing my previous order?

A: YES! You can get the discount coupon-code and instructions from our website under the Members section of our website (see below) or you can email us with the customer information used on the previous order and we can send it directly to you.

Please also see:
FAQ: My License-Code doesn’t work in the Latest Version

 

Q: I purchased BulletProof FTP Server a few years ago and I lost my key. Can you resend it?

A: PERHAPS: First, we need you to send the customer information for the previous order. This will allow us to look it up and see what length of Update/Support Protection with Download Locker was purchased. If it’s within the period you purchased, we can quickly issue a new key and send it to you. If it’s expired, then we can send you a discount coupon-code for renewal.

You can see all of your orders and the Update/Support Protection with Download Locker by logging into the Members section of our website. If the term is expired, you an also get a coupon-code for renewal in this same location. If you are having troubles logging into the Members section of the website, please Contact Us and include your question and customer-information used on the previous order.

FIX: “Out of Memory” or “Low GDI” with BulletProof FTP Server

Is BulletProof FTP Server for Windows crashing when you first run it? You might have it freeze during “Loading” or a “Crash Report” comes up saying “Out of Memory”…

Don’t worry! It’s most likely a problem with running out of GDI-Handles in your installation of Windows…

You may receive an “Out of Memory” error message because of the desktop heap limitation in Windows Vista or in Windows 7
http://support.microsoft.com/kb/947246

“Out of Memory” error message appears when you have a large number of programs running
http://support.microsoft.com/kb/126962

NOTE: You CANNOT have the /3GB option specified in the bootloader, this is often used on Exchange Server installations. (MORE INFO)

This GUIDE will help illustrate the steps to increase the amount of memory..

Vista Hands On #17: Solving a pesky resource problem
http://www.zdnet.com/blog/bott/vista-hands-on-17-solving-a-pesky-resource-problem/269

SharedSection & Max Number of Open Windows on XP & Vista
http://www.mycsharpcorner.com/Post.aspx?postID=50

Windows XP flakiness – solved
http://weblogs.asp.net/kdente/archive/2004/06/04/148145.aspx

Registry setting keeps Windows from wigging out when you open lots of IE7 tabs
http://weblogs.asp.net/jgalloway/archive/2008/01/14/registry-setting-keeps-windows-from-wigging-out-when-you-open-lots-of-ie7-tabs.aspx

INCREASING USER HANDLE AND GDI HANDLE LIMITS
http://weblogs.asp.net/mikedopp/archive/2008/05/16/increasing-user-handle-and-gdi-handle-limits.aspx

Many Thanks to AlexanderL for helping to discover a resolution to this problem!

Create a Login-Message with Recently Uploaded Files

Let’s say you want to create a login-message for a ftp-user, to display the most recently uploaded files. Commonly this is called a “Message-of-the-Day” aka MOTD.

First, you’ll need to setup a directory structure for your BPFTP Server installation. I commonly create a “bin” directory and then go from there. Here is a write-up on setting up directory structures.

Next, you’ll want to create a batch-file with the following lines (save this to your “bin” directory as “event-fileuploaded.cmd”:

@echo off
echo %DATE% %TIME% – File Uploaded [%1] for %2 bytes >> c:\ftp\motd.txt

Thirdly, you’ll need to connect this batch-file to the “OnFileUpload” event under the “Events Manager” in BPFTP Server (Ctrl-E). Scroll down until you see the event, then Check-ON the option “Execute” and put in the following command-line:

c:\ftp\bin\event-fileuploaded.cmd %FILE %FILESIZE

Lastly, you need to edit the user which can see the MOTD in “User Accounts” (Ctrl-U) and click on the ftp-user and then “Links & Messages”, Check-ON the “Show Login Message” and type in “c:\ftp\motd.txt” for the file to display.

That’s it! You can test this by logging into the ftp-server, upload a file and then log-in again to see the MOTD!