robertlacroix.com - Robert Lacroix

Packets dropped with 0xC0040034 FWX_E_SEQ_ACK_MISMATCH result code on Microsoft ISA Server

2010-01-24T15:12:00Z

If you notice performance problems on a web server published through ISA, check if there are packets dropped in the firewall log with the published server as Client IP, the client as Destination IP and a Result Code of 0xC0040034 FWX_E_SEQ_ACK_MISMATCH.

If that's the case, try the following steps and see if the problem disappears:

Update network card drivers to the current version. Reboot afterwards and try if it helped.
Disable IP offloading by setting EnableRSS, EnableTCPA, EnableTCPChimney to 0, and DisableTaskOffload to 1 (create the REG_DWORDs if necessary) in HKEY_LOCAL_MACHINE\ SYSTEM\ CurrentControlSet\ Services\ Tcpip\ Parameters on ISA and the published servers. Reboot afterwards and try if it helped.
In the the To tab of the publishing rule, activate "Requests appear to come from the ISA Server computer" instead of "Requests appear to come from the original client". The last change did fix the problem for me, it has some disadvantages though as you don't have the actual client ip in the web server logs.

How to eliminate re-occurring RSoP warnings in the eventlog on a Windows Server 2008 Server Core

2008-11-18T22:17:00Z

Recently we noticed RSoP warnings on our Windows Server 2008 Server Core Domain Controllers that re-occurred every five minutes in the system as well as in the application eventlog, saying that it was unable to log RSoP-Data for some reason. Resultant Set of Policies (RSoP) is the combined result of all applied group policies, which is logged for analysis reasons and can be seen with the Resultant Set of Policies MMC-snap-in.

Log Name:      System
Source:        Microsoft-Windows-GroupPolicy
Event ID:      1091
Level:         Warning
Description:      Windows could not record the Resultant Set of Policy (RSoP) information for the Group Policy extension <Group Policy Registry>. Group Policy settings successfully applied to the computer or user; however, management tools may not report accurately.

Log Name:      Application
Source:        Group Policy Registry
Event ID:      4099
Level:         Warning
Description:      The client-side extension could not log RSoP data because it failed with error code '0x8004401e <unknown-message-text>'.

After some digging on the net, it turned out that it was related to Group Policy Preference Client Side Extensions we used to distribute some registry settings. The Group Policy Preference Client Side Extensions are an addition to Windows Server 2008 respectively Windows Vista SP1 which can be installed on Windows XP and Windows Server 2003 as well. They add the possibility to distribute "preferences" rather than "policies" in a very smart way. The functionality was named PolicyMaker before, has been bought by Microsoft and is now distributed freely as part of Windows.

Regarding those Group Policy Preference Client Side Extensions, Microsoft obviously has been to aggressive while stripping down server core, as it turns out that in order to generate RSoP of these preferences, a WBEM (WMI) MOF file (Managed Object Format) is needed but missing in server core. Fortunately that can be fixed real easy:

1. Get %SYSTEMROOT%\system32\wbem\polprocl.mof from a full Windows Server 2008 and copy it to the same path on the server core.
2. Get %SYSTEMROOT%\system32\wbem\en-US\polprocl.mfl from a full Windows Server 2008 and copy it to the same path on the server core, where en-US should be replaced with the actual language that is installed (repeat for every language pack installed)
3. On an administrative cmd do mofcomp %SYSTEMROOT%\system32\wbem\polprocl.mof

Afterwards issue a gpupdate /force and the problem should be gone.

check_ganglia, a rudimental approach to combine nagios with ganglia

2008-10-06T18:35:00Z

When it comes to service monitoring, nagios is a pretty good tool, though it can only monitor services from outside. To add monitoring of metrics not exposed to the network, like disk usage, you need an agent running on the monitored server. nagios-statd, written as a python daemon, is such an agent.

From a management and security perspective it is good to keep the number of processes and open network ports as minimal as possible. Those that not only monitor service availability with nagios, but also resource usage with ganglia, already have the ganglia agent (gmond) running on each server. Adding nagios-statd would be the second agent that needs to be configured and kept up-to-date.

But what if you want to be alerted if one of the hard disks get full, load goes through the roof or the server begins using swap space? You would need nagios-statd although ganglia knows about everything. That's why I came up with the idea of using data from ganglia and monitor them with nagios and wrote a small php shellscript that accomplishes this task. There is for sure some work to do but I think it's a good start:

#!/usr/bin/php
<?php
### Get command line arguments
$host = $argv[1];
$metric = $argv[2];
$metric_unit = $argv[3];
$cluster_arg = $argv[4]; #optional
$threshold_warn_arg = $argv[5]; #optional
$threshold_crit_arg = $argv[6]; #optional

### Fill Variables
if (!is_numeric($threshold_warn_arg))
  $threshold_warn = 2;
else
  $threshold_warn = $threshold_warn_arg;

if (!is_numeric($threshold_crit_arg))
  $threshold_crit = 1;
else
  $threshold_crit = $threshold_crit_arg;

if (!$cluster_arg)
  $cluster = "sk";
else
  $cluster = $cluster_args;

### Get data from gmond
$fp = fsockopen("localhost", 8649, $errno, $errstr, 30);
if (!$fp) {
  echo "GANGLIA Unknown - $errstr ($errno)\n";
  exit(3);
} else {
  while (!feof($fp)) {
    $buffer .= fgets($fp, 128);
  }
  fclose($fp);
}

### Get metric out of XML
$xmlobj = simplexml_load_string($buffer);
$metric_value = $xmlobj->xpath("/GANGLIA_XML/CLUSTER[@NAME='$cluster']/HOST[@NAME='$host']/       
METRIC[@NAME='$metric']");

### Convert data (more types tbd)
$metric_value = $metric_value[0]->attributes();
if ($metric_value["TYPE"] == "double")
  $metric_value = doubleval($metric_value["VAL"]);
else
  $metric_value = $metric_value["VAL"];

### Build output strings
$perfcounter = $metric . "=" . $metric_value . $metric_unit . ";" . $threshold_warn
. $metric_unit . ";" . $threshold_crit . $metric_unit;
$text = $metric . " is " . $metric_value . $metric_unit;

### Output
if ($metric_value > $threshold_warn && $metric_value > $threshold_crit)
{
  print("GANGLIA OK - " . $text . " |" . $perfcounter . "\n");
  exit(0);
}

if ($metric_value < $threshold_warn && $metric_value > $threshold_crit)
{
  print("GANGLIA Warning - " . $text . " |" . $perfcounter . "\n");
  exit(1);
}

if ($metric_value < $threshold_warn && $metric_value < $threshold_crit)
{
  print("GANGLIA Critical - " . $text . " |" . $perfcounter . "\n");
  exit(2);
}

echo "GANGLIA Unknown\n";
exit(3);
?>

Save the script in your nagios-plugin directory (Debian defaults to /usr/lib/nagios/plugins) and make it executable (chmod +x check_ganglia). You can then define a new command in your nagios config as follows (in this example disk_free):

define command{
        command_name    check_ganglia_disk_free
        command_line    /usr/lib/nagios/plugins/check_ganglia $ARG1$ disk_free GB                 
        }

After that, add a new service to your host config:

define service{
        use                             generic-service                                           
        host_name                       xxx1
        service_description             DISK
        check_command                   check_ganglia_disk_free!xxx1
        }

Restart Nagios and you should be able to monitor disk usage from then on.

Fixing spontaneous reboots caused by VMware Fusion 2 Beta

2008-08-10T08:44:00Z

There is a bug in VMware Fusion 2 Beta (Beta 1 and Beta 2) that results in spontaneous reboots without even throwing a kernel panic. The problem occurs only when powering up a virtual machine after the Mac has been waked up from sleep. Especially Mac Pro users seem to be hit by this defect.

According to this thread some people found out that it helps to reload one of VMware's kernel extensions after wake up to workaround the crash until VMware has a fix ready.

Because this is very prone to be forgotten I automated it by the following steps:

1. Install SleepWatcher (both PKGs in the DMG)
2. Add the following three lines to the end of /etc/rc.wakeup

LIBDIR="/Library/Application Support/VMware Fusion"
kextunload "$LIBDIR/kexts/vmmon.kext"
kextload "$LIBDIR/kexts/vmmon.kext"

Update: It's fixed in RC1

Behind the scenes of www.sk-gaming.com

2008-03-24T16:51:00Z

(Crosspost on http://www.sk-gaming.com/blog/JFK/16317-Behind_the_scenes_of_wwwskgamingcom)

In my rare spare time I am running the technical infrastructure of the largest eSports web page out there. Since running approximately 4 months on a newly code page I wanted to take the opportunity to get you up to date what basically „runs“ www.sk-gaming.com from a technical perspective and what we did to improve the new page. I will go into very detail which might only be interesting for readers with a technical background.

Virtualized

Our web page runs solely on a bunch of Xen servers. Xen is an open-source virtualization platform (Hypervisor) that allows us to flexibly set up new virtual servers which can share the same physical hardware with other virtual machines. With Xen we can separate services that should better be on separate machines, utilize our physical hardware even better and move servers from one physical machine to another in case of a hardware failure.

Stable

A long time ago we’ve chosen Debian as our Linux distribution because it has proven to be extraordinary stable and it provides a very logical and lean approach to administrate it. Currently we are running the AMD64 release of Debian etch, their current stable version, on both the Parent Domain (dom0) of Xen and the virtual machines (domU’s) to leverage high amount of ram.

Scalable

To scale to the increased demand we’re experiencing since running on the new page we set up the load balancer keepalived to build a redundant firewall (active/passive) which amongst other things distributes the web requests to a farm of web servers. When developing the new page we took care that the code can be spread among multiple web servers safely.

Light

More than a year ago we switched our web server software from Apache to lighttpd which I can say was one of the best decisions ever. Lighttpd, as the name says, has a very low resource footprint, almost no memory usage and a really flexible way of configuring it, without sacrificing any features we need to run www.sk-gaming.com. And best of all, it’s literally fast as light.

Known

That we use PHP as our scripting language is pretty much known and has not changed. Currently we are running on PHP5 and use XCache for caching PHP intermediate code. Interesting maybe for other XCache users is that when we upgrade to the current xcache-1.2.2 we noticed a drop of around 30ms rendertime, even though XCache was already the fastest opcode cacher out there (at least according to this benchmark).

Fast

On the old page we used full page caching on some often loaded pages like the index page, but it was implemented in a very shirt-sleeved way: Every minute a script downloaded the site and pushed the content into the database. The site the user received simply got that rendered site from the database and sent that to the client. The problem with that implementation is obvious: it does not scale beyond a few number of pages cached, needed special coding on these to work and still put certain load on the database.

The approach we needed was a more general one which would work on every page and takes into account, that we have many pages which are user specific (e.g. /member/JFK looks different for you than for you). The solution we now have is leveraging the speed and the scalability that memcached offers , an in-memory cache for data objects, combined with some code (which I might share in a future article) that does basically the following:

Check if memcached has a cached version of the requested page
If yes: deliver it from there and stop processing
If no: go on
Render the page normally but put the output to a buffer
Put that buffer into memcached and deliver the rendered content

With that and using memcached to cache some other functionalities that normally needs database queries, we reduced the number of these queries to zero on cached pages and replaced them with only four, always fast (because non-blocking and distributed), memcached queries. What do zero database queries mean for the users? Page rendering of cached pages only takes around 10ms, which is for most users faster than the page actually needs to travel through the internet to your computers.

Compliant

Most pages are already XHTML 1.0 Transitional compliant which you can easily check with the W3C validator. We still have to work on some issues left, but we are committed to make www.sk-gaming.com completely compliant to current web standards.

Secure

To protect your password from being stolen on logging in, we decided to submit your passwords securely through an SSL encrypted connection. For extra security, we added the “Lock” icon next to the form, where you can enter your username and password, which leads you to a secure login form. This login form allows you to enter your password in a secure environment and makes sure your password is submitted a) only to us and b) only secured with HTTPS. When entering the username and password on the (non-encrypted) front page that cannot be assured, because the content could have been altered after leaving our web servers (man in the middle attack). For that extra security to work it is important that these requirements are fulfilled:

You receive no certificate errors when opening https://www.sk-gaming.com/login/
You have no rootkit or backdoor on your computer which captures key inputs or has installed trusted root certificates

Future-proof

With this solid basis we are looking into implementing and optimizing our infrastructure even more. For example we are evaluating how our users can benefit from new technologies like CardSpace/OpenID or Semantic Web. I will keep you updated.

KisMac 0.2a SVN build

2004-11-21T18:53:00Z

Since the guys from Binaervarianz haven't updated KisMac for long time I decided to build the latest version from their SVN server.

KisMac 0.2a Checkout 40

Using T-Mobile Multimedia Net Card UMTS/GPRS on Mac OS X

2004-11-21T15:02:00Z

There are two ways of using the T-Mobile Multimedia Net Card on Mac. You can either download (and pay for) Mobile High Speed 3G which requries you to reboot during the installation or you use my modem script.
You need to download the modem script to the "Modem Scripts" folder first and assign the proper permissions. To do this paste the following into Terminal.app:

cd /Library/Modem\ Scripts/
sudo curl -o "T-Mobile Cellular" http://www.robertlacroix.com/files/T-Mobile%20Cellular
sudo chmod 664 T-Mobile\ Cellular

1. Disable PIN request on your T-Mobile GSM card, you won't be able to enter it :)
2. Plug in the Multimedia Net Card
3. Go to the Network Preference Pane
4. Choose PC Card Serial in the Show dropdown
5. Use the following settings in the PPP tab:
  Account Name: t-mobile
  Password: t-mobile
  Telephone Number: *99#
6. Choose Modem T-Mobile Cellular

Wait till the LED on the Net Card flashes blue for UMTS or green for GPRS and connect :)

Firefox 1.0 with middle-mouse patch

2004-11-21T14:21:00Z

For those using an external three button mouse on Mac I just compiled Firefox 1.0 with middle-mouse applied. You can choose between two versions:

Fixing "iSync couldn't connect to Exchange" error on Panther

2004-09-05T16:38:00Z

When you set up Address Book to synchronize your contacts with Exchange hourly you probably know the "iSync couldn't connect to Exchange - Could not reach the server specified." for example when you are using your powerbook on the way.
I put a small shell script together which replaces the ExchangeScheduler binary that is called by Apple's scheduler hourly. It has a list of all network locations it should sync in, checks reachability of the outlook web access server and starts the ExchangeScheduler binary then.

Save the following as ExchangeScheduler.sh in
/System/Library/SyncServices/ExchangeConduit.bundle/Contents/Resources/

#!/bin/bash

syncLocations='73A6408D-CE03-11D8-B5C8-000A95D370A4 26415746-FF16-11D8-B1C7-000A95D370A4'
#syncLocations is a space seperated list of network location id's, the script should sync in.
#You can get your id's by entering scselect into Terminal.app.

ExchangeScheduler='/System/Library/SyncServices/ExchangeConduit.bundle/Contents/Resources/ExchangeScheduler.bin'
activeLocation=`scselect 2>&1 | egrep '^\ *\*'| cut -c 4- | cut -c -36`
for i in $syncLocations
do
	if [ "$activeLocation" == "$i" ]
	then
		if curl -k `defaults read com.apple.IISSupport IISServerPath` >/dev/null 2>&1
		then
			$ExchangeScheduler $*
		fi
	fi
done

Open Terminal.app and do the following commands

#change current directory:
cd /System/Library/SyncServices/ExchangeConduit.bundle/Contents/Resources/
#rename ExchangeScheduler to ExchangeScheduler.bin:
sudo mv ExchangeScheduler ExchangeScheduler.bin
#create symlink from ExchangeScheduler.sh to ExchangeScheduler:
sudo ln -s ExchangeScheduler.sh ExchangeScheduler
#grant execute permission to ExchangeScheduler.sh
sudo chmod +x ExchangeScheduler.sh

Test the script by activating a network location that is in $syncLocations and do

./ExchangeScheduler

If the Outlook Web Access Server is reachable the output should be

2004-09-05 10:11:51.302 ExchangeScheduler.bin[14242] Starting exchange sync

Adium X with fancy contact list

2004-08-31T09:42:00Z

The upcoming 0.64 of Adium X introduces an all new contact list, which allows some really cool features. There are a few early screenshots on the Adium homepage that show almost all possibilities.
You can grab my latest Adium X 0.64-pre build (checkout 6459) here.

Hidden Exposé window fixed in Firefox and Mozilla

2004-08-31T08:49:00Z

Finally the Mozilla guys fixed the hidden window on Mac OS X 10.3 which appeared when activating the Exposé functionality.
I made a build (hope it works for you) from the latest CVS checkout with the middle-mouse patch applied, which fixes middle-mouse click behavior on links to open a new tab. So actually two nasty bugs in Firefox on Mac OS X less :)