XP unable to use Samba after recent Windows Update KB2536276

On June 15th, we started getting calls from clients using Samba that had XP workstations unable to connect to shares. These were established clients running Samba 2.x. It didn't seem to affect our inhouse Samba systems, whether Samba 2 or Samba 3.

The inability to connect with the Samba shares was correlated to a Windows XP update KB2536276. This update includes changes to the SMB protocol.

The problem was tracked to Samba systems sharing with unencrypted passwords (I don't think it matters what kind of share security you use, e.g. SHARE or USER). The Samba variable "encrypt passwords" may have defaulted to "No" on older versions of Samba. In any case, the fix is to enable encrypted password shares, and making sure that the secrets.tdb file exists.

To do this, find your Samba bin folder. You should be able to run "ps -ef | grep samba" to and look at the running processes to find both your smb.conf location and the samba bin folder. Run testparm and look for the "encrypt passwords" line. If it says "No" then edit smb.conf to make "encrypt passwords = Yes" in the GENERAL section.

Newer versions of Samba should already have a secrets.tdb. If it does not exist, run "smbpasswd -a user password" to create it. Restart Samba and you should be all set.

Samba 3 on SCO 5.0.6

SCO OpensServer 5.0.6 came with Samba 2.2.2, and you could update to 2.2.8. With the arrival of Vista, Microsoft began defaulting to NTLMv2. There are tweaks (Vista here, Windows 7 here) to allow workstations to connect to Samba 2.x, but ideally you would want to upgrade to Samba 3.x. There is not a release supplement from SCO to do this, but you can install Samba 3.0.14Aa via Skunkware.

On SCO OpenServer 5.0.6 we'll need the following patches (in this order of dependence):

• RS506A - Release Supplement for SCO OpenServer Release 5.0.6 (ver rs506a)
• OSS646C - Execution Environment Supplement (ver 1.2.0a)
• OSS663A - LPD supplement for OpenServer 5.0.6 with OSS646
• gwxlibs-2.1.0Ba - Supplemental Graphics, Web and X11 Libraries (ver 2.1.0Ba)

The gwxlibs-2.1.0Ba update is available from ftp://ftp.sco.com/pub/openserver5/opensrc/gwxlibs-2.1.0Ba/
The OSR 5.0.7 version of Samba 3.0.14Aa is available from ftp://ftp2.sco.com/pub/skunkware/osr5/vols/samba-3.0.14Aa-VOLS.cpio

You'll want to create a directory (say /tmp/samba3) and download the .cpio file there. If you are not familiar with CPIO files, you can examine the man page, or just run this command to produce the VOL files:
cpio -idmv -I samba-3.0.14Aa-VOLS.cpio

I'm going to assume you know how to install VOL files in SCO software manager. All four (rs506a, oss646c, oss663a, gwxlibs) can be installed as media images. For the existing Samba 2.x, I usually just disable it in custom rather than removing it, but YMMV.

The startup files are /etc/init.d/smb and /etc/init.d/nmb. Samba looks for /etc/samba/smb.conf for the configuration. You should be able to backup your original configuration and drop it right in for Samba 3.

Be sure to check /etc/rc2.d for startup files. I create symbolic links to the startup files like so:
ln -s /etc/init.d/smb /etc/rc2.d/S99smbd
ln -s /etc/init.d/nmb /etc/rc2.d/S99nmbd

You can reboot the server, or just run the the rc2.d scripts:
/etc/rc2.d/S99smbd start
/etc/rc2.d/S99nmbd start

I looked for a Samba 3 solution on SCO 5.0.6 for a client that is was having streams memory leak issues, and I'll have to wait to see if that helps upgrading to Samba 3 helped. Neither SCO's knowledge base nor Google was terribly helpful in the memory leak or running Samba 3 on 5.0.6. Frankly, I don't know if Samba 3 would break something else (such as CUPS / LPD). But, Samba 3 is running on my SCO 5.0.6 system now.

UPDATED (04/23/2010):  I added a note about installing OSS663A. Also, it did help with the streams memory leak on that SCO OpenServer system.

UPDATED (11/14/2011):  Updated the links to Samba3 on Skunkware and the patches. Expanded explanation on restarting Samba.