Tuesday, March 31, 2009

Worried about Conficker?

Are you worried about Conficker? It's set to go off on April 1st.

It only affects Windows computers, so if you're running Macs or Linux machines, don't worry about it. (Mac users, don't feel too smug, as the recent PWN to OWN demonstrated an exploit of a fully updated Mac in mere seconds).

There's a fairly detailed analysis at Windows Secrets. If you think you might have Conficker, try visiting http://www.symantec.com/ or http://www.mcafee.com/; if you cannot, you may have it.

To remove Conficker, visit http://www.bdtools.net/ and download BitDefender's single-PC conficker removal tool. If that doesn't work on the infected PC, download it from another and run it on the infected PC.

If you are a network admin, go get Nmap (see also this) and run the following command:
nmap -PN -T4 -p139,445 -n -v --script=smb-check-vulns --script-args safe=1 [subnet]
where subnet is something like -- you can thank Doxpara for that.

I would also suggest that you use OpenDNS. They have been blocking conficker since February, and there's more detail here. Seriously, sign up for OpenDNS if you are not using it yet; it's free and does more than just provide DNS. I've talked about them before, but not enough.

Monday, March 30, 2009

Sphincter Moments

I had a very tense morning on Friday, what my coworker refers to as a Sphincter Moment.

The Background
On Thursday, I was doing a server migration from UNIX to Windows. The UNIX server is getting really old and the hardware is no longer reliable.

The data transfers over without conversion, so I could simply send the files via FTP. For convenience, I usually collect the files into a TAR or GZIP file before sending the files. I got people out of the system at 5 PM, made sure last night's backup had ran OK, then created my TAR files for the transfer.

The Reboot
The FTP download wouldn't complete, so I investigated. The UNIX time was off by 80 minutes (no one applied the new timezone patch), and restarting TCP didn't seem to help. System uptime was 287 days, so I figured, "Well, I have two backups, I'll correct the time, reboot and see if that helps."

I flushed the disk controller's cache to disk via the sync command, then issued a shutdown and restart command. Twenty minutes later, I still can't log in and I can't ping it locally... and my contact person on site answered the phone but just left (argh).

The next morning, the client discovered that the server room was locked and nobody had the key. After getting the vice president who had the key to unlock the server room, he turned the machine back on, followed some onscreen instructions, and we were back in business.

"Oh Crap"
I went to my backup folder to try a transfer before they got busy... and my files were gone. The timezone was still wrong, and some other files and changes I made last night were missing, too. I checked the MAC address to make sure I was on the right system, but it was like someone erased all my work. "Weird," I thought, "someone deleted my changes? Oh well, I'll re-do the backup."

Then came the call, "Hey, Lee, we have nothing in our dispatch board. No appointments, period." Then the call from accounting that the fiscal year was wrong. And as I looked at the system logs, I noted that there was a huge gap between 09/25/2008 and 03/27/2009, as if the server had been off for six months. Even our databases were the same way... no data was entered after the morning of 09/25/2008.

I restored the backup from tape (as my TAR backups disappeared with everything else), and the client lost a day of work. Fortunately, it was a light day and they could re-input their data quickly. Out of the IT director, the CIO, CTO, my manager, and myself, I was the one most bent out-of-shape over the whole thing.

What Happened
We pieced it together after the fact. The IT director is new as the previous one left without warning and without leaving any passwords or configuration information. He knew the UNIX server was having problems, but not what problems.

Nobody was monitoring the RAID array, and it is a RAID-1 with a LSIL on-board controller. The LSIL had a neat feature, in that if the array is broken, both the online drive and the offline drive retain data, and the array can be rebuilt with either drive. This allows you to build an array from existing drives without losing data on the drive you copy from.

Well, DISK 0 probably went offline in 09/2008, while DISK 1 kept working. When the server restarted, it either 1) asked which drive to set as primary and defaulted to the old data on DISK 0, or 2) defaulted to DISK 0 without asking. In either case, the drive from 09/2008 became the primary, thus causing a six-month black hole.

The Moral
The moral of the story? Make sure you have backups, make sure you have OFFLINE backups, and make sure those backups can be accessed and restored -- test your backups! Thank God Almighty that the backup restored OK, or that client would be out of business within months. *whew*

Wednesday, March 25, 2009

NTFS event ID 55 backing up with Volume Shadow Copy (VSS)

OK, I have this client that had failed backups and event viewer showed NTFS errors with event ID 55. The event log entry suggested that I "please run the chkdsk utility on the volume \Device\HarddiskVolumeShadowCopy1".

I performed the obligatory searches of Microsoft's KB database and asked Google, and the closest I got to an exact match was an unanswered call for help on Symantec's forums.

Here's the symptoms:
Backup failed and the indicated the following errors:
Could not access portions of directory System State\COM+ Class Registration Database.
Could not access portions of directory System State\Registry.
Event Viewer showed NTFS errors with event ID 55.
I could make a copy of unopened files.
"VSSADMIN LIST WRITERS" showed no errors.
CHKDSK /F showed no errors.

EventID.NET had a suggestion but it didn't seem to apply. Microsoft has a whole bunch of articles about VSS. The most drastic suggestion was 'do a backup, format and restore' -- except, of course, I couldn't do an online backup.

Anyway, in the end, here's what worked: I emptied the recycle bin, ran disk cleanup, and everything was fine with NTBackup again. Go figure.

Sunday, March 08, 2009

Boaz is a key thief

Boaz, our Russian Blue, is a key thief.

A couple of weeks ago I left my laptop on overnight to compress some video. In the morning, I found I was missing a function key and several other keys were loose. I figured the cats were roughhousing and ran on my laptop, dislodging keys. "Oh well," I thought to myself, "I'll just cover the keyboard from now on." I found it later on the carpet and reattached it, no problem.

The same thing happened to Cathy's keyboard a few days later. Then, it happened again to me, except this time someone actually moved the cover from the keys first.

Then I witnessed it; I actually saw Boaz walk up to Cathy's laptop, angle his head and pry up a key with his teeth, then run away. As weird as it sounds, my cat steals keys off of my laptop keyboard.

Friday, March 06, 2009

Solar-powered lighting at IKEA

We've been thinking of getting some outdoor lighting for the RV. IKEA is beginning to offer solar-powered lighting. Hm... I think we'll hit IKEA this Saturday...