Tying it all together

All applications fit neatly into one of two categories - those that repackage easily and those that are close to impossible. For the past 2 months I’ve been dealing with one of the latter, ArcGIS by Esri, a geographical information system. Basically, it’s a suite of apps for working with mapping data that’s split into many components.

The problems with packaging aren’t the fault of Esri at all. In fact, unlike a lot of companies, an MSI is provided on the disk which would normally make life easier. On an RM network however, the procedure is a little different from a vanilla deployment. Instead of using active directory to deploy the MSI directly, you have two options:

  1. Set up a “package build” workstation  - essentially a computer with a minimal number of packages. You then use the RM Application Wizard to perform a “before scan” of the station, install your software and then run an “after scan”, capturing the changes and creating an MSI file. On the plus side, this lets you change program settings before deployment which is a nice touch. One of the problems I often run into is that occasionally changes are missed, which makes some software think the install is corrupt. The other common problem is that user settings are often pulled into the MSI and during deployment these are sometimes unable to be set and the install will fail.
  2. The Application Wizard is also able to accept MSIs directly, repackaging them on the fly and generating shortcuts and other required files in the RM package format.

Unfortunately, both of these methods failed for ArcGIS. I suspect the software is just too big and complex for a simple repackage job - a basic install is over 1.4GB, with files and registry keys all over. That’s 1.4GB of software - no maps, no data! The MSI deployment would always fail with an MSIInstaller error 11606, complaining that it was unable to find “WindowsVolume%\”.

I decided the best way forward would be to somehow install the original MSI unedited. The RM system does support this by way of its package.ini files. Using these, you can set EXEs to run and MSIs to install. What I couldn’t do was pass the /qb switch to the MSI in order to do a typical, silent install. It was time for some scripting…

The workaround was to tell the package manager to execute my custom batch file, which itself called MSIInstaller /I setup.msi /qb after verifying the sizes of certain installer files. Frustratingly, when the batch file finishes, it seems to return a code that the RM package manager interprets as a failed install! Luckily, though there’s a switch in the package.ini file to ignore return codes and this seemed to do the trick. Big thanks to whoever at RM implemented that feature!

I also had to write an uninstaller batch file, again calling MSIInstaller with the correct product code, since the RM system now has no record of the particular MSI that has been installed.

In an ideal world, that would be it but I also had to install a service pack that was supplied as an MSP file. I couldn’t use method 1 above to package it because I couldn’t do the original install that way and method 2 doesn’t support MSP files. Neither, it seems, do the RM package.ini files, as you can only specify an MSI to run, not an MSP. Batch file time again, this time executing the MSP in the same method as above. Finally, a working installer that brings together the two worlds of vanilla MSIs and the RM system.

It took a little longer than our usual 2 day turnaround on software packaging but I’m pleased with the result, from now on it’s quite simple to install generic MSI files with just a little batch editing. Don’t get me wrong, the RM system has its advantages but this method is sometimes much easier than trying to work around strange repackaging errors.

A Day Out

Took a lovely trip to London the other day with Rach. After Oxford Street we went to Leicester Square and found this little café that does Italian food along with the most enormous ice cream creations! Take a look at the yummy photos.

After lunch we headed to the IMAX cinema down by Waterloo, something I’d heard a lot about but never seen for myself. The screen is just staggeringly huge (puts Stevenage’s cinema to shame!) although when the adverts were running they only took up a fraction of the whole screen and I thought we were in for a let-down. Not at all, I’m definitely going back there again!

That evening we took an ‘Eye flight’, as it’s called now. I’ve lost count of how many times I’ve been on the Eye but it’s still great fun. Luck was definitely on our side since the eye closes at 9 and we just bought our tickets and made it there in time, the last two people allowed on! I’d like to think we were let on because of my sweet smile but it was more likely the fact that we were running towards the queue looking desperate.

Here’s a few pics…

Engadget are missing the point of WWDC

Did Engadget get up on the wrong side of the bed this morning? Have they gone to the wrong conference? I love their live coverage of the event but some of the comments this year are a little snide. Take a look:

And our LAST app [emphasis ours] comes from Digital Legends Entertainment…

ANOTHER medical app. This one by MIMvista. Someone, wake us when Steve’s back.

Forstall back, and another developer demo. Ugh.

Man, please let this string of demos end!

Man, these demos are crazy boring. Throw us a bone here Apple!

Okay, so everyone’s really waiting to see the new iPhone of course but please, Engadget, it’s almost as if it’s a conference for…developers. Sort of like the WorldWide Developers Conference, no? Of course there are going to be loads of app demos, that’s the idea. And as a developer, I like to see them!

Steve Jobs at WWDC 2008

This sort of commentary frustrates me, not because I’m an Apple fan but because it’s damaging to the credability of bloggers as a whole. I was hoping that people would have raised their game after the Gizmodo incident at CES yet these little jabs detract from the professionalism of a great news source.

ThinkGeek and the lure of the 8-bit tie

Sometimes you come across things that are just so quirky, you can’t help but buy them. I finally caved and got myself one of these funky 8-bit ties from ThinkGeek. No, that’s not me modelling. As luck would have it, they’re running a buy one, get one free promotion at the moment so I have two of these retro wonders heading my way - Mario would be proud!

The 8-bit design is a nice throwback to the days of the NES and makes for something a little different around the office. Not wanting to get stung too badly on shipping ($36 to ship a tie is a bit pricey!), I got some other assorted geekery including some highly caffienated candy, a multi-tool, Timmy the ThinkGeek monkey to sit on my monitor and a couple of other bits.

I really wanted to see what all the fuss over Jolt Cola is about as well because I’m a sucker when it comes to shiny packaging - you can see why they’re called Battery Bottles! But sadly it can’t be shipped internationally. Why? I assume it has something to do with pressurised liquids but I’m not sure. Either way, it means no Jolt for me which, considering the 220mg caffeine content per can, is probably no bad thing.

Finally, a proper photo gallery

It’s been a long time since I updated the gallery. I must have literally thousands of photos lying around on my laptop and at least some of them are worth putting online! :P

Now that summer’s rolled around I’m gonna get back on track with my photography as well. My poor old camera has been sitting, gathering dust for most of the winter and it’s about time I dusted it off and went on a trip somewhere. With half term coming up, I’ve no excuse!

For the moment, though, I’ve gone and added about 400 photos from my collection which go back a good couple of years. Who knows, you might even be in one, say cheese! 8)

EduSweep Beta 6 Released

After around 3 weeks of refining code, I’m happy to announce the final beta release of EduSweep. This is the first build that is of proper release quality and from now, only major bugs will be fixed before 1.0.0.

The following bugs have been fixed in this release:

  1. Large sets of results (> 5000 results) cause the results viewer to take a few seconds to appear
  2. Scan engine may fail to recover after multiple inaccessible folders are encountered
  3. Results viewer controls behave properly when selecting multiple items and only some have been deleted or quarantined
  4. A proxy authentication error is encountered while updating the definitions
  5. Two minor bugs that were not fatal but could slow down the scan process

    There are also some notable improvements, some of which I described in detail in the previous post:

    1. The responsiveness of the results viewer has been improved while loading information for files
    2. Online update has been overhauled with a scan log and better handling of proxies
    3. Log files are now written in HTML format and give more detail about scan errors

      Go ahead and grab the download! :)

      EduSweep Beta 6 Changes

      Having taken the overwhelmingly positive feedback from Beta 5 testing over at EduGeek.net, I’ve been hard at work on the next and final beta build. Let’s have a look at what’s changed in this version as it nears release:

      New logging features in the online update module

      More robust definition updates

      The online update feature has had a major re-write, bringing a realtime log on-screen and an error log to help trace any issues. The proxy handling code has been updated and should fix the issue some people were seeing with their Squid proxies and NTLM authentication.

      On startup, the installed definitions are now verified so that if any definitions are missing or damaged, they will be redownloaded as part of the update. The definition files are now also tested before a new scan begins.

      Log files are much, much better

      Previously, log files from a scan would be saved with a rather cryptic filename as text files in the ‘logs’ folder. The actual content of a log looked something like this:

      #### Scan Summary ####

      Scan Date: 10/04/2008 12:02:07
      Files Scanned: 518
      Errors: 0
      Detected Items: 4

      #### Detected Items ####

      \\nbl-sr-02\student$\My Settings\Favorites\proxy\Freedom Proxy.url
      \\nbl-sr-02\student$\My Settings\Favorites\proxy\The OpenDoorNetwork -
      Unlock The Internet With Us! Proxy Sites For Work and School - Visit
      MySpace, Bebo, Facebook and MORE!! F.url
      \\nbl-sr-02\student$\My Settings\Recent\HACKER.lnk
      \\nbl-sr-02\student$\My Settings\Recent\pc virus.lnk

      Scan completed in 2 seconds.

      I’ve been hard at work upgrading the logging code and it’s now able to write (100% valid) html log files, which are just so much easier to work with - now that you can load them in a browser it’s possible to print them with proper formatting and word wrapping, as well as perhaps placing them on a web server for easy access anywhere.

      There’s also some new information that’s written to the logs. First, the scan summary now includes the user that started the scan and the time taken. The scan summary looks like this now:

      Scan date: 06/05/2008 11:49:53

      Scan requested by: paulb

      Scan took 11 minutes and 14 seconds to complete.

      Number of files scanned: 144150

      Number of files detected: 166

      Second, and more importantly, the logs show detailed information on errors instead of just a count. Some of the more common errors you may encounter are:

      • The specified path, file name, or both are too long. The fully qualified file name must be less than 260 characters, and the directory name must be less than 248 characters.
      • Access to the path ‘C:\System Volume Information’ is denied.

      Both of these are normal on a standard filesystem. The second is often triggered by the scanner trying to access system folders and it will continue just fine after skipping them. If a folder is skipped that definitely isn’t a system folder, check that your user account has read access.

      The results viewer is much quicker

      Notice the amount of results and the progress bar for the thread

      It’s hard to design a viewer that has to handle several thousand files and stay responsive. Beta 5 could slow down quite a bit if your scan returned a lot of detected items. I’ve added some threads to make loading data and working with files feel much more responsive throughout.

      The progress bar bottom right shows the progress of gathering data for the files and populating the list.

      Framework folder fudges in .NET

      I love the .NET Framework. I mean that purely in a platonic sense, of course - it’s not *that* good. :lol: As with all platforms though, it falls down in certain areas. Java has its slightly quirky date APIs and I think one of .NET’s quirks has to be folder handling.

      Back in my day…

      Like many people, for better or for worse, I started out programming in VB6 around 2003. I’d managed to write some fairly decent programs - decent more in functionality than code quality, I admit. When Visual Studio .NET 2003 came out, I started learning to program in C# on the then relatively young .NET framework. I was having so much fun, in fact, that I asked for Visual C# Standard for my birthday!

      Back then, the .NET framework made learning a new language much easier. There was a nice set of standard libraries, automated garbage collection and I found that a lot of the methods I was writing in VB were included as standard in the framework.

      Rose-tinted specs

      Recently, though, the framework has been getting on my nerves in a pretty big way. Sometimes the way it handles basic operations is odd, unreliable or just downright broken. One such problem is handling folders.

      EduSweep, my network sweeper, scans entire drives for unwanted files. The process it uses goes something like this:

      • Get the list of targets that the user selected
      • For each target, get all the subfolders and make a list of them
      • For each folder in the list, scan its files

      And that’s pretty much it; it’s not very complex, nor does it need to be. The framework has other ideas, however. By far the simplest way to get all the subfolders of a path would be to use GetDirectories like so:

      GetDirectories(“*.*”, System.IO.SearchOption.AllDirectories);

      That initially looks fine but let’s assume that we have a folder somewhere on the drive that is inaccessible due to insufficient permissions. A good example is the system folder “System Volume Information” in the root of drive C. The code attempts to get the subfolders for the folder but fails because the directory listing is denied.

      We can wrap that code in a try block, catching the exception, but then no results are returned at all! The method is essentially useless because you know that, as soon as it encounters a single folder it can’t access, it throws its toys out of the pram. The only alternative is to manually recurse through the directory tree, enumerating folders and subfolders along the way. :mad:

      Would it be so hard to skip folders along the way and return a string array of failed directories?

      EduSweep 2 Preview

      EduSweep is just approaching release but I already have big plans for the next version. Here’s a look at just 3 of the many new features currently in development:

      Scan Profiles

      Let’s say your users are split over 5 servers and you scan them once a week. You shouldn’t have to babysit the process, adding the scan targets each time, just set up the scan how you like it - choose the definitions to use, the targets to scan and the action to take - and press save. The next time you want to scan those servers, all you need to do is select the profile and away you go!

      Integration with TRID

      TRID is an impressive project, looked after by Marco Pontello. It identifies a file’s type by matching patterns against a known list. For example, TRID detects MP3 files based upon the hex pattern 49 44 33 in the file header.

      For EduSweep, this unlocks a great deal of power. Your students have renamed all their WMVs to TXT files? No problem - the files will be detected and flagged by the TRID library in EduSweep. Truly intelligent file identification will be a great asset to the software and could well pick up on malicious files you never knew you had on your network!

      The TRID library is free for personal and educational use but I’ll be contacting Marco about integrating it into EduSweep, just to be on the safe side. If I do use it, I’ll be sure to send a healthy donation his way.

      Speed, speed and more speed!

      Some of the latest code I’m working on involves multi-threading. EduSweep 1.x is already threaded to keep the UI responsive but the scanner itself is just a single thread. An experimental piece of code that I’ve cooked up tests the scan targets to see if they live on different computers or different physical drives on the same computer. Physically separate targets will each get their own scan thread to really maximise the scan speed for multiple targets and large networks.

      But wait, there’s more!

      Those are just a handful of the cool features I’m working on. I’ll post an update soon with perhaps a few more nifty additions. Watch this space!