getluky.net

Situation:

Extracting data as text from a PDF is never as simple as it first seems. For this project, the data in the PDF, after being extracted, had no natural delimiters except for spaces. However, some of the columns contained proper nouns for cities, which would mess up Excel when going to do a data import from the extracted text.

Starting with a script I found to clear out non-numeric values in a single column with VBA, I came up with this:

After importing space-delimited text, you will naturally be left with an excel document that has unevenly distributed data. If the data you want to get to is purely numeric, then you can run this script.

It jumps through each row, shifting all numeric data left so that only separate numeric values appear on each column. If only numeric data was desired, this should do the trick.

The data set I worked on was California Department of Public Health Service Planning Area by Zip Code data. I’m thinking of making a form for anyone to type in their zip code and get back their SPA, but i’m not sure if it would be in the public interest. If I get any comments that it’s desired, i’ll be happy to put it up.

Uncategorized

I’m temporarily helping out Andy’s Amateur Tsunami video coverage along with Leonard Lin, Nathan Perkins, and Ask Bjorn Hansen, by hosting the videos on a round robin with those guys. When I checked my bandwidth this morning at 6 am, I was at 133GB, and now, at 9 am, i’m at 188GB. That’s a monster load of bandwidth, and i’ll probably have to take down the videos tomorrow to avoid blowing my entire January allocation in the first week. In the meantime, i’ve followed Leonard’s advice by tuning up my ServerLimit and MaxClients directives to 400. It seemed to make my site much more responsive in the meantime.

If you’re looking for places to donate, CNN is maintaining a list of charities. There’s a family in my building that is shipping aid materials directly via a shipping container they have access to, so i’m going to use that method, but I would highly recommend that everyone tries to help out somehow.

Not only is this a tragic loss of life due to natural disaster, it also has the potential to become an immense public health issue that could result in worldwide problems. One of the public health professionals I know commented that her colleagues are already very concerned about what’s going on, even though it’s far away from our world in the US. Not only is it crucial for medical aid to reach the injured and weakened, but the amount of dead that the local governments have to deal with poses a potential disease problem as well.

Uncategorized

Earlier this year, I manually typed in the Office of Management and Budget’s publicly-available FY2005 Historical Budget PDF into Excel, in order to analyze military spending over the past six decades. It struck me as pointless to provide data in PDF only form, and I felt like the government was unnecessarily obfuscating important data from public consumption.

Until today. I had just about given up on this, and was about to go find some PDF to Text conversion tools, then put up a public database of the data so that others could do analysis as well. Then, I stumbled upon the motherlode of government historical budget data, already provided in Excel format.

So, without further ado, here you go:

http://www.whitehouse.gov/omb/budget/fy2005/hist.html

http://www.whitehouse.gov/omb/budget/fy2005/db.html

To all you aspiring data-crunchers out there: this should be enough to get you started with making simple excel graphs to compare budgetary data series over a long enough timespan to actually make some sense of what goes on in Washington.

However, if you were to be provided with a public database that was truly query-able, would you use it? What interface would you prefer? What would really make this data useful to you in your own policy research?

I’d be very willing to take on such a project to get my feet wet with this kind of thing. Please do leave a comment and i’ll take them into consideration whether I should go ahead with something like this.

Uncategorized

In response to the recent coverage in Washington Post and Waxy Links, I wanted to post what I remembered about Kings of Chaos and its predecessors.

One of the primary reasons that Kings of Chaos got so popular so quickly is that it originally “featured” copyrighted images of Lord of the Rings characters, right in the midst of their theatrical releases. This was before KoC’s “Age 1″, which I suppose could be considered a beta period? In any case, they soon disappeared, but not before most of my personal friends had signed up and begun linkspamming the thing around.

See Vigo Mortensen for yourself here through a Jan 18th cache at archive.org: http://web.archive.org/web/20030118181556/http://www.kingsofchaos.com/. Unfortunately, it looks like the rest of the images may be broken, but Vigo does appear, and I clearly remember the rest of the images being LotR characters.

I believe they were present for a couple of weeks while the linkspamming was rampant enough to generate critical mass, then they switched to generic ‘flags’, as you can see here from another cache on Jan 26th: http://web.archive.org/web/20030126121826/http://www.kingsofchaos.com/

There were several other website-based MMORPG’s going strong before that point (anyone remember the name of that gangster-based one? i can’t recall at the time and don’t have my chat logs around).

For each of these services, the value of the game is caused by a large, active player base. Getting to that critical mass determines whether it is destined to suceed or fail. I don’t blame them for using LotR imagery to get started, because it was no doubt very important for generating interest and participation in their particular fantasy world. However, I think the use of these characters to get people registering for the game was a strong factor in the unique success of KoC.

Uncategorized

So I happened upon this guy John La’s deals site the other day, in which he does something I wanted to have for a while.

Mother of All Deals aggregates deal listings from some of the heavy hitters, and puts them all together in a searchable, RSS-enabled way. He’s listened to my suggestions and incorporated them quickly (3-letter search indexing, etc.). Have fun!

Uncategorized

Some notes on currency formatting, in particular adding decimal commas and automatically getting the right decimal character with printf / sprintf:

The Perl Cookbook has a good ‘commify’ starter script that can be modified in order to pass in a separator and decimal character. Just remember that in regex, you might need to add a slash in front of the character to use vars in the pattern matches.

http://www.unix.org.ua/orelly/perl/cookbook/ch02_18.htm

• (Perldoc on the lookahead and lookbehind operators used within)

But before you commify something, you should use Perl’s setlocale (man setlocale) function before your printf. There are multiple LC* locale settings that you can use, but for currency, you’ll want printf to automatically use the right decimal point formatting based on the LCNUMERIC locale.

The locales installed on your box can be found by using locale -a. Personally, I find it useful to store the ones you want to support in a hash, indexed by your preferred name for each.

Unfortunately, the current locale setting is global (not sure how global, though… any mod_perl experts want to comment?), so we’ll store it in a temporary variable during our calculations. I am worried, though, that if it’s too global, this code might not be thread-safe. Got to research that further.

Therefore, to store the old one, you can do:

In mason, you can stick most of the code in the <%init> block, whereas the line that sets the locale back to the old locale can be stuck in the <%cleanup%gt; block.

That should do it!

Uncategorized

I disagree with several pieces of advice on Bruce Schneier’s latest posting, that makes some suggestions apparently targeted to everyday PC users.

In my opinion, some of these suggestions can cause more problems than they attempt to solve. As an IT calvalryman, I often get called in to help out in situations where users who know “just enough to be dangerous” (your self-professed “Power Users”) carry out practices like these and get hopelessly lost in the muddy waters of PC self-maintenance. Let me preface this by saying that I subscribe to Bruce’s blog, and appreciate a lot of what he has to say. It’s just that this particular entry raises my hackles, and I feel compelled to comment.

First of all, if you’re a home PC user without a sysadmin, I feel for you. I don’t think you have much of a chance of surviving without getting your computer infested with spyware. Still, some of these measures are a shot in the dark, and shots in the dark often hit unintended bystanders.

1. When Bruce says “purge unneeded files”, he doesn’t mean for you to go searching through your computer deleting DLL’s or directories that you don’t understand. It’s hard to keep track of everything useful on your PC, especially if you don’t keep track from the very beginning. If you don’t specifically know that something is harmful and isn’t there with your permission, you shouldn’t delete it. I’ve run into far too many people deleting DLL’s from SYSTEM32 and entire directories from Program Files\ without realizing the consequences of not using provided uninstallers. It can break your TCP/IP stack to the point of no return, it can prevent your machine from booting, etc. No offense, but if you don’t know exactly what you’re doing, don’t delete stuff.

2. This goes for COMMAND.COM and CMD.EXE as well. When you screw up your computer by deleting random stuff, these are the tools that professionals will use to help debug and fix it. Deleting them doesn’t make you any more secure, in my opinion.

3. Don’t put CD-R’s in your microwave or a document shredder. I don’t know why Bruce would expose himself to potential liability by advising something like that, but he did, and you shouldn’t. You can cause fires or permanently damage your shredder.

4. I agree that if you’re a home user, a Mac is probably going to require a lot less maintenance. If you need to use a PC, budget some money to get a knowledgeable friend to come and look at it once in a while, and take them out to lunch.

5. Most home users don’t even know what Internet Explorer is. So advising them not to use is it is pretty confusing. A better piece of advice is, if you don’t like popup windows, go and download the Google Toolbar at http://toolbar.google.com/. Better yet, if you understand fundamental browser concepts, go download Firefox at http://www.getfirefox.com, and set it as your default Internet Browser the first time it runs.

6. Passwords. Users don’t pick long, randomly generated passwords, then memorize them. It’s better to provide advice to include at least a two-digit number, along with two or three CamelCase words in their passwords.

7. If you know what SSL is, you’re ahead of the game. For the common user, though, never go to websites that are linked from email. There are bad people out there sending spam with fake websites that look all too real. Before you enter in your password somewhere, make sure that there’s a little “lock” icon in the bottom part of your browser.

8. On firewalls, I agree that everyone should own a little home broadband router that can act as a firewall. But they don’t come in the stores labeled as “Network Address Translation” devices. They’re marketed as Broadband Routers, and you should probably have an aforementioned nerd-friend help you out. If you’ve already bought them lunch, by them a beer next.

9. As far as I know, mentioning a “secret police” in the same paper that contains reasonable advice to home users is a little weird. Home user “best practices” should be separated from “security paranoia”.

10. Don’t find just any spyware-removal tools you can. Two in particular that nerds everywhere recommend are Lavasoft’s Ad-Aware, available at http://www.lavasoftusa.com/, and Spybot Search & Destroy, available at http://www.safer-networking.org/en/index.html.

11. A note about uninstalling applications. Please don’t uninstall everything on your computer. A lot of people see things like “Windows XP Hotfix (XP3418481047)” and think it looks suspicious. Again, if you don’t know that a program or file is specifically malicious, please don’t uninstall it. Ask someone who might know.

And now, a few of my own best practices that keep my own home computing safe and sound:

1. If you use your PC for work, don’t use it for play, or allow family members to use it for play. Personal experiences show that the most common ways viruses, spyware, and other bad stuff gets on people’s PC’s is through “play” stuff. This includes downloaded screensavers. Screensavers are NOT SAFE. This includes MP3 downloading programs, like Kazaa, Gator, and ALMOST ALL OTHERS. This includes “Joke” animations that friends appear to send to you. This basically includes almost anything you might consider doing for fun on the internet. That’s why it’s so hopeless to try to protect the average user - the authors of spyware, viruses, etc. know EXACTLY what most home users like to do, and then bundle their stuff along with the “Fun & Games” type of software. That’s why you never get popups after installing Turbotax.

2. Get someone who works as a professional sysadmin as a friend. Treat them nicely and bribe them to come have a look at your PC every once in a while. They’re used to it, and the fact that you actually compensate for their time will be appreciated. Not all sysadmins are good, but they’re probably decent enough to know how to keep your PC relatively clean.

3. Software firewalls lead to a false sense of security, and often cause more harm than good. They often pop up unintelligible alerts that home users have no chance to understand, which leads to a lot of weird problems that are very tricky for your knowledgeable PC friends to help out. The first thing I often do when helping out is disable software firewalls anyway to try and establish root causes of connectivity problems. Get a hardware broadband router that also acts as a limited firewall. It’s a much better solution to the problem.

I hope Bruce doesn’t mind that I disagree with his advice on most of these things. It’s just that home users have a much greater chance of causing complete havoc when following some of these suggestions.

Uncategorized

In a post I made a few days ago, I referred to a trackback that seemed halfway relevant, but set off alarm bells in my head because of the url: viral-memes.info. I followed the trackback and found a halfway related topic on a very weird website, but no link back to my blog that would have indicated the trackback occurred because of a mention of content.

Not only was there no direct link, there was no mention whatsoever. There was a link to a separate blog. I don’t know if it was malicious or not, but the act of pinging a trackback url manually after searching for related posts on “media mammon” takes some effort. If the guy wrote me and told me that it was a mistake, I would understand, but think about the trackback spamming implications. If you are a comment spammer, why not find a popular link topic from technorati, quote a bunch of bloggers, then hit everyone’s trackback with your text, hidden in a black background (or even out in the open) just to get people clicking to your site. You’d probably fool most people doing cursory summary checks in their trackback logs.

Still, i’m not sure that spammers don’t really worry about blacklist users or careful watchers of their trackback logs… it seems more the opposite, that those of us who prune comments are so few that it’s still a worthwhile business model to increase pagerank of a spammer’s site.

In my view, trackbacks are useful because they help to make blogger’s contributions to knowledge and opinion cumulative. Just using them to direct traffic towards your single, isolated blog without a related link out doesn’t make any sense to me.

Uncategorized

I found this pretty entertaining. If you go to the official Triumph website, http://www.triumphtheinsultcomicdog.com/, then click on “Fansites” on the bottom, you will see a link to http://www.insultcomicdog.com. Follow the link, where you will see the following notice:

Got a message from the company that airs Triumph saying that I have to remove all Triumph stuff. So in the words of my generation — “F#%! NBC”

In fact, I found that two of the sites linked had this kind of directly noted C&D, whereas two more were dead geocities style pages. Pretty funny to sic the lawyers on your own links. Could be, that’s how the lawyers found them in the first place.

Uncategorized

So I noticed that the guy at viral-meme.info wrote a piece about media mammon, and somehow trackback pinged my post about it. However, there’s no link to my site. I think what he’s doing is crawling weblogs and pinging their trackbacks to generate more traffic to his site without including referring links. What a jackass thing to do.

At first, I was confused and couldn’t find the link. Then, I realized, there was no link.

I already deleted his trackback and also added viral-meme.info to my mt-blacklist. I would recommend that you do the same.

Uncategorized