getluky.net

It just occurred to me that RRDs might be really well-suited to some general purpose roles as counters, statistics trackers, etc. for use in webapps. There are statistical analysis tools built in, so I could potentially see it as a way to (locally) rate limit API requests, track certain types of errors, etc. It’s also very fast, writes back to disk, and already plays very nicely with a variety of graphing and monitoring tools.

I’ll do some experimentation soon and report back.

Tech

I was shopping online for a replacement for my bluetooth wireless mouse for mac, and I read that there was an alternative mouse driver that might let me tweak the acceleration settings to something more acceptable.

I ended up downloading and trying Steer Mouse, and it’s actually pretty good at doing that. You can bring down the “Tracking” and bump up the “Sensitivity”, and it won’t be as wonky. Qualitatively, there’s also less noticeable lag between mouse movement and actual input (something that’s been driving me crazy as well). It may hold up better in heavy application use as well, time will tell.

However, one thing I wasn’t willing to live with was the default remapping of middle click to “Move to Close Button”. To change it to work with Firefox new tabs again, you can remap it to the Click action with a apple-key modifier. Seems like that works again. I’ll let it go through the trial period and see if I still like it after some more use.

Tech

My earlier post about the costs of microformats led to some interesting comments that I’d like to respond to at the top level here.

In this comment, André Luís brings up his personal experience with ease of publishing hAtom, and also points out that it may have just been my publishing platform which gave me difficulty. He also mentions that i’ve overlooked added value for power users.

Shira believes that many of the problems I listed will go away with maturity of the microformats project, and are present because of its work-in-progress status.

Yes, it’s true that there will be some cases where publishing microformats will be a technically straightforward matter (especially with say, hAtom or hCard). I don’t wish to say that it’s always technically difficult, but what I’d really love to get across is that there are not only potential technical landmines, but also policy and relationship complications that can result from such a move. In the case of my publishing platform, which was, in my case, in-house written code, there were extreme difficulties in attempting to contort our markup to something that would follow microformat structure expectations in hCalendar and hCard. Additionally, publishing microformats resulted in ongoing maintenance, policy, and backwards-compatibility costs that were not considered at the get-go. No matter what method you use to reach out to developers or power users, you will need to pay the price to keep those relationships healthy, and your data strong. Microformats don’t make your job any easier, because you’ve just intermingled the methods you use to serve your general user base and your developer base. Haphazardly publishing microformats while keeping your developer API users happy is going to cause grief for some users, and will likely be interpreted as sloppy behavior.

I certainly have avoided discussing any of the potential value of publishing microformats, as the title of my previous post may let on. I find that nearly all of the information available solely discusses the benefits without spending due time on the costs. Furthermore, the benefits of microformats (as well as the potentially reduced technical costs that Shira refers to) are, from a pragmatic standpoint, hypothetical. As a writer, it’s my hope that by spreading more complete information, people can make more educated decisions, and I feel that what’s missing in the hype about microformats is the discussion of costs. So, I apologize if I don’t spend much time on the benefits, but I really feel you can get more of that information elsewhere, as long as you remember that those benefits are typically projected onto a vision of the future where microformats are omnipresent. I’ll just say that I don’t see many benefits that aren’t already available to publishers who conform to the existing standards that microformats are based on.

So, what is the likelihood of a world in which microformats are an accepted standard in XHTML publishing? What are the chances that your real, tangible costs will eventually be paid back by real, tangible benefits? How would you begin to evaluate or consider that likelihood?

Typically, standards succeed when there is a massive market need that has not yet been met. However, there already exist viable, well-known methods of serving the needs of both browser users and developers, so I don’t see why these large publishers should invest in serving the “power users” market in between those two, when these power users needs are so close to that of developers. The power users are often served by the standard data formats underlying microformats anyway. Also, the developers often take developer platforms and build products specifically targeted towards power users! So, I don’t see a large, unmet need. To me, it just seems like an product aimed at serving a niche user base that is already being served pretty well.

Also, there is the fundamental matter of data quality that drives the growth of a standard. If your data is questionable, nobody will want it. Or, they’ll collect it and throw it away. If your data is valuable, consumers will go to great lengths to take in and use as much as your policy allows. This natural magnetism that high-value publishers have predates microformats, and explains well why there are dominant platform APIs that don’t seem to play by the rules of standardization. Sure, there is a high technical barrier to entry, but typically, these producers invest heavily on data quality, policy, AND their developer relationships. Relationships between publishers and consumers around valuable data is what drives your average successful Web 2.0 platform, not whether the publishing format is standardized. So, if you want to really predict adoption of microformats, pay attention to the data quality of published microformat data.

If you want to consider the potential of momentum to bring about a microformat-based future, you’ll pay attention to the spread of adoption of microformats, and the hype surrounding them. As mentioned above, I would encourage you to also frame your perspective according to the data quality of data made available via microformats, but that requires a lot of time and energy to really evaluate. The vision here is pretty blurry; I think that’s just the nature of hype. It’s easy to get excited about something before the costs come into the picture.

Essentially, by publishing microformats now, you are becoming an early adopter. These standards are not mature, which Shira is essentially pointing out. The costs may one day be lower, but by getting into the game now, you are not only placing a bet that many others will follow, but also they are subsidizing the costs of publishing for latecomers. I don’t think that the current tangible costs outweighs the current tangible benefits plus the hypothetical benefits multiplied by the likelihood of them coming to pass. But don’t take my word for it. I’d just encourage you to at least make an educated decision, and consider all of the costs I listed in my previous post before taking the plunge.

Tech

I’m done with microformats. From now on, i’m either building separate developer tools and relationship, or i’m not. I say that having been through the cycle of adopting hCalendar and hCard a few times, not just as an industry commentator. My reasons are threefold. First, in the real world, publishing microformats requires you to rewrite DOM structures, publish extraneous invisible elements, adopt new schemas, and adopt data publishing-like structures on frontend pages intended for the browser. Second, the relationship between publisher and developer is not significantly improved by microformats, and would be better served by a separate pathway for developers. Third, its proclaimed benefits as a standard are extraneous because hCard, hCal, etc. are valiantly attempted, but incomplete redefinitions of existing standards like vCard and iCalendar in XHTML-like formats.

The idea of microformats sounds swell, inexpensive, and easy. Take your existing data, and surround the data-ish bits with tags that separate it into parts with semantic meaning. Unfortunately, saying that something is easy does not make it so. I don’t mean to disparage the work that microformats mavens have done, but my experience with being a microformat publisher has shown that things are exponentially more complex than they let on in the “sales pitch.” I don’t think they realized what they were getting into when they started on the process of actually getting publishers to conform to this stuff.

Surrounding your existing markup sounds simple enough, right? But consider how you’ll have to nest things together. Should you publish microformats on both listings and detail pages? Are there required fields that aren’t present in the content you already have on the page? Do you just publish lots of invisible XHTML content to the page to fill in the missing stuff? Do you dare to deal with recursions? What if your data is split up in separate divs? What happens when your data model does not fit with the standard? What if you are not a very good HTML contortionist? What happens when your presentation is different based on varying pieces of available data? Is hCard publishing even useful at all on public pages without private, uniquely identifiable information? Oops, there are no microformat validators, either. This is especially difficult for something with as many ominous implications as a stepchild of iCalendar.

Much like an oral agreement, publishing microformats is an informal agreement between you and (hopefully) a developer community that sets up a relationship with plenty of vagueness, inertial resistance to change, and potential landmines to step on. Would you create a real developer API without a TOS, agreement, or at the very least, guidelines? Are you prepared to deal with objections if, when cutting costs, you rev a frontend design and lose some important aspect of microformat structure on the page (or, god forbid, you just don’t bring microformats over at all). Alternatively, are you prepared to announce all frontend markup changes? Does publishing a microformat without a special agreement mean that you are implicitly allowing comprehensive scraping of your web data? If you spend an hour seriously considering the costs of treating your frontend interface as a programming API for the sake of your relationship with developers, would you then rather spend those costs there, or on proper versioning, documentation, and communication with a developer community over a real publishing protocol or API? Publishing microformats while not having formal consumer support is a commonly what happens, but it is a poor midway point to leave yourself at.

The only place I can still imagine microformats surviving a cost/benefit analysis is in the case of preparing for search engine crawlers. In theory, if everyone publishes their websites according to a few semantic standards, the Big SE’s can embed structured data in their search results and act as aggregators of real data. There are a couple practical issues that you’d have to ignore in order to go for this pipe dream, though. You’d have to hope that the structure of microformats is fault-tolerant enough to survive the endless mangling of random developers trying to publish junk data, and that whatever was clean enough to make it through would be parseable, high quality, not eliminated in dupe checks, and relevant to a big enough segment of searches to justify the cost for the search engines to link to it. Second, you’d have to accept the fact that you probably wouldn’t get any permanent special treatment in search results, lest microformats become a new meta tag for SEO. Third, you’d have to assume that the big SE’s wouldn’t take these handful of high-order data entities and send it to their big topical datastores for aggregation and republishing themselves. I realize the benefit of a “standard.” However, why wouldn’t big aggregators crawl the web for content that conforms to existing, more mature standards that microformats are based on? Especially when these have often been passed through working, real world consumers of vCard/iCalendar before making it to production.

Anyway, here’s the question I want to put into the reader’s mind: should one spend time and effort making a frontend into an informal API through microformats, or to instead spend it on building a fully supported API or data publishing system that exists and operates separately? I think my stance is clear – i’m not against the theory of microformats, but i’m certainly going to differ with anyone who thinks it’s practical. If you can really think all that through, and still think microformats are a good thing to spend your resources on, then by all means, give it a shot. Just don’t say you weren’t warned.

Tech

Quick tip. Since the dev only built a windows binary for version 0.2.1 of uTidyLib, this bugfix for locating the dynamic libtidy on macosx is necessary.

Just edit tidy/lib.py, re-run the setup install script, and it suddenly can find libtidy.

Tech

As some of my friends know, i’ve had a love/hate relationship with the Woot-sold Infocus DLP 61md10 HDTV. I had one bulb fail nearly 10 months in, and I lucked out and got a warranty replacement. After moving this past weekend, I had a weird green colorband floating up the screen very slowly on both component inputs. At the time, I didn’t realize that it might have been related, but I had a very loud hum when putting the audio leads directly to the TV to power the stereo.

I did some research about color weirdness with DLP’s, as well as specific problems associated with the Infocus / RCA model that I have, I was worried that the issue might be the colorwheel on the lamp assembly, or (horror of horrors) possibly the entire “light engine.” I blew out the colorwheel with some compressed air, but didn’t notice any ball bearing noise at all, so I didn’t think that was the issue.

The next day, I was able to hook up an HDMI input and verify that the problem didn’t exist digitally (phew). This post referring to 60hz power line interference made me curious about the cable line coming in from Charter to the cable box. I routed that cable line through my cheesy Monster Cable surge protector (which has 3 coax in/out thingamajigs), and that cleared up the issue – no more weird green color band, and no more loud hum. I am assuming that there’s some sort of voltage regulator or noise filter on those coax passthrus, but this is the first time i’ve been glad that Woot gave away those surge protector kits along with the TV.

Now, hopefully I can enjoy another couple thousand hours of DLP television without spending $500 bucks on a new lamp.

Tech

RPX, a new startup, has come onto the scene offering to aggregate a patent portfolio, and offer a covenant not to sue to companies who license their entire portfolio in aggregate. They are funded by venture capital and also claim to have IBM and Cisco as initial clients.

Dogg, that reminds me of some cards.

Much like Roast Beef’s greeting cards, a technology patent originator can fathomably come up with an infinite pool of them by just applying the same rubric to a new set of circumstances (in the patent trolls’ case, by appending 14 pages of a definition of a computer to tie a business process to a tangible machine — le voila!). This means, as a creator of software technology, you are potentially on the hook for limitless amounts of patent infringement, as nearly everything you can do on a computer has had a patent filed on it in the past 20 years. The mere fact that one entity is aggregating a portfolio of patents has no bearing on the fact that there are potentially unlimited amounts of patents that can be aggregated by others, much as readers of Achewood would probably be able to assist Roast Beef in coming up with limitless amounts of new greeting cards.

It looks like TechCrunch is perhaps appropriately skeptical. Their business model revolves around subscription fees instead of enforcing patents through offensive litigation. If i’m understanding these terms correctly, if you don’t subscribe to their patent portfolio, then you are potentially subject to offensive litigation (which is the practice they are claiming to fight against). Correct me if i’m wrong, but doesn’t this seem to you to just be an extension of existing patent trolling practices (i.e. adding on subscription revenues to existing litigation + settlement proceeds) that is made feasible by the aggregation of large amounts of patents with a large amount of venture capital, instead of an attack on the NPE cottage industry? If so, wouldn’t it seem disingenuous to frame oneself as such?

It is not possible for one entity to aggregate all broad technology patents that pose a threat to oneself. Therefore, the subscription of a company to one particular NPE’s portfolio does not preclude the possibility of being sued for infringement by others. If a subscription model proves to be a more profitable / reliable revenue stream, you can expect other NPEs to adopt it too. Who knows, perhaps portions of the subscription model will be patented by RPX, and they’ll force other NPEs to license their patent as well if they want in to the game. It would be like patenting some aspects of the business process of patent trolling, which would be a fun irony, but still a sad one.

Some days, it seems like buying rights gets you a better return than buying property, but mainly if you’re an asshole.

Tech

I’ve been working on my own PHP-based project for a month or so, and have rewritten some basic generic libraries from scratch in PHP. I’m pondering whether I should open source them, and would appreciate it if you would comment on this post if you’re interested in seeing me put in the effort to write docs, standardize, then toss specific portions or all of it online. I learned my lesson through Freetag; it takes a lot of energy to keep something maintained. Here’s some of what I have:

• A database wrapper singleton around PDO
  • Simplifies code for working with “prepared statements” DBI-style. PDO prepared statements work great for getting around SQL injection, but I found them cumbersome by nature.
  • Consolidated error and exception handling with custom callback support.
  • I’ll eventually add fancier stuff in here, such as handle splitting for replicated architectures in here, and some read-after-write consistency code.
• A simple nonce library for protecting against XSRF/CSRF
• A simple library for handling file and image upload/resize
• A simple code profiling utility
• Some validation routines (I would like to add in some self-documenting API neatness that I wrote about before, when I get some time)
• Some other random handy stuff, like arg signing, link building, human interval descriptions.

It’s all pretty consistent with my philosophy of abstracting away website-related functionality instead of architecture. With this set of limited libraries, i’m building at a pretty fast rate, and I can almost always figure out what’s going on by just looking at one or two files. I’m trying to get a nice foundation that I can build fairly serious stuff on top of without getting completely confused.

Oh, and it all depends on php/filter being enabled with a default filter of “special_chars”. Come to think of it, I might write a separate post about using that, because it’s handy.

Tech

These days, i’m generally a packaged framework-for-the-web curmudgeon. I prefer simple libraries corresponding to simple, common, generic problem sets. It’s one thing to adopt a general-purpose scripting language, or even a general-purpose “hard” programming language, but it’s another thing entirely to download or buy a prepackaged framework on the basis of marketing material or buzz about the shiny new ur-framework for all development. My opinion was arrived at mostly through personal experience, and undoubtedly people will have different opinions based on their own experiences working with such software.

My hypothesis is that frameworks tend to occur naturally out of the needs of specific applications of technology. If you’re using a programming language to build a listing-based website, and you’re building things from scratch, patterns will evolve in your code. If you’re being rigorous about refactoring, you’ll be acutely aware of patterns that may be abstracted away, and it will be very natural to reduce them in some form or another. Eventually, if you add more developers, the lead developer will probably generalize an approach to building features that looks awfully like a framework. It could be lightweight, generator based, MVC based, queue based, or whatever — but this evolved framework, at its essence, is a reduction of noticed patterns about listing data systems to practice. However, why not avoid repeating all this work, and instead skip it by picking up a third party off-the-shelf framework and starting with that off the bat?

A heavily evolved framework is a codification of patterns that work for a specific problem set. This is fundamentally what newcomers to programming languages do not understand when they go reading neato tutorials for the framework du jour. It is also what many framework zealots (and sometimes, evangelists) do not understand when they are aggressively hunting down misguided naysayers who misapply frameworks and complain about it in public.

The risk of having programmers begin their work on top of established frameworks is that they typically do not have the experience necessary to understand what patterns have been adopted, why they have been adopted, and how those patterns would apply to their task at hand. The only people who understand this are usually the most experienced developers of the framework.

Unfortunately, the type of developer most attracted to the marketing claims of frameworks are inexperienced ones. And even more unfortunately, most frameworks’ greatest advocates are experienced developers of the frameworks, whose subtle and nuanced defenses about the proper use of their framework go completely over the heads of most beginners, who only stare googly-eyed at fancy baked demos.

This leads to large, bizarre codebases built on hacks, on top of hacks, on top of frameworks. It also leads to completely unrealistic expectations from businesspeople about the sustainability of development pace for fast, new, framework-based prototypes. It’s par for the course for our industry, and it’ll probably continue for a very long time as the trade of programming grows up in the new century.

Is there a reasonable argument that some “general-purpose” frameworks are good enough for nearly all “straightforward” web applications? Maybe — but this is a tricky subject. Many web applications share the same fundamental characteristics. If you have a general-purpose framework, it may work especially well with the characteristics of web applications. The thing that most web applications have in common is that most of them are dropped far before they get to a very serious type of problem that i’m referring to. So, in the sense that I don’t believe most web applications will get very far, it’s probably fine to use a framework for experimentation or prototyping. But one should be prepared to do the real work at some point, if the project is to be taken seriously.

There is also the annoying fact that when frameworks are popular, they tend to be a big hindrance to prototypes that never got rewritten. The point at which the misapplication of technology becomes a problem too often coincides with surges in popularity of a service. This leaves you open to competition and puts you in a very difficult place, especially if you’ve worked hard to open up a new market.

My advice for anyone starting to get a bad feeling about frameworks: work on real world projects, and start from scratch at least once. Real world projects jolt you out of the 5-minute screencast armchair Internet expert mentality. They have messy requirements, will suddenly strike you with landscape-changing revelations about how you’ve been going about things wrong, and will challenge you to grow more than you thought you ever could. Invest in your frustration up front, and the reward? When you’ve built a mature project, maybe you can generalize it and turn it into a framework… ;P

Tech

I’m so proud of my country tonight. Barack Obama has become the next President-Elect in an historic Presidential race that saw the best and the worst emerge from everyday Americans. His acceptance speech was moving; not only did he emphasize the importance of individuals to his campaign, he also brought up the topic of humility and addressing the country’s problems together. We can indeed find common ground to work on the most threatening problems to our country.

I’ve had a lot of difficulty even listening to the Republican point of view over the past eight years. I don’t doubt that the neoconservatives in the Bush Administration used and abused the differences of opinions between people in America to drive a wedge between us all – public, house, senate, and media. However, Obama’s campaign has gone beyond just earning my vote; it has also made me reconsider my emotionally charged reactions towards GOP talking points.

The first was Joe Biden’s anecdote about motives during the Vice Presidential debate:

I have been able to work across the aisle on some of the most controversial issues and change my party’s mind, as well as Republicans’, because I learned a lesson from Mike Mansfield. Mike Mansfield, a former leader of the Senate, said to me one day — he — I made a criticism of Jesse Helms. He said, “What would you do if I told you Jesse Helms and Dot Helms had adopted a child who had braces and was in real need?” I said, “I’d feel like a jerk.” He said, “Joe, understand one thing. Everyone’s sent here for a reason, because there’s something in them that their folks like. Don’t question their motive.”

Joe’s point is that although we may oppose other peoples’ political views, we should not succumb to prejudice about their character. Barack Obama’s campaign has shown that you can address the American public with maturity, dignity, and calm assertiveness, and still win the highest office in the nation. We can’t mistrust an entire segment of Americans simply for identifying with some policy views of the Republican party.

We can and should argue strongly with logic and passion when we disagree, but we cannot forget that our opponents are Americans too. I believe that both Obama’s acceptance speech and McCain’s concession speech strummed on this chord, and I sincerely hope that the political bases of both parties can be mature enough to rise to a more respectful level of discourse, and heal the wounds of divisiveness that have proven so profitable for the special interests in this country. Now let’s see what we can accomplish in the next four years.

Tech