Addendum: is it still worth using a JS abstraction library like jQuery? Yes.

So my mouse stops working on the MacBook Pro. Oh, I can move it around alright. I can move it around all day and night. I just can’t frakking click. The left-click won’t work but the right-click (I have a two button Bluetooth mouse) will. It’s amazing how quickly one realizes that Right Click isn’t fit to iron Left Click’s pants in the morning, when Left Click isn’t working.

OK. This is just something wrong with that Bluetooth mouse, right? So I try the trackpad. Same behavior… I can move that cursor around wherever I like, but clicking is futile. If you think this won’t make a grown man cry, you weren’t in my basement tonight.

Luckily, my keyboard still worked. So with a little ALT-TAB I could get into my browser, tab around to the location field (thanks Chrome!) and do a Google search on this shiz.

Lots of talk about Bluetooth devices… Bluetooth devices in the next room, Bluetooth devices in the work bag, Bluetooth devices leaning up against stuff with their buttons getting pushed.

Makes sense. So I turn off the power on my Bluetooth mouse. No difference. I yank the USB piece that talks to Bluetooth mouse. No difference… trackpad still produces barren clicks. So I yank every peripheral I have connected…. firewire hard drives, USB hub, audio system, second display. NO DIFFERENCE. A ZOMBIE IS LIVING IN MY MAC AND FEEDING ON ITS LEFT CLICK LIKE BRAINS.

The solution I finally arrived at after more searching, thank you keyboard I love you, was to disable Bluetooth entirely at the command line. This is the same thing as unchecking ‘On’ in System Preferences -> Bluetooth, but trying doing THAT with Left Click pacing the picket line.

COMMAND-SPACE got me into Spotlight, where I typed “Terminal”, arrowed down and hit ENTER. Now at the command line, I typed:

sudo defaults write /Library/Preferences/com.apple.Bluetooth “ControllerPowerState” -bool FALSE

That turned Bluetooth OFF… sort of. That turns the preference off, but to kill any existing Bluetooth connections dead, a little more magic is required:

sudo killall -SIGHUP blued

And with that: my trackpad was restored to normal. I’m pretty happy with my trackpad at this very moment, but I’m sure I’ll start using Bluetooth devices again… with the above experience in the back of my mind.

Writing unit tests is a little bit like flossing.  You know it’s a good thing to do.  It may not feel like fun while you’re doing it, but you’re almost always glad you did it when you’re done.  Like flossing, if you’re going to do it, and do it consistently, you need to try and establish a habit.  And there’s no time like now to get started.

In the directory where you stash your modules, create a subdirectory named “t“.  Yes, you’ve seen this directory name before when installing a module from CPAN.  Don’t panic!  You’re not going to have to build a complete distribution.  You’re just creating a place to stash your tests that is both convenient and follows the conventions used by the Perl prove utility.  Now, the next time you create some new function or method in one of your modules, create a unit test for that function in your “t” directory.  For example, let’s say I’m creating a new utility function call mySum() in the module MyModule.  I’ll also create a new unit test script named mySum.t.  As I write the function in the module, I’ll also start writing the unit tests in the test script.  Writing some logic to handle an edge case?  Write a test to probe that edge.  Test it.  Right now!  Now code up the next hard part, and test that.  Here’s a simple example you can use as a starting template:

#!/usr/bin/env perl
#
#       File: t/mySum.t
#      Usage: prove mySum.t
#   Abstract: Test my wonderful MyMod::mySum() service.

use warnings FATAL => qw(all);   # Make all warnings fatal.
use strict;                      # Keep things squeaky clean.
use Test::More;                  # Lots of testing goodies.
use File::Basename;              # Easy basic filespec parsing.
use lib '..';                    # Where to find our own modules.

my $ownName = basename($0); # For error reporting.

die("? $ownName: no command line args expected\n_ ", join(' ', @ARGV), "\n_")
 if (scalar(@ARGV) > 0);

use_ok('MyMod') or exit(1); # The module we're testing.

is( MyMod::mySum(),                 0,         "sum of nothing is zero"       );
is( MyMod::mySum(undef(), undef()), 0,         "sum of two nothings is zero"  );
is( MyMod::mySum(1,2,3,'fred'),     undef(),   "fred is not a number"         );
is( MyMod::mySum(2,2),              4,         "two small positive integers"  );

my $x = 2;

is( MyMod::mySum($x,$x),            4,         "two small positive integers"  );
is( MyMod::mySum($x,\$x),           undef(),   "can't sum a scalar reference" );

# etc.

done_testing();

# EOF: mySum.t

The example above uses the is() test function which, along with ok(), covers a lot of the sort of simple test cases you’ll want to do.  But there are lots of other very useful test functions beyond these, so be sure to check out the Test::More documentation for more details.

To run your test, from your module directory, you can do either:

$ t/mySum.t
ok 1 – use MyMod;
ok 2 – sum of nothing is zero
ok 3 – sum of two nothings is zero
ok 4 – fred is not a number
ok 5 – two small positive integers
ok 6 – two small positive integers
ok 7 – can’t sum a scalar reference
1..7
$

…or….

$ prove t/mySum.t
t/mySum.t .. ok
All tests successful.
Files=1, Tests=7, 0 wallclock secs ( 0.02 usr 0.00 sys + 0.02 cusr 0.00 CPU)
Result: PASS
$

As your test library grows, you can just enter the prove command with no parameters and it will automatically run all of the *.t files it finds in your test directory.

So, don’t worry about back-filling all of those unit tests you should have written.  But do start getting into the habit of creating unit tests, function by function, as you create new code, particularly for those utility functions that have very narrow, well defined jobs.  For your convenience, here is the toy module that was used with the above unit test.

package MyMod;

# Simple demo module with simple demo function.

use warnings FATAL => qw(all);
use strict;
use Scalar::Util qw(looks_like_number);

sub mySum

#  Precondition: Parameters consist of zero or more numeric values.
#
# Postcondition: The sum of all the numeric values is returned as our
#                functional value.  Undefined parameters are silently
#                ignored.  We return undefined if one or more values
#                do not appear to be numeric (i.e. a usage error).  We
#                return 0 if there are no defined parameters.
#
#     Unit Test: t/mySum.t

{
  my $sum = 0;
  foreach my $x (@_)
    {
      next if (not defined($x));
      return() if (not looks_like_number($x));
      $sum += $x;
    }

  return($sum);
}

1;  # So endith your typical Perl module.

– Why Programmers Work at Night

– Programmer Interrupted

Although now that my spouse is retired, it is getting more difficult to get long stretches of interrupted time at home.  I think there’s a potential market for programmer isolation pods.  An isolation tank could work, if you could solve the problem of needing a waterproof keyboard and mouse.

Here is a tribute to / history of Perl’s first 25 years.

Long live!

-R, –random-sort
sort by random hash of keys

Yes, newer versions of sort will actually shuffle your input data.I’m not sure if that’s a cool thing for a command named sort to do, but I like it anyway.

A quick test (on Red Hat 6) shows that it really is random: you don’t get the same shuffle each time.

Good replacement for the Perl one-liner:

perl -MList::Util -e ‘print List::Util::shuffle <>’

(Added later: many versions of Linux include the “shuf” command.)

A colleague was creating an app that used the Net::LDAP module, a module I have been using myself for a very long time now. In the code there was the statement:

my $msg = $ldap->search(bind_dn => $base_dn, filter => $filter);

…that was returning a “Bad filter” error. Turns out there was no problem with the filter text, but there was with the search statement—the parameter label bind_dn is not a valid keyword for this module.

I did something similar a long time ago, using a parameter label for the bind() method that actually belonged to new(). The parameter and value were accepted without comment. It didn’t cause an error, it was just silently ignored since it was totally unexpected.

As I’m sure you know, when passing named parameters to a method, what you are really passing is a hash. So:

filter => $filter

…is really a hash entry with ‘filter’ as the key getting assigned the value from the scalar $filter. Now what you would like is for the method to throw an error if you misspell a key, but for the Net::LDAP module, and some other CPAN code I’ve seen, no check is done to see if an unexpected key is defined. The justification for this is the possible use of inheritance. If you pass a “foo” parameter to an object, it may not know what to do with “foo” but one of the other methods inherited by that object may. That’s why you’ll see things like…

my($self, %opts) = @_;

and then later…

$self->someOtherMethod($this, $that, %opts);

As for my own code, I’m still working on coming up with a good way to used named parameters with methods where all of the methods that get a set of named parameters have to ‘claim’ the ones that that particular method is using, so that in the end, if there are any unclaimed named parameters, an exception is thrown. Haven’t come with what I think is a clean solution, but I’m still thinking about it.

I hope this is the sort of thing Moose fixes.

Day 1

git

It is Monday morning, and I am attending a(nother) tutorial on git (Get Better at Git). I am fully convinced that git is a better tool than Subversion at this point; I am also convinced it’s harder to fully understand and use. How these things can both be true speaks to the complexity of managing source code. It’s pretty clear that in a highly collaborative project, git is the way to go. That isn’t to say there aren’t thousands of projects using Subversion successfully, but, mastery of git would certainly reduce the pain in reaching the same success.

But mastery of git takes time. Fluency takes time. It occurs to me that the quickest path to fluency is immersion, and taking one or two classes on git does not immersion make. I do long, at times, to work in a hardcore dev shop to become fluent in git… just as I long, yet more often, to live in Quebec City or Paris and become fluent in French.

Moose

My Monday afternoon session is on Moose, the most popular set of modules that extends Perl’s object support. As with git, I won’t write all my technical notes here… I’m just going to soak it in. Immerse myself as it were for the three hours I’ll have.

Okay I lied. I am already sort of liking Moose. When defining an object, you can label its attributes (think ‘values’) as ‘ro’ (read-only) or ‘rw’ (guess), make any of them required, and strongly type them. This is really fantastic. Honestly, I am often fairly lazy on the server side when it comes to validation, having already done the work on the client end in Javascript to try and assure the right values come in. But the client side is not completely in a developer’s controlling hands, so, it cannot be relied upon. Granted, Moose will crash your program if your constraints are violated, but I’d rather crash in such event than enter into an unexpected state.

I’m tempted to rejigger my Remedy and Pinnacle modules to use Moose, but I sort of hope I can get away with Moose::Lite. Performance won’t be much of an issue running atop FastCGI, but the dependency tree for Moose proper is massive, and I don’t want to acquire that much baggage quite yet.

But dang, Moose is neat. I love the concept of ‘advise’. This is a kind of flow control in object land. You can define what happens ‘before’ or ‘after’ a method is run, or even wrap a method with ‘around’ if you need access to its internals. The best analogy I can make is probably applicable to object orientation in general… it’s like stuffing your money deeper in your pocket so it has less chance of falling out. Substitue ‘logic’ for ‘money’.

When you have a class that inherits from more than one, it is called ‘diamond’ or ‘multiple inheritance’. It’s dodgy and error prone. Moose solves this with the concept of ‘roles’. I’m trying to wrap my mind around this, but it seems as though a role is a kind of validation for the relationship between classes. Ah: it enables horizontal code reuse (roles and classes) rather than strictly vertical (classes and sub-classes). Apparently, using roles in Moose is far more common than subclassing at all.

Moose finally broke my mind this afternoon. Much of this I’d need to work with to begin to understand.

Day 2

Modern Perl

Tuesday morning and I am pleased to be attending New Features of the Modern Perls by Damian Conway. Damian’s talks are worth the price of admission in themselves and everything else is gravy.

Something I did not know: when upgrading from 5.8, any XS modules you use need to be recompiled for any new version of Perl you move to. Damian is making a plea for Perl developers to upgrade, commenting that the Perl community will only be supporting the latest release and the previous two releases… meaning, security updates only for older versions of Perl. This means if you encounter an obscure, non-security related bug in an older version of Perl, it may not get fixed, and more importantly, module authors will be requiring more recent versions of Perl and the CPAN will move past you. He recommends targeting 5.14 at this time.

Interestingly, to preserve backward compatibility, you must enable new features explicitly in Perl versions starting with 5.10. This is to prevent new keywords from accidentally clashing with user subroutine names. But if you’re ready to move forward, you can enable all features in addition to requiring a minimum version of Perl simply with:

use v5.14

Now he’s making me want to upgrade just to use say (which is print with an implicit newline). Also, smartmatching (the new ~~ operator) is brilliant, a feature Damian invented for Perl. Another double-character operator added to recent Perls is //, the defined-or operator, which is better for setting default values (as below) than using ||, because || tests for true/false, not defined-ness.

my $number = $number // 42;

The above case would assign $number a value of 0 if it had started off that way, whereas using || would have seen a 0 as false and assigned $number the 42.

Damian on Moose: “It annoys me just enough so that I don’t want to use it.” Now, he also has a lot to say in favor of Moose, but mostly complains of the startup time and isn’t crazy about the declaration syntax.

Another very cool feature in recent Perls is named captures out of regexes… numbered captures are too easy to get wrong. Think using named parameters to pass to subroutines instead of an order-dependent array. It’s just better.

jQuery Mobile

My Tuesday afternoon session is on jQuery Mobile. It’s safe to assume that I’ll be building some mobile apps in the coming decade, and this is one of my likely tools. If I remain at the University during that time, jQuery Mobile is a *more* likely platform than native applications targeted to iOS, Android or some other emerging mobile OS. That is, unless successful alum donate a native apps development budget. But that wouldn’t be the best use of money, would it?

I’ve been following the jQuery projects (core, jQueryUI, and now jQuery Mobile) pretty closely for the past few years, and now load core and UI by default in my webstuff. Bring mobile into the mix is the logical next step. I already have an idea of the app I want to write… but have a ton of backend work to complete before I get proper access to the data I need. I suppose that shouldn’t stop me from prototyping the front end… in my free time (stop laughing).

Day 3

Day 3 begins the shorter sessions of the conference proper, following the keynotes. They had the mayor this year, who proclaimed Portland an ‘open source city’, and who made some funny jokes. There were some other semi-inspiring appearances amongst the keynotes, but as I decided to watch instead of type, I don’t have the time to relay these now.

Perl 5 Today, Tomorrow and Christmas is the first session I am attending this morning. Ricardo Signes is the current Perl 5 project lead (Pumpking), so you might call him a reliable source on the subject. He was also the guy who presented on Moose a couple days ago.

This is mostly a talk about the challenges in managing the Perl project, most of which (surprise!) are human in nature. Apparently the p5p (Perl 5 Porters) mailing list is a great place to get into an argument but not always a great place for a Pumpking to spend his days. There is a lot of talk at these conferences about ‘getting along’ in open source communities. Now he’s making an analogy I’ve often thought of, that of software coming about via ‘intelligent evolution’… a mix between intelligent design and evolution.

Next up, Programming In The Future, a look at how the craft might evolve in the next 25 years through the lens of how Perl has evolved over the past 25. This turned out to be a nit of a meandering talk, but included an interesting history of the most significant moments in Perl history, 1987 to present. It’s hard to imagine you couldn’t write a Perl module until 1994. Now he’s talking about parallel and functional programming, two somewhat advanced programming tasks that people seem to imagine becoming easier in the future, and some examples of how we do these things in Perl today.

Another yummy lunch, this one thanks to Google, and now I am in Test Driven UI Development. This is a very challenging endeavor. This talk centers on a toolset from the Java/Groovy world, but which could be applied to other platforms. I must say though: I cannot watch much of this test-driven development stuff without being secretly glad I don’t do it. I just seems like another codebase to maintain, which itself would always be breaking, perhaps saving me an error here or there, but which would in general dilute my productivity. It is interesting to see the speaker write the test before he writes the HTML for a simple form, though, complete with human readable comments… all very nice documentation. The notion of ‘testable code’ is about being able to select the elements in the DOM that you need to test, so, jQuery-like, you may be adding ids, but in this case, perhaps only to select them in the test suite. Dear me. The speaker discourages uses CSS classes as selectors in your test suite, because you may have front end developers mucking with class names. He mentions testing AJAX can be tricky, which I would have guessed, but that you can access return methods with jQuery and similar in order to check success in many cases.

To finish off the day, I’m attending The Conway Channel, another Damian Conway talk. Entertaining as expected, and I immediately went to play with IO::Prompter and Lingua::EN::Inflect, two of the modules he has updated this year. However IO::Prompter required Perl 5.10+, so… heck, why have I been putting off trying Perlbrew so I can run any version of Perl that I want? Now building and compiling 5.14.2. :)

Day 4

Thursday morning and I had been planning to attend this EFF session, but it was full. So I walked around the Expo Hall, checked out the booths, grabbed some stickers and pens, that kind of stuff.

Next up is the long awaited Taming Perl Regexes, another Damian Conway talk, in which he will is launching a tool called ‘rxrx’ that he had hinted at in the video he sent to YAPC::NA. It’s an incredible debugger for regular expressions, installable from CPAN as Regexp::Debugger. I will try to post a short screen cap of this tool at work once I get it installed.

I’m starting my afternoon with Advanced MySQL Replication Architectures, presented by some Oracle guys. So far this is high-level view of different replication architectures; I was hoping for a little how-to with some commands. Oh well, sometimes you hit a session that isn’t what you’re looking for.

Next up is Cooking Perl with Chef, which I have high hopes for. Having used Perlbrew for the first time today, I am feeling bold about tossing Perl itself around. Here are the problems and their solutions as presented:

• application-specific Perl: Perlbrew
• application-specific @INC Path: local::lib
• versioned application code: git, SVN, etc.
• versioned module dependencies: Carton
• automate the previous four: Chef (scripted using Ruby)

Chef usually requires a server of its own, but if you want to save on that overhead and only have a handful of nodes to deploy to, you can use Chef Solo. Still, I think Chef would be the last piece I’d add to this puzzle, if at all, considering that I already have a scripted deployment routine that is beginning to pull all the same levers and push the same buttons. But adding Perlbrew and Carton (which leverages local::lib) are musts. Hopefully I’ll be deploying with those within a year. We’ll see!

These last session I am attending today is What Every Web Developer Should Know About Database Optimization. This was largely review for me, but I did get a reminder that occasionally JOIN order could be important, and oddly enough, yet another warning about ORMs. Object Relational Mappers can be hell on your database, especially if you don’t give the proper cues, similar to how you’d optimize SQL, just not directly. Explain to me why I want to use an ORM, again? (I know, I know…)

At 7pm (PST of course) is Larry Wall’s State of the Onion address, the official update on Perl’s present and future, but I’m whupped.

Day 5

I skipped the keynotes this morning in favor of a long breakfast. This last day is a half day of sessions.

The first for me is another database session: Optimizing MySQL Configuration. Thus far we’re getting a list of optimizations not to try. The

mysqltuner

I don’t think I’d like to be the steward of a large database that was always needing tuning. It seems so fiddly.

Next up, some attention to the front end of things after dwelling on the back end for while: Designing HTML5 Components. Okay, this was not what I expected at all. He is using a Java framework called Vaadin (the Finnish word for ‘reindeer’) and the goal is apparently to create reusable web components. This is really yet another web framework, this one from Java land, and by reusable he means, reusable in this particular framework’s ecosystem. You write in Java, and it compiles some Javascript for each of your target browsers. Pretty slick, but you’ve got to go ‘all in’ to get the bennies.

The last session I am attending this OSCON is Instantly Better Vim. Another Damian Conway talk, a longer version of which I’ve previously attended, but let’s face it… I still haven’t adopted most of the Vim tricks he taught me the first time, and I could use a refresher.

Wow… by the end of this conference, word has gotten around about Damian’s talks. He is just too entertaining to miss. They had to get a second room and pull out the wall to accomodate his final talk. And as usual he sent me away wanting to try some Vim tricks right away.

Well, that was my OSCON 2012. As usual, a very well organized event, chock full of learning and networking opportunities, jobs for those that want them, and generalized geeky inspiration. I often find myself back in the hotel room coding, or tucked away somewhere in the conference hall, slowly exploring some of the new ideas I’ve been exposed to. Thanks to UNH for my funding.

Jun 11 23:58:15 sunapee MailScanner[7997]: Message q5C3vVli019865 from 71.243.115.147 (info@softwareevals.org) to unh.edu is spam, SpamAssassin (not cached, score=15.565, required 5, BAYES_00 -1.90, DIGEST_MULTIPLE 0.29, HELO_DYNAMIC_IPADDR 1.95, HTML_MESSAGE 0.00, KHOP_DNSBL_BUMP 2.00, MIME_HTML_ONLY 0.72, PYZOR_CHECK 1.39, RAZOR2_CF_RANGE_51_100 0.50, RAZOR2_CF_RANGE_E8_51_100 1.89, RAZOR2_CHECK 0.92, RCVD_IN_HOSTKARMA_BL 1.70, URIBL_BLACK 1.73, URIBL_JP_SURBL 1.25, URIBL_RHS_DOB 1.51, URIBL_WS_SURBL 1.61)

or “ham” (not spam):

Jun 11 23:59:54 sunapee MailScanner[7634]: Message q5C3xp77020291 from 208.117.48.80 (bounces+54769-353f-cfg6=cisunix.unh.edu@email.news.spotifymail.com) to cisunix.unh.edu is not spam, SpamAssassin (not cached, score=-2, required 5, autolearn=not spam, BAYES_00 -1.90, HTML_MESSAGE 0.00, RCVD_IN_HOSTKARMA_NO -0.09, SPF_PASS -0.00, T_RP_MATCHES_RCVD -0.01)

(Lines wrapped for clarity.)

Perl scripts scan through the logs and produce plots of ham/spam traffic, typical example here. Such scripts need to (for lines like the above) extract the date/time information (always the first 15 characters) and whether the line says “is spam” or “is not spam”. My initial approach (years ago) was very simple-minded. Old code snippet:

while (<>) {
    if (/is spam/ || /is not spam/) {
        $date = ParseDate(substr($_, 0, 15));
        $dt = UnixDate($date, "%Y-%m-%d %H:00");
        if (/is spam/) {
             $spamcount{$dt}++;
        else {
             $hamcount{$dt}++;
        }
    }
}

(ParseDate and UnixDate are routines from the Date::Manip package. Without getting bogged down in details, they allow the messages to be counted in hour-sized buckets for the plot linked above.)

For not-particularly-important reasons, I decided to try an “all at once” regular expression on each line instead. New code snippet:

use Readonly;
use Regexp::Common qw/net/;

Readonly my $SPAMLINE_RE => qr {
    ^
    (\w{3} \s+ \d+ \s \d{2}:\d{2}:\d{2}) \s
    \w+ \s MailScanner \[\d+\]: \s
    Message \s \w+ \s
    from \s ($RE{net}{IPv4}) \s
    \([^)]*\) \s to \s \S+ \s
    (is \s (?:not \s)? spam)
}x;

while (<>) {
    if ( $_ =~ $SPAMLINE_RE ) {
        my $d        = $1;
        my $spamflag = $3;
        my $date     = ParseDate($d);
        my $dt       = UnixDate( $date, '%Y-%m-%d %H:00' );
        if ( $spamflag eq 'is spam' ) {
            $spamcount{$dt}++;
        }
        else {
            $hamcount{$dt}++;
        }
    }
}

There’s plenty to criticize about both snippets, but mainly I was thinking: running the second version’s huge, hairy regular expression on each log line will just have to be way slower than the simple fixed-string searching in the first version. (Even though there can be up to three searches per line in the first version, they’re searches for fixed strings, right? Right?)

Wrong, as it turns out. The hairy second version ran slightly faster than the simple-minded first version. (In addition it extracts the IP address of the server that sent us the message; not used here, but important in other scripts.)

Humbling moral: even after many years of Perl coding, my gut feelings about efficiency are not to be trusted.