Perl and Inside-out Objects

Way back, when I was pouring through Damian’s object book, where
he shows all the different ways to built objects in Perl, I became
enamored with the idea of inside-out objects.

Here’s what Randal Schwartz says on the subject…

At the time what I liked about this approach was the idea that the caller couldn’t
grok the object and take short-cuts that would only break later
if I changed the implementation.

All I can say is that I am so happy that for most CPAN modules,
inside-out objects are rarely seen.  That’s because I have found that one of
the best debugging tools I have, when dealing with a complex OO
CPAN module, is being able to call Data::Dumper on the object.

And of course I do it to my own objects too, although I generally
try to always include a  toString() method which can be thought
of as a  smarter and prettier Data::Dumper — only because it understands
the data.

And as for people who grok the object to take unwise shortcuts?  They will
eventually get what they deserve.

Unit Testing is a Little Bit Like Flossing

You know you should floss, but you don’t have time to do it right now.  Sure, you floss when you have an annoyance, like that piece of pulled pork caught between your molars.  And you floss for a couple of days, after your dental hygienist has given you ‘the lecture’.  But in the end, you don’t have time to do it right now.  Maybe later.

Writing unit tests is a little bit like flossing.  You know it’s a good thing to do.  It may not feel like fun while you’re doing it, but you’re almost always glad you did it when you’re done.  Like flossing, if you’re going to do it, and do it consistently, you need to try and establish a habit.  And there’s no time like now to get started.

In the directory where you stash your modules, create a subdirectory named “t“.  Yes, you’ve seen this directory name before when installing a module from CPAN.  Don’t panic!  You’re not going to have to build a complete distribution.  You’re just creating a place to stash your tests that is both convenient and follows the conventions used by the Perl prove utility.  Now, the next time you create some new function or method in one of your modules, create a unit test for that function in your “t” directory.  For example, let’s say I’m creating a new utility function call mySum() in the module MyModule.  I’ll also create a new unit test script named mySum.t.  As I write the function in the module, I’ll also start writing the unit tests in the test script.  Writing some logic to handle an edge case?  Write a test to probe that edge.  Test it.  Right now!  Now code up the next hard part, and test that.  Here’s a simple example you can use as a starting template:

#!/usr/bin/env perl
#       File: t/mySum.t
#      Usage: prove mySum.t
#   Abstract: Test my wonderful MyMod::mySum() service.

use warnings FATAL => qw(all);   # Make all warnings fatal.
use strict;                      # Keep things squeaky clean.
use Test::More;                  # Lots of testing goodies.
use File::Basename;              # Easy basic filespec parsing.
use lib '..';                    # Where to find our own modules.

my $ownName = basename($0); # For error reporting.

die("? $ownName: no command line args expected\n_ ", join(' ', @ARGV), "\n_")
 if (scalar(@ARGV) > 0);

use_ok('MyMod') or exit(1); # The module we're testing.

is( MyMod::mySum(),                 0,         "sum of nothing is zero"       );
is( MyMod::mySum(undef(), undef()), 0,         "sum of two nothings is zero"  );
is( MyMod::mySum(1,2,3,'fred'),     undef(),   "fred is not a number"         );
is( MyMod::mySum(2,2),              4,         "two small positive integers"  );

my $x = 2;

is( MyMod::mySum($x,$x),            4,         "two small positive integers"  );
is( MyMod::mySum($x,\$x),           undef(),   "can't sum a scalar reference" );

# etc.


# EOF: mySum.t

The example above uses the is() test function which, along with ok(), covers a lot of the sort of simple test cases you’ll want to do.  But there are lots of other very useful test functions beyond these, so be sure to check out the Test::More documentation for more details.

To run your test, from your module directory, you can do either:

$ t/mySum.t
ok 1 – use MyMod;
ok 2 – sum of nothing is zero
ok 3 – sum of two nothings is zero
ok 4 – fred is not a number
ok 5 – two small positive integers
ok 6 – two small positive integers
ok 7 – can’t sum a scalar reference


$ prove t/mySum.t
t/mySum.t .. ok
All tests successful.
Files=1, Tests=7, 0 wallclock secs ( 0.02 usr 0.00 sys + 0.02 cusr 0.00 CPU)
Result: PASS

As your test library grows, you can just enter the prove command with no parameters and it will automatically run all of the *.t files it finds in your test directory.

So, don’t worry about back-filling all of those unit tests you should have written.  But do start getting into the habit of creating unit tests, function by function, as you create new code, particularly for those utility functions that have very narrow, well defined jobs.  For your convenience, here is the toy module that was used with the above unit test.

package MyMod;

# Simple demo module with simple demo function.

use warnings FATAL => qw(all);
use strict;
use Scalar::Util qw(looks_like_number);

sub mySum

#  Precondition: Parameters consist of zero or more numeric values.
# Postcondition: The sum of all the numeric values is returned as our
#                functional value.  Undefined parameters are silently
#                ignored.  We return undefined if one or more values
#                do not appear to be numeric (i.e. a usage error).  We
#                return 0 if there are no defined parameters.
#     Unit Test: t/mySum.t

  my $sum = 0;
  foreach my $x (@_)
      next if (not defined($x));
      return() if (not looks_like_number($x));
      $sum += $x;


1;  # So endith your typical Perl module.

Programus Interruptus

This is why, when I have some particularly hard coding to do, I try and do it at home…

Why Programmers Work at Night

– Programmer Interrupted

Although now that my spouse is retired, it is getting more difficult to get long stretches of interrupted time at home.  I think there’s a potential market for programmer isolation pods.  An isolation tank could work, if you could solve the problem of needing a waterproof keyboard and mouse.

Perl pet peeves: named parameters for methods

A colleague was creating an app that used the Net::LDAP module, a module I have been using myself for a very long time now. In the code there was the statement:

my $msg = $ldap->search(bind_dn => $base_dn, filter => $filter);

…that was returning a “Bad filter” error. Turns out there was no problem with the filter text, but there was with the search statement—the parameter label bind_dn is not a valid keyword for this module.

I did something similar a long time ago, using a parameter label for the bind() method that actually belonged to new(). The parameter and value were accepted without comment. It didn’t cause an error, it was just silently ignored since it was totally unexpected.

As I’m sure you know, when passing named parameters to a method, what you are really passing is a hash. So:

filter => $filter

…is really a hash entry with ‘filter’ as the key getting assigned the value from the scalar $filter. Now what you would like is for the method to throw an error if you misspell a key, but for the Net::LDAP module, and some other CPAN code I’ve seen, no check is done to see if an unexpected key is defined. The justification for this is the possible use of inheritance. If you pass a “foo” parameter to an object, it may not know what to do with “foo” but one of the other methods inherited by that object may. That’s why you’ll see things like…

my($self, %opts) = @_;

and then later…

$self->someOtherMethod($this, $that, %opts);

As for my own code, I’m still working on coming up with a good way to used named parameters with methods where all of the methods that get a set of named parameters have to ‘claim’ the ones that that particular method is using, so that in the end, if there are any unclaimed named parameters, an exception is thrown. Haven’t come with what I think is a clean solution, but I’m still thinking about it.

I hope this is the sort of thing Moose fixes.

Adding Vulnerabilities to Your Code for Fun and Profit

The issue of coders purposely adding time bombs and backdoors to their code has been a potential problem for almost as long as programs have been written.  One of my favorite historical exploits is the one described by Ken Thompson about a modified version of the Unix C compiler.

A programmer’s motives for embedding such exploits into their own code can range from wanting to seek revenge for a real or imagined slight, to retain access to a resource, just to see if it can be done, and of course, to make steal money typically with applications that deal with money.

But Bruce Schneler, in his recent blog posting “The Vulnerabilities Market and the Future of Security“, describes an emerging way for unscrupulous coders to monetize vulnerabilities purposely baked right into the code, or at the very least, not report discovered vulnerabilities to their employer.

The Promise, the Limits, the Beauty of Software

In case you missed it, an interesting lecture by Grady Booch…

Software development has been, is, and I believe will remain fundamentally very hard. We have been able to build things we could not have conceived of some years ago because we have improved in our practices and our processes and our tools and in our languages. But every time we’ve overcome those barriers we realize we want to build more and more complex things. And, for many economically interesting systems, the problem is now one of scale. Not just computational scale or platform scale but scale of the sheer amounts of software that we have.

— Grady Booch

Cataloguing Perl Module Dependencies

So you have a bunch of Perl scripts and modules you’ve written over time, and you’d like to quickly determine what non-core Perl modules are required to make them run. Devel::Modlist to the rescue.

$ perl -d:Modlist=nocore some-funky-script
Module::ExtractUse 0.27
Module::ExtractUse::Grammar 0.25
Parse::RecDescent 1.967009
Pod::Strip 1.02
version::vxs 0.95

Unfortunately Devel::Modlist is not itself a core module, but it’s an easy install since it doesn’t have any dependencies of its own. The way it works is pretty clever. Rather than trying to parse the Perl scripts and modules, looking for “use” statements, it loads the code into the Perl debugger. This is a smart approach since the common wisdom is only perl can parse Perl.

But this is not a perfect solution. For one, this only works for code that already runs on your system, so you can’t use it to quickly list what modules are needed for some bit of Perl you just downloaded. It also cannot find and report on modules that get loaded dynamically, as would happen if you do something like:

if ($magicRequired) { require Magic; castSpell(); }

And finally this module is not designed to be used from inside of a Perl script since it would cause the dependency reporting every time that script is executed. The shell command line example shown above is the normal usage model for this module.

But even with those caveats, this is still a handy little tool to have in your Perl toolbox.

Yet Another Documentation System…

I’ve used way too many different documentation systems in my time including RUNOFF, nroff, DECdocument, Interleaf, TeX/LaTex, and PageMaker, just to name some of the older ones. These days my documentation tends to be either just plain old text, org-mode (emacs), HTML, or Perl POD. I find myself, however, once again looking for the ultimate solution for my document formatting needs in anticipation of having to create a lot of documentation for a major project.

At first I was thinking of standardizing on org-mode since that is what I tend to use by default now. Its table editor rocks and the syntax is very readable like Markdown, and other similar lightweight markup languages, with very easy export (for an emacs user) to HTML or LaTex. But I’m concerned about picking an Emacs-centric choice in deference to other maintainers, and the fact that org-mode’s own manual is formatted using GNU’s Textinfo gives me pause.

After casting about, including looking at Texinfo, DocBook, Muse (emacs), and reStructuredText, I find myself gravitating to Markdown and most likely the eventual adoption of Pandoc.

Here’s an article that I found particularly helpful. My current thinking is that the learning curve for Markdown is very shallow so I’m not making an unreasonable assumption for other maintainers, and I can start generating documentation now and worry about pretty formatting, structure and presentation later. I’m also trying to avoid this problem out of the gate…

The General Problem

I’m not happy about putting off the overall document architecture till later, but I’m burning daylight.

Installing Catalyst on OS X

I’ve been looking at Perl based web application frameworks again. About 4-5 years ago I needed to create a report card application for my spouse. I looked at Catalyst and CGI-Application, and decided on CGI-App because it was lighter weight and did not require a persistent process module like mod_perl or FastCGI. I liked CGI-App and found it easy to use and logical.  But recently I’ve been thinking that I needed to adopt an application framework for work projects, so I thought I’d look at Catalyst again for the first time.

To kick the tires on Catalyst, i.e. to try some of the examples found in ‘the book’, we’ll need to install the development version plus some ‘optional’ modules as they are introduced. For your entertainment and enjoyment, I’ve outlined below what it took to get the basics of Catalyst installed on my Mac laptop (OS X, version 10.6.8 aka Snow Leopard).  The prerequisite for these instructions is having MacPorts installed. MacPorts is the best way I’ve found for installing Unix/Linux goodies on my Mac. Be aware that MacPorts wants to install and maintain its own Perl in /opt/local/bin/perl which is a good idea. Your Mac depends on Perl for various miscellaneous system functions, so it is best not to mess with your system’s stock distribution of Perl.

  1. First off we need to know which version of Perl we’ll be installing Catalyst against.
    $ which perl
    $ perl -v
    This is perl 5, version 12, subversion 3 (v5.12.3)...

    This confirms that that the version of Perl in my search path is indeed the one maintained by MacPorts, and reveals its current rev. So in MacPorts, the distributions we’ll be interested in will be labeled with “p5.12″.

  2. Next we fire up the MacPorts utility in interactive mode and search for the exact name of the packages we’ll need. I’ll tell you right now — don’t search just for “catalyst”, it will be a long and overwhelming list!
    $ sudo port
    Password: *******************
    MacPorts 2.0.2
    Entering interactive mode... ("help" for help, "quit" to quit)
    [Users/wfc] > search catalyst-dev
    p5-catalyst-devel @1.310.0 (perl)
        Catalyst Development Tools
    p5.8-catalyst-devel @1.310.0 (perl)
        Catalyst Development Tools
    p5.10-catalyst-devel @1.310.0 (perl)
        Catalyst Development Tools
    p5.12-catalyst-devel @1.310.0 (perl)
        Catalyst Development Tools
    p5.14-catalyst-devel @1.310.0 (perl)
        Catalyst Development Tools
    Found 5 ports.
    [Users/wfc] >
  3. Now that we know the spelling, install the right one for our version of Perl.
    [Users/wfc] > install p5.12-catalyst-devel
    (many dependancies installed)
    [Users/wfc] > quit
  4. Looks like it worked, let’s see if it will talk to us.
    Usage: [options] application-name ...

That’s pretty much it. As you progress with learning Catalyst you’ll discover that you’ll need to start installing ‘optional’ modules, such as your choice of a templater. The drill is pretty much the same, search for the package to get the exact spelling and install it.

Starting a Perl Module

Let’s say you already have a Perl application that is several years old. It grew organically using organizational conventions that you invented along the way. But with the next new module for your app, you’d like to do it ‘the right way’ even if you have no intentions of distributing your module via CPAN. Here’s a recipe that might work for you…

  1. Install Module::Starter
    This module, and its command line utility, provides an easy way to create a directory with all the standard support files for your new module. See this article for more background.
  2. Build the Skeleton
    Create a source directory in a convenient location and create the distribution framework for your new module. For my app, I located this directory within the app’s own tree. Move to your new source directory and do:

    $ module-starter --module=MyNewModule --author="Your Name" \
    > --builder=Module::Build

    In this example we are specifically asking for Module::Build which is a pure Perl (and thus a portable) replacement for the Unix make utility. If you are already familiar with the venerable module ExtUtils::MakeMaker used for building distributions, see this article which compares and contrasts with using the original make based builder/installer. If you are just getting started with Perl module building, see the Module::Build POD for more information about general usage and authoring.

  3. Customize the Install Action for Your App’s Environment
    The goal is to be able to build, test and install the next new version of our module, by doing:

    $ ./Build                   # Build our distribution
    $ ./Build test              # Test our distribution
    $ ./Build install           # Install our distribution

    with that last step installing your new module into wherever your application keeps its own modules.

    Now out of the box, the install action would want to install our module and documentation into standard system libraries. To see this, let’s create our initial default Build script and see where that would be:

    $ perl Build.PL
    Checking whether your kit is complete...
    Looks good
    Checking prerequisites...
    Looks good
    Creating new 'Build' script for 'MyNewModule' version '0.01'
    $ ./Build fakeinstall
    Files will not be installed.
    Installing /usr/lib/perl5/site_perl/5.8.5/
    Installing /usr/share/man/man3/MyNewModule.3pm
    Writing /usr/lib/perl5/site_perl/5.8.5/i386-linux-thread-multi/auto/MyNew...

    To change the install destinations, recreate the Build script by adding the following options to the command line:

    $ perl Build.PL \
    > --install_path lib=../../Lib  \
    > --install_path libdoc=/var/www/html/man3 \
    > --install_path libhtml=/var/www/html \
    > --create_packlist 0
    Checking whether your kit is complete...
    Looks good
    Checking prerequisites...
    Looks good
    Deleting Build
    Removed previous script 'Build'
    Creating new 'Build' script for 'MyNewModule' version '0.01'

    Now use fakeinstall to double check our results:

    $ ./Build fakeinstall
    Installing ../../Lib/
    Installing /var/www/html/man3/MyNewModule.3pm
    Installing /var/www/html/site/lib/MyNewModule.html

    That looks better.

  4. What Did We Just Do and Why Did We Do It?
    Here’s a description of the options used in building our localized Build script:
    • lib=../../Lib
      This causes the module to be installed in the application’s own module directory. Since I want my source files to be in the same directory tree as the application itself, I used a relative directory reference.
    • libdoc=/var/www/html/man3
      I really don’t want or need a standard Unix man page for my application’s modules. I initially tried just to set the path to /dev/null but this did not work, failing with the error “mkdir /dev/null: File exists“. I couldn’t readily find a way to just suppress the man page creation, so it was easier to just create placeholder directory for stashing these.
    • libhtml=/var/www/html
      In contrast I do want HTML documentation. Unfortunately using the default ‘site’ install configuration, it wants to automatically append /site/lib to the path specified. Again, we took the path of least resistance (groan) and have now standardized on that location for all our future module documentation.
    • create_packlist 0
      Lazy me got lucky here. I definitely did not want the install action creating a pack list in the system’s site_perl directory, but I couldn’t find a reference to a command line option for suppressing this. However in the authoring document there is a boolean create_packlist option for the new() object constructor. I took a guess that it could be specified on the command line and… it work!

    That’s not a lot of work to get our files to be installed where we want them.
  5. See If It Really Works
    Now really do an install action to make sure it works. Admire your shiny new installed skeleton module in your application’s module library, and HTMLized POD documentation, all courtesy of Module::Starter.

    In considering these customizations there are some important things to keep in mind:

    • These customizations should stay in place, so long as you do not do a realclean action, or regenerate the Build script without the custom options shown. Just ./Build, ./Build test, ./Build install, repeat.

    • If you want to distribute your customizations to others, then this is not the way to do it. Read the authoring document on how to customize your build, test, and install actions so they always do whatever it is you need to have happen.

    • A possible annoyance is if your new module depends upon peer modules
      that live in your application’s private lib directory. This will require some fun and games with the @INC path so that your new module will be executable in its source location as well as its installed location without having to tinker with the source after installation.

    Even with these caveats, the reason I like the approach illustrated above is because I’m lazy and these customized settings will easily and automatically revert back to the standard defaults should I decide to share this code.
  6. Further Customizations
    Version 0.3608 of Module::Build introduced the manifest_skip action which can be used to automatically generate a boilerplate MANIFEST.SKIP file if one does not already exist. Unfortunately my host’s RHEL comes with version 0.32, so I had to create one manually. See MANIFEST.SKIP in the POD for a good set of starter regular expressions. Things you might want to have automatically skipped could include site specific documentation and configuration settings that would only apply to your particular application.
  7. Adding Flesh to the Skeleton
    Now the fun begins. You’ve got a module file complete with some POD stubs you can fill in as you build your code, and some working test stubs, so you have no excuse not to be creating unit tests as you develop your code. If you want to be really radical — create your unit tests before you write your code!
Personally I don’t know if I will always start a module from scratch this way. After my first module is built and tested under this new (to me) regime, I may find myself cloning it rather than always starting from square one. But even so I’ve taken a small step to laying out code for my future modules in a way that an experienced Perl coder should immediately understand.

Panorama theme by Themocracy