Posts tagged: application development

Unit Testing is a Little Bit Like Flossing

You know you should floss, but you don’t have time to do it right now.  Sure, you floss when you have an annoyance, like that piece of pulled pork caught between your molars.  And you floss for a couple of days, after your dental hygienist has given you ‘the lecture’.  But in the end, you don’t have time to do it right now.  Maybe later.

Writing unit tests is a little bit like flossing.  You know it’s a good thing to do.  It may not feel like fun while you’re doing it, but you’re almost always glad you did it when you’re done.  Like flossing, if you’re going to do it, and do it consistently, you need to try and establish a habit.  And there’s no time like now to get started.

In the directory where you stash your modules, create a subdirectory named “t“.  Yes, you’ve seen this directory name before when installing a module from CPAN.  Don’t panic!  You’re not going to have to build a complete distribution.  You’re just creating a place to stash your tests that is both convenient and follows the conventions used by the Perl prove utility.  Now, the next time you create some new function or method in one of your modules, create a unit test for that function in your “t” directory.  For example, let’s say I’m creating a new utility function call mySum() in the module MyModule.  I’ll also create a new unit test script named mySum.t.  As I write the function in the module, I’ll also start writing the unit tests in the test script.  Writing some logic to handle an edge case?  Write a test to probe that edge.  Test it.  Right now!  Now code up the next hard part, and test that.  Here’s a simple example you can use as a starting template:

#!/usr/bin/env perl
#       File: t/mySum.t
#      Usage: prove mySum.t
#   Abstract: Test my wonderful MyMod::mySum() service.

use warnings FATAL => qw(all);   # Make all warnings fatal.
use strict;                      # Keep things squeaky clean.
use Test::More;                  # Lots of testing goodies.
use File::Basename;              # Easy basic filespec parsing.
use lib '..';                    # Where to find our own modules.

my $ownName = basename($0); # For error reporting.

die("? $ownName: no command line args expected\n_ ", join(' ', @ARGV), "\n_")
 if (scalar(@ARGV) > 0);

use_ok('MyMod') or exit(1); # The module we're testing.

is( MyMod::mySum(),                 0,         "sum of nothing is zero"       );
is( MyMod::mySum(undef(), undef()), 0,         "sum of two nothings is zero"  );
is( MyMod::mySum(1,2,3,'fred'),     undef(),   "fred is not a number"         );
is( MyMod::mySum(2,2),              4,         "two small positive integers"  );

my $x = 2;

is( MyMod::mySum($x,$x),            4,         "two small positive integers"  );
is( MyMod::mySum($x,\$x),           undef(),   "can't sum a scalar reference" );

# etc.


# EOF: mySum.t

The example above uses the is() test function which, along with ok(), covers a lot of the sort of simple test cases you’ll want to do.  But there are lots of other very useful test functions beyond these, so be sure to check out the Test::More documentation for more details.

To run your test, from your module directory, you can do either:

$ t/mySum.t
ok 1 – use MyMod;
ok 2 – sum of nothing is zero
ok 3 – sum of two nothings is zero
ok 4 – fred is not a number
ok 5 – two small positive integers
ok 6 – two small positive integers
ok 7 – can’t sum a scalar reference


$ prove t/mySum.t
t/mySum.t .. ok
All tests successful.
Files=1, Tests=7, 0 wallclock secs ( 0.02 usr 0.00 sys + 0.02 cusr 0.00 CPU)
Result: PASS

As your test library grows, you can just enter the prove command with no parameters and it will automatically run all of the *.t files it finds in your test directory.

So, don’t worry about back-filling all of those unit tests you should have written.  But do start getting into the habit of creating unit tests, function by function, as you create new code, particularly for those utility functions that have very narrow, well defined jobs.  For your convenience, here is the toy module that was used with the above unit test.

package MyMod;

# Simple demo module with simple demo function.

use warnings FATAL => qw(all);
use strict;
use Scalar::Util qw(looks_like_number);

sub mySum

#  Precondition: Parameters consist of zero or more numeric values.
# Postcondition: The sum of all the numeric values is returned as our
#                functional value.  Undefined parameters are silently
#                ignored.  We return undefined if one or more values
#                do not appear to be numeric (i.e. a usage error).  We
#                return 0 if there are no defined parameters.
#     Unit Test: t/mySum.t

  my $sum = 0;
  foreach my $x (@_)
      next if (not defined($x));
      return() if (not looks_like_number($x));
      $sum += $x;


1;  # So endith your typical Perl module.

Cataloguing Perl Module Dependencies

So you have a bunch of Perl scripts and modules you’ve written over time, and you’d like to quickly determine what non-core Perl modules are required to make them run. Devel::Modlist to the rescue.

$ perl -d:Modlist=nocore some-funky-script
Module::ExtractUse 0.27
Module::ExtractUse::Grammar 0.25
Parse::RecDescent 1.967009
Pod::Strip 1.02
version::vxs 0.95

Unfortunately Devel::Modlist is not itself a core module, but it’s an easy install since it doesn’t have any dependencies of its own. The way it works is pretty clever. Rather than trying to parse the Perl scripts and modules, looking for “use” statements, it loads the code into the Perl debugger. This is a smart approach since the common wisdom is only perl can parse Perl.

But this is not a perfect solution. For one, this only works for code that already runs on your system, so you can’t use it to quickly list what modules are needed for some bit of Perl you just downloaded. It also cannot find and report on modules that get loaded dynamically, as would happen if you do something like:

if ($magicRequired) { require Magic; castSpell(); }

And finally this module is not designed to be used from inside of a Perl script since it would cause the dependency reporting every time that script is executed. The shell command line example shown above is the normal usage model for this module.

But even with those caveats, this is still a handy little tool to have in your Perl toolbox.

Yet Another Documentation System…

I’ve used way too many different documentation systems in my time including RUNOFF, nroff, DECdocument, Interleaf, TeX/LaTex, and PageMaker, just to name some of the older ones. These days my documentation tends to be either just plain old text, org-mode (emacs), HTML, or Perl POD. I find myself, however, once again looking for the ultimate solution for my document formatting needs in anticipation of having to create a lot of documentation for a major project.

At first I was thinking of standardizing on org-mode since that is what I tend to use by default now. Its table editor rocks and the syntax is very readable like Markdown, and other similar lightweight markup languages, with very easy export (for an emacs user) to HTML or LaTex. But I’m concerned about picking an Emacs-centric choice in deference to other maintainers, and the fact that org-mode’s own manual is formatted using GNU’s Textinfo gives me pause.

After casting about, including looking at Texinfo, DocBook, Muse (emacs), and reStructuredText, I find myself gravitating to Markdown and most likely the eventual adoption of Pandoc.

Here’s an article that I found particularly helpful. My current thinking is that the learning curve for Markdown is very shallow so I’m not making an unreasonable assumption for other maintainers, and I can start generating documentation now and worry about pretty formatting, structure and presentation later. I’m also trying to avoid this problem out of the gate…

The General Problem

I’m not happy about putting off the overall document architecture till later, but I’m burning daylight.

Installing Catalyst on OS X

I’ve been looking at Perl based web application frameworks again. About 4-5 years ago I needed to create a report card application for my spouse. I looked at Catalyst and CGI-Application, and decided on CGI-App because it was lighter weight and did not require a persistent process module like mod_perl or FastCGI. I liked CGI-App and found it easy to use and logical.  But recently I’ve been thinking that I needed to adopt an application framework for work projects, so I thought I’d look at Catalyst again for the first time.

To kick the tires on Catalyst, i.e. to try some of the examples found in ‘the book’, we’ll need to install the development version plus some ‘optional’ modules as they are introduced. For your entertainment and enjoyment, I’ve outlined below what it took to get the basics of Catalyst installed on my Mac laptop (OS X, version 10.6.8 aka Snow Leopard).  The prerequisite for these instructions is having MacPorts installed. MacPorts is the best way I’ve found for installing Unix/Linux goodies on my Mac. Be aware that MacPorts wants to install and maintain its own Perl in /opt/local/bin/perl which is a good idea. Your Mac depends on Perl for various miscellaneous system functions, so it is best not to mess with your system’s stock distribution of Perl.

  1. First off we need to know which version of Perl we’ll be installing Catalyst against.
    $ which perl
    $ perl -v
    This is perl 5, version 12, subversion 3 (v5.12.3)...

    This confirms that that the version of Perl in my search path is indeed the one maintained by MacPorts, and reveals its current rev. So in MacPorts, the distributions we’ll be interested in will be labeled with “p5.12″.

  2. Next we fire up the MacPorts utility in interactive mode and search for the exact name of the packages we’ll need. I’ll tell you right now — don’t search just for “catalyst”, it will be a long and overwhelming list!
    $ sudo port
    Password: *******************
    MacPorts 2.0.2
    Entering interactive mode... ("help" for help, "quit" to quit)
    [Users/wfc] > search catalyst-dev
    p5-catalyst-devel @1.310.0 (perl)
        Catalyst Development Tools
    p5.8-catalyst-devel @1.310.0 (perl)
        Catalyst Development Tools
    p5.10-catalyst-devel @1.310.0 (perl)
        Catalyst Development Tools
    p5.12-catalyst-devel @1.310.0 (perl)
        Catalyst Development Tools
    p5.14-catalyst-devel @1.310.0 (perl)
        Catalyst Development Tools
    Found 5 ports.
    [Users/wfc] >
  3. Now that we know the spelling, install the right one for our version of Perl.
    [Users/wfc] > install p5.12-catalyst-devel
    (many dependancies installed)
    [Users/wfc] > quit
  4. Looks like it worked, let’s see if it will talk to us.
    Usage: [options] application-name ...

That’s pretty much it. As you progress with learning Catalyst you’ll discover that you’ll need to start installing ‘optional’ modules, such as your choice of a templater. The drill is pretty much the same, search for the package to get the exact spelling and install it.

More than one way to skin the Database Cat

I’m in the planning stages of a Perl application where I need some sort of database engine back-end with pretty basic requirements. I’ve surprised myself in picking what, today, has become an unconventional choice; DBM. For long time Unix practitioners, DBM is well known and there have been many re-implementations of the original idea (NDBM, SDBM, GDBM, etc.). Whatever the iteration, the basic idea is the same; a library of routines that are loaded into the application’s own address space and provides a basic key/value based mechanism for storing and retrieving records. See Wikipedia DBM article for a brief explanation and history of DBM.

Of course many application developers have forgotten all about DBM, assuming that it is an obsolete technology. The assumption today has pretty much become that the database back-end engine will of course be a relational DB with an SQL query language interface. But between applications that need all the power and features of a full-blown relational database, and those that only need low-level read/write operations of general purpose file system, there is a middle ground that a DBM-like database engine fills very nicely.

As it turns out DBM style databases are not dead at all but are actively being developed. QDBM (Quick DataBase Manager) is just one of several modern DBM-like open source database libraries available for deployment. The QDBM web page graciously lists a short description of some of its “brothers” where author Mikio Hirabayashi writes:

There are many followers of UNIX DBM. Select the best suited one for your products. NDBM is ancient and you should not use it. SDBM is maintained by Apache Project, and GDBM is maintained by GNU Project. They are most popular and time-tested. TDB is maintained by Samba Team. It allows multiple simultaneous writers. While CDB does not support updating at a runtime, it is the fastest. Berkeley DB is very multifunctional and ACID compliant. It is used in many commercial products. Finally, QDBM is balanced of performance, functionality, portability, and usability.

For my own project I’ve decided to give Berkeley DB a try. It appears to be feature rich, well supported, and perhaps most importantly, is already installed on the system where I will be deploying my application. :-)

For more information about the DBM approach to database management for applications, and about Berkeley DB in particular, I recommend checking out the first chapter of the Berkeley DB Programmer’s Reference Guide. It’s an interesting read and spells out the case for when a DBM style database is, and is not a good fit an application.

Panorama theme by Themocracy