Posts tagged: programming

Unit Testing is a Little Bit Like Flossing

You know you should floss, but you don’t have time to do it right now.  Sure, you floss when you have an annoyance, like that piece of pulled pork caught between your molars.  And you floss for a couple of days, after your dental hygienist has given you ‘the lecture’.  But in the end, you don’t have time to do it right now.  Maybe later.

Writing unit tests is a little bit like flossing.  You know it’s a good thing to do.  It may not feel like fun while you’re doing it, but you’re almost always glad you did it when you’re done.  Like flossing, if you’re going to do it, and do it consistently, you need to try and establish a habit.  And there’s no time like now to get started.

In the directory where you stash your modules, create a subdirectory named “t“.  Yes, you’ve seen this directory name before when installing a module from CPAN.  Don’t panic!  You’re not going to have to build a complete distribution.  You’re just creating a place to stash your tests that is both convenient and follows the conventions used by the Perl prove utility.  Now, the next time you create some new function or method in one of your modules, create a unit test for that function in your “t” directory.  For example, let’s say I’m creating a new utility function call mySum() in the module MyModule.  I’ll also create a new unit test script named mySum.t.  As I write the function in the module, I’ll also start writing the unit tests in the test script.  Writing some logic to handle an edge case?  Write a test to probe that edge.  Test it.  Right now!  Now code up the next hard part, and test that.  Here’s a simple example you can use as a starting template:

#!/usr/bin/env perl
#
#       File: t/mySum.t
#      Usage: prove mySum.t
#   Abstract: Test my wonderful MyMod::mySum() service.

use warnings FATAL => qw(all);   # Make all warnings fatal.
use strict;                      # Keep things squeaky clean.
use Test::More;                  # Lots of testing goodies.
use File::Basename;              # Easy basic filespec parsing.
use lib '..';                    # Where to find our own modules.

my $ownName = basename($0); # For error reporting.

die("? $ownName: no command line args expected\n_ ", join(' ', @ARGV), "\n_")
 if (scalar(@ARGV) > 0);

use_ok('MyMod') or exit(1); # The module we're testing.

is( MyMod::mySum(),                 0,         "sum of nothing is zero"       );
is( MyMod::mySum(undef(), undef()), 0,         "sum of two nothings is zero"  );
is( MyMod::mySum(1,2,3,'fred'),     undef(),   "fred is not a number"         );
is( MyMod::mySum(2,2),              4,         "two small positive integers"  );

my $x = 2;

is( MyMod::mySum($x,$x),            4,         "two small positive integers"  );
is( MyMod::mySum($x,\$x),           undef(),   "can't sum a scalar reference" );

# etc.

done_testing();

# EOF: mySum.t

The example above uses the is() test function which, along with ok(), covers a lot of the sort of simple test cases you’ll want to do.  But there are lots of other very useful test functions beyond these, so be sure to check out the Test::More documentation for more details.

To run your test, from your module directory, you can do either:

$ t/mySum.t
ok 1 – use MyMod;
ok 2 – sum of nothing is zero
ok 3 – sum of two nothings is zero
ok 4 – fred is not a number
ok 5 – two small positive integers
ok 6 – two small positive integers
ok 7 – can’t sum a scalar reference
1..7
$

…or….

$ prove t/mySum.t
t/mySum.t .. ok
All tests successful.
Files=1, Tests=7, 0 wallclock secs ( 0.02 usr 0.00 sys + 0.02 cusr 0.00 CPU)
Result: PASS
$

As your test library grows, you can just enter the prove command with no parameters and it will automatically run all of the *.t files it finds in your test directory.

So, don’t worry about back-filling all of those unit tests you should have written.  But do start getting into the habit of creating unit tests, function by function, as you create new code, particularly for those utility functions that have very narrow, well defined jobs.  For your convenience, here is the toy module that was used with the above unit test.

package MyMod;

# Simple demo module with simple demo function.

use warnings FATAL => qw(all);
use strict;
use Scalar::Util qw(looks_like_number);

sub mySum

#  Precondition: Parameters consist of zero or more numeric values.
#
# Postcondition: The sum of all the numeric values is returned as our
#                functional value.  Undefined parameters are silently
#                ignored.  We return undefined if one or more values
#                do not appear to be numeric (i.e. a usage error).  We
#                return 0 if there are no defined parameters.
#
#     Unit Test: t/mySum.t

{
  my $sum = 0;
  foreach my $x (@_)
    {
      next if (not defined($x));
      return() if (not looks_like_number($x));
      $sum += $x;
    }

  return($sum);
}

1;  # So endith your typical Perl module.

Programus Interruptus

This is why, when I have some particularly hard coding to do, I try and do it at home…

Why Programmers Work at Night

– Programmer Interrupted

Although now that my spouse is retired, it is getting more difficult to get long stretches of interrupted time at home.  I think there’s a potential market for programmer isolation pods.  An isolation tank could work, if you could solve the problem of needing a waterproof keyboard and mouse.

Perl pet peeves: named parameters for methods

A colleague was creating an app that used the Net::LDAP module, a module I have been using myself for a very long time now. In the code there was the statement:

my $msg = $ldap->search(bind_dn => $base_dn, filter => $filter);

…that was returning a “Bad filter” error. Turns out there was no problem with the filter text, but there was with the search statement—the parameter label bind_dn is not a valid keyword for this module.

I did something similar a long time ago, using a parameter label for the bind() method that actually belonged to new(). The parameter and value were accepted without comment. It didn’t cause an error, it was just silently ignored since it was totally unexpected.

As I’m sure you know, when passing named parameters to a method, what you are really passing is a hash. So:

filter => $filter

…is really a hash entry with ‘filter’ as the key getting assigned the value from the scalar $filter. Now what you would like is for the method to throw an error if you misspell a key, but for the Net::LDAP module, and some other CPAN code I’ve seen, no check is done to see if an unexpected key is defined. The justification for this is the possible use of inheritance. If you pass a “foo” parameter to an object, it may not know what to do with “foo” but one of the other methods inherited by that object may. That’s why you’ll see things like…

my($self, %opts) = @_;

and then later…

$self->someOtherMethod($this, $that, %opts);

As for my own code, I’m still working on coming up with a good way to used named parameters with methods where all of the methods that get a set of named parameters have to ‘claim’ the ones that that particular method is using, so that in the end, if there are any unclaimed named parameters, an exception is thrown. Haven’t come with what I think is a clean solution, but I’m still thinking about it.

I hope this is the sort of thing Moose fixes.

Cataloguing Perl Module Dependencies

So you have a bunch of Perl scripts and modules you’ve written over time, and you’d like to quickly determine what non-core Perl modules are required to make them run. Devel::Modlist to the rescue.

$ perl -d:Modlist=nocore some-funky-script
Module::ExtractUse 0.27
Module::ExtractUse::Grammar 0.25
Parse::RecDescent 1.967009
Pod::Strip 1.02
version::vxs 0.95
$

Unfortunately Devel::Modlist is not itself a core module, but it’s an easy install since it doesn’t have any dependencies of its own. The way it works is pretty clever. Rather than trying to parse the Perl scripts and modules, looking for “use” statements, it loads the code into the Perl debugger. This is a smart approach since the common wisdom is only perl can parse Perl.

But this is not a perfect solution. For one, this only works for code that already runs on your system, so you can’t use it to quickly list what modules are needed for some bit of Perl you just downloaded. It also cannot find and report on modules that get loaded dynamically, as would happen if you do something like:

if ($magicRequired) { require Magic; castSpell(); }

And finally this module is not designed to be used from inside of a Perl script since it would cause the dependency reporting every time that script is executed. The shell command line example shown above is the normal usage model for this module.

But even with those caveats, this is still a handy little tool to have in your Perl toolbox.

Installing Catalyst on OS X

I’ve been looking at Perl based web application frameworks again. About 4-5 years ago I needed to create a report card application for my spouse. I looked at Catalyst and CGI-Application, and decided on CGI-App because it was lighter weight and did not require a persistent process module like mod_perl or FastCGI. I liked CGI-App and found it easy to use and logical.  But recently I’ve been thinking that I needed to adopt an application framework for work projects, so I thought I’d look at Catalyst again for the first time.

To kick the tires on Catalyst, i.e. to try some of the examples found in ‘the book’, we’ll need to install the development version plus some ‘optional’ modules as they are introduced. For your entertainment and enjoyment, I’ve outlined below what it took to get the basics of Catalyst installed on my Mac laptop (OS X, version 10.6.8 aka Snow Leopard).  The prerequisite for these instructions is having MacPorts installed. MacPorts is the best way I’ve found for installing Unix/Linux goodies on my Mac. Be aware that MacPorts wants to install and maintain its own Perl in /opt/local/bin/perl which is a good idea. Your Mac depends on Perl for various miscellaneous system functions, so it is best not to mess with your system’s stock distribution of Perl.

  1. First off we need to know which version of Perl we’ll be installing Catalyst against.
    $ which perl
    /opt/local/bin/perl
    $ perl -v
    This is perl 5, version 12, subversion 3 (v5.12.3)...
    $
    

    This confirms that that the version of Perl in my search path is indeed the one maintained by MacPorts, and reveals its current rev. So in MacPorts, the distributions we’ll be interested in will be labeled with “p5.12″.

  2. Next we fire up the MacPorts utility in interactive mode and search for the exact name of the packages we’ll need. I’ll tell you right now — don’t search just for “catalyst”, it will be a long and overwhelming list!
    $ sudo port
    Password: *******************
    MacPorts 2.0.2
    Entering interactive mode... ("help" for help, "quit" to quit)
    [Users/wfc] > search catalyst-dev
    p5-catalyst-devel @1.310.0 (perl)
        Catalyst Development Tools
    
    p5.8-catalyst-devel @1.310.0 (perl)
        Catalyst Development Tools
    
    p5.10-catalyst-devel @1.310.0 (perl)
        Catalyst Development Tools
    
    p5.12-catalyst-devel @1.310.0 (perl)
        Catalyst Development Tools
    
    p5.14-catalyst-devel @1.310.0 (perl)
        Catalyst Development Tools
    
    Found 5 ports.
    [Users/wfc] >
    
  3. Now that we know the spelling, install the right one for our version of Perl.
    [Users/wfc] > install p5.12-catalyst-devel
         :
         :
    (many dependancies installed)
         :
         :
    [Users/wfc] > quit
    $
    
  4. Looks like it worked, let’s see if it will talk to us.
    $ catalyst.pl
    Usage: catalyst.pl [options] application-name ...
    $
    

That’s pretty much it. As you progress with learning Catalyst you’ll discover that you’ll need to start installing ‘optional’ modules, such as your choice of a templater. The drill is pretty much the same, search for the package to get the exact spelling and install it.

Starting a Perl Module

Let’s say you already have a Perl application that is several years old. It grew organically using organizational conventions that you invented along the way. But with the next new module for your app, you’d like to do it ‘the right way’ even if you have no intentions of distributing your module via CPAN. Here’s a recipe that might work for you…

  1. Install Module::Starter
    This module, and its command line utility, provides an easy way to create a directory with all the standard support files for your new module. See this article for more background.
  2. Build the Skeleton
    Create a source directory in a convenient location and create the distribution framework for your new module. For my app, I located this directory within the app’s own tree. Move to your new source directory and do:

    $ module-starter --module=MyNewModule --author="Your Name" \
    > --email=you@unh.edu --builder=Module::Build
    $
    

    In this example we are specifically asking for Module::Build which is a pure Perl (and thus a portable) replacement for the Unix make utility. If you are already familiar with the venerable module ExtUtils::MakeMaker used for building distributions, see this article which compares and contrasts with using the original make based builder/installer. If you are just getting started with Perl module building, see the Module::Build POD for more information about general usage and authoring.

  3. Customize the Install Action for Your App’s Environment
    The goal is to be able to build, test and install the next new version of our module, by doing:

    $ ./Build                   # Build our distribution
    $ ./Build test              # Test our distribution
    $ ./Build install           # Install our distribution
    $
    

    with that last step installing your new module into wherever your application keeps its own modules.

    Now out of the box, the install action would want to install our module and documentation into standard system libraries. To see this, let’s create our initial default Build script and see where that would be:

    $ perl Build.PL
    Checking whether your kit is complete...
    Looks good
    
    Checking prerequisites...
    Looks good
    
    Creating new 'Build' script for 'MyNewModule' version '0.01'
    $ ./Build fakeinstall
    Files will not be installed.
    Installing /usr/lib/perl5/site_perl/5.8.5/MyNewModule.pm
    Installing /usr/share/man/man3/MyNewModule.3pm
    Writing /usr/lib/perl5/site_perl/5.8.5/i386-linux-thread-multi/auto/MyNew...
    $
    

    To change the install destinations, recreate the Build script by adding the following options to the command line:

    $ perl Build.PL \
    > --install_path lib=../../Lib  \
    > --install_path libdoc=/var/www/html/man3 \
    > --install_path libhtml=/var/www/html \
    > --create_packlist 0
    Checking whether your kit is complete...
    Looks good
    
    Checking prerequisites...
    Looks good
    
    Deleting Build
    Removed previous script 'Build'
    
    Creating new 'Build' script for 'MyNewModule' version '0.01'
    $
    

    Now use fakeinstall to double check our results:

    $ ./Build fakeinstall
    Installing ../../Lib/MyNewModule.pm
    Installing /var/www/html/man3/MyNewModule.3pm
    Installing /var/www/html/site/lib/MyNewModule.html
    $
    

    That looks better.

  4. What Did We Just Do and Why Did We Do It?
    Here’s a description of the options used in building our localized Build script:
    • lib=../../Lib
      This causes the module to be installed in the application’s own module directory. Since I want my source files to be in the same directory tree as the application itself, I used a relative directory reference.
    • libdoc=/var/www/html/man3
      I really don’t want or need a standard Unix man page for my application’s modules. I initially tried just to set the path to /dev/null but this did not work, failing with the error “mkdir /dev/null: File exists“. I couldn’t readily find a way to just suppress the man page creation, so it was easier to just create placeholder directory for stashing these.
    • libhtml=/var/www/html
      In contrast I do want HTML documentation. Unfortunately using the default ‘site’ install configuration, it wants to automatically append /site/lib to the path specified. Again, we took the path of least resistance (groan) and have now standardized on that location for all our future module documentation.
    • create_packlist 0
      Lazy me got lucky here. I definitely did not want the install action creating a pack list in the system’s site_perl directory, but I couldn’t find a reference to a command line option for suppressing this. However in the authoring document there is a boolean create_packlist option for the new() object constructor. I took a guess that it could be specified on the command line and… it work!

    That’s not a lot of work to get our files to be installed where we want them.
  5. See If It Really Works
    Now really do an install action to make sure it works. Admire your shiny new installed skeleton module in your application’s module library, and HTMLized POD documentation, all courtesy of Module::Starter.

    In considering these customizations there are some important things to keep in mind:

    • These customizations should stay in place, so long as you do not do a realclean action, or regenerate the Build script without the custom options shown. Just ./Build, ./Build test, ./Build install, repeat.

    • If you want to distribute your customizations to others, then this is not the way to do it. Read the authoring document on how to customize your build, test, and install actions so they always do whatever it is you need to have happen.

    • A possible annoyance is if your new module depends upon peer modules
      that live in your application’s private lib directory. This will require some fun and games with the @INC path so that your new module will be executable in its source location as well as its installed location without having to tinker with the source after installation.

    Even with these caveats, the reason I like the approach illustrated above is because I’m lazy and these customized settings will easily and automatically revert back to the standard defaults should I decide to share this code.
  6. Further Customizations
    Version 0.3608 of Module::Build introduced the manifest_skip action which can be used to automatically generate a boilerplate MANIFEST.SKIP file if one does not already exist. Unfortunately my host’s RHEL comes with version 0.32, so I had to create one manually. See MANIFEST.SKIP in the POD for a good set of starter regular expressions. Things you might want to have automatically skipped could include site specific documentation and configuration settings that would only apply to your particular application.
  7. Adding Flesh to the Skeleton
    Now the fun begins. You’ve got a module file complete with some POD stubs you can fill in as you build your code, and some working test stubs, so you have no excuse not to be creating unit tests as you develop your code. If you want to be really radical — create your unit tests before you write your code!
Personally I don’t know if I will always start a module from scratch this way. After my first module is built and tested under this new (to me) regime, I may find myself cloning it rather than always starting from square one. But even so I’ve taken a small step to laying out code for my future modules in a way that an experienced Perl coder should immediately understand.

Learning To Program Takes Time

Learning to program takes time… a lot of time. Professors of four year CS programs have lamented the meager progress most can make in that time. Here’s the creator of C++, Bjarne Stroustrup, in an interview just published:

Kalev: You have been a professor at Texas A&M University for several years now. Which new insights and lessons regarding the design of programming languages has the communication with fresh CS students brought to you?

Bjarne Stroustrup : Not as much as I had hoped. Students generally have a somewhat shallow view of programming and programming languages. Bringing students (both undergraduates and graduates) up to date with modern problems and techniques is hard and time consuming.

Tells it like it is, doesn’t he?

The article is well worth a read, but does have a high geek quotient and mostly focuses on the new C++ standard.  Still, you might learn what ‘SCARY iterators’ are… if you dare.

Article: “You’re a Bad Programmer. Embrace It.”

While browsing a Groovy/Grails oriented website recently, I came across this article which I felt obligated to share with everyone here at Software @ UNH. The title of the article is You’re a Bad Programmer. Embrace It..

How many developers think they’re good programmers? We’re not. We’re all fairly bad at what we do. We can’t remember all the methods our code needs to call, so we use autocompleting IDEs to remind us. And the languages we’ve spent years, in some cases decades, learning? We can’t even type the syntax properly, so we have compilers that check and correct it for us.

Don’t even get me started on the correctness of the logic in the code… I’m looking at a chart that shows failure rates for software projects. “Small” projects (about a year for 8 people) have a 46% success rate. That’s almost half. Any longer project just plummets down to a 2% rate for projects costing 10 million or more. In other words, our projects nearly always fail.

Enjoy :-)

CVS vs Bazaar vs Darcs vs Git vs Mercurial vs Subversion

I had no idea there were so many version control systems available. But it does make sense that there are. A text editor is an important tool for programmers, so it is no surprise there are a zillion of them. So I guess it should follow that in the quest for the perfect version control system, programmers have been busy trying to create the optimal tool for this difficult problem.

The subject of this blog entry refers to this specific comparison table which is just one set of the possible combinations possible. See the Version Control Blog comparison page to build your own table from a selection of 29 different version control systems. Have fun comparing your current system to the latest fair haired boy getting all the buzz.

Hacker News

Hacker News is a news site maintained by a venture capital firm for web startups. It’s got a high signal to noise ratio, with most stories about the leading edge of web development.

Panorama theme by Themocracy