Since what you're asking falls into the "customization" category, I can't offer the code to everything you'll need to make it happen.   (Since we'd have to spend some time customizing a test installation in order to tell you what the code was)

But…  I can tell you that the file you're going to be working with to make the changes is "frontdoor.php".

At or around line #14 you can set the number of total products to show. 

$wp_query->query('category_name=products&showposts=24'.'&paged='.$paged);

Ideally, you'll want it to have that number set as a multiple of the # of columns you have.  The stock version shows 6 columns, so you'll see that the initial value is 24 (24 = 6 x 4).

At or around line #45, you can set the number of columns.  The line you're looking for is:

<?php if ($i % 6 == 0)

You'll change the "6" to however many columns you want.

Hopefully this helps.