Startup Tools: Website Testing

To test browsers | landing pages | A/B | performance

Use the test runner you like best!

The only thing your test runner needs to do is to produce TAP output on console.log or process.stdout. Here's an example project using mocha.

You can also use tape to output TAP output in browsers and in node. Here is what a simple tape test looks like:

var test = require('tape'); var myCode = require('../my_code.js'); test('make sure my code works', function (t) { t.plan(2); t.equal(myCode.beep(5), 555); myCode.boop(333, function (n) { t.equal(n, 3); }); });

This code is run through browserify so you can run commonjs-style tests written for node and even published to npm.

Just make sure all the files your test will need are checked into git so that browserify can find them. All the dependencies specified in your `package.json` will be installed with npm so your tests can depend on npm modules in addition to files you've placed directly in your git repo.

If you use tape, make sure to add it to the `devDependencies` field of your `package.json` like so:

{ "name": "my-code", "version": "0.1.4", "devDependencies": { "tape": "~2.3.0" }, "testling": { "files": "test/*.js", "browsers": [ "ie8", "ie9", "ff/13", "chrome/20" ] } }

You'll need to add a "testling" field to a `package.json` file if your project doesn't already have one. Testling-ci looks for 4 keys in the "testling" field: "files", "scripts", "browsers", and "harness". Here is an example `package.json` that uses "files":

{ "name": "pretty-great-module", "version": "0.0.0", "devDependencies": { "tape": "~2.3.0" }, "testling": { "files": "test/*.js", "browsers": { "ie": [ 8, 9 ], "firefox": [ 13 ], "chrome": [ 20 ], "safari": [ 5.1 ], "opera": [ 12 ] } } }

The "files" field is an array or string of file globs which determine the commonjs-style test files to be run. Each file in "files" gets browserified so you can `require()` libraries with a node-style module system.

Each file in the "files" field should output its results as TAP output with `console.log()`.

Here's an example "files" array of globs:

"files": [ "test/*.js", "blarg/**/*.js", "beep/boop.js" ]

If you have more tranditional, non-commonjs-style javascript, use the "scripts" field instead of "files". Each file in "scripts" has the same globbing behavior as "files", but each file is merely placed into a <script> tag on the page instead of being compiled with browserify.

Here's an example of the "scripts" field:

"scripts": [ "scripts/**/*.js", "tests/*.js", "vendor/test/robot_test.js" ]

Each file in the "scripts" field should output its results as TAP output with `console.log()`.

The "browsers" field is an object mapping browser names to arrays of browser versions. Here is an example:

"browsers": { "ie": [ 8, 9 ], "ff": [ 12, 13 ] "chrome": [ 20 ], "safari: [ 5.1 ] }

The "browsers" field may also be an array of strings using an optional "/" to separate browser names from versions. Here is an example of a "browser" field as an array:

"browsers": ["ie8","ie9","ff/12","chrome/20"]

For some test harnesses you can have a "harness" field. The "harness" field sets up test runners that are otherwise tricky to get running with TAP output. Here are the currently supported harnesses:

• "mocha-bdd" - mocha with --ui bdd
• "mocha-tdd" - mocha with --ui tdd
• "mocha-qunit" - mocha with --ui qunit
• "mocha-exports" - mocha with --ui exports
• "mocha" - alias for "mocha-bdd"

If you specify dependencies in the "dependencies" or "devDependencies" flags, they will be installed with npm and each test file will be bundled with browserify before being executed in the chosen browsers.

To configure your github open source repository to use testling-ci, visit:

https://github.com/USER/PROJECT/settings/hooks

then under the "WebHook URLs" add

as a URL. The next time you `git push` new commits, your tests will be run in the latest commit!

If you would like to use testling-ci with your private github repos or with repos not even hosted on github, please contact us to enroll in our TESTLING-CI PROFESH EDITION beta program.

You can let others know which browsers your library works with by displaying a badge on the project readme with the current state of your browser tests and a link to the test status page.

In your `readme.markdown` file, put:

[](https://ci.testling.com/USER/PROJECT)

Your readme should now contain an image like this:

Run a free website speed test from multiple locations around the globe using real browsers (IE and Chrome) and at real consumer connection speeds. You can run simple tests or perform advanced testing including multi-step transactions, video capture, content blocking and much more. Your results will provide rich diagnostic information including resource loading waterfall charts, Page Speed optimization checks and suggestions for improvements.

If you have any performance/optimization questions you should visit the Forums where industry experts regularly discuss Web Performance Optimization.

Uptime matters

Pingdom has the infrastructure to do website monitoring from all over the world, as often as every minute. When something breaks, because it will, you're immediately alerted. We always double-check issues to avoid unnecessary disturbance. Once alerted you can notify your customers and dig in to see server response codes and web server output to analyze the situation.

This post is a primer on the delightful world of testing and experimentation (A/B, Multivariate, and a new term from me: Experience Testing). There is a lot on the web about A/B or Multivariate testing but my hope in this post is to give you some rationale around importance and then a point of view on each methodology along with some tips.

I covered Experimentation and Testing during my recent speech at the Emetrics Summit and here is the text from that slide:

Experiment or Go Home:
• Customers yell our problems (when they call or see us), they bitch, they rarely provide solutions
• Our bosses always think they represent site users and they want to do site design (which all of us promptly implement!!)
• The most phenomenal site experience today is stale tomorrow
• 80% of the time you/we are wrong about what a customer wants / expects from our site experience

That last one is hard to swallow because after all we are quite full of ourselves. But the reality is that we are not our site's customer, we are too close to the company, its products and websites. Experimentation and testing help us figure out we are wrong, quickly and repeatedly and if you think about it that is a great thing for our customers, and for our employers.

Experimentation and testing in the long run will replace most traditional ways of collecting qualitative data on our site experiences such as Lab Usability. Usability (in a lab or in a home or remotely) is great but if our customers like to surf our websites in their underwear then would it not be great if we could do usability on them when they are in their underwear?

It is important to realize that experimentation and testing might sound big and complex but it is not. We are lucky to live at at time when there are options available that allow us to get as deep and as broad as we want to be, and the cost is not prohibitive. There are three types of testing that are most prevalent (the first two are common).

: Usually this is a all encompassing category that seems to represent all kinds of testing. But A/B testing essentially represents testing of more than one version of a web page. Each version of the web page usually uniquely created and stand alone. The goal is to try, for example, three versions of the home page or product page or support FAQ page and see which version of the page works better. Almost always in A/B testing you are measuring one outcome (click thrus to next page or conversion etc). If you do nothing else you should do A/B testing.

How to do A/B Testing: You can simply have your designers/developers create versions of the page and depending on the complexity of your web platform you can put the pages up and measure. If you can't test them at the same time put them up one week after the other and try to control for external factors if you can.

Multivariate Testing: Currently the cool kid on the block, lots of hype, lots of buzz. In A/B above you had to create three pages. Now imagine "modularizing"? your page (break it up into chunks) and being able to just have one page but change dynamically what modules show up on the page, where they show up and to which traffic. Then being able to put that into a complex mathematical engine that will tell you not only which version of the page worked but correlations as well.

For example for my blog I can create "modules" / "containers" of the core page content, the top header, and each element of the right navigation (pages, categories, links, search etc). In a multivariate test I could move each piece around and see which one worked best.

Pro's of doing Multivariate Testing:
• Doing Multivariate turbocharges your ability to do a lot very quickly for a couple of reasons
  • There are free tools like the Google Website Optimizer or paid tools like Offermatica, Optimost, and SiteSpect who can help you get going very quickly by hosting all the functionality remotely (think asp model) such as content, test attributes, analytics, statistics.
  • You don't have to rely on your IT/Development team. All they have to do is put a few lines of javascript on the page and they are done. This is a awesome benefit because most of the times that is a huge hurdle.
• It can be a continuous learning methodology

Con's of doing Multivariate Testing:
• The old computer adage applies, be careful of GIGO (garbage in, garbage out). You still need a clean pool of ideas that are sourced from known customer pain points or strategic business objectives. It is easy to optimize crap quickly.
• Website experiences for most sites are complex multi page affairs. For a e-commerce website it is typical for a entry to a successful purchase to be around 12 to 18 pages, for a support site even more pages (as we thrash around to find a answer!). With Multivariate you are only optimizing one page and no matter how optimized it cannot play a outsized role in final outcome, just the first step or two.

Most definitely do Multivariate but be aware of its limitations (and yes the vendors will tell you that they can change all kinds of things throughout the site experience, take it with a grain of salt and take time to understand what exactly that means).

Experience Testing: New term that I have coined to represent the kind of testing where you have the ability to change the entire site experience of the visitor using capability of your site platform (say ATG, Blue Martini etc). You can not only change things on one page or say the left navigation or a piece of text on each page, but rather you can change all things about the entire experience on your website.

For example lets say you sell computer hardware on your website. Then with this methodology you can create one experience of your website where your site is segmented by Windows and Macintosh versions of products, another experience where the site is segmented by Current customers and New customers and another where the site is purple with white font with no left navigation and smiling babies instead of product box shots.

With experience testing you don't actually have to create three or four websites, but rather using your site platform you can easily create two or three persistent experiences on your websites and see which one your customers react to best. Since any analytics tools you use collect data for all three the analysis is the same you do currently.

Pro's of doing Experience Testing:
• This is Nirvana. You have an ability to test on your customers in their native environment (think underwear) and collect data that is most closely reflective of their true thoughts.
• If your qualitative methods are integrated you can literally read their thoughts about each experience.
• You will get five to ten times more powerful results than any other methodology

Con's of doing Experience Testing:
• You need to have a website platform that supports experience testing, (for example ATG supports this)
• It takes longer than the other two methodology
• It most definitely takes more brain power

Experience testing is very aspirational but companies are getting into it and sooner rather than later the current crop of vendors will start to expand into that space as well.

Agree? Disagree? Counter claims? Please share your feedback via comments.

What is a Landing Page?

A landing page is any page on a website where traffic is sent specifically to prompt a certain action or result. Think of a golf course... a landing page is the putting green that you drive the ball (prospect) to.

Once on the green, the goal is to get the ball into the hole. Likewise, the goal of the copy and design of a landing page is to get the prospect to take your desired action.

Here are a few examples of ways that landing pages are used with various traffic sources:

• Traffic is sent from a pay per click (PPC) search marketing campaign (such as Google AdWords) to multiple landing pages optimized to correspond with the keywords the searcher used.
• Traffic is sent from a banner ad or sponsorship graphic to a landing page specifically designed to address that target audience.
• Traffic is sent from a link in an email to a landing page designed to prompt a purchase.
• Traffic is sent from a blog post or sidebar link to a landing page that pre-sells affiliate products or encourages an opt-in to a sub-list.
• The page you're currently reading is a content landing page designed to organize many related pages around an overall theme.

Our Nine Chapter Landing Page Ebook Will Make You Smarter

This free guide will help you start creating killer landing pages right away:

Landing Pages: How to Turn Traffic into Money

• 5 Landing Page Mistakes that Crush Conversion Rates
• 10 Tips for Writing the Ultimate Landing Page
• 5 Tips for Designing the Ultimate Landing Page
• "Keep it Simple, Stupid" Applies to Your Landing Pages, Too!
• How is an Effective Landing Page Like a Direct Mail Letter?

• 7 Steps to an Email Opt-in Page That Works
• How Crappy Landing Pages Kill Email Campaigns
• How Successful is Your Landing Page? The 3 Key Metrics You Need to Know
• The 10 Commandments of Landing Pages That Work

Get this ebook (and more) free!

• Grab Landing Pages: How to Turn Traffic into Money.
• Also score instant access to 15 other high-impact ebooks on content marketing, copywriting, SEO, effect headlines, email marketing, and keyword research.
• Plus you'll get our 20-part course that lays out the proven framework for online marketing that works, sent directly to your inbox.

Want to add a lot of experiments on the fly and then let the tool figure out which one is the best? I thought so,too. With SubIntent, you're free to add as many variations you want, whenever you want. SubIntent will roll these new experiments into the mix automatically. You don't have to create a special experiment to handle your new variation. Just add and SubIntent takes care of the rest.

Desktop Mac applications:


For map geeks:For Erlang developers:

• Awesome web framework:
  
• Libraries:
  • BossDB - An evented ORM for Erlang
  • ErlyDTL - Django template language
  • Aleppo - Erlang Pre-Processor
  • Jerome - Rich-text processing library

• mod_circle_gif - Serve round corners with colors specified in the URL. The original Nginx module!
• mod_zip - Assemble ZIP archives on the fly. Originally developed for box.net in Palo Alto.
• nginx_upstream_hash - A hashing load balancer.
• mod_rrd_graph - Link RRDtool's graphing facilities directly into Nginx.
• nginx.vim - Syntax highlighting for the world's best text editor.

Engineering Blog:A/B testing articles:Applied math:Random musingsMiscellaneous software notes:Erlang references:
Nginx references:

February 5, 2014.

This year marks 37signals' 15th year in business. And today is Basecamp's 10th birthday. We have a lot to celebrate, and two exciting announcements to share. But first, let's set the scene with some history.

37signals was founded back in 1999 as a web design firm. With the release of Basecamp in 2004, we began our journey to become a software company. Once Basecamp revenue surpassed web design revenue in 2005, the transition was complete.

Since then we've launched Ta-da List (2005), Writeboard (2005), Backpack (2005), Campfire (2006), The Job Board & Gig Board (2006), Highrise (2007), Sortfolio (2009), the all new Basecamp (2012), Know Your Company (2013), and We Work Remotely (2013).

We also created and open sourced Ruby on Rails (2004), wrote a few books (Defensive Design for the Web (2004), Getting Real (2006), REWORK (2010), and REMOTE (2013)), and published thousands of blog posts on Signal vs. Noise.

Fifteen years into it, we're proud of the work we've done and the business we've built. And business has never been better.

However, because we've released so many products over the years, we've become a bit scattered, a bit diluted. Nobody does their best work when they're spread too thin. We certainly don't. We do our best work when we're all focused on one thing.

Further, we've always enjoyed being a small company. Today we're bigger than we've ever been, but we're still relatively small at 43 people. So while we could hire a bunch more people to do a bunch more things, that kind of rapid expansion is at odds with our culture. We want to maintain the kind of company where everyone knows everyone's name. That's one of the reasons why so many of the people who work at 37signals stay at 37signals.

So with that in mind, last August we conducted a thorough review of our products, our customer base, our passions, and our visions of the company for the next 20 years. When we put it all on the table, everything lined up and pointed at one clear conclusion. We all got excited. We knew it was right.

So today, February 5, 2014, exactly ten years to the day since we launched Basecamp, we have a couple of big announcements to make.

Here's the first: Moving forward, we will be a one product company. That product will be Basecamp. Our entire company will rally around Basecamp. With our whole team - from design to development to customer service to ops - focused on one thing, Basecamp will continue to get better in every direction and on every dimension.

Basecamp is our best idea and our biggest winner. Over 15 million people have Basecamp accounts, and just last week another 6,622 companies signed up for new Basecamp accounts. Ten years into it, Basecamp keeps accelerating. We've had other big hits, but nothing quite like Basecamp.

When we meet people, and they ask us what we do, we say we work for 37signals. If they aren't in the tech world, they'll squint and say "what's that?". When we say "we're the folks who make Basecamp", their eyes light up and open wide. "Basecamp! Oh I love Basecamp! My wife uses Basecamp too! Even our church uses Basecamp!" We hear this kind of response over and over. People just love their Basecamp.

So that got us thinking... While 37signals is well known in tech circles, far more people around the world actually know us for Basecamp. And since we're going to be completely focused on Basecamp moving forward, why don't we just go all in on "Basecamp".

So here's the second big announcement: We're changing our name. 37signals is now Basecamp. "37signals" goes into the history books. From now on, we are Basecamp. Basecamp the company, Basecamp the product. We're one and the same.

With this change, we renew our long-term commitment to all things Basecamp. Basecamp on the web, Basecamp on iOS, Basecamp on Android, Basecamp via email, and Basecamp wherever else it makes sense. Each one of us will be dedicated to improving Basecamp, extending Basecamp's reach, expanding Basecamp's capabilities, and making sure our Basecamp customers are treated like royalty.

And we'll never forget what made Basecamp so popular in the first place: It just works. It's simple, it's easy to use, it's easy to understand, it's clear, it's reliable, and it's dependable. We'll continue to make it more of all of those things.

The last fifteen years have been a blast, but with every future moment focused on Basecamp, the next fifteen are going to be even better. We're fired up! We've already got loads of new Basecamp stuff cooking.

Please visit our brand new site at http://basecamp.com and have a look around. If you're not a Basecamp customer yet, give us a try. We'd love to have you. If you already are, we thank you for your business.

Standing by to serve you for decades to come.

Thank you from all of us at Basecamp.

-Jason Fried
Founder & CEO Basecamp

Q: If you're going all in on Basecamp, what happens to Campfire and Highrise?

In the short term, everything stays the same. Business as usual. No interruption in service, no changes that affect our customers.

In the long term, one of three things:

SCENARIO 1: We'll spin them off into separate companies where we'll retain partial ownership, but another fully-dedicated team will run the products and own the majority of the company. This would be our ideal situation as it would ensure continuity and no interruption for our customers, but we'd have to find the right entrepreneur/team with the right experience and enough financing to make it work.

SCENARIO 2: We'll sell the products outright (either separately or together). The key for us in this scenario is that the products, and our customers, are well looked after. We will not sell either of these products to a company that is planning to shut the products down. And since no one from our team goes with the sale, this is not an acqui-hire situation. We're looking to sell to a company that wants to add well-respected, well-established, profitable, growing products to their portfolio.

SCENARIO 3: If we can't find the right partner or buyer, we are committed to continuing to run the products for our existing customers forever. We won't sell the products to new customers, but existing customers can continue to use the products just as they always have. The products will shift into maintenance mode which means there will be no new development, only security updates or minor bug fixes. We did this successfully in 2012 with Ta-da List, Writeboard, and Backpack, so we know how to make it work.

If you're a company or team interested in exploring scenario 1 (spin-off) or scenario 2 (outright purchase), please get in touch. Based on current revenues, current growth rates, and a conservative multiple, Campfire will sell in the single digit millions, and Highrise will sell in the tens of millions, so serious inquires only please. Disclosure: We're currently in early discussions with a few interested parties.

Q: What about Basecamp Classic?

We are fully committed to running Basecamp Classic forever. As long as we're around, Basecamp Classic will be around. A large chunk of our customer base loves Classic and we'll make sure they'll always have their Classic. The same rigorous uptime standards of Basecamp also apply to Basecamp Classic.

Background: Basecamp Classic was the original version of Basecamp we launched in 2004. Then in 2012 we released the all new version of Basecamp. Customers had the choice to stay on the old version, transition to the new version, or use both.

Q: What about other stuff like your Signal vs. Noise Blog, your books, etc?

We will continue publishing to Signal vs. Noise, writing books, and sharing as we always have. We will also be launching another online publication this year called THE DISTANCE. More details on this later.

Q: Will you be downsizing your company as part of downsizing your product lines?

No - our entire team stays intact. There's more than enough work to go around just on Basecamp alone. In fact, we're probably short a few designers now that we're fully committing to supporting Basecamp on multiple platforms. Plus, we're working on a variety of other tools and ideas that'll expand Basecamp in all new ways. We've also begun R&Ding entirely new, future versions of Basecamp, so there's a lot of really interesting work ahead for us. We'll be posting a few open positions in the coming months.

Q: This is a really unusual strategy. Can I talk to you about it?

From the very beginning we've done things differently. From switching from being a client services company to a product company, from being one of the early pioneers of the Software as a Service model, to open-sourcing our Ruby on Rails infrastructure, to signing up thousands of customers without a single salesperson, to being bootstrapped and funded by customer revenues, to being based in Chicago instead of the valley, to having a remote workforce spread out across nearly 30 cities across the world, to eschewing over 100 VC and private equity investment offers over the years (note: we did sell a small piece of the company to Jeff Bezos in 2006), to keeping our company as small as possible when "go big or go home" is all the rage, to writing a New York Times Bestselling business book (REWORK, 2010), etc. We're used to these unusual moves.

If you're a journalist who's interested in a business story unlike any other, please get in touch with our CEO (Jason Fried) at [email protected].