I Split Test My Job Applications — You Should Too

Split testing, a common approach in digital advertising today, became my go-to job hunt method at the end of last month. And it can be yours, too.

Some context: I was in a rough spot. Personally, financially… things were, suffice to say, not doing so well. I needed a quick gig, and I figured I’d leverage my digital marketing skills to help with the job search.

I’ve always considered myself a business guy. I started my first in college (a promotions company, filled with booze, babes, and bad decisions) when I was 19, and I’ve been engulfed in entrepreneurship ever since.

Since then, I’d started two more businesses: an online teaching company, and a digital marketing consultancy. But things all came to a halt two months ago, when some financial troubles meant I needed to find a job. And fast.

The problem? I’d never really applied for jobs before. Hell, I hadn’t taken a single look at my resume since high-school. So I did what any self respecting marketer in 2019 would do — I turned to demographic targeting, multivariate split testing and data analysis to land myself a gig.

Here are the results of around a month’s worth of applications, and how you can do it too:

The Schedule

I applied to ten jobs per day. To increase statistical significance (and leverage some of my old academic skills) I randomized the list of jobs I found on Indeed. I filtered my results based on a few criteria:

No supplementary applications
Only jobs that fall under “Sales & Marketing”
Easily commutable locations

This still left me with a lot of available positions. I estimate at least 500 in the Greater Vancouver Area alone… but who’s counting, right?

Kidding. I did because I’m awesome.

Since I made this a one-month job application challenge, and the month of January has 31 days, I ended up applying to 310 jobs total. Here’s a visualization of how those job applications went.

Second & third interviews not pictured (for simplicity’s sake).

As you can see, my 310 job applications yielded me 29 phone interviews, 16 follow up interviews, and ultimately 6 job offers within my category. Of these offers, one paid $45K with full medical & benefits, and another actually offered me significantly more than I was expecting — $62K, which blew me away.

The entire process took a single month, and has probably been one of the more rewarding experiences of my life. I went through an enormous amount of personal growth during my interview prep (even moreso during the actual interviews themselves!), and I highly recommend a similar approach to anyone currently looking for work.

Here’s how I did it, step-by-step, so you can too.

The first step is to use a job portal like Indeed, where you can filter based on a couple parameters.

Compile a shortlist of 20–30 positions in your local area. Focus on only those jobs without supplementary applications.

This criterion is important because supplementary applications are, by and large, a pain in the butt. A single supplementary application can take twenty or more minutes to finish, whereas most of us can fill out at least ten regular applications in that same time period.

This would be fine if doing a supplemental increased your chances of receiving an interview by at least ten times. But it doesn’t. In fact, I’d say filling out a supplemental app actually decreases your chances of receiving an interview, because it foreshadows that the company you’re applying to doesn’t have it’s sh*t together.

Supplementary application portals are usually a product of poor design & legacy systems. Most portals were designed decades ago (before good OCR was available) and require you to make an account, fill out a bunch of redundant fields, and then painstakingly re-enter every field on your resume into their system for antiquated data processing purposes.

Ask yourself: is that really a company you want to work for? One that can’t be bothered to fix one of the most important aspects of their business — employee intake — and one that has you jumping through meaningless hoops before you’ve even interacted with them? No. I don’t like paying for someone else’s inefficiency, and you shouldn’t either. Avoid supplementaries like the plague.

Once you’ve compiled a shortlist of 20–30 positions in your desired area, it’s onto step 2.

Step 2: Create Templates

Here, go through each job listing and requirement. Look for common keywords and recurring themes — this is important, since a large component of the modern hiring process depends on algorithmic filtering of applicants based on keyword density.

Once you have a decent idea of what you need to do, head on over to resumake.io and create two or more high quality resumes using their available themes. My personal favorite (and the one that ended up getting me a job) is theme one, which you can see below:

Disclaimer: I’m not actually Dwight Schrute.

I suggest rearranging so that work experience is first, because that’s what most employers care about. Then, populate your past work descriptions with as many of the shared keywords and recurring requirements as possible.

Don’t lie, obviously. But include as many keywords as you can so you don’t get auto-filtered out of the applicant pool. Frame all of your work so that it’s as close as possible to the average desired job description (however, don’t go as far as making a custom resume for each job — that’s a huge time sink).

For example, strong keywords in digital marketing today are CPC, CPM, ROAS, and CPA. Each word is domain-specific marketing terminology that, when listed on your resume, implies strong experience and technical know-how. They’re also a few of the terms that algorithms look for when filtering you out from mass candidate lists, so I made sure to include them heavily in my own resume. Your particular industry terms will vary, but don’t forget to pepper them in as much as possible.

Next, create one or more variants. You can change things like keywords, section arrangement, job history, duties — even the way your name is arranged (middle first, anyone?). But the important thing is to keep the number of variables low, so you can more easily understand exactly what led to better performance later on.

In my own resume, I split test the order of my listed job duties. For example, one of the job descriptions on my first resume (variant 1) might been arranged like this:

Created high converting funnel creatives & designs with Photoshop
Managed an outside sales force with HubSpot CRM

Whereas my second would be arranged like this:

Managed an outside sales force with HubSpot CRM
Created high converting funnel creatives & designs with Photoshop

I did this with every piece of work experience I had and ended up with two significantly different-sounding resumes. One accentuated my sales experience, and the other was geared towards digital marketing work. Tweak your own resumes however you like.

Once you have one or more variants, you’re off to step 3.

Step 3: Log Everything

This is by far the most crucial step. Without keeping rigorous measurements of your results, you won’t be able to make informed decisions on which variants performed well and which variants didn’t. So be judicious in your logging — my spreadsheet looked something like this:

Business names blurred for confidentiality reasons.

I logged six things: the name of the company, the time I submitted the application (I tried to keep the same range of times for each submission to minimize variability), which resume variant I used (A or B), whether or not I was offered a phone interview (& a F2F interview), and whether or not I ultimately got offered the job. These were sufficient to provide the level of granularity I needed to accurately assess which variant performed best.

Once you have a detailed logging system, the most important thing is that you stick to it. It’s easy to miss a random, 2am application here or there, but if you leave more than that out you’re left with a potentially statistically significant void in the data that you won’t be able to recoup down the line.

Step 4: Discard the Losers, Keep the Winners

I ran my tests in batches of forty — that means twenty each for A and B resume variants — with four day assessment intervals in between. The four day assessment interval was necessary because most interview offers take a few days to come through, so I would move on to the next test while still waiting for data from my previous one.

In hindsight, this led to my results being even better than stated here. I still get calls for phone interviews probably around once a week; results from the the 310 applications are taking their sweet time to trickle in.

At the end of each four day period, I tallied up the number of interview offers, performed a quick division (number of interview offers divided by 20) and then compared results. Then I chose the winner and turned it into my A variant for the next batch.

The result? I ended up significantly increasing my application effectiveness over the 8-batch period (be warned, though — some of the increase is due to trickle-in from the previous test’s applications):

That’s it. All it took was some elementary mathematics and a process-oriented mindset. I’ve never really considered myself organizationally “gifted”, but after showing my approach to friends & family (and hearing their praise) I’m starting to rethink my personality.

Strength in Data

As marketers, our biggest strength today is data. We no longer need to intuit our target demographic, or spend weeks trying to predict our customer persona. Research is now easier than ever — instead of guessing at the solution like before, businesses today can merely run thousands of iterative tests until they determine the perfect market for their product.

This approach is faster, cheaper, and yields significantly more effective approximations than the old method, which is why large enterprises are beginning to employ huge data science teams in lieu of small groups of more traditional creatives (think Don Draper in Mad Men).

But you needn’t be constrained to utilizing this approach just for marketing. Out-of-the-box thinkers can gain from employing a similar line of thinking to everything, and job hunts are no exception.

As you can see, it doesn’t have to be hard either. Statistical thinking is within your grasp — with tools like Excel available to almost everyone, you can very quickly get your own tests up and running. So take the statistical approach today! You might just find yourself with a new career.

I’ll leave you with this touching quote from Chris Hart:

“All the statistics in the world can’t measure the warmth of a smile.”

And a similarly touching quote from Nick Wells:

“But they sure can measure the size of my wallet!”

Happy hunting!