Geo Tests: Setting Up Your Geo Test

Getting Started

Once you understand the concepts behind geo testing, launching your own test within Measured only takes a few steps. Multiple types of tests can be easily set up to test both the incrementality and contribution of your tactics.

This article covers the steps to create your own geo test.

Click here to read about the concepts and terms involved with geo testing.

Click here to learn about interpreting your geo test results.

Getting to Know the Layout

When you first go to the Geo Tests page in your account, you will see the main screen shown below. It contains various sections.

• New Test is in the upper-left corner and will launch the setup wizard for a new test.
• Market Selection shows the current parameters for the markets that will be involved in the tests you launch. You will also see when the next market selection update occur, which may affect the parameters and any scheduled tests that have not launched yet.
  
  Learn more about the market selection process for geo tests here.
• Recommended Tests are tests that may be useful based on recent factors like how much you have spent on the tactic or how much it has been contributing to your overall business metrics.

Icons within Recommended Tests will specify test conditions, like scale/holdout or manual/automatic. Hovering your cursor over these icons will show you what they signify.

In general, holdout tests will be recommended whenever possible since they provide a more cost-efficient testing solution for the majority of users.

• My Tests is a running log of all of your tests. For any tests listed here, you can click the arrow under the Detail column of its row to see more detailed information.
  If any test needs your attention, it will be tagged as Action Needed. Going to the Detail view will show you the instructions necessary to fix any present issues.
  • Active shows tests either presently running or that are having their results calculated after the test is complete.
  • Scheduled shows any upcoming tests you have planned, either as a draft or as a finalized version with a set launch date.
  • Finished shows an archive of your previous tests alongside relevant topline metrics found during the test.
  • Clicking Select a new test will open a full list of tests you have available beyond our initial recommendations.

Reference Details & Metrics

When looking at the Detail view for any test at any stage, you will see the reference details for along the top of the screen. These will provide helpful context for the test’s conditions, especially when analyzing previous test results.

• Type is the kind of test that will be run.
  • Holdout tests withhold media in certain geographic areas to see if this affects the rate of conversions. These have no cost to run, but will put a very small amount of sales/orders at risk. Their availability is based on if your tactic meets the threshold for how much it contributes to your overall orders or revenue.
  • Scale tests increase the media budget in certain geographic areas to see if a scaled media budget leads to a higher conversion rate. The cost of these tests is based on how much your media budget should increase to see statistically valid results.
• Implementation is the method of execution required to run the test.
  • Auto means Measured will perform tasks to set up and end the test in your ad platforms via API. These actions will modify your existing campaigns for test purposes. Be aware that this process will handle as much of the process as possible, though you may need to take some actions yourself.
  • Manual means you will need to take every step yourself in your ad platforms yourself at the beginning and end of the test. This includes excluding specific markets from existing campaigns for holdout tests, or cloning existing campaigns and setting scaled budgets for scale tests.
• Scale Cost is the estimated increase in your campaigns’ budgets for the test.
  • For scale tests, this is the extra spending being tested. The test’s purpose is to see if this increased amount leads to higher conversions.
  • For holdout tests, this dollar amount is always zero since that type of test is based around withholding spending instead of increasing it.
• Length is how long the test will be active.
  • Four weeks is usually enough time to draw conclusions, but this could extend to a maximum of eight weeks based on the tactic’s recent data.
  • Note that in addition to your selected dates, three days will be necessary prior to the test's launch to ensure valid conditions, and it will take roughly a week after the test's end to calculate the results.
  • If a test is ended prematurely, your results will only reflect the last complete week of the test’s duration.
• Spend is how much you had spent on this tactic in the 30 days prior to the test’s launch.
  • This amount is tied to the test’s start date. It may change between when you schedule a test and when the test begins, especially if scheduling was done far in advance. If the change is drastic enough, it will be reflected in your test details.
  • This metric also corresponds with your Scale Cost. Fluctuations in your Spend amount may also alter your Scale Cost between scheduling a test and its launch.
  • This metric will not change after the test begins. When reviewing a finished test, this number will only reflect the 30 day period before your test began.
• Contribution is the percent of your business’s total orders that this tactic has contributed in the 30 days prior to the test’s launch.
  • Similar to the Spend amount, this metric is tied to your test’s start date. It may fluctuate between when a test is scheduled and when it begins.
  • This metric determines whether or not a holdout test is available for your tactic. If the contribution amount drops below the holdout threshold between when that type of test is scheduled and its launch date, your client success partner will alert you.

Launching a New Test

Selecting Your Test

Under the My Tests section of the main Geo Tests page, click Select a new test to open a list of available options.

At the top of the screen, you can choose the conversion type for the tactic you’d like to test. This will determine the options that populate below.

You’ll also see how many tests you can run simultaneously. This is determined by analyzing your past business data to see how many of your potential markets meet our minimum requirement for being able to draw reliable conclusions.

When viewing your options, icons be shown by certain tests:

• Orange check marks denote Measured’s recommendations based on the tactic’s recent activity.
• The tool icon is for contribution-only tests, meaning they only measure the overall contribution to your incremental sales/orders and incrementality will not be determined.

Additionally, the columns along the top of the list provide the following information (explained further in the Reference Details & Metrics section):

• Type is the kind of test that will be run, either scale or holdout.
• Implementation is the style of execution required to run the test.
• Scale Cost is the amount your campaign budgets will need to increase for the test.
• Spend and Contribution are metrics representing the past 30 days’ worth of activity for the tactic involved in the test.

Scheduling Your Test

Once you’ve chosen a test from the list of available options, you will be able to schedule the dates when it will run.

Note: You will see the settings and reference details for your chosen test along the top of the page. By default, these are set to our recommendations. You can edit certain settings, but make sure you are aware of the impact of any changes you make.

Our recommendations present the safest test conditions for accurate results with minimal impact to your business, so editing them without fully understanding the consequences can have negative results.

Alongside the details about the test at the top of the screen, you’ll see three tabs covering further details:

• Scheduling is a calendar view where you can pick a date for the test to begin. (The end date will be set automatically based on your selection.) Date availability is based on the number of consecutive days that a test can be run, as marked in gray on the calendar view.
  Be aware that there will be week-long periods before and after the selected date range for test preparation and result calculation.
• Test Markets is a list of the geographic market areas that your test conditions will be applied to.
  You can change the default test markets to a different group. Our initial recommendation is what we have found to be the best for your test, but the alternatives we offer will be very similar to the default market group. This way, choosing an alternative market group will not affect your test’s parameters or the accuracy of your results.
  For more information on how these markets are selected, see the Geo Test Overview article.
• Instructions is a breakdown of operations that will take place, or in the case of a manual test, actions you will need to perform.
  • This section includes the Campaigns in the Test section, which lists all campaigns or ad sets involved in your test. For scale tests, the Daily Budget amount applied to your scaled campaigns can be found here.

After you have chosen dates for your test, you can either save your test as a draft or commit it to launching via scheduling. Saving as a draft is ideal for the sake of planning multiple tests over the course of a long period of time.

Please note that two tests cannot run simultaneously, so test dates cannot overlap with other tests, either drafted or scheduled. Drafting a test prior to actual scheduling will still make those dates unavailable for other potential tests.