Skip to main content


A/B Testing#

A/B tests are specialized statistical tests that allow testing of different variations to determine which variant is better. It can essentially be applied to anything, as it aims to answer the question "which one is better?"

For our case, following are examples of questions A/B tests aim to answer:

  • Should I price my product at 10₺ or 8₺?
  • Should I call my product "Lite" or "Premium"?
  • Should I describe my product as "life changing" or "valuable"?

A great way to answer any of these questions is to get feedback from users. A/B tests is a way to do that by splitting the users into two (control and variant) and expose each of these groups to a different product. Drawing from our previous example, an A/B test could be:

  • Group A sees the Product priced at 10₺
  • Group B sees the Product priced at 8₺

At the end of the test, we'd like to see how many people bought each variant and the one with more conversions is assumed to be the better option.

Let's assume Group B had a better conversion rate than A. Our app owners then can decide to move forward with option B (8₺).

What you can do with Appmate A/B testing?#

  • You can test A/B subscription prices and measure how they affect your revenue and lifetime value.
  • You can effortlessly determine winning in-app purchases and subscription prices.
  • You can test different trial periods and promotional offers to increase your conversion rate.
  • You can make smart decisions based on renewals.
  • Find the best pricing and paywall variations.
  • Prove hypotheses without additional software development.

Create An A/B Testing#

You can create an experiment by defining a start date, end date, audience, products, and split rate.

Experiment IDIt is a unique value. Once created, it cannot be updated again.String
ActiveDecides whether the test should continue. If it is disabled in the Running status, the status becomes Pause. Data collection is turned off. No data is collected to the Test report until it is enabled again.Boolean
Experiment NameThe name of A/B Test. Status cannot be changed when Running. It is only updated in WFSD status.String
StatusIndicates the test status. For more detail.String
Start DateIndicates when the test will start. At the earliest, one day later can be selected.Date
End DateIndicates when the test will end. At the earliest, one day later can be selected.Date
AudienceThe audience of A/B Test. For more detail.Audience
A/B Split RateIndicates what percentage of users in the selected audience will see the product in the Group A (control) . The rest see products in Group B (variant). It takes values between 95% and 5%.Rate
Group A Product (Control)The product you want to keep in the control groupProduct
Group B Product (Variant)The product you want to compare with the control group. You can set the product's isVisible property to false and only open it to the relevant percentage of the relevant audience. Other users will not see the product. You cannot select the product in the Group A (Control).Product

You can create an A/B Testing as shown below.


Update A/B Test#

You can make changes to the A/B testing you have created, but with some restrictions.

You can update all parameters of a Test in WFSD (i.e. Waiting for Start Date) status except Experiment ID.

When an experiment is RUNNING, you can only change the active status of a test. You cannot change the date, products, name, id and split rate.


For a test that has passed the start date and is not in FINISHED status, only the "active" status can be changed.


You cannot update any feature of an experiment in FINISHED status.


Technical Requirements of an A/B Test#

A/B tests rely on events to measure conversion rates. This section explains how events can be passed to Appmate.


Best practice for implementing events is to avoid any static declarations.

A dynamic implementation is required to allow no-code experimentation capabilities.

Appmate needs to receive user events for A/B testing calculation. It needs 2 events for A/B testing calculation. These are UserEventType.ViewProduct and UserEventType.PurchaseProduct.

You should send these two events to appmate via the saveAppUserEvents() api. You can view detailed information in Set Appmate Events section on left menu.

UserEventType.VIEWThe event to be sent when the product is displayed.
UserEventType.PURCHASEThe event to be sent when the product is purchased.

A/B Testing List#

You can see all done A/B Testing. You can monitor the status of A/B testing from the status value, update A/B testing if you wish, or view the report if you wish.



You can filter by ExperimentName , Status and Start Date.


You can refresh the page to see the current status of Experiment.

View Report#

You can go Experiment Report for selected Test

View Test Detail#

You can go Update A/B Testing for selected Test

A/B Testing Status#

Some test product selection scenarios have been written for illustrative purposes. You can take a look at the table.

WFSDWaiting for start date. It is the status value from the time you create the A/B testing until the start date.
RUNNINGStatus while A/B Testing is running
INACTIVEInactive. When the active switch box is set to false.
PAUSEIf switchbox "active" is set to disable while A/B testing is in RUNNING status, its status will be paused. During the pause, no data is collected, and the report is empty during this interval.
FINISHEDThe status value after the end-date.

After running status, you can update only in "active" switchbox state. If it is in WFSD state, you can edit everything except A/B testing id.

You Can Create An Audience For Your A/B Testing#

You can determine on which users the A/B testing will be performed and which audiences will participate in the test. When you do not select any audience, your test will be made to include all users. You can find detailed information about creating Audience in the Audience Managament section.

Sample Audiences-Product Scenarios#

Some test product selection scenarios have been written for illustrative purposes. You can take a look at the table.

Product A AudienceProduct A isVisibleProduct B AudienceProduct B isVisibleA/B Test AudienceHandle
Audience 1TrueAudience 1FalseAudience 2- Audience 1 sees product A - Audience 2 sees product A or B If there are users from audience 1 in audience 2, they should only see product A or product B. Therefore, in other steps of product list generation, holdout products should be removed.
Audience1TrueAudience 2TrueAudience 3- Audience 1 sees product A - Audience 2 sees productB - Audience 3 sees product A or B * If there are users from audience 1 or audience 2 in audience 3, they should only see product A or product B. If user 1 is in audience 3 and audience 1 user should see product A or product B due to A/B test.
Audience 1FalseAudience2FalseAudience 3- Audience 1 can't see product A - Audience 2 can't see product B - Audience 3 sees product A or B
Audience 1TrueAll usersFalseAudience1- Audience 1 can see Product A or Product BIf user in Audience 1, user see Product A or B due to contain A/B testing's userListA or userListB
All usersTrueAll usersFalseAudience1- Audience 1 sees Product A or B - All remaining users which are not include in audience 1 see Product A
All usersTrueAll usersFalseAll users- All users sees Product A or B due to A/B testing
All usersFalseAll usersFalseAudience 1- Audience 1 sees product A or B
Audience 1TrueAudience 2TrueAll users- All users sees Product A or B due to A/B testing
Audience 1TrueAll usersFalseAll users- All users sees Product A or B due to A/B testing
Audience 1TrueAudience 2TrueAudience 1- Audience 1 see Product A or B - Audience 2 see Product B If there are users from audience 1 in audience 2 or from audience 2 in audience 1, they should only see product A or product B.

Experiment Report#

This is the part where you see the test results. You see the data collected during the testing process in the form of tables and charts, and the A/B testing report gives you a final inference.


The report is generated one day after you create the test. And it comes out daily at the end of the day.

Chart: Chart gives you cumulative information about the percentage of viewers buying this product, day in and day out, by product. In this way, you can monitor the increase and decrease of user preferences by date, and you can decide to stop the test before the test is concluded.ab_testing_report_chartTable: In the table, you can see information such as *conversion rate*, *confidence interval* of the products in Group A and Group B, which will be useful for estimation.

Confidence Interval helps you estimate generalization when the product is available to your entire user base or selected audience. We would expect conversion rates to fall within the interval.

You can decide which product is advantageous to choose according to the chi-square test result created using this information. The Verdict value is the result of the statistical test and acts as a suggestion.

If the difference is not statistically significant, you may see a no difference verdict. This means, even though purchase counts and conversion rates may be different, no alternative is definitely better than the other.

Table ItemDescription
VisitsShows how many people viewed the product.
PurchasesShows how many people have purchased the product from viewers
Conversion RateIndicates how many of the product viewers purchased the product.
Confidence IntervalThe probability of purchase by those who view a product based on the number of samples.
VerdictChi-square test Estimation result. - It might be "No significant difference" or the winner product
Test Confidence Level%95


You can control which products will be shown to users with offering without requiring an application update. Creating paywalls that are dynamic and responsive to different product configurations gives you maximum flexibility to perform remote updates.

paywall_sampleMain characteristics of offerwalls / paywalls are as follows:
  • They tell about the benefits of a paid service
  • They can act as a blocker to content
  • They consist of at least one product, however it is often preferred to show at least two alternatives
  • IAP purchases can be directly triggered from paywalls

Appmate already includes all relevant product information (content) to fill a paywall. This feature has a grouping mechanism with additional metadata to bundle products into an offerwall / paywall. The app developer will then be able to query a specific offer via the SDK and get all the details about the paywall and products.

When end users are presented with a paywall, application administrators can also see reports for each offer.

What can you do with the offering?
  • Users can list offers.
  • Users can create new offers or edit existing ones.
  • Users can set an offer as the default.
  • Users can add metadata to their offers.
  • Users can edit, reorder and inactive offers.
  • The SDK should be able to invoke any offering by id. If no ID is submitted, the default offering is returned.

Add Offering#

Users can create a offering for the selected app via the appmate console.


The first offer you create is automatically set as the default offering.

Offering IdUnique offer id
DescriptionDescription of the offer
StatusThe status of the relevant offer being SDK active or passive. If you do not want to use an offer you have created, you can take the passive status.
MetadataUsers can add a metadata MetadataType to their offer
ProductsProducts can be added and sorted within the Offering.

Edit Offering#

The created offers can be updated by clicking on the related offer.


The status of the default offer cannot be deactivated. Offering Id cannot be updated once created


List Offerings#

You can see a list of all the offerings you have created in the console. It can set the default offering from this list. You can go to the create, edit and reporting pages.


Offering Report#

Users can view a report for each offer. A date period and currency must be selected for the bid report. The report includes a line chart with view and purchase numbers for each date. The report includes the following metrics:

  • Total Views
  • Total Purchases
  • Conversion Rate
  • Total Revenue
  • Product Based Purchase Counts
  • Product Based Revenues

Date PeriodDate range of the report
Report TypeSpecifies the y-axis of the report. Can be View, Purchases, Revenue, Conversation
Currency TypeRegistered currency types

View Select Offering#

You can access the offering information about the offeringId through your application. If you sent the offeringId empty, the default offering will be returned to you.

Parameter NameType
completion((OfferWall?, GenericError?) -> Void)
Parameter NameType

PurchaseClient.shared.getOfferWall("<OFFERING_ID>") {(offerWall, error) in     PurchaseClient.shared.getOfferWall("<OFFERING_ID>") {(offerWall, error) in            if let offerWall = offerWall {                print(offerWall)            } else {                print(error)            }        }}

if you have a support for html in your offerwall that you have created and wanted to get the html view instead the offerwall object you can call the method getHtmlOfferWall methode as follow.

PurchaseClient.shared.getHtmlOfferWall("<OFFERING_ID>", completion: { htmlView, error in                       // Do whatever you want with the html view                    }, onPurchaseDone: { product in                        // a callback informs that the purchase operation for a specific product is completed                    })

Parameter NameType
Parameter NameType
  • Labels: These are short strings that can be used for titles, bullet points and similar fields in a paywall. Block of Text: This is a formatted text. For MVP, only spacings and breaks should be considered.
  • Number: Could be used for int or float values.
  • Image: Users should be able to upload PNG or JPG/JPEG images. (max size: 5 MB) The image URL should be returned to the SDK.