Experiment report
The report about the experiment shows the current (for the active experiment) or final (if the experiment was stopped) data about it.
The head of the report contains common data about the experiment:
- Name (1)
- Description (2)
- Key (3)
- Current state (4) - the value is "Active Experiment" if the experiment is still working and "Experiment Has Ended" if the experiment has finished.
- Experiment dates and duration (5).
- Experiment audience (6). If the match of players has not completed yet, the audience is divided into 2 part: the number of matched players and the maximum number of players in the experiment.
- How many players have confirmed participation in the experiment (7).
- Time to match the audience - approximate overall time to match experiment audience. The value is not shown if the matching has completed or if the system is not able to estimate the time (for example if the condition by version is set).
Information about the groups is shown as a bar. The tooltip provides what part of the total audience the group has.
Experiment groups option allows getting more details about each group. There is a separate tab for each group. The title shows a group's name and the size of the group (in %).
The form provides the following data:
- Group audience. If the match of players has not completed yet, the data is divided into 2 part: the number of matched players and the expected number of players in the group.
- Description
- Payload
KPI
KPI tab of the report shows the main stats collected during the experiment.
Game of Whales supports the collecting of the following KPIs:
- New users - The number of users included in the experiment when they joined the game.
- DAU (AU) - The number of unique active experiment participants during the experiment. DAU (AU) for every player increases only 1 time per 24 hours. If the experiment period longer than one day, DAU is shown as AU.
- Ads Revenue - The revenue from ads shown in the game. The value includes data from the experiment participants only.
- IAP Revenue - The sum of all gross transactions by the experiment participants.
- ARPDAU (ARPU) - The average revenue per active user ((IAP Revenue + Ads Revenue) / DAU (AU)). If the experiment period longer than one day, ARPDAU is shown as ARPU.
- ARPPU - The average revenue per paying user ((IAP Revenue) / Number of unique paying experiment participants). Each player increases the number of paying users only 1 time per 24 hours.
- Conversion Rate - The percentage of active users converting to paying users (Number of unique paying experiment participants / DAU (AU) * 100).
Every KPI shows:
- the total value for all groups (1)
- the leading group (2)
- the deviation of the results of the leading group from the results of group A (3). If group A is a leading group, the deviation is not shown.
Pay attention that the leading group and the deviation are defined based on normalized values against group A.
The tooltip shows the real KPI value for each group.
KPIs show the value for the whole experiment duration. The graph under the KPIs shows the data by days (or by hours if the experiment duration is less than 24 hours).
The data on the graph depends on the selected KPI.
The data for group A is shown as a line (1). The data for other groups are shown as bars (2).
If the experiment includes 2 groups only, the data for both groups are shown as bars.
Pay attention that the data on the graph is normalized against group A.
The tooltip for every point contains the value for each group at this point. The leader label is shown near the leading group for the KPI for the whole experiment.
Charts
IAP and Ads revenue collected during the experiment is shown in the report with different breakdowns:
- by top countries
- by revenue badge
- by session badge
- by frequency badge
- by platform
- by top products
Each line shows the value distribution between the groups (1) and the total result from all groups (2). The exact values are shown in the tooltip.
By default, the breakdown shows the data for the sum of IAP and Ads revenue. But it's possible to leave only one type by using a filter option near the chart.
Revenue Stream
Revenue stream section of the experiment report shows the list of purchases and ads milestones received during the experiment from the experiment members. The section contains the same data as the dashboard's revenue stream. In addition, each item in the stream has a special label to show the experiment group to which the player belonged.
The data on KPI tab of the report can be selected by using filters. The option to set filters is on the right top corner of the report.
Filters set on the KPI tab of the experiment report influence the data on KPI tab of the report only.
ARPU
The data about the experiment members ARPU is shown on the separate tab of the experiment report.
There are 3 types of ARPU here: IAP ARPU, Ads ARPU, and Holistic ARPU.
- IAP ARPU - the total IAP revenue of the specific user group gotten during the experiment / Number of new game users included in the experiment.
- Ads ARPU - the total Ads revenue of the specific user group gotten during the experiment / Number of new game users included in the experiment.
- Holistic ARPU - the total IAP revenue and Ads revenue of the specific user group gotten during the experiment / Number of new game users included in the experiment.
Pay attention that the data for APRU calculation is gathering while the experiment is active. After the finish of the experiment, the values will not be changed even if the players make new purchases or show the ads.
Each type of ARPU is shown as a separate KPI. It includes:
- the total value for all groups (1).
- the data about the leading group (2)
- the deviation of the results of the leading group from the results of group A (3). If group A is a leading group, the deviation is not shown.
The exact values for each group are shown in the tooltip.
The title of ARPU tab shows the leading group for Holistic ARPU.
There is ARPU breakdown by top countries under the KPIs. The data on the table depends on the selected type of APRU.
The information for each country is divided by the experiment group. The leading group for the country is marked by a special label. If there is no data for some group, it isn't shown.
There is a filter by countries above the table. It allows selecting the data in the table only for specific countries. If there is no data for the selected country, the line with zero values is shown.
Filters set on the KPI tab of the experiment report don't influence the data on ARPU tab of the report.
Retention
The third important part of the experiment report is the data about users retention. It's shown on a separate tab.
The report shows how many players included in the experiment as new game players returned to the game in specific days. The data for each group is shown separately.
The report supports three types of retention:
Each type is shown on a separate tab. The tab's title provides the data about the leading group for the appropriate type of retention.
The data about retention is shown as a graph.
There is a line for each experiment group. The point in the graph shows what part of players returned to the game in the appropriate day or period. The rules of calculation depend on the retention type.
Exact numbers are presented in the tooltip. The data for the leading group (for the whole experiment period) is marked by a special label.
Pay attention that the data for retention calculation is gathering while the experiment is active. After the finish of the experiment, the values will not be changed even if the players continue playing the game.
Filters set on the KPI tab of the experiment report don't influence the data on ARPU tab of the report.
The title of Retention tab shows the leading group for Rolling Retention.