Ksenia Mayorova, an expert in the development of international IT products, former head of IT products at Yandex and VKontakte, told how to effectively conduct experiments in companies so that they become a development tool
Experiments in business are systematic studies or practical tests of hypotheses. They are most often conducted to gain knowledge about a product, audience, to optimize business processes or production, to develop innovations and bring new solutions to market. They can involve products, marketing or operations.
In this article, we will talk about experiments related to launching new products/services or testing changes to them. Most often, they are carried out using the following types of tests:
A/B – division of the experimental object into two groups, where A is the standard version and B is the modified one;
A/A — division of an object into two identical groups to check the stability of the state;
A/B/C – using three options to compare the effectiveness of different changes;
switchback — a method of testing changes to a product in which innovations are enabled only for a limited group of users. This allows one to assess the impact of changes on their behavior and avoid undesirable network effects, when changes to a product affect not only an individual user, but also the interaction between users in the system.
Why conduct experiments?
In general, this approach acts as a driver of progressive changes and helps make strategic decisions. Let’s consider several situations when conducting an experiment would be appropriate:
It is important to avoid large-scale risks when launching a new product feature. Any changes can have both positive and negative consequences. For example, if you work with an online store and plan to launch a new order or payment form for customers. Data shows that people often abandon their purchases at the checkout (once they get to the checkout or payment page). This can happen, among other things, because the form is too complex. Therefore, testing the updated checkout process will be required, since any change in this section can greatly affect your profits.
It is necessary to accurately track how a new change increases business indicators. This request often occurs in corporations or when working with complex algorithmic products using machine learning (ML) models (a subdivision of artificial intelligence, where a machine is trained to recognize certain patterns or behave in a certain way based on the data provided). For example, with the help of ML, we made a new algorithm for assigning a courier to deliver food. And to check that in reality it will work better than the previous one, we conduct an experiment.
You need to test a hypothesis that will show which direction to take or resolve a controversial issue. In case of uncertainty within the company, experiments can serve as a tool for collecting objective data and making informed decisions. This is a scientific way to avoid disputes and one expert opinion. As an example: if you are launching digital customer acquisition funnels, testing several hypotheses will show which onboarding (introducing the user to the product) converts the user into a purchase better.
Need to improve user experience. Experiments can be useful for testing and improving customer experience on websites, mobile apps, and other platforms. For example, marketplaces are experimenting with audience engagement levels through personalized product recommendations. The process is based on user preferences, browsing history, and activity on the platform.
The company has a lot of resources. This is a large staff and financial capacity. For example, corporations such as Booking or Aliexpress can conduct dozens, if not hundreds of experiments in parallel, because they have such an opportunity. In addition, the development of innovation helps such companies to remain flexible.
When you shouldn’t experiment
While experiments are a great tool for business, there are situations where they may not be practical or even possible. Here are some examples:
Business works with a very small amount of data. In such a situation, the results may be statistically insignificant, which makes them uninformative. If the company works in the B2B segment or is a startup with 100 orders per month, then it is too early to think about quantitative experiments. In this case, it is more appropriate to conduct qualitative research: surveys, feedback analysis, interviews, etc.
The company has a limited budget. An experiment is always expensive and labor-intensive. To conduct it, a team of several people must be involved in formulating the correct hypothesis, metrics for tracking the result, and decision-making criteria. Someone must also check the accuracy of the data you receive. All this takes time, costs, and expertise. But it is not a fact that these actions will help achieve the desired result. Therefore, if financial and human resources are limited, it is more appropriate to think about making decisions based on previous experience or best practices in the market.
It is impossible to formulate a correct hypothesis. When conducting an experiment, this is the most difficult process: to understand what exactly you want to do, how to track the result and how it will affect further actions. A hypothesis requires thinking through the work several steps ahead, logic and interpretation of data into actions.
The company cannot find the necessary metrics. In this case, it is impossible to correctly measure and understand the result. If the metric is chosen incorrectly, you may not see the effect from it and the experiment will not be indicative.
Decisions need to be made quickly. The process can take a long time to produce meaningful results. Therefore, in situations where a quick response is required, conducting an experiment may not be practical.
What you need to know before starting an experiment to make it work
First of all, you need to understand that experiments are associated with risks. Very often, they are conducted to find out that changes will not break the main business indicators. Accordingly, you need to be prepared for the fact that the indicators may fall, and have a clear plan of action in advance if they decrease by a critical percentage for your business and industry.
Internal conditions are also important. This includes the creation of an innovative environment, support from managers, budget, team, flexibility in decision-making, access to data, including monitoring user behavior. A special role in the team is given to data-oriented specialists: Data Analyst, Data Scientist, Data Engineer and others. These are experts focused on working with information for its analysis and use in order to make business decisions.
If all of the above is in place, the next step is to learn the principles of conducting experiments.
1. Formulate and test the hypothesis correctly
The hypothesis should come from the problem of clients/users or assume where such a problem comes from. It should be based on knowledge, not on fantasies. It is also important to understand what metrics it can affect.
An example of a good hypothesis: “If we influence the cause, then metric X will change in such and such a direction due to the fact that the client will do something differently.”
An example of a bad hypothesis: “If we paint the booking button green, the conversion rate to creating a booking will increase.”
2. Decide what metrics will be used to judge the outcome
There are no universal metrics, their choice depends on the industry and stage of business development. You should go from large ones, such as the average check or session duration to the conversion funnel (for example, conversion to app installation, registration or purchase). You need to clearly understand: what you do directly affects the indicators you track, including the most important metric. In product businesses, this is the North Star Metric (NSM) – the only important indicator used to assess the overall success of the company and the achievement of its strategic goals. NSM is often associated with how users interact with the product and how well it meets their needs. This could be, for example, the number of active users, customer satisfaction level, total sales volume or another key indicator.
3. Design the experiment (type, sample, duration, geolocation, platforms)
Consists of key stages:
Select the type of experiment: A/B, A/A, A/B/C testing or switchback.
Determine the sample size for statistical significance. The characteristics of the target audience (gender, age, occupation, and other characteristics) may also be taken into account.
Determine duration, locations, and platforms. Includes choosing the timing to collect enough data (here you need to consider seasonality and other time factors that may affect the results). Selecting a geolocation (region, city, country) and platform: determine where the experiment will be conducted (web, iOS, Android) and adapt it to the features of the chosen platform.
4. Test to see if everything works as you intended.
If everything works, then start the process, selecting the person responsible for monitoring (usually a data analyst).
Personal experience
I will tell you using the example of a company that deals with small cargo transportation. In logistics products, in order to increase the share of completed orders, completeness of information about delivery is important, and much is tied to direct communication between the customer and the contractor. We noticed that one of the reasons for order cancellations is due to failure to call from one side or the other. I decided to test the hypothesis: if we give the sender the opportunity to provide the recipient’s phone number immediately when creating an order, this will reduce the share of cancellations, because the contractor will have the opportunity to call another number to clarify the delivery time and other details. To do this, we conducted an A/B test, and it turned out that such a change really has a positive effect. At the same time, it helped our users and improved business metrics.
At the end of experiments, you always need to understand what to do after calculating the results. In our case, we implemented the change to 100% of users and it became part of the product.
If you plan to establish a culture of experiments in your company, it is important to involve the team in this, collaborate around how to test the right hypotheses with minimal risks and resources. Use a scientific approach: for this, each team member must understand what a good hypothesis is, what types of experiments there are, and what metrics to look at. All this helps to understand that the results of the experiment can be trusted and changes can be implemented into the product.
