How to A/B Test Telegram Ads

An A/B test compares two versions of an ad setup while keeping everything else as similar as possible. For Telegram advertisers, the variable can be the creative, the channels or inventory group, the time window, or the amount of repeated exposure.

The method sounds simple. The difficult part is protecting the comparison from other changes. If version B uses a new headline, different channels, a higher CPM, and a later time slot, a better result does not reveal which change caused it.

A practical Telegram testing program needs four things:

1. one primary variable per test;
2. one primary decision metric;
3. a budget calculated from the amount of evidence required;
4. a rule for confirming a winner before scaling it.

This guide provides that system. The numbers in the budget examples are planning assumptions, not universal Telegram benchmarks or platform minimums.

What A/B testing means across Telegram ad buying methods

Telegram advertising is purchased through several routes. The available controls differ, so the same test has to be implemented differently.

Buying route	How to create variants	Typical controls	Main limitation
Official Telegram Ads	Use Create similar ad, then change one element	Text, CPM, budget, selected target channels	Telegram does not document a dedicated A/B-test wizard
Manual channel placements	Book matched posts with channel administrators	Post creative, channel, publication time, repeat schedule	Unequal channel conditions and manual reporting can distort comparisons
Automated Telegram networks	Duplicate campaigns or creatives inside the platform	Depends on the network: creative, targeting, inventory, schedule, reporting	Control names and availability vary by platform

Telegram's official documentation says advertisers can create a similar ad with the same text and parameters, then change the element they want to test. It also says targeting parameters cannot be changed after an ad is created. A new similar ad is therefore required for a placement or targeting comparison.

The official platform reports views and the number of users who joined a channel or started a bot after seeing the sponsored message. Advertisers who need CPC or CPA beyond those events may require trackable links, bot start parameters, CRM records, postbacks, or an external analytics setup, depending on the buying route and destination.

Before building tests, make sure you understand the operational differences between the official platform, manual outreach, and automated Telegram networks.

Start every test with a hypothesis card

Write the test on one screen or one spreadsheet row before spending. A useful hypothesis card contains:

Field	Example
Business goal	Generate qualified bot starts
Primary variable	Creative hook
Control	"Track Telegram ad conversions without spreadsheets"
Variant	"See which Telegram placements produce customers"
Constant conditions	Same placement group, time window, CPM, CTA, destination, and budget
Primary metric	Cost per qualified bot start
Diagnostic metrics	CTR and CPC
Guardrail	Bot-to-qualified-lead rate must not decline
Budget per variant	$150, based on the planning calculation below
Stop rule	Stop for tracking failure, policy issue, or severe quality problem
Decision rule	Confirm the lower-CPA version in a second matched run

The primary metric must match the decision. CTR is suitable when the question is which message earns more clicks. CPA is better when the advertiser needs a lead, purchase, bot start, or another downstream action. CPM is a buying cost, not proof that the audience responded.

Microsoft experimentation researchers describe controlled experiments as comparisons in which control and treatment groups are made statistically equivalent. Telegram media buying rarely gives advertisers perfect random assignment, especially across independent channels. The practical response is to match conditions as closely as possible, document the differences, and treat weak comparisons as directional evidence.

A neatly labeled dashboard does not make a comparison controlled. If the audience, delivery conditions, or measurement changed with the creative, the result is a campaign comparison, not a clean creative test. Record the mismatch instead of giving the winner more certainty than the setup supports.

Sara Al Mansoori

AdTech Strategist at MangoAds.

The metric chain - CPM to CTR, CPC, conversion rate, and CPA

Use one consistent funnel for every variant: Spend -> Impressions -> Clicks -> Conversions

CPM

CPM is the cost of 1,000 impressions.

CPM = Spend / Impressions x 1,000

If a variant spends $150 and receives 25,000 impressions:

CPM = $150 / 25,000 x 1,000 = $6

CTR

CTR is the percentage of impressions that produce clicks.

CTR = Clicks / Impressions x 100%

If 25,000 impressions produce 200 clicks:

CTR = 200 / 25,000 x 100% = 0.8%

CPC

CPC is spend divided by clicks.

CPC = Spend / Clicks

Using the same example:

CPC = $150 / 200 = $0.75

When buying on CPM, CPC can also be estimated from CPM and CTR. Convert the percentage to a decimal first.

Effective CPC = CPM / (1,000 x CTR)

Effective CPC = $6 / (1,000 x 0.008) = $0.75

Conversion rate and CPA

Click-to-conversion rate shows the percentage of clicks that produced the defined action.

Conversion rate = Conversions / Clicks x 100%

CPA = Spend / Conversions

If 200 clicks produce 10 qualified bot starts:

Conversion rate = 10 / 200 x 100% = 5%

CPA = $150 / 10 = $15

The example uses a hypothetical $6 CPM, 0.8% CTR, and 5% click-to-action rate. Replace all three assumptions with your own historical data or conservative planning estimates.

The MangoAds CPC vs CPM guide explains the relationship between the buying models in more detail.

Calculate the test budget from the evidence you need

A useful budget question is not "What is the minimum I can spend?" It is "How much must each variant spend to produce enough clicks or conversions for the decision I need?"

Budget based on target clicks

Required impressions = Target clicks / Expected CTR

Budget per variant = Required impressions / 1,000 x Expected CPM

For 200 target clicks, 0.8% expected CTR, and $6 expected CPM:

Required impressions = 200 / 0.008 = 25,000

Budget per variant = 25,000 / 1,000 x $6 = $150

Two variants therefore require a $300 media budget before any reserve.

Budget based on target conversions

Target clicks = Target conversions / Expected conversion rate

Budget per variant = Target clicks x Expected CPC

For 10 target conversions, a 5% expected conversion rate, and $0.75 expected CPC:

Target clicks = 10 / 0.05 = 200

Budget per variant = 200 x $0.75 = $150

This is an operational planning method, not a statistical significance calculation. A small performance gap may require far more data than a large gap. If the decision carries substantial financial risk, calculate the required sample size with an experiment calculator or analyst before launch.

Three test budget tiers

The tiers below use a $150 cell, based on the hypothetical assumptions above. One cell means one variant receiving its planned evidence budget. Recalculate the cell for your CPM, expected CTR, conversion rate, and target evidence.

Tier	Example media budget	What it can support	Appropriate use
Lean directional test	$300	One two-variant test	Find a promising creative or reject a clearly weak idea
Standard 30-day program	$1,800	Four two-variant tests plus a larger confirmation phase	Test creative, placement, timing, and frequency sequentially
Expanded program	$3,600	More variants, repeated cells, or separate audience segments	Confirm results across inventory groups, GEOs, or funnel stages

Add a reserve for rejected creatives, delivery imbalance, tracking repairs, or a confirmation run. Do not split a lean budget across all four variables. Four underfunded comparisons usually produce more charts but fewer defensible decisions.

If your expected conversion volume is low, use CTR or a qualified click as the primary metric for the first test, then validate the winner against CPA over a longer period. Do not declare a business winner from CTR when the variant attracts low-quality clicks.

Test 1 - Creative

Creative testing asks which message earns the right response from the same audience under comparable delivery conditions.

Possible variables include:

• opening hook;
• value proposition;
• proof point;
• CTA;
• image or video where the buying route supports media;
• short copy versus a more explanatory native post.

Change one element per comparison. If the control uses a benefit-led hook and the variant uses a problem-led hook, keep the destination, CTA, placement group, timing, CPM, and budget stable.

Read the result through the funnel. Higher CTR with unchanged conversion rate can reduce CPA. Higher CTR with a sharply lower conversion rate may mean the hook overpromised or attracted the wrong intent.

Test 2 - Placement

Placement testing compares where the ad appears. Depending on the buying route, that may mean named public channels, channel categories, Telegram Mini App inventory, or matched groups of publisher channels.

Build comparable placement groups before launch. Match them on factors that could influence performance:

• audience topic and language;
• geography;
• channel size;
• recent reach or available inventory quality;
• ad format and position;
• publication day and time;
• pricing model and budget.

Avoid comparing one large finance channel on Monday morning with five small technology channels over a weekend and calling the result a placement test. The audience, concentration, timing, and number of publishers all changed.

For manual placements, compare normalized metrics such as CTR, CPC, conversion rate, and CPA rather than total clicks alone. For automated buying, preserve the same targeting rules except for the inventory dimension under test.

Test 3 - Timing

Timing tests should compare matched windows, not isolated clock hours. Telegram audiences cross time zones, and channel posts can continue accumulating views after publication.

Useful comparisons include:

• weekday versus weekend;
• local morning versus local evening;
• launch-day window versus a steady-state window;
• business hours versus after-hours for a B2B audience.

Use the audience's local time, not the media buyer's office time. Keep the creative and placement stable. Repeat each window when possible so that a single news event, publisher post, or unusual day does not decide the result.

Timing controls vary. Some platforms offer scheduling, while manual buys depend on the publisher and official Telegram Ads may require advertisers to manage active and on-hold states. Document the exact implementation in the test sheet.

Test 4 - Frequency

Frequency is the amount of repeat exposure to the same audience. It is often the hardest Telegram variable to isolate because user-level frequency data and caps are not available in every buying route.

Define the controllable version of frequency before the test. It might be:

• one placement versus two placements in the same matched channel set;
• one campaign wave versus a repeated wave after a fixed interval;
• a lower versus higher platform cap, if the platform exposes that control;
• a narrow versus wider rotation of fresh creatives over the same period.

Do not claim a user-level frequency test if you can only control post count. Call it placement cadence or repeated exposure instead. Track CTR, conversion rate, CPA, and unsubscribe or complaint signals when available. A second exposure can help recall, but repeated low-relevance ads can also reduce response and damage channel trust.

Frequency should be treated as a quality decision, not only a volume lever. If the second wave produces cheaper clicks but more unsubscribes or weaker lead quality, the apparent media efficiency may be borrowing value from the audience relationship.

Sara Al Mansoori

AdTech Strategist at MangoAds.

A sample 30-day Telegram ad testing plan

This plan uses the standard $1,800 example. Each initial A/B phase receives $300, or $150 per variant. The final phase receives $600 to repeat the strongest setup against its control. Adjust the amounts using your own cell calculation.

Days	Test	Keep constant	Budget	Decision
1-3	Setup and baseline	Tracking definitions, destination, naming, attribution window	No test budget assigned	Confirm that impressions, clicks, and conversions reconcile
4-9	Creative A vs B	Placement group, timing, CPM, frequency, destination	$300	Select the message with the better primary metric and acceptable guardrails
10-15	Placement A vs B	Winning creative, timing, budget, destination	$300	Select the inventory group with the stronger normalized result
16-20	Timing A vs B	Winning creative and placement, CPM, destination	$300	Choose the repeatable window, not the best isolated hour
21-25	Frequency or cadence A vs B	Winning creative, placement, timing, measurement window	$300	Choose the exposure level that balances CPA and audience quality
26-30	Confirmation and controlled scale	Winner versus original control under matched conditions	$600	Promote only a result that repeats without breaking guardrails

Do not force every phase to fit the calendar if delivery is slow. A test should end when it reaches its evidence target and covers the required time pattern, not because a spreadsheet says the phase ends on Friday.

The companion planning template can be copied into a spreadsheet and recalculated for each test cell.

Real campaign examples

A large Telegram Mini App traffic case

PropellerAds published a first-party case study in January 2025 covering two Telegram Ads campaigns for a gamified finance Mini App. The campaigns ran from September 20 to October 19, 2024. One reported 9,080,458 impressions, 18,921 conversions, and 2.77% CTR. The other reported 830,873 impressions, 4,272 conversions, and 5.204% CTR.

The second campaign had the higher CTR, but the two rows also differed in CPM, volume, and likely traffic mix. The public report does not establish a clean single-variable A/B test. The useful lesson is methodological: large numbers do not rescue a confounded comparison. Record every changed condition before attributing the difference to creative, placement, or another variable.

An early segment signal that needed more data

A 2026 CPV Lab walkthrough reported $3.71 in early spend and a few conversions from a Telegram traffic campaign, all attributed to Wi-Fi users in the tracker. That observation can generate a useful hypothesis: perhaps connection type identifies a different context or audience.

It should not trigger an immediate conclusion that Wi-Fi traffic is universally better. The sample was tiny, and the vendor itself recommended gathering more volume before separating segments. Early patterns belong in the next-test column, not automatically in the winner column.

How to decide whether a variant won

Use the decision rule written before launch. A simple hierarchy is:

1. Check tracking and delivery integrity.
2. Compare the primary metric.
3. Check conversion quality and other guardrails.
4. Review whether the variants received comparable conditions.
5. Estimate whether the observed difference is large enough to matter economically.
6. Repeat the winner against the control before scaling aggressively.

Do not stop a test only because one variant leads after the first few conversions. Do stop when tracking is broken, a creative violates policy, inventory quality is unacceptable, or the business guardrail is being harmed.

A winner is not permanent. Creative response, publisher inventory, auctions, seasonality, and audience familiarity can change. Save the date and conditions with every conclusion.

Where MangoAds fits

MangoAds is a CPM-based platform for running ads across Telegram channels with campaign targeting, automated channel matching, creative support, and performance reporting. It can reduce the manual work involved in distributing variants across multiple publishers.

The testing logic remains the advertiser's responsibility. Define the hypothesis, isolate the variable, preserve naming, calculate the evidence budget, and judge downstream quality. Automation makes execution easier; it does not turn a mixed comparison into a controlled experiment.

Start with a small, interpretable test. Confirm the result, document the conditions, and scale the combination of creative, placement, timing, and exposure that improves the business metric rather than the most flattering dashboard number.