How to Split Test a Playlist Pitch (and Read the Results)

To split test a playlist pitch, send two versions of your message to two comparable groups of genre-matched curators. Change one element only, such as the opening line, and keep everything else identical: same song, same link, same timing, same follow-up. Decide before you send which single number counts as the win. Then accept that a small test often ends inconclusive, and that this is a real result.

That is harder than it sounds. Spotify’s own split-testing tool exists, but it is built for ads, not for the messages you send to curators. So an artist-run pitch test is a design problem you solve yourself. This guide shows how, with a worked example, a results table, and the mistakes that make most DIY tests meaningless.

TL;DR

• Spotify’s split testing belongs to its ad product. Testing curator pitches is something you design yourself, borrowing its rules: one variable, matched conditions, one chosen metric.
• Test one element at a time: subject line, fit sentence, or story versus sound description. Never rewrite the whole pitch and call it a test.
• Choose your metric before sending: curator reply rate, placement rate, or post-placement saves and follows. Raw streams cannot tell you whether the pitch or the playlist did the work.
• A handful of replies proves nothing. Record inconclusive results instead of forcing a winner.
• Run the test early. Spotify asks for the focus track at least two weeks before release, so the experiment has to finish before release week.

Table of Contents

• What a split test of a playlist pitch actually is
• What you should not claim about Spotify editorial playlists
• Pick one question before you write a single variant
• The variables worth testing
• Build matched groups
• Timing: finish the test before release week
• A worked example
• How much evidence is enough
• Read curator feedback as data
• Perspective: why we treat feedback as the real result
• Where a campaign fits into your testing
• Common mistakes that ruin a pitch test
• After placement: judging listener quality
• A repeatable checklist
• FAQ
• Sources

What a split test of a playlist pitch actually is

In marketing, a split test (or A/B test) compares two versions of one thing, randomly shown to similar audiences, to learn which performs better. Spotify’s guidance for Ads Manager describes it as a controlled comparison: audiences are divided randomly between variants, one variable changes, and targeting, budget and scheduling stay the same. It supports up to four variants and measures a KPI you choose, such as click-through rate, completion rate or cost per conversion.

That tool is for paid ads. It does nothing for the email or submission message you send a playlist curator. Nobody hands you a dashboard for that. What you can borrow is the logic, and the logic is what separates a test from an anecdote.

It also helps to separate three things people lump together under “pitching”:

1. Spotify’s official editorial pitch. You submit one unreleased song through Spotify for Artists, and Spotify’s editors consider it.
2. Independent curator outreach. You (or a service) send your track to people who run their own playlists.
3. Algorithmic discovery. Saves, skips and listener behavior feed recommendation features.

You can only A/B test the second one. The first is a single form, filled once per song. The third responds to what listeners do, not to what you write.

What you should not claim about Spotify editorial playlists

Be careful with language here, because it affects how you read results. Spotify states that no one can pay to be added to its editorial playlists; editors decide with listeners in mind. So a paid campaign can buy outreach, review or consideration from independent curators. It cannot buy a spot on Spotify’s own editorial lists, and any provider that implies otherwise should make you wary.

For your test, this means you should keep editorial, algorithmic and independent-curator outcomes in separate columns. If your song lands on an editorial list the same week you tested pitch wording, that was not your subject line.

Pick one question before you write a single variant

Spotify recommends matching the success metric to the objective. Awareness uses completion rate, traffic uses clicks, engagement uses streams, conversion uses cost-per-action metrics. The same guidance works for pitching if you translate it. Choose one primary question:

Pick one. If you optimize all three at once, you will find a story in the data whichever way it falls.

Raw stream counts are the weakest option. A song added to a playlist with 40,000 followers will outperform one added to a playlist with 400, regardless of how good your pitch was. Streams mix the pitch effect with the playlist effect, and you cannot separate them afterward.

The variables worth testing

A pitch has several parts. Test one per round.

Subject line or first sentence

Curators skim. The first line decides whether the rest gets read. Compare a plain fit statement (“Warm indie folk with fingerpicked guitar, for your late-night acoustic list”) against a story hook (“Written the week my studio flooded”).

Fit sentence

This is the line saying why the track belongs on this playlist. One version can name a mood and an instrument. The other can name a comparable artist already on the list. Same length, different content.

Story versus sound

This is the most useful test for most artists. Version A leads with the song’s origin or your background. Version B leads with what it sounds like. Keep the length equal, because a longer message is a second variable.

Metadata vocabulary

Spotify’s editorial form shows how it thinks about songs. According to Spotify’s published selection limits, you can pick one main genre, up to three subgenres, up to three cultures, up to two moods and up to two song styles, plus instruments, and flag covers, remixes and instrumentals. Those limits are historical, so check the live form before relying on every one. Even so, they hand you a vocabulary: describe your track in the words a curator uses to file music. Testing “two moods, specific instruments” against “loose adjectives” is a clean, useful comparison.

What not to test

Do not vary the track, the link type, the day of the week, or the curator genre between groups. Those are the confounders. Also avoid testing the length and the tone at the same time; if reply rates differ, you will not know which did it.

Build matched groups

Spotify’s rule is identical targeting, budget and schedule. Your equivalent: the two groups of curators must look alike.

• Same genre match. Do not send version A to indie playlists and version B to electronic ones.
• Similar playlist size. Split curators into small, mid and large tiers first, then divide each tier between A and B.
• Same market. A UK-focused list and a US-focused list behave differently, so balance by country too.
• Same send window. Send both versions in the same few days. Do not run A in March and B in April.
• Same follow-up policy. If A gets a nudge after a week, B does too.

Then randomize. The easiest way: list your curators in a spreadsheet, sort by tier, and alternate A, B, A, B down the list. Avoid choosing by hand which curator gets which version. People unconsciously send the better pitch to the curator they like best.

Spotify’s own reasoning applies. Without matched conditions, a better reply rate may reflect a better curator segment or a better release moment, not better copy.

Timing: finish the test before release week

The test competes with a hard deadline. Spotify for Artists says to pitch your focus track at least two weeks before release, including the release story, the sound, genre and vibe, and the playlists it suits. Separately, Spotify says a track pitched at least seven days ahead can automatically appear in your followers’ Release Radar. That is not the same as broader editorial placement.

So plan backward:

1. Six to eight weeks out: lock your track, artwork and metadata. Build your curator list.
2. Five weeks out: send Round 1 (one variable).
3. Four weeks out: collect replies, log results, pick the better version if there is a clear one.
4. Three weeks out: send the winning version to a fresh group, and submit your official Spotify for Artists pitch.
5. Release week: no experiments. Follow up, share, and log placements.

If you start testing in release week, you are optimizing a message for a window that has mostly closed. For a deeper look at the pre-release window, see our guide to pre-release pitching and how to pitch unreleased songs.

A worked example

Say you are an indie-folk singer-songwriter with one focus single. You want to know whether leading with a sound description beats leading with a story.

Held constant: same single, same private link, same three-sentence length, same sign-off, same follow-up after seven days.

Version A (story first): “I wrote this after moving back to my hometown. It is a quiet song about starting over, and I think it suits your late-night acoustic playlist.”

Version B (sound first): “Fingerpicked acoustic guitar, close-mic vocals, a slow, reflective mood. I think it suits your late-night acoustic playlist.”

Metric chosen in advance: curator reply rate within ten days.

Groups: 40 genre-matched curators, sorted by playlist size, alternated A and B, 20 each.

Here is a plausible, hypothetical outcome, to show the reading rather than any real result:

What can you conclude? Very little. A difference of two replies across 40 curators sits well within chance. You would be right to mark this “inconclusive, B slightly ahead” and run a second round, not rewrite everything around B.

How much evidence is enough

There is no magic number for a manual test, but there are sensible guardrails.

• Use Spotify’s logic as a floor. Its Ads Manager only calls a winner when a variant outperforms the others at least 65% of the time; otherwise the test can be inconclusive. That is a modest bar, and a few replies will not clear it.
• Predefine the rule. Write down your metric and your threshold before sending. “B wins if its reply rate is higher by at least ten points across 50 or more pitches per version” is a rule you cannot bend afterward.
• Log inconclusive rounds. They are information. They tell you that this variable does not matter much, so you can stop agonizing over it.
• Repeat. A result that shows up in two rounds with different curators is worth more than a single big gap.

If you cannot reach dozens of curators per version, shrink the ambition. Instead of a formal A/B test, run a sequence of pitches and read the curator comments, which are often more informative than the counts.

Read curator feedback as data

Reply rate is a blunt tool. A written decline can be worth more than a silent non-response, because it tells you why. When a curator says the track is too slow for the list, or the vocal sits too far back, that is information about fit, not wording. Tag each response:

• Fit: wrong genre, mood or tempo
• Quality: mix, production, vocal
• Message: unclear, too long, too generic
• Timing: list is full, not accepting

If most declines are “fit”, your problem is targeting, and no wording test will fix it. If declines cluster under “message”, your test variable is well chosen. This is why writing a pitch that works and picking the right curators are two separate skills.

Perspective: why we treat feedback as the real result

We built Intonality around one belief: a placement is a good outcome, but a clear answer is a good outcome too. Every curator decision on our platform comes with written feedback, and placements and declines are published. That changes how a pitch test works.

When you test wording with silent outreach, a non-reply is ambiguous. The curator may have disliked the song, never opened the message, or run out of room. When each decision has a written reason, you can tell a message problem from a fit problem, and that is the difference between a test that teaches you something and one that gives you a number.

Two more things we have learned from this approach. First, genre match matters more than clever copy. Our curators are manually screened and matched by genre, because a perfect line sent to the wrong playlist still fails. Second, one song at a time beats scattering. Our guide on manual playlist pitching explains why we send one track per campaign and why that keeps feedback clean enough to learn from.

And a plain warning: no service, ours included, can promise Spotify editorial placement. Spotify says that cannot be bought. What a transparent service can do is separate the paid submission from the outcome, and show you what every curator decided and why.

— Einars

Where a campaign fits into your testing

Running your own test means building a curator list, writing variants, tracking replies and chasing follow-ups. That is real work, and the sample sizes are hard to reach by hand.

If you would rather spend that time on music, our campaigns pitch your track to vetted, genre-matched curators, with real-time updates and written feedback on every decision. You choose among the Starter, Standard, Pro and Max tiers on our main page, and you can browse public campaigns with real curator feedback to see what responses look like before you commit. We claim an average of 4 to 11 playlist placements per campaign depending on the tier, and automatic refunds apply when a review is inadequate. Results vary by track and genre.

It can also feed your own experiments. Written declines give you a free read on whether your song or your message is the issue. If you want to start without paying, our piece on drip playlist submissions covers a free-then-feedback route.

Common mistakes that ruin a pitch test

• Changing two things. New subject line and new song link is not a test.
• Unequal groups. Sending A to your warmest contacts and B to cold ones.
• Picking the metric afterward. Declaring victory on whichever number looks best.
• Counting streams. The playlist’s audience dominates the stream count.
• Ignoring the release timeline. A perfect test that ends after the two-week mark helps your next release, not this one.
• Mixing channels. Counting an editorial add or an algorithmic boost as a win for your wording.
• Testing on tiny numbers. Ten pitches cannot distinguish a good message from a lucky one.

Two habits help. Keep a simple log: date, curator, tier, version, response, reason. And change your approach between rounds only after a round is logged.

After placement: judging listener quality

If your goal is the third metric, listener quality, wait. Saves and follows from a new playlist add take days to accumulate, so give each placement the same observation window, say 14 days, before comparing. Compare saves per thousand streams, not total streams, so large playlists do not drown out small ones. Our guide to post-release pitching steps covers what to do once the song is out and the testing continues with the next single.

This is also where a test helps your wider strategy. A pitch that wins a reply but brings listeners who skip within seconds has found the wrong playlists. A pitch that wins fewer adds but brings listeners who save the track has found the right ones. Neither shows up in a reply-rate comparison alone.

A repeatable checklist

1. Choose one track and one primary question.
2. Choose one metric and one threshold, and write them down.
3. Write two versions that differ in one element only.
4. Sort genre-matched curators by playlist size and country, then alternate A and B.
5. Send both within the same few days, with the same follow-up rule.
6. Log every response with a reason tag.
7. Compare at the end of a fixed window. If the gap does not clear your threshold, call it inconclusive.
8. Carry the better wording, or the lesson, into the next round with fresh curators.

One extra idea: if you release often, you can run a round per single and slowly build a picture across songs. Just do not compare results from different songs as if the wording alone differed.

FAQ

Can I use Spotify’s split testing tool to test my playlist pitch?

No. Spotify’s split testing sits inside Ads Manager and compares ad creative. It does not test curator messages or Spotify for Artists pitches. You can borrow its principles, though: change one variable, keep conditions matched, and pick your metric first.

How many pitches do I need before the result means something?

There is no fixed number, but ten per version is too few to separate skill from luck. Aim for dozens per version, set your threshold beforehand, and treat close results as inconclusive. Repeating the test with a new curator group adds more confidence than one large gap.

Does the official Spotify for Artists pitch need a split test?

No. You submit one unreleased song at a time, and Spotify recommends doing it at least two weeks before release. Use that form to describe your song accurately, with genre, mood, instruments and story, and save the testing for independent curator outreach.

Can paying for a campaign get me onto Spotify’s editorial playlists?

No. Spotify says nobody can pay to be added to its editorial playlists. A paid campaign can fund outreach to independent curators and give you their decisions and feedback, but any placement remains the curator’s choice.

Sources

• Split Testing in Spotify Ads Manager
• New Releases, Spotify for Artists
• Share New Music for Playlist Consideration, Spotify for Artists
• Spotify for Artists

Recommended

• Score Playlist Fit Like a Curator, Prioritize 7+ Matches
• Six 500 Character Playlist Pitch Examples With Editor Annotations
• Playlist Pitch Examples for Indie Musicians: What to Write and When
• Weekly Playlist Tracking to Turn Adds Into Streams for Indie Artists