How to Choose a Cold Email Agency: 10 Questions Every Buyer Should Ask

Choosing a cold email agency is a decision that can generate hundreds of thousands in pipeline or waste months of time and budget. The market has hundreds of agencies, most of which look identical on their websites. They all claim "done-for-you outbound," "qualified meetings," and "proven results." The difference between a great agency and a mediocre one only becomes clear when you ask the right questions.

I run Alchemail, a cold email agency. I am writing this guide not to sell you on us, but to give you a framework for evaluating any agency, including ours. After generating $55M+ in pipeline and working with dozens of B2B companies, I know what separates agencies that deliver from agencies that disappoint. Here are the 10 questions that reveal the truth.

Question 1: Who Owns the Sending Infrastructure?

This is the most important technical question. The answer determines your deliverability, your brand safety, and what happens when you leave.

What good looks like: The agency sets up dedicated sending domains and email accounts specifically for your campaigns. You own these domains. When the engagement ends, the infrastructure transfers to you.

Red flags:

• The agency uses shared infrastructure across multiple clients. If another client gets flagged for spam, your campaigns suffer.
• The agency owns the domains and will not transfer them when you leave.
• The agency cannot clearly explain their domain and account setup.

Follow-up questions:

• How many sending domains will you set up for my campaigns?
• How many email accounts per domain?
• Who registers and owns the domains?
• What happens to the infrastructure if we part ways?

At Alchemail, we set up dedicated infrastructure for every client. Typically 3-5 sending domains with 3-5 accounts each, all registered under the client's ownership. When the engagement ends, the infrastructure stays with the client.

Question 2: How Do You Handle Data and List Building?

Data quality directly determines campaign performance. The agency's data process tells you whether your campaigns will reach the right people.

What good looks like: The agency uses multiple data sources, enriches contacts with additional data points, verifies every email address, and segments lists by ICP tier and persona.

Process Step	Good Agency	Mediocre Agency
Data sources	3-5 sources (Apollo, Clay, Sales Nav)	Single source (Apollo only)
Enrichment	Tech stack, funding, hiring signals	Name and title only
Verification	Every email verified (<2% bounce target)	Partial verification or none
Segmentation	By ICP tier, persona, and trigger	One big list, one campaign
Refresh rate	Monthly list updates	Build once, never refresh

Red flags:

• The agency cannot name their data sources.
• They do not verify email addresses before sending.
• They build one list and send the same sequence to everyone.
• Bounce rates in past campaigns exceeded 3%.

At Alchemail, we use Apollo for prospecting, Clay for multi-source enrichment, and LeadMagic for verification. Every contact is verified and enriched with behavioral signals before entering a sequence.

Question 3: What Is Your Pricing Model and What Is Included?

Pricing transparency is a major differentiator. Many agencies hide costs or bundle things in ways that make comparison impossible.

What good looks like: A clear monthly retainer that covers all services: infrastructure setup, data, copywriting, sending, deliverability management, reply management, A/B testing, and weekly reporting. No surprise add-ons.

Common pricing models:

Model	Typical Range	Pros	Cons
Monthly retainer	$3,000-$7,000/month	Predictable cost, aligned incentives	Higher upfront commitment
Pay-per-meeting	$200-$500/meeting	Pay for results only	Quality incentive problem
Hybrid (base + bonus)	$1,500-$4,000 base + per meeting	Balanced incentives	Complexity in "qualified" definition

Red flags:

• Data costs, tool costs, or infrastructure costs are billed separately on top of the retainer.
• Long-term contracts (6-12 months) with no performance guarantees.
• Vague deliverables. "We will run campaigns" is not a deliverable. "10,000 emails per month to 2 ICP segments with 4-email sequences and weekly reporting" is.

For a detailed breakdown, see our cold email agency pricing guide.

At Alchemail, we charge a flat monthly retainer that includes everything. Month-to-month, no lock-in. We believe results should earn the relationship, not a contract.

Question 4: Can You Share Specific Case Studies with Metrics?

Case studies separate agencies with real experience from agencies with good marketing.

What good looks like: Named clients (or clearly described anonymized clients) with specific metrics: emails sent, reply rates, meetings booked, pipeline generated, revenue closed. Including the timeline and any challenges encountered.

Red flags:

• No case studies available.
• Vague results: "We helped Company X grow" with no numbers.
• All metrics are top-of-funnel (open rates, emails sent) with no pipeline or revenue data.
• Cannot share a single reference.

Questions to ask about each case study:

• What was the client's industry and ACV?
• How long was the engagement?
• What were the monthly send volumes?
• What reply rate did you achieve?
• How many meetings did you book per month?
• What pipeline value was generated?
• Can I speak with this client directly?

At Alchemail, we share detailed case studies. For example, our work booking 40 meetings in 90 days for a loyalty app and building $2M in pipeline for an analytics startup.

Question 5: How Do You Define and Track "Qualified Meeting"?

The definition of "qualified" is where most agency-client disputes originate. Without a clear, agreed-upon definition, you will argue about whether the agency is delivering.

What good looks like: The agency works with you to define qualification criteria before launching. Typical criteria include:

• Prospect matches ICP (company size, industry, geography)
• Contact holds a relevant title
• Prospect agreed to a specific meeting time
• Prospect showed up for the meeting

Red flags:

• The agency counts "positive replies" as meetings.
• No formal qualification criteria are defined.
• The agency pushes back on disqualifying meetings that clearly do not match your ICP.
• Meeting quality is not tracked or reported.

At Alchemail, we define qualification criteria with every client during onboarding. We track meeting quality through post-meeting feedback and adjust targeting based on which meetings convert to pipeline.

Question 6: What Does Your Reporting Look Like?

Reporting reveals how transparent and data-driven the agency is.

What good looks like: Weekly reports with:

• Emails sent, delivered, opened, replied
• Reply sentiment breakdown (positive, negative, neutral)
• Meetings booked (with company names and contact details)
• A/B test results and optimization actions taken
• Deliverability metrics (bounce rate, spam complaints)
• Pipeline update (if applicable)

Red flags:

• Monthly reports only (too infrequent to catch and fix problems).
• Reports that only show vanity metrics (emails sent, open rates) without reply and meeting data.
• No access to the sending platform dashboard.
• The agency cannot explain their metrics or how they calculate them.

At Alchemail, we provide weekly reports and give clients direct access to their campaign dashboard. Transparency is non-negotiable.

Question 7: How Do You Handle Deliverability Issues?

Every cold email program encounters deliverability challenges. The question is whether the agency can diagnose and fix them quickly.

What good looks like: The agency monitors deliverability daily, has a documented process for diagnosing issues, and can describe specific actions they take when deliverability drops.

Questions to ask:

• How do you monitor inbox placement?
• What is your process when open rates drop below 30%?
• Have you ever had a domain blacklisted? What did you do?
• How do you handle warmup and sender reputation?

Red flags:

• The agency does not proactively monitor deliverability.
• They cannot describe their troubleshooting process.
• They blame the client's product or market for deliverability issues instead of taking ownership.

Question 8: What Is Your Approach to Copywriting and A/B Testing?

The agency's copywriting process determines your messaging quality. Their testing approach determines whether that quality improves over time.

What good looks like:

• The agency writes custom copy for your specific ICP and personas
• They launch with multiple subject line and email body variants
• They test systematically (one variable at a time, sufficient sample sizes)
• They report test results weekly and implement winners
• They refresh messaging every 6-8 weeks to prevent fatigue

Red flags:

• The agency uses a template they apply to every client with minor tweaks.
• No A/B testing is planned or reported.
• They cannot show examples of how past testing improved results.
• They write one sequence and never change it.

At Alchemail, every campaign launches with 3-4 subject line variants and multiple messaging angles. We test continuously and refresh copy based on data, not guesswork.

Question 9: What Happens During Onboarding?

Onboarding reveals how structured and thorough the agency is. A chaotic onboarding is a preview of chaotic campaign management.

What good looks like: A structured onboarding process with clear steps:

1. Discovery call: Deep dive into your product, ICP, positioning, and competitive landscape
2. ICP workshop: Collaborative session to define or refine your ICP
3. Infrastructure setup: Domains purchased, accounts created, warmup started
4. Messaging development: Custom sequences written and reviewed by your team
5. List building: First lists built, enriched, and verified
6. Campaign review: Final review before launch
7. Launch: Campaigns go live with monitoring

Timeline: Good agencies complete onboarding in 2-3 weeks, with campaigns live by week 3-4.

Red flags:

• No structured onboarding process.
• The agency asks minimal questions about your business before launching.
• They rush to send emails without adequate warmup.
• No opportunity for you to review messaging before it goes out.

Question 10: What Is Your Cancellation Policy?

This reveals the agency's confidence in their own results.

What good looks like: Month-to-month pricing with 30-day notice. No long-term contracts unless you choose one for a discount. The agency is confident that results will keep you, not a contract.

Red flags:

• 6-12 month minimum contracts with no performance clauses.
• Heavy penalties for early termination.
• The agency pressures you to sign a long-term deal before showing any results.
• No trial period or proof-of-concept option.

At Alchemail, we operate month-to-month with no lock-in. We have found that companies stay because of results, not because of contracts.

Bonus: Red Flags to Walk Away From

If you encounter any of these during the evaluation process, walk away:

1. They guarantee specific results ("We guarantee 50 meetings per month"). No agency can guarantee results because too many variables are outside their control.
2. They cannot explain their process. Vague answers to specific questions indicate inexperience.
3. They pressure you to sign immediately. "This pricing is only available today" is a sales tactic, not a sign of quality.
4. No case studies, no references, no proof. If they cannot show past results, they probably do not have them.
5. They use your primary domain for sending. This is a fundamental mistake that puts your entire email reputation at risk.
6. They outsource everything. If the agency is just a middleman outsourcing to freelancers, you are paying a markup for fragmented execution.

The Evaluation Scorecard

Use this scorecard to compare agencies objectively:

Criteria	Weight	Score (1-5)
Infrastructure ownership and quality	15%
Data process and quality	15%
Pricing transparency	10%
Case studies with specific metrics	15%
Meeting qualification framework	10%
Reporting depth and frequency	10%
Deliverability management	10%
Copywriting and testing approach	10%
Cancellation flexibility	5%

Score 40+ out of 50: Strong candidate. Move to reference checks. Score 30-39: Decent but has gaps. Probe weak areas further. Score below 30: Look elsewhere.

For more guidance on the hiring process, see our guide on how to hire a cold email agency.

Frequently Asked Questions

How many agencies should I evaluate?

Evaluate 3-5 agencies. Fewer than 3 does not give you enough comparison. More than 5 creates analysis paralysis. Use the 10 questions in this guide to quickly identify the top 2-3 candidates, then do reference calls.

Should I choose the cheapest agency?

No. The cheapest agency typically cuts corners on data quality, infrastructure, or copywriting. These cuts show up as poor deliverability, low reply rates, and wasted time. Choose the agency with the best process and results at a price that fits your budget.

How long should I give a new agency before evaluating results?

Give them 60-90 days. The first 30 days include onboarding, infrastructure warmup, and initial testing. Results start appearing in weeks 3-4 and reach steady state by week 8-10. Agencies that cannot produce meetings by day 60 have a problem.

Can I switch agencies mid-engagement?

Yes, especially if you own your infrastructure (domains and accounts). The transition involves sharing your campaign data, lists, and playbook with the new agency. Allow 2-3 weeks for the new agency to onboard.

What should I expect in the first 30 days?

In the first 30 days with a good agency: infrastructure setup (week 1), warmup and list building (weeks 1-2), first campaigns live (week 2-3), first replies and meetings (weeks 3-4). By day 30, you should have at least 5-10 meetings booked or in process.

---

Ready to evaluate Alchemail against these 10 questions? Book a call and we will answer every question transparently. No pressure, no long-term contract.