How to Choose an AI Software Development PartnerA checklist you can use before signing anything

Picking an AI software development partner is not like picking a generic dev shop. The failure modes are different. A web-app build that's 30% wrong is annoying. An AI build that's 30% wrong is a confidently incorrect system making decisions your business will trust without thinking.

The difference between a partner who ships production AI and a partner who ships an impressive demo is largely invisible in a sales cycle. This post is the checklist we'd want a prospect to use on us — honest, specific, and quick enough to run in an afternoon.

When you're vetting a partner, look past the marketing and ask to see:

Real production systems. Not pre-recorded demos, not dashboards with fake numbers — actual apps running with actual users. If they have NDAs, that's fair, but they should be able to talk specifically about architecture, traffic, cost per request, and incidents they've handled.

Their take on evaluation. If they can't describe how they measure whether an AI feature is working, you will find out the hard way that it isn't. The good answer involves a golden set, automated runs on every change, and dashboards the team actually watches. The bad answer is "we tested it manually."

Observability and cost controls. Ask how they detect a regression in production, how they track per-user and per-feature cost, and what happens when a model is slow or returning garbage. "We'd add logging" is not a plan; "we use this stack, here's a sanitised screenshot" is.

Comfort with model-provider neutrality. A serious partner will have shipped across OpenAI, Anthropic, Azure, Bedrock, and often a self-hosted open-source model. They'll have opinions about which fits which use case. A partner who only knows one provider is a risk.

Retrieval and RAG maturity. Ask how they chunk documents, handle OCR, combine vector and keyword search, and evaluate retrieval relevance. If they say "we just use OpenAI's file upload," they haven't shipped anything non-trivial.

Agentic systems in production. Not "we've built an agent in a notebook" — "we run agents that take actions for real users, with these guardrails, at this volume." Agentic workflows are where the majority of business value is, and where the most people have been burned.

Technical skill is necessary but not sufficient. You also need a team that works the way your organisation works.

They push back. A partner who agrees with every spec you send is a partner who will happily build the wrong thing. You want someone who reads your brief and comes back with "we'd scope this smaller, here's why" or "this constraint doesn't make sense, can we revisit?".

Clear pricing and scope. Fixed-price for well-defined phases, time-and-materials for exploratory work, and an explicit statement of what "done" means. Anything vaguer is a budget overrun waiting to happen.

Honest about what they don't do. Every team has gaps — too small to run a 24/7 on-call, no mobile team, no designer on staff. Partners who pretend to do everything are less reliable than partners who name their boundaries.

Client references you can actually call. Not logos on a website — people. A 20-minute call with a past client will teach you more than ten proposals.

Communication cadence. Ask how they run projects: weekly demos, written updates, shared task boards, Slack access. If the pitch is "we'll send you an invoice every month," that's not a partnership, it's a black box.

A short list of things that have, in our experience, correlated with projects going wrong:

• Vendor lock-in by default. Code they host, models they host, no access to your own vectors or logs. This is a power play, not a service.
• "Just trust the model." Teams that don't want to talk about failure cases are teams that haven't hit them yet.
• AI buzzword bingo. If every sentence has "autonomous," "transformative," and "revolutionary" without a single concrete number, you're looking at a demo shop.
• No senior engineer in the conversation. If the sales team can't bring a hands-on engineer to a technical call, that's who you'll actually be working with.
• Unwillingness to do a paid pilot. A good partner will scope a small, fixed-price piece before asking for a multi-quarter commitment.

Copy these into the next vendor call:

• "Show me the repo structure of a production AI app you've shipped."
• "What does your evaluation pipeline look like? Can I see a sanitised eval report?"
• "What's the biggest incident you've had in an AI system, and what did you change?"
• "Walk me through how you'd handle our data and access control, step by step."
• "What would you cut from our scope if the budget were halved?"
• "Who will be doing the actual work? Can I meet them?"
• "What's your handoff plan when the engagement ends?"

The answers tell you more than any deck.

Assume you find three candidates that look strong. The best way to compare them is a small, paid pilot — not a bake-off and not a long proposal cycle.

A good pilot is:

• One workflow. A real one, with real stakeholders, not a toy.
• Four to eight weeks. Long enough to produce something usable, short enough that failure is affordable.
• Success criteria on paper. "The agent handles 60% of billing emails end-to-end with zero incorrect auto-sends." Not "an AI thing that's cool."
• Clean handoff. At the end, you own the code, the docs, the eval set, and the deployment. Even if you continue with the partner, you could walk away.

We run engagements exactly like this, and we encourage clients to run the pilot with at least one other shop in parallel if they're uncertain. The comparison is cheaper than a bad multi-year commitment.

Sometimes a partner is the wrong answer. Build the team yourself if:

• AI is your product, not a feature — your company's moat depends on getting better at it than competitors.
• You can credibly hire senior AI engineers (hard in 2025-26, but possible in some markets).
• Your timeline can absorb 6–12 months of team assembly before shipping.
• You have the discipline to run an in-house ML ops practice long term.

If three of those aren't true, a partner gets you to production faster, and you can in-house later once the pattern is proven.

Choosing well is 80% of the project's outcome. Spend a week doing real diligence — it's a tiny investment next to the cost of a failed engagement.

If you'd like us to be one of the partners you evaluate, we'd welcome the chance. Start with our AI software development service to get a feel for how we scope work, or get in touch with the problem you want to solve.