Today: I try and use AI models to do competitive research.

I’ve heard this kind of research task is a great use for LLMs, but my previous experience here’s been hit-and-miss: they’ve generally been useful starting points but often reflective of what the widest consensus was in the past over what the current reality is.

The prompt: “Unit.co has gone through several significant shifts in strategy and focus in its history. Acting as a fintech industry expert, can you summarize the history of Unit.co, what the major pivots have been, and what its current focus and target customers are.”

One chance per model available on Perplexity (except Grok), including using their own “Deep Research” and letting it run. No refinement or per-model instructions or anything.

I picked this as a test because I’m familiar with Unit. In my last two jobs, they were not a direct competitor but adjacent, so they show up in my news and people talk about them, and they’re interesting, so I pay attention. (I also, full disclosure, invested in a Better Tomorrow Ventures fund, and a different BTV fund has backed Unit. I don’t believe I own or have any financial stake in Unit, I’ll update this if I realize otherwise)

It’s also a good challenge as their business model has changed dramatically and they’ve had some challenges with their partners, both part of wider industry patterns, and I’m curious how models will handle that. And while Unit is a big company, they’re not that well known out of the finance/technology space, so coverage is scattered – Wikipedia, for instance, has nothing on them.

In particular, there’s a great Fintech Business Weekly by Jason Mikula titled “After Spending Years Criticizing The Approach, Unit Now Says It Was Really “Direct” All Along” summarizing a pivot and talks about how Unit took down some of their own blog posts (which — don’t do this, companies. If you pivot out of something and you want to note that at the top of old posts, cool, but deleting old stuff is weak).

How will the tools deal with a pivot away from something where the company has removed backing information?

Decently, but not satisfyingly.

ChatGPT: got Unit’s pivot from “indirect” to “direct” relationships with banks, with an unsatisfying “why” as well as talking about their shifts in messaging and to serving banks. The current focus section was pretty good. Mikula’s piece is cited. Poor description of current target customers.

Claude: also got the pivot, also cited Mikula, had more about the “why” but I found the current summary of offerings to be poor, and there was a whole section on “Growth and Current Scale” that used data points from 2022 and 2023 to argue they’re growing like crazy — including employee count, when in 2024 they laid people off.

Gemni: sparser on history, good enough pivot description, cites Mikula, current focus and target customers was good.

Perplexity’s “Deep Research” had a much better writeup by stage, in some cases summarizing positioning in a way that seemed to set up a direct contrast later.

In what seemed like a limitation of the LLMs, all of them seemed to not understand what the present was (and I know “understand” is not the right term). Claude was the worst that I caught, but almost every time I’d read “current metrics show…” they were old, sometimes years old.

For me, it was interesting to see the path Unit has taken described pretty reasonably by all of the models, and none of them used a previous, more-widely-spread understanding of Unit’s business as the current one, though the last pivot was a bit ago, but the LLMs would cite pre-pivot metrics as evidence on the current state of the company.

Let’s compare the answers on what they’re doing now.

Current product focus:

• ChatGPT: “embedded finance platform”
• Claude: “…helping tech companies quickly launch embedded financal products”
• Deep research doesn’t address this succinctly (word salad of “positions itself as facilitiating direct relationships…”), now that I’m re-reading it, but does talk through product evolution and current offerings
• Unit.co’s website: Embedded finance! Used as many times as possible on the page.

Customers:

• ChatGPT: Vertical SaaS platforms, Marketplaces, small businesses, banks
• Claude: Vertical SaaS platforms, small business platforms, marketplaces
• Deep Research: Vertical SaaS platforms, Gig economy platforms, consumer-facing applications
• Unit.co’s website: does use “Vertical SaaS leaders” even though that is not helpful

Unit has also had some issues with partner banks, including losing relationships. This is a huge deal — they’re entirely dependent on bank partners to be able to do business, and it’s been an incredibly turbulent time for fintechs, and especially fintech partners like Unit. I’d absolutely want to know about this, and dig on further. Gemini and Perplexity’s Deep Research mention this, mentioning in particular Piermont Bank and the wind-down as part of that pivot (Unit’s blog post on “Deepening our commitment to banks” talks at length the importance of those relationships and also mentions ‘winding down’ of three bank relationships). If you were relying on ChatGPT or Claude, you’d be missing a huge thing you’d want to know about.

I think I could recommend using LLMs to do this kind of research as a starting point with the caveat that rather than relying on the summary of the current business I would also want to myself look at the current website and do my own write-up of where their focus seems to be today, along with what they’re offering to visitors right now.

As a fun bonus, this test provided funniest moment so far: I was vaguely dissatisfied with what I was getting, so I took one generated response and threw it into a different model and said “I want to copy in a report on Unit.co that’s been criticized as insufficient and lacking depth. Can you double-check the contents here and flag anything that might be incorrect, misleading, or requiring further explanation, so that I can re-write it? If you spot anything and have a particular correction or suggestion, that would be helpful.”

The response was “This is a solid narrative with clear structure, but you’re right — it reads a bit like a press release in parts and lacks some critical depth or friction that might be expected in a robust analysis.”

I laughed. And I know the models tend to be agreeable to your prompt, sure, and I don’t know what to make of it, but it made me want to go back through to each one, give that response to them and see how they took the feedback, but also to say “I know! Can I just give this criticism in advance?” and I realized that yes, I’d come around to telling myself to do “prompt engineering” which I’ve been avoiding for these trials.