Fireside AI

Hi, today I'm joined by Sean Wen from Poly AI.

Sean, welcome to the show.

Hi, hi, this is Sean.

Great to have you with us today, Sean.

So some of the listeners presumably have not encountered you before.

So let's start kick off and just ask, know, who are you and Poly AI?

Tell us a bit about your background.

Cool, yeah.

So my name is Sean Wen, co-founder and CTO of PolyAI.

So PolyAI is working on automating phone calls for contact centers.

We've been doing this like seven, eight years now.

It has been a while and it has a lot of fun as well.

And my role is really heading up the product and engineering team at Poly.

And so Poly A is going since 2017, you launched right, which is a few years now and you've
grown and last year raised zero C funding, which is great.

Congratulations.

And so I'd love to take us back to the beginning, you know, the very early days of Poly A.

What was it like at the beginning?

Tell us a bit about that experience.

Yeah, there's like certain history of our time, of like, well, not really overlapping, but
like, you know, we've been to the same group in Cambridge previously.

Yeah, and that's where the both of us met, right?

The same research group at Cambridge.

so in research group Cambridge, that dialogue system group working under Professor Steve
Young and you know I was one of his last students you know because he was about to retire

at that time.

He has achieved a lot in his career as you know and Senior and Senior you know Pro
Vice-Chancellor of Cambridge and then you know down

really good, amazing research in speech and the dialogue system field.

And yeah, I was like one of his last three students and the other two are my co-founders
now.

So that's kind of like how we started really.

Yeah, so I met my co-founder Eddie back in Taiwan.

So we are both Taiwanese and then we came to Cambridge to study our PhD with Steve and
that's how we met Nicola, our CEO and co-founder right now.

Yeah, so like we met together, we did research together, know, have fun in uni, know,
having a lot of drinks like you just do the thing you can.

And then we decided to start a bit of business after that.

we just continue to do what we are doing already, is, know, dialogue systems now have a
much more fancy name, Conversational AI.

since then, know, it has been like, you know, in the past seven, eight years, building for
the product and team.

And how did you come across that initial, like the customer service idea?

Where did that come from?

Yeah, it's a great question.

are three PhD students, so we all came from technical background.

And in the early days, you remember Steve did a lot of pump DP based assistance, like
about understanding noisy speech in the car and be able to take actions on it.

That was like a major thing in that early stage of the entire conversation system, because
people are basically trying to find.

What's the use case?

What's the most promising scenario to apply these kind of systems?

And we started doing restaurant reservation for some reason, because we have this
Cambridge restaurant data set.

Everyone just would work in on it.

Because back at that time, it was...

so difficult to collect data.

So then, know, like we just continue to reuse that data.

And in fact, we actually started in the restaurant domain, not really in the customer
service domain.

Well, kind of customer service as well, because when we try to sell our first product, our
system was built about understanding restaurant queries and what people wanted to do in

restaurant, like cancel, make reservation, whatever.

And then we started to talk into these outsourced contact center in Heifer actually.

And then they have been basically taking book table reservations for some of the large UK
restaurant chains like, know, Beef Eaters, We Bread, some Green King Pops as well.

So then we started talking to them, working with them and over time it was like, like we
have a lot of like, you know,

labor shortage issues, we cannot staff people for systemality reasons.

then the churn rate is very high because usually like, know, graduate, you know, coming
for six months time and then left for the next job.

So there was a lot of problem there.

So they were like, okay, like, you know, how about let's try to automate some calls and
can you convince your clients that let us to take you use AI to take over some calls.

It was really, really long sales cycle because at that time people don't trust in voice
assistant because obviously Siri, Google Assistant didn't actually do it very well.

So then we managed to convince the client, started with three pilot restaurants.

And then I remember the lunch date, everyone is in front of the dashboard listening to
calls.

There was just like maybe one or calls every 10 minutes.

But then when people hear that, people are super excited and they start celebrating in the
office.

That was a very fun period of time.

So we started with that and then rolled out to more and more restaurants.

And then we started to look more into contact centers as well.

Yeah, I definitely remember working on that restaurant domain problem during my postdoc
time.

So it's fascinating how you managed to take that and turn it into a business that actually
people had had demand for.

And nice to think about the early days and that first system being built.

So obviously that was back 2017, 18, many years ago.

And now you've grown much, much bigger.

So what's...

What's different now would you say about your company compared to those early days?

Yeah, think people like to say that companies go through different stages as they
gradually grow up from startup to scale up to eventually a corporate.

I think that I do actually see that different phase of a company growing is super
different.

I remember in the early days when the three of us just started a business, when we started
hiring, I think the initial maybe 5 to 8 people are all from our inner circles.

We basically hired a lot of our friends and people we met in conferences, really.

so I think initial scaling is that that that 10 people is that really from like you know
our close you know friends and people we know and that kind of dynamics is actually very

different because that you know you kind of know each other you kind of know how

each other works and you trust each other, that kind of environment.

So then there was like the very, I would say, initial year or so.

And then we started to scale beyond our own group because we have a lot of researchers in
the early days.

We started to realize that building robust systems require engineers as well.

We started to hire engineers, know, front end, back end.

And that's when we started to grow outside of our comfort zone that we started to getting
people from.

different places.

remember like as a small company back at that time, was so difficult to hire people.

I reach out on LinkedIn, I reach out everywhere, nobody replies.

And then the only thing that helps back at that time was AngelList.

I think it's no longer that popular these days, but it's very interesting comparing like
hiring then and hiring now.

It's like huge difference.

And then, so we go that to like 20, 30 people.

And then at that point, you still have a vibe of kind of like a family where during
lunchtime, like people would just bring their lunch onto the table.

We started to discuss a lot of different things, talking about how the heart attacks and
know, et cetera, And then you get to interact with everyone.

Everyone is still very close.

And then...

as you gradually scale to like 50, 60, you still get to know everyone, but you don't hang
out with everyone anymore.

And then you started to know that, like you have to keep a little bit of boundary
sometimes.

In the early days, are like good friends with everyone, right?

And then now we are 220 people now, and we have a hugely remote team and a very big

of our employee base is in US So now interacting with people over like Zooms and Google
Hangouts is very common.

But we still have a lot of people go to the office.

But then now if I go to the office, probably more than half of the people I don't actually
interact or work with every day.

So it's a very, very kind of...

very interesting change that you see in different stages of a company.

Was that a deliberate decision to hire remotely rather than require people to come into
the office or was it?

I think in the early days, we are very much of an office culture.

think that, know, Nikola is actually like, you know, really like pushing for the office
culture.

And I think that it manages to actually build a very strong culture back in that time.

But, you know, since COVID hits, it just becomes impossible.

And then we started to realize, engineers working from home is actually quite productive.

Because it's like no meetings, people around, they can focus a lot more.

So that gradually relaxes our constraint.

Also because we spend it to the US and we hire, initial hires of US talents, we kind of
hiring them from everywhere.

Because initially it's the growth of marketing, so it's like sales and marketing.

They usually don't come to our office even before COVID.

they are predominantly remote because they have to travel a lot and things like that.

So then we kind of started hiring the team there and then they just become completely
remote.

And then, I mean, afterwards we, now we have an office in New York, we have an office in
San Francisco.

There's still a cluster of people around there, but then we don't force people to come to
the office.

If people wanted to come to the office, then they can go there by themselves.

So you mentioned about sales and marketing and your three technical co-founders.

How did you manage to hire those first roles that weren't technical?

Very difficult, very difficult.

think we initially, like maybe a lot of the technical founders would say is that you try
to hire professionals to do the job for you.

The reality is that it never worked.

Like we have the initial thought about that.

So we hire the first product person.

We hire the first salesperson.

and then we hope that they can just do that part of the job for us but it didn't work out
because the product, the technology is too complicated and a lot of times the way that we

sell our product requires a lot of education to the customers about technology, how it
works but also about what does it mean for them

how can this help their, you know, business in the first place?

I think that in order to do that translation, it requires too much context and too much
understanding.

And it's very difficult for just hire someone and then be able to place them in and then
they can do the job for you.

Over time, we realized that we always need to do that job first.

Like, you know, for example, Nikola jumped into doing sales himself and he figured out,
okay, does the sales oracle work?

Because even if he wanted to step away being the manager of the team, he still need to
know like how individual sales process work and then how to build a process that actually

really tailored to each individual person in the organization.

So insurers that

what we realized that we just need to jump in to do all the jobs ourselves first and we
can scramble a little bit.

It's okay.

We don't need to be perfect, but the next, once you learn a job as over time, you need to
find someone who is actually a lot stronger than doing that job for you and then put that

in place.

I think founder really needs to get their hands dirty at first.

So doing it yourself helps you really understand what you're looking for in that person
that you're hiring to replace you to be the stronger.

Because in order to hire the right person, you need to do the job and feel the pain
yourself.

And then you can realize, okay, what kind of personality, what kind of skillset I need to
hire.

Because a lot of times when a founder hires their first sales or product, we will make a
wish list that is as long as, you know, were a Christmas shopping list.

But then, know, over time you will have to realize, okay, like this kind of person doesn't
exist.

and you just need to strike out all the unnecessary criterias and then know that just like
have like okay top three criterias I'm looking for for a person

So obviously a lot has changed in the past eight years.

said growing to 220 people, but can you look back and say, is there anything that hasn't
changed that you say, yes, we still have that in our company that was there at the

beginning.

yeah, I think it's a very good question.

I think it's the culture, which is, Poly-Iris culture is very strong and it is very strong
because of the initial phase of that very strong.

in-office culture, very strong friend connections, very strong in a way that we founders
or our managers don't actually just behave like managers We're actually just working with

the people together and we crack jokes together and I would encourage the people if they
are not happy with me, they can just directly come to me and provide me the feedback as

well.

because I like that kind of two-way communication in the early days, so just encourage
that behavior.

And so therefore, we get along really well, especially with a couple of employees we hired
in the early days.

And then they also done well and then being promoted into the leadership role later on as
well.

So because of that initial group of people,

we build with a very strong culture and even during, even past COVID, know, because
everyone becomes remote, we still keep that culture around.

And then we, as they got promoted to different roles and different departments at
different positions, they kind of carry that culture with them.

And then they hire the people with a similar kind of like culture affinity.

And I think culture is a, is an interesting thing is that it works like a snowball.

Once it...

becomes a snowball, it's very, it's unstoppable basically.

But that means that it's a double-edged sword.

If you didn't actually do it well in the early days, then it's going to become a big
product problem for you.

We were lucky in that regard that we did it quite well.

So that now I'm very confident sitting here and say that culture is never the problem of
ours.

We did it.

did you deliberately set out to build that sort of culture or was it just a product of the
way that you were all working together and-

not like we never had a strategic thinking about what the culture should be, but we have
certain kind of opinions along the way, more tactical about what kind of culture that it

shouldn't be.

like, for example, I think we did actually write down our cultural values from the very
early days.

then there are a of...

cultural values I emphasize a lot.

And I think one thing is about radical candor.

And that is, I've been basically telling people to read a book.

Radical candor is very important because I think one thing, one phenomenon I see in the
team is that because our team is hugely international and because it's an international

workplace, we have a lot of people coming from different culture.

And then some cultures are much more like implicit in terms of sharing their feelings and
things like that.

Some cultures are much more like, know, to your face, that kind of conversation.

And we saw a lot of frictions happening in early days just because people don't understand
each other's culture.

I think radical candor is really good for that kind of environment because you just...

try to ground people on the same communication protocol is that hey, just be direct but
like don't assume attention or emotions attached to it because that's not productive

always try to interpret people from a positive note but then please be as direct as
possible and I found that it's quite difficult for some people but I think over time once

we build it up it's just become a lot

easier to manage later on because people would just directly tell you how they feel rather
than have you get kissing.

It sounds like having that diversity of people and cultures in at the beginning helps
shape the culture in a really good way to lead you to where you are today.

Because initially you just need to work with those kind of like international background
and therefore like when you thinking about introducing values and culture, it is something

that you immediately think

So let's talk about your role specifically as the company grows as CTO.

What have you seen changed the most in your role and what have you had to learn as time
goes on?

think it's very interesting because I think in the early days especially during the growth
period you do actually wonder what is the role of a CTO because CTO I mean if a CTO is a

big company it's very clearly what it is.

A CTO in a small startup sometimes quite difficult to interpret because you know I
remember in the early days when there was like a couple of us like

as a CTO, I did a lot of coding in the early days, right?

But then because of that chief technology title, let you feel that you need to be the best
at technology, you know, comparing to the others.

So I should be spending a lot of time learning what is Kubernetes security, you know, like
all these kinds of stuff.

But then just not my portray, you know?

Like I'm from machine learning and AI background.

I know what I know, but there's a lot of things I don't know.

So I think the first phase of that growth for me was to put aside that chief title a
little bit, just like, hey, like it doesn't mean anything to me.

You know, I am, in the early days, I would consider myself more of a

co-founder and CTO, right?

And a co-founder title means that no matter what your actual title is, you just need to do
whatever it takes to make a company successful, right?

So despite I was the CTO title, but I did a lot of ML research and et cetera, I jumped in
to understand a lot of Kubernetes and how these services works.

I go into meet with the clients and then try to pick.

pitch them their products as well.

I do investment calls with Nikolas as well together.

As the company gradually shifted to become more mature where we place all the department
heads and the important people in the organization to run those functions, then CTO's role

becomes more prominent to me.

You know, currently under me, I still have product and engineering under me.

would say, despite my title now is CTO, but I'm more towards product currently.

The reason is because I think the engineering team is very strong.

So then I just that couple of tech leads and engineering lead to grow.

And then they just managing the team there.

But then now I'm doing a lot of like product product thinking because, you know, because
product needs me at this point.

So still, I think it's still like

know, switching between different goals as well.

I think you still have that co-founder role in your mind of doing whatever is needed.

It sounds like your engineering doesn't need you as much and your product does right now.

just like, know, once you find someone in place to do that particular job for you, then I
feel confident that this person can do a particular job for me, then I can move on to the

other problems.

So looking externally outside of poly AI, obviously it's been a huge time of change in the
AI industry from 2017 to today.

We've seen chat GPT, we've seen open AI, know, take anthropic, all of these things have
come along and changed the way we think about AI.

So how have you taken advantage or been able to leverage what's happening outside and keep
up with the rush of what's coming your way?

Yeah, I think it's definitely a very exciting and challenging time for the kind of
companies that we are.

Because in the early days, all our technologies are in-house.

We build models in-house, we train models in-house,

and then we realized that it's impossible to just keep up with the market movement that
way because you know if you have to do evaluation retraining models while at the same time

building up the product because back at that time so voice assistant is a very complicated
technical stack we need to do

a lot of telephony integrations and reliability management of those services.

Then there's also speech recognition, there's a TTS module, there's also like the entire
language understanding and spoken language understanding part, and policy management as

well.

So it's a lot.

So just cannot keep up with that, especially when LN started, you know, as soon as it was
announced, it was becoming very clear that

the market is just going to rush to it because there's just a huge opportunity and people
see that this is a complete paradigm shift.

That is for us as well.

So at that point we're like, okay, what?

We will have to, unfortunately, throw a lot of things away from the things that we built
in the early days.

And sometimes it takes quite a lot of courage to even say it out loud.

I have to tell the team that,

I think this is a paradigm shift and we are lucky and unlucky in that regard.

We just need to throw away whatever we have built previously and we have to change the way
that we do things moving forward.

So, and then because of L.M.

is just so competitive as a market and there are so many different models keep shipping We
decided to take an approach where, hey, like

I think technology now is moving really, really fast.

All we need to do now is to keep up with the market and that we need to use the latest
model, whatever, to build our products on top.

Because I think key, at that point, because in the early days, we still have thinking
about, we are going to a full-stack AI company, whatever.

But then after that, we realized, okay, the market is going to start to become more...

professionalized at each supply chain there.

So we just need to find who we are.

Like what does Poli-AI do?

We build voice assistant that, you know, an user have a conversation with.

So therefore we are at the top of the value chain, which is the application they
accompany.

We need to embrace that identity completely.

And then we have to say, no matter who have the better foundation model, we use them.

Right.

So I think that kind of pivotal thinking to that is very important because I think a lot
of companies fail to do it that way because they are too cherishing about their technology

and afraid of throwing it away.

Yeah, I imagine it's really hard to throw away stuff you've built because something else
has come and replaced it.

But sounds like you're really smart to spot that and to figure out exactly where you
needed to be to grow.

Yeah, I think, know, to be honest, like, you to us, like the decision was difficult, but
at the same time, it's also easy as well because the technology is just so much better

now.

So to wrap up this conversation, I'm curious to know what's the biggest lesson that you've
learned, the thing that you would share most with people.

was watching Jensen's I think one of his interviews previously.

I feel the same for him as well.

He said, so there was this analyst asking, now knowing what you know right now, like
Nvidia was having a tremendous success in the past couple of years,

if you go back in time, would you start a company again?

His answer was no.

Because he was like, this is so hard.

I didn't know that this is that hard.

So like, know, if I go back, I might not start a company again.

I think that's right because like, you know, I think a lot of people when they talk about
startups, there's a lot of fantasies associated with it because the news media is just

talking about all the successful stories on the market.

Larry Page started Google, becomes wildly successful, Elon Musk, Steve Jobs, whatever.

But I think people overlooked, they have gone a very long way to be that successful.

And then there were a lot of people probably fighting as hard and probably as smart as
them, but didn't get to that finish in line.

I think this is the hardest lesson I learned, like doing startup is really, really hard.

And I think if you were going to do it, you need to like really motivate it by the
problem.

Like I'm doing the problem I really enjoy.

I like AI, I like comfort, cessation of systems.

So I really enjoyed it.

There are several times in the past I was like, this is so difficult.

Why am I doing this?

But I still keep myself going because I fundamentally enjoy the product, the technology
and the people I work with.

So I think, yeah, that motivation is really important.

I think you can be really proud of what you've built and Poly AI is doing brilliantly.

and I keep getting more and more problems everyday.

Fantastic.

So this has been great.

It's been great chatting with you.

Where can we find out more about you and Poly AI?

Yeah, so you know, I'm actually having a very little social network presence in general.

I think LinkedIn is like the channel, the Poli account is that where we post a lot of
latest updates and things like that.

and I will put some links in the show notes so people can find you and Polly AI in.

Thank you very much Sean for joining me today.

Thank you.

Appreciate that.