Are Trust & Safety as a Key Part of Your Product Strategy

Desi Velikova 00:53

Hi Neil, welcome to The Product Show.

Neil Shah 00:57

Hello, nice to see you.

Desi Velikova 00:59

I was really keen to have this chat with you. Because trust and safety is such an important aspect of building a product, all the founders and PMs I work with take it very seriously. We all want the products we’re building to instil trust, we want users to feel safe while they’re using them. But we don’t do much until it becomes a problem. And also there are not so many senior leaders in the space comparing to growth, for example. So I’m really excited about our conversation and what you’re going to share with us. Right, so let’s start with your background. What’s been your professional journey so far?

Neil Shah 01:37

Thank you so much for the warm welcome and introduction. So I’m currently at Lyft, working on our product identity, identity services, the intersection between offline and online trust and safety. And we want to keep these two strangers in a vehicle together safe. So how do you make sure that we have the right driver, and we have the right rider in the right car at the same time. And a lot of work goes into making sure that that happens. So I’ve been I’d lived about two years before that I was at Twitter, where I worked on a number of things. But I ended up my career there working on misinformation, information integrity on the product health team, which is as close to trust and safety as you’re going to get. And for me, that was the turning point. That was where I realized there’s a lot of work that needs to happen in trust and safety. That is hard to do. And because previously I’d worked in ADS revenue, and in user growth on the consumer side, I had a lot of experience that would become very relevant, I would learn to this space. And so before that I had worked at some startups. But yeah, that’s my background

Neil Shah 02:56

twist is impacted by all aspects of the business. Isn’t it? branding, tone of voice customer support? The digital experience in Lyft’s case also by the offline experience, because ultimately, that’s what customers are hiring the product for? Where do you start from? How do you go about building trust? do you prioritize the most pressing concerns? First? How do you identify the areas of focus?

Neil Shah 03:25

It can be very nebulous to work on trust and safety. And I often I often want to go back to the customer, the customer problem. And it’s you have to think deeply about what a customer might care for, and not what the company is for. Every tech company wants to grow users wants to grow revenue, that is not a job to be done. From the customer standpoint. Why does somebody hire you to use your product? The classic example from Christian Christensen’s milkshakes is you’re buying a milkshake, not because of the taste and the flavors and how cold it is, but because you’re bored in the car on the way to work. And so you want to slip on something. And so for the same reason, people hire Twitter, for information access they can trust. And if you can’t trust what you see on Twitter, you won’t use the product. And that is a very long term game that you have to play. That’s very hard for people to see. So you go back to the basics. And when you talk to customers in when when I was back at Twitter with people want to know what what is going on with the elections. And if you can’t trust the information you’re reading and you’re seeing and you’re talking to other people, and who is this person talking? Are they a reputable Congress person? Or are they just some fake actor, then you won’t use the product and Twitter wouldn’t be around in a decade. And I think it was that realization that helped, you know, when I was back on that team prioritize trust and safety work with respect to information integrity lifts, you have a similar issue if you don’t feel safe as a driver, right, or as a writer, which are really two interesting problems, that’s to deal with, right? Because you have, you know, in the news recently, you’ve seen riders who are coughing, as if they had, you know, Coronavirus on to a driver. That’s terrible. That’s a really terrible situation for a driver, and why would you drive for Lyft. And if you don’t drive for Lyft, then our driver supply goes down, which increases prices, which is which sucks for everybody else, right. And so, first and foremost, we have to make Lyft safe for our drivers. And we need to make, let’s say, for our riders, and so then you know, what problem you want to solve. And that might be a multi year journey, you have to take it into chunks now and think about what can I solve today? What can I solve in six months. And ultimately, as a trust and safety person, you need to work closely with growth people because 99% of the company is going to be working on growth. And so you want to understand what what are their incentives? Yeah, yeah. And then once you understand those, you have to balance those two out.

Desi Velikova 06:18

Right. In your case, it’s even more challenging, because you have to be thinking about two types of users, the end user, the riders, but also the drivers they have to feel safe to. And I guess some marketplaces, they have even a big challenge, they have to be thinking about three types of users. So can you give us examples of recent campaigns or features that you’ve watched to address the trust and safety concerns for both riders and drivers?

Neil Shah 06:46

Absolutely. So one of the really great features we launched right after COVID hit was a face mask detection. So we were experiencing during Coronavirus, like inconsistency and policies. And people had a difference of opinions in the United States about whether or not to wear a mask. Go figure. So what is the role that Lyft can play in this? Well, drivers should not put their lives at risk, right for the for trying to make money. And so we needed to quickly develop a way to detect whether a person was wearing a mask. And there’s a lot of moving pieces in that equation, you need the ability to report you need the ability to cancel a ride because what if the driver is not wearing a mask, right? It works both ways. And so what we were able to build was a system that allows you to report then challenges a writer or driver to take a selfie with a face mask. And we had to develop that really fast. And we didn’t have any of that kind of technology. And there wasn’t any really publicly available open source or anything. So then we had to build models that detect if somebody is wearing a mask, and we have to worry about Okay, is this accurate and detection of a face mask? And is it biased in any way? I can’t go into too many details. But right, we wanted to make this a really good user experience. And that is something we have to move really fast on. And everybody agrees in the company that it was important to solve, right? Like the it’s rare as a trust and safety person to get a lot of urgency and visibility. And then you don’t have months to solve a problem, you have weeks. So we rolled that out pretty fast last summer. And we were able to increase facemask compliance by a significant amount and you know, make drivers feel safe, make riders feel safe. And then people who don’t want to wear masks more Unfortunately, not able to use our platform.

Desi Velikova 08:48

Trust could be very subjective, it means different things to different people. So how do you measure the results from your work, making sure that the campaigns and the features you worked on had a positive impact on the product? Or in other words, how do you measure trust,

Neil Shah 09:04

This is the hardest and most important problem as a trust and safety person that you’ll encounter. Because you won’t be able to fund your projects, you won’t be able to get an internal alignment unless you do this really well. And there were there are iterations and everybody is okay with that. Right? Like as you think about your goals, everyone will understand you have good intentions. But at the end of the day, if you are saying you will move a number and people don’t really believe you. Or it’s not possible to move that number or they don’t understand the trade off, then you won’t be able to you won’t be able to set yourself up for success. So here’s my playbook for creating trust and safety metrics. First, understand the problem you’re trying to solve and make sure everybody agrees that that is As the right problem, for example, we want to make sure that everyone who is driving, you know, as a Lyft driver, we have all their information on file, and we know that they’re, they’re the ones driving the vehicle at this moment in time, everybody will agree that is a good problem to solve. That is a good problem, like the thing to know. Now, how do you measure that? How do you know if you’re doing well, so you need to set a baseline. And I think this is where you can look to your friends, in your colleagues in growth, to understand what are the metrics they live by. So at Lyft, we think about the number of drivers that are on boarded, the ones that are off boarded, the ones that, you know, leave the platform for whatever reason, the number of people who have you know, let’s just say, accidents or violation, or whatever it is, and their standard terminology. And so then you want to think about your metrics in context to that, and how will the thing you build impact those metrics. And the key thing for us is driver supply. How many hours is a driver driving, and whether the number of drivers on the road by region etc. Now, once you understand how your growth team works, you want to brainstorm with them and basically understand, like, look like this is the problem I’m trying to solve. I understand this is your Northstar metric. If I were to run a test that did something to hurt that, like what would be the cost to you, as a team, that is a common language, this is the most important thing you need to speak common language with growth. Now the problem in trust and safety is often that you cannot, it’s hard to make like a pure ROI trade off, I can’t say, Okay, if I launched the safety feature x, and it costs this amount, but it’s net positive ROI. You don’t want to get into that conversation. But an alternative way to structure this is to say, here are the 10 things we will do. Good, right. And it’s like lower safety reports, increase confidence in the platform boosts sentiment, whatever, right? Like you can come up with a bunch of things in a framework. And then on the other side, you have the cons. So you say, well, we might slightly impact the supply of drivers while we run this test, because we’re causing some friction, and they will need to go and, you know, scan their IDs or something like that, right. And so now, you want to move into, you want to get buy in for an experiment. Try not to go boil the oceans, like we need to launch this 100% tomorrow. But it’s a lot easier for people to swallow. Doing a test first, understanding what will happen. And so like now, as you run the experiment, you’re gonna come up with new metrics, and you’re gonna see some interesting things. And you may have thought there’ll be a really bad impact. But it was fine. It was not a problem, right? And so now you can go back and say, Hey, we were able to boost sentiment, blah, blah, blah, blah, this was a cost. This is a scaled impact. You have to think about it from that framework, because it is the common language of 99% of the company

Desi Velikova 13:15

As someone who is wired to put growth first, I see a little bit of a challenge here. I hate to say but it feels like trust and safety and growth have a little bit of a difficult relationship. So how do you make sure that you strike that delicate balance? Right, that you’re making the right trade offs, the right sacrifices in the right moment? And I guess this questions got two elements really, one is how we make sure that we’re not harming the product too much. But also the inevitable internal battles in politics, people from the trust and safety team might have weight growth in marketing,

Neil Shah 13:55

I think you have to paint a picture of emotion. And so this is one of the skills that has taken me years and years to, to, to learn. But when you have, when you have when you’re in a meeting, and you’re presenting stuff, most people won’t remember what you said. But they will remember how you made them feel. And so if I start off a pic with a picture of let’s just say, you know, user on Twitter believes that the President is a clone, right? Like, which is a real thing that happened in Nigeria, people were like thinking, Oh, this, the President is a clone, there’s real misinformation. And then you paint a picture of how you might solve that and how a person in Nigeria and so all of a sudden this person is, is who you’re presenting to is like, wow, like, that would be terrible. I I this is an important problem to solve. And they will feel as if this is a very noble cause and it’s a worthwhile goal, etc. And so As you work with your your phone, you know your cross functional team, whether it’s marketing, whether it’s comms, whether it’s, you know, the, the founders of the company, you want to paint that picture you want to, you want them to understand, this is the problem you’re trying to solve. And this is why it’s really important. And that will overcome a lot of your problems as you go. And if you’re too incremental, if you’re too focused on metrics, only, you won’t really get there. And I think the the important thing is to get leadership buy in. And so that was something really powerful we were able to do at Twitter is we got buy in for this problem that is really hard to measure. And we needed to just start working on the problem. And when you paint that picture that people are not going to trust Twitter in 10 years, so we have to solve this problem. Then you get the, you get the sign off, and you push forward, you push forward.

Desi Velikova 15:56

Right, so let’s discuss some hypothetical situation. Let’s say I’m a PM, in charge of a trust and safety team. And I realized that I have a huge opportunity to address a very pressing trust issue by adding an extra message to our new users during the onboarding. In practical terms, this means we will have to add an extra screen to the signup flow, how to convince the team and get the buying given that we all know this will add friction.

Neil Shah 16:28

This is very relatable problem I worked on boosting profile pictures for Lyft riders. And it was adding another step at during onboarding that was one of the most metric moving things that we could do but would introduce friction. And so step one is understand the funnel. Okay? Where are users coming from the coming from app installs from marketing campaigns from, you know, whatever it is, and they download the app. Then step one, step two, step three, step four, they book a ride, right? That is the user journey, map that out know, at every single step, what the drop off is, and why there’s a drop off. In our case, for Lyft, the first step is share your location with some people drop off, they don’t want to share their location, well, sorry, we’re a transportation service, you need to know where you are in order to get you a lift, right. And so once you understand that funnel, you want to go into conversations with people who work on that funnel, and understand that you know what you’re talking about. So I can say, hey, there’s, you know, this, this split between Android and iOS, and I see this different thing happening in Android, I noticed this step, blah, blah, blah, right. And like, this is a seven day bookings, right? Like we call it bookings, or somebody who’s booking a Lyft. And then they’re like, wow, like, Neil knows what he’s talking about. He’s done his homework. He understands the funnel. And so I’ve earned credibility with the growth team. Now, I want to say, look, I want to run an experiment, here’s why it’s important. I understand, you know, a book, a 1%, decrease will lead to blah. But I want to run an experiment, because I think of this reasons. And at a company that has some scale, you’re going to, you’re going to run into less resistance on something like that, if you’re really hyper focused at an early company on old growth, growth, growth at all costs. Another way to make this argument is through financial losses. So a lot of companies in commerce deal with financial fraud. And when you don’t know the identity of somebody or the fraudster in your account, then you could that could result in millions of dollars in losses, which is a company though problem like every single person in every company deals with. So then you can speak from that language, you can say that look like? Yeah, we might lose some percentage of users. But we think those a lot of those users are fraudsters that we’re going to be able to keep out. Right. And so you’ve gotten another team interested in a different way. So you have to look for an angle, especially to get to experiment.

Desi Velikova 19:05

During your time at Twitter, you were in charge of fighting misinformation, which I guess in Twitter’s case is the biggest challenge when it comes to trust? Can you tell us a bit more about the projects you run there? And some interesting findings?

Neil Shah 19:19

Yeah, it was, um, it was really interesting. And we were trying to understand how people could trust a tweet, how you could trust a profile. A lot of the content in your timeline is what they call out of network. So you may follow 100 people, and you’re gonna see some of those tweets and your main timeline. And you’re going, they’re going to those 100 people follow other 1000s of people, right? And so that content gets injected into the timeline. Now, if you come across a piece of information, and you’re trying to decide whether or not to trust that information, there’s a lot of different things going on in a tweet there. The profile avatar picture, there is the text of the tweet. Does it sound like a human? Or does it sound like a robot? Right? There is the number of engagements, how many likes how many retweets. Then there at the top now you’ll see there is your friends liked or your friends calm whatever engaged with this tweet, right? There’s, there’s all these pieces of information that go into helping a person decide whether or not to trust this information. And I think this goes to really the heart and debate around whether or not to moderate content on a platform, which is a really hard job. It’s, it’s like if somebody says, you know, something happened? Are you going to take down a piece of information? How do you know what the source of truth is? Right? And so there’s this common like, don’t be the arbiters of truth. So how do you how do you help people make an intelligent choice about content trust? So one of the things we did was when you click on to go into a tweet, on the tweet Details page, you’ll see a little source, it’ll say, Twitter for iPhone, or Twitter for Android, or that it came from a bot. So a bot is an automated way to push out information. So CNN has a bot that pushes out news updates, right? There is an earthquake bot, and an earthquake happens it automatically if this, then that kind of situation. And this can help people make judgment calls about whether or not that this information, something they want to trust. And it’s something that people who are looking to Twitter to do more. I won’t say specifics, but they want Twitter to do more in this space. And this is something you can point to and you can start to have a conversation about like, yeah, like we’re adding these pieces of information like how the composition of a tweet, and and then it can be very valuable in in moving those conversations forward, which are very hard. It’s like when you’re talking to a regulator talking to somebody and there’s like taking out all these tweets, and you’re like, well, we don’t know what’s true. We don’t know what’s not true. But we can add these pieces of information. And look, I’m several years removed from the Twitter work stream. So this is all like many years ago thinking,

Desi Velikova 22:19

yeah, this will surely continue to be a massive challenge for all content platforms. Really. Can you give us other examples of companies that you think are doing an amazing job in the trust and safety space the players who should be learning from, and also the ones that you think are seriously failing?

Neil Shah 22:37

Yeah, I think that this is an important question and gets to a lot about how consumers think about the brand. And if you trust a brand, you’re going to trust that brand, with your business and with your data. Now, do you trust a big social media company with your data, I won’t get into which one, you have some biases here, I had worked on privacy controls over at one of them. And I think that you then can look at Are you going to trust somebody with your credit card information. And if you see on a website, that they’re not using encryption, I hope you don’t put in your credit card information, you want to make sure they’re using stripe, or they’ve encrypted that you know, https. And I think you can look in your ecosystem to quickly figure out places where you trust or you don’t. There’s only a few rideshare companies, which one, do you trust with your data, and which ones don’t you and same with e commerce. And if you if you break that trust, if you lose that trust, you’re gonna have a lot of issues such as, you know, there’s some chat applications recently, which changed their privacy policies and led to some user losses and people switching over to other messaging applications. And that’s the consequence. And it’s really hard to explain that upfront. If you go to your leadership team and say, look like if we mess up on this privacy launch, we’re gonna lose 300 million users like that you don’t, it’s hard to come up with that kind of an estimate. And so you want to as much as possible do the right thing. And it’s really hard to quantify the brand and user attrition. And so instead, think about look like, would you trust your data with this company? Would you trust, you know, using the service, and one, one thing I asked internally is like, Look, if you had a family member who wanted to drive or ride in the left, you know, and we don’t do this feature x or we, we don’t work on this problem, why? And it gets people thinking because now all of a sudden they think really deeply about it. I didn’t answer your question, but specific companies. I think I mentioned some indirectly,

Desi Velikova 24:54

yeah, you got quite a colorful examples that we can all just imagine what you might All right. And if you’re starting a new company now, how we will go about building trust? What’s your advice to founders that are early in their journey of building a company? And they’re not thinking about this enough until it becomes a problem. And it goes back to you know, how I started this interview is, when do we start thinking about it? And what’s your advice, in terms of what would be the first step that someone who wants to get serious about trust and safety should do at their company,

Neil Shah 25:37

I think you should find out, you should, you should hire a good person who believes in this kind of work. Somebody who has deep empathy for user trust. And so that can come in many forms like that could be an operations hire, that can be an analytics hire doesn’t necessarily need to be a product person. But this person will help you get thinking about the problems you might encounter along the way. So if you’re an e commerce, you’re going to want to find a really good person that understands financial fraud, because you’re going to deal with it. And it’s going to cost you millions of dollars, right? Like as you grow as a company. And you want to have those problems, because you want to grow, right. And like every every founder is focused on growth, first and foremost. So if you have a good hire, that has deep empathy for these problems, they can be the issue spotter for you while you focus on growth. So that’s one way to think about it. financial fraud losses, user trust losses, and and it is very meta to think about how that might hurt your growth. And I think that’s okay, because your first and foremost duty right now is to like, get product market fit, right? You’re still a product leader or founder, right? Who needs to figure out what this product does. And so if you focus narrowly on that problem, you can have other people in your company, think about the user trust problems, trust and safety problems and make make that a priority. So if you’re, let’s just say, building a social media application for kids, right? Like, then that’s a different story, you need to have like a very key component of your, of your team working on that from the beginning. just by the nature of and sensitivity of that audience.

Desi Velikova 27:21

Who do you think people should learn from it? They’re interested in the trust and safety space? Where do you learn it?

Neil Shah 27:28

From your customers, right? Like, the first thing I did when I got to lift was listen to customer calls, people who were frustrated about, you know, account access or safety issues. And that brings you closer to the user problem. I also signed up as a Lyft driver, I just started picking people up in San Francisco and I had certain trust and safety things in mind that I wanted to change. So for the rest of your time at this company by by getting close to the customer problem, you are going to be able to base your product thinking on those experiences of listening to calls talking to customers of being you know, being a customer.

Desi Velikova 28:17

Brilliant. That’s a brilliant note to wrap it up. Where can people find you online?

Neil Shah 28:24

You can find me on Twitter @npshah, you can find my writings on medium. And I think if you google me you’ll find some other of my trust and safety talks. Thank you.

Desi Velikova 28:35

Thank you so much for your time knew I wish you all the best.