A Measured Attempt to Productize a Culture of Servicing Customers

SUMMARY

In the midst of his consulting with a range of organizations, Peter Merholz framed a competency model with four distinct stages of organizational maturity. While blunt in nature, it’s compelling in its wide coverage of cultures that exist in the business world. A brief review serves as a stark reminder that we must identify our circumstances in order to meet people where they are.

At Karat, I moved quickly to build a team that could deliver on stakeholder-prioritized initiatives. Simultaneously, I discovered an extreme depth of technical debt and a command & control product culture that would impede the delivery of longer-term value via human-centered design.

This is a case study about adaptability, leadership, and as Seth Godin puts it, knowing when to strategically quit.

Key Learnings

1. Align on Clear Expectations: Before accepting an offer, it's crucial to agree on expectations regarding my role as a Design leader and how Design can impact the business. I don’t have the fortitude to solo change manage cultures anymore. Asking for a model of the company’s CLTV would provide me insight into the organization's maturity level.

2. Balance Assertiveness with Trust: Maintaining healthy relationships with peers is essential. This involves addressing their needs, simplifying design communication, and persevering until alignment is reached. However, as the number of peers increases, hands-on design involvement might decrease.

3. Seek Feedback and Alignment: Actively seek feedback and clarification from leadership to ensure continuous alignment with organizational goals. Creating a proactive feedback loop fosters deep alignment with executives. Resilience is key; setbacks are opportunities for refining strategies and enhancing communication.

4. Establish Baseline Metrics for ROI: Design KPIs pose challenges due to high cross-functional attribution. It's vital to differentiate between starting a new idea from scratch and inheriting an ineffective situation. Setting baseline metrics aligned with business strategy is essential for measuring ROI effectively.

5. Prioritize Short-term Value with Long-term Feasibility: Collaborating with engineering leaders is crucial for influencing architectural solutions that unlock long-term value. While short-term progress is valuable, overlooking accumulating debt—both technical and design—can limit future business solutions. Keeping an eye on both short-term wins and long-term sustainability is paramount.

DISCOVERY

Karat is a zero profit B2B service generating ~$35M in annual revenue, specializing in technical assessments for clients' engineering candidates. I accepted the role of Director of Design (Head) in late 2021—the seventh year of the company, joining six months after Karat secured a $110M Series C funding. The Karat ecosystem—the Interview Cloud™—empowers three primary archetypes to move through their respective journeys centered around the technical assessment: Clients feed their candidates to Karat and gig worker IVEs (interview engineers) facilitate and score the assessment, sending candidate results back to the client within a 48hr SLA.

After successfully delivering a zero to one primary care experience with multiple user archetypes, I figured if all things were equal (e.g. founding culture, executive leadership, transparent alignment across the company, etc.) designing an experience around fair and enjoyable assessments through this multiple archetype ecosystem would be a challenge, yet familiar in complexities.

In leu of being provided with a Day One onboarding document, I proposed to Werner Koepf, my SVP of Engineering, a 30 day plan to meet with key players across the organization, grok the existing product experience, and kick the tires on the tech stack.

A summary of what I discovered:

• Design System: The existing design system was fragmented, relying on an aging Bootstrap framework with inconsistent usage and lacking atomic design principles. This led to varied user experiences due to default Bootstrap components and compromised usability.

• Frontend Service Architecture: The existing FESA relied on a non-modular, monolithic application, causing responsiveness and usability issues. Interfaces were often nested within each other, impacting efficiency and user experience. Modernization and API integration improvements were necessary.

• Backend Implementation Model: The backend heavily relied on hard-coded elements without micro-services, creating dependencies on outdated technologies. This approach hindered data model clarity and efficiency, requiring a shift toward micro-services for modernization.

• Client Experience: The user journey lacked clarity and purpose, closely mirroring internal workflows. For enterprise clients, Time To Value (TTV) was excessively long (months), hindering client engagement. Startups faced shorter TTV but still experienced usability challenges.

• Candidate experience: Candidates lacked a seamless, integrated platform, leading to disconnection between their application experience and Karat’s involvement. The assessment coding environment, unique to Karat, posed familiarity issues for candidates.

• Interview Engineer (IVE) experience: The onboarding process for IVEs was manual and didn’t capture the foundational data necessary for a variety of tasks, including regulatory compliance and assignments based on skill sets. Manual transfer of interview questions into a live assessment and scoring impacted efficiency and objectivity.

• Customer metric issues: One stood out: The rate of candidates receiving offers after Karat's recommendation was lower than expected. Our Head of Solution Engineering attributed it to misalignment between client loop team expectations and Karat's assessment criteria. The insight had not been prioritized or addressed by Product Management.

• Business strategy & Product vision: While the company's business strategy was discernible, an explicit product vision was absent. Product Management was occupied with tactical tasks, hindering the development of a cohesive product vision, translating strategic objectives into actionable plans for the company to rally behind.

• Design debt: Historical design debt, ranging from subpar usability to incomplete user experiences, burdened the ecosystem. Ongoing efforts focused on feature enhancements without addressing fundamental obstacles. The absence of proper platform instrumentation hindered impact assessment, relying solely on overall metrics.

The technical and design debt were substantial—our existing infrastructure was fragile, processes were manual, the UX/UI was poor by any standard, and middle management seemed relatively unconcerned. The collective task ahead for ProdDev seemed daunting and the existing product discovery framework lacked depth & commitment. Product Managers (PMs) planned Engineering sprints a quarter out at a time, leaving little room for investigations and researching user insights, framing problems with desirable outcomes, and testing hypotheses.

If we wanted to close on a Series D round, it felt as though something had to give, and this context—along with Werner’s perspective on Design impact in my interview loop—edified my foundational understanding for why I was hired.

DESIGNING A DESIGN PRACTICE

I designed my operating model to ensure persona coverage across the entire ecosystem, indexing towards rapid prototyping, attaining user insights, and creating narrative consistency. The proposed model, rooted in Human-Centered Design (HCD), required close alignment with PM & ENG peers and leadership.

I was confident that once I had a few designers in play we could deliver in the short-term while I leveled up other functions to understand how we’ll deliver for their customers by putting the user at the center of our collective efforts.

By the end of 2021, we posted JDs and began sourcing candidates—well before the widespread tech industry layoffs. My gut told me that hiring was going to be a challenge given Karat's sales identity + the lack of Design representation on the website’s leadership page + any publicly available screenshots of the product experience.

Collaborating multiple times a week with my recruiter, we dug in to find skilled professionals who might be open to our growth-stage realities. It wasn’t a painless process, but we moved relatively quickly.

• In my first week, ZK Lin transferred to my team from being a SWE, serving as a product design engineer for prototypes and smaller design projects.

• By February, I had recruited my longtime colleague, Ray Mancini, as our principal product designer.

• In April, I onboarded Boney Yeldho, a former IBM interaction designer, as our senior product designer.

• In June, Leo Zhang, a former Facebook UX Researcher joined us, complemented by our research intern, Karina Shirokova from UW’s HCDE program.

We were understaffed (20:1 engineering to designer ratio) but I understood the game—we had to prove our value.

From the start, we established weekly critiques, team meetings, and office hours for colleagues across the company to join, along with on-the-fly Slack huddles and weekly 1:1’s that gave ICs the floor.

Each ritual was co-designed by the team; I wanted them to show up as their authentic selves.

Whether I’m managing managers or ICs, I operate as a servant leader, focusing on enabling their success. In this environment I needed to be add the highly responsive line manager hat to my repertoire—to aid the team navigating the organization, coaching up their methods, providing feedback on approach and estimations, and operationalizing their comms.

I framed unseen context to bridge the team’s knowledge gaps across the company, and methodically collaborated with peers to refine how they might optimally work with designers and researchers while considering our users in the pursuit of business outcomes.

We were all in.

OPERATIONS: FORMATIVE RESEARCH

It takes a seasoned, confident Product Manager (PM) to properly determine value, evaluate risk, and decide which product questions are a priority for an investment in user research. The cost for such exploration is exponentially higher in a startup environment chasing profitability, so within these environments I focus on operationalizing evaluative studies—specifically, formative research—in ways that will clearly empower PMs.

When the team forms a hypothesis—explicitly framing why they believe what they’re about to make will deliver particular outcomes—the PMs must determine how they’ll prove out their hypothesis. This is crucial in zero profit spaces, as there’s little room for error.

While the DNA of each initiative is relatively unique (e.g. a time dependent release that can be tweaked in production vs.a solution that has brand level focus and must work brilliantly out of the gate), the PM will likely lean into the most expedient, resource appropriate approach available to qualify their hypothesis:

• Deploy to production and learn whether their solution works through instrumentation, customer feedback, and/or summative research?

• Rapidly prototype, test, and iterate with users in Figma or a semi-functional prototype prior to making a deep investment in code?

• Employ both options or just ship it and forget it?

As I was recruiting Ray, I began to sketch out the foundation of a formative research framework to empower PMs in Karat’s under-instrumented environment, and as chance would have it, the perfect initiative surfaced to test the framework—improving the conversion rates of Karat’s Brilliant Black Minds (BBM) program.

Interested participants were abandoning the sign-up process at an alarming rate and we needed to stem the bleeding. After sitting with the team, discussing what they saw in the numbers, and listening to what they heard from reaching out to prospects who abandoned, I went through the process myself, and it became readily apparent to me that we were asking prospects for far too much personal information upfront and we were sending emails in ways that prospects had difficulty tracking.

Ray had just come aboard, so I diagrammed for him, and the team, what I had seen, heard, and experienced, and framed up a future journey that would simplify the ask of potential members. As Ray took ownership and worked his way through both a rebranding of BBM and the establishment of a new signup & dashboard flow, he collaborated with his PMs to identify their most pressing research questions before setting up a quick study with users.

The study helped the team make key shifts in approach, ultimately leading to a new sign-up process, with improved conversion metrics and greater engagement, which was the objective from the start. While excited by the results, I was more amped that we had worked out the kinks of the first formative study at Karat, which received rave reviews from everyone involved all the way up to leadership.

In a zero profit company with limited resources, developing a collaborative, risk assessment muscle to mitigate errors of both commission & omission is central to delivering impact.

DESIGNING THE FUTURE

As short-term design value was being experienced by our colleagues—at least to the extent of the initiative we were applied to—the jigsaw puzzle we faced within our ecosystem was multidimensional:

• On the client side, we faced exaggerated TTV rates within an inordinate number of human touch points and low offer rates to candidates we recommended. What might happen if we were to enable clients to choose or create their assessments in the product? And drafting the lead of my DPM colleague, how might we enable our customers to share candidate results across hiring teams?

• On the candidate side, 80%+ of engineering candidates failed to pass our assessment and make it to an onsite loop. What might Karat do with such an untapped population? Might a new revenue model appear if we engage with this cohort or the market in general? Might we be able to double down on the ‘shared candidate’ notion across companies by creating a marketplace?

My DPM peer and I would invariably discuss larger outcome-related issues, such as TTV, and why none of these foundational issues were getting prioritized. Much of the words were spent on feeding engineers, and not having time to focus. I could empathize to a real degree, though not completely as I saw room to change how we operated.

Our conversations invariably led us to the doorstep of tangible product vision, or our lack thereof. I wanted to align around what we believed our products could do for our customers and the business, and create a narrative presentation of this future state of our ecosystem—framing our users engaging with our products in a manner that would rally the company and unlock funded ideation.

Unfortunately this topic became a friction point, and eventually morphed into a territorial issue between the perceived roles of PM and Design, and our trajectory to lead, together, minimized each time we met.

I turned towards my leads and proposed that they develop skin in the future of the product for both the client and candidate experiences. I suggested an interaction design workshop of sorts—an Alan Cooperesque, Goal-Directed Design exercise that would elicit feedback from peers, and optimally, spur ProdDev to strive for deeper alignment on a product vision to address our business strategy.

Ray and I brainstormed the future of the client space together, balancing available user insight with the business outcomes that we understood to date. I took ownership of writing the context scenario and Ray illustrated it to bring focus to individual design personas that had been loosely referred to as the ubiquitous ‘customer.’ Boney and I followed the same approach to exploring the potential of the candidate space.

After a few weeks of 10% investment, we felt there was something worthwhile to share, regardless of our assumptions.

I walked my DPM peer and individual PMs through the deck, generating decent engagement, but the conversations ended there. Karat’s PMs had to operate with their heads down—documenting stakeholder requirements to book the next quarter’s worth of ENG sprints. I positioned these artifacts, context scenarios, as a inexpensive way to facilitate and iterate upon strategic conversations, and my DPM peer questioned why I was treading out of my lane.

As a design consultant who had successfully worked with large clients in similar fashion, and as a design manager who nudged the direction of a Fortune 500 company to rethink their flagship platform through a practice of GDD interaction design and alignment with executives, I humbly agreed to disagree.

I expanded my feedback circle in my 1:1’s with peers—from the heads of Data Science and Solution Engineering, to multiple Engineering Managers, to the Director of Customer Success. Each conversation generated a range of excitement with zero fundamental disagreement with our assumptions, technical or otherwise. Colleagues were excited to step out of the boxes they were operating within and offered nuanced insights into potential user expectations.

It was the type of collaboration that I was hoping for out of the gate.

I was sense making my way through insights that hadn’t been framed up previously, excited by the possibilities of the work that lay in front of us. Yet each person that I discussed this vision of our future with exuded a subtle air of disbelief that we could make either a reality.

It was a derivation of defeatism that was foreign to my experience working with talented people empowered to make the future happen, and the vibe settled into the shadows of my consciousness.

I brought our work to my next 1:1 with Werner, and even though the format was foreign, he asked solid questions and challenged my core assumptions around the notion of productization with a comment to the extent of, “What if our customers don’t want to have to do anything?” I asked what he meant, as it seemed counterintuitive to any progression I understood about scale while decreasing TTV, and he didn’t have anything more to share on the matter.

I followed up by asking what he identified as Series D milestones… I mentioned having tech, a path to scalability, strong revenue, etc. and the room went silent.

The following week, in mixed company, Werner casually shared a perspective on the traditional relationship between founders and vision—that they’re often inseparable. By design, Werner was my only conduit to receiving information from our CEO, so I couldn’t tell if I was being lauded for my efforts, passive aggressively told to stand down, or casually warned about the road ahead.

So I kept things moving.

A LESSON IN HCD

As I shifted my focus between escalating operational matters and recruiting my researcher, and the team continued servicing their PMs, a sense of momentum became palpable after shipping the new BBM brand experience. The launch coincided with Karat’s strategic sponsorship announcement with Serena Williams, which was a brand awareness coup for our founders. The updates to our visual branding and the simplification of the platform experience didn’t seal that particular deal, but the polish had made a difference.

Meanwhile, back in the trenches, engineers were inquiring about how we garnered user requirements—they were used to operating from BRD’s and were excited by the rigor we brought to sharing prototypes with users before investing in full stack development.

It felt as though we were establishing our identity and perceived value as a team—from how we worked collaboratively to our intentional nature to the quality of our designs themselves. And as timing often does, it was my turn to step into the spotlight.

Karat created a cultural tour de force during the pandemic— the previous in-person, weekly, all-hands WeConnect™ meeting moved to Zoom, and it was an experience. Employee emcees played intro theme music, shared company news, facilitated breakouts with ice breaker prompts for people to get to know one another, kicked it to leadership to frame a du jour insight, and once a month leaders of a different function were to provide insights into their team.

I viewed my turn as an opportunity to not only introduce the team and share the work we had accomplished to date, but to lay a foundational understanding of research, and how human-centered design can be a natural fit as a partner to sales and customer service.

How I approached this presentation turned out to be a pivotal moment for me at Karat.

At the 40 minute mark of WeConnect™ it was my turn to present, and over 15 minutes I shared our progress at Karat, with a focused articulation of HCD by illustrating a mock discovery, ideation, and prototyping case study.

I intentionally chose a completely foreign product example—a water pitcher—to avoid questions about domain particulars and to keep everyone’s focus on how HCD works.

Two days later, in my 1:1 with Werner, I received stark feedback. Apparently, our CEO wasn't pleased with the presentation—he felt it did nothing for the sales team and he had decided to freeze hiring. There wasn’t an invite to discuss particulars, and Werner didn’t offer insights into the path forward—he played the messenger and retreated into the background.

The lack of clear communication and even an iota of interest to help me understand the foul I committed was disconcerting. I began questioned my every move for weeks on end. It was on a Saturday afternoon that I came to terms with the situation I now found myself steeped within.

I was knee deep in an eight year-old, zero profit consulting service, leveraging a technology platform to empower a slew of sales people to successfully sell a single vendor service while hunting for opportunities to add more value. Our founders had bootstrapped Karat many years prior to an infusion of VC funding, and they viewed the path to reach profitability as fraught with risk.

And in walks me, insinuating to the entire company that an investment in user research would play a pivotal role in the success of our service offering.

I didn’t have the same access to leadership that I enjoyed at 98point6 and with downward communication so carefully crafted, if not veiled, I decided on that Saturday afternoon that I needed to either find a new home that valued clear communication & alignment, or double down on my upward managing skills, turn my active listening up to 11, and continue to press forward.

After much consternation, I chose the latter.

Tech Debt & DESIGN VALUE

By all heuristic standards and anecdotal accounts, our customers experienced Karat as a Frankenstein of features and human touch points rather than as a unified brand experience resting on a responsive platform.

I needed a blue chip design value win, and without a front-end engineer in play (nine months into my job, Karat hired our first front-end engineer, which was still a few months off on the horizon) I picked up from my 30 days report and a dove deep into the frontend service architecture, hoping to carve a path towards implementing polished UI and establishing an effective design system.

From Day One I had charged my team to use 5% of their time to closely collaborate with the handful of SWEs who met weekly to discuss how they might modernize the frontend. It was this collaborative space that led us to fully understand the state of our legacy design system.

A limited Bootstrap framework had been in play for years to assist engineers as they maintained a basic component library called Vera. It was far from an effective way to code, edit code, and store atomic designs specified by designers. For example, there was no standardization of loading and error states, which meant engineers implemented one-offs with little regard for impact to the user experience.

After the team and I put together a comprehensive deck with definitions, industry ROI data, and loosely projected effectiveness visuals, I found myself in a position to pitch the importance of a robust front-end to my former SVP, Werner, and my DPM peers.

No one saw such debt and lack of infrastructure as a pressing issue.

With all the empathy I could muster, I tried to understand their perspective. The prevailing attitude, which I had to sense make from various conversations, was that client retention in our B2B model didn’t hinge on solid UX/UI, compared to, say, the realities of a B2C.

What wasn’t spoken aloud, though, was that ENG would be challenged to build anything resembling a modern, responsive UI until investments were made in a modern front-end service architecture and a more flexible implementation model.

From a selfish POV: How exactly could me and my team be deemed effective if we can’t improve the experience by shipping polished UIs, let alone doing foundational service and interaction design work?

My assumption joining the company was that Karat was a Series C technology platform, the Interview Cloud™, in a growth stage moving towards SaaS automation and sales efficiencies. Through my various engagements, including that Saturday afternoon ripe with clarity, I couldn’t say what exactly it was that we were doing when it came to the user experience, regardless of the conversations I had upwards or laterally.

As we inched forward to rally support for the front-end and a modern design system, our DPM prioritized a stack of candidate initiatives, which we collectively rolled together and named, ‘Invite to Interview.” Ray dovetailed two existing candidate flow projects into a comprehensive redesign of the candidate experience, from the moment they were invited to their assessment to the moment they joined their Zoom call.

Our value prompt was:

How might we design the Invite-to-Interview experience so that we improve candidate confidence, set clear expectations, and increase the conversion rate of interview scheduling and attendance?

Along with solving for the above needle movers, I directed Ray to design a responsive modern UI, as the current experience was an eyesore. If done right, these novel atomic design elements would constitute a majority percentage of a new design system, which could then unlock a rapid application of new UIs to the client side of our ecosystem.

Ray updated the candidate email with comprehensive affordance, redesigned the scheduling calendar to take advantage of our ‘best IVE’ matching algorithm, added a new automated ADA accommodation flow, and created a white labeled dashboard.

The team engaged in our newly established formative framework and received useful insights into key HCI issues. The broader team was buzzing about the polished consumer grade interfaces, yet after months of work, openly iterating our solutions with with the ENG team, the initiative was quietly dropped into the backlog due to a high cost to unravel the backend implementation model to leverage the algorithm and communicate with novel front-end components.

The irony was not lost on me.

ProductizING A SERVICE CULTURE

When I landed Leo, our UX Researcher, the most valuable and untapped area of our ecosystem was hands down the client experience. Multiple PMs had rotated through or were let go, as it was an unforgiving space, steeped in design and technical debt that included only supported a subset of client touch points. As it turned out, the new PM in that arena was a vocal advocate for research, and made a business case for Leo’s time well before he was hired.

This area had the only full EPD team across all of the PM teams, as Ray and ZK had bifurcated PM focus.

For a few months, I watched the team design low fidelity prototypes, testing solutions that made choosing assessments simple. A relatively unstructured assessment data model would make it difficult to ship anytime soon, but regardless, the team ran their prototypes past the internal teams who built assessments for clients, and they deemed it to be a step in the right direction.

A win of sorts.

From where I sat, though, the friction in the process was less about the mechanics of internal teams making assessment selections, and more about the circus of follow up emails, calls, and Zooms with customers to reach a conclusion about the assessment design itself. I envisioned the team spinning their wheels in this direction for the next year, so I spoke with Boney and asked him to create a simple horizontal line segment that documented the existing client touch points.

With my eyeballs now on the stages from the sales pitch deck to archiving a filled candidate role, there were plenty of stops along the way. That intrigued me. I wondered what percentage of these Client <> Karat interactions were handled in the actual authenticated product, so I green lit another 10% project, and had Boney and Leo dovetail their existing assessment selection research with Karat internal teams into a larger service design initiative.

Without a clear lay of the land there was no way PM could envision, let alone prioritize a path towards an end-to-end product experience that could scale. Product led growth was literally off the table for us in today’s world. Boney, Leo, and Zach, their PM, interviewed 15 SMEs across 11 teams for over 50 hours, and revealed 10 new categories of pain points. I recruited Ray to make a visually digestible blueprint while Boney and Leo collected the last bits of data.

I spent my time coordinating with our new DPM, who had recently joined us from Amazon. He was excited by the work we had done, and recognized a clear line of sight into how we might solve for TTV and other big ticket issues, but even with full ownership of the client experience, he was hesitant to cosign on a clear path moving forward. It felt like to working towards an early conceptual artifact review with a client and the strategy lead is hedging his bet while rallying the team to move forward.

It was a frustrating set of interactions, culminating with his prompt: Am I here to ask for permission or for forgiveness?

Instinctively, I leaned into the latter.

I focused on coordinating the team with daily review sessions to make sure that our data was solid, that the blueprint hadn’t become visually overwhelming, and started writing our deck script featuring callouts of the future opportunities that were as plain as day.

We scheduled time with the heads of the teams that were interviewed, and as a full EPD team, with myself and the DPM on the call, I walked the room through the deck, ending on the following six opportunities that would results from designing an end-to-end experience:

1. Reaching 3x Mid Market Growth

2. Product-Led Growth in the Enterprise

3. Closing the Enterprise Deal

4. Drastically Reduce Time To Value

5. Empower Customers with Actionable Data

6. Further pursue Karat Goals and Mission

When I finished speaking, only one question was raised, coming from our Director of Customer Success, Mike Grafham, who simply asked, “What do you need from us to make this happen, Sean?”

Finally.

Moving Forward

Less than a month after our meeting about the opportunities found in an end-to-end experience, and just three months after telling my new boss that if Design wasn’t central to Karat’s plans, I’d be more than willing to help find Karat a first year manager to replace me, I was laid off—along with 20% of Karat’s workforce.

Three months later, in April 2023, another 20% of staff were laid off, including my former VP of Product. The Design team is now Ray and Leo, reporting into a wonderfully competent SVP of Delivery and Operations. Half of Karat’s engineering managers were also let go, including the head of Data Science. The PM team that had swelled to nine now has three Directors remaining.

I understand business much better than my art schooling might indicate, so none of this came as a shock to me.

During massive economic and technological shifts, such as our post-pandemic inflation and the rise of GenAI, respectively, venture firms weren’t going to continue their speculative anointment of horses into unicorns—the parameters of value, available capital, and investment return expectations measurably shifted.

In response, zero profit unicorns, such as Karat, who weren’t positioned for such economic realities had to quickly reset their assets to extend runway.

Such is life in Silicon Valley, the heart of American capitalism.

“Failure is something we can avoid only by saying nothing, doing nothing, and being nothing.”

- Denis Waitley