Case studyKaratB2B hiring technology

Mapping the 80%: what it would take to productize Karat’s service

Karat sold technical interviews as a service, shaped by sales and built by engineering. As head of design, I established a design and research practice, shipped work inside tight technical constraints, and led a service design study that exposed the biggest blocker to scale: 80% of client interactions happened outside the product.

Company: Karat; Series C, about $35M revenue
My role: Director of Design, head of design and research
When: November 2021 to January 2023
Team: Five: designers, researchers, and a design engineer
Reported to: SVP Engineering, then VP Product

Results

What the work established

80%of client interactions happened outside the product, the clearest blocker to scale
5designers, researchers, and a design engineer in a function I established, at 20 engineers for every designer
6growth opportunities defined for an end-to-end product, from 3x mid-market growth to product-led enterprise sales

From the service design study and team build-out. Shipped work, including a redesigned sign-up for Karat’s Brilliant Black Minds program, is covered in the case study.

The service

Karat ran first-round technical interviews for companies that needed to hire engineers faster than their own teams could interview. Its mission was to make every interview predictive, fair, and enjoyable. This customer story, produced by Karat, shows the promise from a client’s side: interviews scheduled around candidates’ lives, including nights and weekends; a hiring bar calibrated to each client; and engineering hours handed back to the client’s own team.

ContextA Karat customer story with Compass, produced by Karat.

What the story shows less of is how that promise was kept. Calibrating to each client meant solutions engineers working with its hiring managers to design and tune assessments. Settling on an assessment took rounds of emails, calls, and video meetings. Karat called its platform the Interview Cloud, but much of the service lived in the conversations around it. That gap is where this case study focuses.

The system

Three groups moved through every assessment:

ClientSends engineering candidates to Karat for a technical assessment
CandidateSchedules and takes the assessment in Karat’s coding environment
Interview engineerA contract engineer who runs and scores the interview
ClientReceives results and a recommendation within 48 hours

Every step involved people as much as software, and the company had been built around that. There was no onboarding plan when I joined in November 2021, so I proposed my own to the SVP of Engineering: 30 days to meet the key players across the company, learn the product end to end, and understand the tech stack. I shared the shape of the investigation with him up front and stayed aligned as findings came in. What I found set the terms for everything after:

20 engineers per designerand 4.5 product managers per designer
Sprints booked a quarter outfrom stakeholder requirements, with little room for discovery
A monolithic front endon aging Bootstrap, over a hard-coded backend
Months to valuefor enterprise clients, with a journey that mirrored internal workflows
A business strategybut no explicit product vision to translate it
Little instrumentationso design impact could only be read from top-line metrics

What I did

Product decisions at Karat arrived as requirements documents, and the previous design team had been handed them to turn into interfaces. Design wasn’t going to earn a role in shaping the product by arguing for one. My strategy was to meet the company where it was: earn credibility inside the plan, then show the future.

In practice, that meant three efforts, run in parallel:

Serve the roadmap first. Staff the initiatives product managers had already prioritized, and deliver them well, so design built credibility inside the existing plan.
Lower the cost of being right. Introduce research light enough to fit sprints booked a quarter out, so product managers could test an idea before committing engineering time to it.
Make the future visible, cheaply. Protect a small share of the team’s time, about 10%, for artifacts that started conversations the company wasn’t having: product scenarios, a design system case, a service blueprint.

Underneath all three, I led as a servant leader: co-designing the team’s rituals, translating company context the team couldn’t see, and coaching peers on how to work with designers and researchers. The first two became track one; the third became track two.

Track one

Building and shipping

Serve the roadmap and lower the cost of being right: establish the function, give product managers better ways to decide, and ship within the platform as it was.

Design the practice around the people in the system

I organized the function around the persona classes in Karat’s ecosystem (candidates, interview engineers, clients, and internal teams), so every product manager had a design and research partner whose focus held even when priorities shifted. I presented the operating model as where the team could be in 12 to 18 months, not a six-month hiring plan. The team would grow as it proved its value: answering product questions with research and testing ideas with users before they were built.

Hiring started in late 2021. Karat’s identity was sales-led, design had no presence on its leadership page, and the product had no public screenshots, so recruiting meant finding people open to growth-stage realities. Within eight months the team included a design engineer who transferred from software engineering, a principal and a senior product designer, a UX researcher, and a research intern. We co-designed the team’s rituals, from weekly critiques to open office hours for anyone in the company, and I published a leveling rubric so designers, product managers, and engineers knew what to expect at each level.

The 2022–23 hiring plan and operating model: persona classes (candidate, interview engineer, client, internal) across the top, levels of work from vision to prototyping down the side, and each role placed where it covers them. — **Artifact**The operating model, with each role placed against the persona classes and levels of work it covered, and the leveling rubric, published so expectations were clear to designers and the people who worked with them.

Spreadsheet of designer levels from associate to principal product designer, grouped as contributors, drivers, and leader, with rows for description, experience, scope and impact, and execution. — **Artifact**The operating model, with each role placed against the persona classes and levels of work it covered, and the leveling rubric, published so expectations were clear to designers and the people who worked with them.

Give product managers a cheaper way to be right

With engineering capacity booked a quarter out and little instrumentation, product managers tended to learn by shipping. I built a formative research framework so they could test a prototype with users before committing code, at a cost that fit the decision.

The first test case surfaced quickly. Prospects were abandoning sign-up for Brilliant Black Minds, Karat’s free interview-practice program for Black software engineers, at an alarming rate. I went through the process myself. It asked for too much personal information up front and sent emails that were hard to track. I mapped the current and future journeys for the team, and our principal designer took on a rebrand and a new sign-up and dashboard flow, working with his product managers to set the research questions before testing with users.

In testing, every participant completed registration, onboarding, and scheduling. After launch, sign-up conversion and engagement improved, and the relaunch coincided with a high-profile sponsorship announcement for the program. The study became the template for formative research at Karat, and engineers began asking how we gathered requirements.

The formative research framework: six stages (plan, prototype and recruit, test, synthesize, results and reflect, retrospective), each with process steps, deliverables, and notes. — **Artifact**The formative research framework, six stages with a defined deliverable each, and the current and future sign-up journeys for Brilliant Black Minds.

Redesign the candidate experience, and expose what held it back

Product leadership grouped a set of candidate initiatives under one effort, Invite to Interview. The question we set ourselves: how might we improve candidate confidence, set clear expectations, and raise the rate at which candidates schedule and attend? The redesign covered the invitation email, a scheduling calendar built on Karat’s algorithm for matching candidates with the best interview engineer, an automated accommodation flow for candidates with disabilities, and a white-labeled dashboard. The research plan came together in under a week.

I also directed the work toward a modern, responsive UI, because its components could form most of a new design system and speed up the client side. Testing with five recently hired engineers, each with recent interviews elsewhere, confirmed most of the flow: every participant went straight to scheduling from the invite. It also caught the hardest step before any code was written. No one set their availability on the new calendar within two minutes, and the team iterated on it. The design improved. The platform couldn’t carry it: connecting new front-end components to the matching algorithm meant unwinding the backend, and the initiative moved to the backlog.

Title slide reading Invite-To-Interview: Usability Test, conducted August 18–22, 2022. — **Artifact**From the usability test findings. Every hypothesis was stated up front; the calendar missed its target, which is exactly what a formative test is for.

Findings slide for calendar scheduling: the desired outcome, the hypothesis that four of five participants would select availability and confirm within two minutes, and the result that none did, beside a screen of the new availability calendar. — **Artifact**From the usability test findings. Every hypothesis was stated up front; the calendar missed its target, which is exactly what a formative test is for.

Make the case for front-end infrastructure

From the first week, my team spent about 5% of its time with the engineers working on front-end modernization. That’s how we learned the true state of the legacy component library: no standard loading or error states, and one-off implementations throughout. We built a deck defining a design system, with industry ROI data and illustrations of the efficiencies it would unlock, and I took it to engineering and product leadership.

Leadership didn’t see it as pressing. The prevailing view was that client retention in a B2B service didn’t hinge on interface quality. Invite to Interview showed the cost of that view: without investment in the front-end architecture, any modern interface was out of reach.

Current-state chart: under the limited design system, efficiency drops sharply when a project moves from design to engineering and recovers only near launch. — **Artifact**From the design system case: the same project with the limited system and with a working one, tracking efficiency from design through engineering to launch.

Future-state chart: with a working design system, efficiency stays high from design through engineering to launch. — **Artifact**From the design system case: the same project with the limited system and with a working one, tracking efficiency from design through engineering to launch.

Track two

Framing the future

Make the future visible: show the company what its products could become, and what was standing in the way.

Give the product a picture of its future

The biggest questions had no owner. Could clients choose or build their own assessments in the product, and share results across hiring teams? More than 80% of candidates didn’t pass the assessment to an onsite interview. What could Karat offer that untapped population, and could shared candidates across companies become a marketplace?

With the product managers’ attention booked a quarter out, I asked my leads to take a stake in the answers. Using Goal-Directed Design, we wrote and illustrated context scenarios for the client and candidate experiences, giving names and goals to people who had been referred to generically as “the customer.” A few weeks at 10% of the team’s time produced something worth sharing. I walked it through with product leadership and then with the heads of data science, solution engineering, customer success, and several engineering managers. None disagreed with the assumptions, technical or otherwise, and many added insight about what users would expect.

Illustrated scenario slide introducing Seth, a senior engineering manager whose candidates keep failing in the interview loop. — **Artifact**Client experience: Smart Assessments. A hiring manager builds and tunes his own assessment in the product instead of through rounds of calls.

Illustrated scenario slide: Seth chooses an expert backend engineer assessment and weights its competencies using data from previous interviews. — **Artifact**Client experience: Smart Assessments. A hiring manager builds and tunes his own assessment in the product instead of through rounds of calls.

Illustrated scenario slide introducing Jordan, a junior software engineer juggling job applications and a full-time job. — **Artifact**Candidate experience: Marketplace. One Karat assessment opens doors at many companies, turning candidates who didn’t advance into a new market.

Illustrated scenario slide: Karat analyzes Jordan’s self-assessment and surfaces potential matches at companies across its client pool. — **Artifact**Candidate experience: Marketplace. One Karat assessment opens doors at many companies, turning candidates who didn’t advance into a new market.

Map the service before designing the product

The client experience was the most valuable and least served part of the ecosystem. The team assigned to it started with assessment selection: they prototyped a simpler way to choose an assessment and tested it with the internal teams who built assessments for clients, who saw it as a step forward. From where I sat, though, the friction wasn’t in choosing an assessment. It was in the stream of follow-up emails, calls, and video meetings it took to settle on an assessment design at all.

I asked how many client interactions actually happened in the product, and set up a 10% project to find out. A designer, a researcher, and the product manager for the area interviewed 15 subject-matter experts across 11 teams for more than 50 hours, and surfaced 10 new categories of pain points. The service blueprint covered every client interaction from the sales pitch to archiving a filled role: what clients saw, the preparation behind it, the support processes, and the calendar and manual time each step took.

The finding was stark. 80% of client interactions happened outside the product, and every other barrier to scale depended on a decision to productize the service. I took it to the heads of the teams we had interviewed, with the full product, engineering, and design team, and closed on six opportunities an end-to-end experience would unlock:

Reaching 3x mid-market growth
Product-led growth in the enterprise
Closing the enterprise deal
Drastically reducing time to value
Empowering customers with actionable data
Opening new value: fairness and DEI, candidate quality, and global scale

The only question came from the Director of Customer Success: “What do you need from us to make this happen?”

Title slide reading End-to-End Client Product Experience, prepared by Karat Product Design and Research, beside a detail of the service blueprint. — **Artifact**The readout. Each opportunity was shown against the blueprint, highlighting the part of the service it would change; here, reaching 3x mid-market growth. Open the full service blueprint

Slide titled Opportunity: Reaching 3x Mid Market Growth, showing the service blueprint with the early sales and purchase steps highlighted as the area an end-to-end product would change. — **Artifact**The readout. Each opportunity was shown against the blueprint, highlighting the part of the service it would change; here, reaching 3x mid-market growth. Open the full service blueprint

Key learnings

Read the operating model before you design the practice.In a company where sales sets priorities and engineering sets capacity, design’s room to move is decided before you arrive. I now ask how a company models the lifetime value of a customer before accepting a role, and agree up front on what design is expected to change.
Make the case in the company’s language.Human-centered design presented as a method reads as overhead in a sales-led business. The blueprint readout drew a different response because it spoke in growth, time to value, and enterprise deals. The method stays the same; the argument has to be translated.
Design value is bounded by architecture.Invite to Interview was researched, tested, and still couldn’t ship. In a platform with deep technical debt, the most useful early partnership is with engineering leadership on the front end and the implementation model. I’d make that the first investment, with an engineering owner, before the first redesign.
In a service business, map the service before the product.When most of the value is delivered by people, a product backlog optimizes the smallest part of the experience. The blueprint should come first; it shows which product decisions actually matter.
Set baselines before you need to prove ROI.Design impact is hard to attribute across functions, and harder in an under-instrumented product. Inheriting a weak experience is different from starting fresh, and only a baseline set early can show the difference.

Questions about this project? Email me.