Interview with a FAANG engineer.

Design an A/B Experimentation Platform

Watch someone solve the design an A/B experimentation platform problem in an interview with a FAANG engineer and see the feedback their interviewer left them. Explore this problem and others in our library of interview replays.

Interview Summary

Problem type
Design an A/B Experimentation Platform
Interview question
Design an experimentation platform that lets internal teams split user traffic between control and treatment experiences, keep assignments consistent over time, collect outcome metrics, and support both classical experimentation and ML-driven optimization.

Interview Feedback

Feedback about Dapper Badger (the interviewee)
Advance this person to the next round?
No
How were their technical skills?
2/4
How was their problem solving ability?
3/4
What about their communication ability?
3/4

TLDR;
You handled ambiguity well and asked great clarifying questions early. I do expect a senior to cover more ground on things like scale and extensibility but based on your area of expertise I'm not surprised you spoke most in depth around the analytics component (which is great).
To take your system design to the next level, anchor questions in the user experience (“how would a customer interact with this?”) to ground the design. When asking questions, briefly explain why they matter or what tradeoffs you’re considering as it signals depth of thinking.
Early in the session, outline your agenda (functional + non-functional areas, time allocation) to guide the conversation and steer the convo.
As you design, call out key considerations like monitoring, extensibility, and data storage tradeoffs, even if you can’t go deep, mentioning them shows awareness and then gives you a nice list to refer back to if you have bandwidth / time.

RAW NOTES / TIMESTAMPS

  1. A/B Experiment Platform
    • Prompt + functional requirements
      • good clarification, use the framing of asking for a user journey / experience to ground the conversation
    • Partition by user or session
      • great question
    • Good question on metrics component
    • Non-functional requirements
    • Starting to draw
      • app/user
      • Hashing for partitioning
    • Complete pictures
      • setting up databases - metrics database SQL
      • Separate process watching the user interactions here
    • Taking about data comparison
      • Leaning in on the BI side
    • Prompted on the bucket user component
      • We need to store information on the mappings
    • Things to touch on
      • Security
      • Extensibility
      • API calls
      • Splitting out read vs write heavy dbs
      • Scale

Interview Transcript

Dapper Badger: Okay. Well, yeah, thank you for, for making time. I know it was on fairly short notice, so yeah, I appreciate that.
Covert Gecko: No problem. I always love doing these more senior system designs, so looking forward to jumping in.
Dapper Badger: Yeah, me too. All right.
Covert Gecko: Yeah. Talk to me a little bit about— on my end, I'm not sure if you've used the platform before, but all I can see is your pseudonym and that you want to do system design practice, and that's it. So I'd love to hear a little bit more about what you're looking for and what you're hoping to get out of our session today.

Dapper Badger: So, I haven't done a systems design interview. I am more of like a, I started off really as a data scientist and then became more of an engineer kind of over time. And so yeah, I'm applying for a lot of machine learning engineer roles.

Covert Gecko: Cool. The reason I ask is I, um, I usually ask this question of like my more senior and staff group. So I'm just going to pretend and try to evaluate you at the staff level.

Dapper Badger: Sure.

Covert Gecko: So today I want to design an A/B experimentation platform.

Dapper Badger: An A/B experimentation platform. Correct.

Covert Gecko: So let me give you a little bit more context and a user experience example…

...Dapper Badger: So I guess there could be, yeah, there could be more complicated ways to do this, but so I guess what I would want to do is have some way, so I'm gonna start trying to draw something here...

Important considerations:

Final Thoughts

Recommended Resources