Shattering Case Interview Myths with Data

 

We track participants across multiple metrics in all TCO seasons. This allows us to test if common assumptions, myths, and best-practices about case interviews are true. Here are some data highlights from Season 1. We find the same patterns across all seasons.

 

INFOGRAPHIC 1

Incentives and penalties influence behavior

Candidates steadily improved their preparation knowing that session 13 would yield the mid-point decision and potentially work with Kevin Coyne – former McKinsey Worldwide Strategy Co-Leader. That is one of the reasons we have an elimination round in every single TCO season. It forces participants to improve. In seasons where we tried to remove the elimination round, participants did not try as much as they should have.

Slide1 - Copy

Notes: Firmsconsulting online video system tracks viewership and viewership intensity. Session 1 – 4 are resume editing sessions and have no prescribed preparation videos. There were no prescribed videos for sessions 21, 22 and 23.

When candidates knew there was an incentive/penalty in session 13 this modified their behavior to be more diligent. After Session 13 that diligence started to taper off again. Subscribers need to think about a rewards/penalty system they can also use to encourage healthy case practicing. We have seen this in every single season.

Candidates who used the videos carefully were able to understand the entire concept taught in the session, thereby allowing the coach to build on this foundation to teach more complex case techniques. Candidates who rushed through the videos missed key concepts and struggled in the coaching sessions.

Samantha: Did not watch the videos carefully and missed some of the important explanations.

Rafik: Rafik was similar to Samantha but Rafik watched more of the sessions and was more forgetful.

Sanjeev: This candidate was quite strong. He was diligent in his preparation though he missed a few things.

Felix: This candidate was the gold-standard in preparing and extracting information from the videos.

What does it mean for subscribers? If the videos are used, ensure the underlying principles of each technique are properly understood before the session commences.

Felix: “It is consistent with my own impression that I have always watched videos to prepare for the session; thus my graph is pretty stable throughout the sessions. It is interesting that I have several peaks in the graph, and my hypothesis is that I tend to adjust my preparation efforts according to my performance of the session. If I didn’t perform very well in a session, I tended to resume watching more videos to be better prepared. The implications are: (1), I am a person taking feedback seriously and relying on the outside evaluation to adjust my working effort and (2), the preparation videos did help my performance and preparation.”

Michael: This is also a very efficient way to learn cases. If the candidate picks up most of the techniques from the videos then we can focus on very complex cases in the session and also have more time to answer questions. We also tend to be on the same page w.r.t. the language used. Remember that these candidates only had access to our video library with about 70 videos. So Felix, in particular, was very good at extracting lessons from just a few videos.”

 

INFOGRAPHIC 2

Acting on feedback makes a difference

Candidates demonstrated marked differences in their ability to understand, remember and use the feedback provided. Those who acted on feedback immediately, broadly performed the best. If an candidate did not immediately fix the development area, after 2 or 3 sessions all the feedback started accumulating and overwhelms the participant.

Slide2 - Copy

Notes: Session 1 – 4 are resume sessions. The score is a weighted average of (1) the candidate remembering a development area, (2) at least attempting to use our suggestion to fix the problem and (3) their ability to correctly apply our suggestions.

It is important for candidates to always capture the top 2 or 3 areas for improvement and correct these. Trying to focus on all areas leads to little or no improvement because candidates find it difficult to remember and act on that feedback in the subsequent coaching sessions.

Samantha: Really struggled to understand and capture feedback. Due to the lack of understanding, was unable to use the feedback.

Rafik: Very forgetful at using the feedback and needed to be constantly reminded about best-practices.

Sanjeev: Overall good at the following feedback, but struggled in a few important areas like always forgetting to work faster or reduce the length of pauses.

Felix: Showed no problems understanding and immediately acting on feedback correctly.

What does it mean for subscribers? Always operate on the 80/20 rule by focusing on the biggest areas of improvement first and track the progress made.

Michael: Watch Felix’s consistency in using feedback. It is high throughout. The bump at the end is surprising since I would have expected a prolonged dip. Felix’s Ph.D. was wrapping up and she was spending lots of time in the lab. We had to find time for the sessions at 8 am or even 5 pm; usually before breakfast and later in the day when she was tired. So I expected her to forget more. While that happened from about session 16 onward, Felix started her McKinsey application process from session 20 and I think that forced her to be more diligent. That said, Felix was always very effective in her preparation.”

Felix: “Glad to see I respond well to the feedback, which is also consistent with the previous graph.”

Michael: Yes, Felix, you were very good at this. I recall once asking you to do brainstorming without writing out the answer. I was pretty surprised when you followed that feedback about 2 minutes later. Most candidates forget and need to be reminded more than once.”

 

INFOGRAPHIC 3

Good candidates are consistent

Good candidates prepare well and are consistently strong performers from the very first session. Though there are situations where candidates start poorly and steadily improve, that is more an exception than the rule. It is very rare for a participant to dramatically improve.

Slide3 - Copy

Notes: Session 1 – 4 are resume sessions and candidates were not scored on cases, but resume improvement. A single average score was assigned to all 4 resume sessions. Session 9, the big spike in the middle, was not a case score but performance on the debriefing call and should be ignored.

Candidates with a case skills gap in the first session compounded the problem by not understanding key ideas in the videos which followed: their case knowledge “deficit” grew making it tough for them to catch-up. Basically, they missed key ideas between sessions 5 and 8, failed to fix this learning gap and therefore were unable to tackle tougher cases much later.

If someone struggled to understand the principles of brainstorming and estimation right at the beginning, they will just become worse and worse since these are foundational skills with which to solve all full cases. Across every season of TCO we have never seen a participant dramatically improve.

Samantha: Very consistent and strong where a technique could be memorized and followed. Largely inconsistent were logic was involved.

Rafik: Very inconsistent performance driven by non-case issues such as poor planning and preparation before the session.

Sanjeev: Consistent at cases but peaked around session 13 and never fixed the top 3 development areas thereafter.

Felix: Very consistent, then saw a dip in performance around session 14 as she forgot basic skills. Yet, Felix was able to turn this around by session 16.

What does it mean for subscribers? In foundation sessions like estimation cases, brainstorming etc, candidates must understand everything. Since future concepts are built on these foundation concepts, weak understanding of the foundation concepts will not allow candidates to understand nor apply future concepts like reading of graphs.

Michael: Felix was very consistent in her case performance. I cannot say she ever had a bad day. I think she only really struggled from sessions 20 onward when we did the very tough bridge estimation, Siemens IT budget and economic strategy cases. We designed these cases to make her struggle a bit.”

Felix: “After about 10 sessions, my scores were improving steadily. This is consistent with my feeling that I was picking up the essentials for case interview skills.”

Michael: “Even so, this tells only part of the picture. From about session 16 to session 20 Felix’s overall performance, not just cases, was actually dropping as measured by her probability of success further down below in infographic 11. Since she was a good student, we needed to think of ways to improve her already high performance without making her burnout. This is a key problem with great students – keeping them challenged and motivated without giving them cases so difficult that it hurts their confidence.”

 

INFOGRAPHIC 4

Extreme weaknesses can rarely be overcome with another skill

Strong interview candidates tend to be balanced in their skills versus having extremes which balance each other out, which rarely works. Only very strong business judgment may lead to an offer over other weaknesses

Slide4 - Copy

Notes: Communication = conversational skills and language. Technical = the hard skills to complete calculations, brainstorm, build frameworks and generate hypotheses. Confidence = tone, body language, energy, eye contact, and professionalism. Session 1 – 4, 9 and10 were non-case sessions and candidates were not scored on cases

Candidates who performed well in all three areas had the most sustainable case performances and tended to enjoy the sessions the most. It is unlikely that any one skills, besides possible judgment, could help severely weak performances in any one area. This means candidates should not assume their phenomenals math, communication or other skill will make up for below-average performances elsewhere.

Be balanced and just focus on the basics.

Samantha: The extreme weakness was communication and taking feedback. They are linked and drove all other problems.

Rafik: The main weakness which drove all other issues was forgetting simple steps to improve performance.

Sanjeev: Although this cannot be seen in the cases, Sanjeev’s weakness which derailed him were delays on networking.

Felix: The main weakness for Felix was her German language skills. She is working on this but it will delay her applications and is the ultimate reason she withdrew from the process.

What does it mean for subscribers? Cracking the case is insufficient by itself. Pay attention to communication and confidence since these are hard to develop if they are neglected, and lead to unpleasant case experiences – how can your brilliance at cases be conveyed if you cannot communicate it? It cannot be conveyed. That is why communication is so important.

Felix: “This is also interesting that I have a balance of all three areas. Also, I would like to comment that confidence always goes with technical and communication skills. This is very true in my case: the better technical skills I have, the more confidence I have and can establish better communication.”

Michael: “Felix only had two “tells” when her confidence was impacted, she would become quieter and tended to be less concise. Granted, these are far from the worst symptoms she could have displayed. She even managed these very well.”

 

INFOGRAPHIC 5

Focusing on speed hurts candidates

Focusing on technique is more important than speed. As Felix was learning a better estimation technique, her speed initially was slower, yet as she adapted to the new technique, her speed steadily increased. Ignore speed. Just focusing on understanding and you will automatically become faster at cases.

Slide5 - Copy

Notes: All candidates were timed in the sessions in every season.

This exhibit and others from the program like it challenge a major myth about case interviews. We never focused on speed but notice how Felix’s speed improved as she focused on her technique. In other words, the only way to become faster at cases is to ignore the timing of cases and invest time in understanding the underlying mechanics of the case to find ways to make the technique more efficient. Trying to be faster with an inefficient technique, which is what most candidates do, does not help.

Samantha: Since Samantha struggled to understand the concepts explained, it was difficult to help her fix her techniques.

Rafik: In time, Rafik could have improved his speed. This was not such an issue for him.

Sanjeev: Overall, a slower candidate since he liked taking lots of time to build his approach and structure.

Felix: Overall, a fast candidate who could speak as she was thinking. This was the main difference between Felix and the other candidates who needed lots of time to plan their approaches.

What does it mean for subscribers? Ignore websites helping you practice math. Faster math skills only come from good technique: work on your technique. First, understand if the approach you use to solve math problems is efficient. If it is not, change the technique.

Felix: “Yes, an internal improvement of technique can automatically improve the speed. The math speed will not improve if one doesn’t know which question to calculate.”

Michael: “This should stop readers in their tracks! As you can see, not focusing on speed leads to faster times. Better structuring of cases always leads to faster times since inefficient techniques are identified and removed. This is also known as solving a case from first principles.”

 

INFOGRAPHIC 6

Teaching to practicing to maintenance

A good coaching strategy dedicates lots of time to fewer cases in the early stages, to ensure understanding. Maintenance, near the end, must cover many cases per session, but only done to test structuring of the issue. As you become better at cases, focus on the structuring of the problem versus trying to do 3 full cases a day with practice partners.

Slide6 - Copy

Notes: Session 9 was a debrief call and no cases were completed for the candidates, except Samantha who completed 1 case. Session 1 – 4 are resume sessions and no cases were practiced.

Doing many practice cases is not important. It is more important to improve the rate of cases covered per hour. While this is obvious, the compounding benefit is that it helps us cover more cases, which improves a candidate’s morale and allows us to do short drills to test all areas, which in turn leads to less forgetfulness for a client. At the peak of the training, we were covering every type of case with Felix in a single session.

In later seasons of TCO we expected participants to use the huge content library that is TCO to practice before the sessions and we just focused on the big development areas.

Samantha: It was not always clear to me that Samantha understood the material even when we worked quickly. So we slowed things down a lot.

Rafik: The candidate could handle the pace, but only when he was prepared, which was his main obstacle.

Sanjeev: Seemed to thrive in longer sessions at first, but quickly became fatigued as the training continued.

Felix: Longer sessions worked and Felix regularly asked for them to be extended, though her performance did not drop.

What does it mean for subscribers? Doing one of each type of case in every practice session ensures you don’t forget key skills. This only works if the training is layered with easier cases done earlier to slowly build up to more complex cases. Doing a type of case just once, will lead to forgetfulness.

Michael: “As you do more cases, you should become more proficient. It is like pushing a car. At first, it is tough. As you gain momentum, then you need to expend much less effort. Cases are like this. As Felix became better we needed to spend much less time on basic things and could focus on the main issues. We were in a sense doing maintenance in these sessions versus detailed case development. As you do more cases, you should be spending much less time on your preparation, but due to momentum, should see no drop in performance. This only works if you are studying well. Just doing lots of cases without a good learning plan will not lead to this reduction in time spent per case.”

 

INFOGRAPHIC 7

Too much or too little practice can be damaging

A good ratio for managing a training session is 30 minutes to a full case, 30 minutes to brainstorming, math problems etc and about 30 minutes dedicated to questions. Good candidates ask lots of questions.

Slide7 - Copy

 Notes: The times above only apply to the case coaching session. Additional calls or sessions to answer questions have not be included. This would greatly increase Samantha and Sanjeev’s times per session.

Here is a simple rule across TCO and 1-on-1 coaching. Those who ask lots of questions ALWAYS perform better, provided they are not asking questions with no purpose since they read this rule and decided to follow it without thinking about the questions they would ask.

The candidates who focused on technique, versus speed, were doing more cases, but taking far less time per session, which led to less fatigue, and this allowed them to maintain their energy levels far into the program. This also meant they had more time for questions.

Samantha: Spending a lot of time in the sessions but was weaker at taking our advice and using it. Samantha’s development areas were around understanding our feedback, taking notes in a way which led to understanding and communicating her misunderstandings.

Rafik: Development areas were different. I think the sessions were far too short since many times Rafik was not ready for the material and the session had to be shortened. Beyond this, Rafik struggled to recall and use our feedback.

Sanjeev: Sanjeev was fairly efficient in the time we had and I felt the pace was right and we covered the right amount of material.

Felix: Due to Felix’s friendly style and fast pace, we could cover more and try more complex cases. This made her sessions the most productive.

What does it mean for subscribers? Ignoring speed while focusing on technique leads to more cases covered and shorter sessions. This is a good strategy if you want to prepare over a long period since fatigue will lead to poor attention spans after just 2 to 3 weeks.

Michael: “This is more important than it looks. Having a lot of tough, detailed sessions is draining – emotionally and physically. This should always be avoided. It is far more effective to have shorter sessions where you learn smaller clusters of information. This can only be done via great preparation on the side of the client and coach. Fatigue is one of the main reason’s candidates do not perform well.”

 

INFOGRAPHIC 8

Poor preparation makes cases unpleasant

Candidates will rarely perform well in a session if they are under-prepared, and candidates who don’t perform well will rarely enjoy a session. Over time, the candidate falls further behind, leading to even less enjoyment. All of this hurts one’s confidence and leads to problems.

Slide8 - Copy

Notes: Candidates self-report this score immediately after the session is completed. Firmsconsulting does not discuss these scores with the candidates nor alter these self-reported scores.

Candidates who prepared well and performed better had more consistent enjoyment levels in the sessions. If they prepared poorly, they experienced frustration at handling the case questions and this was reflected in their “enjoyment” scores. Provided the scoring is genuine, poor happiness scores are a lead indicator for burnout in the process.

Samantha: The low levels of happiness are not surprising. Poor preparation led to many mistakes which made the cases difficult to complete.

Rafik: Notice how Rafik’s happiness levels spiked at session 6 when we made him focus on 4 non-case related areas to improve. Therefore, cases can be improved by focusing on simple administrative areas.

Sanjeev: The dips for Sanjeev was driven by a repeat session in 15 and session 20 when he suffered from lack of sleep. Both are understandable and outside his general positive experiences.

Felix: Very good at separating herself from the case so that she finds things to enjoy even in a relatively weaker session. This is an important skill to have because even a coach does not enjoy working with a candidate who is struggling.

What does it mean for subscribers? A subscribers success in a session is determined in the days spent preparing before the session itself. Poor preparation will rarely lead to success and happiness.

Felix: “Yes, the graph rightly depicts that I have enjoined the sessions all the time after I established the case working rhythm. Also, I am in general optimistic, and my enjoyment is not totally influenced by my performance. Of course, I enjoyed a lot of sessions with good performance, but during sessions that I didn’t perform as well (for example, Session 10 and 12), I regarded as a good opportunity to learn more, therefore the enjoyment of the sessions was not affected.”

Michael: “Felix does one important thing very, very well. She is able to make the case enjoyable for the interviewer and this is something most candidates forget to do. It is an absolutely vital skill to pass a case and get in. Think about it for a second: who wants to work with someone who makes an engagement less fun that it should be?”

 

INFOGRAPHIC 9

Coaching must adapt to improve learning rates

In adjusting the teaching style we had to constantly determine if the drop in learning rate was driven by the material in the session itself, coaching style or issues in the preparation competed prior to the session

Slide9 - Copy

Notes: This is an index. A score of 10 implies learning as much as the previous session. A score of 0 implies no new learning versus the previous session. Candidates self-report this score immediately after the session is completed. Firmsconsulting does not discuss these scores with the candidates nor alter these self-reported scores .

Every time a candidate reported lower learning rates in a session, we needed to go back and understand what had happened. We expected scores between 20 and 100. Better preparation from the coach ensures candidates have the foundation knowledge to use the information the coach will provide. This leads to improved learning in the sessions. Felix’s scores are interesting. It shows she was learning far more than other candidates.

Samantha: Learning levels where low since Samantha did not have a strong foundation from the videos to understand some of the material.

Rafik: Similar problem to Samantha since Rafik tended to forget the video contents or watch them incorrectly.

Sanjeev: This is an interesting scoring. Learning is fairly stable at first since Sanjeev is only scoring his access to new tools and techniques. When asked that communication learning be included, you can see the scores drop, because Sanjeev was weak in this area.

Felix: Also an interesting scoring. Very high learning rates are surprising since Felix was a strong candidate. This is probably due to the fact that Felix tends to be open-minded and simply absorbs all lessons and reapplies it. She does not assume there is nothing to learn and her level of attentiveness forces her to find useful information in the session.

What does it mean for subscribers? It is tough to follow the sessions if you don’t speak the language of the coach. If you cannot follow the coach, you make mistakes and if you make mistakes, you fail the case. That is never pleasant.

Felix: As a note, since it is a percentage of increase, the curve is steady in later sessions. It will be more impressive if the “total learning” is plotted.

Michael: Good point Felix. The only big problem with doing this is that we need to define total learning. The set of skills you needed to learn may be very different from someone else’s making comparisons very difficult. That is why we do not plot this point. Comparisons are misleading. We indirectly measure this by your scoring. A score of >8 means you knew enough to pass the case.

 

INFOGRAPHIC 10

Delays matter if you do not understand 1st principles

Candidates who were strong on the fundamentals of cases, versus memorizing frameworks, tended to perform consistently well despite any excessive delays between their training sessions

Slide10 - Copy

Notes: Sessions 1-4 are resume sessions and no cases are done.

Candidates strong on fundamental techniques showed no deterioration in case performance over fairly extensive gaps in their session. Candidates who did not understand the core fundamentals easily forgot things.

We see this in 1-on-1 case coaching as well. Clients who understand the basics can take a 2 to 6-month break and return with only a minor lag in their performance.

Samantha: Samantha actually improved overextended delays since she practiced fairly extensively. The challenge for Samantha was her learning efficiency since it was taking longer to learn new material. She even did well in the session 13 Fedex operations profit case but we discounted that down heavily due to the time it took her to learn the concepts.

Rafik: Struggled if there were too long delays. This is because the candidate did not adequately prepare between sessions.

Sanjeev: We saw no material declines between longer apart sessions, except on communication.

Felix: We saw no material declines between longer apart sessions.

What does it mean for subscribers? A true understanding of a concept does not erode over time. So take the time to learn the fundamental techniques since they are harder to forget. Poor candidates assume they will do better if they do more cases. That is an incorrect strategy. If you do many cases but have a poor understanding of the case principles, you will never improve,

Subscribers should take the time to isolate the foundation skills and ensure they have a thorough understanding of the material.

Felix: “Wow, I had a 24 days gap!”

Michael: “Felix, you were on vacation during this period so the gap was decided with us and planned well in advance. That said, I have seen few clients who can remember as much after such long training gaps. In this regard, Felix’s training techniques worked very well.”

 

INFOGRAPHIC 11

The probability score trumps the case score

No matter how well you do in case interview preparation, neglecting things like networking, resumes etc. . will lower your chances of getting an interview. All of these “other” areas are measured in our probability index

Slide11 - Copy

Notes: Weighted score where case performance only contributes 30% of the total. Attitude, adherence to next steps, networking, quality of follow-up etc. all comprise the remaining 70% weighting. Session 1 – 4 are resume editing sessions and it was premature to assign probabilities of success.

Important areas like networking, PEI preparation etc are not tracked in cases but play a huge role in securing an interview and ultimately an offer. We track them separately. The probability score is more important than the case practicing scores since the former measures more variables.

Samantha: Before session 13, the probability score is mostly comprised of case performance since we have not focused on securing the interview. Our feeling is that Samantha would have struggled even if she had secured the interview.

Rafik: The steep rise and drop is driven entirely by Rafik forgetting the 4-point plan we developed in the special session 6 debriefing session we set up. If Rafik had followed this plan, he could very well have been one of the top-2 candidates in the program.

Sanjeev: The entire decline was driven by delays in networking and assumptions made by Sanjeev that his original agreements/discussions with recruiters would apply. By the time he followed up, it was largely too late to do anything in 2013.

Felix: Excellent job of working with us in real-time. I could literally know what Felix was doing per hour and help edit her documents as needed. To be fair, her material needed little editing but as her coach I could at least guide her if needed. This was a another huge difference between Felix and the other candidates. Most took from 24 to 72 hours to respond. Felix responded in 12 hours, even when she was on vacation. So we were never unaware of what was happening.

What does it mean for subscribers? The Consulting Offer = Getting an interview + Dazzling in the cases. You cannot dazzle unless you get the interview, so do not forget about the other areas to help you secure the interview.

Michael: “This is the most important graph. Even if a candidate scores exceptionally well on cases, but was weak on their cover letter and networking, this measure would pick that up and penalize them for it. The case scores may show the opposite, but the probability score is the measure which counts. The probability score indicates whether or not the candidate is doing what is required to eventually secure an interview and offer. It is the aggregate measure we use and was very accurate. We use this to make changes to the program and adjust the learning schedule for the candidates.”

 

INFOGRAPHIC 12

Business judgment matters more than anything else

Business judgment remains the single biggest determinant of case interview performance. Strong judgment is vital to solving cases where frameworks cannot work: 50% of all McKinsey cases

Slide12 - Copy

Notes: A score of 1 = 100% and is the equivalent to an “outstanding” McKinsey associate or BCG consultant. This is the target in mind when coaching candidates.

Being well read is a major advantage and following our reading regimen can fix this gap, which must be fixed. Even Felix showed a huge spike in business judgment after following the program.

About 50% of McKinsey cases cannot be solved with frameworks. They require strong deduction skills. It is tough to master these types of logic cases since they rely almost exclusively on business judgment. In fact, they were developed because candidates started memorizing frameworks.

Samantha: Business judgment could be stronger, but it was not Samantha’s main improvement area. I suspect it would have become more important as a development area later in the program.

Rafik: The same as Samantha. Rafik was not in the program long enough for us to move to cases where this could have been a bigger problem.

Sanjeev: Fairly good at a judgment on estimating numbers and facts. Weaker on judgment around business models and more complex issues with many moving parts, like as we saw in the Telecom’s sweating the assets case.

Felix: This was Felix’s trump card. Even in session 13, when Sanjeev performed better than Felix, we asked Kevin Coyne to train Felix because we knew as the cases became tougher, Felix would overtake Sanjeev. That is exactly what happened.

What does it mean for subscribers? Subscribers must take the time to follow our reading regimen if they are to fulfill this key requirement. Consulting firms hunt for this attribute in case. This is not a skill that can be “crammed” in the last 2-3 weeks. If done correctly, it is developed over 4-5 months.

Felix: “Yes, I agree, business judgment is very important in solving cases. It can be improved by intensive readings from all areas and will lead to a benefit after ~2 month time.”

Michael: “In a career spanning over a decade in management consulting and then 2 years training clients, Felix had one of the highest business judgment skills we have ever seen. The fact that her business judgment materially spiked after session 20 is proof that business judgment can be developed methodically. It is a competitive advantage she must nurture because most applicants/professionals lack good judgment and are unable to develop this skill.”