A monthly or quarterly business review is essential to the operating cadence. Each P&L and operating unit (e.g., manufacturing, customer service, sales) should prepare a document (6 pages or fewer + additional tables in the appendix) to review results from the prior month. In some cases, it makes more sense to do this quarterly. Each group leader writes a BR with assistance from their team and finance counterpart(s). The team should meet with the CEO, their direct reports, and other relevant stakeholders (at the discretion of the CEO) for a one-hour review. Timing: within 5-10 business days (the sooner, the better) after each month ends.
The M/QBR is most effective as a management tool when it is a downstream component of other processes that contribute to a complete operating cadence. Those upstream components are 1) Input Metrics, 2) WBR, 3) PR/FAQ, and 4) annual operating planning (OP1/2).
First, you need to know and identify the controllable inputs to your business and establish a set of input metrics to measure each of them. Once you have a report that holistically covers those input metrics (in a WBR), you can focus on identifying the right initiatives to drive your inputs. The PR/FAQ process is the best practice used at Amazon to conceive of, develop, and decide on the right initiatives to pursue. These proposed initiatives are one of the primary ingredients of each team’s annual operating plan. Once the CEO approves the annual operating plan, the Monthly or Quarterly Business Review enables the CEO and leadership team to have a focused, detailed, one-hour or more discussion with each significant operating unit and function to ensure they are on track to achieve their goals. The M/QBR meeting provides a forum for updates on important results, changes in the customer and competitor landscape, discussions of hurdles and challenges, and alignment on adjustments and modifications to the plan.
A productive monthly or quarterly business review is a review of the business unit or team’s progress vs their annual operating plan. Here are the ingredients for a great operating plan:
Most executives spend nearly every hour of their workday in meetings, which is how they conduct their work. Since executive work involves decisions, and meetings are the mechanism for this work, it is reasonable to argue that their success depends on their ability to run effective meetings.
The goal of a well-structured meeting is to:
a) A clear and concise statement of the (true) state of the business.
b) The right set of well-defined metrics (output and input) and well-defined goals for each.
c) The right set of key initiatives (usually 3-5) to achieve your input metric goals (and correspondingly your output goals) based on your understanding of the flywheel of your business.
d) A plan for the resources (people, marketing spend, OPEX, CAPEX) required to achieve the initiatives (including dependencies).
This is an important context because the M/QBR checks in on the progress of the components and content described in your Annual Operating Plan.
Before describing the components, it is important to note these should be narrative documents, not a collection of tables plus a list of bullet points. Narratives are a forcing function for the writer to tell a story, distill, summarize, and make arguments based on multi-causal data. When I read through these documents, it is like looking at a movie where the projector drops 4 out of 5 frames — there are glimpses of the business, but they are brief. A good business memo can be read and understood by someone unfamiliar with the business.
Page one of the M/QBR should be a chart summarizing the actual vs. planned metrics and a narrative explaining and speaking to these results. The most important inputs and output metrics from the month are presented in a table (and how those metrics compare vs. plan and prior year). The team should summarize and comment on (in prose) any exceptional metrics (above and below plan). This is most effective when teams start managing inputs, not outputs. You can’t apply resources to or control an output, so you can’t discuss the work that the team is doing and whether it is driving the right results. Instead, about 80% of the metrics and discussion should focus on controllable and mostly customer-facing inputs. The primary thrust of the input metrics/WBR process is to address this.
Section two is a recap of the top 3-5 initiatives (for the current operating year) and a detailed update on the progress for each one, including the input metrics corresponding to each. Each major initiative should be listed along with a brief description/summary of activity and progress. These initiatives were (ideally) previously identified in the team’s annual operating plan, and the list should be relatively static from month to month. Status (red, yellow, green) against each initiative and a brief commentary on progress (wins and misses) should be included. Hiring/headcount is typically included here as it is usually one of the critical inputs to a rapidly growing business.
Concerning the initiatives, vague objectives like “Build Differentiated Value” or “Improve Personalization” are not actionable or valuable unless combined with a list of initiatives that define a specific work product, with dates, owners, and corresponding input metrics. This is also the case when the objective is an output like “Grow NPS to 73”. The problem is that none of these describe the specific work product the team is committing to build/launch/complete and its controllable inputs. Here is an example of a well-formed initiative:
Launch personalization engine 2.0 (by 10/15/22) across all product lines for web, iOS, and Android to increase home page click-through rate to 27.3% (+356 bps YoY), while achieving page load time of 240 MS (-10 bps YoY).
14 FTE (10 BIS, 4 TBH)
Team: Sarah G (leader) + 1 PM, 2 Design, 10 SDEs.
Click here to read the full PR/FAQ.
Traditional slide-based presentations often gloss over critical details, emphasizing design and presentation skills more than thoughtful analysis. The linear format leads to questions and interruptions. PowerPoint is a poor medium for describing and understanding complex problems. Typically, 75% of the time is spent on low-bandwidth information transfer, with only 25% focused on meaningful discussion and decision-making.
Narrative documents—composed of prose plus detailed data—encourage clarity, increase information density, and prioritize the quality of discussion. Using narratives, 25% of meeting time is spent reading, leaving 75% of meeting for debate, discussion, and direction. At Amazon, we learned that narrative meetings were a better way to arm leaders with the information needed to make decisions, gain alignment with their teams and set goals and direction.
This is where the owner/team describes any changes in the marketplace, competition, customer needs, results of past initiatives, technology, regulations, etc., impacting the business’s current assumptions and truths. If these changes are profound enough, the owner suggests course changes from the current operating plan. If the team proposes any course changes or would like input from the CEO/leadership, they should include this in the review.
This is where the team describes how they are trending against their budget from a cost point of view, emphasizing headcount hiring/planning. Is the team on track with its hiring plan? Are they over or under budget on fixed and variable OPEX?
Detailed tables and charts with additional metrics, headcount, product roadmaps, etc., can be included here for reference.
You need to distinguish between what your controllable input metrics are and what your output metrics are. Inputs represent the levers your organization has at its disposal to impact performance. Outputs represent the second-order consequences that you can’t directly control. By focusing on inputs, you can prioritize and execute the changes needed to improve your organization’s outputs in the long term.
In the PracticePlus QBR, examples of controllable input metrics include ‘Tele Success%’ and ‘Patient Onboarding Lead Time’ (POLT). Tele Success% measures the proportion of telehealth calls that meet or exceed a predefined quality bar. POLT measures the average days required for a customer’s patient to be onboarded to PatientPlus. Examples of output metrics include Churn Rate, Monthly Recurring Revenue (MRR), and churn rate. PracticePlus knows that improving the quality of telehealth calls and shortening onboarding times are key inputs to customer experience, impacting churn rate and MRR in the long term.
At least 80% of your QBR’s content should be focussed on inputs. Notice how the ‘good’ QBR example doesn’t cover classic output metrics like revenue and net profit until page 5, while the ‘weak’ QBR example leads with this as part of the Q1 overview section. This isn’t because financial outputs aren’t important (they are) but because they don’t offer as much actionable insight as the inputs that drive them.
The purpose of M/QBRs is to reflect the state of the business accurately. Readers should clearly understand how the business is trending against its annual goals. This includes covering progress against metric goals (notably inputs) and key initiatives. Radical transparency is required for leaders to learn the true state of the business. It involves informing them of the lowlights, challenges, and risks that are headwinds against achieving the organization’s goals.
Weak M/QBR memos avoid including bad news. The writer instead focuses on making their team look as good as possible. They use spin and biased perspectives to either hide issues or minimize their significance. They lean on subjective truths in lieu of objective truths.
Weak Examples | Good Examples |
We had yet another excellent quarter, with customer acquisition growing by 20% YoY and beating our goal. Social ad campaigns performed exceptionally well, accounting for most of the YoY growth. The best-performing campaign was Back To School, which achieved a 2.9% CTR on Instagram. | Customer acquisition grew +20% YoY and exceeded OP1 by +1.3%. Marketing campaigns performed weaker than expected, with CAC exceeding the plan by +9.8% and increasing by +6% YoY. This was driven by a mix shift towards social ad campaigns (42% of total spend, +420 bps YoY) following the underperformance of paid search where CPA was 24% higher than plan (+21% YoY). |
Our store’s selection continues to grow and exceed expectations. We added 242 new SKUs in Q2, beating our goal of 210. Notable new additions include the full range of blinky shoes from ShoeDog and soccer cleats from CantonaRox (the global market leader in soccer shoes). | 242 new SKUs were added in Q2 (+15.2% vs. OP2), including 4 soccer cleats from CantonaRox. We are still negotiating with them to secure the remaining 15 SKUs in their catalog, including the ProOne cleats that are our customers’ top selection request (ETA for conclusion: 07/25). ShoeDog added their entire catalog of blinky shoes (218 SKUs). Only 15 have been procured to date since the rest have yet to be viewed by a customer on our site. |
Given how much larger our assortment is, we now have 18,112 SKUs, still the most compelling of any of our competitors. | We now have 18,112 SKUs, which account for 82.7% of all target shoes Nielsen reported having 10+ units sold YTD (425 bps above benchmark). |
The goal of any M/QBR is for your reader to learn as much valuable information as possible in the allocated time. The style of writing used is an essential input to this. It should read more like an objective news report than a subjective opinion piece. You want to be concise and direct. Avoid adverbs and adjectives wherever possible. Replace them with data instead. Find opportunities to say more with fewer words. If you’re describing a vital matter, communicate its importance using numbers, not just words.
Weak Examples | Good Examples |
Our engineering team is working as quickly as possible to fix this bug and expects to deploy a solution by later this month. | A fix will be deployed by 4/13. |
Apple’s iOS tracking policy has been a nightmare for our marketing teams. The most impacted teams are paid social and digital display ads teams, whose performance has been inhibited by Apple’s drastic changes because they’re unable to target customers as effectively as before, hurting ROAS and marketing optimization. | Apple’s iOS tracking policy last year has inflicted a headwind on the performance of our variable channels, particularly Facebook and Google. No longer being able to target customers across platforms resulted in ROAS declining -43% YoY to $3.35 in Q4 2022. |
The functionality encompassing the systematic and automated distribution of email reminders on a day-to-day basis, initially launched in general release on 6/30/2022, has been instrumental in fostering positive progression in year-over-year performance, particularly during the most recent quarter. Furthermore, Patient Onboarding Lead Time experienced a reduction of 9.8% when evaluated on a year-over-year basis, resulting in an updated figure of 3.88. This decrease represents a modest decline when compared to the previous quarter, yet still demonstrates a slight uptick when compared to the OP2 goal we had set. | The automated daily email reminders feature launched on 6/30/2022 continued to drive YoY improvements in Q1. PO lead time declined -9.8% YoY to 3.88 (-0.5% QoQ, +0.2% vs. OP2). |
It’s easy to write a M/QBR that’s too long. A common pitfall is writing about everything, regardless of whether the content is relevant to the reader. You need to work backwards from the audience. For each paragraph, ask yourself, ‘So what?’ ‘What is the key takeaway I want the reader to have?’ ‘Why should they care about this?’.
This will help in three ways. Firstly, it makes your memo more insightful. Instead of just saying you launched something or reached a milestone, you also convey why it matters. Secondly, it prevents sharing details just for its own sake. Senior leaders aren’t interested in understanding how the sausage is made unless it’s necessary to make informed decisions. Thirdly, it ensures only the updates that matter most are included. Avoid writing about everything your team did to make them look productive/hard-working and to make yourself look like an effective leader.
Weak Examples | Good Examples |
We launched a pilot program of FTassist on 2/1/2023. 25% of customers who signed up for a free trial in February were randomly selected to receive support from 2 dedicated CSAs (treatment group). 22.3% of free trialists who qualified for FTassist scheduled 3 sessions with CSA’s support, resulting in 52% of free trialists taking this high-value action. | We launched a pilot program of FTassist on 2/1/2023. 25% of customers who signed up for a free trial in February were randomly selected to receive support from 2 dedicated CSAs (treatment group). Of the 834 free trialists who took part, 56% hadn’t scheduled a session within 10 days of registration and, therefore, qualified for FTassist. Of these customers, 22.3% ended up scheduling 3 sessions with CSAs’ support, resulting in 52% of free trialists taking this high-value action in treatment (vs. 44% in control). Assuming similar results for the control population, the overall FT conversion rate would have increased +540 bps. |
We developed a marketing plan for our Q4 holiday promotion. The events team created an online project management resource using Notion to help coordinate efforts across every team. They added all the key milestones required to make the event successful, and each team contributed to the tasks they needed to complete to meet them. The VPs of marketing, operations, and vendor management reviewed the planner to ensure they were onboard. After some initial disagreements, we made adjustments to align all key stakeholders. We also secured sign-off from IT compliance to use Notion as long as highly confidential data wasn’t used. Each team is expected to update the planner in Notion weekly to ensure we are on track to meet the key dependencies we have in time for the event. An online demo was created to train each project contributor on using the tool. The events lead has already begun emailing weekly status updates. | We developed a marketing plan for our Q4 holiday promotion. This will be the first event we use Notion for project management, which is expected to address ~65% of the operational pain points cited by employees who contributed to last year’s holiday promotion. It will also automate several coordination and reporting tasks (saving ~140 person-hours per event). |
Describing what happened isn’t the priority. Explaining why they happened is. Recognize which results stand out as signals, particularly amongst controllable input metrics. These merit the most attention. They typically satisfy at least one of the following properties: (1) wide gap to plan; (2) trending in a notable direction; (3) statistically significant change from the prior period. They can reflect positive or negative outcomes. Once identified, the M/QBR’s narrative needs to explain what drove these changes.
The same is true for updates provided on key initiatives. Identify the milestones that are at risk, flag them accordingly (e.g., ‘yellow’ status or ‘red’ status), explain why they are at risk, and describe what actions are being taken in response.
Weak Examples | Good Examples |
The Out-Of-Stock (OOS) rate increased 230 bps MoM to 8.2% (vs. 7.3% OP2). The global chip shortage caused this, and we will keep a close eye on it while improving OOS performance in other categories to compensate for the miss. | The Out-Of-Stock (OOS) rate increased 230 bps MoM to 8.2% in March (vs. 7.3% OP2). The increase was driven by challenges with vendor ABC, whose OOS rate increased from 18.5% in February to 89.2% in March (overall OOS impact: 215 bps). ABC’s Purchase Order (PO) fill rates fell to 15%, which they attributed to their supply chain challenges related to chip shortages. They committed to supply us 3 months-of-cover once back in stock (ETA: mid-May) to help mitigate against their unreliable fill rates in the future. |
WW sales increased +15% QoQ (+19% vs. OP1 target), with promising growth in France (+8%), Germany (+4%), Canada (+5%) and Japan (+65%). | WW sales increased +15% QoQ (+19% vs. OP1 target), driven primarily by +65% QoQ growth in Japan (WW impact: 1,132 bps ). Japan’s sales performance was driven by a +86% QoQ surge in web traffic, coinciding with an out-of-home campaign launched in January. A post-mortem will be reviewed on 4/25 and shared with other country teams. |
We launched Vox in beta on 3/5/2023 as a benefit offered to 20% of Platinum members. As of 4/3/2023, 32,520 sessions were opted in (63% opt-in rate) across 2,100 customers (75% of eligible Platinum members). The average WER was 18.5% (P95: 25%), compared to P95 WER success criteria for GA of 15%. | We launched Vox in beta on 3/5/2023 as a benefit offered to 20% of Platinum members. As of 4/3/2023, 32,520 sessions were opted in (63% opt-in rate) across 2,100 customers (75% of eligible Platinum members). The average WER was 18.5% (P95: 25%), compared to P95 WER success criteria for GA of 15%. This gap was larger than expected, putting our launch ETA of 4/30 at risk. Our path to green will depend on how successfully we’re able to fine-tune Whisper using reinforcement learning from human feedback (ETA for first cycle: 4/25). |
Annual goals are typically set as part of the Operating Plan process (OP1/OP2). Teams brainstorm and prioritize initiatives to improve the business’ performance against input metrics. The 3-5 most important initiatives are identified according to a mix of potential impact to inputs, resource allocation, and strategic significance. These goals are earmarked by senior leadership for tracking purposes. M/QBRs are a key mechanism for tracking progress against them, including identifying any risks that may require input from other stakeholders.
Weak Examples | Good Examples |
Make UI changes to the homepage to improve conversion. | Complete at least three A/B experiments on homepage UI that contribute >=2% conversion improvement each by 9/30/2023. |
Improve the Out-Of-Stock rate to 6% without lowering inventory turnover. | Improve Othe ut-Of-Stock rate to <6% during Q4 (-300 bps YoY) without lowering inventory turnover. |
Launch AI-powered transcription feature. | Launch AI-powered transcription feature for English-speaking sessions by 4/30/2023. |
Change product strategy to become mobile-first. | 1) Reach 100% feature parity between desktop and mobile by 6/30. 2) Launch all new features in 2023 on mobile native and web apps first. 3) Launch a responsive web app to unify desktop and mobile web to a single stack by 9/30. |
Multiple teams and individuals typically contribute to an M/QBR. This leaves them susceptible to portraying conflicting views and an inconsistent writing style. The ‘owner’ of the memo needs to edit these inconsistencies out of the document. Contributors should avoid criticizing other teams or individual team members. Blaming others without their input is a shortcut to losing trust and fanning a toxic culture. The process of writing an M/QBR should serve as a mechanism for cross-functional teams to align where there may be disagreements.
Weak Examples | Good Examples |
Out-of-stock (OOS) was 7.3% last month, missing the OP2 goal by 64 bps because the inventory planning team didn’t procure sufficient inventory to meet demand. They are preparing an action plan to correct the course. | Out-of-stock (OOS) was 7.3% last month, missing the OP2 goal by 64 bps. This was driven by a forecasting miss, with units ordered by customers exceeding the plan by +17%. We have since adjusted the safety stock setting to 22 days (+23%) to mitigate against OOS risk resulting from demand variability until the trend stabilizes. |
There was a 2-hour service outage on 3/18, during which customers couldn’t log in. It was caused by a deployment made by the checkout team. Their Quality Assurance Engineer failed to catch the regressive bug during QA. The root cause has yet to be determined. | There was a 2-hour service outage on 3/18, during which time 8,007 customers weren’t able to log in. The cause was a regressive bug found in the checkout pipeline. A COE will be reviewed on 4/5 to determine the root cause and preventative measures. |
While substance matters more than style in memos, style still matters a lot. Work backwards from your readers by making it easy for them to read and review your memo. Here are some guidelines to follow:
PracticePlus is a SaaS product mental health professionals use to manage their practice and patient relationships. Our mission is to empower therapists to invest all their time in providing care for patients by automating everything else. This document provides an overview of performance in Q1 2023 and progress against our operating plan.
Q1 Overview
Perfect sessions% declined -311 bps YoY to 42.39% (+12 bps vs. OP2) due to a mix shift towards non-tele sessions following COVID lockdowns in January 2022 ( impact: ~-680 bps). The rollout of patient text reminders on 11/18 improved patient session attendance and on-time starts, contributing to +60 bps QoQ growth. Self-Serve% remained flat QoQ at 84.9% and missed the OP2 target by -13 bps. We expect to claw back this miss once the mobile web scheduling CX is revamped in April. Tele Success% improved +13 bps QoQ to 94.24% due to fine-tuning our congestion control machine learning model using Q4 data. Patient Onboarding (PO) lead time continued to decline following the launch of automated daily email reminders in Q3, reaching 3.88 (-9.8% YoY, -0.5% QoQ, +0.2% vs. OP2).
Free Trial (FT) sign-ups declined -21.8% QoQ to 8.25k and missed OP2 by -5.5%, driven by (1) comp against the holiday promotion run in Q4 (impact: -988 bps); and (2) reduction in social ad budget (impact: -860 bps) to counter declining customer acquisition cost (CAC) trend in H2 2022. We decided to increase the budget by +25% for the rest of 2023 to achieve the FY goal of 42k FT sign-ups. FT conversion% improved +100 bps YoY to 60.4% (+15 bps vs. OP2) and is expected to grow >500 bps YoY by Q4 following promising results from the FTassist pilot launched in February (see page 3). The holiday promotion also contributed to the churn rate increasing +25 bps QoQ to 3.65% (+18 bps vs. OP2) as a result of +13% higher churn participating paid members (PMs).
Highlights
1. Launched AI-powered transcription feature in beta. Customers can now choose to have English-language telehealth sessions automatically transcribed, summarized and stored in their patients’ Electronic Health Record (EHR). This feature launched on 3/5/2023 and is available to 20% of Platinum members.
2. Promising results from FTassist pilot. We launched a pilot of FTassist across 5% of customers who signed up for a free trial in February. The estimated impact to Free Trial (FT) conversion amongst eligible customers was +540 bps vs 60.5% FT conversion in Q4 2022. We are ramping up our customer success capacity to dial up FTassist in Q2.
3. Podcast sponsorship generated an estimated Return on Ad Spend (ROAS) of $6.3. We sponsored a podcast for the first time, namely “Wellness Pros.” PracticePlus features in three episodes for $10k, resulting in 75 new subscribers and $63k in 12-month Customer Lifetime Value (CLTV). This compares favorably to our blended ROAS of $3.80 YTD.
Lowlights
1. Higher than planned churn rate. Our churn rate was +3.65% in Q1 (+22 bps vs. OP2, -5 bps YoY). This was primarily driven by higher churn amongst customers who joined as part of BFCM promotion in Q4 2022 (7.61%, impact: +18 bps). The promotion yielded positive incremental FCF overall, and we are confident we can meet our FY churn rate goal through new initiatives such as FTassist.
2. France expansion was delayed to 4/15. Practiceplus.fr was initially scheduled to launch on 3/31/2023. However, it was delayed due to a rounding error bug discovered during user acceptance testing on the day of launch (UAT). We are currently green for launching on 4/15/2023, and a COE will be reviewed on 4/20/2023 to aid with future country launches.
3. Attrition in the engineering team. 6 SDEs and 1 SDM from our backend engineering teams resigned in Q1, leaving 42 BIS and 19 TBH across all tech teams. The attrition was attributed to higher compensation offered elsewhere (Google, Facebook, and two Series D competitors). We are reviewing our current compensation policy with Finance and HR on 4/8/2023 (see hot topics in FAQ1).
Key Initiatives
Our strategic priorities for PracticePlus in 2023 are to (1) evolve therapists’ operations towards a hands-off-the-wheel (HOTW) future; (2) increase FT conversion by improving the onboarding experience; (3) increase our rate of product experimentation; and (4) expand the international footprint to cover 45% of global GDP. We selected 4 initiatives to track as C-team goals during OP1. This section provides an update on each one. An additional six org goals are covered in Appendix C.
C-TEAM GOAL #1: Launch AI-powered transcription feature for English-speaking sessions by 4/30/2023
11.5 FTE (9.5 BIS); Team: John C (single-threaded leader) + 1 PM, 0.5 UXD, 1 DS, 8 SDEs
Click here to read the full PR/FAQ.
Status: Yellow
PracticePlus’ mission is to empower therapists to invest all their time in patient care by automating everything else. Our customer surveys have indicated the current suite of features offered by our highest membership tier lowers a therapist’s administrative work by 60%-70%. The next most significant opportunity is for record keeping. Our customers spend 3-5 hours per week taking notes after each patient session for EHR record keeping to comply with HIPAA regulations. 80% of customers complete this task themselves—the remaining 20% delegate drafting to a staff member before editing their notes.
In November 2022, we began developing a new feature that automatically transcribes and summarizes telehealth sessions within PracticePlus’ EHR portal (codename: ‘Vox’). We opted to develop the transcription and text generation backends using OpenAI’s speech-to-text open source model (‘Whisper’) and large language model (GPT-3), respectively. Customers can opt in to have a session transcribed and summarized before starting each telehealth session hosted by PracticePlus.
We launched Vox in beta on 3/5/2023 as a benefit offered to 20% of Platinum members. As of 4/3/2023, 32,520 sessions were opted in (63% opt-in rate) across 2,100 customers (75% of eligible Platinum members). The average WER was 18.5% (P95: 25%). The success criteria we set for GA was a WER P95 of 15%. This gap was more significant than expected, putting our launch ETA of 4/30 at risk. Our path to green will depend on how successfully we can fine-tune Whisper using reinforcement learning from human feedback. Stakeholders will continue to be updated via weekly status updates (alias: VoxUpdates@). See Appendix E for a breakdown by WER category, along with examples for each.
C-TEAM GOAL #2: Launch FTassist in the US by 9/30/2023 to increase FT conversion to 65% in Q4 2023 (+450 bps YoY).
13 FTE (7 BIS, 6 TBH); Team: Sarah G (single-threaded leader) + 1 PM, 2 PgMs, 1 UXD, 8 SDEs
Click here to read the full PR/FAQ.
Status: Green
In 2022, we conducted a deep dive to understand customer behavior during free trials better. We discovered that 92% of free trialists who used PracticePlus to schedule sessions with at least 3 different patients converted to a paid membership. Conversely, only 36% of free trialists who didn’t reach this milestone converted. To improve overall FT conversion, we are launching FTassist, a new onboarding program that takes a proactive approach toward helping free trialists schedule sessions. Instead of relying solely on inbound onboarding measures (e.g., video tutorials), our customer success team will proactively contact free trialists to assist them with onboarding. This involves building a CRM system that (1) automatically identifies free trialists who have yet to schedule a patient session after 10 days of registration, (2) assigning them to customer success agents’ (CSAs) queues, and (3) provides CSAs the tooling they need to help free trialists schedule therapy sessions for them. This includes following up with patients who were invited to schedule a session on PracticePlus but have yet to go through onboarding.
We launched a pilot program of FTassist on 2/1/2023. 25% of customers who signed up for a free trial in February were randomly selected to receive support from 2 dedicated CSAs (treatment group). Of the 834 free trialists who took part, 56% hadn’t scheduled a session within 10 days of registration and, therefore, qualified for FTassist. Of these customers, 22.3% ended up scheduling 3 sessions with CSAs’ support, resulting in 52% of free trialists taking this high-value action in treatment (vs. 44% in control). Assuming similar results for the control population, the overall FT conversion rate would have increased +540 bps. The pilot was conducted using highly manual processes. The variable cost per incremental conversion was $220, which compares favorably to CLTV’s of $3.2k. We are building the tooling and automation required to scale FTassist’s coverage to all free trialists by 9/30/2023. We will also add a new KPI to measure the % of free trialists who scheduled 3 or more sessions in subsequent reviews to track progress. See Appendix F for a breakdown of performance by cohort.
C-TEAM GOAL #3: Increase deployment frequency to weekly by shifting to microservices-based architecture by 9/30/2023.
7 FTE (7 BIS); Team: Heather P (Principal SDE) + 1 TPM, 5 SDEs
Click here to read the full PR/FAQ.
Status: Green
Our pace of innovation has been progressively declining over the last several years despite our product and tech teams having grown from 19 HC in 2020 to 72 HC in 2023. We went from 51 deployments per year to 23 during this period. A tightly coupled architecture has gradually increased the number of technical dependencies. While this architecture allowed us to move quickly in the early stages of our founding, it became a headwind to developer productivity, agility, scalability, and scaling our product as time went by. The frequency of failures and outages also increased.
We began an initiative last year to refactor our software architecture into a services-oriented one. Each team has been given the framework to decouple their service as an independent stack. They cannot launch new features until they have built an API with clear documentation for the rest of the org. Exceptions may only be made to fix customer-facing issues where dependencies exist to meet C-team goals. This initiative has slowed down the rate of new features we’ve been able to develop since H1 2022. However, the investment will pay dividends in the long term as the velocity with which we invent on behalf of customers increases. Teams can deploy independently of each other at a weekly deployment cycle (at least). As of 3/31/2023, this initiative is 45% complete, and we expect to be 100% complete by 7/15/2023. See Appendix F for a breakdown of the status by the team.
C-TEAM GOAL #4: Expand PracticePlus to FR, DE and IT by 12/31/2023
21 FTE (8 BIS); 3 Country Managers, 5 Marketing, 2 TPMs, 2 PMs, 8 SDEs, 1 SDM
Click here to read the full PR/FAQ.
Status: Red
PracticePlus is currently fully localized for customers in the US (.com), UK (.co.uk), and CA (.ca), which collectively account for ~36% of global GDP. We set a goal to expand our coverage to at least 45% of global GDP by the end of 2023. France (FR), Germany (DE), and Italy (IT) were selected because they represented the highest number of customers using PracticePlus.com outside of the US. Customers in each of these countries will be able to use a localized site for language, payments, billing, tax and regulatory compliance. We are also expanding our sales and marketing teams in the UK to grow our business in each European locale.
The first of these expansions will be in FR (update ETA: 4/15), followed by DE (Q3) and IT (Q4). The FR launch was delayed due to a bug caught during UAT. This was an unfortunate incident, considering the PR interviews that needed rescheduling and the sunk cost incurred for fixed marketing campaigns that had to be pulled. We’ve identified the root cause as an oversight in product requirements related to how tax is billed owing to human error. A mechanism to prevent this recurrence in future country launches will be included in a COE scheduled for 4/20/2023.
Our current status for launching in both DE and IT is red. This is due to (1) attrition in our engineering team and slower-than-expected hiring and (2) underestimating the volume of throwaway work that would result from launching new locales while re-architecting our entire tech stack in parallel. Consequently, we have decided to reallocate tech resources towards Goal #3 to accelerate its completion from 9/30/2023 to 7/15/2023. This will leave us enough time to launch in DE by Q4. However, the IT launch has been postponed until Q1 2024.
Customer Experience Tech (CXT)
Perfect Sessions. Perfect Sessions% grew +60 bps QoQ and exceeded OP2 by +12 bps, driven by the rollout of session text reminders in November. Tele no-show % improved -109 bps YoY to 3.41% now that customers are sent SMS reminders of upcoming sessions by default. Tele on-time% also improved +252 bps YoY to 95.3%. We were pleased to exceed 95% for the first time in our history, thanks to dial-in links now being included in session reminder texts (launched 1/22/2023). Conversely, patient self-service scheduling% (Self-Serve%) remained flat QoQ at 84.9% (-13 bps vs. OP2). We are disappointed with our lack of progress here, especially considering 42% of PMs cited scheduling as the top manual task they wish to automate. Our next major milestone is to revamp the mobile CX for patient self-serve scheduling to incorporate feedback from usability studies, which we forecast will improve self-serve% by 162 bps (ETA: 4/30/2023).
Patient Onboarding. The automated daily email reminders feature launched on 6/30/2022 continued to drive YoY improvements in Q1. PO lead time declined -9.8% YoY to 3.88 (-0.5% QoQ, +0.2% vs. OP2). Our next major initiative will be to launch an automated calling service that contacts patients who have yet to complete onboarding 4 days after receiving an invite (ETA: 05/31/2023). This segment represents 18.2% of new patients in Q1 2023 and 2.1 days of aggregate PO lead time. An estimated 25% reduction in lead time will result in overall PO lead time reaching our Q4 OP2 goal of 3.35.
Tele Session Quality. We improved P90 latency by -15% after fine-tuning the congestion control model using Q4 data on 2/15/2023. This improved tele success% +13 bps QoQ to 94.24% (+104 bps YoY, +7bps vs. OP2). We are on track to launch an adaptive bitrate streaming feature by 10/31/2023. This will improve customers’ tele session quality by becoming more responsive to users’ network connection speeds, thereby improving P90 FPS to 30, allowing overall Tele success% to exceed 95% for the first time.
Marketing
Apple’s iOS tracking policy last year has inflicted a headwind on the performance of our variable channels, particularly Facebook and Google. No longer being able to target customers across platforms resulted in ROAS declining -43% YoY to $3.35 in Q4 2022. As part of OP2, we adjusted our OP1 plan by lowering the FY 2023 variable marketing budget from $10.3MM to $8.5MM with a target ROAS of $3.95. However, our ROAS has been $3.80 YTD, and FT sign-ups were down -18.5% YoY in Q1. We estimate that if we had stuck with our OP1 budget, ROAS would have declined to $3.55, and FT sign-ups would have been down by 7.2% YoY. We have responded by reverting our variable marketing budget to OP1 for the rest of the year. This will be funded by improved utilization in our customer success team and by improving the performance of our ad campaigns. For example, we will experiment with a new ads creative optimization product that Facebook announced on 3/5/2023, which is reported to have increased beta customers’ CTRs by over 15%.
This year represents the first time we have invested in fixed marketing channels in PracticePlus’ history. The goal is to raise brand awareness and elevate brand perception amongst therapists across the US. We ran our first audio ad in Q1, sponsoring three episodes of “Wellness Pros.” The host shared a custom landing page (practiceplus.com/wellnesspros) that was used to measure impact. As of 3/31, 75 customers who visited this URL have become paid members, resulting in a ROAS of $6.30 (vs. $3.80 in variable ROAS YTD). We will sponsor an additional ten episodes in Q2, which will be tested, and sponsor two additional podcasts, “This Week in Healing” and “Marketing for Therapists.” Lastly, we secured a prominent booth at the AMHCA conference that will take place in Orlando, FL, in August.
Customer Success
We delivered promising results for customers in Q1 2023. A revamp of our CSAs’ standard operating procedures (SOP) helped both positive response rate (PRR) and average response time (ART) improve +127 bps and -7.9% QoQ to reach 93.8% and 3.2 work hours, respectively. Utilization also improved +314 bps QoQ to 84.6%, driven by the automated sales & operations planning (S&OP) system we rolled out on 12/15/2022. Our 4-week S&OP is now more accurate in forecasting customer contact volume, with a Mean Absolute Error rate ~55% lower than the prior 12-month period when human forecasters were used.
Scaling CSA operations to meet the expected volume for FTassist outbound contacts is our priority in Q2. We are developing software to automate the routing system required to make the SOP seamless. We are also testing different scripts with customers participating in the pilot to optimize its output in time for dialing up FTassist in Q3. Furthermore, we are on track to distribute the Annual PracticePlus Voice of Customer (VoC) Survey by 5/15/2023. The format will be almost identical to last year’s version to monitor key trends like-for-like.
People/Org
Net fixed full-time employees (FTEs) grew +19.7% QoQ to 152. We hired 33 FTEs and had 8 departures (1 unregretted). All 7 regretted attrition came from tech teams, which now have 19 FTEs,TBH. The slowdown in hiring tech talent is a major concern, with only 9 new SDE hires at a lead time of 4.5 months (+32% QoQ). The attrition has been attributed to more competitive compensation being offered elsewhere since employee satisfaction amongst scores continues to trend high (4.2/5 amongst tech FTEs vs. 4.0/5 overall). The offer acceptance rate declined among SDEs to 25% (-820 bps QoQ) due to competing offers with higher compensation packages. We are taking the following actions as a matter of urgency to preserve our 2023 product roadmap: (1) hiring an additional tech talent recruitment agency to increase the pipeline; (2) organizing our first online recruitment fair to appeal to remote candidates; and (3) increasing employee referral fees by +33%. We also propose increasing SDE and SDM compensation packages by +18% to improve offer acceptance rates. See Appendix D for a breakdown of HC by org.
Financial Summary
Monthly Recurring Revenue (MRR) increased 37.3% YoY to $4.29MM (+11.3% QoQ, +0.8% vs. OP2), driven by new PM acquisitions (impact: 3,320 bps) and a mix shift towards platinum subscriptions (impact: 285 bps). The share of platinum members increased +90 bps YoY (+80 bps QoQ, +24 bps vs. OP2), and we expect this growing trend to continue as we focus on inputs such as Perfect Sessions%. Revenue also increased by +34.3% YoY to $11.95MM (+8.8% QoQ, +0.3% vs. OP2). Contribution Profit grew +37.5% YoY to $8.07MM (+1.9% vs. OP2). The contribution Margin increased by +156 bps YoY due to higher customer success agents’ utilization while preserving the Problem Resolution Rate (PRR) and Average Response Time (ART). Operating Margin also exceeded OP2 by +91 bps due to slower than forecasted net headcount growth in Q1 (see page 5).
TOPIC #1: Whether to increase the price for the platinum membership tier.
We have been debating whether to increase the price for platinum membership from $129/month to $149/month in the US. This would be our first increased membership fees since launching PracticePlus in 2017. The change is merited for 3 reasons: Firstly, inflation was 6.5% in 2022 and continues to trend above 5%. Secondly, we have increased the value of our platinum product by launching 16 features since it was first introduced. Each feature is used at least once by 70-75% of MAUs monthly. Our NPS score of 54 amongst platinum members continues to be the highest in our vertical. Thirdly, our AI-powered transcription service will incur ~$5 in incremental computation costs per month per platinum member. Conversely, raising pricing may hurt brand sentiment amongst our most loyal and valued customers. The estimated incremental revenue impact in year 1 would be $1.8MM. Our recommendation is to make this pricing change and to coincide it with the AI-powered transcription launch. We are not recommending raising prices for our other tiers in light of disappointing customer acquisition results.
TOPIC #2: Where to increase SDE compensation offers.
We recommend increasing total compensation targets (TCT) for SDEs and SDMs by +18%, primarily through increased sign-on bonuses and RSUs grants. This will bring us to par with the 50th-75th percentile in the industry. The additional investment amounts to ~$440k in 2023, a 91 bps headwind against our operating margin OP2 goal. The higher TCT will also serve as a lever for ‘dive and save’ offers amongst existing employees.
We have noticed an uptick in Practice Management Software products in international markets that are cloning our UI and gaining market share in countries we have yet to localize in fully. Examples include Hubterapia (market leader across 5 Latam countries) and Xinli Yun in China. We will need to accelerate our international expansion efforts to avoid the cost of entry becoming too high a barrier for us in 2024.
Expense tracking. We are still using a manual expense tracking system that involves employees submitting their expenses to HR via email. Receipts are sent as an attachment, and HR has to add them to an Excel spreadsheet before issuing reimbursements in payroll. We have yet to prioritize addressing this by adopting a system such as Concur but will look to do so in H2 this year.
Lost hardware. We do not have a suitable mechanism for ensuring company property (namely computing hardware) is returned by remote employees before ending their employment. 8 of the 14 former employees who left in the trailing 12 months have yet to return equipment. This hasn’t been a high enough priority given the low financial impact. However, HR is working with legal to find a reasonable resolution before a more robust system is put in place.
The Drug Enforcement Agency (DEA) is set to end the COVID-19 public health emergency status in May 2023. As part of this, schedule II drugs will no longer be prescribed via telehealth to patients who haven’t had an in-person assessment take place. This will likely impact our tele-sessions% amongst psychiatrists (~8% of PMs). However, we believe the impact will be negligible since subsequent sessions can continue via our telehealth system.
We are investigating 3 potential third-party AI solutions that would assist CSAs with responding to customer queries (via chat, social, and email). The solution would involve aggregating a corpus of historical customer contact data that would be used to feed recommended answers to CSAs each time they compose a reply. We are also investigating whether to offer this as a self-service solution for customers.
Appendix A: PracticePlus Tenets (unless you know better ones…):
1. We care more about customer KPIs than subscription revenue. We consistently seek ways to improve the quality of our product for existing customers because long-term customer satisfaction is perfectly aligned with long-term shareholder value.
2. We put patient confidentiality first. We never compromise on our commitment to protect our customers’ patients’ confidentiality, even at the expense of short-term commercial gains. Regarding security, paranoia is a feature, not a bug.
3. We price our product fairly for all customers. We don’t reward customers who attempt to churn by offering them lower pricing than loyal customers. All pricing promotions include a benefit for existing subscribers, too.
4. We work backwards from customers and patients. Customers trust us to launch features that directly or indirectly raise the bar for patient experience.
Appendix B: Key Metrics
PM: PracticePlus Paid Member
Booked sessions: sessions that were scheduled using PracticePlus
Perfect session %: Proportion of booked sessions that were tele-sessions, scheduled self-service by patient, confirmed by patient, started within 5 mins of schedule
Tele session %: proportion of booked sessions that included PracticePlus powered video call
Tele No-Shows: Tele-sessions the patient confirmed but didn’t dial in to
Tele on time%: Proportion of patient attended tele-sessions that started on time
ePM: established Paid Members, namely PMs who have been active for at least T3 months
Self-Serve%: Proportion of booked sessions that were scheduled self-service (SSS) by patients
Billed online %: Proportion of out-of-pocket sessions that were billed online
PO lead time: Average no. of days for patient onboarding completion after the first invite sent
Billed wkly: Total time scheduled for booked sessions per week across established members
MAU: Monthly Active Users
FT: Free trial
FT Conversion: % of expiring FTs that converted to paid plan
Platinum PM: Paid members who subscribe to platinum membership tier
Appendix C: 2023 Goals Status
Appendix D: Headcount Table
Hiring plan:
Get in touch for bulk orders, speaking events or advisory services.
© 2021 Working Backwards LLC. All rights reserved.
See our Privacy Policy and Terms of use