This is the first of a four-part series on aligning business goals and machine learning outcomes.

One critical task for machine learning leaders is to align project outcomes with the business’ goals. This is already challenging for traditional projects, but even harder for ML since their outcome is often unpredictable. So how can you align ML project milestones with business goals? How can you ensure different teams are all going in the same direction?

‍

In this series, we’ll present a tool called OKRs (Objectives & Key Results) which provides company direction and accountability under times of uncertainty. We’ll discover how to apply OKRs to machine learning projects, because there are key differences with non-ML projects. In the final article, we’ll consider ethics and safety problems when using these powerful tools.

‍

What are OKRs and KPIs?

OKRs and KPIs are often misunderstood terms. Let’s define OKRs as favorable outcomes or goals, and KPIs as interim measurements towards that goal. Each objective should be bold and inspiring, but also have a few measurable results attached. This allows teams to dream big, while also encouraging progress and holding contributors accountable. Here’s an example:

‍

Objective: Build a superior manufacturing production line.

Key Result #1: Achieve below 1% production failure rate.

Key Result #2: Produce 1.5x more than current capacity.

Key Result #3: Deliver the first 5.000 products to customers.

‍

OKRs were originally designed by Andy Grove at Intel Corporation, a company which fueled innovation in Silicon Valley for decades. OKRs were later adopted by early Google to manage a startup that was growing 10X many times over. This system has allowed companies to set bold objectives and even achieve or exceed them.‍

‍

Who should dictate Objectives and Key Results?

Ideally, objectives are defined top-down and key results are defined bottom-up. This allows a small group of executives to choose the overall direction of a company, while allowing teams to create key results that contribute to this direction. Brainstorming new objectives can happen in dialogue with employees, but leadership must ultimately converge on the most impactful ones.

‍

Conversely, teams working closely on the problem are in the best position to define key results. They should collaborate with their team managers to choose the results which move the needle on projects. Managers should be prepared to explain these key results to leadership and stakeholders, but leadership must ultimately trust them that these are the most critical results.

‍

How to balance OKRs with the right KPIs?

Some of the most complex work in modern companies is to find the right KPIs that support moving towards an OKR. Whereas OKRs are qualitative goals, KPIs are quantitative metrics. If not done carefully, pairing objectives with the wrong key results can lead to adverse outcomes.

‍

Imagine a firm that wants to improve their documentation process of ML projects. They want to capture all assets and document them for future reuse. They could reassign engineers to handle the tasks of tracking, annotating and unifying all important assets. An OKR could look like this:

‍

Objective: Capture all ML assets and make them interpretable

KR1: Track and store all assets generated by ML team

KR2: Annotate documents and track important metrics

KR3: Create weekly status reports for review meetings

‍

However, while capturing ML assets is an important objective, reassigning ML engineers to create documentation can have adverse effects. To start, hiring engineers is expensive and diverting their efforts to manually track assets that could be spent on developing models can end up hurting the organization.

‍

Furthermore, data scientists want to perform higher-level work they are trained to do. They resent doing tedious and laborious work that is “somewhat” related to data science (that might include painstakingly collecting and annotating ML assets). It’s one of the top reasons why data scientists are leaving companies, an event which is even more costly than the first outcome.

‍

So while this objective is ambitious and meaningful, the attached KPIs can unwillingly lead to negative externalities which actually hurt progress. This is why organizations should spend a long time defining favorable outcomes before starting a project. Companies should also have a rigorous review process to discover which metrics are the most impactful for their business.

‍