Blog

What is a scoring sheet? In a nutshell is a method for quantifying actions against a set of standards.

Why using it? Sometimes decisions are taken emotionally but motivated rationally. A good scoring sheet will distill the emotions in actions!

When? This method is highly valuable when faced with significant decisions—those that are challenging to reverse and have long-term implications. It becomes particularly useful in situations where you encounter contradictory signals and multiple objectives that require optimization. It’s ideal for moments when each option presents appealing advantages, yet their drawbacks cannot be overlooked. Furthermore, this method serves as a bridge between emotion and rationality in decision-making. It not only helps justify past decisions but also aids in identifying actions that can lead to satisfaction.

Overall, this tool helps you make an objective decision, clarify why a certain decision is more appealing, and provides you with a ‘negotiation’ tool to fine-tune existing options!

Two forces are interacting: the decisions you can make at this point (your options) and your preferences (evaluation criteria). Some criteria might be more important than others, and simultaneously, some actions may align more closely with a criterion. A well-designed scoring sheet will capture these interactions. The best decision is the one with the highest scores!

Let’s explore an example within the context of machine learning (ML), where hardware costs are significant. The dilemma often boils down to whether to rent machines, buy hardware, use services like Google Colab, or Kaggle notebooks. Each option comes with its own set of advantages and drawbacks. With numerous benchmarks and cloud providers listing their prices, the challenge is determining the best option for our specific needs.

To start, we first list our options, which in this scenario are straightforward:

• Buy a ready-made Deep Learning (DL) machine.
• Rent bare-metal machines.
• Subscribe to Google Colab.
• Utilize free tiers of services.

Now, we approach the more challenging part: identifying our criteria. Let’s outline the obvious ones to begin with:

• Price: The cost of each option, considering both upfront and ongoing expenses.
• Availability: The accessibility and readiness of the option to be deployed for our use.

These criteria are just the starting point. To make a well-informed decision, we would also consider other factors such as performance, scalability, ease of use, and support for specific ML frameworks or tools. A comprehensive scoring sheet would allow us to quantify how well each option meets our criteria, guiding us to the most suitable choice.

Indeed, relying on only one or two criteria for making a decision is often insufficient for a nuanced analysis. A rule of thumb is to have at least five criteria to ensure a well-rounded evaluation. Let’s delve deeper into our goal, which is to do deep learning (DL). This requires powerful machines, hence a new criterion emerges: Machine Performance.

However, while intensive training sessions demand high computing power, coding and debugging phases do not necessitate such robust resources, leading us to add Flexibility as another criterion. Additionally, the nature of the data we work with introduces the need for Data Protection as a crucial factor.

To summarize, our expanded list of criteria now includes:

• Price: Considering both upfront and ongoing costs.
• Availability: The readiness of the machine for deployment.
• Machine Performance: The computational power and capability to handle DL tasks.
• Flexibility: The ability to scale resources according to the task at hand, from coding to intensive training.
• Data Protection: Ensuring the security and privacy of sensitive data.

These criteria provide a more comprehensive framework for evaluating our options, ensuring that the chosen solution not only meets our budgetary constraints but also supports our technical and operational needs effectively.

Creating a table to organize your options and criteria is a great way to visualize and compare the potential choices. Here’s a simplified structure for our scoring sheet. This table includes a column for the total score and leaves the first row beneath the criteria empty, for now. Also, there is an empty column right next to the list of actions.

Actions ↓	Total score	Price	Availability	Machine Performance	Flexibility	Data Protection
Weights →
Buy
Rent
Subscribe
Free tiers

This table is structured to allow for a comprehensive evaluation of each option against the set criteria. You can assign scores to each option based on how well they meet each criterion. In the end, we will sum up the scores in the “Total Score” column to help identify the most suitable choice. This setup also leaves room for adding more criteria or options later, as needed.

To populate the cells with scores that reflect how well each option fits the criteria, using a scale from 0 (no fit) to 5 (strong fit), Let’s consider hypothetical scores based on common considerations for each criterion. The scoring on a vertical should be consistent across all options, for each criterion. This ensures that the evaluations are comparable and meaningful.

Here is an example of how you might score each option, keeping in mind that these scores must be personalized for your own needs!

• Price: A score of 0 represents the most cost-effective option (free), while 5 indicates very expensive.
• Availability: Higher scores indicate greater ease of access and readiness for immediate use.
• Machine Performance: Scores reflect the computational power and efficiency for deep learning tasks.
• Flexibility: Higher scores denote better adaptability for various computing needs, from coding to training.
• Data Protection: Scores indicate the level of security and privacy measures in place.

Indeed, the process of filling in each cell with a score can range from straightforward tasks, such as determining the price, to more nuanced evaluations like flexibility or data protection. A rough estimate, when faced with uncertainty, is preferable to exhaustive analysis for each cell, especially in the initial stages of decision-making. This approach allows for a balanced consideration of both quantitative and qualitative aspects without getting bogged down in details.

Actions ↓	Total score	Price	Availability	Machine Performance	Flexibility	Data Protection
Weights →
Buy		5	5	3	3	5
Rent		2	4	5	5	3
Subscribe		2	4	4	1	3
Free tiers		0	1	2	1	0

As you proceed with scoring, it’s common to uncover new criteria that weren’t initially considered. This iterative process is valuable, as it ensures a comprehensive evaluation of your options. For instance, you might realize the importance of customer support, software compatibility, or community and ecosystem around the platforms as you delve deeper. If new criteria emerge, they should be integrated into the scoring sheet to reflect these additional dimensions of decision-making.

Let’s assume we identify “Customer Support” and “Ecosystem & Community Support” as new criteria.

Again don’t spend much time on this, we will iterate again.

Now, let’s shift our attention to the criteria. In the blank row at the top of the table, assign a +1 or -1 to each criterion to indicate whether it’s a desirable feature (+1) or something to avoid (-1). For instance, assign Price a -1, as we aim to minimize expenses, and Availability a +1, reflecting our preference for continuous access to the machine.

Actions ↓	Total score	Price	Availability	Machine Performance	Flexibility	Data Protection
Weights →		-1	1	1	1	1
Buy	11	5	5	3	3	5
Rent	15	2	4	5	5	3
Subscribe	10	2	4	4	1	3
Free tiers	4	0	1	2	1	0

After completing this analysis, proceed to calculate a weighted sum for each option. Utilize the Total Score column for this purpose. Multiply the score of each action by its corresponding criterion weight, then sum these values for each action. This calculation yields a comprehensive score for each option, facilitating a more informed decision-making process based on all factors.

We can do this by hand, multiplying and adding values, or, we can use SUMPRODUCT function from our favorite spreadsheet IDE. Make sure that you put a $ sign before the weights row. For example the Total Score cell for the Buy action can be computed using:

=SUMPRODUCT(C3:G3, C$2:G$2)

Now we can fill the rest of the Total Score cells by dragging. The $ sign will “fix” the weights row (row with index 2) so one can “move” the formula around without changing the place where the weights will be picked up. Only the cell scores will be updated, for the new location.

As we fill the table, we can see the scores on the left column! Are we happy with the decision (rent a machine)? Well, let’s leave that for part 3.

In conclusion, the process of evaluating options through a scoring sheet provides a structured and quantifiable method to make informed decisions, especially in complex scenarios like selecting the optimal hardware for machine learning tasks. By systematically assessing each option against a set of criteria, we ensure that our decision aligns with our priorities, from minimizing costs to maximizing availability and performance.

In the second part, where we delve into the art of weighting our criteria. By carefully calibrating the significance we assign to each factor, we’ll uncover how to optimize our decision-making process further, ensuring that our final choice isn’t just good on paper, but perfect for our unique situation. Get ready to elevate your decision-making skills to the next level!

We have to recognize that not all criteria hold equal importance in every context. The next step in our journey involves fine-tuning the weights of these criteria to better reflect our specific needs and goals. This nuanced approach allows us to adjust the influence of each criterion on the final decision, ensuring that our selection process is as precise and tailored as possible.

First, we turn our attention to the criteria. They can range from essential to merely desirable. I assign them rankings from -2 to 2, with 0 indicating indifference—a clear indication that it may not be necessary on the scoring sheet. A negative score implies that a high compatibility with this criterion is undesirable. Take price, for example. Our goal is to minimize expenses; thus, a score of -0.5 indicates a moderate concern about costs, while a -2 signifies a strong intention to avoid expenditure.

Let’s refine our assessment by adjusting the preferences under each criterion. Replace the initial -1/+1 values with more detailed preferences to reflect the varying degrees of importance. This process doesn’t require extensive deliberation for each criterion at this stage; a preliminary adjustment will suffice, as we plan to iterate and further refine these values. The goal is to capture the essence of each criterion’s significance more accurately—whether it’s critically important, moderately necessary, or less desired. This step ensures our scoring system more precisely aligns with our specific needs and priorities, laying the groundwork for a more tailored decision-making process.

Below, an example where Data Protection is paramount!

Actions ↓	Total score	Price	Availability	Machine Performance	Flexibility	Data Protection
Weights →		-0.5	1	0.5	0.5	2
Buy	15.5	5	5	3	3	5
Rent	14	2	4	5	5	3
Subscribe	11.5	2	4	4	1	3
Free tiers	2.5	0	1	2	1	0

In our next iteration, we begin to analyze the scores across criteria more critically. A score of -0.5 for price combined with a 2 for availability suggests a strong preference for constant access to the computer, overshadowing cost concerns. Conversely, a significant emphasis on minimizing costs (e.g., a score of -2 for price) alongside a minimal score for availability (e.g., 0.1) implies a willingness to opt for the free version, accepting the risk of occasional access limitations. Start comparing the criteria between themselves and adjust the weights accordingly.

Now, take a moment to relax and clear your head. Creating a good scoring sheet can take from few good hours to days. It’s essential to approach each step with a fresh perspective. Give yourself a well-deserved break before diving into the next steps, as it will significantly improve the final decision.

In the process of fine-tuning our decision-making criteria can sometimes reveal unexpected insights. It’s essential to challenge our instincts and reevaluate the initial scores assigned to actions, especially for the most important criteria.

For instance, let’s consider the criterion of availability. While initially assigning a score of 5 might seem ideal for buying a computer, it’s essential to factor in the potential for power issues, software/hardware failures, or the need for maintenance. Perhaps a revised score of 4.5 aligns better with this reality. Similarly, cloud services can experience downtime, so a score of 4.9 might be more accurate.

A new criterion might emerge, such as the need for self-repair when owning a machine. This additional dimension can be introduced into the scoring system. Assign scores to actions based on this new criterion, and then determine its weight based on its importance.

Is the scoring board aligning with your gut feeling? There’s a good chance that it’s getting closer to the right answer!

I recommend going through an additional iteration or two over the scoring board, with a specific focus on the top 2-3 actions. Sometimes, you may find that even the criteria-objective grades need some fine-tuning (table cells). Don’t hesitate to make these necessary adjustments to further refine your decision-making process. Keep in mind the need for column-wise consistency!

Once the scoring board stabilizes, with no significant changes in action rankings, it’s time to make your decision. The top-ranked action is often the choice we should take.

In the advanced topic, we’ll explore how to leverage our scoring sheet as a valuable tool—a kind of magic mirror that helps confirm our intuitions and aligns the decisions with our desires. This ensures that our final decision isn’t solely dictated by the numbers but takes into account the boarder emotional context.

A magic mirror

A magic mirror reflects your desires. A magic mirror serves as a reflective tool, showing you your deepest desires and preferences. It is a form of self-awareness: It encourages introspection, allowing you to make choices that genuinely resonate with your values and goals.

A Crystal ball

A crystal ball tends to present predictions or outcomes with a bias, often influenced by preconceived notions or personal beliefs. It can reinforce existing biases and discourage critical thinking, potentially resulting in poor choices.

In essence, a magic mirror aids in decision-making by aligning your choices with your genuine desires and values, fostering self-awareness and balance. In contrast, a crystal ball may present biased or unfounded predictions that can lead to misguided decisions and reinforce existing biases.

We want a scoring sheet to behave like a magic mirror, revealing information about ourselves rather than a echo chamber that amplifies our fears and biases.

Some individuals rely on their intuition, and it’s a perfectly valid approach. The subconscious mind is a powerful tool in decision-making. But how does this approach align with the use of a scoring sheet? It’s simple! We can leverage our past decisions to ‘learn’ the weights assigned to criteria. By populating the scoring board with decisions we’ve already made, including some of our past choices, we gain valuable insights.

Since these decisions are already taken, we can also assess how accurate our intuitions were. Typically, a good past decision should rank among the top choices! Yes, there is a “criteria” drift but we can account for it, when scoring.

Having few past choices, can help us tune the criteria weights! Once this is done, we can take a step back and examine what is important to us. Start searching for actions that will satisfy those criteria! Very powerful tool! Because focus will yield results!

In the example above, we use the scoring sheet to examine the past and surface our desires (and aversions)

Incorporating past decisions into the scoring sheet serves another crucial purpose—maintaining consistency. When our emotions favor action X over Y, we may subconsciously adjust the criteria weights to boost X’s ranking, akin to using a ‘crystal ball’ approach. However, including a few past well-made decisions on the sheet acts as a valuable check.

These high-quality past decisions act as both a warning and an anchor. If we start manipulating the criteria weights excessively, the scores of these past decisions serve as a red flag. They remind us to stay true to our proven decision-making process and resist undue emotional bias.

Fairy tales often depict heroes who must sacrifice something to gain a much-desired skill. While real life may not be as dramatic, the concept of trade-offs is quite common. Occasionally, it’s possible to negotiate a better deal—for example, securing lower prices for ongoing rentals. But how do you determine the value of an offer? Or decide what it would take for an offer to be compelling enough to accept? These may seem like difficult questions, but they become manageable with the use of the cold, analytical scoring sheet. 

By simply adding a row for the new proposal and adjusting the criteria accordingly, you can effectively counter-offer and feel confident in your decision. It’s possible that the seller may compromise on certain aspects, such as availability, in favor of a feature you prioritize. With a scoring sheet at your disposal, you can swiftly evaluate whether the revised proposal remains attractive. For instance, if considerations like safe harbor or latency are not priorities for you, an option located in Asia might suffice for a minimum viable product (MVP). This tool enables quick and informed decision-making, ensuring you can navigate negotiations with clarity and satisfaction.

Regularly revisiting your chosen strategy is crucial to ensure it continues to align with your objectives, allowing for a better assessment of how well actual outcomes match your criterias. Additionally, it’s important to remain open to incorporating new options as they emerge, such as considering a new cloud service provider.

However, a challenge arises when one option begins to slightly outperform another, leading to a situation of high uncertainty. A simple solution to manage this is to introduce a form of hysteresis or to account for the “cost to change” in your evaluation. This approach is particularly effective when the cost of transitioning is relatively low, such as changing cloud providers, assuming your software is platform-independent.

On the other hand, if the cost of making a switch is high, or if the decision is difficult to reverse—like altering the course of your current project—it’s still advisable to use the scoring sheet. In these cases, it’s wise to apply a larger threshold for change. This method provides a structured framework for decision-making, ensuring that you carefully weigh the implications of significant changes or investments.

Keep in mind that the scoring sheet is a live tool! Keep it updated as the business moves forward!