Article 1 of 24: Should Fernanda Follow a Rule or Learn from Experience?

Understanding Rules and Machine Learning

Fernanda is an estate agent (a girl’s gotta do what a girl’s gotta do!). She knows her neighborhood, enjoys helping people find homes and has developed an admirable patience with common phrases such as “deceptively spacious” and “easily-maintained garden”.

Like most experienced professionals, Fernanda uses a mixture of direct instructions she has to follows, and lessons from her previous work. More specifically, some of Fernanda’s decisions follow instructions supplied by her agency, while others draw on patterns she has noticed through experience (for example, which property details tend to be associated with higher sale prices).

Now suppose Fernanda wants a computer to help her make an initial assessment of new properties. How should she teach it what to do?

She could give it an explicit instruction, such as “Follow this rule when assessing a property”.

Alternatively, she could give it records of previous sales and a procedure for finding a useful pattern. In this case her instruction is less explicit: “Use these examples to work out how to assess another property”.

That brings us to the question at the heart of this article:

What is the difference between giving a computer a rule to follow and giving it examples from which to learn?

Before answering, let us give Fernanda a specific problem:

1. The problem: Which homes are likely to sell for at least $300,000?

Fernanda’s agency wants to organize incoming properties into two categories. Why? Well, I am not an estate agent but I guess this may help match properties to buyers’ budgets (maybe Fernanda is a bit posh and prefers to only work on the houses that rich clients can afford?). Anyway, let’s call the 2 possible categories YES and NO (see Table 1 below):

Category	Description
YES	Includes houses which are expected to sell for at least $300,000
NO	Includes houses which are expected to sell for less than $300,000

Table 1: Categorizing Houses Based on Price

Note that at this stage, the agency does not need an exact price estimate. It simply wants an initial category to help organize its work.

Now, imagine that for our example, Fernanda receives only two details about each property (see Table 2 below):

Available Information	Possible Values in Our Example
Number of Bedrooms	2, 3, 4
Location	Standard or Prime

Table 2: Property Details Available to Fernanda

Her required answer, for each property under consideration, is simply YES (it is at least $300,000) or NO (It is less than $300,000).

Of course, actual property prices depend on many other things, such as floor area, condition, transport links, whether the “peaceful outlook” overlooks a nightclub, etc., but for our purpose here, considering only 2 features are good enough. In other words, we are deliberately simplifying the situation so that we can see what happens at every step of the decision-making process. We can make the world more complicated later. Just wait and we get there!

“What is your objective?”, I hear you yelling at me. Here it is:

Our objective:

Using the available property details to predict its price category.

2. First approach: Someone supplies a rule

Suppose the agency manager (good chap but not important enough to have a specific name for now!) gives Fernanda this instruction:

Property Price Rule:

If a property has at least three bedrooms, predict YES. Otherwise, predict NO.

The reasoning sounds plausible. Larger homes often sell for more, and bedroom count seems a convenient starting point. Beside plausibility, this rule is rather simple. It will not take much to remember and apply it to new houses so, it certainly sounds useful.

With this rule in place, Fernanda can now apply the instruction immediately (see Table 3 below):

Number of Bedrooms	Location	Apply the Rule	Prediction
2	Prime	Fewer than three bedrooms	NO
3	Standard	At least three bedrooms	YES
4	Prime	At least three bedrooms	YES

Table 3: Applying a Property Price Rule to Selected Properties

Notice from the results in Table 3 above that location makes no difference under this rule. The information about location is available, but the instruction does not use it at all.

A computer could also follow exactly the same procedure:

1. Receive the bedroom count,
2. Check whether it is at least three,
3. Return YES if it is, and NO otherwise.

This is a rule-based approach. A person has supplied the decision rule, and the computer applies it.

There is nothing inherently wrong with that. The instruction is certainly clear, easy to inspect and inexpensive to apply. The question (an important one) is whether it gives useful answers.

The manager’s confidence, although impressive, does not answer that question (managers rarely gives their charges confidence so, in some sense, this is not surprising😊).

3. Let us check the rule against experience

Fernanda being who she is (meticulous and not blindly trusting her manager), collects six completed sales. Because these properties have already sold, she can compare a prediction with what actually happened (see Table 4 below).

Property	Number of Bedrooms	Location	Actual Sales Price	Correct Category	Prediction: Using “At Least 3-Bedroom” Rule
A	2	Standard	$200,000	NO	NO (correct)
B	3	Standard	$250,000	NO	YES (incorrect)
C	4	Standard	$280,000	NO	YES (incorrect)
D	2	Prime	$300,000	YES	NO (incorrect)
E	3	Prime	$400,000	YES	YES (correct)
F	4	Prime	$450,000	YES	YES (correct)

Table 4: First Round of Prediction

As you can see from Table 4 above, the rule gets three out of six cases right.

That is to say that the manager’s particular rule (based on the rule-based approach) is 50% correct as seen below:

No. of Correct Predictions ÷ Total Number of Cases = 3 ÷ 6 = 0.5, or 50% correct

For now, counting mistakes is enough. We will examine more detailed ways to assess predictions later in the series.

Look closely at the three errors:

• The three-bedroom Standard property (Property B) sold for less than $300,000.
• So did the four-bedroom Standard property (Property C).
• The two-bedroom Prime property (Property D) reached $300,000 despite having fewer bedrooms.

Fernanda begins to suspect that bedroom count alone is not a particularly useful basis for this decision.

Note something important:

The rule is being followed perfectly. It is the rule’s predictive usefulness that is in question.

That distinction matters. A computer can execute an instruction flawlessly and still produce a poor answer.

Now what?

4. What exactly does “learning from experience” mean?

As we just observed, the manager’s rule has made three mistakes across six properties. What should Fernanda do next?

She could inspect the historical sales herself, decide that location looks more useful and write a replacement rule. That would be Fernanda learning from experience. A computer given her replacement rule would simply follow this new instruction.

But we want to explore a different possibility and ask:

Can the computer use those historical examples to choose a prediction rule?

To see how this might work, let’s start with something small enough that we can follow every step.

Let’s assume that Fernanda has three suggestions:

• Perhaps the manager’s original bedroom rule is still worth considering, or
• Perhaps requiring at least four bedrooms would produce better results, or
• Perhaps location is more useful than bedroom count.

Each suggestion proposes a possible relationship between information about a property and its price category. We can express those suggestions as three alternative prediction rules (see Table 5 below):

Candidate Rule	Rule Description
Rule 1	Predict YES if the property has at least three bedrooms, otherwise predict NO
Rule 2	Predict YES if the property has at least four bedrooms, otherwise, predict NO
Rule 3	Predict YES if the property is in a Prime location, otherwise, predict NO

Table 5: Enhanced Property Price Rules

Note that these rules are alternatives. The computer will compare them and select one. It will not apply all three in sequence to reach a single answer.

Now we give the computer the following procedure:

1. Apply each candidate rule to all six historical properties,
2. Compare each prediction with the correct category, obtained from the actual sale price,
3. Count how many mistakes each candidate makes,
4. Select the candidate with the fewest mistakes. If candidates tie for the fewest mistakes, select the one listed first (the tie instruction simply makes our procedure complete. It does not mean that an earlier position in the table makes a rule more trustworthy. This, also demonstrates that sometime there are more than one way to get to a reasonable answer😊).

But hang on. Aren’t we still using rules?

Yes. But the important change is how the prediction rule is chosen.

Previously, the manager chose a rule and instructed the computer to use it (no fuss, no muss). Its performance on the historical examples played no part in that choice.

Now, we supply several candidates and a method for comparing them. The computer uses the historical examples to determine which candidate to select. With different examples, a different candidate could win (ain’t life wonderful!).

There are therefore two separate activities to consider this time:

• Learning: Using historical examples with known answers to select a prediction rule.
• Predicting: Applying the selected rule to another property whose sale price is not yet known.

The selected rule is what we call the learned model in this example (it is “learned” because it has used actual examples to “learn” what works best and it is “model” because it aims to model or capture or predict what happens with new or unseen examples).

We should also be clear about the limits of what the computer is doing. We supplied the three candidates. The computer is testing the possible relationships we suggested. So, it is not inventing new ones or discovering other property characteristics for itself.

This is a deliberately limited (stylized) example of Machine Learning. Later, we will explore methods that adjust numerical settings rather than select from a short list of complete rules. For now, this small example lets us see exactly what is supplied by people and what is determined from the data (existing examples).

Learning from examples can therefore produce a rule. There is no contradiction: The rule is the result of the learning procedure.

Let’s now run that procedure and determine which of our three candidates makes the fewest mistakes on the six historical sales.

5. Let the candidates compete

We already know that Rule 1 makes three mistakes. What happens with the other two? Table 6 below summarizes the results for all 3 rules:

Property	Number of Bedrooms	Location	Actual Sales Price	Correct Category	Rule 1: At Least 3 Bedrooms	Rule 2: At Least 4 Bedrooms	Rule 3: Prime Location
A	2	Standard	$200,000	NO	NO	NO	NO
B	3	Standard	$250,000	NO	YES	NO	NO
C	4	Standard	$280,000	NO	YES	YES	NO
D	2	Prime	$300,000	YES	NO	NO	YES
E	3	Prime	$400,000	YES	YES	NO	YES
F	4	Prime	$450,000	YES	YES	YES	YES
Number of Mistakes (out of 6)					3	3	0

Table 6: Second Round of Predictions

It is clear now: Rule 3 wins😊.

So, the selected rule is:

If the property is in a Prime location, predict YES. Otherwise, predict NO.

Note that the computer did not select this rule because it understands the appeal of a neighborhood café! It selected it because, among the permitted candidates, this rule made the fewest mistakes on the supplied examples.

And this is the important transformation:

• We began with examples of properties and their known outcomes,
• We applied a procedure for comparing possible rules,
• We ended with a selected rule that can be used on another (new) property.

CRITICAL POINT TO STRESS: If the historical outcomes had been different, another candidate could have won. That dependence on the examples is what makes this a learning process.

6. Time to introduce a few names

We have already encountered several ideas that appear throughout Machine Learning. Now we can attach their usual names (see Table 7 below).

What We Have Just Used	Equivalent Machine Learning Term
Bedrooms and location	Features: The input details available for making a prediction
The known YES or NO category for each historical property	Label (or Target): The answer used to assess a prediction
The six historical examples used to choose a rule	Training Data
The procedure that compares candidates and selects one	Learning Algorithm
The selected rule	Trained Model
Applying that selected rule to another property	Prediction (or Inference)

Table 7: Some Useful Machine Learning Terminologies

Let us pause on the difference between the learning algorithm and the learned model as this is a common point of confusion among many students:

• A learning algorithm is the procedure we used to select a rule.
  • Example: Try the three candidates and select the one with the fewest mistakes.
• A learned model is the rule we obtained:
  • Example: Predict YES for Prime locations and NO for Standard locations.

These two concepts are related, but they are not the same thing.

Also notice that the actual sale price helped us establish the correct historical category. It is not an input to our prediction. This is an important distinction to make because for a new property, we do not yet know its eventual sale price (there is no actual price yet). That is what we want to predict (output, rather than input to our model). If we supplied that price to predict its price category, our apparent brilliance would be rather less impressive than advertised😊.

OK. Here is another Machine Learning definition that I could have (maybe prematurely) added to Table 7 above:

Because we learn from examples that include known answers, this whole approach is called Supervised Learning.

The “supervision” consists of those supplied answers. It does not require Fernanda to stand beside a laptop saying, “Come on, you can do better” (knowing her, she may actually do that but that is not the standard practice in Machine Learning!).

7. Did we still program the computer or not? I am confused!

Yes. Absolutely. We did not remove human choices from the process. In particular, people decided:

• What question to answer,
• Which property details to provide,
• Which historical examples to use,
• Which candidate rules to consider,
• How to compare their performance.

Our learning procedure searches only among the possibilities we allow. In this example, it cannot suddenly invent a sophisticated calculation involving floor area, interest rates and the emotional value of a south-facing balcony. After all, we did not provide those inputs or permit that kind of calculation.

The difference is which part people specify directly (see Table 8 below).

Approach	What People Specify	What the Computer Does
Supplied decision rule	The rule that produces YES or NO	Applies the rule
Learning from examples	The candidate rules, examples and selection procedure	Uses the examples to select a rule, then applies it

Table 8: Some Useful Machine Learning

Machine Learning is still implemented through programmed procedures. The defining feature here is that the resulting predictive behavior is determined through the examples.

We will come back to this during this 24-topic series we are embarking on but for now, let’s just note that in more complex methods, the procedure may adjust thousands or millions of numerical settings rather than choose among three short rules. We will reach those methods gradually. So, let’s hold our horses and simply appreciate the fact that the essential idea is already visible through this simple example, thanks to Fernanda with a little help from her, as yet, nameless boss!

8. So, a learned model can itself be a rule?

Yes, my dear and this is worth making explicit:

If we inspect Fernanda’s final model, it looks like an ordinary instruction:

If Prime, predict YES. Otherwise, predict NO.

Someone could have written exactly that instruction without running a learning procedure (I mean someone really clever). But the final wording of the rule alone would not tell us how it was obtained.

The distinction lies in its origin: Was it supplied directly, or selected through a learning procedure using examples?

That is why “rules versus learning” is a useful opening question, but not a complete division of the world. Learning can produce rules.

In fact, some methods produce rules that are easy to read. Others produce numerical relationships that are much harder to inspect. Both can be learned from examples.

9. Does “zero mistakes” mean Fernanda has solved the problem?

Hmmm. Err! It means that the selected rule made zero mistakes on the six examples used (training data) to select it. That is certainly encouraging. And yet, it is also a very specific statement. I mean, it is specific to these 6 samples only. What would happen if we can extend the training data from 6 to, say, 8?

Let’s for example, assume that Fernanda now checks two additional historical sales that she deliberately kept out of the selection process (if I forgot to tell you before, let’s settle this now: Fernanda is quite clever. Can you see why keeping some historical examples out of the training data for further verification after a rule is selected is a particularly useful idea?). Now let’s see the 2 new examples left out of the original list (see Table 9 below):

Property	Number of Bedrooms	Location	Actual Sales Price	Correct Category	Model’s Prediction
G	2	Standard	$220,000	NO	NO
H	2	Prime	$320,000	YES	YES

Table 9: Second Round of Predictions

Well, well, well! Both predictions are correct.

These examples provide a small check of how the model performs on cases it did not use for learning. They do not establish that it will perform well on every future property. Let’s face it: Two successful checks are only two successful checks, however enthusiastically the agency’s marketing department might describe them.

Later, we will examine how to separate data for learning, choosing between approaches and evaluating the final result.

For now, just remember this:

Doing well on the examples used for learning is not enough. We also need evidence from other examples.

10. And one day another property arrives…

And today is the day when Fernanda receives a special two-bedroom property in a Prime location. And as expected by now, the model predicts YES. Loud and clear.

But eventually, the property sells for $260,000. What can I say. Sometimes you are desperate to cash in and can’t wait for the price to reach or exceed the minimum value Fernanda is expecting! The world does not revolve around Fernanda, does it?

The bottom line is that in this case the prediction was wrong.

There are legitimate reasons for this: Perhaps the property needed major repairs. Perhaps its floor area was unusually small. Perhaps market conditions had changed. We would need evidence to find out, of course.

Whatever the explanation though, our model had access only to bedrooms and location. And it could not take account of details it never received.

As for Fernanda, she now has several possible responses:

• Collect more examples to see whether the mistake reflects a wider pattern,
• Investigate whether important information is missing,
• Consider a broader set of possible models,
• Decide whether this simple method is useful enough for its intended role.

Notice what she should not do: Assume that every mistake can be fixed by shouting “more AI” at the problem!

The truth is that sometimes we need a better learning method. Sometimes we need better data. Sometimes we need a clearer question. Often times we need a combination of these 3.

And sometimes a simple prediction should remain a suggestion for a person to review and reject or approve (we call this human in the loop).

One more point: Our model does not automatically change because a sale has occurred. In this example, someone must record the new outcome and run the learning procedure again. Automatic updating would be an additional design choice. It is certainly possible but it has less to do with Artificial Intelligence or Machine Learning and more to do with the Engineering aspect of predictive work.

11. When should Fernanda simply follow a supplied rule?

Not every task requires learning from examples. Sometimes, the agency has already decided what must happen, and Fernanda’s job is to follow that instruction. For example:

Only send a client’s contact details to an outside photographer if the client has given permission.

In this case Fernanda checks whether permission has been recorded. If it has, she may send the details. Otherwise, she must not. It is really that simple.

In other words, there is nothing here for the computer to learn about whether permission should be required. That requirement comes from the agency’s policy. Even if employees sometimes ignored it in the past, their behavior would not make it acceptable and should not be considered as valid training data for extracting the correct patterns regarding the application of the policy (it may be used to identify patterns of bad or wrong behavior but that is a different question altogether). A bad habit does not become a good policy just because we have plenty of examples of it!

Predicting a property’s price category is different. The agency cannot issue an instruction that makes a house sell for at least $300,000 (although I suspect the manager would welcome that option😊). The sale price is an unknown outcome, and historical examples may help us predict it.

The key is to ask what the task needs:

• Applying an established requirement, or
• Estimating an unknown outcome?

Even for prediction, Machine Learning is not always necessary. A simple supplied rule may work well enough. We should choose an approach because it helps solve the problem, not because it sounds more impressive.

12. Often, the sensible answer is to use both

For example, Fernanda’s agency might combine:

• A learned model that suggests a property’s price category,
• A supplied rule requiring certain checks before a listing is published,
• A person who reviews unusual cases and remains responsible for the final action.

Each part performs a different role.

The prediction draws on patterns in examples. The checks enforce explicit requirements. The person deals with context and responsibility beyond the narrow prediction task.

As we progress through this series, we will return to this question:

Which part of the work should be learned from data, which part should be specified directly, and where should human judgment remain?

That is often more useful than asking whether a system is “AI-powered”, even though just mentioning anything AI fits nicely on a company website!

13. Question time

Before moving on, consider these questions:

1. Fernanda writes “If Prime, predict YES” after personally inspecting the sales table. A computer then follows it. Has the computer learned the rule?
2. A programmed procedure compares three candidate rules using historical examples and selects the one with the fewest mistakes. Where does learning occur?
3. Our selected rule gets every training example right. What should we check next?
4. Does the pattern in our small dataset prove that changing a property’s location would cause its price to rise?

14. What have we learned?

We saw how Fernanda began with a simple instruction about bedrooms. By comparing its predictions with historical sales, she discovered that it made several mistakes. We then followed her as she specified a procedure that used those historical sales examples to select a better rule candidate for prediction.

Along the way, we saw that:

• A computer can follow a rule correctly while making poor predictions,
• A learning algorithm uses examples to determine a model,
• The learned model can itself be a readable rule,
• People still choose the objective, data, available inputs and learning procedure,
• Success on training examples does not guarantee success on new cases,
• Explicit rules, learned models and human judgment can work together.

Our example is intentionally small. Real learning systems can search through far more possibilities, and they do not all learn by counting mistakes across three candidate rules.

Nor does our comparison claim that people and computers learn in the same way. Fernanda brings understanding and experience to her work. Our little procedure compares predictions and counts errors. Keeping that boundary clear helps the analogy do its job.

15. Next: Can Fernanda predict an actual price?

A YES or NO category is useful, but sooner or later a seller will ask:

That is lovely, Fernanda. But how much do you think my house will sell for?

And in this case, a simple YES is unlikely to be accepted as a price estimate.

In the next article, Fernanda will combine property details to produce a number. We will examine what each detail contributes, why a starting amount matters and how a simple calculation can be adjusted using past sales.

Question time: Suggested answers

Here are some answers to the questions posed in section 13 above:

1. No. In this description, Fernanda learned from the examples. The computer received and applied her rule.
2. Learning occurs when the procedure uses the examples to select the model. Applying the selected model afterwards is prediction.
3. Check its performance on examples not used to select it. Also consider whether the data represent the properties on which it will be used.
4. No. We have identified a predictive pattern in a small dataset. We have not established a causal effect. We will explore that distinction in a later article.

Feedback please😊.

Next article: How Does Fernanda Put a Price on a Home?
Understanding Linear Regression

Back to the AI Explained Roadmap