DIP+CV · Programming Exercise

EP07_01 — 🟢 Step-by-Step k-NN Classifier

7.18.1 EP07_01 🟢 Step-by-Step k-NN Classifier

The KNeighborsClassifier from scikit-learn, used throughout the chapter, hides behind a single call (.fit / .predict) a quite simple decision rule: for each new observation, compute the distance to all training examples, select the \(k\) closest ones, and vote by the majority class among them.

Before relying on the library, you have been tasked with implementing this rule from scratch, for a two-dimensional feature space, exactly as the chapter’s interactive decision boundary simulator does internally with each user click.

7.18.1.1 📋 Implementation Guidelines

  1. Quantity and parameter: Read the integer \(N\) (number of training examples) and the odd integer \(k\) (number of neighbors).
  2. Training examples: For each of the \(N\) examples, read three values: the \(x\) and \(y\) coordinates (real numbers) and the label \(r\) (integer, \(0\) or \(1\)).
  3. Queries: Read the integer \(Q\) (number of query points) and then the coordinates \(x_q\), \(y_q\) (real numbers) of each query.
  4. Distance: For each query, compute the Euclidean distance to all training examples: \[ d(x_q, x_i) = \sqrt{(x_q - x_i)^2 + (y_q - y_i)^2}. \]
  5. Neighbor selection: Sort the examples by increasing distance and select the first \(k\). In case of a distance tie at the boundary of the \(k\)-th neighbor, break the tie by the example read first in the input (stable reading order).
  6. Majority voting: Count the votes for each class among the \(k\) selected neighbors. If there is a tie in the voting (only possible when \(k\) is even, which should not occur per the guideline in item 1, but handle defensively), assign the class of the closest neighbor among the tied classes.
  7. Output: For each query, in input order, print the predicted class. At the end, print the total number of queries classified as class 1.

7.18.1.2 📌 Computational Constraints

  • Fixed metric: use exclusively the Euclidean distance (not the squared distance) for the sorting, although the comparison result is the same.
  • \(k\) always odd: the input guarantees \(k\) odd and \(k \le N\); still, implement the tie-breaking from item 6 for robustness.
  • Stability: when sorting by distance, preserve the relative order of examples with the same distance (stable sorting).

7.18.1.3 🧠 Theoretical Foundation

Element Role in k-NN
Feature space Set of all possible vectors \((x, y)\)
Euclidean distance Measure of similarity between observations
Small \(k\) Irregular boundary, high variance
Large \(k\) Smooth boundary, high bias
Majority voting Decision rule \(\hat y = \operatorname{mode}\{y_i : x_i \in N_k(x)\}\)

7.18.1.4 📦 Input and Output Specification (VPL)

Input:

  • Line 1: Integers \(N\) and \(k\), separated by spaces.
  • Next \(N\) lines: three values per line — \(x\), \(y\) (real numbers) and \(r\) (integer \(\in \{0,1\}\)), separated by spaces.
  • Next line: integer \(Q\).
  • Next \(Q\) lines: two values per line — \(x_q\), \(y_q\) (real numbers), separated by spaces.

Output:

  • \(Q\) lines, each with the predicted class (0 or 1) for the respective query, in input order.
  • Last line: Total class 1: X.

7.18.1.5 📌 Examples

Input Output Observation
4 3
0 0 0
1 0 0
5 5 1
6 5 1
1
1 1
0
Total class 1: 0
Query close to the class 0 cluster.
4 1
0 0 0
1 0 0
5 5 1
6 5 1
2
0.9 0.1
5.5 5.1
0
1
Total class 1: 1
With \(k=1\), each query inherits the class of its closest neighbor.
🎮 Simulator EP07_01: k-NN Classifier Step by Step Majority Voting
Adjust k and see which training examples (ordered by distance) participate in the voting for the fixed query (★ at x = 3, y = 3).
–
Figure 7.21: EP07_01 Simulator: Step-by-Step k-NN Classifier
%%writefile EP07_01.py
# Python code
Overwriting EP07_01.py
TestSuite("EP07_01.py").run()
✔️ EP07_01.cases already exists in casos/
📋 5 case(s) loaded from casos/EP07_01.cases

🔍 Testing Python: EP07_01.py
⚠️ EP07_01.py: Empty file (fewer than 3 lines). Tests skipped.