• It takes a little time to realize that – (these/those) things were not meant for me / were never meant for me. IMG

So we need a way to turn words into numbers.

That’s where encoding and embedding come in.

1. Step One: What is Encoding?

Think of encoding as giving each word a number tag.

Example:

You have 3 animals:

• cat
• dog
• tiger

Method 1: Label Encoding

We assign numbers:

• cat → 0
• dog → 1
• tiger → 2

But here’s the problem:

• The computer might think tiger (2) > dog (1)
• But animals don’t work like that!

Method 2: One-Hot Encoding

Instead, we use a special trick:

• cat → [1, 0, 0]
• dog → [0, 1, 0]
• tiger → [0, 0, 1]

Now:

• No number is “bigger” than another
• Everything is treated equally

But there is still a big problem…

Look at:

• cat → [1, 0, 0]
• tiger → [0, 0, 1]

To the computer, these are completely different.

But in real life:

• cat and tiger are very similar (both are cats!)

Encoding does NOT understand meaning.

2. Step Two: What is Embedding?

Now comes the smart idea.

Instead of assigning numbers manually, we let the computer learn the numbers by itself.

This is called embedding.

Imagine a Map of Words

Think of a 2D map like a game world:

• “cat” is placed here
• “tiger” is placed nearby
• “car” is placed far away

So:

• Similar things are close together
• Different things are far apart

Example Embeddings

• cat → [0.2, 0.8]
• tiger → [0.25, 0.75]
• car → [0.9, 0.1]

Now:

• cat and tiger are close → similar
• car is far → different

This is something encoding could never do.

3. How Does the Computer Learn This?

This is the magical part.

The computer learns embeddings by reading lots of sentences like:

• “The cat drinks milk”
• “The tiger hunts in the jungle”

It notices:

• cat and tiger appear in similar contexts
• so they must be related

And it slowly adjusts numbers to reflect that.

4. Simple Analogy

Encoding is like:

Giving every student in a class a roll number

• Rahul → 1
• Sita → 2

But roll numbers don’t tell you:

• who is good at math
• who likes football

Embedding is like:

Giving each student a profile based on personality

• Rahul → good at math, likes cricket
• Sita → likes art, good at music

Now you can:

• group similar students
• understand relationships

5. Why Embeddings Are Powerful

With embeddings, a computer can:

• Understand similar words

• Do analogies:

  • king − man + woman ≈ queen

• Improve tasks like:

  • translation
  • sentiment analysis
  • chatbots

6. Key Differences

7. In Deep Learning

When you use:

Embedding(vocab_size, vector_size)

You are telling the model:

“Don’t just label words… learn their meaning from data.”

8. Final Intuition

• Encoding = “Here is a number for each word.”
• Embedding = “Here is what this word means in numbers.”

9. One-Line Summary

Encoding helps the computer see words, Embedding helps the computer understand words.

Whether you are a student, a beginner, or just need a solid refresh, this guide will make K-Means crystal clear.

1. Introduction to Clustering

Before we dive into the algorithm, let’s establish the context: unsupervised machine learning.

What is Clustering?

At its heart, Clustering is about finding groups within data. It is the process of grouping similar data points together while keeping dissimilar points separated.

It is a type of unsupervised learning, which means the algorithm works on unlabeled data. We don’t have a “right answer” (a label like “cat” or “dog”); the algorithm must discover the structure, patterns, and relationships within the data on its own.

Applications of Clustering

If you need to make sense of large amounts of diverse data without previous categories, clustering is your go-to tool. It is used in:

• Customer Segmentation: Businesses analyze purchasing behavior to group customers (e.g., “high spenders,” “occasional buyers”) for targeted marketing.
• Image Compression: By grouping pixels with similar colors together and using the group center as a single color, images can be compressed significantly.
• Anomaly Detection: By clustering standard patterns of data (like bank transactions or sensor readings), data points that fall far outside of any cluster can be identified as potential fraud or system errors.

2. What is K-Means?

The Core Concept

K-Means is a specific, popular partitioning-based clustering algorithm. Its objective is to divide a dataset into \(K\) distinct, non-overlapping subsets (clusters) based on the similarity of the data points.

The Objective Function

The main goal of K-Means can be summarized as:

1. Minimize Intra-cluster distance: Make points inside the same cluster as close (similar) as possible to each other. This is measured by how close points are to their cluster center (centroid).
2. Maximize Inter-cluster distance: Keep the clusters themselves as far away (dissimilar) as possible from each other.

3. How K-Means Works (Step-by-Step)

The beauty of K-Means lies in its simple, iterative process. Here is how it functions, step-by-step:

[Image: A diagram showing the iterative steps of K-Means Clustering]

Step 1: Choose K

The first and most critical decision is deciding the number of clusters, \(K\). We will discuss how to choose this optimally later.

Step 2: Initialize Centroids

K-Means starts by placing \(K\) points into the feature space. These initial points are called centroids (the centers of the clusters).

• They can be initialized randomly by picking \(K\) random data points from the dataset.
• Better methods exist, such as K-Means++, which initializes centroids far apart from each other, speeding up convergence and avoiding bad local optima.

Step 3: Assign Points

This is the optimization part. For every single data point in your entire dataset:

1. Calculate its distance to all \(K\) centroids.
2. Assign the point to the cluster of the nearest centroid.

A point is assigned to Cluster \(j\) if the distance to Centroid \(j\) is the minimum among all centroid distances.

Step 4: Recalculate Centroids

Now that points are assigned, each centroid must find the “true center” of its newly assigned members. The position of each centroid is updated by calculating the mean (average) of all data points that are currently members of that cluster.

If a cluster has points \(P_1, P_2, \dots, P_n\), the new centroid is: \(C_{new} = \left(\frac{1}{n} \sum P_{ix}, \frac{1}{n} \sum P_{iy}\right)\)

Step 5: Repeat Until Convergence

Steps 3 and 4 are repeated (iterated) until convergence. Convergence is reached when the centroids stop changing position (or change very little). At this point, the algorithm has stabilized, and the points are fixed in their best-guess clusters.

4. A Mathematical Example Step-by-Step

concept is good, but math makes it real. Let’s walk through the exact data provided in the example above.

We will use Euclidean Distance as our measure of similarity. For two points \((x_1, y_1)\) and \((x_2, y_2)\), the distance \(d\) is: \(d = \sqrt{(x_2 - x_1)^2 + (y_2 - y_1)^2}\)

Dataset and Initial Conditions

• Data Points: \(\{A1(2,10), A2(2,5), A3(8,4), B1(5,8), B2(7,5), B3(6,4), C1(1,2), C2(4,9)\}\)
• Initial Centroids: We choose \(K=3\) and start with specific (fixed) points:
  • Centroid A: \((2, 10)\)
  • Centroid B: \((5, 8)\)
  • Centroid C: \((1, 2)\)

Iteration 1

We calculate the distance of all 8 points to these 3 initial centroids.

Table 1: Iteration 1 Distance Calculations and Assignments

Analysis of Iteration 1:

• Point A2 is closest to Centroid C (\(3.162\)), not A (\(5.000\)), so it joins Cluster C.
• Point C2 is closest to Centroid B (\(1.414\)), not C (\(7.616\)), so it joins Cluster B.

Recalculating New Centroids for Iteration 2

Since points moved, we must re-calculate the means of the new groupings:

• New Group A: \(\{A1\}\) -> Mean: (2, 10)
• New Group B: \(\{A3, B1, B2, B3, C2\}\)
  • \[x = \frac{8+5+7+6+4}{5} = 6.0\]
  • \[y = \frac{4+8+5+4+9}{5} = 6.0\]
  • New Centroid B: (6, 6)
• New Group C: \(\{A2, C1\}\)
  • \[x = \frac{2+1}{2} = 1.5\]
  • \[y = \frac{5+2}{2} = 3.5\]
  • New Centroid C: (1.5, 3.5)

Iteration 2

We use these updated centroids to recalculate all distances.

Table 2: Iteration 2 Distance Calculations and Assignments

Analysis of Iteration 2:

• C2 has shifted again! Centroid B moved far to the right (to \(x=6.0\)), and Centroid C moved slightly up. Point C2 \((4,9)\) is now much closer to Centroid A \((2,10)\) at \(2.236\) than to Centroid B \((3.606)\). It joins Cluster A.

Recalculating New Centroids for Iteration 3

• New Group A: \(\{A1, C2\}\) -> \(x=\frac{2+4}{2}=3.0\), \(y=\frac{10+9}{2}=9.5\) -> Centroid: (3.0, 9.5)
• New Group B: \(\{A3, B1, B2, B3\}\) -> \(x=\frac{8+5+7+6}{4}=6.5\), \(y=\frac{4+8+5+4}{4}=5.25\) -> Centroid: (6.5, 5.25)
• New Group C: \(\{A2, C1\}\) -> (1.5, 3.5) (unchanged)

Iteration 3

We perform the distance check one more time.

Table 3: Iteration 3 Distance Calculations and Assignments

Analysis of Iteration 3:

• Another shift! B1 \((5,8)\) was originally in Cluster B, but as the centroids shifted, it became significantly closer to the new Centroid A \((2.500)\) than to the new Centroid B \((3.132)\). It joins Cluster A.

Recalculating New Centroids for Iteration 4

• New Group A: \(\{A1, B1, C2\}\) -> \(x \approx 3.667\), \(y=9.0\) -> Centroid: (3.667, 9.0)
• New Group B: \(\{A3, B2, B3\}\) -> \(x=7.0\), \(y \approx 4.333\) -> Centroid: (7.0, 4.333)
• New Group C: \(\{A2, C1\}\) -> (1.5, 3.5) (unchanged)

Iteration 4 (Final Check)

Because we calculate new distances again for this step, but none of the cluster assignments change from Iteration 3. For example, B1 remains closest to Centroid A (\(1.667\)), not Centroid B (\(4.216\)).

Since the assignments (and thus the centroids) are exactly the same as the previous step, the algorithm is stable and converged.

Final Summary:

• Cluster A \(\{A1, B1, C2\}\) — Centroid (3.67, 9.00)
• Cluster B \(\{A3, B2, B3\}\) — Centroid (7.00, 4.33)
• Cluster C \(\{A2, C1\}\) — Centroid (1.50, 3.50)

This mathematical journey shows exactly how points move from initial fixed points to find their final clusters over 4 iterations.

5. K-Means Implementation from Scratch in Python

Now that you’ve done the math manually, let’s look at the implementation using Python. This code does exactly what we described above iterative.

import math
import pandas as pd

# 1. Define the Dataset
# We define our data points as a dictionary of coordinate tuples.
points_data = {
    'A1': (2, 10), 'A2': (2, 5), 'A3': (8, 4),
    'B1': (5, 8),  'B2': (7, 5), 'B3': (6, 4),
    'C1': (1, 2),  'C2': (4, 9)
}

# 2. Set Initial Centroids (Iteration 1)
# These are the same starting points we used in our math example.
initial_centroids = {
    'A': (2, 10),
    'B': (5, 8),
    'C': (1, 2)
}

def euclidean_distance(p1, p2):
    """Calculates the Euclidean distance between two points."""
    return math.sqrt((p1[0] - p2[0])**2 + (p1[1] - p2[1])**2)

def run_kmeans_with_tables(points, centroids, max_iterations=10):
    """Executes the K-Means algorithm and prints the state at each step."""
    current_centroids = centroids.copy()
    
    for i in range(max_iterations):
        print(f"\n--- ITERATION {i+1} ---")
        rows = []
        clusters = {name: [] for name in current_centroids}
        
        # --- Step 1: Cluster Assignment ---
        # For every point, calculate distance to current A, B, and C centers.
        for p_name, p_coords in points.items():
            dist_a = euclidean_distance(p_coords, current_centroids['A'])
            dist_b = euclidean_distance(p_coords, current_centroids['B'])
            dist_c = euclidean_distance(p_coords, current_centroids['C'])
            
            # Find the closest of the three
            dists = {'A': dist_a, 'B': dist_b, 'C': dist_c}
            assigned_cluster = min(dists, key=dists.get)
            
            # Record membership
            clusters[assigned_cluster].append(p_coords)
            
            # Prepare this information to display in our table
            rows.append({
                'Point': p_name,
                'x': p_coords[0],
                'y': p_coords[1],
                'dist to A': round(dist_a, 3),
                'dist to B': round(dist_b, 3),
                'dist to C': round(dist_c, 3),
                'Cluster': assigned_cluster
            })
        
        # --- Display the Table for this Iteration ---
        df = pd.DataFrame(rows)
        # Replicates the structure of the input slide/table.
        print(df.to_string(index=False))
        
        # --- Step 2: Calculate New Centroids ---
        new_centroids = {}
        print("\nRecalculating new centroids based on mean positions...")
        for c_name, members in clusters.items():
            if members:
                avg_x = sum(p[0] for p in members) / len(members)
                avg_y = sum(p[1] for p in members) / len(members)
                # Store new centroid, rounding for neat output
                new_centroids[c_name] = (round(avg_x, 3), round(avg_y, 3))
                print(f"Centroid {c_name} moved to: {new_centroids[c_name]}")
            else:
                # If a cluster became empty, keep the centroid where it was
                new_centroids[c_name] = current_centroids[c_name]

        # --- Step 3: Check for Convergence ---
        if new_centroids == current_centroids:
            print("\nConvergence reached! Assignments stop changing.")
            break
        
        # Prep for the next iteration
        current_centroids = new_centroids

# Run the entire process and view the math tables live!
run_kmeans_with_tables(points_data, initial_centroids)

Ouptput:

--- ITERATION 1 ---
Point  x  y  dist to A  dist to B  dist to C Cluster
   A1  2 10      0.000      3.606      8.062       A
   A2  2  5      5.000      4.243      3.162       C
   A3  8  4      8.485      5.000      7.280       B
   B1  5  8      3.606      0.000      7.211       B
   B2  7  5      7.071      3.606      6.708       B
   B3  6  4      7.211      4.123      5.385       B
   C1  1  2      8.062      7.211      0.000       C
   C2  4  9      2.236      1.414      7.616       B

Recalculating new centroids based on mean positions...
Centroid A moved to: (2.0, 10.0)
Centroid B moved to: (6.0, 6.0)
Centroid C moved to: (1.5, 3.5)

--- ITERATION 2 ---
Point  x  y  dist to A  dist to B  dist to C Cluster
   A1  2 10      0.000      5.657      6.519       A
   A2  2  5      5.000      4.123      1.581       C
   A3  8  4      8.485      2.828      6.519       B
   B1  5  8      3.606      2.236      5.701       B
   B2  7  5      7.071      1.414      5.701       B
   B3  6  4      7.211      2.000      4.528       B
   C1  1  2      8.062      6.403      1.581       C
   C2  4  9      2.236      3.606      6.042       A

Recalculating new centroids based on mean positions...
Centroid A moved to: (3.0, 9.5)
Centroid B moved to: (6.5, 5.25)
Centroid C moved to: (1.5, 3.5)

--- ITERATION 3 ---
Point  x  y  dist to A  dist to B  dist to C Cluster
   A1  2 10      1.118      6.543      6.519       A
   A2  2  5      4.610      4.507      1.581       C
   A3  8  4      7.433      1.953      6.519       B
   B1  5  8      2.500      3.132      5.701       A
   B2  7  5      6.021      0.559      5.701       B
   B3  6  4      6.265      1.346      4.528       B
   C1  1  2      7.762      6.388      1.581       C
   C2  4  9      1.118      4.507      6.042       A

Recalculating new centroids based on mean positions...
Centroid A moved to: (3.667, 9.0)
Centroid B moved to: (7.0, 4.333)
Centroid C moved to: (1.5, 3.5)

--- ITERATION 4 ---
Point  x  y  dist to A  dist to B  dist to C Cluster
   A1  2 10      1.944      7.557      6.519       A
   A2  2  5      4.333      5.044      1.581       C
   A3  8  4      6.616      1.054      6.519       B
   B1  5  8      1.666      4.177      5.701       A
   B2  7  5      5.207      0.667      5.701       B
   B3  6  4      5.518      1.054      4.528       B
   C1  1  2      7.491      6.438      1.581       C
   C2  4  9      0.333      5.548      6.042       A

Recalculating new centroids based on mean positions...
Centroid A moved to: (3.667, 9.0)
Centroid B moved to: (7.0, 4.333)
Centroid C moved to: (1.5, 3.5)

Convergence reached! Assignments stop changing.

6. Metrics and Choosing the Right K

In the previous sections, we walked through the mechanics of K-Means using a small dataset. But in the real world, data doesn’t come with labels, and it certainly doesn’t come with a manual telling you how many clusters exist. How do we know if our model is actually “good”? To answer this, we look at WCSS and the Silhouette Score.

Choosing the Right K: The Elbow Method

The most common question in clustering is: “How many clusters (\(K\)) should I use?” To solve this, we track the Within-Cluster Sum of Squares (WCSS). This metric calculates the sum of the squared distances between each data point and its assigned cluster centroid. Mathematically, it looks like this:

The Logic:

• We want WCSS to be low, as a low value means points are very close to their centers (high density).
• However, as \(K\) increases, WCSS naturally drops. If we have \(10\) points and set \(K=10\), WCSS becomes \(0\) because every point is its own centroid. This isn’t helpful—it’s overfitting.

The “Sweet Spot” Example

Think of the Elbow Method like a cost-benefit analysis. When you plot WCSS against \(K\), the graph typically looks like an arm.

1. The Drop: From \(K=1\) to \(K=3\), you usually see a massive drop in WCSS. This indicates that adding these clusters is significantly improving the model.
2. The Elbow: At a certain point (e.g., \(K=4\)), the rate of decrease slows down significantly. The graph “bends.”
3. Diminishing Returns: Past the elbow, adding more clusters only reduces WCSS by a tiny amount, but increases the complexity of your model. We choose the \(K\) value at the elbow.

Silhouette Score: A Sophisticated Metric

While the Elbow Method tells us about the “tightness” of clusters, the Silhouette Score tells us about their separation. It asks: “Is this point closer to its own group than to the next nearest group?”

For any given point \(i\), we calculate two values:

1. Cohesion (\(a(i)\)): The average distance between point \(i\) and all other points in the same cluster. We want this to be small.
2. Separation (\(b(i)\)): The average distance between point \(i\) and all points in the nearest neighboring cluster. We want this to be large.

The Silhouette Coefficient \(s(i)\) is then: \(s(i) = \frac{b(i) - a(i)}{\max(a(i), b(i))}\)

Interpreting the Score

The score ranges from -1 to +1:

• Near +1: The point is far away from neighboring clusters and right in the middle of its own. This is the goal!
• Near 0: The point is on the boundary between two clusters.
• Near -1: The point is likely in the wrong cluster.

Why use this over the Elbow Method? The Elbow Method only looks at how tight the clusters are. The Silhouette Score is more “sophisticated” because it validates if the clusters are distinct enough from one another. If your average Silhouette Score is low, it’s a sign that your clusters are overlapping too much, regardless of what the Elbow plot says.

Final Thoughts

K-Means Clustering is a cornerstone of unsupervised learning. It allows us to uncover hidden structures in data without needing prior labels. Through our journey, we’ve covered:

• The Mechanism: How centroids “pull” points toward them and shift based on the mean.
• The Math: Using Euclidean distance to drive assignments.
• The Code: Implementing the logic from scratch to understand the “brain” of the algorithm.
• The Metrics: Using the Elbow Method and Silhouette Score to ensure we aren’t just grouping data for the sake of it, but finding meaningful, distinct patterns.

You are now equipped with both the theoretical “why” and the technical “how” of K-Means. Whether you are segmenting customers or organizing image pixels, the principles remain the same: minimize the distance within, maximize the distance between.

Here are the major steps to implement Simple Linear Regression from scratch.

1. Prepare the Dataset

You start with input–output data.

• X → independent variable (feature)
• y → dependent variable (target)

The goal of the model is to learn the relationship between X and y so that it can predict the target value for new inputs.

2. Define the Model (Linear Equation)

Simple linear regression assumes a straight-line relationship between the input and output:

Where:

• w → weight (slope of the line)
• b → bias (intercept)
• x → input feature
• ŷ → predicted value

Intuition:

• w (slope) controls how steep the line is.
• b (intercept) shifts the line up or down.
• The algorithm adjusts the line so that it passes as close as possible to all data points.

Visually:

• Data → scattered points
• Model → straight line
• Training → moving the line until it fits the points best

3. Initialize Parameters

Before training begins, we start with initial parameter values:

w = 0
b = 0
α = 0.01  # learning rate

Where:

• w → weight
• b → bias
• α → learning rate (controls how large each update step is)

4. Make Predictions

Using the model equation, we compute predictions for each input:

This gives the predicted output for every data point.

5. Understand Residuals

A residual is the difference between the actual value and predicted value:

Residuals tell us how wrong each prediction is:

• Positive residual → prediction too small
• Negative residual → prediction too large

6. Calculate the Loss (Error)

We measure how far the predictions are from the true values using Mean Squared Error (MSE):

Where:

• n = number of samples
• yᵢ = actual value
• ŷᵢ = predicted value

The goal of training is to find values of w and b that minimize this loss, i.e., make residuals as small as possible.

7. Compute Gradients (Derivatives)

To reduce the loss, we calculate how sensitive the loss is to changes in the parameters.

Gradients show how the parameters should change to reduce the loss.

Why 2/n?

1. The MSE is \((\frac{1}{n}\sum (y_i - \hat{y}_i)^2)\)
2. Differentiating \(((y_i - \hat{y}_i)^2)\) gives \(2*(ŷ - y)\)
3. Multiply by \(1/n\) from the mean → \(2/n\)

This scaling ensures the gradient points in the correct direction and is proportional to the average error.

8. Update Parameters (Gradient Descent)

We update the parameters in the opposite direction of the gradients:

Intuition:

• If prediction > actual → gradient pushes line downward
• If prediction < actual → gradient pushes line upward

The learning rate α controls how big each adjustment is.

9. Repeat for Many Iterations

Each full pass through the dataset is called an epoch.

During each epoch:

1. Predict values
2. Compute loss
3. Compute gradients
4. Update parameters

Repeat for many epochs until the loss converges (stops decreasing significantly).

10. Use the Model for Prediction

After training is complete, the model has learned the optimal w and b.

For any new input x:

This allows the model to estimate target values for unseen data.

Final Algorithm (Summary)

1. Load dataset \((X, y)\)
2. Initialize w, b, and learning rate
3. Loop for a number of epochs
4. Compute predictions
5. Compute residuals and loss (MSE)
6. Compute gradients
7. Update parameters using gradient descent
8. Repeat until loss converges
9. Use the trained model for predictions

Linear Regression from Scratch: 5 Iterations of Gradient

Iteration 1

1. Dataset

We have 5 data points:

We start with initial parameters:

2. Making Predictions

The predicted values \((\hat{y})\) are calculated using the linear model:

Substituting \((w=0, b=0)\):

At this stage, our model predicts all zeros because both weight and bias are zero.

3. Calculating Errors

The error for each data point is:

This shows how far off our predictions are from the actual values.

4. Calculating Loss (Mean Squared Error)

We use Mean Squared Error (MSE):

This is the “cost” of our model with the current parameters.

5. Calculating Gradients

We compute partial derivatives to know the direction in which to adjust parameters.

A. Gradient w.r.t Weight (w)

Step by step:

Negative gradient → we need to increase (w) to reduce loss.

B. Gradient w.r.t Bias (b)

Step by step:

Negative gradient → we also need to increase (b) to reduce loss.

6. Updating Parameters

Using gradient descent with learning rate \((\eta = 0.01)\):

After one iteration, the model has learned a little: weight (w = 0.5) and bias (b = 0.14).

Iteration 2

1. Making Predictions

Using updated parameters from Iteration 1:

Using the linear model:

Substituting \((w = 0.5, b = 0.14)\):

The predictions are now slightly closer to the true values, but still far away.

2. Calculating Errors

3. Calculating Loss (MSE)

Loss has decreased from 57 → 33.23. The model is learning.

4. Calculating Gradients

A. Gradient w.r.t Weight (w)

B. Gradient w.r.t Bias (b)

5. Updating Parameters

\((\eta = 0.01)\)

After iteration 2: \((w = 0.8816, b = 0.2472)\)

Iteration 3

1. Making Predictions

Using updated parameters from Iteration 2:

Using the linear model:

Substituting \((w = 0.8816, b = 0.2472)\):

\(= [1.1288, \enspace 2.0104, \enspace 2.892, \enspace 3.7736, \enspace 4.6552]\) \(= [1.1288, \enspace 2.0104, \enspace 2.892, \enspace 3.7736, \enspace 4.6552]\)

The predictions continue moving closer to the actual values.

2. Calculating Errors

3. Calculating Loss (MSE)

Loss has decreased further, showing the model is improving.

4. Calculating Gradients

A. Gradient w.r.t Weight (w)

B. Gradient w.r.t Bias (b)

5. Updating Parameters

After iteration 3: \((w = 1.172816, b = 0.32936)\)

Iteration 4

1. Making Predictions

Using updated parameters from Iteration 3:

Using the linear model:

Substituting \((w = 1.172816, b = 0.32936)\):

The predictions continue to move closer to the actual values.

2. Calculating Errors

3. Calculating Loss (MSE)

Loss continues decreasing, indicating the model is learning the correct relationship.

4. Calculating Gradients

A. Gradient w.r.t Weight (w)

B. Gradient w.r.t Bias (b)

5. Updating Parameters

After iteration 4: \((w = 1.39503488, b = 0.39240384)\)

Iteration 5

1. Making Predictions

Using updated parameters from Iteration 4:

Using the linear model:

Substituting \((w = 1.39503488, b = 0.39240384)\):

The predictions are getting closer to the true values.

2. Calculating Errors

3. Calculating Loss (MSE)

Loss keeps decreasing as the model parameters improve.

4. Calculating Gradients

A. Gradient w.r.t Weight (w)

B. Gradient w.r.t Bias (b)

5. Updating Parameters

After iteration 5: \((w = 1.564582976, b = 0.4408536704)\)

Python Implementation

import numpy as np
import matplotlib.pyplot as plt

# Dataset
X = np.array([1, 2, 3, 4, 5], dtype=float)
y = np.array([3, 5, 7, 9, 11], dtype=float)

# Parameters
weights = 0.0
bias = 0.0
learning_rate = 0.01
epochs = 15
n = len(X)

# Store MSE for plotting
mse_list = []

# Print table header
print(f"{'Epoch':^5} | {'Init W':^8} | {'Init B':^8} | {'Pred y':^27}    | {'MSE':^10} | {'Grad W':^10} | {'Grad B':^10} | {'Upd W':^8} | {'Upd B':^8}")
print("-"*120)

plt.figure(figsize=(15,8))

for i in range(epochs):
    # Store initial weights/bias for table
    init_w = weights
    init_b = bias
    
    # Prediction
    y_pred = weights * X + bias
    
    # MSE
    mse = np.mean((y_pred - y) ** 2)
    mse_list.append(mse)
    
    # Gradients
    dw = (2/n) * np.sum((y_pred - y) * X)
    db = (2/n) * np.sum(y_pred - y)
    
    # Update parameters
    weights -= learning_rate * dw
    bias -= learning_rate * db
    
    # Round predicted values for display
    y_pred_str = "[" + ", ".join(f"{v:.2f}" for v in y_pred) + "]"
    
    # Print table row
    print(f"{i+1:^5} | {init_w:^8.4f} | {init_b:^8.4f} | {y_pred_str:^27} | {mse:^10.4f} | {dw:^10.4f} | {db:^10.4f} | {weights:^8.4f} | {bias:^8.4f}")
    
    # Plot predictions every 5 epochs + first epoch
    if (i+1) % 5 == 0 or i == 0:
        plt.plot(X, y_pred, marker='o', linestyle='--', label=f'Epoch {i+1} (MSE={mse:.2f})')

# Plot actual data
plt.scatter(X, y, color='black', label='Actual')
plt.xlabel('X')
plt.ylabel('y')
plt.title('Linear Regression Learning Over Epochs')
plt.legend()
plt.grid(True)
plt.show()

# Plot MSE over epochs
plt.figure(figsize=(15,6))
plt.plot(range(1, epochs+1), mse_list, marker='o', linestyle='-')
plt.xlabel('Epoch')
plt.ylabel('Mean Squared Error')
plt.title('MSE Decrease Over Epochs')
plt.grid(True)
plt.show()

Oputput:

Epoch |  Init W  |  Init B  |           Pred y               |    MSE     |   Grad W   |   Grad B   |  Upd W   |  Upd B  
------------------------------------------------------------------------------------------------------------------------
  1   |  0.0000  |  0.0000  | [0.00, 0.00, 0.00, 0.00, 0.00] |  57.0000   |  -50.0000  |  -14.0000  |  0.5000  |  0.1400 
  2   |  0.5000  |  0.1400  | [0.64, 1.14, 1.64, 2.14, 2.64] |  33.2296   |  -38.1600  |  -10.7200  |  0.8816  |  0.2472 
  3   |  0.8816  |  0.2472  | [1.13, 2.01, 2.89, 3.77, 4.66] |  19.3773   |  -29.1216  |  -8.2160   |  1.1728  |  0.3294 
  4   |  1.1728  |  0.3294  | [1.50, 2.67, 3.85, 5.02, 6.19] |  11.3048   |  -22.2219  |  -6.3044   |  1.3950  |  0.3924 
  5   |  1.3950  |  0.3924  | [1.79, 3.18, 4.58, 5.97, 7.37] |   6.6004   |  -16.9548  |  -4.8450   |  1.5646  |  0.4409 
  6   |  1.5646  |  0.4409  | [2.01, 3.57, 5.13, 6.70, 8.26] |   3.8589   |  -12.9341  |  -3.7308   |  1.6939  |  0.4782 
  7   |  1.6939  |  0.4782  | [2.17, 3.87, 5.56, 7.25, 8.95] |   2.2612   |  -9.8647   |  -2.8801   |  1.7926  |  0.5070 
  8   |  1.7926  |  0.5070  | [2.30, 4.09, 5.88, 7.68, 9.47] |   1.3300   |  -7.5217   |  -2.2307   |  1.8678  |  0.5293 
  9   |  1.8678  |  0.5293  | [2.40, 4.26, 6.13, 8.00, 9.87] |   0.7873   |  -5.7331   |  -1.7347   |  1.9251  |  0.5466 
 10   |  1.9251  |  0.5466  | [2.47, 4.40, 6.32, 8.25, 10.17] |   0.4709   |  -4.3677   |  -1.3561   |  1.9688  |  0.5602 
 11   |  1.9688  |  0.5602  | [2.53, 4.50, 6.47, 8.44, 10.40] |   0.2865   |  -3.3254   |  -1.0669   |  2.0020  |  0.5708 
 12   |  2.0020  |  0.5708  | [2.57, 4.57, 6.58, 8.58, 10.58] |   0.1789   |  -2.5298   |  -0.8460   |  2.0273  |  0.5793 
 13   |  2.0273  |  0.5793  | [2.61, 4.63, 6.66, 8.69, 10.72] |   0.1162   |  -1.9225   |  -0.6773   |  2.0466  |  0.5861 
 14   |  2.0466  |  0.5861  | [2.63, 4.68, 6.73, 8.77, 10.82] |   0.0795   |  -1.4589   |  -0.5484   |  2.0612  |  0.5916 
 15   |  2.0612  |  0.5916  | [2.65, 4.71, 6.78, 8.84, 10.90] |   0.0581   |  -1.1051   |  -0.4499   |  2.0722  |  0.5961 

Charts:

• Bag of Words is a feature extraction method that converts text into a vector of word counts.
• It is a simple way to represent text as numbers by counting how many times each word appears in a document.
• It ignores grammar and word order and only keeps:
  • which words appear, and
  • how often they appear.
• This is the base idea behind models like text classification, spam detection, and sentiment analysis.

Example

Suppose you have these documents:

• D1: "Blue sky"
• D2: "Bright sun today"
• D3: "Bright sun sky"
• D4: "Can see bright sun shining"

Step 1: Build the vocabulary (all unique words)

Remove stop words (common words that don’t add much meaning).

Then, build the vocabulary from the remaining words:

• Vocabulary: [blue, sky, bright, sun, today, can, see, shining]
• Sorted alphabetically: [blue, bright, can, see, shining, sky, sun, today]

Step 2: Represent each document as word counts

So:

• D1 → [1, 0, 0, 0, 0, 1, 0, 0]
• D2 → [0, 1, 0, 0, 0, 0, 1, 1]
• D3 → [0, 1, 0, 0, 0, 1, 1, 0]
• D4 → [0, 1, 1, 1, 1, 0, 1, 0]

This numeric vector is the Bag of Words representation.

Key points

• Word order is ignored:
  • “bright sun today” gives the same vector as “today bright sun”.
• Only frequency matters.
• Very easy to compute and widely used in basic NLP and ML models.

In one line

Bag of Words = vocabulary of all words + frequency of each word in a text.

Python implementation

from sklearn.feature_extraction.text import CountVectorizer

# Sample documents
documents = [
    "Blue sky",
    "Bright sun today",
    "Bright sun sky",
    "Can see bright sun shining"
]

# Create Bag of Words representation
vectorizer = CountVectorizer()
X = vectorizer.fit_transform(documents)

# Get feature names (vocabulary)
# Vocabulary should be fitted or provided beforehand
feature_names = vectorizer.get_feature_names_out() 

print("Vocabulary:", feature_names)
print("\nDocument vectors:")
print(X.toarray())

Output:

Vocabulary: ['blue' 'bright' 'can' 'see' 'shining' 'sky' 'sun' 'today']

Document vectors:
[[1 0 0 0 0 1 0 0]
 [0 1 0 0 0 0 1 1]
 [0 1 0 0 0 1 1 0]
 [0 1 1 1 1 0 1 0]]

• TF-IDF is a feature extraction method that reflects how important a word is in a document relative to a collection of documents (corpus).

• It combines two ideas:

  • Term Frequency (TF): How often a word appears in a document.
  • Inverse Document Frequency (IDF): How unique or rare the word is across all documents.
• Unlike Bag of Words, TF-IDF reduces the weight of common words and gives higher weight to rare but meaningful words.
• TF-IDF is widely used in text mining, search engines, and information retrieval.

Example

Suppose you have these documents:

• D1: "Blue sky"
• D2: "Bright sun today"
• D3: "Bright sun sky"
• D4: "Can see bright sun shining"

Step 1: Compute Term Frequency (TF)

TF is the count of a word in a document divided by the total number of words in that document.

Step 2: Compute Inverse Document Frequency (IDF)

IDF measures how unique a word is across all documents:

For our example (4 documents):

Step 3: Compute TF-IDF

Multiply TF by IDF for each word in each document:

Now each document has a TF-IDF vector, showing the importance of words, not just frequency.

Key points

• Common words like “sun” or “bright” get lower weight.
• Rare or unique words like “blue” or “can” get higher weight.
• TF-IDF is very effective for search engines, text similarity, and document clustering.

In one line

TF-IDF = Term Frequency × Inverse Document Frequency (importance of a word in a document relative to corpus).

Python implementation

from sklearn.feature_extraction.text import TfidfVectorizer

# Sample documents
documents = [
    "Blue sky",
    "Bright sun today",
    "Bright sun sky",
    "Can see bright sun shining"
]

# Create TF-IDF representation
vectorizer = TfidfVectorizer()
X = vectorizer.fit_transform(documents)

# Get feature names (vocabulary)
feature_names = vectorizer.get_feature_names_out() 

print("Vocabulary:", feature_names)
print("\nTF-IDF matrix:")
print(X.toarray())

Output (approx.):

Vocabulary: ['blue' 'bright' 'can' 'see' 'shining' 'sky' 'sun' 'today']

TF-IDF matrix:
[[0.693 0.    0.    0.    0.    0.693 0.    0.   ]
 [0.    0.096 0.    0.    0.    0.    0.096 0.462]
 [0.    0.096 0.    0.    0.    0.231 0.096 0.   ]
 [0.    0.057 0.277 0.277 0.277 0.    0.057 0.   ]]

Built-in options (default)

VS Code only includes:

• Uppercase
• Lowercase
• Title Case

You can access them via:

• Windows/Linux: Ctrl + Shift + P
• macOS: Cmd + Shift + P → type “Transform to…”

You won’t see Sentence Case there.

How to get Sentence Case in VS Code

ption 1: Install an extension (recommended)

Install “Change Case” extension.

Steps:

1. Open Extensions (Ctrl + Shift + X)
2. Search for Change Case
3. Install it
4. Select your text
5. Press Ctrl + Shift + P
6. Choose Change Case → Sentence case

This converts text like:

this is a test. this is another test.

This is a test. This is another test.

ption 2: Keyboard shortcut (after extension)

Once installed, you can bind a shortcut:

• Go to Keyboard Shortcuts
• Search for sentence case
• Assign your own key combo

If you want a quick workaround (no extension)

You can:

• Convert to lowercase
• Manually capitalize the first letter

…but this won’t handle multiple sentences correctly.

Bootstrap offers a robust collection of CSS and JavaScript components, such as its grid system and responsive UI components.

2. Bulma

Bulma is a modern CSS framework based on Flexbox known for its simplicity, clean syntax, and ease of use.

3. Foundation

Foundation Features a responsive grid system, a wide range of UI components, and advanced features like interchangeable layouts and modular architecture. Foundation is the closest alternative to Bootstrap.

4. Tailwind CSS

Tailwind CSS is a utility-first CSS framework that provides low-level utility classes to style web applications. As a utility-first framework, Tailwind abstracts from the constraints of a component-based framework, for example, Bootstrap.

5. Material UI

Material UI is a component-based CSS framework for building user interfaces in React applications. It is based on Google’s open- source design system, Material Design, and provides a rich collection of pre-built components and styles.

CSS positioning controls the placement of elements on a web page, allowing developers to override the normal document flow. The core property is position, which accepts five main values: static, relative, absolute, fixed, and sticky.

The top, right, bottom, and left properties are then used to specify the final location, but they behave differently depending on the position value.

Types of CSS Positioning

• static: This is the default value for all elements. Elements are positioned according to the normal flow of the page and the top, right, bottom, and left properties have no effect.
• relative: The element remains in the normal flow but can be offset from its original position using top, right, bottom, or left. The space the element would have occupied is preserved, and the offset does not affect the position of other elements.
• absolute: The element is removed entirely from the normal document flow. It is then positioned relative to its nearest positioned ancestor (any ancestor with a position other than static). If no such ancestor exists, it is positioned relative to the initial containing block (usually the <html> element/viewport). Other elements on the page will behave as if the absolutely positioned element does not exist.
• fixed: This is similar to absolute positioning, but the element is positioned relative to the viewport (the browser window itself) and stays in the same place even when the page is scrolled. This is commonly used for persistent navigation menus or floating buttons.
• sticky: This value toggles between relative and fixed positioning based on the user’s scroll position. It behaves as a relative element until it reaches a specified scroll threshold (e.g., top: 10px), at which point it becomes fixed to that position within its parent container.

Key Concepts

• Containing Block: The ancestor element relative to which an absolute or fixed element is positioned. For absolute positioning to work as expected within a specific container, that container must have its position set to something other than static (usually position: relative is used for this purpose).
• Z-index: Positioned elements (anything but static) can overlap other elements. The z-index property defines the stack order along the imaginary z-axis (coming out of the screen), with higher values appearing on top of lower values.