TODAY'S ISSUE
TODAY’S DAILY DOSE OF DATA SCIENCE
Discriminative vs. Generative Models
Here’s a visual that depicts how generative and discriminative models differ:
We have seen this topic come up in several interviews, so today, let’s understand the details.
#1) Discriminative models
Discriminative models are primarily centered around discriminating between values of the outcome.
While their general notion appeals towards classification, regression models are also discriminative models.
For instance, even though linear regression does not involve a decision boundary between two classes, it is still discriminating between an outcome of “0.5” vs an outcome of “1.5” vs an outcome of “-0.6”.
Mathematically speaking, they maximize the conditional probability P(Y|X), which is read as follows: “Given an input X, maximize the probability of label Y.”
Popular examples include:
- Logistic regression
- Random Forest
- Decision Trees, etc.
#2) Generative models
Generative models primarily focus on learning the class-conditional distribution.
Thus, they maximize the joint probability P(X, Y) by learning the class-conditional distribution P(X|Y):
Popular examples include:
- Naive Bayes
- Linear Discriminant Analysis (LDA)
- Gaussian Mixture Models, etc.
We formulated Gaussian Mixture Models and implemented them from scratch here: ​Gaussian Mixture Models (GMMs)​.
Since generative models learn the underlying distribution, they can generate new samples.
For instance, consider a GAN:
Once the model has been trained, you can remove the discriminator and just use the generator to generate real-looking images:
However, this is not possible with discriminative models.
Furthermore, generative models possess discriminative properties, i.e., they can be used for classification tasks (if needed).
However, discriminative models do not possess generative properties.
Let’s consider an example to better understand them.
Imagine you are a language classification system.
There are two ways to classify languages:
- Learn every language and then classify a new language based on acquired knowledge.
- Understand some distinctive patterns in each language without truly learning the language. Once you do that, classify a new language based on the learned patterns.
The first approach is generative. This is because you learned the underlying distribution of each language.
In other words, you learned the joint distribution P(Words, Language).
Moreover, since you understand the underlying distribution, you can also generate new sentences.
The second approach is discriminative. This is because you only learned some distinctive patterns for each language:
- If so and so words appear, it is likely “Langauge A.”
- If this specific set of words appears, it is likely “Langauge B.”
- and so on.
In this case, since you learned the conditional distribution P(Language|Words), you cannot generate new sentences.
This is the difference between generative and discriminative models.
Also, the above description might persuade you that generative models are more generally useful, but it is not true.
Generative models have their own modeling complications—like requiring much more data than discriminative models.
Relate it to the language classification example again.
Imagine the amount of data you would need to learn all languages (generative approach) vs. the amount of data you would need to understand some distinctive patterns (discriminative approach).
👉 Over to you: What are some other problems while training generative models?
IN CASE YOU MISSED IT
​Function calling & MCP for LLMs
Before MCPs became mainstream (or popular like they are right now), most AI workflows relied on traditional Function Calling.
Now, MCP (Model Context Protocol) is introducing a shift in how developers structure tool access and orchestration for Agents.
Here’s a visual that explains Function calling & MCP:
​Learn more about it with visual and code in our recent issue here →
PRIVACY-PRESERVING ML
​Train models on private data with federated learning
There’s so much data on your mobile phone right now — images, text messages, etc.
And this is just about one user—you.
But applications can have millions of users. The amount of data we can train ML models on is unfathomable.
The problem?
This data is private.
So consolidating this data into a single place to train a model.
The solution?
​Federated learning​ is a smart way to address this challenge.
The core idea is to ship models to devices, train the model on the device, and retrieve the updates:
But this isn't as simple as it sounds.
1) Since the model is trained on the client side, how to reduce its size?
2) How do we aggregate different models received from the client side?
3) [IMPORTANT] Privacy-sensitive datasets are always biased with personal likings and beliefs. For instance, in an image-related task:
- Some devices may only have pet images.
- Some devices may only have car images.
- Some people may love to travel, and may primarily have travel-related images.
- How to handle such skewness in data distribution?
​Learn how to implement federated learning systems (beginner-friendly) →
THAT'S A WRAP
No-Fluff Industry ML resources to
Succeed in DS/ML roles
At the end of the day, all businesses care about impact. That’s it!
- Can you reduce costs?
- Drive revenue?
- Can you scale ML models?
- Predict trends before they happen?
We have discussed several other topics (with implementations) in the past that align with such topics.
Here are some of them:
- Learn sophisticated graph architectures and how to train them on graph data in this crash course.
- So many real-world NLP systems rely on pairwise context scoring. Learn scalable approaches here.
- Run large models on small devices using Quantization techniques.
- Learn how to generate prediction intervals or sets with strong statistical guarantees for increasing trust using Conformal Predictions.
- Learn how to identify causal relationships and answer business questions using causal inference in this crash course.
- Learn how to scale and implement ML model training in this practical guide.
- Learn 5 techniques with implementation to reliably test ML models in production.
- Learn how to build and implement privacy-first ML systems using Federated Learning.
- Learn 6 techniques with implementation to compress ML models.
All these resources will help you cultivate key skills that businesses and companies care about the most.
SPONSOR US
Advertise to 600k+ data professionals
Our newsletter puts your products and services directly in front of an audience that matters — thousands of leaders, senior data scientists, machine learning engineers, data analysts, etc., around the world.
