TODAY'S ISSUE
TODAYโS DAILY DOSE OF DATA SCIENCE
FireDucks with Seaborn
We have talked about FireDucks a few times before.
For starters, while Pandas is the most popular DataFrame library, it is terribly slow.
- It only uses a single CPU core.
- It has bulky DataFrames.
- It eagerly executes code, which prevents any possible optimization.
FireDucks is a highly optimized, drop-in replacement for Pandas with the same API.
You just need to change one line of code โ ๐ข๐ฆ๐ฉ๐จ๐ซ๐ญ ๐ณ๐ถ๐ฟ๐ฒ๐ฑ๐๐ฐ๐ธ๐.๐ฉ๐๐ง๐๐๐ฌ ๐๐ฌ ๐ฉ๐
Done!
One thing we havenโt covered yet is its seamless integration with third-party libraries like Seaborn.
Letโs look at a quick demo below (hereโs the Colab notebook with code).
We start by installing it:
Next, we download a sample dataset to work with.
Moving on, we load this data with Pandas and FireDucks:
Creating a correlation heatmap on the Pandas DataFrame takes 4.36 seconds:
Doing the same on FireDucks DataFrame takes over 60% less time:
Technically, Seaborn doesn't recognize a FireDucks DataFrame.
However, because of the import-hook (via %load_ext fireducks.pandas), FireDucks can be integrated seamlessly with a third-party library like Seaborn that expects a Pandas DataFrame.
And everything while accelerating the overall computation.
We have several benchmarking before and in all tests, FireDucks comes out to be the fastest:
- Here's Pandas and FireDucks head-to-head on several common tabular operations under eager evaluation.
- Here's Pandas vs. cuDF (a GPU DataFrame library):
- And here's FireDucks vs. Pandas vs. DuckDB vs. Polars:
In all cases, FireDucks is the fastest.
You can find the Colab notebook for Seaborn integration here โ
Thanks for reading!
IN CASE YOU MISSED IT
โTraditional RAG vs. HyDEโ
One critical problem with the traditional โโโโRAG systemโโโโ is that questions are not semantically similar to their answers.
As a result, several irrelevant contexts get retrieved during the retrieval step due to a higher cosine similarity than the documents actually containing the answer.
HyDE solves this.
The following visual depicts how it differs from traditional RAG and HyDE.
โWe covered this in detail in a newsletter issue published last week โโ
TRULY REPRODUCIBLE ML
Data Version Controlโ
Versioning GBs of datasets is practically impossible with GitHub because it imposes an upper limit on the file size we can push to its remote repositories.
That is why Git is best suited for versioning codebase, which is primarily composed of lightweight files.
However, ML projects are not solely driven by code.
Instead, they also involve large data files, and across experiments, these datasets can vastly vary.
To ensure proper reproducibility and experiment traceability, it is also necessary to โversion datasetsโ.
Data version control (DVC) solves this problem.
The core idea is to integrate another version controlling system with Git, specifically used for large files.
โHere's everything you need to know โโ(with implementation)โโ about building 100% reproducible ML projects โโ
THAT'S A WRAP
No-Fluff Industry ML resources to
Succeed in DS/ML roles
At the end of the day, all businesses care about impact. Thatโs it!
- Can you reduce costs?
- Drive revenue?
- Can you scale ML models?
- Predict trends before they happen?
We have discussed several other topics (with implementations) in the past that align with such topics.
Here are some of them:
- Learn sophisticated graph architectures and how to train them on graph data in this crash course.
- So many real-world NLP systems rely on pairwise context scoring. Learn scalable approaches here.
- Run large models on small devices using Quantization techniques.
- Learn how to generate prediction intervals or sets with strong statistical guarantees for increasing trust using Conformal Predictions.
- Learn how to identify causal relationships and answer business questions using causal inference in this crash course.
- Learn how to scale and implement ML model training in this practical guide.
- Learn 5 techniques with implementation to reliably test ML models in production.
- Learn how to build and implement privacy-first ML systems using Federated Learning.
- Learn 6 techniques with implementation to compress ML models.
All these resources will help you cultivate key skills that businesses and companies care about the most.
SPONSOR US
Advertise to 600k+ data professionals
Our newsletter puts your products and services directly in front of an audience that matters โ thousands of leaders, senior data scientists, machine learning engineers, data analysts, etc., around the world.
