feat: clean roadmap content and sync scripts (#10181)

* refactor: clean to roadmap content

* refactor: move shared helpers into scripts/lib
This commit is contained in:
Arik Chakma
2026-07-28 05:45:17 +06:00
committed by GitHub
parent 6f745dc1ea
commit 274082fd77
12643 changed files with 440 additions and 537705 deletions
@@ -0,0 +1,9 @@
# Airflow
Apache Airflow is an open-source platform for authoring, scheduling, and monitoring data pipelines. Pipelines are defined as DAGs (Directed Acyclic Graphs) in Python, where each node is a task and edges define dependencies. Airflow provides a web UI for monitoring runs, retrying failures, and tracking execution history. It is the standard orchestration tool for production Python data pipelines.
Visit the following resources to learn more:
- [@official@Airflow 101: Building Your First Workflow](https://airflow.apache.org/docs/apache-airflow/stable/tutorial/fundamentals.html)
- [@article@Introduction to Apache Airflow](https://www.dataquest.io/blog/introduction-to-apache-airflow/)
- [@video@Airflow Tutorial For Beginners (2026)](https://www.youtube.com/watch?v=IiczxlbQb8s)
@@ -0,0 +1,3 @@
# Altair
Altair is a declarative statistical visualization library for Python based on the Vega-Lite grammar. Charts are built by binding data columns to visual channels (x, y, color, size) and specifying the mark type. Altair produces interactive charts by default and generates JSON specifications that render in notebooks and web browsers.
@@ -0,0 +1,9 @@
# APIs with requests
The `requests` library is the standard Python tool for making HTTP requests. It is used to call REST APIs that return JSON or XML data. A typical workflow involves calling `requests.get(url, params=params)`, checking the response status, and parsing the JSON body with `.json()` before loading it into a DataFrame.
Visit the following resources to learn more:
- [@article@Python Requests Module](https://www.w3schools.com/python/module_requests.asp)
- [@article@Python's Requests Library (Guide)](https://realpython.com/python-requests/)
- [@video@Master Python Requests In 15 Minutes. Call Any API](https://www.youtube.com/watch?v=Xnbef8F_Yfc)
@@ -0,0 +1,9 @@
# args & kwargs
`*args` allows a function to accept any number of positional arguments as a tuple. `**kwargs` allows any number of keyword arguments as a dictionary. They make functions flexible when the number or names of arguments are not known in advance, and are widely used in Python libraries for passing options through layers of function calls.
Visit the following resources to learn more:
- [@article@Args and Kwargs in Python – Function Calling Made Easy](https://towardsdatascience.com/args-and-kwargs-in-python-function-calling-made-easy-acfe736f988a/)
- [@article@Python args & kwargs](https://www.w3schools.com/python/python_args_kwargs.asp)
- [@video@Python *ARGS & **KWARGS are awesome!](https://www.youtube.com/watch?v=Vh__2V2tXUM)
@@ -0,0 +1,10 @@
# Arithmetic
Arithmetic operators perform mathematical calculations: `+` (addition), `-` (subtraction), `*` (multiplication), `/` (division), `//` (floor division), `%` (modulo), and `**` (exponentiation). They are used constantly for computing derived columns, normalizing values, and performing aggregations.
Visit the following resources to learn more:
- [@article@Python Arithmetic Operators](https://www.w3schools.com/python/python_operators_arithmetic.asp)
- [@article@Python Exponent: 5 Methods for Exponentiation + Applications](https://roadmap.sh/python/exponent)
- [@article@Python Division: Operators, Floor Division, and Examples](https://roadmap.sh/python/division)
- [@article@Python Modulo Operator (%): Complete Guide with Examples](https://roadmap.sh/python/modulo)
@@ -0,0 +1,9 @@
# Array Operations
NumPy supports a wide range of array operations: element-wise arithmetic, aggregation functions (`sum`, `mean`, `std`, `min`, `max`), reshaping, stacking, and splitting. These operations are vectorized, meaning they apply to the entire array at once without explicit loops, making them highly efficient.
Visit the following resources to learn more:
- [@article@NumPy Arithmetic Array Operations](https://www.programiz.com/python-programming/numpy/basic-array-operations)
- [@article@Doing Math with Arrays](https://towardsdatascience.com/introducing-numpy-part-4-doing-math-with-arrays-5e77ac595641/)
- [@video@Ultimate Guide to NumPy Arrays](https://www.youtube.com/watch?v=lLRBYKwP8GQ)
@@ -0,0 +1,9 @@
# Arrays & ndarray
The `ndarray` is NumPy's core data structure: a multi-dimensional, homogeneously typed array stored in contiguous memory. It supports element-wise operations, broadcasting, and vectorized computation far faster than Python lists. Understanding ndarray is fundamental to working efficiently with numerical data in Python.
Visit the following resources to learn more:
- [@official@NumPy: the absolute basics for beginners](https://numpy.org/doc/stable/user/absolute_beginners.html)
- [@official@numpy.array](https://numpy.org/doc/stable/reference/generated/numpy.array.html)
- [@video@Ultimate Guide to NumPy Arrays](https://www.youtube.com/watch?v=lLRBYKwP8GQ&pp=ygUMbnVtcHkgYXJyYXlz)
@@ -0,0 +1,9 @@
# BeautifulSoup
BeautifulSoup is a Python library for parsing HTML and XML documents. It provides methods for navigating the document tree, searching for elements by tag, class, or attribute, and extracting text and links. BeautifulSoup is used for web scraping when the target website does not provide an API.
Visit the following resources to learn more:
- [@official@Beautiful Soup Documentation](https://beautiful-soup-4.readthedocs.io/en/latest/)
- [@article@Beautiful Soup: Build a Web Scraper With Python](https://realpython.com/beautiful-soup-web-scraper-python/)
- [@video@Web Scraping with Python - Beautiful Soup Crash Course](https://www.youtube.com/watch?v=XVv6mJpFOb0)
@@ -0,0 +1,7 @@
# Big Data Tools
Big data tools process datasets too large to fit in a single machine's memory using distributed or out-of-core computation. Dask and PySpark are the primary Python tools for scaling beyond what Pandas and NumPy can handle. They provide familiar DataFrame-like APIs while distributing computation across cores or clusters.
Visit the following resources to learn more:
- [@article@4 Types of Big Data Technologies (+ Management Tools)](https://www.coursera.org/articles/big-data-technologies)
@@ -0,0 +1,8 @@
# Booleans
Booleans (`bool`) have two values: `True` and `False`. They are the result of comparison and logical operations and are used to control flow and filter data.
Visit the following resources to learn more:
- [@official@Built-in Types](https://docs.python.org/3/library/stdtypes.html)
- [@article@Python Booleans: Use Truth Values in Your Code](https://realpython.com/ref/builtin-types/str/)
@@ -0,0 +1,9 @@
# Boxplot
A box plot displays the five-number summary of a variable: minimum, Q1, median, Q3, and maximum. The box covers the IQR, a line marks the median, and whiskers extend to the data range. Points beyond the whiskers are plotted individually as potential outliers. Box plots are effective for comparing distributions across groups.
Visit the following resources to learn more:
- [@official@matplotlib.pyplot.boxplot](https://matplotlib.org/stable/api/_as_gen/matplotlib.pyplot.boxplot.html)
- [@article@Create and customize boxplots with Python’s Matplotlib](https://towardsdatascience.com/create-and-customize-boxplots-with-pythons-matplotlib-to-get-lots-of-insights-from-your-data-d561c9883643/)
- [@video@Seaborn boxplot | Box plot explanation, box plot demo, and how to make a box plot in Python seaborn](https://www.youtube.com/watch?v=Vo-bfTqEFQk)
@@ -0,0 +1,8 @@
# Built-in Functions
Python's built-in functions are available without any imports and cover common operations: `len()`, `sum()`, `min()`, `max()`, `sorted()`, `enumerate()`, `zip()`, `map()`, `filter()`, and others. These functions simplify common tasks and are used constantly alongside data analysis libraries.
Visit the following resources to learn more:
- [@official@Built-in Functions](https://docs.python.org/3/library/functions.html)
- [@video@All 71 built-in Python functions](https://www.youtube.com/watch?v=7Qu_KXc7xSI)
@@ -0,0 +1,8 @@
# Casting Types
Casting types in Pandas converts a column from one data type to another using `.astype()`. Common conversions include converting string columns to numeric with `pd.to_numeric()`, converting to datetime with `pd.to_datetime()`, and converting integers to categories. Correct data types are required for accurate calculations and efficient memory use.
Visit the following resources to learn more:
- [@official@pandas.DataFrame.astype](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.astype.html)
- [@article@How To Change Column Type in Pandas DataFrames](https://towardsdatascience.com/how-to-change-column-type-in-pandas-dataframes-d2a5548888f8/)
@@ -0,0 +1,7 @@
# Categorical Plots
Seaborn's categorical plots visualize the relationship between a numeric variable and one or more categorical variables. `sns.boxplot()`, `sns.violinplot()`, `sns.barplot()`, `sns.stripplot()`, and `sns.countplot()` cover the main patterns. They are used to compare distributions or averages across groups.
Visit the following resources to learn more:
- [@official@Visualizing categorical data](https://seaborn.pydata.org/tutorial/categorical.html)
@@ -0,0 +1,8 @@
# Comparison
Comparison operators evaluate the relationship between two values and return a boolean. They include `==`, `!=`, `<`, `>`, `<=`, and `>=`. They form the building blocks of filtering conditions applied to DataFrames and arrays.
Visit the following resources to learn more:
- [@article@Python Comparison Operators](https://www.w3schools.com/python/gloss_python_comparison_operators.asp)
- [@video@Comparison Operators in Python](https://www.youtube.com/watch?v=6ZQtBK-dM9c)
@@ -0,0 +1,9 @@
# conda
conda is an open-source package and environment manager included with the Anaconda and Miniconda distributions. It manages both Python packages and non-Python dependencies, making it well suited for scientific computing. conda environments isolate project dependencies and can be exported to `environment.yml` for reproducibility.
Visit the following resources to learn more:
- [@official@Anaconda](https://www.anaconda.com/)
- [@video@Anaconda (Conda) for Python - What & Why?](https://www.youtube.com/watch?v=23aQdrS58e0)
- [@video@Master the basics of Conda environments in Python](https://www.youtube.com/watch?v=1VVCd0eSkYc)
@@ -0,0 +1,9 @@
# Conditionals
Conditionals execute different code paths based on whether a condition is true. Python uses `if`, `elif`, and `else` for this. Python 3.10 introduced `match/case`, a structural pattern matching statement that cleanly handles multiple specific value checks as an alternative to long `elif` chains. They appear in custom functions applied to DataFrames, in filtering logic, and in branching pipeline code.
Visit the following resources to learn more:
- [@article@Conditional Statements in Python](https://realpython.com/python-conditional-statements/)
- [@article@Python Switch Statement 101: Match-case and alternatives](https://roadmap.sh/python/switch)
- [@video@Control Flow in Python - If Elif Else Statements](https://www.youtube.com/watch?v=Zp5MuPOtsSY)
@@ -0,0 +1,8 @@
# Correlation & Covariance
Correlation measures the strength and direction of the linear relationship between two variables, scaled between −1 and +1. Covariance measures the same relationship but is not normalized, making it harder to interpret across variables with different scales. `df.corr()` and `df.cov()` compute these matrices in Pandas, and heatmaps are used to visualize them.
Visit the following resources to learn more:
- [@article@Statistics in Python – Understanding Variance, Covariance, and Correlation](https://towardsdatascience.com/statistics-in-python-understanding-variance-covariance-and-correlation-4729b528db01/)
- [@video@Covariance and Correlation in Probability](https://www.youtube.com/watch?v=QKPdk57y7Ck)
@@ -0,0 +1,8 @@
# Correlation Matrix
A correlation matrix shows the pairwise correlation coefficients between all numeric columns in a dataset. It is computed with `df.corr()` and typically visualized as a heatmap using Seaborn. It is a key EDA tool for identifying which features are strongly related, which helps with feature selection and multicollinearity detection.
Visit the following resources to learn more:
- [@article@Correlation Matrix, Demystified](https://medium.com/data-science/correlation-matrix-demystified-3ae3405c86c1)
- [@article@Data Science - Statistics Correlation Matrix](https://www.w3schools.com/datascience/ds_stat_correlation_matrix.asp)
@@ -0,0 +1,9 @@
# Cross-tabulation
Cross-tabulation (crosstab) counts the frequency of combinations of values across two or more categorical variables. `pd.crosstab()` produces a table of frequencies or proportions. It is used to examine relationships between categorical variables, such as how customer segments differ across product categories.
Visit the following resources to learn more:
- [@official@pandas.crosstab](https://pandas.pydata.org/docs/reference/api/pandas.crosstab.html)
- [@article@The Power of Crosstab Function in Pandas](https://medium.com/geekculture/the-power-of-crosstab-function-in-pandas-for-data-analysis-and-visualization-6c085c269fcd)
- [@video@Python Pandas Tutorial 13. Crosstab](https://www.youtube.com/watch?v=I_kUj-MfYys)
@@ -0,0 +1,9 @@
# CSV
CSV (Comma-Separated Values) is the most common format for tabular data exchange. `pd.read_csv()` loads a CSV file into a DataFrame and accepts dozens of parameters for handling separators, missing values, date parsing, and data types. CSV files are human-readable but lack type information, so columns often need type correction after loading.
Visit the following resources to learn more:
- [@official@pandas.read_csv](https://pandas.pydata.org/docs/reference/api/pandas.read_csv.html)
- [@article@Pandas Read CSV](https://www.w3schools.com/python/pandas/pandas_csv.asp)
- [@video@How to Read a CSV file into a Pandas DataFrame](https://www.youtube.com/watch?v=4YI9z1qUpew)
@@ -0,0 +1,10 @@
# Customizing Plots
Matplotlib allows extensive customization: titles with `set_title()`, axis labels with `set_xlabel()` / `set_ylabel()`, tick formatting, color palettes, line styles, font sizes, legends, and annotations. Customizing plots ensures they communicate clearly and meet the standards required for reports and presentations.
Visit the following resources to learn more:
- [@course@Advanced Matplotlib: Design & Customize Visualizations](https://www.coursera.org/learn/advanced-matplotlib-design-customize-visualizations)
- [@article@Customizing Plots](https://apxml.com/courses/intermediate-python-programming-ml/chapter-4-data-visualization-matplotlib-seaborn/matplotlib-customization)
- [@article@3 Tricks to Prettify Matplotlib Plots](https://towardsdatascience.com/3-tricks-to-prettify-matplotlib-plots-d0a73b861c09/)
- [@video@Matplotlib customization is easy! 🎨](https://www.youtube.com/watch?v=hunq_UOdmoo)
@@ -0,0 +1,8 @@
# Customizing Plots
Seaborn plots are customized through function parameters (palette, hue, size, style) and by accessing the underlying Matplotlib axes after creation. `sns.set_theme()` and `sns.set_style()` change the global appearance. Seaborn's theming system makes it easy to produce clean, publication-ready charts with minimal code.
Visit the following resources to learn more:
- [@official@Controlling figure aesthetics](https://seaborn.pydata.org/tutorial/aesthetics.html)
- [@article@5 Ways to Transform Your Seaborn Data Visualisations](https://towardsdatascience.com/5-ways-to-transform-your-seaborn-data-visualisations-1ed2cb484e38/)
@@ -0,0 +1,8 @@
# Dash
Dash is a Python framework for building analytical web applications, developed by Plotly. It combines Plotly charts with reactive UI components and runs as a Flask web server. Dash provides more control and customization than Streamlit and is better suited for production-grade dashboards with complex interactivity.
Visit the following resources to learn more:
- [@official@Dash in 20 Minutes](https://dash.plotly.com/tutorial)
- [@video@Introduction to Dash Plotly - Data Visualization in Python](https://www.youtube.com/watch?v=hSPmj7mK6ng)
@@ -0,0 +1,8 @@
# Dashboards
Dashboards combine multiple visualizations and controls into a single interface for monitoring and exploring data. Python provides several tools for building data dashboards that can be shared as web applications without requiring frontend development skills. The main options are Streamlit, Dash, and connecting to BI tools like Power BI and Tableau.
Visit the following resources to learn more:
- [@roadmap@Visit the Dedicated BI Analyst Roadmap](https://roadmap.sh/bi-analyst)
- [@article@What is a dashboard? A complete overview](https://www.tableau.com/dashboard/what-is-dashboard)
@@ -0,0 +1,8 @@
# Dask
Dask is a parallel computing library for Python that scales Pandas, NumPy, and Scikit-learn to larger-than-memory datasets. It breaks data into chunks and builds a task graph that is executed lazily. Dask DataFrames mirror the Pandas API, making it easy to adapt existing code for larger datasets without switching ecosystems.
Visit the following resources to learn more:
- [@official@Dask Tutorial](https://tutorial.dask.org/)
- [@video@Intro to Dask](https://www.youtube.com/watch?v=z18qjLu-Mw4&list=PLeDTMczuyDQ8S73cdc0PrnTO80kfzpgz2)
@@ -0,0 +1,9 @@
# Data Cleaning
Data cleaning identifies and resolves quality issues in raw data so it is accurate and consistent enough for analysis. In Python, cleaning is done primarily with Pandas and string processing tools. Common tasks include handling missing values, fixing data types, standardizing text, removing duplicates, and detecting outliers.
Visit the following resources to learn more:
- [@article@Complete Guide to Data Cleaning in Python](https://www.dataquest.io/guide/data-cleaning-in-python-tutorial/)
- [@article@Pandas Data Cleaning](https://www.w3schools.com/python/pandas/pandas_cleaning.asp)
- [@video@Data Cleaning in Pandas | Python Pandas Tutorials](https://www.youtube.com/watch?v=bDhvCp3_lYw)
@@ -0,0 +1,8 @@
# Data Pipelines
Data pipelines automate the sequence of steps that move and transform data from sources to destinations. They encapsulate the full workflow — ingestion, cleaning, transformation, and output — as code. Orchestration tools like Airflow schedule and monitor these pipelines in production, ensuring they run reliably and their failures are caught and handled.
Visit the following resources to learn more:
- [@article@What is a data pipeline?](https://www.ibm.com/think/topics/data-pipeline)
- [@video@Data Pipelines Explained](https://www.youtube.com/watch?v=6kEGUCrBEU0)
@@ -0,0 +1,8 @@
# Defining Functions
User-defined functions are created with the `def` keyword and encapsulate reusable logic. A function takes parameters, executes a body, and returns a value with `return`. Writing well-scoped functions makes analysis code modular, testable, and easier to apply across a dataset using Pandas' `apply()` method.
Visit the following resources to learn more:
- [@article@Python Return Multiple Values: 4 Methods & Examples](https://roadmap.sh/python/return-multiple-values)
- [@video@Python Functions - Visually Explained](https://www.youtube.com/watch?v=KW6qncswzHw)
@@ -0,0 +1,10 @@
# Dictionaries
Dictionaries store key-value pairs and provide fast lookup by key. They are used extensively in Python for mapping labels to values, building frequency counts, and configuring function arguments.
Visit the following resources to learn more:
- [@official@Dictionaries](https://docs.python.org/3/tutorial/datastructures.html#dictionaries)
- [@article@Hashmaps in Python: Master Implementation and Use Cases](https://roadmap.sh/python/hashmap)
- [@article@Python KeyError Exceptions: Causes and Fixes](https://roadmap.sh/python/keyerror)
- [@video@Python dictionaries are easy 📙](https://www.youtube.com/watch?v=MZZSMaEAC2g)
@@ -0,0 +1,7 @@
# Distribution plots
Seaborn's distribution plots visualize the distribution of one or two variables. `sns.histplot()` and `sns.kdeplot()` show the shape of a single variable's distribution. `sns.displot()` combines both. `sns.pairplot()` shows pairwise distributions and relationships across all numeric columns in a DataFrame.
Visit the following resources to learn more:
- [@official@Visualizing distributions of data](https://seaborn.pydata.org/tutorial/distributions.html)
@@ -0,0 +1,9 @@
# Dropping vs. Imputing
When handling missing values, dropping removes rows or columns with `dropna()`, while imputing fills them with a substitute value using `fillna()` or `SimpleImputer` from Scikit-learn. Dropping is appropriate when missing data is rare or random. Imputing is preferred when data is valuable or missing systematically, using the mean, median, mode, or a predicted value.
Visit the following resources to learn more:
- [@official@pandas.DataFrame.fillna](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.fillna.html)
- [@article@Pandas DataFrame fillna() Method](https://www.w3schools.com/python/pandas/ref_df_fillna.asp)
- [@video@Python Pandas Tutorial 5: Handle Missing Data: fillna, dropna, interpolate](https://www.youtube.com/watch?v=EaGbS7eWSs0)
@@ -0,0 +1,9 @@
# DuckDB
DuckDB is an in-process analytical database designed for fast SQL queries on large datasets stored as files or in memory. It can query CSV, Parquet, and Pandas DataFrames directly with SQL syntax. DuckDB is increasingly used in data analysis workflows as a fast alternative to loading data into a full database system.
Visit the following resources to learn more:
- [@official@DuckDB Docs](https://duckdb.org/docs/current/clients/python/overview)
- [@article@Introducing DuckDB](https://realpython.com/python-duckdb/)
- [@video@Try DuckDB for SQL on Pandas](https://www.youtube.com/watch?v=8SYQtpSk_OI)
@@ -0,0 +1,8 @@
# Encoding Categories
Categorical encoding converts text category labels into numerical values that machine learning algorithms can process. Common approaches include label encoding (assigning each category an integer), one-hot encoding (creating binary columns for each category with `pd.get_dummies()`), and ordinal encoding for ordered categories.
Visit the following resources to learn more:
- [@official@Categorical data](https://pandas.pydata.org/docs/user_guide/categorical.html)
- [@article@Encoding Categorical Variables: One-hot vs Dummy Encoding](https://towardsdatascience.com/encoding-categorical-variables-one-hot-vs-dummy-encoding-6d5b9c46e2db/)
@@ -0,0 +1,7 @@
# Environment Setup
Setting up a proper Python environment for data analysis involves choosing a package manager, managing dependencies, and selecting a development environment. A well-configured environment ensures reproducibility and avoids package conflicts. The main tools are pip and conda for package management, and virtual environments for isolation.
Visit the following resources to learn more:
- [@video@Setting Up A Python Environment for Data Analysis and Machine Learning](https://www.youtube.com/watch?v=NDFMa5FSQuI)
@@ -0,0 +1,8 @@
# Excel
Excel files (`.xlsx`, `.xls`) are loaded with `pd.read_excel()`, which supports selecting sheets, skipping rows, and reading specific columns. The `openpyxl` library is required for `.xlsx` files. Excel is common in business environments, and analysts frequently need to read and write it as part of reporting workflows.
Visit the following resources to learn more:
- [@official@pandas.read_excel](https://pandas.pydata.org/docs/reference/api/pandas.read_excel.html)
- [@article@Pandas read_excel: Reading Excel Files in Python](https://www.digitalocean.com/community/tutorials/pandas-read_excel-reading-excel-file-in-python)
@@ -0,0 +1,9 @@
# Exploratory Data Analysis
Exploratory Data Analysis (EDA) is the process of examining a dataset to understand its structure, distributions, and relationships before formal modeling. It combines descriptive statistics and visualizations to surface patterns, anomalies, and hypotheses. EDA guides subsequent cleaning decisions and model choices.
Visit the following resources to learn more:
- [@article@Exploratory Statistical Data Analysis with a Real Dataset using Pandas](https://medium.com/data-science/exploratory-statistical-data-analysis-with-a-real-dataset-using-pandas-208007798b92)
- [@article@Pandas Profiling – Easy Exploratory Data Analysis in Python](https://towardsdatascience.com/pandas-profiling-easy-exploratory-data-analysis-in-python-65d6d0e23650/)
- [@video@Exploratory Data Analysis in Pandas | Python Pandas Tutorials](https://www.youtube.com/watch?v=Liv6eeb1VfE)
@@ -0,0 +1,10 @@
# Filtering & Querying
Filtering in Pandas selects rows that meet specified conditions. Boolean masks, the `.query()` method, and `.isin()` are common approaches. Multiple conditions can be combined with `&` (and) and `|` (or), and the `.query()` method allows SQL-like string syntax for readable filtering expressions.
Visit the following resources to learn more:
- [@official@pandas.DataFrame.query](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.query.htmlme.query)
- [@official@How do I select a subset of a DataFrame?](https://pandas.pydata.org/docs/getting_started/intro_tutorials/03_subset_data.html)
- [@article@10 Elegant Ways to Filter Pandas DataFrames](https://towardsdatascience.com/stop-writing-messy-boolean-masks-10-elegant-ways-to-filter-pandas-dataframes/)
- [@video@Filtering Columns and Rows in Pandas](https://www.youtube.com/watch?v=kB7FV-ijdqE)
@@ -0,0 +1,8 @@
# Floats
Floats (`float`) represent real numbers with a decimal point. Most numerical data in analysis involves floats, including prices, measurements, and probabilities. Floating-point arithmetic has precision limitations that can cause small rounding errors, which are important to be aware of in financial and scientific calculations.
Visit the following resources to learn more:
- [@official@Built-in Types](https://docs.python.org/3/library/stdtypes.html)
- [@article@float](https://realpython.com/ref/builtin-types/float/)
@@ -0,0 +1,9 @@
# Forward / Backward Fill
Forward fill (`ffill`) propagates the last valid value forward to fill subsequent missing entries. Backward fill (`bfill`) does the reverse, filling from the next valid value. Both are commonly used for time series data where missing values represent periods where the previous or next observation is the best estimate.
Visit the following resources to learn more:
- [@official@pandas.DataFrame.ffill](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.ffill.html)
- [@official@pandas.DataFrame.bfill](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.bfill.html)
- [@article@How to Fill Missing Data with Pandas](https://towardsdatascience.com/how-to-fill-missing-data-with-pandas-8cb875362a0d/)
@@ -0,0 +1,9 @@
# Functions & Methods
Functions are reusable blocks of code that take inputs, perform operations, and return outputs. They encapsulate cleaning steps, transformations, and calculations that need to be applied consistently. Python supports built-in functions, user-defined functions, and anonymous lambda functions.
Visit the following resources to learn more:
- [@article@Python Functions](https://www.w3schools.com/python/python_functions.asp)
- [@article@Python Methods, Functions, & Libraries](https://mode.com/python-tutorial/python-methods-functions-and-libraries)
- [@video@Functions in Python are easy](https://www.youtube.com/watch?v=89cGQjB5R4M)
@@ -0,0 +1,9 @@
# GeoPandas
GeoPandas is an open-source library that extends the capabilities of pandas by allowing spatial operations on geometric types. It simplifies working with geospatial data by enabling the use of familiar data structures like GeoSeries and GeoDataFrame, which store and manipulate vector-based geographic information. Through its integration with libraries like Shapely and PyGEOS, it allows you to perform complex geometric operations such as spatial joins, projections, and distance calculations using straightforward syntax.
Visit the following resources to learn more:
- [@official@GeoPandas Docs](https://geopandas.org/en/stable/docs.html)
- [@official@Introduction to GeoPandas](https://geopandas.org/en/stable/getting_started/introduction.html)
- [@video@Geospatial Python - Full Course for Beginners with Geopandas](https://www.youtube.com/watch?v=0mWgVVH_dos)
@@ -0,0 +1,9 @@
# Geospatial Analysis
Geospatial analysis is the process of gathering, manipulating, and mapping data that is tied to specific geographic locations on the Earth's surface. It involves using tools and libraries to perform spatial operations, such as calculating distances between coordinates, analyzing geographic patterns, or visualizing datasets on interactive maps. By integrating coordinate systems and geometric shapes into data workflows, this analysis allows for a deeper understanding of how physical location influences various trends and phenomena.
Visit the following resources to learn more:
- [@article@Introduction to Python for Geographic Data Analysis](https://pythongis.org/)
- [@article@Spatial analysis with Python](https://spatial-analytics.readthedocs.io/en/develop/lessons/L1/intro-to-python-geostack.html)
- [@video@GeoSpatial Analysis With Python For Beginners](https://www.youtube.com/watch?v=IRJC67zm6nk)
@@ -0,0 +1,9 @@
# Google Colab
Google Colab is a cloud-hosted Jupyter notebook environment from Google. It requires no local setup and provides free access to GPUs and TPUs, making it popular for machine learning work. Colab notebooks are stored in Google Drive and can be shared like any other document.
Visit the following resources to learn more:
- [@official@Google Colab](https://colab.research.google.com/)
- [@official@Python Basics in Colab](https://colab.research.google.com/github/data-psl/lectures2020/blob/master/notebooks/01_python_basics.ipynb)
- [@video@Google Colab Tutorial for Beginners | Get Started with Google Colab](https://www.youtube.com/watch?v=RLYoEyIHL6A)
@@ -0,0 +1,9 @@
# Groupby & Aggregation
`groupby()` splits a DataFrame into groups based on one or more columns, applies a function to each group, and combines the results. Common aggregation functions include `sum()`, `mean()`, `count()`, `min()`, `max()`, and custom functions via `agg()`. This split-apply-combine pattern is one of the most powerful features of Pandas.
Visit the following resources to learn more:
- [@official@pandas.DataFrame.groupby](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.groupby.html)
- [@article@All About Pandas Groupby Explained with 25 Examples](https://towardsdatascience.com/all-about-pandas-groupby-explained-with-25-examples-494e04a8ef56/)
- [@video@Group By and Aggregate Functions in Pandas |](https://www.youtube.com/watch?v=VRmXto2YA2I)
@@ -0,0 +1,9 @@
# Heatmaps
`sns.heatmap()` visualizes matrix-style data using color intensity. It is most commonly used to display correlation matrices and pivot tables. Color maps, annotations, and masking options allow the heatmap to be customized for readability. Heatmaps are an effective way to show patterns across two categorical dimensions.
Visit the following resources to learn more:
- [@official@seaborn.heatmap](https://seaborn.pydata.org/generated/seaborn.heatmap.html)
- [@article@Data Visualization with Seaborn: Heatmaps](https://medium.com/@1zeyneper/data-visualization-with-seaborn-heatmaps-58abfadd79d5)
- [@video@Seaborn heatmap | How to make a heatmap in Python Seaborn and adjust the heatmap style](https://www.youtube.com/watch?v=0U9cs2V-Mqc)
@@ -0,0 +1,9 @@
# Histogram
A histogram groups numeric values into bins and shows the count or frequency of each bin as a bar. It is the primary tool for visualizing the distribution of a single variable: its shape, center, spread, and whether it is skewed or has multiple peaks. `df['col'].hist()` and Matplotlib's `plt.hist()` are the standard ways to create one.
Visit the following resources to learn more:
- [@official@matplotlib.pyplot.hist](https://matplotlib.org/stable/api/_as_gen/matplotlib.pyplot.hist.html)
- [@article@Matplotlib Histograms](https://www.w3schools.com/PYTHON/matplotlib_histograms.asp)
- [@video@Matplotlib histograms in 6 minutes! 🔔](https://www.youtube.com/watch?v=2E6fDoz7LuU)
@@ -0,0 +1,10 @@
# Indexing & Slicing
Pandas provides two primary indexing systems: `.loc[]` for label-based selection and `.iloc[]` for position-based selection. Both work on rows, columns, or both simultaneously. Boolean indexing with a condition (e.g., `df[df['age'] > 30]`) is the most common way to filter rows.
Visit the following resources to learn more:
- [@official@pandas.DataFrame.loc](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.loc.html)
- [@official@pandas.DataFrame.iloc](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.iloc.html)
- [@article@Iloc vs Loc in Pandas: A Guide with Examples](https://www.analyticsvidhya.com/blog/2026/03/iloc-vs-loc-in-pandas/)
- [@video@Pandas loc and iloc](https://www.youtube.com/watch?v=naRQyRZrXCE)
@@ -0,0 +1,10 @@
# Array Operations
NumPy supports a wide range of array operations: element-wise arithmetic, aggregation functions (`sum`, `mean`, `std`, `min`, `max`), reshaping, stacking, and splitting. These operations are vectorized, meaning they apply to the entire array at once without explicit loops, making them highly efficient.
Visit the following resources to learn more:
- [@official@Indexing on ndarrays](https://numpy.org/doc/stable/user/basics.indexing.html)
- [@article@NumPy Array Indexing](https://www.w3schools.com/python/numpy/numpy_array_indexing.asp)
- [@article@NumPy Array Slicing](https://www.w3schools.com/python/numpy/numpy_array_slicing.asp)
- [@article@Indexing Arrays](https://towardsdatascience.com/introducing-numpy-part-2-indexing-arrays-5b381b90d1d0/)
@@ -0,0 +1,9 @@
# Integers
Integers (`int`) are whole numbers without a decimal point. They appear as counts, indices, IDs, and categorical encodings. Python integers have arbitrary precision, meaning they do not overflow like integers in lower-level languages.
Visit the following resources to learn more:
- [@official@Built-in Types](https://docs.python.org/3/library/stdtypes.html)
- [@article@int](https://realpython.com/ref/builtin-types/int/)
- [@article@"Python Max Int: Understanding Arbitrary Precision Integers "](https://roadmap.sh/python/max-int)
@@ -0,0 +1,8 @@
# Interactive Visualization
Interactive visualizations allow users to explore data by hovering, zooming, panning, and filtering directly in the chart. They are more engaging than static plots for dashboards and reports where the audience needs to examine specific data points. Python's main interactive visualization libraries are Plotly and Altair.
Visit the following resources to learn more:
- [@article@Top 10 Python Data Visualization Libraries](https://reflex.dev/blog/top-10-data-visualization-libraries/)
- [@article@4 Key Players in Python Data Visualization Ecosystem: Matplotlib, Seaborn, Altair, and Plotly](https://towardsdatascience.com/4-key-players-in-python-data-visualization-ecosystem-matplotlib-seaborn-altair-and-plotly-23ae37a68227/)
@@ -0,0 +1,10 @@
# Introduction
Python is the dominant language for data analysis due to its readable syntax, rich ecosystem of libraries, and strong community support. Getting started requires understanding the core language features: operators, data types, control flow, and data structures. These fundamentals apply directly to every data manipulation and analysis task that follows.
Visit the following resources to learn more:
- [@book@Python for Data Analysis](https://www.lkhibra.ma/books/Python-for-Data-Analysis.pdf)
- [@article@What Does a Data Analyst Do?](https://roadmap.sh/data-analyst/what-does-a-data-analyst-do)
- [@article@How Long Does It Really Take To Learn Python? My Experience](https://roadmap.sh/python/how-long-does-it-take-to-learn)
- [@video@Python for Data Analytics - Full Course for Beginners](https://www.youtube.com/watch?v=wUSDVGivd-8)
@@ -0,0 +1,9 @@
# IQR
The Interquartile Range (IQR) is the difference between the 75th percentile (Q3) and 25th percentile (Q1) of a dataset. Outliers are commonly defined as values below Q1 − 1.5×IQR or above Q3 + 1.5×IQR. The IQR method is robust to extreme values and is the basis for the box plot's whiskers.
Visit the following resources to learn more:
- [@article@Using Pandas IQR: 3 Essential Steps](https://medium.com/@heyamit10/using-pandas-iqr-3-essential-steps-f5cf73a390ba)
- [@article@How to detect outliers using IQR and Boxplots?](https://machinelearningplus.com/machine-learning/how-to-detect-outliers-using-iqr-and-boxplots/)
- [@video@Outlier detection and removal using IQR](https://www.youtube.com/watch?v=A3gClkblXK8)
@@ -0,0 +1,10 @@
# isnull, isna
`isnull()` and `isna()` are equivalent Pandas methods that return a boolean DataFrame or Series indicating which values are missing (NaN). They are the first step in assessing data completeness. Combined with `.sum()`, they give a count of missing values per column, and with boolean indexing they select rows with missing data.
Visit the following resources to learn more:
- [@official@Working with missing data](https://pandas.pydata.org/docs/user_guide/missing_data.html)
- [@official@pandas.isnull](https://pandas.pydata.org/docs/reference/api/pandas.isnull.html)
- [@official@pandas.isna](https://pandas.pydata.org/docs/reference/api/pandas.isna.html)
- [@article@Handling Missing Values with Pandas](https://towardsdatascience.com/handling-missing-values-with-pandas-b876bf6f008f/)
@@ -0,0 +1,8 @@
# JSON
JSON (JavaScript Object Notation) is a text format for structured data commonly returned by APIs. `pd.read_json()` converts JSON into a DataFrame, though nested structures often require normalization with `pd.json_normalize()`. JSON is flexible but can be irregular in structure, requiring careful handling of missing fields.
Visit the following resources to learn more:
- [@official@pandas.read_json](https://pandas.pydata.org/docs/reference/api/pandas.read_json.html)
- [@article@Pandas Read JSON](https://www.w3schools.com/python/pandas/pandas_json.asp)
@@ -0,0 +1,8 @@
# JupyterLab
JupyterLab is the modern, full-featured successor to the classic Jupyter Notebook interface. It supports notebooks where code, output, and narrative text coexist in a single document, while adding a tabbed layout, a file browser, a terminal, and support for multiple file types side by side. It is the standard environment for exploratory data analysis because results are visible immediately after each cell is run.
Visit the following resources to learn more:
- [@official@Get Started](https://jupyterlab.readthedocs.io/en/stable/getting_started/overview.html)
- [@video@Jupyter Notebook Complete Beginner Guide](https://www.youtube.com/watch?v=5pf0_bpNbkw)
@@ -0,0 +1,8 @@
# Lambda Functions
Lambda functions are anonymous, single-expression functions defined with the `lambda` keyword. They are used for short, throwaway operations, particularly as arguments to functions like `map()`, `filter()`, and Pandas' `apply()`. For example: `df['col'].apply(lambda x: x * 2)`.
Visit the following resources to learn more:
- [@article@Python Lambda](https://www.w3schools.com/python/python_lambda.asp)
- [@video@Python Lambda Functions Explained](https://www.youtube.com/watch?v=HQNiSfb795A)
@@ -0,0 +1,8 @@
# Linear Algebra Basics
NumPy provides linear algebra operations, including matrix multiplication (`np.dot`, `@`), matrix inversion, determinants, and eigenvalues. These are used in statistics (covariance matrices), machine learning (feature transformations), and scientific computing. Understanding the basics of matrix operations is useful for reading ML algorithm implementations.
Visit the following resources to learn more:
- [@article@Numpy Linear Algebra](https://www.programiz.com/python-programming/numpy/linear-algebra)
- [@video@Learn NumPy in 1 hour! 🔢](https://www.youtube.com/watch?v=VXU4LSAQDSc)
@@ -0,0 +1,9 @@
# List Comprehensions
List comprehensions provide a concise syntax for creating lists by applying an expression to each item in an iterable, optionally filtering with a condition. For example: `[x**2 for x in range(10) if x % 2 == 0]`. They are faster and more readable than equivalent `for` loops for simple transformations.
Visit the following resources to learn more:
- [@official@List Comprehensions](https://docs.python.org/3/tutorial/datastructures.html#list-comprehensions)
- [@article@When to Use a List Comprehension in Python Quiz](https://realpython.com/quizzes/list-comprehension-python/)
- [@article@List Comprehensions in Python](https://towardsdatascience.com/list-comprehensions-in-python-28d54c9286ca/)
@@ -0,0 +1,8 @@
# Lists
Lists are ordered, mutable sequences that can hold elements of any type. They are one of the most used data structures in Python for storing collections of values. Typical uses include holding column names, storing results from loops, and passing multiple values to functions.
Visit the following resources to learn more:
- [@official@List](https://docs.python.org/3/tutorial/datastructures.html)
- [@video@How to Use Lists in Python](https://www.youtube.com/watch?v=9OeznAkyQz4)
@@ -0,0 +1,9 @@
# Logical
Logical operators combine boolean expressions. Python uses `and`, `or`, and `not` for this purpose. They are used heavily in data filtering conditions, such as selecting rows where multiple criteria are true simultaneously.
Visit the following resources to learn more:
- [@article@Python Logical Operators](https://www.w3schools.com/python/python_if_logical.asp)
- [@article@Python not Operator: The Complete Guide to Logical Negation](https://roadmap.sh/python/not-operator)
- [@video@Logical operators in Python are easy 🔣](https://www.youtube.com/watch?v=W7luvtXeQTA)
@@ -0,0 +1,10 @@
# Loops
Loops execute a block of code repeatedly. Python provides `for` loops for iterating over sequences and `while` loops for condition-based repetition. They are used for batch processing files, iterating over grouped data, and automating repetitive tasks, though vectorized operations are preferred for performance.
Visit the following resources to learn more:
- [@article@Python while Loops: Repeating Tasks Conditionally](https://realpython.com/python-while-loop/)
- [@article@Python for Loops: The Pythonic Way](https://realpython.com/python-for-loop/#the-guts-of-the-python-for-loop)
- [@article@Understand Loops in Python with One Article](https://towardsdatascience.com/understand-loops-in-python-with-one-article-bace2ddba789/)
- [@video@Learn Python for loops in 5 minutes!](https://www.youtube.com/watch?v=KWgYha0clzw)
@@ -0,0 +1,9 @@
# Matplotlib
Matplotlib is Python's foundational plotting library. It provides a MATLAB-like interface for creating static, animated, and interactive visualizations. While more verbose than higher-level libraries, Matplotlib offers the most control over every aspect of a plot and is the basis for understanding how other Python visualization tools work.
Visit the following resources to learn more:
- [@official@Matplotlib Tutorials](https://matplotlib.org/stable/tutorials/index.html)
- [@article@Matplotlib Tutorial](https://www.w3schools.com/python/matplotlib_intro.asp)
- [@video@Matplotlib Full Python Course - Data Science Fundamentals](https://www.youtube.com/watch?v=OZOOLe2imFo)
@@ -0,0 +1,8 @@
# Mean, Median, Mode
Mean, median, and mode are measures of central tendency that describe the typical value in a distribution. Pandas computes these with `mean()`, `median()`, and `mode()` on Series or DataFrame columns. Comparing them reveals distribution shape: in a symmetric distribution they are equal; in a skewed one they diverge.
Visit the following resources to learn more:
- [@article@A Guide to Metrics in Exploratory Data Analysis](https://towardsdatascience.com/a-guide-to-metrics-in-exploratory-data-analysis-250b33f72297/)
- [@article@Understanding Your Data: The Essentials of Exploratory Data Analysis](https://dev.to/nderitugichuki/understanding-your-data-the-essentials-of-exploratory-data-analysis-400i)
@@ -0,0 +1,9 @@
# Merging & Joining
Pandas provides `merge()` and `join()` for combining DataFrames based on common columns or indices. Merge supports inner, left, right, and outer joins, mirroring SQL JOIN behavior. `concat()` stacks DataFrames vertically or horizontally. These operations are used to combine data from multiple sources into a single analysis-ready table.
Visit the following resources to learn more:
- [@official@Merge, join, concatenate and compare](https://pandas.pydata.org/docs/user_guide/merging.html)
- [@article@Combining Data in pandas With merge(), .join(), and concat()](https://realpython.com/pandas-merge-join-and-concat/)
- [@article@Pandas: Combining Data](https://towardsdatascience.com/pandas-combining-data-b190d793b626/)
@@ -0,0 +1,8 @@
# Null Values
Null values represent missing or undefined data within a dataset, signaling that a specific observation or entry is absent. In Python, this is typically represented by `None`, while libraries like pandas utilize `NaN` (Not a Number) to denote missing numeric information. Handling these values is a fundamental step in data cleaning, as they must be identified and addressed to ensure that statistical calculations and machine learning models perform accurately.
Visit the following resources to learn more:
- [@article@Python Null (None): Guide to Missing Values and NoneType](https://roadmap.sh/python/null#the-most-common-uses-of-none)
- [@article@Null in Python: Understanding Python's NoneType Object](https://realpython.com/null-in-python/)
@@ -0,0 +1,10 @@
# NumPy
NumPy is the foundational numerical computing library for Python. It provides the `ndarray`, a fast, multi-dimensional array, and a comprehensive library of mathematical functions that operate on arrays without Python loops. NumPy underpins Pandas, Scikit-learn, and most other scientific Python libraries.
Visit the following resources to learn more:
- [@official@NumPy quickstart](https://numpy.org/doc/stable/user/quickstart.html)
- [@article@NumPy Tutorial](https://www.w3schools.com/python/numpy/default.asp)
- [@article@NumPy for Absolute Beginners: A Project-Based Approach to Data Analysis](https://towardsdatascience.com/numpy-for-absolute-beginners-a-project-based-approach-to-data-analysis/)
- [@video@Python NumPy Tutorial for Beginners](https://www.youtube.com/watch?v=QUT1VHiLmmI)
@@ -0,0 +1,9 @@
# OOP for Data Analysis
Object-oriented programming (OOP) organizes code around classes and objects rather than standalone functions and procedures. A class defines a blueprint with attributes (data) and methods (behavior), and objects are instances of that class. For data analysis work, OOP is useful when building reusable data processing components, custom dataset loaders, or analysis pipelines that need to maintain state across multiple steps. Most of the libraries used daily, including Pandas, NumPy, and Scikit-learn, are built around classes, so understanding OOP helps in reading documentation, subclassing existing components, and writing cleaner, more maintainable analysis code.
Visit the following resources to learn more:
- [@article@Object-Oriented Programming (OOP) in Python](https://towardsdatascience.com/object-oriented-programming-oop-in-python-56b1f3229c0f/)
- [@article@How data scientists can leverage object oriented programming (OOP)](https://medium.com/@lawjimmy123/how-data-scientists-can-leverage-object-oriented-programming-oop-design-pattern-to-write-better-699166910882)
- [@article@Object-Oriented Programming (OOP) in Python](https://realpython.com/python3-object-oriented-programming/)
@@ -0,0 +1,9 @@
# Operators
Operators are symbols that perform operations on values and variables. Python supports arithmetic, comparison, and logical operators, each serving a different purpose in data analysis code. Understanding how operators work and combine is necessary for writing correct filtering conditions, calculations, and control flow logic.
Visit the following resources to learn more:
- [@article@Python Operators](https://www.w3schools.com/python/python_operators.asp)
- [@article@Python Operators from Scratch!!! – A Beginner’s Guide](https://towardsdatascience.com/python-operators-from-scratch-a-beginners-guide-8471306f4278/)
- [@video@Python Tutorial for Beginners | Operators in Python](https://www.youtube.com/watch?v=v5MR5JnKcZI)
@@ -0,0 +1,9 @@
# Pandas String Methods
Pandas exposes string methods through the `.str` accessor on Series, allowing vectorized text operations on entire columns. Methods include `.str.strip()`, `.str.lower()`, `.str.contains()`, `.str.replace()`, `.str.split()`, and `.str.extract()`. These methods avoid the need to loop over rows for string cleaning.
Visit the following resources to learn more:
- [@official@Working with text data](https://pandas.pydata.org/docs/user_guide/text.html)
- [@article@5 Must-Know Pandas Operations on Strings](https://towardsdatascience.com/5-must-know-pandas-operations-on-strings-4f88ca6b8e25/)
- [@video@How do I use string methods in pandas?](https://www.youtube.com/watch?v=bofaC0IckHo)
@@ -0,0 +1,9 @@
# Pandas
Pandas is the primary data manipulation library for Python. It provides two core data structures: Series (one-dimensional) and DataFrame (two-dimensional tabular data). Pandas supports loading data from many formats, cleaning, transforming, grouping, merging, and exporting data, covering the full data preparation workflow.
Visit the following resources to learn more:
- [@official@10 minutes to pandas](https://pandas.pydata.org/docs/user_guide/10min.html)
- [@article@Pandas Tutorial](https://www.w3schools.com/python/pandas/)
- [@video@Learn Pandas in 1 hour! 🐼](https://www.youtube.com/watch?v=VXtjG_GzO7Q)
@@ -0,0 +1,8 @@
# Pandas
Pandas integrates with SQL through `pd.read_sql()`, which executes a SQL query against a database connection and returns the result as a DataFrame. This allows analysts to leverage SQL for initial data extraction and filtering while using Pandas for downstream manipulation and analysis.
Visit the following resources to learn more:
- [@official@pandas.read_sql](https://pandas.pydata.org/docs/reference/api/pandas.read_sql.html)
- [@video@Read Write Data From Database (read_sql, to_sql)](https://www.youtube.com/watch?v=M-4EpNdlSuY)
@@ -0,0 +1,8 @@
# Parquet
Parquet is a columnar file format optimized for analytical workloads. It stores data with type information, supports efficient compression, and reads much faster than CSV for large datasets. `pd.read_parquet()` requires the `pyarrow` or `fastparquet` library and is the preferred format for storing processed DataFrames on disk.
Visit the following resources to learn more:
- [@official@pandas.read_parquet](https://pandas.pydata.org/docs/reference/api/pandas.read_parquet.html)
- [@video@Reading Parquet Files in Python](https://www.youtube.com/watch?v=XFO5jdGsMek)
@@ -0,0 +1,9 @@
# Parsing Dates
Date columns loaded from CSV are typically read as strings and must be converted to datetime objects for time-based operations. `pd.to_datetime()` parses date strings in many formats and accepts a `format` parameter for custom patterns. Once parsed, datetime columns enable operations like extracting year/month, computing differences, and resampling time series.
Visit the following resources to learn more:
- [@official@pandas.to_datetime](https://pandas.pydata.org/docs/reference/api/pandas.to_datetime.html)
- [@article@Working with Dates and Time Series Data](https://www.youtube.com/watch?v=UFuo7EHI8zc)
- [@article@Dealing with Date and Time in Pandas DataFrames](https://towardsdatascience.com/dealing-with-date-and-time-in-pandas-dataframes-7d140f711a47/)
@@ -0,0 +1,9 @@
# pip
pip is Python's default package installer. It installs packages from the Python Package Index (PyPI) using `pip install package-name`. Libraries like NumPy, Pandas, Matplotlib, and Scikit-learn are all installed this way. A `requirements.txt` file captures all dependencies for a project.
Visit the following resources to learn more:
- [@official@Installing Packages](https://packaging.python.org/en/latest/tutorials/installing-packages/)
- [@opensource@pip](https://github.com/pypa/pip)
- [@video@Python pip 🏗️](https://www.youtube.com/watch?v=9z7gGUbAj5U)
@@ -0,0 +1,9 @@
# Plot Categories
Matplotlib supports a wide range of plot types: line plots (`plot`), bar charts (`bar`, `barh`), scatter plots (`scatter`), histograms (`hist`), box plots (`boxplot`), pie charts (`pie`), and more. Choosing the right plot type depends on the data structure and the relationship being communicated.
Visit the following resources to learn more:
- [@official@Plot types](https://matplotlib.org/stable/plot_types/index.html)
- [@article@Python Gallery: Matplotlib](https://python-graph-gallery.com/matplotlib/)
- [@article@Matplotlib: Part 3. Exploring Different Plot Types](https://medium.com/@ebimsv/mastering-matplotlib-3-exploring-different-plot-types-bd13d18ff613)
@@ -0,0 +1,9 @@
# Plotly
Plotly is a Python library for creating interactive charts and dashboards. It produces web-based visualizations using JavaScript under the hood, with a Python API. Plotly Express provides a high-level interface for common chart types, while the `graph_objects` module offers full control. Plotly integrates with Dash for building full web dashboards.
Visit the following resources to learn more:
- [@official@Getting Started with Plotly in Python](https://plotly.com/python/getting-started/)
- [@official@Plotly Python Graphing Library Fundamentals](https://plotly.com/python/plotly-fundamentals/)
- [@video@Plotly Tutorial - Basics in 7 Minutes!](https://www.youtube.com/watch?v=PqUaDvbczbI)
@@ -0,0 +1,8 @@
# Polars
Polars is a fast DataFrame library for Python written in Rust. It is designed as a high-performance alternative to Pandas, with a more consistent API and significantly better performance on large datasets. Polars uses lazy evaluation and query optimization to process data efficiently without loading everything into memory at once.
Visit the following resources to learn more:
- [@official@Getting started](https://docs.pola.rs/user-guide/getting-started/)
- [@video@Learning the Polars DataFrame Library!](https://www.youtube.com/watch?v=OTVDmA6CRlQ)
@@ -0,0 +1,8 @@
# Power BI / Tableau
Power BI and Tableau are enterprise BI platforms for building interactive dashboards and reports. Python integrates with both: Power BI supports Python visuals and data transformation scripts, and Tableau supports Python through TabPy for custom calculations. Analysts who prepare data in Python can visualize and distribute it through these platforms for business audiences.
Visit the following resources to learn more:
- [@video@Power BI Tutorials for Beginners](https://www.youtube.com/playlist?list=PLUaB-1hjhk8HqnmK0gQhfmIdCbxwoAoys)
- [@video@Learn Tableau in Under 2 hours](https://www.youtube.com/watch?v=j8FSP8XuFyk)
@@ -0,0 +1,10 @@
# Printing Variables
Printing variables is done with Python's built-in `print()` function. During analysis, printing intermediate values helps verify that transformations are working as expected. F-strings (`f"value: {variable}"`) provide a clean way to format output with variable values embedded in strings.
Visit the following resources to learn more:
- [@article@Variables in Python: Usage and Best Practices](https://realpython.com/python-variables/)
- [@article@Python Print New Line: Methods, Examples, and Best Practices](https://roadmap.sh/python/print-new-line)
- [@article@Python Multiline Strings: The Complete Guide](https://roadmap.sh/python/multiline-strings)
- [@video@Data Types & Variables in Python](https://www.youtube.com/playlist?list=PLBlnK6fEyqRhN-sfWgCU1z_Qhakc1AGOn)
@@ -0,0 +1,9 @@
# PySpark
PySpark is the Python API for Apache Spark, the distributed data processing engine. It allows Python code to run Spark jobs on clusters, processing datasets at the scale of terabytes. PySpark provides DataFrame and SQL APIs similar to Pandas and integrates with MLlib for distributed machine learning. It is used when data volume exceeds what Dask or a single machine can handle.
Visit the following resources to learn more:
- [@official@PySpark Tutorials](https://spark.apache.org/docs/latest/api/python/tutorial/index.html)
- [@article@PySpark for Beginners: Beyond the Basics](https://towardsdatascience.com/pyspark-for-beginners-beyond-the-basics/)
- [@video@PySpark Tutorial](https://www.youtube.com/watch?v=wNRjR6Cds5s&list=PL2IsFZBGM_IHCl9zhRVC1EXTomkEp_1zm)
@@ -0,0 +1,9 @@
# Random Module
NumPy's `random` module generates pseudorandom numbers and samples. It provides functions for creating random arrays, sampling from distributions (normal, uniform, binomial), and setting a seed for reproducibility. Random number generation is used in simulation, bootstrapping, and initializing machine learning models.
Visit the following resources to learn more:
- [@article@Random Numbers in NumPy](https://www.w3schools.com/python/numpy/numpy_random.asp)
- [@article@Numpy Random](https://www.programiz.com/python-programming/numpy/random)
- [@video@Random numbers in NumPy are easy! 🎲](https://www.youtube.com/watch?v=Ql5zGPtxlHY&pp=ygUNIG51bXB5IHJhbmRvbQ%3D%3D)
@@ -0,0 +1,10 @@
# re
The `re` module provides regular expression support for pattern matching and text manipulation. It is used for extracting structured data from unstructured text, validating formats, and performing complex find-and-replace operations. Key functions include `re.match()`, `re.search()`, `re.findall()`, and `re.sub()`.
Visit the following resources to learn more:
- [@official@re — Regular expression operations](https://docs.python.org/3/library/re.html)
- [@official@Regular expression HOWTO](https://docs.python.org/3/howto/regex.html)
- [@article@Python RegEx](https://www.w3schools.com/python/python_regex.asp)
- [@video@Python Tutorial: re Module - How to Write and Match Regular Expressions (Regex)](https://www.youtube.com/watch?v=K8L6KVGG-7o)
@@ -0,0 +1,9 @@
# Reading Data
Pandas provides functions to load data from many formats: `pd.read_csv()`, `pd.read_excel()`, `pd.read_json()`, `pd.read_parquet()`, `pd.read_sql()`, and others. Each function returns a DataFrame and accepts parameters for handling headers, delimiters, encoding, and data types. Reading data is always the first step in a Pandas workflow.
Visit the following resources to learn more:
- [@official@How do I read and write tabular data?](https://pandas.pydata.org/docs/getting_started/intro_tutorials/02_read_write.html)
- [@article@Pandas 101: How to Read Data from Multiple Sources](https://riyoma.medium.com/pandas-101-how-to-read-data-from-multiple-sources-ba75e5497ad5)
- [@video@Reading in Files in Pandas | Python Pandas Tutorials](https://www.youtube.com/watch?v=dUpyC40cF6Q)
@@ -0,0 +1,8 @@
# Reading Local Files
Reading local files loads data stored on disk into Python for analysis. Pandas supports the most common file formats used in data work. The right function to use depends on the file format, and parameters like delimiter, encoding, and header row often need to be specified.
Visit the following resources to learn more:
- [@article@pandas: How to Read and Write Files](https://realpython.com/pandas-read-write-files/)
- [@video@How to Navigate File Paths: Reading Data With Pandas](https://www.youtube.com/watch?v=39pwKSJ7T1Y)
@@ -0,0 +1,7 @@
# Reading Web Data
Reading web data involves fetching data from URLs, REST APIs, and web pages directly into Python. This allows analysis workflows to incorporate live or frequently updated data without manual downloads. The main tools are the `requests` library for APIs and `BeautifulSoup` or `scrapy` for web scraping.
Visit the following resources to learn more:
- [@article@An Efficient Way to Read Data from the Web Directly into Python](https://medium.com/data-science/an-efficient-way-to-read-data-from-the-web-directly-into-python-a526a0b4f4cb)
@@ -0,0 +1,9 @@
# Regression Plots
Seaborn's regression plots visualize the relationship between two numeric variables with a fitted regression line. `sns.regplot()` plots data points and a linear regression fit with confidence interval. `sns.lmplot()` extends this to support faceting by a categorical variable, enabling comparison across groups.
Visit the following resources to learn more:
- [@official@Visualizing statistical relationships](https://seaborn.pydata.org/tutorial/relational.html)
- [@article@Seaborn Relplot in Python: Visualising Relationships in Data](https://towardsdatascience.com/seaborn-relplot-in-python-visualising-relationships-in-data-ee39138d53aa/)
- [@video@Seaborn scatter plot | How to make and style a scatterplot in Python seaborn](https://www.youtube.com/watch?v=4yz4cMXCkuw)
@@ -0,0 +1,9 @@
# Reshaping
Reshaping transforms the structure of a DataFrame without changing its data. `pivot()` and `pivot_table()` convert long-format data to wide format. `melt()` does the reverse, converting wide to long. `stack()` and `unstack()` move index levels to columns or vice versa. Reshaping is often needed to prepare data for specific visualizations or models.
Visit the following resources to learn more:
- [@official@Reshaping and pivot tables](https://pandas.pydata.org/docs/user_guide/reshaping.html)
- [@article@Reshaping a Pandas Dataframe: Long-to-Wide and Vice Versa](https://towardsdatascience.com/reshaping-a-pandas-dataframe-long-to-wide-and-vice-versa-517c7f0995ad/)
- [@video@Pandas - Reshape Dataframe](https://www.youtube.com/watch?v=oY62o-tBHF4&list=PL6icxsSf2sRe2qqFmtZwU_VKChEChzhEP)
@@ -0,0 +1,8 @@
# Saving figures
Figures are saved to disk with `plt.savefig('filename.png', dpi=300, bbox_inches='tight')`. Supported formats include PNG, PDF, SVG, and JPEG. Saving high-resolution figures is important when embedding charts in reports or publications. The `bbox_inches='tight'` parameter prevents axis labels from being cut off.
Visit the following resources to learn more:
- [@article@matplotlib.pyplot.savefig](https://matplotlib.org/stable/api/_as_gen/matplotlib.pyplot.savefig.html)
- [@video@How to save a matplotlib figure and fix text cutting off || Matplotlib Tips](https://www.youtube.com/watch?v=C8MT-A7Mvk4)
@@ -0,0 +1,9 @@
# Scatterplot
A scatter plot displays two numeric variables as points on an x-y axis to reveal their relationship. It is used during EDA to detect correlations, clusters, and outliers. A trend line or regression line can be added to show the direction and strength of the linear relationship between the variables.
Visit the following resources to learn more:
- [@article@matplotlib.pyplot.scatter](https://matplotlib.org/stable/api/_as_gen/matplotlib.pyplot.scatter.html)
- [@article@How to Make a Scatter Plot in Python With plt.scatter()](https://realpython.com/visualizing-python-plt-scatter/)
- [@video@Seaborn scatter plot | How to make and style a scatterplot in Python seaborn](https://www.youtube.com/watch?v=4yz4cMXCkuw)
@@ -0,0 +1,8 @@
# Scikit-learn
Scikit-learn is the standard machine learning library for Python. It provides a consistent API for classification, regression, clustering, dimensionality reduction, model selection, and preprocessing. Models are trained with `.fit()`, used to predict with `.predict()`, and evaluated with a suite of metrics. Scikit-learn also provides tools for pipelines, cross-validation, and hyperparameter tuning.
Visit the following resources to learn more:
- [@article@Scikit-learn](https://scikit-learn.org/stable/)
- [@video@Scikit-learn Crash Course - Machine Learning Library for Python](https://www.youtube.com/watch?v=0B5eIE_1vpU)
@@ -0,0 +1,9 @@
# SciPy
SciPy is a scientific computing library built on NumPy. It provides modules for statistics (`scipy.stats`), optimization (`scipy.optimize`), linear algebra, signal processing, and numerical integration. `scipy.stats` is used for hypothesis tests (t-tests, chi-square, ANOVA), probability distributions, and descriptive statistics beyond what NumPy provides.
Visit the following resources to learn more:
- [@official@SciPy User Guide](https://docs.scipy.org/doc/scipy/tutorial/)
- [@article@SciPy Tutorial](https://www.w3schools.com/python/scipy/index.php)
- [@video@SciPy Tutorial: For Physicists, Engineers, and Mathematicians](https://www.youtube.com/watch?v=jmX4FOUEfgU)
@@ -0,0 +1,9 @@
# scrapy
Scrapy is a Python framework for large-scale web scraping. Unlike BeautifulSoup, which parses individual pages, Scrapy manages the full crawling workflow: following links, handling pagination, managing request queues, and exporting data. It is used when scraping requires collecting data from many pages across a site.
Visit the following resources to learn more:
- [@official@Scrapy Docs](https://docs.scrapy.org/en/latest/)
- [@article@A Minimalist End-to-End Scrapy Tutorial (Part I)](https://medium.com/data-science/a-minimalist-end-to-end-scrapy-tutorial-part-i-11e350bcdec0)
- [@video@Scrapy for Beginners - A Complete How To Example Web Scraping Project](https://www.youtube.com/watch?v=s4jtkzHhLzY)
@@ -0,0 +1,10 @@
# Seaborn
Seaborn is a Python visualization library built on Matplotlib that provides a higher-level interface for statistical graphics. It handles common plot types with less code and integrates tightly with Pandas DataFrames. Seaborn is particularly strong for visualizing statistical relationships, distributions, and grouped comparisons.
Visit the following resources to learn more:
- [@official@Seaborn Tutorials](https://seaborn.pydata.org/tutorial.html)
- [@official@Introduction to Seaborn for dataviz with Python](https://python-graph-gallery.com/seaborn/)
- [@article@Data Visualisation Tutorial Using Seaborn](https://towardsdatascience.com/data-visualisation-tutorial-using-seaborn-26e1ef9043db/)
- [@video@Introduction to Seaborn](https://www.youtube.com/watch?v=vaf4ir8eT38&list=PLtPIclEQf-3cG31dxSMZ8KTcDG7zYng1j)
@@ -0,0 +1,10 @@
# Series and DataFrame
A Series is a one-dimensional labeled array, analogous to a single column in a spreadsheet. A DataFrame is a two-dimensional table of Series that share an index, analogous to a spreadsheet or SQL table. These two structures are the foundation of all Pandas operations.
Visit the following resources to learn more:
- [@official@pandas.Series](https://pandas.pydata.org/docs/reference/api/pandas.Series.html)
- [@official@pandas.DataFrame](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.html)
- [@article@Pandas Series](https://www.w3schools.com/python/pandas/pandas_series.asp)
- [@article@Pandas DataFrames](https://www.w3schools.com/python/pandas/pandas_dataframes.asp)
@@ -0,0 +1,8 @@
# Sets
Sets are unordered collections of unique values. They support mathematical set operations like union, intersection, and difference. They are useful for finding unique values, checking membership, and comparing two groups of items.
Visit the following resources to learn more:
- [@official@Sets](https://docs.python.org/3/tutorial/datastructures.html#sets)
- [@video@What are Sets in Python? Python Tutorial for Absolute Beginners](https://www.youtube.com/watch?v=t9j8lCUGZXo)
@@ -0,0 +1,9 @@
# SQL Fundamentals
SQL (Structured Query Language) is the standard language for querying relational databases. Data analysts use SQL to extract, filter, aggregate, and join data from databases before loading it into Python for further analysis. Python provides several libraries for running SQL queries directly from code.
Visit the following resources to learn more:
- [@article@SQL Tutorial - Mode](https://www.thoughtspot.com/sql-tutorial)
- [@article@SQL Tutorial](https://www.sqltutorial.org/)
- [@video@SQL Tutorial - Full Database Course for Beginners](https://www.youtube.com/watch?v=HXV3zeQKqGY)
@@ -0,0 +1,9 @@
# SQLAlchemy
SQLAlchemy is a Python SQL toolkit and object-relational mapper (ORM) that provides a unified interface for connecting to many database backends including PostgreSQL, MySQL, SQLite, and others. It is used with Pandas via `pd.read_sql()` to load query results directly into DataFrames.
Visit the following resources to learn more:
- [@official@SQLAlchemy](https://www.sqlalchemy.org/)
- [@article@Mastering SQLAlchemy: A Comprehensive Guide for Python Developers](https://medium.com/@ramanbazhanau/mastering-sqlalchemy-a-comprehensive-guide-for-python-developers-ddb3d9f2e829)
- [@video@SQLAlchemy: The BEST SQL Database Library in Python](https://www.youtube.com/watch?v=aAy-B6KPld8)
@@ -0,0 +1,8 @@
# sqlite3
`sqlite3` is Python's built-in library for working with SQLite databases. SQLite is a lightweight, file-based relational database that requires no server setup. It is commonly used for local data storage, prototyping, and working with small to medium datasets entirely within Python.
Visit the following resources to learn more:
- [@official@sqlite3](https://docs.python.org/3/library/sqlite3.html#sqlite3-tutorial)
- [@video@Sqlite 3 Python Tutorial in 5 minutes](https://www.youtube.com/watch?v=girsuXz0yA8)
@@ -0,0 +1,9 @@
# Statistics & ML
Python has a rich ecosystem of libraries for statistical analysis and machine learning. SciPy extends NumPy with statistical tests, optimization, and signal processing. Scikit-learn provides a consistent API for building, evaluating, and deploying machine learning models. Together they cover the analytical needs of most data analysis work.
Visit the following resources to learn more:
- [@article@Python Statistics Fundamentals: How to Describe Your Data](https://realpython.com/python-statistics/)
- [@video@Mastering Probability and Statistics in Python](https://www.youtube.com/playlist?list=PLVgEzPHodXi1wT9OK8B_W6Hs8Xc-gaG6N)
- [@video@Machine Learning Tutorial Python | Machine Learning For Beginners](https://www.youtube.com/playlist?list=PLeo1K3hjS3uvCeTYTeyfe0-rN5r8zn9rw)

Some files were not shown because too many files have changed in this diff Show More