feat: clean roadmap content and sync scripts (#10181)

* refactor: clean to roadmap content * refactor: move shared helpers into scripts/lib
2026-10-28 02:43:08 +08:00 · 2026-07-28 05:45:17 +06:00
parent 6f745dc1ea
commit 274082fd77
12643 changed files with 440 additions and 537705 deletions
@@ -0,0 +1,8 @@
+# A/B Testing
+
+A/B testing is a way to compare two versions of something to see which one works better. You split your audience into two groups, one sees version A, the other sees version B — and then you measure which version gets better results, like more clicks, sales, or sign-ups. This helps you make decisions based on real data instead of guesses.
+
+Visit the following resources to learn more:
+
+- [@article@A software engineer's guide to A/B testing](https://posthog.com/product-engineers/ab-testing-guide-for-engineers)
+- [@video@A/B Testing for Beginners](https://www.youtube.com/watch?v=VpTlNRUcIDo)
@@ -0,0 +1,8 @@
+# Amazon EC2 (Compute)
+ 
+Amazon EC2 (Elastic Compute Cloud) provides virtual servers in the AWS cloud. Users can choose instance types optimized for compute, memory, or storage, and pay only for what they run. EC2 is used for running data processing jobs, hosting databases, and building custom data infrastructure on AWS.
+
+Visit the following resources to learn more:
+
+- [@official@EC2 - User Guide](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/concepts.html)
+- [@video@Introduction to Amazon EC2](https://www.youtube.com/watch?v=eaicwmnSdCs)
@@ -0,0 +1,7 @@
+# Amazon RDS (Database)
+ 
+Amazon RDS (Relational Database Service) is a managed relational database service from AWS that supports MySQL, PostgreSQL, MariaDB, Oracle, and MS SQL Server. It handles provisioning, backups, patching, and replication automatically. RDS is used for transactional databases that require minimal database administration overhead.
+
+Visit the following resources to learn more:
+
+- [@official@Amazon RDS](https://aws.amazon.com/rds/)
@@ -0,0 +1,7 @@
+# Amazon RDS (Database)
+
+Amazon RDS (Relational Database Service) is a web service from Amazon Web Services. It's designed to simplify the setup, operation, and scaling of relational databases in the cloud. This service provides cost-efficient, resizable capacity for an industry-standard relational database and manages common database administration tasks. RDS supports six database engines: Amazon Aurora, PostgreSQL, MySQL, MariaDB, Oracle Database, and SQL Server. These engines give you the ability to run instances ranging from 5GB to 6TB of memory, accommodating your specific use case. It also ensures the database is up-to-date with the latest patches, automatically backs up your data and offers encryption at rest and in transit.
+
+Visit the following resources to learn more:
+
+- [@official@Amazon RDS](https://aws.amazon.com/rds/)
@@ -0,0 +1,8 @@
+# Amazon Redshift
+
+Amazon Redshift is a cloud-based data warehouse service from Amazon that lets you store and analyze large amounts of data quickly. It’s designed for running complex queries on huge datasets, so businesses can use it to turn raw data into useful reports and insights. You can load data into Redshift from many sources, and then use SQL to explore it, just like you would with a regular database — but it’s optimized to handle much bigger data and run faster.
+
+Visit the following resources to learn more:
+
+- [@official@Amazon Redshift](https://aws.amazon.com/redshift/)
+- [@video@Getting Started with Amazon Redshift - AWS Online Tech Talks](https://www.youtube.com/watch?v=dfo4J5ZhlKI)
@@ -0,0 +1,7 @@
+# Apache Airflow
+
+Apache Airflow is an open-source tool that helps you schedule, organize, and monitor workflows. Think of it like a to-do list for your data tasks, but smarter — you can set tasks to run in a specific order, track their progress, and see what happens if something fails. It’s often used for automating data pipelines so that data moves, gets processed, and is ready for use without manual work.
+
+Visit the following resources to learn more:
+
+- [@official@Apache Airflow](https://airflow.apache.org/)
@@ -0,0 +1,7 @@
+# Apache Hadoop YARN
+
+Apache Hadoop YARN (Yet Another Resource Negotiator) is the part of Hadoop that manages resources and runs jobs on a cluster. It has a ResourceManager that controls all cluster resources and an ApplicationMaster for each job that schedules and runs tasks. YARN lets different tools like MapReduce and Spark share the same cluster, making it more efficient, flexible, and reliable.
+
+Visit the following resources to learn more:
+
+- [@video@Hadoop Yarn Tutorial](https://www.youtube.com/watch?v=6bIF9VwRwE0)
@@ -0,0 +1,10 @@
+# Apache Kafka
+
+Apache Kafka is an open-source stream-processing software platform developed by LinkedIn and donated to the Apache Software Foundation. It is written in Scala and Java and operates based on a message queue, designed to handle real-time data feeds. Kafka functions as a kind of message broker service in between the data producers and the consumers, facilitating efficient transmission of data. It can be viewed as a durable message broker where applications can process and reprocess streamed data. Kafka is a highly scalable and fault-tolerant system which ensures data delivery without loss.
+
+Visit the following resources to learn more:
+
+- [@official@Apache Kafka Docs](https://kafka.apache.org/43/getting-started/introduction/)
+- [@article@Kafka Streams Confluent](https://kafka.apache.org/documentation/streams/)
+- [@video@Apache Kafka Fundamentals](https://www.youtube.com/watch?v=B5j3uNBH8X4)
+- [@video@Kafka in 100 Seconds](https://www.youtube.com/watch?v=uvb00oaa3k8)
@@ -0,0 +1,8 @@
+# Apache Spark
+ 
+Apache Spark is a distributed data processing engine for large-scale batch and streaming workloads. It processes data in memory across a cluster, making it significantly faster than MapReduce for many workloads. Spark supports Python, Scala, Java, and R, and provides APIs for SQL, streaming, machine learning, and graph processing.
+
+Visit the following resources to learn more:
+
+- [@official@ApacheSpark](https://spark.apache.org/documentation.html)
+- [@article@Spark By Examples](https://sparkbyexamples.com)
@@ -0,0 +1,9 @@
+# APIs
+ 
+APIs (Application Programming Interfaces) expose data from external services in a structured format, typically JSON or XML over HTTP. Many data pipelines pull data from third-party APIs such as payment processors, marketing platforms, or social networks. Rate limits, authentication, and schema changes are common challenges when ingesting from APIs.
+
+Visit the following resources to learn more:
+
+- [@roadmap@Visit the Dedicated Java Roadmap](https://roadmap.sh/api-design)
+- [@article@What is an API?](https://aws.amazon.com/what-is/api/)
+- [@article@A Beginner's Guide to APIs](https://www.postman.com/what-is-an-api/)
@@ -0,0 +1,9 @@
+# ArgoCD
+ 
+Argo CD is a declarative GitOps continuous delivery tool for Kubernetes. It continuously monitors a Git repository and ensures that the state of the Kubernetes cluster matches the desired state defined in code. Argo CD is used in data engineering to manage Kubernetes-based pipeline deployments and infrastructure changes through Git.
+
+Visit the following resources to learn more:
+
+- [@official@Argo CD - Argo Project](https://argo-cd.readthedocs.io/en/stable/)
+- [@video@ArgoCD Tutorial for Beginners](https://www.youtube.com/watch?v=MeU5_k9ssrs)
+- [@video@What is ArgoCD](https://www.youtube.com/watch?v=p-kAqxuJNik)
@@ -0,0 +1,8 @@
+# Async vs Sync Communication
+ 
+Synchronous communication means the sender waits for a response before continuing. Asynchronous communication means the sender sends a message and continues without waiting. Messaging systems enable asynchronous communication, which is better suited for high-throughput pipelines where blocking would create bottlenecks.
+
+Visit the following resources to learn more:
+
+- [@article@Synchronous And Asynchronous Data Transmission: The Differences And How to Use Them](https://www.computer.org/publications/tech-news/trends/synchronous-asynchronous-data-transmission)
+- [@article@Synchronous vs Asynchronous Communication: What’s the Difference?](https://www.getguru.com/reference/synchronous-vs-asynchronous-communication)
@@ -0,0 +1,7 @@
+# Aurora DB
+
+Amazon Aurora (Aurora) is a fully managed relational database engine that's compatible with MySQL and PostgreSQL. Aurora includes a high-performance storage subsystem. Its MySQL- and PostgreSQL-compatible database engines are customized to take advantage of that fast distributed storage. The underlying storage grows automatically as needed. Aurora also automates and standardizes database clustering and replication, which are typically among the most challenging aspects of database configuration and administration.
+
+Visit the following resources to learn more:
+
+- [@official@SAmazon Aurora](https://aws.amazon.com/rds/aurora/)
@@ -0,0 +1,8 @@
+# Authentication vs Authorization
+
+Authentication and authorization are popular terms in modern computer systems that often confuse people. **Authentication** is the process of confirming the identity of a user or a device (i.e., an entity). During the authentication process, an entity usually relies on some proof to authenticate itself, i.e. an authentication factor. In contrast to authentication, **authorization** refers to the process of verifying what resources entities (users or devices) can access, or what actions they can perform, i.e., their access rights.
+
+Visit the following resources to learn more:
+
+- [@article@Basic Authentication](https://roadmap.sh/guides/basic-authentication)
+- [@article@What is Authentication vs Authorization?](https://auth0.com/intro-to-iam/authentication-vs-authorization)
@@ -0,0 +1,11 @@
+# AWS CDK
+
+The AWS Cloud Development Kit (AWS CDK) is an open-source software development framework used to provision cloud infrastructure resources in a safe, repeatable manner through AWS CloudFormation. AWS CDK offers the flexibility to write infrastructure as code in popular languages like Python, Java, Go, and C#.
+
+Visit the following resources to learn more:
+
+- [@course@AWS CDK Crash Course for Beginners](https://www.youtube.com/watch?v=D4Asp5g4fp8)
+- [@official@AWS CDK](https://aws.amazon.com/cdk/)
+- [@official@AWS CDK Documentation](https://docs.aws.amazon.com/cdk/index.html)
+- [@opensource@AWS CDK Examples](https://github.com/aws-samples/aws-cdk-examples)
+- [@feed@Explore top posts about AWS](https://app.daily.dev/tags/aws?ref=roadmapsh)
@@ -0,0 +1,8 @@
+# AWS EKS
+ 
+Amazon EKS (Elastic Kubernetes Service) is AWS's managed Kubernetes service. It runs the Kubernetes control plane across multiple availability zones and integrates with AWS services like IAM, VPC, and ECR. EKS is used to run containerized data pipelines and services on AWS without managing the Kubernetes control plane directly.
+
+Visit the following resources to learn more:
+
+- [@official@Amazon Elastic Kubernetes Service (EKS)](https://aws.amazon.com/eks/)
+- [@official@Concepts of Amazon EKS](https://docs.aws.amazon.com/eks/)
@@ -0,0 +1,9 @@
+# AWS SNS
+ 
+Amazon SNS (Simple Notification Service) is a fully managed pub/sub messaging service from AWS. It allows a single message to be sent to multiple subscribers simultaneously through topics. SNS is commonly used alongside SQS to fan out messages to multiple queues or trigger downstream processing in Lambda functions and data pipelines.
+
+Visit the following resources to learn more:
+
+- [@official@Amazon Simple Notification Service (SNS)](http://aws.amazon.com/sns/)
+- [@official@Send Fanout Event Notifications](https://aws.amazon.com/getting-started/hands-on/send-fanout-event-notifications/)
+- [@article@What is Pub/Sub Messaging?](https://aws.amazon.com/what-is/pub-sub-messaging/)
@@ -0,0 +1,8 @@
+# AWS SQS
+
+Amazon Simple Queue Service (Amazon SQS) offers a secure, durable, and available hosted queue that lets you integrate and decouple distributed software systems and components. Amazon SQS offers common constructs such as dead-letter queues and cost allocation tags. It provides a generic web services API that you can access using any programming language that the AWS SDK supports.
+
+Visit the following resources to learn more:
+
+- [@official@Amazon Simple Queue Service](https://aws.amazon.com/sqs/)
+- [@official@What is Amazon Simple Queue Service?](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/welcome.html)
@@ -0,0 +1,9 @@
+# Azure Blob Storage
+
+Azure Blob Storage is Microsoft's object storage solution for the cloud. “Blob” stands for Binary Large Object, a term used to describe storage for unstructured data like text, images, and video. Azure Blob Storage is Microsoft Azure’s solution for storing these blobs in the cloud. It offers flexible storage—you only pay based on your usage. Depending on the access speed you need for your data, you can choose from various storage tiers (hot, cool, and archive). Being cloud-based, it is scalable, secure, and easy to manage.
+
+Visit the following resources to learn more:
+
+- [@official@Azure Blob Storage](https://azure.microsoft.com/en-us/products/storage/blobs)
+- [@official@Introduction to Azure Blob Storage](https://learn.microsoft.com/en-us/azure/storage/blobs/storage-blobs-introduction)
+- [@video@A Beginners Guide to Azure Blob Storage](https://www.youtube.com/watch?v=ah1XqItWkuc&t=300s)
@@ -0,0 +1,9 @@
+# Azure SQL Database
+
+Azure SQL Database is a fully managed Platform as a Service (PaaS) offering. It abstracts the underlying infrastructure, enabling developers to focus on building and deploying applications without worrying about database maintenance tasks.
+
+Visit the following resources to learn more:
+
+- [@official@Azure SQL Database](https://azure.microsoft.com/en-us/products/azure-sql/database)
+- [@official@What is Azure SQL Database?](https://learn.microsoft.com/en-us/azure/azure-sql/database/sql-database-paas-overview?view=azuresql)
+- [@video@Azure SQL for Beginners](https://www.youtube.com/playlist?list=PLlrxD0HtieHi5c9-i_Dnxw9vxBY-TqaeN)
@@ -0,0 +1,9 @@
+# Azure Virtual Machines
+
+Azure Virtual Machines (VMs) enable virtualization without requiring hardware investments. They provide customizable environments for development, testing, and cloud applications so you can run different operating systems like Ubuntu on a Windows host based on your needs. One of the key advantages of Azure VMs is the pay-as-you-go pricing model. It allows you to scale resources up or down as needed, ensuring cost efficiency without wasting resources.
+
+Visit the following resources to learn more:
+
+- [@official@Azure Virtual Machines](https://azure.microsoft.com/en-us/products/virtual-machines)
+- [@official@Virtual Machines in Azure](https://learn.microsoft.com/en-us/azure/virtual-machines/overview)
+- [@video@AVirtual Machines in Azure | Beginner's Guide](https://www.youtube.com/watch?v=_abaWXoQFZU)
@@ -0,0 +1,8 @@
+# Batch
+
+Batch processing is a method in which large volumes of collected data are processed in chunks or batches. This approach is especially effective for resource-intensive jobs, repetitive tasks, and managing extensive datasets where real-time processing isn’t required. It is ideal for applications like data warehousing, ETL (Extract, Transform, Load), and large-scale reporting. Data batch processing is mainly automated, requiring minimal human interaction once the process is set up. Tasks are predefined, and the system executes them according to a scheduled timeline, typically during off-peak hours when computing resources are readily available.
+
+Visit the following resources to learn more:
+
+- [@article@What is Batch Processing?](https://aws.amazon.com/what-is/batch-processing/)
+- [@article@Batch And Streaming Demystified For Unification](https://towardsdatascience.com/batch-and-streaming-demystified-for-unification-dee0b48f921d/)
@@ -0,0 +1,7 @@
+# Best Practices
+ 
+Best practices for messaging systems include designing idempotent consumers to handle duplicate delivery, setting appropriate retention and replication policies, monitoring consumer lag, and planning for schema evolution. Dead-letter queues are used to handle messages that fail processing repeatedly without losing them.
+
+Visit the following resources to learn more:
+
+- [@article@Best Practices for Message Queue Architecture](https://abhishek-patel.medium.com/best-practices-for-message-queue-architecture-f69d47e3565)
@@ -0,0 +1,8 @@
+# Big Data Tools
+ 
+Big data tools are designed to process and analyze datasets too large to handle with traditional single-machine tools. They distribute computation across clusters and are optimized for throughput at scale. The most widely used big data processing framework is Apache Spark, with Hadoop-based tools remaining common in legacy environments.
+
+Visit the following resources to learn more:
+
+- [@article@What is Big Data?](https://cloud.google.com/learn/what-is-big-data?hl=en)
+- [@video@Introduction to Big Data with Spark and Hadoop](http://youtube.com/watch?v=vHlwg4ciCsI&t=80s&ab_channel=freeCodeAcademy)
@@ -0,0 +1,8 @@
+# BigTable
+
+Bigtable is a high-performance, scalable database that excels at capturing, processing, and analyzing data in real-time. It aggregates data as it's written, providing immediate insights into user behavior, A/B testing results, and engagement metrics. This real-time capability also fuels AI/ML models for interactive applications. Bigtable integrates seamlessly with both Dataflow, enriching streaming pipelines with low-latency lookups, and BigQuery, enabling real-time serving of analytics in user-facing applications and ad-hoc querying on the same data.
+
+Visit the following resources to learn more:
+
+- [@official@Bigtable: Fast, Flexible NoSQL](https://cloud.google.com/bigtable?hl=en#scale-your-latency-sensitive-applications-with-the-nosql-pioneer)
+- [@article@Google Bigtable](https://www.techtarget.com/searchdatamanagement/definition/Google-BigTable)
@@ -0,0 +1,10 @@
+# Business Intelligence
+ 
+Business intelligence (BI) refers to the tools and processes used to collect, analyze, and visualize business data to support decisions. BI platforms connect to data warehouses and allow business users to build reports and dashboards without writing code. Common BI tools include Tableau, Power BI, Looker, and Streamlit.
+
+Visit the following resources to learn more:
+
+- [@roadmap@Visit the Dedicated BI Analyst Roadmap](https://roadmap.sh/bi-analyst)
+- [@article@What is business intelligence (BI)?](https://www.ibm.com/think/topics/business-intelligence)
+- [@article@Business intelligence: A complete overview](https://www.tableau.com/business-intelligence/what-is-business-intelligence)
+- [@video@What is business intelligence?](https://www.youtube.com/watch?v=l98-BcB3UIE)
@@ -0,0 +1,10 @@
+# CAP Theorem
+ 
+The CAP theorem states that a distributed system can provide at most two of three guarantees: Consistency, Availability, and Partition Tolerance. In practice, network partitions are unavoidable, so systems must choose between consistency and availability when a partition occurs. This trade-off shapes the design of distributed databases like Cassandra, DynamoDB, and HBase.
+
+Visit the following resources to learn more:
+
+- [@article@What is CAP Theorem?](https://www.bmc.com/blogs/cap-theorem/)
+- [@article@An Illustrated Proof of the CAP Theorem](https://mwhittaker.github.io/blog/an_illustrated_proof_of_the_cap_theorem/)
+- [@article@CAP Theorem and its applications in NoSQL Databases](https://www.ibm.com/uk-en/cloud/learn/cap-theorem)
+- [@video@What is CAP Theorem?](https://www.youtube.com/watch?v=_RbsFXWRZ10)
@@ -0,0 +1,9 @@
+# Cassandra
+ 
+Apache Cassandra is an open-source distributed wide-column database designed for high availability and linear scalability. It has no single point of failure and is optimized for fast writes across multiple data centers. Cassandra is used for time-series data, IoT workloads, and applications requiring continuous uptime.
+
+Visit the following resources to learn more:
+
+- [@official@Apache Cassandra](https://cassandra.apache.org/_/index.html)
+- [@article@Cassandra - Quick Guide](https://www.tutorialspoint.com/cassandra/cassandra_quick_guide.htm)
+- [@video@Apache Cassandra - Course for Beginners](https://www.youtube.com/watch?v=J-cSy5MeMOA)
@@ -0,0 +1,9 @@
+# Census
+
+Census is a reverse ETL platform that synchronizes data from a data warehouse to various business applications and SaaS apps like Salesforce and Hubspot. It's a crucial part of the modern data stack, enabling businesses to operationalize their data by making it available in the tools where teams work, like CRMs, marketing platforms, and more.
+
+Visit the following resources to learn more:
+
+- [@official@Census Documentation](https://developers.getcensus.com/getting-started/introduction)
+- [@article@A starter guide to reverse ETL with Census](https://www.getcensus.com/blog/starter-guide-for-first-time-census-users)
+- [@video@How to "Reverse ETL" with Census](https://www.youtube.com/watch?v=XkS7DQFHzbA)
@@ -0,0 +1,9 @@
+# Choosing the Right Technologies
+ 
+Selecting the right technology stack depends on data volume, team size, latency requirements, and budget. There is no universal best choice; a small startup may do well with a simple Postgres setup, while a large enterprise may need distributed processing and a cloud data warehouse. The decision involves evaluating trade-offs between cost, complexity, scalability, and maintainability.
+
+Visit the following resources to learn more:
+
+- [@book@Fundamentals of Data Engineering](https://www.oreilly.com/library/view/fundamentals-of-data/9781098108298/)
+- [@article@Build hybrid and multicloud architectures using Google Cloud](https://cloud.google.com/architecture/hybrid-multicloud-patterns)
+- [@article@The Unfulfilled Promise of Serverless](https://www.lastweekinaws.com/blog/the-unfulfilled-promise-of-serverless/)
@@ -0,0 +1,9 @@
+# CI/CD
+ 
+CI/CD (Continuous Integration and Continuous Delivery) is a set of practices and tools for automating the testing and deployment of code changes. In data engineering, CI/CD pipelines validate pipeline code, run tests, and deploy updates to production automatically. This reduces manual errors and accelerates the delivery of pipeline changes.
+
+Visit the following resources to learn more:
+
+- [@article@What is CI/CD? Continuous Integration and Continuous Delivery](https://www.guru99.com/continuous-integration.html)
+- [@article@Continuous Integration vs Delivery vs Deployment](https://www.guru99.com/continuous-integration-vs-delivery-vs-deployment.html)
+- [@article@CI/CD Pipeline: Learn with Example](https://www.guru99.com/ci-cd-pipeline.html)
@@ -0,0 +1,9 @@
+# CircleCI
+
+CircleCI is a CI/CD service that can be integrated with GitHub, BitBucket and GitLab repositories. The service that can be used as a SaaS offering or self-managed using your own resources.
+
+Visit the following resources to learn more:
+
+- [@official@CircleCI](https://circleci.com/)
+- [@official@CircleCI Documentation](https://circleci.com/docs)
+- [@official@Configuration Tutorial](https://circleci.com/docs/config-intro)
@@ -0,0 +1,8 @@
+# Cloud Architectures
+ 
+Cloud architectures describe how systems are designed to run on cloud infrastructure. Common patterns include multi-tier architectures, microservices, event-driven designs, and serverless functions. Good cloud architecture balances cost, reliability, scalability, and security.
+
+Visit the following resources to learn more:
+
+- [@article@What is cloud architecture? - Google](https://cloud.google.com/learn/what-is-cloud-architecture)
+- [@video@WWhat is Cloud Architecture and Common Models?](https://www.youtube.com/watch?v=zTP-bx495hU)
@@ -0,0 +1,9 @@
+# Cloud Computing
+ 
+Cloud computing refers to the delivery of computing resources, including servers, storage, databases, networking, and software, over the internet. Major cloud providers offer on-demand infrastructure that scales with usage and is billed per consumption. Cloud platforms are the dominant environment for modern data engineering work.
+
+Visit the following resources to learn more:
+
+- [@article@Cloud Computing - IBM](https://www.ibm.com/think/topics/cloud-computing)
+- [@article@What is Cloud Computing? - Azure](https://azure.microsoft.com/en-gb/resources/cloud-computing-dictionary/what-is-cloud-computing)
+- [@video@What is Cloud Computing? - Amazon Web Services](https://www.youtube.com/watch?v=mxT233EdY5c)
@@ -0,0 +1,9 @@
+# Cloud SQL (Database)
+
+Google Cloud SQL is a fully-managed, cost-effective and scalable database service that makes it easy to set-up, maintain, manage and administer MySQL, PostgreSQL, and SQL Server databases in the cloud. Hosted on Google Cloud Platform, Cloud SQL provides a database infrastructure for applications running anywhere.
+
+Visit the following resources to learn more:
+
+- [@course@Cloud SQL](https://www.cloudskillsboost.google/course_templates/701)
+- [@official@Cloud SQL](https://cloud.google.com/sql)
+- [@official@Cloud SQL overview](https://cloud.google.com/sql/docs/introduction)
@@ -0,0 +1,3 @@
+# Cluster Computing Basics
+ 
+Cluster computing refers to using a group of connected machines that work together as a single system to process data. It enables workloads that are too large or slow for a single machine by distributing computation across multiple nodes. Concepts like job scheduling, distributed file systems, and resource management are central to working with clusters.
@@ -0,0 +1,3 @@
+# Cluster Management Tools
+
+Cluster management software maximizes the work that a cluster of computers can perform. A cluster manager balances workload to reduce bottlenecks, monitors the health of the elements of the cluster, and manages failover when an element fails. A cluster manager can also help a system administrator to perform administration tasks on elements in the cluster.
@@ -0,0 +1,9 @@
+# Column
+ 
+Column-family databases (also called wide-column stores) organize data into rows and dynamic columns grouped into column families. They are optimized for read and write operations on large datasets spread across many machines. This model is well suited for time-series data, logging, and analytical workloads.
+
+Visit the following resources to learn more:
+
+- [@article@What are columnar databases? Here are 35 examples.](https://www.tinybird.co/blog-posts/what-is-a-columnar-database)
+- [@article@Columnar Databases](https://www.techtarget.com/searchdatamanagement/definition/columnar-database)
+- [@video@WWhat is a Columnar Database? (vs. Row-oriented Database)](https://www.youtube.com/watch?v=1MnvuNg33pA)
@@ -0,0 +1,9 @@
+# Compute Engine (Compute)
+ 
+Google Cloud Compute Engine provides virtual machine instances on Google's infrastructure. It supports custom machine types, preemptible VMs for cost savings, and integration with other Google Cloud services. Compute Engine is used for running custom workloads, data processing jobs, and services that require full control over the operating environment.
+
+Visit the following resources to learn more:
+
+- [@course@The Basics of Google Cloud Compute](https://www.cloudskillsboost.google/course_templates/754)
+- [@official@Compute Engine overview](https://cloud.google.com/compute/docs/overview)
+- [@video@WCompute Engine in a minute](https://www.youtube.com/watch?v=IuK4gQeHRcI)
@@ -0,0 +1,10 @@
+# Containers & Orchestration
+ 
+Containers package an application and its dependencies into a portable, isolated unit that runs consistently across environments. Container orchestration automates the deployment, scaling, and management of these containers across a cluster. Together, containers and orchestration form the foundation for running modern data workloads in cloud and hybrid environments.
+
+Visit the following resources to learn more:
+
+- [@article@What are Containers?](https://cloud.google.com/learn/what-are-containers)
+- [@article@Containers - The New Stack](https://thenewstack.io/category/containers/)
+- [@video@What are Containers?](https://www.youtube.com/playlist?list=PLawsLZMfND4nz-WDBZIj8-nbzGFD4S9oz)
+- [@video@Why You Need Data Orchestration](https://www.youtube.com/watch?v=ZtlS5-G-gng)
@@ -0,0 +1,9 @@
+# CosmosDB
+
+Azure Cosmos DB is a native No-SQL database service and vector database for working with the document data model. It can arbitrarily store native JSON documents with flexible schema. Data is indexed automatically and is available for query using a flavor of the SQL query language designed for JSON data. It also supports vector search. You can access the API using SDKs for popular frameworks such [as.NET](http://as.NET), Python, Java, and Node.js.
+
+Visit the following resources to learn more:
+
+- [@official@What are Containers?](https://azure.microsoft.com/en-us/products/cosmos-db#FAQ)
+- [@official@CAzure Cosmos DB - Database for the AI Era](https://learn.microsoft.com/en-us/azure/cosmos-db/introduction)
+- [@video@What is Azure Cosmos DB?](https://www.youtube.com/watch?v=hBY2YcaIOQM&)
@@ -0,0 +1,8 @@
+# CouchDB
+ 
+Apache CouchDB is an open-source document database that uses JSON for documents and HTTP as its API. It is designed for reliability and offline-first use cases, with a built-in replication protocol that syncs data between devices and servers. CouchDB is used in scenarios where data needs to be available and writable even without a network connection.
+
+Visit the following resources to learn more:
+
+- [@official@CouchDB Documentation](https://docs.couchdb.org/en/stable/intro/overview.html)
+- [@article@What is CouchDB?](https://www.ibm.com/think/topics/couchdb)
@@ -0,0 +1,10 @@
+# Data Analytics
+ 
+Data analytics is the process of examining datasets to draw conclusions and support decision-making. It covers a spectrum from descriptive analytics (what happened) to diagnostic (why it happened), predictive (what might happen), and prescriptive (what to do). Data engineers build the infrastructure that makes analytics possible by ensuring clean, accessible, and timely data.
+
+Visit the following resources to learn more:
+
+- [@course@Introduction to Data Analytics](https://www.coursera.org/learn/introduction-to-data-analytics)
+- [@article@The 4 Types of Data Analysis: Ultimate Guide](https://careerfoundry.com/en/blog/data-analytics/different-types-of-data-analysis/)
+- [@video@Descriptive vs Diagnostic vs Predictive vs Prescriptive Analytics: What's the Difference?](https://www.youtube.com/watch?v=QoEpC7jUb9k)
+- [@video@Types of Data Analytics](https://www.youtube.com/watch?v=lsZnSgxMwBA)
@@ -0,0 +1,7 @@
+# Data Collection Considerations
+ 
+When collecting data, engineers must account for reliability, latency, volume, and schema consistency. Other considerations include data privacy regulations, deduplication, and handling of missing or malformed records. Good collection design reduces problems downstream in the pipeline.
+
+Visit the following resources to learn more:
+
+- [@book@Fundamentals of Data Engineering](https://www.oreilly.com/library/view/fundamentals-of-data/9781098108298/)
@@ -0,0 +1,9 @@
+# Data Engineering Lifecycle
+ 
+The data engineering lifecycle describes the stages data moves through from creation to consumption. These stages typically include generation, ingestion, storage, transformation, and serving. Each stage has its own tools, failure modes, and design considerations. Understanding the full lifecycle helps engineers make better decisions about architecture and tooling.
+
+Visit the following resources to learn more:
+
+- [@book@Fundamentals of Data Engineering](https://www.oreilly.com/library/view/fundamentals-of-data/9781098108298/)
+- [@article@Data Engineering Lifecycle](https://medium.com/towards-data-engineering/data-engineering-lifecycle-d1e7ee81632e)
+- [@video@Getting Into Data Engineering](https://www.youtube.com/watch?v=hZu_87l62J4)
@@ -0,0 +1,9 @@
+# Data Engineering Lifecycle
+ 
+The data engineering lifecycle describes the stages data moves through from creation to consumption. These stages typically include generation, ingestion, storage, transformation, and serving. Each stage has its own tools, failure modes, and design considerations. Understanding the full lifecycle helps engineers make better decisions about architecture and tooling.
+
+Visit the following resources to learn more:
+
+- [@book@Fundamentals of Data Engineering](https://www.oreilly.com/library/view/fundamentals-of-data/9781098108298/)
+- [@article@Data Engineering Lifecycle](https://medium.com/towards-data-engineering/data-engineering-lifecycle-d1e7ee81632e)
+- [@video@Getting Into Data Engineering](https://www.youtube.com/watch?v=hZu_87l62J4)
@@ -0,0 +1,7 @@
+# Data Engineering vs Data Science
+
+Data engineering and data science are distinct but complementary roles within the field of data. Data engineering focuses on building and maintaining the infrastructure for data collection, storage, and processing, essentially creating the systems that make data available for downstream users. On the other hand, data science professionals, like data analysts and data scientists, uses that data to extract insights, build predictive models, and ultimately inform decision-making.
+
+Visit the following resources to learn more:
+
+- [@video@Should You Be a Data Scientist, Analyst or Engineer?](https://www.youtube.com/watch?v=dUnKYhripIE)
@@ -0,0 +1,9 @@
+# Data Fabric
+ 
+Data fabric is an architectural concept that aims to provide a unified layer for accessing and managing data across heterogeneous environments, including on-premises and multiple clouds. It uses metadata, automation, and integration patterns to connect disparate data sources. Data fabric focuses on making data discoverable and accessible without requiring it to be moved to a central location.
+
+Visit the following resources to learn more:
+
+- [@article@What is a data fabric?](http://ibm.com/think/topics/data-fabric)
+- [@article@Data Fabric defined](https://www.jamesserra.com/archive/2021/06/data-fabric-defined/)
+- [@article@How Data Fabric Can Optimize Data Delivery](https://www.gartner.com/en/data-analytics/topics/data-fabric)
@@ -0,0 +1,9 @@
+# Data Factory (ETL)
+
+Data Factory, most commonly referring to Microsoft's Azure Data Factory, is a cloud-based data integration service that allows you to create, schedule, and orchestrate workflows to move and transform data from various sources into a centralized location for analysis. It provides tools for building Extract, Transform, and Load (ETL) pipelines, enabling businesses to prepare data for analytics, business intelligence, and other data-driven initiatives without extensive coding, thanks to its visual, code-free interface and native connectors.
+
+Visit the following resources to learn more:
+
+- [@course@Microsoft Azure - Data Factory](https://www.coursera.org/learn/microsoft-azure---data-factory)
+- [@official@What is Azure Data Factory?](https://learn.microsoft.com/en-us/azure/data-factory/introduction)
+- [@official@Azure Data Factory Documentation](https://learn.microsoft.com/en-gb/azure/data-factory/)
@@ -0,0 +1,8 @@
+# Data Generation
+ 
+Data generation refers to how raw data is produced and originates in a system. Data can come from user interactions, application logs, IoT sensors, databases, APIs, and many other sources. Understanding where data comes from and how it is structured at the source is the starting point for any data pipeline design.
+
+Visit the following resources to learn more:
+
+- [@article@The Concept of Data Generation](https://www.marktechpost.com/2023/02/27/the-concept-of-data-generation/)
+- [@video@Analog vs. Digital](https://www.youtube.com/watch?v=zzvglgC5ut0)
@@ -0,0 +1,8 @@
+# Data Hub
+ 
+A data hub is a centralized platform that acts as an integration point for data flowing between multiple systems. Unlike a data warehouse, a data hub focuses on data movement and integration rather than storage for analytics. It often combines features of a message broker, metadata catalog, and integration layer.
+
+Visit the following resources to learn more:
+
+- [@article@Data hub](https://en.wikipedia.org/wiki/Data_hub)
+- [@article@What is a Data Hub? Definition, 7 Key Benefits & Why You Might Need One](https://www.cdata.com/blog/what-is-a-data-hub)
@@ -0,0 +1,8 @@
+# Data Ingestion
+
+Data ingestion is the third step in the data engineering lifecycle. It entails the process of collecting and importing data files from various sources into a database for storage, processing and analysis. The goal of data ingestion is to clean and store data in an accessible and consistent central repository to prepare it for use within the organization.
+
+Visit the following resources to learn more:
+
+- [@article@What is Data Ingestion?](https://www.ibm.com/think/topics/data-ingestion)
+- [@article@Data Ingestion](https://www.qlik.com/us/data-ingestion)
@@ -0,0 +1,8 @@
+# Data Interoperability
+
+Data interoperability is the ability of diverse systems and applications to access, exchange, and cooperatively use data in a coordinated and meaningful way, even across organizational boundaries. It ensures that data can flow freely, maintaining its integrity and context, allowing for improved efficiency, collaboration, and decision-making by breaking down data silos. Achieving data interoperability often relies on data standards, metadata, and common data elements to define how data is collected, formatted, and interpreted.
+
+Visit the following resources to learn more:
+
+- [@article@Data Interoperability](https://www.sciencedirect.com/topics/computer-science/data-interoperability)
+- [@article@What is Data Interoperability? – Exploring the Process and Benefits](https://www.codelessplatforms.com/blog/what-is-data-interoperability/)
@@ -0,0 +1,8 @@
+# Data Lake
+ 
+A data lake is a centralized storage repository that holds large amounts of raw data in its native format, including structured, semi-structured, and unstructured data. Unlike a data warehouse, a data lake does not enforce a schema on ingestion. Data is stored cheaply at scale and processed when needed, which enables flexibility for future analysis.
+
+Visit the following resources to learn more:
+
+- [@article@Data Lake Definition](https://azure.microsoft.com/en-gb/resources/cloud-computing-dictionary/what-is-a-data-lake)
+- [@video@What is a Data Lake?](https://www.youtube.com/watch?v=LxcH6z8TFpI)
@@ -0,0 +1,7 @@
+# Data Lineage
+
+**Data Lineage** refers to the life-cycle of data, including its origins, movements, characteristics and quality. It's a critical component in Data Engineering for tracking the journey of data through every process in a pipeline, from raw input to model output. Data lineage helps in maintaining transparency, ensuring compliance, and facilitating data debugging or tracing data related bugs. It provides a clear representation of data sources, transformations, and dependencies thereby aiding in audits, governance, or reproduction of machine learning models.
+
+Visit the following resources to learn more:
+
+- [@article@What is Data Lineage? - IBM](https://www.ibm.com/topics/data-lineage)
@@ -0,0 +1,9 @@
+# Data Mart
+
+A data mart is a subset of a data warehouse, focused on a specific business function or department. A data mart is streamlined for quicker querying and a more straightforward setup, catering to the specialized needs of a particular team, or function. Data marts only hold data relevant to a specific department or business unit, enabling quicker access to specific datasets, and simpler management
+
+Visit the following resources to learn more:
+
+- [@article@What is a Data Mart?](https://www.ibm.com/think/topics/data-mart)
+- [@article@WData Mart vs Data Warehouse: a Detailed Comparison](https://www.datacamp.com/blog/data-mart-vs-data-warehouse)
+- [@video@Data Lake VS Data Warehouse VS Data Marts](https://www.youtube.com/watch?v=w9-WoReNKHk)
@@ -0,0 +1,8 @@
+# Data Masking
+
+Data masking is a process that creates a copy of real data but replaces sensitive information with false but realistic-looking data, preserving the format and structure of the original data for non-production uses like software testing, training, and development. The goal is to protect confidential information and ensure compliance with data protection regulations by preventing unauthorized access to real sensitive data without compromising the usability of the data for other business functions.
+
+Visit the following resources to learn more:
+
+- [@article@Data masking](https://en.wikipedia.org/wiki/Data_masking)
+- [@article@What is data masking?](https://aws.amazon.com/what-is/data-masking/)
@@ -0,0 +1,8 @@
+# Data Mesh
+
+A data mesh is a modern approach to data architecture that shifts data management from a centralized model to a decentralized one. It emphasizes domain-oriented ownership, where data management aligns with specific business areas. This alignment makes data operations more scalable and flexible, leveraging the knowledge and expertise of those closest to the data. Data mesh is defined by four principles: data domains, data products, self-serve data platform, and federated computational governance.
+
+Visit the following resources to learn more:
+
+- [@article@What Is a Data Mesh? - AWS](https://aws.amazon.com/what-is/data-mesh)
+- [@video@Data Mesh Architecture](https://www.datamesh-architecture.com/)
@@ -0,0 +1,7 @@
+# Data Modelling Techniques
+ 
+Data modelling is the process of defining how data is structured and related within a storage system. Common techniques include entity-relationship (ER) modelling for transactional databases and dimensional modelling (star and snowflake schemas) for analytics. The choice of model affects query performance, flexibility, and how easy it is to evolve the schema over time.
+
+Visit the following resources to learn more:
+
+- [@article@7 data modeling techniques and concepts for business](https://www.techtarget.com/searchdatamanagement/tip/7-data-modeling-techniques-and-concepts-for-business)
@@ -0,0 +1,8 @@
+# Data Normalization
+ 
+Data normalization is the process of organizing a relational database to reduce redundancy and improve data integrity. It involves decomposing tables into smaller, related ones according to normal forms (1NF, 2NF, 3NF, etc.). Normalized schemas are easier to maintain but may require more joins when querying.
+
+Visit the following resources to learn more:
+
+- [@article@What is Normalization in DBMS (SQL)? 1NF, 2NF, 3NF, BCNF Database with Example](https://www.guru99.com/database-normalization.html)
+- [@video@Complete guide to Database Normalization in SQL](https://www.youtube.com/watch?v=rBPQ5fg_kiY)
@@ -0,0 +1,3 @@
+# Data Obfuscation
+
+Statistical data obfuscation involves altering the values of sensitive data in a way that preserves the statistical properties and relationships within the data. It ensures that the masked data maintains the overall distribution, patterns, and correlations of the original data for accurate statistical analysis. Statistical data obfuscation techniques include applying mathematical functions or perturbation algorithms to the data.
@@ -0,0 +1,8 @@
+# Data Pipelines
+
+Data pipelines are a series of automated processes that transport and transform data from various sources to a destination for analysis or storage. They typically involve steps like data extraction, cleaning, transformation, and loading (ETL) into databases, data lakes, or warehouses. Pipelines can handle batch or real-time data, ensuring that large-scale datasets are processed efficiently and consistently. They play a crucial role in ensuring data integrity and enabling businesses to derive insights from raw data for reporting, analytics, or machine learning.
+
+Visit the following resources to learn more:
+
+- [@article@What is a Data Pipeline? - IBM](https://www.ibm.com/topics/data-pipeline)
+- [@video@What are Data Pipelines?](https://www.youtube.com/watch?v=oKixNpz6jNo)
@@ -0,0 +1,3 @@
+# Data Quality
+ 
+Data quality refers to how well data meets the requirements for its intended use in terms of accuracy, completeness, consistency, timeliness, and validity. Poor data quality leads to incorrect analysis and poor decisions. Data engineers implement quality checks at ingestion and transformation stages to catch and prevent data issues.
@@ -0,0 +1,7 @@
+# Data Quality
+
+Data quality refers to the degree to which a dataset is accurate, complete, consistent, relevant, and timely, making it fit for its intended use. High-quality data is reliable and trustworthy, enabling better decision-making, accurate analysis, and effective strategies, while poor data quality can lead to flawed insights, wasted resources, and negative consequences for an organization.
+
+Visit the following resources to learn more:
+
+- [@article@What is Data Quality?](https://www.ibm.com/think/topics/data-quality)
@@ -0,0 +1,3 @@
+# Data Serving
+
+Data serving is the last step in the data engineering process. Once the data is stored in your data architectures and transformed into coherent and useful format, it's time for get value from it. Data serving refers to the different ways data is used by downstream applications and users to create value. There are many ways companies can extract value from data, including training machine learning models, BI Analytics, and reverse ETL.
@@ -0,0 +1,7 @@
+# Data Storage
+ 
+Data storage in the engineering lifecycle refers to where and how data is persisted after it is generated or ingested. The choice of storage system depends on access patterns, data volume, latency requirements, and cost. Options range from relational databases to object storage, data lakes, and columnar warehouses.
+
+Visit the following resources to learn more:
+
+- [@article@What is data storage?](https://www.ibm.com/think/topics/data-storage)
@@ -0,0 +1,10 @@
+# Data Structures and Algorithms
+ 
+Data structures and algorithms form the foundation for writing efficient code. This knowledge is relevant for data engineers when optimizing queries, designing storage schemas, and building processing logic that scales. Common topics include arrays, hash maps, trees, sorting, and complexity analysis.
+
+Visit the following resources to learn more:
+
+- [@roadmap@Visit the Dedicated DSA Roadmap](https://roadmap.sh/datastructures-and-algorithms)
+- [@article@Interview Questions about Data Structures](https://www.csharpstar.com/csharp-algorithms/)
+- [@video@Data Structures Illustrated](https://www.youtube.com/watch?v=9rhT3P1MDHk&list=PLkZYeFmDuaN2-KUIv-mvbjfKszIGJ4FaY)
+- [@video@Intro to Algorithms](https://www.youtube.com/watch?v=rL8X2mlNHPM)
@@ -0,0 +1,8 @@
+# Data Warehouse
+ 
+A data warehouse stores structured, processed data from operational systems, optimized for analytical queries. It typically uses columnar storage and is populated through ETL or ELT processes. Common cloud data warehouses include Google BigQuery, Snowflake, and Amazon Redshift.
+
+Visit the following resources to learn more:
+
+- [@article@What Is a Data Warehouse?](https://www.oracle.com/database/what-is-a-data-warehouse/)
+- [@video@What is a Data Warehouse?](https://www.youtube.com/watch?v=k4tK2ttdSDg)
@@ -0,0 +1,3 @@
+# Data Warehousing Architectures
+ 
+Data warehousing architectures describe how data is organized, stored, and accessed across a warehouse system. Common patterns include traditional ETL-based warehouses, cloud-native warehouses, data lakehouse architectures, and federated query systems. The choice of architecture affects cost, query performance, scalability, and how fresh the data available for analysis is.
@@ -0,0 +1,10 @@
+# Database Fundamentals
+ 
+Database fundamentals cover the core concepts that apply across most relational database systems: how data is organized into tables, how queries are executed, and how the database ensures consistency and durability. Topics include normalization, indexing, transactions, and query optimization. These concepts apply whether using PostgreSQL, MySQL, or any other relational system.
+
+Visit the following resources to learn more:
+
+- [@article@Oracle: What is a Database?](https://www.oracle.com/database/what-is-database/)
+- [@article@NoSQL Explained](https://www.mongodb.com/nosql-explained)
+- [@video@What is Relational Database](https://youtu.be/OqjJjpjDRLc)
+- [@video@How do NoSQL Databases work](https://www.youtube.com/watch?v=0buKQHokLK8)
@@ -0,0 +1,3 @@
+# Database
+
+A database is an organized, structured collection of electronic data that is stored, managed, and accessed via a computer system, usually controlled by a Database Management System (DBMS). Databases organize various types of data, such as words, numbers, images, and videos, allowing users to easily retrieve, update, and modify it for various purposes, from managing customer information to analyzing business processes.
@@ -0,0 +1,9 @@
+# Databricks Delta Lake
+
+Delta Lake is the optimized storage layer that provides the foundation for tables in a lakehouse on Databricks. Delta Lake is open source software that extends Parquet data files with a file-based transaction log for ACID transactions and scalable metadata handling. Delta Lake is fully compatible with Apache Spark APIs, and was developed for tight integration with Structured Streaming, allowing you to easily use a single copy of data for both batch and streaming operations and providing incremental processing at scale.
+
+Visit the following resources to learn more:
+
+- [@book@The Delta Lake Series — Fundamentals and Performance](https://www.databricks.com/resources/ebook/the-delta-lake-series-fundamentals-performance)
+- [@official@What is Delta Lake in Databricks?](https://docs.databricks.com/aws/en/delta)
+- [@video@Delta Lake](https://www.databricks.com/resources/demos/videos/lakehouse-platform/delta-lake)
@@ -0,0 +1,7 @@
+# Datadog
+
+Datadog is a monitoring and analytics platform for large-scale applications. It encompasses infrastructure monitoring, application performance monitoring, log management, and user-experience monitoring. Datadog aggregates data across your entire stack with 400+ integrations for troubleshooting, alerting, and graphing.
+
+Visit the following resources to learn more:
+
+- [@official@Datadog Documentation](https://docs.datadoghq.com/)
@@ -0,0 +1,9 @@
+# Dataflow
+
+Dataflow is a Google Cloud service that provides unified stream and batch data processing at scale. Typical use cases for Dataflow include Data movement,ETL processes, BI dashboarding, and applying ML in real time to streaming data.
+
+Visit the following resources to learn more:
+
+- [@official@Dataflow](https://cloud.google.com/products/dataflow)
+- [@article@Dataflow](https://en.wikipedia.org/wiki/Google_Cloud_Dataflow)
+- [@video@What is Google Dataflow](https://www.youtube.com/watch?v=KalJ0VuEM7s)
@@ -0,0 +1,8 @@
+# dbt
+
+dbt, also known as the data build tool, is designed to simplify the management of data warehouses and transform the data within. This is primarily the T, or transformation, within ELT (or sometimes ETL) processes. It allows for easy transition between data warehouse types, such as Snowflake, BigQuery, Postgres, or DuckDB. dbt also provides the ability to use SQL across teams of multiple users, simplifying interaction. In addition, dbt translates between SQL dialects as appropriate to connect to different data sources and warehouses.
+
+Visit the following resources to learn more:
+
+- [@course@dbt Official Courses](https://learn.getdbt.com/catalog)
+- [@official@dbt Documentation](https://docs.getdbt.com/docs/build/documentation)
@@ -0,0 +1,12 @@
+# Declarative vs Imperative
+
+When it comes to Infrastructure as Code (IaC), there are two fundamental styles: imperative and declarative.
+
+In **imperative IaC**, you specify a list of steps the IaC tool should follow to provision a new resource. You tell your IaC tool how to create each environment using a sequence of command imperatives. Imperative IaC can offer more flexibility as it allows you to dictate each step. However, this can result in increased complexity. Popular imperative IaC tools are Chef and Puppet
+
+In **declarative IaC**, you specify the name and properties of the infrastructure resources you wish to provision, and then the IaC tool figures out how to achieve that end result on its own. You declare to your IaC tool what you want, but not how to get there. Declarative IaC, while less flexible, tends to be simpler and more manageable. Terraform is the most popular declarative IaC tool
+
+Visit the following resources to learn more:
+
+- [@article@Infrastructure as Code: From Imperative to Declarative and Back Again](https://thenewstack.io/infrastructure-as-code-from-imperative-to-declarative-and-back-again/)
+- [@article@Declarative vs Imperative Programming for Infrastructure as Code (IaC)](https://www.copado.com/resources/blog/declarative-vs-imperative-programming-for-infrastructure-as-code-iac)
@@ -0,0 +1,7 @@
+# Distributed File Systems
+
+A Distributed File System (DFS) allows multiple computers to access and share files across a network as if they were stored on a single local machine. It distributes data across multiple servers, enhancing accessibility and data redundancy. This enables users to access files from various locations and devices, promoting collaboration and data availability.
+
+Visit the following resources to learn more:
+
+- [@article@What is a Distributed File System (DFS)? A Complete Guide](http://starwindsoftware.com/blog/what-is-a-distributed-file-system-dfs-a-complete-guide/)
@@ -0,0 +1,9 @@
+# Distributed Systems
+
+A distributed system is a collection of independent computers that communicate and coordinate to appear as a single unified system. They are widely used for scalability, fault tolerance, and high availability in modern applications. However, they bring challenges such as synchronization, consistency trade-offs (CAP theorem), concurrency, and network latency.
+
+Visit the following resources to learn more:
+
+- [@article@Introduction to Distributed Systems](https://www.freecodecamp.org/news/a-thorough-introduction-to-distributed-systems-3b91562c9b3c/)
+- [@article@Distributed Systems Guide](https://www.baeldung.com/cs/distributed-systems-guide)
+- [@video@Distributed Systems Explained](https://www.youtube.com/watch?v=IJWwfMyPu1c)
@@ -0,0 +1,10 @@
+# Docker
+ 
+Docker is the most widely used platform for building, shipping, and running containers. It packages code and its dependencies into a lightweight, portable image that runs the same in any environment. Data engineers use Docker to containerize pipeline code, ensure reproducible environments, and simplify deployment.
+
+Visit the following resources to learn more:
+
+- [@roadmap@Visit Dedicated Docker Roadmap](https://roadmap.sh/docker)
+- [@official@Docker Documentation](https://docs.docker.com/)
+- [@video@Docker Tutorial](https://www.youtube.com/watch?v=RqTEHSBrYFw)
+- [@video@Docker simplified in 55 seconds](https://youtu.be/vP_4DlOH1G4)
@@ -0,0 +1,8 @@
+# Document
+
+\*\*Document Databases are a type of No-SQL databases that store data in JSON, BSON, or XML formats, allowing for flexible, semi-structured and hierarchical data structures. These databases are characterized by their dynamic schema, scalability through distribution, and ability to intuitively map data models to application code. Popular examples include MongoDB, which allows for easy storage and retrieval of varied data types without requiring a rigid, predefined schema.
+
+Visit the following resources to learn more:
+
+- [@article@What is a Document Database?](https://www.mongodb.com/resources/basics/databases/document-databases)
+- [@article@Document-oriented database](https://en.wikipedia.org/wiki/Document-oriented_database)
@@ -0,0 +1,7 @@
+# DynamoDB
+ 
+Amazon DynamoDB is a fully managed key-value and document database service on AWS. It provides single-digit millisecond performance at any scale and handles replication and scaling automatically. DynamoDB is commonly used for applications that require predictable performance and high availability without database administration.
+
+Visit the following resources to learn more:
+
+- [@official@Amazon DynamoDB](https://aws.amazon.com/dynamodb/)
@@ -0,0 +1,9 @@
+# ECPA
+ 
+The Electronic Communications Privacy Act (ECPA) is a US federal law that regulates government access to electronic communications and stored data. It sets rules for when law enforcement can intercept communications or compel disclosure of stored data from service providers. Data engineers working with communication data must be aware of ECPA requirements when designing storage and access controls.
+
+Visit the following resources to learn more:
+
+- [@official@California Consumer Privacy Act (CCPA)](https://oag.ca.gov/privacy/ccpa)
+- [@article@What is the California Consumer Privacy Act (CCPA)?](https://www.ibm.com/think/topics/ccpa-compliance)
+- [@video@What is the California Consumer Privacy Act? | CCPA Explained?](https://www.youtube.com/watch?v=dpzsAgrDAO4)
@@ -0,0 +1,10 @@
+# Elasticsearch
+ 
+Elasticsearch is a distributed search and analytics engine built on Apache Lucene. It is designed for full-text search, log analysis, and real-time data exploration. Elasticsearch is commonly used as the backend for search features in applications and as a centralized store for log and event data, often alongside Kibana.
+
+Visit the following resources to learn more:
+
+- [@roadmap@Visit the Dedicated Elasticsearch Roadmap](https://roadmap.sh/elasticsearch)
+- [@official@Elasticsearch Website](https://www.elastic.co/elasticsearch/)
+- [@official@Elasticsearch Documentation](https://www.elastic.co/guide/index.html)
+- [@video@What is Elasticsearch](https://www.youtube.com/watch?v=ZP0NmfyfsoM)
@@ -0,0 +1,8 @@
+# Encryption
+
+Encryption is used to protect data from being stolen, changed, or compromised and works by scrambling data into a secret code that can only be unlocked with a unique digital key. Encrypted data can be protected while at rest on computers or in transit between them, or while being processed, regardless of whether those computers are located on-premises or are remote cloud servers.
+
+Visit the following resources to learn more:
+
+- [@article@What is Encryption?](https://cloud.google.com/learn/what-is-encryption)
+- [@video@What is Encryption?](https://www.youtube.com/watch?v=9chKCUQ8_VQ)
@@ -0,0 +1,8 @@
+# End-to-End Testing
+
+End-to-end or (E2E) testing is a form of testing used to assert your entire application works as expected from start to finish or "end-to-end". E2E testing differs from unit testing in that it is completely decoupled from the underlying implementation details of your code. It is typically used to validate an application in a way that mimics the way a user would interact with it.
+
+Visit the following resources to learn more:
+
+- [@article@End to End Testing](https://microsoft.github.io/code-with-engineering-playbook/automated-testing/e2e-testing/)
+- [@article@End to End Testing: Importance, Process, Best Practices & Frameworks](https://testgrid.io/blog/end-to-end-testing-a-detailed-guide/)
@@ -0,0 +1,7 @@
+# Environmental Management
+
+Environmental management, or Environment as Code (EaC) takes the concept of Infrastructure as Code (IaC) one step further. EaC applies DevOps principles to manage and automate entire software environments—including infrastructure, applications, and configurations—using code, making them reproducible, versionable, and reliable. It extends IaC by focusing not just on the underlying servers and networks but on the complete, connected system of services and applications that run on top of it. This approach helps increase efficiency, speeds up deployments, and provides a consistent, auditable process for creating and managing development, testing, and production environments.
+
+Visit the following resources to learn more:
+
+- [@article@EWhat Is Environment as Code (EaaC)?](https://www.bunnyshell.com/blog/what-is-environment-as-code-eaac/)
@@ -0,0 +1,9 @@
+# ETL vs Reverse ETL
+ 
+ETL (Extract, Transform, Load) moves data from operational systems into a data warehouse for analysis. Reverse ETL goes in the opposite direction, syncing processed data from the warehouse back into operational tools like Salesforce, HubSpot, or Intercom. The two patterns are complementary and together form a complete data activation workflow.
+
+Visit the following resources to learn more:
+
+- [@article@What is ETL?](https://www.snowflake.com/guides/what-etl)
+- [@article@ETL vs Reverse ETL vs Data Activation](https://airbyte.com/data-engineering-resources/etl-vs-reverse-etl-vs-data-activation)
+- [@article@ETL vs Reverse ETL: An Overview, Key Differences, & Use Cases](https://portable.io/learn/etl-vs-reverse-etl)
@@ -0,0 +1,12 @@
+# EU AI Act
+
+The Artificial Intelligence Act of the European Union, also known as the EU AI Act, is a comprehensive regulatory framework that is established to ensure safety and that fundamental human rights are upheld in the use of AI technologies. It governs the development and/or use of AI in the European Union. The act takes a risk-based approach to regulation, applying different rules to AI systems according to the risk they pose.
+
+Considered the world's first comprehensive regulatory framework for AI, the EU AI Act prohibits some AI uses outright and implements strict governance, risk management and transparency requirements for others.
+
+Visit the following resources to learn more:
+
+- [@official@The EU AI Act Explorer](https://artificialintelligenceact.eu/ai-act-explorer/)
+- [@article@AI Act - European Commission](https://digital-strategy.ec.europa.eu/en/policies/regulatory-framework-ai)
+- [@article@Artificial Intelligence Act](https://en.wikipedia.org/wiki/Artificial_Intelligence_Act)
+- [@video@The EU AI Act Explained](https://www.youtube.com/watch?v=s_rxOnCt3HQ)
@@ -0,0 +1,3 @@
+# Extract Data
+
+The first step in ETL processes involves extract data from data sources to a staging area. Data can come in various types and formats, from SQL or NoSQL databases and plan text to image and video files.
@@ -0,0 +1,8 @@
+# Functional Testing
+
+Functional testing is a type of software testing that validates the software system against the functional requirements/specifications. The purpose of functional tests is to test each function of the software application by providing appropriate input and verifying the output against the functional requirements.
+
+Visit the following resources to learn more:
+
+- [@article@What is Functional Testing? Types & Examples](https://www.guru99.com/functional-testing.html)
+- [@article@Functional Testing : A Detailed Guide](https://www.browserstack.com/guide/functional-testing)
@@ -0,0 +1,8 @@
+# GDPR
+ 
+GDPR (General Data Protection Regulation) is a data privacy law enacted by the European Union that governs how personal data of EU residents is collected, stored, processed, and shared. It grants individuals rights over their data, including the right to access, correct, and delete it. Data engineers must design systems that support these rights and comply with GDPR requirements such as data minimization and purpose limitation.
+
+Visit the following resources to learn more:
+
+- [@official@GDPR](https://gdpr-info.eu/)
+- [@article@What is GDPR Compliance in Web Application and API Security?](https://probely.com/blog/what-is-gdpr-compliance-in-web-application-and-api-security/)
@@ -0,0 +1,10 @@
+# Git and GitHub
+ 
+Git is a distributed version control system that tracks changes to code over time. GitHub is a platform built on top of Git that adds collaboration features like pull requests, code review, and CI/CD integrations. Data engineers use Git to manage pipeline code, infrastructure configurations, and shared scripts across teams.
+
+Visit the following resources to learn more:
+
+- [@roadmap@Visit Dedicated Git & GitHub Roadmap](https://roadmap.sh/git-github)
+- [@article@Git by Example - Learn Version Control with Bite-sized Lessons](https://antonz.org/git-by-example/)
+- [@video@Git & GitHub Crash Course For Beginners](https://www.youtube.com/watch?v=SWYqp7iY_Tc)
+- [@course@Why use Git? (Interactive Lesson)](https://inter-git.com/lessons/introduction)
@@ -0,0 +1,7 @@
+# GitHub Actions
+ 
+GitHub Actions is a CI/CD platform built into GitHub. It allows developers to define automated workflows as YAML files that trigger on code events like pushes and pull requests. GitHub Actions is widely used to run tests, lint code, build Docker images, and deploy data pipelines.
+
+Visit the following resources to learn more:
+
+- [@official@GitHub Actions Documentation](https://docs.github.com/en/actions)
@@ -0,0 +1,10 @@
+# GitLab CI
+
+GitLab offers a CI/CD service that can be used as a SaaS offering or self-managed using your own resources. You can use GitLab CI with any GitLab hosted repository, or any BitBucket Cloud or GitHub repository in the GitLab Premium self-managed, GitLab Premium SaaS and higher tiers.
+
+Visit the following resources to learn more:
+
+- [@official@GitLab Documentation](https://docs.gitlab.com/)
+- [@official@Get Started with GitLab CI](https://docs.gitlab.com/ee/ci/quick_start/)
+- [@official@Learn GitLab Tutorials](https://docs.gitlab.com/ee/tutorials/)
+- [@official@GitLab CI/CD Examples](https://docs.gitlab.com/ee/ci/examples/)
@@ -0,0 +1,7 @@
+# Glue (ETL)
+ 
+AWS Glue is a fully managed ETL service that automates the discovery, cataloging, and transformation of data. It includes a data catalog for storing metadata, a job scheduler, and a serverless Spark environment for running transformations. Glue is commonly used to move and transform data between S3, Redshift, and other AWS services.
+
+Visit the following resources to learn more:
+
+- [@official@Amazon RDS](https://aws.amazon.com/rds/)
@@ -0,0 +1,10 @@
+# Go
+ 
+Go is a compiled language developed by Google, known for its simplicity, fast execution, and strong concurrency model. In data engineering, it is used to build lightweight, high-throughput services and tools. Its performance characteristics make it a good fit for data pipeline components where latency and resource efficiency matter.
+
+Visit the following resources to learn more:
+
+- [@roadmap@Visit Dedicated Go Roadmap](https://roadmap.sh/golang)
+- [@official@Go Documentation](https://go.dev/doc/)
+- [@article@Go, the Programming Language of the Cloud](https://thenewstack.io/go-the-programming-language-of-the-cloud/)
+- [@video@Go Programming â€“ Golang Course with Bonus Projects](https://www.youtube.com/watch?v=un6ZyFkqFKo)
@@ -0,0 +1,9 @@
+# Google BigQuery
+
+BigQuery is a managed, serverless data warehouse product by Google, offering scalable analysis over large quantities of data. It is a Platform as a Service (PaaS) that supports querying using a dialect of SQL. BigQuery is NoOps, meaning there is no infrastructure to manage and you don't need a database administrator. BigQuery lets you focus on analyzing data to find meaningful insights while using familiar SQL and built-in machine learning at unmatched price-performance.
+
+Visit the following resources to learn more:
+
+- [@official@BigQuery overview](https://cloud.google.com/bigquery/docs/introduction)
+- [@official@From data warehouse to autonomous data and AI platform](https://cloud.google.com/bigquery)
+- [@video@What is BigQuery?](https://www.youtube.com/watch?v=d3MDxC_iuaw)
@@ -0,0 +1,8 @@
+# Google Cloud GKE
+ 
+Google Kubernetes Engine (GKE) is Google Cloud's managed Kubernetes service. It handles cluster provisioning, upgrades, and scaling automatically, reducing the operational burden of running Kubernetes. GKE is used to run containerized data workloads on Google Cloud infrastructure.
+
+Visit the following resources to learn more:
+
+- [@official@GKE](https://cloud.google.com/kubernetes-engine)
+- [@video@What is Google Kubernetes Engine (GKE)?](https://www.youtube.com/watch?v=Rl5M1CzgEH4)
@@ -0,0 +1,9 @@
+# Google Cloud Storage
+
+Google Cloud Storage (GCS) is a scalable, secure, and durable object storage service within Google Cloud Platform (GCP) designed for storing and retrieving unstructured data of any type or size. It allows users to store data in "buckets" and access it through APIs, web interfaces, or command-line tools for applications, backups, media hosting, and big data analytics. GCS offers different storage classes to optimize costs based on data access frequency, strong security with encryption, and high availability through redundant data storage across multiple locations.
+
+Visit the following resources to learn more:
+
+- [@article@Cloud Storage](https://cloud.google.com/storage)
+- [@article@Google Cloud Storage](https://en.wikipedia.org/wiki/Google_Cloud_Storage)
+- [@article@Cloud Storage in a minute](https://www.youtube.com/watch?v=wNOs3LlsH6k)
@@ -0,0 +1,10 @@
+# Google Deployment  Mgr.
+
+Google Cloud Deployment Manager is an infrastructure deployment service that automates the creation and management of Google Cloud resources. It provides users with flexible template and configuration files to create deployments that have a variety of Google Cloud services, such as Cloud Storage, Compute Engine, and Cloud SQL, configured to work together.
+
+Important, Google Deployment Manager will reach end of support on 31 December 2025. An alternative to this tool is **Google Infrastructure Manager**. Infrastructure Manager (Infra Manager) automates the deployment and management of Google Cloud infrastructure resources using Terraform. Infra Manager allows users to deploy programmatically to Google Cloud, allowing to use this service rather than maintaining a different toolchain to work with Terraform on Google Cloud.
+
+Visit the following resources to learn more:
+
+- [@official@Infrastructure Manager Overview](https://cloud.google.com/infrastructure-manager/docs/overview)
+- [@official@Google Cloud Deployment Manager documentation](https://cloud.google.com/deployment-manager/docs)
--- a/Show More
+++ b/Show More