- The Databricks Platform
- Databricks - Data Engineer Associate Certification
- Useful Resources
- Useful Reference Links
Databricks is one of the most popular emerging platforms in the data industry. It enables teams to collaborate effectively on big data and machine learning projects. With its integration of Apache Spark, it offers powerful tools for data analysis, making it easier for organizations to derive insights from large datasets.
It simplifies the creation of modern data warehouses that enable organizations to provide self-service analytics and machine learning across their global data with enterprise-grade performance and governance.
The Databricks Platform
The core of the Databricks platform consists of open-source tools wrapped into an enterprise-friendly package delivered as a service on the cloud.
Apache Spark
The core of Databricks is Apache Spark, an open-source big data processing engine.
It allows for large-scale distributed computing on large datasets. Even when working with large datasets, the engine is extremely flexible and scalable. It unifies both batch and streaming data, incorporates many different processing models, and supports SQL. These characteristics make it much easier to use.
DeltaLake
DeltaLake is an open-source storage layer that runs on top of data lakes to deliver greater reliability, security, and performance.
It is fully compatible with Apache Spark APIs and runs both streaming and batch operations.
MLFlow
MLFlow is an open-source tool that manages the lifecycle of machine learning pipelines and applications.
It enables data scientists to run multiple experiments, deploy ML models in different ways (integrating them into existing apps or creating new services), train the algorithms, and so on, all using a wide variety of tooling.
Databricks - Data Engineer Associate Certification
The Databricks Fundamentals Learning Plan provides a conceptual introduction to what Databricks is and the components that make up the Platform.
Similar to the Google Cloud - Professional Data Engineer certification I was expecting questions on AI/ML but the exam focussed solely on core data engineering concepts and tasks within Databricks.
Here are some of the topics I encountered in the Exam
-
Delta Lake format : main features & advantages, tech questions about how it is stored, Definition of Bronze/Silver/Gold according to Databricks, time-travel, vacuum etc…
-
Delta Live Tables : Table Properties, Incremental Live Tables…
-
Data Quality : How is it handled in the platform, Alerting…
-
Structured Streaming : checkpoints, hopping, write ahead logs, autoloader…
-
Unity / SQL Table Access Management : GRANT SELECT / GRANT VIEW etc… What can an Admin do ?
-
Jobs and Production Pipelines - Types of clusters, Task and Dependency management, Debugging failures…
Useful Resources
- Databricks Academy Training: Start with the Databricks Lakehouse Fundamentals - its a free ondemand training offered by databricks, which covers Data Lakehouse Platform, Architecture, Security and Workloads.
- Exam Guide: Thoroughly review the Databricks Exam Guide, as there is typically one question for each key concept.
- Practice Exam: Resources like Practice exams on Udemy, offering questions that closely resemble the actual certification exam.
Useful Reference Links
- Lakehouse Fundamentals
- DatabricksSQL
- Delta Live Tables
- Git Repos
- Time Travel
- Vacuum Command
- Delta Optimize
- Notebooks
- Auto Loader
- Copy Into