How to protect databases from external threats?

    Network Security (First Line of Defense)

    Network security is the first layer of database protection and focuses on limiting exposure to external threats. Access should be restricted using firewalls and IP allowlists, with connectivity routed through private networks, VPNs, or private endpoints. Databases should never be publicly accessible, and network segmentation using subnets and security groups helps isolate systems and reduce the attack surface.

    Identity & Access Control

    Implement strong identity and access control by enforcing least privilege through RBAC, enabling multi-factor authentication, and eliminating shared or root accounts. Regularly rotate credentials and use managed identities or IAM roles to reduce security risks, improve accountability, and ensure secure access to systems and resources.

    Encryption Everywhere

    Implement encryption across all layers by enabling encryption at rest (such as TDE or disk encryption) and encryption in transit using TLS/SSL to protect data from unauthorized access. Manage encryption keys securely through key vaults or hardware security modules (HSM) and never store passwords or keys in code to reduce the risk of data exposure and security breaches.

    Database Hardening

    Database hardening involves disabling default accounts, unused ports, and nonessential features to reduce attack surfaces, while enforcing strong password policies, account lockout rules, audit logging, and data masking. These controls strengthen security, improve compliance, and help protect sensitive data from unauthorized access and misuse.

    Patch & Update Regularly

    Regular patching is critical to database security and stability. Organizations must routinely apply operating system and database updates, monitor vendor security advisories, and promptly upgrade unsupported versions to reduce vulnerabilities, prevent exploits, and ensure ongoing performance, compliance, and system reliability.

    Monitoring & Threat Detection

    Monitoring and threat detection should be implemented to continuously protect databases from security risks. This includes enabling intrusion detection, configuring SQL firewall alerts, and monitoring database activity for suspicious behavior. Native security tools such as Defender, Guard, or built-in threat detection features should be used to identify threats early and respond quickly to potential security incidents.

    Backup & Disaster Recovery

    Implement a robust backup and disaster recovery strategy by automating regular backups, encrypting all backup data, and validating recovery through routine restore testing. Use geo-redundant storage to ensure data availability and business continuity in the event of system failures, disasters, or regional outages.

    Access Logging & Auditing

    Enable centralized access logging and auditing to monitor login attempts, privilege changes, and failed queries in real time. Store logs securely in a centralized system to support investigation, compliance, and threat detection, ensuring all access activity is traceable, protected, and available for audit and incident response purposes.

    Application Security

    To ensure strong application security, organizations must prevent SQL injection and avoid hard-coded secrets within code. Using parameterized queries helps protect databases from malicious queries, while validating all user inputs ensures only trusted data is processed, reducing the risk of unauthorized access and data breaches.

    Compliance & Governance

    Compliance and governance ensure databases meet regulatory and security standards by following frameworks such as HIPAA, PCI, GDPR, and SOC 2. This includes classifying data based on sensitivity, enforcing retention policies for proper data handling, and conducting regular audits to maintain compliance, reduce risk, and ensure accountability.

    Why do we need SQL databases in Fabric if Fabric can already connect to on-prem SQL and Azure SQL?

    Fabric supports several built-in analytical data stores, such as SQL databases, Warehouses, Lakehouses, and Eventhouses, each optimized for different workloads. Microsoft Learn

    • SQL databases in Fabric provide a T-SQL relational engine well suited for structured data at moderate volumes (GB–TB), supporting transactional consistency, high-frequency updates, stored procedures, referential integrity, and granular access controls (object-, column-, row-level). Microsoft Learn
    • They offer low-latency queries and highly selective lookups, ideal for operational or metadata workloads and scenarios where quick, transactional-style interactions are needed. Microsoft Learn
    • Because of automatic integration with Fabric’s underlying lake storage (OneLake), Fabric SQL databases are natively part of the unified Fabric ecosystem — meaning data can be shared across compute engines (e.g., Spark, Warehouse), used in cross-database queries, or consumed by Power BI semantic models / Direct Lake. Microsoft Learn
    • In contrast, traditional on-prem or Azure SQL databases are optimized for transactional workloads, but are not integrated with Fabric’s broader data lake architecture, and may not deliver the same flexibility, scalability, or governance when supporting analytics, BI, and mixed workloads.

    In short: Fabric SQL databases exist because they combine relational (transactional + structured) capabilities with seamless integration into the unified, scalable, lake-backed Fabric analytics platform — enabling small/moderate data workloads, quick lookups and updates, and smooth interoperability with Warehouses, Lakehouses, Spark, governance, and BI tools

    Data Governance Policies

    Data Governance:

    Data governance establishes clear rules for data quality, security, access, and lifecycle management. It ensures data is accurate, compliant, and trustworthy by enforcing standards, ownership, lineage, and controls across the organization.

    Data Governance Policy:

    A data governance policy is a formal set of rules and guidelines that defines how an organization manages, protects, accesses, and uses its data. It establishes standards for data quality, security, privacy, ownership, lifecycle, and compliance, ensuring data is reliable, consistent, and handled responsibly across all systems and teams.

    Steps to Establish and Enforce Data Governance Policies:

    Step 1: Define Governance Objectives and Regulatory Requirements

    • Identify which regulations apply
      • Global & International Regulations:
        • ISO/IEC 27001 – Information security management
        • ISO/IEC 38505 – Governance of data
        • ISO/IEC 27701 – Privacy information management
        • OECD Privacy Guidelines
        • Basel III – Financial data and risk reporting (banking)
        • PCI-DSS – Payment card data security
        • ITIL – Data/IT service governance framework
      • United States Regulations:
        • HIPAA / HITECH – Health data privacy and security
        • SOX (Sarbanes-Oxley) – Financial data accuracy and controls
        • GLBA (Gramm-Leach-Bliley Act) – Protection of customer financial data
        • FISMA – Federal data security standards
        • FedRAMP – Cloud security controls for federal data
      • European Union Regulations:
        • GDPR – General Data Protection Regulation
        • DORA – Digital Operational Resilience Act (financial sector)
        • NIS2 Directive – Network & information systems security
        • EIDAS – Digital identity and trust services
    • Determine governance goals: data integrity, lineage, access control, retention, audit trails, data quality.

    Outcome: A regulatory requirements matrix mapping each requirement to controls you must implement.

    Step 2: Establish a Data Governance Framework

    • Choose a governance model (e.g., centralized, federated, or hybrid).
    • Define governance roles:
      • Data Owners (business accountability)
      • Data Stewards (quality + documentation)
      • Data Custodians (IT + engineering)
      • Governance Council (cross-functional oversight)
    • Set formal processes for approvals, standards, and escalations.

    Outcome: A governance operating model that is repeatable and auditable.

    Step 3: Implement Data Classification and Metadata Standards

    • Define data categories (PII, PHI, Confidential, Restricted, Public).
    • Create metadata standards covering:
      • schema naming
      • data definitions
      • business rules
      • quality thresholds
    • Use tools like:
      • Microsoft Purview (Fabric/ADF-native lineage & classification)
      • Azure Data Catalog
      • Collibra / Alation (if in enterprise environment)

    Outcome: All sensitive data is discoverable and classified for Regulation controls.

    Step 4. Design Technical Controls in Azure/Fabric

    Access & Security

    • Implement RBAC and ABAC in Azure AD.
    • Enforce least privilege with managed identities.
    • Use column-level and row-level security for PII (GDPR).

    Data Protection

    • Encryption:
      • At rest (Azure-managed keys or BYOK/HSM for SOX)
      • In transit (TLS 1.2+)
    • Masking:
      • Dynamic Data Masking (SQL/Fabric Warehouse)
      • Tokenization (for PCI/PHI)

    Monitoring

    • Enable:
      • Azure Monitor
      • Defender for Cloud
      • Purview Data Loss Prevention
      • Audit logs for DML/DDL changes (SOX requirement)

    Outcome: Technical enforcement is embedded in the data pipeline and platform.

    Databricks: Roles overview

    • Account Administrator
      • Manages the entire Databricks account across all workspaces.
      • Handles billing, workspace creation, user provisioning, and global settings like SSO and identity management.
    • Metastore Administrator
      • Controls the Unity Catalog metastore — manages catalogs, schemas, permissions, and data governance policies for all data assets.
      • Ensures secure and consistent access control across workspaces.
    • Workspace Administrator
      • Manages settings within a specific workspace, including cluster policies, user/group permissions, workspace configuration, and job or notebook access.
    • Owner (Object Owner)
      • The user who creates or owns a resource (e.g., table, cluster, notebook).
      • Has full control over it — can read, modify, delete, or grant permissions to others.

    Databricks: Control Plane vs Data Plane

    Control Plane

    The Control Plane is where Databricks manages and orchestrates your workspace and infrastructure.
    It contains all the metadata and configuration required to run workloads.

    Key responsibilities:

    • Stores notebooks, jobs, cluster configurations, and workspace settings.
    • Handles authentication, authorization, and user management.
    • Manages job scheduling, monitoring, and logs.
    • Orchestrates cluster creation but does not access your actual data.

    Essentially, the Control Plane is Databricks-managed and ensures your workspace runs smoothly, without hosting your business data.

    Data Plane

    The Data Plane is where your data is processed and stored.
    This is typically located within your cloud account (Azure, AWS, or GCP), providing data isolation and security.

    Key responsibilities:

    • Executes Spark jobs, notebooks, and SQL queries.
    • Stores and reads data from Delta Lake, ADLS, S3, or other storage.
    • Performs data transformations and ML workloads.
    • Keeps data within your organization’s security boundary.

    The Data Plane is customer-controlled, ensuring compliance and governance, especially in regulated industries.

    Summary

    PlaneManaged byContainsKey Purpose
    Control PlaneDatabricks
    Metadata, notebooks, job configs
    Management & Orchestration
    Data PlaneCustomerActual data, execution environmentProcessing & Storage

    Databricks Key Components & Concepts

    Data Lake

    A centralized storage repository (often on ADLS, S3, or GCS) that holds raw structured and unstructured data at scale. It serves as the foundation for analytics, ML, and data warehousing on Databricks.

    Delta Lake

    An open-source storage layer that brings ACID transactions, schema enforcement, time travel, and data reliability to the Data Lake. It transforms a basic lake into a Lakehouse, combining the best of data lakes and data warehouses.

    Unity Catalog

    Databricks’ unified governance layer providing centralized data access control, lineage, and auditing across workspaces. It manages permissions at the table, column, and data asset levels — ensuring compliance and consistent data governance.

    Data Intelligence

    The Databricks Data Intelligence Platform integrates AI, ML, and analytics on top of unified data — enabling intelligent data discovery, semantic understanding, and AI-assisted development through tools like Databricks Assistant.

    Roles in Databricks Ecosystem

    • Data Engineer – Builds and optimizes ETL/ELT pipelines, manages Delta tables, and ensures data quality/performance.
    • Data Analyst – Uses SQL Analytics and notebooks for querying, dashboarding, and reporting.
    • Data Scientist – Develops ML models using Python, R, or MLflow on shared datasets.
    • Data Steward / Admin – Manages Unity Catalog, governance, and access control.
    • ML Engineer / Architect – Designs scalable ML pipelines and integrates AI workloads within the Lakehouse.

    Core Data Processing Challenges

    1. Scalability and Performance

    • Managing data volume growth without compromising processing speed.
    • Optimizing Spark, SQL, or pipeline jobs for parallel execution and partitioning.
    • Handling variable workloads and real-time streaming data efficiently.

    2. Data Integration Complexity

    • Ingesting and harmonizing data from diverse structured and unstructured sources.
    • Maintaining schema consistency and preventing drift across ETL/ELT pipelines.
    • Managing dependencies and orchestration across multiple systems (ADF, Fabric Pipelines, Databricks, etc.).

    3. Data Quality and Governance

    • Ensuring data accuracy, completeness, and consistency at scale.
    • Implementing automated validation, cleansing, and lineage tracking.
    • Meeting compliance requirements (HIPAA, GDPR) while maintaining agility.

    4. Cost Optimization

    • Controlling compute, storage, and data movement costs in cloud environments.
    • Managing resource autoscaling and workload scheduling efficiently.
    • Balancing on-demand vs. reserved capacity for predictable budgets.

    5. Security and Access Control

    • Enforcing role-based access, encryption, and secure data sharing across environments.
    • Integrating identity and access management (IAM) between cloud and on-prem systems.
    • Monitoring for compliance breaches and unauthorized access.

    6. Operational Complexity and Automation

    • Managing complex pipelines across multiple tools and environments.
    • Reducing manual interventions through automation (CI/CD, PowerShell, Logic Apps).
    • Ensuring reliable monitoring, alerting, and failure recovery.

    7. Vendor Lock-In and Interoperability

    • Dependence on proprietary tools or formats that limit portability.
    • Difficulty migrating workloads across cloud providers or open frameworks.
    • Limited integration between legacy systems and modern Lakehouse architectures.