Azure Updates - Latest from Azure Charts

Migrating Azure Data Factory pipelines to Fabric directly from your Fabric workspace

Wed, 10 Jun 2026 21:00:00 Z

Migrating Azure Data Factory (ADF) pipelines to Fabric has traditionally started in the ADF portal—running assessments and then switching to Fabric to complete the process. This approach continues to support scenarios where upfront readiness insights are important.
Update Type: Announcement, Services: Microsoft Fabric, Data Factory, Categories:

Orchestration in Fabric Data Factory: Build 2026 recap

Thu, 04 Jun 2026 16:00:00 Z

Microsoft Fabric Data Factory's Build 2026 release reshapes what pipelines can do — moving beyond data movement & process orchestration into the people and processes around your data. New Approval activities bring human decision-making directly into pipeline flows, while conditional retries deliver smarter resilience against transient failures. Materialized Lake View and SQL endpoint refresh activities keep analytics in lockstep with orchestration, a modern canvas makes complex pipelines easier to navigate, and expanded Variable Libraries now cover connections, items, and Airflow for seamless promotion across dev, test, and prod. Plus, in-workspace ADF migration makes getting started faster than ever.
Update Type: Announcement, Services: Microsoft Fabric, Data Factory, Categories:

Build 2026: From data to intelligence—Faster with Fabric Data Factory

Tue, 02 Jun 2026 19:00:00 Z

At Microsoft Build 2026, we're focused on what every developer and data team already knows: the most ambitious AI applications are only as good as the data behind them. Fabric Data Factory is the integration foundation of Microsoft Fabric—built to help builders move data with confidence, ground their AI on trusted sources, and ship intelligent experiences faster.
Update Type: Announcement, Services: Microsoft Fabric, Data Factory, Categories:

Outbound access protection for Data Factory

Wed, 22 Apr 2026 11:00:00 Z

Co-author: Abhishek Narain Workspace outbound access protection (OAP) is widely accessible for Data Factory workloads—including Pipelines, Copy Job, and Dataflows—as well as for Mirrored Databases such as Mirrored SQL Database and Mirrored Snowflake. Key benefits Enhanced outbound security: By leveraging OAP rules, organizations can ensure that the Data Factory items from the protected workspace can … Continue reading “Outbound access protection for Data Factory (Generally Available)”
Update Type: GA, Services: Microsoft Fabric, SQL Database, Data Factory, Categories: Security

Answers to common questions about Fabric Data Factory

Thu, 16 Apr 2026 09:00:00 Z

As the Data Integration Customer Advisory Team (CAT) lead, I spent a lot of time talking to customers at the recent FabCon/SQLCon about Fabric Data Factory, and I came away with a clear picture of what’s on customers’ minds when it comes to the future of data integration. Many of the same questions came up … Continue reading “Answers to common questions about Fabric Data Factory”
Update Type: Announcement, Services: Microsoft Fabric, Data Factory, Categories:

Announcing the latest innovations in Fabric Data Factory: Apache Airflow jobs and pipelines

Tue, 31 Mar 2026 11:30:00 Z

Introduction The world of data integration is rapidly evolving, and staying up to date with the latest technologies is crucial for organizations seeking to make the most of their data assets. Available now are the newest innovations in Fabric Data Factory pipelines and Apache Airflow job orchestration, designed to empower data engineers, architects, and analytics … Continue reading “Announcing the latest innovations in Fabric Data Factory: Apache Airflow jobs and pipelines”
Update Type: Announcement, Services: Data Factory, Categories:

New migration experience from Azure Data Factory to Fabric

Thu, 26 Mar 2026 08:00:00 Z

As more organizations standardize Microsoft Fabric for end-to-end analytics, upgrading existing Azure Data Factory (ADF) workloads is a key step in modernization.
Update Type: Preview, Services: Microsoft Fabric, Data Factory, Categories:

Higher Performance with Copy job in Fabric Data Factory Auto Partitioning

Wed, 25 Mar 2026 08:30:00 Z

Copy job is the go-to solution in Microsoft Fabric Data Factory for simplified data movement across multiple clouds. With native support for bulk copy, incremental copy, and change data capture (CDC) replication, it can handle a wide range of movement scenarios through an intuitive, easy-to-use experience.
Update Type: Preview, Services: Microsoft Fabric, Data Factory, Categories:

Simplifying data movement across multiple Clouds with richer CDC in Copy job in Fabric Data Factory Oracle source, Fabric Data Warehouse sink and SCD Type 2

Wed, 25 Mar 2026 08:00:00 Z

Outstanding connectivity for data movement in Fabric Data Factory

Tue, 24 Mar 2026 13:00:00 Z

In today’s data landscape, success depends not just on powerful analytics, but on how seamlessly and reliably you can move data across systems.
Update Type: Announcement, Services: Data Factory, Categories:

Incremental copy gets more flexible—New watermark column types in Copy job in Fabric Data Factory

Tue, 24 Mar 2026 11:00:00 Z

Copy job is the go-to solution in Microsoft Fabric Data Factory for simplified data movement across multiple clouds. With native support for bulk copy, incremental copy, and change data capture (CDC) replication, it can handle a wide range of movement scenarios through an intuitive, easy-to-use experience.
Update Type: GA, Services: Microsoft Fabric, Data Factory, Categories:

Modernizing pipelines: New activities and innovations in Fabric Data Factory pipelines

Fri, 20 Mar 2026 11:45:00 Z

If you haven’t already, check out Arun Ulag’s hero blog “FabCon and SQLCon 2026: Unifying databases and Fabric on a single, complete platform” for a complete look at all of our FabCon and SQLCon announcements across both Fabric and our database offerings. Introducing the Lakehouse utility suite in Fabric pipelines (Preview) A healthier, more reliable … Continue reading “Modernizing pipelines: New activities and innovations in Fabric Data Factory pipelines”
Update Type: Announcement, Services: Data Factory, Categories:

Audit columns in Copy job in Fabric Data Factory—every row is traceable for data lineage and compliance

Thu, 19 Mar 2026 10:00:00 Z

Copy job is the go-to solution in Microsoft Fabric Data Factory for simplified data movement across multiple clouds. With native support for bulk copy, incremental copy, and change data capture (CDC) replication, it can handle a wide range of movement scenarios through an intuitive, easy-to-use experience.
Update Type: Announcement, Services: Microsoft Fabric, Data Factory, Categories: Compliance

Fabric Data Factory at FabCon Atlanta: Built for modern data integration

Wed, 18 Mar 2026 05:49:18 Z

If you haven’t already, check out Arun Ulag’s hero blog “FabCon and SQLCon 2026: Unifying databases and Fabric on a single, complete platform” for a complete look at all of our FabCon and SQLCon announcements across both Fabric and our database offerings. At FabCon Atlanta, we’re doubling down on a simple truth: every successful analytics, … Continue reading “Fabric Data Factory at FabCon Atlanta: Built for modern data integration”
Update Type: Announcement, Services: Data Factory, Categories:

From Azure Synapse and Azure Data Factory to Microsoft Fabric: The Next-Gen Analytics Leap

Wed, 18 Mar 2026 05:49:01 Z

If you haven’t already, check out Arun Ulag’s hero blog “FabCon and SQLCon 2026: Unifying databases and Fabric on a single, complete platform” for a complete look at all of our FabCon and SQLCon announcements across both Fabric and our database offerings. Coauthor: Bogdan Crivat Across industries, organizations are rethinking their analytics foundation. The era … Continue reading “From Azure Synapse and Azure Data Factory to Microsoft Fabric: The Next-Gen Analytics Leap”
Update Type: Announcement, Services: Synapse Analytics, Microsoft Fabric, Data Factory, Categories:

Workspace Outbound Access Protection for Data Factory and OneLake Shortcuts

Thu, 29 Jan 2026 15:00:00 Z

We announced Outbound Access Protection for Spark (Generally Available) and recently extended it to support SQL Endpoint and Warehouse. Now, Pipelines, Copy job, Dataflows, OneLake Shortcuts as well as Mirrored Databases (such as Mirrored SQL Database, Mirrored Snowflake) support Workspace level Outbound Access Protection (Preview). Key Benefits What to expect with Outbound access protection (OAP) … Continue reading “Workspace Outbound Access Protection for Data Factory and OneLake Shortcuts (Preview)”
Update Type: Preview, Services: SQL Database, Data Factory, Categories:

Assessing Your Azure Data Factory for Migration to Fabric Data Factory

Wed, 10 Dec 2025 16:30:00 Z

As organizations modernize their data integration workflows, moving from Azure Data Factory (ADF) to Fabric Data Factory is an important milestone. Before starting the migration, it’s crucial to determine which pipelines and activities are ready to move, which require modifications, and which are not yet supported in Fabric. The built-in migration assessment tool in ADF … Continue reading “Assessing Your Azure Data Factory for Migration to Fabric Data Factory”
Update Type: Announcement, Services: Data Factory, Categories:

Announcing Copy Job Activity in Data Factory Pipeline

Mon, 24 Nov 2025 11:00:00 Z

This milestone marks a major step forward in unifying and simplifying data movement experiences across Data Factory. With Copy Job Activity, users can now enjoy the simplicity and speed of Copy Job while leveraging the orchestration power and flexibility of Data Factory pipelines. What is the Copy job Activity Copy Job Activity allows you to … Continue reading “Announcing Copy Job Activity in Data Factory Pipeline (Generally Available)”
Update Type: GA, Services: Data Factory, Categories:

Igniting Your Pipelines: New Data Factory Features Announced at Ignite

Wed, 19 Nov 2025 11:30:00 Z

The Microsoft Ignite conference gives us a chance to announce several new and exciting features in Microsoft’s Data Factory data integration service in Microsoft Fabric. Pipelines represent the code-free data orchestration feature in Fabric while Airflow Jobs represent the code-first approach to data pipelines called ‘DAGs’. These amazing new features will empower you to take … Continue reading “Igniting Your Pipelines: New Data Factory Features Announced at Ignite”
Update Type: Announcement, Services: Microsoft Fabric, Data Factory, Categories: Features, Services

Advancing Data Integration: Innovation in Data Factory in MS Fabric at Ignite 2025

Tue, 18 Nov 2025 08:00:00 Z

Microsoft Fabric isn’t just another data platform—it’s a game-changer, delivering enterprise-grade data integration at scale. At its foundation is Microsoft OneLake, a unified data lake that enables zero-copy and zero-ETL strategies—eliminating redundant data movement and complex pipelines. This approach helps organizations break down silos and maximize the value of their data, whether it resides in … Continue reading “Advancing Data Integration: Innovation in Data Factory in MS Fabric at Ignite 2025”
Update Type: Announcement, Services: Microsoft Fabric, Data Factory, Categories:

Simplifying Data Ingestion with Copy job – Copy data across tenants using Copy job in Fabric Data Factory

Mon, 20 Oct 2025 12:00:00 Z

Copy job is the go-to solution in Microsoft Fabric Data Factory for simplified data movement, whether you’re moving data across clouds, from on-premises systems, or between services. With native support for multiple delivery styles, including bulk copy, incremental copy, and change data capture (CDC) replication, Copy job offers the flexibility to handle a wide range … Continue reading “Simplifying Data Ingestion with Copy job – Copy data across tenants using Copy job in Fabric Data Factory “
Update Type: Announcement, Services: Microsoft Fabric, Data Factory, Categories: Services

Azure Data Factory, SQL Managed Instance, and SSIS Implementation

Mon, 06 Oct 2025 21:02:00 Z

Enabling the ADF Instance Create an Azure Data Factory Follow the official Microsoft Learn guide to set up your ADF instance. Required Roles You must have Contributor, Owner, or Admi...
Update Type: Announcement, Services: Data Factory, Categories:

Mission-Critical Data Integration: What’s New in Fabric Data Factory

Thu, 02 Oct 2025 17:00:00 Z

Why Mission-Critical Features Matter In the era of AI and cloud-scale analytics, organizations need a data integration platform that is secure, resilient, and ready for the most demanding workloads. Fabric Data Factory is engineered to deliver just that—enabling enterprises to unify, protect, and orchestrate their data across clouds, firewalls, and hybrid environments. Security, Isolation, and … Continue reading “Mission-Critical Data Integration: What’s New in Fabric Data Factory”
Update Type: Announcement, Services: Data Factory, Categories: Features, Security

Unlocking Seamless Data Integration with the Latest Fabric Data Factory Connector Innovations

Thu, 02 Oct 2025 09:00:00 Z

In the world of modern analytics, connectivity is everything. Your data ecosystem is only as powerful as your ability to connect to diverse sources, move data securely, and keep it fresh for business insights. That’s why Microsoft Fabric has been doubling down on Data Factory connectors—expanding coverage, improving performance, and enabling new enterprise-grade integration scenarios. … Continue reading “Unlocking Seamless Data Integration with the Latest Fabric Data Factory Connector Innovations”
Update Type: Announcement, Services: Microsoft Fabric, Data Factory, Categories:

AI-powered development with Fabric Data Factory – ingest, transform and understand your data with Copilot

Wed, 01 Oct 2025 08:00:00 Z

The new Copilot capabilities of Fabric Data Factory to efficiently ingest your data, easily apply transformation with natural language and understand your data with Copilot explanation. In September 2024, we made Copilot for Data Factory generally available, new features are rolling out to make your data experience even more powerful and intuitive. In this blog, discover … Continue reading “AI-powered development with Fabric Data Factory – ingest, transform and understand your data with Copilot”
Update Type: Announcement, Services: Data Factory, Categories: Features

Cross-Cloud Data Movement with Best-in-Class Connectivity: What’s New and What’s Next in Fabric Data Factory

Mon, 29 Sep 2025 09:00:00 Z

Microsoft Fabric is redefining the modern, AI‑ready data estate—bringing data, governance, and compute together on a unified SaaS platform built for mission‑critical workloads. By converging analytics, security, and operations on OneLake, Fabric accelerates time‑to‑value with consistent governance. Trusted by thousands of customers across industries, Fabric turns fragmented data into a strategic advantage for any business. … Continue reading “Cross-Cloud Data Movement with Best-in-Class Connectivity: What’s New and What’s Next in Fabric Data Factory”
Update Type: Announcement, Services: Microsoft Fabric, Data Factory, Categories: Security

New Innovations for Fabric Data Factory Orchestration at Fabric Conference Europe 2025

Thu, 25 Sep 2025 09:00:00 Z

Unveiling the Future of Data Integration and Pipelines at FabCon Europe 2025
Update Type: Announcement, Services: Data Factory, Categories:

Unify your data estate for the era of AI with Fabric Data Factory

Tue, 16 Sep 2025 01:00:00 Z

Microsoft Fabric provides industry leading data integration capabilities that are unique in the market. Fabric is built on OneLake, which provides data integration at the root, with shortcuts and mirroring that enable Zero-ETL approaches to data unification. Fabric Data Factory provides the industry’s largest, most widely adopted data integration capability offered as a single, cohesive … Continue reading “Unify your data estate for the era of AI with Fabric Data Factory”
Update Type: Announcement, Services: Microsoft Fabric, Data Factory, Categories:

Upsert and Script Activity in Azure Data Factory and Azure Synapse Analytics for Azure Database for PostgreSQL

Mon, 11 Aug 2025 15:15:03 Z

Announcing the general availability of the Upsert method and Script activity support in Azure Data Factory and Azure Synapse Analytics when using Azure Database for PostgreSQL. This enhancement enables you to perform efficient, scalable, and declarative d
Update Type: GA, Services: Synapse Analytics, Database for PostgreSQL, Data Factory, Categories: Features

Call Function App from Azure Data Factory with Managed Identity Authentication

Wed, 23 Jul 2025 04:28:00 Z

Integrating Azure Function Apps into your Azure Data Factory (ADF) workflows is a common practice. To enhance security beyond the use of function API keys, leveraging managed identity authentication ...
Update Type: Announcement, Services: Azure Functions, Data Factory, Categories:

Azure Database for PostgreSQL - Azure Data Factory managed identity

Thu, 05 Jun 2025 15:15:26 Z

Announcing that you can now use user-assigned managed identity and system-assigned managed identity Entra authentication to connect your Azure Database for PostgreSQL instance with Azure Synapse Analytics and Azure Data Factory. This feature enhances secu
Update Type: GA, Services: Synapse Analytics, Database for PostgreSQL, Data Factory, Categories: Features

Azure Data Factory Item Mounting (Generally Available)

Tue, 27 May 2025 15:00:49 Z

We’re excited to announce the General Availability (GA) of the Azure Data Factory (Mounting) feature in Microsoft Fabric! This powerful capability allows you to seamlessly connect your existing Azure Data Factory (ADF) pipelines to Fabric workspaces, eliminating the need to manually rebuild or migrate them. What’s New with GA? Once your ADF instance is linked to a Fabric workspace, you … Continue reading “Azure Data Factory Item Mounting (Generally Available)”
Update Type: GA, Services: Microsoft Fabric, Data Factory, Categories:

Azure Database for PostgreSQL - Azure Data Factory service principal authentication

Wed, 23 Apr 2025 16:30:16 Z

We are excited to announce that you can now use service principal Entra authentication to connect your Azure Database for PostgreSQL instance with Azure Synapse Analytics and Azure Data Factory. This feature enhances security and simplifies the authentica
Update Type: GA, Services: Synapse Analytics, Database for PostgreSQL, Data Factory, Categories: Features, Services

Recap of Data Factory Announcements at Fabric Conference US 2025

Thu, 10 Apr 2025 09:30:00 Z

We had such an exciting week for Fabric during the Fabric Conference US, filled with several product announcements and sneak previews of upcoming new features. Thanks to all of you who participated in the conference, either in person or by being part of the many virtual conversations through blogs, Community forums, social media and other … Continue reading “Recap of Data Factory Announcements at Fabric Conference US 2025”
Update Type: Announcement, Services: Data Factory, Categories: Features

AI-powered development with Data Factory Microsoft Fabric

Fri, 04 Apr 2025 12:00:11 Z

In today’s data-driven landscape, organizations are constantly seeking ways to streamline their data integration processes, enhance productivity, and democratize access to powerful data engineering capabilities with Copilot for Data Factory. At Microsoft, we’re committed to empowering data engineers and analysts with intelligent tools that reduce complexity and accelerate development. We’re excited to share the latest … Continue reading “AI-powered development with Data Factory Microsoft Fabric”
Update Type: Announcement, Services: Microsoft Fabric, Data Factory, Categories:

DataOps in Fabric Data Factory

Thu, 03 Apr 2025 16:30:00 Z

We’ve made several significant updates to our Fabric Data Factory artifacts stories, including Continuous Integration/Continuous Deployment (CI/CD) and APIs support! These updates are designed to automate the integration, testing, and deployment of code changes, ensuring efficient and reliable development. In Fabric Data Factory, we currently support two key features in collaboration with the Application Lifecycle … Continue reading “DataOps in Fabric Data Factory”
Update Type: Announcement, Services: Data Factory, Categories: Features

Best-in-class connectivity and data movement with Data Factory in Fabric

Thu, 03 Apr 2025 16:00:00 Z

In the fast-evolving data integration landscape, Data Factory continues to enhance the existing connectors to provide high-throughput data ingestion experience with no-code, low-code experience. With a focus on improving connector efficiency and expanding capabilities, recent updates bring significant advancements to a number of connectors. These improvements focus on: Latest innovations 1. Lakehouse connector now supports … Continue reading “Best-in-class connectivity and data movement with Data Factory in Fabric”
Update Type: Announcement, Services: Data Factory, Categories:

Fabric Data Factory: What’s New and Latest Roadmap

Mon, 31 Mar 2025 07:45:00 Z

Fabric Data Factory enables every organization to tackle the complexities of ingesting data, applying data transformations, and orchestrating these different data-related activities, needed to support every organization’s data integration needs, and delivering modern data management architectures. Thank you to all customers and partners who shared their Ideas, and suggestions, as we work together on shaping … Continue reading “Fabric Data Factory: What’s New and Latest Roadmap”
Update Type: Announcement, Services: Data Factory, Categories: Management

Simplify data transformation and management with Copilot for Data Factory

Mon, 24 Mar 2025 10:17:26 Z

The process of extracting, transforming, and loading (ETL) data is important for turning raw data into actionable insights. ETL allows data to be collected from various sources, cleansed and formatted into a standard structure, and then loaded into a data warehouse for analysis. This process ensures that data is accurate, consistent, and ready for business … Continue reading “Simplify data transformation and management with Copilot for Data Factory”
Update Type: Announcement, Services: Data Factory, Categories: Management

Azure Data Factory – Azure Database for PostgreSQL Connector Version 2.0

Wed, 12 Mar 2025 18:45:01 Z

We are excited to announce the release of the new Azure Data Factory – Azure Database for PostgreSQL connector. This updated connector now supports the latest TLS version 1.3 and multiple SSL modes for enhanced authentication. To ensure the best performan
Update Type: GA, Services: Cosmos DB, Database for PostgreSQL, Data Factory, Categories: Features

Enhancing data quality with Copilot for Data Factory

Tue, 21 Jan 2025 10:00:00 Z

In today’s world, cleaning and transforming data is incredibly important. Clean data means removing errors, duplicates, and irrelevant information to ensure its correct and complete, making the data more trustworthy for analysis. Transforming data means changing it into a format that is easy to use and analyze. Together, these steps improve data quality, leading to … Continue reading “Enhancing data quality with Copilot for Data Factory”
Update Type: Announcement, Services: Data Factory, Categories:

Microsoft Fabric Runtime 1.1 End of Support

Fri, 17 Jan 2025 14:00:13 Z

End of support for Microsoft Fabric Runtime 1.1 was announced July 12, 2024. We recommend that you upgrade your Fabric workspace and environments to use Runtime 1.3 (Apache Spark 3.5 and Delta Lake 3.2). This latest Fabric Runtime 1.3 also offers a Native
Update Type: Deprecation, Services: Synapse Analytics, Power BI Embedded, Microsoft Fabric, Data Factory, Categories: Retirements

Data Factory Announcements at Ignite 2024 Recap

Wed, 04 Dec 2024 07:00:00 Z

A couple of weeks ago we had such an exciting week for Fabric during the Ignite Conference, filled with several product announcements and sneak previews of upcoming new features. Thanks to all of you who participated in the conference, either in person or by being part of the many virtual conversations through blogs, Community forums, … Continue reading “Data Factory Announcements at Ignite 2024 Recap”
Update Type: Announcement, Services: Data Factory, Categories: Features

Fabric Data Factory now supports writing data in Iceberg format via Azure Data Lake Storage Gen2 Connector in Data pipeline

Fri, 22 Nov 2024 08:00:00 Z

We’ve made a significant enhancement in Fabric Data Factory: Data pipeline can now write data in Iceberg format via the Azure Data Lake Storage (ADLS) Gen2 connector! This addition provides a powerful new option for users who need to manage and optimize large datasets with a high level of flexibility, reliability, and performance. Iceberg format … Continue reading “Fabric Data Factory now supports writing data in Iceberg format via Azure Data Lake Storage Gen2 Connector in Data pipeline”
Update Type: Announcement, Services: Data Lake Storage, Data Factory, Categories:

Boosting Data Ingestion in Data Factory: Continuous Innovations in Performance Optimization

Fri, 22 Nov 2024 07:00:00 Z

At Fabric Data Factory, we’re committed to delivering continuous improvements to optimize data ingestion performance and streamline the user experience. Our latest technical innovations enhance performance across various aspects of Data Factory, from handling binary data more effectively to delivering faster data preview and navigation. Here’s a closer look at how these advancements are transforming … Continue reading “Boosting Data Ingestion in Data Factory: Continuous Innovations in Performance Optimization”
Update Type: Announcement, Services: Data Factory, Categories:

Boosting Data Ingestion in Data Factory: Continuous Innovations in Performance Optimization

Fri, 22 Nov 2024 07:00:00 Z

Best-of-class Connectivity in Fabric Data Factory

Thu, 21 Nov 2024 11:00:00 Z

Fabric Data Factory now features an expanded lineup of connectors and enhanced capabilities. Fabric SQL Database Connector Both Data Pipeline and Dataflow Gen 2 now natively support the Fabric SQL Database connector as both a source and sink. Now, you can directly connect to your Fabric SQL databases, allowing for smooth data movement and transformation across platforms without additional setup. Designed … Continue reading “Best-of-class Connectivity in Fabric Data Factory”
Update Type: Announcement, Services: SQL Database, Data Factory, Categories: Features

Efficiently build and maintain your Data pipelines with Copilot for Data Factory: new capabilities and experiences

Tue, 19 Nov 2024 08:00:00 Z

The new Data pipeline capabilities in Copilot for Data Factory are now available in preview. These features function as an AI expert to help users build, troubleshoot, and maintain data pipelines. What can new capabilities in Copilot for Data Factory do for you? Let’s start the AI-powered journey with Copilot for Data Factory. After creating … Continue reading “Efficiently build and maintain your Data pipelines with Copilot for Data Factory: new capabilities and experiences”
Update Type: Announcement, Services: Data Factory, Categories: Features

New Features for Fabric Data Factory Pipelines Announced at Ignite

Wed, 13 Nov 2024 09:30:00 Z

Table and partition refreshes added to Semantic Model Refresh activity One of the most popular features that we built in Fabric Data Factory came from our customer patterns that we observed being used in ADF and from our community. That is the Semantic Model Refresh activity. After first releasing this pipeline activity, we heard your … Continue reading “New Features for Fabric Data Factory Pipelines Announced at Ignite”
Update Type: Announcement, Services: Data Factory, Categories: Features

Migrating Azure Data Factory’s Snowflake Connector from Legacy to latest V2

Tue, 22 Oct 2024 16:35:38 Z

Recently we announced the new updated V2 of our Popular Snowflake Connector in Azure Data Factory. The new connector promises better performance and security. The new V2 Connector supports the Pipelines and Data Flows and comes with the new Go-based Snowflake driver. Differences between the Legacy and V2 Connector The V2 connector comes with enhancement in terms of performance and security. The following are the major differences between legacy and V2 Connector. Snowflake (V2) Snowflake (legacy) ODBC Driver Go Driver Support Basic and Key pair authentication. Support Basic authentication. Script parameters are not supported in Script activity currently. As an alternative, utilize dynamic expressions for script parameters. Support script parameters in Script activity. Support BigDecimal in Lookup activity. The NUMBER type, as defined in Snowflake, will be displayed as a string in Lookup activity. BigDecimal is not supported in Lookup activity. Connection Parameters include Account, Warehouse, Database, Schema, and Role Connection Parameter is Connection String Additional Connection Properties are not supported Additional Connection Properties are Supported Only Password / Keypair keys can be stored in KeyVault Passwords and Connection Strings can be stored in KeyVault The V2 version offers several enhancements over the legacy version including Autoscaling, Multi-Availability, Static IP, support for enhanced Storage Integration and upcoming features like Iceberg support. Find more information on the Connector Docs. Deprecation of Legacy Connector The Snowflake Legacy Connector will have the End of Support by Oct 31st, 2024, in end of support, the connector is considered as deprecated, and no longer supported. Customers are advised to upgrade the Connector to the new V2 Connector to continue using the connector and experience the new enhanced features. Read the official announcement to know more. Migra
Update Type: Announcement, Services: Data Factory, Categories: Features, Security

We're retiring Memory Optimized Data Flows in Azure Data Factory - use General Purpose for better performance at a lower price.

Fri, 29 Mar 2024 00:00:00 Z

We're retiring Memory Optimized Data Flows in Azure Data Factory
Update Type: Deprecation, Services: Data Factory, Categories: Retirements

Azure Data Factory CI-CD using YAML template

Thu, 17 Mar 2022 10:54:09 Z

Purpose of this Blog Defined the end to end Git integration and Deployment flow using YAML template from a project implementation perspective This can be easily implemented as you just need to include this YAML template and do a few changes as this has been successfully implemented for Deployments across some projects. Developer Workflow(CI/CD) Integration of Code from Data Factory UI(Continuous Integration) 1. A sandbox Data Factory is created for development of data pipelines with Datasets and Linked Services.The Data Factory is configured with Azure Dev-ops Git.(Collaboration and publish branch) and the root folder where the data factory code is committed 2. A feature branch is created based on the main/collaboration branch for development. The branch in the Data Factory UI is changed to feature branch. 3. After the developer tests the pipelines and is satisfied with the changes , a pull request is raised to mergethe changes with the main/collaboration branch 4. After the PR gets approved by the concerned product leads, the changes made in feature branch aremerged into main branch. 5. The branch is changed to main branch in Data Factory UI . The changes are published to main branch byclicking on publish button. 6. The changes are reflected in ARM template located in adf_publish branch Note: If some parameters are missing in the arm template follow the steps below : I. then we should go to manage in ADF UI II. click on the ARM template III. click on Edit parameter configuration in the DataFactory UI and include parameters that were missing in the ARM template in the json configuration file. IV. We need to include this json configuration file in the main branch at root folder configured in Git configuration, let us consider we keep it in 'src/DataFactory/{DataFactoryName}/arm-template-parameters-definition.json' - Follow this documentation on how to use the use custom parameters with the Resource Manager
Update Type: Announcement, Services: Data Factory, Categories: Services

Azure Data Factory Mapping Data Flows are now generally available

Mon, 07 Oct 2019 00:00:00 -0700

In today’s data-driven world, big data processing is a critical task for every organization. To unlock transformational insights and embrace a data-driven culture, companies need tools to help them easily integrate and transform data at scale, without requiring specialized skills.
Update Type: Announcement, Services: Data Factory, Categories:

Azure Data Factory new capabilities are now generally available

Wed, 27 Jun 2018 00:00:00 -0700

Microsoft is excited to announce the general availability of new Azure Data Factory (ADF) features that will make data integration in the cloud easier than ever before with a browser-based drag and drop user experience.
Update Type: Announcement, Services: Data Factory, Categories:

New Azure Data Factory self-paced hands-on lab for UI

Wed, 31 Jan 2018 00:00:00 -0800

Announcing a new self-paced hands-on Lab for the Azure Data Factory UI to visually build scale-out data pipelines in the cloud using ADF V2.
Update Type: Announcement, Services: Data Factory, Categories:

Azure Data Factory offers SAP HANA and Business Warehouse data integration

Tue, 21 Mar 2017 00:00:00 -0700

Today, we are excited to announce that Azure Data Factory newly enable loading data from SAP HANA and SAP Business Warehouse (BW) into various Azure data stores for advanced analytics and reporting.
Update Type: Announcement, Services: Data Factory, Categories: