Azure Updates - Latest from Azure Charts

Azure HDInsight – Enterprise Security Package

Wed, 16 Jul 2025 11:00:34 Z

On July 31, 2026, Support for Enterprise Security Package on Azure HDInsight will end, clusters running ESP on HDInsight will start facing service disruptions, as they would be deleted and associated data will be lost. Required Action To avoid potent
Update Type: Deprecation, Services: HDInsight, Microsoft Fabric, Kubernetes Service, Categories: Security, Services, Retirements

Upgrade your Azure HDInsight clusters with TLS versions 1 dot 1 and below by March 31, 2025

Wed, 29 Jan 2025 13:15:34 Z

Customers that have Azure HDInsight cluster is using TLS version 1.1 or lower are requested to upgrade to clusters have TLS version 1.2.  To enhance security and provide best-in-class encryption for your data, we'll require all connections to Azure HDInsi
Update Type: Deprecation, Services: HDInsight, Kubernetes Service, HDInsight on AKS, Categories: Retirements

New Zealand North region added to Azure HDInsight

Fri, 20 Dec 2024 14:15:55 Z

HDInsight is now generally available in New Zealand North. Azure HDInsight is a managed, full-spectrum, open-source analytics service in the cloud for enterprises. You can use open-source frameworks such as Hadoop, Apache Spark, Apache Hive, LLAP, Apache
Update Type: GA, Services: HDInsight, Kubernetes Service, HDInsight on AKS, Categories: Features, Regions & Datacenters

Azure HDInsight on AKS will retire on January 31, 2025

Thu, 29 Aug 2024 07:00:00 Z

On January 31, 2025, we will retire Azure HDInsight on AKS. The remaining clusters on your subscription will be stopped and removed from the host.Before January 31, 2025, you will need to migrate your workloads to Microsoft Fabric or an equivalent Azure p
Update Type: Deprecation, Services: HDInsight, Microsoft Fabric, Kubernetes Service, HDInsight on AKS, Categories: Retirements

Azure Monitor experience (preview) in HDInsight is retiring by February 1, 2025

Thu, 29 Aug 2024 07:00:00 Z

On February 1, 2025, Azure HDInsight is retiring the Log Analytics used in Azure Monitor experience (preview).In case you migrated from Classic Log Analytics to New Azure monitor experience (preview), you would have completed adjustments to the new table
Update Type: Deprecation, Services: HDInsight, Azure Monitor, Categories: Features, Retirements

Retirement: Azure HDInsight on AKS will retire on January 31, 2025

Thu, 29 Aug 2024 00:00:00 Z

Azure HDInsight on AKS is nearing retirement.
Update Type: Deprecation, Services: HDInsight, Kubernetes Service, HDInsight on AKS, Categories: Retirements

Retirement: Azure Monitor experience (preview) in HDInsight is retiring by February 1, 2025

Thu, 29 Aug 2024 00:00:00 Z

Migrate from Azure Monitor experience (preview) in Azure HDInsight to Azure Monitor Agent (AMA) before January 31, 2025
Update Type: Deprecation, Services: HDInsight, Azure Monitor, Categories: Features

Generally Availabile: Spain Central region added to Azure HDInsight

Fri, 05 Jul 2024 00:00:00 Z

We are now introducing a new region - Spain Central for Azure HDInsight.
Update Type: Announcement, Services: HDInsight, Categories: Features, Regions & Datacenters

Generally Available: Spain Central region added to Azure HDInsight

Thu, 20 Jun 2024 16:47:53 Z

Status: Now Available HDInsight is now generally available in Spain Central.  Azure HDInsight is a managed, full-spectrum, open-source analytics service in the cloud for enterprises. You can use open-source frameworks such as Hadoop, Apache Spark, Apache Hive, LLAP, Apache Kafka, and more.  Learn more. ________________________________________________________________________________________________________________________________
Update Type: GA, Services: HDInsight, Categories: Services

Public Preview: Azure HDInsight on AKS is now available for preview in 6 new regions

Mon, 27 May 2024 00:00:00 Z

Azure HDInsight on AKS is a modern, reliable, secure, and fully managed Platform as a Service (PaaS) that runs on Azure Kubernetes Service (AKS). HDInsight on AKS allows you to deploy popular Open-Source Analytics workloads like Apache Spark™, Apache Flink®️, and Trino without the overhead of managing and monitoring containers. For more information about Azure HDInsight on AKS, see the documentation
Update Type: Preview, Services: HDInsight, Kubernetes Service, HDInsight on AKS, Categories: Features, Regions & Datacenters, Open Source

Scale Real-Time Streams to Delta Lakehouse with Apache Flink on Azure HDInsight on AKS

Wed, 15 May 2024 09:46:57 Z

This post is co-authored with Keshav Singh, Principal Engineering Lead, Microsoft Purview Data Governance In this blog, we turn the page and learn about enabling delta format as source and sink for stream processing with Apache Flink. Delta has become a DeFacto ACID compliant Lakehouse format for ecosystem enabling Petabyte scale processing while turning it a single source of truth, it becomes essential to bring it all together on top of Microsoft Fabric. Data engineering in delta format unifies diverse data sources into singular mode for analytics. Lastly as technologies such as Fabric endpoint, Synapse Serverless SQL will get efficient by the day, direct mode delta access will get cheaper and faster with no real need for an edge copy analytics. Streaming Events can now be unified in Delta format as a sink for enabling Realtime analytics. Let us consider a Sales Event Scenario, the event has demonstrated structure. Sales Source Event The Sales Source Event is stored in Delta Format on ADLS Gen2. Delta Storage for SaleSource Events About 20 Million Events HDInsight on AKS Cluster Pool Create a cluster pool to host a set of clusters, these could be Spark, Trino, Flink clusters. With a cluster pool as a concept, and a Platform as a service offering, HDInsight on AKS allows developers to quickly build up a data estate with all their favorite open source workloads, with full configurability and SKU sizing of their choice. Let's Provision the Pool. HDInsight on AKS Cluster Pool Next, provision a Flink Cluster, we went with a Session cluster. In nutshell a session cluster can share resources amongst multiple jobs while an Application Cluster will be resource dedicated towards a particular application. Flink Session Cluster Once the cluster is provisioned update the flink-configs to add/load the Hadoop class path and ensure to load the cluster’s native class load
Update Type: Announcement, Services: HDInsight, Microsoft Fabric, Kubernetes Service, Container Instances, HDInsight on AKS, Categories: Services, Open Source

Getting started with Private Clusters on HDInsight on AKS for securing your analytics workloads

Wed, 15 May 2024 04:08:46 Z

HDInsight on AKS is a managed Platform as a Service (PaaS) that runs on Azure Kubernetes Service (AKS). HDInsight on AKS allows you to deploy popular Open-Source Analytics workloads like Apache Spark™, Apache Flink:registered:, and Trino without the overhead of managing and monitoring containers. HDInsight on AKS clusters allow you to setup outbound network connections from cluster to any destination, if the destination is reachable from the node's network interface. This means that cluster resources can access any public or private IP address, domain name, or URL on the internet or on your virtual network.However, in some scenarios, you may want to control or restrict the egress traffic from your cluster for security, compliance reasons. For example, you may want to: Prevent clusters from accessing malicious or unwanted services. Enforce network policies or firewall rules on the outbound traffic. Monitor or audit the egress traffic from cluster for troubleshooting or compliance purposes. There are different methods for managing the traffic flow. You can learn more about it here. In this blog, we will discuss about how to control or restrict the egress traffic from your HDInsight on AKS cluster using User Defined Routing (UDR) in your virtual network. With this setup, there won't be any Public IP created when you spin up an HDInsight on AKS cluster.Note: UDR setup requires you to setup firewall rules and define the routing using custom VNet and subnet before creating an HDInsight on AKS clusterLet's get started. Step 1: Setup the virtual network (VNet). Required if you don't have existing VNet From the Azure portal, search for virtual networks and click to create new. Create a VNet named "contoso-hdi-vnet". Step 2: Setup the firewall. Deploy the firewall in your virtual network (contoso-hdi-vnet).To deploy a firewall into the integrated virtual network, you need a subnet called AzureFirewallSubnet Navigate to your VNet
Update Type: Announcement, Services: HDInsight, Kubernetes Service, Virtual Network, HDInsight on AKS, Categories: Compliance, Security, Services

General availability: Mexico Central region added to Azure HDInsight

Tue, 30 Apr 2024 00:00:00 Z

HDInsight is now generally available in Mexico Central. Azure HDInsight is a managed, full-spectrum, open-source analytics service in the cloud for enterprises. You can use open-source frameworks such as Hadoop, Apache Spark, Apache Hive, LLAP, Apache Kafka, and more. For more information about HDInsight, see the Azure HDInsight documentation.
Update Type: GA, Services: HDInsight, Categories: Features, Regions & Datacenters

Public Preview: Azure HDInsight on AKS is now available for preview in 7 new regions

Tue, 30 Apr 2024 00:00:00 Z

Azure HDInsight on AKS is now available for preview in seven new regions - East US 2 EUAP, West US, Japan East, Australia East, Canada Central, North Europe, Brazil South. Azure HDInsight on AKS is a modern, reliable, secure, and fully managed Platform as a Service (PaaS) that runs on Azure Kubernetes Service (AKS). HDInsight on AKS allows you to deploy popular Open-Source Analytics workloads like Apache Spark™, Apache Flink®️, and Trino without the overhead of managing and monitoring containers. For more information about Azure HDInsight on AKS, see the documentation.
Update Type: Preview, Services: HDInsight, Kubernetes Service, HDInsight on AKS, Categories: Features, Regions & Datacenters, Open Source

Run federated queries on Trino with HDInsight on AKS

Tue, 02 Apr 2024 15:46:30 Z

HDInsight on AKS is a modern, reliable, secure, and fully managed Platform as a Service (PaaS) that runs on Azure Kubernetes Service (AKS). HDInsight on AKS allows you to deploy popular Open-Source Analytics workloads like Apache Spark, Apache Flink, and Trino without the overhead of managing and monitoring containers. This blog explains how you can use Trino with HDInsight on AKS to experience the flavor of running federated queries.Trino is an open-source distributed SQL query engine and well known for its capability to query data from multiple sources without moving your data. That is cool!! Isn't it?And of course! you can use Trino for other use cases as well. Interactive and ad-hoc querying. Batch ETL across heterogenous systems. Reporting and Dashboarding. With Trino on HDinsight on AKS, you can set up your cluster easily and fast with a few clicks. You can also enjoy the advantages of auto scaling, an intuitive UI to configure your cluster, and monitoring tools like Azure monitor and managed Prometheus and Grafana. Moreover, there are some exclusive features, such as a special connector for Power BI, result caching for all connectors, query scan statistics to analyze query performance, and Shared SQL connector.There is lot more coming on your way with intuitive UI for catalog management, fine-grained access control with Apache Ranger, and an upgraded OSS version. Stay tuned!!Let's see in action how you can run federated queries on Trino with HDInsight on AKS. Scenario: Analyze NYC taxi data to understand the average fare and the passenger count at different zones.Summary of steps for the demo: Take NYC taxi data from the official source. Land the data in ADLS Gen 2 and expose it as a Hive table in Trino. Prepare zone data and land in Azure Database for PostgresSQL. Run a federated query on two data sources ADLS Gen2 and Azure Database for PostgresSQL. To build the demo yourself and experience the power of Trino firsthand, follow the step-
Update Type: Announcement, Services: HDInsight, Power BI Embedded, Kubernetes Service, HDInsight on AKS, Categories: Features, Services, Management

Azure HDInsight 5.0 will be retired on 31 March 2025

Fri, 29 Mar 2024 00:00:00 Z

Azure HDInsight 5.0 for cluster shapes Kafka, Interactive Query, and Spark will retire on 31 March 2025. HDInsight, 5.1 is Generally available from November 1, 2023, this release contains all the latest versions of supported software. It comes with all the improvements made in open-source versions and the integrations from Microsoft.
Update Type: Deprecation, Services: HDInsight, Categories: Retirements

Basic and Standard A-series VMs on HDInsight will retire on 31 August 2024

Fri, 22 Mar 2024 00:00:00 Z

On 31 August 2024, Azure will retire Basic and Standard A-series VMs as announced earlier on 24 August 2021. Before 31 August 2024, you need to migrate your HDInsight A series VMs to Av2-series VMs, which provide more memory per vCPU and faster storage on solid-state drives (SSDs). To avoid service disruptions, migrate your workloads from Basic and Standard A-series VMs to Av2-series VMs before 31 August 2024.
Update Type: Deprecation, Services: HDInsight, Virtual Machines, Categories: Retirements

Azure HDInsight 4.0 will be retired on 31 March 2025 - migrate your HDInsight clusters to 5.1

Fri, 22 Mar 2024 00:00:00 Z

Azure HDInsight 4.0 will be retired on 31 March 2025 - migrate your HDInsight clusters to 5.1
Update Type: Deprecation, Services: HDInsight, Categories: Features, SDK and Tools, Retirements

Apache Kafka Cluster - TLS encryption on HDInsight

Wed, 20 Mar 2024 16:22:34 Z

In this blog we will go through how to set up Transport Layer Security (TLS) encryption for HDInsight Apache Kafka cluster (between Apache Kafka brokers and Apache Zookeepers). Prerequisite 1. Create Apache Kafka Cluster 2. SSH Access to the cluster The blog covers self-sign certificate, the process remains same for certificates issues from Certificate Authority (CA). TLS Setup for Apache Kafka Brokers User can generate TLS for each broker Domian/IP address/Node Name or Subject Alternative Name (SAN) or Wildcard certificate. The choice between individual vs Wildcard vs SAN is depends on your cluster behavior. If you are planning to have scale out feature of the HDInsight Apache Kafka cluster, then it is less hassle if you use Wildcard or SAN certificate. At the time of scale out, you would not require generating new certificate of each additional node, if you are using Wildcard or SAN certificate. The HDInsight Apache Kafka cluster broker's DNS names follow similar pattern within given cluster wnX-..bx.internal.cloudapp.net, that means you can generate either wildcard or SAN certificate for the cluster without worrying about cluster scale out. Before we jump into certificate generation process, let's understand few points: Certificate can be generated outside of the cluster nodes, via different build pipeline and make them available to the cluster (head node, brokers, and zookeeper nodes) or from the head node. Over here we are assuming you are generating self-sign SAN certificate from the head node. Self-sign vs CA cert process - The self-signed certificates are public key certificates that are not issued by a certificate authority (CA). These self-signed certificates are easy to make and do not cost money. However, they do not provide any trust value. It is recommended to use CA cert for the production workload. Steps to generate SAN certificate for the
Update Type: Announcement, Services: HDInsight, Categories: Security

Migration of Apache Spark from HDInsight 5.0 to HDInsight 5.1

Tue, 27 Feb 2024 16:27:15 Z

Azure HDInsight Spark 5.0 to HDI 5.1 Migration A new version of HDInsight 5.1 is released with Spark 3.3.1. This release improves join query performance via Bloom filters, increases the Pandas API coverage with the support of popular Pandas features such as datetime.timedelta and merge_asof, simplifies the migration from traditional data warehouses by improving ANSI compliance and supporting dozens of new built-in functions. In this article we will discuss about the migration of user applications from HDInsight 5.0(Spark 3.1) to HDInsight 5.1 (Spark 3.3). The sections include, 1. Changes which are compatible with minor changes 2. Changes in Spark that require application changes Application Changes with backport. The below changes are part of HDI 5.1 release. If these functions are used in applications, the given steps can be taken to avoid the changes in application code. Since Spark 3.3, the histogram_numeric function in Spark SQL returns an output type of an array of structs (x, y), where the type of the ‘x’ field in the return value is propagated from the input values consumed in the aggregate function. In Spark 3.2 or earlier, x’ always had double type. Optionally, use the configuration spark.sql.legacy.histogramNumericPropagateInputType since Spark 3.3 to revert to the previous behavior. Spark 3.1 (pyspark) Spark 3.3: In Spark 3.3, the timestamps subtraction expression such as timestamp '2021-03-31 23:48:00' - timestamp '2021-01-01 00:00:00' returns values of DayTimeIntervalType. In Spark 3.1 and earlier, the type of the same expression is CalendarIntervalType. To restore the behavior before Spark 3.3, you can set spark.sql.legacy.interval.enabled to true. Since Spark 3.3, the functions lpad and rpad have been overloaded to support byte sequences. When the first argument is a byte sequence, the optional padding pattern must also be a byte sequence and
Update Type: Announcement, Services: HDInsight, Categories: Features, Compliance

Securing Azure HDInsight: ESM Support with Ubuntu 18.04, Cluster Updates, and Best Practices

Tue, 27 Feb 2024 15:35:08 Z

Azure HDInsight, Microsoft's cloud-based big data analytics platform, continues to advance its features to provide users with a secure and efficient environment. In this article, we will explore the latest enhancements, focusing on Expanded Security Maintenance (ESM) support, the importance of regular cluster updates, and best practices recommended by Microsoft to fortify HDInsight deployments. The foundation of a secure Azure HDInsight environment lies in its ability to address critical vulnerabilities promptly. Microsoft ensures this by shipping the latest HDInsight images with Expanded Security Maintenance (ESM) support, which provides a framework for ongoing support, stability with minimal changes specifically targeting critical, high and some medium-level fixes. This ensures that HDInsight users benefit from a continuously updated and secure environment. ESM Support in Latest Images: Azure HDInsight 5.0 and 5.1 versions use Ubuntu 18.04 pro image. Ubuntu Pro includes security patching for all Ubuntu packages due to Expanded Security Maintenance (ESM) for Infrastructure and Applications. Ubuntu Pro 18.04 LTS will remain fully supported until April 2028. For more information on what's new in the latest HDInsight images with ESM support, users can refer to the official release notes on the Azure HDInsight Release Notes Archive. Periodic Cluster Updates: Maintaining a secure HDInsight environment requires diligence in keeping clusters up to date. Microsoft facilitates this process through the HDInsight OS patching mechanism. Periodically updating clusters using the procedures outlined in the official documentation ensures that users benefit from the latest features, performance improvements, and crucial security patches. Learn more about updating HDInsight clusters through the Azure HDInsight OS Patching documentation. ESM and HDI Release Integration: Extended Security Maintenance is seamlessly
Update Type: Announcement, Services: HDInsight, Categories: Features, Security

Migration Guide for HDInsight 5.1 - Hadoop cluster

Tue, 27 Feb 2024 09:43:13 Z

Azure HDInsight is a managed, full-spectrum, open-source analytics service in the cloud for enterprises. With HDInsight, you can use open-source frameworks such as, Apache Spark, Apache Hive, LLAP, Apache Kafka, Hadoop and more, in your Azure environment. Microsoft periodically upgrades the open source frameworks and the HDInsight Resource provider to include new improvements and features. HDInsight, 5.1 is Generally available from November 1, 2023, this release contains all the latest versions of supported software. It comes with all the improvements made in open-source versions and the integrations from Microsoft. This version is fully compatible with HDI 5.0 and HDI 4.0 versions. HDInsight has discontinued Sqoop and Pig add-ons from HDInsight 5.1 version. This guide contains detailed information about new version and migration steps for Hadoop cluster Introduction Key advantages Latest Version – HDInsight 5.1 comes with the latest stable open-source version available. Customers can benefit from all latest features, improvements, and Bug fixes. Secure – The new versions come with more secure controls and fixes. Open-source security fixes are part of this release as well the improvements in security by Microsoft. Lower TCO – new version is better in performance. Customers can leverage performance improvements to reduce the operating cost. For Hadoop cluster upgrade the below points needs to be considered: Migration guide for Ambari and Oozie Oozie Migration Migration guide for HIVE/LLAP Migration guide for Ambari and Oozie Ambari User Configs Migration After setting up HDI 5.1, it is necessary to update the user-defined configurations from the HDI 4.0 cluster. Unfortunately, Ambari does not currently provide a feature to export and import configurations. To overcome this limitation, we have created a script that facilitates downloading the configurations and comparing them across clusters. However, this process involves a few manual
Update Type: Announcement, Services: HDInsight, Categories: Features, Security, Services, Open Source

General availability: Israel Central region added to Azure HDInsight

Tue, 30 Jan 2024 00:00:00 Z

.
Update Type: GA, Services: HDInsight, Categories: Features

General availability: Italy North region added to Azure HDInsight

Tue, 30 Jan 2024 00:00:00 Z

.
Update Type: GA, Services: HDInsight, Categories: Features

Public Preview: Azure Data Explorer connector for Apache Flink

Mon, 08 Jan 2024 00:00:00 Z

Azure Data Explorer connector for Apache Flink (Public Preview)
Update Type: Preview, Services: HDInsight, Kubernetes Service, Data Explorer, Categories: Features, Open Source, SDK and Tools

Manage your big data needs with HDInsight on AKS

Thu, 21 Sep 2023 08:00:00 -0700

Today, we’re excited to announce the public preview of HDInsight on AKS, our cloud-native, open-source big data service, completely rearchitected on Azure Kubernetes Service infrastructure with two new workloads and numerous improvements across the stack.
Update Type: Announcement, Services: HDInsight, Kubernetes Service, HDInsight on AKS, Categories:

HDInsight support in Azure CLI now out of preview

Tue, 17 Sep 2019 00:00:00 -0700

We are pleased to share that support for HDInsight in Azure CLI is now generally available. The addition of the `az hdinsight` command group allows you to easily manage your HDInsight clusters using simple commands while taking advantage of all that Azure CLI has to offer, such as cross-platform support and tab completion.
Update Type: Announcement, Services: HDInsight, Categories:

Drive higher utilization of Azure HDInsight clusters with autoscale

Tue, 21 May 2019 00:00:00 -0700

We are excited to announce the General Availability of the autoscale feature for Azure HDInsight. This feature enables enterprises to become more productive and cost-efficient by automatically scaling clusters up or down based on the load or a customized schedule.
Update Type: Announcement, Services: HDInsight, Categories:

Manage Azure HDInsight clusters using .NET, Python, or Java

Thu, 18 Apr 2019 00:00:00 -0700

We are pleased to announce the general availability of the new Azure HDInsight management SDKs for .NET, Python, and Java.
Update Type: Announcement, Services: HDInsight, Categories:

Announcing general availability of Apache Hadoop 3.0 on Azure HDInsight

Mon, 15 Apr 2019 00:00:00 -0700

Today we’re announcing the general availability of Apache Hadoop 3.0 on Azure HDInsight. In partnership with Cloudera, Microsoft Azure is the first cloud provider to offer customers the benefit of the latest innovations in the most popular open source analytics projects, with unmatched scalability, flexibility, and security.
Update Type: Announcement, Services: HDInsight, Categories:

HDInsight Tools for Visual Studio Code now generally available

Wed, 23 Jan 2019 00:00:00 -0800

We are pleased to announce the general availability for Azure HDInsight Tools for Visual Studio Code (VSCode). HDInsight Tools for VSCode give developers a cross-platform lightweight code editor for developing HDInsight PySpark and Hive batch jobs and interactive query.
Update Type: Announcement, Services: HDInsight, Categories:

HDInsight Enterprise Security Package now generally available

Mon, 01 Oct 2018 00:00:00 -0700

HDInsight team is excited to reveal general availability (GA) of Enterprise Security Package (ESP) for Apache Spark, Apache Hadoop and Interactive Query clusters in HDI 3.6. When enterprise customers share clusters between multiple employees, Hadoop admins must ensure those employees have the right set of accesses and permissions to perform big data operations.
Update Type: Announcement, Services: HDInsight, Categories: Security

New Azure HDInsight management SDK now in public preview

Wed, 26 Sep 2018 00:00:00 -0700

Today, we are excited to announce the preview of the new Azure HDInsight Management SDK. This preview SDK brings support for new languages and can be used to easily manage your HDInsight clusters.
Update Type: Announcement, Services: HDInsight, Categories: Management

Deep dive into Azure HDInsight 4.0

Tue, 25 Sep 2018 00:00:00 -0700

We are thrilled to announce that HDInsight 4.0 is now available in public preview. HDInsight 4.0 brings latest Apache Hadoop 3.0 innovations representing over 5 years of work from the open source community and our partner Hortonworks across key apache frameworks to solve ever-growing big data and advanced analytics challenges.
Update Type: Announcement, Services: HDInsight, Categories:

Azure HDInsight brings next generation Apache Hadoop 3.0 and enterprise security to the cloud

Mon, 24 Sep 2018 00:00:00 -0700

Hadoop 3.0 is a major upgrade to the Hadoop stack enabling developers to build deep learning systems and business applications on top of very large datasets using the most modern compute infrastructure available in the cloud.
Update Type: Announcement, Services: HDInsight, Categories: Security

Announcing general availability of Azure HDInsight 3.6

Tue, 04 Apr 2017 00:00:00 -0700

This is a release post announcing availability of Azure HDInsight 3.6
Update Type: Announcement, Services: HDInsight, Categories: