Azure Data Factory Skill

This skill provides expert guidance for Azure Data Factory. Covers troubleshooting, best practices, decision making, architecture & design patterns, limits & quotas, security, configuration, integrations & coding patterns, and deployment. It combines local quick-reference content with remote documentation fetching capabilities.

How to Use This Skill

IMPORTANT for Agent: Use the Category Index below to locate relevant sections. For categories with line ranges (e.g., L35-L120), use read_file with the specified lines. For categories with file links (e.g., [security.md](security.md)), use read_file on the linked reference file

IMPORTANT for Agent: If metadata.generated_at is more than 3 months old, suggest the user pull the latest version from the repository. If mcp_microsoftdocs tools are not available, suggest the user install it: Installation Guide

This skill requires network access to fetch documentation content:

• Preferred: Use mcp_microsoftdocs:microsoft_docs_fetch with query string from=learn-agent-skill. Returns Markdown.
• Fallback: Use fetch_webpage with query string from=learn-agent-skill&accept=text/markdown. Returns Markdown.

Category Index

Category	Lines	Description
Troubleshooting	L37-L91	Diagnosing and fixing ADF failures: connector/format errors, CDC, pipelines, triggers, data flows, SHIR/SSIS IR, performance, security, and known issues with concrete error patterns and workarounds.
Best Practices	L92-L114	Performance tuning, schema drift/error handling, reusable patterns, and migration best practices for Data Factory mapping data flows, copy activity, runtimes, and data lake writes.
Decision Making	L115-L133	Guidance on cost planning, connector upgrades, integration runtime and compute choices, and migration/modernization of ADF, SSIS, Synapse, and pipelines to Fabric.
Architecture & Design Patterns	L134-L142	Patterns and architectures for ADF: efficient mapping data flows, metadata‑driven copy, SSIS IR with SQL MI/failover/VNet, Cosmos DB migration, and SAP CDC design.
Limits & Quotas	L143-L150	Configuring Until/Wait activity limits and timeouts, understanding data flow reservation discounts, and reviewing ADF service quotas and capacity constraints.
Security	L151-L182	Securing Data Factory: auth, roles, encryption, Key Vault, managed identities, VNets/private endpoints, firewall/Policy configs, and secure access to SQL, Purview, and SSIS IR.
Configuration	L183-L294	Configuring ADF: integration runtimes, SSIS/SHIR, triggers, monitoring/logging, copy/data flows, formats/connectors, parameters/expressions, and pipeline/control activities.
Integrations & Coding Patterns	L295-L463	Patterns and how-tos for connecting ADF to many data sources (SAP, SaaS apps, DBs, files), using mapping data flow functions, running SSIS, Databricks/ML/Synapse jobs, and pipeline templates.
Deployment	L464-L477	CI/CD and deployment for ADF: ARM/linked templates, Azure DevOps pipelines, hotfix flows, pre/post scripts, SSIS job migration, and integration runtime install/migration.

Troubleshooting

Topic	URL
Troubleshoot Azure Data Factory change data capture	https://learn.microsoft.com/en-us/azure/data-factory/change-data-capture-troubleshoot
Troubleshoot CI/CD, Azure DevOps, and GitHub integration for ADF	https://learn.microsoft.com/en-us/azure/data-factory/ci-cd-github-troubleshoot-guide
Resolve common Azure Data Factory connector upgrade issues	https://learn.microsoft.com/en-us/azure/data-factory/connector-deprecation-frequently-asked-questions
Diagnose and fix Azure Data Factory Amazon S3 connector errors	https://learn.microsoft.com/en-us/azure/data-factory/connector-troubleshoot-amazon-simple-storage-service
Fix Azure Blob Storage connector issues in ADF	https://learn.microsoft.com/en-us/azure/data-factory/connector-troubleshoot-azure-blob-storage
Diagnose and fix Azure Cosmos DB connector issues	https://learn.microsoft.com/en-us/azure/data-factory/connector-troubleshoot-azure-cosmos-db
Resolve Azure Data Explorer connector problems in ADF	https://learn.microsoft.com/en-us/azure/data-factory/connector-troubleshoot-azure-data-explorer
Troubleshoot Azure Data Lake Storage connectors in ADF	https://learn.microsoft.com/en-us/azure/data-factory/connector-troubleshoot-azure-data-lake
Resolve Azure Data Factory Azure Files connector issues	https://learn.microsoft.com/en-us/azure/data-factory/connector-troubleshoot-azure-files
Troubleshoot Azure Data Factory Azure Table Storage connector	https://learn.microsoft.com/en-us/azure/data-factory/connector-troubleshoot-azure-table-storage
Resolve DB2 connector problems in Azure Data Factory	https://learn.microsoft.com/en-us/azure/data-factory/connector-troubleshoot-db2
Fix delimited text connector errors in Data Factory	https://learn.microsoft.com/en-us/azure/data-factory/connector-troubleshoot-delimited-text
Resolve Dynamics 365 and Dataverse connector issues	https://learn.microsoft.com/en-us/azure/data-factory/connector-troubleshoot-dynamics-dataverse
Troubleshoot file system connector in Azure Data Factory	https://learn.microsoft.com/en-us/azure/data-factory/connector-troubleshoot-file-system
Troubleshoot FTP, SFTP, and HTTP connectors in ADF	https://learn.microsoft.com/en-us/azure/data-factory/connector-troubleshoot-ftp-sftp-http
Fix common Azure Data Factory Google Ads connector problems	https://learn.microsoft.com/en-us/azure/data-factory/connector-troubleshoot-google-ads
Diagnose and fix Azure Data Factory connector issues	https://learn.microsoft.com/en-us/azure/data-factory/connector-troubleshoot-guide
Diagnose Azure Data Factory Hive connector connection and query errors	https://learn.microsoft.com/en-us/azure/data-factory/connector-troubleshoot-hive
Troubleshoot Microsoft Fabric Lakehouse connector in ADF	https://learn.microsoft.com/en-us/azure/data-factory/connector-troubleshoot-microsoft-fabric-lakehouse
Fix Microsoft Fabric Warehouse connector issues in ADF	https://learn.microsoft.com/en-us/azure/data-factory/connector-troubleshoot-microsoft-fabric-warehouse
Resolve Azure Data Factory MongoDB connector errors	https://learn.microsoft.com/en-us/azure/data-factory/connector-troubleshoot-mongodb
Resolve Oracle connector issues in Azure Data Factory	https://learn.microsoft.com/en-us/azure/data-factory/connector-troubleshoot-oracle
Troubleshoot ORC format connector in Azure Data Factory	https://learn.microsoft.com/en-us/azure/data-factory/connector-troubleshoot-orc
Fix Parquet format connector issues in Data Factory	https://learn.microsoft.com/en-us/azure/data-factory/connector-troubleshoot-parquet
Fix Azure Database for PostgreSQL connector issues	https://learn.microsoft.com/en-us/azure/data-factory/connector-troubleshoot-postgresql
Diagnose REST connector failures in Azure Data Factory	https://learn.microsoft.com/en-us/azure/data-factory/connector-troubleshoot-rest
Resolve Salesforce and Service Cloud connector issues	https://learn.microsoft.com/en-us/azure/data-factory/connector-troubleshoot-salesforce
Resolve SAP connector issues in Azure Data Factory	https://learn.microsoft.com/en-us/azure/data-factory/connector-troubleshoot-sap
Fix ServiceNow connector problems in Azure Data Factory	https://learn.microsoft.com/en-us/azure/data-factory/connector-troubleshoot-servicenow
Fix SharePoint Online list connector problems in Data Factory	https://learn.microsoft.com/en-us/azure/data-factory/connector-troubleshoot-sharepoint-online-list
Resolve Shopify connector issues in Azure Data Factory	https://learn.microsoft.com/en-us/azure/data-factory/connector-troubleshoot-shopify
Troubleshoot Snowflake connector in Azure Data Factory	https://learn.microsoft.com/en-us/azure/data-factory/connector-troubleshoot-snowflake
Troubleshoot SQL-based connectors in Azure Data Factory	https://learn.microsoft.com/en-us/azure/data-factory/connector-troubleshoot-synapse-sql
Fix Teradata connector failures in Azure Data Factory	https://learn.microsoft.com/en-us/azure/data-factory/connector-troubleshoot-teradata
Troubleshoot Azure Data Factory XML format connector parsing and schema issues	https://learn.microsoft.com/en-us/azure/data-factory/connector-troubleshoot-xml
Use Fail activity to surface custom errors in ADF	https://learn.microsoft.com/en-us/azure/data-factory/control-flow-fail-activity
Troubleshoot Copy activity performance issues	https://learn.microsoft.com/en-us/azure/data-factory/copy-activity-performance-troubleshooting
Troubleshoot external control activities in Azure Data Factory	https://learn.microsoft.com/en-us/azure/data-factory/data-factory-troubleshoot-guide
Diagnose and fix Azure Data Factory Studio issues	https://learn.microsoft.com/en-us/azure/data-factory/data-factory-ux-troubleshoot-guide
Troubleshoot mapping data flow connector and format issues	https://learn.microsoft.com/en-us/azure/data-factory/data-flow-troubleshoot-connector-format
Resolve mapping data flow issues in Azure Data Factory	https://learn.microsoft.com/en-us/azure/data-factory/data-flow-troubleshoot-guide
Work around known Azure Data Factory issues	https://learn.microsoft.com/en-us/azure/data-factory/known-issues-troubleshoot-guide
Troubleshoot Azure Data Factory pipeline orchestration and triggers	https://learn.microsoft.com/en-us/azure/data-factory/pipeline-trigger-troubleshoot-guide
Troubleshoot Azure Data Factory SAP CDC via SHIR logs	https://learn.microsoft.com/en-us/azure/data-factory/sap-change-data-capture-debug-shir-logs
Resolve security and access control issues in Azure Data Factory	https://learn.microsoft.com/en-us/azure/data-factory/security-and-access-control-troubleshoot-guide
Use the self-hosted integration runtime diagnostic tool	https://learn.microsoft.com/en-us/azure/data-factory/self-hosted-integration-runtime-diagnostic-tool
Troubleshoot self-hosted integration runtime issues	https://learn.microsoft.com/en-us/azure/data-factory/self-hosted-integration-runtime-troubleshoot-guide
Troubleshoot SSIS IR connectivity with diagnose feature	https://learn.microsoft.com/en-us/azure/data-factory/ssis-integration-runtime-diagnose-connectivity-faq
Diagnose and fix SSIS IR management issues	https://learn.microsoft.com/en-us/azure/data-factory/ssis-integration-runtime-management-troubleshoot
Troubleshoot SSIS package execution on SSIS IR	https://learn.microsoft.com/en-us/azure/data-factory/ssis-integration-runtime-ssis-activity-faq
Interpret pipeline failure status and error messages in ADF	https://learn.microsoft.com/en-us/azure/data-factory/tutorial-pipeline-failure-error-handling

Best Practices

Topic	URL
Use column patterns in mapping data flows	https://learn.microsoft.com/en-us/azure/data-factory/concepts-data-flow-column-pattern
Reuse logic with flowlets in mapping data flows	https://learn.microsoft.com/en-us/azure/data-factory/concepts-data-flow-flowlet
Tune mapping data flow performance in Data Factory	https://learn.microsoft.com/en-us/azure/data-factory/concepts-data-flow-performance
Optimize sink performance in mapping data flows	https://learn.microsoft.com/en-us/azure/data-factory/concepts-data-flow-performance-sinks
Optimize source performance in mapping data flows	https://learn.microsoft.com/en-us/azure/data-factory/concepts-data-flow-performance-sources
Improve transformation performance in mapping data flows	https://learn.microsoft.com/en-us/azure/data-factory/concepts-data-flow-performance-transformations
Handle schema drift in Data Factory mapping data flows	https://learn.microsoft.com/en-us/azure/data-factory/concepts-data-flow-schema-drift
Tune integration runtime performance for data flows	https://learn.microsoft.com/en-us/azure/data-factory/concepts-integration-runtime-performance
Tune Azure-SSIS integration runtime for high performance	https://learn.microsoft.com/en-us/azure/data-factory/configure-azure-ssis-integration-runtime-performance
Optimize Azure Data Factory copy activity performance	https://learn.microsoft.com/en-us/azure/data-factory/copy-activity-performance
Optimize Azure Data Factory copy activity performance	https://learn.microsoft.com/en-us/azure/data-factory/copy-activity-performance-features
Migrate on-premises HDFS data to Azure Storage	https://learn.microsoft.com/en-us/azure/data-factory/data-migration-guidance-hdfs-azure-storage
Migrate Netezza data to Azure with Data Factory	https://learn.microsoft.com/en-us/azure/data-factory/data-migration-guidance-netezza-azure-sqldw
Migrate data from Amazon S3 to Azure Storage with ADF	https://learn.microsoft.com/en-us/azure/data-factory/data-migration-guidance-s3-azure-storage
Dedupe rows and detect nulls using ADF data flow snippets	https://learn.microsoft.com/en-us/azure/data-factory/how-to-data-flow-dedupe-nulls-snippets
Handle error rows and truncation in ADF data flows	https://learn.microsoft.com/en-us/azure/data-factory/how-to-data-flow-error-rows
Implement BCDR strategies for Data Factory pipelines	https://learn.microsoft.com/en-us/azure/data-factory/pipelines-disaster-recovery
Apply advanced SAP CDC features and best practices	https://learn.microsoft.com/en-us/azure/data-factory/sap-change-data-capture-advanced-topics
Apply data flow best practices for writing to data lake	https://learn.microsoft.com/en-us/azure/data-factory/tutorial-data-flow-write-to-lake

Decision Making

Topic	URL
Use automatic connector upgrades in Azure Data Factory	https://learn.microsoft.com/en-us/azure/data-factory/automatic-connector-upgrade
Interpret pricing examples for ADF integration runtimes	https://learn.microsoft.com/en-us/azure/data-factory/better-understand-different-integration-runtime-charges
Choose the right integration runtime architecture	https://learn.microsoft.com/en-us/azure/data-factory/choose-the-right-integration-runtime-configuration
Choose compute environments for Data Factory	https://learn.microsoft.com/en-us/azure/data-factory/compute-linked-services
Use connector upgrade advisor for ADF and Synapse	https://learn.microsoft.com/en-us/azure/data-factory/connector-upgrade-advisor
Decide how and when to upgrade ADF connectors	https://learn.microsoft.com/en-us/azure/data-factory/connector-upgrade-guidance
Buy and use ADF data flow reserved capacity for cost savings	https://learn.microsoft.com/en-us/azure/data-factory/data-flow-reserved-capacity-overview
Plan data lake and EDW migration to Azure with ADF	https://learn.microsoft.com/en-us/azure/data-factory/data-migration-guidance-overview
Assess Azure Data Factory pipelines for Fabric migration	https://learn.microsoft.com/en-us/azure/data-factory/how-to-assess-your-azure-data-factory-to-fabric-data-factory-migration
Choose and provision Enterprise Edition Azure-SSIS IR	https://learn.microsoft.com/en-us/azure/data-factory/how-to-configure-azure-ssis-ir-enterprise-edition
Upgrade Azure Data Factory pipelines to Fabric Data Factory	https://learn.microsoft.com/en-us/azure/data-factory/how-to-upgrade-your-azure-data-factory-pipelines-to-fabric-data-factory
Modernize Synapse pipelines with Fabric Data Factory	https://learn.microsoft.com/en-us/azure/data-factory/how-to-upgrade-your-azure-synapse-analytics-pipelines-to-fabric-data-factory
Plan and manage Azure Data Factory costs	https://learn.microsoft.com/en-us/azure/data-factory/plan-manage-costs
Assess and plan SSIS migration to Azure Data Factory	https://learn.microsoft.com/en-us/azure/data-factory/scenario-ssis-migration-overview
Apply SSIS migration assessment rules for ADF/Synapse	https://learn.microsoft.com/en-us/azure/data-factory/scenario-ssis-migration-rules

Architecture & Design Patterns

Topic	URL
Design efficient pipelines using mapping data flows	https://learn.microsoft.com/en-us/azure/data-factory/concepts-data-flow-performance-pipelines
Configure Azure-SSIS IR with SQL failover groups	https://learn.microsoft.com/en-us/azure/data-factory/configure-bcdr-azure-ssis-integration-runtime
Build metadata-driven large-scale copy pipelines	https://learn.microsoft.com/en-us/azure/data-factory/copy-data-tool-metadata-driven
Migrate Azure SQL schemas to Azure Cosmos DB with ADF	https://learn.microsoft.com/en-us/azure/data-factory/how-to-sqldb-to-cosmosdb
Decide when and how to join Azure-SSIS integration runtime to a virtual network	https://learn.microsoft.com/en-us/azure/data-factory/join-azure-ssis-integration-runtime-virtual-network

Limits & Quotas

Topic	URL
Configure Until activity loops and timeouts in ADF	https://learn.microsoft.com/en-us/azure/data-factory/control-flow-until-activity
Configure Wait activity durations in Data Factory	https://learn.microsoft.com/en-us/azure/data-factory/control-flow-wait-activity
Understand reservation discounts for ADF data flows	https://learn.microsoft.com/en-us/azure/data-factory/data-flow-understand-reservation-charges
Review Azure Data Factory limits and quotas from FAQ	https://learn.microsoft.com/en-us/azure/data-factory/frequently-asked-questions

Security

Topic	URL
Allow Azure Integration Runtime IP ranges through firewalls	https://learn.microsoft.com/en-us/azure/data-factory/azure-integration-runtime-ip-addresses
Configure VNet injection for Azure-SSIS IR securely	https://learn.microsoft.com/en-us/azure/data-factory/azure-ssis-integration-runtime-virtual-network-configuration
Configure roles and permissions for Azure Data Factory	https://learn.microsoft.com/en-us/azure/data-factory/concepts-roles-permissions
Configure outbound FQDN allow lists with Azure Policy for ADF	https://learn.microsoft.com/en-us/azure/data-factory/configure-outbound-allow-list-azure-policy
Use Azure credentials and managed identities in ADF	https://learn.microsoft.com/en-us/azure/data-factory/credentials
Plan secure data access strategies for Azure Data Factory	https://learn.microsoft.com/en-us/azure/data-factory/data-access-strategies
Configure Azure Private Link for Azure Data Factory	https://learn.microsoft.com/en-us/azure/data-factory/data-factory-private-link
Configure managed identities for Azure Data Factory	https://learn.microsoft.com/en-us/azure/data-factory/data-factory-service-identity
Security infrastructure and data movement in ADF	https://learn.microsoft.com/en-us/azure/data-factory/data-movement-security-considerations
Enable Microsoft Entra authentication for Azure-SSIS integration runtime	https://learn.microsoft.com/en-us/azure/data-factory/enable-aad-authentication-azure-ssis-ir
Enable customer-managed keys (BYOK) for ADF encryption	https://learn.microsoft.com/en-us/azure/data-factory/enable-customer-managed-key
Encrypt credentials for self-hosted integration runtime	https://learn.microsoft.com/en-us/azure/data-factory/encrypt-credentials-self-hosted-integration-runtime
Access secured Purview from Data Factory via private endpoints	https://learn.microsoft.com/en-us/azure/data-factory/how-to-access-secured-purview-account
Use Azure Key Vault secrets in ADF pipeline activities	https://learn.microsoft.com/en-us/azure/data-factory/how-to-use-azure-key-vault-secrets-pipeline-activities
Join Azure-SSIS IR to VNet using PowerShell	https://learn.microsoft.com/en-us/azure/data-factory/join-azure-ssis-integration-runtime-virtual-network-powershell
Join Azure-SSIS IR to VNet via portal	https://learn.microsoft.com/en-us/azure/data-factory/join-azure-ssis-integration-runtime-virtual-network-ui
Secure Azure Data Factory with managed VNets and private endpoints	https://learn.microsoft.com/en-us/azure/data-factory/managed-virtual-network-private-endpoint
Configure modern trusted service firewall bypass in Data Factory	https://learn.microsoft.com/en-us/azure/data-factory/modern-trusted-service
Use built-in Azure Policy definitions for Data Factory	https://learn.microsoft.com/en-us/azure/data-factory/policy-reference
Implement security controls for Azure Data Factory	https://learn.microsoft.com/en-us/azure/data-factory/secure-your-azure-data-factory
Detect and mask PII data with ADF solution template	https://learn.microsoft.com/en-us/azure/data-factory/solution-template-pii-detection-and-masking
Configure Windows authentication for SSIS packages in Azure	https://learn.microsoft.com/en-us/azure/data-factory/ssis-azure-connect-with-windows-auth
Store ADF credentials securely in Azure Key Vault	https://learn.microsoft.com/en-us/azure/data-factory/store-credentials-in-key-vault
Use private endpoints to securely copy data with ADF	https://learn.microsoft.com/en-us/azure/data-factory/tutorial-copy-data-portal-private
Configure Azure-SSIS integration runtime in virtual network	https://learn.microsoft.com/en-us/azure/data-factory/tutorial-deploy-ssis-virtual-network
Secure self-hosted integration runtime with TLS from intranet	https://learn.microsoft.com/en-us/azure/data-factory/tutorial-enable-remote-access-intranet-tls-ssl-certificate
Secure on-prem SQL access via Data Factory managed VNet	https://learn.microsoft.com/en-us/azure/data-factory/tutorial-managed-virtual-network-on-premise-sql-server
Configure private endpoint access to SQL Managed Instance from Data Factory	https://learn.microsoft.com/en-us/azure/data-factory/tutorial-managed-virtual-network-sql-managed-instance