Azure Storage Filesystem

[martindut]

Hi. I just want to ask if you have any plans to add Azure storage to the supported filesystems?

[roee88]

Hi,

It seems like adding support should be simple using https://github.com/dask/adlfs. Can you share what you're trying to do?

[martindut]

Thanks for the reply. We are busy developing an data pipeline for all our client data that we push to Azure Datalake Gen 2. We use Databricks and Spark to Delta the data. Our business apps needs to be able to query the data, which it can, but I want to explore the use of Arrow Flight because of the speed. Secondly, we are based in South Africa and currently our tenant is setup in North Europe, so accessing the data can be a bit slow.

[cdoron]

Hi. Adding support for Azure should be quite simple using an fsspec-compatible python library such as adlfs.
However, it would be difficult for us to write and test flight support for Azure without Azure credentials.
Could you share with us a short python program to access a parquet or csv file on Azure? Maybe something similar to:
https://github.com/dask/adlfs/blob/master/adlfs/spec.py#L70