NAME

       azure-datalake-store - azure-datalake-store Documentation

       A  pure-python  interface  to the Azure Data-lake Storage system, providing pythonic file-system and file
       objects,  seamless  transition  between  Windows  and  POSIX  remote  paths,  high-performance  up-   and
       down-loader.

       This software is under active development and not yet recommended for general use.

INSTALLATION

       Using pip:

          pip install azure-datalake-store

       Manually (bleeding edge):

       • Download the repo from https://github.com/Azure/azure-data-lake-store-python

       • checkout the dev branch

       • install the requirements (pip install -r dev_requirements.txt)

       • install in develop mode (python setup.py develop)

       • optionally: build the documentation (including this page) by running make html in the docs directory.

AUTH

       Although  users  can  generate  and supply their own tokens to the base file-system class, and there is a
       password-based function in the lib module for generating  tokens,  the  most  convenient  way  to  supply
       credentials  is via environment parameters. This latter method is the one used by default in library. The
       following variables are required:

       • azure_tenant_id

       • azure_username

       • azure_password

       • azure_store_name

       • azure_url_suffix (optional)

PYTHONIC FILESYSTEM

       The AzureDLFileSystem object is the main API for library usage  of  this  package.  It  provides  typical
       file-system operations on the remote azure store

          token = lib.auth(tenant_id, username, password)
          adl = core.AzureDLFileSystem(store_name, token)
          # alternatively, adl = core.AzureDLFileSystem()
          # uses environment variables

          print(adl.ls())  # list files in the root directory
          for item in adl.ls(detail=True):
              print(item)  # same, but with file details as dictionaries
          print(adl.walk(''))  # list all files at any directory depth
          print('Usage:', adl.du('', deep=True, total=True))  # total bytes usage
          adl.mkdir('newdir')  # create directory
          adl.touch('newdir/newfile') # create empty file
          adl.put('remotefile', '/home/myuser/localfile') # upload a local file

       In  addition,  the file-system generates file objects that are compatible with the python file interface,
       ensuring compatibility with libraries that work on python files. The recommended way to use this is  with
       a context manager (otherwise, be sure to call close() on the file object).

          with adl.open('newfile', 'wb') as f:
              f.write(b'index,a,b\n')
              f.tell()   # now at position 9
              f.flush()  # forces data upstream
              f.write(b'0,1,True')

          with adl.open('newfile', 'rb') as f:
              print(f.readlines())

          with adl.open('newfile', 'rb') as f:
              df = pd.read_csv(f) # read into pandas.

       To  seamlessly  handle remote path representations across all supported platforms, the main API will take
       in numerous path types: string, Path/PurePath, and AzureDLPath. On Windows in particular, you can pass in
       paths separated by either forward slashes or backslashes.

          import pathlib  # only >= Python 3.4
          from pathlib2 import pathlib  # only <= Python 3.3

          from azure.datalake.store.core import AzureDLPath

          # possible remote paths to use on API
          p1 = '\\foo\\bar'
          p2 = '/foo/bar'
          p3 = pathlib.PurePath('\\foo\\bar')
          p4 = pathlib.PureWindowsPath('\\foo\\bar')
          p5 = pathlib.PurePath('/foo/bar')
          p6 = AzureDLPath('\\foo\\bar')
          p7 = AzureDLPath('/foo/bar')

          # p1, p3, and p6 only work on Windows
          for p in [p1, p2, p3, p4, p5, p6, p7]:
            with adl.open(p, 'rb') as f:
                print(f.readlines())

PERFORMANT UP-/DOWN-LOADING

       Classes  ADLUploader  and  ADLDownloader  will  chunk large files and send many files to/from azure using
       multiple threads. A whole directory tree can be transferred, files matching a  specific  glob-pattern  or
       any particular file.

          # download the whole directory structure using 5 threads, 16MB chunks
          ADLDownloader(adl, '', 'my_temp_dir', 5, 2**24)

AUTHOR

       TBD

COPYRIGHT

       TBD