d6tpipe package

Submodules

d6tpipe.api module

class d6tpipe.api.APIClient(token='config', config=None, profile=None, filecfg='~/d6tpipe/cfg.json')[source]

Bases: d6tpipe.api._APIBase

Manager to interface with the remote API.

Parameters:
  • token (str) – your API token
  • config (dict) – manually pass config object
  • profile (str) – name of profile to use
  • filecfg (str) – path to where config file is stored
forgotToken(username, password)[source]

Retrieve your API token

Parameters:
  • username (str) –
  • password (str) –
login(username, password)[source]

Login if already registered

Parameters:
  • username (str) –
  • password (str) –
register(username, email, password)[source]

Register a new API user

Parameters:
  • username (str) –
  • email (str) –
  • password (str) –
setToken(token)[source]
class d6tpipe.api.APILocal(config=None, profile=None, filecfg='~/d6tpipe/cfg.json')[source]

Bases: d6tpipe.api._APIBase

As an alternative to the remote API, you can store everything locally. It mirrors the basic funtionality of the remote API but is not as feature rich.

Parameters:
  • config (dict) – manually pass config object
  • profile (str) – name of profile to use
  • filecfg (str) – path to where config file is stored
class d6tpipe.api.ConfigManager(profile=None, filecfg='~/d6tpipe/cfg.json')[source]

Bases: object

Manage local config. The config is stored in JSON and can be edited directly filecfg location, by default ‘~/d6tpipe/cfg.json’

Parameters:
  • profile (str) – name of profile to use
  • filecfg (str) – path to where config file is stored
init(config=None, server='https://pipe.databolt.tech', reset=False)[source]

Initialize config with content

Parameters:
  • config (dict) – manually pass config object
  • server (str) – location of REST API server
  • reset (bool) – force reset of an existing config
load()[source]

Loads config

Returns:config
Return type:dict
update(config)[source]

Update config. Only keys present in the new dict will be updated, other parts of the config will be kept as is. In other words you can pass in a partial dict to update just the parts you need to be updated.

Parameters:config (dict) – updated config
d6tpipe.api.list_profiles(filecfg='~/d6tpipe/cfg.json')[source]
d6tpipe.api.upsert_permissions(api, pipe_name, settings)[source]

Convenience function to create or update pipe permissions

Parameters:
  • api (obj) – api
  • settings (dict) – permission settings
Returns:

server response data (dict): json returned by server
Return type:

response (obj)
d6tpipe.api.upsert_pipe(api, settings)[source]

Convenience function to create or update a pipe

Parameters:
  • api (obj) – api
  • settings (dict) – pipe settings
Returns:

server response data (dict): json returned by server
Return type:

response (obj)
d6tpipe.api.upsert_pipe_json(api, path_json, name)[source]

Convenience function to create or update a resource. Loads settings from config file to secure credentials

Parameters:
  • api (obj) – api
  • path_json (str) – path to config file in json format
  • name (str) – name of json entry
Returns:

server response data (dict): json returned by server
Return type:

response (obj)
d6tpipe.api.upsert_resource(apiroot, settings)[source]

Convenience function to create or update a resource

Parameters:
  • apiroot (obj) – API endpoint root eg api.cnxn.pipes
  • settings (dict) – resource settings
Returns:

server response data (dict): json returned by server
Return type:

response (obj)

d6tpipe.pipe module

class d6tpipe.pipe.Pipe(api, name, mode='default', sortby='filename', credentials=None)[source]

Bases: d6tpipe.pipe.PipeBase

Managed data pipe

Parameters:
  • api (obj) – API manager object
  • name (str) – name of the data pipe. Has to be created first
  • mode (str) – sync mode
  • sortby (str) – sort files this key. name, mod, size
  • credentials (dict) – override credentials

Note

  • mode: controls which files are synced
    • ‘default’: modified and new files
    • ‘new’: new files only
    • ‘mod’: modified files only
    • ‘all’: all files
is_synced(israise=False)[source]

Check if local is in sync with remote

Parameters:israise (bool) – raise an exception
Returns:pipe is updated
Return type:bool
pull(files=None, dryrun=False, include=None, exclude=None, nrecent=0, merge_mode=None, cached=True)[source]

Pull remote files to local

Parameters:
  • files (list) – override list of filenames
  • dryrun (bool) – preview only
  • include (str) – pattern of files to include, eg *.csv or a-*.csv|b-*.csv
  • exclude (str) – pattern of files to exclude
  • nrecent (int) – use n newest files by mod date. 0 uses all files. Negative number uses n old files
  • merge_mode (str) – how to deal with pull conflicts ie files that changed both locally and remotely? ‘keep’ local files or ‘overwrite’ local files
  • cached (bool) – if True, use cached remote information, default 5mins. If False forces remote scan
Returns:

filenames with attributes
Return type:

list
pull_preview(files=None, include=None, exclude=None, nrecent=0, cached=True)[source]

Preview of files to be pulled

Parameters:
  • files (list) – override list of filenames
  • include (str) – pattern of files to include, eg *.csv or a-*.csv|b-*.csv
  • exclude (str) – pattern of files to exclude
  • nrecent (int) – use n newest files by mod date. 0 uses all files. Negative number uses n old files
  • cached (bool) – if True, use cached remote information, default 5mins. If False forces remote scan
Returns:

filenames with attributes
Return type:

list
push(files=None, dryrun=False, fromdb=False, include=None, exclude=None, nrecent=0, cached=True)[source]

Push local files to remote

Parameters:
  • files (list) – override list of filenames
  • dryrun (bool) – preview only
  • fromdb (bool) – use files from local db, if false scans all files in pipe folder
  • include (str) – pattern of files to include, eg *.csv or a-*.csv|b-*.csv
  • exclude (str) – pattern of files to exclude
  • nrecent (int) – use n newest files by mod date. 0 uses all files. Negative number uses n old files
  • cached (bool) – if True, use cached remote information, default 5mins. If False forces remote scan
Returns:

filenames with attributes
Return type:

list
push_preview(files=None, include=None, exclude=None, nrecent=0, cached=True)[source]

Preview of files to be pushed

Parameters:
  • files (list) – override list of filenames
  • include (str) – pattern of files to include, eg *.csv or a-*.csv|b-*.csv
  • exclude (str) – pattern of files to exclude
  • nrecent (int) – use n newest files by mod date. 0 uses all files. Negative number uses n old files
  • cached (bool) – if True, use cached remote information, default 5mins. If False forces remote scan
Returns:

filenames with attributes
Return type:

list
remove_orphans(files=None, direction='local', dryrun=None)[source]

Remove file orphans locally and/or remotely. When you remove files, they don’t get synced because pull/push only looks at new or modified files. Use this to clean up any removed files.

Parameters:
  • direction (str) – where to remove files
  • dryrun (bool) – preview only

Note

  • direction:
    • ‘local’: remove files locally, ie files that exist on local but not in remote
    • ‘remote’: remove files remotely, ie files that exist on remote but not in local
    • ‘both’: combine local and remote
reset(delete=False)[source]

Resets by deleting all files and pulling

scan_remote(attributes=False, cached=True)[source]

Get file attributes from remote. To run before doing a pull/push

Parameters:cached (bool) – use cached results
Returns:filenames with attributes in remote
Return type:list
setmode(mode)[source]

Set sync mode

Parameters:mode (str) – sync mode

Note

  • mode: controls which files are synced
    • ‘default’: modified and new files
    • ‘new’: new files only
    • ‘mod’: modified files only
    • ‘all’: all files
update_settings(settings)[source]

Update settings. Only keys present in the new dict will be updated, other parts of the config will be kept as is. In other words you can pass in a partial dict to update just the parts you need to be updated.

Parameters:config (dict) – updated config
class d6tpipe.pipe.PipeBase(name, sortby='filename')[source]

Bases: object

Abstract class, use the functions but dont instantiate the class

delete_files(files=None, confirm=True, all_local=None)[source]

Delete files, local and remote

Parameters:
  • files (list) – filenames, if empty delete all
  • confirm (bool) – ask user to confirm delete
  • all_local (bool) – delete all files or just synced files? (local only)
delete_files_local(files=None, confirm=True, delete_all=None, ignore_errors=False)[source]

Delete all local files and reset file database

Parameters:
  • files (list) – filenames, if empty delete all
  • confirm (bool) – ask user to confirm delete
  • delete_all (bool) – delete all files or just synced files?
  • ignore_errors (bool) – ignore missing file errors
delete_files_remote(files=None, confirm=True)[source]

Delete all remote files

Parameters:
  • files (list) – filenames, if empty delete all
  • confirm (bool) – ask user to confirm delete
filepaths(include=None, exclude=None, sortby=None, check_exists=True, aspathlib=True, aspath=True, fromdb=True)[source]

Full path to Files synced in local repo

Parameters:
  • include (str) – pattern of files to include, eg *.csv or a-*.csv|b-*.csv
  • exclude (str) – pattern of files to exclude
  • sortby (str) – sort files this key. name, mod, size
  • check_exists (bool) – check files exist locally
  • aspathlib (bool) – return as pathlib object
Returns:

path to file, either Pathlib or str
Return type:

path
files(include=None, exclude=None, sortby=None, check_exists=True, fromdb=True)[source]

Files synced in local repo

Parameters:
  • include (str) – pattern of files to include, eg *.csv or a-*.csv|b-*.csv
  • exclude (str) – pattern of files to exclude
  • sortby (str) – sort files this key. name, mod, size
  • check_exists (bool) – check files exist locally
Returns:

filenames
Return type:

list
import_dir(dir, move=False)[source]

Import a directory including subdirs

Parameters:
  • dir (str) – directory
  • move (bool) – move or copy
import_file(file, subdir=None, move=False)[source]

Import a single file to repo

Parameters:
  • files (str) – path to file
  • subdir (str) – sub directory to import into
  • move (bool) – move or copy
import_files(files, subdir=None, move=False)[source]

Import files to repo

Parameters:
  • files (list) – list of files, eg from glob.iglob(‘folder/**/*.csv’)
  • subdir (str) – sub directory to import into
  • move (bool) – move or copy
scan_local(include=None, exclude=None, attributes=False, sortby=None, files=None, fromdb=False, on_not_exist='warn')[source]

Get file attributes from local. To run before doing a pull/push

Parameters:
  • include (str) – pattern of files to include, eg *.csv or a-*.csv|b-*.csv
  • exclude (str) – pattern of files to exclude
  • attributes (bool) – return filenames with attributes
  • sortby (str) – sort files this key. name, mod, size
  • files (list) – override list of filenames
  • fromdb (bool) – use files from local db, if false scans all files in pipe folder
  • on_not_exist (bool) – how to handle missing files when creating file attributes
Returns:

filenames with attributes
Return type:

list
class d6tpipe.pipe.PipeLocal(name, config=None, profile=None, filecfg='~/d6tpipe/cfg.json', sortby='filename')[source]

Bases: d6tpipe.pipe.PipeBase

Managed data pipe, LOCAL mode for accessing local files ONLY

Parameters:
  • api (obj) – API manager object
  • name (str) – name of the data pipe
  • profile (str) – name of profile to use
  • filecfg (str) – path to where config file is stored
  • sortby (str) – sort files this key. name, mod, size

Module contents