Push Files from Local to Remote

Adding Local Files

To add data to a pipe, you need to add files to the central pipe directory at pipe.dir. d6tpipe makes it easy to add new data with a set of convenience functions.

pipe = d6tpipe.pipe.Pipe(api, 'pipe-name')

# save new data directly to pipe
df.to_csv(pipe.dirpath/'new.csv')

# import files from another folder
pipe.import_dir('/some/folder'))

# import files a list of files
pipe.import_files(glob.iglob('folder/**/*.csv'), move=True)

You can also use any other (manual) methods to copy files to the pipe directory pipe.dir.

Pushing Data

After you’ve added data to a pipe (see above), pushing data is just as simple as pulling data.

pipe.push_preview() # preview
pipe.push() # execute

For this to work, you do need to have write permissions to the pipe or registered your own pipe.

Expired Token

If you might get this error: ClientError: An error occurred (ExpiredToken) when calling the PutObject operation: The provided token has expired.. For security reasons you receive short-term credentials. You can force renewal

pipe._reset_credentials()

Customize Push

You can manually control what gets pushed.

# advanced push options
pipe.push(['a.csv']) # force push/pull on selected files
pipe.push(include='*.csv',exclude='backup*.csv') # apply file filters