Push Files from Local to Remote¶
Adding Local Files¶
To add data to a pipe, you need to add files to the central pipe directory at
pipe.dir. d6tpipe makes it easy to add new data with a set of convenience functions.
pipe = d6tpipe.pipe.Pipe(api, 'pipe-name') # save new data directly to pipe df.to_csv(pipe.dirpath/'new.csv') # import files from another folder pipe.import_dir('/some/folder')) # import files a list of files pipe.import_files(glob.iglob('folder/**/*.csv'), move=True)
You can also use any other (manual) methods to copy files to the pipe directory
After you’ve added data to a pipe (see above), pushing data is just as simple as pulling data.
pipe.push_preview() # preview pipe.push() # execute
If you might get this error:
ClientError: An error occurred (ExpiredToken) when calling the PutObject operation: The provided token has expired.. For security reasons you receive short-term credentials. You can force renewal
You can manually control what gets pushed.
# advanced push options pipe.push(['a.csv']) # force push/pull on selected files pipe.push(include='*.csv',exclude='backup*.csv') # apply file filters