This page shows you how to manage all the generated output files from your jobs. Effective management of your output files has numerous benefits, including reduced file size (and hence transfer speed), effective housekeeping of data, and easy access to files on Rescale Cloud Files.
When running a job on Rescale, the working directory is located at
$HOME/work/. This is where the running software dumps all its output files, as well as the working directory of the command line prompt that you are exposed to by default.
Once the job has run to completion, the Rescale platform saves and transfers everything that is in the aforementioned work directory. This is usually the last step in a job, and the time taken to do so is dependent on the total filesize and number of files.
Removing unwanted files
A straightforward way to effectively manage your files would be to remove unwanted files at the end of your job. You can do this by appending additional commands below your simulation command.
A quick tip would be to use the wildcard
* to match any character and delete multiple files in one command.
A sample command to include in your software settings page is shown below:
Please note that deleted files cannot be recovered.
Compressing your files
There are a number of benefits to compressing your files. Firstly, it lowers the file size and number of files, thereby speeding up the upload and download process. This is particulary important when your output files are expected to be very large (>10GB). Secondly, compressing your work directory into a single compressed folder helps preserve your directory structure. If you are managing your files on Rescale Cloud Files, it will be advantageous to store zip files instead if you want to keep the directory structure.
Rescale supports the following compression types:
- .tar, .tar.gz, .tgz, .gz
For example, if you want to archive all your d3plot files, you can use a command similar to:
tar -czf d3plot-archive.tar.gz *d3plot* rm *d3plot*
You can also utilize post-processing scripts to run more complex cleanup and post-processing commands. This script will only be executed once all your commands in the software settings page have run. You can use this feature/script to clean up your job output, or parse out variables for easy viewing in the results page.
You can read more about using our post-processing feature here.