I have been using #xarray for quite sometime, but until now I had only used it sequentially. I had copied the open_mfdataset function to #pymech, but without properly trying out the parallel=True option.
Today I spawned some #dask distributed workers using dask-mpi and there was a 3x speedup for loading a collection of 100 files using 6 cores.
I should try out dask-jobqueue in the #HPC clustef next.
The social network of the future: No ads, no corporate surveillance, ethical design, and decentralization! Own your data with Mastodon!