3
���h�E �
@ s� d dl Z d dlZd dlZd dlZd dlmZ d dlmZ ddlm Z m
Z
ddl m Z ddl
mZ dd lmZmZ dd
lmZ d d d
�Zej� Zdd� Zdd� Zd dd�ZG dd� de�Zd!dd�Z e j e j e j d�Z d"dd �Z!d#d d �Z"dS )$� N)�glob)�BytesIO� )�backends� conventions� )� ArrayWriter)� auto_combine)�close_on_error�
is_remote_uri)�
basestringFc C s� |rbt | �rbydd l}d}W q� tk
r^ ydd l}d}W n tk
rX td��Y nX Y q�X nTydd l}d}W nB tk
r� ydd l}d}W n tk
r� td��Y nX Y nX |S )Nr �netcdf4�pydapzFnetCDF4 or pydap is required for accessing remote datasets via OPeNDAP�scipyzKcannot read or write netCDF files without netCDF4-python or scipy installed)r �netCDF4� ImportErrorr �
ValueErrorZscipy.io.netcdf)�path� allow_remoter �enginer r � r �3/tmp/pip-build-5_djhm0z/xray/xarray/backends/api.py�_get_default_engine s( r c C sV | j d�rd}nB|d kr$t| dd�}|dkr@t| �r:d}qRt}n|dkrNt}nd}|S )Nz.gzFT)r r
�h5netcdf)�endswithr r � _global_lock)�filenamer �lockr r r �
_default_lock, s
r c C s" dd� }x| D ] }||� qW dS )z8DataArray.name and Dataset keys must be a string or Nonec S s, t | t�r| s(td��n| d k r(td��d S )NzoInvalid name for DataArray or Dataset key: string must be length 1 or greater for serialization to netCDF filesz_DataArray.name or Dataset key must be either a string or None for serialization to netCDF files)�
isinstancer r � TypeError)�namer r r �
check_nameC s
z+_validate_dataset_names.<locals>.check_nameNr )�datasetr" �kr r r �_validate_dataset_namesA s
r% Tc s� �sd� d�d�d�d� ��������� f
dd� } t �tj�rH�} �n�t �t��r��jd�r�dk rv�dkrvtd��tjdd� dk r�td
��ytjt j
���} W n8 t k
r� }
z d |
j kr�td ��n� W Y dd}
~
X nX n��dkr�t
�d
d���dk�rtj��d�} nh�dk�r$tj��} nR�dk�r:tj��} n<�dk�rTtj��d�} n"�dk�rjtj��} n td� ��| dk�r�t���} t| �� | | | �S Q R X n&�dk �rdk�r�td��tj��} | | �S )a/ Load and decode a dataset from a file or file-like object.
Parameters
----------
filename_or_obj : str, file or xarray.backends.*DataStore
Strings are interpreted as a path to a netCDF file or an OpenDAP URL
and opened with python-netCDF4, unless the filename ends with .gz, in
which case the file is gunzipped and opened with scipy.io.netcdf (only
netCDF3 supported). File-like objects are opened with scipy.io.netcdf
(only netCDF3 supported).
group : str, optional
Path to the netCDF4 group in the given file to open (only works for
netCDF4 files).
decode_cf : bool, optional
Whether to decode these variables, assuming they were saved according
to CF conventions.
mask_and_scale : bool, optional
If True, replace array values equal to `_FillValue` with NA and scale
values according to the formula `original_values * scale_factor +
add_offset`, where `_FillValue`, `scale_factor` and `add_offset` are
taken from variable attributes (if they exist). If the `_FillValue` or
`missing_value` attribute contains multiple values a warning will be
issued and all array values matching one of the multiple values will
be replaced by NA.
decode_times : bool, optional
If True, decode times encoded in the standard NetCDF datetime format
into datetime objects. Otherwise, leave them encoded as numbers.
concat_characters : bool, optional
If True, concatenate along the last dimension of character arrays to
form string arrays. Dimensions will only be concatenated over (and
removed) if they have no corresponding variable and if they are only
used as the last dimension of character arrays.
decode_coords : bool, optional
If True, decode the 'coordinates' attribute to identify coordinates in
the resulting dataset.
engine : {'netcdf4', 'scipy', 'pydap', 'h5netcdf', 'pynio'}, optional
Engine to use when reading files. If not provided, the default engine
is chosen based on available dependencies, with a preference for
'netcdf4'.
chunks : int or dict, optional
If chunks is provided, it used to load the new dataset into dask
arrays. This is an experimental feature; see the documentation for more
details.
lock : False, True or threading.Lock, optional
If chunks is provided, this argument is passed on to
:py:func:`dask.array.from_array`. By default, a per-variable lock is
used when reading data from netCDF files with the netcdf4 and h5netcdf
engines to avoid issues with concurrent access when using dask's
multithreaded backend.
drop_variables: string or iterable, optional
A variable or list of variables to exclude from being parsed from the
dataset. This may be useful to drop variables with problems or
inconsistent values.
Returns
-------
dataset : Dataset
The newly created dataset.
See Also
--------
open_mfdataset
Fc s� t j| � ����d�}� d k r�yddlm} W n$ tk
rR dd l}td��Y nX t�t�rvt�� rvt j
j ��}n�}||��� ����� ��
}d��p�df }|j � |||d�}|j
|_
n|}|S )N)�mask_and_scale� decode_times�concat_characters�
decode_coords�drop_variablesr )�tokenizez)xarray requires dask version 0.6 or newerz%s:%s/� )� name_prefix�tokenr )r � decode_cfZ dask.baser+ r �daskr r r �osr �getmtime�chunk� _file_obj) �storer �dsr+ r0 Zfile_argr. r- Zds2)
�chunksr( r/ r) r' r* r �filename_or_obj�groupr& r r �maybe_decode_store� s.
z(open_dataset.<locals>.maybe_decode_storez.gzNr zHcan only read gzipped netCDF files with default engine or engine='scipy'r � z4reading a gzipped netCDF not supported on Python 2.6z is not a valid NetCDF 3 filez2gzipped file loading only supports NetCDF 3 files.T)r r
)r9 r r Zpynioz(unrecognized engine for open_dataset: %rzEcan only read file-like objects with default engine or engine='scipy')F)r r; )r r ZAbstractDataStorer r r �sys� version_info�ScipyDataStore�gzip�openr �messager �NetCDF4DataStoreZPydapDataStore�
H5NetCDFStoreZ NioDataStorer r
)r8 r9 r/ r& r' r( r) r r7 r r* r: r5 �er )
r7 r( r/ r) r' r* r r8 r9 r& r � open_datasetQ sV C
rE c @ s e Zd Zdd� Zdd� ZdS )�_MultiFileCloserc C s
|| _ d S )N)� file_objs)�selfrG r r r �__init__� s z_MultiFileCloser.__init__c C s x| j D ] }|j� qW d S )N)rG �close)rH �fr r r rJ � s z_MultiFileCloser.closeN)�__name__�
__module__� __qualname__rI rJ r r r r rF � s rF c
s� t | t�rtt| ��} | s"td���dkr8t| d ���� ���fdd�| D �}dd� |D �}�dk rx�fdd�|D �}t||d�} t|�| _| S ) a Open multiple files as a single dataset.
Experimental. Requires dask to be installed.
Parameters
----------
paths : str or sequence
Either a string glob in the form "path/to/my/files/*.nc" or an explicit
list of files to open.
chunks : int or dict, optional
Dictionary with keys given by dimension names and values given by chunk
sizes. In general, these should divide the dimensions of each dataset.
If int, chunk each dimension by ``chunks``.
By default, chunks will be chosen to load entire input files into
memory at once. This has a major impact on performance: please see the
full documentation for more details.
concat_dim : str or DataArray or Index, optional
Dimension to concatenate files along. This argument is passed on to
:py:func:`xarray.auto_combine` along with the dataset objects. You only
need to provide this argument if the dimension along which you want to
concatenate is not a dimension in the original datasets, e.g., if you
want to stack a collection of 2D arrays along a third dimension.
preprocess : callable, optional
If provided, call this function on each dataset prior to concatenation.
engine : {'netcdf4', 'scipy', 'pydap', 'h5netcdf', 'pynio'}, optional
Engine to use when reading files. If not provided, the default engine
is chosen based on available dependencies, with a preference for
'netcdf4'.
lock : False, True or threading.Lock, optional
This argument is passed on to :py:func:`dask.array.from_array`. By
default, a per-variable lock is used when reading data from netCDF
files with the netcdf4 and h5netcdf engines to avoid issues with
concurrent access when using dask's multithreaded backend.
**kwargs : optional
Additional arguments passed on to :py:func:`xarray.open_dataset`.
Returns
-------
xarray.Dataset
See Also
--------
auto_combine
open_dataset
zno files to openNr c s( g | ] }t |f�� pi �d �����qS ))r r7 r )rE )�.0�p)r7 r �kwargsr r r �
<listcomp>* s z"open_mfdataset.<locals>.<listcomp>c S s g | ]
}|j �qS r )r4 )rO r6 r r r rR , s c s g | ] }� |��qS r r )rO r6 )�
preprocessr r rR / s )�
concat_dim) r r �sortedr �IOErrorr r rF r4 )
�pathsr7 rT rS r r rQ �datasetsrG Zcombinedr )r7 r rQ r rS r �open_mfdataset� s /
rY )r
r r �wc C s� |dkr i }|dkr>t � }|dkr(d}qN|dk rNtd| ��n|dkrNt|�}t| � y t| }W n tk
r� td| ��Y nX |dk r�|j� }|dk} ||||||�}
z&| j|
| |d� t|t �r�|j � S W d| r�|
j
� X | s�|
S dS )z�This function creates an appropriate datastore for writing a dataset to
disk as a netCDF file
See `Dataset.to_netcdf` for full API docs.
The ``writer`` argument is only for the private use of save_mfdataset.
Nr zlinvalid engine for creating bytes with to_netcdf: %r. Only the default engine or engine='scipy' is supportedz%unrecognized engine for to_netcdf: %r)�sync�encoding) r r r r% �WRITEABLE_STORES�KeyError�upperZ
dump_to_storer �getvaluerJ ) r# r �mode�formatr9 r �writerr\ Z store_clsr[ r5 r r r � to_netcdf; s8
rd c s� �dkr$t t|��t |�k r$td��|dkr:dgt | � }t tt | �t |�t |�g��dkrdtd��t� �� ���fdd�t| ||�D �}z"�j� x|D ] }|j� q�W W dx|D ] }|j� q�W X dS )a� Write multiple datasets to disk as netCDF files simultaneously.
This function is intended for use with datasets consisting of dask.array
objects, in which case it can write the multiple datasets to disk
simultaneously using a shared thread pool.
When not using dask, it is no different than calling ``to_netcdf``
repeatedly.
Parameters
----------
datasets : list of xarray.Dataset
List of datasets to save.
paths : list of str
List of paths to which to save each corresponding dataset.
mode : {'w', 'a'}, optional
Write ('w') or append ('a') mode. If mode='w', any existing file at
these locations will be overwritten.
format : {'NETCDF4', 'NETCDF4_CLASSIC', 'NETCDF3_64BIT', 'NETCDF3_CLASSIC'}, optional
File format for the resulting netCDF file:
* NETCDF4: Data is stored in an HDF5 file, using netCDF4 API
features.
* NETCDF4_CLASSIC: Data is stored in an HDF5 file, using only
netCDF 3 compatibile API features.
* NETCDF3_64BIT: 64-bit offset version of the netCDF 3 file format,
which fully supports 2+ GB files, but is only compatible with
clients linked against netCDF version 3.6.0 or later.
* NETCDF3_CLASSIC: The classic netCDF 3 file format. It does not
handle 2+ GB files very well.
All formats are supported by the netCDF4-python library.
scipy.io.netcdf only supports the last two formats.
The default format is NETCDF4 if you are saving a file to disk and
have the netCDF4-python library available. Otherwise, xarray falls
back to using scipy to write netCDF files and defaults to the
NETCDF3_64BIT format (scipy does not support netCDF4).
groups : list of str, optional
Paths to the netCDF4 group in each corresponding file to which to save
datasets (only works for format='NETCDF4'). The groups will be created
if necessary.
engine : {'netcdf4', 'scipy', 'h5netcdf'}, optional
Engine to use when writing netCDF files. If not provided, the
default engine is chosen based on available dependencies, with a
preference for 'netcdf4' if writing to a file on disk.
Examples
--------
Save a dataset into one netCDF per year of data:
>>> years, datasets = zip(*ds.groupby('time.year'))
>>> paths = ['%s.nc' % y for y in years]
>>> xr.save_mfdataset(datasets, paths)
rZ zCcannot use mode='w' when writing multiple datasets to the same pathNr zcmust supply lists of the same length for the datasets, paths and groups arguments to save_mfdatasetc
s& g | ] \}}}t ||��|� ���qS r )rd )rO r6 r r9 )r rb ra rc r r rR � s z"save_mfdataset.<locals>.<listcomp>)�len�setr r �zipr[ rJ )rX rW ra rb �groupsr Zstoresr5 r )r rb ra rc r �save_mfdatasetl s : "
ri )F)
NTTTTTNNNN)NNNNN)NrZ NNNNN)rZ NNN)#r< r? �os.pathr1 � threadingr �ior r, r r �commonr Z core.combiner Z
core.utilsr
r Z
core.pycompatr r �Lockr r r% rE �objectrF rY rB r> rC r] rd ri r r r r �<module> s:
A
0 |