3
���h&= � @ s� d dl Z d dlZddlmZ ddlmZmZmZm Z ddl
m Z m Z m
Z
dd d
�Zd d � Zd
d� Zdd� Zdd� Zddd�Zddd�ZdS )� N� )�utils)� iteritems�reduce� OrderedDict�
basestring)�Variable� as_variable�
Coordinate�all� different�equalsc
C s� ddl m} ddlm}
ytj| �\} } W n tk
rF td��Y nX |dkrdtj dt
dd� d }|dk r�tj d
t
dd� |}|dk r�td ��|dk r�td ��t | |
�r�t } nt | | �r�t
} ntd
��| | |||||�S )a� Concatenate xarray objects along a new or existing dimension.
Parameters
----------
objs : sequence of Dataset and DataArray objects
xarray objects to concatenate together. Each object is expected to
consist of variables and coordinates with matching shapes except for
along the concatenated dimension.
dim : str or DataArray or pandas.Index
Name of the dimension to concatenate along. This can either be a new
dimension name, in which case it is added along axis=0, or an existing
dimension name, in which case the location of the dimension is
unchanged. If dimension is provided as a DataArray or Index, its name
is used as the dimension to concatenate along and the values are added
as a coordinate.
data_vars : {'minimal', 'different', 'all' or list of str}, optional
These data variables will be concatenated together:
* 'minimal': Only data variables in which the dimension already
appears are included.
* 'different': Data variables which are not equal (ignoring
attributes) across all datasets are also concatenated (as well as
all for which dimension already appears). Beware: this option may
load the data payload of data variables into memory if they are not
already loaded.
* 'all': All data variables will be concatenated.
* list of str: The listed data variables will be concatenated, in
addition to the 'minimal' data variables.
If objects are DataArrays, data_vars must be 'all'.
coords : {'minimal', 'different', 'all' o list of str}, optional
These coordinate variables will be concatenated together:
* 'minimal': Only coordinates in which the dimension already appears
are included.
* 'different': Coordinates which are not equal (ignoring attributes)
across all datasets are also concatenated (as well as all for which
dimension already appears). Beware: this option may load the data
payload of coordinate variables into memory if they are not already
loaded.
* 'all': All coordinate variables will be concatenated, except
those corresponding to other dimensions.
* list of str: The listed coordinate variables will be concatenated,
in addition the 'minimal' coordinates.
compat : {'equals', 'identical'}, optional
String indicating how to compare non-concatenated variables and
dataset global attributes for potential conflicts. 'equals' means
that all variable values and dimensions must be the same;
'identical' means that variable attributes and global attributes
must also be equal.
positions : None or list of integer arrays, optional
List of integer arrays which specifies the integer positions to which
to assign each dataset along the concatenated dimension. If not
supplied, objects are concatenated in the provided order.
indexers, mode, concat_over : deprecated
Returns
-------
concatenated : type of objs
See also
--------
auto_combine
r )�Dataset)� DataArrayz.must supply at least one object to concatenateNz�the `dim` argument to `concat` will be required in a future version of xarray; for now, setting it to the old default of 'concat_dim'� )�
stacklevel� concat_dimsz_indexers has been renamed to positions; the alias will be removed in a future version of xarrayzt`mode` is no longer a valid argument to xarray.concat; it has been split into the `data_vars` and `coords` argumentsz{`concat_over` is no longer a valid argument to xarray.concat; it has been split into the `data_vars` and `coords` argumentsz9can only concatenate xarray Dataset and DataArray objects)�datasetr Z dataarrayr r Zpeek_at�
StopIteration�
ValueError�warnings�warn�
FutureWarning�
isinstance�_dataarray_concat�_dataset_concat� TypeError)
�objs�dim� data_vars�coords�compat� positionsZindexers�mode� concat_overr r Z first_obj�f� r& �3/tmp/pip-build-5_djhm0z/xray/xarray/core/combine.py�concat
s0 C
r( c C sv t | t�rd}n^t| d�sBt| dd�}|dkr2d}t|| �}|} n,t| d�sbt| �j� }|j\} n | }|j\} | |fS )z}
Infer the dimension name and 1d coordinate variable (if appropriate)
for concatenating along the new dimension.
N�dims�name�
concat_dim)r r �hasattr�getattrr
r Zto_coordr) )r �coordZdim_namer& r& r' �_calc_concat_dim_coordu s
r/ c sz � �fdd�}t � � x*�D ]"}� j�fdd�|jj� D �� qW � j||d�� � j||d�� ��d krv� j�� � S )z�
Determine which dataset variables need to be concatenated in the result,
and which can simply be taken from the first dataset.
c s� �dkrd}nd}t | t�r�| dkrT�fdd�� t�� fdd�t�d ��D ��}q�| d
kr~tt�d ���t�d j� }q�| d kr�t� }q�td �| f ��n2��fd
d�| D �}|r�td�||f ��t| �}|S )Nr Z coordinateszdata variablesr c s. �d j � � t� �fdd��dd � D ��S )Nr c 3 s | ]}|j � j� � V qd S )N)� variablesr
)�.0�ds)�v�vnamer& r' � <genexpr>� s zQ_calc_concat_over.<locals>.process_subset_opt.<locals>.differs.<locals>.<genexpr>r )r0 �any)r4 )�datasets)r3 r4 r' �differs� s z>_calc_concat_over.<locals>.process_subset_opt.<locals>.differsc 3 s" | ]}|� kr�|�r|V qd S )Nr& )r1 �k)r$ r8 r& r' r5 � s z@_calc_concat_over.<locals>.process_subset_opt.<locals>.<genexpr>r r Zminimalz"unexpected value for concat_%s: %sc s" g | ]}|t � d ��kr|�qS )r )r- )r1 r9 )r7 �subsetr& r' �
<listcomp>� s zA_calc_concat_over.<locals>.process_subset_opt.<locals>.<listcomp>z8some variables in %s are not %s on the first dataset: %s)r r �setr- r) r )�optr: Zsubset_long_nameZ
concat_newZ invalid_vars)r$ r7 )r8 r: r' �process_subset_opt� s(
$z-_calc_concat_over.<locals>.process_subset_optc 3 s | ]\}}� |j kr|V qd S )N)r) )r1 r9 r3 )r r& r' r5 � s z$_calc_concat_over.<locals>.<genexpr>r r r )r< �updater0 �items�add)r7 r r r r> r2 r& )r$ r7 r r' �_calc_concat_over� s $
"
rB c s> ddl m}m� |dkr$td| ��� fdd��D ��t��\�}t��||�}���fdd �} t� �t�d
j���d
j }
x,�d
j
j � D ]\�} �|kr�| �| � q�W xֈdd � D ]�} |dkr�t j
| j |
� r�td ��x�t| j
�D ]�\�} ��k�r�|k�rtd
� ��q��k�| jkk�r6td� ��q��kr��kr�t| |��� � r�|dk�rhdn|}
td�|
f ��q�W q�W �fdd��D ��i �x�D ]} �j| j� �q�W �j�d � ���fdd�}x<|D ]4�|�fdd��D ��}tj|�|�}| �|� �q�W |�|
d�}|j��}|d k �r:|||j< |S )zN
Concatenate a sequence of datasets along a new or existing dimension
r )r �
as_datasetr
� identicalz2compat=%r invalid: must be 'equals' or 'identical'c s g | ] }� |��qS r& r& )r1 r2 )rC r& r' r; � s z#_dataset_concat.<locals>.<listcomp>c s2 t |t�st�| � d jkr&�j| � |�| < d S )Nr )r r �AssertionErrorr rA )r9 r3 )r7 �result_coord_names� result_varsr& r' �insert_result_variable� s
z/_dataset_concat.<locals>.insert_result_variabler Nz#dataset global attributes not equalz"encountered unexpected variable %rz2%r is a coordinate in some datasets but not others�equalz"variable %r not %s across datasetsc s g | ]}|j j� d ��qS )r )r) �get)r1 r2 )r r& r' r; � s c 3 sz t tjdd� | D ���}�|kr*�f| }xJt| ��D ]<\}� |j|krlt � �fdd�|D ��}|j||�}|V q6W d S )Nc S s g | ]}|j D ]}|�qqS r& )r) )r1 r3 �dr& r& r' r; � s z?_dataset_concat.<locals>.ensure_common_dims.<locals>.<listcomp>c 3 s | ]}�j |� �V qd S )N)rJ )r1 rK )�dim_len�non_concat_dimsr& r' r5 s z>_dataset_concat.<locals>.ensure_common_dims.<locals>.<genexpr>)�tuple�pd�unique�zipr) Z expand_dims)�varsZ common_dims�varZ common_shape)r � dim_lengthsrM )rL r' �ensure_common_dims� s
z+_dataset_concat.<locals>.ensure_common_dimsc s g | ]}|j � �qS r& )r0 )r1 r2 )r9 r& r' r; s )�attrs)r
rD )r r rC r r/ rB r r< r rV r0 r@ r Z
dict_equivr r- r? r) �popr r( Z
set_coordsr* )r7 r r r r! r" r r. r$ rH Z result_attrsr3 r2 ZverbrU rR Zcombined�resultr& )rC r7 r rT r9 rM rF rG r' r � sX
r c C s� t | �} |dkrtd��g }xXt| �D ]L\}}|dkr>|j} n&| |jkrd|dkrZtd��n
|j| �}|j|j� � q&W t||||||�}
| d j|
| �S )Nr zFdata_vars is not a valid argument when concatenating DataArray objectsr rD zarray names not identical) �listr � enumerater* �rename�appendZ_to_temp_datasetr Z_from_temp_dataset) Zarraysr r r r! r" r7 �nZarrr* r2 r& r& r' r s
r c s� t | �dkr| d S |d kr�| d � | d �t� j�}� j�jkrpt� jj� �t�jj� � }tdd� |D ��}t |�dkr�t� �fdd�|D ��}t |�dkr�td| ��nt |�dkr�td��|\}t| |d�S d S ) Nr r c s s | ]\}}|V qd S )Nr& )r1 �i�_r& r& r' r5 ; s z _auto_concat.<locals>.<genexpr>c 3 s$ | ] }� | j �| �s|V qd S )N)r
)r1 rK )�ds0�ds1r& r' r5 = s z0too many different dimensions to concatenate: %szTcannot infer dimension to concatenate: supply the ``concat_dim`` argument explicitly)r )�lenr< r) r@ r r( )r7 r r Z
dim_tuplesr& )r` ra r' � _auto_concat1 s$
rc c sD ddl m} |jdd� | �j� }� fdd�|D �}tdd� |�}|S )a* Attempt to auto-magically combine the given datasets into one.
This method attempts to combine a list of datasets into a single entity by
inspecting metadata and using a combination of concat and merge.
It does not concatenate along more than one dimension or align or sort data
under any circumstances. It will fail in complex cases, for which you
should use ``concat`` and ``merge`` explicitly.
When ``auto_combine`` may succeed:
* You have N years of data and M data variables. Each combination of a
distinct time period and test of data variables is saved its own dataset.
Examples of when ``auto_combine`` fails:
* In the above scenario, one file is missing, containing the data for one
year's data for one variable.
* In the most recent year, there is an additional data variable.
* Your data includes "time" and "station" dimensions, and each year's data
has a different set of stations.
Parameters
----------
datasets : sequence of xarray.Dataset
Dataset objects to merge.
concat_dim : str or DataArray or Index, optional
Dimension along which to concatenate variables, as used by
:py:func:`xarray.concat`. You only need to provide this argument if the
dimension along which you want to concatenate is not a dimension in
the original datasets, e.g., if you want to stack a collection of
2D arrays along a third dimension.
Returns
-------
combined : xarray.Dataset
See also
--------
concat
Dataset.merge
r )� itertoolzc S s t t| j��S )N)rN �sortedr )r2 r& r& r' �<lambda>v s z auto_combine.<locals>.<lambda>c s g | ]}t |� d ��qS ))r )rc )r1 r2 )r+ r& r' r; x s z auto_combine.<locals>.<listcomp>c S s
| j |�S )N)�merge)r2 �otherr& r& r' rf y s )Ztoolzrd �groupby�valuesr )r7 r+ rd ZgroupedZ concatenatedZmergedr& )r+ r' � auto_combineJ s +
rk )Nr r r
NNNN)N)N)r ZpandasrO � r Zpycompatr r r r �variabler r r
r( r/ rB r r rc rk r& r& r& r' �<module> s
i5X
|