3
���h5O � @ s6 d dl mZ d dlZd dlmZ d dlZd dlmZm Z
d dl m Z d dl
mZ d dlmZmZmZmZmZmZmZmZmZ d dlmZ d d lm Z d dl j j Z d d
l!m"Z"m#Z# d d l$m%Z% d d l&m'Z' e(e)e'd
�dd�Z*dd� Z+G dd� d�Z,dd� Z-dd� Z.ee, e)d�dd�Z/e)d�d d �Z0d d � Z1d d!� Z2dS )"� )� defaultdictN)�List)�NaT� internals)�cache_readonly)�
maybe_promote) �
_get_dtype�is_categorical_dtype�is_datetime64_dtype�is_datetime64tz_dtype�is_extension_array_dtype�is_float_dtype�is_numeric_dtype� is_sparse�is_timedelta64_dtype)�
concat_compat)�isna)�
DatetimeArray�ExtensionArray)�
make_block)� BlockManager)� concat_axis�copy�returnc
C s dd� | D �}t ||�}g }x�|D ]�\}}t|�dkrx|d j rx|d j} | j}
|r`|
j� }
n|
j� }
| j|
|d�} n�t|�r�|d j} dd� |D �} | j s�t
| | j d d�}
n"t
| �}
t |
t
�s�|
jdt|
��}
t|
|| j d�} ntt|||d �|d�} |j| � q"W t||�S )
z�
Concatenate block managers into one.
Parameters
----------
mgrs_indexers : list of (BlockManager, {axis: indexer,...}) tuples
axes : list of Index
concat_axis : int
copy : bool
Returns
-------
BlockManager
c S s g | ]\}}t ||��qS � )�_get_mgr_concatenation_plan)�.0�mgr�indexersr r �>/tmp/pip-build-5_djhm0z/pandas/pandas/core/internals/concat.py�
<listcomp>1 s z.concatenate_block_managers.<locals>.<listcomp>� r )� placementc S s g | ] }|j j�qS r )�block�values)r �jur r r r B s )�axis)r" �ndim)r )�_combine_concat_plans�lenr r# r$ r �viewZmake_block_same_class�_is_uniform_join_units� is_extensionr r' �
isinstancer Zreshaper �_concatenate_join_units�appendr )
Z
mgrs_indexersZaxesr r Z concat_plansZ concat_plan�blocksr" �
join_units�br$ �blk�valsr r r �concatenate_block_managers s2
r5 c C s� t | j�}x |j� D ]\}}t|�||< qW t|�}d|krl|jd�}tj| j|dd�}tj| j |dd�}n4| j
r�| j d }|j t
|||�fgS d}| j}| j }g } x�tj|dd�D ]�\}
} | js�t�|j� } t |�}
t| �|
d< t|
�}
|
d k�r t
d|
�}n�| j |
}|| j }t| �t|j �k�o\|dk�rL|j j�rL|j jjdk�p\tj|�dkj� }|�rr| jdd� n|| d< t
||
| �}| j| |f� q�W | S )
z�
Construct concatenation plan for given block manager and indexers.
Parameters
----------
mgr : BlockManager
indexers : dict of {axis: indexer}
Returns
-------
plan : list of (BlockPlacement, JoinUnit) tuples
r r! )�
fill_valueNF)�group���r8 r8 )�list�shape�itemsr) �tuple�pop�algosZtake_1d�blknos�blklocsZ_is_single_blockr0 Zmgr_locs�JoinUnit� libinternalsZget_blkno_placementsZ
is_slice_like�AssertionErrorr �indexer�as_slice�step�npZdiff�allr/ )r r Z mgr_shape�axrD Z ax0_indexerr? r@ r3 �planZblkno�
placementsZjoin_unit_indexersr: �unitZax0_blk_indexerZunit_no_ax0_reindexingr r r r W sH
r c @ sP e Zd Zddd�Zed�dd�Zedd� �Zed d
� �Zed d � �Z d
d� Z
dS )rA Nc C s" |d kr i }|| _ || _|| _d S )N)r# r r: )�selfr# r: r r r r �__init__� s
zJoinUnit.__init__)r c C s$ t | �j� dt| j�� d| j� d�S )N�(z, �))�type�__name__�reprr# r )rM r r r �__repr__� s zJoinUnit.__repr__c C s( x"| j j� D ]}|dkj� r dS q W dS )Nr! TFr8 )r r$ �any)rM rD r r r �
needs_filling� s zJoinUnit.needs_fillingc C s> | j d krtd��| js | j jS tt| j j| j j�d �S d S )NzBlock is None, no dtyper )r# rC rV �dtyper r r6 )rM r r r rW � s
zJoinUnit.dtypec C s� | j d krdS | j jsdS | j j}| j jr2|j}n,t| j jj�rDdS | j jrR|}n |jdd�}|j d }t
|d d�}x0t d||�D ] }t |||| � �j
� s�dS q�W dS )NTF�K)�orderr �( i� )r# Z _can_hold_nar$ �is_categorical�
categoriesr rW r, �ravelr: �max�ranger rH )rM r$ Z values_flatZ total_lenZ chunk_len�ir r r �is_na� s$
zJoinUnit.is_nac C s� |d kr | j j}| j j� }�n�|}| j�r@t| j dd�r`| j jjdd�}t|�r`|d d kr`d }t| j dd�svt|�r�| j d kr�|j � }|t
j | j d |j
�|d�S n�t| j d d�r�n�t| j d
d�r�nzt|��r"|j � jg |d�}| j \}}|dks�t|��dt
j|ft
jd� } |j| d |d �S t
j| j |d�}|j|� |S | j �r^| j j �r^| j jS | j j�r�| j j �r�| j jt
j�j}n | j j �r�| j j}n| j j}| j�s�|j � }n,x*| jj � D ] \}
} t j || |
|d
�}�q�W |S )NZ is_objectFrX )rY r Z
is_datetimetzr! )rW r[ r, T)Z
allow_fillr6 )r& r6 r8 )!r# r6 Z
get_valuesra �getattrr$ r] r) r Zconstruct_array_typerG �fullr: �valuer Z_from_sequencerC ZonesZintpZtake�empty�fillr Z_can_consolidateZis_boolr[ Zastype�object_r, r* r; r> Ztake_nd) rM � empty_dtype� upcasted_nar6 r$ �arrayZ missing_arrZncolsZnrowsZ empty_arrrI rD r r r �get_reindexed_values� sT
z JoinUnit.get_reindexed_values)N) rR �
__module__� __qualname__rN �strrT r rV rW ra rk r r r r rA � s
rA c s� |dkr t | �dkr td��t| �\� �� �fdd�| D �}t |�dkr~|d }|r�t|tj�rt|jdk r||j� }q�|j� }ndtdd� |D ��r�d d� |D �}t |dd
�}t|t
� s�t|t �r�|j dkr�tj
|�}n t ||d
�}|S ) zI
Concatenate values from several join units along selected axis.
r r! z$Concatenating join units along axis0c s g | ]}|j � �d ��qS ))rh ri )rk )r r% )rh ri r r r = s z+_concatenate_join_units.<locals>.<listcomp>Nc s s | ]}t |t�V qd S )N)r- r )r �tr r r � <genexpr>L s z*_concatenate_join_units.<locals>.<genexpr>c S s* g | ]"}t |t�r|n|d dd�f �qS )r N)r- r )r ro r r r r O s )r& )r) rC �_get_empty_dtype_and_nar- rG Zndarray�baser rU r r r �tzZ
atleast_2d)r1 r r Z to_concatZ
concat_valuesr )rh ri r r. 2 s(
r. c C s> t | �dkr0| d j}|dkr0tjtj�tjfS t| �rX| d jj}| d jj}||fS d}dgt | � }x.t| �D ]"\}}|jdkr�d}qt|j||< qtW t t
�}t t
�} x�t || �D ]�\}
}|
dkr�q�t |
�r�d} n�t
|
�r�d} n�t|
�r�d} n�t|
jtj��rd } njt|
jtj��r d
} nTt|
��r.d } nDt|
��r>d } n4t|
��rR|
jj} n t|
��sft|
��rn|
j} nd
} |j�r�| | j|
� q�|| j|
� q�W |�s�| }d|k�r�t |�dk�r�|d d } | | j fS tjd
�tjfS �nFd
|k�rtjtj�tjfS d |k�r:|�r&tjtj�tjfS tjtj�dfS � n�d|k�rVtjtj�tjfS d|k�rt|d }
|
d t fS d |k�r�tjd�tj dd�fS d |k�r�tjd�tj dd�fS ytj |g �}
W n$ t!k
�r� tjtj�tjfS X t|
��r|
|
jtj�fS t|
��r.|�r&tjtj�tjfS |
dfS d}t"|��dS )z�
Return dtype and N/A values to use when concatenating specified units.
Returned N/A value may be None which means there was no casting involved.
Returns
-------
dtype
na
r! r NFT�categoryZ
datetimetz� extension�bool�object�datetime� timedelta�floatzM8[ns]r �nszm8[ns]z/invalid dtype determination in get_concat_dtype)#r) r# rG rW Zfloat64�nan�_is_uniform_reindexr6 � enumerater r9 �zipr r r �
issubclassrQ Zbool_rg r
r r �subtype�namer
r ra r/ Zna_valuer Z
datetime64Z timedelta64Zfind_common_type� TypeErrorrC )r1 r3 rh ri Zhas_none_blocksZdtypesr` rL Zupcast_classesZnull_upcast_classesrW Z
upcast_cls�cls�g�msgr r r rq _ s�
rq )r1 r c sF t � fdd�� D ��oDt dd� � D ��oDt dd� � D ��oDt� �dkS )z�
Check if the join units consist of blocks of uniform type that can
be concatenated using Block.concat_same_type instead of the generic
_concatenate_join_units (which uses `concat_compat`).
c 3 s& | ] }t |j�t � d j�kV qdS )r N)rQ r# )r r% )r1 r r rp � s z)_is_uniform_join_units.<locals>.<genexpr>c s s | ]}|j p|jjV qd S )N)ra r# r, )r r% r r r rp � s c s s | ]}|j V qd S )N)r )r r% r r r rp � s r! )rH r) )r1 r )r1 r r+ � s r+ )r c C s( t dd� | D ��o&tdd� | D ��dkS )Nc s s | ]}|j o|j jV qd S )N)r# r, )r r% r r r rp � s z&_is_uniform_reindex.<locals>.<genexpr>c S s h | ]}|j jj�qS r )r# rW r� )r r% r r r � <setcomp>� s z&_is_uniform_reindex.<locals>.<setcomp>r! )rH r) )r1 r r r r} � s r} c C s� d| j krF| j }| jdkr d}q�| jjt|d��}| jjt|��| _n>| j}tj| j �}|d |d� |d< | j d d|� | j d< | jd | f| jdd� }|f| jdd� | _t|||d�S )z�
Reduce join_unit's shape along item axis to length.
Extra items that didn't fit are returned as a separate block.
r Nr! )r# r r: )r r# Z
getitem_block�slicer r: rA )Z join_unit�lengthZextra_indexersZ extra_blockZ extra_shaper r r �_trim_join_unit� s
r� c # s� t | �dkr6x$| d D ]}|d |d gfV qW �n�|dkr�d}xL| D ]D}d}x&|D ] \}}|j|�|gfV |}qVW |dk rH||jj7 }qHW �n2dg� � fdd�}ttt| ��} tt|| ��} �x � d t | �k�rĈ d dkr�td��t| � \}
} ttt |
��} t | �t
| � }
}|
|k�rD|
d | fV t|| �| dd�< q�d}dgt | � }x`t | �D ]T\}\}}|||< t |�|
k�r�||
d� t ||
�f| |< n|}|| | �| |< �q`W ||fV q�W dS )z`
Combine multiple concatenation plans into one.
existing_plan is updated in-place.
r! r Nc s& t | d �}|d kr"� d d7 < |S )Nr r! )�next)�seq�retval)� num_endedr r �
_next_or_none& s
z,_combine_concat_plans.<locals>._next_or_nonezPlan shapes are not aligned)
r) �addrE �stopr9 �map�iter�
ValueErrorr �minr^ r~ r� )Zplansr �p�offsetrJ Zlast_plcZplcrL r� Z
next_itemsrK Zunits�lengthsZmin_len�max_lenZyielded_placementZ
yielded_unitsr` r )r� r r(
sD
r( )3� collectionsr r Ztypingr ZnumpyrG Z pandas._libsr r rB Zpandas.util._decoratorsr Zpandas.core.dtypes.castr Zpandas.core.dtypes.commonr r r
r r r
r r r Zpandas.core.dtypes.concatr Zpandas.core.dtypes.missingr Zpandas.core.algorithms�coreZ
algorithmsr> Zpandas.core.arraysr r Z pandas.core.internals.blocksr Z pandas.core.internals.managersr �intrv r5 r rA r. rq r+ r} r� r( r r r r �<module> s0 , 6V -r |