3
���h# � @ s� d Z ddlZddlmZ ddlZddljjZddl m
Z
m Z m Z ddl
mZmZmZmZ dZed�dd �Zd
d ed
feeee ed �d
d�Zd efed�dd�Zeed�dd�Zd ed
feeed�dd�ZdS )z"
data hash pandas / numpy objects
� N)�Optional)�is_categorical_dtype�is_extension_array_dtype� is_list_like)� ABCDataFrame�
ABCIndexClass�
ABCMultiIndex� ABCSeriesZ0123456789123456)� num_itemsc
C s� y t | �}W n tk
r, tjg tjd�S X tj|g| �} tjd�}tj|�tjd� }xBt| �D ]6\}}|| }||N }||9 }|tjd| | �7 }qdW |d |ks�t d��|tjd�7 }|S )z�
Parameters
----------
arrays : generator
num_items : int
Should be the same as CPython's tupleobject.c
)�dtypeiCB ixV4 iXB � zFed in wrong num_itemsi�| )
�next�
StopIteration�np�array�uint64� itertools�chainZ
zeros_like� enumerate�AssertionError)�arraysr
�firstZmult�out�i�aZ inverse_i� r �:/tmp/pip-build-5_djhm0z/pandas/pandas/core/util/hashing.py�_combine_hash_arrays s
r T�utf8)�index�encoding�hash_key�
categorizec
sx ddl m} �dkrt�t�t�r8|t����ddd�S t�t�rpt�j��� �j ddd�}||�ddd�}�nt�t
�r�t�j��� �j ddd�}|rȇ ���fd d
�dD �}t j |g|�}t
|d �}||�jddd�}n�t�t��rbd d
� �j� D �} t�j�}
|�rD� ���fd
d
�dD �} |
d7 }
t j | | �} dd
� | D �} t
| |
�}||�jddd�}ntdt��� ���|S )aX
Return a data hash of the Index/Series/DataFrame.
Parameters
----------
index : bool, default True
Include the index in the hash (if Series/DataFrame).
encoding : str, default 'utf8'
Encoding for data & key when strings.
hash_key : str, default _default_hash_key
Hash_key for string key to encode.
categorize : bool, default True
Whether to first categorize object arrays before hashing. This is more
efficient when the array contains duplicate values.
Returns
-------
Series of uint64, same length as the object
r )�SeriesNr F)r �copy)r$ )r r r$ c 3 s$ | ] }t �jd ��� d�jV qdS )F)r r r! r" N)�hash_pandas_objectr �_values)�.0�_)r" r r! �objr r � <genexpr>e s z%hash_pandas_object.<locals>.<genexpr>� c s s | ]\}}t |j�V qd S )N)�
hash_arrayr& )r' r( Zseriesr r r r* t s c 3 s$ | ] }t �jd ��� d�jV qdS )F)r r r! r" N)r% r r& )r' r( )r" r r! r) r r r* x s r c s s | ]
}|V qd S )Nr )r' �xr r r r* � s z Unexpected type for hashing )N)N)�pandasr# �_default_hash_key�
isinstancer � hash_tuplesr r, r& �astyper r r r r r �items�len�columns� TypeError�type)
r) r r r! r" r# �hZ
index_iterr �hashesr
Zindex_hash_generatorZ_hashesr )r" r r! r) r r% 7 s>
r% )r! c s� d}t �t�r�g�d}nt��s*td��ddlm� m} t �t�sN|j���� �fdd�t �j
�D ����fdd ��D �}t |t ���}|r�|d }|S )
a
Hash an MultiIndex / list-of-tuples efficiently
Parameters
----------
vals : MultiIndex, list-of-tuples, or single tuple
encoding : str, default 'utf8'
hash_key : str, default _default_hash_key
Returns
-------
ndarray of hashed values array
FTz'must be convertible to a list-of-tuplesr )� Categorical�
MultiIndexc s( g | ] }� �j | �j| d dd��qS )FT)�ordered�fastpath)�codesZlevels)r' �level)r: �valsr r �
<listcomp>� s z hash_tuples.<locals>.<listcomp>c 3 s | ]}t |� �d �V qdS ))r r! N)�_hash_categorical)r' �cat)r r! r r r* � s z hash_tuples.<locals>.<genexpr>)
r0 �tupler r6 r. r: r; r � from_tuples�rangeZnlevelsr r4 )r@ r r! Zis_tupler; r9 r8 r )r: r r! r@ r r1 � s
r1 )r r! c C sl t j| jj�}t|||dd�}| j� }t|�r<|j| j�}nt j t|�dd�}|j
� rht j t j �j
||< |S )a
Hash a Categorical by hashing its categories, and then mapping the codes
to the hashes
Parameters
----------
c : Categorical
encoding : str
hash_key : str
Returns
-------
ndarray of hashed values array, same size as len(c)
F)r" r )r )r Zasarray�
categoriesr& r, Zisnar4 Ztaker> �zeros�anyZiinfor �max)�cr r! �values�hashed�mask�resultr r r rB � s rB )r r! r" c C s� t | d�std��| j}t|�r,t| ||�S t|�rF| j� \} }| j}tj|tj �rtt
tj | ��dt
tj | �� S t
|t�r�| jd�} n�t|jtjtjf�r�| jd�jddd�} n�t|jtj�r�|jdkr�| jd | jj� ��jd�} n�|�r2d
d lm}m}m} || dd �\} }
|| ||
�dd
d�} t| ||�S ytj | ||�} W n0 tk
�rt tj | jt �jt �||�} Y nX | | d? N } | tj d�9 } | | d? N } | tj d�9 } | | d? N } | S )a9
Given a 1d array, return an array of deterministic integers.
Parameters
----------
vals : ndarray, Categorical
encoding : str, default 'utf8'
Encoding for data & key when strings.
hash_key : str, default _default_hash_key
Hash_key for string key to encode.
categorize : bool, default True
Whether to first categorize object arrays before hashing. This is more
efficient when the array contains duplicate values.
Returns
-------
1d uint64 numpy array of hash values, same length as the vals
r zmust pass a ndarray-like� �u8�i8F)r$ � �ur )r: �Index� factorize)�sortT)r< r= � l �e�9� �z � l �b&�&�& � ) �hasattrr6 r r rB r Z_values_for_factorizer Z
issubdtypeZ
complex128r, �real�imagr0 �boolr2 �
issubclassr7 Z
datetime64Z timedelta64�view�number�itemsizer. r: rU rV �hashingZhash_object_array�str�objectr ) r@ r r! r" r r( r: rU rV r> rG rC r r r r, � s@
r, )�__doc__r Ztypingr Znumpyr Zpandas._libs.hashingZ_libsrc Zpandas.core.dtypes.commonr r r Zpandas.core.dtypes.genericr r r r r/ �intr r^ rd r% r1 rB r, r r r r �<module> s" R+( |