3
���h=n � @ s� d Z ddlZddlmZ ddlmZ ddlZddlmZ ddl Z
ddl Z ddl j
jZddlZddljZddlmZmZmZmZmZ yddlZdZ W n e k
r� dZ Y nX yddl Z dZ W n e k
r� dZ Y nX e j j!d �Z"e j#e j$d
e j j%e d d �d
�e j$de j j%e dd �d
�gd�dd� �Z&e j#dd� �Z'e j#dd� �Z(e j#dd� �Z)e j#dd� �Z*e j#dd � �Z+d8d d �Z,d d!� Z-d"d#� Z.d$d%� Z/d&d'� Z0d(d)� Z1d*d+� Z2d,d-� Z3d.d/� Z4G d0d1� d1�Z5G d2d3� d3e5�Z6G d4d5� d5e5�Z7G d6d7� d7e5�Z8dS )9z test parquet compat � N)� LooseVersion)�BytesIO)�catch_warnings)�FastParquetImpl� PyArrowImpl�
get_engine� read_parquet�
to_parquetTFz4ignore:RangeIndex.* is deprecated:DeprecationWarning� fastparquetz fastparquet is not installed)�reason)Zmarks�pyarrowzpyarrow is not installed)�paramsc C s | j S )N)�param)�request� r �>/tmp/pip-build-5_djhm0z/pandas/pandas/tests/io/test_parquet.py�engine- s r c C s t stjd� dS )Nzpyarrow is not installedr )�
_HAVE_PYARROW�pytest�skipr r r r �paA s
r c C s t stjd� dS )Nz fastparquet is not installedr
)�_HAVE_FASTPARQUETr r r r r r �fpH s
r c C s t jdddgdd��S )N� � � �foo)�A�B)�pd� DataFramer r r r � df_compatO s r! c C sD t jtd�ttdd��tjdddd�dd dgt jd
d d �d
��} | S )N�abcr � g @g @�float64)�dtypeTF�20130101r )�periods)�a�b�d�e�f)r r �list�range�np�arange�
date_range)�dfr r r �df_cross_compatT s r3 c C s� t jtd�dtjdgdd dgdddgddd gttd
d ��tjd d
�jd�tjdddd�dtjdgdddgt jdd d�t j d�t j
t j d�gd� �S )Nr" r( �cs foos bars bazr �bar�bazr r# r � �u1g @g @r$ )r% g @g @TFr& )r' Z20130103) �stringZstring_with_nanZstring_with_none�bytes�unicode�intZuint�floatZfloat_with_nan�bool�datetimeZdatetime_with_nat) r r r- r/ �nanr. r0 �astyper1 � TimestampZNaTr r r r �df_fullf s
rC r c
s~ �p
ddi��pi ��dkr ��|r4|�d< |�d< � ������fdd�} �dkrrt j� ��| |� W dQ R X n| |� dS )a Verify parquet serializer and deserializer produce the same results.
Performs a pandas to disk and disk to pandas round trip,
then compares the 2 resulting DataFrames to verify equality.
Parameters
----------
df: Dataframe
engine: str, optional
'pyarrow' or 'fastparquet'
path: str, optional
write_kwargs: dict of str:str, optional
read_kwargs: dict of str:str, optional
expected: DataFrame, optional
Expected deserialization result, otherwise will be equal to `df`
check_names: list of str, optional
Closed set of column names to be compared
check_like: bool, optional
If True, ignore the order of index & columns.
repeat: int, optional
How many times to repeat the test
� compressionNr c sX xRt | �D ]F}�j�f�� tdd�� t�f��}W d Q R X tj�|�� d� q
W d S )NT)�record)� check_names�
check_like)r. r r r �tm�assert_frame_equal)�repeat�_�actual)rG rF r2 �expected�path� read_kwargs� write_kwargsr r �compare� s z!check_round_trip.<locals>.compare)rH � ensure_clean)
r2 r rN rP rO rM rF rG rJ rQ r )rG rF r2 rM rN rO rP r �check_round_trip~ s !
rS c C s& t jt�� t| dd� W d Q R X d S )Nr r5 )r �raises�
ValueErrorrS )r! r r r �test_invalid_engine� s rV c C s$ t jdd�� t| � W d Q R X d S )Nzio.parquet.enginer )r �option_contextrS )r! r r r r �test_options_py� s rX c C s$ t jdd�� t| � W d Q R X d S )Nzio.parquet.enginer
)r rW rS )r! r r r r �test_options_fp� s rY c C s$ t jdd�� t| � W d Q R X d S )Nzio.parquet.engine�auto)r rW rS )r! r r r r r �test_options_auto� s r[ c C s t td�t�st�t td�t�s$t�tjdd��< t td�t�sDt�t td�t�sVt�t td�t�sht�W d Q R X tjdd��< t td�t�s�t�t td�t�s�t�t td�t�s�t�W d Q R X tjdd��> t td�t�s�t�t td�t�s�t�t td�t��st�W d Q R X d S )Nr r
zio.parquet.enginerZ )�
isinstancer r �AssertionErrorr r rW )r r r r r �test_options_get_engine� s r^ c C s0 ddl m} | jd�}| jd�}ts(dnttj�t|�k }tsBdnttj�t|�k }to\| }tof| }| or| �r,|r�d|� d�}t j
t |d�� t d � W d Q R X n&d
}t j
t |d�� t d � W d Q R X |�rd|� d �}t j
t |d�� t d � W d Q R X n&d }t j
t |d�� t d � W d Q R X d S )
Nr )�VERSIONSr r
FzPandas requires version .z. or newer of .pyarrow.)�matchrZ z%Missing optional dependency .pyarrow.z. or newer of .fastparquet.z)Missing optional dependency .fastparquet.)
Zpandas.compat._optionalr_ �getr r r � __version__r r
r rT � ImportErrorr )r_ Z
pa_min_verZ
fp_min_verZhave_pa_bad_versionZhave_fp_bad_versionZhave_usable_paZhave_usable_fpr` r r r �"test_get_engine_auto_error_message� s0
rd c C sj | }t j� �T}|j||d d� t||d�}t j||� t||ddgd�}t j||ddg � W d Q R X d S )N)r rD )r r( r* )r �columns)rH rR r r rI )r3 r r r2 rN �resultr r r �test_cross_engine_pa_fp s
rg c C s� t tj�dk r&t tj�dkr&tjd� | }tj� �j}|j||d d� tdd��D t ||d�}tj
||� t ||dd gd
�}tj
||dd g � W d Q R X W d Q R X d S ) Nz0.15z0.13z`Reading fastparquet with pyarrow in 0.14 fails: https://issues.apache.org/jira/browse/ARROW-6492)r rD T)rE )r r( r* )r re ) r r rb r ZxfailrH rR r r r rI )r3 r r r2 rN rf r r r �test_cross_engine_fp_pa! s
rh c @ s e Zd Zdd� ZdS )�Basec C s> t j� �,}tj|�� t|||d d� W d Q R X W d Q R X d S )N)rD )rH rR r rT r )�selfr2 r �excrN r r r �check_error_on_write: s
zBase.check_error_on_writeN)�__name__�
__module__� __qualname__rl r r r r ri 9 s ri c @ sr e Zd Zdd� Zdd� Zdd� Zejjddd d
d g�d d
� �Z dd� Z
dd� Z dd� Z dd� Z
dd� Zdd� ZdS )� TestBasicc C sH xBt jdddg�ddt jd�tjdddg�gD ]}| j||t� q.W d S )Nr r r r r& )r �SeriesrB r/ �arrayrl rU )rj r �objr r r �
test_errorB s zTestBasic.test_errorc C s6 t jtd�ttdd��d��}ddg|_t||� d S )Nr" r r# )r9 r< r r5 )r r r- r. re rS )rj r r2 r r r �test_columns_dtypesL s
z TestBasic.test_columns_dtypesc C s� t jtd�ttdd��d��}ddg|_| j||t� ddg|_| j||t� tjddddd�tjddddd�g|_| j||t� d S ) Nr" r r# )r9 r< r s foos bari� )r r r- r. re rl rU r? )rj r r2 r r r �test_columns_dtypes_invalidS s
z%TestBasic.test_columns_dtypes_invalidrD N�gzip�snappy�brotlic C sP |dkrt jd� n|dkr&t jd� tjddddgi�}t||d|id� d S ) Nrx ry r r r r rD )rP )r � importorskipr r rS )rj r rD r2 r r r �test_compressione s
zTestBasic.test_compressionc C sJ t jtd�ttdd��d��}t jdtd�i�}t|||ddgid� d S )Nr" r r# )r9 r< r9 re )rM rO )r r r- r. rS )rj r r2 rM r r r �test_read_columnsq s zTestBasic.test_read_columnsc C s� |dk}t jddddgi�}t||� dddgt jddd�td �dddgg}x:|D ]2}||_t|t j�rv|jjd �|_t|||d
� qRW d ddg|_d |j_ t||� d S )
Nr
r r r r r# r& )r' r" )rF r r )
r r rS r1 r- �indexr\ Z
DatetimeIndex�
_with_freq�name)rj r rF r2 �indexesr} r r r �test_write_indexz s
zTestBasic.test_write_indexc C s> |}t jddddgi�}t jjddd g�}||_t||� d S )
Nr r r r r( r) )r( r )r( r )r) r )r r �
MultiIndex� from_tuplesr} rS )rj r r r2 r} r r r �test_write_multiindex� s
z TestBasic.test_write_multiindexc C s< t jjdd d
g�}t jtjjdd�|d�}| j||t� d S ) Nr( r r r) r# r )re )r( r )r( r )r) r ) r r� r� r r/ �random�randnrl rU )rj r Z
mi_columnsr2 r r r � test_write_column_multiindex� s z&TestBasic.test_write_column_multiindexc C s� |}t jdddd�}t jtjjdt|� d�td�d�}t jj d d
g|gd d gd
�}|j
d d
�}x@||gD ]4}||_ t ||� t ||dddgi|ddg d� qjW d S )Nz 01-Jan-2018z 01-Dec-2018ZMS)�freqr r �ABC)re ZLevel1ZLevel2�level�date)�namesre r r )rO rM )
r r1 r r/ r� r� �lenr- r� Z from_product�copyr} rS )rj r r �datesr2 Zindex1Zindex2r} r r r � test_multiindex_with_columns� s $
z&TestBasic.test_multiindex_with_columnsc C s� t jdddgdddgd��}d dd �}|jd
d �}t||||d � t jdddgdddgd�d
ddgd�}t||||d � ddddddddgddddddddgg}t jttd��dd� td�D �d�|d�}|jd
d �}t||||d � d S )Nr r r �q�r�s)r( r) F)rD r} T)Zdrop)rP rM ZzyxZwvuZtsr)r} r5 r6 r Zqux�one�two� c S s g | ]
}| �qS r r )�.0�ir r r �
<listcomp>� s z7TestBasic.test_write_ignoring_index.<locals>.<listcomp>)r� r� )r r Z reset_indexrS r- r. )rj r r2 rP rM Zarraysr r r �test_write_ignoring_index� s
"& z#TestBasic.test_write_ignoring_index)rm rn ro rt ru rv r �mark� parametrizer{ r| r� r� r� r� r� r r r r rp A s
rp c @ s e Zd Zdd� Zdd� Zdd� Zdd� Zd d
� Zd d � Zd
d� Z e
j d�e j
jddgg g�dd� ��Zeje
j d�dd� ��Ze
j d�dd� �Ze
j d�dd� �Zdd � Zd d � Zd d � Zd!d"� Ze
j dd#d$�d%d&� �Ze
j dd'd$�d(d)� �Ze
j dd*d$�d+d,� �Ze
j dd-d$�d.d/� �Z d0S )1�TestParquetPyArrowc C sB |}t jdddd�}|jd �}||d< dd dg|d< t||� d S )Nr& r zEurope/Brussels)r' �tz� datetime_tzTZbool_with_none)r r1 r~ rS )rj r rC r2 �dtir r r �
test_basic� s
z TestParquetPyArrow.test_basicc C s< |}t jdddd�|d< t|||ddg dddgid � d S )
Nr& r zEurope/Brussels)r' r� r� r9 r< re )rM rO )r r1 rS )rj r rC r2 r r r �test_basic_subset_columns� s
z,TestParquetPyArrow.test_basic_subset_columnsc C s6 t jtjd�jdd�td�d�j� }| j||t� d S )N� r# r �aaa)re ) r r r/ r0 �reshaper- r� rl rU )rj r r2 r r r �test_duplicate_columns� s $z)TestParquetPyArrow.test_duplicate_columnsc C s� t tj�t d�k r:tjdtjdddd�i�}| j||t� tjdtjddd�i�}| j||t � tjddd d
gi�}| j||t� d S ) Nz
0.15.1.devr( �2013�Mr )r� r' z1 day)r' r g @)
r r rb r r � period_rangerl � Exception�timedelta_range�NotImplementedError)rj r r2 r r r �test_unsupported� s z#TestParquetPyArrow.test_unsupportedc C s� t j� }t jtd��|d< t jddddd dgt jdddg�d�|d< t jddddddgddd gd
d �|d< ttj�td �kr�t||� n|j t
�}t|||d
� d S )NZabcdefr( r5 r r6 )r% r) r4 r* T)�
categories�orderedz0.15.0)rM ) r r � Categoricalr- ZCategoricalDtyper r rb rS rA �object)rj r r2 rM r r r �test_categorical s "
z#TestParquetPyArrow.test_categoricalc C s2 t jd�}|j� }t|d�}t||d||d� d S )N�s3fs)�
filesystemzpandas-test/pyarrow.parquet)rN rO rP )r rz Z S3FileSystem�dictrS )rj r! � s3_resourcer r� Zs3�kwr r r � test_s3_roundtrip_explicit_fs s
z0TestParquetPyArrow.test_s3_roundtrip_explicit_fsc C s t ||dd� d S )Nz s3://pandas-test/pyarrow.parquet)rN )rS )rj r! r� r r r r �test_s3_roundtrip( s z$TestParquetPyArrow.test_s3_roundtripr� �
partition_colr c C sp |j � }ttj�td�ko*ttj�td�k }|rP|r:d}nd}|| j|�||< t|||d|d d�ddd � d S )
Nz1.0.0z2.0.0Zint32�categoryz s3://pandas-test/parquet_dir)�partition_colsrD Tr )rM rN rP rG rJ )r� r r rb rA rS )rj r! r� r r� Z expected_dfZpa10Zpartition_col_typer r r �test_s3_roundtrip_for_dir, s"
z,TestParquetPyArrow.test_s3_roundtrip_for_dirr c C s d}t j|�}tj||� d S )Nzfhttps://raw.githubusercontent.com/pandas-dev/pandas/master/pandas/tests/io/data/parquet/simple.parquet)r r rH rI )rj r! �urlr2 r r r �test_parquet_read_from_urlO s
z-TestParquetPyArrow.test_parquet_read_from_urlc C s* t � }|j|� tj|�}tj||� d S )N)r r r r rH rI )rj r! �bufferZ df_from_bufr r r � test_read_file_like_obj_supportY s
z2TestParquetPyArrow.test_read_file_like_obj_supportc C sd |j dd� |j dd� tjtdd�� tjd� W d Q R X tjtdd�� |jd� W d Q R X d S )N�HOMEZ TestingUserZ USERPROFILEz.*TestingUser.*)r` z~/file.parquet)Zsetenvr rT �OSErrorr r r )rj r! Z monkeypatchr r r �test_expand_user` s z#TestParquetPyArrow.test_expand_userc C st ddg}|}t j� �V}|j||d d� dd lj}|j|dd�}t|jj�dksRt �|jjt
|�ksft �W d Q R X d S )Nr> r< )r� rD r F)�validate_schemar ) rH �ensure_clean_dirr �pyarrow.parquet�parquet�ParquetDatasetr� �
partitions�partition_namesr] �set)rj r rC r� r2 rN �pq�datasetr r r � test_partition_cols_supportedi s
z0TestParquetPyArrow.test_partition_cols_supportedc C sv d}|g}|}t j� �V}|j||d d� dd lj}|j|dd�}t|jj�dksTt �|jjt
|�ksht �W d Q R X d S )Nr> )r� rD r F)r� r ) rH r� r r� r� r� r� r� r� r] r� ) rj r rC r� Zpartition_cols_listr2 rN r� r� r r r �test_partition_cols_stringu s
z-TestParquetPyArrow.test_partition_cols_stringc C s t j� }t||� d S )N)r r rS )rj r r2 r r r �test_empty_dataframe� s z'TestParquetPyArrow.test_empty_dataframec C sV dd l }tjdddgi�}|j|jd|j� d�g�}|jt�}t||d|i|d� d S )Nr �xr )�type�schema)rP rM ) r r r r� �fieldZbool_rA r> rS )rj r r r2 r� Zout_dfr r r �test_write_with_schema� s
z)TestParquetPyArrow.test_write_with_schemaz0.15.0)� min_versionc C s� t jt jdddgdd�t jdddgdd�t jdd dgd d�d
��}ttj�td �krZ|}n(|j|jjd �|j jd �|j
jd
�d
�}t |||d� t jdt jdddd gdd�i�}ttj�td �kr�|}n|j|jjd�d�}t |||d� d S )Nr r r ZInt64)r% ZUInt32r( r4 r9 )r( r) r4 z0.16.0Zint64r� )rM r$ )r( ) r r rq r r rb Zassignr( rA r) r4 rS )rj r r2 rM r r r � test_additional_extension_arrays� s $ z3TestParquetPyArrow.test_additional_extension_arraysz0.16.0c C s( t jdt jdddd�i�}t||� d S )Nr* z
2012-01-01r �D)r' r� )r r r� rS )rj r r2 r r r � test_additional_extension_types� s z2TestParquetPyArrow.test_additional_extension_typesz0.14c C s0 t jdt jdddd�i�}t||ddid� d S ) Nr( z
2017-01-01Z1n�
)r� r' �versionz2.0)rP )r r r1 rS )rj r r2 r r r �test_timestamp_nanoseconds� s z-TestParquetPyArrow.test_timestamp_nanosecondsz0.17c
C s^ t jdttdd��i�}tj� �$}|j||� t||dgdd�}W d Q R X t|�dksZt �d S ) Nr( r r �==F)�filtersZuse_legacy_datasetr )r( r� r )
r r r- r. rH rR r r r� r] )rj r r2 rN rf r r r �test_filter_row_groups� s
z)TestParquetPyArrow.test_filter_row_groupsN) rm rn ro r� r� r� r� r� r� r� �td�
skip_if_nor r� r� r� rH �networkr� r� r� r� r� r� r� r� r� r� r� r r r r r� � s* "
r� c @ s� e Zd Zejddd�dd� �Zejjdd�dd � �Z d
d � Z
d d
� Z dd� Z dd� Z
dd� Zdd� Zdd� Zdd� Zdd� Zd d � Zd S ) �TestParquetFastParquetr
z0.3.2)r� c C sF |}t jdddd�}|jd �}||d< t jddd�|d< t||� d S ) Nr& r z
US/Eastern)r' r� r� z1 day)r' � timedelta)r r1 r~ r� rS )rj r rC r2 r� r r r r� � s
z!TestParquetFastParquet.test_basicz
not supported)r c C s6 t jtjd�jdd�td�d�j� }| j||t� d S )Nr� r# r r� )re ) r r r/ r0 r� r- r� rl rU )rj r r2 r r r r� � s $z-TestParquetFastParquet.test_duplicate_columnsc C s@ t jddd dgi�}t jddtjdgidd�}t|||d� d S ) Nr( TFg �?g Zfloat16)r% )rM )r r r/ r@ rS )rj r r2 rM r r r �test_bool_with_none� s z*TestParquetFastParquet.test_bool_with_nonec C sN t jdt jdddd�i�}| j||t� t jddddgi�}| j||t� d S )Nr( r� r� r )r� r' r g @)r r r� rl rU )rj r r2 r r r r� � s z'TestParquetFastParquet.test_unsupportedc C s& t jdt jtd��i�}t||� d S )Nr( r" )r r r� r- rS )rj r r2 r r r r� � s z'TestParquetFastParquet.test_categoricalc
C sf dt tdd��i}tj|�}tj� �(}|j||d dd� t||dgd�}W d Q R X t|�dksbt �d S ) Nr( r r r )rD Zrow_group_offsets�==)r� )r( r� r )
r- r. r r rH rR r r r� r] )rj r r* r2 rN rf r r r r� � s
z-TestParquetFastParquet.test_filter_row_groupsc C s t ||dd� d S )Nz$s3://pandas-test/fastparquet.parquet)rN )rS )rj r! r� r r r r r� � s z(TestParquetFastParquet.test_s3_roundtripc
C sl ddg}|}t j� �N}|j|d|d d� tjj|�s8t�dd l}|j|d�j }t
|�dks^t�W d Q R X d S )Nr> r< r
)r r� rD r Fr ) rH r� r �osrN �existsr] r
� ParquetFile�catsr� )rj r rC r� r2 rN r
�actual_partition_colsr r r r� � s
z4TestParquetFastParquet.test_partition_cols_supportedc
C sh d}|}t j� �N}|j|d|d d� tjj|�s4t�dd l}|j|d�j }t
|�dksZt�W d Q R X d S )Nr> r
)r r� rD r Fr ) rH r� r r� rN r� r] r
r� r� r� )rj r rC r� r2 rN r
r� r r r r�
s
z1TestParquetFastParquet.test_partition_cols_stringc
C sl ddg}|}t j� �N}|j|dd |d� tjj|�s8t�dd l}|j|d�j }t
|�dks^t�W d Q R X d S )Nr> r< r
)r rD � partition_onr Fr ) rH r� r r� rN r� r] r
r� r� r� )rj r rC r� r2 rN r
r� r r r �test_partition_on_supported s
z2TestParquetFastParquet.test_partition_on_supportedc C sN ddg}|}t jt��. tj� �}|j|dd ||d� W d Q R X W d Q R X d S )Nr> r< r
)r rD r� r� )r rT rU rH r� r )rj r rC r� r2 rN r r r �3test_error_on_using_partition_cols_and_partition_on/ s
zJTestParquetFastParquet.test_error_on_using_partition_cols_and_partition_onc C s* t j� }|j� }d|j_t|||d� d S )Nr} )rM )r r r� r} r rS )rj r r2 rM r r r r� = s z+TestParquetFastParquet.test_empty_dataframeN)rm rn ro r� r� r� r r� r r� r� r� r� r� r� r� r� r� r� r� r r r r r� � s
r� )NNNNNTFr )9�__doc__r? Zdistutils.versionr �ior r� �warningsr Znumpyr/ r Z pandas.util._test_decorators�utilZ_test_decoratorsr� Zpandasr Zpandas._testingZ_testingrH Zpandas.io.parquetr r r r r r r rc r
r r� �filterwarningsZ
pytestmarkZfixturer Zskipifr r r r! r3 rC rS rV rX rY r[ r^ rd rg rh ri rp r� r� r r r r �<module> sv
3+ z |