3
���hh8 � @ s< d Z ddlZddlZddlZddlZddlZddlZddlZddlm Z m
Z
ddl Z ddl Z ddl
mZmZmZ ddljjZddlZddlmZ ddljZddlm Z m Z e� Z e j dd�d d
� �Z dPd d �Z!d
d� Z"dd� Z#dd� Z$dd� Z%dd� Z&dd� Z'dd� Z(dd � Z)ejej*j+ej*j,e-�d d d d ��Z.e j e.d!�d"d � �Z/d#d$� Z0d%d&� Z1d'd(� Z2d)d*� Z3d+d,� Z4e j d-d.� �Z5G d/d0� d0�Z6G d1d2� d2�Z7e j8j9d3d4gd5ejd6d7d8g�fd9ej:j;d6d7d8gd:d;d<gg�fg�d=d>� �Z<d?d@� Z=e j8j9dAdBdCdDg�dEdF� �Z>ej?dG�dHdI� �Z@G dJdK� dKejA�ZBdLdM� ZCdNdO� ZDdS )Qa<
manage legacy pickle tests
How to add pickle tests:
1. Install pandas version intended to output the pickle.
2. Execute "generate_legacy_storage_files.py" to create the pickle.
$ python generate_legacy_storage_files.py <output_dir> pickle
3. Move the created pickle to "data/legacy_pickle/<version>" directory.
� N)�catch_warnings� simplefilter)�_get_lzma_file� _import_lzma�is_platform_little_endian)�Index)�Day�MonthEnd�module)Zscopec C s ddl m} | � S )Nr )�create_pickle_data)Z-pandas.tests.io.generate_legacy_storage_filesr )r � r �=/tmp/pip-build-5_djhm0z/pandas/pandas/tests/io/test_pickle.py�current_pickle_data% s r c C s� t |t�rtj|| � d S |jd�r6tj}|| |� n`|dkrv|tjkrX| tjkstt�q�| |ksdt�| j |j ks�t�n t
td|� d�tj �}|| |� d S )NZsp_� timestampZassert_Z_equal) �
isinstancer �tm�assert_index_equal�
startswithZ assert_equal�pdZNaT�AssertionError�freq�getattrZassert_almost_equal)�result�expected�typ�version�
comparatorr r r
�compare_element0 s
r c
C sz t j|�} t� }xd| j� D ]X\}}xN|j� D ]B\}}| | | }d|� d|� �} |j| |d �} | ||||� q,W qW | S )NZcompare_�_r )r � read_pickle�globals�items�get)
�dataZvfr �mr �dv�dtr r r r r r
�compareC s
r' c C s� t j| |� | jj|jjks t�| jjj s0t�t j| dk|dk� | jj}|td� td�ksdt�|tjdd� }t |tj�s�t�|tjddd�ks�t�|tjdd� }t |tj�s�t�|tjddd�ks�t�d S )Nr � � )�hours)�daysr* )� nanoseconds)r+ r, )
r �assert_series_equal�indexr r � normalizer r Z Timedeltar )r r r r r �resr r r
�compare_series_tsU s r1 c C s t j| |� d S )N)r r- )r r r r r r r
�compare_series_dt_tzi s r2 c C s t j| |� d S )N)r r- )r r r r r r r
�compare_series_catm s r3 c C s t j| |� d S )N)r �assert_frame_equal)r r r r r r r
�compare_frame_dt_mixed_tzsq s r5 c C s t j| |� d S )N)r r4 )r r r r r r r
�compare_frame_cat_onecolu s r6 c C s t | |||� d S )N)r6 )r r r r r r r
�compare_frame_cat_and_floaty s r7 c C sV t j| |� t| jt�s t�| jt� ks,t�| jdks:t�t j| jd�|jd�� d S )N�Mr) )r r r r r r Zfreqstr�shift)r r r r r r r
�compare_index_period} s
r: r# �
legacy_pickle�*z*.pickle)�paramsc C s
|| j �S )N)�param)�request�datapathr r r
r; � s c C sR t � stjd� tjjtjj|��}tdd�� td� t | ||� W d Q R X d S )Nz"known failure on non-little endianT)�record�ignore)
r �pytest�skip�os�path�basename�dirnamer r r' )r r; r r r r
� test_pickles� s
rI c C s� dd� }dd� }| }x�|j � D ]�\}}xz|j � D ]n\}}xdtj|gD ]V}|d krRqDtj� �:} ||| � tj| �}
t|
||� || �}
t|
||� W d Q R X qDW q0W q W d S )Nc S s* t |d��}tj| |dd� W d Q R X d S )N�wbr( )�protocol���)�open�pickle�dump)�objrF �fhr r r
�python_pickler� s z/test_round_trip_current.<locals>.python_picklerc S s* t | d��}|jd� tj|�S Q R X d S )N�rbr )rM �seekrN �load)rF rQ r r r
�python_unpickler� s
z1test_round_trip_current.<locals>.python_unpickler)r! r � to_pickler � ensure_cleanr r ) r rR rV r# r r% r& r �writerrF r r r r
�test_round_trip_current� s
rZ c C s( t j� } t j| jtj�}t j| |� d S )N)r �
makeDataFrameZround_trip_pathlibrW r r r4 )�dfr r r r
�test_pickle_path_pathlib� s r] c C s( t j� } t j| jtj�}t j| |� d S )N)r r[ Zround_trip_localpathrW r r r4 )r\ r r r r
�test_pickle_path_localpath� s r^ c C st t jt��( tdt� tj| dddd�dd� W dQ R X t jt��( tdt� tj| dddd �dd� W dQ R X dS )
ap
Generated with
>>> df = pd.DataFrame({"A": [1, 2, 3, 4], "B": [0, 0, 1, 1]}).to_sparse()
>>> df.to_pickle("pandas/tests/io/data/pickle/sparseframe-0.20.3.pickle.gz",
... compression="gzip")
>>> s = df['B']
>>> s.to_pickle("pandas/tests/io/data/pickle/sparseseries-0.20.3.pickle.gz",
... compression="gzip")
rB �ior# rN z sparseseries-0.20.3.pickle.gz�gzip)� compressionNz sparseframe-0.20.3.pickle.gz)r Zassert_produces_warning�
FutureWarningr �DeprecationWarningr r )r@ r r r
�test_legacy_sparse_warning� s
rd c C s dt jd�� d�S )N�__�
z __.pickle)r Zrandsr r r r
�get_random_path� s rg c @ s� e Zd Zdddddd�Zdd� Zd d
� Zejjd d d
ddg�dd� �Z ejjdd ddddg�dd� �Z
dd� Z ejjdd dddddg�dd� �Z dS )�TestCompressionz.nonez.gzz.bz2z.zipz.xz)Nr` �bz2�zip�xzc C s� |d krt j||� d S |dkr.tj|d�}n||dkrDtj|d�}nf|dkr�tj|dtjd��}|j |t
j j |�� W d Q R X n*|dkr�t
t�|d�}nd|� �}t|��|dkr�t|d��$}|� |j |j� � W d Q R X W d Q R X d S ) Nr` �wri rj )ra rk z Unrecognized compression type: rS )�shutil�copyfiler` rM ri �BZ2File�zipfile�ZipFile� ZIP_DEFLATED�writerE rF rG r �lzma�
ValueError�read)�selfZsrc_path� dest_pathra �f�msgrQ r r r
�
compress_file� s"
z TestCompression.compress_filec C s� |}|d }|d }t j|���}t j|��t}t j� }|j||d� t j||d��*} t|d��}
|
j| j� � W d Q R X W d Q R X tj |d d�} t j
|| � W d Q R X W d Q R X d S )Nz .compressedz.raw)ra rJ ) r rX r[ rW �decompress_filerM rs rv r r r4 ) rw ra rg �base�path1�path2�p1�p2r\ ry rQ �df2r r r
�test_write_explicit
s "z#TestCompression.test_write_explicitra � �None�badZ7zc C sJ t jtdd��2 tj|�� }tj� }|j||d� W d Q R X W d Q R X d S )Nz Unrecognized compression type)�match)ra )rC Zraisesru r rX r[ rW )rw ra rg rF r\ r r r
�test_write_explicit_bad" s z'TestCompression.test_write_explicit_bad�extz .no_compressc C s� |}|| }|d }d }x"| j D ]}| j | |kr |}P q W tj|���}tj|��p} tj� }
|
j|� tj||d��*} t| d��} | j| j� � W d Q R X W d Q R X t j
| d d�}
tj |
|
� W d Q R X W d Q R X d S )Nz.raw)ra rJ ) �_compression_to_extensionr rX r[ rW r| rM rs rv r r r4 )rw r� rg r} r~ r ra �cr� r� r\ ry rQ r� r r r
�test_write_infer) s
"z TestCompression.test_write_inferc
C s� |}|d }|d }t j|��\}t j|��F}t j� }|j|d d� | j|||d� tj||d�} t j|| � W d Q R X W d Q R X d S )Nz.rawz .compressed)ra )r rX r[ rW r{ r r r4 )
rw ra rg r} r~ r r� r� r\ r� r r r
�test_read_explicitD s z"TestCompression.test_read_explicitc C s� |}|d }|| }d }x"| j D ]}| j | |kr |}P q W tj|��X}tj|��B} tj� }
|
j|d d� | j|| |d� tj| �} tj|
| � W d Q R X W d Q R X d S )Nz.raw)ra ) r� r rX r[ rW r{ r r r4 ) rw r� rg r} r~ r ra r� r� r� r\ r� r r r
�test_read_inferW s
z TestCompression.test_read_inferN)
�__name__�
__module__� __qualname__r� r{ r� rC �mark� parametrizer� r� r� r� r r r r
rh � s rh c @ s* e Zd Zejjdddddg�dd� �ZdS ) � TestProtocolrK r( r r) c C sF t j|��2}t j� }|j||d� tj|�}t j||� W d Q R X d S )N)rK )r rX r[ rW r r r4 )rw rK rg rF r\ r� r r r
� test_readw s
zTestProtocol.test_readNrL )r� r� r� rC r� r� r� r r r r
r� v s r� � pickle_file�excolsz
test_py27.pkl�a�br� ztest_mi_py27.pkl�A�B�Cc C s* | ddd|�}t j|�}tj|j|� d S )Nr_ r# rN )r r r r �columns)r@ r� r� rF r\ r r r
�test_unicode_decode_error� s
r� c C sl t j� �Z} t j� }t| d��}|j|� W d Q R X t| d��}tj|�}W d Q R X t j||� W d Q R X d S )NrJ rS )r rX r[ rM rW r r r4 )rF r\ rQ r r r r
� test_pickle_buffer_roundtrip� s
r� �mockurlzhttp://url.comzftp://test.comzhttp://gzip.comc
sp dd� }G dd� d�� t j� �H�� �fdd�}t j� }||�� | jd|� tj|�}t j||� W d Q R X d S )Nc S s* t |d��}tj| |dd� W d Q R X d S )NrJ r( )rK rL )rM rN rO )rP rF rQ r r r
rR � s z3test_pickle_generalurl_read.<locals>.python_picklerc @ s$ e Zd Zdd� Zdd� Zdd� ZdS )z5test_pickle_generalurl_read.<locals>.MockReadResponsec S s. t |d�| _d|kr ddi| _n
dd i| _d S )NrS r` zContent-Encoding)rM �file�headers)rw rF r r r
�__init__� s z>test_pickle_generalurl_read.<locals>.MockReadResponse.__init__c S s
| j j� S )N)r� rv )rw r r r
rv � s z:test_pickle_generalurl_read.<locals>.MockReadResponse.readc S s
| j j� S )N)r� �close)rw r r r
r� � s z;test_pickle_generalurl_read.<locals>.MockReadResponse.closeN)r� r� r� r� rv r� r r r r
�MockReadResponse� s r� c s � ��S )Nr )�args�kwargs)r� rF r r
�mock_urlopen_read� s z6test_pickle_generalurl_read.<locals>.mock_urlopen_readzurllib.request.urlopen)r rX r[ �setattrr r r4 )Z monkeypatchr� rR r� r\ r r )r� rF r
�test_pickle_generalurl_read� s
r� Zfsspecc
C sD t j� �2 d} t j� }|j| � tj| �}t j||� W d Q R X d S )Nzmemory://afile)r rX r[ rW r r r4 )r� r\ r r r r
� test_pickle_fsspec_roundtrip� s
r� c @ s e Zd Zdd� ZdS )�MyTzc C s d S )Nr )rw r r r
r� � s z
MyTz.__init__N)r� r� r� r� r r r r
r� � s r� c C sF t jtd�t� f} tj| �}tj|d | d � t|d t�sBt�d S )N)Zdtyper r( ) r �Series�objectr� r �round_trip_pickler- r r )r r r r r
� test_read_pickle_with_subclass� s
r� c C sh t jtd��jd�jdg } tj| �}|jjd j dks<t
�|jjd j dksRt
�tj |dg | � d S )N�abc�categoryr r( T)r( )
r r� �listZastypeZilocr r� Z_mgr�blocks�ndimr �shaper- )Zserr0 r r r
� test_pickle_preserves_block_ndim� s
r� )N)E�__doc__ri �datetime�globr` rE rN rm �warningsr r rp rC Z
pandas.compatr r r Z pandas.util._test_decorators�utilZ_test_decorators�tdZpandasr r Zpandas._testingZ_testingr Zpandas.tseries.offsetsr r rt Zfixturer r r' r1 r2 r3 r5 r6 r7 r: rF �joinrH �__file__�filesr; rI rZ r] r^ rd rg rh r� r� r� Z
MultiIndexZ from_arraysr� r� r� Z
skip_if_nor� �tzinfor� r� r� r r r r
�<module> sf
!
*!
|