3
���hJH � @ sb d Z ddlZddlmZ ddlZddlmZmZmZ ddl Z ddl
Z
ddl Z ddl m
Z
mZmZmZmZmZmZmZmZmZmZ ddlmZmZm Z m Z m Z ddl Z ddl m!Z! ddl"m#Z#m$Z$ dd l%m&Z& dd
l'm(Z( e$� Z)e*e e e �Z+e+j,d � e�r
dd lm-Z- e.d
�dd�Z/e!e e!e d�dd�Z0dd
�dd�Z1e!e e!e d�dd�Z2dd� Z3e!e.d�dd�Z4d6e!ee5 ee5 ee5 eee5ef d �d d �Z6e5e5d �d d!�Z7d"d#d$d%d&�Z8eee5ee5e5f f eee5 ee5e5f f d'�d(d)�Z9e!ee5 ee5 d*�d+d,�Z:d7e5eee5ee5ef f e.e.d/�d0d1�Z;G d2d3� d3e j<e�Z=G d4d5� d5ej>�Z?dS )8zCommon IO api utilities� N)�abc)�BufferedIOBase�BytesIO� RawIOBase) �IO�
TYPE_CHECKING�Any�AnyStr�Dict�List�Mapping�Optional�Tuple�Type�Union)�urljoin�urlparse� uses_netloc� uses_params�
uses_relative)�FilePathOrBuffer)�_get_lzma_file� _import_lzma)�import_optional_dependency)� is_file_like� )�IOBase)�returnc C s t | t�sdS t| �jtkS )z�
Check to see if a URL has a valid protocol.
Parameters
----------
url : str or unicode
Returns
-------
isurl : bool
If `url` has a valid protocol return True otherwise False.
F)�
isinstance�str� parse_url�scheme� _VALID_URLS)�url� r$ �2/tmp/pip-build-5_djhm0z/pandas/pandas/io/common.py�is_url1 s
r& )�filepath_or_bufferr c C s t | t�rtjj| �S | S )a]
Return the argument with an initial component of ~ or ~user
replaced by that user's home directory.
Parameters
----------
filepath_or_buffer : object to be converted if possible
Returns
-------
expanded_filepath_or_buffer : an expanded filepath or the
input if not expandable
)r r �os�path�
expanduser)r' r$ r$ r% � _expand_userC s
r+ c C s t | t�rtd��d S )Nz�Passing a bool to header is invalid. Use header=None for no header or header=int or list-like of ints to specify the row(s) making up the column names)r �bool� TypeError)�headerr$ r$ r% �validate_header_argX s
r/ c C s. t | d�r| j� S t| tj�r&t| �S t| �S )ad
Attempt to convert a path-like object to a string.
Parameters
----------
filepath_or_buffer : object to be converted
Returns
-------
str_filepath_or_buffer : maybe a string version of the object
Notes
-----
Objects supporting the fspath protocol (python 3.6+) are coerced
according to its __fspath__ method.
For backwards compatibility with older pythons, pathlib.Path and
py.path objects are specially coerced.
Any other object is passed through unchanged, which includes bytes,
strings, buffers, or anything else that's not even path-like.
�
__fspath__)�hasattrr0 r �pathlib�Pathr r+ )r' r$ r$ r% �stringify_patha s
r4 c O s ddl }|jj| |�S )z`
Lazy-import wrapper for stdlib urlopen, as that imports a big chunk of
the stdlib.
r N)�urllib.request�request�urlopen)�args�kwargs�urllibr$ r$ r% r7 � s r7 )r# r c C s t | t�o d| ko | jd� S )zR
Returns true if the given URL looks like
something fsspec can handle
z://�http://�https://)r; r< )r r �
startswith)r# r$ r$ r% �
is_fsspec_url� s
r> )r' �encoding� compression�mode�storage_optionsc C s� t | �} t| t�r\t| �r\t| �}|jjdd�}|dkr<d}t|j� �}|j � |||dfS t
| ��r�t| t�stt �| j d�r�| j
dd�} | j d�r�| j
dd�} td�}g } y&td � d
d lm}
m} |
| tg} W n tk
r� Y nX y(|j| fd |p�d
i|�pi ��j� } W nd t| �k
�rt |dk�r:ddi}nt|�}d|d< |j| fd |�p\d
i|�pfi ��j� } Y nX | ||dfS t| tttjf��r�t| �d|dfS t| ��s�dt| �� �}
t |
��| d|dfS )a�
If the filepath_or_buffer is a url, translate and return the buffer.
Otherwise passthrough.
Parameters
----------
filepath_or_buffer : a url, filepath (str, py.path.local or pathlib.Path),
or buffer
compression : {{'gzip', 'bz2', 'zip', 'xz', None}}, optional
encoding : the encoding to use to decode bytes, default is 'utf-8'
mode : str, optional
storage_options: dict, optional
passed on to fsspec, if using it; this is not yet accessed by the public API
Returns
-------
Tuple[FilePathOrBuffer, str, str, bool]
Tuple containing the filepath or buffer, the encoding, the compression
and should_close.
zContent-EncodingN�gzipTzs3a://zs3://zs3n://�fsspecZbotocorer )� ClientError�NoCredentialsErrorrA �rbZanonFz)Invalid file path or buffer object type: ) r4 r r r& r7 �headers�getr �read�closer> �AssertionErrorr= �replacer Zbotocore.exceptionsrE rF �PermissionError� ImportError�open�tuple�dict�bytes�mmapr+ r �type�
ValueError)r' r? r@ rA rB �req�content_encoding�readerrD Z err_types_to_retry_with_anonrE rF Zfile_obj�msgr$ r$ r% �get_filepath_or_buffer� sV
$
(
r[ )r) r c C s ddl m} td|| ��S )z�
converts an absolute native path to a FILE URL.
Parameters
----------
path : a path in native format
Returns
-------
a valid FILE URL
r )� pathname2urlzfile:)r5 r\ r )r) r\ r$ r$ r% �file_path_to_url� s
r] z.gzz.bz2z.zipz.xz)rC �bz2�zip�xz)r@ r c C s\ t | t�rPt| �}y|jd�} W qT tk
rL } ztd�|�W Y dd}~X qTX ni }| |fS )a
Simplifies a compression argument to a compression method string and
a mapping containing additional arguments.
Parameters
----------
compression : str or mapping
If string, specifies the compression method. If mapping, value at key
'method' specifies compression method.
Returns
-------
tuple of ({compression method}, Optional[str]
{compression arguments}, Dict[str, str])
Raises
------
ValueError on mapping missing 'method' key
�methodz.If mapping, compression must have key 'method'N)r r rR �pop�KeyErrorrV )r@ �compression_args�errr$ r$ r% �get_compression_method
s
rf )r' r@ r c C s� |dkr dS |dkrVt | �} t| t�s*dS x&tj� D ]\}}| j� j|�r4|S q4W dS |tkrb|S d|� �}ddgtt� }|d|� �7 }t|��dS )a�
Get the compression method for filepath_or_buffer. If compression='infer',
the inferred compression method is returned. Otherwise, the input
compression method is returned unchanged, unless it's invalid, in which
case an error is raised.
Parameters
----------
filepath_or_buffer : str or file handle
File path or object.
compression : {'infer', 'gzip', 'bz2', 'zip', 'xz', None}
If 'infer' and `filepath_or_buffer` is path-like, then detect
compression from the following extensions: '.gz', '.bz2', '.zip',
or '.xz' (otherwise no compression).
Returns
-------
string or None
Raises
------
ValueError on invalid compression specified.
NZinferz Unrecognized compression type: z
Valid compression types are ) r4 r r �_compression_to_extension�items�lower�endswith�sortedrV )r' r@ � extensionrZ Zvalidr$ r$ r% �infer_compression. s
rm FT)rA r@ �
memory_map�is_textc C s� yddl m} tt|f}W n tk
r6 ttf}Y nX t� } | }
t| �} t| t�} t |�\}} | rnt
| |�}|�r�|dkr�| r�t j | |f| �}
nt j
f d| i| ��}
n�|dkr�| r�tj| |f| �}
ntj| f| �}
n�|dk�rjt| |f| �}
| j|
� |
jdk�r
|
}
n^|
jdk�r�|
j� }t|�d k�r<|
j |j� �}
n,t|�dk�rZtd
| � ���ntd |� ���n,|d k�r�tt�| |�}
nd
|� �}t|��| j|
� nL| �r�|�r�t | |||dd�}
n"|�r�t | |ddd�}
n
t | |�}
| j|
� |�r@|�st|
|��r@ddlm} ||
||dd�}t|
ttf��s<| j|� |}
|�r�t|
d��r�yt |
�}|
j � |}
W n t k
�r� Y nX |
| fS )af
Get file handle for given path/buffer and mode.
Parameters
----------
path_or_buf : str or file handle
File path or object.
mode : str
Mode to open path_or_buf with.
encoding : str or None
Encoding to use.
compression : str or dict, default None
If string, specifies compression mode. If dict, value at key 'method'
specifies compression mode. Compression mode must be one of {'infer',
'gzip', 'bz2', 'zip', 'xz', None}. If compression mode is 'infer'
and `filepath_or_buffer` is path-like, then detect compression from
the following extensions: '.gz', '.bz2', '.zip', or '.xz' (otherwise
no compression). If dict and compression mode is one of
{'zip', 'gzip', 'bz2'}, or inferred as one of the above,
other entries passed as additional compression options.
.. versionchanged:: 1.0.0
May now be a dict with key 'method' as compression mode
and other keys as compression options if compression
mode is 'zip'.
.. versionchanged:: 1.1.0
Passing compression options as keys in dict is now
supported for compression modes 'gzip' and 'bz2' as well as 'zip'.
memory_map : boolean, default False
See parsers._parser_params for more information.
is_text : boolean, default True
whether file/buffer is in text format (csv, json, etc.), or in binary
mode (pickle, etc.).
errors : str, default 'strict'
Specifies how encoding and decoding errors are to be handled.
See the errors argument for :func:`open` for a full list
of options.
.. versionadded:: 1.1.0
Returns
-------
f : file-like
A file-like object.
handles : list of file-like objects
A list of file-like object that were opened in this function.
r )�S3FilerC �fileobjr^ r_ �w�r� z Zero files found in ZIP file z9Multiple files found in ZIP file. Only one file per ZIP: r` z Unrecognized compression type: r )r? �errors�newlinerM )ru rv )�
TextIOWrapper�fileno) Zs3fsrp r r rO �listr4 r r rf rm rC rP �GzipFiler^ �BZ2File�
_BytesZipFile�appendrA �namelist�lenrb rV r �lzma�iorw r1 � _MMapWrapperrK � Exception)Z path_or_bufrA r? r@ rn ro ru rp Zneed_text_wrappingZhandles�fZis_pathrd �zfZ zip_namesrZ rw �g�wrappedr$ r$ r% �
get_handled s~ =
r� c sH e Zd ZdZd
eeee d�� fdd�
Z� fdd�Ze dd � �Z
� Z S ) r| a
Wrapper for standard library class ZipFile and allow the returned file-like
handle to accept byte strings via `write` method.
BytesIO provides attributes of file-like object and ZipFile.writestr writes
bytes strings into a member of the archive.
N)�filerA � archive_namec s4 |dkr|j dd�}|| _t� j||tjf|� d S )N�wbrG �br )r� rG )rM r� �super�__init__�zipfile� ZIP_DEFLATED)�selfr� rA r� r9 )� __class__r$ r% r� s z_BytesZipFile.__init__c s( | j }| jd k r| j}t� j||� d S )N)�filenamer� r� �writestr)r� �datar� )r� r$ r% �write$ s
z_BytesZipFile.writec C s
| j d kS )N)�fp)r� r$ r$ r% �closed* s z_BytesZipFile.closed)N) �__name__�
__module__� __qualname__�__doc__r r r
r� r� �propertyr� �
__classcell__r$ r$ )r� r% r| s r| c @ sH e Zd ZdZed�dd�Zed�dd�Zd d�d d
�Zed�d d �Z d
S )r� a
Wrapper for the Python's mmap class so that it can be properly read in
by Python's csv.reader class.
Parameters
----------
f : file object
File object to be mapped onto memory. Must support the 'fileno'
method or have an equivalent attribute
)r� c C s t j |j� dt jd�| _ d S )Nr )�access)rT rx Z ACCESS_READ)r� r� r$ r$ r% r� < s z_MMapWrapper.__init__)�namec C s t | j|�S )N)�getattrrT )r� r� r$ r$ r% � __getattr__? s z_MMapWrapper.__getattr__)r c C s | S )Nr$ )r� r$ r$ r% �__iter__B s z_MMapWrapper.__iter__c C s$ | j j� }|jd�}|dkr t�|S )Nzutf-8r )rT �readline�decode�
StopIteration)r� Znewbytesrv r$ r$ r% �__next__E s
z_MMapWrapper.__next__N)
r� r� r� r� r r� r r� r� r� r$ r$ r$ r% r� / s
r� )NNNN)NNFTN)@r� r^ � collectionsr rC r� r r r rT r( r2 Ztypingr r r r r
r r r
r r r � urllib.parser r r r r r r� Zpandas._typingr Z
pandas.compatr r Zpandas.compat._optionalr Zpandas.core.dtypes.commonr r� �setr" �discardr r, r& r+ r/ r4 r7 r> r r[ r] rg rf rm r� �ZipFiler| �Iteratorr� r$ r$ r$ r% �<module> sZ 4
0[" 7 * $! |