3
���ho8 � @ s d Z ddlmZ ddlmZ ddlZddlZddlZddlm Z ddl
Z ddl m
Z
ddlmZ dZd Zd
Zd Zd d
ddddddddddddddgZd Zd Zd Zd Zd e� d!e� d!e� d!e� d"� Zd#e� d!e� d$�Zd%Ze ed&�d'd(�Z e d)�d*d+�Z d,d-� Z d.d/� Z G d0d1� d1eej!�Z"dS )2z�
Read a SAS XPort format file into a Pandas DataFrame.
Based on code from Jack Cushman (github.com/jcushman/xport).
The file format is defined here:
https://support.sas.com/techsup/technote/ts140.pdf
� )�abc)�datetimeN)�Appender)�get_filepath_or_buffer)�
ReaderBasezPHEADER RECORD*******LIBRARY HEADER RECORD!!!!!!!000000000000000000000000000000 zKHEADER RECORD*******MEMBER HEADER RECORD!!!!!!!000000000000000001600000000zPHEADER RECORD*******DSCRPTR HEADER RECORD!!!!!!!000000000000000000000000000000 zPHEADER RECORD*******OBS HEADER RECORD!!!!!!!000000000000000000000000000000 �ntypeZnhfun� field_lengthZnvar0�name�labelZnformZnflZ num_decimalsZnfjZnfillZniformZniflZnifdZnpos�_z�Parameters
----------
filepath_or_buffer : string or file-like object
Path to SAS file or object implementing binary read method.z�index : identifier of index column
Identifier of column that should be used as index of the DataFrame.
encoding : string
Encoding for text data.
chunksize : int
Read file `chunksize` lines at a time, returns iterator.zEformat : string
File format, only `xport` is currently supported.z_iterator : boolean, default False
Return XportReader object for reading file incrementally.z#Read a SAS file into a DataFrame.
�
a
Returns
-------
DataFrame or XportReader
Examples
--------
Read a SAS Xport file:
>>> df = pd.read_sas('filename.XPT')
Read a Xport file in 10,000 line chunks:
>>> itr = pd.read_sas('filename.XPT', chunksize=10000)
>>> for chunk in itr:
>>> do_something(chunk)
z$Class for reading SAS Xport files.
z�
Attributes
----------
member_info : list
Contains information about the file
fields : list
Contains information about the variables in the file
z�Read observations from SAS Xport file, returning as data frame.
Parameters
----------
nrows : int
Number of rows to read from data file; if None, read whole
file.
Returns
-------
A DataFrame.
)�datestr�returnc C s( y t j| d�S tk
r" tjS X dS )z3 Given a date in xport format, return Python date. z%d%b%y:%H:%M:%SN)r �strptime�
ValueError�pdZNaT)r
� r �9/tmp/pip-build-5_djhm0z/pandas/pandas/io/sas/sas_xport.py� _parse_date� s r )�sc C sD i }d}x0|D ](\}}| ||| � j � ||< ||7 }qW |d= |S )a
Parameters
----------
s: str
Fixed-length string to split
parts: list of (name, length) pairs
Used to break up string, name '_' will be filtered from output.
Returns
-------
Dict of name:contents of string at given location.
r r )�strip)r �parts�out�startr �lengthr r r � _split_line� s
r c C sT |dkrPt jt| �t jd��}t jd|� dd| � ��}|j|d�}| |d< |S | S )N� ZS8�Sz,S)�dtype�f0)�np�zeros�lenr �view)�vec�nbytes�vec1r Zvec2r r r �_handle_truncated_float_vec� s r' c C s t jd�}| j|d�}|d }|d }|d@ }t jt| �t jd�}d|t j|d@ �< d|t j|d @ �< d
|t j|d @ �< ||L }||? |d @ d
d
| > B }|dM }||d? d@ d d> | d d> |d@ B O }t jt|�fdd�}||d< ||d< |jdd�}|jd�}|S )zf
Parse a vector of float values representing IBM 8 byte floats into
native 8 byte floats.
z>u4,>u4)r r �f1i��� � i � i @ � i � � � l �� � � �A i� � l z>f8Zf8) r r r# r! r" Zuint8�where�emptyZastype) r$ r r& Zxport1Zxport2Zieee1�shiftZieee2Zieeer r r �_parse_float_vec� s(
r5 c @ sl e Zd ZeZddd�Zdd� Zdd� Zd d
� Zd d � Z e
d
�dd�Z ddd�Z dd� Z
ee�ddd��ZdS )� XportReaderN�
ISO-8859-1c C sd || _ d| _|| _|| _t|t�r6t||d�\}}}}t|ttf�rRt|d�| _ n|| _ | j
� d S )Nr )�encoding�rb) � _encoding� _lines_read�_index�
_chunksize�
isinstance�strr �bytes�open�filepath_or_buffer� _read_header)�selfrB �indexr8 � chunksize� compressionZ should_closer r r �__init__� s
zXportReader.__init__c C s | j j� d S )N)rB �close)rD r r r rI s zXportReader.closec C s | j jd�j� S )N�P )rB �read�decode)rD r r r �_get_row s zXportReader._get_rowc C sd | j jd� | j� }|tkr,| j� td��| j� }ddgddgddgddgd d
gg}t||�}|d d kr|| j� td ��t|d �|d < || _| j� }t|d d
� �|d
< | j� }| j� }|j t
�}|t k} |o�| s�| j� td��t |d&d'� �}
ddgddgddgddgddgddgd d
gg} t| j� | �} d
d
gdd
gddgddgg} | j
t| j� | �� t| d
�| d
< t| d �| d < | | _ddd�}
t | j� dd� �}|
| }|d �r�|d|d 7 }| j j|�}g }d}�xt|�|
k�r�|d |
� ||
d � }}|jd �}tjd |�}ttt|��}|d= |
|d |d < |d }|d dk�r�|dk �sn|dk�r�| j� d |� d!�}t|��x>|j� D ]2\}}y|j� ||< W n tk
�r� Y nX �q�W ||d 7 }||g7 }�q�W | j� }|tk�s| j� td"��|| _ || _ | j j � | _ | j � | _!d#d$� | j D �| _"d%d$� t#| j �D �}t$j%|�}|| _&d S )(Nr z#Header record is not an XPORT file.�prefixr. �versionr ZOSr �created� zSAS SAS SASLIBz!Header record has invalid prefix.�modifiedzMember header not found� r* �set_nameZsasdatar
�( �type�numeric�char)r) r* �6 �: rJ � z>hhhh8s40s8shhh2s8shhl52sr r zFloating field width z is not between 2 and 8.z Observation header not found.c S s g | ]}|d j � �qS )r )rL )�.0�xr r r �
<listcomp>y s z,XportReader._read_header.<locals>.<listcomp>c S s, g | ]$\}}d t |� dt |d � f�qS )r r r )r? )r\ �i�fieldr r r r^ } s ������)'rB �seekrM �_correct_line1rI r r r � file_info�
startswith�_correct_header1�_correct_header2�int�update� member_inforK r" �ljust�struct�unpack�dict�zip�
_fieldkeys� TypeError�itemsr �AttributeError�_correct_obs_header�fields�
record_length�tell� record_start�
_record_count�nobs�columns� enumerater r �_dtype) rD Zline1Zline2Zfifre Zline3Zheader1Zheader2Z headflag1Z headflag2ZfieldnamelengthZmemrk �typesZ
fieldcountZ
datalengthZ fielddatarv Z
obs_lengthr` Z fieldstruct�fl�msg�k�v�headerZdtypelr r r r rC s� "
"
zXportReader._read_headerc C s | j | jp dd�S )Nr) )�nrows)rK r= )rD r r r �__next__� s zXportReader.__next__)r c C s� | j jdd� | j j� | j }|d dkr4tjd� | jdkrV| j j| j� || j S | j jdd� | j jd�}tj |tj
d�}tj |dk�}t |�dkr�d}n dt |� }| j j| j� || | j S ) z�
Get number of records in file.
This is maybe suboptimal because we have to seek to the end of
the file.
Side effect: returns file position to record_start.
r r* rJ zxport file may be corrupted)r l @@� r i����)
rB rc rx ry �warnings�warnrw rK r �
frombufferZuint64Z flatnonzeror" )rD Ztotal_records_lengthZ last_card�ixZtail_padr r r rz � s
zXportReader._record_countc C s |dkr| j }| j|d�S )a
Reads lines from Xport file and returns as dataframe
Parameters
----------
size : int, defaults to None
Number of lines to read. If None, reads whole file.
Returns
-------
DataFrame
N)r� )r= rK )rD �sizer r r � get_chunk� s
zXportReader.get_chunkc C sl |j dd�}|d dk|d dk@ |d dk@ }|d dk|d d k@ |d d
kB |d d kB }||M }|S ) Nz u1,u1,u2,u4)r r( r �f2Zf3r r0 �Z �_ �. )r# )rD r$ r� �missZmiss1r r r �_missing_double� s
$0zXportReader._missing_doublec
sd |d kr� j }t|� j � j �}|� j }|dkr>� j� t�� jj|�}tj |� j
|d�}t j t
|�d�}x�t� j�D ]�\}}|dt|� } � j| d }
|
dkr�t| � j| d �} � j| �} t| �} tj| | < n@� j| d dk�rd d
� | D �} � jd k �r� fd d
�| D �} | ||< qxW � jd k�rFt
� j� j| �|_n |j� j�}� j|7 _|S ) Nr )r �count)rE r r rW r rX c S s g | ] }|j � �qS r )�rstrip)r\ �yr r r r^ � s z$XportReader.read.<locals>.<listcomp>c s g | ]}|j � j��qS r )rL r: )r\ r� )rD r r r^ � s )r{ �minr; rw rI �
StopIterationrB rK r r� r~ r Z DataFrame�ranger} r| r? rv r' r� r5 �nanr: r< rE Z set_index)
rD r� �
read_lines�read_len�raw�dataZdf�jr] r$ r r� r� r )rD r rK � s8
zXportReader.read)Nr7 N)N)N)�__name__�
__module__� __qualname__�_xport_reader_doc�__doc__rH rI rM rC r� ri rz r� r� r �_read_method_docrK r r r r r6 � s
m#
r6 )#r� � collectionsr r rm r� Znumpyr Zpandas.util._decoratorsr Zpandasr Zpandas.io.commonr Zpandas.io.sas.sasreaderr rd rg rh ru rq Z_base_params_docZ _params2_docZ_format_params_docZ
_iterator_docZ
_read_sas_docr� r� r? r r r' r5 �Iteratorr6 r r r r �<module> sR 9 |