3
���h�� � @ s� d Z ddlmZ ddlmZmZmZ ddlmZ ddlZddl m
Z
m Z m Z m
Z
ddlZddlZddljjZddlmZmZmZ ddlmZ dd lmZ dd
l m Z m Z dd l m Z dd l!m"Z" G d
d� de#�Z$G dd� de%�Z&da'dd� Z(dd� Z)dd� Z*dQdd�Z+dd� Z,dRd d �Z-dSd d �Z.e
dTde d �d!d"��Z/e
dUe0e
e d �d$d"��Z/dVe e0 e e e
e f d �d%d"�Z/e
dWde d �d&d'��Z1e
dXe0e
e d �d(d'��Z1dYe e0 e e e
e f d �d)d'�Z1e
dZde d �d*d+��Z2e
d[e0e
e d �d,d+��Z2d\e e0 e e e
e f d �d-d+�Z2d]dd/�d0d1�Z3d^d2d3�Z4e4Z5d4d5� Z6d_d7d8�Z7G d9d:� d:e �Z8G d;d<� d<e �Z9G d=d>� d>e9�Z:d?d@dAdBdCdDdAdE�Z;dFdG� Z<dHdI� Z=dJZ>G dKdL� dLe8�Z?G dMdN� dNe9�Z@d`dOdP�ZAdS )az|
Collection of query wrappers / abstractions to both facilitate data
retrieval and to reduce dependency on DB-specific API.
� )�contextmanager)�date�datetime�time)�partialN)�Iterator�Optional�Union�overload)�is_datetime64tz_dtype� is_dict_like� is_list_like)�DatetimeTZDtype)�isna)� DataFrame�Series)� PandasObject)� to_datetimec @ s e Zd ZdS )�SQLAlchemyRequiredN)�__name__�
__module__� __qualname__� r r �//tmp/pip-build-5_djhm0z/pandas/pandas/io/sql.pyr s r c @ s e Zd ZdS )�
DatabaseErrorN)r r r r r r r r s r c C sT t d kr2ydd l}da W n tk
r0 da Y nX t rLdd l}t| |jj�S dS d S )Nr TF)�_SQLALCHEMY_INSTALLED�
sqlalchemy� ImportError�
isinstance�engineZ Connectable)�conr r r r �_is_sqlalchemy_connectable( s
r! c C s6 | g}|dk r2t |d�r$||g7 }n|t|�g7 }|S )z9Convert SQL and params args to DBAPI2.0 compliant format.N�keys)�hasattr�list)�sql�params�argsr r r �_convert_params: s
r( c C s2 | dks| dks| dkr g } nt | d�s.| g} | S )z3Process parse_dates argument for read_sql functionsTNF�__iter__)r# )� parse_datesr r r � _process_parse_dates_argumentE s
r+ c C s� t |t�r t| fddi|��S |d krJt| jjtj�sFt| jjtj�rJd}|dkrbt| d ||d �S t | j�rxt| d
d�S t| d ||d�S d S )N�errors�ignore�s�D�d�h�m�ms�us�nsZcoerce)r, �unit�utcT)r7 )r, �formatr7 )r/ r0 r1 r2 r. r3 r4 r5 )
r �dictr �
issubclass�dtype�type�np�floating�integerr )�colr7 r8 r r r �_handle_date_columnP s
rA c
C sj t |�}x\| j� D ]P\}}t|j�s,||kry || }W n tk
rP d}Y nX t||d�| |< qW | S )zz
Force non-datetime columns to be read as such.
Supports both string formatted and integer timestamp columns.
N)r8 )r+ �itemsr r; � TypeErrorrA )Z
data_framer* �col_name�df_col�fmtr r r �_parse_date_columnse s
rG Tc C s4 t j| ||d�}t||�}|dk r0|j|dd� |S )z(Wrap result set of query in a DataFrame.)�columns� coerce_floatNT)�inplace)r � from_recordsrG � set_index)�datarH � index_colrI r* �framer r r � _wrap_resultz s
rP c C s2 |dkrt |�}n t |dd�}t| |�}|j|� S )aY
Execute the given SQL query using the provided connection object.
Parameters
----------
sql : string
SQL query to be executed.
con : SQLAlchemy connectable(engine/connection) or sqlite3 connection
Using SQLAlchemy makes it possible to use any DB supported by the
library.
If a DBAPI2 object, only sqlite3 is supported.
cur : deprecated, cursor is obtained from connection, default: None
params : list or tuple, optional, default: None
List of parameters to pass to execute method.
Returns
-------
Results Iterable
NT)� is_cursor)�pandasSQL_builderr( �execute)r% r �curr& �
pandas_sqlr' r r r rS � s
rS )� chunksize�returnc C s d S )Nr )�
table_namer �schemarN rI r* rH rV r r r �read_sql_table� s rZ � c C s d S )Nr )rX r rY rN rI r* rH rV r r r rZ � s c C s� t |�}t|�std��ddl}ddlm} | ||d�}
y|
j| gdd� W n8 |jjk
r� } zt d| � d ��| �W Y dd} ~ X nX t
||
d
�} | j | |||||d �}
|
dk r�|
S t d| � d �|��dS ) a�
Read SQL database table into a DataFrame.
Given a table name and a SQLAlchemy connectable, returns a DataFrame.
This function does not support DBAPI connections.
Parameters
----------
table_name : str
Name of SQL table in database.
con : SQLAlchemy connectable or str
A database URI could be provided as as str.
SQLite DBAPI connection mode not supported.
schema : str, default None
Name of SQL schema in database to query (if database flavor
supports this). Uses default schema if None (default).
index_col : str or list of str, optional, default: None
Column(s) to set as index(MultiIndex).
coerce_float : bool, default True
Attempts to convert values of non-string, non-numeric objects (like
decimal.Decimal) to floating point. Can result in loss of Precision.
parse_dates : list or dict, default None
- List of column names to parse as dates.
- Dict of ``{column_name: format string}`` where format string is
strftime compatible in case of parsing string times or is one of
(D, s, ns, ms, us) in case of parsing integer timestamps.
- Dict of ``{column_name: arg dict}``, where the arg dict corresponds
to the keyword arguments of :func:`pandas.to_datetime`
Especially useful with databases without native Datetime support,
such as SQLite.
columns : list, default None
List of column names to select from SQL table.
chunksize : int, default None
If specified, returns an iterator where `chunksize` is the number of
rows to include in each chunk.
Returns
-------
DataFrame or Iterator[DataFrame]
A SQL table is returned as two-dimensional data structure with labeled
axes.
See Also
--------
read_sql_query : Read SQL query into a DataFrame.
read_sql : Read SQL query or database table into a DataFrame.
Notes
-----
Any datetime values with time zone information will be converted to UTC.
Examples
--------
>>> pd.read_sql_table('table_name', 'postgres:///db_name') # doctest:+SKIP
z9read_sql_table only supported for SQLAlchemy connectable.r N)�MetaData)rY T)�onlyZviewszTable z
not found)�meta)rN rI r* rH rV ) �_engine_builderr! �NotImplementedErrorr �sqlalchemy.schemar\ �reflect�excZInvalidRequestError�
ValueError� SQLDatabase�
read_table)rX r rY rN rI r* rH rV r r\ r^ �errrU �tabler r r rZ � s, A $ c C s d S )Nr )r% r rN rI r& r* rV r r r �read_sql_query! s
ri c C s d S )Nr )r% r rN rI r& r* rV r r r ri . s
c C s t |�}|j| |||||d�S )a7
Read SQL query into a DataFrame.
Returns a DataFrame corresponding to the result set of the query
string. Optionally provide an `index_col` parameter to use one of the
columns as the index, otherwise default integer index will be used.
Parameters
----------
sql : str SQL query or SQLAlchemy Selectable (select or text object)
SQL query to be executed.
con : SQLAlchemy connectable, str, or sqlite3 connection
Using SQLAlchemy makes it possible to use any DB supported by that
library. If a DBAPI2 object, only sqlite3 is supported.
index_col : str or list of str, optional, default: None
Column(s) to set as index(MultiIndex).
coerce_float : bool, default True
Attempts to convert values of non-string, non-numeric objects (like
decimal.Decimal) to floating point. Useful for SQL result sets.
params : list, tuple or dict, optional, default: None
List of parameters to pass to execute method. The syntax used
to pass parameters is database driver dependent. Check your
database driver documentation for which of the five syntax styles,
described in PEP 249's paramstyle, is supported.
Eg. for psycopg2, uses %(name)s so use params={'name' : 'value'}.
parse_dates : list or dict, default: None
- List of column names to parse as dates.
- Dict of ``{column_name: format string}`` where format string is
strftime compatible in case of parsing string times, or is one of
(D, s, ns, ms, us) in case of parsing integer timestamps.
- Dict of ``{column_name: arg dict}``, where the arg dict corresponds
to the keyword arguments of :func:`pandas.to_datetime`
Especially useful with databases without native Datetime support,
such as SQLite.
chunksize : int, default None
If specified, return an iterator where `chunksize` is the number of
rows to include in each chunk.
Returns
-------
DataFrame or Iterator[DataFrame]
See Also
--------
read_sql_table : Read SQL database table into a DataFrame.
read_sql : Read SQL query or database table into a DataFrame.
Notes
-----
Any datetime values with time zone information parsed via the `parse_dates`
parameter will be converted to UTC.
)rN r& rI r* rV )rR �
read_query)r% r rN rI r& r* rV rU r r r ri ; s =c C s d S )Nr )r% r rN rI r& r* rH rV r r r �read_sql� s rk c C s d S )Nr )r% r rN rI r& r* rH rV r r r rk � s c
C s� t |�}t|t�r(|j| |||||d�S y|j| �} W n tk
rN d} Y nX | rz|jj| gd� |j| |||||d�S |j| |||||d�S dS )a�
Read SQL query or database table into a DataFrame.
This function is a convenience wrapper around ``read_sql_table`` and
``read_sql_query`` (for backward compatibility). It will delegate
to the specific function depending on the provided input. A SQL query
will be routed to ``read_sql_query``, while a database table name will
be routed to ``read_sql_table``. Note that the delegated function might
have more specific notes about their functionality not listed here.
Parameters
----------
sql : str or SQLAlchemy Selectable (select or text object)
SQL query to be executed or a table name.
con : SQLAlchemy connectable, str, or sqlite3 connection
Using SQLAlchemy makes it possible to use any DB supported by that
library. If a DBAPI2 object, only sqlite3 is supported. The user is responsible
for engine disposal and connection closure for the SQLAlchemy connectable. See
`here <https://docs.sqlalchemy.org/en/13/core/connections.html>`_.
index_col : str or list of str, optional, default: None
Column(s) to set as index(MultiIndex).
coerce_float : bool, default True
Attempts to convert values of non-string, non-numeric objects (like
decimal.Decimal) to floating point, useful for SQL result sets.
params : list, tuple or dict, optional, default: None
List of parameters to pass to execute method. The syntax used
to pass parameters is database driver dependent. Check your
database driver documentation for which of the five syntax styles,
described in PEP 249's paramstyle, is supported.
Eg. for psycopg2, uses %(name)s so use params={'name' : 'value'}.
parse_dates : list or dict, default: None
- List of column names to parse as dates.
- Dict of ``{column_name: format string}`` where format string is
strftime compatible in case of parsing string times, or is one of
(D, s, ns, ms, us) in case of parsing integer timestamps.
- Dict of ``{column_name: arg dict}``, where the arg dict corresponds
to the keyword arguments of :func:`pandas.to_datetime`
Especially useful with databases without native Datetime support,
such as SQLite.
columns : list, default: None
List of column names to select from SQL table (only used when reading
a table).
chunksize : int, default None
If specified, return an iterator where `chunksize` is the
number of rows to include in each chunk.
Returns
-------
DataFrame or Iterator[DataFrame]
See Also
--------
read_sql_table : Read SQL database table into a DataFrame.
read_sql_query : Read SQL query into a DataFrame.
)rN r& rI r* rV F)r] )rN rI r* rH rV N) rR r �SQLiteDatabaserj � has_table� Exceptionr^ rb rf )
r% r rN rI r& r* rH rV rU Z_is_table_namer r r rk � s: A
�fail)rW c
C sj |d
krt d|� d���t||d�}
t| t�r8| j� } nt| t�sJtd��|
j| |||||||| d� d S ) a;
Write records stored in a DataFrame to a SQL database.
Parameters
----------
frame : DataFrame, Series
name : str
Name of SQL table.
con : SQLAlchemy connectable(engine/connection) or database string URI
or sqlite3 DBAPI2 connection
Using SQLAlchemy makes it possible to use any DB supported by that
library.
If a DBAPI2 object, only sqlite3 is supported.
schema : str, optional
Name of SQL schema in database to write to (if database flavor
supports this). If None, use default schema (default).
if_exists : {'fail', 'replace', 'append'}, default 'fail'
- fail: If table exists, do nothing.
- replace: If table exists, drop it, recreate it, and insert data.
- append: If table exists, insert data. Create if does not exist.
index : boolean, default True
Write DataFrame index as a column.
index_label : str or sequence, optional
Column label for index column(s). If None is given (default) and
`index` is True, then the index names are used.
A sequence should be given if the DataFrame uses MultiIndex.
chunksize : int, optional
Specify the number of rows in each batch to be written at a time.
By default, all rows will be written at once.
dtype : dict or scalar, optional
Specifying the datatype for columns. If a dictionary is used, the
keys should be the column names and the values should be the
SQLAlchemy types or strings for the sqlite3 fallback mode. If a
scalar is provided, it will be applied to all columns.
method : {None, 'multi', callable}, optional
Controls the SQL insertion clause used:
- None : Uses standard SQL ``INSERT`` clause (one per row).
- 'multi': Pass multiple values in a single ``INSERT`` clause.
- callable with signature ``(pd_table, conn, keys, data_iter)``.
Details and a sample callable implementation can be found in the
section :ref:`insert method <io.sql.method>`.
.. versionadded:: 0.24.0
ro �replace�append�'z ' is not valid for if_exists)rY z9'frame' argument should be either a Series or a DataFrame)� if_exists�index� index_labelrY rV r; �methodN)ro rp rq )rd rR r r Zto_framer r` �to_sql) rO �namer rY rs rt ru rV r; rv rU r r r rw s$ :
rw c C s t ||d�}|j| �S )a-
Check if DataBase has named table.
Parameters
----------
table_name: string
Name of SQL table.
con: SQLAlchemy connectable(engine/connection) or sqlite3 DBAPI2 connection
Using SQLAlchemy makes it possible to use any DB supported by that
library.
If a DBAPI2 object, only sqlite3 is supported.
schema : string, default None
Name of SQL schema in database to write to (if database flavor supports
this). If None, use default schema (default).
Returns
-------
boolean
)rY )rR rm )rX r rY rU r r r rm Z s rm c C sB t | t�r>y ddl}W n tk
r. daY nX |j| �} | S | S )zw
Returns a SQLAlchemy engine from a URI (if con is a string)
else it just return con without modifying it.
r NF)r �strr r r Z
create_engine)r r r r r r_ u s
r_ Fc C sB t | �} t| �r t| ||d�S t| t�r2td��n t| |d�S dS )zm
Convenience function to return the correct PandasSQL subclass based on the
provided parameters.
)rY r^ z.Using URI string without sqlalchemy installed.)rQ N)r_ r! re r ry r rl )r rY r^ rQ r r r rR � s
rR c @ s� e Zd ZdZd(dd�Zdd � Zd
d � Zd d
� Zdd� Zdd� Z dd� Z
dd� Z d)dd�Z d*dd�Z
d+dd�Zd d � Zd d � Zd d!� Zd,d"d#�Zd$d%� Zd&d'� ZdS )-�SQLTablez�
For mapping Pandas tables to SQL tables.
Uses fact that table is reflected by SQLAlchemy to
do better type conversions.
Also holds various flags needed to avoid having to
pass them between functions all the time.
NTro �pandasc C s� || _ || _|| _|| _| j||�| _|| _|| _| | _|
| _ |d k rR| j
� | _ n| jj | j | j�| _ | j d kr�t
d|� d���d S )NzCould not init table 'rr )rx �pd_sql�prefixrO � _index_namert rY rs r" r; �_create_table_setuprh � get_tablerd ) �selfrx Zpandas_sql_enginerO rt rs r} ru rY r" r; r r r �__init__� s
zSQLTable.__init__c C s | j j| j| j�S )N)r| rm rx rY )r� r r r �exists� s zSQLTable.existsc C s$ ddl m} t|| j�j| jj��S )Nr )� CreateTable)ra r� ry rh �compiler| � connectable)r� r� r r r �
sql_schema� s zSQLTable.sql_schemac C s | j j| jj�| _ | j j� d S )N)rh Z
tometadatar| r^ �create)r� r r r �_execute_create� s zSQLTable._execute_createc C sx | j � rl| jdkr&td| j� d���qt| jdkrL| jj| j| j� | j� qt| jdkrXqttd| j� d���n| j� d S )Nro zTable 'z' already exists.rp rq rr z ' is not valid for if_exists)r� rs rd rx r| �
drop_tablerY r� )r� r r r r� � s
zSQLTable.createc s( � fdd�|D �}|j | jj� |� dS )a<
Execute SQL statement inserting data
Parameters
----------
conn : sqlalchemy.engine.Engine or sqlalchemy.engine.Connection
keys : list of str
Column names
data_iter : generator of list
Each item contains a list of values to be inserted
c s g | ]}t t� |���qS r )r9 �zip)�.0�row)r" r r �
<listcomp>� s z,SQLTable._execute_insert.<locals>.<listcomp>N)rS rh �insert)r� �connr" � data_iterrM r )r" r �_execute_insert� s zSQLTable._execute_insertc s( � fdd�|D �}|j | jj|�� dS )a
Alternative to _execute_insert for DBs support multivalue INSERT.
Note: multi-value insert is usually faster for analytics DBs
and tables containing a few columns
but performance degrades quickly with increase of columns.
c s g | ]}t t� |���qS r )r9 r� )r� r� )r" r r r� � s z2SQLTable._execute_insert_multi.<locals>.<listcomp>N)rS rh r� )r� r� r" r� rM r )r" r �_execute_insert_multi� s z SQLTable._execute_insert_multic C s0 | j d k rd| jj� }| j |j _y|jdd� W qj tk
r` } ztd|� ��|�W Y d d }~X qjX n| j}ttt|j ��}t
|�}d g| }x�t |j � �D ]�\}\}}|j
} | jjdkr�| j� }
n(| jjdkr�| jd�jt�}
n
| jt�}
t|
tj��stt|
���|j�r t|
�} d |
| < |
||< q�W ||fS )NT)rJ z!duplicate name in index/columns: �Mr2 �i8)rt rO �copy�namesZ reset_indexrd r$ �mapry rH �len� enumeraterB Z_valuesr; �kindZ
to_pydatetime�view�astype�objectr r= Zndarray�AssertionErrorr< Z _can_hold_nar ) r� �temprg � column_namesZncols� data_list�i�_Zser�valsr0 �maskr r r � insert_data� s0
$
zSQLTable.insert_datac s |d kr| j }n2|dkr | j}n"t|�r4t|| �}ntd|� ���| j� \}}t| j�}|dkrdd S |d krr|}n|dkr�td��t|| �d }| j j
� �`}xXt |�D ]L} | | �t | d | |�� �� kr�P t
� �fdd�|D �� }
||||
� q�W W d Q R X d S )NZmultiz Invalid parameter `method`: r z%chunksize argument should be non-zeror[ c s g | ]}|�� � �qS r r )r� Zarr)�end_i�start_ir r r� = s z#SQLTable.insert.<locals>.<listcomp>)r� r� �callabler rd r� r� rO �intr| �run_transaction�range�minr� ) r� rV rv Z exec_insertr" r� Znrows�chunksr� r� Z
chunk_iterr )r� r� r r� s0
zSQLTable.insertc c s^ xX|j |�}|sP qtj|||d�| _| j|d� | jdk rN| jj| jdd� | jV qW dS )z,Return generator through chunked result set.)rH rI )r* NT)rJ )� fetchmanyr rK rO �_harmonize_columnsrt rL )r� �resultrV rH rI r* rM r r r �_query_iterator@ s
zSQLTable._query_iteratorc s� |d k rtt |�dkrtddlm} � fdd�|D �}� jd k rjx,� jd d d � D ]}|jd� jj| � qNW ||�}n
� jj� }� jj|�} | j � }
|d k r�� j
| ||
||d�S | j � } t j
| |
|d�� _� j|d� � jd k r� jj� jd d
� � jS d S ) Nr )�selectc s g | ]}� j j| �qS r )rh �c)r� �n)r� r r r� Y s z!SQLTable.read.<locals>.<listcomp>r[ )rI r* )rH rI )r* T)rJ ���)r� r r� rt r� rh r� r| rS r" r� �fetchallr rK rO r� rL ) r� rI r* rH rV r� �cols�idxZ
sql_selectr� r� rM r )r� r �readT s0
z
SQLTable.readc C s� |dkr�| j jj}|d k rJt|t�s*|g}t|�|krFtd|� ���n|S |dkrrd| j jkrr| j jjd krrdgS dd� t | j jj
�D �S n"t|t �r�|gS t|t�r�|S d S d S )NTz@Length of 'index_label' should match number of levels, which is r[ rt c S s& g | ] \}}|d k r|nd|� ��qS )NZlevel_r )r� r� �lr r r r� � s z(SQLTable._index_name.<locals>.<listcomp>) rO rt �nlevelsr r$ r� rd rH rx r� r� ry )r� rt ru r� r r r r~ y s(
zSQLTable._index_namec st g }�j d k rLx<t�j �D ].\}}� �jj j|��}|jt|�|df� qW |� �fdd�tt�jj��D �7 }|S )NTc s6 g | ].}t �jj| �� �jjd d �|f �df�qS )NF)ry rO rH Ziloc)r� r� )� dtype_mapperr� r r r� � s z8SQLTable._get_column_names_and_types.<locals>.<listcomp>) rt r� rO Z_get_level_valuesrq ry r� r� rH )r� r� �column_names_and_typesr� Z idx_labelZidx_typer )r� r� r �_get_column_names_and_types� s
z$SQLTable._get_column_names_and_typesc
s� ddl m� m}m} | j| j�}� fdd�|D �}| jd k rtt| j�sP| jg}n| j}||d| jd i�}|j |� | j
p�| j j j
}ddl
m} || j |d�} || j| f|�d |i�S )
Nr )�Column�PrimaryKeyConstraint�Tablec s g | ]\}}}� |||d ��qS ))rt r )r� rx �typ�is_index)r� r r r� � s z0SQLTable._create_table_setup.<locals>.<listcomp>rx Z_pk)r\ )rY rY )r r� r� r� r� �_sqlalchemy_typer" r
rx rq rY r| r^ ra r\ )
r� r� r� r� rH r" ZpkcrY r\ r^ r )r� r r � s
z SQLTable._create_table_setupc C s, t |�}�x | jjD �]}|j}y�| j| }||krny || }W n tk
rX d}Y nX t||d�| j|< w| j|j�}|t ks�|t
ks�|t kr�|t k}t||d�| j|< n\|t kr�|j
|dd�| j|< n>t|�|j� k�r
|tjd�ks�|tk�r
|j
|dd�| j|< W q tk
�r" Y qX qW dS )a
Make the DataFrame's column types align with the SQL table
column types.
Need to work around limited NA value support. Floats are always
fine, ints must always be floats if there are Null values.
Booleans are hard because converting bool column with None replaces
all Nones with false. Therefore only convert bool if there are no
NA values.
Datetimes should already be converted to np.datetime64 if supported,
but here we also force conversion if required.
N)r8 )r7 F)r� �int64)r+ rh rH rx rO rC rA �
_get_dtyper< r r r �floatr� r� �countr= r; �bool�KeyError)r� r* Zsql_colrD rE rF �col_typer7 r r r r� � s2
zSQLTable._harmonize_columnsc
C s^ | j pi }|j|kr | j |j S tj|dd�}ddlm}m}m}m}m }m
} m }
m } m
} |dksj|dkr�y|jjd k r�|dd�S W n. tk
r� t|dd �d k r�|dd�S Y nX |S |d kr�tjd
td d � |S |d
k� r�|j dkr�| dd�S | dd�S n\|dk�r |j dk�r|
S |S n<|dk�r,|S |dk�r:|S |dk�rH| S |dk�rZtd��| S )NT)�skipnar ) � TIMESTAMP�
BigInteger�Boolean�Date�DateTime�Float�Integer�Text�Time�
datetime64r )�timezone�tz� timedelta64zlthe 'timedelta' type is not supported, and will be written as integer values (ns frequency) to the database.� )�
stacklevelr> Zfloat32� )Z precision�5 r? Zint32�booleanr r �complexz Complex datatypes not supported)r; rx �lib� infer_dtype�sqlalchemy.typesr� r� r� r� r� r� r� r� r� �dtr� �AttributeError�getattr�warnings�warn� UserWarningrd )
r� r@ r; r� r� r� r� r� r� r� r� r� r� r r r r� � sH
,
zSQLTable._sqlalchemy_typec C s� ddl m}m}m}m}m}m} t||�r.tS t||�rBt j
d�S t||�rZ|j sVt S t
S t||�rht S t||�rvtS t||�r�tS tS )Nr )r� r� r� r� r� r� r� )r� r� r� r� r� r� r� r r� r= r; r� r r r r� r� )r� Zsqltyper� r� r� r� r� r� r r r r� 9 s
zSQLTable._get_dtype)NTro r{ NNNN)NN)TN)TNNN)N)r r r �__doc__r� r� r� r� r� r� r� r� r� r� r� r~ r� r r� r� r� r r r r rz � s2
$
%
%$
3?rz c @ s e Zd ZdZdd� Zdd� ZdS )� PandasSQLz7
Subclasses Should define read_sql and to_sql.
c O s t d��d S )NzMPandasSQL must be created with an SQLAlchemy connectable or sqlite connection)rd )r� r' �kwargsr r r rk U s zPandasSQL.read_sqlc O s t d��d S )NzMPandasSQL must be created with an SQLAlchemy connectable or sqlite connection)rd )r� r' r� r r r rw [ s zPandasSQL.to_sqlN)r r r r� rk rw r r r r r� P s r� c @ s� e Zd ZdZd dd�Zedd� �Zdd� Zd d
d �Ze d d d
��Z
d dd�Z e Z d!dd�Z
edd� �Zd"dd�Zd#dd�Zd$dd�Zd%dd �ZdS )&re a�
This class enables conversion between DataFrame and SQL databases
using SQLAlchemy to handle DataBase abstraction.
Parameters
----------
engine : SQLAlchemy connectable
Connectable to connect with the database. Using SQLAlchemy makes it
possible to use any DB supported by that library.
schema : string, default None
Name of SQL schema in database to write to (if database flavor
supports this). If None, use default schema (default).
meta : SQLAlchemy MetaData object, default None
If provided, this MetaData object is used instead of a newly
created. This allows to specify database flavor specific
arguments in the MetaData object.
Nc C s. || _ |s$ddlm} || j |d�}|| _d S )Nr )r\ )rY )r� ra r\ r^ )r� r rY r^ r\ r r r r� v s
zSQLDatabase.__init__c
c s4 | j j� � }t|d�r |V n| j V W d Q R X d S )NrS )r� �beginr# )r� Ztxr r r r� s
zSQLDatabase.run_transactionc O s | j jdd�j||�S )z,Simple passthrough to SQLAlchemy connectableT)Z
no_parameters)r� Zexecution_optionsrS )r� r' r� r r r rS � s zSQLDatabase.executeTc C s" t || ||d�}|j||||d�S )a�
Read SQL database table into a DataFrame.
Parameters
----------
table_name : string
Name of SQL table in database.
index_col : string, optional, default: None
Column to set as index.
coerce_float : boolean, default True
Attempts to convert values of non-string, non-numeric objects
(like decimal.Decimal) to floating point. This can result in
loss of precision.
parse_dates : list or dict, default: None
- List of column names to parse as dates.
- Dict of ``{column_name: format string}`` where format string is
strftime compatible in case of parsing string times, or is one of
(D, s, ns, ms, us) in case of parsing integer timestamps.
- Dict of ``{column_name: arg}``, where the arg corresponds
to the keyword arguments of :func:`pandas.to_datetime`.
Especially useful with databases without native Datetime support,
such as SQLite.
columns : list, default: None
List of column names to select from SQL table.
schema : string, default None
Name of SQL schema in database to query (if database flavor
supports this). If specified, this overwrites the default
schema of the SQL database object.
chunksize : int, default None
If specified, return an iterator where `chunksize` is the number
of rows to include in each chunk.
Returns
-------
DataFrame
See Also
--------
pandas.read_sql_table
SQLDatabase.read_query
)rt rY )rI r* rH rV )rz r� ) r� rX rN rI r* rH rY rV rh r r r rf � s 4zSQLDatabase.read_tablec c s0 x*| j |�}|sP qt|||||d�V qW dS )z+Return generator through chunked result set)rN rI r* N)r� rP )r� rV rH rN rI r* rM r r r r� � s
zSQLDatabase._query_iteratorc C s\ t ||�}| j|� }|j� } |dk r:| j||| |||d�S |j� }
t|
| |||d�} | S dS )a$
Read SQL query into a DataFrame.
Parameters
----------
sql : string
SQL query to be executed.
index_col : string, optional, default: None
Column name to use as index for the returned DataFrame object.
coerce_float : boolean, default True
Attempt to convert values of non-string, non-numeric objects (like
decimal.Decimal) to floating point, useful for SQL result sets.
params : list, tuple or dict, optional, default: None
List of parameters to pass to execute method. The syntax used
to pass parameters is database driver dependent. Check your
database driver documentation for which of the five syntax styles,
described in PEP 249's paramstyle, is supported.
Eg. for psycopg2, uses %(name)s so use params={'name' : 'value'}
parse_dates : list or dict, default: None
- List of column names to parse as dates.
- Dict of ``{column_name: format string}`` where format string is
strftime compatible in case of parsing string times, or is one of
(D, s, ns, ms, us) in case of parsing integer timestamps.
- Dict of ``{column_name: arg dict}``, where the arg dict
corresponds to the keyword arguments of
:func:`pandas.to_datetime` Especially useful with databases
without native Datetime support, such as SQLite.
chunksize : int, default None
If specified, return an iterator where `chunksize` is the number
of rows to include in each chunk.
Returns
-------
DataFrame
See Also
--------
read_sql_table : Read SQL database table into a DataFrame.
read_sql
N)rN rI r* )r( rS r" r� r� rP ) r� r% rN rI r* r& rV r' r� rH rM rO r r r rj � s& 2
zSQLDatabase.read_queryro c
sn � r t � � r � fdd�|D �� � dk rlddlm}
m} x2� j� D ]&\} }
t| |
�|
�sBtd| � d���qBW t|| |||||� d�}|j� dd l m
} y|j || d
� W nN |j k
r� } z0d }t
|j�}tj||�r�td �|�n|�W Y dd}~X nX |j� �rj|j� �rj| jj}| jj� � }|j|�p6| jj|d
�}W dQ R X ||k�rjd|� d�}tj|t� dS )a�
Write records stored in a DataFrame to a SQL database.
Parameters
----------
frame : DataFrame
name : string
Name of SQL table.
if_exists : {'fail', 'replace', 'append'}, default 'fail'
- fail: If table exists, do nothing.
- replace: If table exists, drop it, recreate it, and insert data.
- append: If table exists, insert data. Create if does not exist.
index : boolean, default True
Write DataFrame index as a column.
index_label : string or sequence, default None
Column label for index column(s). If None is given (default) and
`index` is True, then the index names are used.
A sequence should be given if the DataFrame uses MultiIndex.
schema : string, default None
Name of SQL schema in database to write to (if database flavor
supports this). If specified, this overwrites the default
schema of the SQLDatabase object.
chunksize : int, default None
If not None, then rows will be written in batches of this size at a
time. If None, all rows will be written at once.
dtype : single type or dict of column name to SQL type, default None
Optional specifying the datatype for columns. The SQL type should
be a SQLAlchemy type. If all columns are of the same type, one
single value can be used.
method : {None', 'multi', callable}, default None
Controls the SQL insertion clause used:
* None : Uses standard SQL ``INSERT`` clause (one per row).
* 'multi': Pass multiple values in a single ``INSERT`` clause.
* callable with signature ``(pd_table, conn, keys, data_iter)``.
Details and a sample callable implementation can be found in the
section :ref:`insert method <io.sql.method>`.
.. versionadded:: 0.24.0
c s i | ]
}� |�qS r r )r� rD )r; r r �
<dictcomp>^ s z&SQLDatabase.to_sql.<locals>.<dictcomp>Nr )�
TypeEngine� to_instancez The type of z is not a SQLAlchemy type)rO rt rs ru rY r; )rc )rv z.(1054, "Unknown column 'inf' in 'field list'")z inf cannot be used with MySQL)rY �
connectionzThe provided table name 'z�' is not found exactly as such in the database after writing the table, possibly due to case sensitivity issues. Consider using lower case table names.) r r� r� r� rB r rd rz r� r rc r� ZSQLAlchemyErrorry �orig�re�search�isdigit�islowerr� r �connect� table_namesr^ rY r� r� r� )r� rO rx rs rt ru rY rV r; rv r� r� r@ �my_typerh rc rg �msgZerr_textr r� r� r )r; r rw ( sD 5
zSQLDatabase.to_sqlc C s | j jS )N)r^ �tables)r� r r r r� � s zSQLDatabase.tablesc C s | j j| j jj||p| jj�S )N)r� Z run_callable�dialectrm r^ rY )r� rx rY r r r rm � s zSQLDatabase.has_tablec C sl |p
| j j}|r*| j jjdj||g��}n| j jj|�}ddlm} x"|jD ]}t|j |�rLd|j _
qLW |S )N�.r )�NumericF) r^ rY r� �get�joinr r� rH r r< Z asdecimal)r� rX rY Ztblr� �columnr r r r� � s zSQLDatabase.get_tablec C sH |p
| j j}| j||�rD| j j|g|d� | j||�j� | j j� d S )N)r] rY )r^ rY rm rb r� Zdrop�clear)r� rX rY r r r r� � s
zSQLDatabase.drop_tablec C s t || |d||d�}t|j� �S )NF)rO rt r" r; )rz ry r� )r� rO rX r" r; rh r r r �_create_sql_schema� s z SQLDatabase._create_sql_schema)NN)NTNNNN)NTN)NTNNN)ro TNNNNN)N)N)N)NN)r r r r� r� r r� rS rf � staticmethodr� rj rk rw �propertyr� rm r� r� r r r r r re b s<
4
D
_
re ZTEXTZREALZINTEGERr� ZDATEZTIME)�stringr> r? r r r r� c C sT yt | �jdd�jd�}W n4 tk
rN } ztd| � d��|�W Y d d }~X nX |S )Nzutf-8�strictz%Cannot convert identifier to UTF-8: 'rr )ry �encode�decode� UnicodeErrorrd )rx �unamerg r r r �_get_unicode_name� s
$r
c C sF t | �}t|�std��|jd�}|dkr2td��d|jdd� d S )Nz$Empty table or column name specified� r z%SQLite identifier cannot contain NULs�"z"")r
r� rd �findrp )rx r Z nul_indexr r r �_get_valid_sqlite_name� s
r zvThe spaces in these column names will not be changed. In pandas versions < 0.14, spaces were converted to underscores.c sX e Zd ZdZ� fdd�Zdd� Zdd� Zdd � Zd
d � Zd d
� Z dd� Z
dd� Z � Z S )� SQLiteTablezw
Patch the SQLTable for fallback support.
Instead of a table variable just use the Create Table statement.
c s* dd l }|jtdd� � t� j||� d S )Nr c S s
| j d�S )Nz %H:%M:%S.%f)�strftime)r� r r r �<lambda>� s z&SQLiteTable.__init__.<locals>.<lambda>)�sqlite3Zregister_adapterr �superr� )r� r' r� r )� __class__r r r� � s zSQLiteTable.__init__c C s t dj| j��S )Nz;
)ry r rh )r� r r r r� � s zSQLiteTable.sql_schemac
C s4 | j j� � }x| jD ]}|j|� qW W d Q R X d S )N)r| r� rh rS )r� r� Zstmtr r r r� � s zSQLiteTable._execute_createc s� t tt| jj��}d}t� | jd k rJx$| jd d d � D ]}|jd|� q6W � fdd�|D �}dj|�}dj|gt |� ��dj�fdd�t
|�D ��}d � | j �� d
|� d |� �}|S )
N�?r[ r c s g | ] }� |��qS r r )r� r )�escaper r r� s z0SQLiteTable.insert_statement.<locals>.<listcomp>�,c 3 s | ]}d � � d�V qdS )�(�)Nr )r� r� )�
row_wildcardsr r � <genexpr> s z/SQLiteTable.insert_statement.<locals>.<genexpr>z INSERT INTO z (z ) VALUES r� ) r$ r� ry rO rH r rt r� r r� r� rx ) r� �num_rowsr� �wldr� Zbracketed_namesZ col_names� wildcards�insert_statementr )r r r r" � s
z SQLiteTable.insert_statementc C s t |�}|j| jdd�|� d S )Nr[ )r )r$ Z executemanyr" )r� r� r" r� r� r r r r� s zSQLiteTable._execute_insertc C s2 t |�}dd� |D �}|j| jt|�d�|� d S )Nc S s g | ]}|D ]}|�q qS r r )r� r� �xr r r r� s z5SQLiteTable._execute_insert_multi.<locals>.<listcomp>)r )r$ rS r" r� )r� r� r" r� r� Zflattened_datar r r r� s z!SQLiteTable._execute_insert_multic
sR | j | j�}tjd�}dd� |D �}tt|j|��rBtjt dd� t
� � fdd�|D �}| j dk r�t | j �r�t
| j �s�| j g}n| j }dj� fd d
�|D ��}|jd | j� d |� d
�� d� | j� d dj|� d g}dd� |D �}t |��rNdj|�} dj� fdd
�|D ��}|jd� d| j d | � d � | j� d | d
� |S )z�
Return a list of SQL statements that creates a table reflecting the
structure of a DataFrame. The first entry will be a CREATE TABLE
statement while the rest will be CREATE INDEX statements.
z\s+c S s g | ]\}}}|�qS r r )r� rD r� r r r r� # s z3SQLiteTable._create_table_setup.<locals>.<listcomp>� )r� c s" g | ]\}}}� |�d | �qS )� r )r� �cname�ctyper� )r r r r� * s Nz, c 3 s | ]}� |�V qd S )Nr )r� r� )r r r r 2 s z2SQLiteTable._create_table_setup.<locals>.<genexpr>z CONSTRAINT z_pk PRIMARY KEY (r z
CREATE TABLE z (
z,
z
)c S s g | ]\}}}|r|�qS r r )r� r& r� r� r r r r� ? s r� r c 3 s | ]}� |�V qd S )Nr )r� r� )r r r r B s z
CREATE INDEX Zix_zON z ()r� �_sql_type_namer� r� �anyr� r� r� r� �_SAFE_NAMES_WARNINGr r" r� r
r rq rx )
r� r� �patr� Zcreate_tbl_stmtsr" Z cnames_brZ create_stmtsZix_colsZcnamesr )r r r s,
"
6z SQLiteTable._create_table_setupc C s� | j pi }|j|kr ||j S tj|dd�}|dkrJtjdtdd� d}n,|dkrXd }n |d
krfd }n|d krvtd
��|tkr�d }t| S )NT)r� r� zlthe 'timedelta' type is not supported, and will be written as integer values (ns frequency) to the database.r� )r� r? r� r �emptyr r� z Complex datatypes not supported) r; rx r� r� r� r� r� rd �
_SQL_TYPES)r� r@ r; r� r r r r( O s&
zSQLiteTable._sql_type_name)
r r r r� r� r� r� r" r� r� r r( �
__classcell__r r )r r r � s
5r c @ s� e Zd ZdZd dd�Zedd� �Zdd� Zed d d ��Z d d
d�Z
dd� Z d dd�Z d dd�Z
d!dd�Zd"dd�Zd#dd�Zd S )$rl z�
Version of SQLDatabase to support SQLite connections (fallback without
SQLAlchemy). This should only be used internally.
Parameters
----------
con : sqlite connection object
Fc C s || _ || _d S )N)rQ r )r� r rQ r r r r� { s zSQLiteDatabase.__init__c c sT | j j� }z:y|V | j j� W n tk
r@ | j j� � Y nX W d |j� X d S )N)r �cursor�commitrn �rollback�close)r� rT r r r r� s
z SQLiteDatabase.run_transactionc O s� | j r| j}n
| jj� }y|j||� |S tk
r� } zvy| jj� W nB tk
r� } z&td|d � d|� d��}||�W Y d d }~X nX td|d � d|� ��}||�W Y d d }~X nX d S )NzExecution failed on sql: r �
z
unable to rollbackzExecution failed on sql 'z': )rQ r r/ rS rn r1 r )r� r' r� rT rc Z inner_exc�exr r r rS � s
zSQLiteDatabase.executeNTc c sL xF| j |�}t|�tkr t|�}|s0| j� P qt|||||d�V qW dS )z+Return generator through chunked result set)rN rI r* N)r� r< �tupler$ r2 rP )r/ rV rH rN rI r* rM r r r r� � s
z SQLiteDatabase._query_iteratorc C sn t ||�}| j|� }dd� |jD �} |d k rB| j||| |||d�S | j|�}
|j� t|
| |||d�} | S d S )Nc S s g | ] }|d �qS )r r )r� Zcol_descr r r r� � s z-SQLiteDatabase.read_query.<locals>.<listcomp>)rN rI r* )r( rS � descriptionr� �_fetchall_as_listr2 rP ) r� r% rN rI r& r* rV r' r/ rH rM rO r r r rj � s(
zSQLiteDatabase.read_queryc C s |j � }t|t�st|�}|S )N)r� r r$ )r� rT r� r r r r7 � s
z SQLiteDatabase._fetchall_as_listro c
s� � r t � � r � fdd�|D �� � dk r\x2� j� D ]&\}
} t| t�s2t|
� d| � d���q2W t|| ||||� d�} | j� | j|| � dS )ai
Write records stored in a DataFrame to a SQL database.
Parameters
----------
frame: DataFrame
name: string
Name of SQL table.
if_exists: {'fail', 'replace', 'append'}, default 'fail'
fail: If table exists, do nothing.
replace: If table exists, drop it, recreate it, and insert data.
append: If table exists, insert data. Create if it does not exist.
index : boolean, default True
Write DataFrame index as a column
index_label : string or sequence, default None
Column label for index column(s). If None is given (default) and
`index` is True, then the index names are used.
A sequence should be given if the DataFrame uses MultiIndex.
schema : string, default None
Ignored parameter included for compatibility with SQLAlchemy
version of ``to_sql``.
chunksize : int, default None
If not None, then rows will be written in batches of this
size at a time. If None, all rows will be written at once.
dtype : single type or dict of column name to SQL type, default None
Optional specifying the datatype for columns. The SQL type should
be a string. If all columns are of the same type, one single value
can be used.
method : {None, 'multi', callable}, default None
Controls the SQL insertion clause used:
* None : Uses standard SQL ``INSERT`` clause (one per row).
* 'multi': Pass multiple values in a single ``INSERT`` clause.
* callable with signature ``(pd_table, conn, keys, data_iter)``.
Details and a sample callable implementation can be found in the
section :ref:`insert method <io.sql.method>`.
.. versionadded:: 0.24.0
c s i | ]
}� |�qS r r )r� rD )r; r r r� s z)SQLiteDatabase.to_sql.<locals>.<dictcomp>Nz (z) not a string)rO rt rs ru r; )r rB r ry rd r r� r� )
r� rO rx rs rt ru rY rV r; rv r@ r� rh r )r; r rw � s 4
zSQLiteDatabase.to_sqlc C s* d}d|� d�}t | j||g�j� �dkS )Nr z;SELECT name FROM sqlite_master WHERE type='table' AND name=�;r )r� rS r� )r� rx rY r �queryr r r rm & s zSQLiteDatabase.has_tablec C s d S )Nr )r� rX rY r r r r� 0 s zSQLiteDatabase.get_tablec C s dt |�� �}| j|� d S )Nz DROP TABLE )r rS )r� rx rY Zdrop_sqlr r r r� 3 s zSQLiteDatabase.drop_tablec C s t || |d||d�}t|j� �S )NF)rO rt r" r; )r ry r� )r� rO rX r" r; rh r r r r 7 s z!SQLiteDatabase._create_sql_schema)F)NTN)NTNNN)ro TNNNNN)N)N)N)NN)r r r r� r� r r� rS r r� rj r7 rw rm r� r� r r r r r rl p s.
>
rl c C s t |d�}|j| |||d�S )a�
Get the SQL db table schema for the given frame.
Parameters
----------
frame : DataFrame
name : string
name of SQL table
keys : string or sequence, default: None
columns to use a primary key
con: an open SQL database connection object or a SQLAlchemy connectable
Using SQLAlchemy makes it possible to use any DB supported by that
library, default: None
If a DBAPI2 object, only sqlite3 is supported.
dtype : dict of column name to SQL type, default None
Optional specifying the datatype for columns. The SQL type should
be a SQLAlchemy type, or a string for sqlite3 fallback connection.
)r )r" r; )rR r )rO rx r" r r; rU r r r �
get_schema> s
r: )NN)NTN)NN)NNTNNN)NNTNNr[ )NNTNNN)NTNNN)NTNNr[ )NTNNN)NTNNNN)NTNNNr[ )NTNNNN)Nro TNNNN)N)NNF)NNN)Br� �
contextlibr r r r � functoolsr r� Ztypingr r r r
r� Znumpyr= Zpandas._libs.libZ_libsr� Zpandas.core.dtypes.commonr r r
Zpandas.core.dtypes.dtypesr Zpandas.core.dtypes.missingr Zpandas.core.apir r Zpandas.core.baser Zpandas.core.tools.datetimesr r r �IOErrorr r r! r( r+ rA rG rP rS rZ r� ri rk rw rm Z table_existsr_ rR rz r� re r- r
r r* r rl r: r r r r �<module> s�
V @ c H
< \ O |