3
���hf^ � @ s� d Z ddlZddlmZmZ ddlmZ ddlZddlm Z m
Z
m Z m Z m
Z
mZ ddlZddljjZddlmZmZmZmZmZmZm Z m Z m Z m Z m Z m!Z!m"Z"m#Z#m$Z$m%Z% ddl&m'Z'm(Z( ddl)m*Z* ddl+j,j-j.Z.e e/e0f e e/e0f d �d
d �Z1e e/e0f e e/e0f d �d d
�Z2e e/e0f e e/e0f d �dd�Z3dd� Z4dd� Z5e5e3e2e1e'�fe0e0d�dd�Z6dd� Z7e7e8�Z9e7e0�Z:e;e<dd� dd � e=e�D ���Z>e>fd d �Z?e;e@d d� e>��ZAe?ejB�ZCe?ejD�ZEe?ejF�ZGe?ejH�ZIe?ejJ�ZKe?ejL�ZMe?ejN�ZOe?ejP�ZQe?ejR�ZSe?ejT�ZUe?ejV�ZWe?ejX�ZYe?ej�ZZe?ej[�Z\e;d d!d"g�Z]e;d#d$d%d&d'd(d)d*d+d,d-g �Z^eEeCB eWB eYB eZB e\B eIB e^B e] Z_eAe_ e]B Z`e_e`@ Zad.ea� �Zbea �s�tceb��e0e dL d0�d1d2�Zded3d4d5�Zee e0 e e
ee ge
ee f d6�d7d8�Zfd9d:� Zgee d;�Zhd<d=� Ziefe_�eieh�G d>d4� d4ejj���Zke;d?d@dAdBg�Zle;e#e" �Zmefe_elB eMe;d@dCdAdBdDg�B �G dEdF� dFek��Znefe_elB e;dGg�B �G dHdI� dIek��ZoG dJd"� d"�ZpeoendK�ZqdS )Mz
:func:`~pandas.eval` parsers.
� N)�partial�reduce)� iskeyword)�Callable�Optional�Set�Tuple�Type�TypeVar)�
_LOCAL_TAG�BinOp�Constant�Div�FuncNode�Op�Term�UnaryOp�UndefinedVariableError�_arith_ops_syms�_bool_ops_syms�
_cmp_ops_syms�_mathops� _reductions�_unary_ops_syms�is_term)�clean_backtick_quoted_toks�tokenize_string)�Scope)�tok�returnc C s | \}}||dkrdn|fS )a`
Rewrite the assignment operator for PyTables expressions that use ``=``
as a substitute for ``==``.
Parameters
----------
tok : tuple of int, str
ints correspond to the all caps constants in the tokenize module
Returns
-------
tuple of int, str
Either the input or token or the replacement values
�=z==� )r �toknum�tokvalr! r! �>/tmp/pip-build-5_djhm0z/pandas/pandas/core/computation/expr.py�_rewrite_assign& s r% c C sF | \}}|t jkr>|dkr$t jdfS |dkr6t jdfS ||fS ||fS )aj
Replace ``&`` with ``and`` and ``|`` with ``or`` so that bitwise
precedence is changed to boolean precedence.
Parameters
----------
tok : tuple of int, str
ints correspond to the all caps constants in the tokenize module
Returns
-------
tuple of int, str
Either the input or token or the replacement values
�&�and�|�or)�tokenize�OP�NAME)r r" r# r! r! r$ �_replace_booleans9 s
r- c C s, | \}}|t jkr$|dkr$t jtfS ||fS )a(
Replace local variables with a syntactically valid name.
Parameters
----------
tok : tuple of int, str
ints correspond to the all caps constants in the tokenize module
Returns
-------
tuple of int, str
Either the input or token or the replacement values
Notes
-----
This is somewhat of a hack in that we rewrite a string such as ``'@a'`` as
``'__pd_eval_local_a'`` by telling the tokenizer that ``__pd_eval_local_``
is a ``tokenize.OP`` and to replace the ``'@'`` symbol with it.
�@)r* r+ r )r r" r# r! r! r$ �_replace_localsR s
r/ c s � �fdd�S )z
Compose 2 callables.
c s � �| |��S )Nr! )�args�kwargs)�f�gr! r$ �<lambda>p s z_compose2.<locals>.<lambda>r! )r2 r3 r! )r2 r3 r$ � _compose2l s r5 c G s t | �dkstd��tt| �S )z&
Compose 2 or more callables.
� z.At least 2 callables must be passed to compose)�len�AssertionErrorr r5 )�funcsr! r! r$ �_composes s r: )�sourcer c s, t � �std��tj� fdd�t| �D ��S )a�
Compose a collection of tokenization functions.
Parameters
----------
source : str
A Python source code string
f : callable
This takes a tuple of (toknum, tokval) as its argument and returns a
tuple with the same structure but possibly different elements. Defaults
to the composition of ``_rewrite_assign``, ``_replace_booleans``, and
``_replace_locals``.
Returns
-------
str
Valid Python source code
Notes
-----
The `f` parameter can be any callable that takes *and* returns input of the
form ``(toknum, tokval)``, where ``toknum`` is one of the constants from
the ``tokenize`` module and ``tokval`` is a string.
zf must be callablec 3 s | ]}� |�V qd S )Nr! )�.0�x)r2 r! r$ � <genexpr>� s z _preparse.<locals>.<genexpr>)�callabler8 r* �
untokenizer )r; r2 r! )r2 r$ � _preparse{ s rA c s � fdd�S )zO
Factory for a type checking function of type ``t`` or tuple of types.
c s t | j� �S )N)�
isinstance�value)r= )�tr! r$ r4 � s z_is_type.<locals>.<lambda>r! )rD r! )rD r$ �_is_type� s rE c C s t | t�ot| tj�S )N)rB �type�
issubclass�ast�AST)r= r! r! r$ r4 � s r4 c c s | ]}t t|�V qd S )N)�getattrrH )r< �noder! r! r$ r> � s r> c s � fdd�|D �}t |�S )zE
Filter out AST nodes that are subclasses of ``superclass``.
c 3 s | ]}t |� �r|jV qd S )N)rG �__name__)r< rK )�
superclassr! r$ r> � s z _filter_nodes.<locals>.<genexpr>)� frozenset)rM Z all_nodesZ
node_namesr! )rM r$ �
_filter_nodes� s rO c C s | j S )N)rL )r= r! r! r$ r4 � s ZAssignZModule�ExprZYieldZ GeneratorExpZIfExpZDictCompZSetComp�Repr�Lambdar rI ZIsZIsNotz$cannot both support and not support .)� node_namer c s � fdd�}|S )zV
Return a function that raises a NotImplementedError with a passed node name.
c s t d� � d���d S )N�'z' nodes are not implemented)�NotImplementedError)�selfr0 r1 )rS r! r$ r2 � s z _node_not_implemented.<locals>.fr! )rS r2 r! )rS r$ �_node_not_implemented� s rW �_T�BaseExprVisitor)Zbound)�nodesr c s t t t t d�� fdd� }|S )z�
Decorator to disallow certain nodes from parsing. Raises a
NotImplementedError instead.
Returns
-------
callable
)�clsr c sF f | _ x:� D ]2}t|�}d|� �}| j |f7 _ t| ||� q W | S )N�visit_)�unsupported_nodesrW �setattr)r[ rK Z
new_method�name)rZ r! r$ �
disallowed s
z disallow.<locals>.disallowed)r rX )rZ r` r! )rZ r$ �disallow s
ra c s � �fdd�}|S )zw
Return a function to create an op class with its symbol already passed.
Returns
-------
callable
c s t � �f|�|�S )z�
Return a partial function with an Op subclass with an operator already passed.
Returns
-------
callable
)r )rV rK r0 r1 )�op_class� op_symbolr! r$ r2 ! s z_op_maker.<locals>.fr! )rb rc r2 r! )rb rc r$ � _op_maker s
rd )�binaryZunaryc s � fdd�}|S )z9
Decorator to add default implementation of ops.
c st xn� j � D ]b\}}t| |� d��}t| |� d��}x8|D ]0}|| }|d k r8t||�}t| d|� �|� q8W q
W | S )NZ_opsZ
_op_nodes_mapr\ )�itemsrJ rd r^ )r[ Z op_attr_namerb �opsZops_map�opZop_nodeZmade_op)�
op_classesr! r$ r2 6 s
zadd_ops.<locals>.fr! )ri r2 r! )ri r$ �add_ops1 s rj c @ st e Zd ZU dZeZee eZe e
e Z dZZ
eee e
��ZeZd[Zeeee��ZejejejejejejejejiZee df e fdd�Z d d � Z d d � Z!d d!� Z"d"d#� Z#d\d$d%�Z$d&d'� Z%d(d)� Z&d_d2d3�Z'd4d5� Z(d6d7� Z)d8d9� Z*d:d;� Z+d<d=� Z,d>d?� Z-d@dA� Z.dBdC� Z/dDdE� Z0e0Z1dFdG� Z2dHdI� Z3dJdK� Z4dLdM� Z5dNdO� Z6d`dPdQ�Z7dRdS� Z8dTdU� Z9dVdW� Z:dXdY� Z;dS )arY z�
Custom ast walker. Parsers of other engines should subclass this class
if necessary.
Parameters
----------
env : Scope
engine : str
parser : str
preparser : callable
�Gt�Lt�GtE�LtE�Eq�NotEq�In�NotIn�BitAnd�BitOr�And�Or�Add�Sub�MultN�Pow�FloorDiv�Mod�UAdd�USub�Invert�Not.c C s" || _ || _|| _|| _d | _d S )N)�env�engine�parser� preparser�assigner)rV r� r� r� r� r! r! r$ �__init__{ s
zBaseExprVisitor.__init__c K s� t |t�rl| j|�}ytjtj|��}W nB tk
rj } z&tdd� |j� D ��rVd|_ |�W Y d d }~X nX dt
|�j }t | |�}||f|�S )Nc s s | ]}t |�V qd S )N)r )r< r= r! r! r$ r> � s z(BaseExprVisitor.visit.<locals>.<genexpr>z4Python keyword not valid identifier in numexpr queryr\ )
rB �strr� rH Zfix_missing_locations�parse� SyntaxError�any�split�msgrF rL rJ )rV rK r1 �clean�e�method�visitorr! r! r$ �visit� s
zBaseExprVisitor.visitc K s. t |j�dkrtd��|jd }| j|f|�S )Nr6 z#only a single expression is allowedr )r7 �bodyr� r� )rV rK r1 �exprr! r! r$ � visit_Module� s
z BaseExprVisitor.visit_Modulec K s | j |jf|�S )N)r� rC )rV rK r1 r! r! r$ �
visit_Expr� s zBaseExprVisitor.visit_Exprc C s� |j }t|�}t|�r�t|�r�|| jkr�tt||f�\}}tt||f�\}} |s\|s\|s\| rh| j| � }| r�| jj|j g�}
| j
|
| j�}|r�| jj|j g�}
| j
|
| j�}| j |�} | |||fS )N) rh rF r � rewrite_map�map�_is_list�_is_strr� �add_tmprC � term_typer� ) rV rK �left�rightZ op_instanceZop_typeZ left_listZ
right_listZleft_strZ right_strr_ rh r! r! r$ �_rewrite_membership_op� s
z&BaseExprVisitor._rewrite_membership_opc C sR |d kr| j |jdd�}|d kr0| j |jdd�}| j|||�\}}}}||||fS )Nr� )�sider� )r� r� r� r� )rV rK r� r� rh rb r! r! r$ �_maybe_transform_eq_ne� s z&BaseExprVisitor._maybe_transform_eq_nec C s� t jt j�}|jrPt|d�rP|j rP|j|krP| jjt j|j��}| j || j�}|jr�t|d�r�|j r�|j|kr�| jjt j|j��}| j || j�}||fS )NrC )
�npZdtypeZfloat32Z is_scalar�hasattr� return_typer� r� rC r� )rV r� r� Zf32r_ r! r! r$ �_maybe_downcast_constants� s
z)BaseExprVisitor._maybe_downcast_constantsc C s |j | j| j| j| j|�S )N)�evaluater� r� r� r� )rV �binop�eval_in_pythonr! r! r$ � _maybe_eval� s zBaseExprVisitor._maybe_eval�in�not in�==�!=�<�>�<=�>=c C s� |||�}|j r2td|j� d|j� d|j� d���| jdkrl|jtkrRt|dd�s^t|dd�rl| j|| j�S |j|kr�| j||�S | jdkr�t|dd �t ks�t|dd �t kr�| j||| �S |S ) Nz unsupported operand type(s) for z: 'z' and 'rT ZpytablesZ is_datetimeFr� )
Zhas_invalid_return_type� TypeErrorrh rF r� r rJ r� �
binary_ops�object)rV rh rb �lhs�rhsr� Zmaybe_eval_in_python�resr! r! r$ �_maybe_evaluate_binop� s
z%BaseExprVisitor._maybe_evaluate_binopc K s2 | j |�\}}}}| j||�\}}| j||||�S )N)r� r� r� )rV rK r1 rh rb r� r� r! r! r$ � visit_BinOp s zBaseExprVisitor.visit_BinOpc K s dd� S )Nc S s
t | |�S )N)r )r� r� r! r! r$ r4
s z+BaseExprVisitor.visit_Div.<locals>.<lambda>r! )rV rK r1 r! r! r$ � visit_Div s zBaseExprVisitor.visit_Divc K s | j |j�}| j |j�}||�S )N)r� rh �operand)rV rK r1 rh r� r! r! r$ �
visit_UnaryOp s z BaseExprVisitor.visit_UnaryOpc K s | j |j| jf|�S )N)r� �idr� )rV rK r1 r! r! r$ �
visit_Name s zBaseExprVisitor.visit_Namec K s | j |j| j�S )N)�
const_typerC r� )rV rK r1 r! r! r$ �visit_NameConstant s z"BaseExprVisitor.visit_NameConstantc K s | j |j| j�S )N)r� �nr� )rV rK r1 r! r! r$ � visit_Num s zBaseExprVisitor.visit_Numc K s | j |j| j�S )N)r� r� r� )rV rK r1 r! r! r$ �visit_Constant s z BaseExprVisitor.visit_Constantc K s | j j|j�}| j|| j �S )N)r� r� �sr� )rV rK r1 r_ r! r! r$ � visit_Str s zBaseExprVisitor.visit_Strc s* � j j� fdd�|jD ��}� j|� j �S )Nc s g | ]}� j |�� j��qS r! )r� r� )r< r� )rV r! r$ �
<listcomp>% s z.BaseExprVisitor.visit_List.<locals>.<listcomp>)r� r� Zeltsr� )rV rK r1 r_ r! )rV r$ �
visit_List$ s zBaseExprVisitor.visit_Listc K s | j |j�S )z
df.index[4] )r� rC )rV rK r1 r! r! r$ � visit_Index* s zBaseExprVisitor.visit_Indexc
K s� dd l }| j|j�}| j|j�}|j|| j| j| jd�}y|j| }W n4 tk
rz |j|| j| j| jd�}|| }Y nX | jj |�} | j
| | jd�S )Nr )Z
local_dictr� r� )r� ) �pandasr� rC �slice�evalr� r� r� �AttributeErrorr� r� )
rV rK r1 �pdrC Zslobj�result�vr� r_ r! r! r$ �visit_Subscript. s z BaseExprVisitor.visit_Subscriptc K sZ |j }|dk r| j|�j}|j}|dk r4| j|�j}|j}|dk rN| j|�j}t|||�S )z df.index[slice(4,6)] N)�lowerr� rC �upper�stepr� )rV rK r1 r� r� r� r! r! r$ � visit_SliceB s zBaseExprVisitor.visit_Slicec K s� t |j�dkrtd��t|jd tj�s0td��| jjdkrDtd��y| j |jd f|�}W n t
k
r| |jd j }Y nX t |d|�| _
| j
dkr�td��| j |jf|�S ) z�
support a single assignment node, like
c = a + b
set the assigner at the top level, must be a Name node which
might or might not exist in the resolvers
r6 z#can only assign a single expressionr z5left hand side of an assignment must be a single nameNz%cannot assign without a target objectr_ z@left hand side of an assignment must be a single resolvable name)r7 �targetsr� rB rH �Namer� �target�
ValueErrorr� r r� rJ r� rC )rV rK r1 r� r! r! r$ � visit_AssignP s
z BaseExprVisitor.visit_Assignc K s� |j }|j}|j}t|tj�r�| j|�j}y$t||�}| jj |�}| j
|| j�S t k
r~ t|tj �rx|j
|krx|S � Y nX tdt|�j� ���d S )NzInvalid Attribute context )�attrrC �ctxrB rH ZLoadr� rJ r� r� r� r� r� r� r� rF rL ) rV rK r1 r� rC r� �resolvedr� r_ r! r! r$ �visit_Attributen s
z BaseExprVisitor.visit_Attributec sr t |jtj�r(|jjdkr(� j|j�}nft |jtj�s@td��nNy� j|j�}W n< t k
r� yt
|jj �}W n t k
r� � Y nX Y nX |d kr�t d|jj � ���t
|d�r�|j}t |t
�r�� fdd�|jD �}|jr�td|j� d���||� S � fd d�|jD �}xN|jD ]D}t |tj��s8t d
|jj � d ���|j�r� j|j�j||j< �qW � j|||�� j�S d S ) N�__call__z"Only named functions are supportedzInvalid function call rC c s g | ]}� j |��qS r! )r� )r< �arg)rV r! r$ r� � s z.BaseExprVisitor.visit_Call.<locals>.<listcomp>z
Function "z$" does not support keyword argumentsc s g | ]}� j |�j�qS r! )r� rC )r< r� )rV r! r$ r� � s z keyword error in function call 'rT )rB �funcrH � Attributer� r� r� r� r� r r r� r� r� rC r0 �keywordsr_ �keywordr� r� r� )rV rK r� r1 r� Znew_args�keyr! )rV r$ �
visit_Call� s8
zBaseExprVisitor.visit_Callc C s |S )Nr! )rV rh r! r! r$ � translate_In� s z BaseExprVisitor.translate_Inc K s� |j }|j}t|�dkrF| j|d �}tj||j|d d�}| j|�S |j}g }xDt||�D ]6\}} | jtj | g|| j|�gd��}
| }|j
|
� q\W | jtj tj � |d��S )Nr6 r )rh r� r� )� comparatorsr� rg )rh �values)
rg r� r7 r� rH r r� r� �zipZCompare�append�BoolOpru ) rV rK r1 rg �compsrh r� r� r� �compZnew_noder! r! r$ �
visit_Compare� s
z BaseExprVisitor.visit_Comparec C s t |ttf�r|S | j|�S )N)rB r r r� )rV Zbopr! r! r$ �_try_visit_binop� s z BaseExprVisitor._try_visit_binopc s � �fdd�}� j }t||�S )Nc s< �j | �}�j |�}�j� ||�\}}}}�j|� j||�S )N)r� r� r� rh )r= �yr� r� rh rb )rK rV r! r$ r� � s
z-BaseExprVisitor.visit_BoolOp.<locals>.visitor)r� r )rV rK r1 r� Zoperandsr! )rK rV r$ � visit_BoolOp� s z BaseExprVisitor.visit_BoolOp)rk rl rm rn ro rp rq rr rs rt ru rv rw rx ry Nrz r{ r| )r} r~ r r� )NN�r� r� �r� r� r� r� r� r� )r� r� )N)<rL �
__module__� __qualname__�__doc__r
r� r r r� r r r r� Zbinary_op_nodes�dictr� Zbinary_op_nodes_mapr Z unary_opsZunary_op_nodesZunary_op_nodes_maprH ro rq rp rr r� r r� r] rA r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� Z visit_Tupler� r� r� r� r� r� r� r� r� r� r! r! r! r$ rY D s|
!
.�Dictr� rq rr r� r c s0 e Zd Zeeeeee�d�f� fdd� Z � Z
S )�PandasExprVisitor)r2 c s t � j||||� d S )N)�superr� )rV r� r� r� r� )� __class__r! r$ r� � s
zPandasExprVisitor.__init__) rL r� r� r rA r: r/ r- r r� �
__classcell__r! r! )r� r$ r� � s
r� r� c s$ e Zd Zdd� f� fdd� Z� ZS )�PythonExprVisitorc C s | S )Nr! )r= r! r! r$ r4 � s zPythonExprVisitor.<lambda>c s t � j||||d� d S )N)r� )r� r� )rV r� r� r� r� )r� r! r$ r� � s zPythonExprVisitor.__init__)rL r� r� r� r� r! r! )r� r$ r� � s r� c @ s| e Zd ZU dZeeedeee e e
d�dd�Z e d d
� �Z
d d � Zed
�dd�Ze
d
�dd�Zdd� Ze dd� �ZdS )rP a
Object encapsulating an expression.
Parameters
----------
expr : str
engine : str, optional, default 'numexpr'
parser : str, optional, default 'pandas'
env : Scope, optional, default None
level : int, optional, default 2
�numexprr� Nr )r� r� r� �levelc C sL || _ |pt|d d�| _|| _|| _t| | j| j| j�| _| j� | _d S )Nr6 )r� ) r� r r� r� r� �_parsers�_visitorr� �terms)rV r� r� r� r� r� r! r! r$ r� s z
Expr.__init__c C s t | jdd �S )Nr� )rJ r )rV r! r! r$ r� s z
Expr.assignerc C s | j | j�S )N)r r� )rV r! r! r$ r� s z
Expr.__call__)r c C s t j| j�S )N)�printingZ pprint_thingr )rV r! r! r$ �__repr__ s z
Expr.__repr__c C s
t | j�S )N)r7 r� )rV r! r! r$ �__len__ s z Expr.__len__c C s | j j| j�S )z&
Parse an expression.
)r r� r� )rV r! r! r$ r� " s z
Expr.parsec C s2 t | j�rt| jjg�S tdd� tj| j�D ��S )z1
Get the names in an expression.
c s s | ] }|j V qd S )N)r_ )r< Ztermr! r! r$ r> / s z Expr.names.<locals>.<genexpr>)r r rN r_ �com�flatten)rV r! r! r$ �names( s
z
Expr.names)r� r� Nr )rL r� r� r� r r� r� r� r� r �intr� �propertyr� r� r r r� r r! r! r! r$ rP � s
)�pythonr� ).N)rr� rH � functoolsr r r� r r* Ztypingr r r r r r
Znumpyr� Zpandas.core.common�core�commonr Zpandas.core.computation.opsr r r
r r r r r r r r r r r r r Z pandas.core.computation.parsingr r Z pandas.core.computation.scoper Zpandas.io.formats.printing�io�formatsr r r� r% r- r/ r5 r: rA rE �listr� r� rN �filter�dirZ
_all_nodesrO r� Z_all_node_names�modZ
_mod_nodesZstmtZ _stmt_nodesr� Z _expr_nodesZ expr_contextZ_expr_context_nodesr� Z _slice_nodesZboolopZ
_boolop_nodes�operatorZ_operator_nodesZunaryopZ_unary_op_nodesZcmpopZ
_cmp_op_nodesZ
comprehensionZ_comprehension_nodesZ
excepthandlerZ_handler_nodesZ argumentsZ_arguments_nodesZ_keyword_nodes�aliasZ _alias_nodesZ
_hacked_nodesZ_unsupported_expr_nodesZ_unsupported_nodesZ_base_supported_nodes� intersection�_msgr8 rW rX ra rd Z _op_classesrj Z NodeVisitorrY Z_python_not_supportedZ_numexpr_supported_callsr� r� rP r� r! r! r! r$ �<module> s� H
&
" = |