3
���h�0 � @ s d dl mZ d dlZd dlmZmZmZmZmZm Z m
Z
d dl Z d dl
mZ d dlmZ d dlmZ d dlZd dlmZ dd � Zdeeee e d �d
d�Zde
eee f e e
eef e e
eee
eee f f e e e e eee e dd� dd�Zededd�ZdS )� )� defaultdictN)�Any� DefaultDict�Dict�Iterable�List�Optional�Union)�convert_json_to_lines)�Scalar)� deprecate)� DataFramec C s2 | d dk r | d dkr | S | dd� } t | �S )zJ
Helper function that converts JSON lists to line delimited JSON.
r �[� �]���r )r
)�s� r �;/tmp/pip-build-5_djhm0z/pandas/pandas/io/json/_normalize.py�convert_to_line_delimits s r � �.)�prefix�sep�level� max_levelc C s� d}t | t�r| g} d}g }x�| D ]�}tj|�}x�|j� D ]�\} }
t | t�sTt| �} |dkrb| } n || | } t |
t� s�|dk r�||kr�|dkr:|j| �}
|
|| < q:q:|j| �}
|jt|
| ||d |�� q:W |j |� q"W |r�|d S |S )a�
A simplified json_normalize
Converts a nested dict into a flat dict ("record"), unlike json_normalize,
it does not attempt to extract a subset of the data.
Parameters
----------
ds : dict or list of dicts
prefix: the prefix, optional, default: ""
sep : str, default '.'
Nested records will generate names separated by sep,
e.g., for sep='.', { 'foo' : { 'bar' : 0 } } -> foo.bar
level: int, optional, default: 0
The number of levels in the json string.
max_level: int, optional, default: None
The max depth to normalize.
.. versionadded:: 0.25.0
Returns
-------
d - dict or list of dicts, matching `ds`
Examples
--------
IN[52]: nested_to_record(dict(flat1=1,dict1=dict(c=1,d=2),
nested=dict(e=dict(c=1,d=2),d=2)))
Out[52]:
{'dict1.c': 1,
'dict1.d': 2,
'flat1': 1,
'nested.d': 2,
'nested.e.c': 1,
'nested.e.d': 2}
FTr Nr )
�
isinstance�dict�copy�deepcopy�items�str�pop�update�nested_to_record�append) Zdsr r r r Z singletonZnew_ds�dZnew_d�k�vZnewkeyr r r r$ s2 ,
r$ �raiser
) �data� record_path�meta� meta_prefix�
record_prefix�errorsr r �returnc s� t ttf tttf tttf d�dd��t ttf tttf td��fdd� �t| t�rf| rft � S t| t
�rv| g} |dkr�t dd� | D ��r�t | � �d �} t | �S t|t�s�|g}|dkr�g }nt|t�s�|g}d
d � |D �� g �
g �t
t��� fd d �� D ��d� ���������
� f dd� ��| |i d
d� t �
�}� dk �r\|j� fdd�d�}xZ�j� D ]N\} }
|dk �r�|| } | |k�r�td| � d���tj|
td�j��|| < �qfW |S )a�
Normalize semi-structured JSON data into a flat table.
Parameters
----------
data : dict or list of dicts
Unserialized JSON objects.
record_path : str or list of str, default None
Path in each object to list of records. If not passed, data will be
assumed to be an array of records.
meta : list of paths (str or list of str), default None
Fields to use as metadata for each record in resulting table.
meta_prefix : str, default None
If True, prefix records with dotted (?) path, e.g. foo.bar.field if
meta is ['foo', 'bar'].
record_prefix : str, default None
If True, prefix records with dotted (?) path, e.g. foo.bar.field if
path to records is ['foo', 'bar'].
errors : {'raise', 'ignore'}, default 'raise'
Configures error handling.
* 'ignore' : will ignore KeyError if keys listed in meta are not
always present.
* 'raise' : will raise KeyError if keys listed in meta are not
always present.
sep : str, default '.'
Nested records will generate names separated by sep.
e.g., for sep='.', {'foo': {'bar': 0}} -> foo.bar.
max_level : int, default None
Max number of levels(depth of dict) to normalize.
if None, normalizes all levels.
.. versionadded:: 0.25.0
Returns
-------
frame : DataFrame
Normalize semi-structured JSON data into a flat table.
Examples
--------
>>> data = [{'id': 1, 'name': {'first': 'Coleen', 'last': 'Volk'}},
... {'name': {'given': 'Mose', 'family': 'Regner'}},
... {'id': 2, 'name': 'Faye Raker'}]
>>> pandas.json_normalize(data)
id name name.family name.first name.given name.last
0 1.0 NaN NaN Coleen NaN Volk
1 NaN NaN Regner NaN Mose NaN
2 2.0 Faye Raker NaN NaN NaN NaN
>>> data = [{'id': 1,
... 'name': "Cole Volk",
... 'fitness': {'height': 130, 'weight': 60}},
... {'name': "Mose Reg",
... 'fitness': {'height': 130, 'weight': 60}},
... {'id': 2, 'name': 'Faye Raker',
... 'fitness': {'height': 130, 'weight': 60}}]
>>> json_normalize(data, max_level=0)
fitness id name
0 {'height': 130, 'weight': 60} 1.0 Cole Volk
1 {'height': 130, 'weight': 60} NaN Mose Reg
2 {'height': 130, 'weight': 60} 2.0 Faye Raker
Normalizes nested data up to level 1.
>>> data = [{'id': 1,
... 'name': "Cole Volk",
... 'fitness': {'height': 130, 'weight': 60}},
... {'name': "Mose Reg",
... 'fitness': {'height': 130, 'weight': 60}},
... {'id': 2, 'name': 'Faye Raker',
... 'fitness': {'height': 130, 'weight': 60}}]
>>> json_normalize(data, max_level=1)
fitness.height fitness.weight id name
0 130 60 1.0 Cole Volk
1 130 60 NaN Mose Reg
2 130 60 2.0 Faye Raker
>>> data = [{'state': 'Florida',
... 'shortname': 'FL',
... 'info': {'governor': 'Rick Scott'},
... 'counties': [{'name': 'Dade', 'population': 12345},
... {'name': 'Broward', 'population': 40000},
... {'name': 'Palm Beach', 'population': 60000}]},
... {'state': 'Ohio',
... 'shortname': 'OH',
... 'info': {'governor': 'John Kasich'},
... 'counties': [{'name': 'Summit', 'population': 1234},
... {'name': 'Cuyahoga', 'population': 1337}]}]
>>> result = json_normalize(data, 'counties', ['state', 'shortname',
... ['info', 'governor']])
>>> result
name population state shortname info.governor
0 Dade 12345 Florida FL Rick Scott
1 Broward 40000 Florida FL Rick Scott
2 Palm Beach 60000 Florida FL Rick Scott
3 Summit 1234 Ohio OH John Kasich
4 Cuyahoga 1337 Ohio OH John Kasich
>>> data = {'A': [1, 2]}
>>> json_normalize(data, 'A', record_prefix='Prefix.')
Prefix.0
0 1
1 2
Returns normalized data with columns prefixed with the given string.
)�js�specr0 c S s2 | }t |t�r&x |D ] }|| }qW n|| }|S )z Internal function to pull field)r �list)r1 r2 �result�fieldr r r � _pull_field� s
z$_json_normalize.<locals>._pull_fieldc sB � | |�}t |t�s>tj|�r$g }nt| � d|� d|� d���|S )z�
Internal function to pull field for records, and similar to
_pull_field, but require to return list. And will raise error
if has non iterable value.
z has non list value z
for path z. Must be list or null.)r r3 �pdZisnull� TypeError)r1 r2 r4 )r6 r r �
_pull_records� s
z&_json_normalize.<locals>._pull_recordsNc s s | ]}d d� |j � D �V qdS )c S s g | ]}t |t��qS r )r r )�.0�xr r r �
<listcomp> s z-_json_normalize.<locals>.<genexpr>.<listcomp>N)�values)r: �yr r r � <genexpr> s z"_json_normalize.<locals>.<genexpr>)r r c S s g | ]}t |t�r|n|g�qS r )r r3 )r: �mr r r r< s z#_json_normalize.<locals>.<listcomp>c s g | ]}� j |��qS r )�join)r: �val)r r r r< ' s r c
sv t | t�r| g} t|�dkr�xj| D ]b}x8t� ��D ]*\}}|d t|�kr2�||d �||< q2W �||d |dd � ||d d� q"W n�x�| D ]�}�||d �}��
fdd�|D �}�jt|�� x�t� ��D ]�\}}|d t|�kr�|| }n`y�|||d � �}W nH tk
�rP } z*�dk�r.tj}ntd| � d��| �W Y d d } ~ X nX �| j|� q�W � j|� q�W d S )
Nr r )r c s( g | ] }t |t�r t|�� d �n|�qS ))r r )r r r$ )r: �r)r r r r r< 7 s z?_json_normalize.<locals>._recursive_extract.<locals>.<listcomp>�ignorez(Try running with errors='ignore' as key z is not always presentr ) r r �len�zipr% �KeyError�np�nan�extend)
r* �pathZ seen_metar �objrB �keyZrecsZmeta_val�e) �_metar6 r9 �_recursive_extractr/ �lengthsr � meta_keys� meta_vals�recordsr r r rP ) s4
*
z+_json_normalize.<locals>._recursive_extract)r c s � � | � �S )Nr )r; )r. r r �<lambda>U s z!_json_normalize.<locals>.<lambda>)�columnszConflicting metadata name z , need distinguishing prefix )Zdtype)r )r r! r r r r r r r3 r
r �anyr$ r �renamer �
ValueErrorrH �array�object�repeat) r* r+ r, r- r. r/ r r r4 r' r( r ) rO r6 r9 rP r/ rQ r rR rS r. rT r r �_json_normalizep sF w
&
"'
r] z pandas.io.json.json_normalizez1.0.0zpandas.json_normalize)r r r N)NNNNr) r N) � collectionsr r Ztypingr r r r r r r ZnumpyrH Zpandas._libs.writersr
Zpandas._typingr Zpandas.util._decoratorsr Zpandasr7 r
r r! �intr$ r] Zjson_normalizer r r r �<module> s0 $ N R l |