uawdijnntqw1x1x1
IP : 3.147.27.154
Hostname : host45.registrar-servers.com
Kernel : Linux host45.registrar-servers.com 4.18.0-513.18.1.lve.2.el8.x86_64 #1 SMP Sat Mar 30 15:36:11 UTC 2024 x86_64
Disable Function : None :)
OS : Linux
PATH:
/
home
/
.
/
..
/
lib64
/
mysql
/
.
/
.
/
..
/
python2.7
/
.
/
ensurepip
/
.
/
..
/
unittest
/
..
/
robotparser.pyc
/
/
� z fc@s}dZddlZddlZdgZdd d��YZdd d��YZdd d��YZd ejfd ��YZdS(s& robotparser.py Copyright (C) 2000 Bastian Kleineidam You can choose between two licenses when using this package: 1) GNU GPLv2 2) PSF license for Python 2.2 The robots.txt Exclusion Protocol is implemented as specified in http://www.robotstxt.org/norobots-rfc.txt i����NtRobotFileParsercBsbeZdZdd�Zd�Zd�Zd�Zd�Zd�Zd�Z d �Z d �Z RS( ss This class provides a set of methods to read, parse and answer questions about a single robots.txt file. tcCs>g|_d|_t|_t|_|j|�d|_dS(Ni(tentriestNonet default_entrytFalset disallow_allt allow_alltset_urlt last_checked(tselfturl((s#/usr/lib64/python2.7/robotparser.pyt__init__s cCs|jS(s�Returns the time the robots.txt file was last fetched. This is useful for long-running web spiders that need to check for new robots.txt files periodically. (R (R ((s#/usr/lib64/python2.7/robotparser.pytmtime!scCs ddl}|j�|_dS(sYSets the time the robots.txt file was last fetched to the current time. i����N(ttimeR (R R((s#/usr/lib64/python2.7/robotparser.pytmodified*s cCs/||_tj|�dd!\|_|_dS(s,Sets the URL referring to a robots.txt file.iiN(R turlparsethosttpath(R R ((s#/usr/lib64/python2.7/robotparser.pyR2s cCs�t�}|j|j�}g|D]}|j�^q"}|j�|j|_|jdkrkt|_nO|jdkr�|jdkr�t|_n%|jdkr�|r�|j |�ndS(s4Reads the robots.txt URL and feeds it to the parser.i�i�i�i�i�N(i�i�( t URLopenertopenR tstriptcloseterrcodetTrueRRtparse(R topenertftlinetlines((s#/usr/lib64/python2.7/robotparser.pytread7s cCsAd|jkr-|jdkr=||_q=n|jj|�dS(Nt*(t useragentsRRRtappend(R tentry((s#/usr/lib64/python2.7/robotparser.pyt _add_entryEscCs&d}d}t�}|j�x�|D]�}|d7}|s�|dkrZt�}d}q�|dkr�|j|�t�}d}q�n|jd�}|dkr�|| }n|j�}|s�q&n|jdd�}t|�dkr&|dj�j�|d<tj |dj��|d<|ddkru|dkrX|j|�t�}n|j j |d�d}q|ddkr�|dkr�|j j t |dt��d}q�q|ddkr|dkr�|j j t |dt��d}q�qq&q&W|dkr"|j|�nd S( s�parse the input lines from a robots.txt file. We allow that a user-agent: line is not preceded by one or more blank lines.iiit#t:s user-agenttdisallowtallowN(tEntryRR#tfindRtsplittlentlowerturllibtunquoteR R!t rulelinestRuleLineRR(R R tstatet linenumberR"R ti((s#/usr/lib64/python2.7/robotparser.pyRNsP cCs�|jr tS|jrtS|js'tStjtj|��}tjdd|j |j |j |j f�}tj |�}|s�d}nx-|jD]"}|j|�r�|j|�Sq�W|jr�|jj|�StS(s=using the parsed robots.txt decide if useragent can fetch urlRt/(RRRRR RR-R.t urlunparseRtparamstquerytfragmenttquoteRt applies_tot allowanceR(R t useragentR t parsed_urlR"((s#/usr/lib64/python2.7/robotparser.pyt can_fetch�s$ cCsE|j}|jdk r+||jg}ndjtt|��dS(Ns (RRRtjointmaptstr(R R((s#/usr/lib64/python2.7/robotparser.pyt__str__�s ( t__name__t __module__t__doc__R R RRR R#RR>RB(((s#/usr/lib64/python2.7/robotparser.pyRs 4 R0cBs)eZdZd�Zd�Zd�ZRS(soA rule line is a single "Allow:" (allowance==True) or "Disallow:" (allowance==False) followed by a path.cCsS|dkr | r t}ntjtj|��}tj|�|_||_dS(NR(RRR5R-R9RR;(R RR;((s#/usr/lib64/python2.7/robotparser.pyR �s cCs |jdkp |j|j�S(NR (Rt startswith(R tfilename((s#/usr/lib64/python2.7/robotparser.pyR:�scCs |jrdpdd|jS(NtAllowtDisallows: (R;R(R ((s#/usr/lib64/python2.7/robotparser.pyRB�s(RCRDRER R:RB(((s#/usr/lib64/python2.7/robotparser.pyR0�s R(cBs2eZdZd�Zd�Zd�Zd�ZRS(s?An entry has one or more user-agents and zero or more rulelinescCsg|_g|_dS(N(R R/(R ((s#/usr/lib64/python2.7/robotparser.pyR �s cCsjg}x'|jD] }|jd|dg�qWx*|jD] }|jt|�dg�q:Wdj|�S(Ns User-agent: s R(R textendR/RAR?(R trettagentR ((s#/usr/lib64/python2.7/robotparser.pyRB�s cCs]|jd�dj�}x=|jD]2}|dkr9tS|j�}||kr#tSq#WtS(s2check if this entry applies to the specified agentR4iR (R*R,R RR(R R<RL((s#/usr/lib64/python2.7/robotparser.pyR:�s cCs.x'|jD] }|j|�r |jSq WtS(sZPreconditions: - our agent applies to this entry - filename is URL decoded(R/R:R;R(R RGR ((s#/usr/lib64/python2.7/robotparser.pyR;�s (RCRDRER RBR:R;(((s#/usr/lib64/python2.7/robotparser.pyR(�s RcBs#eZd�Zd�Zd�ZRS(cGs tjj||�d|_dS(Ni�(R-tFancyURLopenerR R(R targs((s#/usr/lib64/python2.7/robotparser.pyR �scCsdS(N(NN(R(R Rtrealm((s#/usr/lib64/python2.7/robotparser.pytprompt_user_passwd�scCs(||_tjj||||||�S(N(RR-RMthttp_error_default(R R tfpRterrmsgtheaders((s#/usr/lib64/python2.7/robotparser.pyRQ�s (RCRDR RPRQ(((s#/usr/lib64/python2.7/robotparser.pyR�s (((( RERR-t__all__RR0R(RMR(((s#/usr/lib64/python2.7/robotparser.pyt<module> s �$
/home/./../lib64/mysql/././../python2.7/./ensurepip/./../unittest/../robotparser.pyc