|
Server : LiteSpeed System : Linux server51.dnsbootclub.com 4.18.0-553.62.1.lve.el8.x86_64 #1 SMP Mon Jul 21 17:50:35 UTC 2025 x86_64 User : nandedex ( 1060) PHP Version : 8.1.33 Disable Function : NONE Directory : /opt/cppython/lib/python3.8/test/__pycache__/ |
U
>��g�* � @ s d dl Z d dlZd dlZd dlZd dlZd dlmZ d dlm Z m
Z
G dd� d�Z G dd� de ej �Z
G dd � d e ej �ZG d
d � d e ej �ZG d d
� d
e ej �ZG dd� de �ZG dd� deej �ZG dd� deej �ZG dd� de�ZG dd� de ej �ZG dd� de ej �ZG dd� de ej �ZG d d � d e ej �ZG d d � d e�ZG d d!� d!e ej �ZG d"d#� d#e ej �ZG d$d%� d%e ej �Z G d&d'� d'e ej �Z G d(d)� d)eej �Z G d*d+� d+e ej �Z G d,d-� d-e �Z G d.d/� d/ej �Z!G d0d1� d1ej �Z"e#d2k�re�$� dS )3� N)�support)�BaseHTTPRequestHandler�
HTTPServerc @ sH e Zd ZdZdZg Zg ZdZdd� Zdd� Z dd � Z
d
d � Z d d
� Z dS )�
BaseRobotTest� Ztest_robotparserNc C s, t �| j��� }tj�� | _| j�|� d S �N) �io�StringIO�
robots_txt� readlines�urllib� robotparser�RobotFileParser�parser�parse)�self�lines� r �4/opt/cppython/lib/python3.8/test/test_robotparser.py�setUp s zBaseRobotTest.setUpc C s$ t |t�r|\}}||fS | j|fS r )�
isinstance�tuple�agent�r �urlr r r r �get_agent_and_url s
z BaseRobotTest.get_agent_and_urlc
C sL | j D ]@}| �|�\}}| j||d�� | �| j�||�� W 5 Q R X qd S �N)r r )�goodr �subTest�
assertTruer � can_fetchr r r r �test_good_urls s
z BaseRobotTest.test_good_urlsc
C sL | j D ]@}| �|�\}}| j||d�� | �| j�||�� W 5 Q R X qd S r )�badr r � assertFalser r r r r r �
test_bad_urls" s
zBaseRobotTest.test_bad_urlsc C s | � | j�� | j� d S r )� assertEqualr � site_maps�r r r r �test_site_maps( s z BaseRobotTest.test_site_maps)
�__name__�
__module__� __qualname__r
r r r" r& r r r! r$ r( r r r r r
s r c @ s" e Zd ZdZddgZdddgZdS )�UserAgentWildcardTestz�User-agent: *
Disallow: /cyberworld/map/ # This is an infinite virtual URL space
Disallow: /tmp/ # these will soon disappear
Disallow: /foo.html
�/�
/test.html�/cyberworld/map/index.htmlz/tmp/xxx� /foo.htmlN�r) r* r+ r
r r" r r r r r, , s r, c @ s e Zd ZdZdddgZdgZdS )� CrawlDelayAndCustomAgentTestz�# robots.txt for http://www.example.com/
User-agent: *
Crawl-delay: 1
Request-rate: 3/15
Disallow: /cyberworld/map/ # This is an infinite virtual URL space
# Cybermapper knows where to go.
User-agent: cybermapper
Disallow:
r- r. )Z cybermapperr/ r/ Nr1 r r r r r2 7 s
r2 c @ s& e Zd ZdZddgZdgZddgZdS )� SitemapTesta # robots.txt for http://www.example.com/
User-agent: *
Sitemap: http://www.gstatic.com/s2/sitemaps/profiles-sitemap.xml
Sitemap: http://www.google.com/hostednews/sitemap_index.xml
Request-rate: 3/15
Disallow: /cyberworld/map/ # This is an infinite virtual URL space
r- r. r/ z7http://www.gstatic.com/s2/sitemaps/profiles-sitemap.xmlz2http://www.google.com/hostednews/sitemap_index.xmlN)r) r* r+ r
r r" r& r r r r r3 H s
�r3 c @ s e Zd ZdZg ZdddgZdS )�RejectAllRobotsTestz(# go away
User-agent: *
Disallow: /
r/ r- �/tmp/Nr1 r r r r r4 Y s r4 c @ s e Zd ZdZdZdd� ZdS )�BaseRequestRateTestNc
C s� | j }| j| j D ]�}| �|�\}}| j||d��p | �|�|�| j� |�|�}| �|| j� | jd k r�| �|t j
j � | �|j | jj � | �|j
| jj
� W 5 Q R X qd S r )r r r" r r r% � crawl_delay� request_rateZassertIsInstancer r
� RequestRateZrequestsZseconds)r r r r Zparsed_request_rater r r �test_request_rateg s(
���z%BaseRequestRateTest.test_request_rate)r) r* r+ r8 r7 r: r r r r r6 c s r6 c @ s e Zd ZdZdgZdS )�
EmptyFileTestr z/fooN)r) r* r+ r
r r r r r r; s r; c @ s> e Zd ZdZdZej�dd�ZdZ dgZ
ddd d
d d d
gZ dS )� CrawlDelayAndRequestRateTestz�User-agent: figtree
Crawl-delay: 3
Request-rate: 9/30
Disallow: /tmp
Disallow: /a%3cd.html
Disallow: /a%2fb.html
Disallow: /%7ejoe/index.html
�figtree� � � )r= r0 �/tmpz /tmp.html� /tmp/a.html� /a%3cd.html� /a%3Cd.htmlz /a%2fb.htmlz/~joe/index.htmlN) r) r* r+ r
r r r
r9 r8 r7 r r" r r r r r< � s
�r< c @ s e Zd ZdZdS )�DifferentAgentTestz FigTree Robot libwww-perl/5.04N�r) r* r+ r r r r r rE � s rE c @ s* e Zd ZdZdgZddddddgZd Zd
S ) �InvalidRequestRateTestz�User-agent: *
Disallow: /tmp/
Disallow: /a%3Cd.html
Disallow: /a/b.html
Disallow: /%7ejoe/index.html
Crawl-delay: 3
Request-rate: 9/banana
rA r5 rB rC rD z /a/b.htmlz/%7Ejoe/index.htmlr@ N)r) r* r+ r
r r" r7 r r r r rG � s
�rG c @ s e Zd ZdZdgZg ZdS )�InvalidCrawlDelayTestz2User-Agent: *
Disallow: /.
Crawl-delay: pears
r0 Nr1 r r r r rH � s rH c @ s e Zd ZdZdZdgZdgZdS )� AnotherInvalidRequestRateTestzeUser-agent: Googlebot
Allow: /folder1/myfile.html
Disallow: /folder1/
Request-rate: whale/banana
� Googlebot�/folder1/myfile.html�/folder1/anotherfile.htmlN�r) r* r+ r
r r r" r r r r rI � s rI c @ s e Zd ZdZdZdgZdS )�UserAgentOrderingTestzMUser-agent: Googlebot
Disallow: /
User-agent: Googlebot-Mobile
Allow: /
rJ z/something.jpgN)r) r* r+ r
r r" r r r r rN � s rN c @ s e Zd ZdZdS )�UserAgentGoogleMobileTestzGooglebot-MobileNrF r r r r rO � s rO c @ s e Zd ZdZdZdgZdgZdS )�GoogleURLOrderingTestzJUser-agent: Googlebot
Allow: /folder1/myfile.html
Disallow: /folder1/
Z googlebotrK rL NrM r r r r rP � s rP c @ s e Zd ZdZdgZdgZdS )�DisallowQueryStringTestz2User-agent: *
Disallow: /some/path?name=value
�
/some/pathz/some/path?name=valueNr1 r r r r rQ � s rQ c @ s e Zd ZdZdgZdgZdS )� UseFirstUserAgentWildcardTestzNUser-agent: *
Disallow: /some/path
User-agent: *
Disallow: /another/path
z
/another/pathrR Nr1 r r r r rS � s rS c @ s e Zd ZdZdgZdgZdS )�EmptyQueryStringTestz>User-agent: *
Allow: /some/path?
Disallow: /another/path?
z /some/path?z/another/path?Nr1 r r r r rT � s rT c @ s0 e Zd ZdZej�dd�ZdZddgZ dgZ
dS ) �DefaultEntryTestzOUser-agent: *
Crawl-delay: 1
Request-rate: 3/15
Disallow: /cyberworld/map/
r@ � � r- r. r/ N) r) r* r+ r
r r
r9 r8 r7 r r" r r r r rU s
rU c @ s e Zd ZdZdZdd� ZdS )�StringFormattingTestz�User-agent: *
Crawl-delay: 1
Request-rate: 3/15
Disallow: /cyberworld/map/ # This is an infinite virtual URL space
# Cybermapper knows where to go.
User-agent: cybermapper
Disallow: /some/path
zxUser-agent: cybermapper
Disallow: /some/path
User-agent: *
Crawl-delay: 1
Request-rate: 3/15
Disallow: /cyberworld/map/c C s | � t| j�| j� d S r )r% �strr �expected_outputr' r r r �test_string_formatting( s z+StringFormattingTest.test_string_formattingN)r) r* r+ r
rZ r[ r r r r rX s
rX c @ s e Zd Zdd� Zdd� ZdS )� RobotHandlerc C s | � dd� d S )Ni� zForbidden access)Z
send_errorr' r r r �do_GET. s zRobotHandler.do_GETc G s d S r r )r �format�argsr r r � log_message1 s zRobotHandler.log_messageN)r) r* r+ r] r` r r r r r\ , s r\ c @ s* e Zd Zdd� Zdd� Zejdd� �ZdS )� PasswordProtectedSiteTestCasec C sB t tjdft�| _tjd| jjddid�| _d| j_ | j�
� d S )Nr zHTTPServer servingZ
poll_intervalg{�G�z�?)�name�target�kwargsT) r r �HOSTr\ �server� threading�ThreadZ
serve_forever�t�daemon�startr' r r r r 7 s �z#PasswordProtectedSiteTestCase.setUpc C s" | j �� | j�� | j �� d S r )rf �shutdownri �joinZ server_closer' r r r �tearDownD s
z&PasswordProtectedSiteTestCase.tearDownc C s\ | j j}dtj d t|d � }|d }tj�� }|�|� |� � | �
|� d|�� d S )Nzhttp://�:rW z /robots.txt�*) rf Zserver_addressr re rY r r
r Zset_url�readr# r )r �addrr Z
robots_urlr r r r �testPasswordProtectedSiteI s
z7PasswordProtectedSiteTestCase.testPasswordProtectedSiteN)r) r* r+ r rn r Z reap_threadsrs r r r r ra 5 s
ra c @ sF e Zd ZdZd�e�Zedd� �Zdd� Zdd� Z d d
� Z
d d � Z d
S )�NetworkTestCasezhttp://www.pythontest.net/z{}elsewhere/robots.txtc C s@ t �d� t �| j�� tj�| j�| _| j� � W 5 Q R X d S )NZnetwork)
r ZrequiresZtransient_internet�base_urlr r
r r
r rq )�clsr r r �
setUpClassY s
zNetworkTestCase.setUpClassc C s$ d� | j|tj�|�d s dnd�S )Nz{}{}{}rW r- r )r^ ru �os�path�splitext)r ry r r r r ` s
�zNetworkTestCase.urlc C sV | � | jj� | � | jj� | �| j�� d� | � | j�d�� | � | j�d�� d S )Nr rp )r# r � disallow_all� allow_allZ
assertGreater�mtimer7 r8 r' r r r �
test_basice s
zNetworkTestCase.test_basicc C s� | � | j�d| �d��� | �| j�d| j�� | �| j�d| �d��� | �| j�d| �d��� | �| j�d| �d��� | � | j�d| j�� d S )Nrp Z elsewhereZNutchZbrianZwebstats)r r r r r# ru r' r r r �test_can_fetchl s z NetworkTestCase.test_can_fetchc C sf t j�| �d��}|�� | �|j� | �|j� | � |�
� d� | � |� d�� | � |�
d�� d S )Nz i-robot.txtr rp )r r
r r rq r r| r# r{ r% r} Z assertIsNoner7 r8 )r r r r r �
test_read_404t s z NetworkTestCase.test_read_404N) r) r* r+ ru r^ r
� classmethodrw r r~ r r� r r r r rt T s
rt �__main__)%r rx rg ZunittestZurllib.robotparserr �testr Z http.serverr r r ZTestCaser, r2 r3 r4 r6 r; r< rE rG rH rI rN rO rP rQ rS rT rU rX r\ ra rt r) �mainr r r r �<module> s> "
)