HEX
Server: LiteSpeed
System: Linux premium69.web-hosting.com 4.18.0-553.44.1.lve.el8.x86_64 #1 SMP Thu Mar 13 14:29:12 UTC 2025 x86_64
User: swifizcd (1555)
PHP: 8.2.30
Disabled: NONE
Upload Files
File: //opt/cloudlinux/venv/lib64/python3.11/site-packages/ssa/__pycache__/db.cpython-311.pyc
�

�e�i� ��f�ddlZddlZddlZddlZddlmZmZddlmZmZm	Z	m
Z
mZmZm
Z
mZmZddlmZddlmZddlmZddlmZddlmZd	Zed
zZdZe��Zejd��ZGd
�de��Z dd�Z!d�Z"d�Z#d�Z$d�Z%d�Z&dd�Z'd�Z(d�Z)d�Z*ej+defd���Z,dS)�N)�datetime�	timedelta)	�Column�Boolean�DateTime�Integer�String�
create_engine�event�func�text)�declarative_base)�	Inspector)�Session)�close_all_sessions)�
DatabaseErrorz/var/lve/ssa.dbz.old��cleanup_old_datac���eZdZdZdZeed���Zeedd���Z	eedd���Z
eed���Zeed���Zee
d���Zee
d���Zeed���Zeed���Zee
d���Zeed���ej��d�	��Zeed���ej��ej���
��ZdS)�
RequestResultay
    Describes processed request stored in database file.

    E.g.
    {
        "timestamp": "1650008727",
        "url": "http://mydomain.com/index.php",
        "duration": 162077,
        "hitting_limits": false,
        "throttled_time": 0,
        "io_throttled_time": 0,
        "wordpress": true
    }

    Note: created_at, updated_at is saved in local TZ format
    �
scrape_resultT)�primary_keyF)�index�nullable)r)�timezone)�server_defaultr)�onupdaterN)�__name__�
__module__�__qualname__�__doc__�
__tablename__rr�idr	�domain�path�	timestamp�durationr�is_slow_request�hitting_limits�throttled_time�io_throttled_time�	wordpressrr�now�
created_at�
updated_at���=/opt/cloudlinux/venv/lib64/python3.11/site-packages/ssa/db.pyrr'sR�������� $�M�	���T�	*�	*�	*�B�
�V�F�$��
7�
7�
7�F��6�&��u�5�5�5�D���w��/�/�/�I��v�g��.�.�.�H��f�W�u�5�5�5�O��V�G�e�4�4�4�N��V�G�e�4�4�4�N���w��7�7�7����w��/�/�/�I�����$�/�/�/����
�
�RV�W�W�W�J�����$�/�/�/�(�$�(�*�*�U]�UY�U]�U_�U_�`�`�`�J�J�Jr1r�'c��tj��tt���z
}d}	t	|��5}|�t
d��||d���}|j}||z
}|dkr
	ddd��n	ddd��n#1swxYwY�nt�	d|��t|��dS)a
    Removes outdated records from database, saving disk space.
    Deletions are batched to avoid loading the entire database into
    the cgroup's page cache, which causes OOM on large databases.
    Runs VACUUM after deletion to reclaim disk space.
    )�daysrTzrDELETE FROM scrape_result WHERE rowid IN (SELECT rowid FROM scrape_result WHERE created_at < :cutoff LIMIT :batch))�cutoff�batchNzCleanup deleted %d old records)r�todayr�RETENTION_TIME_DAYS�
session_scope�executer
�rowcount�logger�info�vacuum_database)�engine�
batch_size�
n_days_ago�
total_deleted�session�result�deleteds       r2rrMs5����!�!�I�3F�$G�$G�$G�G�J��M�	�
�6�
"�
"�	�g��_�_�T�\�&�&�%�z�:�:�<�<�F��o�G��W�$�M��!�|�|��	�	�	�	�	�	�	��	�	�	�	�	�	�	�	�	�	�	����	�	�	�	�	��K�K�0�-�@�@�@��F�����s�:B�B�Bc��|���}|j}	d|_|���}tj�tj�t����}tj|tj	��r7|�
d|�d���t�d|����nt�
d|�d���t�d��|�
d��t�d	��|���nA#t$r4}t�d
t#|������Yd}~nd}~wwxYw||_|���dS#||_|���wxYw)a
    Run VACUUM command to reclaim disk space after deletions.
    VACUUM must be run outside of a transaction, so we use raw connection.

    SQLite VACUUM creates a full temporary copy of the database.
    By default, the temp file goes to /var/tmp or /tmp which may be
    tmpfs (RAM-backed). On systems with MemoryMax cgroup limits
    (like ssa-agent.service), this causes OOM kills for large databases.
    Setting temp_store_directory to the database's own directory ensures
    the temp file is written to disk, not RAM.
    NzPRAGMA temp_store_directory = '�'z$Set SQLite temp_store_directory to: z
Directory z0 not writable. VACUUM will use default /var/tmp.zStarting VACUUM operation...�VACUUMzVACUUM completed successfully.zVACUUM failed: )�raw_connection�isolation_level�cursor�osr%�dirname�abspath�SSA_DB�access�W_OKr;r=r>�warning�close�	Exception�error�str)r@�raw_conn�original_isolation_levelrL�db_dir�es      r2r?r?ds����$�$�&�&�H�'�7���#'�� ����"�"�����������!8�!8�9�9��
�9�V�R�W�%�%�	b��N�N�F�V�F�F�F�G�G�G��K�K�G�v�G�G�H�H�H�H��N�N�`��`�`�`�a�a�a����2�3�3�3����x� � � ����4�5�5�5����������1�1�1����/�s�1�v�v�/�/�0�0�0�0�0�0�0�0�����1����$<�� ����������$<�� ����������s0�D-E�
F)�
F	�*F�?F)�F	�	F)�)Gc�f�t|��s!tj�|��dSdS�N)�
is_db_present�Base�metadata�
create_all)r@s r2�create_db_if_not_existrb�s8���� � �)��
� � ��(�(�(�(�(�)�)r1c���tj�t��sdSt	j|��}d�|���D��}t|��dkS)NFc��g|]}|��Sr0r0)�.0�tables  r2�
<listcomp>z!is_db_present.<locals>.<listcomp>�s��
G�
G�
G��e�
G�
G�
Gr1r)rMr%�isfilerPr�from_engine�get_table_names�len)r@�database_inspection�tabless   r2r^r^�s^��
�7�>�>�&�!�!���u�#�/��7�7��
G�
G�!4�!D�!D�!F�!F�
G�
G�
G�F��v�;�;��?�r1c�0�|�d��dS)NzPRAGMA journal_mode = WAL)r;)�	dbapi_con�
con_records  r2�setup_wal_moderq�s��
���1�2�2�2�2�2r1c���dt���|r��d���fd�}td|d���}tj|dt��t|��|S)Nzfile:z?mode=roc�0��tj�d���S)NT)�uri)�sqlite3�connect)�connection_strings�r2�<lambda>z!_setup_database.<locals>.<lambda>�s���g�o�&7�T�B�B�B�r1�sqlite:////F)�creator�echorv)rPr
r�listenrqrb)�readonlyrzr@rws   @r2�_setup_databaser~�sz���(��(�(���;�0�:�:�:��B�B�B�B�G�
��w�U����F�
�L���N�3�3�3��6�"�"�"��Mr1Fc� �t|��Sr])r~)r}s r2�setup_databaser��s���8�$�$�$r1c���tj�t��r-t	��tjtt��t��}|���tdt����}	t|��5}t|��5}|�t���
��t||��ddd��n#1swxYwYddd��n#1swxYwYn#t$rYnwxYwttdztdzfD]5}tj�|��rtj|���6dS)z�
    Restore database by establish connections to old and new databases,
    merge data to new one if possible and delete old one.
    ryNz-walz-shm)rMr%�existsrPr�replace�
OLD_SSA_DBr��disposer
r:�queryr�first�!merge_unharmed_data_from_databaser�remove)r@�
new_engine�
old_engine�session_old�session_newr%s      r2�restore_databaser��s���

�w�~�~�f���'�����
�
�6�:�&�&�&��!�!�J�
�N�N�����9�Z�9�9�:�:�J�
�
�:�
&�
&�	H�+�}�Z�7P�7P�	H�T_����m�,�,�2�2�4�4�4�-�k�;�G�G�G�	H�	H�	H�	H�	H�	H�	H�	H�	H�	H�	H����	H�	H�	H�	H�	H�	H�	H�	H�	H�	H�	H�	H�	H�	H�	H����	H�	H�	H�	H����
�
�
���
�����V�f�_�f�v�o�>����
�7�>�>�$���	��I�d�O�O�O���sZ�D�D�+=C4�(D�4C8	�8D�;C8	�<D�?D�D�D�D�D�
D$�#D$c��d}d}	|�t���|���|��}	|���}|sdS|D]}|�|���nv#t$rit|��D]V}	|�||z�����}|sn&|�|���G#t$rY�SwxYwYnwxYw|�	��||z
}��)z_
    Scrape all unharmed records from malformed database and merge them into new database.
    r�
TN)
r�r�offset�limit�all�merger�ranger��commit)	r�r�r�rAr��records_to_save�record�pos_in_batch�record_to_saves	         r2r�r��sN���F��J���!�!�-�0�0�7�7��?�?�E�E�j�Q�Q��	�#�i�i�k�k�O�"�
���)�
*�
*���!�!�&�)�)�)�)�
*���	�	�	� %�j� 1� 1�
�
���%*�\�\�&�<�2G�%H�%H�%N�%N�%P�%P�N�)�����%�%�n�5�5�5�5��$�����D�������	����	�������*���%sH�A:�A:�:C-�,C�C-�C�C-�
C'�$C-�&C'�'C-�,C-c���tj�t��r=tj�t��rtjt��	t
|��5}|�td����}|�	��}|dddkcddd��S#1swxYwYdS#t$rYdSwxYw)z�
    Try integrity check of database file to see if it is malformed.
    If database unable to execute it, will also count as malformed.
    zPRAGMA integrity_checkr�okNT)rMr%r�r�rPr�r:r;r
�fetchallr)r@�dbrE�errorss    r2�is_malformed_databaser��s��

�w�~�~�j�!�!��b�g�n�n�V�&<�&<��
�	�*�����
�6�
"�
"�	(�b��Z�Z��%=� >� >�?�?�F��_�_�&�&�F��!�9�Q�<�4�'�	(�	(�	(�	(�	(�	(�	(�	(�	(�	(�	(�	(����	(�	(�	(�	(�	(�	(�������t�t����s7�#C�2AC�:C�C�C�C�C�
C"�!C"�returnc#��K�t|���}	|V�|���n#|����xYw	|���dS#|���wxYw)zF
    Provide a transactional scope around a series of operations.
    )�bindN)rr��rollbackrT)r@rDs  r2r:r:�s�����
�6�"�"�"�G���
�
�
����������������
����	�
	�
�
��������
�
�������s�-�A�A�A�A4)r3)F)-rM�
contextlibru�loggingrr�
sqlalchemyrrrrr	r
rrr
�sqlalchemy.ext.declarativer�sqlalchemy.engine.reflectionr�sqlalchemy.ormr�sqlalchemy.orm.sessionr�sqlalchemy.excrrPr�r9r_�	getLoggerr=rrr?rbr^rqr~r�r�r�r��contextmanagerr:r0r1r2�<module>r�sN��
�	�	�	�������������(�(�(�(�(�(�(�(�����������������������8�7�7�7�7�7�2�2�2�2�2�2�"�"�"�"�"�"�5�5�5�5�5�5�(�(�(�(�(�(�	��
�f�_�
��������	��	�-�	.�	.��#a�#a�#a�#a�#a�D�#a�#a�#a�L����.&�&�&�T)�)�)�
���3�3�3�
�
�
�%�%�%�%����8���6
�
�
� ���W��������r1