pywb.manager package

Submodules

pywb.manager.aclmanager module

class pywb.manager.aclmanager.ACLManager(r)

Bases: pywb.manager.manager.CollectionsManager

DEFAULT_FILE = 'access-rules.aclj'

SURT_RX = re.compile('([^:.]+[,)])+')

VALID_ACCESS = ('allow', 'block', 'exclude', 'allow_ignore_embargo')

add_excludes(r)

Import old-style excludes, in url-per-line format

Parameters:r (argparse.Namespace) – Parsed result from ArgumentParser

add_rule(r)

Adds a rule the ACL manager

Parameters:r (argparse.Namespace) – The argparse namespace representing the rule to be added Return type:None

find_match(r)

Finds a matching acl rule

Parameters:r (argparse.Namespace) – Parsed result from ArgumentParser Return type:None

classmethod init_parser(parser)

Initializes an argument parser for acl commands

Parameters:parser (argparse.ArgumentParser) – The parser to be initialized Return type:None

is_valid_auto_coll(coll_name)

Returns T/F indicating if the supplied collection name is a valid collection

Parameters:coll_name – The collection name to check Returns:T/F indicating a valid collection Return type:bool

list_rules(r)

Print the acl rules to the stdout

Parameters:r (argparse.Namespace|None) – Not used Return type:None

load_acl(must_exist=True)

Loads the access control list

Parameters:must_exist (bool) – Does the acl file have to exist Returns:T/F indicating load success Return type:bool

print_rule(rule)

Prints the supplied rule to the std out

Parameters:rule (CDXObject) – The rule to be printed Return type:None

process(r)

Process acl command

Parameters:r (argparse.Namespace) – Parsed result from ArgumentParser Return type:None

remove_rule(r)

Removes a rule from the acl file

Parameters:r (argparse.Namespace) – Parsed result from ArgumentParser Return type:None

save_acl(r=None)

Save the contents of the rules as cdxj entries to the access control list file

Parameters:r (argparse.Namespace|None) – Not used Return type:None

to_key(url_or_surt, exact_match=False)

If ‘url_or_surt’ already a SURT, use as is If exact match, add the exact match suffix

Parameters:

• url_or_surt (str) – The url or surt to be converted to an acl key
• exact_match (bool) – Should the exact match suffix be added to key

Return type:

str

validate(log=False, correct=False)

Validates the acl rules returning T/F if the list should be saved

Parameters:

• log (bool) – Should the results of validating be logged to stdout
• correct (bool) – Should invalid results be corrected and saved

Return type:

None

validate_access(access)

Returns true if the supplied access value is valid otherwise terminates the process

Parameters:access (str) – The access value to be validated Returns:True if valid Return type:bool

validate_save(r=None, log=False)

Validates the acl rules and saves the file

Parameters:

• r (argparse.Namespace|None) – Not used
• log (bool) – Should a report be printed to stdout

Return type:

None

pywb.manager.autoindex module

class pywb.manager.autoindex.AutoIndexer(colls_dir=None, interval=30, keep_running=True)

Bases: object

AUTO_INDEX_FILE = 'autoindex.cdxj'

EXT_RX = re.compile('.*\\.w?arc(\\.gz)?$')

check_path()

do_index(files)

is_newer_than(path1, path2, track=False)

run()

start()

stop()

pywb.manager.locmanager module

class pywb.manager.locmanager.LocManager

Bases: object

compile_catalog()

extract_loc(locale, no_csv)

extract_text()

init_catalog(loc)

classmethod init_parser(parser)

Initializes an argument parser for acl commands

Parameters:parser (argparse.ArgumentParser) – The parser to be initialized Return type:None

list_loc()

process(r)

remove_loc(locale)

update_catalog(loc)

update_loc(locale, no_csv)

pywb.manager.manager module

class pywb.manager.manager.CollectionsManager(coll_name, colls_dir=None, must_exist=True)

Bases: object

This utility is designed to simplify the creation and management of web archive collections

It may be used via cmdline to setup and maintain the directory structure expected by pywb

COLLS_DIR = 'collections'

COLL_RX = re.compile('^[\\w][-\\w]*$')

DEF_INDEX_FILE = 'index.cdxj'

add_collection()

add_template(template_name, force=False, ignore=False)

add_warcs(warcs)

change_collection(coll_name)

index_merge(filelist, index_file)

list_colls()

list_templates()

migrate_cdxj(path, force=False)

reindex()

remove_template(template_name, force=False)

set_metadata(namevalue_pairs)

pywb.manager.manager.get_input(msg)

pywb.manager.manager.get_version()

Get version of the pywb

pywb.manager.manager.main(args=None)

pywb.manager.manager.main_wrap_exc()

pywb.manager.migrate module

class pywb.manager.migrate.MigrateCDX(dir_)

Bases: object

convert_to_cdxj()

count_cdx()

iter_cdx_files()

Module contents