s3workers package

Submodules

s3workers.cli module

s3workers.jobs module

class s3workers.jobs.Job

Bases: object

run(*args, **kwargs)

stop()

class s3workers.jobs.S3ListJob(bucket, prefix, selector, key_handler, progress)

Bases: s3workers.jobs.Job

Iterate through S3 objects invoking a callback for each

Parameters:

• bucket – the S3 bucket manager
• prefix – the key prefix to use when listing from the S3 bucket
• selector – optional callback to be evaluated for when an object is “interesting”
• key_handler – the callback to be invoked for each selected object (or all if no selector provided)
• progress – the callback to be invoked for reporting progress through the listing

s3workers.manager module

class s3workers.manager.Manager(worker_count, stop_signals=[2, 15, 13], listen_for_unhandled_exceptions=True)

Bases: object

Manage several worker threads and their shared job work queue.

Parameters:

• worker_count – number of worker threads to use (i.e. concurrency)
• stop_signals – list of signals to act on for automatically stopping workers
• listen_For_unhandled_exceptions – when enabled, automatically stop workers when an exception is thrown and not processed (will call orignal handler)

add_work(job)

Add a new job into the shared work queue.

start_workers()

Tell workers to start listening and handling jobs posted to the shared work queue.

stop_workers(*_ignored)

Immediately request that all workers stop pulling jobs off the shared work queue and stop themselves.

Workers will finish jobs in progress but will stop accepting new ones and with terminate themselves.

wait_for_workers(join_timeout=1)

Wait for workers to finish all outstanding jobs in the shared work queue.

Should be called after all work has been submitted and the caller is ready to wait for all workers to gracefully stop.

s3workers.progress module

class s3workers.progress.S3KeyProgress

Bases: s3workers.progress.SimpleProgress

finish()

report(final=False)

write(msg, *args)

class s3workers.progress.SimpleProgress

Bases: object

finish()

report(msg, *args)

write(msg, *args)

s3workers.reducer module

class s3workers.reducer.Reducer(reduction_string, accumulation_string='0')

Bases: object

Execute reduction logic against an accumulator value.

This abstraction allows for a caller to perform a reduction of many values into one or a smaller set. For example, to provide summation of values, collect interesting values into an array, or produce grouped summations in a dictionary.

Parameters:

• reduction_string – exec’d to perform accumulation logic (must set the accumulator during each call or nothing will aggregate)
• accumulation_string – eval’d to an initial value to accumulate the reduction results

reduce(name, size, md5, last_modified)

s3workers.work_queue module

class s3workers.work_queue.WorkQueue(*args)

Bases: Queue.Queue

Simple wrapper to also provide ability to indicate when no more work is expected.

all_jobs_submitted()

Indicate that no more work is expected on this queue.

is_done()

Determine if there is more work expected on this queue.

s3workers.worker module

class s3workers.worker.Worker(work_queue)

Bases: threading.Thread

Simple thread to continuously pull jobs off a work queue until told to stop or that no more jobs will be submitted.

Parameters:work_queue – the job queue to query for work

run()

Run until a stop is requested or there is no more work expected.

stop()

Finish any job currently in progress and then terminate.

Module contents