Ecosyste.ms: Repos
An open API service providing repository metadata for many open source software ecosystems.
Package Usage: go: github.com/highlight-run/workerpool
Package workerpool queues work to a limited number of goroutines.
The purpose of the worker pool is to limit the concurrency of tasks
executed by the workers. This is useful when performing tasks that require
sufficient resources (CPU, memory, etc.), and running too many tasks at the
same time would exhaust resources.
A task is a function submitted to the worker pool for execution. Submitting
tasks to this worker pool will not block, regardless of the number of tasks.
Incoming tasks are immediately dispatched to an available
worker. If no worker is immediately available, or there are already tasks
waiting for an available worker, then the task is put on a waiting queue to
wait for an available worker.
The intent of the worker pool is to limit the concurrency of task execution,
not limit the number of tasks queued to be executed. Therefore, this unbounded
input of tasks is acceptable as the tasks cannot be discarded. If the number
of inbound tasks is too many to even queue for pending processing, then the
solution is outside the scope of workerpool, and should be solved by
distributing load over multiple systems, and/or storing input for pending
processing in intermediate storage such as a database, file system, distributed
message queue, etc.
This worker pool uses a single dispatcher goroutine to read tasks from the
input task queue and dispatch them to worker goroutines. This allows for a
small input channel, and lets the dispatcher queue as many tasks as are
submitted when there are no available workers. Additionally, the dispatcher
can adjust the number of workers as appropriate for the work load, without
having to utilize locked counters and checks incurred on task submission.
When no tasks have been submitted for a period of time, a worker is removed by
the dispatcher. This is done until there are no more workers to remove. The
minimum number of workers is always zero, because the time to start new workers
is insignificant.
It is advisable to use different worker pools for tasks that are bound by
different resources, or that have different resource use patterns. For
example, tasks that use X Mb of memory may need different concurrency limits
than tasks that use Y Mb of memory.
When there are no available workers to handle incoming tasks, the tasks are put
on a waiting queue, in this implementation. In implementations mentioned in
the credits below, these tasks were passed to goroutines. Using a queue is
faster and has less memory overhead than creating a separate goroutine for each
waiting task, allowing a much higher number of waiting tasks. Also, using a
waiting queue ensures that tasks are given to workers in the order the tasks
were received.
This implementation builds on ideas from the following:
http://marcio.io/2015/07/handling-1-million-requests-per-minute-with-golang
http://nesv.github.io/golang/2014/02/25/worker-queues-in-go.html
6 versions
Latest release: over 2 years ago
View more package details: https://packages.ecosyste.ms/registries/proxy.golang.org/packages/github.com/highlight-run/workerpool
View more repository details: https://repos.ecosyste.ms/hosts/GitHub/repositories/highlight%2Fworkerpool
Dependent Repos 1
highlight/highlight
highlight.io: The open source, full-stack monitoring platform. Error monitoring, session replay, logging, distributed tracing, and more.- v1.3.0 backend/go.mod
- v1.3.0 backend/go.sum
Size: 477 MB - Last synced: 4 days ago - Pushed: 4 days ago
mysticaltech/highlight Fork of highlight/highlight
highlight.io: The open source, full-stack monitoring platform. Error monitoring, session replay, logging and more.- v1.3.0 backend/go.mod
- v1.3.0 backend/go.sum
Size: 120 MB - Last synced: 10 months ago - Pushed: over 1 year ago
bejewel-kyoungmin/highlight Fork of highlight/highlight
highlight.io: The open source, full-stack monitoring platform. Error monitoring, session replay, logging and more.- v1.3.0 backend/go.mod
- v1.3.0 backend/go.sum
Size: 121 MB - Last synced: 2 months ago - Pushed: about 1 year ago
euvl/highlight Fork of highlight/highlight
highlight.io: The open source, full-stack monitoring platform. Error monitoring, session replay, logging and more.- v1.3.0 backend/go.mod
- v1.3.0 backend/go.sum
Size: 323 MB - Last synced: about 1 month ago - Pushed: 10 months ago
phultquist/highlight Fork of highlight/highlight
highlight.io: The open source, full-stack monitoring platform. Error monitoring, session replay, logging and more.- v1.3.0 backend/go.mod
- v1.3.0 backend/go.sum
Size: 327 MB - Last synced: 9 months ago - Pushed: 9 months ago
lalitkumawat1m/highlight Fork of highlight/highlight
highlight.io: The open source, full-stack monitoring platform. Error monitoring, session replay, logging and more.- v1.3.0 backend/go.mod
- v1.3.0 backend/go.sum
Size: 375 MB - Last synced: about 1 month ago - Pushed: 4 months ago
KaranJagtiani/highlight Fork of highlight/highlight
highlight.io: The open source, full-stack monitoring platform. Error monitoring, session replay, logging and more.- v1.3.0 backend/go.mod
- v1.3.0 backend/go.sum
Size: 274 MB - Last synced: 8 months ago - Pushed: 9 months ago
Checkfer-Limited/highlight Fork of highlight/highlight
highlight.io: The open source, full-stack monitoring platform. Error monitoring, session replay, logging and more.- v1.3.0 backend/go.mod
- v1.3.0 backend/go.sum
Size: 273 MB - Last synced: about 1 month ago - Pushed: 9 months ago
AyushAgrawal-A2/highlight Fork of highlight/highlight
highlight.io: The open source, full-stack monitoring platform. Error monitoring, session replay, logging and more.- v1.3.0 backend/go.mod
- v1.3.0 backend/go.sum
Size: 279 MB - Last synced: about 1 month ago - Pushed: 8 months ago
yugasun/highlight Fork of highlight/highlight
highlight.io: The open source, full-stack monitoring platform. Error monitoring, session replay, logging and more.- v1.3.0 backend/go.mod
- v1.3.0 backend/go.sum
Size: 300 MB - Last synced: about 2 months ago - Pushed: 10 months ago
ayusham001/highlight Fork of highlight/highlight
highlight.io: The open source, full-stack monitoring platform. Error monitoring, session replay, logging and more.- v1.3.0 backend/go.mod
- v1.3.0 backend/go.sum
Size: 274 MB - Last synced: about 1 month ago - Pushed: 9 months ago
MAYANK-TRIPATH/highlight Fork of highlight/highlight
highlight.io: The open source, full-stack monitoring platform. Error monitoring, session replay, logging and more.- v1.3.0 backend/go.mod
- v1.3.0 backend/go.sum
Size: 274 MB - Last synced: about 2 months ago - Pushed: 9 months ago
neo773/highlight Fork of highlight/highlight
highlight.io: The open source, full-stack monitoring platform. Error monitoring, session replay, logging and more.- v1.3.0 backend/go.mod
- v1.3.0 backend/go.sum
Size: 384 MB - Last synced: about 1 month ago - Pushed: 3 months ago
arjunbharti/highlight Fork of highlight/highlight
highlight.io: The open source, full-stack monitoring platform. Error monitoring, session replay, logging and more.- v1.3.0 backend/go.mod
- v1.3.0 backend/go.sum
Size: 274 MB - Last synced: about 1 month ago - Pushed: 9 months ago
vikasrathod4299/highlight Fork of highlight/highlight
highlight.io: The open source, full-stack monitoring platform. Error monitoring, session replay, logging and more.- v1.3.0 backend/go.mod
- v1.3.0 backend/go.sum
Size: 274 MB - Last synced: about 1 month ago - Pushed: 9 months ago
hirajanwin/highlight Fork of highlight/highlight
highlight.io: The open source, full-stack monitoring platform. Error monitoring, session replay, logging and more.- v1.3.0 backend/go.mod
- v1.3.0 backend/go.sum
Size: 118 MB - Last synced: about 2 months ago - Pushed: over 1 year ago
lexciobotariu/highlight Fork of highlight/highlight
highlight.io: The open source, full-stack monitoring platform. Error monitoring, session replay, logging and more.- v1.3.0 backend/go.mod
- v1.3.0 backend/go.sum
Size: 267 MB - Last synced: about 1 month ago - Pushed: 12 months ago
akashsiddamshetty/highlight Fork of highlight/highlight
highlight.io: The open source, full-stack monitoring platform. Error monitoring, session replay, logging and more.- v1.3.0 backend/go.mod
- v1.3.0 backend/go.sum
Size: 279 MB - Last synced: about 1 month ago - Pushed: 8 months ago
piyush-debuide/highlight Fork of highlight/highlight
highlight.io: The open source, full-stack monitoring platform. Error monitoring, session replay, logging and more.- v1.3.0 backend/go.mod
- v1.3.0 backend/go.sum
Size: 332 MB - Last synced: about 1 month ago - Pushed: 8 months ago
Vinz000/highlight Fork of highlight/highlight
highlight.io: The open source, full-stack monitoring platform. Error monitoring, session replay, logging and more.- v1.3.0 backend/go.mod
- v1.3.0 backend/go.sum
Size: 364 MB - Last synced: about 1 month ago - Pushed: 4 months ago