github.com/prefecthq/prefect ↗

About prefecthq/prefect

Prefect is a Python-based workflow orchestration framework designed to transform scripts into production-ready data pipelines. It provides developers with simple decorators for defining tasks and flows, enabling them to build resilient, dynamic pipelines with built-in support for scheduling, caching, retries, event-based automation, and dependency management. The framework handles complex branching logic and allows workflows to react to and recover from unexpected changes.

The project targets data teams and engineers who need to automate data processes with confidence. Workflows can be monitored through a self-hosted Prefect server instance with a local UI or through Prefect Cloud, a managed dashboard service. Users can run workflows on schedules, trigger them manually via UI or CLI, or execute them in response to events. Prefect requires Python 3.10 or later and integrates with existing tools through a growing ecosystem of integrations.

Prefect Cloud is the commercial offering, processing over 200 million data tasks monthly for organizations ranging from Fortune 50 companies like Progressive Insurance to startups like Cash App. The project maintains an active open-source community of over 25,000 practitioners and provides multiple engagement channels including documentation, Slack community, contribution guidelines, integration development pathways, and a development blog.