Back to overview
Degraded

Exercises repository updates can be slow

Aug 14 at 10:00am CEST
Affected services
dodona.be

Resolved
Aug 22 at 09:12am CEST

The problem has been resolved. Git changes should now be processed as before.

Updated
Aug 21 at 06:18pm CEST

We have a final solution in place and are now manually processing all repositories to make sure we didn't miss any exercise updates. After that's done, the webhooks should operate as normal.

Updated
Aug 18 at 02:37pm CEST

We are working on a solution where a dedicated machine with low latency storage handles all git operations and syncs the result to a network drive used by our other machines. Initial benchmarks indicate a 700x performance improvement over the current situation.

Updated
Aug 14 at 09:34pm CEST

We finalized a pull of all repositories, but noticed that triggering the webhook large exercise repositories is still slow. We hope to have a solution in place early next week.

Created
Aug 14 at 10:00am CEST

After the migration, there was an issue where triggering the webhooks for exercise repository updates failed. This has since been resolved.

As a side effect of the migration, the first time the webhook for a repository is triggered, this can take a (very) long time. Subsequent webhook triggers should be acceptably fast. We are now manually triggering the webhooks for all repositories to minimize user impact.