Building Beacon resilience

Beacon users will, regrettably, be well aware of the issues we are currently experiencing with Beacon. Users are occasionally being confronted with a ‘502’ error screen. This means that the Beacon server has not properly responded to your request.

We have discovered that this is being driven by an overwhelming increase in the number of ‘events’ sent to the Beacon server by our email delivery agent as part of the email sending process. When you send an email through Beacon, it is processed by our delivery agent and this agent returns information to Beacon about the delivery of your email.

When the volume of events exceeds a certain threshold, the system locks up. This can prevent users from carrying out tasks (which manifests as a 502 error) and may result in the email delivery status not being correctly updated – despite a successful delivery.

This is in all likelihood due to the unprecedented increase in the amount of use Beacon is seeing – from both existing and new users. Over the last year, the number of Beacon user u3as has increased by over 15 per cent. However, the number of emails handled in January this year as compared to January last year, for example, was over 59 per cent greater.

Please be assured that we are continuing to work to resolve these problems as soon as we can.

So far, we have:

  • Increased the computing resource available to Beacon
  • Implemented enhanced error logging to help us diagnose current and future issues
  • Established a plan of action to resolve our current issues
  • Provisioned a new server to handle email acknowledgement processes
  • Started work to implement a queue system for email acknowledgement events
Our next steps are to:
  • Reroute email acknowledgement events to the new, dedicated server
  • Complete the work to implement the queue system
  • Review the hosting infrastructure to improve the scalability and capacity of the system

We expect to make further progress against these steps next week and will continue to keep users informed via the Beacon service notifications in the Beacon Help Centre.

While we recognise and regret the disruption you may be facing, we are thankful for your patience while we work to resolve the underlying issues.

We are confident that once these tasks have been completed, Beacon will be a stronger and more resilient system, capable of supporting all our existing users and many more into the future.

We are finalising plans to hold an event for the Beacon community once the present system issues are resolved. In the session, we will present and discuss both the work so far this year and the future pipeline of developments. Watch this space for more details.