What do "Relocating dyno to a new server" messages mean?

Issue

You saw a "Relocating dyno to a new server" message in your application logs and aren't sure what it means.

Resolution

That message is logged under a couple of different conditions. The most common condition involves the case where the backend server your dyno is on is experiencing problems. Server health is monitored by our systems and if one is determined to be unhealthy, an automated process will be initiated which replaces the server. When that happens, all dynos on the server are evacuated to another server. In some situations, the conditions on the server are such that dynos being moved may not get the typical SIGTERM signal and will simply get shutdown. This sort of operation also precludes preboots (which require a controlled environment and 4-6 minutes to complete – neither of which is possible in this situation). If it happens to a Scheduler dyno, the job will get interrupted and won't resume again until the next scheduled execution time.

The next most common situation is a rolling restart of our fleet. That doesn't happen too often, but is done occasionally to address security vulnerabilities or other issues that can crop up. When that happens, you'll likely see this error more frequently – particularly if you have a lot of dynos running. This happens because dynos are evacuated to a random server. Near the beginning of the fleet roll, most servers still need to be restarted so the odds are high that the dyno will get moved to a server that still needs to be restarted. As the fleet roll proceeds, it becomes less and less likely for this to happen though. A full fleet roll can take between 1-3 days depending upon the speed we're performing it (which usually correlates with how urgent the required maintenance is).

The error itself isn't really something you can do to prevent on your end and isn't really something to be alarmed about – it's just related to processes that run on our platform as part of day-to-day operations. That said, these sorts of errors do happen more frequently in our shared fleet. On Performance tier dynos, you will see this error much less frequently due to their single tenant architecture, so if this is proving to be an issue for your use case, then the Performance tier might be something worth investigating.