| TYOC035 and TYOC033 update:
These are being rebooted now in 10~ minutes, and will have two disks taken out of the disk pool and VMs rebooted. The XPG disks are receiving another error. They will be replaced by Samsung Pro or WD or equivalent. We don’t suspect anyone to have data loss as it appears the disks failed relatively quickly. Once again, we’re throwing all these disks out of the servers we’re sending to the datacenters so hopefully we won’t have further issues with them.
So to confirm, this is not the same issue as before, it appears it is a new issue related to the disks failing. If anyone does have important data on it that they need to retrieve, the disk will be kept and sent back to us so we can likely still recover data, just open a ticket.
This means that we just lost some space in Tokyo. We’re sending more servers this week, and we should still get close to all deployments as I’m setting up another server now and waiting for IPMI reset by the DC on another. I’m checking with the DC if they have the other disks we sent on-hand so we can put the new disks in these new nodes and avoid further problems but they may be at an office for storage away from the datacenter and it’s the weekend. We’ll see. (Later on we’ll streamline this process.)
Everyone "offline" on these will be recreated on another node. |
