Update 21-Jan-2015: Quota reporting issue has been resolved.
Update 12-Jan-2015 12:08 EST: The downtime is complete. Space is being recalculated and must complete before quotas are re-enabled.
Update 12-Jan-2015 10:49 EST: The following filesystems will be unmounted and then re-mounted at 12:00EST today:
- /scratch
- /scratch2
- /home2
- /data2
Update 08-Jan-2015 12:51 EST: The quota remediation steps will be executed at 9:00AM Fri 09-Jan-2015. Please note that this work may affect running jobs. More details to follow.
Update 08-Jan-2015 10:47 EST: An action plan has been formulated by Hitachi Support. The plan is being reviewed and will be executed soon.
Update 03-Jan-2015 19:56 EST: The quotas have been temporarily removed until the next filesystem check step is complete. You should not experience any false “Disk quota exceeded” messages in the near term.
Update: The quota on Virtual Volume “/scratch2″ was increased to 220TB in an attempt to compensate for the quote mis-reporting. This is nearly double the size of the underlying filesystem. In the interim, Hitachi engineers has us execute a “checkfs” on the filesystem. By our estimates, it should complete in ~24 hours. Unfortunately, the quota limit was reached and we cannot increase it until the checkfs completes. At this point, we will likely delete all quotas on /scratch2 until the issue is resolved.
Hitachi has two potential fixes for the quota mis-reporting issue. One of them will require unmounting the underlying filesystem. We will report back as soon as we know if that course of action is recommended.
We are currently battling a quota issue on Louise’s Hitachi storage. For several users, /scratch2 is reporting much higher usage than actual. If you experience disk quota issues in /scratch2 and are under quota, please contact hpc@yale.edu and we will temporarily increase your quote so processing can continue.
We are actively working with Hitachi to resolve this issue.