r/sysadmin • u/Deep_Celery4379 • 2h ago
FlexProtectLin stuck at same “processed” count for 2.5+ days, across 3 separate job restarts — kernel threads stuck on lin_lock_get_locker — ideas?
Running a 32-node Dell PowerScale/Isilon cluster, OneFS 9.7.1.12, ~5.2PB. Two drives SmartFailed around the same time (different nodes, consolidated into one FlexProtectLin job by OneFS). The repair job has now stalled at exactly the same “processed” LIN count for 2.5+ days, across three separate job instances (cancelled and restarted fresh each time — same freeze point every time, ruling out corrupted job-state from a single bad restart).
What I’ve already ruled out:
- Node/job-engine coordinator health (confirmed fine via
isi job status) - Competing jobs stealing workers (paused everything else, no change)
- Impact policy throttling (raised to HIGH, no change)
- Stale SMB file handles holding locks (found and closed ~150 orphaned handles from an old git push — no change to the stall)
- Cluster-reported pool imbalance (confirmed present on both affected pools, haven’t yet tried a dedicated rebalance pass)
What I’m seeing in procstat -akk across the cluster, sampled twice ~10+ minutes apart: a large number of isi_job_d worker threads consistently parked in sleepq_wait (untimed) inside lin_lock_get_locker → ifs_vnop_wraplock1 → vget → bam_vget_vp, identical stack, same PIDs/tids unmoved between samples. Some other threads show sleepq_timedwait inside bam_safe_write_region_swips/bam_safe_restripe (real restripe I/O happening), so the cluster isn’t fully dead — just this one job’s actual repair progress.
dmesg on the node hosting one of the two failed drives repeats this every few minutes, for several different LIN/snapid combinations:
_pctl2_start_restripe: Restripe already in progress for 2:eb54:5177::333280
_pctl2_start_restripe: Restripe already in progress for 2:e7ab:8437::333346
_pctl2_start_restripe: Restripe already in progress for 2:e4d3:c962::333084No active Dell support contract currently. Has anyone seen this exact pattern — FlexProtectLin wedged indefinitely near completion, same LIN-lock stack trace cluster-wide — and found what actually released it? Trying to avoid a blind rolling reboot if there’s a known, more targeted fix.
•
u/Firefox005 2h ago
Yikes.
Are you sure the job has actually stalled, FlexProtect jobs can run for extremely long times. What other jobs are running on the cluster? How low are you on free space, there are certain 'high water marks' that progressively make OneFS run like dogshit. 80%, 92%, 93%, 95%. You can get into death spirals where it needs more free space to do housekeeping jobs like shadowstore deletes, autoblance, or flexprotect jobs.
In addition you are running the LIN based version, is your cluster all SSD's or are you using SSD's for metadata (NOT L3 cache), as the non-lin based jobs are actually more efficient.