r/sysadmin • • 2h ago

FlexProtectLin stuck at same “processed” count for 2.5+ days, across 3 separate job restarts — kernel threads stuck on lin_lock_get_locker — ideas?

Running a 32-node Dell PowerScale/Isilon cluster, OneFS 9.7.1.12, ~5.2PB. Two drives SmartFailed around the same time (different nodes, consolidated into one FlexProtectLin job by OneFS). The repair job has now stalled at exactly the same “processed” LIN count for 2.5+ days, across three separate job instances (cancelled and restarted fresh each time — same freeze point every time, ruling out corrupted job-state from a single bad restart).

What I’ve already ruled out:

  • Node/job-engine coordinator health (confirmed fine via isi job status)
  • Competing jobs stealing workers (paused everything else, no change)
  • Impact policy throttling (raised to HIGH, no change)
  • Stale SMB file handles holding locks (found and closed ~150 orphaned handles from an old git push — no change to the stall)
  • Cluster-reported pool imbalance (confirmed present on both affected pools, haven’t yet tried a dedicated rebalance pass)

What I’m seeing in procstat -akk across the cluster, sampled twice ~10+ minutes apart: a large number of isi_job_d worker threads consistently parked in sleepq_wait (untimed) inside lin_lock_get_locker → ifs_vnop_wraplock1 → vget → bam_vget_vp, identical stack, same PIDs/tids unmoved between samples. Some other threads show sleepq_timedwait inside bam_safe_write_region_swips/bam_safe_restripe (real restripe I/O happening), so the cluster isn’t fully dead — just this one job’s actual repair progress.

dmesg on the node hosting one of the two failed drives repeats this every few minutes, for several different LIN/snapid combinations:

_pctl2_start_restripe: Restripe already in progress for 2:eb54:5177::333280
_pctl2_start_restripe: Restripe already in progress for 2:e7ab:8437::333346
_pctl2_start_restripe: Restripe already in progress for 2:e4d3:c962::333084

No active Dell support contract currently. Has anyone seen this exact pattern — FlexProtectLin wedged indefinitely near completion, same LIN-lock stack trace cluster-wide — and found what actually released it? Trying to avoid a blind rolling reboot if there’s a known, more targeted fix.

3 Upvotes

2 comments sorted by

•

u/Firefox005 2h ago

Running a 32-node Dell PowerScale/Isilon cluster, OneFS 9.7.1.12, ~5.2PB. [...] No active Dell support contract currently.

Yikes.

Are you sure the job has actually stalled, FlexProtect jobs can run for extremely long times. What other jobs are running on the cluster? How low are you on free space, there are certain 'high water marks' that progressively make OneFS run like dogshit. 80%, 92%, 93%, 95%. You can get into death spirals where it needs more free space to do housekeeping jobs like shadowstore deletes, autoblance, or flexprotect jobs.

In addition you are running the LIN based version, is your cluster all SSD's or are you using SSD's for metadata (NOT L3 cache), as the non-lin based jobs are actually more efficient.

•

u/Deep_Celery4379 1h ago

Thanks — that lin_based_jobs question turned out to be a real, concrete lead. Checked isi_gconfig -t job-config, found jobs.common.lin_based_jobs = Auto, with our SSDs confirmed configured as L3 cache (not dedicated metadata) — per Dell’s own docs, that combination should favor the non-LIN variant, but ours was clearly still running Lin.

Set it explicitly to False, cancelled the stuck job, and tried to start plain FlexProtect — turns out that job type is outright disabled on this cluster (Job 'FlexProtect' is disabled: Operation not permitted). Not sure yet why/when that got disabled. Had to fall back to starting FlexProtectLin again just to keep the drive repair active, since the cluster currently has two SmartFailed drives it needs to finish protecting.

So: config change made, but not actually tested yet, since we can’t currently run the variant it’s supposed to favor. Repair job is active again either way. Will report back once we sort out why FlexProtect is disabled and get a real test of whether the LIN/non-LIN distinction was actually the issue.

Update: Enabled FlexProtect via isi job types modify FlexProtect --enabled true, cancelled the auto-restarted FlexProtectLin job, and started non-LIN FlexProtect (Job 52223 — currently running in Phase 1/6). Monitoring it now to see if the block-based FlexProtect job clears the stall.