ZFS – HDD performance strongly varies in 3-way-mirror among vendors

Question:

Prerequisite

We have a FreeNAS System with a Pool consiting of 5x”3-way-mirror”. The disks per mirror are from different vendors but with the same “on-paper” performace-characteristics.

Pool-Layout:

Tank0 ?mirror-0 ?Western Digital WDC WD20SPZX (2TB, 128MB Cache, 5400 RPM) ?Seagate ST2000LM015 (2TB, 128MB Cache, 5400 RPM) ?Toshiba HDWL120 (2TB, 128MB Cache, 5400 RPM) ?mirror-1 ?Western Digital WDC WD20SPZX (2TB, 128MB Cache, 5400 RPM) ?Seagate ST2000LM015 (2TB, 128MB Cache, 5400 RPM) ?Toshiba HDWL120 (2TB, 128MB Cache, 5400 RPM) ?mirror-2 ?Western Digital WDC WD20SPZX (2TB, 128MB Cache, 5400 RPM) ?Seagate ST2000LM015 (2TB, 128MB Cache, 5400 RPM) ?Toshiba HDWL120 (2TB, 128MB Cache, 5400 RPM) ?mirror-3 ?Western Digital WDC WD20SPZX (2TB, 128MB Cache, 5400 RPM) ?Seagate ST2000LM015 (2TB, 128MB Cache, 5400 RPM) ?Toshiba HDWL120 (2TB, 128MB Cache, 5400 RPM) ?mirror-4 ?Western Digital WDC WD20SPZX (2TB, 128MB Cache, 5400 RPM) ?Seagate ST2000LM015 (2TB, 128MB Cache, 5400 RPM) ?Toshiba HDWL120 (2TB, 128MB Cache, 5400 RPM) ?logs ?mirror-5 ?Seagate ST200FM0002 (200GB SAS SSD) ?Seagate ST200FM0002 (200GB SAS SSD) ?cache ?Samsung 850 PRO (512GB SATA SSD)

The 3 different drive vendors per mirror where chosen to minimize the risk of failing specific models of drives. WD and Seagate drives were inserted at creation time of the mirror, Toshiba drives have been added later to form a 3-way-mirror.

All drives are attached on a backplane of a HP SE326M1-Server. The backplane itself is atached to a HP H220 HBA flashed with IT-Mode Firmware.

ZFS is able to interact with each drive directly without any raid-cards.

Smart-Values of Drives

Western Digital

=== START OF INFORMATION SECTION ===Device Model: ?WDC WD20SPZX-00CRAT0Serial Number: ?WD-WXA1E17A5SPCLU WWN Device Id: 5 0014ee 65db69aedFirmware Version: 01.01A01User Capacity: ?2,000,398,934,016 bytes [2.00 TB]Sector Sizes: ?512 bytes logical, 4096 bytes physicalRotation Rate: ?5400 rpmForm Factor: ?2.5 inchesDevice is: ?Not in smartctl database [for details use: -P showall]ATA Version is: ?ACS-3 T13/2161-D revision 5SATA Version is: ?SATA 3.1, 6.0 Gb/s (current: 3.0 Gb/s)Local Time is: ?Wed Apr 24 13:04:40 2019 CESTSMART support is: Available – device has SMART capability.SMART support is: Enabled=== START OF READ SMART DATA SECTION ===SMART overall-health self-assessment test result: PASSEDGeneral SMART Values:Offline data collection status: ?(0x00) Offline data collection activity ???was never started. ???Auto Offline Data Collection: Disabled.Self-test execution status: ?( ?0) The previous self-test routine completed ???without error or no self-test has ever ???been run.Total time to complete Offlinedata collection: ?(21120) seconds.Offline data collectioncapabilities: ??(0x71) SMART execute Offline immediate. ???No Auto Offline data collection support. ???Suspend Offline collection upon new ???command. ???No Offline surface scan supported. ???Self-test supported. ???Conveyance Self-test supported. ???Selective Self-test supported.SMART capabilities: ?(0x0003) Saves SMART data before entering ???power-saving mode. ???Supports SMART auto save timer.Error logging capability: ?(0x01) Error logging supported. ???General Purpose Logging supported.Short self-test routinerecommended polling time: ?( ?2) minutes.Extended self-test routinerecommended polling time: ?( 368) minutes.Conveyance self-test routinerecommended polling time: ?( ?3) minutes.SCT capabilities: ?(0x3035) SCT Status supported. ???SCT Feature Control supported. ???SCT Data Table supported.SMART Attributes Data Structure revision number: 16Vendor Specific SMART Attributes with Thresholds:ID# ATTRIBUTE_NAME ?FLAG ?VALUE WORST THRESH TYPE ?UPDATED ?WHEN_FAILED RAW_VALUE ?1 Raw_Read_Error_Rate ?0x002f ?200 ?200 ?051 ?Pre-fail ?Always ?- ?0 ?3 Spin_Up_Time ?0x0027 ?100 ?253 ?021 ?Pre-fail ?Always ?- ?0 ?4 Start_Stop_Count ?0x0032 ?100 ?100 ?000 ?Old_age ?Always ?- ?4 ?5 Reallocated_Sector_Ct ?0x0033 ?200 ?200 ?140 ?Pre-fail ?Always ?- ?0 ?7 Seek_Error_Rate ?0x002e ?200 ?200 ?000 ?Old_age ?Always ?- ?0 ?9 Power_On_Hours ?0x0032 ?091 ?091 ?000 ?Old_age ?Always ?- ?7047 10 Spin_Retry_Count ?0x0032 ?100 ?253 ?000 ?Old_age ?Always ?- ?0 11 Calibration_Retry_Count 0x0032 ?100 ?253 ?000 ?Old_age ?Always ?- ?0 12 Power_Cycle_Count ?0x0032 ?100 ?100 ?000 ?Old_age ?Always ?- ?4192 Power-Off_Retract_Count 0x0032 ?200 ?200 ?000 ?Old_age ?Always ?- ?2193 Load_Cycle_Count ?0x0032 ?200 ?200 ?000 ?Old_age ?Always ?- ?20194 Temperature_Celsius ?0x0022 ?111 ?109 ?000 ?Old_age ?Always ?- ?36196 Reallocated_Event_Count 0x0032 ?200 ?200 ?000 ?Old_age ?Always ?- ?0197 Current_Pending_Sector ?0x0032 ?200 ?200 ?000 ?Old_age ?Always ?- ?0198 Offline_Uncorrectable ?0x0030 ?100 ?253 ?000 ?Old_age ?Offline ?- ?0199 UDMA_CRC_Error_Count ?0x0032 ?200 ?200 ?000 ?Old_age ?Always ?- ?0200 Multi_Zone_Error_Rate ?0x0008 ?100 ?253 ?000 ?Old_age ?Offline ?- ?0SMART Error Log Version: 1No Errors LoggedSMART Self-test log structure revision number 1Num ?Test_Description ?Status ??Remaining ?LifeTime(hours) ?LBA_of_first_error# 1 ?Short offline ?Completed without error ?00% ?2 ?-SMART Selective self-test log data structure revision number 1 SPAN ?MIN_LBA ?MAX_LBA ?CURRENT_TEST_STATUS ?1 ?0 ?0 ?Not_testing ?2 ?0 ?0 ?Not_testing ?3 ?0 ?0 ?Not_testing ?4 ?0 ?0 ?Not_testing ?5 ?0 ?0 ?Not_testingSelective self-test flags (0x0): ?After scanning selected spans, do NOT read-scan remainder of disk.If Selective self-test is pending on power-up, resume after 0 minute delay.

Seagate

=== START OF INFORMATION SECTION ===Model Family: ?Seagate Barracuda 2.5 5400Device Model: ?ST2000LM015-2E8174Serial Number: ?WDZBHCXKLU WWN Device Id: 5 000c50 0acdec7dfFirmware Version: SDM1User Capacity: ?2,000,398,934,016 bytes [2.00 TB]Sector Sizes: ?512 bytes logical, 4096 bytes physicalRotation Rate: ?5400 rpmForm Factor: ?2.5 inchesDevice is: ?In smartctl database [for details use: -P show]ATA Version is: ?ACS-3 T13/2161-D revision 3bSATA Version is: ?SATA 3.1, 6.0 Gb/s (current: 3.0 Gb/s)Local Time is: ?Wed Apr 24 13:05:23 2019 CESTSMART support is: Available – device has SMART capability.SMART support is: Enabled=== START OF READ SMART DATA SECTION ===SMART overall-health self-assessment test result: PASSEDGeneral SMART Values:Offline data collection status: ?(0x00) Offline data collection activity ???was never started. ???Auto Offline Data Collection: Disabled.Self-test execution status: ?( ?0) The previous self-test routine completed ???without error or no self-test has ever ???been run.Total time to complete Offlinedata collection: ?( ?0) seconds.Offline data collectioncapabilities: ??(0x71) SMART execute Offline immediate. ???No Auto Offline data collection support. ???Suspend Offline collection upon new ???command. ???No Offline surface scan supported. ???Self-test supported. ???Conveyance Self-test supported. ???Selective Self-test supported.SMART capabilities: ?(0x0003) Saves SMART data before entering ???power-saving mode. ???Supports SMART auto save timer.Error logging capability: ?(0x01) Error logging supported. ???General Purpose Logging supported.Short self-test routinerecommended polling time: ?( ?1) minutes.Extended self-test routinerecommended polling time: ?( 331) minutes.Conveyance self-test routinerecommended polling time: ?( ?2) minutes.SCT capabilities: ?(0x3035) SCT Status supported. ???SCT Feature Control supported. ???SCT Data Table supported.SMART Attributes Data Structure revision number: 10Vendor Specific SMART Attributes with Thresholds:ID# ATTRIBUTE_NAME ?FLAG ?VALUE WORST THRESH TYPE ?UPDATED ?WHEN_FAILED RAW_VALUE ?1 Raw_Read_Error_Rate ?0x000f ?078 ?064 ?006 ?Pre-fail ?Always ?- ?57137976 ?3 Spin_Up_Time ?0x0003 ?100 ?100 ?000 ?Pre-fail ?Always ?- ?0 ?4 Start_Stop_Count ?0x0032 ?100 ?100 ?020 ?Old_age ?Always ?- ?4 ?5 Reallocated_Sector_Ct ?0x0033 ?100 ?100 ?036 ?Pre-fail ?Always ?- ?0 ?7 Seek_Error_Rate ?0x000f ?086 ?060 ?045 ?Pre-fail ?Always ?- ?426983688 ?9 Power_On_Hours ?0x0032 ?092 ?092 ?000 ?Old_age ?Always ?- ?7054 (57 92 0) 10 Spin_Retry_Count ?0x0013 ?100 ?100 ?097 ?Pre-fail ?Always ?- ?0 12 Power_Cycle_Count ?0x0032 ?100 ?100 ?020 ?Old_age ?Always ?- ?4184 End-to-End_Error ?0x0032 ?100 ?100 ?099 ?Old_age ?Always ?- ?0187 Reported_Uncorrect ?0x0032 ?100 ?100 ?000 ?Old_age ?Always ?- ?0188 Command_Timeout ?0x0032 ?100 ?100 ?000 ?Old_age ?Always ?- ?0189 High_Fly_Writes ?0x003a ?100 ?100 ?000 ?Old_age ?Always ?- ?0190 Airflow_Temperature_Cel 0x0022 ?067 ?064 ?040 ?Old_age ?Always ?- ?33 (Min/Max 23/36)191 G-Sense_Error_Rate ?0x0032 ?100 ?100 ?000 ?Old_age ?Always ?- ?0192 Power-Off_Retract_Count 0x0032 ?100 ?100 ?000 ?Old_age ?Always ?- ?2193 Load_Cycle_Count ?0x0032 ?100 ?100 ?000 ?Old_age ?Always ?- ?15194 Temperature_Celsius ?0x0022 ?033 ?040 ?000 ?Old_age ?Always ?- ?33 (0 23 0 0 0)197 Current_Pending_Sector ?0x0012 ?100 ?100 ?000 ?Old_age ?Always ?- ?0198 Offline_Uncorrectable ?0x0010 ?100 ?100 ?000 ?Old_age ?Offline ?- ?0199 UDMA_CRC_Error_Count ?0x003e ?200 ?200 ?000 ?Old_age ?Always ?- ?0240 Head_Flying_Hours ?0x0000 ?100 ?253 ?000 ?Old_age ?Offline ?- ?7054 (68 231 0)241 Total_LBAs_Written ?0x0000 ?100 ?253 ?000 ?Old_age ?Offline ?- ?74035391620242 Total_LBAs_Read ?0x0000 ?100 ?253 ?000 ?Old_age ?Offline ?- ?65426356249254 Free_Fall_Sensor ?0x0032 ?100 ?100 ?000 ?Old_age ?Always ?- ?0SMART Error Log Version: 1No Errors LoggedSMART Self-test log structure revision number 1Num ?Test_Description ?Status ??Remaining ?LifeTime(hours) ?LBA_of_first_error# 1 ?Short offline ?Completed without error ?00% ?2 ?-SMART Selective self-test log data structure revision number 1 SPAN ?MIN_LBA ?MAX_LBA ?CURRENT_TEST_STATUS ?1 ?0 ?0 ?Not_testing ?2 ?0 ?0 ?Not_testing ?3 ?0 ?0 ?Not_testing ?4 ?0 ?0 ?Not_testing ?5 ?0 ?0 ?Not_testingSelective self-test flags (0x0): ?After scanning selected spans, do NOT read-scan remainder of disk.If Selective self-test is pending on power-up, resume after 0 minute delay.

Toshiba

=== START OF INFORMATION SECTION ===Device Model: ?TOSHIBA HDWL120Serial Number: ?88K6P0OYTLU WWN Device Id: 5 000039 8b2409181Firmware Version: JT000AUser Capacity: ?2,000,398,934,016 bytes [2.00 TB]Sector Sizes: ?512 bytes logical, 4096 bytes physicalRotation Rate: ?5400 rpmForm Factor: ?2.5 inchesDevice is: ?Not in smartctl database [for details use: -P showall]ATA Version is: ?ACS-3 T13/2161-D revision 5SATA Version is: ?SATA 3.3, 6.0 Gb/s (current: 1.5 Gb/s)Local Time is: ?Wed Apr 24 13:09:44 2019 CESTSMART support is: Available – device has SMART capability.SMART support is: Enabled=== START OF READ SMART DATA SECTION ===SMART overall-health self-assessment test result: PASSEDGeneral SMART Values:Offline data collection status: ?(0x00) Offline data collection activity ???was never started. ???Auto Offline Data Collection: Disabled.Self-test execution status: ?( ?0) The previous self-test routine completed ???without error or no self-test has ever ???been run.Total time to complete Offlinedata collection: ?( ?120) seconds.Offline data collectioncapabilities: ??(0x5b) SMART execute Offline immediate. ???Auto Offline data collection on/off support. ???Suspend Offline collection upon new ???command. ???Offline surface scan supported. ???Self-test supported. ???No Conveyance Self-test supported. ???Selective Self-test supported.SMART capabilities: ?(0x0003) Saves SMART data before entering ???power-saving mode. ???Supports SMART auto save timer.Error logging capability: ?(0x01) Error logging supported. ???General Purpose Logging supported.Short self-test routinerecommended polling time: ?( ?2) minutes.Extended self-test routinerecommended polling time: ?( 322) minutes.SCT capabilities: ?(0x003d) SCT Status supported. ???SCT Error Recovery Control supported. ???SCT Feature Control supported. ???SCT Data Table supported.SMART Attributes Data Structure revision number: 16Vendor Specific SMART Attributes with Thresholds:ID# ATTRIBUTE_NAME ?FLAG ?VALUE WORST THRESH TYPE ?UPDATED ?WHEN_FAILED RAW_VALUE ?1 Raw_Read_Error_Rate ?0x000b ?100 ?100 ?050 ?Pre-fail ?Always ?- ?0 ?2 Throughput_Performance ?0x0005 ?100 ?100 ?050 ?Pre-fail ?Offline ?- ?0 ?3 Spin_Up_Time ?0x0027 ?100 ?100 ?001 ?Pre-fail ?Always ?- ?1621 ?4 Start_Stop_Count ?0x0032 ?100 ?100 ?000 ?Old_age ?Always ?- ?2 ?5 Reallocated_Sector_Ct ?0x0033 ?100 ?100 ?050 ?Pre-fail ?Always ?- ?0 ?7 Seek_Error_Rate ?0x000b ?100 ?100 ?050 ?Pre-fail ?Always ?- ?0 ?8 Seek_Time_Performance ?0x0005 ?100 ?100 ?050 ?Pre-fail ?Offline ?- ?0 ?9 Power_On_Hours ?0x0032 ?092 ?092 ?000 ?Old_age ?Always ?- ?3577 10 Spin_Retry_Count ?0x0033 ?100 ?100 ?030 ?Pre-fail ?Always ?- ?0 12 Power_Cycle_Count ?0x0032 ?100 ?100 ?000 ?Old_age ?Always ?- ?2191 G-Sense_Error_Rate ?0x0032 ?100 ?100 ?000 ?Old_age ?Always ?- ?0192 Power-Off_Retract_Count 0x0032 ?100 ?100 ?000 ?Old_age ?Always ?- ?0193 Load_Cycle_Count ?0x0032 ?099 ?099 ?000 ?Old_age ?Always ?- ?16896194 Temperature_Celsius ?0x0022 ?100 ?100 ?000 ?Old_age ?Always ?- ?30 (Min/Max 24/34)196 Reallocated_Event_Count 0x0032 ?100 ?100 ?000 ?Old_age ?Always ?- ?0197 Current_Pending_Sector ?0x0032 ?100 ?100 ?000 ?Old_age ?Always ?- ?0198 Offline_Uncorrectable ?0x0030 ?100 ?100 ?000 ?Old_age ?Offline ?- ?0199 UDMA_CRC_Error_Count ?0x0032 ?200 ?253 ?000 ?Old_age ?Always ?- ?0220 Disk_Shift ?0x0002 ?100 ?100 ?000 ?Old_age ?Always ?- ?0222 Loaded_Hours ?0x0032 ?092 ?092 ?000 ?Old_age ?Always ?- ?3540223 Load_Retry_Count ?0x0032 ?100 ?100 ?000 ?Old_age ?Always ?- ?0224 Load_Friction ?0x0022 ?100 ?100 ?000 ?Old_age ?Always ?- ?0226 Load-in_Time ?0x0026 ?100 ?100 ?000 ?Old_age ?Always ?- ?268240 Head_Flying_Hours ?0x0001 ?100 ?100 ?001 ?Pre-fail ?Offline ?- ?0SMART Error Log Version: 1No Errors LoggedSMART Self-test log structure revision number 1No self-tests have been logged. ?[To run self-tests, use: smartctl -t]SMART Selective self-test log data structure revision number 1 SPAN ?MIN_LBA ?MAX_LBA ?CURRENT_TEST_STATUS ?1 ?0 ?0 ?Not_testing ?2 ?0 ?0 ?Not_testing ?3 ?0 ?0 ?Not_testing ?4 ?0 ?0 ?Not_testing ?5 ?0 ?0 ?Not_testingSelective self-test flags (0x0): ?After scanning selected spans, do NOT read-scan remainder of disk.If Selective self-test is pending on power-up, resume after 0 minute delay.

camcontrol output

camcontrol identify

WD

pass6: ACS-3 ATA SATA 3.x devicepass6: 300.000MB/s transfers, Command Queueing Enabledprotocol ?ATA/ATAPI-10 SATA 3.xdevice model ?WDC WD20SPZX-00CRAT0firmware revision ?01.01A01serial number ?WD-WX91E17FJNXDWWN ??50014ee6086142a7cylinders ?16383heads ?16sectors/track ?63sector size ?logical 512, physical 4096, offset 0LBA supported ?268435455 sectorsLBA48 supported ?3907029168 sectorsPIO supported ?PIO4DMA supported ?WDMA2 UDMA6media RPM ?5400Feature ??Support ?Enabled ?Value ?Vendorread ahead ??yes ?yeswrite cache ??yes ?yesflush cache ??yes ?yesoverlap ??noTagged Command Queuing (TCQ) ?no ?noNative Command Queuing (NCQ) ?yes ?32 tagsNCQ Queue Management ?noNCQ Streaming ??noReceive & Send FPDMA Queued ?noSMART ??yes ?yesmicrocode download ?yes ?yessecurity ??yes ?nopower management ?yes ?yesadvanced power management ?yes ?no ?254/0xFEautomatic acoustic management ?no ?nomedia status notification ?no ?nopower-up in Standby ?yes ?nowrite-read-verify ?no ?nounload ??yes ?yesgeneral purpose logging ?yes ?yesfree-fall ??no ?noData Set Management (DSM/TRIM) yesDSM – max 512byte blocks ?yes ?10DSM – deterministic read ?yes ?any valueHost Protected Area (HPA) ?no

Seagate

pass4: ACS-3 ATA SATA 3.x devicepass4: 300.000MB/s transfers, Command Queueing Enabledprotocol ?ATA/ATAPI-10 SATA 3.xdevice model ?ST2000LM015-2E8174firmware revision ?SDM1serial number ?WDZBHCXKWWN ??5000c500acdec7dfcylinders ?16383heads ?16sectors/track ?63sector size ?logical 512, physical 4096, offset 0LBA supported ?268435455 sectorsLBA48 supported ?3907029168 sectorsPIO supported ?PIO4DMA supported ?WDMA2 UDMA6media RPM ?5400Feature ??Support ?Enabled ?Value ?Vendorread ahead ??yes ?yeswrite cache ??yes ?yesflush cache ??yes ?yesoverlap ??noTagged Command Queuing (TCQ) ?no ?noNative Command Queuing (NCQ) ?yes ?32 tagsNCQ Queue Management ?noNCQ Streaming ??noReceive & Send FPDMA Queued ?noSMART ??yes ?yesmicrocode download ?yes ?yessecurity ??yes ?nopower management ?yes ?yesadvanced power management ?yes ?no ?0/0x00automatic acoustic management ?no ?nomedia status notification ?no ?nopower-up in Standby ?yes ?nowrite-read-verify ?yes ?no ?0/0x0unload ??yes ?yesgeneral purpose logging ?yes ?yesfree-fall ??no ?noData Set Management (DSM/TRIM) noHost Protected Area (HPA) ?yes ?no ?3907029168/3907029168HPA – Security ?no

Toshiba

pass16: ACS-3 ATA SATA 3.x devicepass16: 150.000MB/s transfers, Command Queueing Enabledprotocol ?ATA/ATAPI-10 SATA 3.xdevice model ?TOSHIBA HDWL120firmware revision ?JT000Aserial number ?88K6P0O0TWWN ??50000398b240915fcylinders ?16383heads ?16sectors/track ?63sector size ?logical 512, physical 4096, offset 0LBA supported ?268435455 sectorsLBA48 supported ?3907029168 sectorsPIO supported ?PIO4DMA supported ?WDMA2 UDMA5media RPM ?5400Feature ??Support ?Enabled ?Value ?Vendorread ahead ??yes ?yeswrite cache ??yes ?yesflush cache ??yes ?yesoverlap ??noTagged Command Queuing (TCQ) ?no ?noNative Command Queuing (NCQ) ?yes ?32 tagsNCQ Queue Management ?noNCQ Streaming ??noReceive & Send FPDMA Queued ?noSMART ??yes ?yesmicrocode download ?yes ?yessecurity ??yes ?nopower management ?yes ?yesadvanced power management ?yes ?yes ?128/0x80automatic acoustic management ?no ?nomedia status notification ?no ?nopower-up in Standby ?yes ?nowrite-read-verify ?yes ?no ?0/0x0unload ??yes ?yesgeneral purpose logging ?yes ?yesfree-fall ??no ?noData Set Management (DSM/TRIM) noHost Protected Area (HPA) ?yes ?no ?3907029168/3907029168HPA – Security ?no

Symptoms

The pool itself is stable for months. But we observed massive performance-differences on vendor drives.

• Disk-Busy: The WD-Drives are busy most of the time (95-100% Disk Busy) while the others (Seagate and Toshiba) have a max of 30%.

• Disk Latency: WD Drives show latencys between 2000 and 8000 msec. for Writes and Deletes while the others (Seagate and Toshiba) are in Ranges of 20-50msec.

• All Drives of the same vendors behave simillar.

  • All WD drives in this system show bad performance. Graphs are almost identical
  • All Seagate and Toshiba Drives show almost identical performance data among thier vendors.
  • Seagate and Toshiba behave almost the same with very slight differences.

Question(s)

• Why do the WD Drives have such bad performance in this Setup.
• Did i miss something in choosing “same drive characteristics”?
  • yes i did: Only WD supports DSM/TRIM! Info about TRIM on WD drives
• Is there a setting for WD-Drives to behave better?
• Why does the WD drive support DSM/TRIM while the others do not?
  • Is there a way to disable that feature to test without?

Answer: CAUSE: The Western Digital Drives support TRIM

Other than Seagate and Toshiba the WD-drive supports TRIM: Data Set Management (DSM/TRIM) yes

ZFS on FreeBSD (FreeNAS) supports TRIM too. The WD drives have plenty to do tiding up deleted Data.

SOLUTION: Disable TRIM

1) My attempt to disable TRIM on the drives itself at runtime ended in a kernel panic. Do this at your own risk!

sysctl kern.cam.da.6.delete_method=DISABLE

2) Disabling TRIM for ZFS did succeed, but leaves my Cache-SSD without TRIM.

Goto FreeNAS – System – Tunables and click “Add Tuneable”

Variable: vfs.zfs.trim.enabledValue: 0Type: LoaderEnabled: checked

After rebooting ZFS does not TRIM anymore. The storage is reactig much quicker. “Disk Busy” stays at a high level, but “Disk Latency” dropped from Seconds to milliseconds!

“Pending I/O requests” dropped from avg. 130 to not more than 20.

The Storage itself feels much more agile.

Edit: last Scrub took about 11 hours while the one before took 24 days.