'Scanning' Speed on network folders

Feb 21, 2016 at 6:17 PM
Hi

I've setup a test backup... first time 7zBackup user so be patient. :-)

I setup a Windows Server 2012 R2 on a VirtualBox machine with 4 cores & 4Gb RAM..
The script runs on this VM and has the Source network drive mounted (network drive is on the Virtualbox Host server). The Backup target is my mounted OneDrive 4 Business account..

I started the script at around 9:30 am and now it's 8pm and the script isn't finished scanning the network drive.. there's approx. 700Gb of pictures... jpgs, nef, bmp's etc etc...

it's gotten around 45% of the drive now..after 11 hrs..

anyway to speed this up?
Feb 21, 2016 at 7:42 PM
so.. now it finished and is archiving...
this is the scanning result:

Scanning ...

Selected 282.155 out of 282.155 files in 59.748 folders. 631.363,57 MBytes to backup
Completed in 0 days, 11 hours, 21 minutes, 14 seconds

Selection Details

Extension Count Total MB Abs % Inc %

.jpg 259.055 307.868,39 48,76 48,76
.NEF 10.771 117.476,87 18,61 67,37
.MOV 864 107.170,61 16,97 84,34
.mp4 1.176 74.051,86 11,73 96,07
.flv 404 6.849,23 1,08 97,16
.psd 131 5.733,58 0,91 98,07
.zip 15 5.222,91 0,83 98,89
.wmv 18 3.089,77 0,49 99,38
.rar 29 1.045,14 0,17 99,55
.3gp 46 820,28 0,13 99,68
.BridgeCacheT 180 727,76 0,12 99,79
.bmp 60 518,43 0,08 99,88
.png 425 315,26 0,05 99,93
.db 946 156,47 0,02 99,95
.AVI 2 90,10 0,01 99,96
.tmp 14 35,24 0,01 99,97
.MPG 5 34,54 0,01 99,98
.tif 21 30,90 0,00 99,98
.html 1.655 30,83 0,00 99,98
.dll 10 18,37 0,00 99,99
.xmp 4.470 16,65 0,00 99,99
.docx 8 12,81 0,00 99,99
.jpeg 6 9,55 0,00 99,99
.pdf 8 7,33 0,00 100,00
.BridgeCache 180 5,71 0,00 100,00
.thm 567 5,41 0,00 100,00
.log 3 5,28 0,00 100,00
.bak 1 4,49 0,00 100,00
.jar 70 2,58 0,00 100,00
.gif 166 1,19 0,00 100,00
.java 130 1,13 0,00 100,00
.amr 4 0,85 0,00 100,00
.wav 6 0,79 0,00 100,00
.doc 2 0,57 0,00 100,00
.xml 39 0,41 0,00 100,00
.rb 108 0,31 0,00 100,00
.js 27 0,31 0,00 100,00
.info 1 0,23 0,00 100,00
.DS_Store 4 0,20 0,00 100,00
.properties 15 0,13 0,00 100,00
.unpack 1 0,12 0,00 100,00
.gz 2 0,12 0,00 100,00
.pack 1 0,12 0,00 100,00
.css 12 0,11 0,00 100,00
.a 1 0,10 0,00 100,00
.rtf 2 0,09 0,00 100,00
                                      67              0,09    0,00  100,00
.eeprom 1 0,06 0,00 100,00
.dat 21 0,05 0,00 100,00
.icns 1 0,05 0,00 100,00
.csv 1 0,04 0,00 100,00
.class 15 0,04 0,00 100,00
.txt 10 0,03 0,00 100,00
.ini 41 0,03 0,00 100,00
.libraries 1 0,01 0,00 100,00
.security 1 0,01 0,00 100,00
.dtd 2 0,01 0,00 100,00
.ps1 1 0,01 0,00 100,00
.wlmp 1 0,00 0,00 100,00
.au 2 0,00 0,00 100,00
.apdisk 2 0,00 0,00 100,00
.TemporaryItems 1 0,00 0,00 100,00
.501 1 0,00 0,00 100,00
.jad 4 0,00 0,00 100,00
.policy 2 0,00 0,00 100,00
.CTG 1 0,00 0,00 100,00
.db@SynoEAStream 7 0,00 0,00 100,00
.BridgeSort 1 0,00 0,00 100,00
.plist 1 0,00 0,00 100,00
.res 1 0,00 0,00 100,00
.c 1 0,00 0,00 100,00
.conf 1 0,00 0,00 100,00
.bean 2 0,00 0,00 100,00
.manifest 1 0,00 0,00 100,00
.10 1 0,00 0,00 100,00
.14 1 0,00 0,00 100,00
.19 1 0,00 0,00 100,00
.18 1 0,00 0,00 100,00
.9 1 0,00 0,00 100,00
.13 1 0,00 0,00 100,00
.15 1 0,00 0,00 100,00
.12 1 0,00 0,00 100,00
.8 1 0,00 0,00 100,00
.11 1 0,00 0,00 100,00
.17 1 0,00 0,00 100,00
.16 1 0,00 0,00 100,00
.5 1 0,00 0,00 100,00
.4 1 0,00 0,00 100,00
.7 1 0,00 0,00 100,00
.2 1 0,00 0,00 100,00
.3 1 0,00 0,00 100,00
.6 1 0,00 0,00 100,00
.1 1 0,00 0,00 100,00
.document 1 0,00 0,00 100,00
.fail 278 0,00 0,00 100,00
.DPS 5 0,00 0,00 100,00
                              ----------- ----------------- 
Total 282.155 631.363,57
                              =========== ================= 

should I seriously expect 11 hrs for each backup... ?

What I want is to set up a schedule like I am using today with rsnapshot on unix..

1 daily backup .. 7 day rotation
1 weekly.. 4 week rotation
1 montly
1 yearly
Coordinator
Feb 23, 2016 at 11:00 AM
Hi Symbiot,
I must admit 11 hours are a bit of pain in the ass.
Nevertheless your environment is quite prone to performance issues: in fact the same pc acts both as a host and as a virtualized guest. Network communications between the two is shared via a single bridged network card and disk access is shared. Therefore the slow response.
There may also be other factors which may impact on performance : just for example disk speed, disk access caching and protection software.

In my environments I measure 250k files examined roughly in 1hour (AVG).

You should test, if you have the opportunity, the backup solution using two phisycal different HW.

Let me know.
Coordinator
Jun 8, 2016 at 6:54 AM
Hi Symbiot,
new release 2.0.1 completely refactored the scanning process enforcing more speed.

If you have the opportunity please test it and let me know.