Speed of export when running --update

[chrismanderson]

Hello - just started using the tool. Did an initial export and everything worked great (for about 500 GB of media).

I then exported again a few days later and noticed that the tool evaluates every photo and sidecar file. I have the verbose logs enabled, and saw each photo being evaluated. Now, each photo was then reported as 'Skipped up to date file' so nothing was actually copied. However, I assumed that subsequent exports after the initial one would have been much faster, but in this case, because each photo is still evaluated, it took roughly the same amount of time.

Is that expected behavior? It's not a big ideal if it is as my syncs will run overnight anyways, but I would like to know if I'm doing something wrong. Thanks!

[RhetTbull]

Hi @chrismanderson welcome to osxphotos. The answer is "it depends". If on an internal SSD drive, the update will be much faster. If on a NAS or slower external drive, it may take the same amount of time. Depending on the options you used for export, here's what osxphotos evaluates when doing an update:

1. did the file signature (size, modification time) change? For this it needs to do a stat() on the filesystem. Very fast on internal SSD, not so fast on a NAS.
2. If writing sidecars or using the exiftool option, did any of the metadata change? To do this it evaluates the metadata in the Photos library against the metadata stored in the osxphotos export database (normally in /your/export/path/.osxphotos_export.db. If this export database (a SQLite database) is on a slow external drive/NAS, then this lookup process can be slow.

If you are exporting to a NAS or external drive, I suggest using the --ramdb option which stores the export database in memory until the export is completed. You could also use the --exportdb DB_PATH option to store the export database on your internal drive and this can be used with or without --ramdb. If doing this you'd want to move the export database from the root of the current export path to the new location so you don't lose the update state info. See osxphotos help export exportdb for more details.

I've made a couple attempts to improve export speed with multi-threading, etc. however this greatly complicates the code and it's not something I need so I do not spend a lot of time worrying about the export speed. As you said, for repeat use, just let it run overnight.

[RhetTbull]

Also, to be clear, each photo is not evaluated in the sense that the photo is not loaded from disk and examined. Only the data about the photo is evaluated. For example, if you added a keyword to a previously exported photo osxphotos would need to write a new sidecar. The only way it can do this is by examining the previously exported data (in the export database). If you're exporting to external disk I expect --ramdb will speed things up.

[chrismanderson]

Thanks for the information! I am backing up to a NAS, and the --ramdb option seems to be having the time of a re-export, which is great. Thanks.

[RhetTbull]

@chrismanderson by the way, if you've not made any metadata changes (keywords, albums, etc.) and you simply want to update the export with any newly added photos, you can use the --only-new option and this will run in a fraction of the time as osxphotos won't even check the previously exported files.

[chrismanderson]

Is it 'safe' to alternate the use of --only-new with exports?

E.g. - I'm imagining that I run --only-new fairly regularly, to ensure that the file themselves get captured in the export. Then, run a full export, without the --only-new weekly or some less-frequent cadence to capture all metadata changes. That seems to be a great way to balance export speed with completeness. But, I'd love to confirm if that approach makes sense and won't cause any export issues or conflicts down the road.

[RhetTbull]

Yes, that's safe as long as you always include --update.

You can save all the settings in a TOML file using --save-config when you run export then the next time, you can use --load-config to load the settings and all you have to do is provide the export path. for example: osxphotos export /path/to/export --load-config full_backup.toml or ospxhotos export /path/to/export --load-config only_new.toml. You could thus create separate toml files to make it easy to run the version of the backup you want.

[chrismanderson]

💯 Fantastic, thanks so much for the help and for the amazing work here.