Config recommendation for export and restore

[nkxco]

Hi,

tl;dr: I am trying to find the osxphotos configuration that saves the largest amount of information, so that a photo library can be restored as closely as possible to the original if there's an issue.

I recently discovered osxphotos, and I've started to use it for periodic backups of my photo library from iCloud to an external cloud for disaster recovery:

• I am using iCloud with the "optimize storage" setting on all my devices, which means that iCloud is currently the only place where my originals are stored
• While this is working well most of the time, I've already suffered from data loss with iCloud, and this config doesn't really feel safe, so I'd like to have another external backup
• I don't need this external backup to be accessible day-to-day, but only for disaster recovery. My plan is to use Amazon S3 Glacier for cold storage, with periodic backups (for instance weekly)

What I'm trying to protect from is:

• malfunctioning Apple device: a device somehow deleting part of my library, where the deletion would then get synced with iCloud and on all devices (user error, bug in software, malware, etc.)
• malfunctioning iCloud server: deleting or corrupting part of my online library (server-side bug, hack, etc.)

Now my question is: what would be the safest osxphotos configuration, to save the maximum amount of information from my library, so that it could be restored in case it is damaged? Ideally, after the restore, the photo library would appear "as before" (so with people, albums, dates, location, etc.)

And follow-up question, what are the limitations of osxphotos for this use case? (Things that cannot be restored and would be lost for instance)

I've read the documentation and some issues and discussions, and I'm currently using this:

osxphotos export
    --library <path> --report <path>
    --cleanup --verbose --timestamp
    --export-by-date --from-date <start> --to-date <end>
    --download-missing --use-photokit
    --exiftool --exiftool-merge-keywords --exiftool-merge-persons --favorite-rating
    --touch-file --retry 5 --strip
    <outpath>

Let me know if there's something to improve. And thanks a lot for the very useful project. I've seen in another discussion that you don't accept donations, if that changes please make it visible in the front page, as I'd like to contribute.

[RhetTbull]

Hi  @nkxco I'm glad you found osxphotos and that it's useful. Your command looks like a very good start. A few thoughts on what else I would include:

--ramdb if exporting to an external drive or NAS
this will improve performance
--album-keyword: store album names as keywords

--person-keyword: store person name as keywords

--sidecar xmp: write XMP sidecars
--sidecar exiftool: write exiftool JSON sidecars

I wouldn't use --cleanup for your use case because deletions (even accidental deletions) would be synced to the backup and deleted.

If you use albums and folders a lot in photos, consider using --directory "{folder_album}" as Photos can restore albims from a path when importing. However, note that you'll get duplicates if an asset is in multiple albums. If. you don't use folders, just albums, you could use --directory "{album}". You cannot use both --directory and --export-by-date.

Instead of --album-keyword, you could also do something like: --keyword-template "{album?album:{folder_album},}". This creates a new keyword named "album:Name of album" for each photo. in an album. This would make it easier to find these on import and recreate albums if necessary. You could do the same with `--keyword-template "{person?person:{person},}" which would create keywords like "person:Person Name" for each person in an image and this could help with re-applying the person tags (which Photos does not import and osxphotos cannot do anything about this).

Note that currently, persons/faces cannot be reimported into Photos. osxphotos will export them to XMP and the embedded metadata with --exiftool but Photos won't read them. Other apps like Lightroom or DigiKam will read them.

I'm working on some new code that would improve the current osxphotos import command and allow it to create a new Photos library from the backup with all metadata (except persons) preserved and restored as it was before backup. Will be some time before this is ready for release.

The limitations you'll run into are:

1. As noted above, Photos does not import persons/face regions and while osxphotos exports these, there's nothing I can do about Photos' operation in this regard.
2. Edits will get exported but upon re-import you'll get 2 copies of edited photos: the original and the edited version, losing the ability to use "restore to original" (though the original will still be there).

The new code I'm working on will solve this because it will be able to create edited photos in the Photos library so upon re-import it could recreate the edits. If this interests you (and again, this capability is a ways off in the future), you might also want to use --export-aae which exports the adjustments files which photos stores with each edited image. It's possible in the future that osxphotos could add these back to the library when imported edited versions as described above. These are in a proprietary Apple format and are not currently used by osxphotos but it can export them.

[RhetTbull]

I am using iCloud with the "optimize storage" setting on all my devices,

In this case, osxphotos will be very ineffecient. Yes it will download the original (lots of bandwidth) but Photos will delete it again. Not a huge issue after the first export as you'll only have to download new / updated photos but something to be aware of.

Another option that doesn't involve osxphotos (and could still be used with osxphotos) is to create a separate user account on your Mac. In this new account, login to your regular iCloud account. Move the system Photos library to an external disk and the in Photos, select "Keep originals on this Mac". Once it syncs, you'll have a second copy of your Photos library that has all the originals on the external drive. You can then log out of the account and unplug the drive. This gives you a good backup of your Photos library that would be much easier to restore from than an osxphotos backup. You can use fast user switching to stay logged in the new account in the background while using the regular account. Periodically, you'd plug in the drive, login and let it sync. This gives an extra layer of protection (and also syncs persons/faces). There are some caveats if you need to restore for if you plug this drive in to iCloud and you've deleted a bunch of photos in iCloud, they'll get deleted in the backup.

[nkxco]

Thanks a lot for the very detailed answer. I am using albums a lot, so your recommendations are useful. I'll try all the options you mention.

I'm working on some new code that would improve the current osxphotos import command

I'm interested! By any chance, is there an issue tracking this so I can check for updates?

In this case, osxphotos will be very ineffecient. Yes it will download the original (lots of bandwidth) but Photos will delete it again.

Thanks for the suggestion. I actually realised exactly what you described, and recently enabled "Keep originals" on the Mac where I perform osxphotos backups, with a separate user account. For now I can afford the storage space, but will probably need an external disk.

[RhetTbull]

I'm interested! By any chance, is there an issue tracking this so I can check for updates?

There's a collection of loosely related issues: #959, #938, #909

The work is being done in a separate repo over at photokit