-
|
Hi there. Excellent TOOL. Now experimenting with import!! I'm running the following command and on a ~90000 Photos libraryosxphotos import --exiftool --verbose --timestamp --no-progress --report import.csv --album CONVERT "/Volumes/Macintosh HD/Users/Shared/Pictures/Pictures/iPhoto/(...)/MOV01046.MOV" "/Volumes/Macintosh HD/Users/Shared/Pictures/Pictures/iPhoto/(...)/MOV04014.MOV" "/Volumes/Macintosh HD/Users/Shared/Pictures/Pictures/iPhoto/(...)/MOV00374.MOV" "/Volumes/Macintosh HD/Users/Shared/Pictures/Pictures/iPhoto/(...)/MOV01585.MOV"I get a fast upload on the 1st movie, but then one minute delay in between imports. Also at the end to write the report it takes 1 minute. MOV file sizes are not that big.
2023-01-11 20:27:36.142965 -- Importing /Users/Shared/Pictures/Pictures/iPhoto/(...)/MOV01046.MOV
2023-01-11 20:27:36.641196 -- Imported MOV01046.MOV with UUID F0C94316-9B8F-40F3-8311-1C74C364B6FB
2023-01-11 20:27:36.641894 -- Setting metadata and location from EXIF for MOV01046.MOV
2023-01-11 20:27:37.030127 -- No metadata to set for MOV01046.MOV
2023-01-11 20:27:37.031395 -- No location to set for MOV01046.MOV
2023-01-11 20:27:37.032802 -- Adding photo MOV01046.MOV to 1 album
2023-01-11 20:27:37.033163 -- Adding photo MOV01046.MOV to album CONVERT
2023-01-11 20:27:37.290056 -- Added MOV01046.MOV (F0C94316-9B8F-40F3-8311-1C74C364B6FB) to album CONVERT
2023-01-11 20:28:34.770483 -- Importing /Users/Shared/Pictures/Pictures/iPhoto/(...)MOV04014.MOV
2023-01-11 20:28:35.095857 -- Imported MOV04014.MOV with UUID E07D4B76-B3FC-4D6E-950E-D9F8D75906EA
2023-01-11 20:28:35.096320 -- Setting metadata and location from EXIF for MOV04014.MOV
2023-01-11 20:28:35.185843 -- No metadata to set for MOV04014.MOV
2023-01-11 20:28:35.186234 -- No location to set for MOV04014.MOV
2023-01-11 20:28:35.186904 -- Adding photo MOV04014.MOV to 1 album
2023-01-11 20:28:35.187195 -- Adding photo MOV04014.MOV to album CONVERT
2023-01-11 20:28:35.403891 -- Added MOV04014.MOV (E07D4B76-B3FC-4D6E-950E-D9F8D75906EA) to album CONVERT
2023-01-11 20:29:33.105296 -- Importing /Users/Shared/Pictures/Pictures/iPhoto/(...)/MOV00374.MOV
2023-01-11 20:29:33.456947 -- Imported MOV00374.MOV with UUID E9B7D7F8-3D7C-4174-9748-7D7EAFC701AD
2023-01-11 20:29:33.457495 -- Setting metadata and location from EXIF for MOV00374.MOV
2023-01-11 20:29:33.551093 -- No metadata to set for MOV00374.MOV
2023-01-11 20:29:33.551473 -- No location to set for MOV00374.MOV
2023-01-11 20:29:33.552007 -- Adding photo MOV00374.MOV to 1 album
2023-01-11 20:29:33.552286 -- Adding photo MOV00374.MOV to album CONVERT
2023-01-11 20:29:33.769769 -- Added MOV00374.MOV (E9B7D7F8-3D7C-4174-9748-7D7EAFC701AD) to album CONVERT
2023-01-11 20:30:31.449675 -- Importing /Users/Shared/Pictures/Pictures/iPhoto/(...)/MOV01585.MOV
2023-01-11 20:30:31.650394 -- Imported MOV01585.MOV with UUID 3C58AB3D-712A-461F-A313-8FA65E1B38D3
2023-01-11 20:30:31.650996 -- Setting metadata and location from EXIF for MOV01585.MOV
2023-01-11 20:30:31.760262 -- No metadata to set for MOV01585.MOV
2023-01-11 20:30:31.760662 -- No location to set for MOV01585.MOV
2023-01-11 20:30:31.761362 -- Adding photo MOV01585.MOV to 1 album
2023-01-11 20:30:31.761659 -- Adding photo MOV01585.MOV to album CONVERT
2023-01-11 20:30:32.014197 -- Added MOV01585.MOV (3C58AB3D-712A-461F-A313-8FA65E1B38D3) to album CONVERT
2023-01-11 20:31:30.476684 -- Wrote import report to import.csv
2023-01-11 20:31:30.477293 -- Done: imported 4 files, 0 errors, |
Beta Was this translation helpful? Give feedback.
Answered by
oPromessa
Jan 21, 2023
Replies: 6 comments 12 replies
-
|
import uses AppleScript behind the scenes which is slow. I'll have to look at what's happening between each photo but I'm surprised it takes a minute between photo -- that doesn't happen in my test case. That said, I do know that album & folder lookup is very slow if you have a lot of folders or albums and I've done a complete rewrite of the relevant parts of photoscript to significantly speed up folder/album access -- see this issue. The code is done I just have not gotten around to deploying it as I need to rewrite all the tests for Ventura before releasing. (I've been distracted on other interesting projects!) Once I get the new PhotoScript version pushed you should see some speed up. When I get a chance I'll add some instrumentation to import to figure out where that 1 minute delay is coming from. |
Beta Was this translation helpful? Give feedback.
0 replies
-
|
Would be related with a database update? osxphotos/osxphotos/cli/import_cli.py Lines 1548 to 1549 in 106b258 |
Beta Was this translation helpful? Give feedback.
1 reply
-
|
Maybe but the import database is kept on the home volume in |
Beta Was this translation helpful? Give feedback.
-
|
Great. Just added some clumsy ad-hoc timestamps before and after the Also checked that I had some Smart Albums which yield large number of pics/videos and were catching the imported videos on their filters. But I remove them but had no impact. 2023-01-14 00:16:56.477652 -- Adding photo MOV03474.MOV to album XXX
2023-01-14 00:16:56.664193 -- Creating album 'XXX'
2023-01-14 00:16:56.791115 -- Added MOV03474.MOV (2C005D3A-88AF-4E39-943C-722132FD162B) to album XXX
2023-01-14 00:16:56.791663 -- Before update_report_record /Users/Shared/Pictures/Pictures/(...)/MOV03474.MOV
2023-01-14 00:17:56.424713 -- After update_report_record /Users/Shared/Pictures/Pictures/(...)/MOV03474.MOV
2023-01-14 00:17:56.427133 -- After import_db /Users/Shared/(...)/MOV03474.MOV
2023-01-14 00:17:56.427511 -- After progress.advance /Users/Shared/Pictures/(...)/MOV03474.MOV
2023-01-14 00:17:56.427990 -- Importing /Users/Shared/Pictures/(...)/MOV04186.MOVAdded a bit more of debug... at it seems the time is taken while entering the function and executing but photo.albums has just one entry. report_record.albums = [a.title for a in photo.albums] |
Beta Was this translation helpful? Give feedback.
1 reply
-
Yes, unfortunately this call has to traverse all albums in the Photos library to look up the one album the photo is in. This is a limitation of Apple's very klunky AppleScript interface. I'm working on a fix (see notes below) that should help a lot. |
Beta Was this translation helpful? Give feedback.
-
|
Further to this I've commented out the following... runs a bit faster but still sometime in between imports. Not 1 minute... but still ~25 seconds. Maybe my DB is just too big ?? ~90K pics. if False:
update_report_record(report_data[filepath], photo, filepath)
import_db.set(str(filepath), report_data[filepath])2023-01-14 01:16:53.164497 -- Importing /Users/Shared/Pictures/Pictures/(...)/MOV04495.MOV
2023-01-14 01:16:53.425340 -- Imported MOV04495.MOV with UUID 2CEC04E2-F1D2-4A75-8D23-D89F4F68ECAB
2023-01-14 01:16:53.425851 -- Setting metadata and location from EXIF for MOV04495.MOV
2023-01-14 01:16:53.826976 -- No metadata to set for MOV04495.MOV
2023-01-14 01:16:53.860147 -- Set location for MOV04495.MOV: 42.5769, 1.667764
2023-01-14 01:16:53.862213 -- Adding photo MOV04495.MOV to 1 album
2023-01-14 01:16:53.862599 -- Adding photo MOV04495.MOV to album iPhoto Events/XXX
2023-01-14 01:17:17.850100 -- Added MOV04495.MOV (2CEC04E2-F1D2-4A75-8D23-D89F4F68ECAB) to album iPhoto Events/XXX
2023-01-14 01:17:17.851715 -- Importing /Users/Shared/Pictures/Pictures/(...)/MOV04494.MOV
2023-01-14 01:17:18.107963 -- Imported MOV04494.MOV with UUID FBCEF30F-0D1C-434B-BD2D-457E91905D27
2023-01-14 01:17:18.108490 -- Setting metadata and location from EXIF for MOV04494.MOV
2023-01-14 01:17:18.226821 -- No metadata to set for MOV04494.MOV |
Beta Was this translation helpful? Give feedback.
1 reply
-
|
I've released a new version v0.56.0 that includes a
The output of this command will show profiling data which will allow us to figure out exactly where the delays are. (Though, based on your previous debugging, I think it's going to be in getting the album names from PhotoScript) |
Beta Was this translation helpful? Give feedback.
-
|
I'm testing the new release with extra profiling/debug options now but your data supports what I thought the likely issue was: namely, in libraries with many albums and/or folders, the lookup with PhotoScript which is my wrapper around AppleScript and how I interact with the library to add photos, is very slow. I'm working on a refactor (See #36) and need to get that out. That fix should significantly boost performance. I have a design for an option that completely does away with AppleScript but that is months away. |
Beta Was this translation helpful? Give feedback.
0 replies
-
|
Sorry for the delay. With version osxphotos --profile import --no-progress --verbose --timestamp --no-progress --exiftool --album CONVERT --report "imported_mpg_files_{today}.csv" file1 file2 file2and output is: Profile output |
Beta Was this translation helpful? Give feedback.
9 replies
-
|
I edited your comment to hide the profile content behind a spoiler tag for easier reading of the comment thread. When you have a chance, try the following command which will launch the debugger when the code hits the
When you run this, as soon as the code hits the problem function, it'll pop open the debugger, which will look something like this: Sample debugger outputAt the
Each time the code comes to this function, it will open the debugger prompt. Once you've collected enough data to find the problem line, you can hit Output will look something like this: Sample debugger output(Pdb++) s
--Call--
[12] > ...osxphotos/cli/import_cli.py(633)update_report_record()
-> def update_report_record(report_record: ReportRecord, photo: Photo, filepath: Path):
(Pdb++) n
[12] > ...osxphotos/cli/import_cli.py(635)update_report_record()
-> report_record.albums = [a.title for a in photo.albums]
(Pdb++) n
[12] > ...osxphotos/cli/import_cli.py(636)update_report_record()
-> report_record.description = photo.description
(Pdb++) n
[12] > ...osxphotos/cli/import_cli.py(637)update_report_record()
-> report_record.filename = filepath.name
(Pdb++) n
[12] > ...osxphotos/cli/import_cli.py(638)update_report_record()
-> report_record.filepath = filepath
(Pdb++) n
[12] > ...osxphotos/cli/import_cli.py(639)update_report_record()
-> report_record.imported = True
(Pdb++) n
[12] > ...osxphotos/cli/import_cli.py(640)update_report_record()
-> report_record.keywords = photo.keywords
(Pdb++) n
[12] > ...osxphotos/cli/import_cli.py(641)update_report_record()
-> report_record.location = photo.location
(Pdb++) n
[12] > ...osxphotos/cli/import_cli.py(642)update_report_record()
-> report_record.title = photo.title
(Pdb++) n
[12] > ...osxphotos/cli/import_cli.py(643)update_report_record()
-> report_record.uuid = photo.uuid
(Pdb++) n
[12] > ...osxphotos/cli/import_cli.py(645)update_report_record()
-> return report_record
(Pdb++) n
--Return--
[12] > ...osxphotos/cli/import_cli.py(645)update_report_record()->ReportRecord(...764E404A24F5')
-> return report_record
(Pdb++) c |
Beta Was this translation helpful? Give feedback.
-
|
After reviewing the code, I'm almost certain the issue is going to be in this line: report_record.albums = [a.title for a in photo.albums]The issue is that this line is retrieving the albums a photo belongs to and that's a very inefficient process that scales with the number of albums and folders in a user's library. Apple Photos does not provide a way to do this in the app itself, nor in the AppleScript interface nor in the native PhotoKit API. osxphotos is using PhotoScript which is a python wrapper I wrote around the Photos AppleScript interface. Because Photos doesn't provide a way to determine the albums a photo is in, the only method available is to get a list of all albums and query each one to see if the photo is contained in the album. In Mojave and earlier, Photos would return all albums in the Library when queried. In Catalina and later, Photos only returns top-level albums which means that for albums contained in folders, the code needs to walk the entire folder/album tree to find every album. In AppleScript, this is very slow, even on a modern M1 Mac. For "static" commands like export, osxphotos reads the Photos SQL database to extract the album information but this isn't done in import because the osxphotos design requires reading the entire database and this would be horribly slow to do for every imported photo. I can think of three alternatives to fix this issue:
I guess the 4th option is to do nothing and live with the very slow import code. For now, I'll take a look at what can be done to implement #3 and continue to think about the design of #2 which has much broader uses beyond this use case. For example, it would allow you to query the Photos database on non-Mac devices as it would rely only on SQL queries unlike the current osxphotos code which is an ugly mix of SQL, python, AppleScript, and Objective-C. |
Beta Was this translation helpful? Give feedback.
-
|
I have a good start on implementing # 3, should have a fix out this week. Still need to come up with a more elegant solution for the AppleScript slowness but I'm confident I can fix this issues with slow import for now. |
Beta Was this translation helpful? Give feedback.
-
Sorry I wasn't able to provide you with the debug. But I've just now tested out version $ osxphotos --profile import --exiftool --verbose --timestamp --no-progress --report import.csv --album CONVERT "/Volumes/(...)/MOV01046.MOV" /Volumes/(...)/MOV04014.MOV /Volumes/(...)/MOV00374.MOV "/Volumes/(...)/MOV01585.MOV"
Profiling...
Profile sort_stats order: ['cumulative']
2023-01-21 16:04:07.486164 -- Importing /Volumes/(...)/MOV01046.MOV
2023-01-21 16:04:08.031624 -- Imported MOV01046.MOV with UUID B94DF56E-9BFC-4F39-8B96-B3F145900F84
2023-01-21 16:04:08.032462 -- Setting metadata and location from EXIF for MOV01046.MOV
2023-01-21 16:04:08.561089 -- Set metadata for MOV01046.MOV:
2023-01-21 16:04:08.562493 -- title='', description='Album:(...)']
2023-01-21 16:04:08.601276 -- Set location for MOV01046.MOV: (...)
2023-01-21 16:04:08.604368 -- Adding photo MOV01046.MOV to 1 album
2023-01-21 16:04:08.605735 -- Adding photo MOV01046.MOV to album CONVERT
2023-01-21 16:04:08.799014 -- Creating album 'CONVERT'
2023-01-21 16:04:08.910013 -- Added MOV01046.MOV (B94DF56E-9BFC-4F39-8B96-B3F145900F84) to album CONVERT
2023-01-21 16:04:08.930400 -- Importing /Volumes/(...)MOV04014.MOV
2023-01-21 16:04:09.162008 -- Imported MOV04014.MOV with UUID 94D2E74C-1B08-4FA4-BCF6-786623F66C7B
2023-01-21 16:04:09.162700 -- Setting metadata and location from EXIF for MOV04014.MOV
2023-01-21 16:04:09.516486 -- Set metadata for MOV04014.MOV:
2023-01-21 16:04:09.517110 -- title='', description='(...)']
2023-01-21 16:04:09.517900 -- No location to set for MOV04014.MOV
2023-01-21 16:04:09.518883 -- Adding photo MOV04014.MOV to 1 album
2023-01-21 16:04:09.519370 -- Adding photo MOV04014.MOV to album CONVERT
2023-01-21 16:04:09.795705 -- Added MOV04014.MOV (94D2E74C-1B08-4FA4-BCF6-786623F66C7B) to album CONVERT
2023-01-21 16:04:09.817753 -- Importing /Volumes(...)/MOV00374.MOV
2023-01-21 16:04:10.133043 -- Imported MOV00374.MOV with UUID 8D6D797F-8F48-4022-9F26-8E690A50C653
2023-01-21 16:04:10.133747 -- Setting metadata and location from EXIF for MOV00374.MOV
2023-01-21 16:04:10.336382 -- Set metadata for MOV00374.MOV:
2023-01-21 16:04:10.337009 -- title='', description='(...)']
2023-01-21 16:04:10.374359 -- Set location for MOV00374.MOV: -23, -44.310001
2023-01-21 16:04:10.375906 -- Adding photo MOV00374.MOV to 1 album
2023-01-21 16:04:10.376469 -- Adding photo MOV00374.MOV to album CONVERT
2023-01-21 16:04:10.599707 -- Added MOV00374.MOV (8D6D797F-8F48-4022-9F26-8E690A50C653) to album CONVERT
2023-01-21 16:04:10.621172 -- Importing /Volumes/(...)/MOV01585.MOV
2023-01-21 16:04:10.929210 -- Imported MOV01585.MOV with UUID 7F51B3F8-2C0A-4250-97EF-166297B8C49B
2023-01-21 16:04:10.929909 -- Setting metadata and location from EXIF for MOV01585.MOV
2023-01-21 16:04:11.185895 -- Set metadata for MOV01585.MOV:
2023-01-21 16:04:11.186507 -- title='', description='(...)
2023-01-21 16:04:11.187091 -- No location to set for MOV01585.MOV
2023-01-21 16:04:11.187935 -- Adding photo MOV01585.MOV to 1 album
2023-01-21 16:04:11.188422 -- Adding photo MOV01585.MOV to album CONVERT
2023-01-21 16:04:11.417793 -- Added MOV01585.MOV (7F51B3F8-2C0A-4250-97EF-166297B8C49B) to album CONVERT
2023-01-21 16:04:11.437719 -- Wrote import report to import.csv
2023-01-21 16:04:11.438582 -- Done: imported 4 files, 0 errors,
Profiling completed
106422 function calls (98904 primitive calls) in 4.000 secondsProfile output |
Beta Was this translation helpful? Give feedback.
Answer selected by
oPromessa
-
The changes I just pushed weren't yet released. Did you do a I'm working on #937 now and planned to release those two changes today. |
Beta Was this translation helpful? Give feedback.
-
Yes. Sorry. Tested on the local repo with the latest changes. |
Beta Was this translation helpful? Give feedback.
-
Great -- glad the fix worked. Building the new release now! |
Beta Was this translation helpful? Give feedback.
-
|
Fixed in v0.56.4 |
Beta Was this translation helpful? Give feedback.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Sorry I wasn't able to provide you with the debug.
But I've just now tested out version
0.56.3you just released and it's a breeze ;) GREAT!