This blog is for future me and for anyone else wanting to understand some iBooks structure. It’s not an attack on Apple – I know I’m an extreme case.
Some of you may know my fondness for books. A habit that led to me buying so many books when the iPad came out I actually broke the iBooks app (too many books to display on the “purchased” screen) which took a year to fix. Fast forward several years..
It’s been an unexpected few days of technical support. Rumour is that Apple will be changing the iBooks app in an upcoming release and that always makes me nervous. I buy around 30 books a month and have 3859 on my iPad and iPhone. Probably about 60/40 iTunes and Amazon. Losing my books would be equivalent to someone who cares about music losing all their music or a gamer losing all their games. It would be bad. Give her space. Don’t try and talk to her. Back away slowly. Bad.
I carefully backup (and have to remove DRM to do it) about once a month. Why? Because Apple may decide to drop iBooks at any time and then where would I be with 4000 (or at least 2000) unreadable books?
So I needed to backup and since upgrading to High Sierra that’s been impossible. The technology I used only worked up to Sierra. That’s OK, I use Parallels , can download Sierra at no cost from the App Store and create a VM running Sierra. Of course I had to authorise that VM with my iTunes account so it could read the books which meant deauthorising everything else first since I was at 5 devices. Top tip, if you buy new kit, make sure you deactivate iTunes before flattening the old kit.
Step 1: Getting the books into my VM
In theory because I sync my books to the cloud I should be able to just launch iBooks and auto redownload. Unfortunately that didn’t happen. The books display as in the cloud but have to be manually downloaded. Understandably selecting nearly 4000 books and telling iBooks to download them all caused it to crash. Repeatedly. So I needed a better way.
Step 2: Why not just copy the books from my laptop which is the host machine for the VM?
Some digging uncovered that my epubs are stored in
so surely I can just copy them over from one machine to another? Why yes I can and when I launch iBooks they all display – kind of. They can’t be read and most of the covers are missing but otherwise.. great! Some more digging later and I realised that although I had copied over the books.plist (which is a preferences file containing and index of all the books iBooks knows about), I didn’t get the SQL database that iBooks uses that is in ~/Library/Containers/com.apple.iBooksX.
So that isn’t going to work. A few hours of trying to get covers to appear or books to be readable and I realised I needed to take a step back.
Step 3: Maybe I was overthinking this. iBooks builds the index when you add books to the app by choosing “add to library” or just drag and drop them so why not drag the 4000 epubs into iBooks. I knew they were already there but I tested and it does prompt you with the option to “Replace” all books that are already there instead of creating duplicates (of course what I could really do with is “Skip” rather than “Replace” but I get i’m in a niche situation).
So – drag 4000 books to iBooks and choose “Replace” and wait. There’s no progress bar. Nothing. The only way I can see that anything is happening is by launching activity monitor and noting that bkagentservice was consuming 80+% CPU. Eventually “lots” of books appear. This is the point where I realise there’s no way to count how many books are in iBooks. I knew “lots” wasn’t all because I got this dialog “<epub filename> couldn’t be opened because you don’t have permission to view it”
I click OK and got another, and another and another. Eventually having to Force Quit iBooks and restart.
Fair enough. Maybe when copying over the files from host to guest the permissions came with them and my new guest account doesn’t have permissions. I spend some time making sure all permissions are OK, applying my new account as well as “Everyone” to that folder and all files contained in it. I finally test by dragging and dropping individual files into iBooks that work with no error so I decide that error is a red herring – it’s more a “gah! iBooks can’t handle you doing that and has tripped over itself – try adding fewer books”
So now I have a new problem. What books are missing? If I knew what books were missing I could manually add them. Unfortunately not only do I not know what books are missing, I don’t know if it’s 10 books or 2000.
Step 4: The search for the missing books
Those filenames aren’t terribly helpful but I know what books I have so I search in iBooks for certain book titles and discover some that aren’t there that should be (and are in my iBooks on my host machine). How do I find the filename that matches the book title if I know I have the epub in the correct directory? Here we head to terminal. In the directory
I type grep “some phrase” ./*.epub -r
tell application “System Events”
tell property list file “/Users/gabrielladavis/Library/Containers/com.apple.BKAgentService/Data/Documents/iBooks/Books/Books.plist”
set Booklist to value of property list item “Books”
set Output to “”
set Counter to 0
repeat with a from 1 to length of Booklist
set theCurrentListItem to item a of Booklist
set author to artistname of theCurrentListItem
set booktitle to itemname of theCurrentListItem
set thefile to sourcepath of theCurrentListItem
set Output to Output & author & “,” & booktitle & “,” & thefile & return
set Counter to Counter + 1
if Counter mod 50 = 0 then
The counter was so I could see it was actually doing something as it ran. The “try” was to check if the item has an author etc since my PDFs often didn’t and the code would fail otherwise.
It may not be pretty but it gave me what I wanted which was thousands of lines like this
Pamela Hartshorne,Time’s Echo,/Users/gabrielladavis/Library/Containers/com.apple.BKAgentService/Data/Documents/iBooks/Books/1F31185F755DD6B65C00B1CF641409B4.epub
Riggs, Ransom,Miss Peregrine’s Home for Peculiar Children,/Users/gabrielladavis/Library/Containers/com.apple.BKAgentService/Data/Documents/iBooks/Books/46D721416EA9EBB037E767DF155A4395.epub
Step 7: An afternoon with Excel
Running the agent twice against the host and guest books.plist gives me the data I need. The host machine plist gives me 3789 entries and the guest machine 5443 entries. It appears every time I attempted to drag and drop a file in the guest copy of iBooks it created a new plist entry. I enjoy data manipulation in Excel and after cleaning things up and playing with INDEX/MATCH I discover…. it’s not going to work.
The problem is that the plist filename is only updated when the books are added to the library so there was an unreliable mismatch between the guest and host plists.
Step 8: Take a step back and try playing by Apple’s rules
I take a copy of the iBooks directory into another folder (“movedbooks”)then I launch iBooks itself and (making sure iCloud is completely disabled on the guest machine so there’s absolutely not syncing to any device) I remove every.single.book from within iBooks. Several scary minutes later iBooks is empty and so is the iBooks folder and the plist file.
Meanwhile I still have a copy of all the books in “movedbooks” – I know iBooks didn’t like me dropping 4000 books in but at this point I’m prepared to meet it half way. After some trial and error, I copy the books in 250 or so at a time. I verify they are added correctly by checking the books count that appears in the iBooks folder. It takes about an hour but when I’m done, the iBooks folder is 170 items smaller than the movedbooks backup.
Step 9: The search for the missing books
I now need a tool to compare the contents of the movedbooks folder to the ibooks folder and tell me which files re present in the first but missing in the second i.e. are missing from iBooks. A free app called “Compare Folders” does that for me nicely. Unfortunately it won’t let me export the list but at list I can see the list of missing files.
Step 10: The final piece
170 is a manageable number so now, one by one, I find the missing files and drop them into iBooks. That works and I end up 3849 books in iBooks and in the directory. If you’ve spotted that’s 10 less than I should have then congratulations, that’s not a typo. 10 books completely resisted being added to the guest, no error, nothing, they just won’t add. Even weird when I check my Excel spreadsheet and decide I don’t care about those 10. But I make a note in case in care in the future.
So that’s it. I shouldn’t need to do this again as I can add books in small numbers as I buy them and never again have to add all books I’ve bought. In theory.
A final note. If you have a Mac , buy yourself a copy of DiskWarrior, but that’s a story for another day.