#googletakeout — Public Fediverse posts

Google už zase prudí s tím, že mi na jeho službách zbývá už jen 1,45 GB místa. Drive i Photos mám v zanedbatelných velikostech, ale Gmail má 9,3 GB.

Dal jsem druhou šanci Google One Storage Manager. Ano, našlo to pár bizarních e-mailů, které měly víc než 20 MB, ale kdybych si tím měl uvolnit gigabajty, musel bych si s tím hrát týden. Nejde to ani seřadit od největších! Zjevně je to tam jen na oko, aby člověk prostě zmáčkl všudypřítomné tlačítko „Get more storage“ a neřešil to.

Asi nemám sílu přecházet s e-mailem jinam, ale konečně mě napadlo aspoň polovičaté řešení. Přes https://takeout.google.com/ si stáhnu zálohu celého Gmailu v otevřeném formátu a pak prostě udělám nějaký cutoff a smažu z Gmailu vše, co je víc jak 10 let staré. Kdybych to potřeboval, budu to mít na disku jako MBOX.

Je v tomto řešení nějaká mezera, kterou přehlížím?

#gmail #googletakeout #googleonestoragemanager #storagefull #cloud #enshittification

Migrating from Google Photos to Immich Via Google Takeout

In an ideal world Google Photos, iCloud Photos, MyCloud (Swisscom) and other photo backup solutions would back up your photos into a directory structure that you can access and download from with ease and convenience. Unfortunately non of them want to offer that. That is why finding a workflow to get data out is worthwhile.

Remember, a cloud solution, where you can't get media files back out, is not a backup solution. It's just a synching solution. That's why Immich and kDrive are interesting alternatives.

The Problem

If you're like me you have been taking videos and photos for decades and in that time you have taken gigabytes worth of photos and terabytes worth of video. If we save everything to a hard drive then we're fine but because we're in the age of laptops and mobile phones we're backing up to the cloud, and applications are offloading our data from MacOS, iOS and Android. The result is that we no longer have local copies of files and that's where Google Takeout comes in.

Google Takeout

Google Takeout is a tool provided by Google One to export data from Google services to have a local backup. A few years ago when I first considered leaving Google One for kDrive, because that solution was cheaper I found that exporting all my data would be time consuming.

The reason for this is that I had over a hundred zip files to download from Google Drive and that would take hours, if not days to do. Instead, at the time, I was able to backup Google Photos directly to Infomaniak kDrive. With kDrive I could download the files automatically via the kDrive desktop app. I won't elaborate any further because that solution is no longer available.

Takeout Options

Google Takeout gives you the option of downloading 1GB, 2GB, 10GB, 50GB zip files and more. If you have 360 GB of files, as I do, the most convenient solution is to download the 50GB files as you only have eight files to download. With the 2gb version I would have had 186 files to download. It takes half a day or more for the files to be ready.

Unzipping and Metadata

Safari will want to automatically "open" the zip files but I chose not to have this option because I want to keep the zips. I downloaded all eight files, and before doing anything else I backed up the eight files to a second disk as a backup. I then unzipped all eight zips. This takes time so let it work overnight. This is a good time to work on something else.

Folders and Json

Google Takeout creates directories for each album but it also creates albums per year. This means that plenty of files are duplicated, at the very least.

My natural instinct, as a media asset manager was to see this file organisation and tidy up. Don't make this mistake if you're going to use Immich CLI or Exif tool. The mistake is not critical but if you want to save time then it is not required.

The JSON files that are created by Google Takeout expect the folder structure that it creates. If you tidy things up then Exif tool will look for files and not find them because it's looking in "Google Takeout 5" whereas you have everything in "Google Takeout". The result is that exif will fail to update exif data for thousands of photos.

Getting AI Help

Using the Exif tool might be daunting, and that's where Google Gemini or another AI solution can help. Sometimes you can read the fabulous manual but feel confused. With Gemini, Le Chat by Mistral, or other you can ask for help and it will.

Of course, with AI, it might say "Use this prompt" and it might be wrong. That's why it's good to test on a small folder first, see if there are any errors. If you see there are errors you can fix them. When things work well, then you can try in a second and third directory.

The Limitation

With event specific folders that span a single folder this approach worked fine but it's when I was exporting "photos from 2010" that I noticed that it was looking for the folder structure that I had habitually destroyed as a media asset manager.

That's why you keep the zip files. With the zip files intact you can extract the files and folders and start again, this time knowing that the command prompt is the right one and you can let Exif tool work in the background.

Double Checking

When Exiftool is done there are three ways to see if the operation was a success. The first is the summary after running the command. The second is to check individual files to see if the creation date is correct and the first test is to upload to immich and see if the files are uploaded with the correct date and time. When these three steps are complete you can scale up.

Populating Immich

The Fun part is when you start populating Immich. It is gratifying to see images sorted by date and time and for people to be recognised, and for us to see if we remember the names of friends from 15-20 years ago. It's also fun to see how mobile phones and photo cameras have improved over the years. It's also nice not to be trapped by iCloud or Google Photos anymore.

Have a Test Instance or a Test Account

I recommend having a test instance of Immich or at the very least a test account. If something goes wrong it's better to see it go wrong on a test instance rather than in production. I'd rather make sure all photos load into the right year, before seeing thousands of photos added to the day the Google Takeout archive was created.

Things to Watch Out For

• Duplicates: If you edited the rotation, adjusted histograms or anything else you might have three or four copies of the same image showing up.

• Creation Date: With some photos I found that they are sorted by the creation date for the zip, rather than their original creation date. In these cases it's worth checking to see if they are duplicates or if you need to correct the date and time information.

• Keep the Zips: If you make a mistake you can start over

• Keep the file and folder structure: The JSON files are optimised for Google Takeout Folder structures.

• Test Small and then scale up

And Finally

When I hit the 200 GB limit many years ago I upgraded to the 2TB plan and I was happy with it, but I didn't use all of that space so I was paying for more than I needed. As many people have said before me, if iCloud, Google One and other solutions had more tiers then we could upgrade one small step at a time, rather than catapulting from 200 GB to Terabytes.

Another point is that with larger file export sizes breaking out of Google Photos is almost easy, in contrast to before. When you had too download hundreds of zip files, unzip them, and then re-consolidate everything you were speaking of a huge investment in time. Now, within a day or two, you can be out of Google Photos and into Immich.

One of my key reservations about using cloud synching tools as a backup solution is that it was very hard to get data out, whereas now it is considerably easier.

Upon completion of this process, if I tidy up my e-mails I could revert to Google's free tier.

Me he hecho un script rapido en python para pasar los mensajes de #Gmail bajados en formato #mbox con #googletakeout con fecha a partir del 1 de enero de 2023 a un nuevo fichero mbox para importar en @Tutanota el mobox completo de 15GB ya está #7z con contraseña en un @veracrypt si todo va bien compartiré el python en un gist (es mu tonto y muy a lo rápido eso si).

#dataprivacy #ethicaldata

Break free from Big Tech 🔓 ! Use Google Takeout to liberate your data and migrate to #Nextcloud for more control and privacy.

Here's how: https://cloud68.co/blog/migrate-from-bigtech-to-nextcloud-with-googletakeout

#GoogleTakeout #RiseAgainstBigTech

Reading Time: 5 minutes

Yesterday I started the proper migration of my Google Photo assets from Google Takeout to PhotoPrism. The first step was to mount the drives to the linux system, the second was to transfer the photos from the external hard drive to the internal SD card, unzip them, and then start imposing assets.

The first bottle neck is exporting 800 gigabytes from Google drive to a local drive. I chose to download the files in one gigabyte packages in fifty gigabyte sets over many hours. This depends on your connection so the experience will vary.

Moving Between Disks

The second bottle neck is when moving the files from an external hard drive to a microsd card. The transfer can be quite time consuming which is part of my reason for using a Pi, rather than a laptop. A laptop would be much faster but it will time out unless you tell it not to sleep. The issue with that is that laptops are not designed to be un for days at a time, without sleeping every so often. Once the Pi is working you can leave it to work.

Ideally I would keep the files on the external drive and skip this time consuming step. I wanted to test the feasability of using an SD card, to keep things tidy once the time consuming phase is over.

Unzipping

PhotoPrism needs files to be unzipped to work. This can be a time consuming task if you do so with the Pi, rather than a laptop or desktop. I would recommend unzipping the files ahead of moving the files from an external drive to an internal drive. At the time of writing I did not find a quick way of unzipping files with a single command.

Importing From the Import Folder

PhotoPrism has an import folder. This is where you extract your unzipped Google Takeout Folders to. Select “move files” to delete all files that have been automatically imported. Click import and then PhotoPrism does the rest. This is the stage that takes the most time. PhotoPrism orgqanises the photos by year, month, date, location, tags, and people. This is the stage where you can go for a walk, or enjoy a good night of sleep because it will take hours to complete.

I am not clear whether the JSON files for images are always zipped within the same archive or whether they are zipped once every few files. This is part of my reason for experimenting with ingesting several gigabytes at a time, rather than one folder at a time. The second reason is that if I give it tasks that take hours it gives me no excuse to procrastinate. That’s what I did before writing this blog post.

Recognising Faces

PhotoPrism recognises faces but it doesn’t create a “person” until it has several images. When it does have several images it gives you the opportunity to name that face, or add it to a pre-existing face. It’s nice to watch as faces from your past re-appear, and amusing when you realise how many names you have forgotten, as well as seeing which ones you clearly remember.

Recognising Places

At the moment when I am writing this post it has recognised 200 places and 15 states. When it recognises a place you can search by country via the search tool, or by location with the map. States are towns, villages or regions. This is a useful way of organising photos because it shows how much you travel, but also how many places you have been to, once you zoom in enough.

Cameras

This tool, by reading EXIF info, makes it possible to search for photos by camera. You can search for 360 photos with insta360 or photos from years ago with canon s70 or Canon EOS 5 MkII etc.

Categories

If you’re looking for photos of food, or aircrafts, or goats, or sheep, or monuments, or historical, you can. It also allows you to seach by colours, for example teal for grass and diving or blue for sunny days and more.

Years and Months

It is possible to search by year or by month, or both. You can search for December images to see if you can find images of snow, or you can search for June and yellow to find images of drought.

Import and Log

Two tabs that I spent time watching last night when I first started to import Google Photos properly were “import” and “log”.

The import tab is important because it allows you to know whether files are being imported or not so this allows you to decide whether to import more images or not. I prefer to import 50 gigabytes, clear the import directory, and then import the next 50 gigabytes, rather than to leave it to run for hours and assume that everything was imported correctly. If something fails I like to know when it’s easier to fix. So far I am under the impression that photos are, at the very least, imported into folders by year and month, as well as indexed automatically based on content.

I like to watch the logs to see if errors occur, but also to see when new faces are recognised, whether thumbnails are being generated correctly as well as when a new face cluster is ready to be named.

Making Photos and Videos Private

With PhotoPrism you can set several images or videos as private at once. You select the first image and then before clicking the last image press shift as you click and it will select that range. You can then click on the lock and those images or videos will be marked as private. Having the ability to select a range of images and apply changes, such as privacy is useful. I use privacy as an example but you could just as easily add a country, or keyword, or anything else.

The Index Tab

I realise that I should mention the index tab. Indexing runs automatically, as soon as it detects images in a directory or a mounted drive it will attempt to index those files which is both useful and likely to cause a mess. Luckily PhotoPrism comes with a “cleanup” tool to make clearing up orphan files and tidying the index easier.

It’s Fast

Despite indexing thousands of files, adding metadata, recognising colours, adding categories and labels, as well as generating thumbnails it’s fast. I can load images almost instantly when scrolling up and down. This is despite running on a Pi whilst it’s working hard. This is a great self-hosted alternative to Google Photos and iPhotos. Remember, before cloud storage was a default tools like Picasa existed, and these were great for organising photos. The difference is that now they’re cloud solutions where the cloud is your home Raspberry Pi rather than an app on your phone or laptop.

User Roles

When you use PhotoPrism for free you can have just one user. If you pay 2 Euros per month you can have “Super Admin, Admin, User, Viewer, Guest” accounts. This allows you to create individual users, to add friends and family.

Cloud Option

If you don’t desire to have your own private instance running at home you have a cloud based solution that starts at 6.50 Euros per month but it’s unclear what the cost is for storage.

And Finally

Initially I thought that I would use Nextcloud for media asset management but when it failed to display video files as thumbnails and when I saw that I couldn’t easily get rid of orphan index entries I hesitated between Immich and PhotoPrism. What made me commit to PhotoPrism is that I saw that they had a tool to import from Google Takeout built in. Instead of spending hours or even days or weeks re-organising photos the software would do it for me.

Adobe Lightroom costs 99 CHF per year. Kyno by LessPain Software costs 150 Euros per year and CatDV costs several thousand CHF to purchase. If you know how to setup PhotoPrism you can save money, or upgrade the hard drive to a higher capacity once per year, to ensure less risk of drive failure due to age.

https://www.main-vision.com/richard/blog/migrating-media-assets-from-google-photos-to-photoprism/

#bottleNecks #day415 #googlePhotos #googleTakeout #photoprism #workflow

@brunomiguel yea I'd prefer anonymized 3rd party data like the telcos have so we didn't have to guess about the biases in the self-reported survey data at all. Wish some nonprofit had a way to #anonymize and aggregate data voluntarily shared by users with researchers (#googletakeout #facebookexport).

#ActuLibre Google Accidentally Shared Private Videos of Some Users With Others -> http://feedproxy.google.com/~r/TheHackersNews/~3/dXVZsXxs3Mo/google-photos-videos.html #googlephotosapp #cybersecurity #GoogleTakeout #PrivacyBreach #Vulnerability