Imports from Gmail
Create access credentials | Google Workspace | Google for Developers
If you need to access multiple user mailboxes via IMAP to backfill the email Archive via the “Mailbox Reader” service, you MUST set up a Google Service Account with the authority to do this. Otherwise, you will be required to enter the password for every user mailbox that you need to access.
A Service Account is not interactive and therefore cannot perform 2 Factor Authentication. So, it is configured to use OAuth with a special key file. In the future, a more dynamic access control method will be available – but for now an access key file will be used.
Return to your Google Cloud Project (https://console.developers.google.com/apis) (see part 1 above), and access the credentials panel. Here we select the “Create Credentials” button again – like we did earlier to create the OAuth Client – but this time we will create a “Service Account”.
Note – this selection will route the browser to the “IAM & Admin” section of your Google Cloud Project.
The Service Account Name that you enter here will be used to create a “Service Account ID” – which looks like a very long email address.
The remaining details can be left to default, or you can add a description.
Here is a completed Service Account that will be used for the YOUR ARCHIVE’s Mailbox Reader service:
Linking the Service Account to your Workspace
You now have a Service Account – but it is not yet associated with your Workspace. It is SEPARATE to the OAuth configuration that you made earlier. So now we need to connect the Service Account to your Workspace. We do this in a very similar way to the method we did for OAuth.
You will need to obtain the Client-ID value of your Service Account as follows:
In the API Project area, where you created the Service Account, click on the Service Account Email to access the Configuration Panel of this account:
In the Service Account Details panel is a Unique ID. We will use this as the ID in your Workspace.
Copy this Unique ID, which we will now register in your Workspace as a Domain Wide Delegate.
Add Service Account as a Domain Wide Delegated app
You will now need to return to your Workspace Admin. If you have multiple Workspace Domains that you wish the Email Archive to access, then you can register the SAME Service Account into each of them.
This is the most critical step! Without this, the mailbox reader will display a connection or permission error.
In your Workspace Admin, navigate to Security > API controls > MANAGE DOMAIN-WIDE DELEGATION
On the Domain Wide Delegation Panel, click the Add new option.
On the “Add new client ID” panel paste in that Unique ID of the Service Account (that you copied just a few pages back!), then enter the OAuth scopes as https://mail.google.com,email,profile.
The “OAuth scopes” defines the areas of the Workspace for which you are allowing the Service Account to access. If you search for Google Workspace Scopes, you will find that there are several to choose from. But for the Email Archive, it only needs to access user mailboxes and their emails. So, we only need to grant those 3 scopes:
Email
Profile
Once these have been Authorised, then – and only then – will the Service Account be granted access to User Mailbox data.
Service Account Key
In order to use the Service Account in an application you will need to obtain a Key File. The key holds details that allow the connecting application to authenticate without needing user interaction (no password or 2FA prompts). For our purposes, we create a Key file and register this key in the Archive so it is able to read the content of user mailboxes.
For the Security account that you have added, in the IAM and admin panel select the “Keys” tab and add a Key.
Click the “ADD KEY” button.
Obtain the JSON file for this key. As it states here, look after this file but do not copy it widely, as it can be used by anyone to remotely access your Workspace services that it has the scopes for.
The Service Account must now be registered as an API Service Account – just like we did for the OAuth.
YOUR ARCHIVE Mailbox Reader settings
From YOUR ARCHIVE version 9.6.1, it is now possible to access multiple Google Workspace mailboxes using the Service Account Key.
To use a Service Account in YOUR ARCHIVE’s Mailbox Reader, please do the following:
Add a Mailbox Reader connection, and select the IMAP protocol.
Enter “imap.gmail.com” as the server
Set the Port to “993” (this is an encrypted port).
Unless you only require a specific set of mail from a known folder, Set the Include Folders to “[Gmail]/All Mail”.
Blank out the Exclude Folders.
Tick the User Google Service Account option
And copy paste the complete text of the JSON key file into the “service account key” field. You can open the Key File in Notepad – it is just plain text. Ensure that any blank spaces, rows, n/ are removed and that the opening and closing brackets for JSON are in place.
After this connection is Saved, the system will display the Client Email / Client ID and Private Key Id values extracted from the Key File JSON text. These details can be shared publicly – they are not the secret portions of this data.
Set the required Date Range to collect. Typically this will be: “All Mail Up-To” and set the end date to the date/time that Google’s BCC/Journaling was enabled. WARNING: If set to “All Mail” then the system will continually re-connect to user mailboxes fetching any new mails - forever. This is probably NOT the correct mode to use if you have enabled the Google Journaling options.
Now save the connection and move on to entering the accounts to collect mail from.
You are now ready to start importing mail from Google Workspace Mailboxes into you archive.
User Configuration
Now that we have the IMAP connection with that Service Account Key, we can specify the set of mailboxes to read. Click the User Configuration menu. Here you have 2 choices to add mailbox names:
Manual Entry, for adding 1 mail box at a time.
Import a list of accounts from a CSV file.
Because we are using IMAP, the Username and the Email Address values will be the same. For each mailbox, simply enter the email address for both Username AND Email Address.
If you test by using the Manual entry – then you will see a “Test” link on the right of this account. Press this to see if the Mailbox Reader can successfully connect to this account. PLEASE use this Test link against a user’s account rather than the Test Connection on the Mailbox Reader Connections panel.
If this test is successful, then just click the “Start Download” button.
Creating a csv list of mailboxes
You can Export the list of Mailboxes from your Workspace Admin area. Trim the export down to JUST the set of Primary Email Addresses. Using Excel, you can copy the column of email addresses so you end up with 2 columns of the same email addresses.
You can then import this list into the Mailbox Reader.
Running and Checking Progress
After clicking the Start Download:
Some accounts show “Waiting” or “Not Started”. Just wait for these to get queued.
Some accounts show “Running”. These accounts are actively being read and emails being archived.
Some accounts show “Completed”. These have finished downloading mails.
Some may show “Exited” – though many emails may have been collected, if multiple attempts to download emails raises error responses from Google it will eventually stop with the Exited status.
“Connection Failed” – this means that the system was unable to access that account for some reason.
“Sleeping” – this means that the Connection Settings does not have an END DATE set – it is a “polling” connection. PLEASE USE JOURNALING instead of a polling mailbox reader.
For any account that shows Running, Completed, Sleeping or Exited, you can see the progress the mail reader made in that account by pressing the Probe or History link.
This pop-up panel (here via the probe link) shows the details of the last time this account was read.
Last updated