Find out how to get object change notifications in Google Drive and send notifications as messages to Pub/Sub topic
At Clairvoyant, we have been working with GCP (Google Cloud Platform) to solve unique use-cases for several of our customers. Our expertise in multiple clouds and mapping the correct tech-stack to the business problems has helped our customers expedite their cloud adoption.
In my last post, we learned how to configure pub/sub notifications/triggers for Google Cloud storage events. Let us explore how these can be achieved for different source systems, i.e., Google Drive.
Traditionally, if anyone wanted to monitor changes made in file contents or file meta-data, or if any new files had to be added or existing to be deleted, the only option was polling the resources on Google Drive at regular intervals that come with extra network and computed costs.
But don’t worry, GCP has come up with a better solution for this through one of its offerings — Google Drive APIs. Using Google Drive API, you can watch any of these changes in Google Drive resources, like Google Sheets, Google Docs, etc.
Google Drive APIs supports the following change logs:
-
changes in contents of each file within a folder/shared drive
-
changes in metadata (i.e. name change etc.) of file/folder/shared drive
-
changes in file access permissions
Let’s see how you can set up the push notifications for the change mentioned above events with the following simple steps and sample workflow:
Sample Workflow to set up Google Drive notifications and publishing to Pub/Sub for processing
Step 1: Creating a service account in GCP
This is the basic step one needs to follow while working on any GCP feature. NNavigate to Cloud IAM and create a service account with the required roles and permissions to generate a credentials file. The key credential file is used to generate the GCP auth token. This auth token is required to connect to your GCP project and authenticate API calls.
See below the sample credentials file — sample-gdrive-notifications.json:
{ "type": "service_account", "project_id": "sample-gdrive-notifications", "private_key_id": "*************************", "private_key": "-----BEGIN PRIVATE KEY----- ************* -----END PRIVATE KEY-----\n", "client_email": "sample-gdrive@sample-gdrive-notifications.iam.gserviceaccount.com", "client_id": "**************", "auth_uri": "https://accounts.google.com/o/oauth2/auth", "token_uri": "https://oauth2.googleapis.com/token", "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs", "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/sample- gdrive%40sample-gdrive-notifications.iam.gserviceaccount.com" }
Given below is the java code to generate auth token from credentials file:
ServiceAccountCredentials cred = ServiceAccountCredentials.fromStream(getClass().
getResourceAsStream("
/sample-gdrive-notifications.json"));
Instant now = Instant.now();
return Jwts.builder()
.setHeaderParam("kid", cred.getPrivateKeyId())
.setIssuer(cred.getClientEmail())
.setAudience("hostName")
.setSubject(cred.getClientEmail())
.setIssuedAt(Date.from(now))
.setExpiration(Date.from(Instant.ofEpochSecond(now.getEpochSecond() +
9000L)))
.claim("target_audience", cred.getClientId())
.signWith(SignatureAlgorithm.RS256, cred.getPrivateKey())
.compact();
Step 2: Verify if the domain URL used to set up notifications is owned by you using the Google Search Console.
-
Click here to navigate to the search console and hit on start now
-
Click on add property, enter the domain url in the URL Prefix tab and hit continue.
Verification — Step2
-
From the list of verification methods, select the ‘HTML tag’ option and copy the meta tag in the header section of your application and deploy it. Do not click on ‘verify’ yet.
Verification — Step3
-
After the successfully deployment of your domain application hit on ‘verify’ as in the above-shown step. You will see the Ownership Verified message.
Verification Successful
Step 3: Domain Registration
-
Navigate to GCP -> APIs & Services -> Domain Verification and click on ‘Add domain’.
Domain Registration — Step1
-
Enter the above ownership verified domain in the text box and click on ‘Add domain’.
Domain Registration — Step2
-
If the URL is verified, you will see the domain added to the list.
Successful Domain Registration
Step 4: Enabling push notifications by making an API call to watch the request of Drive API
Here is the API which has the watch method associated with the URI:
https://www.googleapis.com/apiName/apiVersion/resourcePath/watch
If you want to watch changes to all the file resources:
POST https://www.googleapis.com/drive/v3/changes/watch
Authorization: Bearer auth_token_for_current_user
Content-Type: application/json
Request Body
{
"id": "98765432-01cdef-9876543210ab", // Your channel ID.
"type": "web_hook",
"address": "https://sample-notification-app.com/notifications", //
Your receiving URL.
"token": "target=myApp-myFilesChannelDest", // (Optional) token.
"expiration": 1426325213000 // (Optional) expiration time.
}
If you want to watch changes to a specific file resource
POST https://www.googleapis.com/drive/v3/files/fileId/watch
Authorization: Bearer auth_token_for_current_user
Content-Type: application/json
Request Body
{
"id": "98765432-01cdef-9876543210ab", // Your channel ID.
"type": "web_hook",
"address": "https://sample-notification-app.com/notifications", //
Your receiving URL.
"token": "target=myApp-myFilesChannelDest", // (Optional) token.
"expiration": 1426325213000 // (Optional) expiration time.
}
Let’s take a moment to describe each request body attribute here:
id to identify each notification uniquely
type always a web_hook
address call-back web-hook URL to listen and respond to the notification which should always be HTTPS
token optional property to verify each token from the same notification channel
expiration timestamp to identify when to disable/stop the notification channel
Here’s my java code to invoke the watch request for a specific file resource:
String gcpOauthToken = generateEndpointsToken(); //from step 1 NotifyRequest notifyRequest = NotifyRequest.builder() .id("4ba78bf0-6a47-11e2-bcfd-0800200c9a77") .type("web_hook") .address("https://sample-notification-app/notifications") .token("qwertyuiop-9867542301") .expiration(1426325213000) .build(); HttpHeaders httpHeaders = new HttpHeaders(); httpHeaders.setContentType(MediaType.APPLICATION_JSON); httpHeaders.add(AUTHORIZATION, "Bearer " + gcpOauthToken); HttpEntity<NotifyRequest> requestHttpEntity = new HttpEntity<> (notifyRequest, httpHeaders); ResponseEntity<String> responseEntity = testRestTemplate.exchange("https://www.googleapis.com/drive/v3/files/ fileId/watch", HttpMethod.POST, requestHttpEntity, String.class);
Once you get 200 response code for the above request, you have successfully configured Google Drive notifications! Congrats!
Here’s a sample response of the above request:
{
"kind": "api#channel",
"id": "4ba78bf0-6a47-11e2-bcfd-0800200c9a77", // ID you specified
for this channel.
"resourceId": "bx3hgv8765ngfrd", // ID of the watched resource.
"resourceUri": "https://www.googleapis.com/drive/v3/files/o3hgv1538sdjfh", //
Version-specific ID of the watched resource.
"token": "target=qwertyuiop-9867542301", // token provided in the
request
"expiration": 1426325213000, // expiration time provided in the
request
}
Step 5: Processing response data from resource change notifications
-
Store each notification channel id to datastore to keep track of the subscribed notifications and resources for any other requirements like stopping/disabling any specific notification channel.
-
Whenever any watch resource changes its state, your application will receive a notification message in the POST request form to the registered address URI, mentioned in the request.
Here’s the sample notification message format received for changes in file contents:
POST https://sample-notification-app/notifications // Your receiving URL. Content-Type: application/json; utf-8 Content-Length: 319 X-Goog-Channel-ID: 4ba78bf0-6a47-11e2-bcfd-0800200c9a77 X-Goog-Channel-Token: qwertyuiop-9867542301 X-Goog-Channel-Expiration: Tue, 07 Mar 2021 01:13:52 GMT X-Goog-Resource-ID: bx3hgv8765ngfrd X-Goog-Resource-URI: https://www.googleapis.com/drive/v3/changes X-Goog-Resource-State: change X-Goog-Message-Number: 9 { "kind": "drive#changes" }
X-Goog-Channel-ID identifier provided to notification channel in the request
X-Goog-Channel-Token token set by the application when setting up request, used to verify the source of notification
X-Goog-Resource-ID uniquely identifies the watched resource
X-Goog-Resource-State event which triggers the notification [sync, add, remove, update, trash, untrash, or change]
X-Goog-Channel-Expiration expiration date-time of notification
X-Goog-Message-Number message identifier for a specific notification channel
-
The above-received notification message can further be sent to different Pub/Sub topics, which will be consumed by other applications for processing each file with updated contents/data.
Wait, we are not done yet!!
In the future, you may want to stop receiving these notifications before their expiration time for a few resources.
To get this done, Google Drive API has provided a stop method which can be used as shown in the sample below:
POST https://www.googleapis.com/drive/v3/channels/stop
Authorization: Bearer {auth_token_for_current_user}
Content-Type: application/json
{
"id": "4ba78bf0-6a47-11e2-bcfd-0800200c9a77", //channel-id
"resourceId": "bx3hgv8765ngfrd"
}
Now you have all the detailed steps to enable/disable Google Drive changed event notifications for your drive account and application. As discussed above, it can be sent over Pub/Sub topics likewise and you can send it over any other application like slack alerts, email alerts, etc., bundled in the form of reports as required. So, it’s time for you to try this out!
Reach out to us for all your business requirements of cloud based services.