BUILDING FINANCIAL STOCK SCANNER WITH RUBY ON RAILS AND R. PART 19. GOOGLE DRIVING.

In the previous post we have learned how to generate a PDF view of the List model. Our next task is to integrate our app with the cloud, so that we can store and view our reports externally.

For that I decided to use Google Drive. There is no particular reason why that and not Dropbox.

My original code for that had used an excellent google_drive gem.

It looked like this:


def send_to_drive
  require 'open-uri'
  require 'google_drive'
  session = GoogleDrive.login("my_email@gmail.com","MyVeryHard2GuessPassword")
  List.each do |l|
    list_name = l.name
    open("/tmp/#{list_name}.pdf",'wb') do |file|
      file << open("http://172.31.177.2/lists/#{list_name}.pdf").read
    end
    file = session.file_by_title("#{list_name}.pdf")
    file.update_from_file("/tmp/#{list_name}.pdf")
  end
end

Then, back in April 2015, Google made changes to the API and version 0.3.x of the gem stopped working.

With the new version 1.0.x I could not figure out how to avoid copy-pasting authorisation code each time I run the script. Plus I wanted to be able to upload my PDFs into some directory other than the root one.
Therefore I decided to dig into the actual Google Drive API and write my own wraparound.
For that task I have formulated my requirements as being equivalent to these two BASH commands:

mkdir -p /some/destination/dir
cp /some/source/dir/*.pdf /some/destinaiton/dir

In plain English I wanted:
1. Be able to create any directory/subdirectory structure – including parents.
2. Copy a local file to that subdirectory. If file already exists – overwrite it.

The end result looked like this:

def send_to_gdrive(site_url,tpm_dir,remote_dir)
  require 'gdrive'
  require 'open-uri'
  gd = Google_drive.new
  List.each do |list|
    local_file = "#{tpm_dir}/#{list.name}.pdf"
    pdf_url = "#{site_url}/lists/#{list.name}.pdf"
    open(local_file,'wb') do |file|
      file << open(pdf_url).read end gd.copy_to_gdrive(local_file, remote_dir) 
  end 
end 

And here is my Google_drive class for that:

 
class Google_drive
  require 'google/api_client'
  require 'google/api_client/client_secrets'
  require 'google/api_client/auth/installed_app'
  require 'google/api_client/auth/storage'
  require 'google/api_client/auth/storages/file_store'
  require 'fileutils'

  APPLICATION_NAME = 'Stock Scanner'
  CLIENT_SECRETS_PATH = 'client_secret.json'
  CREDENTIALS_PATH = File.join(Dir.home, '.credentials',
                               "stock-scanner.json")
  SCOPE = 'https://www.googleapis.com/auth/drive'

  def authorize
    FileUtils.mkdir_p(File.dirname(CREDENTIALS_PATH))
    file_store = Google::APIClient::FileStore.new(CREDENTIALS_PATH)
    storage = Google::APIClient::Storage.new(file_store)
    auth = storage.authorize
    if auth.nil? || (auth.expired? && auth.refresh_token.nil?)
      app_info = Google::APIClient::ClientSecrets.load(CLIENT_SECRETS_PATH)
      flow = Google::APIClient::InstalledAppFlow.new({
        :client_id => app_info.client_id,
        :client_secret => app_info.client_secret,
        :scope => SCOPE})
      auth = flow.authorize(storage)
      puts "Credentials saved to #{CREDENTIALS_PATH}" unless auth.nil?
    end
    auth
  end

  def initialize
    @client = Google::APIClient.new(:application_name => APPLICATION_NAME)
    @client.authorization = authorize
    @drive = @client.discovered_api('drive', 'v2')
  end

  def get_file_id(file_name, parent_id)
    file_id = nil
    results = @client.execute!(
      :api_method => @drive.files.list,
      :parameters => { :q => "title \= '#{file_name}' and trashed \= false and '#{parent_id}' in parents"})
    file_id = results.data.items.last.id unless results.data.items.empty?
  end

  def create_folder(folder_name, parent_folder_id)
    file = @drive.files.insert.request_schema.new({
      'title' => folder_name,
      'mimeType' => "application/vnd.google-apps.folder",
      'parents' => [{'id' => parent_folder_id}]
    })
    result = @client.execute(
      :api_method => @drive.files.insert,
      :body_object => file
      )
      if result.status == 200
        return result.data.id
      else
        return nil
      end
  end

  def get_or_create_parent_id(folders)
    parent_id = 'root'
    return parent_id if folders.length == 0
    folders = [folders] if folders.is_a?(String)
    folders.each do |folder|
      result = self.get_file_id(folder, parent_id)
      if result == nil
        parent_id = self.create_folder(folder, parent_id)
      else
        parent_id = result
      end
    end
    parent_id
  end

  def insert_file(local_file, file_mime_type, parent_id)
    file_name = File.basename local_file
    file = @drive.files.insert.request_schema.new({
      'title' => file_name,
      'mimeType' => file_mime_type
    })
    file.parents = [{'id' => parent_id}] unless parent_id.nil?
    media = Google::APIClient::UploadIO.new(local_file, file_mime_type, file_name)
    result = @client.execute(
      :api_method => @drive.files.insert,
      :body_object => file,
      :media => media,
      :parameters => {
        'uploadType' => 'multipart',
        'alt' => 'json'}
      )
      if result.status == 200
        return result.data.id
      else
        return nil
      end
  end

  def update_file(local_file, file_mime_type, file_id)
    file_name = File.basename local_file
    result = @client.execute(
      :api_method => @drive.files.get,
      :parameters => { 'fileId' => file_id })
    if result.status == 200
      file = result.data
      file.title = file_name
      file.mime_type = file_mime_type
      media = Google::APIClient::UploadIO.new(local_file, file_mime_type, file_name)
      result = @client.execute(
        :api_method => @drive.files.update,
        :body_object => file,
        :media => media,
        :parameters => {
          'fileId' => file_id,
          'newRevision?' => TRUE,
          'uploadType' => 'multipart',
          'alt' => 'json'}
        )
      if result.status == 200
        return result.data.id
      else
        return nil
      end
    end
  end

  def get_mime_type(file)
    ext_to_mime_type = {
      ".csv" =>"text/csv",

      ".doc" =>"application/msword",
      ".docx" => "application/vnd.openxmlformats-officedocument.wordprocessingml.document",
      ".htm" =>"text/html",
      ".html" =>"text/html",
      ".ods" =>"application/x-vnd.oasis.opendocument.spreadsheet",
      ".odt" =>"application/vnd.oasis.opendocument.text",
      ".pdf" =>"application/pdf",
      ".png" =>"image/png",
      ".ppt" =>"application/vnd.ms-powerpoint",
      ".pps" =>"application/vnd.ms-powerpoint",
      ".rtf" =>"application/rtf",
      ".swf" =>"application/x-shockwave-flash",
      ".sxw" =>"application/vnd.sun.xml.writer",
      ".txt" =>"text/plain",
      ".tsv" =>"text/tab-separated-values",
      ".tab" =>"text/tab-separated-values",
      ".xls" =>"application/vnd.ms-excel",
      ".xlsx" => "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet",
      ".zip" =>"application/zip"
    }
    mime_type = ext_to_mime_type[File.extname(file)]
  end

  def copy_to_gdrive(local_file, remote_path, file_mime_type)
    # convert '/foo/bar' into ['foo','bar'] or '/' into []
    parent_folders = remote_path.gsub(/^\//, '').split('/')
    parent_id = get_or_create_parent_id(parent_folders)
    file_name = File.basename(local_file)
    file_mime_type = self.get_mime_type(local_file)
    file_id = get_file_id(file_name, parent_id)
    if file_id
      update_file(local_file, file_mime_type, file_id)
    else
      file_id = insert_file(local_file, file_mime_type, parent_id)
    end
    file_id
  end

end

First you will need google-api-ruby-client gem.
At the time of writing version 0.9 was still in alfa, therefore I used 0.8.
You will also need follow the quickstart instructions on how to enable Drive API, generate and save client_secret.son

Now let’s look at the Google_drive class and it’s instance methods.

authorize and initialize are almost exact copy/paste from the quickstart guide.
It checks if authorisation credentials to use Drive API exist and valid. If so a new client session is initialized and API is discovered.
If there are no valid credentials exist then OAuth2 authorisation request is generated and a page with the request approval will be loaded in a default browser session.
get_file_id uses file.list API method with a search query (name, not in a thrash and a parent id) to locate a file and return it’s ID. If no file located then this method returns nil.
create_folder actually uses file.insert API call to create a file with a MIME type of “application/vnd.google-apps.folder” which represents Google Drive folder.
It returns folder ID or nil (if unsuccessful).

get_or_create_parent_id takes an array of folders as an argument (i.e. /some/remote/folder is represented as [‘some’,’remote’,’folder’]. Then it uses the above two methods to crawl folders starting with root to find an ID of the last one. If one or more folders along the path do not exist – they are created on the fly.

insert_file – once we know the parent folder ID, we can make another file.insert call – this time to copy content of our local file to the Google Drive.

update_file  makes files.update call to overwrite file by a given file ID.

get_mime_type translates given file extension into a MIME type.

copy_to_gdrive uses the above methods to make decision if a remote file is to be created or to be updated. It also accepts remote path as a traditional notation string (“/path/to/destination”).

Now let’s update rake tasks list (lib/tasks/asx.rake):


desc "send PDFs to Google Drive"  
task :send_to_gdrive => :environment do
  $SITE_URL = APP_CONFIG[:site_url]
  require 'send_to_gdrive'
  send_to_gdrive($SITE_URL,'/tmp','/Stock Scanner Reports')
end

Running the task:


vint@belka:~/sscanner/lib/tasks$ rake exchange:asx:send_to_gdrive
(in /Users/vint/sscanner)
...
vint@belka:~/sscanner/lib/tasks$

Success! Let’s see what we have at the Google Drive:

Screen Shot 2015-09-03 at 10.32.15 PM

We have a couple of test lists – both ended with the PDFs generated and sent to the Google Drive.

Viewing the first one:

Screen Shot 2015-09-03 at 10.36.35 PM

Looks good. We almost there. Please come back!

Advertisements
BUILDING FINANCIAL STOCK SCANNER WITH RUBY ON RAILS AND R. PART 19. GOOGLE DRIVING.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s