Building A Financial Stock Scanner With Ruby On Rails and R. Part 20. On the Home[Page] Run.

This is a 20th Jubilee post. And we are on the home run. There is one last major feature that is left to be implemented – a home page that will display few of the Australian stockmarket financial indices, namely – All Ordinaries (AORD), Small Ordinaries (AXSO) and the Industry sector indices listed here. This will provide ‘at a glance’ view of the ASX/Sectors performance. For the charts I will use the same GMMAs as for the companies.

First thing, we need another model for an Index. If you remember my diagram in the Part 3, the original ideal was to have it as a completely separate from the Company model, but with one to many relationship with the same Price model. My line of thought back then, perhaps, was – an Index is not a company, it does not have all the attributes of the company(i.e. ‘active’, ‘ignore’, ‘industry’ – well it is in the name). So let’s have it as a new model. But wait, index prices (Open, High, Low, Close) and volume are essentially the same as for the  company stock prices. So let’s point both of them to the same Price model.

Bad idea! It is not DRY at all, as you would need to code price data loader and charting scripts all again. You can’t reuse existing ones as they have references to the Company model all over the place.

Sure you can pass model name as a parameter, but that’s ugly.

The more I thought about it the more my hidden Rembrandt (somewhere deep inside me) rebelled against going that direction.

But what instead? Well, let’s think again about companies and indices. They are not that dissimilar, right? Both have tickers, names, prices. We can even extend concept of ‘ignoring’  companies in our searches to indices too.

Perhaps we can re-use the Company model for indices too? Just add another attribute  – say :index (false/true) to our existing model.

But then we would need to take that attribute into consideration each time we work with the database. For instance to display ASX companies charts we would need to search for those ones that are not ignored and are not indices. It’s possible, but still awkward.

We can use scopes, and have a scope for a ‘true’ company and a scope for an index defined in the Company model. Then we can query ‘Company.companies’ and ‘Company.indices’. Nah, that does not read good.

So, let’s think again what we want. Ideally –

  • we should be able to query Company.all and Index.all separately
  • we should populate prices for both models in one go
  • generate charts for both models in one go too, and
  • preferably, do not change the code too much

And the answer is…

(drums beating…)

… inheritance.

Yep, it’s the model inheritance, which is a simple but powerful concept in RoR.

Let’s introduce a top level, called, mmmm.. ‘Entity’ (sorry, could not find a better word for that – I guess no Shakespeare  inside me today… or yesterday).

Entity will have one-to-many relationship with the Price model. It will also have most of the attributes (including the default scope) of the current Company model.

Then we will have new lean Company model and a completely new Index model, which will inherit from the Entity model.

That will gives us all the flexibility we need. For instance we can do both ‘Entity.find_by(ticker: ‘NAB.AX’)’ and ‘Company.find_by(ticker: ‘NAB.AX’) – and we get the same result.

Behind the scene Company and Index will use the same single ‘entities’ MongoDB collection. When creating new company or index records Mongoid will automatically add an additional _type attribute that will refer to the child model.

This means we will be able to work with the Entity model when we want to do something with the both companies and indices, for instance update prices, or generate charts. At the same time we can work with just the Company or just the Index models – for instance, when listing them on the web site.

Here is how we do it.

First, let’s scaffold Index and Entity models/controllers/views:

vint@belka:~/sscanner$ rails g scaffold Index


vint@belka:~/sscanner$ rails g scaffold Entity


This is our new Entity model:

class Entity
include Mongoid::Document
field :ticker, type: String
field :name, type: String
field :roar, type: Float
field :ignore, type: Boolean, default: false
field :_id, type: String, default: ->{ticker}
index({ticker: 1}, { unique: true })

belongs_to :exchange
has_many :prices

default_scope ->{ where(ignore: false) }


Here is our ‘lean’ Company model:

class Company < Entity
field :industry, type: String
field :active, type: Boolean

And this is our new Index model:

class Index < Entity
      field :category, type: String

The ‘category’ attribute will, for now, have value of ‘ordinary’ or ‘GICS’.


Remember that we only interact with the Price model for the two reasons:

  • to load day prices.
  • to generate JSONs for R/Quantmod

We talk about the fist task later – when discussing changes to the price loader script.

For the second task, since we want the same code to generate JSON for both companies and indices, there is a small change required to the prices_controller:

  def index
    @e_ids = params[:entity_id]
    @e_ids = [@e_ids] if @e_ids.is_a?(String)
    @prices = Price.where(entity_id: {'$in': @e_ids}) if @e_ids
    respond_to do |format|

Here we changed params[:company_id] to params[:entity_id] and changed name of the array from ‘@c_ids’ to ‘@e_ids’ – just to be consistent.

Loader scripts

There is a new load_indices.rb script, used to seed indices in the entity collection. It is not an ideal solution, as it has hardcoded tickers listed inside. It will do for now, but, perhaps, I need to mode that function to the web frontend – possible future ‘Settings’ section.

def load_indices
  indices = [
    {ticker: '^AORD', name: "All Ordinaries", category: "ordinary"},
    {ticker: '^AXSO', name: "Small Ordinaries", category: "ordinary"},
    {ticker: '^AXEJ', name: "Energy", category: "GICS"},
    {ticker: '^AXMJ', name: "Materials", category: "GICS"},
    {ticker: '^AXNJ', name: "Industrials", category: "GICS"},
    {ticker: '^AXHJ', name: "Healh Care", category: "GICS"},
    {ticker: '^AXDJ', name: "Consumer Discretionary", category: "GICS"},
    {ticker: '^AXSJ', name: "Consumer Staples", category: "GICS"},
    {ticker: '^AXFJ', name: "Financials", category: "GICS"},
    {ticker: '^AXIJ', name: "Information Technology", category: "GICS"},
    {ticker: '^AXTJ', name: "Telecommunications", category: "GICS"},
    {ticker: '^AXPJ', name: "A-REIT", category: "GICS"},
    {ticker: '^AXUJ', name: "Utilities", category: "GICS"}

   indices.each do |i|
       :name => i[:name],
       :ticker => i[:ticker],
       :category => i[:category]
     }) unless Index.where(ticker: "#{i[:ticker]}").exists?
  return true

Next, we need to update update_prices.rb to change word Company to Entity.

module Update_prices

  require 'csv'
  require 'progress_bar'
  @entity_count = Entity.count
  @bar =

  class Dat <, :body)

  def Update_prices.finance_yahoo_url(ticker, start_date, end_date)
      url = ""
      url << "/table.csv?s=#{ticker}&g=d"
      url << "&a=#{start_date.month - 1}"
      url << "&b=#{start_date.mday}"
      url << "&c=#{start_date.year}"
      url << "&d=#{end_date.month - 1}"
      url << "&e=#{end_date.mday}"
      url << "&f=#{end_date.year.to_s}" url.gsub('^','%5E') end def Update_prices.get_prices(max_days) per_batch = 20 0.step(@entity_count, per_batch) do |offset| response = prices_array = hydra = => 20)
          Entity.asc(:ticker).limit(per_batch).skip(offset).each do |e|
              ticker = e.ticker
              end_date =
              prices = e.prices.asc(:date)
              if prices.empty?
                  start_date = end_date - max_days
                  last_record = prices.last
                  start_date = + 1
              next if start_date > end_date
              url = Update_prices.finance_yahoo_url(ticker,start_date,end_date)
              request =, :method => :get)
              request.on_complete do |resp|
                  next if resp.code != 200
                  next if resp.body[0..40] != "Date,Open,High,Low,Close,Volume,Adj Close"
                  dat =
                  dat.entity = e
                  dat.body = resp.body

          response.each do |dat|
              prices_array.concat Update_prices.update_entity(dat)
          Price.collection.insert_many( unless prices_array.size == 0
          response = []
          prices_array = []

  def Update_prices.update_entity(dat)
          entity = dat.entity
          res_array =
, :headers => true).reverse_each do |line|
              date = line[0]
              open = (line[1].to_f * 1000).to_i
              high = (line[2].to_f * 1000).to_i
              low = (line[3].to_f * 1000).to_i
              close = (line[4].to_f * 1000).to_i
              volume = line[5].to_i
              day ={
                  :date => date,
                  :open => open,
                  :high => high,
                  :low => low,
                  :close => close,
                  :volume => volume
              day = {}
           return res_array

Now we need to update app/views/indices/index.html.erb (remember it was created by scaffolding) so that it would print GMMA charts for the indices.

<h4>All and Small Ordinaries Indices</h4>
<% aord=@indices.find_by(ticker: '^AORD') %>
<% sord=@indices.find_by(ticker: '^AXSO') %>

<table >
<th><%= aord.ticker %></th>
<th><%= %></th>
<th><%= sord.ticker %></th>
<th><%= %></th>
<td colspan=2><%=image_tag("^AORD_mma.png") %></td>
<td colspan=2><%=image_tag("^AXSO_mma.png") %></td>
<h4>GICS Indices</h4>
<% @indices.where(category: 'GICS').desc(:roar).each_slice(2).to_a.each do |s| %>
    <% s.each do |x| %>
<th><%= x.ticker %></th>
<th><%= %></th>
    <% end %>
    <% s.each do |x| %>
<td colspan="2"><%=image_tag("#{x.ticker}_mma.png")%></td>
    <% end %>
<% end %>

I used indices to be printed in a two-column table. Hence the use of ‘each_slice’ method.

The last bit for today is to update config/routes.rb to accomodate for the new ‘Entity’ model, as well as to point root of the website to the indices#index view.

Rails.application.routes.draw do
#  resources :indices do
#    resources :prices
#  end
  resources :entities, :constraints => { :id => /.*/ } do
    resources :prices
  resources :lists
  resources :prices
  resources :companies , :constraints => { :id => /.*/ } do
    resources :prices
    collection do
      get 'autocomplete'
  resources :exchanges
  root :to => 'indices#index'

Now we can test it all. But before that we need to re-populate our database with the new entities collection. We, either need to drop the entire database, or, just, companies, prices and lists collections.
After that run:

vint@belka:~/sscanner$ rake exchange:asx:update_companies
vint@belka:~/sscanner$ rake exchange:asx:update_indices
vint@belka:~/sscanner$ rake exchange:asx:update_prices
vint@belka:~/sscanner$ rake exchange:asx:plot_graphs

Now navigate to http://localhost:3000/ and you should get a webpage similar to this:

Screen Shot 2015-10-11 at 8.37.12 PM

That’s all for today. Next we will be discussing moving our stock scanner to production environment.

Building A Financial Stock Scanner With Ruby On Rails and R. Part 20. On the Home[Page] Run.

2 thoughts on “Building A Financial Stock Scanner With Ruby On Rails and R. Part 20. On the Home[Page] Run.

    1. Thanks Jon. Glad you liked it. I am thinking about re-designing the architecture to be more flexible with the approach geared towards the microservices architecture. Can’t promise you when, though. Perhaps you have suggestion what you would like to see in the part 21?


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s