Blake Erickson

Rake Task with an Arbitrary Amount of Arguments

2019-07-12T07:00:00+00:00

I’m currently working on a rake task where I need to pass in an arbitrary amount of values. I couldn’t even remember at first how to even pass in arguments to a rake task, so let’s cover that first.

For this example we are going to create a file called raketask.rb inside of an empty directory. And inside of that file we can create our first task like so:

desc "A sample task"
task :print_hello do
  puts "hello"
end

Once you save and close the file we can run rake -T and it will print out a list of tasks that is knows about:

rake print_hello      # A sample task

And we can call this task with the following command:

rake print_hello
hello

Now let’s make another task with an argument:

desc "Say hi"
task :say_hello, [:name] do |t, args|
  puts "Hello #{args.name}"
end

If we run rake -T again:

rake print_hello      # A sample task
rake say_hello[name]  # Say hi

Unlike command line bash scripts or even calling a ruby script with multiple args you can’t just separate them by spaces in a rake task like this:

ruby my-sample-script.rb arg1 arg2 arg3

Rake task arguments go inside brackets:

rake say_hello[blake]
Hello blake

Now how do we specify multiple arguments? By specifying each one inside of the task declaration:

desc "Two args"
task :two_args, [:fname, :lname] do |t, args|
  puts "Hello #{args.fname} #{args.lname}"
end

And you can run it by specifying args in a comma separated list (with no spaces!):

rake two_args[blake,erickson]
Hello blake erickson

Now onto the tricky part that I’m still not 100% satisfied with, is how do you pass in an arbitrary amount of arguments to a rake task? For my case I need to pass in a list of values, but I’m not sure how many there will be.

To do this you will not specify the args like we did before inside of the task declaration. The problem with this is now it won’t show the args when running rake -T. So to handle an arbitrary amount of args you can use the args.extras method like so:

desc "Arbitrary args, pass in a comma separated list"
task :arbitrary_args do |t, args|
  puts "There are #{args.extras.count} args"
  args.extras.each do |arg|
    puts arg
  end
end

Now you can run the arbitrary args command with:

rake arbitrary_args[banana,apple,orange,mango]
There are 4 args
banana
apple
orange
mango

I’m not totally satisfied with specifying an arbitrary amount of arguments this way, but it does get the job done. Hopefully this helps you in your rake task adventures.

Using Ruby to Save Cookies

2019-07-11T07:00:00+00:00

Like I mentioned in my previous post about saving cookies with cURL in this post I’m going to show you how to do it with ruby.

I found this stackoverflow answer that talks about using http-cookie which doesn’t look like it has been updated in over 3 years, but it did work just fine.

I created the golds_checkin repo to store the ruby code of this project. Currently it is just an app.rb file, but I will most likely turn it into an actual ruby gem.

Here is the output of my app.rb file:

require 'yaml'
require 'net/http'
require 'http-cookie'
require 'cgi'

class GoldsCheckin
  def initialize
    @config = YAML.load_file('config.yml')
    @jar = HTTP::CookieJar.new
  end

  def login
    uri = URI('https://mico.myiclubonline.com/iclub/j_spring_security_check')

    Net::HTTP.start(uri.host, uri.port, use_ssl: true) do |http|
      req = Net::HTTP::Post.new uri
      req.form_data = { "j_username": @config['username'], "j_password": @config['password'] }
      res = http.request req
      res.get_fields('Set-Cookie').each do |value|
        @jar.parse(value, req.uri)
      end
    end
  end

  def fetch_checkins
    low_date = CGI::escape(Time.now.strftime("%m/%d/%Y"))
    high_date = CGI::escape(Time.now.strftime("%m/%d/%Y"))
    uri = URI("https://mico.myiclubonline.com/iclub/account/checkInHistory.htm?lowDate=#{low_date}&highDate=#{high_date}")

    Net::HTTP.start(uri.host, uri.port, use_ssl: true) do |http|
      req = Net::HTTP::Get.new(uri)
      req['Cookie'] = HTTP::Cookie.cookie_value(@jar.cookies(uri))
      res = http.request req
      res.body
    end
  end


end

golds_checkin = GoldsCheckin.new

golds_checkin.login
puts golds_checkin.fetch_checkins

Basically there are two methods, ‘login’ and ‘fetch_checkins’. ‘login’ reads my credentials out of a yaml file and makes a POST request wich then sets the returned cookies into the @jar instance variable so that we can use it for subsequent requests. Once we are logged in then we can call ‘fetch_checkins’ which I have set to only fetch today’s date. This call reads from the cookie jar in order to make an authenticated request.

From these two posts (this one and yesterdays) you can start to see my process for making an app. I usually start with just playing around on the command line. Then I translate all of that knowledge into a simple app.rb file. From there I think it kind of depends what my end goal is, but then the app.rb file evolves in some way where it no longer just lives on my local computer, but gets turned into a ruby gem to be used in another project, or it evovles into a larger web application.

Using cURL to Save Cookies

2019-07-10T07:00:00+00:00

Just for fun I want to automatically track if I go to the gym or not. Every time I go to Golds Gym I have to scan my card that I keep on my key chain so there is an electronic record of me going to the gym or not. I did not know about this when I signed up, but Golds Gym does have a crappy website for managing your membership and on there you can see a record of every time you have checked into the gym.

The problem I have with this website is that they don’t have an API and it mostly requires javascript to use it. This makes automatically fetching the data I need very tricky. Using Firefox’s developer tools you can inspect the http requests and copy them as cURL so that you can kind of reverse engineer how to authenticate and fetch data.

This works pretty well and I was able to get everything I needed from the command line using cURL commands, but unfortunatley it only worked for the lifetime of the session cookie. Even though I have worked with web technology for many years I actually haven’t done a whole lot with cookies. The copy as cURL command from Firefox will actually copy over all of the cookies into the cURL request, but that wasn’t exactly what I was after. Through some digging around I found a great resource on cURL that mentions how to save and retreive cookies from a text file.

To save a cookie you can use the -c flag followed by the name of the cookie file you want to save. Here is an example:

curl -c cookie.txt https://example.com?creds=asdf

If you then want to use those cookies you can use the -b flag like this:

curl -b cookie.txt https://example.com/data

So now I can run one cURL command to authenticate with my username and password which will save a session token in my cookie file. And then I can run another cURL command using the cookie file to fetch the data I’m after. Pretty cool.

Now my next steps are to do this all in ruby. After that I’m hoping I can add this to my private Discourse instance somehow so that I can be awarded a custom gym badge every day I go or for going so many days in a row.

Bug Crushing Adventure

2018-10-31T07:00:00+00:00

At Discourse we have a new “game” that we play where we have a running bug crushing topic on our internal team Discourse site that is always assigned to someone. If this topic is assigned to you then you need to look through the application logs of all the sites that we host to find a bug and then fix it. Once the bug is fixed, post a reply to the topic with the details and then assign the topic to a new person. So far it has been working great and it has given me, a developer who is often busy working on infrastructure related ops work, an opportunity to commit more code changes to the core Discourse app. Also this helps spread the bug fixing responsibility across many more team members so it doesn’t just end up being a couple people fixing all the bugs.

Yesterday was my turn. I actually found a bug that I noticed was happening on a couple of sites. I then spent a large chunk of my day trying to fix it and replicate it locally. The initial hard part was getting the plugin installed locally with the right environment variables. Once I was able to debug the code locally something just didn’t add up. LIke this literally could not be happening. After much frustration and even pairing with my awesome coworker we figured out that this bug was actually fixed 27 days ago. So we then checked the deploy history for those sites and sure enough those sites just hadn’t been deployed since then and didn’t have the fix.

I guess the lesson to learn here is to check git blame to see the last time a file I’m working on was last modified. Maybe I’ll look into something like vim-fugitive so that it is easier for me to see the git blame history without leaving my trusty editor.

The good thing about all of this is that I actually learned quite a bit about the code base by “investing” all of this time trying to fix that bug.

Luckily I was able to find another bug (which I originally was going to talk more about in this post) that was showing up in the logs and crush it.

Rails Collection Routes

2018-10-28T07:00:00+00:00

While working on Pistachio, the todo app I’m building to practice rails, I wanted to work on a new feature that would allow me to order lists. I already had the ability to order items in a list but not the lists themselves. As I was working on this feature I realized that maybe my original route for sorting items wasn’t actually the best route to be using especially since it would mean that I would have two sort routes on my lists controller.

Here is what I originally had for sorting list items in my config/routes.rb file:

resources :lists, only: [:index, :show, :new, :create, :destroy, :edit, :update] do
  patch :sort
end

which means the route looks like this lists/:list_id/sort.

This actually worked fine, but it was a bit weird that I had my sort item logic inside of my lists controller. I’m sure I original did it this way because I wasn’t sure how to make a collection route that would hit my items controller without requiring an item id, nor did I know it was called a collection route that I was looking for.

Before we fix the lists route let’s go ahead and create this collection based sort route under the items resource:

resources :items, only: [:create, :destroy, :edit, :update] do
  patch :complete
  collection do
    patch :sort
  end
end

This will create a route for us that looks like /lists/:list_id/items/sort. Which then means I can move the contents of the current sort method in my lists controller to the sort method of my items controller where it should have been in the first place.

And now we can move the current patch :sort route that is part of the lists resources into a collection so that we can use it for sorting lists. It should now look like:

resources :lists, only: [:index, :show, :new, :create, :destroy, :edit, :update] do
  patch :sort
  collection do
    patch :sort
  end
end

This will give us a route of ‘/lists/sort` which is exactly what we want so that the sort route isn’t tied to any specific route id, but instead to the whole collection of lists.

Finally Turning AudioGrab Into a Rails App

2018-08-26T06:00:00+00:00

6:45 PM

Okay today I’m finally going to make a Rails app for AudioGrab. I think creating a cli app was a great first step, but having a rails app that I could have a bookmarklet talk to is what I really want. The original audiograb I made back in 2013. My current cli version I just opened sourced.

My goal for tonight is to make the most basic rails app and get it deployed to production. All I want it to do is have an input box for the youtube url with a submit button. It then will take that url and download the youtube video, convert it to mp3, and then upload it to my overcast account.

This means that I won’t have user accounts yet, but maybe I can just protect it with basic auth or something.

7:08 PM

Okay I just ran rails new audio_grab_app to generate my new app, ran rails server to verify that it runs, and now I’m going to push it up to github.

Okay I created my public repo on github and added a license.

Then I ran git remote add origin <repo-name>. Followed by git pull --rebase origin master to pull down my license. Then I committed and pushed my current changes.

Okay now its dinner time.

9:17 PM

Now that I’m back from dinner and playing bed wars with my 9 year old, let’s see if we can write some rails code.

bin/rails generate controller Links create

Oops I probably should create the new action for my form and make that the first page you go to. Create will handle the form submission.

Okay I just created new.html.erb and changed the route in routes.rb to `get ‘links/new’.

Now I need to make my form. I’m not going to persist anything to the database yet for this because I don’t want to remember any of these links. So I’m going to create a new Link model, but not have it inherit from active record.

So my model class is called Link and I was also going to call my value I’m storing called link as well, but I think that will get confusing so I’m going to just call it url.

Okay my form is working and I’m able to submit it. Now I need to hook up all the stuff in my command line app after the form is submitted, but first I’m going to commit what I have so far.

11:02 PM

Just got back from a walk. I think the first thing I need to do is be able to load some secrets for overcast. For obvious reasons I don’t want them to be checked into source control, but I need my controller or rake task to be able to read from them.

Ohh maybe I can use the new credentials feature:

https://medium.com/@wintermeyer/goodbye-secrets-welcome-credentials-f4709d9f4698

Maybe I should also watch (or listen) this YouTube video on the subject:

https://www.youtube.com/watch?v=fS92ZDfLhng

Okay, cool I got credentails working. What I want to happen is create a background process to download and the YouTube video so that the web page doesn’t stay open forever if I had it do all the work in the controller. I don’t really want sidekiq right now, so I think I can just do this in a rake task or something. Maybe I could use Active Job?

Sweet I just created my active job called ProcessVideo and pretty much just copied what I had in this initial commit from audio_grab.

youtube-dl ran just fine and created a .mp3 file at the application root. I wonder if I should store it in the /tmp directory? I then ran into an error I think trying to load the file. Ohh I see what I did. The variable was still called files instead of just file. I also just changed it to use the /tmp directory instead. Let’s see if it runs this time.

Wow it actually worked. My form returned right away while it processed the video in the background and then uploaded it to overcast.

There are still some improvements like making sure I don’t just grab the first .mp3 file from the tmp directory but the actual video that was downloaded incase I’m in the middle of processing multiple videos.

Also I still need to deploy it, but I’ll have to do that another day. Here is my last commit.

How to Clone Git Repositores with Puppet on OS X

2018-08-26T06:00:00+00:00

I’m currently in the process of puppetizing my macOS based laptop, but I’m noticing some external modules like vcsrepo don’t have OS X support.

To clone git repositories using puppet on OS X I’m doing:

exec { 'git clone git@github.com:org/repo.git /full-path/repo':
  path    => '/usr/local/bin',
  user    => 'blake'
  creates => '/full-path/repo'
}

Which seems to be getting the job done.

Installing Puppet on OS X

2018-08-25T06:00:00+00:00

My main computer right now is a Linux based desktop computer, but my laptop is a 13” MacBook Pro running the latest version of OS X. Because I use my desktop the most I feel like whenever I do need to be portable and grab my laptop, it is always out of sync with my desktop. Then rather than being productive and getting done what I need to get done I spend my time configuring my laptop.

I’ve been trying to be really good with my current desktop and configure everything via puppet, my goal is that I can also puppettize my OS X based laptop, so I set out to do just that today. I didn’t realize that full support for Puppet and external modules really is only on Linux distros, but they do have an official OS X installer, so this is looking promising.

To install Puppet on OS X visit the offical Installing Puppet agent: MacOS page. And on step #2 click the package link that takes you to this download page. Actully just visit this /mac/puppet/10.13/x86_64 page and download puppet-agent-latest.dmg.

This will download the puppet agent installer. Click on it and follow the install instructions. I’m actually running a masterless puppet install so I don’t actually need the puppet agent running. Once puppet is installing you can then turn off the puppet agent with this command:

sudo puppet resource service pxp-agent ensure=stopped enable=false

You can also verify that puppet is installed by running:

puppet --version

Now you can browse to /etc/puppetlabs and edit your puppet config.

Azure Table Storage

2018-08-24T06:00:00+00:00

One of the things that I like about Azure Table Storage is that it is cheap and it can be quite fast too if you know the partition key and row key of the thing you are searching for.

You can actually store several gigs of data for just a few cents every single month.

One use for table storage could be application log data. Or perhaps there is some database table in your sql database that is trastically larger than any other table in your db that you could extract out and store that data in azure table storage instead?

Here is some key information about Azure Table Storage

Key-Value based
Can have 255 properties
Each entity size can be up to 1mb
Data is stored by partition key and row key
- Improve scalability by limiting number of partitions
- Batch operations can be affected by having a lot of partitions

Supposedly you are only charged the amount of data you are storing, but on the pricing page it says:

We charge $0.00036 per 10,000 transactions for tables. Any type of operation against the storage is counted as a transaction, including reads, writes, and deletes.

So on the Azure 70-532 exam you might need to be careful how you answer a question about this.

One of the things that I don’t always like about using cloud only resources like Azure Table Storage is that you can’t really run them locally on your dev machine. There is however a storage emulater you can run locally on Windows and there is also a non-Microsoft open source version you can run on Linux, OS X, and Windows.

Another tool to checkout is the Storage Explorer which you can use instead of the Azure portal to view/edit data in your Table Storage account.

Getting started with Azure Cosmos DB

2018-08-23T06:00:00+00:00

In this post I’m going to walk you through how to use the Azure portal to create a new Cosmos DB and then how to create a database, a collection, a document, and then query for the document, all from the portal.

(1) Create a new Cosmos DB

In the azure portal, click on “Create a resource” and start searching for “Cosmos DB”. Once it shows up in the search results, possibly as “Azure Cosmos DB” click on it and you should see this “New Account” blade:

You will then need to create a globally unique id for your Cosmos DB. Because this is just a demo db that I’m going to throw away I just created a 31 character random string “45c6c5768a11937faa708eb2fb0587e” for my db id.

Next you will need to choose which Cosmos DB API type you would like to use. Here are the options you can choose from:

For this example we are going to pick “SQL” which is the “DocumentDB” type.

Now go ahead and choose the resource group you want to use or create a new one. For this example let’s just create a new resource group called: “blog-20180823”.

Now choose the location for you Document DB. I’m going to choose “Central US”.

I’m also going to turn off geo-redundancy and not turn on Multi Master. I’m also not going to configure a virtual network.

Now you can click “Create”.

(2) Create a database

Once our cosmos db instance has been created click on it in the azure portal and then select “Data Explorer”. Your screen should now look something like:

From this screen select “New Database”. A new blade will open up. Give it a “Database id” and then click on “OK”.

(3) Create a collection

Now that we hava database, which is really just a holder for multiple collections, we can create our collection by hovering over our database, selecting the “…” and selecting “New Collection”. The following new collection blade will appear:

Let’s call the Coolection Id “Items”. Set the Storage capacity to “Fixed (10 GB)”. And set the throughput to “400” RU/s. This will make our bill come out to around $23.10 per month.

(4) Create a document

Now that we have a collection let’s add a document to it. Normally you would probably do this through code, bet we can also do this in the portal. Click on “New Document” and add the following json:

{
    "id": "1",
    "name": "Buy milk"
}

Then press “Save”.

(5) Create a query

I know we only have one document in our collection, but we can still query for it. Click on the Items collection then click on “New SQL Query” and enter in the following SQL:

SELECT *
FROM items i
WHERE i.id = "1"

Press “Execute Query” and it should return our item.