Scraper CLI Project

M*A*S*H Scraper Project


Project Parameters:
Build a CLI program using Ruby that utilizes website scraping or an API

This was the first project we had to do, the first time we would utilize the things we had learned so far.

The project goal was to use scraping or an API to gather information from a website of our choice and then use that information tp build a CLI(Command Line Interface) program.

The first thing I had to learn was how to use a gem called "Nokogiri" to scrap the website.  Nokogiri creates special nodes that are sorted into arrays that make it so they can be accessed and iterate over.  The result being the information you wanted.  This presented my first big challenge: CSS selectors.

To filter through these Nokogiri nodes you use CSS selectors.  To determine what selectors to use you have to use a combination of the pry method and inspecting the code of the website.  I had difficulty at first with this as none of the selectors I was trying to use was returning the information I wanted.  I discovered though, that if you right-click on an element in the inspect panel of the website, then go to "Copy" there is a "Copy Selector" option.  While this option can sometimes be too specific it gives you a much better idea of what selectors you need versus just trying to guess at it.


Once you have that done you have to iterate over the information because it will often contain special characters, code specific tags and so on.  One of the easiest ways to do this to first convert it to text using .text. Once you have the string (text) then you need to iterate.  In my case I had a few special characters such as "\n"(new line).  To do this I used .gsub.  This allows you to pass in a condition that the .gsub iterator will remove from the string.  I found the most efficient way to remove multiple different items from the string was to pass an array with the parts I wanted to remove using regex syntax then since .gsub actually replaces instead of removes, I replaced each one with nothing using "".

Once that was done building the CLI was just a matter of building it out and plugging in the scraped data as needed.

Comments

Post a Comment

Popular posts from this blog

Introducing Dir

Image
While learning JavaScript I learned about console.log() . In the course of using JS I used the quit often.  It's a great debugging tool for doing thing like seeing exactly what a function or variable is returning or even as a build tool to see what a fetch request returns.  I've even used it to see what order my code runs in when using asynchronous code. Recently I learned about a different version of this, console.dir() . Here is what the official docs have to say about it: The  Console  method  dir()  displays an interactive list of the properties of the specified JavaScript object.  The output is presented as a hierarchical listing with disclosure triangles that let you see the contents of child objects. You can read more from the official docs  here .  This is something I hadn't seen before through my entire time in school nor in any of the tutorials I found online.  I'm sure there is some reason I am unaware of that no one s...
Read more

Flatiron Final Feelings

Image
My feeling about my time at Flatiron Well I'm at the end. Six months ago I took a giant leap and went back to school.  No small feat for someone nearing 40 with bills to pay, but I did it.  There where lots of ups and downs.  I met a lot of great people along the way and learned more than I could ever think I could. The Good: The people, the help, the lessons.  My favorite section was on Rails.  I aced and soared through that.  I got really good at everything to do with Rails to where I was helping other students daily. Which leads me to the bad..... The Bad: While JS isn't that hard for me now at the time I was riding this kind of confidence high from Ruby and Rails.  Then I hit Javascript. I wasn't catching on the way I was with Rails.  I even had to take a step back and go over everything again.  It put me behind.  My confidence was shot. There where tears. At the time I was devastated.   I though I...
Read more

Teaching to Learn

    Recently I have been helping new comers to my alma mater learn to code.  This is my way of giving back and helping to pull others up along with me.  Along the way I have discovered something about teaching: I learn more from teaching than I do from just going to classes or watching tutorial videos.     Teaching forces you to keep up with the newest trends like syntax changes and different ways to write code you already learned.  It also enforces good debugging techniques and thinking out loud, forming your thoughts into questions.  It keeps the basics fresh while still being able to flex all the fancy code and techniques you learned along the way.     It also forces you to think outside your own box.  For example when I was in school we had to build a project using only Ruby on Rails for frontend and backend.  One of the requirements is to setup 3rd party authentication through Oauth.  When I build mine and for a few o...
Read more