More HTML Parsing

class: center, middle, inverse, title-slide

# More HTML Parsing
## Pragmatic Datafication - DSVIL 2018
### John Little
### 2018-05-30

---

class: softblue

.pull-left[

## Scrape

![scraping propolis](images/Scraping_propolis.jpg)

Using tools to gather data you can see on a webpage

]

.pull-right[

## Parse

Analyzing the strings and symbols to reveal only the data you need

![strain honey from comb](images/strain_comb.jpg)

]

???

Image Credits:
- [Scraping Propolis](https://commons.wikimedia.org/wiki/File:Scraping_propolis.jpg) by [fishermansdaughter](https://www.flickr.com/photos/fishermansdaughter/)

---
class: softblue

## HTML

Hypertext Markup Language

```html
<!DOCTYPE html>
<html>

 <head>
  <title>sample</title>
 </head>
 <body>
  <p>Voluptatem accusantium
   totam rem aperiam.</p>
 </body>
</html>
```

---

## URL

![elements of a URL](images/URL.PNG)

- Server sends back marked-up content

- HTML
    - CSS StyleSheets
    - JavaScript
    
- Browser parses the HTML

???

Image Credit:
- [URL](https://commons.wikimedia.org/wiki/File:Uniform_Resource_Locator_%28URL%29_example.PNG)
    
---
## OpenRefine (Fetch & Parse)

- Scrape

- Add column by fetching URL...
    
- Parse

- BeautifulSoup Libraries
         - jSoup is a Java library implementation of BeautifulSoup -- a tool for HTML Extraction

- Documentation

- [OpenRefine official documentation](https://github.com/OpenRefine/OpenRefine/wiki/StrippingHTML) on HTML parsing
    - [Learn CSS through gamification](http://flukeout.github.io/)

---
## Process Outline

1. Fetch HTML by URL from Web Server

- Edit column > Add column by fetching URL...
    
3. Identify CSS class and id values.  Typically using the [Chrome Browser's `Inspect`](https://www.youtube.com/results?search_query=Chrome+Inspect+) feature works best

- in Chrome:  highlight text > right-click > inspect (Chrome Developer Mode)

2. Parse HTML

- Edit column > Add column based on this column
        - `parseHTML()`
        - `select("HTML/CSS handle")`
        - choose an array element, typically `[0]`
        - convert element:
            - `htmlText()`
            - `htmlAttr("attribute handle")`

---
###  Parsing by HTML elements and CSS

- Edit Column > Add column based on this column

- `value.parseHtml().select("title")[0].htmlText()`

###  Parsing by Brute Force

- Edit Column > Add column based on this column

1. `value.split("</title>")[0]`
    2. Edit Cells > Transform > `value.split("<title>")[1]`

---
## Demonstration

- https://libjohn.github.io/openrefine/demonstration.html#web-scrape-api

#### use data from

- https://raw.githubusercontent.com/libjohn/openrefine/master/data/bicycle-subset-phm-collection.tsv

---
class: green

## Now You Try It

1. HTML Parsing

- Complete Steps 1-6 (Import, Crawl, Parse) in the [*OpenRefine Workbook*](https://libjohn.github.io/openrefine/hands-on-html-parsing.html)

1. If you're speedy complete as many other steps as suit you

### FYI

- Steps 7 & 8 implement looping control

- convert the display to **records** instead of **rows**
 
- *Inspect your work*. See section (7.3) in the *Workbook*

---
## Review

From price-press_releases.csv

```
date2
  value.parseHtml().select("div.pane-content")[0].htmlText()

dateline 
  value.parseHtml().select("div.field-item.even p strong")[0].htmlText()

Links 
  forEach(
      value.parseHtml().select("div#block-system-main")[0].select("a"),
      v,
      v.htmlAttr("href")
  ).join("|")

Link Text
  forEach(value.parseHtml().select("div#block-system-main")[0].select("a"),
      v,
      v.htmlText()
  ).join("|")
```
    
---
class: middle, center
[Top / Table of Contents](index.html#3)

---

# John Little

.pull-left[
### I am ...

**John Little**
- Data Analytics Librarian
- https://johnlittle.info/
- https://github.com/libjohn

#### Your Rfun host...
You can make **Rfun** with our resources for R and data science analytics.  See the [R we having fun yet‽](https://rfun.library.duke.edu/) resource pages.

]

.pull-right[
### Duke Univesrity...
**Data & Visualization Services**
- https://library.duke.edu/data/
- askData@Duke.edu
- The /Edge, Bostock (1st Floor)
- [Past Workshops](https://library.duke.edu/data/news/past-workshops)
- [Guides & Tutorials](https://library.duke.edu/data/tutorials)

]

---
class: center, middle
## Shareable under CC BY-NC license

Data, presentation, and handouts are shareable under [CC BY-NC license](https://creativecommons.org/licenses/by-nc/4.0/)

![This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.](images/cc-by-nc.png "This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License")