class: center, middle, inverse, title-slide

# Web Scraping with WebScraper.io
## Pragmatic Datafication - DSVIL 2018
### John Little
### 2018-05-29

---

class: middle, center

### Scraping = Crawling + Parsing

---
class: softblue
## Crawl

.pull-left[ 
Moving across or through a website in an attempt to gather data from more than one page (URL)
]

.pull-right[

![sloth](images/crawling_med.jpg)

]

---
class: middle, center

For our hands-on and demonstration

# WebScraper.io

Use the Chrome Browser

---
## Demonstration

Congressional Press Releases

- Representative Nancy Pelosi’s Press Releases
    - CONTENT
        - Structure of the [Press Release subsection](http://pelosi.house.gov/news/press-releases/) of the site
            - Pagination
            - Links to each release
            - Information Content Structure:  Web Site & Web Page(s)
    - TOOL
        - [Webscraper.io](http://Webscraper.io) tool works inside of Chrome
            - Tutorials
            - Documentation
            - Community
            - Free 
                - Alternatively, Fee for Service

---
background-image: url(images/selector_graph.png)

---
class: green

## Now You Try It

1. Download & Install [Webscraper.io](http://Webscraper.io) ; restart your Chrome browser

2. Use one of the following sites

- https://pelosi.house.gov/news/press-releases/ 
    - https://butterfield.house.gov/media-center/press-releases 
    - https://walker.house.gov/media-center/press-releases 
    - https://www.donnelly.senate.gov/newsroom/press 
    - https://price.house.gov/press-releases/

3. Follow Intructions 1-8

- Substitute the site you picked above
    
    - instructions: [v.gd/webscraping1111](https://docs.google.com/document/d/17iKQG9JfREEEThz_Qeai5lvv5mYPRsixi0MKbS6aIsM/edit#heading=h.hdqpl186tuyo)

---
class: middle, center
[Top / Table of Contents](index.html#3) | [NEXT OpenRefine: Basics (Part 1)](openrefine_cleaning_basics_20.html)

---

# John Little

.pull-left[
### I am ...

**John Little**
- Data Analytics Librarian
- https://johnlittle.info/
- https://github.com/libjohn

#### Your Rfun host...
You can make **Rfun** with our resources for R and data science analytics.  See the [R we having fun yet‽](https://rfun.library.duke.edu/) resource pages.

]

.pull-right[
### Duke Univesrity...
**Data & Visualization Services**
- https://library.duke.edu/data/
- askData@Duke.edu
- The /Edge, Bostock (1st Floor)
- [Past Workshops](https://library.duke.edu/data/news/past-workshops)
- [Guides & Tutorials](https://library.duke.edu/data/tutorials)

]

---
class: center, middle
## Shareable under CC BY-NC license

Data, presentation, and handouts are shareable under [CC BY-NC license](https://creativecommons.org/licenses/by-nc/4.0/)

![This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.](images/cc-by-nc.png "This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License")

Notes for current slide

Notes for next slide

Web Scraping with WebScraper.ioPragmatic Datafication - DSVIL 2018John Little2018-05-291 / 10

Scraping = Crawling + Parsing2 / 10

Crawl

Moving across or through a website in an attempt to gather data from more than one page (URL)

sloth

3 / 10

For our hands-on and demonstration

WebScraper.io

Use the Chrome Browser

4 / 10

Demonstration

Congressional Press Releases

Representative Nancy Pelosi’s Press Releases
- CONTENT
  - Structure of the Press Release subsection of the site
    - Pagination
    - Links to each release
    - Information Content Structure: Web Site & Web Page(s)
- TOOL
  - Webscraper.io tool works inside of Chrome
    - Tutorials
    - Documentation
    - Community
    - Free
      - Alternatively, Fee for Service

5 / 10

6 / 10

Now You Try It

Download & Install Webscraper.io ; restart your Chrome browser
Use one of the following sites
Follow Intructions 1-8
- Substitute the site you picked above
- instructions: v.gd/webscraping1111

7 / 10

Top / Table of Contents | NEXT OpenRefine: Basics (Part 1)

8 / 10

John Little

I am ...

John Little

Your Rfun host...

You can make Rfun with our resources for R and data science analytics. See the R we having fun yet‽ resource pages.

Duke Univesrity...

Data & Visualization Services

https://library.duke.edu/data/
askData@Duke.edu
The /Edge, Bostock (1st Floor)
Past Workshops
Guides & Tutorials

9 / 10

Shareable under CC BY-NC license

Data, presentation, and handouts are shareable under CC BY-NC license

This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.

10 / 10

Scraping = Crawling + Parsing2 / 10

Paused

Help

Keyboard shortcuts

↑, ←, Pg Up, k	Go to previous slide
↓, →, Pg Dn, Space, j	Go to next slide
Home	Go to first slide
End	Go to last slide
Number + Return	Go to specific slide
b / m / f	Toggle blackout / mirrored / fullscreen mode
c	Clone slideshow
p	Toggle presenter mode
t	Restart the presentation timer
?, h	Toggle this help

Esc	Back to slideshow