Note that there are some explanatory texts on larger screens.

plurals
  1. POHow to design a "web spider" with state in Haskell?
    primarykey
    data
    text
    <p>I am learning Haskell after years of OOP.</p> <p>I am writing a dumb web spider with few functions and state.<br> I am not sure how to do it right in FP world.</p> <p>In OOP world this spider could be designed like this (by usage):</p> <pre><code>Browser b = new Browser() b.goto(“http://www.google.com/”) String firstLink = b.getLinks()[0] b.goto(firstLink) print(b.getHtml()) </code></pre> <p>This code loads <a href="http://www.google.com/" rel="noreferrer">http://www.google.com/</a>, then “clicks” the first link, loads content of second page and then prints the content.</p> <pre><code>class Browser { goto(url: String) : void // loads HTML from given URL, blocking getUrl() : String // returns current URL getHtml() : String // returns current HTML getLinks(): [String] // parses current HTML and returns a list of available links (URLs) private _currentUrl:String private _currentHtml:String } </code></pre> <p>It’s possbile to have 2 or “browsers” at once, with its own separate state:</p> <pre><code>Browser b1 = new Browser() Browser b2 = new Browser() b1.goto(“http://www.google.com/”) b2.goto(“http://www.stackoverflow.com/”) print(b1.getHtml()) print(b2.getHtml()) </code></pre> <p><strong>QUESTION</strong>: show how would you design such a thing in Haskell from scracth (Browser-like API with possibility to have several independent instances)? Please, give a code snippet.</p> <p><strong>NOTE</strong>: For simplicity, skip the details on getLinks() function (its trivial and not interesting).<br>Also let’s assume there is an API function</p> <pre><code>getUrlContents :: String -&gt; IO String </code></pre> <p>that opens HTTP connection and returns an HTML for given URL. <br> <br> <br></p> <p><strong>UPDATE</strong>: why to have state (or may be not)?</p> <p>The API can have more functions, not just single "load-and-parse results".<br> I didn't add them to avoid complexity.</p> <p>Also it could care about HTTP Referer header and cookies by sending them with each request in order to emulate real browser behavior.</p> <p>Consider the following scenario:</p> <ol> <li>Open <a href="http://www.google.com/" rel="noreferrer">http://www.google.com/</a></li> <li>Type "haskell" into first input area</li> <li>Click button "Google Search"</li> <li>Click link "2"</li> <li>Click link "3"</li> <li>Print HTML of current page (google results page 3 for "haskell")</li> </ol> <p>Having a scenario like this on hands, I as a developer would like to transfer it to code as close as possible:</p> <pre><code>Browser b = new Browser() b.goto("http://www.google.com/") b.typeIntoInput(0, "haskell") b.clickButton("Google Search") // b.goto(b.finButton("Google Search")) b.clickLink("2") // b.goto(b.findLink("2")) b.clickLink("3") print(b.getHtml()) </code></pre> <p>The goal of this scenario is to get HTML of the last page after a set of operations. Another less visible goal is to keep code compact.</p> <p>If Browser has a state, it can send HTTP Referer header and cookies while hiding all mechanics inside itself and giving nice API.</p> <p>If Browser has no state, the developer is likely to pass around all current URL/HTML/Cookies -- and this adds noise to scenario code.</p> <p>NOTE: I guess there are libraries outside for scrapping HTML in Haskell, but my intention was not to scrap HTML, but learn how these "black-boxed" things can be designed properly in Haskell.</p>
    singulars
    1. This table or related slice is empty.
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
 

Querying!

 
Guidance

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload