StackOverflow2013

Note that there are some explanatory texts on larger screens.

plurals

POHow can I make a function that repeats itself until it finds ALL the information?
text
Body
copied!<p>I want to create a PHP function that goes through a website's homepage, finds all the links in the homepage, goes through the links that it finds and keeps going until all the links on said website are final. I really need to build something like this so I can spider my network of sites and supply a "one stop" for searching.</p> <p>Here's what I got so far -</p> <pre><code>function spider($urltospider, $current_array = array(), $ignore_array = array('')) { if(empty($current_array)) { // Make the request to the original URL $session = curl_init($urltospider); curl_setopt($session, CURLOPT_RETURNTRANSFER, true); $html = curl_exec($session); curl_close($session); if($html != '') { $dom = new DOMDocument(); @$dom->loadHTML($html); $xpath = new DOMXPath($dom); $hrefs = $xpath->evaluate("/html/body//a"); for($i = 0; $i < $hrefs->length; $i++) { $href = $hrefs->item($i); $url = $href->getAttribute('href'); if(!in_array($url, $ignore_array) && !in_array($url, $current_array)) { // Add this URL to the current spider array $current_array[] = $url; } } } else { die('Failed connection to the URL'); } } else { // There are already URLs in the current array foreach($current_array as $url) { // Connect to this URL // Find all the links in this URL // Go through each URL and get more links } } } </code></pre> <p>The only problem is, I can't seem to get my head around how to proceed. Can anyone help me out? Basically, this function will repeat itself until everything has been found.</p>

Querying!

Guidance

An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload