jump to navigation

PHP Basics: Accessing Remote URLs using cURL October 17, 2010

Posted by Tournas Dimitrios in PHP.

In this post, I’m going back to basics to talk about accessing remote URLs with cURL.In PHP, there are actually four ways to access a remote URL :

  1. The 4 file manipulation functions (aka fopen(), file_get_contents() , file() , readfile() ) see this article .
  2. fsockopen()
  3. HTTP_Request class from the PEAR library (OOP approach)
  4. cURL extensions

Now, choosing one way over another really depends on your needs for simplicity, control, and portability. Generally, I prefer using cURL simply because it’s easy to understand and it’s really powerful. cURL supports a number of different protocols and gives your access to the response headers. It even has the capability to do parallel access if you call the ‘curl_multi’ group of functions (I’ll save this for another post one day).

To use cURL, you need to make sure that your PHP install has the library enabled. Read this posting on how to enable cURL for your PHP.

cURL is a library which allows you to connect and communicate to many different types of servers with many different types of protocols. Using cURL you can:

  • Implement payment gateways’ payment notification scripts.
  • Download and upload files from remote servers.
  • Login to other websites and access members only sections.

PHP cURL library is definitely the odd man out. Unlike other PHP libraries where a whole plethora of functions is made available, PHP cURL wraps up a major parts of its functionality in just four functions.

A typical PHP cURL usage follows the following sequence of steps.

curl_init – Initializes the session and returns a cURL handle which can be passed to other cURL functions.

curl_opt – This is the main work horse of cURL library. This function is called multiple times and specifies what we want the cURL library to do.

curl_exec – Executes a cURL session.

curl_close – Closes the current cURL session.

Below are some examples which should make the working of cURL more clearer.

Download file or web page using PHP cURL :
The below piece of PHP code uses cURL to download Google’s RSS feed.

* Initialize the cURL session
$ch = curl_init();
* Set the URL of the page or file to download.
curl_setopt($ch, CURLOPT_URL,
* Ask cURL to return the contents in a variable
* instead of simply echoing them to the browser.
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
* Execute the cURL session
$contents = curl_exec ($ch);
* Close cURL session
curl_close ($ch);

As you can see, curl_setopt is the pivot around which the main cURL functionality revolves. cURL functioning is controlled by way of passing predefined options and values to this function.

The above code uses two such options.

  • CURLOPT_URL: Use it to specify the URL which you want to process. This could be the URL of the file you want to download or it could be the URL of the script to which you want to post some data.
  • CURLOPT_RETURNTRANSFER: Setting this option to 1 will cause the curl_exec function to return the contents instead of echoing them to the browser.

Download file or web page using PHP cURL and save it to file :

The below PHP code is a slight variation of the above code. It not only downloads the contents of the specified URL but also saves it to a file.

* Initialize the cURL session
$ch = curl_init();
* Set the URL of the page or file to download.
curl_setopt($ch, CURLOPT_URL,
* Create a new file
$fp = fopen('rss.xml', 'w');
* Ask cURL to write the contents to a file
curl_setopt($ch, CURLOPT_FILE, $fp);
* Execute the cURL session
curl_exec ($ch);
* Close cURL session and file
curl_close ($ch);

Here we have used another of the cURL options, CURLOPT_FILE. Obtain a file handler by creating a new file or opening an existing one and then pass this file handler to the curl_set_opt function.cURL will now write the contents to a file as it downloads a web page or file.

Submitting forms using PHP cURL :

To submit forms using cURL, we need to follow the below steps:

  1. Prepare the data to be posted
  2. Connect to the remote URL
  3. Post (submit) the data
  4. Fetch response and display it to the user

Prepare data to be posted:

Form data is essentially sent as name value pairs in the format “field1=field1_value&field2=field2_value&field3=field3_value”.

field1, field 2 etc. refer to the form fields and the field1_value, field2_value etc. refer to values of these fields. For our example we will assume that the data we want to post to the remote URL is contained in an associative array like so:

$data = array();
$data['first_name'] = 'Jatinder';
$data['last_name'] = 'Thind';
$data['password'] = 'secret';
$data['email'] = 'me@abc.com';

When a browser submits the form, it automatically urlencodes the data before sending it off. Similarly we will need to urlencode all data before posting it through cURL. You can find more about urlencode here.

$post_str = '';
foreach($data as $key=>$val) {
    $post_str .= $key.'='.urlencode($val).'&';
$post_str = substr($post_str, 0, -1);

The above code will leave us with string “first_name=Jatinder&last_name=Thind&password=secret&email=me%40abc.com” which can now be sent to the remote URL through cURL.

Connect to the remote URL

$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, 'http://example.com/form-handler.php' );

Post (submit) the form data :

First we instruct cURL to a regular HTTP POST.Next we tell cURL which data to send in the HTTP POST.

curl_setopt($ch, CURLOPT_POST, TRUE);
curl_setopt($ch, CURLOPT_POSTFIELDS, $post_str);


Execute request and fetch the response :

$result = curl_exec($ch);
echo $result;

The above code snippet executes the cURL request, fetches the response of the form handler script and displays it to the user.


1. ლევან - May 13, 2011

how to install curl. help me please. i dont understand you

tournasdimitrios1 - May 14, 2011

It all depends on what your working environment is . You have to provide me information like : have you a locally enabled server or are you using an hosting provider ?
If you have a local server , what is your operating system and your server ?
A golden role to explore the features that your environment supports is to run a file with the phpinfo() directive : http://gr.php.net/phpinfo%28%29

2. NikoGreko - June 8, 2011

I v found your examples veay usefull
I need to know if there is a way to use curl to request and grab a specific HTML value from a remote HTML code.
For example in the next part of code

how can i grab the value of ‘active’ hidden field in a variable.?

tournasdimitrios1 - June 8, 2011

The php code that I have seen so far , used Curl to send GET/POST data ( to automate some functionality ) . It’s unusual to use Curl to parse form data , anyway ……

Curl has no build-in functionality to parse the returned data , though PHP has a bunch of regular expressions functions that can handle that kind of functionality . See the official site for more information >>>>>

A possible starting code for your request would be :

$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, 'http://www.yoursite.com/form.html');
curl_setopt($ch, CURLOPT_HEADER, 0);
$match = implode("\r\n",$elements[0]);
print $match ;

Try another example to extract hidden elements from forms >>>>

ps WordPress is very sensitive with uploaded code , so include your script in code tags …

3. NikoGreko - June 9, 2011

Thank you !!!

4. Andy Ingham - June 17, 2011

Hi Toumas,

Found the post helpful, as currently setting something up to act as a ‘gateway’ to a system by allowing an XML message to be posted to a URL which will then parse the XML and create appropriate db records.

Would you recommend any standard approach for securing the URL processing, to prevent unauthorised usage?


tournasdimitrios1 - June 17, 2011

Web-application security is an huge topic and can be implemented with different approaches . Many factors can influence the strategies that has to be followed . For example , this article >>>>> and this article >>>>> demonstrated a basic http authentication mechanism ( with user interaction ) . Another article demonstrated mcrypt >>>> to encrypt large files or data streams .
The by far most common protocol for doing secure communications is generally known as HTTPS ( HTTP over SSL ) . Curl supports encrypted fetches thanks to the freely available OpenSSL Libraries .

I have to emphasize , I don’t have Enterprise level application development experience , so take it in account when evaluating my articles .

5. Sudhir - July 22, 2011

Tried the example above. It works perfectly for local URL(same webserver) but returns nothing from remote URLs. What could be the issue? Is this something to do with my settings?

tournasdimitrios1 - July 22, 2011

The first think I would suspect are firewall rules of your computer and router .Keep in mind that some administrators configure their servers not to respond to specific ” user agents ” , ie if the request isn’t from a browser it’s just ignored .
These restrictions can easily configured in the server’s .htaccess file . There are many on-line web-applications that emulate requests for specific “user agents>>>>>.

6. Build a Visitor Tracking System for your Website with PHP « Tournas Dimitrios - January 16, 2012

[…] support the curl extension , use the class provided by ipinfo.com . My previous article ” Accessing Remote Url’s using Curl ” made a good introduction to this topic , read that article if you like to learn more about […]

7. How To Enable the cURL Extension on XAMP / WAMP (Windows) « Tournas Dimitrios - March 4, 2012

[…] PHP Basics: Accessing Remote URLs using cURL […]

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s