SaltyCrane Blog — Notes on JavaScript and web development

Using curl over ftp took 3+ minutes for a 4 byte file w/ EPSV

Date:  |  Tags: linux

It took over 3 minutes to download a 4 byte file with curl via ftp. It took less than a second with wget.

$ time curl -o testfile.txt -u myusername:mypassword ftp://ftp.myserver.com/path/to/testfile.txt 

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100     4    0     4    0     0      0      0 --:--:--  0:03:09 --:--:--     0

real    3m9.411s
user    0m0.000s
sys     0m0.050s
Running curl with the verbose option

Running curl with the verbose option showed it was failing in "Extended Passive Mode".

$ curl -v -o testfile.txt -u myusername:mypassword ftp://ftp.myserver.com/path/to/testfile.txt

* About to connect() to ftp.myserver.com port 21 (#0)
*   Trying 10.1.2.102...   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0connected
* Connected to ftp.myserver.com (10.1.2.102) port 21 (#0)
< 220 Welcome! 01-srv.
> USER myusername
< 331 Please specify the password.
> PASS mypassword
< 230 Login successful.
> PWD
< 257 "/"
* Entry path is '/'
> CWD dmi
< 250 Directory successfully changed.
> EPSV
* Connect data stream passively
< 229 Entering Extended Passive Mode (|||58226|)
  0     0    0     0    0     0      0      0 --:--:--  0:03:09 --:--:--     0Connection timed out
* couldn't connect to host
* got positive EPSV response, but can't connect. Disabling EPSV
> PASV
< 227 Entering Passive Mode (10,1,2,102,190,42)
*   Trying 10.1.2.102... connected
* Connecting to 10.1.2.102 (10.1.2.102) port 48682
> TYPE I
< 200 Switching to Binary mode.
> SIZE testfile.txt
< 213 4
> RETR testfile.txt
< 150 Opening BINARY mode data connection for testfile.txt (4 bytes).
* Maxdownload = -1
* Getting file with size: 4
{ [data not shown]
* Remembering we are in dir "path/to/"
< 226 File send OK.
Disabling EPSV

I googled and found that some servers have a problem using Extended Passive Mode (EPSV). To disable Extended Passive Mode, use the --disable-epsv option. With this option, the download took 0.046 seconds.

$ time curl --disable-epsv -o testfile.txt -u myusername:mypassword ftp://ftp.myserver.com/path/to/testfile.txt

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100     4  100     4    0     0    112      0 --:--:-- --:--:-- --:--:--   133

real    0m0.046s
user    0m0.010s
sys     0m0.000s
Disabling EPSV with libcurl

Set CURLOPT_FTP_USE_EPSV to 0. See http://curl.haxx.se/libcurl/c/curl_easy_setopt.html.

Disabling EPSV with pycurl
import pycurl

c = pycurl.Curl()
c.setopt(c.FTP_USE_EPSV, 0)

Comments