In PowerShell, you can download a file via HTTP, HTTPS, and FTP with the Invoke-WebRequest cmdlet

Michael Pietroforte

Michael Pietroforte is the founder and editor in chief of 4sysops. He has more than 35 years of experience in IT management and system administration.

Download with SMB ^

If you are working in a hybrid IT environment, you often need to download or upload files from or to the cloud in your PowerShell scripts. If you only use Windows servers that communicate through the Server Message Block (SMB) protocol, you can simply use the Copy-Item cmdlet to copy the file from a network share:

This assumes that you have a VPN solution in place so that your cloud network virtually belongs to your intranet. Things get a bit more complicated if we are leaving the intranet and have to download from an extranet or the Internet.

Download in PowerShell 2 ^

The next simple case is where you have to download a file from the web or from an FTP server. In PowerShell 2, you had to use the New-Object cmdlet for this purpose:

As of PowerShell 3, we have the Invoke-WebRequest cmdlet, which is more convenient to work with. It is PowerShell’s counterpart to GNU wget, a popular tool in the Linux world, which is probably the reason Microsoft decided to use its name as an alias for Invoke-WebRequest. This is perhaps an understatement; Invoke-WebRequest is more powerful than wget because it allows you to not only download files but also parse them. But this is a topic for another post.

Download with Invoke-WebRequest ^

To simply download a file through HTTP, you can use this command:

In the example, we just download the HTML page that the web server at www.contoso.com generates. Note that, if you only specify the folder without the file name, as you can do with Copy-Item, PowerShell will error:

Invoke-WebRequest : Could not find a part of the path

The shorter version for the command line is:

If you omit the local path to the folder, Invoke-WebRequest will just use your current folder. The -Outfile parameter is always required if you want to save the file. The reason is that, by default, Invoke-WebRequest sends the downloaded file to the pipeline.

However, the pipeline will then not just contain the contents of the file. Instead, you will find an object with a variety of properties and methods that allow you to analyze text files. If you send a binary file through the pipeline, PowerShell will treat it as a text file and you won’t be able to use the data in the file.

To only read the contents of the text file, we need to read the Content property of the object in the pipeline:

This command does the same thing as the previous one. The -ExpandProperty parameter ensures that the header (in this case, “Content”) won’t be stored in the file.

If you want to have the file in the pipeline and store it locally, you have to use -PassThru parameter:

This command stores the web page in a file and displays the HTML code.

Download and display file

Download and display file

Authenticating at a web server ^

If the web server requires authentication, you have to use the -Credential parameter:

Note that, if you omit the -Credential parameter, PowerShell will not prompt you for a user name and password and will throw this error:

Invoke-WebRequest : Authorization Required

You have to at least pass the user name with the -Credential parameter. PowerShell will then ask for the password. If you want to avoid a dialog window in your script, you can store the credentials in a PSCredential object:

You can use the -UseDefaultCredentials parameter instead of the -Credential parameter if you want to use the credentials of the current user. To add a little extra security, you might want to encrypt the password. Make sure to always use HTTPS instead of HTTP if you have to authenticate on a remote server. If the web server uses basic authentication, your password will be transmitted in clear text if you download via HTTP.

Note that this method only works if the web server manages authentication. Nowadays, most websites use the features of a content management system (CMS) to authenticate users. Usually, you then have to fill out an HTML form. I will explain in one of my next posts how you can do this with Invoke-WebRequest.

Downloading files through FTP works analogous to HTTP. You also shouldn’t use this protocol if security matters. To download multiple files securely, you had better work with SFTP or SCP. Invoke-WebRequest doesn’t support these protocols. However, third-party PowerShell modules exist that step into the breach.

In my next post I will show you can use Invoke-WebRequest to parse HTML pages and scrape content from websites.

Join the 4sysops PowerShell group!

6+

Users who have LIKED this post:

  • avatar
Share
57 Comments
  1. Spyros 3 years ago

    Is it possible to copy a directory (possibly with grandchildren etc...)  in powershell via HTTP Using PSCredential object credentials? If so, how? I have tried using Start-BitsTransfer for this instead but with no luck so far.

    0

    • Spyros 3 years ago

      'Download' or 'Get' is what I meant when I said copy.

      0

    • Author
      Michael Pietroforte 3 years ago

      I don't think you can do it with Invoke-WebRequest via HTTP because the cmdlet lacks the -r and the --no-parent parameters of its Linux counterpart. You probably could do it using the FTP protocol, but I didn't a try that. Another option is the free wget for Windows. After you install wget, you have to navigate to "C:\Program Files (x86)\GnuWin32\bin>" and then run this command:

      wget -r -np -P "C:\yourLocalFolder http://yourWebServer.com/folder/

      0

  2. ed 3 years ago

    Hi Michael,

    Thanks for that posting.

    I'm having difficulty invoking a download from sites that have restricted TLS versions due to recent vulnerabilities in SSL

    see https://www.ssllabs.com/ssltest/analyze.html?d=zeustracker.abuse.ch for an example.

    Any idea how force a particular TLS version for a download? as it is you can see a handshake failure in a packet trace and get the following error in powershell itself

    + CategoryInfo          : InvalidOperation: (:) [Start-BitsTransfer], Exception
    + FullyQualifiedErrorId : StartBitsTransferCOMException,Microsoft.BackgroundIntelligentTransfer.Management.NewBitsTransferCommand

     

    Any help appreciated

    Ed

    1+

    • Author
      Michael Pietroforte 3 years ago

      Invoke-WebRequest works fine with TLS. I used the cmdlet on sites where SSL is disabled without problems. So I think you have another problem. The issue could be related to the cipher suite. Try to reach this website over HTTPS with Internet Explorer and see if that works. If not, the webmaster of this website removed important cipher suites.

      To force Invoke-WebRequest to use TLS (or a specific TLS version), you can use this command:

      [System.Net.ServicePointManager]::SecurityProtocol = [System.Net.SecurityProtocolType]'Tls,Tls11,Tls12'

      3+

      • ed 3 years ago

        Thanks Michael,
        I'm now thinking it is the cyphers as well- could it be a limitation of server 2003 build that I'm using for the download?

        Regards,
        ED

        1+

      • Chris 2 years ago

        Thanks Michael! This line is what I was looking for to get past the same error on my download.

        [System.Net.ServicePointManager]::SecurityProtocol = [System.Net.SecurityProtocolType]'Tls,Tls11,Tls12'

        2+

        Users who have LIKED this comment:

        • avatar
  3. prb 3 years ago

    Why am I only get the first 5 lines of the webpage?

    powershell -Command "(New-Object Net.WebClient).DownloadFile('http://nwpapps.usace.army.mil/environment/fishdata/report.asp?op=runsum&from=1%2F1%2F2012&to=3%2F6%2F2017&prj=LWG&subsubop=salmon','COE.txt')"

    0

    • Author
      Michael Pietroforte 3 years ago

      I am getting the entire webpage with your command. Maybe a connection issue?

      0

  4. Marc 3 years ago

    It seems DownloadFile() does not work for relative paths - i.e. '.\op.txt' will not be saved in the current folder. So you have to supply a full path:

    $WebClient.DownloadFile("http://url",(pwd).path+"\relative-file.txt")

    0

  5. Lucas 3 years ago

    Hi,

    I'm trying to use this powershell command to download a web file, but the server has access control by user/pass credentials. What changes I must to do in my command to inform user and password and download a file correctly? I try to put "user:pass@" after "//" but this didn't work.

    powershell -command "& { (New-Object Net.WebClient).DownloadFile('http://portalextranet.ibge.gov.br/bog/scripts/visao/Espelho_Setores.php?UF=31&MUN=70206&DISTR=0&SUBDISTR=0&SETOR=0&NIVEL=2&TIPO=X&ANO_BOG=2011200020132010', 'C:\temp\test.xls') }"

    2+

    • Author
      Michael Pietroforte 3 years ago

      It depends on the authentication method the web server uses if this will work. If you have to sign in through form fields, you can use the method described in example 2 here.

      0

    • LUCHO 2 years ago

      Hi Lucas,  did you solve this issue? i have the same problem and still looking a solution, please let me know.

      0

  6. Tom 3 years ago

    Hi Michael,

    Any idea how to get and use the default file name?  I would love to be able to save a file using it's default file name!

    Thanks!!

    0

  7. CuriousITGuy 3 years ago

    I am using powershell to test if a malicious website is blocked, specifically for urls that directly download a file.
    When a URL is a direct download, why does the Invoke-WebRequest command when used via command line, open up IE browser and download a file without asking a user?

    Sample script:
    C:\Users\user1> Invoke-WebRequest http://www.badURL.com/

    The command I use is exactly as listed above (of course with different urls)

    0

    • Author
      Michael Pietroforte 3 years ago

      What do you mean with "direct download"? I never saw Invoke-WebRequest open Internet Explorer.

      0

  8. Joey 2 years ago

    I am hoping to get some clarification on the download process. I have a distributor license for Adobe, and want to distribute Flash Player to my company's new PC's via a powershell command. I have it set up to do the rest of the files I need and they all work, including other Adobe files. But when I try to use WebClient.Download I get the proper named output file, in the proper place, but instead of it being an install file of Adobe Flash Player, it is a 100KB file that errors out when opening. I have also tried BITS Transfer and go the exact same results. The URL is correct and when I paste it into the address bar and hit enter it immediately starts a download of the real file, and it works to install. But not via powershell. Can anyone offer advice?

    0

  9. Chris P 2 years ago

    Is it possible to download the entire contents of a website that is secure? It requires a user name and password to access the site. I have a membership to the site and they offer pre-made video animation templates which I am allowed to use, alter, resell, etc as I see fit. I want to be able to put all the animation templates on my site as I will be offering to customize these animations for clients. Thanks.

    -Chris

    0

    • Author
      Michael Pietroforte 2 years ago

      Before you download an entire site, you should ask the site owner if this is okay. Most sites owners don't really like that. The fact that you can use those templates does not necessarily mean that you can download them all. Then you have to essentially build your own crawler following every link on the website. There are many tools available for this purpose and I guess it doesn't make sense to reinvent the wheel with PowerShell here.

      1+

      Users who have LIKED this comment:

      • avatar
  10. Ranjit 2 years ago

    $dlink="https://AAAAA.com/cc-download-invoice?ie=UTF8&format=csv&invoiceNumber="+$invoiceNo[0].innerHTML

    $output = "$PSScriptRoot\"+$invoiceNo[0].innerHTML.Trim()+".csv"

    #Invoke-WebRequest -Uri $dlink  -Method Post -Outfile $output
    #  $WebClient = New-Object System.Net.WebClient
    # $WebClient.DownloadFile($dlink,$output)

    wget $dlink -outfile $output

    have tried all but never use

    I try download csv file but not able to download it downloads some irrelevant data. but manual click it works fine

    0

    • Author
      Michael Pietroforte 2 years ago

      If this is an invoice, you probably have to be signed in to be able to download the file. Thus, I guess one problem is that you are not signed in on the PowerShell console. Your download link also doesn't make sense. Where did you assign the $invoice variable?

      0

  11. Stan 2 years ago

    Can I use this to download images from all my website pages that have attachments displayed as .png, .jppeg or .jpg?

    0

    • Author
      Michael Pietroforte 2 years ago

      In theory, you can use PowerShell for this purpose. However, if you want to download all images from your website, you have write a crawler that follows all the links. You probably will be faster, if you simply use a web scraping tool.

      0

  12. ITJoe 2 years ago

    Mkdir "C:\Tempdownload"
    $URI="https://get.teamviewer.com/vnurc53"
    $Loc="C:\tempdownload\TeamViewer_Host_Setup.exe"

    $WC = New-Object net.webclient

    $HTMLCode = $WC.Downloadfile("$URI",$Loc)

    This will not download the full .exe it stops after downloading 6kb

    0

  13. Browny 2 years ago

    Hi

    Having some issues similar to above here basically can manually download the file no problem.

    However trying to write a basic script to do this (p.S I'm new to this and self teaching) any help would be gratefully appreciated.

    so my script so far is using the get credential part above and I get the pop up box with the username field filled in however i have to manually type in password.

    it's mentioned above about using PSCredential object how do you add that in? I've tried following the example above and testing but cannot get it to work.

    thanks

    0

    • Author
      Michael Pietroforte 2 years ago

      Which example does not work? If you want to store credentials in your PowerShell script, you can learn here how to do it.

      1+

      • Browny 2 years ago

        Hi,

        Thanks for the reply. This is my script so far:

        $WebClient = New-Object System.Net.Webclient
        $Credentials = New-Object System.Management.Automation.PSCredential $Username, $Password,
        Invoke-WebRequest -Uri "https://www.Webaddress.com" -OutFile -PassThru "C:\Downloads\File=1" -Credential $Credentials

        I've tried this line by line in powershell locally with no errors coming back,

        My next issue is to increment the downloaded file so for example J48File1 would become J49File1 (where J is the Year and the 48 is the week) this is to be downloaded weekly.

        Is this possible?

        Thanks in advance

        0

        • Author
          Michael Pietroforte 2 years ago

          Try this:

          0

          • Browny 2 years ago

            when i use the following line

            Invoke-Webrequest -Uri"https.......

            I get the following error Invoke-Webrequest : the remote name could not be resolved: '-urihttps' Ive tried it with a space and with out to no joy Should I change the wed address part to a $Dlink and put a line in above like

            $Dlink="Https....?

            thanks in advance

             

            0

          • Author
            Michael Pietroforte 2 years ago

            Remove-item -Path "C:\Downloads\*-include *.csv" looks fine. I guess you forgot the last quote.

            1+

        • Author
          Michael Pietroforte 2 years ago

          Can you post the exact command you tried?

          1+

          • Browny 2 years ago

            Hi

            Here is My Script so far:-

            $WebClient = New-Object System.Net.Webclient
            $Credentials = New-Object System.Management.Automation.PSCredential $UserName, $Password, Filename
            $Invoke-WebRequest -URL "https://www.webaddress.com/request_for_data.php?" -OutFile -PassThru "C:\Downloads\File1" -Credential $Credentials
            $Credentials = New-Object System.Management.Automation.PSCredential $Username, $Password, Filename
            $Invoke-WebRequest -URL "https://www.webaddress.com/request_for_data.php?" -OutFile -PassThru "C:\Downloads\File2" -Credential $Credentials
            $filename = "$(Get-Date -UFormat %Y)$(Get-Date -UFormat %V)File1"
            $filename = "$(Get-Date -UFormat %Y)$(Get-Date -UFormat %V)File2"
            $WebClient Copy-Item -Path C:\Downloads\*.csv -Destination "E:\Downloads\Test"
            $WebClient remove-item C:\Downloads*.*

            Thanks in advance.

            0

        • Author
          Michael Pietroforte 2 years ago

          You have several errors in the Invoke-Webrequest line (I didn't check the other lines). There shouldn't be $ at the beginning of the line, the parameter is -Uri, not -URL, and the file name has to be placed right after the -OutFile parameter:

          2+

          • Browny 2 years ago

            Thanks again for the reply, would you be bale to check the other lines to see if I've made any further mistakes please?

            Thanks in advance

            0

          • Browny 2 years ago

            HI Popped powershell into unrestricted mode to test, and all looked good until the Copy-Item part.

            Unexpected token "copy-Item" in expression or statement.

            0

        • Author
          Michael Pietroforte 2 years ago

          The Copy-Item command is OK. I would just put the parameter values in quotes to be on the safe side. But why did you put the $WebClient variable in front of the command? By the way, you don't need the System.Net.Webclient object if you work with Invoke-Webrequest. The object was only needed in PowerShell 2.

          1+

          • Browny 2 years ago

            Thanks again for your help, I had put the $Webclient in as I read from a blog it was better to pop it in.

            I've carried it out as you mentioned above so now looks like :-

            Copy-Item -Path "C:\Downloads\*.csv -Destination "E:\Downloads\Test"
            Remove-item -Path "C:\Downloads\*-include *.csv"

            Copy line now working fine no errors in power shell, but i now get a

            +Remove-item -Path "C:\Downloads\*-include *.csv <<<< "

            is missing the terminator: ".

            At

            +Remove-item -Path "C:\Downloads\*-include *.csv <<<< "

            +CategoryInfo                         : ParserError: (:String) [], ParentContainsErrorRecordException

            +FullyQualifiedErrorID        : TerminatorExpectedAtEndOfString

            0

  14. Browny 2 years ago

    Hi

    Think I've found a way to rename / increment the file as I need to above,

    $WebClient set /A File1=%J44File1%+1

    can someone advise if I'm going in the right direction

    Any help appreciated, Thanks in advance.

     

    1+

  15. LaVerne Sanders 2 years ago

    I have a server that refuses to complete an HTTP download.  The filename is created and ISE shows bytes coming down, but, when it times out, the file has 0 bytes.  This script works on several other systems but the one it needs to work on. (Windows server 2012 R2).  This is the script:

    Invoke-WebRequest "http://ir.eia.gov/wpsr/psw01.xls?api_key=70a7a191d22a1a5ccf689dc939ab3145" -OutFile "D:\SourceDataDownloads\EIA_WeeklyPetroleumStatusReport\RawDataTable01.xls"

    The file can be manually downloaded and opened in Excel thru FireFox.

    Thanks.

     

    0

    • Author
      Michael Pietroforte 2 years ago

      I had no problem downloading the Excel file. Maybe the destination folder in your command is wrong?

      0

  16. LaVerne Sanders 2 years ago

    On the HTTP download issue, I'm convinced that it is not a PowerShell script issue.  I have a c# program that fails in a similar manner.  It seems to have something to do with the machine not allowing that file type to be written from an HTTP site.  However, it will allow FireFox to download and write/open the XLS file.  And, I can open an Excel file and do a save as successfully.

    0

  17. LaVerne Sanders 2 years ago

    Another note for the HTTP download is  Only Excel files cause this problem.  Other files types (csv, png,jpeg) download OK.

    0

  18. LaVerne Sanders 2 years ago

    Other file types download from the same site OK.

    0

  19. Browny 2 years ago

    Hi

    when you say I forgot the last quote are your referring to " I would just put the parameter values in quotes to be on the safe side"?

    thanks in advance

    0

    • Author
      Michael Pietroforte 2 years ago

      No. There is just one quotation mark missing in your command. This is correct:

      1+

      • Browny 2 years ago

        Thanks SO the line that I posted above the remove item line is missing the quotation marks as well so not working so it should be

        Remove -Item "C:\Downloads\*.*"

        0

  20. Hoang 2 years ago

    can some help on how to download the content from the site which has self signed certificate assigned, without disabling the SSL/TLS using the invoke-webrequest command?

    0

    • Author
      Michael Pietroforte 2 years ago

      I think PowerShell changed its behavior with regard to self-signed certificates recently. What error message did you get?

      0

  21. Sid 1 year ago

    Hi Michael,

    I am trying to transfer a file from a Windows 2012 R2 server to a path in http service (EXAMPLE: https://e190.cbee/service/cbee-transport/ -) using powershell. How to make that possible? Is there any particular command that I should be using?
    Also, when I am trying to access the https from the Windows server, it is blocked by IE Enhanced Security Configuration(Server Manager), which I tried but was not allowed to shut off.

    0

    • Author

      Sid, it seems you want to upload (and not download) a file to a web server. The methods you can use for this task depend on the kind of web server you use and what APIs this server provides. Are you managing e190.cbee?

      0

  22. Reddy 1 year ago

    I need some help on how to Restore a(same) database daily from a website.

    There is a database backup(Zip file) in the website and all I want to do is to restore the same database daily on my server. ( Overwrite the same database daily)

    I want that to be scheduled so that the users have fresh data daily.

    Attached is the screenshot where I am getting the backup from.

    Appreciate the help.

    0

    • Author

      You can just schedule a task with a little PowerShell script that downloads the database file.

      0

  23. Thomas 1 year ago

    If I download an exe file with PowerShell, it seems to be corrupt. It throws an error when I try to execute it. If I download the same file through a browser it works. Any ideas why this could be the case?

    $uri = "https://dl.google.com/chrome/install/ChromeStandaloneSetup.exe"
    if (-not $PSScriptRoot)
    {
    $PSScriptRoot = Split-Path -Parent -Path $script:MyInvocation.MyCommand.Definition
    }
    $outFile = "$PSScriptRoot\ChromeStandaloneSetup.exe"
    Invoke-WebRequest -Uri $uri -OutFile $outFile
    & $outFile /silent /install

    0

  24. Rick 5 days ago

    Hi Michael, I need a PowerShell 5 script that will download 185,000 files (all types) plus any versions that may exist. 

    Any ideas?

    0

    • Author

      The more interesting question is why you want to download so many files from a website? If this server belongs to your organization why not simply copy the files from the webserver?

      0

Leave a reply

Your email address will not be published. Required fields are marked *

*

© 4sysops 2006 - 2019

CONTACT US

Please ask IT administration questions in the forums. Any other messages are welcome.

Sending

Log in with your credentials

or    

Forgot your details?

Create Account