Powershell help downloading 2016 MS free ebooks

josephhays2 · 2017-07-17T16:40:59.000Z

############################################################### 
# Eric Ligmans Amazing Free Microsoft eBook Giveaway 
# https://blogs.msdn.microsoft.com/mssmallbiz/2017/07/11/largest-free-microsoft-ebook-giveaway-im-giving-away-millions-of-free-microsoft-ebooks-again-including-windows-10-office-365-office-2016-power-bi-azure-windows-8-1-office-2013-sharepo/
# Link to download list of eBooks 
# http://ligman.me/2sZVmcG 
# Thanks David Crosby for the template (https://social.technet.microsoft.com/profile/david%20crosby/)
############################################################### 
$dest = "D:\Users\johnh\Desktop\books2016\" 
 
# Download the source list of books 
#$downLoadList = "http://www.mssmallbiz.com/ericligman/Key_Shorts/MSFTFreeEbooks.txt" 
#$downLoadList = "https://www.dropbox.com/s/752jcqwzwjcp2by/Ligman_eBooks_2017.txt?dl=1"
#$bookList = Invoke-WebRequest $downLoadList 
$bookList = "D:\Users\johnh\Desktop\MSFTFreeEbooks.txt"
 
# Convert the list to an array 
[string[]]$books = "" 
$books = $bookList.Content.Split("`n") 
# Remove the first line - it's not a book 
$books = $books[1..($books.Length -1)] 
$books # Here's the list 
 
# Download the books 
foreach ($book in $books) { 
    $hdr = Invoke-WebRequest $book -Method Head 
    $title = $hdr.BaseResponse.ResponseUri.Segments[-1] 
    $title = [uri]::UnescapeDataString($title) 
    $saveTo = $dest + $title 
    Invoke-WebRequest $book -OutFile $saveTo 
}

So, I suck at PowerShell haha. Today I was using it to download free ebooks from the article posted on SpiceWorks (thanks again). I thought it would be easy enough to get the URL list from 2016 and modify the script to download all of last years free ebooks. Anyway, I can’t seem to figure out why it is not working for me. I have the list saved locally, which is the only real difference. I also have the destination directory in place, so I know that isn’t the problem. I will post the error I am getting below. Any help would be appreciated. (ps: I know I can just use a download extension for my browser, but I like to tinker with PowerShell)

You cannot call a method on a null-valued expression.
At D:\Users\johnh\Desktop\2016Eligman_DownloadAll_PowerShell.ps1:18 char:1
+ $books = $bookList.Content.Split("`n")
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidOperation: (:) [], RuntimeException
    + FullyQualifiedErrorId : InvokeMethodOnNull
 

Invoke-WebRequest : Invalid URI: The hostname could not be parsed.
At D:\Users\johnh\Desktop\2016Eligman_DownloadAll_PowerShell.ps1:25 char:12
+     $hdr = Invoke-WebRequest $book -Method Head
+            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Invoke-WebRequest], UriFormatException
    + FullyQualifiedErrorId : System.UriFormatException,Microsoft.PowerShell.Commands.InvokeWebRequestCommand
 
Cannot index into a null array.
At D:\Users\johnh\Desktop\2016Eligman_DownloadAll_PowerShell.ps1:26 char:5
+     $title = $hdr.BaseResponse.ResponseUri.Segments[-1]
+     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidOperation: (:) [], RuntimeException
    + FullyQualifiedErrorId : NullArray
 
Invoke-WebRequest : Invalid URI: The hostname could not be parsed.
At D:\Users\johnh\Desktop\2016Eligman_DownloadAll_PowerShell.ps1:29 char:5
+     Invoke-WebRequest $book -OutFile $saveTo
+     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Invoke-WebRequest], UriFormatException
    + FullyQualifiedErrorId : System.UriFormatException,Microsoft.PowerShell.Commands.InvokeWebRequestCommand

Neally · 2017-07-17T16:56:22.000Z

$books = $bookList.Content.Split("`n")

the ‘.content’ is part of the output if the ‘invoke-webrequest’. Since you get it from a textfile you do not have that object.
Try like so:

$books = $bookList.Split("`n")

Also, you’ll need to get the content of the text file

$bookList = "D:\Users\johnh\Desktop\MSFTFreeEbooks.txt"
# should be:
$bookList = Get-Content "D:\Users\johnh\Desktop\MSFTFreeEbooks.txt"

ALso you will not have to remove the first line after that as I assume since it is from your list, it is a book?

# Remove the first line - it's not a book 
$books = $books[1..($books.Length -1)] 

# ^ not needed?

josephhays2 · 2017-07-17T17:13:29.000Z

Thanks @alexw . I would have never figured out the.content I feel like I should have seen the missing get-content, but that’s what you get from a PS newb. The first line of the txt file actually is not a URL and just a line of text, so I believe that portion is correct. I will post back momentarily with results.

josephhays2 · 2017-07-17T17:28:13.000Z

Okay. Definitely got past at least part of the problem. I am now getting errors in the foreach loop. I think it might be an issue saving to the destination, but I am not sure. I did try a few of the links in the list manually, and the ones I have tried work in the browser but won’t download in the script. I will attach a few of the errors.

Invoke-WebRequest : Could not find a part of the path 'D:\Users\jhays\Desktop\books2016\9780735697225.PDF'.
At D:\Users\johnh\Desktop\2016Eligman_DownloadAll_PowerShell.ps1:29 char:5
+     Invoke-WebRequest $book -OutFile $saveTo
+     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Invoke-WebRequest], DirectoryNotFoundException
    + FullyQualifiedErrorId : System.IO.DirectoryNotFoundException,Microsoft.PowerShell.Commands.InvokeWebRequestCommand
 
Invoke-WebRequest : Could not find a part of the path 'D:\Users\jhays\Desktop\books2016\9780735697225.EPUB'.
At D:\Users\johnh\Desktop\2016Eligman_DownloadAll_PowerShell.ps1:29 char:5
+     Invoke-WebRequest $book -OutFile $saveTo
+     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Invoke-WebRequest], DirectoryNotFoundException
    + FullyQualifiedErrorId : System.IO.DirectoryNotFoundException,Microsoft.PowerShell.Commands.InvokeWebRequestCommand
 
Invoke-WebRequest : Could not find a part of the path 'D:\Users\jhays\Desktop\books2016\9780735697225.MOBI'.
At D:\Users\johnh\Desktop\2016Eligman_DownloadAll_PowerShell.ps1:29 char:5
+     Invoke-WebRequest $book -OutFile $saveTo
+     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Invoke-WebRequest], DirectoryNotFoundException
    + FullyQualifiedErrorId : System.IO.DirectoryNotFoundException,Microsoft.PowerShell.Commands.InvokeWebRequestCommand
 
Invoke-WebRequest : Could not find a part of the path 'D:\Users\jhays\Desktop\books2016\9780735698178.EPUB'.
At D:\Users\johnh\Desktop\2016Eligman_DownloadAll_PowerShell.ps1:29 char:5
+     Invoke-WebRequest $book -OutFile $saveTo
+     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Invoke-WebRequest], DirectoryNotFoundException
    + FullyQualifiedErrorId : System.IO.DirectoryNotFoundException,Microsoft.PowerShell.Commands.InvokeWebRequestCommand
 
Invoke-WebRequest : Could not find a part of the path 'D:\Users\jhays\Desktop\books2016\9780735698178.MOBI'.
At D:\Users\johnh\Desktop\2016Eligman_DownloadAll_PowerShell.ps1:29 char:5
+     Invoke-WebRequest $book -OutFile $saveTo
+     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Invoke-WebRequest], DirectoryNotFoundException
    + FullyQualifiedErrorId : System.IO.DirectoryNotFoundException,Microsoft.PowerShell.Commands.InvokeWebRequestCommand
 
Invoke-WebRequest : Could not find a part of the path 'D:\Users\jhays\Desktop\books2016\9780735698178.PDF'.
At D:\Users\johnh\Desktop\2016Eligman_DownloadAll_PowerShell.ps1:29 char:5
+     Invoke-WebRequest $book -OutFile $saveTo
+     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Invoke-WebRequest], DirectoryNotFoundException
    + FullyQualifiedErrorId : System.IO.DirectoryNotFoundException,Microsoft.PowerShell.Commands.InvokeWebRequestCommand
 
Invoke-WebRequest : Could not find a part of the path 'D:\Users\jhays\Desktop\books2016\MICROSOFT_PRESS_EBOOK_AZURE_AUTOMATION_PDF.PDF'.
At D:\Users\johnh\Desktop\2016Eligman_DownloadAll_PowerShell.ps1:29 char:5
+     Invoke-WebRequest $book -OutFile $saveTo
+     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Invoke-WebRequest], DirectoryNotFoundException
    + FullyQualifiedErrorId : System.IO.DirectoryNotFoundException,Microsoft.PowerShell.Commands.InvokeWebRequestCommand
 
Invoke-WebRequest : Could not find a part of the path 'D:\Users\jhays\Desktop\books2016\9780735698154.EPUB'.
At D:\Users\johnh\Desktop\2016Eligman_DownloadAll_PowerShell.ps1:29 char:5
+     Invoke-WebRequest $book -OutFile $saveTo
+     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Invoke-WebRequest], DirectoryNotFoundException
    + FullyQualifiedErrorId : System.IO.DirectoryNotFoundException,Microsoft.PowerShell.Commands.InvokeWebRequestCommand
 
Invoke-WebRequest : Could not find a part of the path 'D:\Users\jhays\Desktop\books2016\9780735698154.MOBI'.
At D:\Users\johnh\Desktop\2016Eligman_DownloadAll_PowerShell.ps1:29 char:5
+     Invoke-WebRequest $book -OutFile $saveTo
+     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Invoke-WebRequest], DirectoryNotFoundException
    + FullyQualifiedErrorId : System.IO.DirectoryNotFoundException,Microsoft.PowerShell.Commands.InvokeWebRequestCommand
 
Invoke-WebRequest : Could not find a part of the path 'D:\Users\jhays\Desktop\books2016\Bring-the-cloud-to-your-datacenter-Microsoft-Azure-Stack.pdf'.
At D:\Users\johnh\Desktop\2016Eligman_DownloadAll_PowerShell.ps1:29 char:5
+     Invoke-WebRequest $book -OutFile $saveTo
+     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Invoke-WebRequest], DirectoryNotFoundException
    + FullyQualifiedErrorId : System.IO.DirectoryNotFoundException,Microsoft.PowerShell.Commands.InvokeWebRequestCommand

Neally · 2017-07-17T17:30:42.000Z

huh , what are you trying to do?

Invoke-WebRequest : Could not find a part of the path 'D:\Users\jhays\Desktop\books2016\9780735697225.PDF'.

It looks like you are trying to invoke a web request on a local file?

It should be something like

Invoke-WebRequest -Uri "http://somewebsite.com"

josephhays2 · 2017-07-17T17:51:17.000Z

That is simply the output of the script after making the changes you suggested with get-content and removing .content.

I thought that it looked like it was invoking a webrequest to a local file as well, but that just didn’t make sense to me so I imagined I didn’t know what I was reading and was having errors writing the output to the local location.

Perhaps the array from $booklist.split is not sending the correct URLS to invoke-webrequest.

Current script I am using:

############################################################### 
# Eric Ligmans Amazing Free Microsoft eBook Giveaway 
# https://blogs.msdn.microsoft.com/mssmallbiz/2017/07/11/largest-free-microsoft-ebook-giveaway-im-giving-away-millions-of-free-microsoft-ebooks-again-including-windows-10-office-365-office-2016-power-bi-azure-windows-8-1-office-2013-sharepo/
# Link to download list of eBooks 
# http://ligman.me/2sZVmcG 
# Thanks David Crosby for the template (https://social.technet.microsoft.com/profile/david%20crosby/)
############################################################### 
$dest = "D:\Users\johnh\Desktop\books2016\" 
 
# Download the source list of books 
#$downLoadList = "http://www.mssmallbiz.com/ericligman/Key_Shorts/MSFTFreeEbooks.txt" 
#$downLoadList = "https://www.dropbox.com/s/752jcqwzwjcp2by/Ligman_eBooks_2017.txt?dl=1"
#$bookList = Invoke-WebRequest $downLoadList 
$bookList = Get-Content "D:\Users\johnh\Desktop\MSFTFreeEbooks.txt"
 
# Convert the list to an array 
[string[]]$books = "" 
$books = $bookList.Split("`n") 
# Remove the first line - it's not a book 
$books = $books[1..($books.Length -1)] 
$books # Here's the list 
 
# Download the books 
foreach ($book in $books) { 
    $hdr = Invoke-WebRequest $book -Method Head 
    $title = $hdr.BaseResponse.ResponseUri.Segments[-1] 
    $title = [uri]::UnescapeDataString($title) 
    $saveTo = $dest + $title 
    Invoke-WebRequest $book -OutFile $saveTo 
}

Current output when script is run (truncated for brevity):

PS D:\Users\jhays\Desktop> D:\Users\jhays\Desktop\2016Eligman_DownloadAll_PowerShell.ps1
http://ligman.me/29ngkYn
http://ligman.me/29hV6ZM
http://ligman.me/29d1qAV
http://ligman.me/29pfpbI
http://ligman.me/29v8nwX
http://ligman.me/29uq452
http://ligman.me/29d1wc3
http://ligman.me/29Y1O7a
http://ligman.me/29a7wRA
http://ligman.me/29pnEEG
http://ligman.me/1FYtDD8
http://ligman.me/1HByNKS
http://ligman.me/1NCfcKC
http://ligman.me/1HCDxl9
http://ligman.me/1HCCCRP
http://ligman.me/1H4Q0e5
http://ligman.me/1JI6V77
http://ligman.me/1CSMobd
http://ligman.me/1jWMJA2
http://ligman.me/1m6xucg
http://ligman.me/1onTg9n
http://ligman.me/1pxniH4
http://ligman.me/1dG2ZZ9
http://ligman.me/29v9igV
http://ligman.me/29hofVk
http://ligman.me/29idcee
http://ligman.me/29rOvx0
http://ligman.me/29ctkki
http://ligman.me/29okalt
http://ligman.me/29i2CEe
http://ligman.me/29FT71n
http://ligman.me/29NBSj1
http://ligman.me/29H3O3o
http://ligman.me/29OHLKa
http://ligman.me/29uHKKV
Invoke-WebRequest : Could not find a part of the path 'D:\Users\johnh\Desktop\books2016\MICROSOFT_PRESS_EBOOK_DEPLOYING_WINDOWS_10_PDF.PDF'.
At D:\Users\johnh\Desktop\2016Eligman_DownloadAll_PowerShell.ps1:29 char:5
+     Invoke-WebRequest $book -OutFile $saveTo
+     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Invoke-WebRequest], DirectoryNotFoundException
    + FullyQualifiedErrorId : System.IO.DirectoryNotFoundException,Microsoft.PowerShell.Commands.InvokeWebRequestCommand
 
Invoke-WebRequest : Could not find a part of the path 'D:\Users\johnh\Desktop\books2016\MICROSOFT_PRESS_EBOOK_DEPLOYING_WINDOWS_10_EPUB.EPUB'.
At D:\Users\johnh\Desktop\2016Eligman_DownloadAll_PowerShell.ps1:29 char:5
+     Invoke-WebRequest $book -OutFile $saveTo
+     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Invoke-WebRequest], DirectoryNotFoundException
    + FullyQualifiedErrorId : System.IO.DirectoryNotFoundException,Microsoft.PowerShell.Commands.InvokeWebRequestCommand
 
Invoke-WebRequest : Could not find a part of the path 'D:\Users\johnh\Desktop\books2016\MICROSOFT_PRESS_EBOOK_DEPLOYING_WINDOWS_10_MOBI.MOBI'.
At D:\Users\johnh\Desktop\2016Eligman_DownloadAll_PowerShell.ps1:29 char:5
+     Invoke-WebRequest $book -OutFile $saveTo
+     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Invoke-WebRequest], DirectoryNotFoundException
    + FullyQualifiedErrorId : System.IO.DirectoryNotFoundException,Microsoft.PowerShell.Commands.InvokeWebRequestCommand
 
Invoke-WebRequest : Could not find a part of the path 'D:\Users\johnh\Desktop\books2016\MICROSOFT_PRESS_EBOOK_INTRODUCING_WINDOWS_10_PDF.PDF'.
At D:\Users\johnh\Desktop\2016Eligman_DownloadAll_PowerShell.ps1:29 char:5
+     Invoke-WebRequest $book -OutFile $saveTo
+     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Invoke-WebRequest], DirectoryNotFoundException
    + FullyQualifiedErrorId : System.IO.DirectoryNotFoundException,Microsoft.PowerShell.Commands.InvokeWebRequestCommand
 
Invoke-WebRequest : Could not find a part of the path 'D:\Users\johnh\Desktop\books2016\MICROSOFT_PRESS_EBOOK_INTRODUCING_WINDOWS_10_EPUB.EPUB'.
At D:\Users\johnh\Desktop\2016Eligman_DownloadAll_PowerShell.ps1:29 char:5
+     Invoke-WebRequest $book -OutFile $saveTo
+     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Invoke-WebRequest], DirectoryNotFoundException
    + FullyQualifiedErrorId : System.IO.DirectoryNotFoundException,Microsoft.PowerShell.Commands.InvokeWebRequestCommand
 
Invoke-WebRequest : Could not find a part of the path 'D:\Users\johnh\Desktop\books2016\MICROSOFT_PRESS_EBOOK_INTRODUCING_WINDOWS_10_MOBI.MOBI'.
At D:\Users\johnh\Desktop\2016Eligman_DownloadAll_PowerShell.ps1:29 char:5
+     Invoke-WebRequest $book -OutFile $saveTo
+     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Invoke-WebRequest], DirectoryNotFoundException
    + FullyQualifiedErrorId : System.IO.DirectoryNotFoundException,Microsoft.PowerShell.Commands.InvokeWebRequestCommand

(only parts truncated are error output and the full list of URLS)

Looking at it now, it looks like the URL list might not be piped to the for each loop properly, but that is merely a guess as it is way over my head.

Neally · 2017-07-17T18:00:01.000Z

But does it download some of the books? It does for me.
Look at it verbose, it seems some books are not available:

foreach ($book in $books) { 
    $hdr = Invoke-WebRequest $book -Method Head -Verbose
    $title = $hdr.BaseResponse.ResponseUri.Segments[-1] 
    $title = [uri]::UnescapeDataString($title) 
    $saveTo = $dest + $title 
    Invoke-WebRequest $book -OutFile $saveTo -Verbose
}

Invoke-WebRequest : The remote server returned an error: (404) Not Found.

josephhays2 · 2017-07-17T18:08:41.000Z

It is not downloading any files for me. I am getting this as verbose output.

VERBOSE: GET http://ligman.me/29olaWI with 0-byte payload
VERBOSE: received 39132034-byte response of content type application/octet-stream

However, if I visit http://ligman.me/29olaWI via a web browser, the download starts.

gungnir · 2017-07-17T18:16:45.000Z

Does the path ‘D:\Users\johnh\Desktop\books2016’ exist? PowerShell usually doesn’t like implicitly creating directories. Testing the command with an non-existing destination directory produced the same error message for me.

Neally · 2017-07-17T18:21:59.000Z

$dest = "D:\Users\johnh\Desktop\books2016\" 
$bookList = Get-Content "D:\Users\johnh\Desktop\MSFTFreeEbooks.txt"
[string[]]$books = "" 
$books = $bookList.Split("`n") 
$books = $books[1..($books.Length -1)] 

 
# Download the books 
foreach ($book in $books) {
 
      Try{
          $test = Invoke-WebRequest $book -Method Head -ErrorAction Stop
          $hdr = $test
      }catch{
          write-warning "Unable to reach $book"
      }
      $title  = $hdr.BaseResponse.ResponseUri.Segments[-1]
      $title  = [uri]::UnescapeDataString($title)
      $saveTo = $dest + $title
      Invoke-WebRequest $book -OutFile $saveTo

} 
Write-Output "Downloaded $((get-childitem D:\Users\johnh\Desktop\books2016).count)/$($books.Count) books"

WARNING: Unable to reach http://ligman.me/29hV6ZM
WARNING: Unable to reach http://ligman.me/29d1qAV
WARNING: Unable to reach http://ligman.me/29pfpbI
Downloaded 5/8 books

http://ligman.me/29olaWI # remember, you excluded the first line
http://ligman.me/29ngkYn
http://ligman.me/29hV6ZM
http://ligman.me/29d1qAV
http://ligman.me/29pfpbI
http://ligman.me/29v8nwX
http://ligman.me/29uq452
http://ligman.me/29d1wc3

Neally · 2017-07-17T18:25:31.000Z

Gungnir:

Does the path ‘D:\Users\johnh\Desktop\books2016’ exist? PowerShell usually doesn’t like implicitly creating directories. Testing the command with an non-existing destination directory produced the same error message for me.

Good point…

if(-not(test-path "D:\Users\johnh\Desktop\books2016"){
    New-Item "D:\Users\johnh\Desktop\books2016" -ItemType Directory -Force
}

josephhays2 · 2017-07-17T18:35:12.000Z

I think I have found the problem (other than my lack of expert powershell skills and caffeine levels). Thanks to both of you. I did have the directory, BUT I failed to notice the difference in my script and the actual directory ( books2016 and books.2016). All is working now (except for whatever links are broken).

Thanks again. I’ll go put on my dunce hat until quitting time.