Script or Program to scrape internal web site data

jeff79 · 2014-01-22T14:05:19.000Z

I’m looking for a free program or a script(s) to allow me to scrape internal web site data and then email it using smtp. I know how to send email via powershell but I’m having some issues trying to figure out how to grab specific web page data and package it up to be sent off. If anyone has an idea or a solution I would be very interested in learning how to do this.

The data is the copy count for a Ricoh C2551 Multi-function device. This data is accessible via the web interface of the device.

Thanks,

J

jeff79 · 2014-01-22T14:09:53.000Z

Actually in my continued search I found this link that might be something interesting. If anyone has used this functionality as well would be interesting.

leeholmes.com

Lee Holmes | HTML Agility Pack Rocks Your Screen Scraping World

I had to do some deep data extraction from a web page today, and naturally leaned on PowerShell for some assistance. PowerShell is a great language for text munging, and web content is no different. There are tons of examples online, but here’s an...

chrisseiter · 2014-01-22T14:13:14.000Z

Invoke-WebRequest may be able to get you in the right direction.

The Surly Admin – 21 Dec 13

Playing with Invoke-WebRequest – What is my IP Address?

Playing with Invoke-WebRequest and finding out What my Public IP address is.

By our resident PoSH hero, Martin9700.

Also, if it’s web accessible it may be SNMP accessible and you wouldn’t need the extra heavy lifting of parsing a web page first.

jeff79 · 2014-01-22T15:02:32.000Z

It is accessible through SNMP and since I use a managed print house due to these being leased machines there really should be some way they can get this. Anyhow, they call and ask for counts every month and it’s just a pain. I guess I would need to get the MIB information if I were to do it this way. This seems like an easier way to go then grabbing it off the web page.

martin9700 · 2014-01-22T15:30:45.000Z

Jeff79:

It is accessible through SNMP and since I use a managed print house due to these being leased machines there really should be some way they can get this. Anyhow, they call and ask for counts every month and it’s just a pain. I guess I would need to get the MIB information if I were to do it this way. This seems like an easier way to go then grabbing it off the web page.

It might be worth it to contact the printer vendor, they may have a program that can do this kind of thing. Maybe even get it in CSV format (then you can use PowerShell to tweak as you need).

andreygrehov8835 · 2014-01-23T02:00:48.000Z

We are currently building a service/API to scrape data from any web site. Subscribe for beta testing: http://textract.me/

zabolyx · 2014-03-21T18:01:15.000Z

There is also iMacros that could be used for data gathering… If you use it to scrape you can have it create a CSV file of the data it gathers. I use it to automate some functions in the company so I can be lazy… I mean efficient.

We lease 7 Ricohs at the moment… I really should build something for that as we have 3 locations and each has to gather data on the machines. I could just run the script (or give it to someone to run) and get all of them in one file.

I’ve already built a spreadsheet that helps setup extensions, inbound routes in bulk for Trix/FreePBX (never seem to get the bulk add function to work) and it also programs the phones (same issues with the EPM).