Batch files for automating data feeds

Today I had to come up with a system/process that could retrieve data from a web site after logging in, retrieve the data and then upload the data into a SQL database on a daily basis.

Instead of building a complex application that could do it all. I decided on writing a batch file that combined a custom command line download utility, an unzip utility and SQL BCP utility to download and update the data file.

This approach is quick and simple. Plus most routine data processing tasks require some amount of maintenance and updating from time to time. Using a batch file makes it relatively easy to access the process and manage any changes in the future.

However the interesting portion of the whole exercise was developing the custom download tool that could login to a website and download the required file/s. Its pretty simple actually what you need to do is simulate a form post using the Http Request class and stream the response to a file on your local server/machine.

While this is pretty simple it got me thinking about how you can start interfacing different web applications using this method. I'm sure most applications expose some kind of web service to do this but for those that don't and require some kind of authentication this approach opens up a host of possibilities.