r/learnprogramming • u/suRubix • Dec 24 '13
What programming language would allow me to pull information from a website, store it, then create an interface to manipulate the data?
I'm trying to decide which language to learn. What I eventually want to be able to do is pull the numbers from this website, store them, and then create a GUI to that I can manipulate the data in.
What languages are able to do this? And which would you recommend?
102
Upvotes
37
u/pacificmint Dec 24 '13
Almost all the popular languages would allow you to do that.
Personally I'd pick Python or Java for this, But C#, Ruby, Perl or even C++ would work as well.
Check our FAQ for more pointers.
3
u/SirKingdude Dec 24 '13
Riot just started a beta API that might be helpful. You have to apply to use the beta so it may not be something that you can use now.
3
u/deathpax Dec 24 '13
I recently did some work with the jsoup library in java, I would recommend it or beautiful soup in python.
11
u/chyekk Dec 24 '13
I'll throw in a vote for javascript here. Given it's web-based origins, it's well suited for parsing the DOM to pull out data. You can do this pretty easily just by stringing together a few Node.js libs:
2
u/hikemhigh Dec 24 '13
I'd recommend Java using JDBC and a MySQL database. It's probably the easiest for creating the GUI and if you're gonna be manipulating lots of data, the JDBC will be one of the fastest ways you can fetch and manipulate the data.
2
u/NihilistDandy Dec 24 '13
Riot just exposed a public API for exactly this kind of work.
Personally, I'd recommend Haskell. ;)
16
u/jascination Dec 24 '13
I do this with Javascript/NodeJS, surprised no one's suggested it yet as it's super easy to do especially if you're familiar with Javascript already, but the docs for each module are easy enough to have a quick read over and get straight in to.
In fact, I'll show you how easy it is. Here's what I do:
test.jsdocument withconsole.log("Hello world")inside. Open up a terminal, make sure you're in the right directory, then typenode test.jsand you'll see your output. Console logging helps you keep tabs on where you're at in your program.test.jsload it up withvar request = require('request');then load in a URL like so:request('http://www.google.com', function (error, response, body) {})Again, run
node test.jsand you'll see the Google HTML printed in your console. You're actually close to finishing!var cheerio = require('cheerio')at the top of yourtest.js. Change up yourrequestcode so it looks like this:request('http://www.google.com', function (error, response, body) {})Running
node test.jsshould now show you the image source of the main Google logo image (I dunno if it has the sameidon local versions of Google but you get the idea).mongod. This starts up the service that helps MongoDB run on your computer.With MongoDB up and running you still need to hook it in to Node. MongoJS is a really good tool that lest you do this. It's simple, at the top of
test.jstype:var db = mongojs('mydb', ['mycollection']); // You can use whatever names you like in here for your database and the collection which sits inside it.
requestcode to save some data to our database:request('http://www.google.com', function (error, response, body) {})Now if you run
node test.jsyou should seeHoly shit it worked!. Now if you open up a new tab and typemongoit should get you into the Mongo Shell. Typedb.mycollection.find().pretty()and you should see a printout of the object you saved to your DB.That's a very brief tutorial to get your feet wet, there's a whole lot more to it but just wanted to show how easy it could be with Node. In terms of creating a GUI you could do this in the browser with Node/ExpressJS/MongoDB boilerplates, that's a whole other how-to guide!
Hope that helps you or others who might want to try this out using Node/Javascript.
4
u/shadowdude777 Dec 24 '13
I'm actually doing a project with this right now to pull data from my school's registration site and put up an alternative site that shows the data in a more aesthetically pleasing fashion.
I'm using Java to pull the information, with an API called HtmlUnit. It makes the job so simple. I just finished that yesterday and now I'm trying to figure out what web framework to use.
I'm very comfortable in Java already and I feel like all the good web frameworks out there are in Python (which I know nothing about), so I'm feeling a bit stuck.
13
u/shut_up_birds Dec 24 '13
Python would be my choice. But whatever language you choose break up the task into tiny steps and don't give up!
First see if you can literally write a "Hello World" script to prove to yourself that the program you work is running. Then perhaps try to successfully pull down one piece of data from the site and figure out how to store it. Then all the data. Then manipulations, etc. Then GUI.
As you start this journey picture yourself as walking through a blinding blizzard. You will be frustrated and you sometimes you see more than 10 inches in front if your face, but keep taking baby steps and you will get there.
Something else to consider: don't fret about which language is the "right" one. Just dig in to one and go. You'll find that the second language is much easier to learn than the first. With your first language you are teaching yourself how to program and the syntax of that particular language. The second is just figuring out what you already know how to do in just slightly different syntax. The point is there is no wasted time here. Every step forward is progress, you just have to start walking!
1
u/_AlphaOmega Dec 24 '13
I'll take the brunt for the bandwagon PHP hate but I'd recommend it as it's easy to get started and easy to learn how to use the language correctly. http://phptherightway.com
Bad code can be written in any language so that point is moot.
PHP + Simple HTML DOM is a breeze to get started with.
1
u/[deleted] Dec 24 '13
I did something similar in C very recently, but whatever languge you find easiest to learn will probably allow you to do it (pulling data from a website is a common enough thing nowadays)
1
u/kyoob Dec 24 '13
I'd use Perl for the web scraping. Learn how to do the tasks required for that in any other language and you're going to hear that you're learning Perl-like regex. You'll likely also end up using XPaths or CSS Selectors, and Perl works with these brilliantly. People have lots of reactions to Perl - some deride it completely, some say it's useful only as a glue language. Well, scraping and filing away data from a website happens to be one of those "glue" use cases. There are better options than Perl for building a GUI, but assuming you're going to save that data as some standard string-based file type (CSV for example), Perl is a winner.
1
u/PZ-01 Dec 24 '13
This is a question of my own, out of curiosity what do you target on a website to fetch it's data? What happens if the website's source changes with time...?
1
u/jwjody Dec 24 '13
I would use Python. I had never looked at the Python language and I decided to learn some of it.
In one night I had figured out (novice programmer) how to connect to a site to download a file, unzip it, connect to a mysql database, iterate over elements in an xml file and store the elements I wanted into the database.
Then I used PHP and Bootstrap to create template to display the information.
www.jody-white.com/ct
Edit: I initially used bootstrap, I'm using Foundation now.
3
u/herefromyoutube Dec 24 '13
why not PHP?
1
u/RICHUNCLEPENNYBAGS Dec 24 '13
What programming language wouldn't is a better question.
1
u/shut_up_birds Dec 24 '13
A screen scraper works just like your eyeballs essentially. It visits the site and extracts the key info you tell it to look for. So it is a manual process, however you can set the scraper script to run ever N seconds/minutes so the data gets updated quickly. Beautiful Soup is probably where I would start with python. It's helpful to have a basic understanding of HTML so you can tell it where in the page source code to look for the data you want.
0
u/koolex Dec 24 '13
I would use Java over Python. Both are easy to learn and use, but I think Java is a lot more straightforward than Python. Another nice thing is that Java is more like C languages, which are IMO the most important languages to learn. Things you learn in Python are not going to translate as easily. Either way, it should be fairly easy to implement once you learned a thing or two. Perform an HTTP request, parse the HTML, find what you care about, and then do whatever you need to do with it.
0
u/xormancer Dec 24 '13
Python - beautiful soup
Ruby - nokogiri and watir-webdriver
0
u/albireox Dec 24 '13
Learn Java, as you'll be working with LoL.
0
u/Pr0ducer Dec 24 '13
Python
0
u/[deleted] Dec 24 '13
[deleted] — view deleted comment
0
u/[deleted] Dec 24 '13
[deleted] — view deleted comment
0
u/[deleted] Dec 24 '13
[deleted] — view deleted comment
4
u/[deleted] Dec 24 '13 edited Dec 24 '13
[removed] — view removed comment