필요성

Jsoup을 이용해서 웹페이지의 정보를 가져오고 싶은데

Javascript가 실행되지 않아서 가져오지 못하는 정보를

headless browser를 통해서 javascript가 실행됨으로써 더 완벽하게 웹페이지의 정보를 가져올 수 도 있고

버튼 클릭도 할 수 있고 로그인도 할 수 있다 등등

Here are a list of headless browsers that I know about:

• HtmlUnit - Java. Custom browser engine. Limited JavaScript support/DOM emulated. Open source.
• Ghost - Python only. WebKit-based. Full JavaScript support. Open source.
• Twill - Python/command line. Custom browser engine. No JavaScript. Open source.
• PhantomJS - Command line/all platforms. WebKit-based. Full JavaScript support. Open source.
• Awesomium - C++/.NET/all platforms. Chromium-based. Full JavaScript support. Commercial/free.
• SimpleBrowser - .NET 4/C#. Custom browser engine. No JavaScript support. Open source.
• ZombieJS - Node.js. Custom browser engine. JavaScript support/emulated DOM. Open source. Based on jsdom.
• EnvJS - JavaScript via Java/Rhino. Custom browser engine. JavaScript support/emulated DOM. Open source.
• Watir-webdriver with headless gem - Ruby via WebDriver. Full JS Support via Browsers (Firefox/Chrome/Safari/IE).
• Spynner - Python only. PyQT and WebKit.
• jsdom - Node.js. Custom browser engine. Supports JS via emulated DOM. Open source.
• TrifleJS - port of PhantomJS using MSIE (Trident) and V8. Open source.
• ui4j - Pure Java 8 solution. A wrapper library around the JavaFx WebKit Engine incl. headless modes.
• Chromium Embedded Framework - Full up-to-date embedded version of Chromium with off-screen rendering as needed. C/C++, with .NET wrappers (and other languages). As it is Chromium, it has support for everything. BSD licensed.
• Selenium WebDriver - Full support for JavaScript via browsers (Firefox, IE, Chrome, Safari, Opera). Officially supported bindings are C#, Java, JavaScript, Haskell, Perl, Ruby, PHP, Python, Objective-C, and R. Unofficial bindings are available for Qt and Go. Open source.

Headless browsers that have JavaScript support via an emulated DOM generally have issues with some sites that use more advanced/obscure browser features, or have functionality that has visual dependencies (e.g. via CSS positions and so forth), so whilst the pure JavaScript support in these browsers is generally complete, the actual supported browser functionality should be considered as partial only.

(Note: Original version of this post only mentioned HtmlUnit, hence the comments. If you know of other headless browser implementations and have edit rights, feel free to edit this post and add them.)

Selenium

https://github.com/SeleniumHQ/selenium

JAR files

https://github.com/SeleniumHQ/selenium/tree/master/third_party/java

zip files

https://selenium-release.storage.googleapis.com/index.html?path=3.1/

1. github

https://github.com/blog/2047-language-trends-on-github

1-1. githut

http://githut.info/

2. stack overflow

http://jkunst.com/r/what-do-we-ask-in-stackoverflow/#the-topics-this-year

http://stackoverflow.com/research/developer-survey-2016

3. Tiobe index

(대중적인 언어에 대한 인덱스이며 전문프로그래머 수 와 구글,야후,위키디피아 ,유튜브, 바이두 등을  이용하여 지수를 매기며 가장 좋은 언어라든지 가장 많이 코딩된 언어를 말하지 않습니다)

http://www.tiobe.com/tiobe-index/

기준 : http://www.tiobe.com/tiobe-index/programming-languages-definition/

4. redmonk

(깃허브와 스택오버플로우 기반으로 순위를 매깁니다)

http://redmonk.com/sogrady/category/programming-languages/

5. codeeval

http://blog.codeeval.com/codeevalblog/2016/2/2/most-popular-coding-languages-of-2016

6. IEEE Spectrum

Interactive: The Top Programming Languages 2016

http://spectrum.ieee.org/static/interactive-the-top-programming-languages-2016